Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/26/21 4:38 PM, Christoph Paasch wrote:
> Hi Bob,
> 
>> On Oct 26, 2021, at 4:23 PM, Bob McMahon <bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com>> wrote:
>> I'm confused. I don't see any blocking nor partial writes per the write() at the app level with TCP_NOTSENT_LOWAT set at 4 bytes. The burst is 40K, the write size is 4K and the watermark is 4 bytes. There are ten writes per burst.
> 
> You are on Linux here, right?
> 
> AFAICS, Linux will still accept whatever fits in an skb. And that is likely more than 4K (with GSO on by default).

This (max payload per skb) can be tuned at the driver level, at least for experimental purposes or dedicated devices.

ip link set dev eth0 gso_max_size 8000

To fetch current values :

ip -d link sh dev eth0


> 
> However, do you go back to select() after each write() or do you loop over the write() calls?
> 
> 
> Christoph
> 
>> The S8 histograms are the times waiting on the select().  The first value is the bin number (multiplied by 100usec bin width) and second the bin count. The worst case time is at the end and is timestamped per unix epoch.
>>
>> The second run is over a controlled WiFi link where a 99.7% point of 4-8ms for a WiFi TX op arbitration win is in the ballpark. The first is 1G wired and is in the 600 usec range. (No media arbitration there.)
>>
>>  [root@localhost iperf2-code]# src/iperf -c 10.19.87.9 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>> ------------------------------------------------------------
>> Client connecting to 10.19.87.9, TCP port 5001 with pid 2124 (1 flows)
>> Write buffer size: 4096 Byte
>> Bursting: 40.0 KByte every 1.00 seconds
>> TCP window size: 85.0 KByte (default)
>> Event based writes (pending queue watermark at 4 bytes)
>> Enabled select histograms bin-width=0.100 ms, bins=10000
>> ------------------------------------------------------------
>> [  1] local 10.19.87.10%eth0 port 33166 connected with 10.19.87.9 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=0.54 ms) on 2021-10-26 16:07:33 (PDT)
>> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/5368 us  8
>> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,2:5,3:2,4:1,11:1 (5.00/95.00/99.7%=1/11/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/569 us  72
>> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:1,3:4,4:1,7:1,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.736 ms/1635289654.928088)
>> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/312 us  131
>> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.548 ms/1635289655.927776)
>> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/302 us  136
>> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:2,3:5,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.584 ms/1635289656.927814)
>> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/316 us  130
>> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:2,4:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.572 ms/1635289657.927810)
>> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/253 us  162
>> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:4,5:1 (5.00/95.00/99.7%=1/5/5,Outliers=0,obl/obu=0/0) (0.417 ms/1635289658.927630)
>> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/290 us  141
>> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:3,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.573 ms/1635289659.927771)
>> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/359 us  114
>> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,3:4,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.570 ms/1635289660.927753)
>> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/349 us  117
>> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:5,4:1,7:1 (5.00/95.00/99.7%=1/7/7,Outliers=0,obl/obu=0/0) (0.608 ms/1635289661.927843)
>> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/347 us  118
>> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:1,3:5,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.725 ms/1635289662.927861)
>> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/1519 us  27
>> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:25,2:13,3:36,4:11,5:5,6:5,7:2,8:2,11:1 (5.00/95.00/99.7%=1/7/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>>
>> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>> ------------------------------------------------------------
>> Client connecting to 192.168.1.1, TCP port 5001 with pid 2131 (1 flows)
>> Write buffer size: 4096 Byte
>> Bursting: 40.0 KByte every 1.00 seconds
>> TCP window size: 85.0 KByte (default)
>> Event based writes (pending queue watermark at 4 bytes)
>> Enabled select histograms bin-width=0.100 ms, bins=10000
>> ------------------------------------------------------------
>> [  1] local 192.168.1.4%eth1 port 45518 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.48 ms) on 2021-10-26 16:07:56 (PDT)
>> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10339 us  4
>> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,40:1,47:1,49:2,50:3,51:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.990 ms/1635289676.802143)
>> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4853 us  8
>> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,39:1,44:1,45:1,49:1,51:1,52:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.937 ms/1635289677.802274)
>> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4991 us  8
>> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:2,50:2,51:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.307 ms/1635289678.794326)
>> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4610 us  9
>> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:3,56:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.362 ms/1635289679.794335)
>> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5028 us  8
>> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.367 ms/1635289680.794399)
>> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5113 us  8
>> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:2,58:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.442 ms/1635289681.794392)
>> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5054 us  8
>> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,51:1,60:2,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.374 ms/1635289682.794335)
>> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5138 us  8
>> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:2,40:1,49:2,50:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.396 ms/1635289683.794338)
>> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,45:2,49:1,50:3,63:1 (5.00/95.00/99.7%=1/63/63,Outliers=0,obl/obu=0/0) (6.292 ms/1635289684.794262)
>> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:3,84:1 (5.00/95.00/99.7%=1/84/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/6331 us  6
>> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,38:2,39:5,40:2,44:1,45:3,47:1,48:1,49:26,50:17,51:4,52:1,56:1,58:1,59:1,60:7,63:1,64:5,65:1,84:1 (5.00/95.00/99.7%=1/64/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>>
>> Bob
>>
>> On Tue, Oct 26, 2021 at 11:45 AM Christoph Paasch <cpaasch@apple.com <mailto:cpaasch@apple.com>> wrote:
>>
>>     Hello,
>>
>>     > On Oct 25, 2021, at 9:24 PM, Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>> wrote:
>>     >
>>     >
>>     >
>>     > On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
>>     >> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net>> wrote:
>>     >>
>>     >>> Hi All,
>>     >>>
>>     >>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>>     >>>
>>     >>> Does this seem reasonable?
>>     >>
>>     >> I’m not 100% sure what you’re asking, but I will try to help.
>>     >>
>>     >> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.
>>     >
>>     >
>>     > When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
>>     > stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
>>     > was above the threshold.
>>     >
>>     > So it looks like Apple implementation is different, based on your description ?
>>
>>     Yes, TCP_NOTSENT_LOWAT only impacts the wakeup on iOS/macOS/...
>>
>>     An app can still fill the send-buffer if it does a sendmsg() with a large buffer or does repeated calls to sendmsg().
>>
>>     Fur Apple, the goal of TCP_NOTSENT_LOWAT was to allow an app to quickly change the data it "scheduled" to send. And thus allow the app to write the smallest "logical unit" it has. If that unit is 512KB large, the app is allowed to send that.
>>     For example, in case of video-streaming one may want to skip ahead in the video. In that case the app still needs to transmit the remaining parts of the previous frame anyways, before it can send the new video frame.
>>     That's the reason why the Apple implementation allows one to write more than just the lowat threshold.
>>
>>
>>     That being said, I do think that Linux's way allows for an easier API because the app does not need to be careful at how much data it sends after an epoll/kqueue wakeup. So, the latency-benefits will be easier to get.
>>
>>
>>     Christoph
>>
>>
>>
>>     > [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36>
>>     >
>>     > netperf does not use epoll(), but rather a loop over sendmsg().
>>     >
>>     > One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
>>     > max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
>>     > by 10x, allowing for autotune to increase BDP for big RTT flows, this without
>>     > increasing memory needs for flows with small RTT.
>>     >
>>     > In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
>>     >>
>>     >> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
>>     >>
>>     >> How to use TCP_NOTSENT_LOWAT is explained in this video:
>>     >>
>>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>>
>>     >>
>>     >> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
>>     >>
>>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>>
>>     >>
>>     >> Stuart Cheshire
>>     >> _______________________________________________
>>     >> Bloat mailing list
>>     >> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>>     >> https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>
>>     >>
>>     > _______________________________________________
>>     > Bloat mailing list
>>     > Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
>>     > https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>
>>
>>
>> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/26/21 8:45 PM, Bob McMahon wrote:
> This is linux. The code flow is burst writes until the burst size, take a timestamp, call select(), take second timestamp and insert time delta into histogram, await clock_nanosleep() to schedule the next burst. (actually, the deltas, inserts into the histogram and user i/o are done in another thread, i.e. iperf 2's reporter thread.)
> 
> I still must be missing something.  Does anything else need to be set to reduce the skb size? Everything seems to be indicating 4K writes even when gso_max_size is 2000 (I assume these are units of bytes?) There are ten writes, ten reads and ten  RTTs for the bursts.  I don't see partial writes at the app level. 
> 
>     [root@localhost iperf2-code]# ip link set dev eth1 gso_max_size 2000

You could check with tcpdump on eth1, that outgoing packets are no longer 'TSO/GSO', but single MSS ones.

(Note: this device gso_max_size is only taken into account for flows established after the change)

> 
>     [root@localhost iperf2-code]# ip -d link sh dev eth1
>     9: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
>         link/ether 00:90:4c:40:04:59 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 1500 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 2000 gso_max_segs 65535
>     [root@localhost iperf2-code]# uname -r
>     5.0.9-301.fc30.x86_64
> 
> 
> It looks like RTT is being driven by WiFi TXOPs as doubling the write size increases the aggregation by two but has no significant effect on the RTTs.
> 
>     4K writes: tot_mpdus 328 tot_ampdus 209 mpduperampdu 2
> 
> 
>     8k writes:  tot_mpdus 317 tot_ampdus 107 mpduperampdu 3
> 
> 
> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1%eth1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001 with pid 5145 via eth1 (1 flows)
> Write buffer size: 4096 Byte
> Bursting: 40.0 KByte every 1.00 seconds
> TCP window size: 85.0 KByte (default)
> Event based writes (pending queue watermark at 4 bytes)
> Enabled select histograms bin-width=0.100 ms, bins=10000
> ------------------------------------------------------------
> [  1] local 192.168.1.4%eth1 port 45680 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.30 ms) on 2021-10-26 20:25:29 (PDT)
> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10091 us  4
> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,36:1,40:1,44:1,46:1,48:1,49:1,50:2,52:1 (5.00/95.00/99.7%=1/52/52,Outliers=0,obl/obu=0/0) (5.121 ms/1635305129.152339)
> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4990 us  8
> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,45:1,49:5,50:1 (5.00/95.00/99.7%=1/50/50,Outliers=0,obl/obu=0/0) (4.991 ms/1635305130.153330)
> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4904 us  8
> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,29:1,49:4,50:1,59:1,75:1 (5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.455 ms/1635305131.147353)
> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4964 us  8
> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:4,50:2,59:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.460 ms/1635305132.146338)
> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4970 us  8
> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.404 ms/1635305133.146335)
> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4986 us  8
> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:1,50:4,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.395 ms/1635305134.146343)
> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5059 us  8
> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:2,60:1,85:1 (5.00/95.00/99.7%=1/85/85,Outliers=0,obl/obu=0/0) (8.417 ms/1635305135.148343)
> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5407 us  8
> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,40:1,49:4,50:1,59:1,75:1 (5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.428 ms/1635305136.147343)
> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5188 us  8
> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,40:1,49:3,50:3,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.388 ms/1635305137.146284)
> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5306 us  8
> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:2,50:2,51:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.422 ms/1635305138.146316)
> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/5939 us  7
> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,29:1,36:1,39:3,40:3,44:1,45:1,46:1,48:2,49:33,50:18,51:1,52:1,59:5,60:2,64:2,65:3,75:2,85:1 (5.00/95.00/99.7%=1/65/85,Outliers=0,obl/obu=0/0) (8.417 ms/1635305135.148343)
> 
> [root@localhost iperf2-code]# src/iperf -s -i 1 -e -B 192.168.1.1%eth1
> ------------------------------------------------------------
> Server listening on TCP port 5001 with pid 6287
> Binding to local address 192.168.1.1 and iface eth1
> Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
> TCP window size:  128 KByte (default)
> ------------------------------------------------------------
> [  1] local 192.168.1.1%eth1 port 5001 connected with 192.168.1.4 port 45680 (MSS=1448) (burst-period=1.0000s) (trip-times) (sock=4) (peer 2.1.4-master) on 2021-10-26 20:25:29 (PDT)
> [ ID] Burst (start-end)  Transfer     Bandwidth       XferTime  (DC%)     Reads=Dist          NetPwr
> [  1] 0.0001-0.0500 sec  40.1 KBytes  6.59 Mbits/sec  49.848 ms (5%)    12=12:0:0:0:0:0:0:0  0
> [  1] 1.0002-1.0461 sec  40.0 KBytes  7.14 Mbits/sec  45.913 ms (4.6%)    10=10:0:0:0:0:0:0:0  0
> [  1] 2.0002-2.0491 sec  40.0 KBytes  6.70 Mbits/sec  48.876 ms (4.9%)    11=11:0:0:0:0:0:0:0  0
> [  1] 3.0002-3.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.886 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 4.0002-4.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.887 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 5.0002-5.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.881 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 6.0002-6.0511 sec  40.0 KBytes  6.44 Mbits/sec  50.895 ms (5.1%)    10=10:0:0:0:0:0:0:0  0
> [  1] 7.0002-7.0501 sec  40.0 KBytes  6.57 Mbits/sec  49.889 ms (5%)    10=10:0:0:0:0:0:0:0  0
> [  1] 8.0002-8.0481 sec  40.0 KBytes  6.84 Mbits/sec  47.901 ms (4.8%)    11=11:0:0:0:0:0:0:0  0
> [  1] 9.0002-9.0491 sec  40.0 KBytes  6.70 Mbits/sec  48.872 ms (4.9%)    10=10:0:0:0:0:0:0:0  0
> [  1] 0.0000-10.0031 sec   400 KBytes   328 Kbits/sec               104=104:0:0:0:0:0:0:0
> 
> Bob
> 
> On Tue, Oct 26, 2021 at 6:12 PM Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>> wrote:
> 
> 
> 
>     On 10/26/21 4:38 PM, Christoph Paasch wrote:
>     > Hi Bob,
>     >
>     >> On Oct 26, 2021, at 4:23 PM, Bob McMahon <bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com> <mailto:bob.mcmahon@broadcom.com <mailto:bob.mcmahon@broadcom.com>>> wrote:
>     >> I'm confused. I don't see any blocking nor partial writes per the write() at the app level with TCP_NOTSENT_LOWAT set at 4 bytes. The burst is 40K, the write size is 4K and the watermark is 4 bytes. There are ten writes per burst.
>     >
>     > You are on Linux here, right?
>     >
>     > AFAICS, Linux will still accept whatever fits in an skb. And that is likely more than 4K (with GSO on by default).
> 
>     This (max payload per skb) can be tuned at the driver level, at least for experimental purposes or dedicated devices.
> 
>     ip link set dev eth0 gso_max_size 8000
> 
>     To fetch current values :
> 
>     ip -d link sh dev eth0
> 
> 
>     >
>     > However, do you go back to select() after each write() or do you loop over the write() calls?
>     >
>     >
>     > Christoph
>     >
>     >> The S8 histograms are the times waiting on the select().  The first value is the bin number (multiplied by 100usec bin width) and second the bin count. The worst case time is at the end and is timestamped per unix epoch.
>     >>
>     >> The second run is over a controlled WiFi link where a 99.7% point of 4-8ms for a WiFi TX op arbitration win is in the ballpark. The first is 1G wired and is in the 600 usec range. (No media arbitration there.)
>     >>
>     >>  [root@localhost iperf2-code]# src/iperf -c 10.19.87.9 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>     >> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>     >> ------------------------------------------------------------
>     >> Client connecting to 10.19.87.9, TCP port 5001 with pid 2124 (1 flows)
>     >> Write buffer size: 4096 Byte
>     >> Bursting: 40.0 KByte every 1.00 seconds
>     >> TCP window size: 85.0 KByte (default)
>     >> Event based writes (pending queue watermark at 4 bytes)
>     >> Enabled select histograms bin-width=0.100 ms, bins=10000
>     >> ------------------------------------------------------------
>     >> [  1] local 10.19.87.10%eth0 port 33166 connected with 10.19.87.9 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=0.54 ms) on 2021-10-26 16:07:33 (PDT)
>     >> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>     >> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/5368 us  8
>     >> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,2:5,3:2,4:1,11:1 (5.00/95.00/99.7%=1/11/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>     >> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/569 us  72
>     >> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:1,3:4,4:1,7:1,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.736 ms/1635289654.928088)
>     >> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/312 us  131
>     >> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.548 ms/1635289655.927776)
>     >> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/302 us  136
>     >> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:2,3:5,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.584 ms/1635289656.927814)
>     >> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/316 us  130
>     >> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:2,4:2,5:2,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.572 ms/1635289657.927810)
>     >> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/253 us  162
>     >> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:4,5:1 (5.00/95.00/99.7%=1/5/5,Outliers=0,obl/obu=0/0) (0.417 ms/1635289658.927630)
>     >> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/290 us  141
>     >> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:3,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.573 ms/1635289659.927771)
>     >> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/359 us  114
>     >> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,3:4,4:3,6:1 (5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.570 ms/1635289660.927753)
>     >> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/349 us  117
>     >> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:5,4:1,7:1 (5.00/95.00/99.7%=1/7/7,Outliers=0,obl/obu=0/0) (0.608 ms/1635289661.927843)
>     >> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/347 us  118
>     >> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:1,3:5,8:1 (5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.725 ms/1635289662.927861)
>     >> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/1519 us  27
>     >> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:25,2:13,3:36,4:11,5:5,6:5,7:2,8:2,11:1 (5.00/95.00/99.7%=1/7/11,Outliers=0,obl/obu=0/0) (1.089 ms/1635289653.928360)
>     >>
>     >> [root@localhost iperf2-code]# src/iperf -c 192.168.1.1 --trip-times -i 1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
>     >> WARN: option of --burst-size without --burst-period defaults --burst-period to 1 second
>     >> ------------------------------------------------------------
>     >> Client connecting to 192.168.1.1, TCP port 5001 with pid 2131 (1 flows)
>     >> Write buffer size: 4096 Byte
>     >> Bursting: 40.0 KByte every 1.00 seconds
>     >> TCP window size: 85.0 KByte (default)
>     >> Event based writes (pending queue watermark at 4 bytes)
>     >> Enabled select histograms bin-width=0.100 ms, bins=10000
>     >> ------------------------------------------------------------
>     >> [  1] local 192.168.1.4%eth1 port 45518 connected with 192.168.1.1 port 5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.48 ms) on 2021-10-26 16:07:56 (PDT)
>     >> [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
>     >> [  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0          0       14K/10339 us  4
>     >> [  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,40:1,47:1,49:2,50:3,51:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.990 ms/1635289676.802143)
>     >> [  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4853 us  8
>     >> [  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,39:1,44:1,45:1,49:1,51:1,52:1,60:1 (5.00/95.00/99.7%=1/60/60,Outliers=0,obl/obu=0/0) (5.937 ms/1635289677.802274)
>     >> [  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4991 us  8
>     >> [  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,48:1,49:2,50:2,51:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.307 ms/1635289678.794326)
>     >> [  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/4610 us  9
>     >> [  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:3,56:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.362 ms/1635289679.794335)
>     >> [  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5028 us  8
>     >> [  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.367 ms/1635289680.794399)
>     >> [  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5113 us  8
>     >> [  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:3,50:2,58:1,60:1,65:1 (5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.442 ms/1635289681.794392)
>     >> [  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5054 us  8
>     >> [  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,51:1,60:2,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.374 ms/1635289682.794335)
>     >> [  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5138 us  8
>     >> [  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:2,40:1,49:2,50:1,60:1,64:1 (5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.396 ms/1635289683.794338)
>     >> [  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>     >> [  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,38:1,45:2,49:1,50:3,63:1 (5.00/95.00/99.7%=1/63/63,Outliers=0,obl/obu=0/0) (6.292 ms/1635289684.794262)
>     >> [  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0          0       14K/5329 us  8
>     >> [  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,49:3,50:3,84:1 (5.00/95.00/99.7%=1/84/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>     >> [  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0          0       14K/6331 us  6
>     >> [  1] 0.00-10.01 sec S8(f)-PDF: bin(w=100us):cnt(100)=1:19,38:2,39:5,40:2,44:1,45:3,47:1,48:1,49:26,50:17,51:4,52:1,56:1,58:1,59:1,60:7,63:1,64:5,65:1,84:1 (5.00/95.00/99.7%=1/64/84,Outliers=0,obl/obu=0/0) (8.306 ms/1635289685.796315)
>     >>
>     >> Bob
>     >>
>     >> On Tue, Oct 26, 2021 at 11:45 AM Christoph Paasch <cpaasch@apple.com <mailto:cpaasch@apple.com> <mailto:cpaasch@apple.com <mailto:cpaasch@apple.com>>> wrote:
>     >>
>     >>     Hello,
>     >>
>     >>     > On Oct 25, 2021, at 9:24 PM, Eric Dumazet <eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com> <mailto:eric.dumazet@gmail.com <mailto:eric.dumazet@gmail.com>>> wrote:
>     >>     >
>     >>     >
>     >>     >
>     >>     > On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
>     >>     >> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net> <mailto:make-wifi-fast@lists.bufferbloat.net <mailto:make-wifi-fast@lists.bufferbloat.net>>> wrote:
>     >>     >>
>     >>     >>> Hi All,
>     >>     >>>
>     >>     >>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>     >>     >>>
>     >>     >>> Does this seem reasonable?
>     >>     >>
>     >>     >> I’m not 100% sure what you’re asking, but I will try to help.
>     >>     >>
>     >>     >> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.
>     >>     >
>     >>     >
>     >>     > When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
>     >>     > stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
>     >>     > was above the threshold.
>     >>     >
>     >>     > So it looks like Apple implementation is different, based on your description ?
>     >>
>     >>     Yes, TCP_NOTSENT_LOWAT only impacts the wakeup on iOS/macOS/...
>     >>
>     >>     An app can still fill the send-buffer if it does a sendmsg() with a large buffer or does repeated calls to sendmsg().
>     >>
>     >>     Fur Apple, the goal of TCP_NOTSENT_LOWAT was to allow an app to quickly change the data it "scheduled" to send. And thus allow the app to write the smallest "logical unit" it has. If that unit is 512KB large, the app is allowed to send that.
>     >>     For example, in case of video-streaming one may want to skip ahead in the video. In that case the app still needs to transmit the remaining parts of the previous frame anyways, before it can send the new video frame.
>     >>     That's the reason why the Apple implementation allows one to write more than just the lowat threshold.
>     >>
>     >>
>     >>     That being said, I do think that Linux's way allows for an easier API because the app does not need to be careful at how much data it sends after an epoll/kqueue wakeup. So, the latency-benefits will be easier to get.
>     >>
>     >>
>     >>     Christoph
>     >>
>     >>
>     >>
>     >>     > [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36> <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36 <https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36>>
>     >>     >
>     >>     > netperf does not use epoll(), but rather a loop over sendmsg().
>     >>     >
>     >>     > One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
>     >>     > max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
>     >>     > by 10x, allowing for autotune to increase BDP for big RTT flows, this without
>     >>     > increasing memory needs for flows with small RTT.
>     >>     >
>     >>     > In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
>     >>     >>
>     >>     >> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
>     >>     >>
>     >>     >> How to use TCP_NOTSENT_LOWAT is explained in this video:
>     >>     >>
>     >>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>>>
>     >>     >>
>     >>     >> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
>     >>     >>
>     >>     >> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520 <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>>>
>     >>     >>
>     >>     >> Stuart Cheshire
>     >>     >> _______________________________________________
>     >>     >> Bloat mailing list
>     >>     >> Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net> <mailto:Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>>
>     >>     >> https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat> <https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>>
>     >>     >>
>     >>     > _______________________________________________
>     >>     > Bloat mailing list
>     >>     > Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net> <mailto:Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>>
>     >>     > https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat> <https://lists.bufferbloat.net/listinfo/bloat <https://lists.bufferbloat.net/listinfo/bloat>>
>     >>
>     >>
>     >> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[Codel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Dave Taht]

The program committee members are *amazing*. Perhaps, finally, we can
move the bar for the internet's quality metrics past endless, blind
repetitions of speedtest.

For complete details, please see:
https://www.iab.org/activities/workshops/network-quality/

Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
Invitations Issued by: Monday 16th August 2021

Workshop Date: This will be a virtual workshop, spread over three days:

1400-1800 UTC Tue 14th September 2021
1400-1800 UTC Wed 15th September 2021
1400-1800 UTC Thu 16th September 2021

Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira

The Program Committee members:

Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.

Send Submissions to: network-quality-workshop-pc@iab.org.

Position papers from academia, industry, the open source community and
others that focus on measurements, experiences, observations and
advice for the future are welcome. Papers that reflect experience
based on deployed services are especially welcome. The organizers
understand that specific actions taken by operators are unlikely to be
discussed in detail, so papers discussing general categories of
actions and issues without naming specific technologies, products, or
other players in the ecosystem are expected. Papers should not focus
on specific protocol solutions.

The workshop will be by invitation only. Those wishing to attend
should submit a position paper to the address above; it may take the
form of an Internet-Draft.

All inputs submitted and considered relevant will be published on the
workshop website. The organisers will decide whom to invite based on
the submissions received. Sessions will be organized according to
content, and not every accepted submission or invited attendee will
have an opportunity to present as the intent is to foster discussion
and not simply to have a sequence of presentations.

Position papers from those not planning to attend the virtual sessions
themselves are also encouraged. A workshop report will be published
afterwards.

Overview:

"We believe that one of the major factors behind this lack of progress
is the popular perception that throughput is the often sole measure of
the quality of Internet connectivity. With such narrow focus, people
don’t consider questions such as:

What is the latency under typical working conditions?
How reliable is the connectivity across longer time periods?
Does the network allow the use of a broad range of protocols?
What services can be run by clients of the network?
What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
What security mechanisms are available for local services, such as DNS?
To what degree are the privacy, confidentiality, integrity and
authenticity of user communications guarded?

Improving these aspects of network quality will likely depend on
measurement and exposing metrics to all involved parties, including to
end users in a meaningful way. Such measurements and exposure of the
right metrics will allow service providers and network operators to
focus on the aspects that impacts the users’ experience most and at
the same time empowers users to choose the Internet service that will
give them the best experience."


-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

Re: [Codel] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Dave Taht]

In terms of trying to find "Quality" I have tried to encourage folk to
both read "zen and the art of motorcycle maintenance"[0], and Deming's
work on "total quality management".

My own slice at this network, computer and lifestyle "issue" is aiming
for "imperceptible latency" in all things. [1]. There's a lot of
fallout from that in terms of not just addressing queuing delay, but
caching, prefetching, and learning more about what a user really needs
(as opposed to wants) to know via intelligent agents.

[0] If you want to get depressed, read Pirsig's successor to "zen...",
lila, which is in part about what happens when an engineer hits an
insoluble problem.
[1] https://www.internetsociety.org/events/latency2013/



On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com> wrote:
>
> Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling rate at the physical layer or throughput of FTP transfers.
>
>
>
> But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the hardware or the routers or even the routing algorithms *to its users*.
>
>
>
> By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will likely fall into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be what users actually will do given the chance.
>
>
>
> I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to optimize ASR33 logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing systems, and "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer, because someone thought that TV broadcast would be the dominant application.
>
>
>
> Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in system design". Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users at all times, now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might expect to experience that they don't experience.
>
>
>
> There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as quality, so applications could avoid having to solve such issues.
>
>
>
> What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different uses to *cooperate* in some way.
>
>
>
> Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>
> (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but never, ever explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation of the so-called end-to-end principle).
>
>
>
> Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing. Quality can be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end applications to affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the other end-to-end users sharing the infrastructure.
>
>
>
> This conference won't talk about it this way. So don't waste your time.
>
>
>
>
>
>
>
> On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com> said:
>
> > The program committee members are *amazing*. Perhaps, finally, we can
> > move the bar for the internet's quality metrics past endless, blind
> > repetitions of speedtest.
> >
> > For complete details, please see:
> > https://www.iab.org/activities/workshops/network-quality/
> >
> > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
> > Invitations Issued by: Monday 16th August 2021
> >
> > Workshop Date: This will be a virtual workshop, spread over three days:
> >
> > 1400-1800 UTC Tue 14th September 2021
> > 1400-1800 UTC Wed 15th September 2021
> > 1400-1800 UTC Thu 16th September 2021
> >
> > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> >
> > The Program Committee members:
> >
> > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
> > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
> > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
> > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
> > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
> >
> > Send Submissions to: network-quality-workshop-pc@iab.org.
> >
> > Position papers from academia, industry, the open source community and
> > others that focus on measurements, experiences, observations and
> > advice for the future are welcome. Papers that reflect experience
> > based on deployed services are especially welcome. The organizers
> > understand that specific actions taken by operators are unlikely to be
> > discussed in detail, so papers discussing general categories of
> > actions and issues without naming specific technologies, products, or
> > other players in the ecosystem are expected. Papers should not focus
> > on specific protocol solutions.
> >
> > The workshop will be by invitation only. Those wishing to attend
> > should submit a position paper to the address above; it may take the
> > form of an Internet-Draft.
> >
> > All inputs submitted and considered relevant will be published on the
> > workshop website. The organisers will decide whom to invite based on
> > the submissions received. Sessions will be organized according to
> > content, and not every accepted submission or invited attendee will
> > have an opportunity to present as the intent is to foster discussion
> > and not simply to have a sequence of presentations.
> >
> > Position papers from those not planning to attend the virtual sessions
> > themselves are also encouraged. A workshop report will be published
> > afterwards.
> >
> > Overview:
> >
> > "We believe that one of the major factors behind this lack of progress
> > is the popular perception that throughput is the often sole measure of
> > the quality of Internet connectivity. With such narrow focus, people
> > don’t consider questions such as:
> >
> > What is the latency under typical working conditions?
> > How reliable is the connectivity across longer time periods?
> > Does the network allow the use of a broad range of protocols?
> > What services can be run by clients of the network?
> > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
> > What security mechanisms are available for local services, such as DNS?
> > To what degree are the privacy, confidentiality, integrity and
> > authenticity of user communications guarded?
> >
> > Improving these aspects of network quality will likely depend on
> > measurement and exposing metrics to all involved parties, including to
> > end users in a meaningful way. Such measurements and exposure of the
> > right metrics will allow service providers and network operators to
> > focus on the aspects that impacts the users’ experience most and at
> > the same time empowers users to choose the Internet service that will
> > give them the best experience."
> >
> >
> > --
> > Latest Podcast:
> > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> >
> > Dave Täht CTO, TekLibre, LLC
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >



--
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

Re: [Codel] [Starlink] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Ben Greear]

Hello,

I am interested to hear wish lists for network testing features.  We make test equipment, supporting lots
of wifi stations and a distributed architecture, with built-in udp, tcp, ipv6, http, ... protocols,
and open to creating/improving some of our automated tests.

I know Dave has some test scripts already, so I'm not necessarily looking to reimplement that,
but more fishing for other/new ideas.

Thanks,
Ben

On 7/2/21 4:28 PM, Bob McMahon wrote:
> I think we need the language of math here. It seems like the network power metric, introduced by Kleinrock and Jaffe in the late 70s, is something useful. 
> Effective end/end queue depths per Little's law also seems useful. Both are available in iperf 2 from a test perspective. Repurposing test techniques to actual 
> traffic could be useful. Hence the question around what exact telemetry is useful to apps making socket write() and read() calls.
> 
> Bob
> 
> On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> wrote:
> 
>     In terms of trying to find "Quality" I have tried to encourage folk to
>     both read "zen and the art of motorcycle maintenance"[0], and Deming's
>     work on "total quality management".
> 
>     My own slice at this network, computer and lifestyle "issue" is aiming
>     for "imperceptible latency" in all things. [1]. There's a lot of
>     fallout from that in terms of not just addressing queuing delay, but
>     caching, prefetching, and learning more about what a user really needs
>     (as opposed to wants) to know via intelligent agents.
> 
>     [0] If you want to get depressed, read Pirsig's successor to "zen...",
>     lila, which is in part about what happens when an engineer hits an
>     insoluble problem.
>     [1] https://www.internetsociety.org/events/latency2013/ <https://www.internetsociety.org/events/latency2013/>
> 
> 
> 
>     On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com <mailto:dpreed@deepplum.com>> wrote:
>      >
>      > Well, nice that the folks doing the conference  are willing to consider that quality of user experience has little to do with signalling rate at the
>     physical layer or throughput of FTP transfers.
>      >
>      >
>      >
>      > But honestly, the fact that they call the problem "network quality" suggests that they REALLY, REALLY don't understand the Internet isn't the hardware or
>     the routers or even the routing algorithms *to its users*.
>      >
>      >
>      >
>      > By ignoring the diversity of applications now and in the future, and the fact that we DON'T KNOW what will be coming up, this conference will likely fall
>     into the usual trap that net-heads fall into - optimizing for some imaginary reality that doesn't exist, and in fact will probably never be what users
>     actually will do given the chance.
>      >
>      >
>      >
>      > I saw this issue in 1976 in the group developing the original Internet protocols - a desire to put *into the network* special tricks to optimize ASR33
>     logins to remote computers from terminal concentrators (aka remote login), bulk file transfers between file systems on different time-sharing systems, and
>     "sessions" (virtual circuits) that required logins. And then trying to exploit underlying "multicast" by building it into the IP layer, because someone
>     thought that TV broadcast would be the dominant application.
>      >
>      >
>      >
>      > Frankly, to think of "quality" as something that can be "provided" by "the network" misses the entire point of "end-to-end argument in system design".
>     Quality is not a property defined or created by The Network. If you want to talk about Quality, you need to talk about users - all the users at all times,
>     now and into the future, and that's something you can't do if you don't bother to include current and future users talking about what they might expect to
>     experience that they don't experience.
>      >
>      >
>      >
>      > There was much fighting back in 1976 that basically involved "network experts" saying that the network was the place to "solve" such issues as quality,
>     so applications could avoid having to solve such issues.
>      >
>      >
>      >
>      > What some of us managed to do was to argue that you can't "solve" such issues. All you can do is provide a framework that enables different uses to
>     *cooperate* in some way.
>      >
>      >
>      >
>      > Which is why the Internet drops packets rather than queueing them, and why diffserv cannot work.
>      >
>      > (I know the latter is conftroversial, but at the moment, ALL of diffserv attempts to talk about end-to-end applicaiton specific metrics, but never, ever
>     explains what the diffserv control points actually do w.r.t. what the IP layer can actually control. So it is meaningless - another violation of the
>     so-called end-to-end principle).
>      >
>      >
>      >
>      > Networks are about getting packets from here to there, multiplexing the underlying resources. That's it. Quality is a whole different thing. Quality can
>     be improved by end-to-end approaches, if the underlying network provides some kind of thing that actually creates a way for end-to-end applications to
>     affect queueing and routing decisions, and more importantly getting "telemetry" from the network regarding what is actually going on with the other
>     end-to-end users sharing the infrastructure.
>      >
>      >
>      >
>      > This conference won't talk about it this way. So don't waste your time.
>      >
>      >
>      >
>      >
>      >
>      >
>      >
>      > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" <dave.taht@gmail.com <mailto:dave.taht@gmail.com>> said:
>      >
>      > > The program committee members are *amazing*. Perhaps, finally, we can
>      > > move the bar for the internet's quality metrics past endless, blind
>      > > repetitions of speedtest.
>      > >
>      > > For complete details, please see:
>      > > https://www.iab.org/activities/workshops/network-quality/ <https://www.iab.org/activities/workshops/network-quality/>
>      > >
>      > > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
>      > > Invitations Issued by: Monday 16th August 2021
>      > >
>      > > Workshop Date: This will be a virtual workshop, spread over three days:
>      > >
>      > > 1400-1800 UTC Tue 14th September 2021
>      > > 1400-1800 UTC Wed 15th September 2021
>      > > 1400-1800 UTC Thu 16th September 2021
>      > >
>      > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
>      > >
>      > > The Program Committee members:
>      > >
>      > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
>      > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
>      > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
>      > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
>      > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
>      > >
>      > > Send Submissions to: network-quality-workshop-pc@iab.org <mailto:network-quality-workshop-pc@iab.org>.
>      > >
>      > > Position papers from academia, industry, the open source community and
>      > > others that focus on measurements, experiences, observations and
>      > > advice for the future are welcome. Papers that reflect experience
>      > > based on deployed services are especially welcome. The organizers
>      > > understand that specific actions taken by operators are unlikely to be
>      > > discussed in detail, so papers discussing general categories of
>      > > actions and issues without naming specific technologies, products, or
>      > > other players in the ecosystem are expected. Papers should not focus
>      > > on specific protocol solutions.
>      > >
>      > > The workshop will be by invitation only. Those wishing to attend
>      > > should submit a position paper to the address above; it may take the
>      > > form of an Internet-Draft.
>      > >
>      > > All inputs submitted and considered relevant will be published on the
>      > > workshop website. The organisers will decide whom to invite based on
>      > > the submissions received. Sessions will be organized according to
>      > > content, and not every accepted submission or invited attendee will
>      > > have an opportunity to present as the intent is to foster discussion
>      > > and not simply to have a sequence of presentations.
>      > >
>      > > Position papers from those not planning to attend the virtual sessions
>      > > themselves are also encouraged. A workshop report will be published
>      > > afterwards.
>      > >
>      > > Overview:
>      > >
>      > > "We believe that one of the major factors behind this lack of progress
>      > > is the popular perception that throughput is the often sole measure of
>      > > the quality of Internet connectivity. With such narrow focus, people
>      > > don’t consider questions such as:
>      > >
>      > > What is the latency under typical working conditions?
>      > > How reliable is the connectivity across longer time periods?
>      > > Does the network allow the use of a broad range of protocols?
>      > > What services can be run by clients of the network?
>      > > What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
>      > > What security mechanisms are available for local services, such as DNS?
>      > > To what degree are the privacy, confidentiality, integrity and
>      > > authenticity of user communications guarded?
>      > >
>      > > Improving these aspects of network quality will likely depend on
>      > > measurement and exposing metrics to all involved parties, including to
>      > > end users in a meaningful way. Such measurements and exposure of the
>      > > right metrics will allow service providers and network operators to
>      > > focus on the aspects that impacts the users’ experience most and at
>      > > the same time empowers users to choose the Internet service that will
>      > > give them the best experience."
>      > >
>      > >
>      > > --
>      > > Latest Podcast:
>      > > https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
>      > >
>      > > Dave Täht CTO, TekLibre, LLC
>      > > _______________________________________________
>      > > Cerowrt-devel mailing list
>      > > Cerowrt-devel@lists.bufferbloat.net <mailto:Cerowrt-devel@lists.bufferbloat.net>
>      > > https://lists.bufferbloat.net/listinfo/cerowrt-devel <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
>      > >
> 
> 
> 
>     --
>     Latest Podcast:
>     https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/ <https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> 
>     Dave Täht CTO, TekLibre, LLC
>     _______________________________________________
>     Make-wifi-fast mailing list
>     Make-wifi-fast@lists.bufferbloat.net <mailto:Make-wifi-fast@lists.bufferbloat.net>
>     https://lists.bufferbloat.net/listinfo/make-wifi-fast <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
> 
> 
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
> the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
> restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
> you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
> received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: [Codel] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 17734 bytes --]

For those who might be interested in Little's law
there is a nice paper by John Little on the occasion
of the 50th anniversary  of the result.

https://www.informs.org/Blogs/Operations-Research-Forum/Little-s-Law-as-Viewed-on-its-50th-Anniversary

https://www.informs.org/content/download/255808/2414681/file/little_paper.pdf

Nice read.
Luca

P.S.
Who has not a copy of L. Kleinrock's books? I do have and am not ready to
lend them!

On Fri, Jul 9, 2021 at 11:01 AM Leonard Kleinrock <lk@cs.ucla.edu> wrote:

> David,
>
> I totally appreciate  your attention to when and when not analytical
> modeling works. Let me clarify a few things from your note.
>
> First, Little's law (also known as Little’s lemma or, as I use in my book,
> Little’s result) does not assume Poisson arrivals -  it is good for *any*
> arrival process and any service process and is an equality between time
> averages.  It states that the time average of the number in a system (for a
> sample path *w)* is equal to the average arrival rate to the system
> multiplied by the time-averaged time in the system for that sample path.
> This is often written as   NTimeAvg =λ·TTimeAvg .  Moreover, if the
> system is also ergodic, then the time average equals the ensemble average
> and we often write it as N ̄ = λ T ̄ .  In any case, this requires
> neither Poisson arrivals nor exponential service times.
>
> Queueing theorists often do study the case of Poisson arrivals.  True, it
> makes the analysis easier, yet there is a better reason it is often used,
> and that is because the sum of a large number of independent stationary
> renewal processes approaches a Poisson process.  So nature often gives us
> Poisson arrivals.
>
> Best,
> Len
>
>
>
> On Jul 8, 2021, at 12:38 PM, David P. Reed <dpreed@deepplum.com> wrote:
>
> I will tell you flat out that the arrival time distribution assumption
> made by Little's Lemma that allows "estimation of queue depth" is totally
> unreasonable on ANY Internet in practice.
>
>
> The assumption is a Poisson Arrival Process. In reality, traffic arrivals
> in real internet applications are extremely far from Poisson, and, of
> course, using TCP windowing, become highly intercorrelated with crossing
> traffic that shares the same queue.
>
>
> So, as I've tried to tell many, many net-heads (people who ignore
> applications layer behavior, like the people that think latency doesn't
> matter to end users, only throughput), end-to-end packet arrival times on a
> practical network are incredibly far from Poisson - and they are more like
> fractal probability distributions, very irregular at all scales of time.
>
>
> So, the idea that iperf can estimate queue depth by Little's Lemma by just
> measuring saturation of capacity of a path is bogus.The less Poisson, the
> worse the estimate gets, by a huge factor.
>
>
>
>
> Where does the Poisson assumption come from?  Well, like many theorems, it
> is the simplest tractable closed form solution - it creates a simplified
> view, by being a "single-parameter" distribution (the parameter is called
> lambda for a Poisson distribution).  And the analysis of a simple queue
> with poisson arrival distribution and a static, fixed service time is the
> first interesting Queueing Theory example in most textbooks. It is
> suggestive of an interesting phenomenon, but it does NOT characterize any
> real system.
>
>
> It's the queueing theory equivalent of "First, we assume a spherical
> cow...". in doing an example in a freshman physics class.
>
>
> Unfortunately, most networking engineers understand neither queuing theory
> nor application networking usage in interactive applications. Which makes
> them arrogant. They assume all distributions are poisson!
>
>
>
>
> On Tuesday, July 6, 2021 9:46am, "Ben Greear" <greearb@candelatech.com>
> said:
>
> > Hello,
> >
> > I am interested to hear wish lists for network testing features. We make
> test
> > equipment, supporting lots
> > of wifi stations and a distributed architecture, with built-in udp, tcp,
> ipv6,
> > http, ... protocols,
> > and open to creating/improving some of our automated tests.
> >
> > I know Dave has some test scripts already, so I'm not necessarily
> looking to
> > reimplement that,
> > but more fishing for other/new ideas.
> >
> > Thanks,
> > Ben
> >
> > On 7/2/21 4:28 PM, Bob McMahon wrote:
> > > I think we need the language of math here. It seems like the network
> > power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something
> > useful.
> > > Effective end/end queue depths per Little's law also seems useful.
> Both are
> > available in iperf 2 from a test perspective. Repurposing test
> techniques to
> > actual
> > > traffic could be useful. Hence the question around what exact telemetry
> > is useful to apps making socket write() and read() calls.
> > >
> > > Bob
> > >
> > > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <dave.taht@gmail.com
> > <mailto:dave.taht@gmail.com <dave.taht@gmail.com>>> wrote:
> > >
> > > In terms of trying to find "Quality" I have tried to encourage folk to
> > > both read "zen and the art of motorcycle maintenance"[0], and Deming's
> > > work on "total quality management".
> > >
> > > My own slice at this network, computer and lifestyle "issue" is aiming
> > > for "imperceptible latency" in all things. [1]. There's a lot of
> > > fallout from that in terms of not just addressing queuing delay, but
> > > caching, prefetching, and learning more about what a user really needs
> > > (as opposed to wants) to know via intelligent agents.
> > >
> > > [0] If you want to get depressed, read Pirsig's successor to "zen...",
> > > lila, which is in part about what happens when an engineer hits an
> > > insoluble problem.
> > > [1] https://www.internetsociety.org/events/latency2013/
> > <https://www.internetsociety.org/events/latency2013/>
> > >
> > >
> > >
> > > On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <dpreed@deepplum.com
> > <mailto:dpreed@deepplum.com <dpreed@deepplum.com>>> wrote:
> > > >
> > > > Well, nice that the folks doing the conference  are willing to
> > consider that quality of user experience has little to do with
> signalling rate at
> > the
> > > physical layer or throughput of FTP transfers.
> > > >
> > > >
> > > >
> > > > But honestly, the fact that they call the problem "network quality"
> > suggests that they REALLY, REALLY don't understand the Internet isn't
> the hardware
> > or
> > > the routers or even the routing algorithms *to its users*.
> > > >
> > > >
> > > >
> > > > By ignoring the diversity of applications now and in the future,
> > and the fact that we DON'T KNOW what will be coming up, this conference
> will
> > likely fall
> > > into the usual trap that net-heads fall into - optimizing for some
> > imaginary reality that doesn't exist, and in fact will probably never be
> what
> > users
> > > actually will do given the chance.
> > > >
> > > >
> > > >
> > > > I saw this issue in 1976 in the group developing the original
> > Internet protocols - a desire to put *into the network* special tricks
> to optimize
> > ASR33
> > > logins to remote computers from terminal concentrators (aka remote
> > login), bulk file transfers between file systems on different
> time-sharing
> > systems, and
> > > "sessions" (virtual circuits) that required logins. And then trying to
> > exploit underlying "multicast" by building it into the IP layer, because
> someone
> > > thought that TV broadcast would be the dominant application.
> > > >
> > > >
> > > >
> > > > Frankly, to think of "quality" as something that can be "provided"
> > by "the network" misses the entire point of "end-to-end argument in
> system
> > design".
> > > Quality is not a property defined or created by The Network. If you
> want
> > to talk about Quality, you need to talk about users - all the users at
> all times,
> > > now and into the future, and that's something you can't do if you don't
> > bother to include current and future users talking about what they might
> expect
> > to
> > > experience that they don't experience.
> > > >
> > > >
> > > >
> > > > There was much fighting back in 1976 that basically involved
> > "network experts" saying that the network was the place to "solve" such
> issues as
> > quality,
> > > so applications could avoid having to solve such issues.
> > > >
> > > >
> > > >
> > > > What some of us managed to do was to argue that you can't "solve"
> > such issues. All you can do is provide a framework that enables
> different uses to
> > > *cooperate* in some way.
> > > >
> > > >
> > > >
> > > > Which is why the Internet drops packets rather than queueing them,
> > and why diffserv cannot work.
> > > >
> > > > (I know the latter is conftroversial, but at the moment, ALL of
> > diffserv attempts to talk about end-to-end applicaiton specific metrics,
> but
> > never, ever
> > > explains what the diffserv control points actually do w.r.t. what the
> IP
> > layer can actually control. So it is meaningless - another violation of
> the
> > > so-called end-to-end principle).
> > > >
> > > >
> > > >
> > > > Networks are about getting packets from here to there, multiplexing
> > the underlying resources. That's it. Quality is a whole different thing.
> Quality
> > can
> > > be improved by end-to-end approaches, if the underlying network
> provides
> > some kind of thing that actually creates a way for end-to-end
> applications to
> > > affect queueing and routing decisions, and more importantly getting
> > "telemetry" from the network regarding what is actually going on with
> the other
> > > end-to-end users sharing the infrastructure.
> > > >
> > > >
> > > >
> > > > This conference won't talk about it this way. So don't waste your
> > time.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wednesday, June 30, 2021 8:12pm, "Dave Taht"
> > <dave.taht@gmail.com <mailto:dave.taht@gmail.com <dave.taht@gmail.com>>>
> said:
> > > >
> > > > > The program committee members are *amazing*. Perhaps, finally,
> > we can
> > > > > move the bar for the internet's quality metrics past endless,
> > blind
> > > > > repetitions of speedtest.
> > > > >
> > > > > For complete details, please see:
> > > > > https://www.iab.org/activities/workshops/network-quality/
> > <https://www.iab.org/activities/workshops/network-quality/>
> > > > >
> > > > > Submissions Due: Monday 2nd August 2021, midnight AOE
> > (Anywhere On Earth)
> > > > > Invitations Issued by: Monday 16th August 2021
> > > > >
> > > > > Workshop Date: This will be a virtual workshop, spread over
> > three days:
> > > > >
> > > > > 1400-1800 UTC Tue 14th September 2021
> > > > > 1400-1800 UTC Wed 15th September 2021
> > > > > 1400-1800 UTC Thu 16th September 2021
> > > > >
> > > > > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> > > > >
> > > > > The Program Committee members:
> > > > >
> > > > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire,
> > Sam
> > > > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen,
> > Geoff
> > > > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja
> > Kuehlewind,
> > > > > Jason Livingood, Matt Mathias, Randall Meyer, Kathleen
> > Nichols,
> > > > > Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.
> > > > >
> > > > > Send Submissions to: network-quality-workshop-pc@iab.org
> > <mailto:network-quality-workshop-pc@iab.org
> <network-quality-workshop-pc@iab.org>>.
> > > > >
> > > > > Position papers from academia, industry, the open source
> > community and
> > > > > others that focus on measurements, experiences, observations
> > and
> > > > > advice for the future are welcome. Papers that reflect
> > experience
> > > > > based on deployed services are especially welcome. The
> > organizers
> > > > > understand that specific actions taken by operators are
> > unlikely to be
> > > > > discussed in detail, so papers discussing general categories
> > of
> > > > > actions and issues without naming specific technologies,
> > products, or
> > > > > other players in the ecosystem are expected. Papers should not
> > focus
> > > > > on specific protocol solutions.
> > > > >
> > > > > The workshop will be by invitation only. Those wishing to
> > attend
> > > > > should submit a position paper to the address above; it may
> > take the
> > > > > form of an Internet-Draft.
> > > > >
> > > > > All inputs submitted and considered relevant will be published
> > on the
> > > > > workshop website. The organisers will decide whom to invite
> > based on
> > > > > the submissions received. Sessions will be organized according
> > to
> > > > > content, and not every accepted submission or invited attendee
> > will
> > > > > have an opportunity to present as the intent is to foster
> > discussion
> > > > > and not simply to have a sequence of presentations.
> > > > >
> > > > > Position papers from those not planning to attend the virtual
> > sessions
> > > > > themselves are also encouraged. A workshop report will be
> > published
> > > > > afterwards.
> > > > >
> > > > > Overview:
> > > > >
> > > > > "We believe that one of the major factors behind this lack of
> > progress
> > > > > is the popular perception that throughput is the often sole
> > measure of
> > > > > the quality of Internet connectivity. With such narrow focus,
> > people
> > > > > don’t consider questions such as:
> > > > >
> > > > > What is the latency under typical working conditions?
> > > > > How reliable is the connectivity across longer time periods?
> > > > > Does the network allow the use of a broad range of protocols?
> > > > > What services can be run by clients of the network?
> > > > > What kind of IPv4, NAT or IPv6 connectivity is offered, and
> > are there firewalls?
> > > > > What security mechanisms are available for local services,
> > such as DNS?
> > > > > To what degree are the privacy, confidentiality, integrity
> > and
> > > > > authenticity of user communications guarded?
> > > > >
> > > > > Improving these aspects of network quality will likely depend
> > on
> > > > > measurement and exposing metrics to all involved parties,
> > including to
> > > > > end users in a meaningful way. Such measurements and exposure
> > of the
> > > > > right metrics will allow service providers and network
> > operators to
> > > > > focus on the aspects that impacts the users’ experience
> > most and at
> > > > > the same time empowers users to choose the Internet service
> > that will
> > > > > give them the best experience."
> > > > >
> > > > >
> > > > > --
> > > > > Latest Podcast:
> > > > >
> >
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> > <
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> > > > >
> > > > > Dave Täht CTO, TekLibre, LLC
> > > > > _______________________________________________
> > > > > Cerowrt-devel mailing list
> > > > > Cerowrt-devel@lists.bufferbloat.net
> > <mailto:Cerowrt-devel@lists.bufferbloat.net
> <Cerowrt-devel@lists.bufferbloat.net>>
> > > > > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> > <https://lists.bufferbloat.net/listinfo/cerowrt-devel>
> > > > >
> > >
> > >
> > >
> > > --
> > > Latest Podcast:
> > >
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
> > <
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/>
> > >
> > > Dave Täht CTO, TekLibre, LLC
> > > _______________________________________________
> > > Make-wifi-fast mailing list
> > > Make-wifi-fast@lists.bufferbloat.net
> > <mailto:Make-wifi-fast@lists.bufferbloat.net
> <Make-wifi-fast@lists.bufferbloat.net>>
> > > https://lists.bufferbloat.net/listinfo/make-wifi-fast
> > <https://lists.bufferbloat.net/listinfo/make-wifi-fast>
> > >
> > >
> > > This electronic communication and the information and any files
> transmitted
> > with it, or attached to it, are confidential and are intended solely for
> the use
> > of
> > > the individual or entity to whom it is addressed and may contain
> information
> > that is confidential, legally privileged, protected by privacy laws, or
> otherwise
> > > restricted from disclosure to anyone else. If you are not the intended
> > recipient or the person responsible for delivering the e-mail to the
> intended
> > recipient,
> > > you are hereby notified that any use, copying, distributing,
> dissemination,
> > forwarding, printing, or copying of this e-mail is strictly prohibited.
> If you
> > > received this e-mail in error, please return the e-mail to the sender,
> delete
> > it from your computer, and destroy any printed copy of it.
> > >
> > > _______________________________________________
> > > Starlink mailing list
> > > Starlink@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/starlink
> > >
> >
> >
> > --
> > Ben Greear <greearb@candelatech.com>
> > Candela Technologies Inc http://www.candelatech.com
> >
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

[-- Attachment #2: Type: text/html, Size: 26493 bytes --]

Re: [Codel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Dave Taht]

I just wanted to comment on how awesome this thread was, and how few
people outside this group deeply grok what was discussed here. I would
so like
to somehow construct an educational TV series explaining "How the
Internet really works" to a wider, and new audience, consisting of
animations, anecdotes,
and interviews with the key figures of its evolution.

While I deeply understood Len Kleinrock's work in the period
2011-2015, and tried to pass on analogies and intuition without using
the math since, inspired by van jacobson's
analogies and Radia Perlman's poetry, it's hard for me now to follow
the argument. Queue theory in particular, is not well known or taught
anymore, despite its obvious applications to things like the Covid
crisis.

But that would be just one thing! The end to end argument, the side
effects of spitting postscript into a lego robot, what actually
happens during a web page load, how a cpu actually works, are all
things that are increasingly lost in multiple mental models, and in my
mind many could be taught in kindergarden, if we worked at explaining
it hard enough.

Re: [Codel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Eric Dumazet]

On 10/25/21 8:11 PM, Stuart Cheshire via Bloat wrote:
> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <make-wifi-fast@lists.bufferbloat.net> wrote:
> 
>> Hi All,
>>
>> Sorry for the spam. I'm trying to support a meaningful TCP message latency w/iperf 2 from the sender side w/o requiring e2e clock synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to help with this. It seems that this event goes off when the bytes are in flight vs have reached the destination network stack. If that's the case, then iperf 2 client (sender) may be able to produce the message latency by adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled RTT.
>>
>> Does this seem reasonable?
> 
> I’m not 100% sure what you’re asking, but I will try to help.
> 
> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your endpoint as writable (e.g., via kqueue or epoll) until less than that threshold of data remains unsent. It won’t stop you writing more bytes if you want to, up to the socket send buffer size, but it won’t *ask* you for more data until the TCP_NOTSENT_LOWAT threshold is reached.


When I implemented TCP_NOTSENT_LOWAT back in 2013 [1], I made sure that sendmsg() would actually
stop feeding more bytes in TCP transmit queue if the current amount of unsent bytes
was above the threshold.

So it looks like Apple implementation is different, based on your description ?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=c9bee3b7fdecb0c1d070c7b54113b3bdfb9a3d36

netperf does not use epoll(), but rather a loop over sendmsg().

One of the point of TCP_NOTSENT_LOWAT for Google was to be able to considerably increase
max number of bytes in transmit queues (3rd column of /proc/sys/net/ipv4/tcp_wmem)
by 10x, allowing for autotune to increase BDP for big RTT flows, this without
increasing memory needs for flows with small RTT.

 In other words, the TCP implementation attempts to keep BDP bytes in flight + TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in flight is necessary to fill the network pipe and get good throughput. The TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the source software some advance notice that the TCP implementation will soon be looking for more bytes to send, so that the buffer doesn’t run dry, thereby lowering throughput. (The old SO_SNDBUF option conflates both “bytes in flight” and “bytes buffered and ready to go” into the same number.)
> 
> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification, that will tell you roughly how long it took n bytes to depart the machine. You won’t know why, though. The bytes could depart the machine in response for acks indicating that the same number of bytes have been accepted at the receiver. But the bytes can also depart the machine because CWND is growing. Of course, both of those things are usually happening at the same time.
> 
> How to use TCP_NOTSENT_LOWAT is explained in this video:
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2199>
> 
> Later in the same video is a two-minute demo (time offset 42:00 to time offset 44:00) showing a “before and after” demo illustrating the dramatic difference this makes for screen sharing responsiveness.
> 
> <https://developer.apple.com/videos/play/wwdc2015/719/?time=2520>
> 
> Stuart Cheshire
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>