From dave.taht at gmail.com Sat Jun 3 04:03:54 2023 From: dave.taht at gmail.com (Dave Taht) Date: Sat, 3 Jun 2023 02:03:54 -0600 Subject: [Rpm] receive window bug fix Message-ID: these folk do good work, and I loved the graphs https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From rjmcmahon at rjmcmahon.com Sat Jun 3 14:56:12 2023 From: rjmcmahon at rjmcmahon.com (rjmcmahon) Date: Sat, 03 Jun 2023 11:56:12 -0700 Subject: [Rpm] receive window bug fix In-Reply-To: References: Message-ID: <709fd354372c370233f4b4eb33bab216@rjmcmahon.com> > these folk do good work, and I loved the graphs > > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ Very cool. Thanks for sharing. I've been considering adding stress tests to iperf 2. Looks like Cloudfare has at least two Small reads & writes with short delay to stress receive window processing per At the sending host, run a TCP program with an infinite loop, sending 1500B packets, with a 1 ms delay between each send. At the receiving host, run a TCP program with an infinite loop, reading 1B at a time, with a 1 ms delay between each read. And then, rx buffer limit tests, from https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/ reads as fast as it can, for five seconds this is called fast mode, opens up the window calculates 5% of the high watermark of the bytes reader during any previous one second for each second of the next 15 seconds: this is called slow mode reads that 5% number of bytes, then stops reading sleeps for the remainder of that particular second most of the second consists of no reading at all steps 1-3 are repeated in a loop three times, so the entire run is 60 seconds This has the effect of highlighting any issues in the handling of packets when the buffers repeatedly hit the limit. Curious about any other traffic scenarios driven by socket read/write behaviors that could be useful. Or any others that might apply to WiFi aggregation. Then, if there is a way to generalize these types of send/read/delay graphs with a parametric command line? Bob From woody77 at gmail.com Sat Jun 3 14:20:43 2023 From: woody77 at gmail.com (Aaron Wood) Date: Sat, 3 Jun 2023 11:20:43 -0700 Subject: [Rpm] receive window bug fix In-Reply-To: References: Message-ID: This is good work! I love reading their posts on scale like this. It’s wild to me that the Linux kernel has (apparently) never implemented shrinking the receive window, or handling the case of userspace starting a large transfer and then just not ever reading it… the latter is less surprising, I guess, because that’s an application bug that you probably would catch separately, and would be focused on fixing in the application layer… -Aaron On Sat, Jun 3, 2023 at 1:04 AM Dave Taht via Rpm wrote: > these folk do good work, and I loved the graphs > > > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ > > -- > Podcast: > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ > Dave Täht CSO, LibreQos > _______________________________________________ > Rpm mailing list > Rpm at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm > -- - Sent from my iPhone. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rjmcmahon at rjmcmahon.com Sat Jun 3 15:15:31 2023 From: rjmcmahon at rjmcmahon.com (rjmcmahon) Date: Sat, 03 Jun 2023 12:15:31 -0700 Subject: [Rpm] receive window bug fix In-Reply-To: References:

Message-ID: <6968461fa2076430aa8d379709488c5a@rjmcmahon.com> I think better tooling can help and I am always interested in suggestions on what to add to iperf 2 for better coverages. I've thought it good for iperf 2 to support some sort of graph which drives socket read/write/delays vs a simplistic pattern of AFAP. It for sure stresses things differently, even in drivers. I've seen huge delays in some 10G drivers where some UDP packets seem to get stuck in queues and where the e2e latency is driven by the socket write rates vs the network delays. This is most obvious using burst patterns where the last packet of a latency burst is coupled to the first packet of the subsequent burst. The coupling between the syscalls to network performance is nonobvious and sometimes hard to believe. We've been adding more "traffic profile" knobs for socket testing and have much of the latency metrics incorporated. Most don't use these. They seem to be hard to generalize. Cloudflare seems to have crafted specific tests after obtaining knowledge of causality. Bob PS. As a side note, I'm now being asked how to generate "AI loads" into switch fabrics, though there it probably won't be based upon socket syscalls but maybe using io_urings - not sure. > This is good work! I love reading their posts on scale like this. > > It’s wild to me that the Linux kernel has (apparently) never > implemented shrinking the receive window, or handling the case of > userspace starting a large transfer and then just not ever reading > it… the latter is less surprising, I guess, because that’s an > application bug that you probably would catch separately, and would be > focused on fixing in the application layer… > > -Aaron > > On Sat, Jun 3, 2023 at 1:04 AM Dave Taht via Rpm > wrote: > >> these folk do good work, and I loved the graphs >> >> > https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ >> >> -- >> Podcast: >> > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ >> Dave Täht CSO, LibreQos >> _______________________________________________ >> Rpm mailing list >> Rpm at lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/rpm > -- > - Sent from my iPhone. > _______________________________________________ > Rpm mailing list > Rpm at lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm From dave.taht at gmail.com Tue Jun 6 08:21:05 2023 From: dave.taht at gmail.com (Dave Taht) Date: Tue, 6 Jun 2023 06:21:05 -0600 Subject: [Rpm] data growth report in the eu Message-ID: https://www.linkedin.com/feed/update/urn:li:activity:7071576961424203777/ -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos From dave.taht at gmail.com Fri Jun 30 11:46:34 2023 From: dave.taht at gmail.com (Dave Taht) Date: Fri, 30 Jun 2023 09:46:34 -0600 Subject: [Rpm] Instant Messaging Meets Video Conferencing: Studying the Performance of IM Video Calls Message-ID: This paper does a really good job of measuring the impacts of tcp cross traffic, including BBRv2, against the videoconferencing subsystems in signal, telegram, and whatsapp. https://tma.ifip.org/2023/wp-content/uploads/sites/12/2023/06/tma2023-final37.pdf Naturally fq_codel shines. The paper also introduces a new means of measuring videoconferenceQoE, called BRISQUE. About my only kvetch is they didn't try cake, which has a different shaper, drops less packets, has less collisions, as well as diffserv support. Seeing BBRv2 stats and impact was very interesting also. -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos