[Bloat] [Cerowrt-devel] better business bufferbloat monitoring tools?

Thu May 14 10:28:00 EDT 2015

Tools, tools, tools.  Make it trivially easy to capture packets in the home (don't require cerowrt, for obvious reasons).  For example, an iPhone app that does a tcpdump and sends it to us would be fantastic to diagnose "make wifi fast" issues and also bufferbloat issues.  Give feedback that is helpful to every one who contributes data.  (That's what made netalyzr work so well... you got feedback ASAP that could be used to understand your own situation).

Not sure an iPhone app can be disseminated.  An Android app might be, as could a MacBook app and a WIndows app.

Linux/FreeBSD options: One could  generate a memstick app that would boot Linux on a standard windows laptop to run tcpdump and upload the results, or something that would run in Parallels or VMWare fusion on a Mac.

I've started looking at a hardware measurement platform for my "make WiFi fast" work - currently looks like a Rangely board will do the trick.  But that won't scale well outside my home since it costs a few hundred bucks for the hardware.

On Wednesday, May 13, 2015 11:30am, "Jim Gettys" <jg at freedesktop.org> said:

On Wed, May 13, 2015 at 9:20 AM, Bill Ver Steeg (versteb) <[ versteb at cisco.com ]( mailto:versteb at cisco.com )> wrote:
Time scales are important. Any time you use TCP to send a moderately large file, you drive the link into congestion. Sometimes this is for a few milliseconds per hour and sometimes this is for 10s of minutes per hour.

 For instance, watching a 3 Mbps video (Netflix/YouTube/whatever) on a 4 Mbps link with no cross traffic can cause significant bloat, particularly on older tail drop middleboxes.  The host code does an HTTP get every N seconds, and drives the link as hard as it can until it gets the video chunk. It waits a second or two and then does it again. Rinse and Repeat. You end up with a very characteristic delay plot. The bloat starts at 0, builds until the middlebox provides congestion feedback, then sawtooths around at about the buffer size. When the burst ends, the middlebox burns down its buffer and bloat goes back to zero. Wait a second or two and do it again.

It's time to do some packet traces to see what the video providers are doing.  In YouTube's case, I believe the traffic is using the new sched_fq qdisc, which does packet pacing; but exactly how this plays out by the time packets reach the home isn't entirely clear to me. Other video providers/CDN's may/may not have started generating clues.

Also note that so far, no one is trying to pace the IW transmission at all.

 You can't fix this by adding bandwidth to the link. The endpoint's TCP sessions will simply ramp up to fill the link. You will shorten the congested phase of the cycle, but TCP will ALWAYS FILL THE LINK (given enough time to ramp up)

That has been the behavior in the past, but it's no longer safe to presume we should tar everyone with the same brush, rather, we should do a bit of science, and then try to hold people's feet to the fire that do not "play nice" with the network.
Some packet captures in the home can easily sort this out.
Jim

 The new AQM (and FQ_AQM) algorithms do a much better job of controlling the oscillatory bloat, but you can still see ABR video patterns in the delay figures.

 Bvs

 -----Original Message-----
 From: [ bloat-bounces at lists.bufferbloat.net ]( mailto:bloat-bounces at lists.bufferbloat.net ) [mailto:[ bloat-bounces at lists.bufferbloat.net ]( mailto:bloat-bounces at lists.bufferbloat.net )] On Behalf Of Dave Taht
 Sent: Tuesday, May 12, 2015 12:00 PM
 To: bloat; [ cerowrt-devel at lists.bufferbloat.net ]( mailto:cerowrt-devel at lists.bufferbloat.net )
 Subject: [Bloat] better business bufferbloat monitoring tools?

 One thread bothering me on [ dslreports.com ]( http://dslreports.com ) is that some folk seem to think you only get bufferbloat if you stress test the network, where transient bufferbloat is happening all the time, everywhere.

 On one of my main sqm'd network gateways, day in, day out, it reports about 6000 drops or ecn marks on ingress, and about 300 on egress.
 Before I doubled the bandwidth that main box got, the drop rate used to be much higher, and a great deal of the bloat, drops, etc, has now moved into the wifi APs deeper into the network where I am not monitoring it effectively.

 I would love to see tools like mrtg, cacti, nagios and smokeping[1] be more closely integrated, with bloat related plugins, and in particular, as things like fq_codel and other ecn enabled aqms deploy, start also tracking congestive events like loss and ecn CE markings on the bandwidth tracking graphs.

 This would counteract to some extent the classic 5 minute bandwidth summaries everyone looks at, that hide real traffic bursts, latencies and loss at sub 5 minute timescales.

 mrtg and cacti rely on snmp. While loss statistics are deeply part of snmp, I am not aware of there being a mib for CE events and a quick google search was unrevealing. ?

 There is also a need for more cross-network monitoring using tools such as that done by this excellent paper.

[ http://www.caida.org/publications/papers/2014/measurement_analysis_internet_interconnection/measurement_analysis_internet_interconnection.pdf ]( http://www.caida.org/publications/papers/2014/measurement_analysis_internet_interconnection/measurement_analysis_internet_interconnection.pdf )

 [1] the network monitoring tools market is quite vast and has many commercial applications, like intermapper, forks of nagios, vendor specific producs from cisco, etc, etc. Far too many to list, and so far as I know, none are reporting ECN related stats, nor combining latency and loss with bandwidth graphs. I would love to know if any products, commercial or open source, did....

 --
 Dave Täht
 Open Networking needs **Open Source Hardware**

[ https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 ]( https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 )
 _______________________________________________Bloat mailing list
[ Bloat at lists.bufferbloat.net ]( mailto:Bloat at lists.bufferbloat.net )
[ https://lists.bufferbloat.net/listinfo/bloat ]( https://lists.bufferbloat.net/listinfo/bloat )

_______________________________________________
 Cerowrt-devel mailing list
[ Cerowrt-devel at lists.bufferbloat.net ]( mailto:Cerowrt-devel at lists.bufferbloat.net )
[ https://lists.bufferbloat.net/listinfo/cerowrt-devel ]( https://lists.bufferbloat.net/listinfo/cerowrt-devel )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20150514/cc4e030d/attachment-0003.html>