From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 2A22E21F1BA; Wed, 13 May 2015 08:30:46 -0700 (PDT) Received: by widdi4 with SMTP id di4so203930889wid.0; Wed, 13 May 2015 08:30:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Hvt72HB9NYvNzBtzREULTtWV2kXsdSGUWZFEdXpTelQ=; b=ctdYJfM1rbeGZfRHWIkeO1HXeFJ6O4TEMhACWp2YWn2o95y1Fu4G2scyheCOyjwKJe E5Ouyj4nqVeuqtahnvBB0NJI9gNg32ELRojxq34wEkRtm7GcSsqVogRh4jpzL62OY/7D h/MVuSP6CVUxJhlUPBA2DUHOpEujadLAxiAHOvj7Drz0GumVEsoD72MAXy4KfGOlGWjc XQg6x8aRxESIJFIHcoapv2WgJQxcNFWmTcRYAfHOmk8H1n5SRfeO3ocIqtNVmZN1M/tk oxIwBWNXldLCLx2WwcBmWD0Kjg/dzOyFoflaO/Jx0k3SBYMQ17u3axS2FzK9NJk/F+kX LZzg== MIME-Version: 1.0 X-Received: by 10.180.14.4 with SMTP id l4mr14726774wic.14.1431531040455; Wed, 13 May 2015 08:30:40 -0700 (PDT) Sender: gettysjim@gmail.com Received: by 10.194.118.166 with HTTP; Wed, 13 May 2015 08:30:40 -0700 (PDT) In-Reply-To: References: Date: Wed, 13 May 2015 11:30:40 -0400 X-Google-Sender-Auth: 4fHhVeLB_4j0JGSSU60drPzFAVQ Message-ID: From: Jim Gettys To: "Bill Ver Steeg (versteb)" Content-Type: multipart/alternative; boundary=f46d040f9f5226c0b90515f84ba3 Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] [Bloat] better business bufferbloat monitoring tools? X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 May 2015 15:31:16 -0000 --f46d040f9f5226c0b90515f84ba3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, May 13, 2015 at 9:20 AM, Bill Ver Steeg (versteb) wrote: > Time scales are important. Any time you use TCP to send a moderately larg= e > file, you drive the link into congestion. Sometimes this is for a few > milliseconds per hour and sometimes this is for 10s of minutes per hour. > > For instance, watching a 3 Mbps video (Netflix/YouTube/whatever) on a 4 > Mbps link with no cross traffic can cause significant bloat, particularly > on older tail drop middleboxes. The host code does an HTTP get every N > seconds, and drives the link as hard as it can until it gets the video > chunk. It waits a second or two and then does it again. Rinse and Repeat. > You end up with a very characteristic delay plot. The bloat starts at 0, > builds until the middlebox provides congestion feedback, then sawtooths > around at about the buffer size. When the burst ends, the middlebox burns > down its buffer and bloat goes back to zero. Wait a second or two and do = it > again. > =E2=80=8BIt's time to do some packet traces to see what the video providers= are doing. In YouTube's case, I believe the traffic is using the new sched_fq qdisc, which does packet pacing; but exactly how this plays out by the time packets reach the home isn't entirely clear to me. Other video providers/CDN's may/may not have started generating clues. Also note that so far, no one is trying to pace the IW transmission at all. =E2=80=8B > > You can't fix this by adding bandwidth to the link. The endpoint's TCP > sessions will simply ramp up to fill the link. You will shorten the > congested phase of the cycle, but TCP will ALWAYS FILL THE LINK (given > enough time to ramp up) > =E2=80=8BThat has been the behavior in the past, but it's no longer safe to presume=E2=80=8B we should tar everyone with the same brush, rather, we sho= uld do a bit of science, and then try to hold people's feet to the fire that do not "play nice" with the network. =E2=80=8BSome packet captures in the home can easily sort this out. Jim =E2=80=8B > > The new AQM (and FQ_AQM) algorithms do a much better job of controlling > the oscillatory bloat, but you can still see ABR video patterns in the > delay figures. > > Bvs > > > -----Original Message----- > From: bloat-bounces@lists.bufferbloat.net [mailto: > bloat-bounces@lists.bufferbloat.net] On Behalf Of Dave Taht > Sent: Tuesday, May 12, 2015 12:00 PM > To: bloat; cerowrt-devel@lists.bufferbloat.net > Subject: [Bloat] better business bufferbloat monitoring tools? > > One thread bothering me on dslreports.com is that some folk seem to think > you only get bufferbloat if you stress test the network, where transient > bufferbloat is happening all the time, everywhere. > > On one of my main sqm'd network gateways, day in, day out, it reports > about 6000 drops or ecn marks on ingress, and about 300 on egress. > Before I doubled the bandwidth that main box got, the drop rate used to b= e > much higher, and a great deal of the bloat, drops, etc, has now moved int= o > the wifi APs deeper into the network where I am not monitoring it > effectively. > > I would love to see tools like mrtg, cacti, nagios and smokeping[1] be > more closely integrated, with bloat related plugins, and in particular, a= s > things like fq_codel and other ecn enabled aqms deploy, start also tracki= ng > congestive events like loss and ecn CE markings on the bandwidth tracking > graphs. > > This would counteract to some extent the classic 5 minute bandwidth > summaries everyone looks at, that hide real traffic bursts, latencies and > loss at sub 5 minute timescales. > > mrtg and cacti rely on snmp. While loss statistics are deeply part of > snmp, I am not aware of there being a mib for CE events and a quick googl= e > search was unrevealing. ? > > There is also a need for more cross-network monitoring using tools such a= s > that done by this excellent paper. > > > http://www.caida.org/publications/papers/2014/measurement_analysis_intern= et_interconnection/measurement_analysis_internet_interconnection.pdf > > [1] the network monitoring tools market is quite vast and has many > commercial applications, like intermapper, forks of nagios, vendor specif= ic > producs from cisco, etc, etc. Far too many to list, and so far as I know, > none are reporting ECN related stats, nor combining latency and loss with > bandwidth graphs. I would love to know if any products, commercial or ope= n > source, did.... > > -- > Dave T=C3=A4ht > Open Networking needs **Open Source Hardware** > > https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel > --f46d040f9f5226c0b90515f84ba3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Wed, Ma= y 13, 2015 at 9:20 AM, Bill Ver Steeg (versteb) <versteb@cisco.com>= wrote:
Time scales are important.= Any time you use TCP to send a moderately large file, you drive the link i= nto congestion. Sometimes this is for a few milliseconds per hour and somet= imes this is for 10s of minutes per hour.

For instance, watching a 3 Mbps video (Netflix/YouTube/whatever) on a 4 Mbp= s link with no cross traffic can cause significant bloat, particularly on o= lder tail drop middleboxes.=C2=A0 The host code does an HTTP get every N se= conds, and drives the link as hard as it can until it gets the video chunk.= It waits a second or two and then does it again. Rinse and Repeat. You end= up with a very characteristic delay plot. The bloat starts at 0, builds un= til the middlebox provides congestion feedback, then sawtooths around at ab= out the buffer size. When the burst ends, the middlebox burns down its buff= er and bloat goes back to zero. Wait a second or two and do it again.

=E2=80=8BIt's time to do some packet traces t= o see what the video providers are doing.=C2=A0 In YouTube's case, I be= lieve the traffic is using the new sched_fq qdisc, which does packet pacing= ; but exactly how this plays out by the time packets reach the home isn'= ;t entirely clear to me. Other video providers/CDN's may/may not have s= tarted generating clues.

Also note that so far, = no one is trying to pace the IW transmission at all.
=E2=80=8B=C2=A0

You can't fix this by adding bandwidth to the link. The endpoint's = TCP sessions will simply ramp up to fill the link. You will shorten the con= gested phase of the cycle, but TCP will ALWAYS FILL THE LINK (given enough = time to ramp up)

=E2=80=8BThat has been the = behavior in the past, but it's no longer safe to presume=E2=80=8B we sh= ould tar everyone with the same brush, rather, we should do a bit of scienc= e, and then try to hold people's feet to the fire that do not "pla= y nice" with the network.

=E2=80=8BSome packet captures in the h= ome can easily sort this out.

Jim

=E2=80=8B
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
The new AQM (and FQ_AQM) algorithms do a much better job of controlling the= oscillatory bloat, but you can still see ABR video patterns in the delay f= igures.

Bvs


-----Original Message-----
From: bloat-bounces@= lists.bufferbloat.net [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Dave Ta= ht
Sent: Tuesday, May 12, 2015 12:00 PM
To: bloat; cerowrt-d= evel@lists.bufferbloat.net
Subject: [Bloat] better business bufferbloat monitoring tools?

One thread bothering me on dslreports.com is that some folk seem to think you only get bufferb= loat if you stress test the network, where transient bufferbloat is happeni= ng all the time, everywhere.

On one of my main sqm'd network gateways, day in, day out, it reports a= bout 6000 drops or ecn marks on ingress, and about 300 on egress.
Before I doubled the bandwidth that main box got, the drop rate used to be = much higher, and a great deal of the bloat, drops, etc, has now moved into = the wifi APs deeper into the network where I am not monitoring it effective= ly.

I would love to see tools like mrtg, cacti, nagios and smokeping[1] be more= closely integrated, with bloat related plugins, and in particular, as thin= gs like fq_codel and other ecn enabled aqms deploy, start also tracking con= gestive events like loss and ecn CE markings on the bandwidth tracking grap= hs.

This would counteract to some extent the classic 5 minute bandwidth summari= es everyone looks at, that hide real traffic bursts, latencies and loss at = sub 5 minute timescales.

mrtg and cacti rely on snmp. While loss statistics are deeply part of snmp,= I am not aware of there being a mib for CE events and a quick google searc= h was unrevealing. ?

There is also a need for more cross-network monitoring using tools such as = that done by this excellent paper.

http://www.caida.org/publications/papers/2014/measure= ment_analysis_internet_interconnection/measurement_analysis_internet_interc= onnection.pdf

[1] the network monitoring tools market is quite vast and has many commerci= al applications, like intermapper, forks of nagios, vendor specific producs= from cisco, etc, etc. Far too many to list, and so far as I know, none are= reporting ECN related stats, nor combining latency and loss with bandwidth= graphs. I would love to know if any products, commercial or open source, d= id....

--
Dave T=C3=A4ht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat
____________________________= ___________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.= bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

--f46d040f9f5226c0b90515f84ba3--