From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relais-inet.francetelecom.com (relais-ias92.francetelecom.com [193.251.215.92]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 24C6221F3E1; Thu, 14 May 2015 08:40:51 -0700 (PDT) Received: from omfedm05.si.francetelecom.fr (unknown [xx.xx.xx.1]) by omfedm09.si.francetelecom.fr (ESMTP service) with ESMTP id B5E372DC683; Thu, 14 May 2015 17:40:46 +0200 (CEST) Received: from Exchangemail-eme1.itn.ftgroup (unknown [10.114.1.186]) by omfedm05.si.francetelecom.fr (ESMTP service) with ESMTP id 85CAF35C082; Thu, 14 May 2015 17:40:46 +0200 (CEST) Received: from PEXCVZYM12.corporate.adroot.infra.ftgroup ([fe80::81f:1640:4749:5d13]) by PEXCVZYH01.corporate.adroot.infra.ftgroup ([::1]) with mapi id 14.03.0235.001; Thu, 14 May 2015 17:40:46 +0200 From: To: "Bill Ver Steeg (versteb)" , Dave Taht Thread-Topic: RE : [Bloat] better business bufferbloat monitoring tools? Thread-Index: AQHQjlxXQBvr8GaXjUqv8qk2YddAaw== Date: Thu, 14 May 2015 15:40:45 +0000 Message-ID: <25912_1431618046_5554C1FE_25912_15728_1_uku1cu5mnfsh21qmib4oxebl.1431618039727@email.android.com> References: , In-Reply-To: Accept-Language: en-US, fr-FR Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: multipart/alternative; boundary="_000_uku1cu5mnfsh21qmib4oxebl1431618039727emailandroidcom_" MIME-Version: 1.0 X-PMX-Version: 6.2.1.2478543, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2015.5.14.140316 Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: [Cerowrt-devel] RE : [Bloat] better business bufferbloat monitoring tools? X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 May 2015 15:41:27 -0000 --_000_uku1cu5mnfsh21qmib4oxebl1431618039727emailandroidcom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Bill I beleive you hit the limit of what you can do with AQM w/o FQ. something more can be achieved with paced sources as said in this thread. I do not see incentives for ABR folks to do true pacing however. doing partial pacing to fix the TSO/GSO problem is of course a must but won= 't solve the problem you mention. see you on monday at the conference. I 'm giving a talk right before you. Luca -------- Message d'origine -------- De : "Bill Ver Steeg (versteb)" Date :2015/05/14 00:54 (GMT+01:00) =C0 : Dave Taht Cc : cerowrt-devel@lists.bufferbloat.net, bloat Objet : Re: [Bloat] better business bufferbloat monitoring tools? Dave That said - It has generally been my hope that most of the big movie s= treaming folk have moved to some form of pacing by now but have no data on = it. (?) Bill VerSteeg replies - Based on my recent tests, the production ABR flows = are still quite bursty. There has been some work done in this area, but I d= o not think bloat is top-of-mind for the ABR folks, and I do not think it h= as made it into production systems. Some of the work is in the area of paci= ng TCP's micro-bursts using sch_fq-like methods. Some has been in the area = of application rate estimation. Some of the IW10 pacing stuff may also be u= seful. I am actually giving a talk on AQM to a small ABR video conference next wee= k. The executive summary of my talk is "AQM makes bursty ABR flows less imp= actful to the network buffers (and thus cross traffic), but the bursts stil= l cause problems. The problems are really bad on legacy buffer management a= lgorithms. The new AQM algorithms take care of most of the issues, but burs= ts of data make the new algorithms work harder and do cause some second-ord= er problems." The main problem that I have seen in my testing has been in the CoDel/PIE (= as opposed to FQ_XXX) variants. When the bottleneck link drops packets as t= he elephant bursts, the mice flows suffer. Rather than completing in a hand= ful of RTTs, it takes several times longer for the timeouts and rexmits to = complete the transfer. When running FQ_Codel or FQ_PIE, the elephant flow o= nly impacts itself, as the mice are on their own queues. There are also som= e corner cases when the offered load is extremely high, but these seem to b= e third order effects. I will let the list know what the current state of the art on pacing is aft= er next week's conference, but I suspect that the ABR folks are still on a = learning curve here. Bvs -----Original Message----- From: Dave Taht [mailto:dave.taht@gmail.com] Sent: Wednesday, May 13, 2015 9:37 AM To: Bill Ver Steeg (versteb) Cc: bloat; cerowrt-devel@lists.bufferbloat.net Subject: Re: [Bloat] better business bufferbloat monitoring tools? On Wed, May 13, 2015 at 6:20 AM, Bill Ver Steeg (versteb) wrote: > Time scales are important. Any time you use TCP to send a moderately larg= e file, you drive the link into congestion. Sometimes this is for a few mil= liseconds per hour and sometimes this is for 10s of minutes per hour. > > For instance, watching a 3 Mbps video (Netflix/YouTube/whatever) on a 4 M= bps link with no cross traffic can cause significant bloat, particularly on= older tail drop middleboxes. The host code does an HTTP get every N secon= ds, and drives the link as hard as it can until it gets the video chunk. It= waits a second or two and then does it again. Rinse and Repeat. You end up= with a very characteristic delay plot. The bloat starts at 0, builds until= the middlebox provides congestion feedback, then sawtooths around at about= the buffer size. When the burst ends, the middlebox burns down its buffer = and bloat goes back to zero. Wait a second or two and do it again. The dslreports tests are opening 8 or more full rate streams at once. Not pretty results. Web browsers expend most of their flows entirely in slow start. Etc. I am very concerned with what 4k streaming looks like, and just got an amaz= on box to take a look at it. (but have not put out the cash for a suitable = monitor) > You can't fix this by adding bandwidth to the link. The endpoint's TCP > sessions will simply ramp up to fill the link. You will shorten the > congested phase of the cycle, but TCP will ALWAYS FILL THE LINK (given > enough time to ramp up) It is important to keep stressing this point as the memes propagate outward= s. > > The new AQM (and FQ_AQM) algorithms do a much better job of controlling t= he oscillatory bloat, but you can still see ABR video patterns in the delay= figures. It has generally been my hope that most of the big movie streaming folk hav= e moved to some form of pacing by now but have no data on it. (?) Certainly I'm happy with what I saw of quic and have hope that http/2 will = cut the number of simultaneous flows in progress. But I return to my original point in that I would like to continue to find = more ways to make the sub 5 minute behaviors visible and comprehensible to = more people... > Bvs > > > -----Original Message----- > From: bloat-bounces@lists.bufferbloat.net > [mailto:bloat-bounces@lists.bufferbloat.net] On Behalf Of Dave Taht > Sent: Tuesday, May 12, 2015 12:00 PM > To: bloat; cerowrt-devel@lists.bufferbloat.net > Subject: [Bloat] better business bufferbloat monitoring tools? > > One thread bothering me on dslreports.com is that some folk seem to think= you only get bufferbloat if you stress test the network, where transient b= ufferbloat is happening all the time, everywhere. > > On one of my main sqm'd network gateways, day in, day out, it reports abo= ut 6000 drops or ecn marks on ingress, and about 300 on egress. > Before I doubled the bandwidth that main box got, the drop rate used to b= e much higher, and a great deal of the bloat, drops, etc, has now moved int= o the wifi APs deeper into the network where I am not monitoring it effecti= vely. > > I would love to see tools like mrtg, cacti, nagios and smokeping[1] be mo= re closely integrated, with bloat related plugins, and in particular, as th= ings like fq_codel and other ecn enabled aqms deploy, start also tracking c= ongestive events like loss and ecn CE markings on the bandwidth tracking gr= aphs. > > This would counteract to some extent the classic 5 minute bandwidth summa= ries everyone looks at, that hide real traffic bursts, latencies and loss a= t sub 5 minute timescales. > > mrtg and cacti rely on snmp. While loss statistics are deeply part of snm= p, I am not aware of there being a mib for CE events and a quick google sea= rch was unrevealing. ? > > There is also a need for more cross-network monitoring using tools such a= s that done by this excellent paper. > > http://www.caida.org/publications/papers/2014/measurement_analysis_int > ernet_interconnection/measurement_analysis_internet_interconnection.pd > f > > [1] the network monitoring tools market is quite vast and has many commer= cial applications, like intermapper, forks of nagios, vendor specific produ= cs from cisco, etc, etc. Far too many to list, and so far as I know, none a= re reporting ECN related stats, nor combining latency and loss with bandwid= th graphs. I would love to know if any products, commercial or open source,= did.... > > -- > Dave T=E4ht > Open Networking needs **Open Source Hardware** > > https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat -- Dave T=E4ht Open Networking needs **Open Source Hardware** https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67 _______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat ___________________________________________________________________________= ______________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confiden= tielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu= ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages el= ectroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou = falsifie. Merci. This message and its attachments may contain confidential or privileged inf= ormation that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and dele= te this message and its attachments. As emails may be altered, Orange is not liable for messages that have been = modified, changed or falsified. Thank you. --_000_uku1cu5mnfsh21qmib4oxebl1431618039727emailandroidcom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Bill

I beleive you hit the limit of what you can do with AQM w/o FQ. <= /div>

something more can be achieved with paced sources as said in this thre= ad. 

I do not see  incentives for ABR folks to do true pacing however.=  

doing partial pacing to fix the TSO/GSO problem is of course a must bu= t won't solve the problem you mention. 

see you on monday at the conference.  I 'm giving a talk right be= fore you. 

Luca




-------- Message d'origine --------
De : "Bill Ver Steeg (versteb)"
Date :2015/05/14 00:54 (GMT+01:00)
=C0 : Dave Taht
Cc : cerowrt-devel@lists.bufferbloat.net, bloat
Objet : Re: [Bloat] better business bufferbloat monitoring tools?

Dave That said - It has generally been my hope tha= t most of the big movie streaming folk have moved to some form of pacing by= now but have no data on it. (?)

Bill VerSteeg replies - Based on my recent tests, the production ABR flows = are still quite bursty. There has been some work done in this area, but I d= o not think bloat is top-of-mind for the ABR folks, and I do not think it h= as made it into production systems. Some of the work is in the area of pacing TCP's micro-bursts using sch_fq-= like methods. Some has been in the area of application rate estimation. Som= e of the IW10 pacing stuff may also be useful.

I am actually giving a talk on AQM to a small ABR video conference next wee= k. The executive summary of my talk is "AQM makes bursty ABR flows les= s impactful to the network buffers (and thus cross traffic), but the bursts= still cause problems. The problems are really bad on legacy buffer management algorithms. The new AQM algorithms = take care of most of the issues, but bursts of data make the new algorithms= work harder and do cause some second-order problems."

The main problem that I have seen in my testing has been in the CoDel/PIE (= as opposed to FQ_XXX) variants. When the bottleneck link drops packets as t= he elephant bursts, the mice flows suffer. Rather than completing in a hand= ful of RTTs, it takes several times longer for the timeouts and rexmits to complete the transfer. When running= FQ_Codel or FQ_PIE, the elephant flow only impacts itself, as the mice are= on their own queues. There are also some corner cases when the offered loa= d is extremely high, but these seem to be third order effects.

I will let the list know what the current state of the art on pacing is aft= er next week's conference, but I suspect that the ABR folks are still on a = learning curve here.

Bvs


-----Original Message-----
From: Dave Taht [mailto:dave.taht@gm= ail.com]
Sent: Wednesday, May 13, 2015 9:37 AM
To: Bill Ver Steeg (versteb)
Cc: bloat; cerowrt-devel@lists.bufferbloat.net
Subject: Re: [Bloat] better business bufferbloat monitoring tools?

On Wed, May 13, 2015 at 6:20 AM, Bill Ver Steeg (versteb) <versteb@cisco= .com> wrote:
> Time scales are important. Any time you use TCP to send a moderately l= arge file, you drive the link into congestion. Sometimes this is for a few = milliseconds per hour and sometimes this is for 10s of minutes per hour.
>
> For instance, watching a 3 Mbps video (Netflix/YouTube/whatever) on a = 4 Mbps link with no cross traffic can cause significant bloat, particularly= on older tail drop middleboxes.  The host code does an HTTP get every= N seconds, and drives the link as hard as it can until it gets the video chunk. It waits a second or two and then= does it again. Rinse and Repeat. You end up with a very characteristic del= ay plot. The bloat starts at 0, builds until the middlebox provides congest= ion feedback, then sawtooths around at about the buffer size. When the burst ends, the middlebox burns down it= s buffer and bloat goes back to zero. Wait a second or two and do it again.=

The dslreports tests are opening 8 or more full rate streams at once.
Not pretty results.

Web browsers expend most of their flows entirely in slow start.

Etc.

I am very concerned with what 4k streaming looks like, and just got an amaz= on box to take a look at it. (but have not put out the cash for a suitable = monitor)

> You can't fix this by adding bandwidth to the link. The endpoint's TCP=
> sessions will simply ramp up to fill the link. You will shorten the > congested phase of the cycle, but TCP will ALWAYS FILL THE LINK (given=
> enough time to ramp up)

It is important to keep stressing this point as the memes propagate outward= s.

>
> The new AQM (and FQ_AQM) algorithms do a much better job of controllin= g the oscillatory bloat, but you can still see ABR video patterns in the de= lay figures.

It has generally been my hope that most of the big movie streaming folk hav= e moved to some form of pacing by now but have no data on it.
(?)

Certainly I'm happy with what I saw of quic and have hope that http/2 will = cut the number of simultaneous flows in progress.

But I return to my original point in that I would like to continue to find = more ways to make the sub 5 minute behaviors visible and comprehensible to = more people...

> Bvs
>
>
> -----Original Message-----
> From: bloat-bounces@lists.bufferbloat.net
> [mailto:bloat-b= ounces@lists.bufferbloat.net] On Behalf Of Dave Taht
> Sent: Tuesday, May 12, 2015 12:00 PM
> To: bloat; cerowrt-devel@lists.bufferbloat.net
> Subject: [Bloat] better business bufferbloat monitoring tools?
>
> One thread bothering me on dslreports.com is that some folk seem to th= ink you only get bufferbloat if you stress test the network, where transien= t bufferbloat is happening all the time, everywhere.
>
> On one of my main sqm'd network gateways, day in, day out, it reports = about 6000 drops or ecn marks on ingress, and about 300 on egress.
> Before I doubled the bandwidth that main box got, the drop rate used t= o be much higher, and a great deal of the bloat, drops, etc, has now moved = into the wifi APs deeper into the network where I am not monitoring it effe= ctively.
>
> I would love to see tools like mrtg, cacti, nagios and smokeping[1] be= more closely integrated, with bloat related plugins, and in particular, as= things like fq_codel and other ecn enabled aqms deploy, start also trackin= g congestive events like loss and ecn CE markings on the bandwidth tracking graphs.
>
> This would counteract to some extent the classic 5 minute bandwidth su= mmaries everyone looks at, that hide real traffic bursts, latencies and los= s at sub 5 minute timescales.
>
> mrtg and cacti rely on snmp. While loss statistics are deeply part of = snmp, I am not aware of there being a mib for CE events and a quick google = search was unrevealing. ?
>
> There is also a need for more cross-network monitoring using tools suc= h as that done by this excellent paper.
>
> http://www.caida.org/publications/papers/2014/measurement_analysis_int<= br> > ernet_interconnection/measurement_analysis_internet_interconnection.pd=
> f
>
> [1] the network monitoring tools market is quite vast and has many com= mercial applications, like intermapper, forks of nagios, vendor specific pr= oducs from cisco, etc, etc. Far too many to list, and so far as I know, non= e are reporting ECN related stats, nor combining latency and loss with bandwidth graphs. I would love to know= if any products, commercial or open source, did....
>
> --
> Dave T=E4ht
> Open Networking needs **Open Source Hardware**
>
> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists= .bufferbloat.net/listinfo/bloat



--
Dave T=E4ht
Open Networking needs **Open Source Hardware**

= https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.buff= erbloat.net/listinfo/bloat
 
______________________________________________________________________=
___________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confiden=
tielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu=
 ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages el=
ectroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou =
falsifie. Merci.

This message and its attachments may contain confidential or privileged inf=
ormation that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and dele=
te this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been =
modified, changed or falsified.
Thank you.
--_000_uku1cu5mnfsh21qmib4oxebl1431618039727emailandroidcom_--