From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-bk0-f49.google.com (mail-bk0-f49.google.com [209.85.214.49]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 001DD21F0BD for ; Sun, 10 Feb 2013 08:55:53 -0800 (PST) Received: by mail-bk0-f49.google.com with SMTP id w11so2310233bku.8 for ; Sun, 10 Feb 2013 08:55:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=Ih7fWViMVzcOuIMaBGnqP21Ol/T2iIPw+uW9HBWaMKk=; b=cbb7oVoTD+26nxG0/vjUv1KTfU+KA5xlVUtzwpClnagWzmFm9YqWXWnZ+vnAAXJXSd LxPxwuPp+WWJmeFkVwwPfXFWtgyBLFeYimgiWO0ZGcjE7v4NAmC73CJWyZ07VflhfeZ3 i7gEL/1qkAFElasGv91OPO9uoY5IqfJt+qzQgnxXsT9RMY71/ZDIpj4kszXt8cmdJ8Sc 4KPEmtYnzGN3ojQ8Ho+a5dtnuWmUIr4TEnOou5O1236g/GO7rG+Zyo01Cz3UsKsopbCo VUoMF+FTTXJIIvFs6YVay4SvbI1P/2eWh/2F5q/yMi7t55xX6u3hfwMku9ncK2KpiIeU 15pw== MIME-Version: 1.0 X-Received: by 10.204.9.156 with SMTP id l28mr3266268bkl.40.1360515351142; Sun, 10 Feb 2013 08:55:51 -0800 (PST) Received: by 10.204.50.150 with HTTP; Sun, 10 Feb 2013 08:55:50 -0800 (PST) Received: by 10.204.50.150 with HTTP; Sun, 10 Feb 2013 08:55:50 -0800 (PST) In-Reply-To: References: Date: Sun, 10 Feb 2013 18:55:50 +0200 Message-ID: From: Jonathan Morton To: Forums1000 Content-Type: multipart/alternative; boundary=00151758850e37634e04d561aa4e Cc: bloat Subject: Re: [Bloat] I am unable to pinpoint the source of bufferbloat X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Feb 2013 16:55:54 -0000 --00151758850e37634e04d561aa4e Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable If you are traffic shaping - and thus limiting traffic to a level that the hardware can always deal with immediately - then the hardware/driver queues will always be empty except immediately after a packet has been queued. So then their size doesn't matter. There is an initiative within Linux called BQL (Byte Queue Limits) which aims to deal with the driver queue problem (which for a lot of hardware, but not all, is synonymous with the hardware queue). This requires small edits to each driver though. - Jonathan Morton On Feb 10, 2013 6:43 PM, "Forums1000" wrote: > Firstly, I can confirm uploading to Google Drive saturates my upload (I > checked the outgoing rate in the router) and also tried Dropbox and got > the same result. > > Anyway, performing an upload with Windows 7 and then pinging the remote > router with windows XP, added 50ms to the "unbloated" 20ms roundtrip time= s. > So the lowest "most represented" RTT numbers I was getting were around > 70ms, the worst well over 100ms. > I said "most represented" in the previous sentence, as there was also a > significant delay variance ("jitter") in the RTT's. The RTT's regularly > dived under 70ms, sometimes even hitting an odd 30ms. The was also a larg= e > swing in the other direction with regular ventures to 110-120ms. Needless > to say, the balance of those ventures tilted more towards the higher > numbers. > > So Windows 7, being more modern definitely induces more bufferbloat > (thanks to TCP window scaling:)). I'll be repeating this when running > Wireshark as it slipped my mind to fire it up. > > As a sidenote to the above: > I still don't see how an AQM-algorithm will combat the buffering in > drivers and hardware (actually, are the issues pertaining to "drivers" an= d > "hardware' distinct or referring to the same?). I understand that feeding > less packets to a device will prevent the hardware buffer from filling up > as fast as without AQM, but we cannot actually influence its (the HW > buffer) size, can we? If so, it can still introduce a bottleneck that > cannot be prevented... > > -------------------------------------------------------------------------= ---------------------------------------------------------------------------= --------------------- > >> On Sat, Feb 9, 2013 at 2:06 PM, Forums1000 wrote: >> >>> Excellent information. So an AQM-algorithm will sort things on the OS >>> level of the router and should make things considerably better. However= , >>> from reading around on the matter, it seems drivers for the network dev= ice >>> and the hardware itself also contain buffers. Since, Dave (and respect = for >>> that) is developing CeroWRT, is there anything that can be done about t= hat? >>> Do we have any idea on how severe the buffering in drivers and hardware= is? >>> >>> >> In linux, it's got a lot better in the past 2 years. Work continues. >> >> I have some data on OSX now, and some on windows, but not a lot. >> >> In APs, routers, and switches, it's not looking very good. >> >> >> However I note that the second biggest place we see the issue is on the >> edge home gateways, dslams, cable head-ends, and many of those use softw= are >> rate limiting, which generally has 1 buffer or less native to it, and th= e >> underlying buffering doesn't matter. >> >> (then on top of the software rate limiters are big fat dumb drop tails >> queues. currently. sigh) >> >> >> >> >>> A little test I just performed using Windows XP now, indeed shows that >>> Netalyzr is showing me a worst case scenario: >>> >>> >> Meh. Try something with window scaling. Or, try a netanalyzer test from >> some other machine at the same time you do this one. >> >> >>> - a continuous ping (1 ping per second) between 2 routers under my >>> control has an RTT of 20ms (give or take). The remote router I'm pingin= g >>> sits pretty much idle and has nothing better to do than answering the p= ing. >>> - uploading a large file to Google drive (thereby saturating my uplink >>> bandwidth) adds +-10ms of additional latency. >>> >> >> I think this is an invalid assumption without actually measuring your >> transfer rate to the the gdrive. It would not surprise me if their TCP w= as >> pretty sensitive to latency swings, however, they are very on top of the >> bufferbloat issue. >> >> Wouldn't mind a packet capture of that upload while doing the above test= . >> >> >> >>> Sure it varies a bit between 20 and 30ms and goes to 35ms or even 40ms >>> regularly. Moreover, every now and then I get a spike to 70-80ms but th= at >>> spike never lasts more than 3 pings. >>> >>> All in all considerably lower bloat than the 550ms Netalyzr is >>> indicating. In order to mimic the worst case scenario, I'd have to tran= sfer >>> using UDP then? >>> >> >> Just run a more modern OS.... >> >> >>> >>> _______________________________________________ >>> Bloat mailing list >>> Bloat@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >> >> >> -- >> Dave T=E4ht >> >> Fixing bufferbloat with cerowrt: >> http://www.teklibre.com/cerowrt/subscribe.html >> > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > > --00151758850e37634e04d561aa4e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

If you are traffic shaping - and thus limiting traffic to a level that t= he hardware can always deal with immediately - then the hardware/driver que= ues will always be empty except immediately after a packet has been queued.= So then their size doesn't matter.

There is an initiative within Linux called BQL (Byte Queue Limits) which= aims to deal with the driver queue problem (which for a lot of hardware, b= ut not all, is synonymous with the hardware queue). This requires small edi= ts to each driver though.

- Jonathan Morton

On Feb 10, 2013 6:43 PM, "Forums1000" = <forums1000@gmail.com> wr= ote:
Firstly, I can confirm uploading to Google = Drive saturates my upload (I checked the outgoing rate in the router) and= =A0 also tried Dropbox and got the same result.

Anyway, performing = an upload with Windows 7 and then pinging the remote router with windows XP= , added 50ms to the "unbloated" 20ms roundtrip times. So the lowe= st "most represented" RTT numbers I was getting were around 70ms,= the worst well over 100ms.
I said "most represented" in the previous sentence, as there was = also a significant delay variance ("jitter") in the RTT's. Th= e RTT's regularly dived under 70ms, sometimes even hitting an odd 30ms.= The was also a large swing in the other direction with regular ventures to= 110-120ms. Needless to say, the balance of those ventures tilted more towa= rds the higher numbers.

So Windows 7, being more modern definitely induces more bufferbloat (th= anks to TCP window scaling:)). I'll be repeating this when running Wire= shark as it slipped my mind to fire it up.

As a sidenote to the abo= ve:
I still don't see how an AQM-algorithm will combat the buffering in dri= vers and hardware (actually, are the issues pertaining to "drivers&quo= t; and "hardware' distinct or referring to the same?). I understan= d that feeding less packets to a device will prevent the hardware buffer fr= om filling up as fast as without AQM, but we cannot actually influence its = (the HW buffer) size, can we? If so, it can still introduce a bottleneck th= at cannot be prevented...
---------------------------------------------------------------------------= ---------------------------------------------------------------------------= -------------------
On Sat, Feb 9, 2013 at 2:06 PM, Forums1000 = <forums1000@gmail.com> wrote:
Excellent information. So an AQM-algorithm will sort things on th= e OS=20 level of the router and should make things considerably better. However, from reading around on the matter, it seems drivers for the network=20 device and the hardware itself also contain buffers. Since, Dave (and=20 respect for that) is developing CeroWRT, is there anything that can be=20 done about that? Do we have any idea on how severe the buffering in=20 drivers and hardware is?


In linux, it's got a lot be= tter in the past 2 years. Work continues.

I have some data on OSX no= w, and some on windows, but not a lot.

In APs, routers, and switches= , it's not looking very good.


However I note that the second biggest place we see the issue is on= the edge home gateways, dslams, cable head-ends, and many of those use sof= tware rate limiting, which generally has 1 buffer or less native to it, and= the underlying buffering doesn't matter.

(then on top of the software rate limiters are big fat dumb drop tails = queues. currently. sigh)


=A0
A little test I just performed using Windows XP now, indeed shows that= Netalyzr is showing me a worst case scenario:


Meh. Try something with window scaling. Or, try a netana= lyzer test from some other machine at the same time you do this one.
=A0
- a continuous ping (1 ping per second) between 2 routers under my=20 control has an RTT of 20ms (give or take). The remote router I'm pingin= g sits pretty much idle and has nothing better to do than answering the=20 ping.
- uploading a large file to Google drive (thereby saturating my uplink=20 bandwidth) adds +-10ms of additional latency.

I think this is an invalid assumption without actually measuring = your transfer rate to the the gdrive. It would not surprise me if their TCP= was pretty sensitive to latency swings, however, they are very on top of t= he bufferbloat issue.

Wouldn't mind a packet capture of that upload while doing the above= test.

=A0
Sure it varies a bit=20 between 20 and 30ms and goes to 35ms or even 40ms regularly. Moreover,=20 every now and then I get a spike to 70-80ms but that spike never lasts=20 more than 3 pings.

All in all considerably lower bloat than the 550ms Netalyzr is=20 indicating. In order to mimic the worst case scenario, I'd have to=20 transfer using UDP then?

Just run a more modern OS....
= =A0

______________________= _________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
= https://lists.bufferbloat.net/listinfo/bloat




--
Dav= e T=E4ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/= cerowrt/subscribe.html=20


_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat

--00151758850e37634e04d561aa4e--