From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-iw0-f171.google.com (mail-iw0-f171.google.com [209.85.214.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 482C6201759; Wed, 11 May 2011 07:30:18 -0700 (PDT) Received: by iwn8 with SMTP id 8so838864iwn.16 for ; Wed, 11 May 2011 07:37:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=i4ppRf0RJYEG3CX2QsC7OLTOnyuA8pUTQcdAckvScCM=; b=RCQm2tzowohbOIvYsclvEEvnRFgqdHYGcPzzyNdkuwognzurgk3gmxfM0bWsHU9+hs fjlyYlQ9+keedob+ClxXjtMoKaizL20RMh/Hs+IEYX33Oo8jW6dyVd1o4yQpWUqCatmr /e0o79hGjL652wmb0WkoPGhU8OEZiVGbkUPhk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=WGuROTkGIxp0JTT2RoSZGjFjEHj4S6fex//5MTcj/dI5Gc2pM+50ptIBM2YKAEZ4Oo q5NY4EOmh2JGdTqHrsfa2PdOWVPALaY1vZiNekEOanxTVB6DH4AN8BVhOkGKwJrpCLJC XKIDtuCWRnSO3JeDlIju2ogadqSfWuN0UsrJM= MIME-Version: 1.0 Received: by 10.42.66.207 with SMTP id q15mr1158991ici.359.1305124657271; Wed, 11 May 2011 07:37:37 -0700 (PDT) Received: by 10.231.31.201 with HTTP; Wed, 11 May 2011 07:37:37 -0700 (PDT) Date: Wed, 11 May 2011 08:37:37 -0600 Message-ID: From: Dave Taht To: bloat , bismark-devel@lists.bufferbloat.net, bloat-devel@lists.bufferbloat.net Content-Type: multipart/alternative; boundary=90e6ba614b529593ad04a301032a Subject: [Bismark-devel] QoS for mission critical packets on wireless networks X-BeenThere: bismark-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: BISMark related software development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2011 14:30:19 -0000 --90e6ba614b529593ad04a301032a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I think it would be more useful for me to describe the context behind the fe80:: thread also on this list. In the case of wireless interfaces, very little QoS and packet prioritization has been taking place with the various qdiscs being used. (Linux uses "mq" by default, which is severely underdocumented, and the features it has are rarely implemented by packet classification before it hits that qdisc, and it seems mostly intended to take advantage of hardware QoS, (which at least on wireless n is something of a lose anyway - aggregation is better) Result: On a busy wireless network, mission critical, statistically rare packets, are getting delayed or lost. If it already wasn't apparent to me before I started on this trip a few weeks ago, it is glaringly obvious now. Yesterday I was getting 3.5KB(!)/sec throughput on Georgia Tech's internal wireless lan. I could "hear" over 32 access points. And everywhere I've been, hotels, airports, etc, I've seen DHCP take multiple attempts to get an address, and DNS timing out a lot. I'm sure others have observed this sort of behavior!!! On a busy wireless lan, and/or one with many access points and clients on the same channel, DHCP queries go unanswered, ARP!! times out, DNS times out, RA and ND fail, NTP gets delayed... While some of this is due to interference (my god, "hearing" 32+APs!), I believe that a great deal of it is due to large queue lengths and contention for the link the user is connected to. The default behavior for many of these protocols is to re-send the packets after a short time - thus adding to the problem! So... Felix Fietkau reduced queue lengths significantly in the ath9k wireless drivers used in the uberwrt/bismark/iscswrt projects and that has helped a lot (still quite a ways to go to match my original patch but getting packet aggregation to work well requires larger queues) On the current build of bismark, I get 32ms avg delay on a latency under load test, on an access point I'm connected to at 54Mbit/second. I'm going to start doing connects at slower speeds today and expect the results to be rather awful. While this is not good (my old patches got around 4ms, debloat-testing hits around 12ms), this is STILL better than what many are seeing in the field..= . So... I think some *more* of the problems can be fixed by packet prioritization o= n the wireless side. So I started working on a set of wireless QoS ideas to see if I could improve the situation. (wilst working on red for external, wired interfaces= ) I started off with a hard case - a broadcast multicast protocol that runs over IPv6 called babel. It has the interesting property in that it sends a broadcast every 20 seconds from an identifiable address (theoretically - se= e the fe80 discussion also on this list) and thus I can easily (or so I thought) observe, with enough nodes - delays and packet loss. The idea I'm toying with now is to reserve a few percentage points of the bandwidth to mission critical packets such as those listed so far, using packet marking and a queuing discipline that does more of the "right thing" (hsfc or htb+sfb) (strict prioritization of things like ping would lead to trivial DDOSes on any of these protocols, so a tiny bandwidth reserve strikes me as a better approach) Wireless multicast is additionally weird in that it falls back to a very lo= w rate in order to do its work. Suggestions? comments? The current build of the "capetown" release of bismark is very usable, has many cool features, including SFB, and supports several off the shelf routers commonly available via best-buy and new-egg. Anyone that's interested, please join in the effort. Certainly mere qdiscs are something more folk can fiddle with at a scripting and tcpdump level... Bismark documentation (supports Netgear wndr3700, wndr3700v2, nanostation M5, buffalo, and I just build an x86 kvm virtual machine that needs love) http://www.bufferbloat.net/projects/bismark/wiki Capetown builds: http://huchra.bufferbloat.net/~bismark/ --=20 Dave T=E4ht SKYPE: davetaht US Tel: 1-239-829-5608 http://the-edge.blogspot.com --90e6ba614b529593ad04a301032a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I think it would be more useful for me to describe the context behind the f= e80:: thread also
on this list.

In the case of wireless interfac= es, very little QoS and packet prioritization has been taking place
with= the various qdiscs being used. (Linux uses "mq" by default, whic= h is severely underdocumented, and the features it has are rarely implement= ed by packet classification before
it hits that qdisc, and it seems mostly intended to take advantage of hardw= are QoS, (which at least on wireless n is something of a lose anyway - aggr= egation is better)

Result: On a busy wireless network, mission criti= cal, statistically rare packets, are getting delayed or lost.

If it already wasn't apparent to me before I started on this trip a= few weeks ago,=A0 it is glaringly obvious now.

Yesterday I was get= ting 3.5KB(!)/sec throughput on Georgia Tech's=20 internal wireless lan. I could "hear" over 32 access points.
<= br>And everywhere I've been, hotels, airports, etc, I've seen DHCP take multiple attempts to get an address, and DNS timing out a lot. I'm sure others have observed this sort of behavior!!!

On = a busy wireless lan, and/or one with many access points and clients on the = same channel, DHCP queries go unanswered, ARP!! times out, DNS times out, R= A and ND fail, NTP gets delayed... While some of this is due to interferenc= e (my god, "hearing" 32+APs!), I believe that a great deal of it = is due to large queue lengths and contention for the link the user is conne= cted to.

The default behavior for many of these protocols is to re-send the pack= ets after a short time - thus adding to the problem!

So...

Fe= lix Fietkau reduced queue lengths significantly in the ath9k wireless drive= rs used in the uberwrt/bismark/iscswrt projects and that has helped a lot (= still quite a ways to go to match my original patch but getting packet aggr= egation to work well requires larger queues)

On the current build of bismark, I get 32ms avg delay on a latency unde= r load test, on an access point I'm connected to at 54Mbit/second. I= 9;m going to start doing connects at slower speeds today and expect the res= ults to be rather awful.

While this is not good (my old patches got around 4ms, debloat-testing = hits around 12ms), this is STILL better than what many are seeing in the fi= eld...

So...

I think some *more* of the problems can be fixed= by packet prioritization on the wireless side.

So I started working on a set of wireless QoS ideas to see if I could i= mprove the situation. (wilst working on red for external, wired interfaces)=

I started off with a hard case - a broadcast multicast protocol tha= t runs over IPv6 called babel. It has the interesting property in that it s= ends a broadcast every 20 seconds from an identifiable address (theoretical= ly - see the fe80 discussion also on this list) and thus I can easily (or s= o I thought) observe, with enough nodes - delays and packet loss.

The idea I'm toying with now is to reserve a few percentage points = of the bandwidth to mission critical packets such as those listed so far, u= sing packet marking and a queuing discipline that does more of the "ri= ght thing" (hsfc or htb+sfb)

(strict prioritization of things like ping would lead to trivial DDOSes= on any of these protocols, so a tiny bandwidth reserve strikes me as a bet= ter approach)

Wireless multicast is additionally weird in that it fa= lls back to a very low rate in order to do its work.

Suggestions? comments? The current build of the "capetown" re= lease of bismark is very usable, has many cool features, including SFB, and= supports several off the shelf routers commonly available via best-buy and= new-egg.

Anyone that's interested, please join in the effort. Certainly mere= qdiscs are something more folk can fiddle with at a scripting and tcpdump = level...

Bismark documentation (supports Netgear wndr3700, wndr3700v= 2, nanostation M5, buffalo, and I just build an x86 kvm virtual machine tha= t needs love)

http://www= .bufferbloat.net/projects/bismark/wiki

Capetown builds:
http://huchra.bufferbloat.ne= t/~bismark/

--
Dave T=E4ht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://the-edge.blog= spot.com
--90e6ba614b529593ad04a301032a--