[Cerowrt-devel] Bufferbloat at upcoming LUG talk

Mon Nov 26 06:49:38 EST 2012

All of Jims presos are licences under one of the creative commons licenses.

Mine would be, too, if I had bothered to mark them as such. I will go
back and correct this. Feel free to reuse whatever you like, with the
caveat that all these presos do reflect enhanced thinking and research
over time, and sometimes contain errors or ideas that have been
revised.

Original libreoffice and powerpoint sources for jim's stuff are at
http://mirrors.bufferbloat.net/

Mine are mostly at: http://www.teklibre.com/~d/bloat/Talks/ - the
linuxcon talk is not up in source form because I wanted to illustrate
several of the graphs better.

On Mon, Nov 26, 2012 at 2:11 AM, Richard Brown
<richard.e.brown at dartware.com> wrote:
> Folks,
>
> I am planning to give a talk about Bufferbloat to the local Linux User Group next week (http://dlslug.org). All this traffic on the list is fantastic, because it gives me a lot of background on the current state of bufferbloat. I've pulled together a bunch of general questions about CeroWrt that I would like to be able to cover if they come up:
>
> - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from mid-September? (My router is using this build - r33460.)

Yes. A lot of updates have accumulated, I do hope to get a new release
out soon and start up 3.6.x+ development. I'm extremely encouraged by
what is in 3.6 and later.

I note there is a nasty hole in the sugarland config that new users
should close immediately on a new install.

http://www.bufferbloat.net/issues/411

> - I see the "QoS" item in the Network tab of the web GUI. Is this important for Sugarland? Or does some other router configuration take care of this now?

Openwrt QoS works ok on ipv4 traffic. (yes, it uses fq_codel). However
the simple_qos.sh script I have been prototyping works on everything
and seems to be mildly better. In part that's due to handling ipv6
traffic correctly, and possibly due to the fact it has "nfq_codel" in
it, rather than fq_codel, which is a respin of fq_codel using the
latest ns2 model of codel behind it, which differs in some important
respects.

It was obvious after working on that that better benchmarks were
needed, which has been a focus these past 2 months.

simple_qos.sh does require manual configuration, however, and testing
for an accurate bandwidth estimate is problematic even with the rrul
test. I would really like someone to make this available via a gui,
and more people to beat on it...

(and I'll rename this ceroshaper in the next release)

And: I would prefer to stress to all audiences that this stuff is WIP,
and try to get them to understand that the goal is to do smart
queueing over top the smallest possible amount of buffering, and a
multitude of factors can lead to sub-optimal results.

rather than "install this script on your random hardware, expect
magic". Or: "Download this firmware, expect magic".

As one example, I was completely blindsided the other day, when
working with david woodhouse, by the new GRO network offload
"feature", on his device, which wedges enormous packets into the
stack. That needs to get turned off on linux based routers, generally.
Somehow. Preferably, automatically.

(I am delighted that multiple ADSL folk, including david, just made a
huge dent in adsl latencies over on the netdev mailing list)

> - What's the relationship between the QoS GUI item above and the debloat.sh and simple_qos.sh scripts that have been mentioned on this list? What's the best practice here for getting a router up and running?

the debloat (written in lua) script in the ceropackages repository
makes sure that various latency inducing network offloads (on by
default in linux) are OFF. I recently learned that the GRO offload
could induce a problem, I don't remember if that's off in sugarland.
It runs on ALL interfaces by default in cerowrt....

People trying to make debloating a router work on other platforms than
cerowrt, should modify that script to run on all their interfaces.
Or...

debloat.sh is a simpler version of debloat that doesn't do all the
stuff debloat does, yet, but works on debian/ubuntu. It needs to be
better, turning off gro for example on ALL devices, and I like that it
lacks the lua dependency.

I incidentally note that I no longer remember what's in sugarland for
simple_qos and debloat, what I am fiddling with now is not checked in.
I think a difference is that I'm using a different quantum, am only
using ecn on ingress.

>
> - I can see how the CeroWrt de-bloating algorithms help protect against bad latency when I'm *uploading* big files. I'm not sure whether using CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files, though. What can I say about this?

Most shapers in the world try hard to address ingress and egress. It
is very byzantine to configure the default ingress shaper "IFB", but
that's what is done.

Shaping ingress traffic has been one of the few places where I have
seen ECN give a benefit. ECN on egress, seems to be a lose at most
bandwidths I've tried.

The right place for ingress shaping, however, really is at the head
end or dslam.

>
> - I believe the default DNS server in Sugarland is dnsmasq,

Bind ate way too much memory. Writing a gui for it proved nearly
impossible. Dnsmasq has evolved over the past year to include a nifty
AAAA and dhcpv6 naming scheme (and added dhcpv6, and has nearly enough
functionality to replace the radvd daemon now). I'm very happy with
the directions dnsmasq is going.

(I'd really like to get some help on getting AHCP into it)

> not bind. Is DNSSEC enabled by default?

DNSSEC is temporarily gone.

Simon kelly of dnsmasq is making some progress towards making it fit
into dnsmasq. As he lacks time and resources on his side, I can't give
an estimate as to arrival time, but I'm sure it will be a lot more
tiny and more web configurable, whenever it's done. There's partial
DNSSEC support in a branch of dnsmasq, patches would be welcomed....

I learned this week that dnsmasq is the default dns/dhcp server in
android, for tethering, and is also heavily used in libvirt and
various clustering solutions. That's in addition to near dominance of
the linux based home router market...

And for all that market penetration, (at least 1/4 of the internet)
simon and his crew have made such solid software, in their spare time,
as for nobody to know who they are. It's sad, and frustrating...

I admit that I like a few features of bind, such as views, and
nsupdate... Bind can be optionally installed via installing
bind-latest and bind-latest-config via opkg.
I certainly plan to leave bind available for those that want it, but
going forward,
unless MBs of ram drop from the sky for free for everyone, it's not going to be
the default in cerowrt.

> Also: there's a report (Bug #411) that says that DNS is leaking internal names to the outside world. What's the best advice for closing this? ("list notinterface 'ge00'" is one recommendation…)

yes. EVERYBODY please do that.

>
> - I've been assembling information about the various de-bloating techniques implemented in CeroWrt. It seems that Infoblox has recently reorganized their blogs, and the links published earlier this week have all broken. Here are updates:
>
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> http://www.infoblox.com/community/blog/router-buffer-tuning
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports

Thx.

> My plan is to give a little of the science behind bufferbloat mitigation and also put in a plug for CeroWrt. Any topics I haven't already mentioned that I should? Thanks!
>
> Rich Brown
> Hanover, NH USA
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html