[Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Richard Brown]

Folks,

I am planning to give a talk about Bufferbloat to the local Linux User Group next week (http://dlslug.org). All this traffic on the list is fantastic, because it gives me a lot of background on the current state of bufferbloat. I've pulled together a bunch of general questions about CeroWrt that I would like to be able to cover if they come up:

- Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from mid-September? (My router is using this build - r33460.) 

- I see the "QoS" item in the Network tab of the web GUI. Is this important for Sugarland? Or does some other router configuration take care of this now?

- What's the relationship between the QoS GUI item above and the debloat.sh and simple_qos.sh scripts that have been mentioned on this list? What's the best practice here for getting a router up and running?

- I can see how the CeroWrt de-bloating algorithms help protect against bad latency when I'm *uploading* big files. I'm not sure whether using CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files, though. What can I say about this?

- I believe the default DNS server in Sugarland is dnsmasq, not bind. Is DNSSEC enabled by default? Also: there's a report (Bug #411) that says that DNS is leaking internal names to the outside world. What's the best advice for closing this? ("list notinterface 'ge00'" is one recommendation…)

- I've been assembling information about the various de-bloating techniques implemented in CeroWrt. It seems that Infoblox has recently reorganized their blogs, and the links published earlier this week have all broken. Here are updates:

http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
http://www.infoblox.com/community/blog/router-buffer-tuning
http://www.infoblox.com/community/blog/rethinking-interface-error-reports

My plan is to give a little of the science behind bufferbloat mitigation and also put in a plug for CeroWrt. Any topics I haven't already mentioned that I should? Thanks!

Rich Brown
Hanover, NH USA

Re: [Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Dave Taht]

All of Jims presos are licences under one of the creative commons licenses.

Mine would be, too, if I had bothered to mark them as such. I will go
back and correct this. Feel free to reuse whatever you like, with the
caveat that all these presos do reflect enhanced thinking and research
over time, and sometimes contain errors or ideas that have been
revised.

Original libreoffice and powerpoint sources for jim's stuff are at
http://mirrors.bufferbloat.net/

Mine are mostly at: http://www.teklibre.com/~d/bloat/Talks/ - the
linuxcon talk is not up in source form because I wanted to illustrate
several of the graphs better.



On Mon, Nov 26, 2012 at 2:11 AM, Richard Brown
<richard.e.brown@dartware.com> wrote:
> Folks,
>
> I am planning to give a talk about Bufferbloat to the local Linux User Group next week (http://dlslug.org). All this traffic on the list is fantastic, because it gives me a lot of background on the current state of bufferbloat. I've pulled together a bunch of general questions about CeroWrt that I would like to be able to cover if they come up:
>
> - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from mid-September? (My router is using this build - r33460.)

Yes. A lot of updates have accumulated, I do hope to get a new release
out soon and start up 3.6.x+ development. I'm extremely encouraged by
what is in 3.6 and later.

I note there is a nasty hole in the sugarland config that new users
should close immediately on a new install.

http://www.bufferbloat.net/issues/411

> - I see the "QoS" item in the Network tab of the web GUI. Is this important for Sugarland? Or does some other router configuration take care of this now?

Openwrt QoS works ok on ipv4 traffic. (yes, it uses fq_codel). However
the simple_qos.sh script I have been prototyping works on everything
and seems to be mildly better. In part that's due to handling ipv6
traffic correctly, and possibly due to the fact it has "nfq_codel" in
it, rather than fq_codel, which is a respin of fq_codel using the
latest ns2 model of codel behind it, which differs in some important
respects.

It was obvious after working on that that better benchmarks were
needed, which has been a focus these past 2 months.

simple_qos.sh does require manual configuration, however, and testing
for an accurate bandwidth estimate is problematic even with the rrul
test. I would really like someone to make this available via a gui,
and more people to beat on it...

(and I'll rename this ceroshaper in the next release)

And: I would prefer to stress to all audiences that this stuff is WIP,
and try to get them to understand that the goal is to do smart
queueing over top the smallest possible amount of buffering, and a
multitude of factors can lead to sub-optimal results.

rather than "install this script on your random hardware, expect
magic". Or: "Download this firmware, expect magic".

As one example, I was completely blindsided the other day, when
working with david woodhouse, by the new GRO network offload
"feature", on his device, which wedges enormous packets into the
stack. That needs to get turned off on linux based routers, generally.
Somehow. Preferably, automatically.

(I am delighted that multiple ADSL folk, including david, just made a
huge dent in adsl latencies over on the netdev mailing list)

> - What's the relationship between the QoS GUI item above and the debloat.sh and simple_qos.sh scripts that have been mentioned on this list? What's the best practice here for getting a router up and running?

the debloat (written in lua) script in the ceropackages repository
makes sure that various latency inducing network offloads (on by
default in linux) are OFF. I recently learned that the GRO offload
could induce a problem, I don't remember if that's off in sugarland.
It runs on ALL interfaces by default in cerowrt....

People trying to make debloating a router work on other platforms than
cerowrt, should modify that script to run on all their interfaces.
Or...

debloat.sh is a simpler version of debloat that doesn't do all the
stuff debloat does, yet, but works on debian/ubuntu. It needs to be
better, turning off gro for example on ALL devices, and I like that it
lacks the lua dependency.

I incidentally note that I no longer remember what's in sugarland for
simple_qos and debloat, what I am fiddling with now is not checked in.
I think a difference is that I'm using a different quantum, am only
using ecn on ingress.

>
> - I can see how the CeroWrt de-bloating algorithms help protect against bad latency when I'm *uploading* big files. I'm not sure whether using CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files, though. What can I say about this?

Most shapers in the world try hard to address ingress and egress. It
is very byzantine to configure the default ingress shaper "IFB", but
that's what is done.

Shaping ingress traffic has been one of the few places where I have
seen ECN give a benefit. ECN on egress, seems to be a lose at most
bandwidths I've tried.

The right place for ingress shaping, however, really is at the head
end or dslam.


>
> - I believe the default DNS server in Sugarland is dnsmasq,

Bind ate way too much memory. Writing a gui for it proved nearly
impossible. Dnsmasq has evolved over the past year to include a nifty
AAAA and dhcpv6 naming scheme (and added dhcpv6, and has nearly enough
functionality to replace the radvd daemon now). I'm very happy with
the directions dnsmasq is going.

(I'd really like to get some help on getting AHCP into it)

> not bind. Is DNSSEC enabled by default?

DNSSEC is temporarily gone.

Simon kelly of dnsmasq is making some progress towards making it fit
into dnsmasq. As he lacks time and resources on his side, I can't give
an estimate as to arrival time, but I'm sure it will be a lot more
tiny and more web configurable, whenever it's done. There's partial
DNSSEC support in a branch of dnsmasq, patches would be welcomed....

I learned this week that dnsmasq is the default dns/dhcp server in
android, for tethering, and is also heavily used in libvirt and
various clustering solutions. That's in addition to near dominance of
the linux based home router market...

And for all that market penetration, (at least 1/4 of the internet)
simon and his crew have made such solid software, in their spare time,
as for nobody to know who they are. It's sad, and frustrating...

I admit that I like a few features of bind, such as views, and
nsupdate... Bind can be optionally installed via installing
bind-latest and bind-latest-config via opkg.
I certainly plan to leave bind available for those that want it, but
going forward,
unless MBs of ram drop from the sky for free for everyone, it's not going to be
the default in cerowrt.

> Also: there's a report (Bug #411) that says that DNS is leaking internal names to the outside world. What's the best advice for closing this? ("list notinterface 'ge00'" is one recommendation…)

yes. EVERYBODY please do that.

>
> - I've been assembling information about the various de-bloating techniques implemented in CeroWrt. It seems that Infoblox has recently reorganized their blogs, and the links published earlier this week have all broken. Here are updates:
>
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> http://www.infoblox.com/community/blog/router-buffer-tuning
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports

Thx.

> My plan is to give a little of the science behind bufferbloat mitigation and also put in a plug for CeroWrt. Any topics I haven't already mentioned that I should? Thanks!
>
> Rich Brown
> Hanover, NH USA
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 9076 bytes --]

On Mon, Nov 26, 2012 at 6:49 AM, Dave Taht <dave.taht@gmail.com> wrote:

> All of Jims presos are licences under one of the creative commons licenses.
>
> Mine would be, too, if I had bothered to mark them as such. I will go
> back and correct this. Feel free to reuse whatever you like, with the
> caveat that all these presos do reflect enhanced thinking and research
> over time, and sometimes contain errors or ideas that have been
> revised.
>
> Original libreoffice and powerpoint sources for jim's stuff are at
> http://mirrors.bufferbloat.net/


Me bad; I haven't uploaded more recent versions of my talks.  I'll add it
to my "todo list", to do RSN...
                                - Jim


>
> Mine are mostly at: http://www.teklibre.com/~d/bloat/Talks/
>
> - the
> linuxcon talk is not up in source form because I wanted to illustrate
> several of the graphs better.
>
>
>
> On Mon, Nov 26, 2012 at 2:11 AM, Richard Brown
> <richard.e.brown@dartware.com> wrote:
> > Folks,
> >
> > I am planning to give a talk about Bufferbloat to the local Linux User
> Group next week (http://dlslug.org
>
> ). All this traffic on the list is fantastic, because it gives me a lot of
> background on the current state of bufferbloat. I've pulled together a
> bunch of general questions about CeroWrt that I would like to be able to
> cover if they come up:
> >
> > - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from
> mid-September? (My router is using this build - r33460.)
>
> Yes. A lot of updates have accumulated, I do hope to get a new release
> out soon and start up 3.6.x+ development. I'm extremely encouraged by
> what is in 3.6 and later.
>
> I note there is a nasty hole in the sugarland config that new users
> should close immediately on a new install.
>
> http://www.bufferbloat.net/issues/411
>
>
>
> > - I see the "QoS" item in the Network tab of the web GUI. Is this
> important for Sugarland? Or does some other router configuration take care
> of this now?
>
> Openwrt QoS works ok on ipv4 traffic. (yes, it uses fq_codel). However
> the simple_qos.sh script I have been prototyping works on everything
> and seems to be mildly better. In part that's due to handling ipv6
> traffic correctly, and possibly due to the fact it has "nfq_codel" in
> it, rather than fq_codel, which is a respin of fq_codel using the
> latest ns2 model of codel behind it, which differs in some important
> respects.
>
> It was obvious after working on that that better benchmarks were
> needed, which has been a focus these past 2 months.
>
> simple_qos.sh does require manual configuration, however, and testing
> for an accurate bandwidth estimate is problematic even with the rrul
> test. I would really like someone to make this available via a gui,
> and more people to beat on it...
>
> (and I'll rename this ceroshaper in the next release)
>
> And: I would prefer to stress to all audiences that this stuff is WIP,
> and try to get them to understand that the goal is to do smart
> queueing over top the smallest possible amount of buffering, and a
> multitude of factors can lead to sub-optimal results.
>
> rather than "install this script on your random hardware, expect
> magic". Or: "Download this firmware, expect magic".
>
> As one example, I was completely blindsided the other day, when
> working with david woodhouse, by the new GRO network offload
> "feature", on his device, which wedges enormous packets into the
> stack. That needs to get turned off on linux based routers, generally.
> Somehow. Preferably, automatically.
>
> (I am delighted that multiple ADSL folk, including david, just made a
> huge dent in adsl latencies over on the netdev mailing list)
>
> > - What's the relationship between the QoS GUI item above and the
> debloat.sh and simple_qos.sh scripts that have been mentioned on this list?
> What's the best practice here for getting a router up and running?
>
> the debloat (written in lua) script in the ceropackages repository
> makes sure that various latency inducing network offloads (on by
> default in linux) are OFF. I recently learned that the GRO offload
> could induce a problem, I don't remember if that's off in sugarland.
> It runs on ALL interfaces by default in cerowrt....
>
> People trying to make debloating a router work on other platforms than
> cerowrt, should modify that script to run on all their interfaces.
> Or...
>
> debloat.sh is a simpler version of debloat that doesn't do all the
> stuff debloat does, yet, but works on debian/ubuntu. It needs to be
> better, turning off gro for example on ALL devices, and I like that it
> lacks the lua dependency.
>
> I incidentally note that I no longer remember what's in sugarland for
> simple_qos and debloat, what I am fiddling with now is not checked in.
> I think a difference is that I'm using a different quantum, am only
> using ecn on ingress.
>
> >
> > - I can see how the CeroWrt de-bloating algorithms help protect against
> bad latency when I'm *uploading* big files. I'm not sure whether using
> CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files,
> though. What can I say about this?
>
> Most shapers in the world try hard to address ingress and egress. It
> is very byzantine to configure the default ingress shaper "IFB", but
> that's what is done.
>
> Shaping ingress traffic has been one of the few places where I have
> seen ECN give a benefit. ECN on egress, seems to be a lose at most
> bandwidths I've tried.
>
> The right place for ingress shaping, however, really is at the head
> end or dslam.
>
>
> >
> > - I believe the default DNS server in Sugarland is dnsmasq,
>
> Bind ate way too much memory. Writing a gui for it proved nearly
> impossible. Dnsmasq has evolved over the past year to include a nifty
> AAAA and dhcpv6 naming scheme (and added dhcpv6, and has nearly enough
> functionality to replace the radvd daemon now). I'm very happy with
> the directions dnsmasq is going.
>
> (I'd really like to get some help on getting AHCP into it)
>
> > not bind. Is DNSSEC enabled by default?
>
> DNSSEC is temporarily gone.
>
> Simon kelly of dnsmasq is making some progress towards making it fit
> into dnsmasq. As he lacks time and resources on his side, I can't give
> an estimate as to arrival time, but I'm sure it will be a lot more
> tiny and more web configurable, whenever it's done. There's partial
> DNSSEC support in a branch of dnsmasq, patches would be welcomed....
>
> I learned this week that dnsmasq is the default dns/dhcp server in
> android, for tethering, and is also heavily used in libvirt and
> various clustering solutions. That's in addition to near dominance of
> the linux based home router market...
>
> And for all that market penetration, (at least 1/4 of the internet)
> simon and his crew have made such solid software, in their spare time,
> as for nobody to know who they are. It's sad, and frustrating...
>
> I admit that I like a few features of bind, such as views, and
> nsupdate... Bind can be optionally installed via installing
> bind-latest and bind-latest-config via opkg.
> I certainly plan to leave bind available for those that want it, but
> going forward,
> unless MBs of ram drop from the sky for free for everyone, it's not going
> to be
> the default in cerowrt.
>
> > Also: there's a report (Bug #411) that says that DNS is leaking internal
> names to the outside world. What's the best advice for closing this? ("list
> notinterface 'ge00'" is one recommendation…)
>
> yes. EVERYBODY please do that.
>
> >
> > - I've been assembling information about the various de-bloating
> techniques implemented in CeroWrt. It seems that Infoblox has recently
> reorganized their blogs, and the links published earlier this week have all
> broken. Here are updates:
> >
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
>
>
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
>
>
> > http://www.infoblox.com/community/blog/router-buffer-tuning
>
>
> >
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports
>
>
>
> Thx.
>
> > My plan is to give a little of the science behind bufferbloat mitigation
> and also put in a plug for CeroWrt. Any topics I haven't already mentioned
> that I should? Thanks!
> >
> > Rich Brown
> > Hanover, NH USA
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt:
> http://www.teklibre.com/cerowrt/subscribe.html
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>

[-- Attachment #2: Type: text/html, Size: 12149 bytes --]

Re: [Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Michael Richardson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 2170 bytes --]


>>>>> "Richard" == Richard Brown <richard.e.brown@dartware.com> writes:
    Richard> - I can see how the CeroWrt de-bloating algorithms help
    Richard> protect against bad latency when I'm *uploading* big
    Richard> files. I'm not sure whether using CeroWrt with its
    Richard> CoDel/FQ/SFQ/etc. helps when I'm downloading big files,
    Richard> though. What can I say about this? 

If the link from the broadband to the laptop is wireless, than it's
quite possible that the wireless link experiences bufferbloat. 
This would be true:
     - if the laptop is far from the base station the rate could be
       lower than the broadband download link. (Especially now that
       cable offers 50Mb/s downlinks...)
     - if the wireless is bridged to wired, and there are many windows
       boxes, broadcasting a lot, then the wireless link may be
       otherwise saturated

bad uplink latency will affect TCP ACKs, and can totally ruin your
interactive ssh day too.

But, in general, either the ISP has to debloat too, or it has to rate
limit to below the actual bandwidth.

    Richard> - I believe the default DNS server in Sugarland is dnsmasq,
    Richard> not bind. Is DNSSEC enabled by default? Also: there's a
    Richard> report (Bug #411) that says that DNS is leaking internal
    Richard> names to the outside world. What's the best advice for
    Richard> closing this? ("list notinterface 'ge00'" is one
    Richard> recommendation…) 

(In general, leaking names is really not that much of a worry...)

    Richard> My plan is to give a little of the science behind
    Richard> bufferbloat mitigation and also put in a plug for
    Richard> CeroWrt. Any topics I haven't already mentioned that I
    Richard> should? Thanks! 

Use the fountain images that Van Jacobson used at IETF84.

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition. 

Re: [Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Dave Taht]

On Mon, Nov 26, 2012 at 1:26 PM, Michael Richardson <mcr@sandelman.ca> wrote:
>
>>>>>> "Richard" == Richard Brown <richard.e.brown@dartware.com> writes:
>     Richard> - I can see how the CeroWrt de-bloating algorithms help
>     Richard> protect against bad latency when I'm *uploading* big
>     Richard> files. I'm not sure whether using CeroWrt with its
>     Richard> CoDel/FQ/SFQ/etc. helps when I'm downloading big files,
>     Richard> though. What can I say about this?
>
> If the link from the broadband to the laptop is wireless, than it's
> quite possible that the wireless link experiences bufferbloat.
> This would be true:
>      - if the laptop is far from the base station the rate could be
>        lower than the broadband download link. (Especially now that
>        cable offers 50Mb/s downlinks...)
>      - if the wireless is bridged to wired, and there are many windows
>        boxes, broadcasting a lot, then the wireless link may be
>        otherwise saturated

One point of the rrul tests are that netserver runs out of xinetd on
the router itself,
so it's possible to test wifi performance in the presence of multiple workloads.

 However, the overhead of running netserver on such a small box is too
extreme, presently. I hope to produce a simpler test that can, indeed,
work right on cerowrt, so you can easily diagnose the inside path on
your network.

You can certainly install netperf 2.6 or later on a heftier box,
locally on your network, and test wifi and wired that way.

>
> bad uplink latency will affect TCP ACKs, and can totally ruin your
> interactive ssh day too.

s/can/does

> But, in general, either the ISP has to debloat too, or it has to rate
> limit to below the actual bandwidth.

Rate limiting below the ISP's provided downlink-to-you bandwidth does
work, but tends to chop off 10-15% of what the ISP claims they are
providing.

>     Richard> - I believe the default DNS server in Sugarland is dnsmasq,
>     Richard> not bind. Is DNSSEC enabled by default? Also: there's a
>     Richard> report (Bug #411) that says that DNS is leaking internal
>     Richard> names to the outside world. What's the best advice for
>     Richard> closing this? ("list notinterface 'ge00'" is one
>     Richard> recommendation…)
>
> (In general, leaking names is really not that much of a worry...)

Names, no. Amplification attacks are a serious problem with DNS.

The internet is rife with worms and daemons that are leveraging open
dns servers to amplification attacks. In a few short weeks that macej
had left the port open,

http://www.bufferbloat.net/issues/411

"Having DNS open for a while made some evil forces notice it and use my IP
for DNS amplification attacks. I secured dnsmasq not to listen on ge00, but
I'm still getting over 300 UDP packets/s!"

I really hate having contributed to this problem with sugarland.
Nobody wants an extra 300 packets/s hitting their home network for any
reason. Please close this immediately upon installing sugarland.

I've tried very hard to respond to CVEs over the course of this
project (bind alone, had 5), but I'm away from the lab, in the middle
of a trip, in between a major upgrade of functionality to cerowrt and
trying to get funding to re-invigorate this project.

I haven't had much time to hack. None to test.

I would to get to where we had infrastructure to easily create, test,
and push out security related fixes.

>
>     Richard> My plan is to give a little of the science behind
>     Richard> bufferbloat mitigation and also put in a plug for
>     Richard> CeroWrt. Any topics I haven't already mentioned that I
>     Richard> should? Thanks!
>
> Use the fountain images that Van Jacobson used at IETF84.

In my own preso at the lincs, I used my coffee cup...

There is an interesting preso that shemminger is using that uses soda
bottles to do something similar to both concepts. Jamming holes into
it randomly to simulate red....

I may adopt this - however in explaining fq_codel, I think I need to
add multiple cups, and an eye-dropper for the ant packets.
>
> --
> ]       He who is tired of Weird Al is tired of life!           |  firewalls  [
> ]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
> ] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
>    Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
>                        then sign the petition.
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Bufferbloat at upcoming LUG talk

[Michael Richardson]

>>>>> "Dave" == Dave Taht <dave.taht@gmail.com> writes:
    >> (In general, leaking names is really not that much of a worry...)

    Dave> Names, no. Amplification attacks are a serious problem with DNS.

    Dave> The internet is rife with worms and daemons that are leveraging open
    Dave> dns servers to amplification attacks. In a few short weeks that macej
    Dave> had left the port open,

so, there is a difference between leaking names, and providing recursive
service to everyone...

    Dave> I've tried very hard to respond to CVEs over the course of this
    Dave> project (bind alone, had 5), but I'm away from the lab, in the middle
    Dave> of a trip, in between a major upgrade of functionality to cerowrt and
    Dave> trying to get funding to re-invigorate this project.

Understood.

    Dave> I haven't had much time to hack. None to test.

    Dave> I would to get to where we had infrastructure to easily create, test,
    Dave> and push out security related fixes.

I wonder if part of the issue is that flashing hardware is a pain in the
butt to do automagically.  Were I able to spend paid time on this, I'd
want to do a cerowrt build for x86, or some other trivially virtualized
processor.

    >> Use the fountain images that Van Jacobson used at IETF84.

    Dave> In my own preso at the lincs, I used my coffee cup...

    Dave> There is an interesting preso that shemminger is using that uses soda
    Dave> bottles to do something similar to both concepts. Jamming holes into
    Dave> it randomly to simulate red....

    Dave> I may adopt this - however in explaining fq_codel, I think I need to
    Dave> add multiple cups, and an eye-dropper for the ant packets.

sure... in explaining fq_codel, I can see that you need more than the
fountain.
I think that most people need bufferbloat clearly articulated before
they will see that there is a problem that needs fixing, and most
non-network people are still there. 

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] mcr@sandelman.ottawa.on.ca http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition. 

Re: [Cerowrt-devel] Bufferbloat at LUG talk - Meeting Report

[Richard Brown]

Folks,

I gave the talk to the local Linux User Group on Thursday, and it went really well. Two people came up to me after the talk and said, in effect, "You know, I think I've seen this. But I've always blamed something else." Their experience:

- Attempting to Skype with a bunch of web browser tabs open gives bad results. Closing the tabs made things better. (They been blaming the browser for "using too much memory". Now it's possible to think that it's a network problem.)

- Another person reported that his network connection (wireless ISP, two hops to a wired network) seemed to work OK as long as his household was mostly downloading. But uploading much of anything really made things bad.

I posted the slides at http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf

Rich

PS I've updated the CeroWrt site to include links to a bunch of relevant videos. (http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos) Please let me know if there are others that we should point to.

On Nov 25, 2012, at 8:11 PM, Richard E. Brown <rbrown@intermapper.com> wrote:

> Folks,
> 
> I am planning to give a talk about Bufferbloat to the local Linux User Group next week (http://dlslug.org). All this traffic on the list is fantastic, because it gives me a lot of background on the current state of bufferbloat. I've pulled together a bunch of general questions about CeroWrt that I would like to be able to cover if they come up:
> 
> - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from mid-September? (My router is using this build - r33460.) 
> 
> - I see the "QoS" item in the Network tab of the web GUI. Is this important for Sugarland? Or does some other router configuration take care of this now?
> 
> - What's the relationship between the QoS GUI item above and the debloat.sh and simple_qos.sh scripts that have been mentioned on this list? What's the best practice here for getting a router up and running?
> 
> - I can see how the CeroWrt de-bloating algorithms help protect against bad latency when I'm *uploading* big files. I'm not sure whether using CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files, though. What can I say about this?
> 
> - I believe the default DNS server in Sugarland is dnsmasq, not bind. Is DNSSEC enabled by default? Also: there's a report (Bug #411) that says that DNS is leaking internal names to the outside world. What's the best advice for closing this? ("list notinterface 'ge00'" is one recommendation…)
> 
> - I've been assembling information about the various de-bloating techniques implemented in CeroWrt. It seems that Infoblox has recently reorganized their blogs, and the links published earlier this week have all broken. Here are updates:
> 
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> http://www.infoblox.com/community/blog/router-buffer-tuning
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports
> 
> My plan is to give a little of the science behind bufferbloat mitigation and also put in a plug for CeroWrt. Any topics I haven't already mentioned that I should? Thanks!
> 
> Rich Brown
> Hanover, NH USA

Re: [Cerowrt-devel] Bufferbloat at LUG talk - Meeting Report

[Maciej Soltysiak]

[-- Attachment #1: Type: text/plain, Size: 4723 bytes --]

Excellent job Richard! Those slides are very clean and informative and you
got fantastic real life user reports!
Point #1 is very common: lots of behind-the-scenes javascript, buffering,
asynchronous requests, facebook chat box and updates, etc.

I was trying to make a mock conversation for the purpose of providing a
story backing up the debloating efforts so that end users realize better
what's going on. Please, guys, have a look and comment:
https://soltysiak.com/wiki/index.php/BB_dialog

Part 1 is an intro, also touching on tiered ISP services. Part 2 would be
what bufferbloat is all about. Part 3 is an outro to have the users have a
take-home message, also touching on DPI and other evil stuff ISPs do trying
to workaround the issues.

You can edit that wiki.

I couldn't post it on bufferbloat.net wiki because I don't seem to have
privilege to create new pages so I setup my own.

Regards,
Maciej


On Sun, Dec 9, 2012 at 5:56 PM, Richard Brown
<richard.e.brown@dartware.com>wrote:

> Folks,
>
> I gave the talk to the local Linux User Group on Thursday, and it went
> really well. Two people came up to me after the talk and said, in effect,
> "You know, I think I've seen this. But I've always blamed something else."
> Their experience:
>
> - Attempting to Skype with a bunch of web browser tabs open gives bad
> results. Closing the tabs made things better. (They been blaming the
> browser for "using too much memory". Now it's possible to think that it's a
> network problem.)
>
> - Another person reported that his network connection (wireless ISP, two
> hops to a wired network) seemed to work OK as long as his household was
> mostly downloading. But uploading much of anything really made things bad.
>
> I posted the slides at
> http://www.bufferbloat.net/attachments/download/148/Bufferbloat-DLSLUG-Dec2012.pdf
>
> Rich
>
> PS I've updated the CeroWrt site to include links to a bunch of relevant
> videos. (http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos)
> Please let me know if there are others that we should point to.
>
> On Nov 25, 2012, at 8:11 PM, Richard E. Brown <rbrown@intermapper.com>
> wrote:
>
> > Folks,
> >
> > I am planning to give a talk about Bufferbloat to the local Linux User
> Group next week (http://dlslug.org). All this traffic on the list is
> fantastic, because it gives me a lot of background on the current state of
> bufferbloat. I've pulled together a bunch of general questions about
> CeroWrt that I would like to be able to cover if they come up:
> >
> > - Is it true that the latest CeroWrt is Sugarland 3.3.8-26 from
> mid-September? (My router is using this build - r33460.)
> >
> > - I see the "QoS" item in the Network tab of the web GUI. Is this
> important for Sugarland? Or does some other router configuration take care
> of this now?
> >
> > - What's the relationship between the QoS GUI item above and the
> debloat.sh and simple_qos.sh scripts that have been mentioned on this list?
> What's the best practice here for getting a router up and running?
> >
> > - I can see how the CeroWrt de-bloating algorithms help protect against
> bad latency when I'm *uploading* big files. I'm not sure whether using
> CeroWrt with its CoDel/FQ/SFQ/etc. helps when I'm downloading big files,
> though. What can I say about this?
> >
> > - I believe the default DNS server in Sugarland is dnsmasq, not bind. Is
> DNSSEC enabled by default? Also: there's a report (Bug #411) that says that
> DNS is leaking internal names to the outside world. What's the best advice
> for closing this? ("list notinterface 'ge00'" is one recommendation…)
> >
> > - I've been assembling information about the various de-bloating
> techniques implemented in CeroWrt. It seems that Infoblox has recently
> reorganized their blogs, and the links published earlier this week have all
> broken. Here are updates:
> >
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-1
> >
> http://www.infoblox.com/community/blog/application-analysis-using-tcp-retransmissions-part-2
> > http://www.infoblox.com/community/blog/router-buffer-tuning
> >
> http://www.infoblox.com/community/blog/rethinking-interface-error-reports
> >
> > My plan is to give a little of the science behind bufferbloat mitigation
> and also put in a plug for CeroWrt. Any topics I haven't already mentioned
> that I should? Thanks!
> >
> > Rich Brown
> > Hanover, NH USA
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 6200 bytes --]

Re: [Cerowrt-devel] Bufferbloat at LUG talk - Meeting Report

[Richard Brown]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

Hello Maciej,

Thanks for the kind words about my presentation.

Your imagined dialog is spot on. It's a good way to let people recognize this common situation, and what the fix might be.

Best regards,

Rich

On Dec 9, 2012, at 12:32 PM, Maciej Soltysiak <maciej@soltysiak.com<mailto:maciej@soltysiak.com>> wrote:

Excellent job Richard! Those slides are very clean and informative and you got fantastic real life user reports!
Point #1 is very common: lots of behind-the-scenes javascript, buffering, asynchronous requests, facebook chat box and updates, etc.

I was trying to make a mock conversation for the purpose of providing a story backing up the debloating efforts so that end users realize better what's going on. Please, guys, have a look and comment:
https://soltysiak.com/wiki/index.php/BB_dialog

Part 1 is an intro, also touching on tiered ISP services. Part 2 would be what bufferbloat is all about. Part 3 is an outro to have the users have a take-home message, also touching on DPI and other evil stuff ISPs do trying to workaround the issues.

You can edit that wiki.

I couldn't post it on bufferbloat.net<http://bufferbloat.net/> wiki because I don't seem to have privilege to create new pages so I setup my own.

Regards,
Maciej


[-- Attachment #2: Type: text/html, Size: 1906 bytes --]