[Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[David Collier-Brown]

  I suspect not enough people are aware of the later efforts of the 
bufferbloat team, so I'm thinking of one or two articles, starting with 
LWN and an audience of aficionados.

The core community is aware of what we've done, but in my view we 
haven't converted "grandma". Grandma, as well as a whole bunch of 
ordinary engineers and partners of engineers, are dependent on debloated 
performance because they're working at home now, and competing with 
granddaughter playing video games while they're trying to hold a video call.

Right now, my colleagues at work suffer from more than a second of 
bloat-related lag. They therefore tend to speak over each other on 
con-calls, apologize, start again and talk over each other, again. After 
a little while, the picture becomes a distinctly silly one: a bunch of 
grown adults putting their hands up and waving, like little kids in 
school. No-one has called out “me, me, teacher” yet, but I expect it any 
time.

I propose we show the results in terms that we can explain to Grandma, 
specifically concentrating on functioning VOIP. I just upgraded to 
Fedora 31, and the networking is absolutely stock, so I make a perfect 
victim/guinea-pig (;-))

Who's interested?


--dave
-- 
David Collier-Brown,                     | Always do right. This will 
gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net                  |                      -- Mark Twain

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Tom Henderson]

On 8/9/20 11:27 AM, David Collier-Brown wrote:
>  I suspect not enough people are aware of the later efforts of the 
> bufferbloat team, so I'm thinking of one or two articles, starting 
> with LWN and an audience of aficionados.
>
> The core community is aware of what we've done, but in my view we 
> haven't converted "grandma". Grandma, as well as a whole bunch of 
> ordinary engineers and partners of engineers, are dependent on 
> debloated performance because they're working at home now, and 
> competing with granddaughter playing video games while they're trying 
> to hold a video call.
>
> Right now, my colleagues at work suffer from more than a second of 
> bloat-related lag. They therefore tend to speak over each other on 
> con-calls, apologize, start again and talk over each other, again. 
> After a little while, the picture becomes a distinctly silly one: a 
> bunch of grown adults putting their hands up and waving, like little 
> kids in school. No-one has called out “me, me, teacher” yet, but I 
> expect it any time.
>
> I propose we show the results in terms that we can explain to Grandma, 
> specifically concentrating on functioning VOIP. I just upgraded to 
> Fedora 31, and the networking is absolutely stock, so I make a perfect 
> victim/guinea-pig (;-))
>
> Who's interested?

Are the risks and tradeoffs well enough understood (and visible enough 
for troubleshooting) to recommend broader deployment?

I recently gave openwrt a try on some hardware that I ultimately 
concluded was insufficient for the job.  Fairly soon after changing out 
my access point, I started getting complaints of Wi-Fi dropping in my 
household, especially when someone was trying to videoconference.  I 
discovered that my AP was spontaneously rebooting, and the box was 
getting hot.

I am also wondering about what features will be lost if/when the device 
is flashed from the commercial firmware.  Does openwrt have access to 
all of the available Wi-Fi 11ac rates, and how does rate control 
compare?  The commercial devices offer proprietary media/device 
prioritization settings; does CAKE/SQM outperform them?  Many commercial 
APs or mesh systems have smartphone apps with features like parental 
controls; that kind of control will be lost.  What about 11ax support?

- Tom

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Jonathan Morton]

> Are the risks and tradeoffs well enough understood (and visible enough 
> for troubleshooting) to recommend broader deployment?
> 
> I recently gave openwrt a try on some hardware that I ultimately 
> concluded was insufficient for the job.  Fairly soon after changing out 
> my access point, I started getting complaints of Wi-Fi dropping in my 
> household, especially when someone was trying to videoconference.  I 
> discovered that my AP was spontaneously rebooting, and the box was 
> getting hot.

Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.

It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.

Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 5148 bytes --]

On 2020-08-09 5:35 p.m., Jonathan Morton wrote:

>> Are the risks and tradeoffs well enough understood (and visible enough
>> for troubleshooting) to recommend broader deployment?
>>
>> I recently gave openwrt a try on some hardware that I ultimately
>> concluded was insufficient for the job.  Fairly soon after changing out
>> my access point, I started getting complaints of Wi-Fi dropping in my
>> household, especially when someone was trying to videoconference.  I
>> discovered that my AP was spontaneously rebooting, and the box was
>> getting hot.
> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
>
> It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
>
> Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.

I'm primarily thinking of /this week's/ version of the home router 
problem (;-))

Because of the degree to which we're working from home and 
videoconferencing, a lot of low-price, medium-performance devices are 
suddenly too wimpy for their new role.

A (very!) draft version is up in Google docs, at 
https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing

Using myself as the guinea-pig, running pfifo-fast was clearly bad, 
fq_codel was better, and cake was good with a newish Fedora and the 
stock Rogers router.  It's been a while since I did rrul tests, and in 
any case, I think that to convince readers we need a very practical way 
of making it clear that they have a problem. I'm thinking that making 
VOIP fail might do the trick (;-))

The hard part, IMHO, is constructing a test that immediately 
communicates the idea that the reader has a problem, and that CAKE 
addresses it.

Returning to the hardware question, https://evenroute.com/iqrv3 seems to 
be capable of handling up to ~300 Mbit/S connections, and my ISP only 
delivers 170 (and advertises 150, which is mildly surprising!)

I just ordered one, so I'll have a 'plug in" example, along with 
reflashing my linksys for the umpty-thousandth time.

--dave

>  I suspect not enough people are aware of the later efforts of the 
> bufferbloat team, so I'm thinking of one or two articles, starting 
> with LWN and an audience of aficionados.
>
> The core community is aware of what we've done, but in my view we 
> haven't converted "grandma". Grandma, as well as a whole bunch of 
> ordinary engineers and partners of engineers, are dependent on 
> debloated performance because they're working at home now, and 
> competing with granddaughter playing video games while they're trying 
> to hold a video call.
>
> Right now, my colleagues at work suffer from more than a second of 
> bloat-related lag. They therefore tend to speak over each other on 
> con-calls, apologize, start again and talk over each other, again. 
> After a little while, the picture becomes a distinctly silly one: a 
> bunch of grown adults putting their hands up and waving, like little 
> kids in school. No-one has called out “me, me, teacher” yet, but I 
> expect it any time.
>
> I propose we show the results in terms that we can explain to Grandma, 
> specifically concentrating on functioning VOIP. I just upgraded to 
> Fedora 31, and the networking is absolutely stock, so I make a perfect 
> victim/guinea-pig (;-))
>
> Who's interested? 



-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain


[-- Attachment #2: Type: text/html, Size: 6364 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Daniel Sterling]

So I've been wanting to write up what I did to improve my home network
for a while.

Here's a quick overview:

I'm running a small laptop-class sandy bridge CPU in a small desktop
computer, running openwrt, running cake. It has two NICs -- the
built-in realtek NIC, and an old Intel gigabit NIC in the PCI slot.

Internet goes into the realtek NIC and out the Intel NIC. (WAN / LAN in openwrt)

my internet is AT&T gigabit fiber, but I throttle that heavily  with
cake (see below)

I manually apply cake with my own scripts. I'll post those on gist and
reply to this email with that info, just wanted to write this up
quickly this morning. but it's basically just, apply two simple cake
tc lines to the NICs.

For wifi I use UBNT's SOHO line -- Amplifi HD units.

it works really rather well; after some tweaking I've managed to
essentially get rid of the things that I've empirically found really
hurt home network performance:

1. wifi dead zones -- solved by using as many amplifi HD units as you
like, meshed or wired together. obviously wires are better than mesh
and a dedicated backhaul set of APs is better than mesh but mesh works
too.

2. wifi trying to use 5ghz when it's too slow and refusing to switch
to 2ghz -- solved by amplifi AP having a setting where it kicks
devices off the 5ghz network proactively to convice them to switch to
2ghz. thank you UBNT!

3. TCP not dropping enough packets. (or rather, not having good queue
management)

4. TCP (or rather, the network) dropping too many TCP packets --
streams / apps / web sites will get "stuck"

so after much tweaking, I've got cake set to 40mbit down, 20mbit up,
enforced by two cakes (one for each NIC). that's fairly low --

it's low to highly throttle bulk streams so that I  can play
latency-sensitive games with basically no jitter and low latency, even
if other people are using the wifi. even if I can't wire an xbox, I
can still get low latency gaming on wifi

but it's still high enough that we can stream HD video.

and of course low jitter and low latency across the board means good
ssh and video cal performance.

just wanted to write this up quickly to reply to this thread -- cake
really is amazing and I'd bet people would be willing to pay for a
magic box like I've set up that they can stick in between their
existing CPE and a decent AP that applies cake. or if AP vendors would
put cake in their APs themselves, that would be good too.

but as you note #1 and #2 on my list are important, even before queue
management comes into play. you have to be willing to buy a good AP
before cake really starts to matter, I think

-- Dan

On Mon, Aug 10, 2020 at 8:57 AM David Collier-Brown <davecb.42@gmail.com> wrote:
>
> On 2020-08-09 5:35 p.m., Jonathan Morton wrote:
>
> Are the risks and tradeoffs well enough understood (and visible enough
> for troubleshooting) to recommend broader deployment?
>
> I recently gave openwrt a try on some hardware that I ultimately
> concluded was insufficient for the job.  Fairly soon after changing out
> my access point, I started getting complaints of Wi-Fi dropping in my
> household, especially when someone was trying to videoconference.  I
> discovered that my AP was spontaneously rebooting, and the box was
> getting hot.
>
> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
>
> It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
>
> Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.
>
> I'm primarily thinking of this week's version of the home router problem (;-))
>
> Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.
>
> A (very!) draft version is up in Google docs, at https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing
>
> Using myself as the guinea-pig, running pfifo-fast was clearly bad, fq_codel was better, and cake was good with a newish Fedora and the stock Rogers router.  It's been a while since I did rrul tests, and in any case, I think that to convince readers we need a very practical way of making it clear that they have a problem. I'm thinking that making VOIP fail might do the trick (;-))
>
> The hard part, IMHO, is constructing a test that immediately communicates the idea that the reader has a problem, and that CAKE addresses it.
>
> Returning to the hardware question, https://evenroute.com/iqrv3 seems to be capable of handling up to ~300 Mbit/S connections, and my ISP only delivers 170 (and advertises 150, which is mildly surprising!)
>
> I just ordered one, so I'll have a 'plug in" example, along with reflashing my linksys for the umpty-thousandth time.
>
> --dave
>
>  I suspect not enough people are aware of the later efforts of the bufferbloat team, so I'm thinking of one or two articles, starting with LWN and an audience of aficionados.
>
> The core community is aware of what we've done, but in my view we haven't converted "grandma". Grandma, as well as a whole bunch of ordinary engineers and partners of engineers, are dependent on debloated performance because they're working at home now, and competing with granddaughter playing video games while they're trying to hold a video call.
>
> Right now, my colleagues at work suffer from more than a second of bloat-related lag. They therefore tend to speak over each other on con-calls, apologize, start again and talk over each other, again. After a little while, the picture becomes a distinctly silly one: a bunch of grown adults putting their hands up and waving, like little kids in school. No-one has called out “me, me, teacher” yet, but I expect it any time.
>
> I propose we show the results in terms that we can explain to Grandma, specifically concentrating on functioning VOIP. I just upgraded to Fedora 31, and the networking is absolutely stock, so I make a perfect victim/guinea-pig (;-))
>
> Who's interested?
>
>
>
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net           |                      -- Mark Twain
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Tom Henderson]

> so after much tweaking, I've got cake set to 40mbit down, 20mbit up,
> enforced by two cakes (one for each NIC). that's fairly low --

Thanks for sharing this configuration experience, but it leads me back 
to a question I have about best current practice for deployment.  Can 
CAKE/SQM handle dynamic Wi-Fi bandwidth due to Wi-Fi rate control 
selecting lower MCS to increase range, or does it rely on first getting 
the Wi-Fi deployed so that it has strong signal everywhere, and then 
finding a CAKE shaping rate that shaves off a few Mb/s from the highest 
capacity MCS so that the bottleneck always lands on the CAKE AQM?  It 
seems like the deployment that you shared with a separate router will 
require a predictable Wi-Fi rate, but I am wondering more about the case 
in which CAKE is deployed on the AP.

- Tom

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Sebastian Moeller]

Hi Tom,


On 10 August 2020 17:08:49 CEST, Tom Henderson <tomh@tomh.org> wrote:
>
>> so after much tweaking, I've got cake set to 40mbit down, 20mbit up,
>> enforced by two cakes (one for each NIC). that's fairly low --
>
>Thanks for sharing this configuration experience, but it leads me back 
>to a question I have about best current practice for deployment.  Can 
>CAKE/SQM handle dynamic Wi-Fi bandwidth due to Wi-Fi rate control 
>selecting lower MCS to increase range, or does it rely on first getting
>
>the Wi-Fi deployed so that it has strong signal everywhere, and then 
>finding a CAKE shaping rate that shaves off a few Mb/s from the highest
>
>capacity MCS so that the bottleneck always lands on the CAKE AQM?  It 
>seems like the deployment that you shared with a separate router will 
>require a predictable Wi-Fi rate, but I am wondering more about the
>case 
>in which CAKE is deployed on the AP.
>
>- Tom

No, neither cake nor SQM can deal all too well with variable rate links, if we talk about link rate variations in the second to second range. But in OpenWrt both the ath9k and the ath10k WiFi drivers gained airtime fairness modes, which do a pretty good job at keeping link sharing fair and WiFi bufferbloat low. 
The issue here is not really that closely related to cake or sqm, but that WiFi without AQL (airtime queueing limits, analog to wired Ethernet's byte queue limits) does not give the required pushback for an upstream qdisc to keep the WiFi queues at an acceptable/reasonable size and it also does not really offer a reliable fast estimator of achievable rate, as far as I can tell.
The current best practice seems to be to instantiate cake/SQM on a reasonably fixed rate wan link and select WiFi cards/socs that offer decent airtime fairness.
Works pretty well in practice...

Best regards
        Sebastian


>
>_______________________________________________
>Bloat mailing list
>Bloat@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/bloat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Jonathan Morton]

> The current best practice seems to be to instantiate cake/SQM on a reasonably fixed rate wan link and select WiFi cards/socs that offer decent airtime fairness.
> Works pretty well in practice...

Yes, AQL does essentially the right thing here, again along the lines of limiting the influence of one machine's load on another's performance, and completely automatically since it has faurly direct information and control over the relevant hardware.  Cake is designed to deal with wired links where the capacity doesn't change much, but the true bottleneck is typically not at the device exerting control.

On that note, there is a common wrinkle whereby the bottleneck may shift between the private last mile link and some shared backhaul in the ISP at different times of day and/or days of week.  Locally I've seen it vary between 20M (small hours, weekday) and 1Mbps (weekend evening).  When Cake is configured for one case but the situation is different, the results are obviously suboptimal.  I'm actually now trying a different ISP to see if they do better in the evenings.

Evenroute's product includes automatic detection of and scheduling for this case, assuming that it follows a consistent pattern over a weekly period.  Once set up, it is essentially a cronjob adjusting Cake's parameters dynamically, so providing a manual setup for the general OpenWRT community should be feasible.  On “tc qdisc change”, Cake usually doesn't drop any packets, so parameters can be changed frequently if you have a reason for it.

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Tom Henderson]

On 8/10/20 8:57 AM, Jonathan Morton wrote:
>> The current best practice seems to be to instantiate cake/SQM on a reasonably fixed rate wan link and select WiFi cards/socs that offer decent airtime fairness.
>> Works pretty well in practice...
> Yes, AQL does essentially the right thing here, again along the lines of limiting the influence of one machine's load on another's performance, and completely automatically since it has faurly direct information and control over the relevant hardware.  Cake is designed to deal with wired links where the capacity doesn't change much, but the true bottleneck is typically not at the device exerting control.
>
> On that note, there is a common wrinkle whereby the bottleneck may shift between the private last mile link and some shared backhaul in the ISP at different times of day and/or days of week.  Locally I've seen it vary between 20M (small hours, weekday) and 1Mbps (weekend evening).  When Cake is configured for one case but the situation is different, the results are obviously suboptimal.  I'm actually now trying a different ISP to see if they do better in the evenings.
>
> Evenroute's product includes automatic detection of and scheduling for this case, assuming that it follows a consistent pattern over a weekly period.  Once set up, it is essentially a cronjob adjusting Cake's parameters dynamically, so providing a manual setup for the general OpenWRT community should be feasible.  On “tc qdisc change”, Cake usually doesn't drop any packets, so parameters can be changed frequently if you have a reason for it.

Thanks for your insights, they have been helpful.  I also just found 
these references:

http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf

https://blog.tohojo.dk/slides/llc19-linux-wifi-bloat.pdf

- Tom

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Daniel Sterling]

as promised here is the script I run after rebooting my openwrt box,
to set up cake

https://gist.github.com/eqhmcow/c378c46a41aa5716767a0da811087dd4




On Mon, Aug 10, 2020 at 10:00 AM Daniel Sterling
<sterling.daniel@gmail.com> wrote:
>
> So I've been wanting to write up what I did to improve my home network
> for a while.
>
> Here's a quick overview:
>
> I'm running a small laptop-class sandy bridge CPU in a small desktop
> computer, running openwrt, running cake. It has two NICs -- the
> built-in realtek NIC, and an old Intel gigabit NIC in the PCI slot.
>
> Internet goes into the realtek NIC and out the Intel NIC. (WAN / LAN in openwrt)
>
> my internet is AT&T gigabit fiber, but I throttle that heavily  with
> cake (see below)
>
> I manually apply cake with my own scripts. I'll post those on gist and
> reply to this email with that info, just wanted to write this up
> quickly this morning. but it's basically just, apply two simple cake
> tc lines to the NICs.
>
> For wifi I use UBNT's SOHO line -- Amplifi HD units.
>
> it works really rather well; after some tweaking I've managed to
> essentially get rid of the things that I've empirically found really
> hurt home network performance:
>
> 1. wifi dead zones -- solved by using as many amplifi HD units as you
> like, meshed or wired together. obviously wires are better than mesh
> and a dedicated backhaul set of APs is better than mesh but mesh works
> too.
>
> 2. wifi trying to use 5ghz when it's too slow and refusing to switch
> to 2ghz -- solved by amplifi AP having a setting where it kicks
> devices off the 5ghz network proactively to convice them to switch to
> 2ghz. thank you UBNT!
>
> 3. TCP not dropping enough packets. (or rather, not having good queue
> management)
>
> 4. TCP (or rather, the network) dropping too many TCP packets --
> streams / apps / web sites will get "stuck"
>
> so after much tweaking, I've got cake set to 40mbit down, 20mbit up,
> enforced by two cakes (one for each NIC). that's fairly low --
>
> it's low to highly throttle bulk streams so that I  can play
> latency-sensitive games with basically no jitter and low latency, even
> if other people are using the wifi. even if I can't wire an xbox, I
> can still get low latency gaming on wifi
>
> but it's still high enough that we can stream HD video.
>
> and of course low jitter and low latency across the board means good
> ssh and video cal performance.
>
> just wanted to write this up quickly to reply to this thread -- cake
> really is amazing and I'd bet people would be willing to pay for a
> magic box like I've set up that they can stick in between their
> existing CPE and a decent AP that applies cake. or if AP vendors would
> put cake in their APs themselves, that would be good too.
>
> but as you note #1 and #2 on my list are important, even before queue
> management comes into play. you have to be willing to buy a good AP
> before cake really starts to matter, I think
>
> -- Dan
>
> On Mon, Aug 10, 2020 at 8:57 AM David Collier-Brown <davecb.42@gmail.com> wrote:
> >
> > On 2020-08-09 5:35 p.m., Jonathan Morton wrote:
> >
> > Are the risks and tradeoffs well enough understood (and visible enough
> > for troubleshooting) to recommend broader deployment?
> >
> > I recently gave openwrt a try on some hardware that I ultimately
> > concluded was insufficient for the job.  Fairly soon after changing out
> > my access point, I started getting complaints of Wi-Fi dropping in my
> > household, especially when someone was trying to videoconference.  I
> > discovered that my AP was spontaneously rebooting, and the box was
> > getting hot.
> >
> > Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
> >
> > It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
> >
> > Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.
> >
> > I'm primarily thinking of this week's version of the home router problem (;-))
> >
> > Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.
> >
> > A (very!) draft version is up in Google docs, at https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing
> >
> > Using myself as the guinea-pig, running pfifo-fast was clearly bad, fq_codel was better, and cake was good with a newish Fedora and the stock Rogers router.  It's been a while since I did rrul tests, and in any case, I think that to convince readers we need a very practical way of making it clear that they have a problem. I'm thinking that making VOIP fail might do the trick (;-))
> >
> > The hard part, IMHO, is constructing a test that immediately communicates the idea that the reader has a problem, and that CAKE addresses it.
> >
> > Returning to the hardware question, https://evenroute.com/iqrv3 seems to be capable of handling up to ~300 Mbit/S connections, and my ISP only delivers 170 (and advertises 150, which is mildly surprising!)
> >
> > I just ordered one, so I'll have a 'plug in" example, along with reflashing my linksys for the umpty-thousandth time.
> >
> > --dave
> >
> >  I suspect not enough people are aware of the later efforts of the bufferbloat team, so I'm thinking of one or two articles, starting with LWN and an audience of aficionados.
> >
> > The core community is aware of what we've done, but in my view we haven't converted "grandma". Grandma, as well as a whole bunch of ordinary engineers and partners of engineers, are dependent on debloated performance because they're working at home now, and competing with granddaughter playing video games while they're trying to hold a video call.
> >
> > Right now, my colleagues at work suffer from more than a second of bloat-related lag. They therefore tend to speak over each other on con-calls, apologize, start again and talk over each other, again. After a little while, the picture becomes a distinctly silly one: a bunch of grown adults putting their hands up and waving, like little kids in school. No-one has called out “me, me, teacher” yet, but I expect it any time.
> >
> > I propose we show the results in terms that we can explain to Grandma, specifically concentrating on functioning VOIP. I just upgraded to Fedora 31, and the networking is absolutely stock, so I make a perfect victim/guinea-pig (;-))
> >
> > Who's interested?
> >
> >
> >
> >
> > --
> > David Collier-Brown,         | Always do right. This will gratify
> > System Programmer and Author | some people and astonish the rest
> > davecb@spamcop.net           |                      -- Mark Twain
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Kenneth Porter]

--On Tuesday, August 11, 2020 9:43 AM -0400 Daniel Sterling 
<sterling.daniel@gmail.com> wrote:

> as promised here is the script I run after rebooting my openwrt box,
> to set up cake
>
> https://gist.github.com/eqhmcow/c378c46a41aa5716767a0da811087dd4

How does this differ from the sqm-scripts available in Fedora and OpenWrt?

https://github.com/tohojo/sqm-scripts

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Daniel Sterling]

as far as I know it's much simpler; I literally just turn on cake per
NIC with the correct settings applied for my environment

the env settings that matter are: which NIC is WAN vs LAN; and how
much bandwidth you want cake to enforce.

Also I find I get best results with "besteffort" vs using any of
cake's internal additional queues. I don't have enough visibility into
how cake is working to know what exactly is happening, and it may be
I'm confounding cake with some other env issue, but I've stuck with
besteffort due to seeing more latency when I turn on multiple cake
queues

On Tue, Aug 11, 2020 at 9:57 AM Kenneth Porter <shiva@sewingwitch.com> wrote:
>
> --On Tuesday, August 11, 2020 9:43 AM -0400 Daniel Sterling
> <sterling.daniel@gmail.com> wrote:
>
> > as promised here is the script I run after rebooting my openwrt box,
> > to set up cake
> >
> > https://gist.github.com/eqhmcow/c378c46a41aa5716767a0da811087dd4
>
> How does this differ from the sqm-scripts available in Fedora and OpenWrt?
>
> https://github.com/tohojo/sqm-scripts
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Daniel Sterling]

https://github.com/tohojo/sqm-scripts/blob/master/src/piece_of_cake.qos
looks to be more or less what I'm doing

On Tue, Aug 11, 2020 at 10:09 AM Daniel Sterling
<sterling.daniel@gmail.com> wrote:
>
> as far as I know it's much simpler; I literally just turn on cake per
> NIC with the correct settings applied for my environment
>
> the env settings that matter are: which NIC is WAN vs LAN; and how
> much bandwidth you want cake to enforce.
>
> Also I find I get best results with "besteffort" vs using any of
> cake's internal additional queues. I don't have enough visibility into
> how cake is working to know what exactly is happening, and it may be
> I'm confounding cake with some other env issue, but I've stuck with
> besteffort due to seeing more latency when I turn on multiple cake
> queues
>
> On Tue, Aug 11, 2020 at 9:57 AM Kenneth Porter <shiva@sewingwitch.com> wrote:
> >
> > --On Tuesday, August 11, 2020 9:43 AM -0400 Daniel Sterling
> > <sterling.daniel@gmail.com> wrote:
> >
> > > as promised here is the script I run after rebooting my openwrt box,
> > > to set up cake
> > >
> > > https://gist.github.com/eqhmcow/c378c46a41aa5716767a0da811087dd4
> >
> > How does this differ from the sqm-scripts available in Fedora and OpenWrt?
> >
> > https://github.com/tohojo/sqm-scripts
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Kenneth Porter]

--On Tuesday, August 11, 2020 11:11 AM -0400 Daniel Sterling 
<sterling.daniel@gmail.com> wrote:

> https://github.com/tohojo/sqm-scripts/blob/master/src/piece_of_cake.qos
> looks to be more or less what I'm doing

I'm using the "layer cake" script on my OpenWrt router but I have nothing 
to dynamically change the bandwidth caps based on ISP "weather". I suspect 
that would be a useful thing to capture in a cron job or systemd timer unit 
as I still see Youtube stutter from time to time.

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Jonathan Foulkes]

[-- Attachment #1: Type: text/plain, Size: 10472 bytes --]

Hi David, 

Great topic, and glad you brought it up, as increasing awareness of all the goodness that de-bloating brings to end users is important, especially with all the WFH and soon, teaching/learning going on these days.

Background / disclosure: I’m the founder and CEO of Evenroute, so first, thank you for your order :-)  Second, please let me know if you have any questions once you get the unit. Happy to personally support you, or any member of this list.

Given I have access to the combined data of the deployed IQrouter fleet, I have a pretty good view of how well Cake performs in the real world, as we are on nearly every ISP domestically (US) and a surprising number of International deployments as well (even though we only market in the US).
as you might imagine, given our marketing, we have a huge number of users with really bad lines, or on challenging tech, like WISPs & Satellite. So we get to see the worst of the worst.
But interestingly enough, we also have a certain amount of users with extremely low and stable latencies with no QoS, yet they still continue to deploy their IQrouters, likely because the prior ISP-supplied device *added* latencies, and the benefits of fairness in the per-device, per-host settings in Cake, plus correctly prioritizing based on type & DSCP marks (most WiFi-calling smartphone traffic is correctly marked).
>>> Are the risks and tradeoffs well enough understood (and visible enough 
>>> for troubleshooting) to recommend broader deployment?
We’ve been deploying SQM / CAKE for 4+ years in the IQrouter, and while we have evolved a lot of our algorithms do deal with the millions of permutations in configurations and settings, I’ve seen SQM and lately CAKE likewise mature, and my assessment is they are indeed ready for prime time in terms of foundational tech.
The challenge is not the core tech, it’s accessibility. That was my take in 2015 when I first discovered it, and led to the founding of my company.

As for troubleshooting visibility, check out the Status->Ping Stats page once your IQrouter has been running for a few days. Very helpful in triaging modem and line issues. Basically its a line capacity usage monitor and ping plotter.

> but in my view we haven't converted "grandma".

Because until I produced a product with zero user configuration requirements (relative to QoS), ‘grandma’ was never a viable user.
Back story: I live in a large (3,000 homes) development in a rural area, most residents are retired professionals and DSL was the only choice up until recently, so a bunch of non-technical grandma's and grandpas are my neighbors. The IQrouter was developed to meet the needs of that audience. Grandma should be able to deploy in 15 minutes and have a de-bloated DSL connection that got rid of the 5,000ms+ lag spikes. So initial config workflow, and all the tuning and dynamic line adaptation were largely born of dealing with my local needs.

Funny enough, our focus on non-techie usability led to a skeptical backlash from some ’techies’, who find our marketing messaging and simple UI too off-putting for them. Even though we do expose the full native UI for OpenWRT under the ‘Advanced Menus’ option. Heck, you can even instal OpenWRT packages on this thing.
Smart techies love though, a recent IT guy wrote in response to our support team letting him know about OpenWRT support was: "Now, I know that this really is the best router for all consumers to use in their homes!”. We made his WISP service usable.

> Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.
> 
Big time. As noted above, it not just the CPE devices, it’s the congestion on the backhauls that causes issues, and it’s everything from DSL (slammed DSLAMs are endemic) to cable systems with oversubscribed local loops and congested CMTS backhaul. Hell, even fiber to the home ISPs manage to have variable capacity (and bloat) in the evenings.

Traffic management is a requirement these days.

>> I propose we show the results in terms that we can explain to Grandma, specifically concentrating on functioning VOIP.

This is desperately needed, as there need to be more points of proof and articles outlining the problem, and the impacts of resolving it with effective traffic management.

Stuff like this article from Jim Gettys on the needs of teachers. BTW- he calls out the IQrouter as his ‘Go to’ recommendation for non-techies. He runs one himself and gave one to his non-techie brother. Bufferbloat in Action due to Covid-19 <https://gettys.wordpress.com/2020/04/22/bufferbloat-in-action-due-to-covid-19/>

More comments on other response later, got work to do ;-)

Cheers,

Jonathan Foulkes


> On Aug 10, 2020, at 8:57 AM, David Collier-Brown <davecb.42@gmail.com> wrote:
> 
> On 2020-08-09 5:35 p.m., Jonathan Morton wrote:
> 
>>> Are the risks and tradeoffs well enough understood (and visible enough 
>>> for troubleshooting) to recommend broader deployment?
>>> 
>>> I recently gave openwrt a try on some hardware that I ultimately 
>>> concluded was insufficient for the job.  Fairly soon after changing out 
>>> my access point, I started getting complaints of Wi-Fi dropping in my 
>>> household, especially when someone was trying to videoconference.  I 
>>> discovered that my AP was spontaneously rebooting, and the box was 
>>> getting hot.
>> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
>> 
>> It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
>> 
>> Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.
> I'm primarily thinking of this week's version of the home router problem (;-)) 
> 
> Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.
> 
> A (very!) draft version is up in Google docs, at https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing <https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing>
> Using myself as the guinea-pig, running pfifo-fast was clearly bad, fq_codel was better, and cake was good with a newish Fedora and the stock Rogers router.  It's been a while since I did rrul tests, and in any case, I think that to convince readers we need a very practical way of making it clear that they have a problem. I'm thinking that making VOIP fail might do the trick (;-))
> 
> The hard part, IMHO, is constructing a test that immediately communicates the idea that the reader has a problem, and that CAKE addresses it. 
> 
> Returning to the hardware question, https://evenroute.com/iqrv3 <https://evenroute.com/iqrv3> seems to be capable of handling up to ~300 Mbit/S connections, and my ISP only delivers 170 (and advertises 150, which is mildly surprising!)
> 
> I just ordered one, so I'll have a 'plug in" example, along with reflashing my linksys for the umpty-thousandth time.
> 
> --dave
> 
> 
>>  I suspect not enough people are aware of the later efforts of the bufferbloat team, so I'm thinking of one or two articles, starting with LWN and an audience of aficionados. 
>> 
>> The core community is aware of what we've done, but in my view we haven't converted "grandma". Grandma, as well as a whole bunch of ordinary engineers and partners of engineers, are dependent on debloated performance because they're working at home now, and competing with granddaughter playing video games while they're trying to hold a video call. 
>> 
>> Right now, my colleagues at work suffer from more than a second of bloat-related lag. They therefore tend to speak over each other on con-calls, apologize, start again and talk over each other, again. After a little while, the picture becomes a distinctly silly one: a bunch of grown adults putting their hands up and waving, like little kids in school. No-one has called out “me, me, teacher” yet, but I expect it any time. 
>> 
>> I propose we show the results in terms that we can explain to Grandma, specifically concentrating on functioning VOIP. I just upgraded to Fedora 31, and the networking is absolutely stock, so I make a perfect victim/guinea-pig (;-)) 
>> 
>> Who's interested?
> 
> 
> 
> 
> -- 
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net <mailto:davecb@spamcop.net>           |                      -- Mark Twain
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


[-- Attachment #2: Type: text/html, Size: 14036 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Carlos R. Pasqualini]

[-- Attachment #1: Type: text/plain, Size: 11559 bytes --]

Great topic, indeed!


We (almost?) all know that we can put a linux box between our network and
the Internet and apply cake on the WAN interface (may be an ingress too)
and most of the bufferbloat problems disappear

One thing I suggest is to bring more cake awareness in other communities
and commercial products to support it.

As an example, a was unable to find a way to get cake on pfSense/opnSense
(may be FreeBSD doesn't support cake?) only something similar to fq_codel.

In worst shape, are products like Mikrotik, used by quite a few "network
professionals" and some ISPs, where I was unable to find any way to get
ride of bufferbloat



Thanks!


El lun., 10 ago. 2020 a las 14:58, Jonathan Foulkes (<jf@jonathanfoulkes.com>)
escribió:

> Hi David,
>
> Great topic, and glad you brought it up, as increasing awareness of all
> the goodness that de-bloating brings to end users is important, especially
> with all the WFH and soon, teaching/learning going on these days.
>
> Background / disclosure: I’m the founder and CEO of Evenroute, so first,
> thank you for your order :-)  Second, please let me know if you have any
> questions once you get the unit. Happy to personally support you, or any
> member of this list.
>
> Given I have access to the combined data of the deployed IQrouter fleet, I
> have a pretty good view of how well Cake performs in the real world, as we
> are on nearly every ISP domestically (US) and a surprising number of
> International deployments as well (even though we only market in the US).
> as you might imagine, given our marketing, we have a huge number of users
> with really bad lines, or on challenging tech, like WISPs & Satellite. So
> we get to see the worst of the worst.
> But interestingly enough, we also have a certain amount of users with
> extremely low and stable latencies with no QoS, yet they still continue to
> deploy their IQrouters, likely because the prior ISP-supplied device
> *added* latencies, and the benefits of fairness in the per-device, per-host
> settings in Cake, plus correctly prioritizing based on type & DSCP marks
> (most WiFi-calling smartphone traffic is correctly marked).
>
> Are the risks and tradeoffs well enough understood (and visible enough
> for troubleshooting) to recommend broader deployment?
>
> We’ve been deploying SQM / CAKE for 4+ years in the IQrouter, and while we
> have evolved a lot of our algorithms do deal with the millions of
> permutations in configurations and settings, I’ve seen SQM and lately CAKE
> likewise mature, and my assessment is they are indeed ready for prime time
> in terms of foundational tech.
> The challenge is not the core tech, it’s accessibility. That was my take
> in 2015 when I first discovered it, and led to the founding of my company.
>
> As for troubleshooting visibility, check out the Status->Ping Stats page
> once your IQrouter has been running for a few days. Very helpful in
> triaging modem and line issues. Basically its a line capacity usage monitor
> and ping plotter.
>
> but in my view we haven't converted "grandma".
>
>
> Because until I produced a product with zero user configuration
> requirements (relative to QoS), ‘grandma’ was never a viable user.
> Back story: I live in a large (3,000 homes) development in a rural area,
> most residents are retired professionals and DSL was the only choice up
> until recently, so a bunch of non-technical grandma's and grandpas are my
> neighbors. The IQrouter was developed to meet the needs of that audience.
> Grandma should be able to deploy in 15 minutes and have a de-bloated DSL
> connection that got rid of the 5,000ms+ lag spikes. So initial config
> workflow, and all the tuning and dynamic line adaptation were largely born
> of dealing with my local needs.
>
> Funny enough, our focus on non-techie usability led to a skeptical
> backlash from some ’techies’, who find our marketing messaging and simple
> UI too off-putting for them. Even though we do expose the full native UI
> for OpenWRT under the ‘Advanced Menus’ option. Heck, you can even instal
> OpenWRT packages on this thing.
> Smart techies love though, a recent IT guy wrote in response to our
> support team letting him know about OpenWRT support was: "Now, I know
> that this really is the *best router* for all consumers to use in their
> homes!”. We made his WISP service usable.
>
> Because of the degree to which we're working from home and
> videoconferencing, a lot of low-price, medium-performance devices are
> suddenly too wimpy for their new role.
>
> Big time. As noted above, it not just the CPE devices, it’s the congestion
> on the backhauls that causes issues, and it’s everything from DSL (slammed
> DSLAMs are endemic) to cable systems with oversubscribed local loops and
> congested CMTS backhaul. Hell, even fiber to the home ISPs manage to have
> variable capacity (and bloat) in the evenings.
> Traffic management is a requirement these days.
>
> I propose we show the results in terms that we can explain to Grandma,
> specifically concentrating on functioning VOIP.
>
>
> This is desperately needed, as there need to be more points of proof and
> articles outlining the problem, and the impacts of resolving it with
> effective traffic management.
>
> Stuff like this article from Jim Gettys on the needs of teachers. BTW- he
> calls out the IQrouter as his ‘Go to’ recommendation for non-techies. He
> runs one himself and gave one to his non-techie brother. Bufferbloat in
> Action due to Covid-19
> <https://gettys.wordpress.com/2020/04/22/bufferbloat-in-action-due-to-covid-19/>
>
> More comments on other response later, got work to do ;-)
>
> Cheers,
>
> Jonathan Foulkes
>
>
> On Aug 10, 2020, at 8:57 AM, David Collier-Brown <davecb.42@gmail.com>
> wrote:
>
> On 2020-08-09 5:35 p.m., Jonathan Morton wrote:
>
> Are the risks and tradeoffs well enough understood (and visible enough
> for troubleshooting) to recommend broader deployment?
>
> I recently gave openwrt a try on some hardware that I ultimately
> concluded was insufficient for the job.  Fairly soon after changing out
> my access point, I started getting complaints of Wi-Fi dropping in my
> household, especially when someone was trying to videoconference.  I
> discovered that my AP was spontaneously rebooting, and the box was
> getting hot.
>
> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
>
> It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
>
> Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.
>
> I'm primarily thinking of *this week's* version of the home router
> problem (;-))
>
> Because of the degree to which we're working from home and
> videoconferencing, a lot of low-price, medium-performance devices are
> suddenly too wimpy for their new role.
>
> A (very!) draft version is up in Google docs, at
> https://docs.google.com/document/d/1gWKp9HqTbuHLfgD59WU4KJ8Og3eHuBtIeC7BUK0Ju9w/edit?usp=sharing
>
> Using myself as the guinea-pig, running pfifo-fast was clearly bad,
> fq_codel was better, and cake was good with a newish Fedora and the stock
> Rogers router.  It's been a while since I did rrul tests, and in any case,
> I think that to convince readers we need a very practical way of making it
> clear that they have a problem. I'm thinking that making VOIP fail might do
> the trick (;-))
>
> The hard part, IMHO, is constructing a test that immediately communicates
> the idea that the reader has a problem, and that CAKE addresses it.
>
> Returning to the hardware question, https://evenroute.com/iqrv3 seems to
> be capable of handling up to ~300 Mbit/S connections, and my ISP only
> delivers 170 (and advertises 150, which is mildly surprising!)
>
> I just ordered one, so I'll have a 'plug in" example, along with
> reflashing my linksys for the umpty-thousandth time.
>
> --dave
>
>  I suspect not enough people are aware of the later efforts of the
> bufferbloat team, so I'm thinking of one or two articles, starting with LWN
> and an audience of aficionados.
>
> The core community is aware of what we've done, but in my view we haven't
> converted "grandma". Grandma, as well as a whole bunch of ordinary
> engineers and partners of engineers, are dependent on debloated performance
> because they're working at home now, and competing with granddaughter
> playing video games while they're trying to hold a video call.
>
> Right now, my colleagues at work suffer from more than a second of
> bloat-related lag. They therefore tend to speak over each other on
> con-calls, apologize, start again and talk over each other, again. After a
> little while, the picture becomes a distinctly silly one: a bunch of grown
> adults putting their hands up and waving, like little kids in school.
> No-one has called out “me, me, teacher” yet, but I expect it any time.
>
> I propose we show the results in terms that we can explain to Grandma,
> specifically concentrating on functioning VOIP. I just upgraded to Fedora
> 31, and the networking is absolutely stock, so I make a perfect
> victim/guinea-pig (;-))
>
> Who's interested?
>
>
>
>
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the restdavecb@spamcop.net           |                      -- Mark Twain
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>


-- 
Carlos Pasqualini
+5493454040137
Administración de Redes

[-- Attachment #2: Type: text/html, Size: 14555 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

Hi Johnathan, I remember you from before!

On 2020-08-10 1:58 p.m., Jonathan Foulkes wrote:

Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.

Big time. As noted above, it not just the CPE devices, it’s the congestion on the backhauls that causes issues, and it’s everything from DSL (slammed DSLAMs are endemic) to cable systems with oversubscribed local loops and congested CMTS backhaul. Hell, even fiber to the home ISPs manage to have variable capacity (and bloat) in the evenings.


...

Stuff like this article from Jim Gettys on the needs of teachers. BTW- he calls out the IQrouter as his ‘Go to’ recommendation for non-techies. He runs one himself and gave one to his non-techie brother. Bufferbloat in Action due to Covid-19<https://gettys.wordpress.com/2020/04/22/bufferbloat-in-action-due-to-covid-19/>


Thanks, I hadn't seen that



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 2881 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Michael Yartys]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

The question I struggle with is how to get ISPs to care and put pressure on CPE manufacturers to fix bufferbloat in their products.

I performed some tests on my Zyxel router, and the WiFi link was terribly bloated when there were a few walls between my laptop and the router. The latency almost reached 10 seconds under load! My ISPs response was that some buffering is normal (some = 10 seconds??) and that artificially saturating the link doesn't represent a normal use case. Keep in mind that this was at the edge of the WiFi coverage with a maximum data rate of 15 Mbps, a speed that's relatively easy to reach. I followed up with an email about how there are solutions to this issue and that they should put pressure on Zyxel to implement them, but alas, there has been no response and it has probably fallen on deaf ears.

Michael

-------- Original Message --------
On 10 Aug 2020, 22:28, Dave Collier-Brown wrote:

> Hi Johnathan, I remember you from before!
>
> On 2020-08-10 1:58 p.m., Jonathan Foulkes wrote:
>
>>> Because of the degree to which we're working from home and videoconferencing, a lot of low-price, medium-performance devices are suddenly too wimpy for their new role.
>>
>> Big time. As noted above, it not just the CPE devices, it’s the congestion on the backhauls that causes issues, and it’s everything from DSL (slammed DSLAMs are endemic) to cable systems with oversubscribed local loops and congested CMTS backhaul. Hell, even fiber to the home ISPs manage to have variable capacity (and bloat) in the evenings.
>
> ...
>
>> Stuff like this article from Jim Gettys on the needs of teachers. BTW- he calls out the IQrouter as his ‘Go to’ recommendation for non-techies. He runs one himself and gave one to his non-techie brother. [Bufferbloat in Action due to Covid-19](https://gettys.wordpress.com/2020/04/22/bufferbloat-in-action-due-to-covid-19/)
>
> Thanks, I hadn't seen that
>
> CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 3915 bytes --]

Re: [Bloat] Sidebar to "How about a topical LWN article on demonstrating the real-world goodness of CAKE?"

[David Collier-Brown]

On 2020-08-09 5:35 p.m., Jonathan Morton wrote:

> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
I have an interest in the next step up from mass-market routers, and 
this description made my ears prick up.

At work we have a decent network, carefully designed about 15 years ago 
to get work from Akamai, distribute it over mostly-in-the-same-DC 
service providers, then return it to the end-user customer. The IT folks 
don't know how it works, but are confident that it does (;-))

The nets we use for WFH are a little newer, but we had no idea if we 
were going to be able to work from home or hold conference calls with 
400-odd people. So we had to send everyone home on a Monday to try the 
experiment. Fortunately it worked.

That makes me curious about the customer-premises equipment we and 
others have, and to what degree it can be de-bloated. I know the 
datacenter vendors were being resistant when the problem was first 
solved, but I've not heard anything positive to date.  Cisco makes 
vaguely hand-wavey statements about DOCSIS 3.1 and PIE, but doesn't seem 
to answer customer questions,

What do we know about large-home and small-office devices these days?

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net           |                      -- Mark Twain

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Simon Barber]

I’ve had good success adding GRO into the mix to reduce CPU usage for WiFi APs with complex forwarding rules. The recent Qualcomm chipsets and proprietary drivers support some parts of GRO and TSO in firmware which helps. The kernel’s software GSO function doesn’t work well for this purpose - it reallocates new skbs for headers for all the packets it GSOs - but if you reused the existing space in the skbs for headers, even software GSO could probably be quite fast.

I’ve yet to explore GRO by aggregate but it feels like a natural fit since the wireless packets arrive in aggregates of up to a few 100 frames, to GRO them together into one.

Simon


> On Aug 9, 2020, at 2:35 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> Are the risks and tradeoffs well enough understood (and visible enough 
>> for troubleshooting) to recommend broader deployment?
>> 
>> I recently gave openwrt a try on some hardware that I ultimately 
>> concluded was insufficient for the job.  Fairly soon after changing out 
>> my access point, I started getting complaints of Wi-Fi dropping in my 
>> household, especially when someone was trying to videoconference.  I 
>> discovered that my AP was spontaneously rebooting, and the box was 
>> getting hot.
> 
> Most CPE devices these days rely on hardware accelerated packet forwarding to achieve their published specs.  That's all about taking packets in one side and pushing them out the other as quickly as possible, with only minimal support from the CPU (likely, new connections get a NAT/firewall lookup, that's all).  It has the advantages of speed and power efficiency, but unfortunately it is also incompatible with our debloating efforts.  So debloated CPE will tend to run hotter and with lower peak throughput, which may be noticeable to cable and fibre users; VDSL (FTTC) users might have service of 80Mbps or less where this effect is less likely to matter.
> 
> It sounds like that AP had a very marginal thermal design which caused the hardware to overheat as soon as the CPU was under significant load, which it can easily be when a shaper and AQM are running on it at high throughput.  The cure is to use better designed hardware, though you could also contemplate breaking the case open to cure the thermal problem directly.  There are some known reliable models which could be collected into a list.  As a rule of thumb, the ones based on ARM cores are likely to be designed with CPU performance more in mind than those with MIPS.
> 
> Cake has some features which can be used to support explicit classification and (de)prioritisation of traffic via firewall marking rules, either by rewriting the Diffserv field or by associating metadata with packets within the network stack (fwmark).  This can be very useful for pushing Bittorrent or WinUpdate swarm traffic out of the way.  But for most situations, the default flow-isolating behaviour already works pretty well, especially for ensuring that one computer's network load has only a bounded effect on any other.  We can discuss that in more detail if that would be helpful.
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Taht]

I note that.I pretty much took the summer off. And I'm trying to get
started again. Did this article come off?

On Sun, Aug 9, 2020 at 11:27 AM David Collier-Brown <davecb.42@gmail.com> wrote:
>
>   I suspect not enough people are aware of the later efforts of the
> bufferbloat team, so I'm thinking of one or two articles, starting with
> LWN and an audience of aficionados.
>
> The core community is aware of what we've done, but in my view we
> haven't converted "grandma". Grandma, as well as a whole bunch of
> ordinary engineers and partners of engineers, are dependent on debloated
> performance because they're working at home now, and competing with
> granddaughter playing video games while they're trying to hold a video call.
>
> Right now, my colleagues at work suffer from more than a second of
> bloat-related lag. They therefore tend to speak over each other on
> con-calls, apologize, start again and talk over each other, again. After
> a little while, the picture becomes a distinctly silly one: a bunch of
> grown adults putting their hands up and waving, like little kids in
> school. No-one has called out “me, me, teacher” yet, but I expect it any
> time.
>
> I propose we show the results in terms that we can explain to Grandma,
> specifically concentrating on functioning VOIP. I just upgraded to
> Fedora 31, and the networking is absolutely stock, so I make a perfect
> victim/guinea-pig (;-))
>
> Who's interested?
>
>
> --dave
> --
> David Collier-Brown,                     | Always do right. This will
> gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net                  |                      -- Mark Twain
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 3655 bytes --]

LWN said OK, but I'm stuck on the search for a striking test, one that resonates with "grandma".

My next two thoughts, probably for the current long weekend, is either to call a loop-back number with Skype and/or ask Toke how he got "Big Buck Bunny" to suffer dropouts. I'd love to use the latter, as it aligns with the observation that this is a time when conference-call failures are driving my colleagues to drink (;-))

I also owe you a pull request for a "recipes" page.

--dave

On 2020-09-05 2:52 p.m., Dave Taht wrote:

I note that.I pretty much took the summer off. And I'm trying to get
started again. Did this article come off?

On Sun, Aug 9, 2020 at 11:27 AM David Collier-Brown <davecb.42@gmail.com><mailto:davecb.42@gmail.com> wrote:



  I suspect not enough people are aware of the later efforts of the
bufferbloat team, so I'm thinking of one or two articles, starting with
LWN and an audience of aficionados.

The core community is aware of what we've done, but in my view we
haven't converted "grandma". Grandma, as well as a whole bunch of
ordinary engineers and partners of engineers, are dependent on debloated
performance because they're working at home now, and competing with
granddaughter playing video games while they're trying to hold a video call.

Right now, my colleagues at work suffer from more than a second of
bloat-related lag. They therefore tend to speak over each other on
con-calls, apologize, start again and talk over each other, again. After
a little while, the picture becomes a distinctly silly one: a bunch of
grown adults putting their hands up and waving, like little kids in
school. No-one has called out “me, me, teacher” yet, but I expect it any
time.

I propose we show the results in terms that we can explain to Grandma,
specifically concentrating on functioning VOIP. I just upgraded to
Fedora 31, and the networking is absolutely stock, so I make a perfect
victim/guinea-pig (;-))

Who's interested?


--dave
--
David Collier-Brown,                     | Always do right. This will
gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net<mailto:davecb@spamcop.net>                  |                      -- Mark Twain

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat







--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 4680 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Toke Høiland-Jørgensen]

Dave Collier-Brown <dave.collier-brown@indexexchange.com> writes:

> LWN said OK, but I'm stuck on the search for a striking test, one that resonates with "grandma".
>
> My next two thoughts, probably for the current long weekend, is either
> to call a loop-back number with Skype and/or ask Toke how he got "Big
> Buck Bunny" to suffer dropouts. I'd love to use the latter, as it
> aligns with the observation that this is a time when conference-call
> failures are driving my colleagues to drink (;-))

I assume you're referring to the Dash data included in the Polify paper,
right? :)

Not sure if I ever got it to drop out completely; just did some
measurements of which bitrate it picked, and the context was airtime
prioritisation, not so much the latency improvements. But anyway, the
tests just used the reference dash.js[0] player, with a logger addition
that Flent can parse[1].

I don't recall what exactly is needed on the server side to run this,
but I think it's basically just dropping the right Big Buck Bunny
tarball on a web server along with dash-logger.js from the Flent sources :)

-Toke


[0] https://github.com/Dash-Industry-Forum/dash.js/wiki
[1] https://github.com/tohojo/flent/commit/6b83896cc0df1d468577ef0f35abbab6dd025c3f

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 2634 bytes --]

Many thanks: will try the skype test later today --dave

On 2020-09-07 5:23 a.m., Toke Høiland-Jørgensen wrote:

Dave Collier-Brown <dave.collier-brown@indexexchange.com><mailto:dave.collier-brown@indexexchange.com> writes:



LWN said OK, but I'm stuck on the search for a striking test, one that resonates with "grandma".

My next two thoughts, probably for the current long weekend, is either
to call a loop-back number with Skype and/or ask Toke how he got "Big
Buck Bunny" to suffer dropouts. I'd love to use the latter, as it
aligns with the observation that this is a time when conference-call
failures are driving my colleagues to drink (;-))



I assume you're referring to the Dash data included in the Polify paper,
right? :)

Not sure if I ever got it to drop out completely; just did some
measurements of which bitrate it picked, and the context was airtime
prioritisation, not so much the latency improvements. But anyway, the
tests just used the reference dash.js[0] player, with a logger addition
that Flent can parse[1].

I don't recall what exactly is needed on the server side to run this,
but I think it's basically just dropping the right Big Buck Bunny
tarball on a web server along with dash-logger.js from the Flent sources :)

-Toke


[0] https://github.com/Dash-Industry-Forum/dash.js/wiki
[1] https://github.com/tohojo/flent/commit/6b83896cc0df1d468577ef0f35abbab6dd025c3f


--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 3568 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[David Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

Just FYI, I did two tests today, in my Copious Spare Time

 1. a DSLReports speed test using an IQrouter, and
 2. a Skype "tap" test, with and without the de-bloated router

The router did exactly what I expected: it vaporized the bloat-induced 
delay provided by Rogers.

My previous bloat-induced lag was between 700 and 900 milliseconds under 
load, and an unimpressive 240 milliseconds when idle.  With a de-bloated 
router, lag was 40 milliseconds maximum, whether idle or loaded, and 
about 30 millisecond minimum, or a 3500 km round-trip, appropriate for 
the distances to the measurements points. By comparison, Rodger-Dogers 
was giving me lag times more appropriate for a test to Ankara, Turkey.

The tap test was interesting, but less helpful. I connected to 
1-909-390-0003, which is an instant-echo server, used to test voice 
telephones. If you connect to it and say something, it will echo it back 
to you. If you tap on your microphone, it's fairly easy to get an idea 
of how much delay there is.

Alas, with and without bufferbloat mitigation, it took about a second 
echo back the tap, so it's not the kind of instantly-obvious success 
test I'm searching for, to demonstrate the goodness of CAKE to my 
grandmother (;-))

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
davecb@spamcop.net            |                      -- Mark Twain


[-- Attachment #2: Type: text/html, Size: 1923 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Taht]

Dear David:

I think freeswitch still has a public sip echo server which might be a
better test than skype?  I tend to think there is a lot of bloat on
the endpoints nowadays because it "helps" on not having to deal with
jitter
that the bloat on the rest of the internet induces. But 1sec you got
on skype in your test below is NUTS. I hacked a bit on BBB earlier
this year to get their number down to sanity.

I find endless videoconferencing nowadays to be really tiring and
frustrating. Am I alone in this? I have to go outside and scream after
two hours. Log off entirely after 4.

On Mon, Sep 7, 2020 at 10:20 AM David Collier-Brown <davecb.42@gmail.com> wrote:
>
> Just FYI, I did two tests today, in my Copious Spare Time
>
> a DSLReports speed test using an IQrouter, and
> a Skype "tap" test, with and without the de-bloated router
>
> The router did exactly what I expected: it vaporized the bloat-induced delay provided by Rogers.
>
> My previous bloat-induced lag was between 700 and 900 milliseconds under load, and an unimpressive 240 milliseconds when idle.  With a de-bloated router, lag was 40 milliseconds maximum, whether idle or loaded, and about 30 millisecond minimum, or a 3500 km round-trip, appropriate for the distances to the measurements points. By comparison, Rodger-Dogers was giving me lag times more appropriate for a test to Ankara, Turkey.
>
> The tap test was interesting, but less helpful. I connected to 1-909-390-0003, which is an instant-echo server, used to test voice telephones. If you connect to it and say something, it will echo it back to you. If you tap on your microphone, it's fairly easy to get an idea of how much delay there is.
>
> Alas, with and without bufferbloat mitigation, it took about a second echo back the tap, so it's not the kind of instantly-obvious success test I'm searching for, to demonstrate the goodness of CAKE to my grandmother (;-))
>
> --dave
>
> --
> David Collier-Brown,         | Always do right. This will gratify
> System Programmer and Author | some people and astonish the rest
> davecb@spamcop.net           |                      -- Mark Twain



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Matt Mathis]

[-- Attachment #1: Type: text/plain, Size: 4053 bytes --]

Consider starting from the incorrect default premise in many households:
if you want to get something done, yell at the teenagers to stop
streaming.  Make the point that many people fail to recognize that "manual
traffic management" is really a workaround for busted network devices.  It
would be cool if you had an easily reproduced manual example, such as poor
interactive response on one device while running a speed test on another
device.

To be simplistic, you might just talk about cake vs (bloated) drop tail.
To be thorough, you also need to make the case that cake is better than
other AQMs.  This feels like too much for LWN, but silence on other
solutions might trigger skeptics.

I think your main audience is the extended WRT community - the people who
you want to bake this into future products.   Think about what side issues
might matter to them (no manual config?).    Although gamers also care,
they are an easy sell.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.


On Tue, Sep 8, 2020 at 8:43 AM Dave Taht <dave.taht@gmail.com> wrote:

> Dear David:
>
> I think freeswitch still has a public sip echo server which might be a
> better test than skype?  I tend to think there is a lot of bloat on
> the endpoints nowadays because it "helps" on not having to deal with
> jitter
> that the bloat on the rest of the internet induces. But 1sec you got
> on skype in your test below is NUTS. I hacked a bit on BBB earlier
> this year to get their number down to sanity.
>
> I find endless videoconferencing nowadays to be really tiring and
> frustrating. Am I alone in this? I have to go outside and scream after
> two hours. Log off entirely after 4.
>
> On Mon, Sep 7, 2020 at 10:20 AM David Collier-Brown <davecb.42@gmail.com>
> wrote:
> >
> > Just FYI, I did two tests today, in my Copious Spare Time
> >
> > a DSLReports speed test using an IQrouter, and
> > a Skype "tap" test, with and without the de-bloated router
> >
> > The router did exactly what I expected: it vaporized the bloat-induced
> delay provided by Rogers.
> >
> > My previous bloat-induced lag was between 700 and 900 milliseconds under
> load, and an unimpressive 240 milliseconds when idle.  With a de-bloated
> router, lag was 40 milliseconds maximum, whether idle or loaded, and about
> 30 millisecond minimum, or a 3500 km round-trip, appropriate for the
> distances to the measurements points. By comparison, Rodger-Dogers was
> giving me lag times more appropriate for a test to Ankara, Turkey.
> >
> > The tap test was interesting, but less helpful. I connected to
> 1-909-390-0003 <(909)%20390-0003>, which is an instant-echo server, used
> to test voice telephones. If you connect to it and say something, it will
> echo it back to you. If you tap on your microphone, it's fairly easy to get
> an idea of how much delay there is.
> >
> > Alas, with and without bufferbloat mitigation, it took about a second
> echo back the tap, so it's not the kind of instantly-obvious success test
> I'm searching for, to demonstrate the goodness of CAKE to my grandmother
> (;-))
> >
> > --dave
> >
> > --
> > David Collier-Brown,         | Always do right. This will gratify
> > System Programmer and Author | some people and astonish the rest
> > davecb@spamcop.net           |                      -- Mark Twain
>
>
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> <(831)%20435-0729>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 5270 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Jonathan Morton]

> On 8 Sep, 2020, at 7:48 pm, Matt Mathis via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> To be simplistic, you might just talk about cake vs (bloated) drop tail.  To be thorough, you also need to make the case that cake is better than other AQMs.  This feels like too much for LWN, but silence on other solutions might trigger skeptics.

Personally, my position is:

1: Bloated dumb FIFOs are terrible.

2: Basic AQM is good.  This can be as simple as TBF+WRED; it solves a large part of the basic problem by eliminating multi-second queue delays.  In some cases this can solve very serious problems, such as DNS lookups failing when the link is loaded, quite adequately.  Properly configured, you can keep queue delays below the 100ms threshold for reasonable VoIP performance.

3: FQ-AQM is better.  That generally means HTB+fq_codel, but other forms of this exist.  It means essentially zero added delay for non-saturating flows.  It's an easy way to make DNS, VoIP and online gaming work nicely without having to restrict data-hungry applications.

4: Cake offers some extra tools and aims to be easier (more intuitive) to configure.  Currently, it is the best solution for slow and medium-speed broadband (up to 100Mbps), and can also be used at higher speeds with some care, mostly regarding device performance.

 - Jonathan Morton

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Anthony Minessale II]

[-- Attachment #1: Type: text/plain, Size: 2158 bytes --]

Still willing to host a call on SignalWire Work if you want to check it out.
I barely use this email addr so I keep forgetting to check it.


On Tue, Sep 8, 2020 at 12:09 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 8 Sep, 2020, at 7:48 pm, Matt Mathis via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> >
> > To be simplistic, you might just talk about cake vs (bloated) drop
> tail.  To be thorough, you also need to make the case that cake is better
> than other AQMs.  This feels like too much for LWN, but silence on other
> solutions might trigger skeptics.
>
> Personally, my position is:
>
> 1: Bloated dumb FIFOs are terrible.
>
> 2: Basic AQM is good.  This can be as simple as TBF+WRED; it solves a
> large part of the basic problem by eliminating multi-second queue delays.
> In some cases this can solve very serious problems, such as DNS lookups
> failing when the link is loaded, quite adequately.  Properly configured,
> you can keep queue delays below the 100ms threshold for reasonable VoIP
> performance.
>
> 3: FQ-AQM is better.  That generally means HTB+fq_codel, but other forms
> of this exist.  It means essentially zero added delay for non-saturating
> flows.  It's an easy way to make DNS, VoIP and online gaming work nicely
> without having to restrict data-hungry applications.
>
> 4: Cake offers some extra tools and aims to be easier (more intuitive) to
> configure.  Currently, it is the best solution for slow and medium-speed
> broadband (up to 100Mbps), and can also be used at higher speeds with some
> care, mostly regarding device performance.
>
>  - Jonathan Morton
>
>

-- 

[image: Inline image 1]

Anthony Minessale II | President

FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
<https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>

Email: anthm@freeswitch.com

Mobile: +12623098501

Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>

[image: color-facebook-96.png] <https://www.facebook.com/freeswitch/>[image:
color-twitter-96.png]
<https://twitter.com/freeswitch?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor>

[-- Attachment #2: Type: text/html, Size: 6034 bytes --]

Re: [Bloat] How about a topical LWN article on demonstrating the real-world goodness of CAKE?

[Dave Collier-Brown]

[-- Attachment #1: Type: text/plain, Size: 3626 bytes --]

Thanks, I may take you up on that, but I still have some blockers to work through with my victims^h^h^h^h^h^h^h colleagues at work. (;-))

--dave

On 2020-09-10 11:08 a.m., Anthony Minessale II wrote:
Still willing to host a call on SignalWire Work if you want to check it out.
I barely use this email addr so I keep forgetting to check it.


On Tue, Sep 8, 2020 at 12:09 PM Jonathan Morton <chromatix99@gmail.com<mailto:chromatix99@gmail.com>> wrote:
> On 8 Sep, 2020, at 7:48 pm, Matt Mathis via Bloat <bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net>> wrote:
>
> To be simplistic, you might just talk about cake vs (bloated) drop tail.  To be thorough, you also need to make the case that cake is better than other AQMs.  This feels like too much for LWN, but silence on other solutions might trigger skeptics.

Personally, my position is:

1: Bloated dumb FIFOs are terrible.

2: Basic AQM is good.  This can be as simple as TBF+WRED; it solves a large part of the basic problem by eliminating multi-second queue delays.  In some cases this can solve very serious problems, such as DNS lookups failing when the link is loaded, quite adequately.  Properly configured, you can keep queue delays below the 100ms threshold for reasonable VoIP performance.

3: FQ-AQM is better.  That generally means HTB+fq_codel, but other forms of this exist.  It means essentially zero added delay for non-saturating flows.  It's an easy way to make DNS, VoIP and online gaming work nicely without having to restrict data-hungry applications.

4: Cake offers some extra tools and aims to be easier (more intuitive) to configure.  Currently, it is the best solution for slow and medium-speed broadband (up to 100Mbps), and can also be used at higher speeds with some care, mostly regarding device performance.

 - Jonathan Morton



--

[Inline image 1]

Anthony Minessale II | President

FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045<https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>

Email: anthm@freeswitch.com<mailto:anthm@freeswitch.com>

Mobile: +12623098501<tel:+12623098501>

Website: https://www.FreeSWITCH.com<https://www.freeswitch.com/>

[color-facebook-96.png]<https://www.facebook.com/freeswitch/>[color-twitter-96.png]<https://twitter.com/freeswitch?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor>

--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com<mailto:dave.collier-brown@indexexchange.com> |              -- Mark Twain



CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

[-- Attachment #2: Type: text/html, Size: 8326 bytes --]