[Cake] ER-X now running cake, thanks for the help. :)

[Cake] ER-X now running cake, thanks for the help. :)

[erik.taraldsen]

Just a follow-up from the great support I got here.  I am now running ER-X with cake with the precompiled binaries from Nils.  I need to do some tuning and get it properly into the lab.  Currently I'm dogfooding it at home.


Any other suggestions in order to make the topping of the cake so to speek?  I notice for example that all interfaces pr default use pfifo_fast and run a txqueuelen of 1000.  Which is a bit deep for running a 20/1 adsl line.



And a slight rant on the default fq_codel setup for ER-X, 10240 packets?  Before reducing that I got a one way delay of more than 600ms when I congested the line with udp.  Sane defaults


Default from "smart queue" fq_codel

qdisc htb 1: dev eth4 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 100: dev eth4 parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn


-Erik

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Toke Høiland-Jørgensen]

<erik.taraldsen@telenor.com> writes:

> Just a follow-up from the great support I got here.  I am now running ER-X with cake with the precompiled binaries from Nils.  I need to do some tuning and get it properly into the lab.  Currently I'm dogfooding it at home.
>
> Any other suggestions in order to make the topping of the cake so to speek?  I notice for example that all interfaces pr default use pfifo_fast and run a txqueuelen of 1000.  Which is a bit deep for running a 20/1 adsl line.
>
> And a slight rant on the default fq_codel setup for ER-X, 10240
> packets?  Before reducing that I got a one way delay of more than
> 600ms when I congested the line with udp.  Sane defaults

You're not likely to hit the packet limit for fq-codel, since the AQM
will make TCP flows back off before it gets that far. The exception is
an unresponsive UDP flow, of course, but that is only going to hurt
itself.

Main problem with the 10240 limit on small boxes is the memory; an
explicit memory limit was introduces in later versions to deal with
this. Guess that wasn't backported to the edgerouter, though...

-Toke

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Sebastian Moeller]

Hi Erik,


> On May 5, 2017, at 15:00, <erik.taraldsen@telenor.com> <erik.taraldsen@telenor.com> wrote:
> 
> Just a follow-up from the great support I got here.  I am now running ER-X with cake with the precompiled binaries from Nils.  I need to do some tuning and get it properly into the lab.  Currently I'm dogfooding it at home.
> 
> 
> Any other suggestions in order to make the topping of the cake so to speek?

I think https://lede-project.org/docs/howto/sqm pretty much sums up the current best initial recommendations for sqm-scripts.


>  I notice for example that all interfaces pr default use pfifo_fast and run a txqueuelen of 1000.  Which is a bit deep for running a 20/1 adsl line.

	If you specify a bandwidth parameter to cake that is below the egress interface’s true bandwidth the txqueue is never going to fill up, so you can leave it untouched. Also I believe that BQL will effectively, if available and enabled on an interface, make the exact value of txqueuelen irrelevant (I guess it needs to be large enough for BQL to reach a decent maximum).

> 
> 
> 
> And a slight rant on the default fq_codel setup for ER-X, 10240 packets?  Before reducing that I got a one way delay of more than 600ms when I congested the line with udp.  Sane defaults

	Well, for sqm-scripts’ simplest/simple.qos we reduced this to 1001; but the rationale pretty much is that this is really just a belts and suspender thing, fq_codel should make sure under normal conditions that the queue is never fully filled, so ideally this just constrains the worst case memory requirement (roughly, as a non-giant skb is I believe 2KiB in size, so 10240 packet will at worst require 2*10240/1024 = 20 MB, for GRO/GSO giant packets this approximation does not hold anymore, cake I believe has a memlimit keyword that allows to explicit;y state how much memory one allows an instance to consume worst case, cake also reports how much memory it actually consumed). Why 10240, you ask (as did I): for the default of 1024 hashed queues this allows 10 packets in each queue on average… (now if we had another number of digits on our hands than 10 that might be different, but I digress...)
The only way to ever run into this is flooding with an unelastic load (I first run into the 10240 packets = 20 MB packets when UDP floods with randomised ports knocked out my puny 64mb router reliably in a few seconds). With ehe ERX’s 256 MB I would guess the worst case 20MiB is small change and I would not bother to change that. One problem with the unelastic load is that as far as I can tell no flow on the open internet is allows/assumed to behave like that (isn’t the default tcp-friendly, and inelastic DOS traffic is essentially out-lawed?)


Best Regards
	Sebastian

> 
> 
> Default from "smart queue" fq_codel
> 
> qdisc htb 1: dev eth4 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 direct_qlen 1000
> qdisc fq_codel 100: dev eth4 parent 1:10 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
> 
> 
> -Erik
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] ER-X now running cake, thanks for the help. :)

[erik.taraldsen]

> Fra: Sebastian Moeller <moeller0@gmx.de>

> .. snip <good explanations and tips> 
> One problem with the unelastic load is that as far as I can tell no flow on the open internet is 
> allows/assumed to behave like that (isn’t the default tcp-friendly, and inelastic DOS traffic is 
> essentially out-lawed?)

You are right in that single stream udp without any feedback is a quite unrealistic load.  It is however usually a good indicator of possible trouble with bufferbloat.  When I get delays in the multiple seconds range there usually is no point in running more realistic tests such as flent, so I tend to start with that udp test before setting up a lab.  Current record found in production systems is 34 seconds.

I'll redo the test using flent in the lab next week, using both fq_codel and cake.

-Erik

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Toke Høiland-Jørgensen]

<erik.taraldsen@telenor.com> writes:

>> Fra: Sebastian Moeller <moeller0@gmx.de>
>
>> .. snip <good explanations and tips> 
>> One problem with the unelastic load is that as far as I can tell no flow on the open internet is 
>> allows/assumed to behave like that (isn’t the default tcp-friendly, and inelastic DOS traffic is 
>> essentially out-lawed?)
>
> You are right in that single stream udp without any feedback is a
> quite unrealistic load. It is however usually a good indicator of
> possible trouble with bufferbloat. When I get delays in the multiple
> seconds range there usually is no point in running more realistic
> tests such as flent, so I tend to start with that udp test before
> setting up a lab. Current record found in production systems is 34
> seconds.

How are you measuring latency in this case?

-Toke

Re: [Cake] ER-X now running cake, thanks for the help. :)

[erik.taraldsen]

> Den 5. mai 2017 kl. 15.59 skrev Toke Høiland-Jørgensen <toke@toke.dk>:
> 
> How are you measuring latency in this case?

I use nuttcp with the option -o. It uses system time on the two peers. Make sure they are recently synced to the same ntp server. And do an uncongested test to establish actual network time distance, before the actual congestion starts. 

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Toke Høiland-Jørgensen]

On 5 May 2017 16:30:38 GMT+02:00, erik.taraldsen@telenor.com wrote:
>
>> Den 5. mai 2017 kl. 15.59 skrev Toke Høiland-Jørgensen
><toke@toke.dk>:
>> 
>> How are you measuring latency in this case?
>
>I use nuttcp with the option -o. It uses system time on the two peers.
>Make sure they are recently synced to the same ntp server. And do an
>uncongested test to establish actual network time distance, before the
>actual congestion starts. 

Right, so you're measuring the actual latency experienced by the UDP packets in the unresponsive flow? Yeah, you'll get quite a bit of latency in that case. But not sure that's a case we should be optimising for...

-Toke

Re: [Cake] ER-X now running cake, thanks for the help. :)

[erik.taraldsen]

> Fra: Toke Høiland-Jørgensen <toke@toke.dk>
>
> Right, so you're measuring the actual latency experienced by the UDP packets in the 
> unresponsive flow? Yeah, you'll get quite a bit of latency in that case. But not sure that's a case 
> we should be optimising for...

That was just the worst case scenario, not really meant to emulate user traffic.  And I'm not suggesting that the schedulers should be optimized for that. But we can agree that 20+ seconds of delay is a bit much no matter how it is measured? :)

I also use nuttcp to model other kinds of traffic, such as VIOP, Voice over WiFi, dumb udp vod that Telenor uses.  Then I subject the router to various kinds of line saturation and stress.  Cloud upload, torrent, virus emulation (zmap not really designed for that, but man, if you want to inflict pain on a router... zmap is your daddy).  And we try to make sure that long living tcp/vpn sessions are not torn down due to need for more sessions in nat for yet another torrent session.  Different QoS/schedulers behave different and we try and use the best possible solution for a given HW/access tech.  What works very well for balancing cloud upload and interactive traffic like ssh or http may fail miserably on torrent.  And the other way around.  On our legacy stuff we can't implement fq_codel, cake or sfq even. 

And as you know I also incorporate flent. 


-Erik

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Neil Shepperd]

[-- Attachment #1: Type: text/plain, Size: 2004 bytes --]

Shouldn't unresponsive UDP flows be punished for sending too many packets?
In that case if there's any problem at all it is that the packets get
through with 20 seconds delay instead of being dropped entirely. Although
it shouldn't really matter either way as long as other flows are not
unfairly impacted.

On Fri, 5 May 2017 at 13:12 <erik.taraldsen@telenor.com> wrote:

> > Fra: Toke Høiland-Jørgensen <toke@toke.dk>
> >
> > Right, so you're measuring the actual latency experienced by the UDP
> packets in the
> > unresponsive flow? Yeah, you'll get quite a bit of latency in that case.
> But not sure that's a case
> > we should be optimising for...
>
> That was just the worst case scenario, not really meant to emulate user
> traffic.  And I'm not suggesting that the schedulers should be optimized
> for that. But we can agree that 20+ seconds of delay is a bit much no
> matter how it is measured? :)
>
> I also use nuttcp to model other kinds of traffic, such as VIOP, Voice
> over WiFi, dumb udp vod that Telenor uses.  Then I subject the router to
> various kinds of line saturation and stress.  Cloud upload, torrent, virus
> emulation (zmap not really designed for that, but man, if you want to
> inflict pain on a router... zmap is your daddy).  And we try to make sure
> that long living tcp/vpn sessions are not torn down due to need for more
> sessions in nat for yet another torrent session.  Different QoS/schedulers
> behave different and we try and use the best possible solution for a given
> HW/access tech.  What works very well for balancing cloud upload and
> interactive traffic like ssh or http may fail miserably on torrent.  And
> the other way around.  On our legacy stuff we can't implement fq_codel,
> cake or sfq even.
>
> And as you know I also incorporate flent.
>
>
> -Erik
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 2500 bytes --]

Re: [Cake] ER-X now running cake, thanks for the help. :)

[Sebastian Moeller]

Hi Erik,


> On May 5, 2017, at 15:55, erik.taraldsen@telenor.com wrote:
> 
>> Fra: Sebastian Moeller <moeller0@gmx.de>
> 
>> .. snip <good explanations and tips> 
>> One problem with the unelastic load is that as far as I can tell no flow on the open internet is 
>> allows/assumed to behave like that (isn’t the default tcp-friendly, and inelastic DOS traffic is 
>> essentially out-lawed?)
> 
> You are right in that single stream udp without any feedback is a quite unrealistic load.  It is however usually a good indicator of possible trouble with bufferbloat.  When I get delays in the multiple seconds range there usually is no point in running more realistic tests such as flent, so I tend to start with that udp test before setting up a lab.  Current record found in production systems is 34 seconds.

	I beg to differ somewhat. With fq_codel and cake the high-bandwidth in-elastic UDP flow (heck an in-elastic TCP flow would be the same) really is just testing the DOS case and effectively probing the worst case memory buffer behind the AQM. So I believe this is a valid test, but it is not a show-stopper if this test shows deep buffers. Netalyzr basically does that (and it also measures for short durations so that fq_codel/cake have problems ramping their drop probability high enough in time to make dent into the load). Under normal operating conditions however that buffer will never fill up and hence can be considered harmless (or actually helpfull, as it allows to straighten out bursty traffic).

Best Regards
	Sebastian


> 
> I'll redo the test using flent in the lab next week, using both fq_codel and cake.
> 
> -Erik