[Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Daniel Sterling]

Good day list,

I am looking for input / discussion on how to achieve:

* on a "regular" SoHo network

* first and foremost, to the exclusion of all other goals, consistent
low-latency for non-bulk streams from particular endpoints; usually
those streams are easily identified and differentiated from all other
streams based on UDP/TCP port number,

* and assuming the identified and prioritized streams behave
themselves and stay non-bulk, decent throughput for all other traffic.


That is to say, some endpoints are more important than others; and
moreover some apps on some endpoints are most important.


In my case the traffic that needs to be low-latency no matter what is
xbox FPS gaming traffic (Call of Duty UDP traffic is especially easy
to prioritize, as it typically uses port 3075).


How would you customize your network to achieve those goals?


Here is what I have done; please provide any and all feedback:


I put a linux laptop between CPE (WAN) and LAN. AT&T fiber in my case,
100+ mbit up and down.

I've a tc script that drastically limits bandwidth for non-prioritized
traffic (where priority is based on UDP/TCP port number).

The theory is that this ensures prioritized traffic always has plenty
of available bandwidth to send / receive data, and will never
experience latency unless it's misbehaved / incorrectly classified.



Here is what I have NOT done:

I didn't use tc_cake WITHOUT adding a layer of port-based
prioritization. On a typical SoHo network, I don't see how it would be
possible to ensure xbox traffic never experiences latency w/o
prioritizing streams *before* cake sees them.


Does this make sense?

Here is my script:
https://gist.github.com/eqhmcow/1278a928d11279cb5846688e05dfd363#file-prio-cake-sh


Thanks,
Dan

Re: [Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Toke Høiland-Jørgensen]

Daniel Sterling <sterling.daniel@gmail.com> writes:

> Good day list,
>
> I am looking for input / discussion on how to achieve:
>
> * on a "regular" SoHo network
>
> * first and foremost, to the exclusion of all other goals, consistent
> low-latency for non-bulk streams from particular endpoints; usually
> those streams are easily identified and differentiated from all other
> streams based on UDP/TCP port number,
>
> * and assuming the identified and prioritized streams behave
> themselves and stay non-bulk, decent throughput for all other traffic.
>
>
> That is to say, some endpoints are more important than others; and
> moreover some apps on some endpoints are most important.
>
>
> In my case the traffic that needs to be low-latency no matter what is
> xbox FPS gaming traffic (Call of Duty UDP traffic is especially easy
> to prioritize, as it typically uses port 3075).
>
>
> How would you customize your network to achieve those goals?
>
>
> Here is what I have done; please provide any and all feedback:
>
>
> I put a linux laptop between CPE (WAN) and LAN. AT&T fiber in my case,
> 100+ mbit up and down.
>
> I've a tc script that drastically limits bandwidth for non-prioritized
> traffic (where priority is based on UDP/TCP port number).
>
> The theory is that this ensures prioritized traffic always has plenty
> of available bandwidth to send / receive data, and will never
> experience latency unless it's misbehaved / incorrectly classified.
>
>
>
> Here is what I have NOT done:
>
> I didn't use tc_cake WITHOUT adding a layer of port-based
> prioritization. On a typical SoHo network, I don't see how it would be
> possible to ensure xbox traffic never experiences latency w/o
> prioritizing streams *before* cake sees them.
>
>
> Does this make sense?

First off, on a 100mbit+ link, my guess would be that just putting in
cake with nothing other than a bandwidth parameter would achieve the
gaming performance you want. You'd need a *lot* of other bulk flows to
saturate the link to a point where the gaming traffic would no longer be
considered "sparse" (and thus get automatic priority).

That being said, if you do want to prioritise based on port numbers,
CAKE has its own built-in priority tiers (if you don't run it in
'besteffort' mode), and that does support filtering based on TC rules.
So you don't need the prio qdisc, you can just steer the traffic into
the high-prio cake tin...

-Toke

Re: [Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Jonathan Morton]

> On 12 Feb, 2020, at 6:55 am, Daniel Sterling <sterling.daniel@gmail.com> wrote:
> 
> * first and foremost, to the exclusion of all other goals, consistent
> low-latency for non-bulk streams from particular endpoints; usually
> those streams are easily identified and differentiated from all other
> streams based on UDP/TCP port number,

This is the ideal situation for simply deploying Cake without any special effort.  Just tell it the capacity of the link it's controlling, minus a modest margin (say 1% upstream, 5% downstream).

You should be pleasantly surprised by the results.

 - Jonathan Morton

Re: [Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]


Daniel Sterling <sterling.daniel@gmail.com> wrote:
    > I am looking for input / discussion on how to achieve:
    > * on a "regular" SoHo network

    > * first and foremost, to the exclusion of all other goals, consistent
    > low-latency for non-bulk streams from particular endpoints; usually
    > those streams are easily identified and differentiated from all other
    > streams based on UDP/TCP port number,

    > * and assuming the identified and prioritized streams behave
    > themselves and stay non-bulk, decent throughput for all other traffic.


    > That is to say, some endpoints are more important than others; and
    > moreover some apps on some endpoints are most important.

Distinguishing between apps is difficult in IPv4.
IPv6 lets you naturally have many IP addresses, so it could be easier, but
apps need to be taught to use "their" source address.  Or OSes need to force
them, or "containers".

In the specific case of a single network, I'd just do static DHCPv4
allocation and change the parameters on the qos scripts.

In general, I'd want RFC8520 to announce the type of the device, and suggest
a particular class of service.   In the old days, we'd be talking RSVP to
signal desired Diffserv behaviour ("DiffEdge"), but that specification did
not, unfortunately, gain market momentum.

    > I put a linux laptop between CPE (WAN) and LAN. AT&T fiber in my case,
    > 100+ mbit up and down.

It's not a terrible thing to use a laptop, but at 100Mb/s, an Omnia Turris or
equivalenet running latest OpenWRT may be more foolproof. (I'm always the fool)

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Daniel Sterling]

Thanks to all for the input!

Toke, Jonathan -- you were absolutely right!

I sent this email because I -- I thought it inconceivable that "just"
setting a single bandwidth tunable could both:
* enforce / properly rate limit inbound and outbound traffic
* and, simultaneously, *prevent* non-bulk streams from seeing latency.

I know, I know. You've been telling everyone who will listen that cake
works. I just couldn't wrap my head around that possibly being true --

But boy, I was wrong. cake is as amazing as you say.

I got rid of my complex rules and swapped them out for:

cake bandwidth 60Mbit besteffort internet nat ethernet

Then I monitored my xbox game latency as I streamed videos, etc, to
generate bulk traffic.

There was no observable latency or jitter, and I did not see any
issues during actual game-play either.

Once again, I am truly amazed. Thank you to everyone who worked on
this impressive tool!

Ubuntu 19.10 finally ships with all the pieces in place (kernel, new
iproute2 package) -- so cake is now finally usable "out of the box"
for the average linux user. I look forward to telling everyone I know
to have some cake!

:)

Thanks,
Dan

On Fri, Feb 14, 2020 at 9:18 AM Michael Richardson <mcr@sandelman.ca> wrote:
>
>
> Daniel Sterling <sterling.daniel@gmail.com> wrote:
>     > I am looking for input / discussion on how to achieve:
>     > * on a "regular" SoHo network
>
>     > * first and foremost, to the exclusion of all other goals, consistent
>     > low-latency for non-bulk streams from particular endpoints; usually
>     > those streams are easily identified and differentiated from all other
>     > streams based on UDP/TCP port number,
>
>     > * and assuming the identified and prioritized streams behave
>     > themselves and stay non-bulk, decent throughput for all other traffic.
>
>
>     > That is to say, some endpoints are more important than others; and
>     > moreover some apps on some endpoints are most important.
>
> Distinguishing between apps is difficult in IPv4.
> IPv6 lets you naturally have many IP addresses, so it could be easier, but
> apps need to be taught to use "their" source address.  Or OSes need to force
> them, or "containers".
>
> In the specific case of a single network, I'd just do static DHCPv4
> allocation and change the parameters on the qos scripts.
>
> In general, I'd want RFC8520 to announce the type of the device, and suggest
> a particular class of service.   In the old days, we'd be talking RSVP to
> signal desired Diffserv behaviour ("DiffEdge"), but that specification did
> not, unfortunately, gain market momentum.
>
>     > I put a linux laptop between CPE (WAN) and LAN. AT&T fiber in my case,
>     > 100+ mbit up and down.
>
> It's not a terrible thing to use a laptop, but at 100Mb/s, an Omnia Turris or
> equivalenet running latest OpenWRT may be more foolproof. (I'm always the fool)
>
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>

Re: [Bloat] is extremely consistent low-latency for e.g. xbox possible on SoHo networks w/o manual configuration?

[Toke Høiland-Jørgensen]

Daniel Sterling <sterling.daniel@gmail.com> writes:

> Thanks to all for the input!
>
> Toke, Jonathan -- you were absolutely right!
>
> I sent this email because I -- I thought it inconceivable that "just"
> setting a single bandwidth tunable could both:
> * enforce / properly rate limit inbound and outbound traffic
> * and, simultaneously, *prevent* non-bulk streams from seeing latency.
>
> I know, I know. You've been telling everyone who will listen that cake
> works. I just couldn't wrap my head around that possibly being true --
>
> But boy, I was wrong. cake is as amazing as you say.
>
> I got rid of my complex rules and swapped them out for:
>
> cake bandwidth 60Mbit besteffort internet nat ethernet
>
> Then I monitored my xbox game latency as I streamed videos, etc, to
> generate bulk traffic.
>
> There was no observable latency or jitter, and I did not see any
> issues during actual game-play either.
>
> Once again, I am truly amazed. Thank you to everyone who worked on
> this impressive tool!

This is great to hear! And thank you for providing some awesome "happy
user" quotes! ;)

> Ubuntu 19.10 finally ships with all the pieces in place (kernel, new
> iproute2 package) -- so cake is now finally usable "out of the box"
> for the average linux user. I look forward to telling everyone I know
> to have some cake!

Please do! We still have a few billion devices left to upgrade...

-Toke