[Bloat] cake + ipv6

[Bloat] cake + ipv6

[Daniel Sterling]

As you know, I'm here cuz I have an xbox and y'all created cake, which
I am eternally grateful for, since it makes latency go away.

But I've recently hit an interesting issue --

Microsoft (and/or akamai, or whatever) has recently started pushing
updates to the xbox via ipv6 instead of v4.

As I'm sure you know ipv6 addresses are essentially random on the
internal LAN as compared to v4 -- a box can grab as many v6 addresses
as it wants, and I don't believe my linux router can really know which
box is using which address, can it?

Which means... ipv6 breaks cake's flow isolation.

Cake can't throttle all those xbox downloads correctly cuz it doesn't
know they're all going to/from that one device.

So I suppose this may be similar to the "bittorrent" problem -- which,
is there a general solution for that problem?

In my case the xbox grabs more than its share of bandwidth, which
means other bulk streaming -- that is to say, youtube and netflix :)
-- stops working well

I can think of one general solution -- run more wires to more devices,
and give devices their own VLAN, and tag / prioritize / deprioritize
specific traffic that way...

But.. are there better / more general solutions?

Thanks,
Dan

Re: [Bloat] cake + ipv6

[Jonathan Morton]

On Tuesday, 18 August 2020, Daniel Sterling wrote:
> As you know, I'm here cuz I have an xbox and y'all created cake, which
> I am eternally grateful for, since it makes latency go away.
> 
> But I've recently hit an interesting issue --
> 
> Microsoft (and/or akamai, or whatever) has recently started pushing
> updates to the xbox via ipv6 instead of v4.
> 
> As I'm sure you know ipv6 addresses are essentially random on the
> internal LAN as compared to v4 -- a box can grab as many v6 addresses
> as it wants, and I don't believe my linux router can really know which
> box is using which address, can it?
> 
> Which means... ipv6 breaks cake's flow isolation.
> 
> Cake can't throttle all those xbox downloads correctly cuz it doesn't
> know they're all going to/from that one device.
> 
> So I suppose this may be similar to the "bittorrent" problem -- which,
> is there a general solution for that problem?
> 
> In my case the xbox grabs more than its share of bandwidth, which
> means other bulk streaming -- that is to say, youtube and netflix :)
> -- stops working well
> 
> I can think of one general solution -- run more wires to more devices,
> and give devices their own VLAN, and tag / prioritize / deprioritize
> specific traffic that way...
> 
> But.. are there better / more general solutions?

Does this traffic at least have some consistent means of identification, such as a port number or a remote address range?  If so, you could use fwmark rules and Cake's diffserv3 mode to put that traffic in the Bulk tin, same as with BitTorrent.

I suppose it's also possible to make Cake sensitive to Layer 2 addresses (that is, the Ethernet address) for the purpose of host isolation.  That is presently not implemented, so might take a while to filter through the deployment range.

Re: [Bloat] cake + ipv6

[Kenneth Porter]

--On Monday, August 17, 2020 10:52 PM -0400 Daniel Sterling 
<sterling.daniel@gmail.com> wrote:

> As I'm sure you know ipv6 addresses are essentially random on the
> internal LAN as compared to v4 -- a box can grab as many v6 addresses
> as it wants, and I don't believe my linux router can really know which
> box is using which address, can it?

Is this the usual IPv6 allocation by autoconfig or is the Xbox grabbing 
extra addresses deliberately to break flow isolation? It should only 
advertise one public address for its updates.

Re: [Bloat] cake + ipv6

[Daniel Sterling]

This is a very good question, I will stare at "iftop" (and check the
xbox's network settings, if I can), the next time I notice it's
updating.

The report from my "users" :) was that pausing xbox updates solved
their video streaming issues -- but, it may be that there was other
network traffic or that multiple systems were doing updates, and the
update they paused gave back enough bandwidth for their streams.

I will look at traffic summore and let you know if I was mistaken
about the xbox. It is now using ipv6 for at least some updates, I do
know that much for certain, but hopefully I'm wrong about it using too
many v6 addresses.

Jonathan, thanks for the tips re: using ports or remote addresses --

Hmm, I don't know -- is it possible to identify (and thus classify)
plain old bulk downloads, as separate from video streams? They're both
going to use http / https (or possibly QUIC) -- and they're both
likely to come from CDN networks... I can't think of a simple way to
tell them apart.

Is this enough of a problem that people would try to make a list of
netblocks / prefixes that belong to video vs other CDN content?

I do notice video streams are much more bursty than plain downloads
for me, but that may not hold for all users.

That is, for me at least, a video stream may average 5mbps over, say,
1 minute, but it will sit at 0mbps for a while and then burst at
20mbps for a bit.

I can't think of a way to mark such traffic though. Is there a place
where such complex, custom rules for marking packets would sit?
Perhaps a userspace daemon that looks at traffic with pcap?

Thanks,
Dan

On Mon, Aug 17, 2020 at 10:55 PM Kenneth Porter <shiva@sewingwitch.com> wrote:
>
> --On Monday, August 17, 2020 10:52 PM -0400 Daniel Sterling
> <sterling.daniel@gmail.com> wrote:
>
> > As I'm sure you know ipv6 addresses are essentially random on the
> > internal LAN as compared to v4 -- a box can grab as many v6 addresses
> > as it wants, and I don't believe my linux router can really know which
> > box is using which address, can it?
>
> Is this the usual IPv6 allocation by autoconfig or is the Xbox grabbing
> extra addresses deliberately to break flow isolation? It should only
> advertise one public address for its updates.
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] cake + ipv6

[Jonathan Morton]

On 18/08/2020 06:44, Daniel Sterling wrote:
> ...is it possible to identify (and thus classify)
> plain old bulk downloads, as separate from video streams? They're both
> going to use http / https (or possibly QUIC) -- and they're both
> likely to come from CDN networks... I can't think of a simple way to
> tell them apart.

If there was an easy way to do it, I would already have done so.  We are 
unfortunately hamstrung by some bad design and deployment around 
Diffserv, which might otherwise provide a useful end-to-end visible 
signal here.

> Is this enough of a problem that people would try to make a list of
> netblocks / prefixes that belong to video vs other CDN content?

It's possible that someone is doing this, but I don't specifically know 
of such a source of information.  It would of course be better to find a 
solution that didn't rely on white/black lists, which have a distressing 
habit of going stale.

But one of the more reliable ways might be to use Autonomous System (AS) 
information.  ASes are an organisational unit used for assigning IP 
address ranges and for routing, and usually correspond to a more-or-less 
significant Internet organisation.  It should be feasible to map an 
observed IP address to an AS, then look up the address blocks assigned 
to that AS, thereby capturing a whole range of related IP addresses.

> I do notice video streams are much more bursty than plain downloads
> for me, but that may not hold for all users.
> 
> That is, for me at least, a video stream may average 5mbps over, say,
> 1 minute, but it will sit at 0mbps for a while and then burst at
> 20mbps for a bit.

Correct, YouTube at least likes to fetch a big block of data from disk 
and send it all at once, then rely on the client buffer to tide it over 
while the disk services other requests.  It makes some sense when you 
consider how slow disk seeks are relative to the number of clients they 
need to support, each of which will generally be watching a different 
video (or at least a different part of the same one).

However, this burstiness disappears on the wire just when you would like 
to use it to identify traffic, ie. when the video traffic saturates the 
bandwidth available to it.  If there's only just enough bandwidth, or 
even *less* than what is required, then YouTube sends data continuously 
into the client buffer, trying to keep it as full as possible.

There are no easy answers here.  But I've suggested some things to look 
for and try out.

  - Jonathan Morton

Re: [Bloat] cake + ipv6

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 4169 bytes --]

Hi there,

A few related thoughts. Some people use sldnsmasq to populate ipsets based on dnsnames to figure out which IP addresses are used by, say YouTube, and then dscp mark them to their own desire. Others use iptables rules to detect bulk flows by their accumulated  transfervolume and then dscp mark them. The common thread here is that these techniques fo not work with the IFB way of ingress shaping, since the IFB runs before iptables/dnsmadq/ipset actually has seen the packets and hence they carry their original useless DSCP marks. If your router does not use wifi, you can instantiate the ingress Shaper on the egress side of the interface from the CPU to the lan side, or you can use a veth pair to route ingress traffic into the lan bridge and then instantiate the ingress Shaper on the egress side of that veth...

Best Regards
        Sebastian

P.S.: As far as I can tell IPv6 end hosts will cycle through new addresses, but typically should only use a few concurrently. It would be quite interesting to figure out whether the Xbox updates truly use multiple IP addresses in the first place or whether this is a thundering herd problem caused by synchronized updates by multiple hosts, no?

On 18 August 2020 06:15:55 CEST, Jonathan Morton <chromatix99@gmail.com> wrote:
>On 18/08/2020 06:44, Daniel Sterling wrote:
>> ...is it possible to identify (and thus classify)
>> plain old bulk downloads, as separate from video streams? They're
>both
>> going to use http / https (or possibly QUIC) -- and they're both
>> likely to come from CDN networks... I can't think of a simple way to
>> tell them apart.
>
>If there was an easy way to do it, I would already have done so.  We
>are 
>unfortunately hamstrung by some bad design and deployment around 
>Diffserv, which might otherwise provide a useful end-to-end visible 
>signal here.
>
>> Is this enough of a problem that people would try to make a list of
>> netblocks / prefixes that belong to video vs other CDN content?
>
>It's possible that someone is doing this, but I don't specifically know
>
>of such a source of information.  It would of course be better to find
>a 
>solution that didn't rely on white/black lists, which have a
>distressing 
>habit of going stale.
>
>But one of the more reliable ways might be to use Autonomous System
>(AS) 
>information.  ASes are an organisational unit used for assigning IP 
>address ranges and for routing, and usually correspond to a
>more-or-less 
>significant Internet organisation.  It should be feasible to map an 
>observed IP address to an AS, then look up the address blocks assigned 
>to that AS, thereby capturing a whole range of related IP addresses.
>
>> I do notice video streams are much more bursty than plain downloads
>> for me, but that may not hold for all users.
>> 
>> That is, for me at least, a video stream may average 5mbps over, say,
>> 1 minute, but it will sit at 0mbps for a while and then burst at
>> 20mbps for a bit.
>
>Correct, YouTube at least likes to fetch a big block of data from disk 
>and send it all at once, then rely on the client buffer to tide it over
>
>while the disk services other requests.  It makes some sense when you 
>consider how slow disk seeks are relative to the number of clients they
>
>need to support, each of which will generally be watching a different 
>video (or at least a different part of the same one).
>
>However, this burstiness disappears on the wire just when you would
>like 
>to use it to identify traffic, ie. when the video traffic saturates the
>
>bandwidth available to it.  If there's only just enough bandwidth, or 
>even *less* than what is required, then YouTube sends data continuously
>
>into the client buffer, trying to keep it as full as possible.
>
>There are no easy answers here.  But I've suggested some things to look
>
>for and try out.
>
>  - Jonathan Morton
>_______________________________________________
>Bloat mailing list
>Bloat@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/bloat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 4839 bytes --]

Re: [Bloat] cake + ipv6

[Y]

Hi.

Did you try to use dual-src(dst)host option?

This is my setting.

pi@raspberrypi:~ $ tc -s qdisc show dev eth0 qdisc cake 8021: root
refcnt 2 bandwidth 831Kbit besteffort dual-srchost nonat nowash
ack-filter split-gso rtt 100.0ms atm overhead 10
pi@raspberrypi:~ $ tc -s qdisc show dev ifb4eth0 qdisc cake 8022: root
refcnt 2 bandwidth 10833Kbit besteffort dual-dsthost nonat wash
ack-filter split-gso rtt 100.0ms atm overhead 10 


On Mon, 17 Aug 2020 21:52:43 -0400
Daniel Sterling <sterling.daniel@gmail.com> wrote:

> As you know, I'm here cuz I have an xbox and y'all created cake, which
> I am eternally grateful for, since it makes latency go away.
> 
> But I've recently hit an interesting issue --
> 
> Microsoft (and/or akamai, or whatever) has recently started pushing
> updates to the xbox via ipv6 instead of v4.
> 
> As I'm sure you know ipv6 addresses are essentially random on the
> internal LAN as compared to v4 -- a box can grab as many v6 addresses
> as it wants, and I don't believe my linux router can really know which
> box is using which address, can it?
> 
> Which means... ipv6 breaks cake's flow isolation.
> 
> Cake can't throttle all those xbox downloads correctly cuz it doesn't
> know they're all going to/from that one device.
> 
> So I suppose this may be similar to the "bittorrent" problem -- which,
> is there a general solution for that problem?
> 
> In my case the xbox grabs more than its share of bandwidth, which
> means other bulk streaming -- that is to say, youtube and netflix :)
> -- stops working well
> 
> I can think of one general solution -- run more wires to more devices,
> and give devices their own VLAN, and tag / prioritize / deprioritize
> specific traffic that way...
> 
> But.. are there better / more general solutions?
> 
> Thanks,
> Dan
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] cake + ipv6

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]


Daniel Sterling <sterling.daniel@gmail.com> wrote:
    > As I'm sure you know ipv6 addresses are essentially random on the
    > internal LAN as compared to v4 -- a box can grab as many v6 addresses
    > as it wants, and I don't believe my linux router can really know which
    > box is using which address, can it?

So, IPv6 privacy extensions do this.
But, they aren't really supposed to change that fast, and should ideally be
sticky to an application.  So a single batch of xbox downloads should all use
the same IID in the IPv6.

On a home *LAN*, the L2 ethernet address won't change for the xbox, so ideally,
the home router could build some kind of set based upon that, and then use
that.

Adding a layer of L3 router, of course, defeats that, and this is mostly the
point of how privacy extensions are supposed to work.

    > Which means... ipv6 breaks cake's flow isolation.

    > Cake can't throttle all those xbox downloads correctly cuz it doesn't
    > know they're all going to/from that one device.

Are you explicitely throttling by IP address, or are you letting cake figure
it out?

    > I can think of one general solution -- run more wires to more devices,
    > and give devices their own VLAN, and tag / prioritize / deprioritize
    > specific traffic that way...

    > But.. are there better / more general solutions?

--
Michael Richardson <mcr+IETF@sandelman.ca>, Sandelman Software Works
 -= IPv6 IoT consulting =-

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] cake + ipv6

[Daniel Sterling]

Thanks to everyone for their input,

I ran some updates on the xbox and watched iftop. I found that the
xbox does the following:

* uses up to four http (TCP port 80) connections at once to download data
* connects (seemingly randomly) to both ipv4 and ipv6 update hosts

That means at any given time, the xbox could be downloading solely via
ipv4, solely via ipv6, or a with mix of the two.

I believe this means when it's using both v4 and v6, it's getting
double its "share" of the bandwidth since cake can't tell that the v4
and v6 traffic is coming from the same LAN host -- is that correct?

I'm using the default "triple-isolate" parameter. I can try switching
to dual-src/dest host or even plain srchost / dsthost isolation. In
theory that should limit traffic more per download host, even if cake
can't determine the LAN host that's doing the downloading, right?

For reference here is my tc config:

root@OpenWrt:~# tc -s qdisc | grep cake
qdisc cake 1: dev eth1 root refcnt 2 bandwidth 20Mbit besteffort
triple-isolate nat nowash no-ack-filter split-gso rtt 100.0ms noatm
overhead 38 mpu 84
qdisc cake 1: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort
triple-isolate nat nowash ingress no-ack-filter split-gso rtt 100.0ms
noatm overhead 38 mpu 84

eth1 is WAN, eth0 is LAN. This is openwrt running on a sandy bridge
PC, so both are real NICs. eth0 is an intel PCI NIC, eth1 is
motherboard NIC.

root@OpenWrt:~# uname -a
Linux OpenWrt 4.14.195 #0 SMP Mon Sep 7 18:27:30 2020 x86_64 GNU/Linux

root@OpenWrt:~# cat /proc/cpuinfo | grep name
model name      : Intel(R) Celeron(R) CPU 847 @ 1.10GHz

root@OpenWrt:~# for i in 0 1 ; do ethtool -i eth$i | grep driver ; done
driver: e1000e
driver: r8169

Thanks,
Dan

On Tue, Aug 18, 2020 at 5:55 PM Michael Richardson
<mcr+ietf@sandelman.ca> wrote:
>
>
> Daniel Sterling <sterling.daniel@gmail.com> wrote:
>     > As I'm sure you know ipv6 addresses are essentially random on the
>     > internal LAN as compared to v4 -- a box can grab as many v6 addresses
>     > as it wants, and I don't believe my linux router can really know which
>     > box is using which address, can it?
>
> So, IPv6 privacy extensions do this.
> But, they aren't really supposed to change that fast, and should ideally be
> sticky to an application.  So a single batch of xbox downloads should all use
> the same IID in the IPv6.
>
> On a home *LAN*, the L2 ethernet address won't change for the xbox, so ideally,
> the home router could build some kind of set based upon that, and then use
> that.
>
> Adding a layer of L3 router, of course, defeats that, and this is mostly the
> point of how privacy extensions are supposed to work.
>
>     > Which means... ipv6 breaks cake's flow isolation.
>
>     > Cake can't throttle all those xbox downloads correctly cuz it doesn't
>     > know they're all going to/from that one device.
>
> Are you explicitely throttling by IP address, or are you letting cake figure
> it out?
>
>     > I can think of one general solution -- run more wires to more devices,
>     > and give devices their own VLAN, and tag / prioritize / deprioritize
>     > specific traffic that way...
>
>     > But.. are there better / more general solutions?
>
> --
> Michael Richardson <mcr+IETF@sandelman.ca>, Sandelman Software Works
>  -= IPv6 IoT consulting =-

Re: [Bloat] cake + ipv6

[Jonathan Morton]

> On 23 Sep, 2020, at 8:36 pm, Daniel Sterling <sterling.daniel@gmail.com> wrote:
> 
> I ran some updates on the xbox and watched iftop. I found that the
> xbox does the following:
> 
> * uses up to four http (TCP port 80) connections at once to download data
> * connects (seemingly randomly) to both ipv4 and ipv6 update hosts
> 
> That means at any given time, the xbox could be downloading solely via
> ipv4, solely via ipv6, or a with mix of the two.
> 
> I believe this means when it's using both v4 and v6, it's getting
> double its "share" of the bandwidth since cake can't tell that the v4
> and v6 traffic is coming from the same LAN host -- is that correct?

It fits my mental model, yes, though obviously the ideal would be to recognise that the xbox is a singular machine.  Are you seeing a larger disparity than that?  If so, is it even larger than four connections would justify without host-fairness?

> I'm using the default "triple-isolate" parameter. I can try switching
> to dual-src/dest host or even plain srchost / dsthost isolation. In
> theory that should limit traffic more per download host, even if cake
> can't determine the LAN host that's doing the downloading, right?

Triple-isolate is designed to function reasonably well when the user can't be sure which side of the network is the LAN!  The "dual" modes provide Cake with that information explicitly, so may be more reliable in corner cases.

For your topology, eth0 (LAN egress) should get dual-dsthost, and eth1 (WAN egress) should get dual-srchost.

 - Jonathan Morton

Re: [Bloat] cake + ipv6

[Daniel Sterling]

On Wed, Sep 23, 2020 at 2:13 PM Jonathan Morton <chromatix99@gmail.com> wrote:
> It fits my mental model, yes, though obviously the ideal would be to recognise that the xbox is a singular machine.  Are you seeing a larger disparity than that?  If so, is it even larger than four connections would justify without host-fairness?

Thanks Jonathan,

Unfortunately I fear I've been running an improperly configured cake
all this time-- leave it to me to take a piece of cake and mess it up
:) I should say, even with it being possibly misconfigured, it has
still worked amazingly well for me! I have no end of gratitude for
y'alls work --

So taking your advice, with my cake properly configured, the answer is
no -- it looks like host-fairness is working correctly -- with a
notable exception -- read on for details...

I was using these probably wrong settings:

WAN: besteffort bandwidth $UPBANDWIDTH internet nat egress ethernet
LAN: besteffort bandwidth $BANDWIDTH internet nat ingress ethernet

So, I was using triple-isolate instead of the correct src/dest
isolation options per NIC, and I was using "nat" on the LAN side,
which may have been further breaking host isolation as per the openwrt
wiki:

"don't add the nat option on LAN interfaces (this option should only
be used in the WAN interface) or Per-Host Isolation stops working" -
https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details

I changed my configuration to:

$WAN handle 1: root cake besteffort bandwidth $UPBANDWIDTH internet
nat egress dual-srchost ethernet
$LAN handle 1: root cake besteffort bandwidth $BANDWIDTH internet
ingress dual-dsthost ethernet

So:

root@OpenWrt:~# tc -s qdisc | grep cake
qdisc cake 1: dev eth1 root refcnt 2 bandwidth 20Mbit besteffort
dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm
overhead 38 mpu 84
qdisc cake 1: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort
dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100.0ms
noatm overhead 38 mpu 84

WAN:
* nat option enabled
* egress option (default) enabled
* dual-srchost

LAN:
* nat option disabled (default)
* ingress option enabled
* dual-dsthost

This worked well, but I ran into a bizarre issue:

My xbox FPS gaming UDP stream would start at a low latency (even while
other systems were doing bulk downloads), but after some time, the
latency went from a steady 30ms (+/- <10ms for jitter) to over 100ms !
100ms is of course notably the default cake rtt parameter. Resetting
cake (tc del and then tc add again) fixed this; the same stream's
latency went back down to about 30ms.

This gaming stream uses well under 512kbps down and less than half
that up, normally.

This is worrisome! My use-case may be an edge case, as I have AT&T
gigabit fiber but I limit it to 40mbit with cake, since empirically
that's the highest throughput I've found that I can use while also
maintaining the low jitter / latency over wifi. So certainly the WAN
NIC is getting a lot of traffic that cake on the LAN NIC is throwing
away. That is, there is basically always a large number of bytes in
LAN cake's backlog.

For now I've solved this by using the "prio" tc qdisc such that xbox
traffic bypasses cake, and all other traffic goes thru cake.

Is this a known issue at all? I was quite surprised to see this
behaviour. Let me know if I can help debug this. It likely only
happens when multiple other hosts are doing bulk downloads / video
streaming.

Thanks,
Dan

Re: [Bloat] cake + ipv6

[Daniel Sterling]

I guess the reason I'm surprised is I'm confused about the following:

For UDP streams that use <1mbit down , should I expect cake in ingress
mode to keep those at low latency even in the face of a constantly
full queue / backlog, using "besteffort" but also using host
isolation?

By backlog I mean what I see with tc -s qdisc. I assume that's the
total of all individual flow backlogs, right?

I'm guessing no, but I'm wondering why not. Let's say we have some
hosts doing downloads as fast as they can with as many connections as
they can, and another separate host doing "light" UDP (xbox FPS
traffic).

So that's, say, 4 hosts constantly filling their queue's backlog -- it
will always have as many bytes as the rtt setting allows -- so by
default up to 100ms of bytes at my bandwidth setting, per flow, right?
Or is that per host?

And then another host (the xbox) that will have a constant flow that
doesn't really respond to shaping hints -- it's going to have a steady
state of packets that it wants to receive and send no matter what. It
might go from "high" to "low" update resolution (e.g. 256kbit to
128bkit), but that's about it. It will always want about 256kbit down
and 128kbit up with v4 UDP.

Normally that stream will have an rtt of < 50ms. Sometimes, e.g.
in-between rounds of the same game (thus the same UDP flow), the
server might let the rtt spike to 100+ ms since nothing much needs to
be sent between rounds.

But once the new round starts, we'll want low latency again.

Is it at all possible that cake, seeing the UDP stream is no longer
demanding low latency (in-between rounds), it figures it can let its
rtt stay at 100+ms per its rtt settings, even after the new round
starts and the xbox wants low latency again?

That is, since every host wants some traffic, and most if not all the
queues / backlogs will always be filled, is it possible that once a
flow allows its rtt to rise, cake won't let it back down again until
there's a lull?

As I said, I solved this by giving xbox traffic absolute first
priority with the "prio" qdisc. Obviously this means the xbox traffic
can starve everything else given a malicious flow, but that's not
likely to happen and if it does, I will notice.

Does my theory about cake having pathological behaviour for gaming
flows while the backlog is full hold any water?

Thanks,
Dan

On Wed, Sep 23, 2020 at 9:07 PM Daniel Sterling
<sterling.daniel@gmail.com> wrote:
>
> On Wed, Sep 23, 2020 at 2:13 PM Jonathan Morton <chromatix99@gmail.com> wrote:
> > It fits my mental model, yes, though obviously the ideal would be to recognise that the xbox is a singular machine.  Are you seeing a larger disparity than that?  If so, is it even larger than four connections would justify without host-fairness?
>
> Thanks Jonathan,
>
> Unfortunately I fear I've been running an improperly configured cake
> all this time-- leave it to me to take a piece of cake and mess it up
> :) I should say, even with it being possibly misconfigured, it has
> still worked amazingly well for me! I have no end of gratitude for
> y'alls work --
>
> So taking your advice, with my cake properly configured, the answer is
> no -- it looks like host-fairness is working correctly -- with a
> notable exception -- read on for details...
>
> I was using these probably wrong settings:
>
> WAN: besteffort bandwidth $UPBANDWIDTH internet nat egress ethernet
> LAN: besteffort bandwidth $BANDWIDTH internet nat ingress ethernet
>
> So, I was using triple-isolate instead of the correct src/dest
> isolation options per NIC, and I was using "nat" on the LAN side,
> which may have been further breaking host isolation as per the openwrt
> wiki:
>
> "don't add the nat option on LAN interfaces (this option should only
> be used in the WAN interface) or Per-Host Isolation stops working" -
> https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details
>
> I changed my configuration to:
>
> $WAN handle 1: root cake besteffort bandwidth $UPBANDWIDTH internet
> nat egress dual-srchost ethernet
> $LAN handle 1: root cake besteffort bandwidth $BANDWIDTH internet
> ingress dual-dsthost ethernet
>
> So:
>
> root@OpenWrt:~# tc -s qdisc | grep cake
> qdisc cake 1: dev eth1 root refcnt 2 bandwidth 20Mbit besteffort
> dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm
> overhead 38 mpu 84
> qdisc cake 1: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort
> dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100.0ms
> noatm overhead 38 mpu 84
>
> WAN:
> * nat option enabled
> * egress option (default) enabled
> * dual-srchost
>
> LAN:
> * nat option disabled (default)
> * ingress option enabled
> * dual-dsthost
>
> This worked well, but I ran into a bizarre issue:
>
> My xbox FPS gaming UDP stream would start at a low latency (even while
> other systems were doing bulk downloads), but after some time, the
> latency went from a steady 30ms (+/- <10ms for jitter) to over 100ms !
> 100ms is of course notably the default cake rtt parameter. Resetting
> cake (tc del and then tc add again) fixed this; the same stream's
> latency went back down to about 30ms.
>
> This gaming stream uses well under 512kbps down and less than half
> that up, normally.
>
> This is worrisome! My use-case may be an edge case, as I have AT&T
> gigabit fiber but I limit it to 40mbit with cake, since empirically
> that's the highest throughput I've found that I can use while also
> maintaining the low jitter / latency over wifi. So certainly the WAN
> NIC is getting a lot of traffic that cake on the LAN NIC is throwing
> away. That is, there is basically always a large number of bytes in
> LAN cake's backlog.
>
> For now I've solved this by using the "prio" tc qdisc such that xbox
> traffic bypasses cake, and all other traffic goes thru cake.
>
> Is this a known issue at all? I was quite surprised to see this
> behaviour. Let me know if I can help debug this. It likely only
> happens when multiple other hosts are doing bulk downloads / video
> streaming.
>
> Thanks,
> Dan

Re: [Bloat] cake + ipv6

[Toke Høiland-Jørgensen]

Daniel Sterling <sterling.daniel@gmail.com> writes:

> I guess the reason I'm surprised is I'm confused about the following:
>
> For UDP streams that use <1mbit down , should I expect cake in ingress
> mode to keep those at low latency even in the face of a constantly
> full queue / backlog, using "besteffort" but also using host
> isolation?

It depends. A 'sparse' flow should get consistent priority (and hence
low latency), but what exactly constitutes a sparse flow varies with the
link bandwidth and usage. I wrote a longish analysis of this for
FQ-CoDel[0] which should more or less carry over to CAKE; basically it
boils down to any flow using less than it's "fair share" of the link
should get priority, assuming its packets are spaced evenly. The main
difference between CAKE and FQ-CoDel is that the host fairness changes a
flows 'fair share', and that CAKE has a 'cooldown' period for sparse
flows after they disappear, which may change dynamics slightly. But the
basic mechanism is the same.

[0] https://ieeexplore.ieee.org/document/8469111/

> By backlog I mean what I see with tc -s qdisc. I assume that's the
> total of all individual flow backlogs, right?

Yup.

> I'm guessing no, but I'm wondering why not. Let's say we have some
> hosts doing downloads as fast as they can with as many connections as
> they can, and another separate host doing "light" UDP (xbox FPS
> traffic).
>
> So that's, say, 4 hosts constantly filling their queue's backlog -- it
> will always have as many bytes as the rtt setting allows -- so by
> default up to 100ms of bytes at my bandwidth setting, per flow, right?
> Or is that per host?

That's per flow. But you're misunderstanding the 100ms value. That's the
'interval', which is (simplifying a bit) the amount of time CAKE will
wait until it reacts to a flow building a queue. The actual amount of
queueing CAKE is *aiming for* is the 'target', which is interval/20 so
5ms by default. So in the 'steady state' a flow's backlog should
oscillate around this (ignoring for the moment that "steady state" is an
idealisation that rarely, if ever, exists in the real world).

BTW, you can see per-flow statistics by using 'tc -s class show
$DEVICE'.

> And then another host (the xbox) that will have a constant flow that
> doesn't really respond to shaping hints -- it's going to have a steady
> state of packets that it wants to receive and send no matter what. It
> might go from "high" to "low" update resolution (e.g. 256kbit to
> 128bkit), but that's about it. It will always want about 256kbit down
> and 128kbit up with v4 UDP.
>
> Normally that stream will have an rtt of < 50ms. Sometimes, e.g.
> in-between rounds of the same game (thus the same UDP flow), the
> server might let the rtt spike to 100+ ms since nothing much needs to
> be sent between rounds.

I'm not quite sure what you mean by "the server will let the RTT spike",
actually. Above you seemed to be saying that the gaming flow would just
carry one sending at whatever rate it wants?

> But once the new round starts, we'll want low latency again.
>
> Is it at all possible that cake, seeing the UDP stream is no longer
> demanding low latency (in-between rounds), it figures it can let its
> rtt stay at 100+ms per its rtt settings, even after the new round
> starts and the xbox wants low latency again?

You're attributing a bit more intent to CAKE here than it really
possesses ;)

CAKE doesn't have a notion of "what a flow wants". It just schedules
flows in a certain way, and if a flow happens to use less than its fair
share of bandwidth (the analysis linked above), it'll get temporary
priority whenever a packet from that flow arrives.

> That is, since every host wants some traffic, and most if not all the
> queues / backlogs will always be filled, is it possible that once a
> flow allows its rtt to rise, cake won't let it back down again until
> there's a lull?

Yes, this part is right, sorta. If a flow exceeds its fair share *rate*,
it'll no longer (from CAKEs) PoV be a 'sparse flow', and it'll get the
same treatment as all other flows (round-robin scheduling), and if it
keeps sending at this higher rate, it'll keep being scheduled in this
way. If the flow is non-elastic (i.e., doesn't slows down in response to
packet drops), it'll self-congest and you'll see that as increased
latency. So if what you meant by "the server will let the RTT spike", is
that the flow is bursty at times, and in some periods it'll increase its
rate above what CAKE's notion of fair share is, then yeah, that can lead
to the behaviour you're seeing.

> As I said, I solved this by giving xbox traffic absolute first
> priority with the "prio" qdisc. Obviously this means the xbox traffic
> can starve everything else given a malicious flow, but that's not
> likely to happen and if it does, I will notice.

If it helps to always prioritise the flow, then that seems to be an
indication that what I'm describing above is what you're seeing.

The various diffserv modes of CAKE are meant to be a way to deal with
this, BTW. It's similar to the prio qdisc in that it will use diffserv
markings to prioritise some traffic, but it will also impose a max total
bandwidth that each higher-priority tier can use before being
automatically demoted. If those limits fit your requirements, you may
be able to just use that instead of the prio qdisc (you don't have to
use real diffserv marks either, CAKE can be configured to get the
priority band from fwmark or skb->priority instead).

-Toke

Re: [Bloat] cake + ipv6

[Daniel Sterling]

On Mon, Sep 28, 2020 at 11:14 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> It depends. A 'sparse' flow should get consistent priority

> That's per flow. But you're misunderstanding the 100ms value. That's the
> 'interval', which is (simplifying a bit) the amount of time CAKE will
> wait until it reacts to a flow building a queue.

> If a flow exceeds its fair share *rate*,
> it'll no longer (from CAKEs) PoV be a 'sparse flow', and it'll get the
> same treatment as all other flows (round-robin scheduling), and if it
> keeps sending at this higher rate, it'll keep being scheduled in this
> way. If the flow is non-elastic (i.e., doesn't slows down in response to
> packet drops), it'll self-congest and you'll see that as increased
> latency.

Ah! Thank you *very* much for this explanation. I greatly appreciate
the effort everyone in this group puts into explaining (and tolerating
:) ) new users of cake.

In my case: I am happy to report this is *not* a bug or an issue with
cake, as I originally thought. I am able to reproduce the issue I was
seeing (high ping times as reported by the xbox game's network
monitoring) w/o cake being in the mix at all. So this issue is either
with how I've configured / built openwrt, or with my wireless network
mesh, or with the xbox itself. It is NOT an issue with cake.

Thank you all very much again. I will continue to use and test cake
and let you know if I encounter further issues with cake itself.

Thanks,
Dan

Re: [Bloat] cake + ipv6

[Toke Høiland-Jørgensen]

Daniel Sterling <sterling.daniel@gmail.com> writes:

> On Mon, Sep 28, 2020 at 11:14 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> It depends. A 'sparse' flow should get consistent priority
>
>> That's per flow. But you're misunderstanding the 100ms value. That's the
>> 'interval', which is (simplifying a bit) the amount of time CAKE will
>> wait until it reacts to a flow building a queue.
>
>> If a flow exceeds its fair share *rate*,
>> it'll no longer (from CAKEs) PoV be a 'sparse flow', and it'll get the
>> same treatment as all other flows (round-robin scheduling), and if it
>> keeps sending at this higher rate, it'll keep being scheduled in this
>> way. If the flow is non-elastic (i.e., doesn't slows down in response to
>> packet drops), it'll self-congest and you'll see that as increased
>> latency.
>
> Ah! Thank you *very* much for this explanation. I greatly appreciate
> the effort everyone in this group puts into explaining (and tolerating
> :) ) new users of cake.
>
> In my case: I am happy to report this is *not* a bug or an issue with
> cake, as I originally thought. I am able to reproduce the issue I was
> seeing (high ping times as reported by the xbox game's network
> monitoring) w/o cake being in the mix at all. So this issue is either
> with how I've configured / built openwrt, or with my wireless network
> mesh, or with the xbox itself. It is NOT an issue with cake.
>
> Thank you all very much again. I will continue to use and test cake
> and let you know if I encounter further issues with cake itself.

You're welcome! Happy experimenting :)

-Toke

[Bloat] openwrt e1000e (was: Re: cake + ipv6)

[Daniel Sterling]

> Daniel Sterling <sterling.daniel@gmail.com> writes:
> In my case: I am happy to report this is *not* a bug or an issue with
> cake, as I originally thought. I am able to reproduce the issue I was

Wanted to give an update on this. All my issues (odd latency, slow
throughput, etc) went away when I switched from the in-kernel e1000e
driver to Intel's NAPI driver.

That is, I compiled e1000e driver 3.8.7-NAPI from
https://sourceforge.net/projects/e1000/files/e1000e%20stable/ ,
instead of using the mainline e1000e driver from kernel 5.4.75 in
openwrt dev.

After switching to the Intel driver, my internet has been rock solid.

My NIC from lspci:
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

This is the "Intel EXPI9301CT Desktop Adapter Gigabit" from newegg. I
bought it figuring it would have good linux support. *grin*

And it does, but not with the mainline driver it seems. The in-kernel
driver doesn't (I assume) support NAPI -- so very possibly this is due
to the NAPI support in the out-of-tree driver (vs being an issue with
the driver itself)

-- Dan

[Bloat] finally got the proper cake experience (was: Re: openwrt e1000e (was: Re: cake + ipv6))

[Daniel Sterling]

On Tue, Nov 17, 2020 at 7:00 PM Daniel Sterling
<sterling.daniel@gmail.com> wrote:
> Wanted to give an update on this. All my issues (odd latency, slow
> throughput, etc) went away when I switched from the in-kernel e1000e
> driver to Intel's NAPI driver.

OK y'all, after switching to this driver, I configured my openwrt
snapshot setup as follows:

** e1000e loaded with InterruptThrottleRate=0:
rmmod r8169 ; modprobe r8169
rmmod e1000e ; insmod e1000e InterruptThrottleRate=0

** All ethtool -K offloading turned off:
rx tx sg tso gso gro all off

** flow control / pause frames turned off:
autoneg off rx off tx off

** ring buffers set to smallest setting:
rx 64 tx 64

** byte queue limits set to 3000
echo 3000 > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/net/eth1/queues/tx-0/byte_queue_limits/limit_max
echo 3000 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0/queues/tx-0/byte_queue_limits/limit_max

cake configured with besteffort and no bandwidth limit:
/usr/sbin/tc qdisc add dev $WAN handle 1: root cake besteffort
internet nat egress ack-filter dual-srchost ethernet
/usr/sbin/tc qdisc add dev $LAN handle 1: root cake besteffort
internet ingress dual-dsthost ethernet


AND ALL MY LATENCY IS GONE.

I've finally experienced the Zen of low latency and high throughput of
the Promised Cake.

This is amazing.

Previously, I was able to get low latency by setting the cake
"bandwidth" setting to about 40-50mbps out of the LAN interface.
Higher than that and I'd see latency.

But now, I can run as fast as the wifi / wired connections allow with
no latency anywhere on the network. This is truly wild.

Why did no one tell me I didn't need that bandwidth setting?? I
finally pushed and pushed and messed with things until I got it
working this way. If this is the intended final configuration, it
needs to be proselytized! Truly wire-line speed and no latency. This
is just, there are no words. The dream is real

Enormous thank you to everyone who worked on cake!

Thanks,
Dan