[Cake] Per-host fairness

[Cake] Per-host fairness

[G. Amanakis]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

I am trying the cobalt branch along with the 950-add-cake-to-tc.patch from lede-git on Archlinux. However, I cannot get per-host fairness as expected, neither with IPv4 behind NAT, nor with IPv6. Having host A downloading from 2 sites and host B from 1, A gets always 1/3 of available bandwidth. I am using "bestefforts triple-isolate nat" on egress and ingress ifb of the WAN interface. Kernel 4.4.24 and 4.7.6.

The same setup with the same bandwidth limiting (3300kbit/900kbit) works as expected with fq_codel, tc-flow using mark for hashing, and iptables with HMARK using source egress ip for hashing and CONNMARK on WAN.

[-- Attachment #2: Type: text/html, Size: 645 bytes --]

Re: [Cake] Per-host fairness

[G. Amanakis]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

I meant B gets always 1/3 of the bandwidth. It also gets worse if A is using bittorrent. The setup with cake doesn't involve tc-flow or marking packets with iptables.

On October 16, 2016 11:57:03 AM EDT, "G. Amanakis" <g_amanakis@yahoo.com> wrote:
>I am trying the cobalt branch along with the 950-add-cake-to-tc.patch
>from lede-git on Archlinux. However, I cannot get per-host fairness as
>expected, neither with IPv4 behind NAT, nor with IPv6. Having host A
>downloading from 2 sites and host B from 1, A gets always 1/3 of
>available bandwidth. I am using "bestefforts triple-isolate nat" on
>egress and ingress ifb of the WAN interface. Kernel 4.4.24 and 4.7.6.
>
>The same setup with the same bandwidth limiting (3300kbit/900kbit)
>works as expected with fq_codel, tc-flow using mark for hashing, and
>iptables with HMARK using source egress ip for hashing and CONNMARK on
>WAN.

[-- Attachment #2: Type: text/html, Size: 1137 bytes --]

Re: [Cake] Per-host fairness

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

Hi,

Could you also try dual-srchost on egress and dual-srchost on ingress, please. These promise per-IP fairness at the first level and per-flow fairness "inside" each IP. So if you think through it you can setup the ingress and egress shapers to aim for per-internal-IP address which I believe is what you are after. Triple-isolate is much harder to argue with, partly because it is lacking a clear description what it does short of the source code, so it is somewhat opaque what to expect in a given set of bidirectional flows. In short you might have found a flaw in triple-isolate or it is working as intended, which only JM will be able to answer. Cursory experiments in the past indicated that a properly configured dual-srchost/dsthost pair on ingress and egress work closer to a layman's expectations... Oh, with the new and shiny deNAT features the dual-isolation options should now also work on a wan interface for IPv4, thanks to Kevin's persistence.

Best Regards
        Sebastian



On October 16, 2016 7:30:04 PM GMT+02:00, "G. Amanakis" <g_amanakis@yahoo.com> wrote:
>I meant B gets always 1/3 of the bandwidth. It also gets worse if A is
>using bittorrent. The setup with cake doesn't involve tc-flow or
>marking packets with iptables.
>
>On October 16, 2016 11:57:03 AM EDT, "G. Amanakis"
><g_amanakis@yahoo.com> wrote:
>>I am trying the cobalt branch along with the 950-add-cake-to-tc.patch
>>from lede-git on Archlinux. However, I cannot get per-host fairness as
>>expected, neither with IPv4 behind NAT, nor with IPv6. Having host A
>>downloading from 2 sites and host B from 1, A gets always 1/3 of
>>available bandwidth. I am using "bestefforts triple-isolate nat" on
>>egress and ingress ifb of the WAN interface. Kernel 4.4.24 and 4.7.6.
>>
>>The same setup with the same bandwidth limiting (3300kbit/900kbit)
>>works as expected with fq_codel, tc-flow using mark for hashing, and
>>iptables with HMARK using source egress ip for hashing and CONNMARK on
>>WAN.
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Cake mailing list
>Cake@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cake

[-- Attachment #2: Type: text/html, Size: 2758 bytes --]

Re: [Cake] Per-host fairness

[Georgios Amanakis]

I tried with "besteffort dual-dsthost nat" on ifb ingress and
"besteffort dual-srchost nat" on WAN egress, and when doing simple
things (i.e. downloading from a couple of websites) the bandwidth is
divided fair per-host, i.e 1:1 between A and B. However, when one of
the hosts is using bittorrent, he also gets most of the bandwidth, i.e.
if total ingress bandwidth is 3300kbit, A is using bittorrent and gets
2900kbit, B is downloading from a single website and gets 400kbit.

Re: [Cake] Per-host fairness

[moeller0]

Hi Georgios,


> On Oct 18, 2016, at 03:33 , Georgios Amanakis <g_amanakis@yahoo.com> wrote:
> 
> I tried with "besteffort dual-dsthost nat" on ifb ingress and
> “besteffort dual-srchost nat" on WAN egress,

	Thanks.

> and when doing simple
> things (i.e. downloading from a couple of websites) the bandwidth is
> divided fair per-host, i.e 1:1 between A and B.

	Okay, does this stay fair if one of the hosts open a bunch of websites at the same time? Does the download test also give per-host fairness if the number of downloads is massively imbalanced between the hosts?


> However, when one of
> the hosts is using bittorrent, he also gets most of the bandwidth, i.e.
> if total ingress bandwidth is 3300kbit, A is using bittorrent and gets
> 2900kbit, B is downloading from a single website and gets 400kbit.

	Bit-torrent is a hard problem it seems, especially the cobalts branch that you test is supposed to better deal with it though so this is abit of a puzzle. How long are your tests? Are you looking only at a few seconds or something like >30 seconds (maybe the shaper needs time to reach equilibrium)? What happens in the upload during theses tests? I also wonder how many new torrent connections are established during the test or are the torrent flows long lived?

	I probably would try to instantiate cake not on the wan but on a lan interface of the router and re-do the test without the “nat” option just to reduce the number of tested parts. If you should do this, please remember that ingress and egress are per interface, so downloads from the internet use the LAN-egress while internet-uploads use LAN-ingress, so make sure to switch the bandwidth definitions for your two cake instances accordingly. 
	Finally could you post the results of “tc -s qdisc” before, during and after your bit-torrent tests for the test you just reported and the LAN test in case you ago that route.


Best Regards
	Sebastian

> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Per-host fairness

[Georgios Amanakis]

Hi Sebastian,

> I probably would try to instantiate cake not on the wan but on
> a lan interface of the router and re-do the test without the “nat”
> option just to reduce the number of tested parts. If you should do
> this, please remember that ingress and egress are per interface, so
> downloads from the internet use the LAN-egress while internet-
uploads 
> use LAN-ingress, so make sure to switch the bandwidth definitions for
> your two cake instances accordingly. 
> 	Finally could you post the results of “tc -s qdisc” before,
> during and after your bit-torrent tests for the test you just
> reported and the LAN test in case you ago that route. 

I tried what you suggested on the LAN interface (ingress: 900kbit/s,
egress: 3300kbit/s). Details follow.

> > and when doing simple
> > things (i.e. downloading from a couple of websites) the bandwidth
> > is
> > divided fair per-host, i.e 1:1 between A and B.
> 
> 	Okay, does this stay fair if one of the hosts open a bunch of
> websites at the same time? Does the download test also give per-host
> fairness if the number of downloads is massively imbalanced between
> the hosts?

There seems to be a discrepancy here. If host A is downloading a file
(e.g. kernel.org) and host B opens a bunch of websites simultaneously
(e.g. "firefox arstechnica.com; firefox in.gr; firefox phoronix.com;
firefox cnn.com; firefox theguardian.co.uk") the following happens: for
the first 20 seconds host B gets 90% of the bandwidth and then it
slowly equilibrates to 50%-50%.


> > However, when one of
> > the hosts is using bittorrent, he also gets most of the bandwidth,
> > i.e.
> > if total ingress bandwidth is 3300kbit, A is using bittorrent and
> > gets
> > 2900kbit, B is downloading from a single website and gets 400kbit.
> 
> 	Bit-torrent is a hard problem it seems, especially the cobalts
> branch that you test is supposed to better deal with it though so
> this is abit of a puzzle. How long are your tests? Are you looking
> only at a few seconds or something like >30 seconds (maybe the shaper
> needs time to reach equilibrium)? What happens in the upload during
> theses tests? I also wonder how many new torrent connections are
> established during the test or are the torrent flows long lived?
> 
I am performing this test for 5 minutes, using the torrent for ubuntu-
16.10-desktop. Regarding new established torrent connections, these are
around 50 in this 5-minute interval. You *can* browse the web from host
B but there is notable latency. Pings to 8.8.8.8 are typically 300-
400msec.


Best regards,
George