Re: [Cake] Upstream submission of dual-mode fairness patch

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Thanks for the nudge... I just tested this on the original one-armed router setup I encountered it on (3.16.7, routing from eth0 to eth0.3300). The patch makes a big improvement over the original (which was around 10/80):

IP 1, 1 up: 47.0 Mbit
IP 2, 8 up: 47.0 Mbit
IP 1, 8 down: 46.7 Mbit
IP 2, 1 down: 46.6 Mbit

Curiously, if my one-armed router has cake applied on egress and ingress of eth0.3300, instead of egress of eth0 and egress of eth0.3300 as above, some reduction in fairness occurs, but only for the downstream flows. This increases with the number of flows:

IP 1, 1 up: 47.2 Mbit
IP 2, 8 up: 47.2 Mbit
IP 1, 8 down: 44.5 Mbit
IP 2, 1 down: 48.7 Mbit

IP 1, 1 up: 47.0 Mbit
IP 2, 16 up: 47.0 Mbit
IP 1, 16 down: 42.6 Mbit
IP 2, 1 down: 50.3 Mbit

IP 1, 1 up: 47.2 Mbit
IP 2, 32 up: 47.0 Mbit
IP 1, 32 down: 41.4 Mbit
IP 2, 1 down: 51.3 Mbit

It still happens when ether-vlan is used on the leaf qdiscs with vlan traffic.

That said, unless there’s an obvious reason for this that’s fixable, I’m fine with how it is, considering the improvement. :)

> On Feb 28, 2019, at 12:17 PM, Toke Høiland-Jørgensen via Cake <cake@lists.bufferbloat.net> wrote:
> 
> 
> From: Toke Høiland-Jørgensen <toke@toke.dk>
> Subject: Upstream submission of dual-mode fairness patch
> Date: February 28, 2019 at 12:17:39 PM GMT+1
> To: cake@lists.bufferbloat.net
> 
> 
> Hey everyone
> 
> The dual-mode fairness patch has been in the github repo for a few weeks
> now, and no one has complained. If no one continues to complain, I'll
> submit it upstream tomorrow along with Kevin's fwmark patch.
> 
> So if anyone else wants to test, now would be a good time to do so :)
> 
> -Toke
> 
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


[-- Attachment #2: Type: text/html, Size: 4804 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Toke Høiland-Jørgensen]

Pete Heist <pete@heistp.net> writes:

> Thanks for the nudge... I just tested this on the original one-armed
> router setup I encountered it on (3.16.7, routing from eth0 to
> eth0.3300). The patch makes a big improvement over the original (which
> was around 10/80):
>
> IP 1, 1 up: 47.0 Mbit
> IP 2, 8 up: 47.0 Mbit
> IP 1, 8 down: 46.7 Mbit
> IP 2, 1 down: 46.6 Mbit
>
> Curiously, if my one-armed router has cake applied on egress and
> ingress of eth0.3300, instead of egress of eth0 and egress of
> eth0.3300 as above, some reduction in fairness occurs, but only for
> the downstream flows. This increases with the number of flows:
>
> IP 1, 1 up: 47.2 Mbit
> IP 2, 8 up: 47.2 Mbit
> IP 1, 8 down: 44.5 Mbit
> IP 2, 1 down: 48.7 Mbit
>
> IP 1, 1 up: 47.0 Mbit
> IP 2, 16 up: 47.0 Mbit
> IP 1, 16 down: 42.6 Mbit
> IP 2, 1 down: 50.3 Mbit
>
> IP 1, 1 up: 47.2 Mbit
> IP 2, 32 up: 47.0 Mbit
> IP 1, 32 down: 41.4 Mbit
> IP 2, 1 down: 51.3 Mbit
>
> It still happens when ether-vlan is used on the leaf qdiscs with vlan
> traffic.

Hmm, I have no good explanation for this...

> That said, unless there’s an obvious reason for this that’s fixable,
> I’m fine with how it is, considering the improvement. :)

Cool! And you haven't seen any regressions in other usage? :)

-Toke

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]


> On Mar 1, 2019, at 12:01 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 
> Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> writes:
> 
>> That said, unless there’s an obvious reason for this that’s fixable,
>> I’m fine with how it is, considering the improvement. :)
> 
> Cool! And you haven't seen any regressions in other usage? :)

To be honest, today’s the first time I tried it and I haven’t done any testing on it beyond fairness. (So, ship it!)

At least, I haven’t seen any other problems in this one-armed routing scenario or a regular host to host scenario.

Host fairness seems “mostly good" no matter what values I choose for the number of flows of the four clients, flow fairness still looks good, and I don’t see any problems starting and stopping different numbers of flows mid-test.

Pete


[-- Attachment #2: Type: text/html, Size: 4386 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Georgios Amanakis]

Thanks for testing Pete! The unfairness you see occurs if you shape
ingress using ifb on the same interface you shape egress, right?
Sounds puzzling to me, and I don't have an explanation right now.


On Fri, Mar 1, 2019 at 6:55 AM Pete Heist <pete@heistp.net> wrote:
>
>
> On Mar 1, 2019, at 12:01 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
> That said, unless there’s an obvious reason for this that’s fixable,
> I’m fine with how it is, considering the improvement. :)
>
>
> Cool! And you haven't seen any regressions in other usage? :)
>
>
> To be honest, today’s the first time I tried it and I haven’t done any testing on it beyond fairness. (So, ship it!)
>
> At least, I haven’t seen any other problems in this one-armed routing scenario or a regular host to host scenario.
>
> Host fairness seems “mostly good" no matter what values I choose for the number of flows of the four clients, flow fairness still looks good, and I don’t see any problems starting and stopping different numbers of flows mid-test.
>
> Pete
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

> On Mar 1, 2019, at 3:40 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
> 
> Thanks for testing Pete! The unfairness you see occurs if you shape
> ingress using ifb on the same interface you shape egress, right?
> Sounds puzzling to me, and I don't have an explanation right now.

Yes, same interface, so it’s:

apu2a eth0 <—> apu1a eth0 / apu1a eth0.3300 <—> apu2b eth0.3300

Of course, the VLAN interfaces have 4 bytes of extra overhead, which should be accounted for by ether-vlan.

It’s possible there’s some quirkiness here with apu1a, which has a Realtek Ethernet interface with the r8169 driver. With this driver, I find that setting rx-vlan-offload off (ethtool -K eth0 rxvlan off) doubles routed throughput when one-armed routing. That probably shouldn’t be. Nevertheless, the same thing happens with or without that offload enabled.

Or it’s just something in this old kernel again. If I ever have a rig like this with a newer kernel I’ll try it.

Re: [Cake] Upstream submission of dual-mode fairness patch

[George Amanakis]

I ran some tests with shaping ingress (ifb) and egress on the same
physical interface, albeit without vlans:

IP1, 1 up: 	45.07
IP2, 32 up:	45.06
IP1, 32 down:	45.64 mbit/s
IP2, 1 down:	44.50

This is on x86_64, kernel 4.20.12, iproute 4.20.

qdisc cake 8006: dev enp1s0 root refcnt 2 bandwidth 100Mbit diffserv3
dual-srchost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
noatm overhead 38 mpu
84                                                           

qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
noatm overhead 38 mpu 84 


I will setup a vlan and try again.

Re: [Cake] Upstream submission of dual-mode fairness patch

[George Amanakis]

On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:
> 
> I will setup a vlan and try again.
> 

I replicated Pete's VLAN setup, and I am getting fairness:

IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server

IP1, 1 up:      46.73 mbit/s
IP2, 32 up:     46.91
IP1, 32 down:   46.70
IP2, 1 down:    46.89

qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit
diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark
rtt 100.0ms noatm overhead 4

qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
noatm overhead 4

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

Great, thanks for trying it. That strongly suggests a problem with the kernel (or driver) I’m using. Feels better to know it works as it should in recent kernels though...

> On Mar 2, 2019, at 5:47 AM, George Amanakis <gamanakis@gmail.com> wrote:
> 
> On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:
>> 
>> I will setup a vlan and try again.
>> 
> 
> I replicated Pete's VLAN setup, and I am getting fairness:
> 
> IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server
> 
> IP1, 1 up:      46.73 mbit/s
> IP2, 32 up:     46.91
> IP1, 32 down:   46.70
> IP2, 1 down:    46.89
> 
> qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit
> diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark
> rtt 100.0ms noatm overhead 4
> 
> qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
> dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
> noatm overhead 4
> 
> 
> 

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

Oops, sorry I didn’t notice this before but it’s the ingress keyword that makes the difference:

qdisc cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 

IP1, 1 up: 46.8 Mbit
IP2, 16 up: 46.8 Mbit
IP1, 16 down: 41.9 Mbit
IP2, 1 down: 51.1 Mbit

qdisc cake 803a: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 

IP1, 1 up: 46.6 Mbit
IP2, 16 up: 46.5 Mbit
IP1, 16 down: 46.4 Mbit
IP2, 1 down: 46.5 Mbit

Also, my setup is different in that I use cake as a leaf to either hfsc or htb. But, it’s adding the ingress keyword that causes the imbalance for me.

> On Mar 2, 2019, at 11:20 AM, Pete Heist <pete@heistp.net> wrote:
> 
> Great, thanks for trying it. That strongly suggests a problem with the kernel (or driver) I’m using. Feels better to know it works as it should in recent kernels though...
> 
>> On Mar 2, 2019, at 5:47 AM, George Amanakis <gamanakis@gmail.com> wrote:
>> 
>> On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:
>>> 
>>> I will setup a vlan and try again.
>>> 
>> 
>> I replicated Pete's VLAN setup, and I am getting fairness:
>> 
>> IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server
>> 
>> IP1, 1 up:      46.73 mbit/s
>> IP2, 32 up:     46.91
>> IP1, 32 down:   46.70
>> IP2, 1 down:    46.89
>> 
>> qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit
>> diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark
>> rtt 100.0ms noatm overhead 4
>> 
>> qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
>> dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
>> noatm overhead 4
>> 
>> 
>> 
> 


[-- Attachment #2: Type: text/html, Size: 3336 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]

Mmmh,

Doesn't this look like ingress magic being applied selectively to the users based on number of flows? I thought that the idea behind the ingress keyword is to effectively shape harder the more bulk flows are coming in. 

Just asking...

Best regards
Sebastian

On March 3, 2019 8:19:00 AM GMT+01:00, Pete Heist <pete@heistp.net> wrote:
>Oops, sorry I didn’t notice this before but it’s the ingress keyword
>that makes the difference:
>
>qdisc cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort
>dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt
>100.0ms raw overhead 0 
>
>IP1, 1 up: 46.8 Mbit
>IP2, 16 up: 46.8 Mbit
>IP1, 16 down: 41.9 Mbit
>IP2, 1 down: 51.1 Mbit
>
>qdisc cake 803a: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort
>dual-dsthost nonat nowash no-ack-filter no-split-gso rtt 100.0ms raw
>overhead 0 
>
>IP1, 1 up: 46.6 Mbit
>IP2, 16 up: 46.5 Mbit
>IP1, 16 down: 46.4 Mbit
>IP2, 1 down: 46.5 Mbit
>
>Also, my setup is different in that I use cake as a leaf to either hfsc
>or htb. But, it’s adding the ingress keyword that causes the imbalance
>for me.
>
>> On Mar 2, 2019, at 11:20 AM, Pete Heist <pete@heistp.net> wrote:
>> 
>> Great, thanks for trying it. That strongly suggests a problem with
>the kernel (or driver) I’m using. Feels better to know it works as it
>should in recent kernels though...
>> 
>>> On Mar 2, 2019, at 5:47 AM, George Amanakis <gamanakis@gmail.com>
>wrote:
>>> 
>>> On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:
>>>> 
>>>> I will setup a vlan and try again.
>>>> 
>>> 
>>> I replicated Pete's VLAN setup, and I am getting fairness:
>>> 
>>> IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server
>>> 
>>> IP1, 1 up:      46.73 mbit/s
>>> IP2, 32 up:     46.91
>>> IP1, 32 down:   46.70
>>> IP2, 1 down:    46.89
>>> 
>>> qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit
>>> diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark
>>> rtt 100.0ms noatm overhead 4
>>> 
>>> qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
>>> dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt
>100.0ms
>>> noatm overhead 4
>>> 
>>> 
>>> 
>> 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 3965 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Jonathan Morton]

> On 3 Mar, 2019, at 11:53 am, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Doesn't this look like ingress magic being applied selectively to the users based on number of flows? I thought that the idea behind the ingress keyword is to effectively shape harder the more bulk flows are coming in. 

No, it simply counts dropped packets against the shaper, as well as those actually transmitted.  There shouldn't be that many packets being dropped to make this much of a difference.

 - Jonathan Morton

Re: [Cake] Upstream submission of dual-mode fairness patch

[Sebastian Moeller]

Hi Jonathan,


> On Mar 3, 2019, at 10:58, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 3 Mar, 2019, at 11:53 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Doesn't this look like ingress magic being applied selectively to the users based on number of flows? I thought that the idea behind the ingress keyword is to effectively shape harder the more bulk flows are coming in. 
> 
> No, it simply counts dropped packets against the shaper, as well as those actually transmitted.  

	Sure, but the question is how is the resulting "pressure" to drop/mark distributed over the existing (bulk) flows.

> There shouldn't be that many packets being dropped to make this much of a difference.

	My intuition (probably wrong) is that is not the few packets dropped, but the fact that the the dropping does seem to be restricted to the flows of the IP with more flows, no?

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 

Re: [Cake] Upstream submission of dual-mode fairness patch

[Jonathan Morton]

> On 3 Mar, 2019, at 1:26 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>>> Doesn't this look like ingress magic being applied selectively to the users based on number of flows? I thought that the idea behind the ingress keyword is to effectively shape harder the more bulk flows are coming in. 
>> 
>> No, it simply counts dropped packets against the shaper, as well as those actually transmitted.  
> 
> Sure, but the question is how is the resulting "pressure" to drop/mark distributed over the existing (bulk) flows.
> 
>> There shouldn't be that many packets being dropped to make this much of a difference.
> 
> My intuition (probably wrong) is that is not the few packets dropped, but the fact that the the dropping does seem to be restricted to the flows of the IP with more flows, no?

As long as there is a backoff response to a dropped packet, the extra back-pressure should be contained to the flows experiencing drops, and other flows should see no difference - so you should see a slight reduction in goodput on the 16 flows, but *no increase* on the single flow in parallel.

Even if that were not the case, the single flow should take longer on average to recover from a cwnd reduction (even in CUBIC) to the fair BDP.  That should result in a greater reduction in goodput on the single flow than the many flows - but we see the reverse here.

So I'm not entirely sure what's happening here, but at least the asymmetry isn't too bad; it's achieving significantly better host-fairness than a pure flow-fair system would.

 - Jonathan Morton

Re: [Cake] Upstream submission of dual-mode fairness patch

[Sebastian Moeller]

> On Mar 3, 2019, at 13:13, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 3 Mar, 2019, at 1:26 pm, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>>> Doesn't this look like ingress magic being applied selectively to the users based on number of flows? I thought that the idea behind the ingress keyword is to effectively shape harder the more bulk flows are coming in. 
>>> 
>>> No, it simply counts dropped packets against the shaper, as well as those actually transmitted.  
>> 
>> Sure, but the question is how is the resulting "pressure" to drop/mark distributed over the existing (bulk) flows.
>> 
>>> There shouldn't be that many packets being dropped to make this much of a difference.
>> 
>> My intuition (probably wrong) is that is not the few packets dropped, but the fact that the the dropping does seem to be restricted to the flows of the IP with more flows, no?
> 
> As long as there is a backoff response to a dropped packet, the extra back-pressure should be contained to the flows experiencing drops, and other flows should see no difference - so you should see a slight reduction in goodput on the 16 flows, but *no increase* on the single flow in parallel.

	Sure, it looks as if that single flow would see much less drops than each of the 16 flows of the other IP?

> 
> Even if that were not the case, the single flow should take longer on average to recover from a cwnd reduction (even in CUBIC) to the fair BDP.  That should result in a greater reduction in goodput on the single flow than the many flows - but we see the reverse here.

	If the single flow gets roughly the same per-flow drops/marks as the 16 other flows, that would be the expected behavior. Would it be fair to assume that this looks like dual-host isolation together with the ingress keyword results in unequal drop/mark probability based on the IP-address isolation?

> 
> So I'm not entirely sure what's happening here, but at least the asymmetry isn't too bad; it's achieving significantly better host-fairness than a pure flow-fair system would.
> 
> - Jonathan Morton
> 

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

> On Mar 3, 2019, at 1:13 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> So I'm not entirely sure what's happening here, but at least the asymmetry isn't too bad; it's achieving significantly better host-fairness than a pure flow-fair system would.

Yeah, it’s not a big problem, and IMO it’s in the preferable direction where the 1-flow IP “wins” over the N-flow IP. It’s just a matter of either flushing out or explaining the asymmetry, if possible.

Does it say anything to you that enabling ECN makes the asymmetry go away?

Re: [Cake] Upstream submission of dual-mode fairness patch

[Jonathan Morton]

> On 3 Mar, 2019, at 6:07 pm, Pete Heist <pete@heistp.net> wrote:
> 
> Does it say anything to you that enabling ECN makes the asymmetry go away?

Yes - it tells me that it has something directly to do with how dropped packets are accounted for, not with the congestion response by TCP.  ECN changes the number of dropped packets, and shouldn't change the congestion response.

 - Jonathan Morton

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

> On Mar 3, 2019, at 5:10 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 3 Mar, 2019, at 6:07 pm, Pete Heist <pete@heistp.net> wrote:
>> 
>> Does it say anything to you that enabling ECN makes the asymmetry go away?
> 
> Yes - it tells me that it has something directly to do with how dropped packets are accounted for, not with the congestion response by TCP.  ECN changes the number of dropped packets, and shouldn't change the congestion response.

That suggests it’s possible to fix. It almost seems like in ingress mode, dropped packets are being counted as bytes transferred, when they weren’t.

In the four pcaps I sent, enabling ECN resulted in 0 dropped packets for the N flows down, so it didn’t get to the point where a drop was necessary. Also, the rate of CWR bits set was a bit lower than the rate of drops, and total aggregate throughput was a bit higher. so enabling ECN is a win in this test.

Re: [Cake] Upstream submission of dual-mode fairness patch

[Jonathan Morton]

> On 3 Mar, 2019, at 6:35 pm, Pete Heist <pete@heistp.net> wrote:
> 
> It almost seems like in ingress mode, dropped packets are being counted as bytes transferred, when they weren’t.

Well yes, that's the whole point - at least as far as the shaper is concerned.  Ingress mode is supposed to deal with the case where inbound packets have already traversed the bottleneck link *before* Cake gets to choose what to do with them.

But that shouldn't affect fairness, only total throughput.  Fairness is not handled by the shaper, but by the (very) extended DRR++ algorithm.

 - Jonathan Morton

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]



> On Mar 3, 2019, at 5:40 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 3 Mar, 2019, at 6:35 pm, Pete Heist <pete@heistp.net> wrote:
>> 
>> It almost seems like in ingress mode, dropped packets are being counted as bytes transferred, when they weren’t.
> 
> Well yes, that's the whole point - at least as far as the shaper is concerned.  Ingress mode is supposed to deal with the case where inbound packets have already traversed the bottleneck link *before* Cake gets to choose what to do with them.
> 
> But that shouldn't affect fairness, only total throughput.  Fairness is not handled by the shaper, but by the (very) extended DRR++ algorithm.


It should probably come as no surprise then that if I take pcaps from the client side and sum cumulative bytes transferred with 1514 * the number of drops (according to tcp.analysis.lost_segment), I get pretty close to the same value for each IP. I think that roughly confirms what we think is happening. https://www.heistp.net/downloads/ingress_fairness/client/ <https://www.heistp.net/downloads/ingress_fairness/client/>

IP 1, 16 down:
  54968142 bytes transferred
  5313 drops (according to tcp.analysis.lost_segment)
  54968142 + 1514 * 5313 = 63012024

IP 2, 1 down:
  63769820 bytes transferred
  61 drops (according to tcp.analysis.lost_segment)
  63769820 + 1514 * 61 = 63862174

Knowing this though, maybe the current behavior is, after all, what we want. :)

I mean, if a host has caused more drops with more simultaneous flows, then it has filled the pipe with that data, even if it’s been dropped to signal congestion. Should we penalize other hosts in order to equalize goodput for all hosts, regardless of how many flows they use? If a client wants to ensure that they aren’t “charged" for drops for fairness purposes, they can use ECN.

I think it’s subjective and I’m willing to be corrected or be convinced otherwise, but thanks for bringing me to this point though…


[-- Attachment #2: Type: text/html, Size: 3009 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[gamanakis]

For the record, I can replicate it now. I also think that this behavior is expected.
There is a work-around but it seems like an awful hack:

----------8<----------
diff --git a/sch_cake.c b/sch_cake.c
index 733b897..08e08f4 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -2216,7 +2216,6 @@ retry:
                if (q->rate_flags & CAKE_FLAG_INGRESS) {
                        len = cake_advance_shaper(q, b, skb,
                                                  now, true);
-                       flow->deficit -= len;
                        b->tin_deficit -= len;
                }
                flow->dropped++;
----------8<----------


Then the results are:

IP1, 1 up:	47.18 mbit/s
IP2, 32 up:	46.99
IP1, 32 down:	40.98
IP2, 1 down:	41.34

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

> On Mar 3, 2019, at 8:03 PM, gamanakis@gmail.com wrote:
> 
> For the record, I can replicate it now. I also think that this behavior is expected.
> There is a work-around but it seems like an awful hack:
> 
> ----------8<----------
> diff --git a/sch_cake.c b/sch_cake.c
> index 733b897..08e08f4 100644
> --- a/sch_cake.c
> +++ b/sch_cake.c
> @@ -2216,7 +2216,6 @@ retry:
>                if (q->rate_flags & CAKE_FLAG_INGRESS) {
>                        len = cake_advance_shaper(q, b, skb,
>                                                  now, true);
> -                       flow->deficit -= len;
>                        b->tin_deficit -= len;
>                }
>                flow->dropped++;
> ----------8<----------
> 
> 
> Then the results are:
> 
> IP1, 1 up:	47.18 mbit/s
> IP2, 32 up:	46.99
> IP1, 32 down:	40.98
> IP2, 1 down:	41.34

Yes, thanks for confirming it, this fix equalizes goodput for me as well, although I think we’re in agreement that nothing should actually change in the end(?)

If so, I think we should document this. If it wasn’t obvious to us right away what was happening, then it probably won’t be obvious to others.

If I don’t hear any objection to this in the next couple days, I’ll submit a pull request to tc-adv for tc-cake.8, if that’s the right place to do it.

Re: [Cake] Upstream submission of dual-mode fairness patch

[Georgios Amanakis]

I was just thinking that getting rid of the decrement of flow->deficit
and tin->deficit when we call cake_advance_shaper() to account for
"ingress" traffic might not be that awful after all. That way, the
shaper logic is unaffected but the fairness logic wouldn't account for
that "ingress traffic" and would yield fairer results. I would love to
hear your thoughts.


On Sun, Mar 3, 2019 at 2:49 PM Pete Heist <pete@heistp.net> wrote:
>
>
> > On Mar 3, 2019, at 8:03 PM, gamanakis@gmail.com wrote:
> >
> > For the record, I can replicate it now. I also think that this behavior is expected.
> > There is a work-around but it seems like an awful hack:
> >
> > ----------8<----------
> > diff --git a/sch_cake.c b/sch_cake.c
> > index 733b897..08e08f4 100644
> > --- a/sch_cake.c
> > +++ b/sch_cake.c
> > @@ -2216,7 +2216,6 @@ retry:
> >                if (q->rate_flags & CAKE_FLAG_INGRESS) {
> >                        len = cake_advance_shaper(q, b, skb,
> >                                                  now, true);
> > -                       flow->deficit -= len;
> >                        b->tin_deficit -= len;
> >                }
> >                flow->dropped++;
> > ----------8<----------
> >
> >
> > Then the results are:
> >
> > IP1, 1 up:    47.18 mbit/s
> > IP2, 32 up:   46.99
> > IP1, 32 down: 40.98
> > IP2, 1 down:  41.34
>
> Yes, thanks for confirming it, this fix equalizes goodput for me as well, although I think we’re in agreement that nothing should actually change in the end(?)
>
> If so, I think we should document this. If it wasn’t obvious to us right away what was happening, then it probably won’t be obvious to others.
>
> If I don’t hear any objection to this in the next couple days, I’ll submit a pull request to tc-adv for tc-cake.8, if that’s the right place to do it.
>

Re: [Cake] Upstream submission of dual-mode fairness patch

[Jonathan Morton]

> On 4 Mar, 2019, at 4:55 am, Georgios Amanakis <gamanakis@gmail.com> wrote:
> 
> …the fairness logic wouldn't account for that "ingress traffic" and would yield fairer results.

Well there we have a quandary, since presently it enforces fairness of *load* on the bottleneck link, but the change would alter that to fairness of *delivered* traffic.  The current setup is arguably more robust against adversarial traffic, don't you think?

 - Jonathan Morton

Re: [Cake] Upstream submission of dual-mode fairness patch

[Ryan Mounce]

On Mon, 4 Mar 2019 at 13:47, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 4 Mar, 2019, at 4:55 am, Georgios Amanakis <gamanakis@gmail.com> wrote:
> >
> > …the fairness logic wouldn't account for that "ingress traffic" and would yield fairer results.
>
> Well there we have a quandary, since presently it enforces fairness of *load* on the bottleneck link, but the change would alter that to fairness of *delivered* traffic.  The current setup is arguably more robust against adversarial traffic, don't you think?

And it provides an incentive to use ECN so that congestion signals can
be sent "for free" without dropping packets that have traversed the
bottleneck. I'm firmly in favour of the current setup.

This is analogous to the WiFi airtime fairness problem that Toke knows
all too much about :)
We want fairness of time on the bottleneck medium. Aiming for fairness
of goodput to each host allows less efficient hosts to use more than
their fair share of the medium - reducing the efficiency of the whole
network and weakening incentives to be more efficient.

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

> On Mar 4, 2019, at 5:22 AM, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On Mon, 4 Mar 2019 at 13:47, Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> On 4 Mar, 2019, at 4:55 am, Georgios Amanakis <gamanakis@gmail.com> wrote:
>>> 
>>> …the fairness logic wouldn't account for that "ingress traffic" and would yield fairer results.
>> 
>> Well there we have a quandary, since presently it enforces fairness of *load* on the bottleneck link, but the change would alter that to fairness of *delivered* traffic.  The current setup is arguably more robust against adversarial traffic, don't you think?

I think that’s the best argument for the current behavior.

> And it provides an incentive to use ECN so that congestion signals can
> be sent "for free" without dropping packets that have traversed the
> bottleneck. I'm firmly in favour of the current setup

Agreed. I was going to provide test results of aggressive UDP vs TCP with and without the change, but I’m seeing some odd behavior with UDP that I’ll investigate more and start in a separate thread if needed. :)

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

I added some text for the ingress keyword and related fairness behavior to the cake man page, in case there’s feedback or changes:

https://github.com/dtaht/tc-adv/pull/25/commits/9a7905a8a768fc57926b1875e8be445865e08627 <https://github.com/dtaht/tc-adv/pull/25/commits/9a7905a8a768fc57926b1875e8be445865e08627>


[-- Attachment #2: Type: text/html, Size: 549 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 46 bytes --]

Agreed then! Thank you all for your insights!

[-- Attachment #2: Type: text/html, Size: 68 bytes --]

Re: [Cake] Upstream submission of dual-mode fairness patch

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

As another clue, the un-fairness goes away when ECN is enabled on the client (both results with ingress keyword in use):

sysctl -w net.ipv4.tcp_ecn=2

IP1, 1 up: 46.7 Mbit
IP2, 16 up: 46.7 Mbit
IP1, 16 down: 41.0 Mbit
IP2, 1 down: 49.3 Mbit

sysctl -w net.ipv4.tcp_ecn=1

IP1, 1 up: 47.0 Mbit
IP2, 16 up: 47.0 Mbit
IP1, 16 down: 46.1 Mbit
IP2, 1 down: 46.1 Mbit

Also, this is probably just academic, but here are pcaps of "IP1, 16 down", with and without “ingress” and ECN enabled, along with some throughput and RTT graphs of one of the 16 flows for each: https://www.heistp.net/downloads/ingress_fairness/ <https://www.heistp.net/downloads/ingress_fairness/>

I find the different TCP RTT “slots” interesting (see the rtt graphs, all times in ms):

no ingress: 4, 8.5, 13, 17
no ingress+ecn: 8, 12, 16 (a few outliers at 20)
ingress: 4, 8, 12, 16, 20
ingress+ecn: 8, 12, 16 (outliers at 4 and 20 later in test)

> On Mar 3, 2019, at 8:19 AM, Pete Heist <pete@heistp.net> wrote:
> 
> Oops, sorry I didn’t notice this before but it’s the ingress keyword that makes the difference:
> 
> qdisc cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 
> 
> IP1, 1 up: 46.8 Mbit
> IP2, 16 up: 46.8 Mbit
> IP1, 16 down: 41.9 Mbit
> IP2, 1 down: 51.1 Mbit
> 
> qdisc cake 803a: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash no-ack-filter no-split-gso rtt 100.0ms raw overhead 0 
> 
> IP1, 1 up: 46.6 Mbit
> IP2, 16 up: 46.5 Mbit
> IP1, 16 down: 46.4 Mbit
> IP2, 1 down: 46.5 Mbit
> 
> Also, my setup is different in that I use cake as a leaf to either hfsc or htb. But, it’s adding the ingress keyword that causes the imbalance for me.
> 
>> On Mar 2, 2019, at 11:20 AM, Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> wrote:
>> 
>> Great, thanks for trying it. That strongly suggests a problem with the kernel (or driver) I’m using. Feels better to know it works as it should in recent kernels though...
>> 
>>> On Mar 2, 2019, at 5:47 AM, George Amanakis <gamanakis@gmail.com <mailto:gamanakis@gmail.com>> wrote:
>>> 
>>> On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:
>>>> 
>>>> I will setup a vlan and try again.
>>>> 
>>> 
>>> I replicated Pete's VLAN setup, and I am getting fairness:
>>> 
>>> IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server
>>> 
>>> IP1, 1 up:      46.73 mbit/s
>>> IP2, 32 up:     46.91
>>> IP1, 32 down:   46.70
>>> IP2, 1 down:    46.89
>>> 
>>> qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit
>>> diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark
>>> rtt 100.0ms noatm overhead 4
>>> 
>>> qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3
>>> dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt 100.0ms
>>> noatm overhead 4
>>> 
>>> 
>>> 
>> 
> 


[-- Attachment #2: Type: text/html, Size: 5354 bytes --]