From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8E7DF3B2A4 for ; Sun, 3 Mar 2019 07:06:36 -0500 (EST) Received: by mail-wr1-x443.google.com with SMTP id w17so2433148wrn.12 for ; Sun, 03 Mar 2019 04:06:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=q2bVNWZ9fLCeFLptYgQtSof35DiznBWAw4IVvEqti8g=; b=HzSVPJAzB+JB3NpHoaVqFj1SkkDx7vPzZH4hiRZn7IvCo1e6mBsUezEVbplocfdFbT q99SdolnFbu8SJ5/ILezYTNXSAr7RiaEhq19i1pXYuokWAq4fnEbm/15BffE/QPu7OUQ hW2crq2rp9oPbb0tdcEQ6V9jS7N6OguFMdrXm5a6aB8WMctsebddbMzj7JAH/KqFYyW8 B3fDjHm2sLTYorK7cjXRBmgZYqld80Lj08cKwwWntB+DVYn9MtPWIS1/UsE9Y5U86vOX gZhIyWdWwjI/AC6S77sdDc1lVREB8Ki+u7EotgFMfi7vyQmEzONX+lolOhorazBjcySW bPsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=q2bVNWZ9fLCeFLptYgQtSof35DiznBWAw4IVvEqti8g=; b=e0mgKCzb91OSnRQ24LxZyv5ImhQ9HzMq7cr//2cX9CWrhHnkxvI7KvzH9edhA2UY+9 ooXioYqJ8SvoFWKqEogrO9oyR3jEAxSCBnpIg5twPM62D1cWMzpajmc97pO6cCe0TcqG jXw08L5la6KMUpRBWelWVGjA+tU9nkZ4n0lBsVEuXKU2Q5WGdxvZ3RVG9ZXJwFNVub6W UDbjTcadMwGerNICydbHOZ2HPn0JAsb4GVtyBypu+nsgbMuK8Wvq4H4VbVgNDB37j0IS 8+qrbLd5BlAAFzOrBOS02eZ3HwUrhjBgxtrKJ/7+FsJCCgkdn6sm6YSzUVIq/ejxgKN1 oW5Q== X-Gm-Message-State: APjAAAV5/5JzRJceG36DigAgTjpNZi5lH6sk8Ng4OG68up6KhGWsrCYd KUoJbQ8U/ybZUa/8iLsC2Xwufw== X-Google-Smtp-Source: APXvYqy/63f5OnGZJzoNJIX3qy5Hixc28zSUkkIihCPJbqv+oe7TfwGkdUq1EtQ+UVs3ATrLRpOhSw== X-Received: by 2002:a5d:66c1:: with SMTP id k1mr9071605wrw.120.1551614795654; Sun, 03 Mar 2019 04:06:35 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id z17sm3586885wrs.75.2019.03.03.04.06.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 03 Mar 2019 04:06:35 -0800 (PST) From: Pete Heist Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_6B1AD784-B1A4-417E-875F-21FCC1B1D4AB" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Sun, 3 Mar 2019 13:06:34 +0100 In-Reply-To: Cc: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Cake List To: George Amanakis References: <72193310-7502-47B8-9554-7F8F9FA23204@heistp.net> <874l8mn9iy.fsf@toke.dk> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] Upstream submission of dual-mode fairness patch X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Mar 2019 12:06:36 -0000 --Apple-Mail=_6B1AD784-B1A4-417E-875F-21FCC1B1D4AB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 As another clue, the un-fairness goes away when ECN is enabled on the = client (both results with ingress keyword in use): sysctl -w net.ipv4.tcp_ecn=3D2 IP1, 1 up: 46.7 Mbit IP2, 16 up: 46.7 Mbit IP1, 16 down: 41.0 Mbit IP2, 1 down: 49.3 Mbit sysctl -w net.ipv4.tcp_ecn=3D1 IP1, 1 up: 47.0 Mbit IP2, 16 up: 47.0 Mbit IP1, 16 down: 46.1 Mbit IP2, 1 down: 46.1 Mbit Also, this is probably just academic, but here are pcaps of "IP1, 16 = down", with and without =E2=80=9Cingress=E2=80=9D and ECN enabled, along = with some throughput and RTT graphs of one of the 16 flows for each: = https://www.heistp.net/downloads/ingress_fairness/ = I find the different TCP RTT =E2=80=9Cslots=E2=80=9D interesting (see = the rtt graphs, all times in ms): no ingress: 4, 8.5, 13, 17 no ingress+ecn: 8, 12, 16 (a few outliers at 20) ingress: 4, 8, 12, 16, 20 ingress+ecn: 8, 12, 16 (outliers at 4 and 20 later in test) > On Mar 3, 2019, at 8:19 AM, Pete Heist wrote: >=20 > Oops, sorry I didn=E2=80=99t notice this before but it=E2=80=99s the = ingress keyword that makes the difference: >=20 > qdisc cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited = besteffort dual-dsthost nonat nowash ingress no-ack-filter no-split-gso = rtt 100.0ms raw overhead 0=20 >=20 > IP1, 1 up: 46.8 Mbit > IP2, 16 up: 46.8 Mbit > IP1, 16 down: 41.9 Mbit > IP2, 1 down: 51.1 Mbit >=20 > qdisc cake 803a: dev ifbLink2 parent 1:1 bandwidth unlimited = besteffort dual-dsthost nonat nowash no-ack-filter no-split-gso rtt = 100.0ms raw overhead 0=20 >=20 > IP1, 1 up: 46.6 Mbit > IP2, 16 up: 46.5 Mbit > IP1, 16 down: 46.4 Mbit > IP2, 1 down: 46.5 Mbit >=20 > Also, my setup is different in that I use cake as a leaf to either = hfsc or htb. But, it=E2=80=99s adding the ingress keyword that causes = the imbalance for me. >=20 >> On Mar 2, 2019, at 11:20 AM, Pete Heist > wrote: >>=20 >> Great, thanks for trying it. That strongly suggests a problem with = the kernel (or driver) I=E2=80=99m using. Feels better to know it works = as it should in recent kernels though... >>=20 >>> On Mar 2, 2019, at 5:47 AM, George Amanakis > wrote: >>>=20 >>> On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote: >>>>=20 >>>> I will setup a vlan and try again. >>>>=20 >>>=20 >>> I replicated Pete's VLAN setup, and I am getting fairness: >>>=20 >>> IP1,2 <---> router enp1s0 / router enp1s0.100 <---> server >>>=20 >>> IP1, 1 up: 46.73 mbit/s >>> IP2, 32 up: 46.91 >>> IP1, 32 down: 46.70 >>> IP2, 1 down: 46.89 >>>=20 >>> qdisc cake 8006: dev enp1s0.100 root refcnt 2 bandwidth 100Mbit >>> diffserv3 dual-srchost nonat nowash no-ack-filter split-gso nofwmark >>> rtt 100.0ms noatm overhead 4 >>>=20 >>> qdisc cake 8002: dev ifb0 root refcnt 2 bandwidth 100Mbit diffserv3 >>> dual-dsthost nonat nowash no-ack-filter split-gso nofwmark rtt = 100.0ms >>> noatm overhead 4 >>>=20 >>>=20 >>>=20 >>=20 >=20 --Apple-Mail=_6B1AD784-B1A4-417E-875F-21FCC1B1D4AB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
As another clue, the un-fairness goes away when ECN is = enabled on the client (both results with ingress keyword in = use):

sysctl -w net.ipv4.tcp_ecn=3D2

IP1, 1 up: 46.7 Mbit
IP2, = 16 up: 46.7 Mbit
IP1, 16 down: 41.0 Mbit
IP2, = 1 down: 49.3 Mbit

sysctl -w net.ipv4.tcp_ecn=3D1

IP1, 1 up: 47.0 Mbit
IP2, 16 up: 47.0 Mbit
IP1, 16 down: 46.1 = Mbit
IP2, 1 down: 46.1 Mbit

Also, this is probably just academic, = but here are pcaps of "IP1, 16 down", with and without =E2=80=9Cingress=E2= =80=9D and ECN enabled, along with some throughput and RTT graphs of one = of the 16 flows for each: https://www.heistp.net/downloads/ingress_fairness/

I find = the different TCP RTT =E2=80=9Cslots=E2=80=9D interesting (see the rtt = graphs, all times in ms):

no ingress: 4, 8.5, 13, 17
no = ingress+ecn: 8, 12, 16 (a few outliers at 20)
ingress: 4, 8, 12, 16, 20
ingress+ecn: = 8, 12, 16 (outliers at 4 and 20 later in test)

On Mar 3, 2019, at 8:19 AM, Pete Heist <pete@heistp.net> = wrote:

Oops, sorry I didn=E2=80= =99t notice this before but it=E2=80=99s the ingress keyword that makes = the difference:

qdisc = cake 802c: dev ifbLink2 parent 1:1 bandwidth unlimited besteffort = dual-dsthost nonat nowash ingress no-ack-filter no-split-gso rtt 100.0ms = raw overhead 0 

IP1, 1 up: 46.8 Mbit
IP2, 16 up: 46.8 = Mbit
IP1, 16 down: 41.9 = Mbit
IP2, 1 down: 51.1 = Mbit

qdisc cake 803a: dev ifbLink2 = parent 1:1 bandwidth unlimited besteffort dual-dsthost nonat nowash = no-ack-filter no-split-gso rtt 100.0ms raw overhead = 0 

IP1, 1 up: 46.6 Mbit
IP2, 16 up: 46.5 = Mbit
IP1, 16 down: 46.4 = Mbit
IP2, 1 down: 46.5 Mbit

Also, my setup is different in that I = use cake as a leaf to either hfsc or htb. But, it=E2=80=99s adding the = ingress keyword that causes the imbalance for = me.

On = Mar 2, 2019, at 11:20 AM, Pete Heist <pete@heistp.net> = wrote:

Great, thanks for trying it. That strongly suggests a problem = with the kernel (or driver) I=E2=80=99m using. Feels better to know it = works as it should in recent kernels though...

On Mar 2, 2019, at 5:47 = AM, George Amanakis <gamanakis@gmail.com> wrote:

On Fri, 2019-03-01 at 22:02 -0500, George Amanakis wrote:

I will = setup a vlan and try again.


I replicated Pete's VLAN setup, and I am getting fairness:

IP1,2 <---> router enp1s0 / router = enp1s0.100 <---> server

IP1, 1 up: =      46.73 mbit/s
IP2, 32 up: =     46.91
IP1, 32 down: =   46.70
IP2, 1 down:    46.89

qdisc cake 8006: dev enp1s0.100 root refcnt 2 = bandwidth 100Mbit
diffserv3 dual-srchost nonat nowash = no-ack-filter split-gso nofwmark
rtt 100.0ms noatm = overhead 4

qdisc cake 8002: dev ifb0 root = refcnt 2 bandwidth 100Mbit diffserv3
dual-dsthost nonat = nowash no-ack-filter split-gso nofwmark rtt 100.0ms
noatm = overhead 4






= --Apple-Mail=_6B1AD784-B1A4-417E-875F-21FCC1B1D4AB--