From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 831FA3BA8E for ; Tue, 1 Jan 2019 18:04:41 -0500 (EST) Received: by mail-wr1-x42a.google.com with SMTP id t27so28967961wra.6 for ; Tue, 01 Jan 2019 15:04:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:mime-version:subject:message-id:date:to; bh=KluYZyPLpqwva4nc+nvpix25i7H7MZA9MdGNt+dnpUU=; b=KJdNvpjiqYGUmaaie99xsa9pBvy3P0/FsVIyEDDsOWCwRA7jjZK6ifDvXNoXHtFh/A 7LsIhPrvbqlQoy46DUaTVU3dW2U99ly4fAkjd/jW0Dnzp7Oqv6WF0dvzyHctx//Q6Urh BwATh7shvYJ/+1QHAPLjhrujRtWnjGh1ZFnVvWFE3AgA/Omck9jaAMLI4Yemghp3CuVw TSsydcsTI2wnUO2uHpDTZABdBzr2L6deW7Ise2K+vIs+bQvXULsmuVYc84PUCug+2Ik/ vxa8bFe/5zF6Lnb7My2+vYj34l2AbRbep3/wiymHTwYHGeTCgBIV2S2JOveKs36rxtey 6Xmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:message-id:date:to; bh=KluYZyPLpqwva4nc+nvpix25i7H7MZA9MdGNt+dnpUU=; b=Eow1YFZ/rGzVhvd8Mc2Rr45Na/BIZP53naRh3926AWWYh+cvM2mGdUamZBjqMJgBBT bjWhlEaDmDYGBhZuhXnI5ZY6wD530hllmUd/V4KfpzVMGqw6/+8pU0RFFH9snMKvKQ8f hH0GMDnGW210wbjeJ6V2WnEDTa8eYgtqEFOUJbHrqjhSk/DJCwgtjxVJppPGNAUgboZy +H0pRop+Obxj+PaeDCZdWehF5rJRsBOg0bTGMt6I6DGC1+yoVLi4qTSCSVIaIFAJ7mEq WbaBufMAc+YjjUgdwIR94++xEOCxPKgiAcZ/ylMyylCM+KrR+BhRTBbNY1dzsfJXWJt7 3bLg== X-Gm-Message-State: AJcUukeyuXCaIuIYcXm+FD9kmKt8sLcGluGjZj02E3t8Ghw4sHL6dB9U YSEoYN94GobE2xsj7SN9bOYbdsrjzUY= X-Google-Smtp-Source: ALg8bN5mF0E3jXmZcacSlElaeoBMuP7CEtHc3LvT7U4+mY99K1YgZPffLRgpEiLz6WP0Wrje4u7pBg== X-Received: by 2002:adf:e247:: with SMTP id n7mr34680387wri.205.1546383880312; Tue, 01 Jan 2019 15:04:40 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id c65sm44378134wma.24.2019.01.01.15.04.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Jan 2019 15:04:39 -0800 (PST) From: Pete Heist Content-Type: multipart/mixed; boundary="Apple-Mail=_2E5B4BE8-E2F3-4FB3-81A5-0F02DFD11B94" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Message-Id: <8F9DE6A8-8614-46A8-9E9B-7B7E4CC7414F@heistp.net> Date: Wed, 2 Jan 2019 00:04:38 +0100 To: Cake List X-Mailer: Apple Mail (2.3445.9.1) Subject: [Cake] dual-src/dsthost unfairness, only with bi-directional traffic X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jan 2019 23:04:41 -0000 --Apple-Mail=_2E5B4BE8-E2F3-4FB3-81A5-0F02DFD11B94 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 In my one-armed router setup I=E2=80=99m seeing host fairness work = perfectly with srchost or dsthost, but with dual-srchost or = dual-dsthost, host fairness deviates from the ideal, _only_ when there's = bi-directional traffic. The deviation is then dependent on the number of = flows. Is this expected? I had thought that dual-src/dsthost worked the same as src/dsthost = (fairness between hosts) with the exception that there is also fairness = of flows within each host. Here are some results (all rates aggregate throughput in Mbit): IP1=3D8 up / 1 down IP2=3D1 up / 8 down (post-test tc stats attached): srchost/dsthost, upload only: IP1=3D48.1, IP2=3D47.9 (OK) srchost/dsthost, download only: IP1=3D47.8, IP2=3D47.8 (OK) srchost/dsthost, bi-directional: IP1=3D47.5 up / 43.9 down, = IP2=3D44.7 up / 46.7 down (OK) dual-srchost/dual-dsthost, upload only: IP1=3D48.1, IP2=3D48.0 = (OK) dual-srchost/dual-dsthost, download only: IP1=3D47.9, IP2=3D47.9 = (OK) dual-srchost/dual-dsthost, bi-directional: IP1=3D83.0 up / 10.7 = down, IP2=3D10.6 up / 83.0 down (*** asymmetric ***) Dual-srchost/dual-dsthost, bi-directional tests with different flow = counts: IP1=3D4 up / 1 down IP2=3D1 up / 4 down: IP1=3D74.8 up / 18.8 down, IP2=3D18.8 up / 74.8 down IP1=3D2 up / 1 down IP2=3D1 up / 2 down: IP1=3D62.4 up / 31.3 down, IP2=3D31.3 up / 62.4 down IP1=3D4 up / 1 down IP2=3D1 up / 8 down: IP1=3D81.8 up / 11.5 down, IP2=3D17.4 up / 76.3 down IP1=3D2 up / 1 down IP2=3D1 up / 8 down: IP1=3D79.9 up / 13.5 down, IP2=3D25.7 up / 68.1 down The setup: apu2a (kernel 4.9) <=E2=80=94 default VLAN =E2=80=94> apu1a = (kernel 3.16.7) <=E2=80=94 VLAN 3300 =E2=80=94> apu2b (kernel 4.9) - apu1a is the router, and has cake only on egress of both eth0 and = eth0.3300, rate limited to 100mbit for both - it has no trouble shaping at 100mbit up and down simultaneously, so = that should not be a problem - the same problem occurs at 25mbit or 50mbit) - since apu2a is the client [dual-]dsthost is used on eth0 and = [dual-]srchost is used on eth0.3300 - the fairness test setup seems correct, based on the results of most of = the tests, at least. - note in the qdisc stats attached there is a prio qdisc on eth0 for = filtering out VLAN traffic so it isn=E2=80=99t shaped twice - I also get the exact same results with an htb or hfsc hierarchy on = eth0 instead of adding a qdisc to eth0.3300 - printk=E2=80=99s in sch_cake.c shows values of flow_mode, srchost_hash = and dsthost_hash as expected - I also see it going into allocate_src and allocate_dst as expected, = and later ending up in found_src and found_dst I=E2=80=99m stumped. I know I=E2=80=99ve tested fairness of = dual-src/dsthost before, but that was from the egress of client and = server, and it was on a recent kernel. Time to sleep on it... --Apple-Mail=_2E5B4BE8-E2F3-4FB3-81A5-0F02DFD11B94 Content-Disposition: attachment; filename=qdisc_stats.txt Content-Type: text/plain; x-unix-mode=0644; name="qdisc_stats.txt" Content-Transfer-Encoding: quoted-printable qdisc prio 1: dev eth0 root refcnt 2 bands 2 priomap 1 1 1 1 1 1 1 1 1 = 1 1 1 1 1 1 1 Sent 1502062182 bytes 1525226 pkt (dropped 204373, overlimits 0 = requeues 0)=20 backlog 0b 4294550547p requeues 0 qdisc cake 10: dev eth0 parent 1:1 bandwidth 100Mbit besteffort = dual-dsthost nonat nowash no-ack-filter split-gso rtt 100.0ms raw = overhead 0=20 Sent 751856301 bytes 762993 pkt (dropped 8512, overlimits 2414269 = requeues 0)=20 backlog 0b 0p requeues 0 memory used: 361600b of 5000000b capacity estimate: 100Mbit min/max network layer size: 42 / 1514 min/max overhead-adjusted size: 42 / 1514 average network hdr offset: 14 Tin 0 thresh 100Mbit target 5.0ms interval 100.0ms pk_delay 3.9ms av_delay 2.4ms sp_delay 60us backlog 0b pkts 771505 bytes 764743469 way_inds 6 way_miss 26 way_cols 0 drops 8512 marks 0 ack_drop 0 sp_flows 9 bk_flows 4 un_flows 0 max_len 18168 quantum 1514 qdisc cake 8060: dev eth0.3300 root refcnt 2 bandwidth 100Mbit = besteffort dual-srchost nonat nowash no-ack-filter split-gso rtt 100.0ms = raw overhead 0=20 Sent 750205881 bytes 762233 pkt (dropped 8542, overlimits 904221 = requeues 0)=20 backlog 0b 0p requeues 0 memory used: 279744b of 5000000b capacity estimate: 100Mbit min/max network layer size: 42 / 1514 min/max overhead-adjusted size: 42 / 1514 average network hdr offset: 14 Tin 0 thresh 100Mbit target 5.0ms interval 100.0ms pk_delay 157us av_delay 83us sp_delay 1us backlog 0b pkts 770775 bytes 763138469 way_inds 31166 way_miss 23 way_cols 0 drops 8542 marks 0 ack_drop 0 sp_flows 17 bk_flows 1 un_flows 0 max_len 30280 quantum 1514 --Apple-Mail=_2E5B4BE8-E2F3-4FB3-81A5-0F02DFD11B94--