From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D98EB3CB35 for ; Thu, 3 Jan 2019 08:02:30 -0500 (EST) Received: by mail-wr1-x433.google.com with SMTP id l9so33454625wrt.13 for ; Thu, 03 Jan 2019 05:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=QTRdICX+5jxfGq09gr/HegX6spLO8gTXLMpTKvVExWQ=; b=DF97L++X2JkC6P5yuomxHiQgVo4HFXg9BFnhQ73QKwEibewOk2jUsQFNNSESDEr9ut erDtmrUxXH4wqKioQf7eE0jikV34/2oIf/mb1MaKruS1Z29QV/3onGVeH1KfpRW2t0IO m0Hcq3RprszWHXqMzYHaYGVQynSYRFUAAfi6Ah+PE9t0JxqFNmU5adsFymEWdmuFp4g1 2DiTeozZV0A6umRXCyjvMCwq8EtIxbc8Z5+om97n4baMiBLHAV3tBZG1FAnx5pGoYxAx aWcGWiGFIuJAPDlT23sdyte/YZloMI2t/7doPXxWRDP0J4oFfl8+Qvi0jpNprDUm0fyn o19w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=QTRdICX+5jxfGq09gr/HegX6spLO8gTXLMpTKvVExWQ=; b=QXrYtovVqw8dUUND73OLEk7lF9BLcEOoGYZgvFOO0qh/QLOy5lbkOztq9r8hFngFFJ 1f1eVKx8B8lLONjt7wib6LxrxC+4OrhzMaBLfcsmbeh6BwPmjODhDzhUJzAyFGX9sSyL szzb21qBQkQqu4NIsM18x5tgFfoKR8Nf+uxi8+Aq5bXApBfTIRAt1q9LRDVNun+q926M OWyuaTV/MYk34FZ52rv2G9WzNLzUdKjbs9oZjouEuUttQH1a+SGmqJ83ozno05e/shzF udEgVc0m1sXhJ+HPash8p6QGKlrojR6T2qBVlo3OLYhSyP8yDUheGWvlt9XvKyu/oQfW 6TWw== X-Gm-Message-State: AJcUukc6df/9jhw5IFOJwTbt+udCR0rk4PVyNewKJ59z5QK2BIxczyq6 R7/Bjya8TLFtdISdWzi/ZcPYAg== X-Google-Smtp-Source: ALg8bN5D/IcQbEVn73rQ6RwojnQMTYYX9oxOXdPgO3d5TyqyatASc4zsKQryJF0Qlv7ta44hPjIHeA== X-Received: by 2002:a5d:5607:: with SMTP id l7mr41892945wrv.25.1546520549944; Thu, 03 Jan 2019 05:02:29 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id f187sm34564128wma.4.2019.01.03.05.02.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 05:02:29 -0800 (PST) From: Pete Heist Message-Id: Content-Type: multipart/mixed; boundary="Apple-Mail=_3FA2BA07-F287-459E-B72C-93A07E55BD0E" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Thu, 3 Jan 2019 14:02:27 +0100 In-Reply-To: <87imz6xatw.fsf@toke.dk> Cc: Jonathan Morton , Cake List To: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= References: <8F9DE6A8-8614-46A8-9E9B-7B7E4CC7414F@heistp.net> <43a8ddec5beb962c53fe828363ecc839832de2c0.camel@gmail.com> <3650A136-97A6-43F5-ADD3-B94A19775379@gmail.com> <99C93851-3539-4CB6-BED1-193B56658486@heistp.net> <87imz6xatw.fsf@toke.dk> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] dual-src/dsthost unfairness, only with bi-directional traffic X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jan 2019 13:02:31 -0000 --Apple-Mail=_3FA2BA07-F287-459E-B72C-93A07E55BD0E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jan 3, 2019, at 12:03 PM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >=20 >> Jon, is there anything I can check by instrumenting the code = somewhere >> specific? >=20 > Is there any way you could test with a bulk UDP flow? I'm wondering > whether this is a second-order effect where TCP ACKs are limited in a > way that cause the imbalance? Are you using ACK compression? Not using ack-filter, if that=E2=80=99s what=E2=80=99s meant by ACK = compression. I thought about the TCP ACK traffic, but would be very = surprised if that amount of ACK traffic could cause that large of an = imbalance, although it=E2=80=99s worth trying to find out. I tried iperf3 in UDP mode, but cake is treating these flows = aggressively. I get the impression that cake penalizes flows heavily = that do not respond to congestion control signals. If I pit one 8 TCP = flows against a single UDP flow at 40mbit, the UDP flow goes into a = death spiral with increasing drops over time (iperf3 output attached). I=E2=80=99m not sure there=E2=80=99d be any way I can test fairness with = iperf3 in UDP mode. We=E2=80=99d need something that has some congestion = control feedback, right? Otherwise, I don=E2=80=99t think there are any = rates I can choose to both reach saturation and not be severely = punished. And if it has congestion control feedback, it has the ACK-like = traffic we=E2=80=99re trying to avoid for the test. :) As another test, I took out the one-armed router and just tried from a = client to a server, no VLANs. Same result. So, still stumped. Thank you = for the help... --Apple-Mail=_3FA2BA07-F287-459E-B72C-93A07E55BD0E Content-Disposition: attachment; filename=iperf3_spiral.txt Content-Type: text/plain; x-unix-mode=0644; name="iperf3_spiral.txt" Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- Server listening on 5202 ----------------------------------------------------------- Accepted connection from 10.0.0.239, port 48289 [ 5] local 10.0.0.231 port 5202 connected to 10.72.0.239 port 38334 [ ID] Interval Transfer Bandwidth Jitter = Lost/Total Datagrams [ 5] 0.00-1.00 sec 4.20 MBytes 35.3 Mbits/sec 0.467 ms 21/559 = (3.8%) =20 [ 5] 1.00-2.00 sec 4.27 MBytes 35.8 Mbits/sec 0.555 ms 43/589 = (7.3%) =20 [ 5] 2.00-3.00 sec 4.48 MBytes 37.6 Mbits/sec 0.482 ms 69/642 = (11%) =20 [ 5] 3.00-4.00 sec 3.90 MBytes 32.7 Mbits/sec 0.461 ms 87/586 = (15%) =20 [ 5] 4.00-5.00 sec 3.84 MBytes 32.2 Mbits/sec 0.490 ms 111/603 = (18%) =20 [ 5] 5.00-6.00 sec 3.94 MBytes 33.0 Mbits/sec 0.341 ms 130/634 = (21%) =20 [ 5] 6.00-7.00 sec 3.63 MBytes 30.5 Mbits/sec 0.539 ms 144/609 = (24%) =20 [ 5] 7.00-8.00 sec 3.59 MBytes 30.1 Mbits/sec 0.451 ms 159/618 = (26%) =20 [ 5] 8.00-9.00 sec 3.21 MBytes 26.9 Mbits/sec 0.987 ms 181/592 = (31%) =20 [ 5] 9.00-10.00 sec 3.23 MBytes 27.1 Mbits/sec 0.224 ms 225/639 = (35%) =20 [ 5] 10.00-11.00 sec 3.11 MBytes 26.1 Mbits/sec 0.204 ms 214/612 = (35%) =20 [ 5] 11.00-12.00 sec 2.80 MBytes 23.5 Mbits/sec 0.371 ms 229/587 = (39%) =20 [ 5] 12.00-13.00 sec 2.66 MBytes 22.3 Mbits/sec 0.543 ms 254/594 = (43%) =20 [ 5] 13.00-14.00 sec 2.73 MBytes 22.9 Mbits/sec 0.386 ms 292/642 = (45%) =20 [ 5] 14.00-15.00 sec 2.49 MBytes 20.9 Mbits/sec 0.399 ms 298/617 = (48%) =20 [ 5] 15.00-16.00 sec 2.40 MBytes 20.1 Mbits/sec 0.216 ms 288/595 = (48%) =20 [ 5] 16.00-17.00 sec 2.20 MBytes 18.5 Mbits/sec 0.486 ms 327/609 = (54%) =20 [ 5] 17.00-18.00 sec 2.19 MBytes 18.3 Mbits/sec 0.538 ms 344/624 = (55%) =20 [ 5] 18.00-19.00 sec 2.00 MBytes 16.8 Mbits/sec 0.519 ms 321/577 = (56%) =20 [ 5] 19.00-20.00 sec 1.95 MBytes 16.4 Mbits/sec 0.930 ms 369/619 = (60%) =20 [ 5] 20.00-21.00 sec 1.93 MBytes 16.2 Mbits/sec 0.526 ms 377/624 = (60%) =20 [ 5] 21.00-22.00 sec 1.66 MBytes 13.9 Mbits/sec 0.543 ms 374/586 = (64%) =20 [ 5] 22.00-23.00 sec 1.70 MBytes 14.2 Mbits/sec 0.833 ms 412/629 = (66%) =20 [ 5] 23.00-24.00 sec 1.66 MBytes 13.9 Mbits/sec 0.340 ms 402/614 = (65%) =20 [ 5] 24.00-25.00 sec 1.52 MBytes 12.7 Mbits/sec 0.693 ms 431/625 = (69%) =20 [ 5] 25.00-26.00 sec 1.40 MBytes 11.7 Mbits/sec 0.491 ms 404/583 = (69%) =20 [ 5] 26.00-27.00 sec 1.32 MBytes 11.1 Mbits/sec 1.028 ms 456/625 = (73%) =20 [ 5] 27.00-28.00 sec 1.25 MBytes 10.5 Mbits/sec 0.870 ms 427/587 = (73%) =20 [ 5] 28.00-29.00 sec 1.20 MBytes 10.1 Mbits/sec 0.660 ms 479/633 = (76%) =20 [ 5] 29.00-30.00 sec 1.19 MBytes 9.96 Mbits/sec 0.773 ms 466/618 = (75%) =20 [ 5] 30.00-31.00 sec 1.05 MBytes 8.85 Mbits/sec 1.103 ms 455/590 = (77%) =20 [ 5] 31.00-32.00 sec 1.03 MBytes 8.65 Mbits/sec 0.559 ms 488/620 = (79%) =20 [ 5] 32.00-33.00 sec 888 KBytes 7.27 Mbits/sec 0.415 ms 494/605 = (82%) =20 [ 5] 33.00-34.00 sec 896 KBytes 7.34 Mbits/sec 1.023 ms 489/601 = (81%) =20 [ 5] 34.00-35.00 sec 880 KBytes 7.21 Mbits/sec 0.986 ms 519/629 = (83%) =20 [ 5] 35.00-36.00 sec 776 KBytes 6.36 Mbits/sec 0.414 ms 493/590 = (84%) =20 [ 5] 36.00-37.00 sec 800 KBytes 6.55 Mbits/sec 0.845 ms 506/606 = (83%) =20 [ 5] 37.00-38.00 sec 832 KBytes 6.82 Mbits/sec 1.124 ms 536/640 = (84%) =20 [ 5] 38.00-39.00 sec 768 KBytes 6.29 Mbits/sec 0.577 ms 515/611 = (84%) =20 [ 5] 39.00-40.00 sec 728 KBytes 5.96 Mbits/sec 1.269 ms 496/587 = (84%) =20 [ 5] 40.00-41.00 sec 752 KBytes 6.16 Mbits/sec 0.834 ms 544/638 = (85%) =20 [ 5] 41.00-42.00 sec 528 KBytes 4.32 Mbits/sec 1.533 ms 346/412 = (84%) =20 [ 5] 42.00-43.00 sec 552 KBytes 4.52 Mbits/sec 2.008 ms 722/791 = (91%) =20 [ 5] 43.00-44.00 sec 416 KBytes 3.41 Mbits/sec 2.202 ms 528/580 = (91%) =20 [ 5] 44.00-45.00 sec 408 KBytes 3.34 Mbits/sec 2.075 ms 566/617 = (92%) =20 [ 5] 45.00-46.00 sec 512 KBytes 4.19 Mbits/sec 1.629 ms 517/581 = (89%) =20 [ 5] 46.00-47.00 sec 400 KBytes 3.28 Mbits/sec 1.750 ms 584/634 = (92%) =20 [ 5] 47.00-48.00 sec 408 KBytes 3.34 Mbits/sec 1.587 ms 541/592 = (91%) =20 [ 5] 48.00-49.00 sec 504 KBytes 4.13 Mbits/sec 1.344 ms 587/650 = (90%) =20 [ 5] 49.00-50.00 sec 600 KBytes 4.91 Mbits/sec 1.338 ms 522/597 = (87%) =20 [ 5] 50.00-51.00 sec 384 KBytes 3.15 Mbits/sec 2.033 ms 592/640 = (92%) =20 [ 5] 51.00-52.00 sec 504 KBytes 4.13 Mbits/sec 1.566 ms 529/592 = (89%) =20 [ 5] 52.00-53.00 sec 400 KBytes 3.28 Mbits/sec 1.883 ms 508/558 = (91%) =20 [ 5] 53.00-54.00 sec 424 KBytes 3.47 Mbits/sec 1.833 ms 639/692 = (92%) =20 [ 5] 54.00-55.00 sec 280 KBytes 2.29 Mbits/sec 2.004 ms 539/574 = (94%) =20 [ 5] 55.00-56.00 sec 288 KBytes 2.36 Mbits/sec 2.161 ms 522/558 = (94%) =20 [ 5] 56.00-57.00 sec 288 KBytes 2.36 Mbits/sec 2.693 ms 635/671 = (95%) =20 [ 5] 57.00-58.00 sec 264 KBytes 2.16 Mbits/sec 2.225 ms 554/587 = (94%) =20 [ 5] 58.00-59.00 sec 256 KBytes 2.10 Mbits/sec 2.065 ms 571/603 = (95%) =20 [ 5] 59.00-60.00 sec 696 KBytes 5.70 Mbits/sec 0.154 ms 588/675 = (87%) =20 [ 5] 60.00-60.04 sec 24.0 KBytes 4.98 Mbits/sec 0.136 ms 0/3 (0%) = =20 --Apple-Mail=_3FA2BA07-F287-459E-B72C-93A07E55BD0E--