From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BF7803B29E for ; Sun, 22 Jul 2018 05:57:27 -0400 (EDT) Received: by mail-wr1-x436.google.com with SMTP id v14-v6so15055983wro.5 for ; Sun, 22 Jul 2018 02:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=UgnzqFfBbiyKl34ek/eeGPWEKmuBdgUs7FOikBVFU9o=; b=Cr67VVBN7E2efSnPyuZM47E7oaqTEeYQ8j8oH+wKZ6P6JJs2ZhpTulaMvMMzB2uCO/ 00NkehBSDwv1MWmyqtiWYfTe2rBIQICrKoJBTuiyxF3HgY1dSTlpkk6Lz3VGMLq3z5TA afgjK0YUQIp1xduJPrHUph08m4c6OdxSPLs1w6ZWOE/AZGd8aFy9H2Zp66OU7ssTg0iT ytgl33YsOHev5ksY8IVQZof3J3sisriYzwLk5TyC+8fFsUhwxAKrSCiOZ35kG+VNMMIJ TQk4LsAYQlfeccgVy/frkhQE6urP8z8ha33d89LiJEfWwYXPcRgBj49ja3AwFx0nPB9v WDlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=UgnzqFfBbiyKl34ek/eeGPWEKmuBdgUs7FOikBVFU9o=; b=pCon5+PNn62Ju31cTQFRtNquQrHxEHfO2stHRrEjwDpfmL+7FIcs92cJ+7Z/JFEO7p 6AnkOeeg6y9VMrUyWlvjgssrMkNCsA5dGs97ek5Kjdr5h1+gvmxDzuODpQqoV4oAakPe kTCv6uT45z2S9pN1dXAXkYWaHIVaTA1JS8rJolLVXtslhExHsvbTbIpebQy1zR8zGRUq pdWt2/zHtbt4KI1bGXnCSiQ4uLeGhpZ/SziCD30JEHBbmJtIZ1n7cJhquDz3ezuw3Fsy YesZ1oalUCPao4um/FrkPY+b1gYoH112+RDu98j64eahsHTHsANzLmzJsBMb6Ra8IX6D y01g== X-Gm-Message-State: AOUpUlH+vaGbp8bFYS3DlXrLZQQJqWiGVC1NfMMry1+DtVN/kxIJfobe Av5gzgbEwFXsWLC7RTxzCH7qng== X-Google-Smtp-Source: AAOMgpeAK2nPPBRgX7RYigeMZg0rixHLsfgMAt5439B0T8iaSW0BjU4ECaJQo356oYMBbi0k20YeWw== X-Received: by 2002:adf:a0f3:: with SMTP id n48-v6mr5913214wrn.23.1532253446759; Sun, 22 Jul 2018 02:57:26 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id p184-v6sm4983431wmp.31.2018.07.22.02.57.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Jul 2018 02:57:26 -0700 (PDT) From: Pete Heist Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_76A355F7-18C6-4F85-BDA4-4E49335D4D15" Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Sun, 22 Jul 2018 11:57:24 +0200 In-Reply-To: Cc: Cake List , cerowrt-devel@lists.bufferbloat.net To: Dave Taht References: X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2018 09:57:28 -0000 --Apple-Mail=_76A355F7-18C6-4F85-BDA4-4E49335D4D15 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 21, 2018, at 6:09 PM, Dave Taht wrote: >=20 > PS I also have two other issues going on. This is the first time I've > been using irtt with a 20ms interval, and I regularly see single 50+ms > spikes (in both ping and irtt) data and also see irtt stop > transmitting. irtt should keep sending for the duration of the test. I noticed that it = looks like irtt was actually used in only one of these initial tests: = ping-2018-07-21T082842.445812.flent-newark-reno.flent. In the rest, = netperf UDP_RR was used, which can stop sending upon packet loss. If irtt was configured but didn=E2=80=99t run, that may be because flent = does a connectivity check to the server with =E2=80=9Cirtt client -n=E2=80= =9D, where it sends three requests within 900ms (200ms timeout, then = 300ms then 400ms) and if it doesn=E2=80=99t receive a reply, it falls = back to netperf UDP_RR. Do you think that=E2=80=99s what happened here? > On this front, it could merely be that my (not tested in > months!) test cablemodem setup is malfunctioning also! Or we're > hitting power save. Or (finally) seeing request-grant delays. Or > scheduling delay somewhere in the net-next kernel I'm using... Or.... > (regardless, this seems independent of my main issue, and I've not had > such high res data before) Regarding the spikes both you and Arie you=E2=80=99re seeing, I also saw = in one of your later emails "0 delay via fq would be better than even the 15-40ms I'm getting now with linux flows=E2=80=9D. Those numbers = struck me as similar to an issue I=E2=80=99m still dealing with: = https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spik= es-about-once-per-second-even-just-to/m-p/2359800/highlight/true#M119202 = = To summarize, with airOS on the NanoStation M5, there are isochronous = pauses around once per second in the processing of all packets, not just = for the WiFi device but Ethernet also. Packets are not lost, but queued = for either 20ms, if one Ethernet port is connected, or 40ms, if both are = connected. This behavior is exactly described by the = ar7240sw_phy_poll_reset function in ag71xx_ar7240.c, so it looks to me = like the ar7240 internal switch is being reset once per second for no = apparent reason. So far I=E2=80=99ve gotten crickets in response. The cable modem you=E2=80=99re using doesn=E2=80=99t happen to have = built-in WiFi and an ar7240, does it? If it does and has the same or a = similar driver problem, you could just do a low-interval ping straight = to its Ethernet adapter and check for isochronous spikes, something = like: sudo ping -l 100 -c 5000 -i 0.001 cablemodem Now, back to vacation :)= --Apple-Mail=_76A355F7-18C6-4F85-BDA4-4E49335D4D15 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Jul 21, 2018, at 6:09 PM, Dave Taht <dave.taht@gmail.com>= wrote:

PS I also = have two other issues going on. This is the first time I've
been using irtt with a 20ms interval, and I regularly see = single 50+ms
spikes (in both ping and irtt) data and also = see irtt stop
transmitting.
irtt should keep sending for the = duration of the test. I noticed that it looks like irtt was actually = used in only one of these initial tests: = ping-2018-07-21T082842.445812.flent-newark-reno.flent. In the rest, = netperf UDP_RR was used, which can stop sending upon packet = loss.

If irtt was configured = but didn=E2=80=99t run, that may be because flent does a connectivity = check to the server with =E2=80=9Cirtt client -n=E2=80=9D, where it = sends three requests within 900ms (200ms timeout, then 300ms then 400ms) = and if it doesn=E2=80=99t receive a reply, it falls back to netperf = UDP_RR. Do you think that=E2=80=99s what happened here?

On this front, it could merely be that my (not tested in
months!) test cablemodem setup is malfunctioning also! Or = we're
hitting power save. Or (finally) seeing = request-grant delays. Or
scheduling delay somewhere in the = net-next kernel I'm using... Or....
(regardless, this = seems independent of my main issue, and I've not had
such = high res data before)

Regarding the spikes both you and Arie you=E2=80=99r= e seeing, I also saw in one of your later emails "0 delay via fq would = be better than even
the 15-40ms I'm getting now with linux = flows=E2=80=9D. Those numbers struck me as similar to an issue I=E2=80=99m= still dealing with:


To summarize, with = airOS on the NanoStation M5, there are isochronous pauses around once = per second in the processing of all packets, not just for the WiFi = device but Ethernet also. Packets are not lost, but queued for either = 20ms, if one Ethernet port is connected, or 40ms, if both are connected. = This behavior is exactly described by the ar7240sw_phy_poll_reset = function in ag71xx_ar7240.c, so it looks to me like the ar7240 = internal switch is being reset once per second for no apparent reason. = So far I=E2=80=99ve gotten crickets in response.

The cable modem you=E2=80=99re using doesn=E2=80=99t= happen to have built-in WiFi and an ar7240, does it? If it does and has = the same or a similar driver problem, you could just do a low-interval = ping straight to its Ethernet adapter and check for isochronous spikes, = something like:

sudo ping -l 100 -c = 5000 -i 0.001 cablemodem

Now, back = to vacation :)
= --Apple-Mail=_76A355F7-18C6-4F85-BDA4-4E49335D4D15--