From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4E0803B29E for ; Fri, 28 Dec 2018 16:22:13 -0500 (EST) Received: by mail-wr1-x444.google.com with SMTP id v13so21950220wrw.5 for ; Fri, 28 Dec 2018 13:22:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=BdC9Mhow8say1Rcd9LzVa14fpZVgbVxG3ppInlyXKwg=; b=axmuG8ds5QufTy6pr7hzme+HueTQyhgsRP0B/mstOYVGQbeouuKwd8+OCvc+BcOZ2r udwf4xPhaUOvYakUsPUE0zLb7kMzjVvXDTS9iHv00JALD7t/ety25T/WezLLk5T+AlSS B6fwdpEhkmHdxX+Y6yZbr+qBbc1nAQTpPiLhstCvGIf+5ynms2MoUlz9CCXoIGdPUh87 V2n9x41+W9JV2ZtpPDopyruouwzgycvP9ppuCtqfH93Vlke9fFzjwaTYX6WrgpDrPnZP jPegFLyEO8pKoxyG8HmCtFGx6GT7ZnZcVXWsYfMpSS1MCduy+Tq1RK268mK33+Caw0QG oJcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=BdC9Mhow8say1Rcd9LzVa14fpZVgbVxG3ppInlyXKwg=; b=jvKOyfq7a76LlaZDKg+Wc6+5y3zh9rKLNZnypd8MfOyq2cBJBFN4HxfxnRo/7yGQVw hbiBo1gaJNucQ33R7vWUUfCqNvOXviLxZmwqFszlyjNRLfL3Uw1aj2VcFIkMzGwNJW+O fHeysWGQ+qeZrMTaNeA/nWe6m0SKmpRhj/kIs34niojuxO3Uj/qtNTSdAMhZUARq/htw xJSbPULCrqcoQJ1YMrD6h1lN0PJ06Lf1wg/Vv0HIet9WRI+K4VGKBJPjaZE4NUjGe1qH o2KrI46S8NYl+pJWCW8s7fUcHAn4D2orFFrsAwEMgFnCy9cxKc420olLU7ShLaEVnLOf IAuQ== X-Gm-Message-State: AJcUukePTQh7lBqYRt128cKV7PXvlLiyVPSPklvqft/aCzUJrZv7t3VM vCCAEZja1POFkibC6lhgvoB/Ukj9Ic0= X-Google-Smtp-Source: ALg8bN7G1PilmFtrLuU/aKn3RYYo09LOxVXR2wQrLlgAmsvSPrt060Q2rWDZu4W/9DnPywgqGJeVsQ== X-Received: by 2002:a05:6000:12c4:: with SMTP id l4mr25645543wrx.134.1546032132066; Fri, 28 Dec 2018 13:22:12 -0800 (PST) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id z206sm33243643wmc.18.2018.12.28.13.22.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 13:22:11 -0800 (PST) From: Pete Heist Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Date: Fri, 28 Dec 2018 22:22:09 +0100 References: To: Cake List In-Reply-To: Message-Id: <5482A3CA-9C36-4DDE-A858-24D8467F70C7@heistp.net> X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] cake infinite loop(?) with hfsc on one-armed router X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 21:22:13 -0000 Ok, the lockup goes away if you use no-split-gso on the cake qdiscs for = the default traffic (noted below in the drr and hfsc cases with "!!! = must use no-split-gso here !!!"). Only I=E2=80=99d like my 600 =CE=BCs = back. :) This smells of a bug Toke fixed on Sep 12, 2018 in = 42e87f12ea5c390bf5eeb658c942bc810046160a, but then reverted in the next = commit because it was fixed upstream. However, if I re-apply that = commit, it still doesn=E2=80=99t fix it. Perhaps there are more cases where skb_reset_mac_len(skb) needs to be = called somewhere for VLAN support? I managed to capture some output from what happens to hfsc: [ 683.864456] ------------[ cut here ]------------ [ 683.869116] WARNING: CPU: 1 PID: 11 at net/sched/sch_hfsc.c:1427 = 0xf9ced4ef() [ 683.876267] Modules linked in: cls_u32 em_meta cls_basic sch_cake(O) = sch_drr xt_ACCOUNT(O) sch_hfsc cls_fw sch_sfq sch_prio ipt_Ra [ 683.931317] CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G W O = 3.16.7-ckt9-voyage #1 [ 683.939595] Hardware name: PC Engines APU/APU, BIOS 4.0 09/08/2014 [ 683.945790] 00000000 00000000 f5c8bc9c c13167e9 00000000 f5c8bcb4 = c102a7dd f9ced4ef [ 683.953792] f1907c00 00000000 00000000 f5c8bcc4 c102a803 00000009 = 00000000 f5c8bce4 [ 683.961791] f9ced4ef f1907fc8 732494ae 00000002 f1907c00 00000000 = 00000040 f5c8bd00 [ 683.969783] Call Trace: [ 683.972256] [] dump_stack+0x41/0x52 [ 683.976729] [] warn_slowpath_common+0x5c/0x73 [ 683.982063] [] ? 0xf9ced4ee [ 683.985834] [] warn_slowpath_null+0xf/0x13 [ 683.990905] [] 0xf9ced4ee [ 683.994499] [] __qdisc_run+0x81/0xf0 [ 683.999052] [] __dev_queue_xmit+0x23d/0x35f [ 684.004216] [] dev_queue_xmit+0xa/0xc [ 684.008857] [] register_vlan_dev+0x938/0xe3b [8021q] [ 684.014799] [] dev_hard_start_xmit+0x29e/0x37b [ 684.020223] [] __dev_queue_xmit+0x2a8/0x35f [ 684.025381] [] dev_queue_xmit+0xa/0xc [ 684.030016] [] arp_xmit+0x1c/0x47 [ 684.034307] [] arp_send+0x2e/0x33 [ 684.038598] [] arp_process+0x288/0x4d8 [ 684.043331] [] ? ip_forward_finish+0x66/0x6b [ 684.048581] [] ? __kfree_skb+0x5d/0x5f [ 684.053303] [] arp_rcv+0xca/0x102 [ 684.057597] [] __netif_receive_skb_core+0x467/0x4b6 [ 684.063453] [] __netif_receive_skb+0x48/0x59 [ 684.068698] [] netif_receive_skb_internal+0x59/0x85 [ 684.074557] [] napi_gro_receive+0x31/0x6d [ 684.079549] [] ? text_poke_bp+0xa0/0xa0 [ 684.084369] [] 0xf8086049 [ 684.087974] [] net_rx_action+0x56/0x10e [ 684.092791] [] __do_softirq+0x91/0x175 [ 684.097523] [] run_ksoftirqd+0x16/0x29 [ 684.102255] [] smpboot_thread_fn+0x108/0x11e [ 684.107505] [] ? SyS_setgroups+0xa6/0xa6 [ 684.112403] [] kthread+0x9f/0xa4 [ 684.116615] [] ret_from_kernel_thread+0x21/0x30 [ 684.122126] [] ? kthread_freezable_should_stop+0x40/0x40 [ 684.128407] ---[ end trace cb7778967851e0ad ]--- [ 684.133646] ------------[ cut here ]------------ [ 684.138337] WARNING: CPU: 1 PID: 11 at net/sched/sch_hfsc.c:1427 = 0xf9ced4ef() [ 684.145487] Modules linked in: cls_u32 em_meta cls_basic sch_cake(O) = sch_drr xt_ACCOUNT(O) sch_hfsc cls_fw sch_sfq sch_prio ipt_Ra [ 684.200459] CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G W O = 3.16.7-ckt9-voyage #1 [ 684.208736] Hardware name: PC Engines APU/APU, BIOS 4.0 09/08/2014 [ 684.214933] 00000000 00000000 f5c8be98 c13167e9 00000000 f5c8beb0 = c102a7dd f9ced4ef [ 684.222930] f1907c00 00000000 00000000 f5c8bec0 c102a803 00000009 = 00000000 f5c8bee0 [ 684.230928] f9ced4ef f1907fc8 7364c482 00000002 f1907c00 00000000 = 00000040 f5c8befc [ 684.238926] Call Trace: [ 684.241399] [] dump_stack+0x41/0x52 [ 684.245870] [] warn_slowpath_common+0x5c/0x73 [ 684.251206] [] ? 0xf9ced4ee [ 684.254979] [] warn_slowpath_null+0xf/0x13 [ 684.260055] [] 0xf9ced4ee [ 684.263651] [] __qdisc_run+0x81/0xf0 [ 684.268203] [] net_tx_action+0x91/0xdd [ 684.272927] [] __do_softirq+0x91/0x175 [ 684.277659] [] run_ksoftirqd+0x16/0x29 [ 684.282389] [] smpboot_thread_fn+0x108/0x11e [ 684.287633] [] ? SyS_setgroups+0xa6/0xa6 [ 684.292529] [] kthread+0x9f/0xa4 [ 684.296735] [] ret_from_kernel_thread+0x21/0x30 [ 684.302246] [] ? kthread_freezable_should_stop+0x40/0x40 [ 684.308536] ---[ end trace cb7778967851e0ae ]--- > On Dec 28, 2018, at 1:58 PM, Pete Heist wrote: >=20 > Note that this doesn=E2=80=99t happen when prio is used in place of = hfsc and cake is used in the leafs to do the rate limiting, i.e.: >=20 > tc qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 = 1 1 1 1 1 1 1 1 1 1 > tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth = 100mbit ethernet # !!! must use no-split-gso here !!! > tc qdisc add dev eth0 parent 1:2 handle 11: cake besteffort bandwidth = 100mbit ethernet ether-vlan > tc filter add dev eth0 protocol all parent 1:0 prio 1 basic match = "meta(vlan mask 0xfff eq 0xce4)" flowid 1:2 > tc filter add dev eth0 protocol all parent 1:0 prio 2 u32 match u32 0 = 0 flowid 1:1 >=20 > But it does happen when drr is used instead of prio: >=20 > tc qdisc add dev eth0 root handle 1: drr > tc class add dev eth0 parent 1: classid 1:1 > tc class add dev eth0 parent 1: classid 1:2 > tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth = 100mbit > tc qdisc add dev eth0 parent 1:2 handle 11: cake besteffort bandwidth = 100mbit ether-vlan > tc filter add dev eth0 protocol all parent 1:0 prio 1 basic match = "meta(vlan mask 0xfff eq 0xce4)" flowid 1:2 > tc filter add dev eth0 protocol all parent 1:0 prio 2 u32 match u32 0 = 0 flowid 1:1 >=20 > drr might ultimately be what I want to use for this, so I can use cake = to do the rate limiting instead of htb. prio works well but leads to = starvation when the rate limit is above what the CPU can handle. >=20 > Meanwhile, using htb classes with rate limits way above the actual, = then rate limiting in the cake leafs, works as well, but this seems like = a hack: >=20 > tc qdisc add dev eth0 root handle 1: htb default 10 > tc class add dev eth0 parent 1: classid 1:1 htb rate 10gbit > tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5gbit > tc class add dev eth0 parent 1:1 classid 1:11 htb rate 5gbit > tc filter add dev eth0 protocol ip parent 1:0 prio 1 basic match = "meta(vlan mask 0xfff eq 0xce4)" flowid 1:11 > tc qdisc add dev eth0 parent 1:10 handle 20: cake besteffort bandwidth = 100mbit ethernet # !!! must use no-split-gso here !!! > tc qdisc add dev eth0 parent 1:11 handle 21: cake besteffort bandwidth = 100mbit ethernet ether-vlan >=20 >> On Dec 28, 2018, at 12:30 AM, Pete Heist wrote: >>=20 >> I=E2=80=99m seeing what I think it an infinite loop when cake is used = in a one-armed router configuration with hfsc as the rate limiter. Three = APUs are connected to the same switch and the =E2=80=9Cmiddle=E2=80=9D = APU (apu1a) routes between the default VLAN and a tagged VLAN. >>=20 >> apu2a <=E2=80=94 default VLAN =E2=80=94> apu1a <=E2=80=94 VLAN = 3300 =E2=80=94> apu2b >>=20 >> After qos is set up, ping from apu2a to apu2b still works fine. When = iperf3 is run from apu2a to apu2b it works fine, but when it goes in = reverse (apu2b to apu2a), all traffic stops flowing from apu1a on the = default VLAN. Traffic still flows from apu1a on VLAN 3300 however, with = very high RTT (mean 500ms), leading me to believe that the cake instance = on the default VLAN is in an infinite loop. >>=20 >> It does not happen with hfsc+fq_codel, or with htb+cake in the same = configuration. >>=20 >> Here are the commands that set up qos, and it only locks up when cake = is used as the instance at handle 20, not at handle 21: >>=20 >> ----- >> tc qdisc add dev eth0 root handle 1: hfsc default 10 >> tc class add dev eth0 parent 1: classid 1:1 hfsc sc rate 200mbit ul = rate 200mbit >> tc class add dev eth0 parent 1:1 classid 1:10 hfsc sc rate 100mbit ul = rate 100mbit >> tc class add dev eth0 parent 1:1 classid 1:11 hfsc sc rate 100mbit ul = rate 100mbit >> tc filter add dev eth0 protocol ip parent 1:0 prio 1 \ >> basic match "meta(vlan mask 0xfff eq 0xce4)" flowid 1:11 >> tc qdisc add dev eth0 parent 1:10 handle 20: fq_codel # using cake = here locks up !!! >> tc qdisc add dev eth0 parent 1:11 handle 21: cake >> =E2=80=94=E2=80=94 >>=20 >> I=E2=80=99m using sch_cake and tc-adv from the current HEAD, on = kernel 3.16.7 (yeah, I know). >>=20 >> root@apu1a:~/qos# uname -a >> Linux apu1a 3.16.7-ckt9-voyage #1 SMP Thu Apr 23 11:10:44 HKT 2015 = i686 GNU/Linux >>=20 >> Any ideas just from just this? Otherwise, I can only think to hook up = the serial cable and start with the printk=E2=80=99s=E2=80=A6 >>=20 >=20