* Re: [Cake] Cake on openwrt - falling behind
[not found] <mailman.381.1530347349.3512.cake@lists.bufferbloat.net>
@ 2018-06-30 16:37 ` Georgios Amanakis
2018-06-30 17:26 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Georgios Amanakis @ 2018-06-30 16:37 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]
Dear Kevin and Toke,
First of all thank you for all your efforts!
I recently acquired a linksys wrt1900acs (arm/mvebu), I will test this by
compiling latest openwrt with latest sch_cake/master and tc-adv/master.
Best,
George
On Sat, Jun 30, 2018, 4:29 AM Kevin Darbyshire-Bryant via Cake <
cake@lists.bufferbloat.net> wrote:
>
>
>
> ---------- Forwarded message ----------
> From: Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
> To: Cake List <cake@lists.bufferbloat.net>
> Cc:
> Bcc:
> Date: Sat, 30 Jun 2018 08:29:05 +0000
> Subject: Cake on openwrt - falling behind
> Hi Chaps,
>
> I’m concerned that cake on openwrt is falling behind. Due to strange
> behaviour at least on MIPS since commit
> https://github.com/dtaht/sch_cake/commit/af1d7cde7046af55ec867b29854d754816b64bc8
> Switch rates to 64bit, openwrt has been ‘stuck’ on the prior commit. I
> personally have hack patches in my that effectively reverts the conversion
> to 64 netlink values in cake & tc and I’m pleased to say that works,
> however the wider community are a) not so lucky b) not actually running our
> latest code, there are a number of tweaks to the ack filter code that
> openwrt users are not benefitting from.
>
> Toke & myself dug into this a bit and we don’t, so far, think this is a
> bug in cake but is rather exposing an issue in netlink handling. But we’ve
> hit an information vacuum, in essence I’m only seeing this on MIPS/Openwrt
> *BUT* that’s the only thing I can actually test on. So is this 32bit MIPS
> only? Is it Openwrt only? Is it MIPS & Openwrt only? Is it 32bit archs
> only? And so on.
>
> This really needs to be investigated & solved, so that a) openwrt’s cake
> can be bumped and b) Someone can get on with backporting the class handling
> code (which is being sent upstream) to 4.14/4.9 (maybe earlier)
>
> If *you* have the means of compiling the latest cake & tc code on your
> platform could you please do so and report architecture etc if you do/do
> not see cake’s tin stats from ‘tc -s qdisc’ (that’s the biggest indicator
> of the issue)
>
>
> Cheers,
>
> Kevin D-B
>
> 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
>
>
>
>
> ---------- Forwarded message ----------
> From: Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net>
> To: Cake List <cake@lists.bufferbloat.net>
> Cc:
> Bcc:
> Date: Sat, 30 Jun 2018 01:29:10 -0700 (PDT)
> Subject: [Cake] Cake on openwrt - falling behind
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
[-- Attachment #2: Type: text/html, Size: 3975 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 16:37 ` [Cake] Cake on openwrt - falling behind Georgios Amanakis
@ 2018-06-30 17:26 ` Pete Heist
2018-06-30 18:09 ` Georgios Amanakis
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-06-30 17:26 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]
> On Jun 30, 2018, at 6:37 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> I recently acquired a linksys wrt1900acs (arm/mvebu), I will test this by compiling latest openwrt with latest sch_cake/master and tc-adv/master.
Hi Georgios- I’m also donating tonight to making some progress on this, so let’s see what we can come up with. :) This is actually the first time I’ve had to compile OpenWRT- my VM ran out of memory the first time, but my laptop's fans have been on full blast for 10 minutes now so it looks like all’s well.
I’m not yet sure how to get the head of cake and tc-adv into the tree but I figured I’d cross that bridge when I come to it. Ideally I’d like to be able to recompile sch_cake.ko after the system is running and not have to rebuild an OpenWRT image every time I make a change, in case I want to experiment with the code, so I hope that’s possible. I worked with C back in the day, but with rather different tooling so at the moment I feel like my hands are tied behind my back- hoping that changes. :)
For starters, I’ll just attempt to try it on my OM2P (cpumodel MIPS 24Kc V7.4). I would be thrilled if the tc stats failed.
Pete
[-- Attachment #2: Type: text/html, Size: 2198 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 17:26 ` Pete Heist
@ 2018-06-30 18:09 ` Georgios Amanakis
2018-06-30 18:55 ` Kevin Darbyshire-Bryant
[not found] ` <mailman.384.1530384918.3512.cake@lists.bufferbloat.net>
0 siblings, 2 replies; 77+ messages in thread
From: Georgios Amanakis @ 2018-06-30 18:09 UTC (permalink / raw)
To: Pete Heist; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 1937 bytes --]
Great Pete!
For cake it's not hard to do, previously I was just switching
PKG_SOURCE_VERSION in package/kernel/kmod-sched-cake/Makefile to the latest
git commit. That would be:
PKG_SOURCE_VERSION:=0520a6cbcce759b109568b26df3b6081aed42483
Then I just go for "make package/kmod-sched-cake/compile -j1 V=s" in
openwrt root directory. The version in git master should also compile for
4.14, shouldn't it?
For tc-adv, I don't exactly remember out of the top of my head, but I think
I had to modify the iproute2-full package which also compiles TC. I will
let you know once I get my hands on it.
On Sat, Jun 30, 2018, 1:26 PM Pete Heist <pete@heistp.net> wrote:
>
> On Jun 30, 2018, at 6:37 PM, Georgios Amanakis <gamanakis@gmail.com>
> wrote:
>
> I recently acquired a linksys wrt1900acs (arm/mvebu), I will test this by
> compiling latest openwrt with latest sch_cake/master and tc-adv/master.
>
>
> Hi Georgios- I’m also donating tonight to making some progress on this, so
> let’s see what we can come up with. :) This is actually the first time I’ve
> had to compile OpenWRT- my VM ran out of memory the first time, but my
> laptop's fans have been on full blast for 10 minutes now so it looks like
> all’s well.
>
> I’m not yet sure how to get the head of cake and tc-adv into the tree but
> I figured I’d cross that bridge when I come to it. Ideally I’d like to be
> able to recompile sch_cake.ko after the system is running and not have to
> rebuild an OpenWRT image every time I make a change, in case I want to
> experiment with the code, so I hope that’s possible. I worked with C back
> in the day, but with rather different tooling so at the moment I feel like
> my hands are tied behind my back- hoping that changes. :)
>
> For starters, I’ll just attempt to try it on my OM2P (cpumodel MIPS 24Kc
> V7.4). I would be thrilled if the tc stats failed.
>
> Pete
>
>
[-- Attachment #2: Type: text/html, Size: 2938 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 18:09 ` Georgios Amanakis
@ 2018-06-30 18:55 ` Kevin Darbyshire-Bryant
2018-06-30 19:57 ` Pete Heist
[not found] ` <mailman.384.1530384918.3512.cake@lists.bufferbloat.net>
1 sibling, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-06-30 18:55 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Pete Heist, Cake List
[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]
> On 30 Jun 2018, at 19:09, Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> Great Pete!
>
> For cake it's not hard to do, previously I was just switching PKG_SOURCE_VERSION in package/kernel/kmod-sched-cake/Makefile to the latest git commit. That would be:
> PKG_SOURCE_VERSION:=0520a6cbcce759b109568b26df3b6081aed42483
> Then I just go for "make package/kmod-sched-cake/compile -j1 V=s" in openwrt root directory. The version in git master should also compile for 4.14, shouldn't it?
>
> For tc-adv, I don't exactly remember out of the top of my head, but I think I had to modify the iproute2-full package which also compiles TC. I will let you know once I get my hands on it.
>
> On Sat, Jun 30, 2018, 1:26 PM Pete Heist <pete@heistp.net> wrote:
>
>> On Jun 30, 2018, at 6:37 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>>
>> I recently acquired a linksys wrt1900acs (arm/mvebu), I will test this by compiling latest openwrt with latest sch_cake/master and tc-adv/master.
>
> Hi Georgios- I’m also donating tonight to making some progress on this, so let’s see what we can come up with. :) This is actually the first time I’ve had to compile OpenWRT- my VM ran out of memory the first time, but my laptop's fans have been on full blast for 10 minutes now so it looks like all’s well.
>
> I’m not yet sure how to get the head of cake and tc-adv into the tree but I figured I’d cross that bridge when I come to it. Ideally I’d like to be able to recompile sch_cake.ko after the system is running and not have to rebuild an OpenWRT image every time I make a change, in case I want to experiment with the code, so I hope that’s possible. I worked with C back in the day, but with rather different tooling so at the moment I feel like my hands are tied behind my back- hoping that changes. :)
>
> For starters, I’ll just attempt to try it on my OM2P (cpumodel MIPS 24Kc V7.4). I would be thrilled if the tc stats failed.
>
> Pete
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
May I direct your attention to https://git.openwrt.org/?p=openwrt/staging/ldir.git;a=shortlog;h=refs/heads/cakepatches
A selection of patches including my hack workaround…which you can choose to include/exclude from your test builds… as well as Toke’s ‘extra info’ patch to see what netlink stats blocks are being returned.
Cheers,
Kevin D-B
012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 18:55 ` Kevin Darbyshire-Bryant
@ 2018-06-30 19:57 ` Pete Heist
2018-06-30 20:58 ` Georgios Amanakis
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-06-30 19:57 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Georgios Amanakis, Cake List
> On Jun 30, 2018, at 8:55 PM, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
> May I direct your attention to https://git.openwrt.org/?p=openwrt/staging/ldir.git;a=shortlog;h=refs/heads/cakepatches
>
> A selection of patches including my hack workaround…which you can choose to include/exclude from your test builds… as well as Toke’s ‘extra info’ patch to see what netlink stats blocks are being returned.
Thanks, that’s a big help... Where I’m stuck now is I’ve got a system compiled with three of the four patches (just not the hack for now) but I seem to have a kernel that loves panicing (I think even without these patches). scp’ing a file to it results in a panic about 50% of the time, and of course I have no logs.
Also, I might be going about this the wrong way, but I include kmod-sched-cake and iproute2 as modules in the build but still have to copy the ipks over manually after I install the image, then for starters cake can’t be installed because of some dependency problem. It must be me.
root@LEDE:/tmp# uname -sr
Linux 4.9.109
root@LEDE:/tmp# opkg install ./kmod-sched-cake_4.9.109\+2018-06-26-0520a6cb-1_mips_24kc.ipk
Installing kmod-sched-cake (4.9.109+2018-06-26-0520a6cb-1) to root...
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-sched-cake:
* kernel (= 4.9.109-1-75eefef32bb09870dd8be117d46be950) *
* opkg_install_cmd: Cannot install package kmod-sched-cake.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 19:57 ` Pete Heist
@ 2018-06-30 20:58 ` Georgios Amanakis
2018-06-30 21:37 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Georgios Amanakis @ 2018-06-30 20:58 UTC (permalink / raw)
To: Pete Heist; +Cc: Kevin Darbyshire-Bryant, Cake List
[-- Attachment #1: Type: text/plain, Size: 642 bytes --]
On Sat, Jun 30, 2018, 3:57 PM Pete Heist <pete@heistp.net> wrote:
>
> Also, I might be going about this the wrong way, but I include
> kmod-sched-cake and iproute2 as modules in the build but still have to copy
> the ipks over manually after I install the image, then for starters cake
> can’t be installed because of some dependency problem. It must be me.
>
Usually this is what I do, too. You have to install the base openwrt image
you just compiled and then install the two ipks. Otherwise if you go with
some other precompiled base image it throws dependency errors.
Did you install the base image you just compiled?
[-- Attachment #2: Type: text/html, Size: 1193 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 20:58 ` Georgios Amanakis
@ 2018-06-30 21:37 ` Pete Heist
2018-06-30 22:43 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-06-30 21:37 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jun 30, 2018, at 10:58 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> Usually this is what I do, too. You have to install the base openwrt image you just compiled and then install the two ipks. Otherwise if you go with some other precompiled base image it throws dependency errors.
>
> Did you install the base image you just compiled?
Yes, installed the one that was built into bin/targets/ar71xx/generic:
OpenWrt SNAPSHOT, r7360-e15565a01c
It's the sysupgrade image for OM2P, then I used sysupgrade to install as I’ve done with release builds.
Also not sure if I applied the patches right. I just copied them into the right package directory, but now I’m reading about quilt, if I should have installed them some proper way.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 21:37 ` Pete Heist
@ 2018-06-30 22:43 ` Pete Heist
2018-06-30 23:20 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-06-30 22:43 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jun 30, 2018, at 11:37 PM, Pete Heist <pete@heistp.net> wrote:
>
>
>> On Jun 30, 2018, at 10:58 PM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>>
>> Usually this is what I do, too. You have to install the base openwrt image you just compiled and then install the two ipks. Otherwise if you go with some other precompiled base image it throws dependency errors.
>>
>> Did you install the base image you just compiled?
>
> Yes, installed the one that was built into bin/targets/ar71xx/generic:
>
> OpenWrt SNAPSHOT, r7360-e15565a01c
>
> It's the sysupgrade image for OM2P, then I used sysupgrade to install as I’ve done with release builds.
>
> Also not sure if I applied the patches right. I just copied them into the right package directory, but now I’m reading about quilt, if I should have installed them some proper way.
There are a whole bunch of unknown symbols in dmesg. I’ll just make clean and build from scratch. Not much progress for a spent Saturday night. :)
act_mirred: Unknown symbol __tcf_hash_release (err 0)
act_mirred: Unknown symbol tcf_generic_walker (err 0)
…
act_skbedit: Unknown symbol __tcf_hash_release (err 0)
act_skbedit: Unknown symbol tcf_generic_walker (err 0)
…
sch_ingress: Unknown symbol net_dec_egress_queue (err 0)
sch_ingress: Unknown symbol net_dec_ingress_queue (err 0)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-06-30 22:43 ` Pete Heist
@ 2018-06-30 23:20 ` Pete Heist
[not found] ` <CACvFP_gkdAPKSEO7j9+cMPqTa-fJYd8XFEEBD6ZzLuVvaNwsvg@mail.gmail.com>
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-06-30 23:20 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Kevin Darbyshire-Bryant, Cake List
Ok, same result (unknown symbols and routine crashing) after a make clean and re-build, but I just force installed the kmod-sched-cake package with --force-depends and it seems to work. This reproduces it, right? no stats and the debug output?
root@LEDE:~# tc qdisc add dev eth0 root cake
root@LEDE:~# tc -s -d qdisc show dev eth0
qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
tca_stats 2005579564 tca_stats2 0 tca_xstats 0
calling print_tcstats_attr()
print_tcstats_attr()
got stats
Sent 9070 bytes 66 pkts (dropped 0, overlimits 0)
got xstats 0 tca_stats 2005579564 tca_stats2 0 tca_xstats 0
root@LEDE:~# cat /proc/cpuinfo
system type : Atheros AR7240 rev 2
machine : OpenMesh OM2P
processor : 0
cpu model : MIPS 24Kc V7.4
BogoMIPS : 265.42
wait instruction : yes
microsecond timers : yes
tlb_entries : 16
extra interrupt vector : yes
hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa : mips1 mips2 mips32r1 mips32r2
ASEs implemented : mips16
shadow register sets : 1
kscratch registers : 0
package : 0
core : 0
VCED exceptions : not available
VCEI exceptions : not available
^ permalink raw reply [flat|nested] 77+ messages in thread
* [Cake] Fwd: Cake on openwrt - falling behind
[not found] ` <CACvFP_gkdAPKSEO7j9+cMPqTa-fJYd8XFEEBD6ZzLuVvaNwsvg@mail.gmail.com>
@ 2018-07-01 2:37 ` Georgios Amanakis
2018-07-01 7:18 ` [Cake] " Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Georgios Amanakis @ 2018-07-01 2:37 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant, Cake List
The stats got mangled in by the email, in the terminal they are
properly aligned.
---------- Forwarded message ----------
From: Georgios Amanakis <gamanakis@gmail.com>
Date: Sat, Jun 30, 2018 at 10:35 PM
Subject: Re: [Cake] Cake on openwrt - falling behind
To: Pete Heist <pete@heistp.net>
I just tested this on my wrt1900acs but it is behaving as it should. I
used Kevin's RFC patches for iproute2 and kmod-sched-cake.
root@lede:/tmp# tc -s qdisc show dev eth0
qdisc cake 8004: root refcnt 9 bandwidth 2500Kbit diffserv3
triple-isolate split-gso rtt 100.0ms raw overhead 0
Sent 122745 bytes 691 pkt (dropped 0, overlimits 419 requeues 0)
backlog 0b 0p requeues 0
memory used: 6416b of 4Mb
capacity estimate: 2500Kbit
min/max network layer size: 42 / 1186
min/max overhead-adjusted size: 42 / 1186
average network hdr offset: 13
Bulk Best Effort Voice
thresh 156248bit 2500Kbit 625Kbit
target 116.3ms 7.3ms 29.1ms
interval 232.6ms 102.3ms 124.1ms
pk_delay 0us 196us 8.6ms
av_delay 0us 13us 1.3ms
sp_delay 0us 2us 38us
backlog 0b 0b 0b
pkts 0 414 277
bytes 0 38367 84378
way_inds 0 0 0
way_miss 0 20 2
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 1 0
bk_flows 0 0 1
un_flows 0 0 0
max_len 0 206 1186
quantum 300 300 300
root@lede:/tmp# opkg list-installed | grep cake
kmod-sched-cake - 4.14.50+2018-06-26-0520a6cb-1
root@lede:/tmp# uname -a
Linux lede 4.14.50 #0 SMP Wed Jun 27 21:48:32 2018 armv7l GNU/Linux
George
On Sat, Jun 30, 2018 at 7:20 PM, Pete Heist <pete@heistp.net> wrote:
> Ok, same result (unknown symbols and routine crashing) after a make clean and re-build, but I just force installed the kmod-sched-cake package with --force-depends and it seems to work. This reproduces it, right? no stats and the debug output?
>
> root@LEDE:~# tc qdisc add dev eth0 root cake
> root@LEDE:~# tc -s -d qdisc show dev eth0
> qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
> tca_stats 2005579564 tca_stats2 0 tca_xstats 0
> calling print_tcstats_attr()
> print_tcstats_attr()
> got stats
> Sent 9070 bytes 66 pkts (dropped 0, overlimits 0)
> got xstats 0 tca_stats 2005579564 tca_stats2 0 tca_xstats 0
>
> root@LEDE:~# cat /proc/cpuinfo
> system type : Atheros AR7240 rev 2
> machine : OpenMesh OM2P
> processor : 0
> cpu model : MIPS 24Kc V7.4
> BogoMIPS : 265.42
> wait instruction : yes
> microsecond timers : yes
> tlb_entries : 16
> extra interrupt vector : yes
> hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
> isa : mips1 mips2 mips32r1 mips32r2
> ASEs implemented : mips16
> shadow register sets : 1
> kscratch registers : 0
> package : 0
> core : 0
> VCED exceptions : not available
> VCEI exceptions : not available
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 2:37 ` [Cake] Fwd: " Georgios Amanakis
@ 2018-07-01 7:18 ` Pete Heist
2018-07-01 13:48 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-01 7:18 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Kevin Darbyshire-Bryant, Cake List
Ok, so I’ve got to figure out what’s up with my build in general, so I’m blowing away my source tree and will build master without any patches.
I don’t know if the kernel panics and unknown symbols are related somehow, but they’re not helping(!) so I’ll try to get past that. I don’t actually get logs in /sys/kernel/debug/crashlog, so I can’t even prove it’s a panic, but it’s rebooting that usually happens just as I do something net related, like start typing in ssh or do an scp, after some period of inactivity. If it’s reproducible on an unpatched build I’ll head over to the OpenWRT universe.
(the stats alignment is just proportional vs fixed fonts- they look fine in vim)
> On Jul 1, 2018, at 4:37 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> The stats got mangled in by the email, in the terminal they are
> properly aligned.
>
>
> ---------- Forwarded message ----------
> From: Georgios Amanakis <gamanakis@gmail.com>
> Date: Sat, Jun 30, 2018 at 10:35 PM
> Subject: Re: [Cake] Cake on openwrt - falling behind
> To: Pete Heist <pete@heistp.net>
>
>
> I just tested this on my wrt1900acs but it is behaving as it should. I
> used Kevin's RFC patches for iproute2 and kmod-sched-cake.
>
> root@lede:/tmp# tc -s qdisc show dev eth0
> qdisc cake 8004: root refcnt 9 bandwidth 2500Kbit diffserv3
> triple-isolate split-gso rtt 100.0ms raw overhead 0
> Sent 122745 bytes 691 pkt (dropped 0, overlimits 419 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 6416b of 4Mb
> capacity estimate: 2500Kbit
> min/max network layer size: 42 / 1186
> min/max overhead-adjusted size: 42 / 1186
> average network hdr offset: 13
>
> Bulk Best Effort Voice
> thresh 156248bit 2500Kbit 625Kbit
> target 116.3ms 7.3ms 29.1ms
> interval 232.6ms 102.3ms 124.1ms
> pk_delay 0us 196us 8.6ms
> av_delay 0us 13us 1.3ms
> sp_delay 0us 2us 38us
> backlog 0b 0b 0b
> pkts 0 414 277
> bytes 0 38367 84378
> way_inds 0 0 0
> way_miss 0 20 2
> way_cols 0 0 0
> drops 0 0 0
> marks 0 0 0
> ack_drop 0 0 0
> sp_flows 0 1 0
> bk_flows 0 0 1
> un_flows 0 0 0
> max_len 0 206 1186
> quantum 300 300 300
>
> root@lede:/tmp# opkg list-installed | grep cake
> kmod-sched-cake - 4.14.50+2018-06-26-0520a6cb-1
>
> root@lede:/tmp# uname -a
> Linux lede 4.14.50 #0 SMP Wed Jun 27 21:48:32 2018 armv7l GNU/Linux
>
> George
>
> On Sat, Jun 30, 2018 at 7:20 PM, Pete Heist <pete@heistp.net> wrote:
>> Ok, same result (unknown symbols and routine crashing) after a make clean and re-build, but I just force installed the kmod-sched-cake package with --force-depends and it seems to work. This reproduces it, right? no stats and the debug output?
>>
>> root@LEDE:~# tc qdisc add dev eth0 root cake
>> root@LEDE:~# tc -s -d qdisc show dev eth0
>> qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
>> tca_stats 2005579564 tca_stats2 0 tca_xstats 0
>> calling print_tcstats_attr()
>> print_tcstats_attr()
>> got stats
>> Sent 9070 bytes 66 pkts (dropped 0, overlimits 0)
>> got xstats 0 tca_stats 2005579564 tca_stats2 0 tca_xstats 0
>>
>> root@LEDE:~# cat /proc/cpuinfo
>> system type : Atheros AR7240 rev 2
>> machine : OpenMesh OM2P
>> processor : 0
>> cpu model : MIPS 24Kc V7.4
>> BogoMIPS : 265.42
>> wait instruction : yes
>> microsecond timers : yes
>> tlb_entries : 16
>> extra interrupt vector : yes
>> hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
>> isa : mips1 mips2 mips32r1 mips32r2
>> ASEs implemented : mips16
>> shadow register sets : 1
>> kscratch registers : 0
>> package : 0
>> core : 0
>> VCED exceptions : not available
>> VCEI exceptions : not available
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
[not found] ` <mailman.384.1530384918.3512.cake@lists.bufferbloat.net>
@ 2018-07-01 9:46 ` Magnus Olsson
2018-07-01 12:34 ` Kevin Darbyshire-Bryant
0 siblings, 1 reply; 77+ messages in thread
From: Magnus Olsson @ 2018-07-01 9:46 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
Hi,
Have done some testing. Using the following patches from Kevin's
cakepatches tree:
iproute2: RFC update cake support
kmod-sched-cake: RFC latest cake
'tc -s -d qdisc show' does not give tin status on:
ramips: MediaTek MT7621 (MIPS 1004Kc) / UBNT-ERX-SFP
ath79: Qualcomm Atheros QCA956X (MIPS 74Kc) / Ubiquiti UniFi-AC-LITE
ath79: Atheros AR7161 (MIPS 24Kc) / Netgear WNDR3800
Works on:
mpc85xx: Freescale P1014 (e500v2) / TP-Link TL-WDR4900 v1
Also made another round including the cake 'u32 hacks' patch, and that
solves for all above.
On 2018-06-30 20:55, Kevin Darbyshire-Bryant via Cake wrote:
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 9:46 ` Magnus Olsson
@ 2018-07-01 12:34 ` Kevin Darbyshire-Bryant
0 siblings, 0 replies; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-01 12:34 UTC (permalink / raw)
To: Magnus Olsson; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 850 bytes --]
> On 1 Jul 2018, at 10:46, Magnus Olsson <mbo2@uggenabben.se> wrote:
>
> Hi,
>
> Have done some testing. Using the following patches from Kevin's cakepatches tree:
>
> iproute2: RFC update cake support
> kmod-sched-cake: RFC latest cake
>
> 'tc -s -d qdisc show' does not give tin status on:
>
> ramips: MediaTek MT7621 (MIPS 1004Kc) / UBNT-ERX-SFP
> ath79: Qualcomm Atheros QCA956X (MIPS 74Kc) / Ubiquiti UniFi-AC-LITE
> ath79: Atheros AR7161 (MIPS 24Kc) / Netgear WNDR3800
>
> Works on:
>
> mpc85xx: Freescale P1014 (e500v2) / TP-Link TL-WDR4900 v1
>
> Also made another round including the cake 'u32 hacks' patch, and that solves for all above.
Thankyou Gentlemen for your continuing efforts, much appreciated. Please keep going :-)
So far it looks like it’s MIPS specific, but more test cases needed.
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 7:18 ` [Cake] " Pete Heist
@ 2018-07-01 13:48 ` Pete Heist
2018-07-01 14:02 ` Dave Taht
` (2 more replies)
0 siblings, 3 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-01 13:48 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Georgios Amanakis, mbo2, Cake List
Ok, my reboots were because I need to compile with om-watchdog support, otherwise the OM2P’s hardware watchdog reboots every few minutes. Thus, my sysupgrades were not _actually_ working (watchdog was interrupting them), thus the unknown symbols / dependency problems, etc, so that’s behind me. :/
And yes, I get no tin stats without the patch. With the patch, I get tin stats as expected. This is probably nothing new.
I also compiled on a Raspi 2 (32-bit ARM) running Debian 9.4, and it works as expected without the patch. This probably just confirms what George reported, because I think his wrt1900acs also uses 32-bit ARM.
So yes, MIPS 32-bit, both le and be are both fails. What next? We could test more archs, but it might be more productive to attempt to figure it out in the code when I can at least repro it on MIPS.
You mentioned earlier netlink_parse_nested doesn’t seem to handle 64-bit values but I’m not seeing where that is. I’m getting my head around how stats work so just writing out loud…looks like there are qstats from struct Qdisc then the tin stats which are probably custom for Cake and stored at some address obtained from qdisc_priv, which is right after the regular Qdisc struct. Interesting. I’m searching now for how this data ends up in tc’s hands, which looks like happens in cake_print_xstats with parse_rtattr_nested, then the printing gets underway, and at some point if st[TCA_CAKE_STATS_TIN_STATS] is true it gets into the tin printing business. What’s the condition that fails to make the tin stats not be printed? Best I know is to add debugging statements to figure it out, which I might give a try later today. Am I on the right track?
> On Jul 1, 2018, at 9:18 AM, Pete Heist <pete@heistp.net> wrote:
>
> Ok, so I’ve got to figure out what’s up with my build in general, so I’m blowing away my source tree and will build master without any patches.
>
> I don’t know if the kernel panics and unknown symbols are related somehow, but they’re not helping(!) so I’ll try to get past that. I don’t actually get logs in /sys/kernel/debug/crashlog, so I can’t even prove it’s a panic, but it’s rebooting that usually happens just as I do something net related, like start typing in ssh or do an scp, after some period of inactivity. If it’s reproducible on an unpatched build I’ll head over to the OpenWRT universe.
>
> (the stats alignment is just proportional vs fixed fonts- they look fine in vim)
>
>> On Jul 1, 2018, at 4:37 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
>>
>> The stats got mangled in by the email, in the terminal they are
>> properly aligned.
>>
>>
>> ---------- Forwarded message ----------
>> From: Georgios Amanakis <gamanakis@gmail.com>
>> Date: Sat, Jun 30, 2018 at 10:35 PM
>> Subject: Re: [Cake] Cake on openwrt - falling behind
>> To: Pete Heist <pete@heistp.net>
>>
>>
>> I just tested this on my wrt1900acs but it is behaving as it should. I
>> used Kevin's RFC patches for iproute2 and kmod-sched-cake.
>>
>> root@lede:/tmp# tc -s qdisc show dev eth0
>> qdisc cake 8004: root refcnt 9 bandwidth 2500Kbit diffserv3
>> triple-isolate split-gso rtt 100.0ms raw overhead 0
>> Sent 122745 bytes 691 pkt (dropped 0, overlimits 419 requeues 0)
>> backlog 0b 0p requeues 0
>> memory used: 6416b of 4Mb
>> capacity estimate: 2500Kbit
>> min/max network layer size: 42 / 1186
>> min/max overhead-adjusted size: 42 / 1186
>> average network hdr offset: 13
>>
>> Bulk Best Effort Voice
>> thresh 156248bit 2500Kbit 625Kbit
>> target 116.3ms 7.3ms 29.1ms
>> interval 232.6ms 102.3ms 124.1ms
>> pk_delay 0us 196us 8.6ms
>> av_delay 0us 13us 1.3ms
>> sp_delay 0us 2us 38us
>> backlog 0b 0b 0b
>> pkts 0 414 277
>> bytes 0 38367 84378
>> way_inds 0 0 0
>> way_miss 0 20 2
>> way_cols 0 0 0
>> drops 0 0 0
>> marks 0 0 0
>> ack_drop 0 0 0
>> sp_flows 0 1 0
>> bk_flows 0 0 1
>> un_flows 0 0 0
>> max_len 0 206 1186
>> quantum 300 300 300
>>
>> root@lede:/tmp# opkg list-installed | grep cake
>> kmod-sched-cake - 4.14.50+2018-06-26-0520a6cb-1
>>
>> root@lede:/tmp# uname -a
>> Linux lede 4.14.50 #0 SMP Wed Jun 27 21:48:32 2018 armv7l GNU/Linux
>>
>> George
>>
>> On Sat, Jun 30, 2018 at 7:20 PM, Pete Heist <pete@heistp.net> wrote:
>>> Ok, same result (unknown symbols and routine crashing) after a make clean and re-build, but I just force installed the kmod-sched-cake package with --force-depends and it seems to work. This reproduces it, right? no stats and the debug output?
>>>
>>> root@LEDE:~# tc qdisc add dev eth0 root cake
>>> root@LEDE:~# tc -s -d qdisc show dev eth0
>>> qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
>>> tca_stats 2005579564 tca_stats2 0 tca_xstats 0
>>> calling print_tcstats_attr()
>>> print_tcstats_attr()
>>> got stats
>>> Sent 9070 bytes 66 pkts (dropped 0, overlimits 0)
>>> got xstats 0 tca_stats 2005579564 tca_stats2 0 tca_xstats 0
>>>
>>> root@LEDE:~# cat /proc/cpuinfo
>>> system type : Atheros AR7240 rev 2
>>> machine : OpenMesh OM2P
>>> processor : 0
>>> cpu model : MIPS 24Kc V7.4
>>> BogoMIPS : 265.42
>>> wait instruction : yes
>>> microsecond timers : yes
>>> tlb_entries : 16
>>> extra interrupt vector : yes
>>> hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
>>> isa : mips1 mips2 mips32r1 mips32r2
>>> ASEs implemented : mips16
>>> shadow register sets : 1
>>> kscratch registers : 0
>>> package : 0
>>> core : 0
>>> VCED exceptions : not available
>>> VCEI exceptions : not available
>>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 13:48 ` Pete Heist
@ 2018-07-01 14:02 ` Dave Taht
2018-07-01 15:30 ` Pete Heist
2018-07-01 14:38 ` Kevin Darbyshire-Bryant
[not found] ` <mailman.392.1530455913.3512.cake@lists.bufferbloat.net>
2 siblings, 1 reply; 77+ messages in thread
From: Dave Taht @ 2018-07-01 14:02 UTC (permalink / raw)
To: Pete Heist; +Cc: Kevin Darbyshire-Bryant, Cake List
I'm in no position to help (on the road). But please test also cake's
4 and 8 tin modes on arm and mips. Does cake besteffort work on mips?
On Sun, Jul 1, 2018 at 9:49 AM Pete Heist <pete@heistp.net> wrote:
>
> Ok, my reboots were because I need to compile with om-watchdog support, otherwise the OM2P’s hardware watchdog reboots every few minutes. Thus, my sysupgrades were not _actually_ working (watchdog was interrupting them), thus the unknown symbols / dependency problems, etc, so that’s behind me. :/
>
> And yes, I get no tin stats without the patch. With the patch, I get tin stats as expected. This is probably nothing new.
>
> I also compiled on a Raspi 2 (32-bit ARM) running Debian 9.4, and it works as expected without the patch. This probably just confirms what George reported, because I think his wrt1900acs also uses 32-bit ARM.
>
> So yes, MIPS 32-bit, both le and be are both fails. What next? We could test more archs, but it might be more productive to attempt to figure it out in the code when I can at least repro it on MIPS.
>
> You mentioned earlier netlink_parse_nested doesn’t seem to handle 64-bit values but I’m not seeing where that is. I’m getting my head around how stats work so just writing out loud…looks like there are qstats from struct Qdisc then the tin stats which are probably custom for Cake and stored at some address obtained from qdisc_priv, which is right after the regular Qdisc struct. Interesting. I’m searching now for how this data ends up in tc’s hands, which looks like happens in cake_print_xstats with parse_rtattr_nested, then the printing gets underway, and at some point if st[TCA_CAKE_STATS_TIN_STATS] is true it gets into the tin printing business. What’s the condition that fails to make the tin stats not be printed? Best I know is to add debugging statements to figure it out, which I might give a try later today. Am I on the right track?
>
> > On Jul 1, 2018, at 9:18 AM, Pete Heist <pete@heistp.net> wrote:
> >
> > Ok, so I’ve got to figure out what’s up with my build in general, so I’m blowing away my source tree and will build master without any patches.
> >
> > I don’t know if the kernel panics and unknown symbols are related somehow, but they’re not helping(!) so I’ll try to get past that. I don’t actually get logs in /sys/kernel/debug/crashlog, so I can’t even prove it’s a panic, but it’s rebooting that usually happens just as I do something net related, like start typing in ssh or do an scp, after some period of inactivity. If it’s reproducible on an unpatched build I’ll head over to the OpenWRT universe.
> >
> > (the stats alignment is just proportional vs fixed fonts- they look fine in vim)
> >
> >> On Jul 1, 2018, at 4:37 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
> >>
> >> The stats got mangled in by the email, in the terminal they are
> >> properly aligned.
> >>
> >>
> >> ---------- Forwarded message ----------
> >> From: Georgios Amanakis <gamanakis@gmail.com>
> >> Date: Sat, Jun 30, 2018 at 10:35 PM
> >> Subject: Re: [Cake] Cake on openwrt - falling behind
> >> To: Pete Heist <pete@heistp.net>
> >>
> >>
> >> I just tested this on my wrt1900acs but it is behaving as it should. I
> >> used Kevin's RFC patches for iproute2 and kmod-sched-cake.
> >>
> >> root@lede:/tmp# tc -s qdisc show dev eth0
> >> qdisc cake 8004: root refcnt 9 bandwidth 2500Kbit diffserv3
> >> triple-isolate split-gso rtt 100.0ms raw overhead 0
> >> Sent 122745 bytes 691 pkt (dropped 0, overlimits 419 requeues 0)
> >> backlog 0b 0p requeues 0
> >> memory used: 6416b of 4Mb
> >> capacity estimate: 2500Kbit
> >> min/max network layer size: 42 / 1186
> >> min/max overhead-adjusted size: 42 / 1186
> >> average network hdr offset: 13
> >>
> >> Bulk Best Effort Voice
> >> thresh 156248bit 2500Kbit 625Kbit
> >> target 116.3ms 7.3ms 29.1ms
> >> interval 232.6ms 102.3ms 124.1ms
> >> pk_delay 0us 196us 8.6ms
> >> av_delay 0us 13us 1.3ms
> >> sp_delay 0us 2us 38us
> >> backlog 0b 0b 0b
> >> pkts 0 414 277
> >> bytes 0 38367 84378
> >> way_inds 0 0 0
> >> way_miss 0 20 2
> >> way_cols 0 0 0
> >> drops 0 0 0
> >> marks 0 0 0
> >> ack_drop 0 0 0
> >> sp_flows 0 1 0
> >> bk_flows 0 0 1
> >> un_flows 0 0 0
> >> max_len 0 206 1186
> >> quantum 300 300 300
> >>
> >> root@lede:/tmp# opkg list-installed | grep cake
> >> kmod-sched-cake - 4.14.50+2018-06-26-0520a6cb-1
> >>
> >> root@lede:/tmp# uname -a
> >> Linux lede 4.14.50 #0 SMP Wed Jun 27 21:48:32 2018 armv7l GNU/Linux
> >>
> >> George
> >>
> >> On Sat, Jun 30, 2018 at 7:20 PM, Pete Heist <pete@heistp.net> wrote:
> >>> Ok, same result (unknown symbols and routine crashing) after a make clean and re-build, but I just force installed the kmod-sched-cake package with --force-depends and it seems to work. This reproduces it, right? no stats and the debug output?
> >>>
> >>> root@LEDE:~# tc qdisc add dev eth0 root cake
> >>> root@LEDE:~# tc -s -d qdisc show dev eth0
> >>> qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
> >>> tca_stats 2005579564 tca_stats2 0 tca_xstats 0
> >>> calling print_tcstats_attr()
> >>> print_tcstats_attr()
> >>> got stats
> >>> Sent 9070 bytes 66 pkts (dropped 0, overlimits 0)
> >>> got xstats 0 tca_stats 2005579564 tca_stats2 0 tca_xstats 0
> >>>
> >>> root@LEDE:~# cat /proc/cpuinfo
> >>> system type : Atheros AR7240 rev 2
> >>> machine : OpenMesh OM2P
> >>> processor : 0
> >>> cpu model : MIPS 24Kc V7.4
> >>> BogoMIPS : 265.42
> >>> wait instruction : yes
> >>> microsecond timers : yes
> >>> tlb_entries : 16
> >>> extra interrupt vector : yes
> >>> hardware watchpoint : yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
> >>> isa : mips1 mips2 mips32r1 mips32r2
> >>> ASEs implemented : mips16
> >>> shadow register sets : 1
> >>> kscratch registers : 0
> >>> package : 0
> >>> core : 0
> >>> VCED exceptions : not available
> >>> VCEI exceptions : not available
> >>>
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
> >
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 13:48 ` Pete Heist
2018-07-01 14:02 ` Dave Taht
@ 2018-07-01 14:38 ` Kevin Darbyshire-Bryant
2018-07-01 16:54 ` Pete Heist
[not found] ` <mailman.392.1530455913.3512.cake@lists.bufferbloat.net>
2 siblings, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-01 14:38 UTC (permalink / raw)
To: Pete Heist; +Cc: Georgios Amanakis, mbo2, Cake List
[-- Attachment #1: Type: text/plain, Size: 3884 bytes --]
> On 1 Jul 2018, at 14:48, Pete Heist <pete@heistp.net> wrote:
>
> Ok, my reboots were because I need to compile with om-watchdog support, otherwise the OM2P’s hardware watchdog reboots every few minutes. Thus, my sysupgrades were not _actually_ working (watchdog was interrupting them), thus the unknown symbols / dependency problems, etc, so that’s behind me. :/
>
> And yes, I get no tin stats without the patch. With the patch, I get tin stats as expected. This is probably nothing new.
>
> I also compiled on a Raspi 2 (32-bit ARM) running Debian 9.4, and it works as expected without the patch. This probably just confirms what George reported, because I think his wrt1900acs also uses 32-bit ARM.
>
> So yes, MIPS 32-bit, both le and be are both fails. What next? We could test more archs, but it might be more productive to attempt to figure it out in the code when I can at least repro it on MIPS.
>
> You mentioned earlier netlink_parse_nested doesn’t seem to handle 64-bit values but I’m not seeing where that is. I’m getting my head around how stats work so just writing out loud…looks like there are qstats from struct Qdisc then the tin stats which are probably custom for Cake and stored at some address obtained from qdisc_priv, which is right after the regular Qdisc struct. Interesting. I’m searching now for how this data ends up in tc’s hands, which looks like happens in cake_print_xstats with parse_rtattr_nested, then the printing gets underway, and at some point if st[TCA_CAKE_STATS_TIN_STATS] is true it gets into the tin printing business. What’s the condition that fails to make the tin stats not be printed? Best I know is to add debugging statements to figure it out, which I might give a try later today. Am I on the right track?
I think we’ve all been heading in a similar direction that’s for sure. The symptoms are that netlink doesn’t appear to handle 64bit values on MIPS well because generally as soon as you put a 64 bit value in the netlink stream, things go a bit wonky.
I think this is some sort of alignment problem. e.g. If I get cake kmod to insert a dummy 32bit pad netlink attribute at the top of ‘cake_dump’, then I see the tin stats magically appear, although some of the values ‘capacity’ ‘threshold’ ‘sent bytes’ are stupidly high, as if an endian issue has hit.
In openwrt world, mips is not defined as an architecture that can do ‘efficient unaligned access’, thus if you look in include/net/netlink.h you’ll see
static inline bool nla_need_padding_for_64bit(struct sk_buff *skb)
{
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
/* The nlattr header is 4 bytes in size, that's why we test
* if the skb->data _is_ aligned. A NOP attribute, plus
* nlattr header for next attribute, will make nla_data()
* 8-byte aligned.
*/
if (IS_ALIGNED((unsigned long)skb_tail_pointer(skb), 8))
return true;
#endif
return false;
}
which looks to be designed to align netlink attributes in a friendly way.
I haven’t looked yet to see if working archs define ‘CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS’ and non working archs don’t….but to me it’s an obvious place to start poking, as we clearly have some/most archs wher the iproute/tc & cake code talk well….and one arch where they don’t…. to me that sort of rules out tc/cake doing anything silly.
There’s a ‘kmon-nlmon’ that allows you to create a virtual network interface for the netlink packets and thus capture (tcpdump) and export to wireshark for dissection. I had a look at that a few weeks ago but my head exploded and have only just recovered enough to start looking at this problem again…. the EFFECIENT_UNALIGNED_ACCESS thing is something I discovered about 2 hours ago.
Kevin
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
[not found] ` <mailman.392.1530455913.3512.cake@lists.bufferbloat.net>
@ 2018-07-01 15:17 ` Jonathan Morton
0 siblings, 0 replies; 77+ messages in thread
From: Jonathan Morton @ 2018-07-01 15:17 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Pete Heist, Cake List
> On 1 Jul, 2018, at 5:38 pm, Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net> wrote:
>
> In openwrt world, mips is not defined as an architecture that can do ‘efficient unaligned access’, thus if you look in include/net/netlink.h you’ll see
That shouldn't cause any special problems for a 64-bit value on a 32-bit architecture, because the CPU treats it as a pair of 32-bit values which only need to be 4-byte aligned.
- Jonathan Morton
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 14:02 ` Dave Taht
@ 2018-07-01 15:30 ` Pete Heist
0 siblings, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-01 15:30 UTC (permalink / raw)
To: Dave Taht; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jul 1, 2018, at 4:02 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> I'm in no position to help (on the road). But please test also cake's
> 4 and 8 tin modes on arm and mips. Does cake besteffort work on mips?
On mips, it does not (none of the diffserv keywords change that), on arm/Debian it does…
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 14:38 ` Kevin Darbyshire-Bryant
@ 2018-07-01 16:54 ` Pete Heist
2018-07-01 19:41 ` Kevin Darbyshire-Bryant
[not found] ` <mailman.397.1530474091.3512.cake@lists.bufferbloat.net>
0 siblings, 2 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-01 16:54 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]
> On Jul 1, 2018, at 4:38 PM, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
> I haven’t looked yet to see if working archs define ‘CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS’ and non working archs don’t….but to me it’s an obvious place to start poking, as we clearly have some/most archs wher the iproute/tc & cake code talk well….and one arch where they don’t…. to me that sort of rules out tc/cake doing anything silly.
I tried a hail mary and compiled with CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y, but didn’t see a change. Don’t take that for 100% because CONFIG_IKCONFIG_PROC wasn’t set when I went so I can’t prove it’s actually set. Anyway, I’ll probably get back to narrowing down a few basics so I understand it better.
> There’s a ‘kmon-nlmon’ that allows you to create a virtual network interface for the netlink packets and thus capture (tcpdump) and export to wireshark for dissection. I had a look at that a few weeks ago but my head exploded and have only just recovered enough to start looking at this problem again….
Cool, I might eventually get to that point. My head also approached some thermal limits last night with this unfamiliar build system but it’s getting a little better… :)
[-- Attachment #2: Type: text/html, Size: 2903 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 16:54 ` Pete Heist
@ 2018-07-01 19:41 ` Kevin Darbyshire-Bryant
2018-07-02 10:19 ` Pete Heist
[not found] ` <mailman.397.1530474091.3512.cake@lists.bufferbloat.net>
1 sibling, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-01 19:41 UTC (permalink / raw)
To: Pete Heist; +Cc: Cake List
[-- Attachment #1.1: Type: text/plain, Size: 654 bytes --]
> On 1 Jul 2018, at 17:54, Pete Heist <pete@heistp.net> wrote:
>
>
>
>> There’s a ‘kmon-nlmon’ that allows you to create a virtual network interface for the netlink packets and thus capture (tcpdump) and export to wireshark for dissection. I had a look at that a few weeks ago but my head exploded and have only just recovered enough to start looking at this problem again….
>
> Cool, I might eventually get to that point. My head also approached some thermal limits last night with this unfamiliar build system but it’s getting a little better… :)
For those who fancy dissecting some netlink packets, see attached :-)
[-- Attachment #1.2: cakenetlinkbroken --]
[-- Type: application/octet-stream, Size: 25895 bytes --]
No. Time Source Destination Protocol Length Info
1 14:30:13.244 Netlink route 3784
Frame 1: 3784 bytes on wire (30272 bits), 3784 bytes captured (30272 bits)
Encapsulation type: Linux Netlink (158)
Arrival Time: Jun 9, 2018 14:30:13.244467000 BST
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1528551013.244467000 seconds
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 3784 bytes (30272 bits)
Capture Length: 3784 bytes (30272 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: netlink:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route]
Linux netlink (cooked header)
Link-layer address type: Netlink (824)
Family: Route (0x0000)
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 888
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 320
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 888
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551015
Port ID: 8644
0000 00 04 03 38 00 00 00 00 00 00 00 00 00 00 00 00 ...8............
0010 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
0020 00 00 00 00 00 00 00 01 00 00 00 00 ff ff ff ff ................
0030 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0040 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00a0 00 00 00 00 00 00 00 00 00 00 03 78 00 24 00 02 ...........x.$..
00b0 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 02 [..g..!.........
00c0 80 0a 00 00 ff ff ff ff 00 00 00 02 00 09 00 01 ................
00d0 63 61 6b 65 00 00 00 00 00 80 00 02 00 0c 00 02 cake............
00e0 00 00 00 00 00 25 f4 cc 00 08 00 03 00 00 00 00 .....%..........
00f0 00 08 00 04 00 00 00 02 00 08 00 05 00 00 00 05 ................
0100 00 08 00 0b 00 00 00 00 00 08 00 11 00 00 00 01 ................
0110 00 08 00 0d 00 00 00 00 00 08 00 06 00 00 00 1a ................
0120 00 08 00 0e 00 00 00 00 00 08 00 07 00 01 86 a0 ................
0130 00 08 00 08 00 00 13 88 00 08 00 09 00 00 00 00 ................
0140 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 00 ................
0150 00 08 00 0a 00 00 00 00 02 9c 00 09 00 04 00 07 ................
0160 02 60 00 04 00 0c 00 02 00 00 00 00 00 25 f4 cc .`...........%..
0170 00 08 00 03 00 40 00 00 00 08 00 04 00 00 c9 80 .....@..........
0180 00 08 00 05 00 00 00 0e 00 08 00 07 00 00 05 dc ................
0190 00 08 00 09 00 00 06 0e 00 08 00 06 00 00 00 1c ................
01a0 00 08 00 08 00 00 00 37 02 18 00 0a 00 b4 00 01 .......7........
01b0 00 04 00 01 00 0c 00 0c 00 00 00 00 00 02 5f 4c .............._L
01c0 00 04 00 01 00 0c 00 03 00 00 00 00 00 00 00 00 ................
01d0 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 39 0f ..............9.
01e0 00 08 00 0e 00 01 ac 27 00 08 00 02 00 00 00 00 .......'........
01f0 00 08 00 04 00 00 00 00 00 08 00 08 00 00 00 00 ................
0200 00 08 00 06 00 00 00 00 00 08 00 12 00 00 00 00 ................
0210 00 08 00 13 00 00 00 00 00 08 00 14 00 00 00 00 ................
0220 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 00 ................
0230 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 00 ................
0240 00 08 00 16 00 00 00 00 00 08 00 17 00 00 00 00 ................
0250 00 08 00 18 00 00 00 00 00 08 00 19 00 00 01 2c ...............,
0260 00 b0 00 02 00 0c 00 0c 00 00 00 00 00 25 f4 cc .............%..
0270 00 04 00 01 00 0c 00 03 00 00 00 00 00 33 ea 39 .............3.9
0280 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0290 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 4d 97 ..............M.
02a0 00 08 00 04 00 00 00 05 00 08 00 08 00 00 00 00 ................
02b0 00 08 00 06 00 00 00 00 00 08 00 12 00 00 02 53 ...............S
02c0 00 08 00 13 00 00 00 4d 00 08 00 14 00 00 00 05 .......M........
02d0 00 08 00 0f 00 00 02 86 00 08 00 10 00 00 06 d1 ................
02e0 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 01 ................
02f0 00 08 00 16 00 00 00 01 00 08 00 17 00 00 00 00 ................
0300 00 08 00 18 00 00 0b d4 00 08 00 19 00 00 02 5f ..............._
0310 00 b0 00 03 00 0c 00 0c 00 00 00 00 00 09 7d 33 ..............}3
0320 00 04 00 01 00 0c 00 03 00 00 00 00 00 00 18 b3 ................
0330 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0340 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 00 48 ...............H
0350 00 08 00 04 00 00 00 00 00 08 00 08 00 00 00 00 ................
0360 00 08 00 06 00 00 00 00 00 08 00 12 00 00 00 12 ................
0370 00 08 00 13 00 00 00 02 00 08 00 14 00 00 00 02 ................
0380 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 35 ...............5
0390 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 00 ................
03a0 00 08 00 16 00 00 00 00 00 08 00 17 00 00 00 00 ................
03b0 00 08 00 18 00 00 01 67 00 08 00 19 00 00 01 2c .......g.......,
03c0 00 04 00 06 00 14 00 01 00 00 00 00 00 33 ec 9d .............3..
03d0 00 00 4d da 00 00 00 00 00 04 00 06 00 18 00 03 ..M.............
03e0 00 00 00 00 00 00 00 00 00 00 00 05 00 00 00 00 ................
03f0 00 00 12 25 00 2c 00 03 00 00 00 00 00 33 ec 9d ...%.,.......3..
0400 00 00 4d da 00 00 00 05 00 00 12 25 00 00 00 00 ..M........%....
0410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0420 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
0430 00 00 00 00 00 00 00 02 ff ff 00 00 ff ff ff f1 ................
0440 00 00 00 01 00 0c 00 01 69 6e 67 72 65 73 73 00 ........ingress.
0450 00 04 00 02 00 38 00 07 00 04 00 06 00 14 00 01 .....8..........
0460 00 00 00 00 00 fb 02 13 00 00 51 42 00 00 00 00 ..........QB....
0470 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0480 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0490 00 00 00 00 00 fb 02 13 00 00 51 42 00 00 00 00 ..........QB....
04a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
04b0 00 00 00 00 00 00 00 00 00 00 01 40 00 24 00 02 ...........@.$..
04c0 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 03 [..g..!.........
04d0 00 00 00 00 ff ff ff ff 00 00 00 02 00 0d 00 01 ................
04e0 66 71 5f 63 6f 64 65 6c 00 00 00 00 00 44 00 02 fq_codel.....D..
04f0 00 08 00 01 00 00 13 87 00 08 00 02 00 00 28 00 ..............(.
0500 00 08 00 03 00 01 86 9f 00 08 00 04 00 00 00 01 ................
0510 00 08 00 06 00 00 05 ea 00 08 00 08 00 00 00 40 ...............@
0520 00 08 00 09 00 40 00 00 00 08 00 05 00 00 04 00 .....@..........
0530 00 6c 00 09 00 04 00 07 00 04 00 06 00 2c 00 04 .l...........,..
0540 00 00 00 00 00 00 02 2b 00 00 00 00 00 00 00 00 .......+........
0550 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 ................
0560 00 00 00 00 00 00 00 00 00 04 00 06 00 14 00 01 ................
0570 00 00 00 00 01 cd c2 5a 00 00 b3 53 00 00 00 00 .......Z...S....
0580 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0590 00 00 00 00 00 00 00 04 00 00 00 00 00 2c 00 03 .............,..
05a0 00 00 00 00 01 cd c2 5a 00 00 b3 53 00 00 00 00 .......Z...S....
05b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
05c0 00 00 00 00 00 00 00 00 00 04 00 09 00 2c 00 04 .............,..
05d0 00 00 00 00 00 00 02 2b 00 00 00 00 00 00 00 00 .......+........
05e0 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00 00 ................
05f0 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0600 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 09 [..g..!.........
0610 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0620 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0630 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
0640 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
0650 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0660 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0670 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0680 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0690 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
06a0 00 00 00 00 00 00 00 0a 00 00 00 00 ff ff ff ff ................
06b0 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
06c0 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
06d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
06e0 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
06f0 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0700 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0710 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0720 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0730 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 0c [..g..!.........
0740 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0750 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0760 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
0770 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
0780 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0790 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
07a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
07b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
07c0 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
07d0 00 00 00 00 00 00 00 0d 00 00 00 00 ff ff ff ff ................
07e0 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
07f0 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0810 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0820 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0830 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0840 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0850 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0860 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 12 [..g..!.........
0870 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0880 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0890 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
08a0 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
08b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
08c0 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
08d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
08e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
08f0 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
0900 00 00 00 00 00 00 00 16 00 00 00 00 ff ff ff ff ................
0910 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0920 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0930 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0940 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0950 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0960 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0970 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0980 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0990 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 17 [..g..!.........
09a0 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
09b0 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
09c0 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
09d0 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
09e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
09f0 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0a00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a20 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
0a30 00 00 00 00 00 00 00 18 00 00 00 00 ff ff ff ff ................
0a40 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0a50 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0a60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a70 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0a80 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0a90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0ab0 00 00 00 00 00 00 00 00 00 00 03 78 00 24 00 02 ...........x.$..
0ac0 5b 1b d6 67 00 00 21 c4 00 00 00 00 00 00 00 1d [..g..!.........
0ad0 80 0b 00 00 ff ff ff ff 00 00 00 02 00 09 00 01 ................
0ae0 63 61 6b 65 00 00 00 00 00 80 00 02 00 0c 00 02 cake............
0af0 00 00 00 00 00 90 f5 60 00 08 00 03 00 00 00 00 .......`........
0b00 00 08 00 04 00 00 00 02 00 08 00 05 00 00 00 06 ................
0b10 00 08 00 0b 00 00 00 00 00 08 00 11 00 00 00 01 ................
0b20 00 08 00 0d 00 00 00 00 00 08 00 06 00 00 00 1a ................
0b30 00 08 00 0e 00 00 00 00 00 08 00 07 00 01 86 a0 ................
0b40 00 08 00 08 00 00 13 88 00 08 00 09 00 00 00 00 ................
0b50 00 08 00 0f 00 00 00 01 00 08 00 10 00 00 00 00 ................
0b60 00 08 00 0a 00 00 00 00 02 9c 00 09 00 04 00 07 ................
0b70 02 60 00 04 00 0c 00 02 00 00 00 00 00 90 f5 60 .`.............`
0b80 00 08 00 03 00 40 00 00 00 08 00 04 00 00 f0 40 .....@.........@
0b90 00 08 00 05 00 00 00 0e 00 08 00 07 00 00 05 dc ................
0ba0 00 08 00 09 00 00 06 0e 00 08 00 06 00 00 00 2e ................
0bb0 00 08 00 08 00 00 00 4a 02 18 00 0a 00 b4 00 01 .......J........
0bc0 00 04 00 01 00 0c 00 0c 00 00 00 00 00 09 0f 56 ...............V
0bd0 00 04 00 01 00 0c 00 03 00 00 00 00 00 00 00 00 ................
0be0 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0bf0 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 00 00 ................
0c00 00 08 00 04 00 00 00 00 00 08 00 08 00 00 00 00 ................
0c10 00 08 00 06 00 00 00 00 00 08 00 12 00 00 00 00 ................
0c20 00 08 00 13 00 00 00 00 00 08 00 14 00 00 00 00 ................
0c30 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 00 ................
0c40 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 00 ................
0c50 00 08 00 16 00 00 00 00 00 08 00 17 00 00 00 00 ................
0c60 00 08 00 18 00 00 00 00 00 08 00 19 00 00 01 2c ...............,
0c70 00 b0 00 02 00 0c 00 0c 00 00 00 00 00 90 f5 60 ...............`
0c80 00 04 00 01 00 0c 00 03 00 00 00 00 00 ff 70 2b ..............p+
0c90 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0ca0 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 51 33 ..............Q3
0cb0 00 08 00 04 00 00 00 02 00 08 00 08 00 00 00 00 ................
0cc0 00 08 00 06 00 00 00 00 00 08 00 12 00 00 01 56 ...............V
0cd0 00 08 00 13 00 00 00 55 00 08 00 14 00 00 00 04 .......U........
0ce0 00 08 00 0f 00 00 02 e9 00 08 00 10 00 00 06 a3 ................
0cf0 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 01 ................
0d00 00 08 00 16 00 00 00 01 00 08 00 17 00 00 00 00 ................
0d10 00 08 00 18 00 00 05 ea 00 08 00 19 00 00 05 ea ................
0d20 00 b0 00 03 00 0c 00 0c 00 00 00 00 00 24 3d 58 .............$=X
0d30 00 04 00 01 00 0c 00 03 00 00 00 00 00 00 03 84 ................
0d40 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0d50 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 00 0f ................
0d60 00 08 00 04 00 00 00 00 00 08 00 08 00 00 00 00 ................
0d70 00 08 00 06 00 00 00 00 00 08 00 12 00 00 00 09 ................
0d80 00 08 00 13 00 00 00 00 00 08 00 14 00 00 00 00 ................
0d90 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 01 ................
0da0 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 00 ................
0db0 00 08 00 16 00 00 00 00 00 08 00 17 00 00 00 00 ................
0dc0 00 08 00 18 00 00 00 3c 00 08 00 19 00 00 02 43 .......<.......C
0dd0 00 04 00 06 00 14 00 01 00 00 00 00 00 ff 67 db ..............g.
0de0 00 00 51 40 00 00 00 00 00 04 00 06 00 18 00 03 ..Q@............
0df0 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 ................
0e00 00 00 50 15 00 2c 00 03 00 00 00 00 00 ff 67 db ..P..,........g.
0e10 00 00 51 40 00 00 00 02 00 00 50 15 00 00 00 00 ..Q@......P.....
0e20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0e30 00 00 00 98 00 24 00 02 5b 1b d6 67 00 00 21 c4 .....$..[..g..!.
0e40 00 00 00 00 00 00 00 1f 00 00 00 00 ff ff ff ff ................
0e50 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0e60 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0e70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0e80 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0e90 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0ea0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0eb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0ec0 00 00 00 00 00 00 00 00 ........
[-- Attachment #1.3: cakenetlinkbroken.pcap --]
[-- Type: application/octet-stream, Size: 3824 bytes --]
[-- Attachment #1.4: cakenetlinkworking.pcap --]
[-- Type: application/octet-stream, Size: 3864 bytes --]
[-- Attachment #1.5: cakenetlinkworking --]
[-- Type: application/octet-stream, Size: 26611 bytes --]
No. Time Source Destination Protocol Length Info
1 14:41:34.832 Netlink route 3824
Frame 1: 3824 bytes on wire (30592 bits), 3824 bytes captured (30592 bits)
Encapsulation type: Linux Netlink (158)
Arrival Time: Jun 9, 2018 14:41:34.832063000 BST
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1528551694.832063000 seconds
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 3824 bytes (30592 bits)
Capture Length: 3824 bytes (30592 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: netlink:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route:netlink-route]
Linux netlink (cooked header)
Link-layer address type: Netlink (824)
Family: Route (0x0000)
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 832
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 320
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 152
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
Linux rtnetlink (route netlink) protocol
Netlink message header (type: Add queueing discipline)
Length: 832
Message type: Add queueing discipline (36)
Flags: 0x0002
.... .... .... ...0 = Request: 0
.... .... .... ..1. = Multipart message: 1
.... .... .... .0.. = Ack: 0
.... .... .... 0... = Echo: 0
.... .... ...0 .... = Dump inconsistent: 0
.... .... ..0. .... = Dump filtered: 0
Sequence: 1528551696
Port ID: 12057
0000 00 04 03 38 00 00 00 00 00 00 00 00 00 00 00 00 ...8............
0010 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
0020 00 00 00 00 00 00 00 01 00 00 00 00 ff ff ff ff ................
0030 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0040 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0070 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00a0 00 00 00 00 00 00 00 00 00 00 03 40 00 24 00 02 ...........@.$..
00b0 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 02 [...../.........
00c0 80 0d 00 00 ff ff ff ff 00 00 00 02 00 09 00 01 ................
00d0 63 61 6b 65 00 00 00 00 00 7c 00 02 00 08 00 02 cake.....|......
00e0 00 25 f4 cc 00 08 00 03 00 00 00 00 00 08 00 04 .%..............
00f0 00 00 00 02 00 08 00 05 00 00 00 05 00 08 00 0b ................
0100 00 00 00 00 00 08 00 11 00 00 00 01 00 08 00 0d ................
0110 00 00 00 00 00 08 00 06 00 00 00 1a 00 08 00 0e ................
0120 00 00 00 00 00 08 00 07 00 01 86 a0 00 08 00 08 ................
0130 00 00 13 88 00 08 00 09 00 00 00 00 00 08 00 0f ................
0140 00 00 00 00 00 08 00 10 00 00 00 00 00 08 00 0a ................
0150 00 00 00 00 02 68 00 07 02 34 00 04 00 08 00 02 .....h...4......
0160 00 25 f4 cc 00 08 00 03 00 40 00 00 00 08 00 04 .%.......@......
0170 00 00 17 40 00 08 00 05 00 00 00 0b 00 08 00 07 ...@............
0180 00 00 05 34 00 08 00 09 00 00 05 64 00 08 00 06 ...4.......d....
0190 00 00 00 1c 00 08 00 08 00 00 00 37 01 f0 00 0a ...........7....
01a0 00 a4 00 01 00 08 00 0c 00 02 5f 4c 00 08 00 03 .........._L....
01b0 00 00 00 00 00 08 00 0b 00 00 00 00 00 08 00 0d ................
01c0 00 00 39 0f 00 08 00 0e 00 01 ac 27 00 08 00 02 ..9........'....
01d0 00 00 00 00 00 08 00 04 00 00 00 00 00 08 00 08 ................
01e0 00 00 00 00 00 08 00 06 00 00 00 00 00 08 00 12 ................
01f0 00 00 00 00 00 08 00 13 00 00 00 00 00 08 00 14 ................
0200 00 00 00 00 00 08 00 0f 00 00 00 00 00 08 00 10 ................
0210 00 00 00 00 00 08 00 11 00 00 00 00 00 08 00 15 ................
0220 00 00 00 00 00 08 00 16 00 00 00 00 00 08 00 17 ................
0230 00 00 00 00 00 08 00 18 00 00 00 00 00 08 00 19 ................
0240 00 00 01 2c 00 a4 00 02 00 08 00 0c 00 25 f4 cc ...,.........%..
0250 00 08 00 03 00 00 b0 12 00 08 00 0b 00 00 00 00 ................
0260 00 08 00 0d 00 00 13 88 00 08 00 0e 00 01 86 a0 ................
0270 00 08 00 02 00 00 01 9b 00 08 00 04 00 00 00 00 ................
0280 00 08 00 08 00 00 00 00 00 08 00 06 00 00 00 00 ................
0290 00 08 00 12 00 00 00 cb 00 08 00 13 00 00 00 0c ................
02a0 00 08 00 14 00 00 00 05 00 08 00 0f 00 00 00 00 ................
02b0 00 08 00 10 00 00 00 7e 00 08 00 11 00 00 00 00 .......~........
02c0 00 08 00 15 00 00 00 01 00 08 00 16 00 00 00 01 ................
02d0 00 08 00 17 00 00 00 00 00 08 00 18 00 00 05 42 ...............B
02e0 00 08 00 19 00 00 02 5f 00 a4 00 03 00 08 00 0c ......._........
02f0 00 09 7d 33 00 08 00 03 00 00 00 ce 00 08 00 0b ..}3............
0300 00 00 00 00 00 08 00 0d 00 00 13 88 00 08 00 0e ................
0310 00 01 86 a0 00 08 00 02 00 00 00 03 00 08 00 04 ................
0320 00 00 00 00 00 08 00 08 00 00 00 00 00 08 00 06 ................
0330 00 00 00 00 00 08 00 12 00 00 00 05 00 08 00 13 ................
0340 00 00 00 00 00 08 00 14 00 00 00 00 00 08 00 0f ................
0350 00 00 00 00 00 08 00 10 00 00 00 03 00 08 00 11 ................
0360 00 00 00 00 00 08 00 15 00 00 00 00 00 08 00 16 ................
0370 00 00 00 00 00 08 00 17 00 00 00 00 00 08 00 18 ................
0380 00 00 00 5a 00 08 00 19 00 00 01 2c 00 14 00 01 ...Z.......,....
0390 00 00 00 00 00 00 b0 e0 00 00 01 9e 00 00 00 00 ................
03a0 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
03b0 00 00 00 00 00 00 00 00 00 00 00 24 00 2c 00 03 ...........$.,..
03c0 00 00 00 00 00 00 b0 e0 00 00 01 9e 00 00 00 00 ................
03d0 00 00 00 24 00 00 00 00 00 00 00 00 00 00 00 00 ...$............
03e0 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
03f0 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 02 [...../.........
0400 ff ff 00 00 ff ff ff f1 00 00 00 01 00 0c 00 01 ................
0410 69 6e 67 72 65 73 73 00 00 04 00 02 00 38 00 07 ingress......8..
0420 00 04 00 06 00 14 00 01 00 00 00 00 00 01 7d 14 ..............}.
0430 00 00 01 5f 00 00 00 00 00 04 00 06 00 18 00 03 ..._............
0440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0450 00 00 00 00 00 2c 00 03 00 00 00 00 00 01 7d 14 .....,........}.
0460 00 00 01 5f 00 00 00 00 00 00 00 00 00 00 00 00 ..._............
0470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0480 00 00 01 40 00 24 00 02 5b 1b d9 10 00 00 2f 19 ...@.$..[...../.
0490 00 00 00 00 00 00 00 03 00 00 00 00 ff ff ff ff ................
04a0 00 00 00 02 00 0d 00 01 66 71 5f 63 6f 64 65 6c ........fq_codel
04b0 00 00 00 00 00 44 00 02 00 08 00 01 00 00 13 87 .....D..........
04c0 00 08 00 02 00 00 28 00 00 08 00 03 00 01 86 9f ......(.........
04d0 00 08 00 04 00 00 00 01 00 08 00 06 00 00 05 ea ................
04e0 00 08 00 08 00 00 00 40 00 08 00 09 00 40 00 00 .......@.....@..
04f0 00 08 00 05 00 00 04 00 00 6c 00 09 00 04 00 07 .........l......
0500 00 04 00 06 00 2c 00 04 00 00 00 00 00 00 02 2b .....,.........+
0510 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 00 ................
0520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0530 00 04 00 06 00 14 00 01 00 00 00 00 02 d7 f2 25 ...............%
0540 00 01 49 42 00 00 00 00 00 04 00 06 00 18 00 03 ..IB............
0550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04 ................
0560 00 00 00 00 00 2c 00 03 00 00 00 00 02 d7 f2 25 .....,.........%
0570 00 01 49 42 00 00 00 00 00 00 00 00 00 00 00 00 ..IB............
0580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0590 00 04 00 09 00 2c 00 04 00 00 00 00 00 00 02 2b .....,.........+
05a0 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 00 ................
05b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
05c0 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
05d0 00 00 00 00 00 00 00 09 00 00 00 00 ff ff ff ff ................
05e0 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
05f0 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0600 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0610 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0620 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0630 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0640 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0650 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0660 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 0a [...../.........
0670 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0680 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0690 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
06a0 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
06b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
06c0 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
06d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
06e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
06f0 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
0700 00 00 00 00 00 00 00 0c 00 00 00 00 ff ff ff ff ................
0710 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0720 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0730 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0740 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0750 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0770 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0780 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0790 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 0d [...../.........
07a0 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
07b0 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
07c0 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
07d0 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
07e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
07f0 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0800 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0810 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0820 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
0830 00 00 00 00 00 00 00 12 00 00 00 00 ff ff ff ff ................
0840 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0850 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0860 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0870 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0880 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0890 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
08a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
08b0 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
08c0 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 16 [...../.........
08d0 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
08e0 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
08f0 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
0900 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
0910 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0920 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0930 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0940 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0950 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
0960 00 00 00 00 00 00 00 17 00 00 00 00 ff ff ff ff ................
0970 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0980 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0990 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
09a0 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
09b0 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
09c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
09d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
09e0 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
09f0 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 18 [...../.........
0a00 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0a10 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0a20 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
0a30 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
0a40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a50 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0a60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0a80 00 00 00 98 00 24 00 02 5b 1b d9 10 00 00 2f 19 .....$..[...../.
0a90 00 00 00 00 00 00 00 1f 00 00 00 00 ff ff ff ff ................
0aa0 00 00 00 02 00 0c 00 01 6e 6f 71 75 65 75 65 00 ........noqueue.
0ab0 00 3c 00 09 00 04 00 07 00 04 00 06 00 14 00 01 .<..............
0ac0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0ad0 00 04 00 06 00 18 00 03 00 00 00 00 00 00 00 00 ................
0ae0 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 03 .............,..
0af0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0b00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0b10 00 00 00 00 00 00 00 00 00 00 00 98 00 24 00 02 .............$..
0b20 5b 1b d9 10 00 00 2f 19 00 00 00 00 00 00 00 20 [...../........
0b30 00 00 00 00 ff ff ff ff 00 00 00 02 00 0c 00 01 ................
0b40 6e 6f 71 75 65 75 65 00 00 3c 00 09 00 04 00 07 noqueue..<......
0b50 00 04 00 06 00 14 00 01 00 00 00 00 00 00 00 00 ................
0b60 00 00 00 00 00 00 00 00 00 04 00 06 00 18 00 03 ................
0b70 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0b80 00 00 00 00 00 2c 00 03 00 00 00 00 00 00 00 00 .....,..........
0b90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0ba0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0bb0 00 00 03 40 00 24 00 02 5b 1b d9 10 00 00 2f 19 ...@.$..[...../.
0bc0 00 00 00 00 00 00 00 22 80 0e 00 00 ff ff ff ff ......."........
0bd0 00 00 00 02 00 09 00 01 63 61 6b 65 00 00 00 00 ........cake....
0be0 00 7c 00 02 00 08 00 02 00 90 f5 60 00 08 00 03 .|.........`....
0bf0 00 00 00 00 00 08 00 04 00 00 00 02 00 08 00 05 ................
0c00 00 00 00 06 00 08 00 0b 00 00 00 00 00 08 00 11 ................
0c10 00 00 00 01 00 08 00 0d 00 00 00 00 00 08 00 06 ................
0c20 00 00 00 1a 00 08 00 0e 00 00 00 00 00 08 00 07 ................
0c30 00 01 86 a0 00 08 00 08 00 00 13 88 00 08 00 09 ................
0c40 00 00 00 00 00 08 00 0f 00 00 00 01 00 08 00 10 ................
0c50 00 00 00 00 00 08 00 0a 00 00 00 00 02 68 00 07 .............h..
0c60 02 34 00 04 00 08 00 02 00 90 f5 60 00 08 00 03 .4.........`....
0c70 00 40 00 00 00 08 00 04 00 00 0f 80 00 08 00 05 .@..............
0c80 00 00 00 0a 00 08 00 07 00 00 05 dc 00 08 00 09 ................
0c90 00 00 06 0e 00 08 00 06 00 00 00 2e 00 08 00 08 ................
0ca0 00 00 00 4a 01 f0 00 0a 00 a4 00 01 00 08 00 0c ...J............
0cb0 00 09 0f 56 00 08 00 03 00 00 00 00 00 08 00 0b ...V............
0cc0 00 00 00 00 00 08 00 0d 00 00 13 88 00 08 00 0e ................
0cd0 00 01 86 a0 00 08 00 02 00 00 00 00 00 08 00 04 ................
0ce0 00 00 00 00 00 08 00 08 00 00 00 00 00 08 00 06 ................
0cf0 00 00 00 00 00 08 00 12 00 00 00 00 00 08 00 13 ................
0d00 00 00 00 00 00 08 00 14 00 00 00 00 00 08 00 0f ................
0d10 00 00 00 00 00 08 00 10 00 00 00 00 00 08 00 11 ................
0d20 00 00 00 00 00 08 00 15 00 00 00 00 00 08 00 16 ................
0d30 00 00 00 00 00 08 00 17 00 00 00 00 00 08 00 18 ................
0d40 00 00 00 00 00 08 00 19 00 00 01 2c 00 a4 00 02 ...........,....
0d50 00 08 00 0c 00 90 f5 60 00 08 00 03 00 01 90 0a .......`........
0d60 00 08 00 0b 00 00 00 00 00 08 00 0d 00 00 13 88 ................
0d70 00 08 00 0e 00 01 86 a0 00 08 00 02 00 00 01 5e ...............^
0d80 00 08 00 04 00 00 00 00 00 08 00 08 00 00 00 00 ................
0d90 00 08 00 06 00 00 00 00 00 08 00 12 00 00 00 4c ...............L
0da0 00 08 00 13 00 00 00 0b 00 08 00 14 00 00 00 06 ................
0db0 00 08 00 0f 00 00 00 00 00 08 00 10 00 00 00 81 ................
0dc0 00 08 00 11 00 00 00 00 00 08 00 15 00 00 00 03 ................
0dd0 00 08 00 16 00 00 00 01 00 08 00 17 00 00 00 00 ................
0de0 00 08 00 18 00 00 05 ea 00 08 00 19 00 00 05 ea ................
0df0 00 a4 00 03 00 08 00 0c 00 24 3d 58 00 08 00 03 .........$=X....
0e00 00 00 00 3c 00 08 00 0b 00 00 00 00 00 08 00 0d ...<............
0e10 00 00 13 88 00 08 00 0e 00 01 86 a0 00 08 00 02 ................
0e20 00 00 00 01 00 08 00 04 00 00 00 00 00 08 00 08 ................
0e30 00 00 00 00 00 08 00 06 00 00 00 00 00 08 00 12 ................
0e40 00 00 00 02 00 08 00 13 00 00 00 00 00 08 00 14 ................
0e50 00 00 00 00 00 08 00 0f 00 00 00 00 00 08 00 10 ................
0e60 00 00 00 01 00 08 00 11 00 00 00 00 00 08 00 15 ................
0e70 00 00 00 00 00 08 00 16 00 00 00 00 00 08 00 17 ................
0e80 00 00 00 00 00 08 00 18 00 00 00 3c 00 08 00 19 ...........<....
0e90 00 00 02 43 00 14 00 01 00 00 00 00 00 01 90 46 ...C...........F
0ea0 00 00 01 5f 00 00 00 00 00 04 00 06 00 18 00 03 ..._............
0eb0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0ec0 00 00 00 20 00 2c 00 03 00 00 00 00 00 01 90 46 ... .,.........F
0ed0 00 00 01 5f 00 00 00 00 00 00 00 20 00 00 00 00 ..._....... ....
0ee0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
[not found] ` <mailman.397.1530474091.3512.cake@lists.bufferbloat.net>
@ 2018-07-01 23:55 ` Dave Taht
2018-07-02 0:05 ` Dave Taht
0 siblings, 1 reply; 77+ messages in thread
From: Dave Taht @ 2018-07-01 23:55 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Pete Heist, Cake List
Well, I took a look at that cap. It does look like what is supplied is
truncated by quite a few bytes. I am under the impression that
openwrt/lede use a mini library for libmnl (what does tc link to?)
that may well be the source of this bug.
One thing that might help in peering at the raw data this way would be
to send const patterns
for each value like F0F0F0F0F0F0 alternating with E0E0E0E0E0... (full
pattern for 32 bit or 64 bit) to see what shows up.
On Sun, Jul 1, 2018 at 12:41 PM Kevin Darbyshire-Bryant via Cake
<cake@lists.bufferbloat.net> wrote:
>
>
>
>
> ---------- Forwarded message ----------
> From: Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
> To: Pete Heist <pete@heistp.net>
> Cc: Cake List <cake@lists.bufferbloat.net>
> Bcc:
> Date: Sun, 1 Jul 2018 19:41:28 +0000
> Subject: Re: [Cake] Cake on openwrt - falling behind
>
>
> > On 1 Jul 2018, at 17:54, Pete Heist <pete@heistp.net> wrote:
> >
> >
> >
> >> There’s a ‘kmon-nlmon’ that allows you to create a virtual network interface for the netlink packets and thus capture (tcpdump) and export to wireshark for dissection. I had a look at that a few weeks ago but my head exploded and have only just recovered enough to start looking at this problem again….
> >
> > Cool, I might eventually get to that point. My head also approached some thermal limits last night with this unfamiliar build system but it’s getting a little better… :)
>
> For those who fancy dissecting some netlink packets, see attached :-)
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net>
> To: Pete Heist <pete@heistp.net>
> Cc: Cake List <cake@lists.bufferbloat.net>
> Bcc:
> Date: Sun, 01 Jul 2018 12:41:33 -0700 (PDT)
> Subject: Re: [Cake] Cake on openwrt - falling behind
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 23:55 ` Dave Taht
@ 2018-07-02 0:05 ` Dave Taht
0 siblings, 0 replies; 77+ messages in thread
From: Dave Taht @ 2018-07-02 0:05 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Pete Heist, Cake List
On Sun, Jul 1, 2018 at 4:55 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> Well, I took a look at that cap. It does look like what is supplied is
> truncated by quite a few bytes. I am under the impression that
> openwrt/lede use a mini library for libmnl (what does tc link to?)
> that may well be the source of this bug.
Meh. I think you should ignore my first statement
>
> One thing that might help in peering at the raw data this way would be
> to send const patterns
> for each value like F0F0F0F0F0F0 alternating with E0E0E0E0E0... (full
> pattern for 32 bit or 64 bit) to see what shows up.
And pursue the second.
> On Sun, Jul 1, 2018 at 12:41 PM Kevin Darbyshire-Bryant via Cake
> <cake@lists.bufferbloat.net> wrote:
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
> > To: Pete Heist <pete@heistp.net>
> > Cc: Cake List <cake@lists.bufferbloat.net>
> > Bcc:
> > Date: Sun, 1 Jul 2018 19:41:28 +0000
> > Subject: Re: [Cake] Cake on openwrt - falling behind
> >
> >
> > > On 1 Jul 2018, at 17:54, Pete Heist <pete@heistp.net> wrote:
> > >
> > >
> > >
> > >> There’s a ‘kmon-nlmon’ that allows you to create a virtual network interface for the netlink packets and thus capture (tcpdump) and export to wireshark for dissection. I had a look at that a few weeks ago but my head exploded and have only just recovered enough to start looking at this problem again….
> > >
> > > Cool, I might eventually get to that point. My head also approached some thermal limits last night with this unfamiliar build system but it’s getting a little better… :)
> >
> > For those who fancy dissecting some netlink packets, see attached :-)
> >
> >
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net>
> > To: Pete Heist <pete@heistp.net>
> > Cc: Cake List <cake@lists.bufferbloat.net>
> > Bcc:
> > Date: Sun, 01 Jul 2018 12:41:33 -0700 (PDT)
> > Subject: Re: [Cake] Cake on openwrt - falling behind
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-01 19:41 ` Kevin Darbyshire-Bryant
@ 2018-07-02 10:19 ` Pete Heist
2018-07-02 11:38 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-02 10:19 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
> On Jul 1, 2018, at 9:41 PM, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
> For those who fancy dissecting some netlink packets, see attached :-)
Hard to see much in those so far, just found out what netlink is. :) It does look very easy to get alignment and byte order of these messages wrong by using values directly instead of through macros, etc. Also noticed the comment in libnetlink(3) to “use libmnl for new programs”, which tc does not.
What is TCA_STATS2 vs TCA_STATS and where does it come from? That’s at least one rtattr that’s null in print_qdisc on MIPS for some reason. That’s where I’m at, anyway…
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 10:19 ` Pete Heist
@ 2018-07-02 11:38 ` Pete Heist
2018-07-02 11:59 ` Kevin Darbyshire-Bryant
2018-07-02 12:03 ` Toke Høiland-Jørgensen
0 siblings, 2 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-02 11:38 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
> On Jul 2, 2018, at 12:19 PM, Pete Heist <pete@heistp.net> wrote:
>
> What is TCA_STATS2 vs TCA_STATS and where does it come from? That’s at least one rtattr that’s null in print_qdisc on MIPS for some reason. That’s where I’m at, anyway…
I should have written that the rtattr at TCP_STATS2 has a zero len and type in it, not that it’s null.
But, um, I find it curious that tb[TCA_PAD] has valid looking values in it, and if I just go:
tb[TCA_STATS2] = tb[TCA_PAD]
right after parse_rtattr is called, I start getting tin stats printed that look valid. There should be zeroes or invalid values at tb[TCA_PAD], as that’s just supposed to be padding, right?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 11:38 ` Pete Heist
@ 2018-07-02 11:59 ` Kevin Darbyshire-Bryant
2018-07-02 14:01 ` Pete Heist
2018-07-02 12:03 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 11:59 UTC (permalink / raw)
To: Pete Heist; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]
> On 2 Jul 2018, at 12:38, Pete Heist <pete@heistp.net> wrote:
>
>
>> On Jul 2, 2018, at 12:19 PM, Pete Heist <pete@heistp.net> wrote:
>>
>> What is TCA_STATS2 vs TCA_STATS and where does it come from? That’s at least one rtattr that’s null in print_qdisc on MIPS for some reason. That’s where I’m at, anyway…
>
> I should have written that the rtattr at TCP_STATS2 has a zero len and type in it, not that it’s null.
>
> But, um, I find it curious that tb[TCA_PAD] has valid looking values in it, and if I just go:
>
> tb[TCA_STATS2] = tb[TCA_PAD]
>
> right after parse_rtattr is called, I start getting tin stats printed that look valid. There should be zeroes or invalid values at tb[TCA_PAD], as that’s just supposed to be padding, right?
>
Thank you Pete for your persistence in this. I’m afraid I’ve just be zapped by a migraine so will be out of it for a day or two.
My understanding is the padding is only used align 64 bit values, if required… and we’re back to CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
Sorry struggling to type english…which this may not be, can’t see clearly
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 11:38 ` Pete Heist
2018-07-02 11:59 ` Kevin Darbyshire-Bryant
@ 2018-07-02 12:03 ` Toke Høiland-Jørgensen
2018-07-02 14:51 ` Pete Heist
1 sibling, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 12:03 UTC (permalink / raw)
To: Pete Heist, Kevin Darbyshire-Bryant; +Cc: Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 2, 2018, at 12:19 PM, Pete Heist <pete@heistp.net> wrote:
>>
>> What is TCA_STATS2 vs TCA_STATS and where does it come from? That’s at least one rtattr that’s null in print_qdisc on MIPS for some reason. That’s where I’m at, anyway…
>
> I should have written that the rtattr at TCP_STATS2 has a zero len and
> type in it, not that it’s null.
>
> But, um, I find it curious that tb[TCA_PAD] has valid looking values
> in it, and if I just go:
>
> tb[TCA_STATS2] = tb[TCA_PAD]
>
> right after parse_rtattr is called, I start getting tin stats printed
> that look valid. There should be zeroes or invalid values at
> tb[TCA_PAD], as that’s just supposed to be padding, right?
Hmm, that's interesting. Sounds like you are on the right track. What
are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
iproute2, respectively?
And thank you for looking into this! :)
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 11:59 ` Kevin Darbyshire-Bryant
@ 2018-07-02 14:01 ` Pete Heist
0 siblings, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-02 14:01 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
> On Jul 2, 2018, at 1:59 PM, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
>> On 2 Jul 2018, at 12:38, Pete Heist <pete@heistp.net> wrote:
>>
>> I should have written that the rtattr at TCP_STATS2 has a zero len and type in it, not that it’s null.
>>
>> But, um, I find it curious that tb[TCA_PAD] has valid looking values in it, and if I just go:
>>
>> tb[TCA_STATS2] = tb[TCA_PAD]
>>
>> right after parse_rtattr is called, I start getting tin stats printed that look valid. There should be zeroes or invalid values at tb[TCA_PAD], as that’s just supposed to be padding, right?
>
> Thank you Pete for your persistence in this. I’m afraid I’ve just be zapped by a migraine so will be out of it for a day or two.
Ouch, then by all means get away from the computer! I used to get them until I found out for me they’re triggered by some foods with processed sugars- some ice creams would be a sure path to one so I have to avoid it entirely, dangit.
> My understanding is the padding is only used align 64 bit values, if required… and we’re back to CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
Ok, every clue can help, I’ll keep trying on this...
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 12:03 ` Toke Høiland-Jørgensen
@ 2018-07-02 14:51 ` Pete Heist
2018-07-02 16:14 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-02 14:51 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
>> in it, and if I just go:
>>
>> tb[TCA_STATS2] = tb[TCA_PAD]
>>
>> right after parse_rtattr is called, I start getting tin stats printed
>> that look valid. There should be zeroes or invalid values at
>> tb[TCA_PAD], as that’s just supposed to be padding, right?
>
> Hmm, that's interesting. Sounds like you are on the right track. What
> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
> iproute2, respectively?
I never would’ve guessed they could be different when compiled at once :) but true, there are at least five different versions of rtnetlink.h under build_dir based on their md5sums, and I see one belongs to iproute2-full. In all of those, it looks in the source like TCA_STATS2=7 and TCA_PAD=9. So I presume that the value for tb[TCA_STATS2] is out of place by 64 bits in memory. For interest, current output, which includes the values of TCA_STATS2 and TCA_PAD plus all the values in the rtattr struct after parsing:
root@OpenWrt:/tmp# tc -s -d qdisc show dev eth0
TCA_STATS2 val=00000007
TCA_PAD val=00000009
tb[TCA_UNSPEC]=00000000
tb[TCA_KIND]=77f013fc
tb[TCA_OPTIONS]=77f01408
tb[TCA_STATS]=77f0172c
tb[TCA_XSTATS]=00000000
tb[TCA_RATE]=00000000
tb[TCA_FCNT]=00000000
tb[TCA_STATS2]=00000000
tb[TCA_STAB]=00000000
tb[TCA_PAD]=77f01490
tb[TCA_DUMP_INVISIBLE]=00000000
tb[TCA_CHAIN]=00000000
tb[TCA_HW_OFFLOAD]=00000000
tb[TCA_INGRESS_BLOCK]=00000000
tb[TCA_EGRESS_BLOCK]=00000000
// hack happens here to get valid pointer in tb[TCA_STATS2]: tb[TCA_STATS2] = tb[TCA_PAD]
qdisc cake 8019: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
tca_stats 2012223276 tca_stats2 2012222608 tca_xstats 0
calling print_tcstats_attr()
print_tcstats_attr()
got stats2
Sent 7992 bytes 28 pkt (dropped 0, overlimits 0 requeues 0)
xstats 2141024476 tca_stats_app 2012222616
backlog 0b 0p requeues 0
got xstats 2012222616 tca_stats 2012223276 tca_stats2 2012222608 tca_xstats 0
calling print_xstats
memory used: 2240b of 15140Kb
capacity estimate: 0bit
min/max network layer size: 66 / 1154
min/max overhead-adjusted size: 66 / 1154
average network hdr offset: 1
Bulk Best Effort Voice
thresh 0bit 0bit 0bit
target 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 100.0ms
pk_delay 0us 18us 20us
av_delay 0us 0us 1us
sp_delay 0us 0us 1us
backlog 0b 0b 0b
pkts 0 8 20
bytes 0 3376 4616
...
> And thank you for looking into this! :)
Least I can do! I’m afraid I’m still only at the point where we know that “something” is wrong “somewhere” with either the generation or parsing of this netlink message, so I’ll keep exploring…
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 14:51 ` Pete Heist
@ 2018-07-02 16:14 ` Toke Høiland-Jørgensen
2018-07-02 16:59 ` Kevin Darbyshire-Bryant
` (3 more replies)
0 siblings, 4 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 16:14 UTC (permalink / raw)
To: Pete Heist; +Cc: Kevin Darbyshire-Bryant, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Pete Heist <pete@heistp.net> writes:
>>
>>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
>>> in it, and if I just go:
>>>
>>> tb[TCA_STATS2] = tb[TCA_PAD]
>>>
>>> right after parse_rtattr is called, I start getting tin stats printed
>>> that look valid. There should be zeroes or invalid values at
>>> tb[TCA_PAD], as that’s just supposed to be padding, right?
>>
>> Hmm, that's interesting. Sounds like you are on the right track. What
>> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
>> iproute2, respectively?
>
> I never would’ve guessed they could be different when compiled at once
> :) but true, there are at least five different versions of rtnetlink.h
> under build_dir based on their md5sums, and I see one belongs to
> iproute2-full. In all of those, it looks in the source like
> TCA_STATS2=7 and TCA_PAD=9.
Aha! I think I figured out what is going on:
The gen_stats facility will add an nlattr header at the beginning of the
qdisc stats, which is the toplevel TLV that contains all stats (and that
we put our stats inside). It stores a reference to this header, and when
all the per-qdisc callbacks have finished adding their stats, it goes
back and fixes up the length of the containing header.
The problem is that on architectures that need padding, the padding TLV
is added *first*, which means that the nlattr pointer that is stored
before the callbacks are performed points to the padding TLV and not the
stats TLV. And so, when the header is fixed up, the result (from the
parser's perspective) is just a very big padding TLV.
The options TLV is before the stats TLV, so the bug only occurs if the
options happen to have a length that means the stats will need padding.
Which is why messing with the number of options "fixes" the bug.
Could you try applying the patch below (to the kernel) and see if that
resolves the issue, please?
-Toke
@@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
d->lock = lock;
spin_lock_bh(lock);
}
- if (d->tail)
- return gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (d->tail) {
+ int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (!ret)
+ d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
+ return ret;
+ }
return 0;
}
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 16:14 ` Toke Høiland-Jørgensen
@ 2018-07-02 16:59 ` Kevin Darbyshire-Bryant
2018-07-02 17:04 ` Pete Heist
` (2 subsequent siblings)
3 siblings, 0 replies; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 16:59 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Pete Heist, Cake List
[-- Attachment #1: Type: text/plain, Size: 4663 bytes --]
> On 2 Jul 2018, at 17:14, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
>>> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Pete Heist <pete@heistp.net> writes:
>>>
>>>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
>>>> in it, and if I just go:
>>>>
>>>> tb[TCA_STATS2] = tb[TCA_PAD]
>>>>
>>>> right after parse_rtattr is called, I start getting tin stats printed
>>>> that look valid. There should be zeroes or invalid values at
>>>> tb[TCA_PAD], as that’s just supposed to be padding, right?
>>>
>>> Hmm, that's interesting. Sounds like you are on the right track. What
>>> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
>>> iproute2, respectively?
>>
>> I never would’ve guessed they could be different when compiled at once
>> :) but true, there are at least five different versions of rtnetlink.h
>> under build_dir based on their md5sums, and I see one belongs to
>> iproute2-full. In all of those, it looks in the source like
>> TCA_STATS2=7 and TCA_PAD=9.
>
> Aha! I think I figured out what is going on:
>
> The gen_stats facility will add an nlattr header at the beginning of the
> qdisc stats, which is the toplevel TLV that contains all stats (and that
> we put our stats inside). It stores a reference to this header, and when
> all the per-qdisc callbacks have finished adding their stats, it goes
> back and fixes up the length of the containing header.
>
> The problem is that on architectures that need padding, the padding TLV
> is added *first*, which means that the nlattr pointer that is stored
> before the callbacks are performed points to the padding TLV and not the
> stats TLV. And so, when the header is fixed up, the result (from the
> parser's perspective) is just a very big padding TLV.
>
> The options TLV is before the stats TLV, so the bug only occurs if the
> options happen to have a length that means the stats will need padding.
> Which is why messing with the number of options "fixes" the bug.
>
> Could you try applying the patch below (to the kernel) and see if that
> resolves the issue, please?
>
> -Toke
>
>
> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
> d->lock = lock;
> spin_lock_bh(lock);
> }
> - if (d->tail)
> - return gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (d->tail) {
> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (!ret)
> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
> + return ret;
> + }
>
> return 0;
> }
I do believe Sir you have cracked it. Without my u32 hack on a MIPS34kc arch that previously required my hack…..
tc -s qdisc show dev ifb4pppoa-wan
qdisc cake 800b: root refcnt 2 bandwidth 2027Kbit diffserv3 dual-dsthost nat ingress split-gso rtt 100.0ms raw overhead 0
tca_stats 2011972452 tca_stats2 2011971788 tca_xstats 0
calling print_tcstats_attr()
print_tcstats_attr()
got stats2
Sent 144 bytes 3 pkt (dropped 0, overlimits 0 requeues 0)
xstats 2145834940 tca_stats_app 2011971792
backlog 0b 0p requeues 0
got xstats 2011971792 tca_stats 2011972452 tca_stats2 2011971788 tca_xstats 0
memory used: 2464b of 4Mb
capacity estimate: 2027Kbit
min/max network layer size: 48 / 48
min/max overhead-adjusted size: 48 / 48
average network hdr offset: 0
Bulk Best Effort Voice
thresh 126680bit 2027Kbit 506744bit
target 143.4ms 9.0ms 35.9ms
interval 286.8ms 104.0ms 130.9ms
pk_delay 0us 17us 0us
av_delay 0us 0us 0us
sp_delay 0us 0us 0us
backlog 0b 0b 0b
pkts 0 3 0
bytes 0 144 0
way_inds 0 0 0
way_miss 0 1 0
way_cols 0 0 0
drops 0 0 0
marks 0 0 0
ack_drop 0 0 0
sp_flows 0 1 0
bk_flows 0 0 0
un_flows 0 0 0
max_len 0 48 0
quantum 300 300 300
Cheers,
Kevin D-B
012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 16:14 ` Toke Høiland-Jørgensen
2018-07-02 16:59 ` Kevin Darbyshire-Bryant
@ 2018-07-02 17:04 ` Pete Heist
2018-07-02 17:12 ` Kevin Darbyshire-Bryant
2018-07-02 18:24 ` Pete Heist
[not found] ` <mailman.407.1530550780.3512.cake@lists.bufferbloat.net>
2018-07-02 18:39 ` [Cake] Cake on openwrt - falling behind Dave Taht
3 siblings, 2 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-02 17:04 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]
> On Jul 2, 2018, at 6:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Aha! I think I figured out what is going on:
>
> The gen_stats facility will add an nlattr header at the beginning of the
> qdisc stats, which is the toplevel TLV that contains all stats (and that
> we put our stats inside). It stores a reference to this header, and when
> all the per-qdisc callbacks have finished adding their stats, it goes
> back and fixes up the length of the containing header.
>
> The problem is that on architectures that need padding, the padding TLV
> is added *first*, which means that the nlattr pointer that is stored
> before the callbacks are performed points to the padding TLV and not the
> stats TLV. And so, when the header is fixed up, the result (from the
> parser's perspective) is just a very big padding TLV.
>
> The options TLV is before the stats TLV, so the bug only occurs if the
> options happen to have a length that means the stats will need padding.
> Which is why messing with the number of options "fixes" the bug.
>
> Could you try applying the patch below (to the kernel) and see if that
> resolves the issue, please?
Awesome Toke! It looks like from Kevin’s email that it works for him, but it didn’t work for me the first time around. This may have to do with how I added the patch as I’m still not that familiar with OpenWRT’s build system (first kernel patch I tried). I wasn’t sure if it should go into generic or platform, for one, so I tried generic…is that right?
sysadmin@rey:~/src/openwrt/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/linux-4.9.109/patches/generic$ cat 010-gen_stats_fix.patch
--- a/net/core/gen_stats.c
+++ b/net/core/gen_stats.c
@@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_b
d->lock = lock;
spin_lock_bh(lock);
}
- if (d->tail)
- return gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (d->tail) {
+ int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (!ret)
+ d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
+ return ret;
+ }
return 0;
}
[-- Attachment #2: Type: text/html, Size: 16631 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 17:04 ` Pete Heist
@ 2018-07-02 17:12 ` Kevin Darbyshire-Bryant
2018-07-02 18:24 ` Pete Heist
1 sibling, 0 replies; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 17:12 UTC (permalink / raw)
To: Pete Heist; +Cc: Toke Høiland-Jørgensen, Cake List
[-- Attachment #1: Type: text/plain, Size: 2872 bytes --]
> On 2 Jul 2018, at 18:04, Pete Heist <pete@heistp.net> wrote:
>
>
>
>> On Jul 2, 2018, at 6:14 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Aha! I think I figured out what is going on:
>>
>> The gen_stats facility will add an nlattr header at the beginning of the
>> qdisc stats, which is the toplevel TLV that contains all stats (and that
>> we put our stats inside). It stores a reference to this header, and when
>> all the per-qdisc callbacks have finished adding their stats, it goes
>> back and fixes up the length of the containing header.
>>
>> The problem is that on architectures that need padding, the padding TLV
>> is added *first*, which means that the nlattr pointer that is stored
>> before the callbacks are performed points to the padding TLV and not the
>> stats TLV. And so, when the header is fixed up, the result (from the
>> parser's perspective) is just a very big padding TLV.
>>
>> The options TLV is before the stats TLV, so the bug only occurs if the
>> options happen to have a length that means the stats will need padding.
>> Which is why messing with the number of options "fixes" the bug.
>>
>> Could you try applying the patch below (to the kernel) and see if that
>> resolves the issue, please?
>
> Awesome Toke! It looks like from Kevin’s email that it works for him, but it didn’t work for me the first time around. This may have to do with how I added the patch as I’m still not that familiar with OpenWRT’s build system (first kernel patch I tried). I wasn’t sure if it should go into generic or platform, for one, so I tried generic…is that right?
>
> sysadmin@rey:~/src/openwrt/build_dir/target-mips_24kc_musl/linux-ar71xx_generic/linux-4.9.109/patches/generic$ cat 010-gen_stats_fix.patch
> --- a/net/core/gen_stats.c
> +++ b/net/core/gen_stats.c
> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_b
> d->lock = lock;
> spin_lock_bh(lock);
> }
> - if (d->tail)
> - return gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (d->tail) {
> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (!ret)
> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
> + return ret;
> + }
>
> return 0;
> }
>
generic is where it should go, but what I did was:
make package/iproute2/clean
make package/kmod-sched-cake/clean - both to be *really* sure they get rebuilt, shouldn’t be required but!!!
make target/linux/{clean,prepare} (this gets the kernel build tree in a ready to build state with all the openwrt patches)
edit build/target_mips34*/linux*/linux*/net/core/genstat.c directly
make
flash the image - test :-)
I assume tha patch has been tried on previously unbroken arch’s and they remain unb0rked ?
Cheers,
Kevin D-B
012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
[not found] ` <mailman.407.1530550780.3512.cake@lists.bufferbloat.net>
@ 2018-07-02 17:50 ` Kevin Darbyshire-Bryant
2018-07-02 19:33 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 17:50 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 3359 bytes --]
> On 2 Jul 2018, at 17:59, Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net> wrote:
>
>
> From: Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
> Subject: Re: [Cake] Cake on openwrt - falling behind
> Date: 2 July 2018 at 17:59:36 BST
> To: Toke Høiland-Jørgensen <toke@toke.dk>
> Cc: Pete Heist <pete@heistp.net>, Cake List <cake@lists.bufferbloat.net>
>
>
>
>
>> On 2 Jul 2018, at 17:14, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Pete Heist <pete@heistp.net> writes:
>>
>>>> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>>
>>>> Pete Heist <pete@heistp.net> writes:
>>>>
>>>>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
>>>>> in it, and if I just go:
>>>>>
>>>>> tb[TCA_STATS2] = tb[TCA_PAD]
>>>>>
>>>>> right after parse_rtattr is called, I start getting tin stats printed
>>>>> that look valid. There should be zeroes or invalid values at
>>>>> tb[TCA_PAD], as that’s just supposed to be padding, right?
>>>>
>>>> Hmm, that's interesting. Sounds like you are on the right track. What
>>>> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
>>>> iproute2, respectively?
>>>
>>> I never would’ve guessed they could be different when compiled at once
>>> :) but true, there are at least five different versions of rtnetlink.h
>>> under build_dir based on their md5sums, and I see one belongs to
>>> iproute2-full. In all of those, it looks in the source like
>>> TCA_STATS2=7 and TCA_PAD=9.
>>
>> Aha! I think I figured out what is going on:
>>
>> The gen_stats facility will add an nlattr header at the beginning of the
>> qdisc stats, which is the toplevel TLV that contains all stats (and that
>> we put our stats inside). It stores a reference to this header, and when
>> all the per-qdisc callbacks have finished adding their stats, it goes
>> back and fixes up the length of the containing header.
>>
>> The problem is that on architectures that need padding, the padding TLV
>> is added *first*, which means that the nlattr pointer that is stored
>> before the callbacks are performed points to the padding TLV and not the
>> stats TLV. And so, when the header is fixed up, the result (from the
>> parser's perspective) is just a very big padding TLV.
>>
>> The options TLV is before the stats TLV, so the bug only occurs if the
>> options happen to have a length that means the stats will need padding.
>> Which is why messing with the number of options "fixes" the bug.
>>
>> Could you try applying the patch below (to the kernel) and see if that
>> resolves the issue, please?
>>
>> -Toke
>>
>>
>> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
>> d->lock = lock;
>> spin_lock_bh(lock);
>> }
>> - if (d->tail)
>> - return gnet_stats_copy(d, type, NULL, 0, padattr);
>> + if (d->tail) {
>> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
>> + if (!ret)
>> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
>> + return ret;
>> + }
>>
>> return 0;
>> }
> o/cake
Would you like me to bash this into openwrt? Plans for this to go upstream? Perhaps now take off list or onto IRC?
Don’t think premature - TRULY FANTASTIC WORK EVERYONE!!!!
Kevin
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 17:04 ` Pete Heist
2018-07-02 17:12 ` Kevin Darbyshire-Bryant
@ 2018-07-02 18:24 ` Pete Heist
2018-07-02 19:31 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-02 18:24 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
[-- Attachment #1: Type: text/plain, Size: 2889 bytes --]
> On Jul 2, 2018, at 7:04 PM, Pete Heist <pete@heistp.net> wrote:
>
>
>
>> On Jul 2, 2018, at 6:14 PM, Toke Høiland-Jørgensen <toke@toke.dk <mailto:toke@toke.dk>> wrote:
>>
>> Aha! I think I figured out what is going on:
>>
>> The gen_stats facility will add an nlattr header at the beginning of the
>> qdisc stats, which is the toplevel TLV that contains all stats (and that
>> we put our stats inside). It stores a reference to this header, and when
>> all the per-qdisc callbacks have finished adding their stats, it goes
>> back and fixes up the length of the containing header.
>>
>> The problem is that on architectures that need padding, the padding TLV
>> is added *first*, which means that the nlattr pointer that is stored
>> before the callbacks are performed points to the padding TLV and not the
>> stats TLV. And so, when the header is fixed up, the result (from the
>> parser's perspective) is just a very big padding TLV.
>>
>> The options TLV is before the stats TLV, so the bug only occurs if the
>> options happen to have a length that means the stats will need padding.
>> Which is why messing with the number of options "fixes" the bug.
>>
>> Could you try applying the patch below (to the kernel) and see if that
>> resolves the issue, please?
>
> Awesome Toke! It looks like from Kevin’s email that it works for him, but it didn’t work for me the first time around. This may have to do with how I added the patch as I’m still not that familiar with OpenWRT’s build system (first kernel patch I tried). I wasn’t sure if it should go into generic or platform, for one, so I tried generic…is that right?
Ok, I got it to work after re-flashing with tftp. :) It looks like the OM2P is not always successfully performing sysupgrades, perhaps due to its limited memory (64M), but I’m not sure.
I still have my debugging in place and do still have one question. The pointer in TCA_STATS2 is now valid, but there is still a pointer value in TCA_PAD, which is pointing to a place 32 bits before TCA_STATS2. Is that expected?
root@OpenWrt:/tmp# tc -s -d qdisc show dev eth0
TCA_STATS2 val=00000007
TCA_PAD val=00000009
tb[TCA_UNSPEC]=00000000
tb[TCA_KIND]=773a73fc
tb[TCA_OPTIONS]=773a7408
tb[TCA_STATS]=773a772c
tb[TCA_XSTATS]=00000000
tb[TCA_RATE]=00000000
tb[TCA_FCNT]=00000000
tb[TCA_STATS2]=773a7494
tb[TCA_STAB]=00000000
tb[TCA_PAD]=773a7490
tb[TCA_DUMP_INVISIBLE]=00000000
tb[TCA_CHAIN]=00000000
tb[TCA_HW_OFFLOAD]=00000000
tb[TCA_INGRESS_BLOCK]=00000000
tb[TCA_EGRESS_BLOCK]=00000000
qdisc cake 8001: root refcnt 2 bandwidth unlimited diffserv3 triple-isolate rtt 100.0ms raw overhead 0
tca_stats 2000320300 tca_stats2 2000319636 tca_xstats 0
…
Also, if you agree, I’d like to see this tested on 32-bit ARM (George, or my Raspi?) and 64-bit Intel, at least. What do you think?
[-- Attachment #2: Type: text/html, Size: 17141 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 16:14 ` Toke Høiland-Jørgensen
` (2 preceding siblings ...)
[not found] ` <mailman.407.1530550780.3512.cake@lists.bufferbloat.net>
@ 2018-07-02 18:39 ` Dave Taht
2018-07-02 19:11 ` Kevin Darbyshire-Bryant
2018-07-02 19:31 ` Pete Heist
3 siblings, 2 replies; 77+ messages in thread
From: Dave Taht @ 2018-07-02 18:39 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Pete Heist, Cake List
On Mon, Jul 2, 2018 at 12:13 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net> writes:
>
> >> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >>
> >> Pete Heist <pete@heistp.net> writes:
> >>
> >>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
> >>> in it, and if I just go:
> >>>
> >>> tb[TCA_STATS2] = tb[TCA_PAD]
> >>>
> >>> right after parse_rtattr is called, I start getting tin stats printed
> >>> that look valid. There should be zeroes or invalid values at
> >>> tb[TCA_PAD], as that’s just supposed to be padding, right?
> >>
> >> Hmm, that's interesting. Sounds like you are on the right track. What
> >> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
> >> iproute2, respectively?
> >
> > I never would’ve guessed they could be different when compiled at once
> > :) but true, there are at least five different versions of rtnetlink.h
> > under build_dir based on their md5sums, and I see one belongs to
> > iproute2-full. In all of those, it looks in the source like
> > TCA_STATS2=7 and TCA_PAD=9.
>
> Aha! I think I figured out what is going on:
>
> The gen_stats facility will add an nlattr header at the beginning of the
> qdisc stats, which is the toplevel TLV that contains all stats (and that
> we put our stats inside). It stores a reference to this header, and when
> all the per-qdisc callbacks have finished adding their stats, it goes
> back and fixes up the length of the containing header.
>
> The problem is that on architectures that need padding, the padding TLV
> is added *first*, which means that the nlattr pointer that is stored
> before the callbacks are performed points to the padding TLV and not the
> stats TLV. And so, when the header is fixed up, the result (from the
> parser's perspective) is just a very big padding TLV.
>
> The options TLV is before the stats TLV, so the bug only occurs if the
> options happen to have a length that means the stats will need padding.
> Which is why messing with the number of options "fixes" the bug.
>
> Could you try applying the patch below (to the kernel) and see if that
> resolves the issue, please?
This seems like it will introduce problems with stuff that isn't or is
legitimately broken in the first place, pointing to potentially random
data in the wrong place.
would a workaround be adding more padding to the cake stats output so
it's always even?
why does it work as written on arm?
>
> -Toke
>
>
> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
> d->lock = lock;
> spin_lock_bh(lock);
> }
> - if (d->tail)
> - return gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (d->tail) {
> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
> + if (!ret)
> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
> + return ret;
> + }
>
> return 0;
> }
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 18:39 ` [Cake] Cake on openwrt - falling behind Dave Taht
@ 2018-07-02 19:11 ` Kevin Darbyshire-Bryant
2018-07-02 19:23 ` Toke Høiland-Jørgensen
2018-07-02 19:31 ` Pete Heist
1 sibling, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 19:11 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, Cake List
[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]
> On 2 Jul 2018, at 19:39, Dave Taht <dave.taht@gmail.com> wrote:
>
>>
>
> This seems like it will introduce problems with stuff that isn't or is
> legitimately broken in the first place, pointing to potentially random
> data in the wrong place.
>
> would a workaround be adding more padding to the cake stats output so
> it's always even?
>
> why does it work as written on arm?
If I understand correctly: This will only be a problem on architectures that require alignment of 64 bit values to 8 byte boundaries which is achieved by padding the structure by a dummy (4 byte) value if required. So to hit this bug we need kernel symbol CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS undefined *and* we need a netlink stats structure that needs a 4 byte dummy pad value to align to 8 bytes. Of the architectures tested, MIPS is the only one that DOES NOT set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus may be exposed to the bug.
arm sets CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus no padding is ever required/added, thus pointers always point to the correct data location.
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:11 ` Kevin Darbyshire-Bryant
@ 2018-07-02 19:23 ` Toke Høiland-Jørgensen
2018-07-02 19:27 ` Dave Taht
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 19:23 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant, Dave Taht; +Cc: Cake List
Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>> On 2 Jul 2018, at 19:39, Dave Taht <dave.taht@gmail.com> wrote:
>>
>>>
>>
>> This seems like it will introduce problems with stuff that isn't or is
>> legitimately broken in the first place, pointing to potentially random
>> data in the wrong place.
>>
>> would a workaround be adding more padding to the cake stats output so
>> it's always even?
>>
>> why does it work as written on arm?
>
> If I understand correctly: This will only be a problem on
> architectures that require alignment of 64 bit values to 8 byte
> boundaries which is achieved by padding the structure by a dummy (4
> byte) value if required. So to hit this bug we need kernel symbol
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS undefined *and* we need a
> netlink stats structure that needs a 4 byte dummy pad value to align
> to 8 bytes. Of the architectures tested, MIPS is the only one that
> DOES NOT set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus may be
> exposed to the bug.
>
> arm sets CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus no padding is
> ever required/added, thus pointers always point to the correct data
> location.
Yup, exactly. This has always been broken on MIPS, I assume, but because
most other qdiscs send their stats output as a serialised struct, tc
just automatically falls back to the legacy data format, and no one has
noticed. But because we switched to sending each stat as an individual
netlink attribute (and thus no fallback legacy stats struct), we expose
the bug...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:23 ` Toke Høiland-Jørgensen
@ 2018-07-02 19:27 ` Dave Taht
2018-07-02 19:38 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Dave Taht @ 2018-07-02 19:27 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
On Mon, Jul 2, 2018 at 12:23 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>
> >> On 2 Jul 2018, at 19:39, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >>>
> >>
> >> This seems like it will introduce problems with stuff that isn't or is
> >> legitimately broken in the first place, pointing to potentially random
> >> data in the wrong place.
> >>
> >> would a workaround be adding more padding to the cake stats output so
> >> it's always even?
> >>
> >> why does it work as written on arm?
> >
> > If I understand correctly: This will only be a problem on
> > architectures that require alignment of 64 bit values to 8 byte
> > boundaries which is achieved by padding the structure by a dummy (4
> > byte) value if required. So to hit this bug we need kernel symbol
> > CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS undefined *and* we need a
> > netlink stats structure that needs a 4 byte dummy pad value to align
> > to 8 bytes. Of the architectures tested, MIPS is the only one that
> > DOES NOT set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus may be
> > exposed to the bug.
> >
> > arm sets CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus no padding is
> > ever required/added, thus pointers always point to the correct data
> > location.
>
> Yup, exactly. This has always been broken on MIPS, I assume, but because
> most other qdiscs send their stats output as a serialised struct, tc
> just automatically falls back to the legacy data format, and no one has
> noticed. But because we switched to sending each stat as an individual
> netlink attribute (and thus no fallback legacy stats struct), we expose
> the bug...
Well, if you wrap that patch in
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
bla
#endif
I guess I'd sleep better but I do generally get nervous when
arbitrarily subtracting something
from a pointer
>
> -Toke
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 18:24 ` Pete Heist
@ 2018-07-02 19:31 ` Toke Høiland-Jørgensen
2018-07-02 20:09 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 19:31 UTC (permalink / raw)
To: Pete Heist; +Cc: Kevin Darbyshire-Bryant, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 2, 2018, at 7:04 PM, Pete Heist <pete@heistp.net> wrote:
>>
>>
>>
>>> On Jul 2, 2018, at 6:14 PM, Toke Høiland-Jørgensen <toke@toke.dk <mailto:toke@toke.dk>> wrote:
>>>
>>> Aha! I think I figured out what is going on:
>>>
>>> The gen_stats facility will add an nlattr header at the beginning of the
>>> qdisc stats, which is the toplevel TLV that contains all stats (and that
>>> we put our stats inside). It stores a reference to this header, and when
>>> all the per-qdisc callbacks have finished adding their stats, it goes
>>> back and fixes up the length of the containing header.
>>>
>>> The problem is that on architectures that need padding, the padding TLV
>>> is added *first*, which means that the nlattr pointer that is stored
>>> before the callbacks are performed points to the padding TLV and not the
>>> stats TLV. And so, when the header is fixed up, the result (from the
>>> parser's perspective) is just a very big padding TLV.
>>>
>>> The options TLV is before the stats TLV, so the bug only occurs if the
>>> options happen to have a length that means the stats will need padding.
>>> Which is why messing with the number of options "fixes" the bug.
>>>
>>> Could you try applying the patch below (to the kernel) and see if that
>>> resolves the issue, please?
>>
>> Awesome Toke! It looks like from Kevin’s email that it works for him,
>> but it didn’t work for me the first time around. This may have to do
>> with how I added the patch as I’m still not that familiar with
>> OpenWRT’s build system (first kernel patch I tried). I wasn’t sure if
>> it should go into generic or platform, for one, so I tried generic…is
>> that right?
>
> Ok, I got it to work after re-flashing with tftp. :) It looks like the
> OM2P is not always successfully performing sysupgrades, perhaps due to
> its limited memory (64M), but I’m not sure.
Great, thanks for testing, both of you!
> I still have my debugging in place and do still have one question. The
> pointer in TCA_STATS2 is now valid, but there is still a pointer value
> in TCA_PAD, which is pointing to a place 32 bits before TCA_STATS2. Is
> that expected?
Yes, that is expected. The PAD is there to align the subsequent STATS
TLV, so it is being parsed but is unused.
This is the offending spot from Kevin's pcap files:
Working: The header starts at byte 0x17c with a TLV of type 7 (TCA_STATS2)
of length 0x268. No padding TLV needed, so stats work:
00000170: 0000 0000 0008 000a 0000 0000 0268 0007 .............h..
00000180: 0234 0004 0008 0002 0025 f4cc 0008 0003 .4.......%......
Broken: The header starts at byte 0x180 with a PAD TLV (type 9) followed
by the TCA_STATS2 TLV. Both start out with 4-byte lengths (just the
header), but because the PAD TLV is the one being extended, that gets a
new length of 0x29c, which means it now contains the TCA_STATS2 TLV as
the first nested TLV. What should have happened was that the TCA_PAD TLV
should have stayed at 4 bytes, thus aligning the payload of the
TCA_STATS2 TLV at 0x188 bytes, and the TCA_STATS2 TLV should have been
length 0x298. Which is what happens after the patch, and why TCA_PAD is
still set by the parser.
00000180: 029c 0009 0004 0007 0260 0004 000c 0002 .........`......
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 18:39 ` [Cake] Cake on openwrt - falling behind Dave Taht
2018-07-02 19:11 ` Kevin Darbyshire-Bryant
@ 2018-07-02 19:31 ` Pete Heist
1 sibling, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-02 19:31 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, Cake List
[-- Attachment #1: Type: text/plain, Size: 1657 bytes --]
> On Jul 2, 2018, at 8:39 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Jul 2, 2018 at 12:13 PM Toke Høiland-Jørgensen <toke@toke.dk <mailto:toke@toke.dk>> wrote:
>>
>> Could you try applying the patch below (to the kernel) and see if that
>> resolves the issue, please?
>
> This seems like it will introduce problems with stuff that isn't or is
> legitimately broken in the first place, pointing to potentially random
> data in the wrong place.
>
> would a workaround be adding more padding to the cake stats output so
> it's always even?
>
> why does it work as written on arm?
I doubt this, but what if someone else conditionally padded their custom stats in response? Personally I’d say get it right in the kernel and “make them fix their stuff”, but does anyone have visibility on that?
After re-reading Toke’s explanation I see now why tb[TCA_PAD] points to right before TCA_STATS2. That’s where the padding is. :) I was only generally wondering if a padding pointer should point to anything.
>>
>> -Toke
>>
>>
>> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
>> d->lock = lock;
>> spin_lock_bh(lock);
>> }
>> - if (d->tail)
>> - return gnet_stats_copy(d, type, NULL, 0, padattr);
>> + if (d->tail) {
>> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
>> + if (!ret)
>> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
>> + return ret;
>> + }
>>
>> return 0;
>> }
[-- Attachment #2: Type: text/html, Size: 10657 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 17:50 ` Kevin Darbyshire-Bryant
@ 2018-07-02 19:33 ` Toke Høiland-Jørgensen
2018-07-02 19:36 ` Kevin Darbyshire-Bryant
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 19:33 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>> On 2 Jul 2018, at 17:59, Kevin Darbyshire-Bryant via Cake <cake@lists.bufferbloat.net> wrote:
>>
>>
>> From: Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk>
>> Subject: Re: [Cake] Cake on openwrt - falling behind
>> Date: 2 July 2018 at 17:59:36 BST
>> To: Toke Høiland-Jørgensen <toke@toke.dk>
>> Cc: Pete Heist <pete@heistp.net>, Cake List <cake@lists.bufferbloat.net>
>>
>>
>>
>>
>>> On 2 Jul 2018, at 17:14, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Pete Heist <pete@heistp.net> writes:
>>>
>>>>> On Jul 2, 2018, at 2:03 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>>>
>>>>> Pete Heist <pete@heistp.net> writes:
>>>>>
>>>>>> But, um, I find it curious that tb[TCA_PAD] has valid looking values
>>>>>> in it, and if I just go:
>>>>>>
>>>>>> tb[TCA_STATS2] = tb[TCA_PAD]
>>>>>>
>>>>>> right after parse_rtattr is called, I start getting tin stats printed
>>>>>> that look valid. There should be zeroes or invalid values at
>>>>>> tb[TCA_PAD], as that’s just supposed to be padding, right?
>>>>>
>>>>> Hmm, that's interesting. Sounds like you are on the right track. What
>>>>> are the numerical values of TCA_STATS2 and TCA_PAD in the kernel and
>>>>> iproute2, respectively?
>>>>
>>>> I never would’ve guessed they could be different when compiled at once
>>>> :) but true, there are at least five different versions of rtnetlink.h
>>>> under build_dir based on their md5sums, and I see one belongs to
>>>> iproute2-full. In all of those, it looks in the source like
>>>> TCA_STATS2=7 and TCA_PAD=9.
>>>
>>> Aha! I think I figured out what is going on:
>>>
>>> The gen_stats facility will add an nlattr header at the beginning of the
>>> qdisc stats, which is the toplevel TLV that contains all stats (and that
>>> we put our stats inside). It stores a reference to this header, and when
>>> all the per-qdisc callbacks have finished adding their stats, it goes
>>> back and fixes up the length of the containing header.
>>>
>>> The problem is that on architectures that need padding, the padding TLV
>>> is added *first*, which means that the nlattr pointer that is stored
>>> before the callbacks are performed points to the padding TLV and not the
>>> stats TLV. And so, when the header is fixed up, the result (from the
>>> parser's perspective) is just a very big padding TLV.
>>>
>>> The options TLV is before the stats TLV, so the bug only occurs if the
>>> options happen to have a length that means the stats will need padding.
>>> Which is why messing with the number of options "fixes" the bug.
>>>
>>> Could you try applying the patch below (to the kernel) and see if that
>>> resolves the issue, please?
>>>
>>> -Toke
>>>
>>>
>>> @@ -77,8 +77,12 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
>>> d->lock = lock;
>>> spin_lock_bh(lock);
>>> }
>>> - if (d->tail)
>>> - return gnet_stats_copy(d, type, NULL, 0, padattr);
>>> + if (d->tail) {
>>> + int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
>>> + if (!ret)
>>> + d->tail = ((struct nlattr *)skb_tail_pointer(skb)) - 1;
>>> + return ret;
>>> + }
>>>
>>> return 0;
>>> }
>> o/cake
>
> Would you like me to bash this into openwrt? Plans for this to go
> upstream? Perhaps now take off list or onto IRC?
I'll submit a patch for netdev. If you want to pack it up and submit it
for openwrt, that would be helpful! :)
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:33 ` Toke Høiland-Jørgensen
@ 2018-07-02 19:36 ` Kevin Darbyshire-Bryant
2018-07-02 19:39 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Kevin Darbyshire-Bryant @ 2018-07-02 19:36 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Cake List
[-- Attachment #1: Type: text/plain, Size: 618 bytes --]
> On 2 Jul 2018, at 20:33, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>>>
>>
>> Would you like me to bash this into openwrt? Plans for this to go
>> upstream? Perhaps now take off list or onto IRC?
>
> I'll submit a patch for netdev. If you want to pack it up and submit it
> for openwrt, that would be helpful! :)
>
> -Toke
They’ve given me commit access to openwrt (!!!!!) so point me at your netdev patch and I’ll commit as a backport into openwrt, and bump cake/tc as well…. tomorrow. Progress :-)
Cheers,
Kevin D-B
012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:27 ` Dave Taht
@ 2018-07-02 19:38 ` Toke Høiland-Jørgensen
2018-07-02 20:05 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 19:38 UTC (permalink / raw)
To: Dave Taht; +Cc: Kevin Darbyshire-Bryant, Cake List
Dave Taht <dave.taht@gmail.com> writes:
> On Mon, Jul 2, 2018 at 12:23 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>>
>> >> On 2 Jul 2018, at 19:39, Dave Taht <dave.taht@gmail.com> wrote:
>> >>
>> >>>
>> >>
>> >> This seems like it will introduce problems with stuff that isn't or is
>> >> legitimately broken in the first place, pointing to potentially random
>> >> data in the wrong place.
>> >>
>> >> would a workaround be adding more padding to the cake stats output so
>> >> it's always even?
>> >>
>> >> why does it work as written on arm?
>> >
>> > If I understand correctly: This will only be a problem on
>> > architectures that require alignment of 64 bit values to 8 byte
>> > boundaries which is achieved by padding the structure by a dummy (4
>> > byte) value if required. So to hit this bug we need kernel symbol
>> > CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS undefined *and* we need a
>> > netlink stats structure that needs a 4 byte dummy pad value to align
>> > to 8 bytes. Of the architectures tested, MIPS is the only one that
>> > DOES NOT set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus may be
>> > exposed to the bug.
>> >
>> > arm sets CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus no padding is
>> > ever required/added, thus pointers always point to the correct data
>> > location.
>>
>> Yup, exactly. This has always been broken on MIPS, I assume, but because
>> most other qdiscs send their stats output as a serialised struct, tc
>> just automatically falls back to the legacy data format, and no one has
>> noticed. But because we switched to sending each stat as an individual
>> netlink attribute (and thus no fallback legacy stats struct), we expose
>> the bug...
>
> Well, if you wrap that patch in
>
> #ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> bla
> #endif
>
> I guess I'd sleep better but I do generally get nervous when
> arbitrarily subtracting something
> from a pointer
Ah, right; well, that was just a proof of concept to make sure it fixed
the issue. I'll look harder at the code to see if I can find a better
solution before submitting it upstream (at a first glance the API
doesn't make it easy, but I'll have another look).
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:36 ` Kevin Darbyshire-Bryant
@ 2018-07-02 19:39 ` Toke Høiland-Jørgensen
2018-07-02 20:03 ` [Cake] cake at 60gbit Dave Taht
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 19:39 UTC (permalink / raw)
To: Kevin Darbyshire-Bryant; +Cc: Cake List
Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>> On 2 Jul 2018, at 20:33, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>>>>
>>>
>>> Would you like me to bash this into openwrt? Plans for this to go
>>> upstream? Perhaps now take off list or onto IRC?
>>
>> I'll submit a patch for netdev. If you want to pack it up and submit it
>> for openwrt, that would be helpful! :)
>>
>> -Toke
>
> They’ve given me commit access to openwrt (!!!!!)
Neat!
> so point me at your netdev patch and I’ll commit as a backport into
> openwrt, and bump cake/tc as well…. tomorrow. Progress :-)
Will do :)
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* [Cake] cake at 60gbit
2018-07-02 19:39 ` Toke Høiland-Jørgensen
@ 2018-07-02 20:03 ` Dave Taht
2018-07-02 20:09 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Dave Taht @ 2018-07-02 20:03 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
OK, so, like, changing the topic...
has anyone been able to duplicate toke's crash at >40gbit speeds?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:38 ` Toke Høiland-Jørgensen
@ 2018-07-02 20:05 ` Toke Høiland-Jørgensen
0 siblings, 0 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 20:05 UTC (permalink / raw)
To: Dave Taht; +Cc: Kevin Darbyshire-Bryant, Cake List
Toke Høiland-Jørgensen <toke@toke.dk> writes:
> Dave Taht <dave.taht@gmail.com> writes:
>
>> On Mon, Jul 2, 2018 at 12:23 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> writes:
>>>
>>> >> On 2 Jul 2018, at 19:39, Dave Taht <dave.taht@gmail.com> wrote:
>>> >>
>>> >>>
>>> >>
>>> >> This seems like it will introduce problems with stuff that isn't or is
>>> >> legitimately broken in the first place, pointing to potentially random
>>> >> data in the wrong place.
>>> >>
>>> >> would a workaround be adding more padding to the cake stats output so
>>> >> it's always even?
>>> >>
>>> >> why does it work as written on arm?
>>> >
>>> > If I understand correctly: This will only be a problem on
>>> > architectures that require alignment of 64 bit values to 8 byte
>>> > boundaries which is achieved by padding the structure by a dummy (4
>>> > byte) value if required. So to hit this bug we need kernel symbol
>>> > CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS undefined *and* we need a
>>> > netlink stats structure that needs a 4 byte dummy pad value to align
>>> > to 8 bytes. Of the architectures tested, MIPS is the only one that
>>> > DOES NOT set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus may be
>>> > exposed to the bug.
>>> >
>>> > arm sets CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS and thus no padding is
>>> > ever required/added, thus pointers always point to the correct data
>>> > location.
>>>
>>> Yup, exactly. This has always been broken on MIPS, I assume, but because
>>> most other qdiscs send their stats output as a serialised struct, tc
>>> just automatically falls back to the legacy data format, and no one has
>>> noticed. But because we switched to sending each stat as an individual
>>> netlink attribute (and thus no fallback legacy stats struct), we expose
>>> the bug...
>>
>> Well, if you wrap that patch in
>>
>> #ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>> bla
>> #endif
>>
>> I guess I'd sleep better but I do generally get nervous when
>> arbitrarily subtracting something
>> from a pointer
>
> Ah, right; well, that was just a proof of concept to make sure it fixed
> the issue. I'll look harder at the code to see if I can find a better
> solution before submitting it upstream (at a first glance the API
> doesn't make it easy, but I'll have another look).
Here's a safer version; anyone care to do a quick test before I submit
it? :)
-Toke
@@ -77,8 +77,20 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
d->lock = lock;
spin_lock_bh(lock);
}
- if (d->tail)
- return gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (d->tail) {
+ int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
+
+ /* The initial attribute added in gnet_stats_copy() may be
+ * preceded by a padding attribute, in which case d->tail will
+ * end up pointing at the padding instead of the real attribute.
+ * Fix this so gnet_stats_finish_copy() adjusts the length of
+ * the right attribute.
+ */
+ if (ret == 0 && d->tail->nla_type == padattr)
+ d->tail = (struct nlattr *)((char *)d->tail +
+ NLA_ALIGN(d->tail->nla_len));
+ return ret;
+ }
return 0;
}
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 19:31 ` Toke Høiland-Jørgensen
@ 2018-07-02 20:09 ` Pete Heist
2018-07-02 20:11 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-02 20:09 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jul 2, 2018, at 9:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Ok, I got it to work after re-flashing with tftp. :) It looks like the
>> OM2P is not always successfully performing sysupgrades, perhaps due to
>> its limited memory (64M), but I’m not sure.
I see now, my OM2P was actually running out of its “massive” 16 meg flash space on the sysupgrade because I had installed tcpdump, nlmon, etc while making netlink dumps. I got as far as correlating values in the tin stats with segments of the stats blob, but without a wireshark dissector for cake stats in netlink (who’s volunteering there? haha, I’d started considering it) I was still miles away from noticing the padding difference in the messages.
> Great, thanks for testing, both of you!
Surely- you saved me a heck of a lot of discovery time by spotting this, Gandalf, if I ever would've gotten there. :)
Finally, sleep, but I see your new version now so I’ll test it first…
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 20:03 ` [Cake] cake at 60gbit Dave Taht
@ 2018-07-02 20:09 ` Toke Høiland-Jørgensen
2018-07-02 21:16 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 20:09 UTC (permalink / raw)
To: Dave Taht; +Cc: Kevin Darbyshire-Bryant, Cake List
Dave Taht <dave.taht@gmail.com> writes:
> OK, so, like, changing the topic...
>
> has anyone been able to duplicate toke's crash at >40gbit speeds?
I haven't gotten around to looking at that yet, but am planning to later
this week (I'll be without reliable internet for a few days, so can't
do remote experiments until a few days from now).
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 20:09 ` Pete Heist
@ 2018-07-02 20:11 ` Toke Høiland-Jørgensen
2018-07-02 20:46 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 20:11 UTC (permalink / raw)
To: Pete Heist; +Cc: Kevin Darbyshire-Bryant, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 2, 2018, at 9:31 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> Ok, I got it to work after re-flashing with tftp. :) It looks like the
>>> OM2P is not always successfully performing sysupgrades, perhaps due to
>>> its limited memory (64M), but I’m not sure.
>
> I see now, my OM2P was actually running out of its “massive” 16 meg
> flash space on the sysupgrade because I had installed tcpdump, nlmon,
> etc while making netlink dumps. I got as far as correlating values in
> the tin stats with segments of the stats blob, but without a wireshark
> dissector for cake stats in netlink (who’s volunteering there? haha,
> I’d started considering it) I was still miles away from noticing the
> padding difference in the messages.
Yeah, I was dreading the manual hex dump analysis, but figured I'd have
to do it eventually; you guys making such lovely progress prompted me to
finally suck it up and look at it. And the hint that the padding
attribute was being filled was a tremendous help! :)
>> Great, thanks for testing, both of you!
>
> Surely- you saved me a heck of a lot of discovery time by spotting
> this, Gandalf, if I ever would've gotten there. :)
>
> Finally, sleep, but I see your new version now so I’ll test it first…
Great, thanks!
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] Cake on openwrt - falling behind
2018-07-02 20:11 ` Toke Høiland-Jørgensen
@ 2018-07-02 20:46 ` Pete Heist
0 siblings, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-02 20:46 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Kevin Darbyshire-Bryant, Cake List
> On Jul 2, 2018, at 10:11 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> finally suck it up and look at it. And the hint that the padding
> attribute was being filled was a tremendous help! :)
Awesome, it was worth the time now...
>>> Great, thanks for testing, both of you!
>>
>> Surely- you saved me a heck of a lot of discovery time by spotting
>> this, Gandalf, if I ever would've gotten there. :)
>>
>> Finally, sleep, but I see your new version now so I’ll test it first…
>
> Great, thanks!
So yep, I’m seeing the new version work as well, and the use of NLA_ALIGN looks cleaner to these eyes anyway.
I still haven’t tested on any other platform, but I’ll probably save that for tomorrow. Either someone gets to it or it’s up to y’all if you’re sure it’s clear enough to commit, which it sounds like… :)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 20:09 ` Toke Høiland-Jørgensen
@ 2018-07-02 21:16 ` Pete Heist
2018-07-02 21:35 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-02 21:16 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Dave Taht, Cake List
> On Jul 2, 2018, at 10:09 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Dave Taht <dave.taht@gmail.com> writes:
>
>> OK, so, like, changing the topic...
>>
>> has anyone been able to duplicate toke's crash at >40gbit speeds?
>
> I haven't gotten around to looking at that yet, but am planning to later
> this week (I'll be without reliable internet for a few days, so can't
> do remote experiments until a few days from now).
I haven’t reproduced it and not sure I can, but a few questions just in case:
- what’s the MTU of the interface, 9000 for jumbo frames?
- when you add the cake qdisc is it with any parameters?
- how long does it usually take before it locks up, and when doing what?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 21:16 ` Pete Heist
@ 2018-07-02 21:35 ` Toke Høiland-Jørgensen
2018-07-02 22:07 ` Georgios Amanakis
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 21:35 UTC (permalink / raw)
To: Pete Heist; +Cc: Dave Taht, Cake List
On 2 July 2018 23:16:26 CEST, Pete Heist <pete@heistp.net> wrote:
>
>
>> On Jul 2, 2018, at 10:09 PM, Toke Høiland-Jørgensen <toke@toke.dk>
>wrote:
>>
>> Dave Taht <dave.taht@gmail.com> writes:
>>
>>> OK, so, like, changing the topic...
>>>
>>> has anyone been able to duplicate toke's crash at >40gbit speeds?
>>
>> I haven't gotten around to looking at that yet, but am planning to
>later
>> this week (I'll be without reliable internet for a few days, so can't
>> do remote experiments until a few days from now).
>
>I haven’t reproduced it and not sure I can, but a few questions just in
>case:
>
>- what’s the MTU of the interface, 9000 for jumbo frames?
1500 bytes. No jumbo frames involved.
>- when you add the cake qdisc is it with any parameters?
Nope
>- how long does it usually take before it locks up, and when doing
>what?
Not very long - a few seconds at most. Other threads will spin waiting for the qdisc lock.
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 21:35 ` Toke Høiland-Jørgensen
@ 2018-07-02 22:07 ` Georgios Amanakis
2018-07-02 22:12 ` Dave Taht
2018-07-02 22:23 ` Toke Høiland-Jørgensen
0 siblings, 2 replies; 77+ messages in thread
From: Georgios Amanakis @ 2018-07-02 22:07 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Pete Heist, Cake List
[-- Attachment #1: Type: text/plain, Size: 106 bytes --]
I can reliably achieve 40-70gbit/s using veth on Xeons and i7s, but I
cannot reproduce the crash.
George
[-- Attachment #2: Type: text/html, Size: 294 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 22:07 ` Georgios Amanakis
@ 2018-07-02 22:12 ` Dave Taht
2018-07-02 23:48 ` Georgios Amanakis
2018-07-02 22:23 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: Dave Taht @ 2018-07-02 22:12 UTC (permalink / raw)
To: George Amanakis; +Cc: Toke Høiland-Jørgensen, Cake List
On Mon, Jul 2, 2018 at 3:08 PM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> I can reliably achieve 40-70gbit/s using veth on Xeons and i7s, but I cannot reproduce the crash.
Both with and without shaping?
ipv4? ipv6? Both?
Which flent test was it? I have a 12 core box I can attempt to bring
to bear on the problem,
but it was very slow with my existing veth setup.
there were a few changes to toke's device driver (mlx5? I think) since
the last net-next round.
> George
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 22:07 ` Georgios Amanakis
2018-07-02 22:12 ` Dave Taht
@ 2018-07-02 22:23 ` Toke Høiland-Jørgensen
2018-07-03 7:35 ` Pete Heist
2018-07-03 9:18 ` Jonathan Morton
1 sibling, 2 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-02 22:23 UTC (permalink / raw)
To: Georgios Amanakis; +Cc: Pete Heist, Cake List
On 3 July 2018 00:07:48 CEST, Georgios Amanakis <gamanakis@gmail.com> wrote:
>I can reliably achieve 40-70gbit/s using veth on Xeons and i7s, but I
>cannot reproduce the crash.
Thanks for testing!
My hunch is that this has something to do with the way mlx5 uses multiple receive queues (and thus multiple CPUs). Which is probably different from veth...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 22:12 ` Dave Taht
@ 2018-07-02 23:48 ` Georgios Amanakis
0 siblings, 0 replies; 77+ messages in thread
From: Georgios Amanakis @ 2018-07-02 23:48 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, Cake List
[-- Attachment #1: Type: text/plain, Size: 433 bytes --]
On Mon, 2018-07-02 at 15:12 -0700, Dave Taht wrote:
>
> Both with and without shaping?
Yes, both with and without shaping.
> ipv4? ipv6? Both?
ipv4
> Which flent test was it?
tcp_4down, mtu 1500, ecn both on(1,2) and off(0)
I think I get the fastest results by using "besteffort" and "flows".
See the attached scripts, I run them as:
# vim vcake.sh --> modify cake parameters, ecn
# ./vsetup.sh
# ./mm2.sh title_for_test
George
[-- Attachment #2: setup.tgz --]
[-- Type: application/x-compressed-tar, Size: 1120 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 22:23 ` Toke Høiland-Jørgensen
@ 2018-07-03 7:35 ` Pete Heist
2018-07-03 9:18 ` Jonathan Morton
1 sibling, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-03 7:35 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Georgios Amanakis, Cake List
> On Jul 3, 2018, at 12:23 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> My hunch is that this has something to do with the way mlx5 uses multiple receive queues (and thus multiple CPUs). Which is probably different from veth...
It doesn’t reproduce between between 2x APU2c4 (4 hardware queues and igb driver w/ mq support), but that’s 1Gbit, plus this Mellanox thing appears to have all sorts of hardware whatnots if this is the right device:
http://www.mellanox.com/related-docs/prod_silicon/PB_ConnectX-4_EN_IC.pdf
I noticed it has support for some link-layer congestion notification I’d not heard of until now (802.1Qau, https://www.ietf.org/proceedings/67/slides/tsvarea-2.pdf).
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-02 22:23 ` Toke Høiland-Jørgensen
2018-07-03 7:35 ` Pete Heist
@ 2018-07-03 9:18 ` Jonathan Morton
2018-07-03 9:57 ` Pete Heist
2018-07-03 10:27 ` Toke Høiland-Jørgensen
1 sibling, 2 replies; 77+ messages in thread
From: Jonathan Morton @ 2018-07-03 9:18 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Georgios Amanakis, Cake List
> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> My hunch is that this has something to do with the way mlx5 uses multiple receive queues (and thus multiple CPUs). Which is probably different from veth...
At this stage I'm pretty confident it has nothing to do with Cake, and everything to do with the Mellanox hardware and driver. It does strike me that Linux' default handling of multiqueue hardware doesn't map very well to the qdisc interface.
- Jonathan Morton
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-03 9:18 ` Jonathan Morton
@ 2018-07-03 9:57 ` Pete Heist
2018-07-03 10:27 ` Toke Høiland-Jørgensen
1 sibling, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-03 9:57 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Toke Høiland-Jørgensen, Cake List
> On Jul 3, 2018, at 11:18 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> My hunch is that this has something to do with the way mlx5 uses multiple receive queues (and thus multiple CPUs). Which is probably different from veth...
>
> At this stage I'm pretty confident it has nothing to do with Cake, and everything to do with the Mellanox hardware and driver. It does strike me that Linux' default handling of multiqueue hardware doesn't map very well to the qdisc interface.
That’s looking highly likely to me too. I’ll stop trying to repro this unless we come up with something new to try.
I wonder if enabling the “lockless qdisc” support (https://lwn.net/Articles/698135/) we discussed late last year would change anything. It should work either way, even with a single qdisc lock, but just speculating, at those speeds with multiple cores competing for a single qdisc lock I wonder if we’re exposing a problem in the driver or somewhere else.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-03 9:18 ` Jonathan Morton
2018-07-03 9:57 ` Pete Heist
@ 2018-07-03 10:27 ` Toke Høiland-Jørgensen
2018-07-03 10:41 ` Pete Heist
2018-07-05 22:31 ` Toke Høiland-Jørgensen
1 sibling, 2 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-03 10:27 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Georgios Amanakis, Cake List
Jonathan Morton <chromatix99@gmail.com> writes:
>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> My hunch is that this has something to do with the way mlx5 uses
>> multiple receive queues (and thus multiple CPUs). Which is probably
>> different from veth...
>
> At this stage I'm pretty confident it has nothing to do with Cake, and
> everything to do with the Mellanox hardware and driver. It does strike
> me that Linux' default handling of multiqueue hardware doesn't map
> very well to the qdisc interface.
Well, it doesn't happen with fq_codel, so even if it is a driver bug, it
is being triggered by cake specifically...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-03 10:27 ` Toke Høiland-Jørgensen
@ 2018-07-03 10:41 ` Pete Heist
2018-07-05 22:31 ` Toke Høiland-Jørgensen
1 sibling, 0 replies; 77+ messages in thread
From: Pete Heist @ 2018-07-03 10:41 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Jonathan Morton, Cake List
> On Jul 3, 2018, at 12:27 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Jonathan Morton <chromatix99@gmail.com> writes:
>
>>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> My hunch is that this has something to do with the way mlx5 uses
>>> multiple receive queues (and thus multiple CPUs). Which is probably
>>> different from veth...
>>
>> At this stage I'm pretty confident it has nothing to do with Cake, and
>> everything to do with the Mellanox hardware and driver. It does strike
>> me that Linux' default handling of multiqueue hardware doesn't map
>> very well to the qdisc interface.
>
> Well, it doesn't happen with fq_codel, so even if it is a driver bug, it
> is being triggered by cake specifically…
Does fq_codel run at the full 100gbit on your hardware?
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-03 10:27 ` Toke Høiland-Jørgensen
2018-07-03 10:41 ` Pete Heist
@ 2018-07-05 22:31 ` Toke Høiland-Jørgensen
2018-07-05 23:48 ` Georgios Amanakis
2018-07-06 8:55 ` Pete Heist
1 sibling, 2 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-05 22:31 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Georgios Amanakis, Cake List
Toke Høiland-Jørgensen <toke@toke.dk> writes:
> Jonathan Morton <chromatix99@gmail.com> writes:
>
>>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>>
>>> My hunch is that this has something to do with the way mlx5 uses
>>> multiple receive queues (and thus multiple CPUs). Which is probably
>>> different from veth...
>>
>> At this stage I'm pretty confident it has nothing to do with Cake, and
>> everything to do with the Mellanox hardware and driver. It does strike
>> me that Linux' default handling of multiqueue hardware doesn't map
>> very well to the qdisc interface.
>
> Well, it doesn't happen with fq_codel, so even if it is a driver bug, it
> is being triggered by cake specifically...
Right, so finally got some time to investigate this further.
I suspected that cake_dequeue() was looping forever, so I added some
debug statements to investigate this; and turns out I was right. Using
the debug patch below, in unlimited mode I get loop aborts on loop 'i'
for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It
happens pretty reliably, but only when I load up the link sufficiently
(need 4-6 TCP flows which get ~50 Gbps of total throughput).
The weird thing is that what appears to be happening, is that cake
somehow gets into a state where sch->q.qlen is >0 while all tin backlogs
are 0. I have no clue how this happens; as far as I can tell, all
changes to tin_backlog are paired with a change to q.qlen. The only
thing outside of cake itself that modifies q.qlen is peek(), which is
not being used here.
I'm giving up for tonight; if anyone else has any ideas, I'm all ears.
-Toke
Sample debug output:
[ 5456.068281] Loop counter i hit 100k; aborting! i 100001 j 0 k 180 l 3 m 0 qlen 2 qbkllog 33184 tin 2 deficit 172 tot backlog 0
With this debug patch:
@@ -1892,6 +1892,20 @@ static struct sk_buff *cake_dequeue(struct Qdisc *sch)
u64 delay;
u32 len;
+ int i=0,j=0,k=0,l=0,m=0;
+
+#define COUNT_LOOP(v) do { \
+ if (++v > 100000) { \
+ int tot_bkl = 0; \
+ struct cake_tin_data *t; \
+ int n; \
+ for(n=0,t = q->tins; n < CAKE_MAX_TINS; n++,t++) \
+ tot_bkl += t->tin_backlog; \
+ net_warn_ratelimited("Loop counter " #v " hit 100k; aborting! i %d j %d k %d l %d m %d qlen %d qbkllog %d tin %d deficit %d tot backlog %d", i, j, k, l, m, sch->q.qlen, sch->qstats.backlog, q->cur_tin, b->tin_deficit, tot_bkl); \
+ return NULL; \
+ } \
+ } while(0);
+
begin:
if (!sch->q.qlen)
return NULL;
@@ -1912,6 +1926,7 @@ begin:
/* In unlimited mode, can't rely on shaper timings, just balance
* with DRR
*/
+ i=0;
while (b->tin_deficit < 0 ||
!(b->sparse_flow_count + b->bulk_flow_count)) {
if (b->tin_deficit <= 0)
@@ -1923,6 +1938,7 @@ begin:
q->cur_tin = 0;
b = q->tins;
}
+ COUNT_LOOP(i);
}
} else {
/* In shaped mode, choose:
@@ -1960,8 +1976,10 @@ retry:
head = &b->old_flows;
if (unlikely(list_empty(head))) {
head = &b->decaying_flows;
- if (unlikely(list_empty(head)))
+ if (unlikely(list_empty(head))) {
+ COUNT_LOOP(j);
goto begin;
+ }
}
}
}
@@ -2008,6 +2026,7 @@ retry:
flow->set = CAKE_SET_SPARSE_WAIT;
}
}
+ COUNT_LOOP(k);
goto retry;
}
@@ -2050,6 +2069,7 @@ retry:
srchost->srchost_refcnt--;
dsthost->dsthost_refcnt--;
}
+ COUNT_LOOP(l);
goto begin;
}
@@ -2075,6 +2095,8 @@ retry:
kfree_skb(skb);
if (q->rate_flags & CAKE_FLAG_INGRESS)
goto retry;
+
+ COUNT_LOOP(m);
}
b->tin_ecn_mark += !!flow->cvars.ecn_marked;
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-05 22:31 ` Toke Høiland-Jørgensen
@ 2018-07-05 23:48 ` Georgios Amanakis
2018-07-06 1:21 ` Dave Taht
2018-07-06 8:55 ` Pete Heist
1 sibling, 1 reply; 77+ messages in thread
From: Georgios Amanakis @ 2018-07-05 23:48 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Jonathan Morton, Cake List
[-- Attachment #1: Type: text/plain, Size: 5350 bytes --]
I am going to give it a try, with your patch applied tonight and report.
Thank you!
George
On Thu, Jul 5, 2018, 6:31 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Toke Høiland-Jørgensen <toke@toke.dk> writes:
>
> > Jonathan Morton <chromatix99@gmail.com> writes:
> >
> >>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk>
> wrote:
> >>>
> >>> My hunch is that this has something to do with the way mlx5 uses
> >>> multiple receive queues (and thus multiple CPUs). Which is probably
> >>> different from veth...
> >>
> >> At this stage I'm pretty confident it has nothing to do with Cake, and
> >> everything to do with the Mellanox hardware and driver. It does strike
> >> me that Linux' default handling of multiqueue hardware doesn't map
> >> very well to the qdisc interface.
> >
> > Well, it doesn't happen with fq_codel, so even if it is a driver bug, it
> > is being triggered by cake specifically...
>
> Right, so finally got some time to investigate this further.
>
> I suspected that cake_dequeue() was looping forever, so I added some
> debug statements to investigate this; and turns out I was right. Using
> the debug patch below, in unlimited mode I get loop aborts on loop 'i'
> for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It
> happens pretty reliably, but only when I load up the link sufficiently
> (need 4-6 TCP flows which get ~50 Gbps of total throughput).
>
> The weird thing is that what appears to be happening, is that cake
> somehow gets into a state where sch->q.qlen is >0 while all tin backlogs
> are 0. I have no clue how this happens; as far as I can tell, all
> changes to tin_backlog are paired with a change to q.qlen. The only
> thing outside of cake itself that modifies q.qlen is peek(), which is
> not being used here.
>
> I'm giving up for tonight; if anyone else has any ideas, I'm all ears.
>
> -Toke
>
> Sample debug output:
>
> [ 5456.068281] Loop counter i hit 100k; aborting! i 100001 j 0 k 180 l 3 m
> 0 qlen 2 qbkllog 33184 tin 2 deficit 172 tot backlog 0
>
> With this debug patch:
>
> @@ -1892,6 +1892,20 @@ static struct sk_buff *cake_dequeue(struct Qdisc
> *sch)
> u64 delay;
> u32 len;
>
> + int i=0,j=0,k=0,l=0,m=0;
> +
> +#define COUNT_LOOP(v) do { \
> + if (++v > 100000) { \
> + int tot_bkl = 0; \
> + struct cake_tin_data *t; \
> + int n; \
> + for(n=0,t = q->tins; n < CAKE_MAX_TINS; n++,t++)
> \
> + tot_bkl += t->tin_backlog; \
> + net_warn_ratelimited("Loop counter " #v " hit
> 100k; aborting! i %d j %d k %d l %d m %d qlen %d qbkllog %d tin %d deficit
> %d tot backlog %d", i, j, k, l, m, sch->q.qlen, sch->qstats.backlog,
> q->cur_tin, b->tin_deficit, tot_bkl); \
> + return NULL; \
> + } \
> + } while(0);
> +
> begin:
> if (!sch->q.qlen)
> return NULL;
> @@ -1912,6 +1926,7 @@ begin:
> /* In unlimited mode, can't rely on shaper timings, just
> balance
> * with DRR
> */
> + i=0;
> while (b->tin_deficit < 0 ||
> !(b->sparse_flow_count + b->bulk_flow_count)) {
> if (b->tin_deficit <= 0)
> @@ -1923,6 +1938,7 @@ begin:
> q->cur_tin = 0;
> b = q->tins;
> }
> + COUNT_LOOP(i);
> }
> } else {
> /* In shaped mode, choose:
> @@ -1960,8 +1976,10 @@ retry:
> head = &b->old_flows;
> if (unlikely(list_empty(head))) {
> head = &b->decaying_flows;
> - if (unlikely(list_empty(head)))
> + if (unlikely(list_empty(head))) {
> + COUNT_LOOP(j);
> goto begin;
> + }
> }
> }
> }
> @@ -2008,6 +2026,7 @@ retry:
> flow->set = CAKE_SET_SPARSE_WAIT;
> }
> }
> + COUNT_LOOP(k);
> goto retry;
> }
>
> @@ -2050,6 +2069,7 @@ retry:
> srchost->srchost_refcnt--;
> dsthost->dsthost_refcnt--;
> }
> + COUNT_LOOP(l);
> goto begin;
> }
>
> @@ -2075,6 +2095,8 @@ retry:
> kfree_skb(skb);
> if (q->rate_flags & CAKE_FLAG_INGRESS)
> goto retry;
> +
> + COUNT_LOOP(m);
> }
>
> b->tin_ecn_mark += !!flow->cvars.ecn_marked;
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 7083 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-05 23:48 ` Georgios Amanakis
@ 2018-07-06 1:21 ` Dave Taht
2018-07-06 2:55 ` George Amanakis
2018-07-06 9:21 ` Toke Høiland-Jørgensen
0 siblings, 2 replies; 77+ messages in thread
From: Dave Taht @ 2018-07-06 1:21 UTC (permalink / raw)
To: George Amanakis; +Cc: Toke Høiland-Jørgensen, Cake List
0 length packet? maybe coming out of the new GSO/GRO code?
check truesize also?
On Thu, Jul 5, 2018 at 4:48 PM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> I am going to give it a try, with your patch applied tonight and report.
> Thank you!
>
> George
>
> On Thu, Jul 5, 2018, 6:31 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Toke Høiland-Jørgensen <toke@toke.dk> writes:
>>
>> > Jonathan Morton <chromatix99@gmail.com> writes:
>> >
>> >>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> >>>
>> >>> My hunch is that this has something to do with the way mlx5 uses
>> >>> multiple receive queues (and thus multiple CPUs). Which is probably
>> >>> different from veth...
>> >>
>> >> At this stage I'm pretty confident it has nothing to do with Cake, and
>> >> everything to do with the Mellanox hardware and driver. It does strike
>> >> me that Linux' default handling of multiqueue hardware doesn't map
>> >> very well to the qdisc interface.
>> >
>> > Well, it doesn't happen with fq_codel, so even if it is a driver bug, it
>> > is being triggered by cake specifically...
>>
>> Right, so finally got some time to investigate this further.
>>
>> I suspected that cake_dequeue() was looping forever, so I added some
>> debug statements to investigate this; and turns out I was right. Using
>> the debug patch below, in unlimited mode I get loop aborts on loop 'i'
>> for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It
>> happens pretty reliably, but only when I load up the link sufficiently
>> (need 4-6 TCP flows which get ~50 Gbps of total throughput).
>>
>> The weird thing is that what appears to be happening, is that cake
>> somehow gets into a state where sch->q.qlen is >0 while all tin backlogs
>> are 0. I have no clue how this happens; as far as I can tell, all
>> changes to tin_backlog are paired with a change to q.qlen. The only
>> thing outside of cake itself that modifies q.qlen is peek(), which is
>> not being used here.
>>
>> I'm giving up for tonight; if anyone else has any ideas, I'm all ears.
>>
>> -Toke
>>
>> Sample debug output:
>>
>> [ 5456.068281] Loop counter i hit 100k; aborting! i 100001 j 0 k 180 l 3 m 0 qlen 2 qbkllog 33184 tin 2 deficit 172 tot backlog 0
>>
>> With this debug patch:
>>
>> @@ -1892,6 +1892,20 @@ static struct sk_buff *cake_dequeue(struct Qdisc *sch)
>> u64 delay;
>> u32 len;
>>
>> + int i=0,j=0,k=0,l=0,m=0;
>> +
>> +#define COUNT_LOOP(v) do { \
>> + if (++v > 100000) { \
>> + int tot_bkl = 0; \
>> + struct cake_tin_data *t; \
>> + int n; \
>> + for(n=0,t = q->tins; n < CAKE_MAX_TINS; n++,t++) \
>> + tot_bkl += t->tin_backlog; \
>> + net_warn_ratelimited("Loop counter " #v " hit 100k; aborting! i %d j %d k %d l %d m %d qlen %d qbkllog %d tin %d deficit %d tot backlog %d", i, j, k, l, m, sch->q.qlen, sch->qstats.backlog, q->cur_tin, b->tin_deficit, tot_bkl); \
>> + return NULL; \
>> + } \
>> + } while(0);
>> +
>> begin:
>> if (!sch->q.qlen)
>> return NULL;
>> @@ -1912,6 +1926,7 @@ begin:
>> /* In unlimited mode, can't rely on shaper timings, just balance
>> * with DRR
>> */
>> + i=0;
>> while (b->tin_deficit < 0 ||
>> !(b->sparse_flow_count + b->bulk_flow_count)) {
>> if (b->tin_deficit <= 0)
>> @@ -1923,6 +1938,7 @@ begin:
>> q->cur_tin = 0;
>> b = q->tins;
>> }
>> + COUNT_LOOP(i);
>> }
>> } else {
>> /* In shaped mode, choose:
>> @@ -1960,8 +1976,10 @@ retry:
>> head = &b->old_flows;
>> if (unlikely(list_empty(head))) {
>> head = &b->decaying_flows;
>> - if (unlikely(list_empty(head)))
>> + if (unlikely(list_empty(head))) {
>> + COUNT_LOOP(j);
>> goto begin;
>> + }
>> }
>> }
>> }
>> @@ -2008,6 +2026,7 @@ retry:
>> flow->set = CAKE_SET_SPARSE_WAIT;
>> }
>> }
>> + COUNT_LOOP(k);
>> goto retry;
>> }
>>
>> @@ -2050,6 +2069,7 @@ retry:
>> srchost->srchost_refcnt--;
>> dsthost->dsthost_refcnt--;
>> }
>> + COUNT_LOOP(l);
>> goto begin;
>> }
>>
>> @@ -2075,6 +2095,8 @@ retry:
>> kfree_skb(skb);
>> if (q->rate_flags & CAKE_FLAG_INGRESS)
>> goto retry;
>> +
>> + COUNT_LOOP(m);
>> }
>>
>> b->tin_ecn_mark += !!flow->cvars.ecn_marked;
>>
>>
>>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 1:21 ` Dave Taht
@ 2018-07-06 2:55 ` George Amanakis
2018-07-06 3:06 ` George Amanakis
2018-07-06 9:21 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: George Amanakis @ 2018-07-06 2:55 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, Cake List
With Toke's patch I can see these warnings using veth, too.
This is on a 4.14.53 kernel, using sch_cake/master branch.
However I can see them only if the counter is set to throw a warning at
a lower limit, i.e. 1k instead of 100k.
This is what I get:
cake in unlimited mode --> aborts in loop 'k':
[ +0.316779] Loop counter k hit 1k; i 0 j 0 k 1001 l 4 m 0 qlen 1
qbkllog 1514 tin 0 deficit 232 tot backlog 1514
[ +0.000347] Loop counter k hit 1k; i 0 j 0 k 1001 l 1 m 0 qlen 1
qbkllog 56018 tin 0 deficit 216 tot backlog 56018
[ +0.003706] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 2
qbkllog 136260 tin 0 deficit 310 tot backlog 136260
[ +0.000042] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 3
qbkllog 165026 tin 0 deficit 16 tot backlog 165026
[ +0.000042] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 2
qbkllog 56018 tin 0 deficit 702 tot backlog 56018
[ +0.000894] Loop counter k hit 1k; i 0 j 0 k 1001 l 1 m 0 qlen 1
qbkllog 43906 tin 0 deficit 264 tot backlog 43906
[ +0.000596] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 5
qbkllog 254352 tin 0 deficit 724 tot backlog 254352
[ +0.000032] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 3
qbkllog 118092 tin 0 deficit 518 tot backlog 118092
[ +0.001537] Loop counter k hit 1k; i 0 j 0 k 1001 l 1 m 0 qlen 3
qbkllog 190764 tin 0 deficit 106 tot backlog 190764
[ +0.000026] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 1
qbkllog 59046 tin 0 deficit 792 tot backlog 59046
cake with bandwidth limited @50gbit --> aborts in loop 'k', again:
[ +0.000002] Loop counter k hit 1k; i 0 j 0 k 1001 l 4 m 0 qlen 1
qbkllog 68130 tin 0 deficit -5599742 tot backlog 68130
[ +0.000127] Loop counter k hit 1k; i 0 j 0 k 1001 l 4 m 0 qlen 2
qbkllog 101438 tin 0 deficit -5976728 tot backlog 101438
[ +0.000552] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 2
qbkllog 16654 tin 0 deficit -7303854 tot backlog 16654
[ +0.000888] Loop counter k hit 1k; i 0 j 0 k 1001 l 1 m 0 qlen 1
qbkllog 68130 tin 0 deficit -10334948 tot backlog 68130
[ +0.000565] Loop counter k hit 1k; i 0 j 0 k 1001 l 1 m 0 qlen 2
qbkllog 136260 tin 0 deficit -12733190 tot backlog 136260
[ +0.000055] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 3
qbkllog 136260 tin 0 deficit -12930010 tot backlog 136260
[ +0.000354] Loop counter k hit 1k; i 0 j 0 k 1001 l 2 m 0 qlen 4
qbkllog 139288 tin 0 deficit -13950446 tot backlog 139288
[ +0.000515] Loop counter k hit 1k; i 0 j 0 k 1001 l 4 m 0 qlen 1
qbkllog 7570 tin 0 deficit -15529548 tot backlog 7570
[ +0.000068] Loop counter k hit 1k; i 0 j 0 k 1001 l 4 m 0 qlen 1
qbkllog 60560 tin 0 deficit -15749078 tot backlog 60560
[ +0.000766] Loop counter k hit 1k; i 0 j 0 k 1001 l 3 m 0 qlen 1
qbkllog 68130 tin 0 deficit -18088208 tot backlog 68130
On 7/5/2018 9:21 PM, Dave Taht wrote:
> 0 length packet? maybe coming out of the new GSO/GRO code?
When cake is operating in bandwidth limited mode >1gbit, GSO/GRO code
should be bypassed, or am I wrong?
George
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 2:55 ` George Amanakis
@ 2018-07-06 3:06 ` George Amanakis
2018-07-06 9:22 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: George Amanakis @ 2018-07-06 3:06 UTC (permalink / raw)
To: Dave Taht; +Cc: Toke Høiland-Jørgensen, Cake List
I guess the aborts I reported in loop 'k' are normal, just denoting that
a flow's deficit is <=0.
If I increase the counter to >10k, I don't see any warnings anymore.
George
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-05 22:31 ` Toke Høiland-Jørgensen
2018-07-05 23:48 ` Georgios Amanakis
@ 2018-07-06 8:55 ` Pete Heist
2018-07-06 9:29 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-06 8:55 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Jonathan Morton, Cake List
> On Jul 6, 2018, at 12:31 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> I suspected that cake_dequeue() was looping forever, so I added some
> debug statements to investigate this; and turns out I was right. Using
> the debug patch below, in unlimited mode I get loop aborts on loop 'i'
> for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It
> happens pretty reliably, but only when I load up the link sufficiently
> (need 4-6 TCP flows which get ~50 Gbps of total throughput).
>
> The weird thing is that what appears to be happening, is that cake
> somehow gets into a state where sch->q.qlen is >0 while all tin backlogs
> are 0. I have no clue how this happens; as far as I can tell, all
> changes to tin_backlog are paired with a change to q.qlen. The only
> thing outside of cake itself that modifies q.qlen is peek(), which is
> not being used here.
I’m not sure I’ll be much help, but here are the thoughts I start having with loop ‘i’:
- is tin_deficit overflowing at these rates? at 50gbit, 2^31-1 bytes happen in 344 ms (involuntary chuckle)
- what’s the value of tin_quantum_band here? but I suspect it’s ok.
- I’m assuming sparse_flow_count + bulk_flow_count wouldn’t be 0…
Dave’s point about the possibility of 0 length packets could also get sch->q.qlen and tin_backlog out of sync.
I’ll still look at loop ‘l’, but it starting with while(1) does open a door of sorts… ;)
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 1:21 ` Dave Taht
2018-07-06 2:55 ` George Amanakis
@ 2018-07-06 9:21 ` Toke Høiland-Jørgensen
1 sibling, 0 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 9:21 UTC (permalink / raw)
To: Dave Taht, George Amanakis; +Cc: Cake List
Dave Taht <dave.taht@gmail.com> writes:
> 0 length packet? maybe coming out of the new GSO/GRO code?
Nope, these are big juicy GRO packets; the global backlog is also large
(qlen 2 backlog 33184 in the example I posted). It's not actually the
tin backlog that the loops check either, I just used it as a way to see
that the tins are all empty even though the global qlen is not...
I'm going to go investigate some more...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 3:06 ` George Amanakis
@ 2018-07-06 9:22 ` Toke Høiland-Jørgensen
0 siblings, 0 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 9:22 UTC (permalink / raw)
To: George Amanakis, Dave Taht; +Cc: Cake List
George Amanakis <gamanakis@gmail.com> writes:
> I guess the aborts I reported in loop 'k' are normal, just denoting that
> a flow's deficit is <=0.
> If I increase the counter to >10k, I don't see any warnings anymore.
Yeah, some looping is fine, it's the infinite loops we want to avoid ;)
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 8:55 ` Pete Heist
@ 2018-07-06 9:29 ` Toke Høiland-Jørgensen
2018-07-06 10:00 ` Pete Heist
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 9:29 UTC (permalink / raw)
To: Pete Heist; +Cc: Jonathan Morton, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 6, 2018, at 12:31 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> I suspected that cake_dequeue() was looping forever, so I added some
>> debug statements to investigate this; and turns out I was right. Using
>> the debug patch below, in unlimited mode I get loop aborts on loop 'i'
>> for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It
>> happens pretty reliably, but only when I load up the link sufficiently
>> (need 4-6 TCP flows which get ~50 Gbps of total throughput).
>>
>> The weird thing is that what appears to be happening, is that cake
>> somehow gets into a state where sch->q.qlen is >0 while all tin backlogs
>> are 0. I have no clue how this happens; as far as I can tell, all
>> changes to tin_backlog are paired with a change to q.qlen. The only
>> thing outside of cake itself that modifies q.qlen is peek(), which is
>> not being used here.
>
> I’m not sure I’ll be much help, but here are the thoughts I start having with loop ‘i’:
>
> - is tin_deficit overflowing at these rates? at 50gbit, 2^31-1 bytes
> happen in 344 ms (involuntary chuckle)
> - what’s the value of tin_quantum_band here? but I suspect it’s ok.
I thought about overflows, but I don't get any "weird" values, and
everything ends up back at zero when the flows stop. And it's not
actually tin_backlog that's causing the looping...
> - I’m assuming sparse_flow_count + bulk_flow_count wouldn’t be 0…
Yeah, they are; that's why it keeps looping. I've been looking at both
tin_backlog and the *_flow_count vars as different ways of checking
whether the tins are actually empty... they are all 0 when this happens.
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 9:29 ` Toke Høiland-Jørgensen
@ 2018-07-06 10:00 ` Pete Heist
2018-07-06 10:46 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-06 10:00 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Jonathan Morton, Cake List
[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]
> On Jul 6, 2018, at 11:29 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> writes:
>
>> - is tin_deficit overflowing at these rates? at 50gbit, 2^31-1 bytes
>> happen in 344 ms (involuntary chuckle)
>> - what’s the value of tin_quantum_band here? but I suspect it’s ok.
>
> I thought about overflows, but I don't get any "weird" values, and
> everything ends up back at zero when the flows stop. And it's not
> actually tin_backlog that's causing the looping…
Ok, I think tin_deficit is meant here, esp. in light of what follows regarding *_flow_count.
Once we do get past this infinite loop, which it sounds like is not caused by overflow here, I guess it’s still worth reviewing whether tin_backlog or other values _could_ overflow in certain conditions. In your case rtt is probably low, but what if it weren’t? Adding delay with netem might coax something out. In fact, I’ll see if I can add some delay to the 30-40gbit local testing that I _can_ do to see if I notice anything...
>> - I’m assuming sparse_flow_count + bulk_flow_count wouldn’t be 0…
>
> Yeah, they are; that's why it keeps looping. I've been looking at both
> tin_backlog and the *_flow_count vars as different ways of checking
> whether the tins are actually empty... they are all 0 when this happens.
Aha, ok. It does look physically possible for these to both be 0 since there appear to be cases where one is decremented without the other being incremented. That _all_ *_flow_count vars are 0 seems strange logically. I’ll leave this alone now though as don’t yet understand what the values represent well enough… :)
[-- Attachment #2: Type: text/html, Size: 9209 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 10:00 ` Pete Heist
@ 2018-07-06 10:46 ` Toke Høiland-Jørgensen
2018-07-06 11:33 ` Toke Høiland-Jørgensen
0 siblings, 1 reply; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 10:46 UTC (permalink / raw)
To: Pete Heist; +Cc: Jonathan Morton, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 6, 2018, at 11:29 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> writes:
>>
>>> - is tin_deficit overflowing at these rates? at 50gbit, 2^31-1 bytes
>>> happen in 344 ms (involuntary chuckle)
>>> - what’s the value of tin_quantum_band here? but I suspect it’s ok.
>>
>> I thought about overflows, but I don't get any "weird" values, and
>> everything ends up back at zero when the flows stop. And it's not
>> actually tin_backlog that's causing the looping…
>
> Ok, I think tin_deficit is meant here, esp. in light of what follows regarding *_flow_count.
>
> Once we do get past this infinite loop, which it sounds like is not
> caused by overflow here, I guess it’s still worth reviewing whether
> tin_backlog or other values _could_ overflow in certain conditions.
There's a global memory limit which limits the backlog. So no overflows
as long as the accounting is working correctly :)
>> Yeah, they are; that's why it keeps looping. I've been looking at both
>> tin_backlog and the *_flow_count vars as different ways of checking
>> whether the tins are actually empty... they are all 0 when this happens.
>
> Aha, ok. It does look physically possible for these to both be 0 since
> there appear to be cases where one is decremented without the other
> being incremented. That _all_ *_flow_count vars are 0 seems strange
> logically. I’ll leave this alone now though as don’t yet understand
> what the values represent well enough… :)
Yeah, that state machine is a bit too complicated for my taste as well;
not sure it's actually incorrect, though...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 10:46 ` Toke Høiland-Jørgensen
@ 2018-07-06 11:33 ` Toke Høiland-Jørgensen
2018-07-06 11:43 ` Jonathan Morton
2018-07-06 11:58 ` Pete Heist
0 siblings, 2 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 11:33 UTC (permalink / raw)
To: Pete Heist; +Cc: Jonathan Morton, Cake List
AHA! Found the culprit!
The bulk dequeue mechanism in sch_generic.c will dequeue a bunch of
packets at once, then check if they belong on the same hardware txq. If
they don't, they will be put back on a separate queue in the qdisc
structure (sch->skb_bad_txq), and the qlen will be increased, without
telling the qdisc about it.
This obviously only happens on hardware with multiple TXQs, which is why
the bug doesn't happen on veth.
Now, to figure out what to do about it...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 11:33 ` Toke Høiland-Jørgensen
@ 2018-07-06 11:43 ` Jonathan Morton
2018-07-06 11:48 ` Toke Høiland-Jørgensen
2018-07-06 11:58 ` Pete Heist
1 sibling, 1 reply; 77+ messages in thread
From: Jonathan Morton @ 2018-07-06 11:43 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Pete Heist, Cake List
> On 6 Jul, 2018, at 2:33 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Now, to figure out what to do about it...
Okay, if qlen can be manipulated outside Cake, that breaks an assumption I made. Let me see what I can do about changing the decisions involving it.
- Jonathan Morton
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 11:43 ` Jonathan Morton
@ 2018-07-06 11:48 ` Toke Høiland-Jørgensen
0 siblings, 0 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 11:48 UTC (permalink / raw)
To: Jonathan Morton; +Cc: Pete Heist, Cake List
Jonathan Morton <chromatix99@gmail.com> writes:
>> On 6 Jul, 2018, at 2:33 pm, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Now, to figure out what to do about it...
>
> Okay, if qlen can be manipulated outside Cake, that breaks an
> assumption I made. Let me see what I can do about changing the
> decisions involving it.
Yup. The easy fix is to just add an internal counter; I just tried
adding a tot_qlen to cake_sched_data that mirrors sch->q.qlen. This
fixes the issue. But if you have a better idea, by all means go ahead :)
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 11:33 ` Toke Høiland-Jørgensen
2018-07-06 11:43 ` Jonathan Morton
@ 2018-07-06 11:58 ` Pete Heist
2018-07-06 12:04 ` Toke Høiland-Jørgensen
1 sibling, 1 reply; 77+ messages in thread
From: Pete Heist @ 2018-07-06 11:58 UTC (permalink / raw)
To: Toke Høiland-Jørgensen; +Cc: Jonathan Morton, Cake List
> On Jul 6, 2018, at 1:33 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> AHA! Found the culprit!
>
> The bulk dequeue mechanism in sch_generic.c will dequeue a bunch of
> packets at once, then check if they belong on the same hardware txq. If
> they don't, they will be put back on a separate queue in the qdisc
> structure (sch->skb_bad_txq), and the qlen will be increased, without
> telling the qdisc about it.
Solid, nice work!
> This obviously only happens on hardware with multiple TXQs, which is why
> the bug doesn't happen on veth.
It would be nice if veth were mq capable.
For whatever reason, I didn’t see this on my i210at’s (1gbit ethernet with 4 transmit and 4 receive queues).
I’m now playing with netem, cake and veth for the first time (two namespaces with netem as the parent qdisc to cake for each namespace). I’ve gotten the setup not to lock up in an infinite loop but to occasionally stop passing traffic sometimes after a netperf test. This could easily be a problem specific to netns though, so I’ll be playing with it some more and will post if I can narrow it down to something specific.
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: [Cake] cake at 60gbit
2018-07-06 11:58 ` Pete Heist
@ 2018-07-06 12:04 ` Toke Høiland-Jørgensen
0 siblings, 0 replies; 77+ messages in thread
From: Toke Høiland-Jørgensen @ 2018-07-06 12:04 UTC (permalink / raw)
To: Pete Heist; +Cc: Jonathan Morton, Cake List
Pete Heist <pete@heistp.net> writes:
>> On Jul 6, 2018, at 1:33 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> AHA! Found the culprit!
>>
>> The bulk dequeue mechanism in sch_generic.c will dequeue a bunch of
>> packets at once, then check if they belong on the same hardware txq. If
>> they don't, they will be put back on a separate queue in the qdisc
>> structure (sch->skb_bad_txq), and the qlen will be increased, without
>> telling the qdisc about it.
>
> Solid, nice work!
Thanks :)
>> This obviously only happens on hardware with multiple TXQs, which is why
>> the bug doesn't happen on veth.
>
>
> It would be nice if veth were mq capable.
>
> For whatever reason, I didn’t see this on my i210at’s (1gbit ethernet
> with 4 transmit and 4 receive queues).
Well, you have to hit the exact conditions; i.e., a bulk dequeue that
happens to get a bunch of packets that hit different TX queues. So that
depends on both the TXQ hashing, and the queue state, number of flows
etc. I only get a handful of "lockups" (debug lines) on a 10-sec netperf
test with 6 flows.
> I’m now playing with netem, cake and veth for the first time (two
> namespaces with netem as the parent qdisc to cake for each namespace).
> I’ve gotten the setup not to lock up in an infinite loop but to
> occasionally stop passing traffic sometimes after a netperf test. This
> could easily be a problem specific to netns though, so I’ll be playing
> with it some more and will post if I can narrow it down to something
> specific.
Yay, more fun! :P
Please do see if you can narrow this down; it would be good to fix this
as well before we submit another version upstream...
-Toke
^ permalink raw reply [flat|nested] 77+ messages in thread
end of thread, other threads:[~2018-07-06 12:05 UTC | newest]
Thread overview: 77+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <mailman.381.1530347349.3512.cake@lists.bufferbloat.net>
2018-06-30 16:37 ` [Cake] Cake on openwrt - falling behind Georgios Amanakis
2018-06-30 17:26 ` Pete Heist
2018-06-30 18:09 ` Georgios Amanakis
2018-06-30 18:55 ` Kevin Darbyshire-Bryant
2018-06-30 19:57 ` Pete Heist
2018-06-30 20:58 ` Georgios Amanakis
2018-06-30 21:37 ` Pete Heist
2018-06-30 22:43 ` Pete Heist
2018-06-30 23:20 ` Pete Heist
[not found] ` <CACvFP_gkdAPKSEO7j9+cMPqTa-fJYd8XFEEBD6ZzLuVvaNwsvg@mail.gmail.com>
2018-07-01 2:37 ` [Cake] Fwd: " Georgios Amanakis
2018-07-01 7:18 ` [Cake] " Pete Heist
2018-07-01 13:48 ` Pete Heist
2018-07-01 14:02 ` Dave Taht
2018-07-01 15:30 ` Pete Heist
2018-07-01 14:38 ` Kevin Darbyshire-Bryant
2018-07-01 16:54 ` Pete Heist
2018-07-01 19:41 ` Kevin Darbyshire-Bryant
2018-07-02 10:19 ` Pete Heist
2018-07-02 11:38 ` Pete Heist
2018-07-02 11:59 ` Kevin Darbyshire-Bryant
2018-07-02 14:01 ` Pete Heist
2018-07-02 12:03 ` Toke Høiland-Jørgensen
2018-07-02 14:51 ` Pete Heist
2018-07-02 16:14 ` Toke Høiland-Jørgensen
2018-07-02 16:59 ` Kevin Darbyshire-Bryant
2018-07-02 17:04 ` Pete Heist
2018-07-02 17:12 ` Kevin Darbyshire-Bryant
2018-07-02 18:24 ` Pete Heist
2018-07-02 19:31 ` Toke Høiland-Jørgensen
2018-07-02 20:09 ` Pete Heist
2018-07-02 20:11 ` Toke Høiland-Jørgensen
2018-07-02 20:46 ` Pete Heist
[not found] ` <mailman.407.1530550780.3512.cake@lists.bufferbloat.net>
2018-07-02 17:50 ` Kevin Darbyshire-Bryant
2018-07-02 19:33 ` Toke Høiland-Jørgensen
2018-07-02 19:36 ` Kevin Darbyshire-Bryant
2018-07-02 19:39 ` Toke Høiland-Jørgensen
2018-07-02 20:03 ` [Cake] cake at 60gbit Dave Taht
2018-07-02 20:09 ` Toke Høiland-Jørgensen
2018-07-02 21:16 ` Pete Heist
2018-07-02 21:35 ` Toke Høiland-Jørgensen
2018-07-02 22:07 ` Georgios Amanakis
2018-07-02 22:12 ` Dave Taht
2018-07-02 23:48 ` Georgios Amanakis
2018-07-02 22:23 ` Toke Høiland-Jørgensen
2018-07-03 7:35 ` Pete Heist
2018-07-03 9:18 ` Jonathan Morton
2018-07-03 9:57 ` Pete Heist
2018-07-03 10:27 ` Toke Høiland-Jørgensen
2018-07-03 10:41 ` Pete Heist
2018-07-05 22:31 ` Toke Høiland-Jørgensen
2018-07-05 23:48 ` Georgios Amanakis
2018-07-06 1:21 ` Dave Taht
2018-07-06 2:55 ` George Amanakis
2018-07-06 3:06 ` George Amanakis
2018-07-06 9:22 ` Toke Høiland-Jørgensen
2018-07-06 9:21 ` Toke Høiland-Jørgensen
2018-07-06 8:55 ` Pete Heist
2018-07-06 9:29 ` Toke Høiland-Jørgensen
2018-07-06 10:00 ` Pete Heist
2018-07-06 10:46 ` Toke Høiland-Jørgensen
2018-07-06 11:33 ` Toke Høiland-Jørgensen
2018-07-06 11:43 ` Jonathan Morton
2018-07-06 11:48 ` Toke Høiland-Jørgensen
2018-07-06 11:58 ` Pete Heist
2018-07-06 12:04 ` Toke Høiland-Jørgensen
2018-07-02 18:39 ` [Cake] Cake on openwrt - falling behind Dave Taht
2018-07-02 19:11 ` Kevin Darbyshire-Bryant
2018-07-02 19:23 ` Toke Høiland-Jørgensen
2018-07-02 19:27 ` Dave Taht
2018-07-02 19:38 ` Toke Høiland-Jørgensen
2018-07-02 20:05 ` Toke Høiland-Jørgensen
2018-07-02 19:31 ` Pete Heist
[not found] ` <mailman.397.1530474091.3512.cake@lists.bufferbloat.net>
2018-07-01 23:55 ` Dave Taht
2018-07-02 0:05 ` Dave Taht
[not found] ` <mailman.392.1530455913.3512.cake@lists.bufferbloat.net>
2018-07-01 15:17 ` Jonathan Morton
[not found] ` <mailman.384.1530384918.3512.cake@lists.bufferbloat.net>
2018-07-01 9:46 ` Magnus Olsson
2018-07-01 12:34 ` Kevin Darbyshire-Bryant
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox