From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0063.outbound.protection.outlook.com [104.47.1.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8FA7E3B29E for ; Sun, 1 Jul 2018 10:38:32 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=darbyshire-bryant.me.uk; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Vugi0JAws+yHVMw7A+71Oaw1FbLn0JklBC5tIXPMKjs=; b=e2Fyi+D0qbouqa+cfIzNjPpXdDQhMo5GO69POwji0JeHzxfPeMP2jUT4hPukDlCq4HUbrfFvR1YlXpntNLqF+qn+ZARLq+FehawPlT8VzWTRWylS8btCUQcWungyz6+dBRcoSasmdoaiTjb+aOj3og7zFuS3EoT7n8TFgPTVnio= Received: from VI1PR07MB4254.eurprd07.prod.outlook.com (20.176.6.147) by VI1PR07MB1520.eurprd07.prod.outlook.com (10.165.238.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.930.10; Sun, 1 Jul 2018 14:38:29 +0000 Received: from VI1PR07MB4254.eurprd07.prod.outlook.com ([fe80::44fe:35ff:4978:fb9d]) by VI1PR07MB4254.eurprd07.prod.outlook.com ([fe80::44fe:35ff:4978:fb9d%2]) with mapi id 15.20.0930.016; Sun, 1 Jul 2018 14:38:29 +0000 From: Kevin Darbyshire-Bryant To: Pete Heist CC: Georgios Amanakis , "mbo2@uggenabben.se" , Cake List Subject: Re: [Cake] Cake on openwrt - falling behind Thread-Topic: [Cake] Cake on openwrt - falling behind Thread-Index: AQHUEQu6h1e7lfuzPEir60kIjWhE1aR6Ym6AgAAN0wA= Date: Sun, 1 Jul 2018 14:38:29 +0000 Message-ID: <94C9790F-E9BC-4D59-9845-17C305E4B910@darbyshire-bryant.me.uk> References: <6DF9A5E0-EFD5-4519-9889-BC0A7B9BD48E@darbyshire-bryant.me.uk> <1A8BA286-6B31-4581-86C9-6855AC28C245@heistp.net> <673EAD3F-AB09-4B90-88BB-5DCE0BD65534@heistp.net> <6FE8D434-01BE-41A1-BD6B-EFFD67AC8784@heistp.net> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [2a02:c7f:1231:2000::dc83] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR07MB1520; 7:dmTWCWLpatgz74zi49fBc6sQTgiXbEHb5TL1AM9luSrpF2DgY63Ksi/w1JABkQsGabiiy+avplzNvns+Ffw51fxaCYv6zrFN1guUm5Hg5IEQSG88Kp5VDMrdaDvL7BmhSm6ZMR+3GLtmvk8t1rDPxaIhXDCspPqCAeiNhZXRQlohf/IKj5G3SXyufj1iA3xQ/fZHam6HkRRlGgE22i07yzN6MOCu1JCgf98L+k+HDQ6g84Gd4WaqESuxn3+FKNNz x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 87f55754-1785-489f-344c-08d5df604fcf x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652040)(7021125)(8989117)(5600053)(711020)(4534165)(7022125)(4603075)(4627221)(201702281549075)(8990107)(7048125)(7024125)(7027125)(7028125)(7023125)(2017052603328)(7153060)(49563074)(7193020); SRVR:VI1PR07MB1520; x-ms-traffictypediagnostic: VI1PR07MB1520: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(3231254)(944501410)(52105095)(3002001)(149027)(150027)(6041310)(20161123564045)(20161123560045)(20161123562045)(20161123558120)(2016111802025)(6072148)(6043046)(201708071742011)(7699016); SRVR:VI1PR07MB1520; BCL:0; PCL:0; RULEID:; SRVR:VI1PR07MB1520; x-forefront-prvs: 07200C0526 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(39830400003)(366004)(136003)(346002)(376002)(189003)(199004)(105586002)(11346002)(14454004)(99286004)(14444005)(256004)(6512007)(6916009)(54906003)(36756003)(74482002)(99936001)(6436002)(46003)(6116002)(478600001)(486006)(476003)(2900100001)(2616005)(97736004)(446003)(86362001)(83716003)(106356001)(6506007)(53546011)(6246003)(2906002)(81166006)(186003)(81156014)(8936002)(76176011)(53936002)(6346003)(39060400002)(4326008)(33656002)(25786009)(102836004)(5660300001)(93886005)(229853002)(8676002)(5250100002)(82746002)(6486002)(305945005)(316002)(68736007)(7736002); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR07MB1520; H:VI1PR07MB4254.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: darbyshire-bryant.me.uk does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=kevin@darbyshire-bryant.me.uk; x-microsoft-antispam-message-info: Gf0jxDYHVCTfRfUcHH+EEbGMeUS2UWotiNbUl3v2o8bHAjnxxv9wI0XToZrLoLuSah5BsV51PYs/u7nmidIXp5eYZaVGSgDOm/wLsVJGCznHBO17GhwgGqqX1NNu1qb9Fps5HYl/t/nmSQvPt6DmDSF30aytKlRjSiIKk/6Mma4l6BP8wWCb4VoQyqFAA5R56LH25LNCbMHckPbhE/GsStUTfioZukB5qB14BD7IV7hJoQ4Fbhkxt0p5s6HQu+BbDGhLryOmtfSb0uiDbVsbtzIZvwdiO5jTa+hjxZyQ/5z0IgCt08fbZpCdjnTkodTEDn/hC/rzfKFtmPaja8RUCJR86+Q9l/KY6xBAC92hzCs= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/signed; boundary="Apple-Mail=_8574A3BF-3492-4346-8417-CBB6C8E043C1"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-OriginatorOrg: darbyshire-bryant.me.uk X-MS-Exchange-CrossTenant-Network-Message-Id: 87f55754-1785-489f-344c-08d5df604fcf X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2018 14:38:29.1314 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 9151708b-c553-406f-8e56-694f435154a4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB1520 X-List-Received-Date: Sun, 01 Jul 2018 14:38:32 -0000 --Apple-Mail=_8574A3BF-3492-4346-8417-CBB6C8E043C1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 1 Jul 2018, at 14:48, Pete Heist wrote: >=20 > Ok, my reboots were because I need to compile with om-watchdog = support, otherwise the OM2P=E2=80=99s hardware watchdog reboots every = few minutes. Thus, my sysupgrades were not _actually_ working (watchdog = was interrupting them), thus the unknown symbols / dependency problems, = etc, so that=E2=80=99s behind me. :/ >=20 > And yes, I get no tin stats without the patch. With the patch, I get = tin stats as expected. This is probably nothing new. >=20 > I also compiled on a Raspi 2 (32-bit ARM) running Debian 9.4, and it = works as expected without the patch. This probably just confirms what = George reported, because I think his wrt1900acs also uses 32-bit ARM. >=20 > So yes, MIPS 32-bit, both le and be are both fails. What next? We = could test more archs, but it might be more productive to attempt to = figure it out in the code when I can at least repro it on MIPS. >=20 > You mentioned earlier netlink_parse_nested doesn=E2=80=99t seem to = handle 64-bit values but I=E2=80=99m not seeing where that is. I=E2=80=99m= getting my head around how stats work so just writing out loud=E2=80=A6lo= oks like there are qstats from struct Qdisc then the tin stats which are = probably custom for Cake and stored at some address obtained from = qdisc_priv, which is right after the regular Qdisc struct. Interesting. = I=E2=80=99m searching now for how this data ends up in tc=E2=80=99s = hands, which looks like happens in cake_print_xstats with = parse_rtattr_nested, then the printing gets underway, and at some point = if st[TCA_CAKE_STATS_TIN_STATS] is true it gets into the tin printing = business. What=E2=80=99s the condition that fails to make the tin stats = not be printed? Best I know is to add debugging statements to figure it = out, which I might give a try later today. Am I on the right track? I think we=E2=80=99ve all been heading in a similar direction that=E2=80=99= s for sure. The symptoms are that netlink doesn=E2=80=99t appear to = handle 64bit values on MIPS well because generally as soon as you put a = 64 bit value in the netlink stream, things go a bit wonky. I think this is some sort of alignment problem. e.g. If I get cake kmod = to insert a dummy 32bit pad netlink attribute at the top of = =E2=80=98cake_dump=E2=80=99, then I see the tin stats magically appear, = although some of the values =E2=80=98capacity=E2=80=99 =E2=80=98threshold=E2= =80=99 =E2=80=98sent bytes=E2=80=99 are stupidly high, as if an endian = issue has hit. In openwrt world, mips is not defined as an architecture that can do = =E2=80=98efficient unaligned access=E2=80=99, thus if you look in = include/net/netlink.h you=E2=80=99ll see static inline bool nla_need_padding_for_64bit(struct sk_buff *skb) { #ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS /* The nlattr header is 4 bytes in size, that's why we test * if the skb->data _is_ aligned. A NOP attribute, plus * nlattr header for next attribute, will make nla_data() * 8-byte aligned. */ if (IS_ALIGNED((unsigned long)skb_tail_pointer(skb), 8)) return true; #endif return false; } which looks to be designed to align netlink attributes in a friendly = way. I haven=E2=80=99t looked yet to see if working archs define = =E2=80=98CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=E2=80=99 and non working = archs don=E2=80=99t=E2=80=A6.but to me it=E2=80=99s an obvious place to = start poking, as we clearly have some/most archs wher the iproute/tc & = cake code talk well=E2=80=A6.and one arch where they don=E2=80=99t=E2=80=A6= . to me that sort of rules out tc/cake doing anything silly. There=E2=80=99s a =E2=80=98kmon-nlmon=E2=80=99 that allows you to create = a virtual network interface for the netlink packets and thus capture = (tcpdump) and export to wireshark for dissection. I had a look at that = a few weeks ago but my head exploded and have only just recovered enough = to start looking at this problem again=E2=80=A6. the = EFFECIENT_UNALIGNED_ACCESS thing is something I discovered about 2 hours = ago. Kevin --Apple-Mail=_8574A3BF-3492-4346-8417-CBB6C8E043C1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEASyssijGxT6XdZEjs6I4m53iM0oFAls452QACgkQs6I4m53i M0qbzA/7BBl55OJ1mC4RjKIGoluN9OMIR2QXKLmt0VK/zXfZrHVkdldG0LUIQ+6o onBkD7MPMzAyLc8AR3wgEEZfiEzC/lZ10edHZCkSoHlLyw7ymNdIs1B9ZfZrBwAB 6Va2GBXu4tX2Y031ePYoTvwSwQJczk0OHJwYWVPutd97UxPQxGw2DKSpj61HQdOE HvIskx5bq6mK9RP3KokPRxDD7iyYvb9udEfYxB7OyW835EdRLGWc0ReEraWkj3B1 cj8e7MY1VqAXTmrojAut6UHeqtiNhjAi6q1MkHGloB3KC5W9ZgWZHYIL13Bec6Qa nI0aS/h935Or/Cxa7OJ8XCaiR3Z91ZzKt0XW4iEDrdLcY7H3SK8y+GIxjrT4zWO3 Xa5A68BxZj7a5RTPQK5vLBM6+Jhls9tO3IMIWbfJzBM7lVPqPky9NRR8kbeN3Nod aWQ1zT7Cq6HTGHO+4Fzh6b0Yp5O+rErEEZ7vUIcxtmDOWJdfsxllEsWi0pnnudNx uAUsPh6UFT+RZPFCjwan9KTf+q5T8M4+k1PD/02dXASyG4Po+06707Eu+JjgK4oA LREQjBSP1axWCHpeL1PyEbhRiQFjaTwjL/jVdLb/AZ7X6G6NAVSR7iYL8bw2EfJe Q+XCiU0L0iFrWAu9YDnLRoDOSO8HBXvPMwqUrCrq4/0Q6P1IDnA= =Xtc/ -----END PGP SIGNATURE----- --Apple-Mail=_8574A3BF-3492-4346-8417-CBB6C8E043C1--