From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20073.outbound.protection.outlook.com [40.107.2.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 44EDB3B29E for ; Tue, 19 Jun 2018 09:41:40 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=darbyshire-bryant.me.uk; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3kX8fhWZvdlIBVPNgcXclV7PEjT1FVVwFkTlo17jtrs=; b=cihWII7Ym6uMq1Al577RB3NRpyC0Erlr+b83WEZF6tasTwgv82PiImQB9ah307dy7fgoO0S9tvqlzny7WoUkK31rSPr8dKGLqwKUd8ZPialsDsQSwQfXWhLamVcVTInAOBKi9Hi7r9m+Cf4KfrDdgSGt7n4/GjYRyD3g6xq7m2g= Received: from VI1PR07MB4254.eurprd07.prod.outlook.com (20.176.6.147) by VI1PR07MB0878.eurprd07.prod.outlook.com (10.161.108.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.884.17; Tue, 19 Jun 2018 13:41:37 +0000 Received: from VI1PR07MB4254.eurprd07.prod.outlook.com ([fe80::2d37:3474:cf1e:b21c]) by VI1PR07MB4254.eurprd07.prod.outlook.com ([fe80::2d37:3474:cf1e:b21c%4]) with mapi id 15.20.0884.010; Tue, 19 Jun 2018 13:41:37 +0000 From: Kevin Darbyshire-Bryant To: Pete Heist CC: =?utf-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= , Cake List Subject: Re: [Cake] the Cake stalemate Thread-Topic: [Cake] the Cake stalemate Thread-Index: AQHUB7BiTz9PUHXGDE6KnfwWgbqNdqRnV+aAgAAbOACAAAYhAIAACP8AgAAU3QA= Date: Tue, 19 Jun 2018 13:41:36 +0000 Message-ID: <1CC8F6F6-94E1-4987-928E-29BDE01787BC@darbyshire-bryant.me.uk> References: <87in6ft81g.fsf@toke.dk> <0FF888EC-4FF0-4AAE-B2B8-79A10B0B5A9E@gmail.com> <877emvt2ij.fsf@toke.dk> <6D5348AC-921E-49D3-B6D0-9CC30874B80D@heistp.net> In-Reply-To: <6D5348AC-921E-49D3-B6D0-9CC30874B80D@heistp.net> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [2a02:c7f:1231:2000::dc83] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR07MB0878; 7:f13fy1PqA7MHbyL7Xj0xxoHw/JenM/GBxJkQTz04SSU9CFhEfYTjk4YGGNxRBoWdjlnxibtN1JR2pMgCqGJNJX1H+LLEXCsT3IY1ZQIikcdM8cOZSKG8BUkpASBfsGax1e2+TIH8KRylYOSCjr8mXb54ahB4AiyK+fWRJAj4DOgmOEaZMP9SNFzk+fAwpqxpGeavl7ez86DHzUmzOqCJhekrR7xOQqobTf8BqXFfjM9yr70hYtFB40F/mjLWkCce x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: f5a67b8e-410f-41ec-198b-08d5d5ea611f x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(7021125)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(5600026)(711020)(2017052603328)(7153060)(49563074)(7193020); SRVR:VI1PR07MB0878; x-ms-traffictypediagnostic: VI1PR07MB0878: authentication-results: spf=none (sender IP is ) smtp.mailfrom=kevin@darbyshire-bryant.me.uk; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(166708455590820)(84791874153150)(226747003789276); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415395)(6040522)(2401047)(5005006)(8121501046)(10201501046)(3231254)(944501410)(52105095)(93006095)(93001095)(3002001)(149027)(150027)(6041310)(20161123564045)(20161123558120)(2016111802025)(20161123560045)(20161123562045)(6043046)(6072148)(201708071742011)(7699016); SRVR:VI1PR07MB0878; BCL:0; PCL:0; RULEID:; SRVR:VI1PR07MB0878; x-forefront-prvs: 07083FF734 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(396003)(376002)(39380400002)(39830400003)(366004)(199004)(189003)(52314003)(82746002)(6506007)(6486002)(4326008)(6246003)(8676002)(6916009)(102836004)(6436002)(53546011)(305945005)(7736002)(76176011)(97736004)(11346002)(2616005)(83716003)(81156014)(966005)(478600001)(6306002)(446003)(53936002)(486006)(8936002)(6512007)(476003)(81166006)(186003)(3280700002)(54906003)(36756003)(74482002)(229853002)(575784001)(93886005)(3660700001)(86362001)(5250100002)(14454004)(6116002)(2900100001)(99936001)(25786009)(68736007)(316002)(105586002)(5660300001)(46003)(33656002)(106356001)(2906002)(99286004); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR07MB0878; H:VI1PR07MB4254.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: darbyshire-bryant.me.uk does not designate permitted sender hosts) x-microsoft-antispam-message-info: ykujhPGnaXmcEeBUXFYDZ5jVz7L8Earx76YWUID7CN7TR82E3ukP2qibcr8wKQ4m8U4pVFfEi62zIo5ozsG1kwnz6RVMQK4fqmV7g7BQb/FJEcvd8Wu/FlsJLkoeMZsl/+pp/X3T3awlW9jq4F78aVidaU0HsqeujvQIMTmdKTyjpJEd8YrLTc+I3pvP9XwM5aaHQB5IxF3lZxlIPFvTzC/mt6vdo11TnsKonQ53jGE4MadS5SQ4bfhiZIn/2nEsdrIkSfjUMfNBanA9WeZiMESkV9cBnDeoa/5J6liHdoI8dLgQJ5UNELnLqk58CXnJ2nwBtITxsx0RsTJgsWbwRw== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/signed; boundary="Apple-Mail=_8DFDC1E5-3604-4E68-9EAF-4EA5D0D530F8"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-OriginatorOrg: darbyshire-bryant.me.uk X-MS-Exchange-CrossTenant-Network-Message-Id: f5a67b8e-410f-41ec-198b-08d5d5ea611f X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jun 2018 13:41:36.8913 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 9151708b-c553-406f-8e56-694f435154a4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR07MB0878 X-List-Received-Date: Tue, 19 Jun 2018 13:41:40 -0000 --Apple-Mail=_8DFDC1E5-3604-4E68-9EAF-4EA5D0D530F8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 19 Jun 2018, at 13:26, Pete Heist wrote: >=20 >=20 >> On Jun 19, 2018, at 1:54 PM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >>=20 >> We also saw a bug on 32-bit MIPS where some combinations of 64-bit >> netlink attributes would cause stats display in tc to fail. However, = I >> believe this is more a case of Cake exposing a latent bug somewhere = in >> the tc or kernel netlink code (alignment issues, perhaps?), and so = I'm >> not sure it is necessarily a blocker for merging Cake. However, if >> someone could take a look that would be very helpful. I forget if the >> current head of the cobalt branch exposes the bug, but I think it = does. >> It's quite obvious when it happens: no stats output whatsoever... >=20 > I have a 32-bit MIPS in my ER-X, but it sounds like what I saw = (outrageous refcnt values) was something different: Yes it was. At one point iproute=E2=80=99s tc was doing hidden type = promotions in printing from 32bit to 64bit types and neglecting to tell = the printf formatter of the change, thus printf was starting at the = wrong point in memory in big endian environments. This was part of the = move to JSON output. Toke took my bug report & patch and made it acceptable to upstream where = it now lives as: = https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/?id=3D= 4db2ff0db46f6368d89cfb3498a700e1256d2a04 and is included in iproute2 = v.4.17 > However, if there=E2=80=99s a way I should try to reproduce something = on this hardware to take a look, send any info you=E2=80=99ve got (how = to add 64-bit netlink attributes?). I even have a spare ER-X on which I = could put OpenWRT in case I need to be working with a more modern = kernel=E2=80=A6 The lack of stats on recent (ie post = https://github.com/dtaht/sch_cake/commit/af1d7cde7046af55ec867b29854d75481= 6b64bc8 May 15th) with MIPS BE & LE 32 bit arch is a mystery. My hack = workaround to that for my own personal openwrt builds is = https://github.com/ldir-EDB0/openwrt/tree/tokesiproutedebug - which also = includes a debug commit from Toke. I considered bumping openwrt=E2=80=99s master branch to point at latest = commit of =E2=80=98cobalt=E2=80=99 like my build does, so we could judge = from the resultant screaming if it was just MIPS affected or other 32 = bit arch=E2=80=99s. I was dissuaded from doing so. I got a little further into collecting info on this courtesy = =E2=80=98kmod-netnl=E2=80=99 which allows packet capture of netlink = packets as if on a network interface - captures sent to Toke IIRC but = they require hand disassembly to determine where the packet formatting = is going wrong. And there $real_life intervened and I=E2=80=99ve not = looked at since/had some more pressing bugs to ponder. Openwrt nearly bumped to iproute v4.17 but I haven=E2=80=99t yet got = around to seeing if that makes any difference. It looks like = netlink_parse_nested cannot cope with 64bit netlink attributes=E2=80=A6. = but this requires a person who can code rather than me to go any = further. RE: the stalemate. I swing between an absolute hatred of anything = linux/open source/mail lists and finding some people *incredibly* = helpful and thinking =E2=80=98it=E2=80=99s not so bad, actually this is = fun=E2=80=99. I offer a very recent example of this where I worked with = David Woodhouse on a kernel PPPoATM bug (caused by a ticking timebomb = that one E Dumazet left behind ;-) that stretched me to my absolute = limits but was executed in a spirit of helpfulness, curiosity & fun. So = it seems to be about finding the right person in kernel land who can = both see the errors in our code but also see the value and effort in = what we have achieved. Maybe I=E2=80=99m being unfair and not = interpreting the kernel mailing list environment correctly but to me it = comes across as abrasive at best (and I swore I'd put my head in a = tiger=E2=80=99s mouth and tickle its testicles with a spanner before I = even think of trying to submit another patch upstream) On the other hand I can also see that had we approached/involved the = kernel people earlier on then some of the blind alleys we=E2=80=99ve = travelled (I=E2=80=99m thinking passing of netlink stats here) could = have been avoided. Instead we=E2=80=99ve invested years of work and = just presented a fait accompli. Whether that would have yielded some of = the layer breaking stuff we=E2=80=99ve ended up with I very much doubt = and cobalt would have been much, much poorer as a result. The beauty of cake/cobalt is that it does a number of sensible things = all in one command line (and has to work around some of linux=E2=80=99s = layering decisions.. IFB) Anyway, there=E2=80=99s my opinion. KDB >=20 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake Cheers, Kevin D-B 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A --Apple-Mail=_8DFDC1E5-3604-4E68-9EAF-4EA5D0D530F8 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEASyssijGxT6XdZEjs6I4m53iM0oFAlspCBAACgkQs6I4m53i M0ruPBAAxLsP4MZcdbRUnHsdBOUlUr3Xp8zODJaWQn8BUT3/saWhHP7K7PeQIiAI 7Q/lGADyrx+8btbXtDkr2EGwOzB9S4eThGngmD3a2zV6CcSWuHCOC1RGYoJvZZkN LrZOdoCRS4h6jb5ndV9TC/sAbjT23THNzm1bR3d1uxZlfIdgjQRme+zawbs8PsYE FwdvRPZ2mYfQk+ejPVVXQmhEaAS0Dby0Ka+kebNpH0+vE8LqB/JEtqsDbc0OztLo AzhRNNLHueOzpT6TxroDZiLRkIQjYmmlIZJk7FFSgNRiUl5MZQbCmjf7Xeu2boQg 32B7X1ripo/kHwQuGIOYZXYoP/GCLaT+H+yVw9QCAFw2IV+8rsnq6XdKzzJGq3uq KMHgkDSMN2H61po9IwWBjad2vI1IPz8G0US3o4vbyOZboc9LCNja49BjSw7uCjU7 AXI/Xvx0M7nqiHNprwTLRMhPKAyceywinCnCdRAefdEKefbFOj4eE9ZnGWWtxdlf a70+ktzTbIeEigvMw4ch0PF+vMom3TAphr7FQfhaDNmzx2SFY64xKrVQ0B2xmchP /2UbnjeGx8MqM110IS1erVjtVSSvMGBqhKSNr077Z6lE4ambtZ4iCdrEiKDDBXef iCdBSo6LslH4cyTQcEsXHclIy+XGhmPv/UTEi+zZX+xjpXaxy5s= =US1Q -----END PGP SIGNATURE----- --Apple-Mail=_8DFDC1E5-3604-4E68-9EAF-4EA5D0D530F8--