From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 978783B29D for ; Thu, 17 Nov 2022 12:34:05 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1668706439; bh=ft49yfwRrLbfK02ppHpPa5Cc5Au5ORaG/Gb280Z41rg=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=uia4lUH1dxzCOXOsUAVSTcEc5YQEiU62tZJnt8Tb81bxFqi/trp173Wqfv1LnBZ76 Tv78HNm7z9+22ZxUCEpH1P2VP7pARPTjHYh2/O0tPnsyJK68naS9Vvr0Z5HxROFiur 6iiiysaCqH7YQuBrlzazclHlG+PKT/DQyFtKVeAEsmoHzSFYD0hnIvJdA4oY09ozBX eiXsY7lSvMTZ+V9qYOrMleiMKWA8FvjgleGSoirQ6OB9OOlGJxTXMCTS+4A1ayVsK3 pkFLh4RTinCiKIzZm8bBO6Px1gWUk+BzaRc/dB/PyD3KAztxn69BbdyuKKvXTlrlN+ Wlr3wDwLjybjQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([95.119.225.252]) by mail.gmx.net (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N5mGB-1p24h22EBf-017A8O; Thu, 17 Nov 2022 18:33:59 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Thu, 17 Nov 2022 18:33:58 +0100 Cc: openwrt-devel , Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <84641688-4969-45B6-AE88-5CD8D3CCD61A@gmx.de> References: <386F2ED9-3D39-4A42-8982-742B5D4B417F@slashdirt.org> <4CF53FE3-D26A-4E32-8378-3B27BBB70AEF@slashdirt.org> <4D5AA477-1547-441F-900C-6DC70C824960@slashdirt.org> <3550010F-3EB3-4CF3-B107-367802DF39A1@slashdirt.org> <94702DB0-7CD6-469D-8C13-B30831CAEB38@gmx.de> To: Thibaut X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:nVTxxdllxgqpkhAY400X5gw5Uh/203ONjQr9H1MHc8Xe9hQPbIu ebMMddl8NLS6TNK5GzcazTvyQ5rqeqwqrWrlV+jCx3o0yTi+AllGF6Jq0js6f7Lqg2HLif/ QmwK35dPNVeOxGL60icoZRcG7lzoflTqVTWewLwzaSLRxv5CbmrKH3NxOmznYMHcqsZVEVH TUbm6OE7ZazK/E7IcKpMA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:gpy1pj+FsXE=;mrIALrcHYKz8beXN5el2G/Josly AIbXlSg7dDlCAm4NaQqY0TGf1FaquPQXU50L6Rhk+lUEvugC8xnTi8kEiVB3b/8uW8B3qJvYr pSoASq5qMxvd+BAOM+n60ptdu2eGxxN2/8I7Zv0Oq5eJFziKwXftIemZAEYrInjfNOrRNbA2o Xmpmx1JUiGKSR55bBt4VFODGkHas0PdflRoU4EVbB7YimC8enMl1sH940xIbLPdsD4V9vOz94 E4bLT5eLMdUy6HkXP+TpHoPMUCnG9VZdjPCnr2WfNYoYyI8LqylaSXVvSih9BpK5B6XEOOoVJ PrKsNbhbTogt9mBHUhvbBsZRGFzojy0xuZFvX9RoXBSoLOVG6jkTg0Z6mQCaQJyjI2lCV9TIc xazYt1cdYodQ8Srk0PXbOF+oOzsiDJAXgAmqXpx7AlB9JQpme1+49wnKd6eBUnGdzQXbMYmYm 8yGIs45+ujRge6sAJUP+qimkHKfUp1AW/9UZMI8FFdjRQpl5hPy4uyfPPPX117ZpxgAT8PJsW Ii+frpW2y5PD1g5R4Z7jJaFr7/DBxWOhP4CdwgkU1cg8h7YDAu0X8fk02v5VjUVnG6+atSNKc OR+pMHrU2tp24Dm7bas1+17epz7ZFSxd/tcPWoRSQnv8Aokl75RJR4nA8fyQeo2b1c7OuLWmP 6Rr9ebvFZiECFJsr5DWBAnES1JlIEf/B6RT0fKGhvc2IrxdNUBCsI59VoxM5t7XH2AyK2kTYL IQeioImMSLlGHFCWp1bgTi84O/mKKmC4MwJ/LMxsvEH+ZtvUnYESw0WGZ5laU4fToSe70Lv1e ZWYvX4ijJ9aLWxsrcejzlqNCtiouoynssxQLQ8O3YIr5yz0mIbv0f+Dh9wrPgO3/kSh1oFeIR vkLlpRHUu2JA8EpJy30NWI55adTtV7glrowq9iz0pfuGpVj7T1e+WQWHPvCjlijehwp42riuj tCNK0OS+eE+ENb/KTlzlpiJRh9k= Subject: Re: [Cake] Help untangling CAKE settings for FTTH X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2022 17:34:06 -0000 Hi Thibaut, > On Nov 17, 2022, at 16:53, Thibaut wrote: >=20 > Hi Sebastian, >=20 >> Le 17 nov. 2022 =C3=A0 16:19, Sebastian Moeller a = =C3=A9crit : >>=20 >> Hi Thibaut, >=20 > [...] >>>>> Although I must confess that it certainly feels counter-intuitive = that for ethernet (and FTTH) we suggest a higher overhead than e.g. = VDSL2/cable (which themselves run off an ethernet interface). >>>>=20 >>>> That is simply because for (ADSL) DOCSIS VDSL2 we have a much = clear picture about what we need to account for. And AON can be = essentially using true ethernet encapsulation so we can expect an = unspecified "FTTH" class to encompass a broad set of different = encapsulations, if we want to recommend a single number that should also = cover AON (the PONs are much less clear*). Why do you assume that FTTH = necessarily would have a smaller per-packet-overhead than other link = technologies?=20 >>>=20 >>> I=E2=80=99m not (necessarily) making that assumption for FTTH = (although I would expect it to be the case, from my limited = understanding of FTTH technologies), >>=20 >> Well, all I can say about encapsulations and per-packet-overhead = is: it's complicated. >=20 > I can see that now :) >=20 >>> I am however certainly making that assumption for plain ethernet, = which is included in the 44-byte documentation case. I think it=E2=80=99s = a reasonable one? >>=20 >> Yes, the goal is to give one number that works everywhere so 44 = is what we recommend. As I said overestimating the overhead is benign, = underestimating it is not. True ethernet has an effective* = per-packet-overhead of: >> 7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte = inter frame gap (IFG) + 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) = + 6 (src MAC) + 2 (ethertype) >> 7 + 1 + 12 + 4 + 6 + 6 + 2 =3D 38 bytes >=20 > And now I=E2=80=99m even more confused: why do we suggest an overhead = of 34/26 for VDSL or 22 for docsis cable then? I assume the respective = modems are connected via ethernet, aren=E2=80=99t they? For DOCSIS you will be correct as there are no DOCSIS = modem-routers that run OpenWrt, for DSL we do have some devices (older = xrx200 and in development newer vrx518 lantiq/maxlinear SoCs) that = actually have DSL interfaces. This is a bit of a judgement call, we only need to account for those = overheads that actually happen on the bottleneck link of a path. For DSL/PTM the dsl overhead is smaller than the full ethernet overhead = as some parts of ethernet encapsulation are not transmitted via the DSL = link. So let's run the numbers for a router with gigabit ethernet port = connected to a DSL modem with gigabit ethernet port. At what DSL rate = will the added ethernet overhead actually matter and to save some time = let's only look at the worst case of small packets (as there the = relative overhead cost is larger): Ethernet max payload rate for minimal packet size:=20 1000 * ((64-18)/(64+20)) =3D 547.619047619 Mbps (true ethernet adds 20 = bytes equivalent on top of the L2 ethernet frame) DOCSIS: (1000 * ((64-18)/(64+20))) * ((64)/(64-18)) =3D 761.904761905 Mbps Now use this payload rate to see how much raw DSL rate we would need to = meet/exceed that rate: (1000 * ((64-18)/(64+20))) * (65/64) * ((64+4)/(64-18)) =3D = 822.172619048 Mbps (PTM adds 4 bytes on top of the L2 ethernet frame = https://en.wikipedia.org/wiki/Ethernet_frame) Currently VDSL2 is limited to ~250 Mbps, so we are far off from the true = ethernet overhead becoming relevant for our traffic shaper. HOWEVER, if the link between router and modem is only 100 Mbps ethernet = the picture changes, and indeed when using a 100 Mbps modem with a = 100Mbps VDSL2 link I used `rate 95 overhead 50` (why 50 and not 34+20=3D54= you might ask? because on the link between modem and router I did not = use a VLAN tag, but had the modem add this tag later). As I said, it's complicated. It is similar for DOCSIS, except there rates in the >=3D 1 Gbps range = are a thing since some time already and unless rputer and modem use a = 2.5/5/10/... Gbps connection the full ethernet overhead becomes = relevant. Time to change the docsis recommendations for say rates > 500 = Mbps. ADSL/ATM/AAL5 is essentially only used for speeds below 25 Mbps with = devices with at least 100Mbps ethernet ports so let's ignore this for = now: >=20 > Another confusing point in the current detailed doc (and possibly the = one that got me started): > " =E2=80=A2 Link Layer Adaptation calculates the proper overheads = for WAN links such as DSL and ADSL. If you use any kind of DSL link, you = should review this section." >=20 > This seems to imply LLA is only necessary for =C2=AB exotic =C2=BB = links such as ADSL/VDSL. This is historic, LLA is exceptionally important for ATM/AAL5 as = there the effectove overhead depends on the actual packet size, the = importance of proper overhead accounting for all link technologies only = became obvious later (one naively assumes that this can be fixed by = setting the shaper somewhat lower, but as the section about overhead and = shaper rate shows it is not as easy). > This also brought me to the (na=C3=AFve) comment that it would be nice = if CAKE would account for local NIC L2 overhead internally (since that = info is readily available to it, is it not?), and one would only have to = configure the actual =C2=AB overhead =C2=BB of whatever is connected to = the local NIC. There would be less margin for error I believe. You need to account for the relevant overhead on the bottleneck = link, which often is not an L1/L2 interface of the router itself. Cake = has a number of technology specific link layer accounting "sets" = prepared you can activate with a keyword, but that does not solve the = hard question, which overhead is actually to be accounted for (see the = example of VDSL connected over ethernet above with the to be accounted = for overhead depending on the ratio od DSL rate to ethernet rate). > Then I saw this: >=20 > "Please note that as of middle 2018 cake, and cake only, will try to = interpret any given overhead to be applied on top of IP packets, all = other qdiscs (and cake if configured with the =E2=80=9Craw=E2=80=9D = keyword) will add the specified overhead on top of the overhead the = kernel already accounted for. This seems confusing, because it is ;) so = if in doubt stick to cake." >=20 > Well yes, it *is* very confusing :P Yepp, on ethernet interfaces the kernel will typically only = account for 14 bytes of the total 38 bytes of ethernet framing, as the = kernel maintain these fields (MAC addresses and ethertype) itself while = the other parts are mostly handled by the driver of offloaded onto the = NIC, so the kernel is not doing anything wrong by not reporting these = other parts. >=20 >> Now if a VLAN tag is added you end up with 38+4 =3D 42 so both pure = ethernet and ethernet+VLAN have a er-packet-overhead close to 44 bytes. >>=20 >> *) The IFG is not real bytes, but transmission silence for the time = it would take to transmit 12 octets. >>=20 >>>=20 >>>> Now, if you have reliable information for say GPON-overhead, by = all means add it (but to be useful this also should contain an easy = identifier for other users to figure out whether they use GPON in the = first place). >>>>=20 >>>> The bigger point however is IMHO, from the perspective of = minimizing bufferbloat/latency-under-load-increase using a slightly too = large per-packet-overhead is fully benign, while specifying to small an = overhead can easily result in measurable bufferbloat increase. So IMHO = it is fine to err on the side of too large when estimating the = per-packet-overhead. >>>=20 >>> OK. Although I would think people reading the detailed explanation = are looking for precise info and explanations, not one-stop-shop = approximations. >>=20 >> Sure, but the problem is we can not give them that "precise = information" them, you , me would like to have, so we do the best we = can. >>=20 >>> Not to mention the kind of users who want to squeeze the maximum = performance out of their setup. >>=20 >> Just read the section where I explain how per-packet-overhead = and shaper-rate are not orthogonal variables to understand how = problematic the whoe thing is, add to it that ISPs often employ traffic = shapers with potentially independent overhead-assumptions making the = whole problem even harder. >=20 > Yes, I see it=E2=80=99s a bit of a headache :) +1; the whole wiki would read easier if the subject matter was = less complicated ;) >=20 >>>> *) The core problem is that we have no straight forward way to = empirically deduce the effective per-packet-overhead over an arbitrary = network path, so if a link technology defines/documents the overhead = well and conclusively (like docsis) we are in luck, but if the details a = vague or leave many options to the ISP we have to make an educated = guess. >>>>=20 >>>>=20 >>>>> I would also like to see some info about ppp vs ethernet interface = in there (matching your previous email): unless you beat me to it I will = add it. >>>>=20 >>>> I will not beat you to it, because for users of cake it does not = matter >>>=20 >>> You said the exact opposite in your previous email (persistence of = statistics) :) >>=20 >> I misunderstood your argument here and thought you wanted a = section explaining the issues with getting the overhead numbers for = pppoe** and ethernet interfaces set for HTB+fq_codel, sorry. If you want = to explain differences between cake on pppoe or on ethN go for it. >=20 > OK but I also believe it=E2=80=99s important to clarify that the = overhead setting needs not be adjusted whether we shape the pppoe = interface or its underlying ethernet one: that certainly was = counter-intuitive for me. +1; go for it! >=20 > Thanks for bearing with my ignorance, This is how wikis improve if open questions are discussed and = text changed. > T > _______________________________________________ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel