From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 29F3A3B29D; Thu, 28 Sep 2023 15:31:28 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1695929485; x=1696534285; i=moeller0@gmx.de; bh=tqVSqQLZF+AEdzOwi/tZrzC1l74bJViYlOjmD8G3tRA=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=MiMSw0CGz6GdxptY8FE+n9ORTME07HaHXqG/QyNAva3o6tpsP01Z99CPeXBLxMMJVQW/kEtIh1G 5/icK7ZUrGpY17jzHwSQ7LtDVLj+asMmxRLcU16gEw0mAfwNv5EyPUbhsy/0hR9IsHGsKS6QpW3oU 427pzaB++3nS2DNX7On7z3dl3VxGLTfybC0W0SYGoHjdD4e/fqAJApTBUMUXQGY82XhTmNKKQx88I RbekagX4ZuojVPwH7QxpYeqtqGTHjDv+A9xMTaQwNQz58/3Fsi5q2gIZRbjAu+L0EIiZVXpGxJYcu xqI7r14KW0wwDwOvqlEuiTOyZvSosAHC7hjA== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([77.3.239.225]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1N3siA-1rlN1R28S8-00zpK5; Thu, 28 Sep 2023 21:31:25 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) From: Sebastian Moeller In-Reply-To: Date: Thu, 28 Sep 2023 21:31:24 +0200 Cc: libreqos , Dave Taht via Starlink , bloat , Rpm , Jamal Hadi Salim Content-Transfer-Encoding: quoted-printable Message-Id: References: To: =?utf-8?Q?Dave_T=C3=A4ht?= X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Provags-ID: V03:K1:8Nm7sVKnxD8+RaoKiiKt8HwJ3A9SbicyZrkwv2d3FaWUYqT+D+B AIFXvbkcsGoalyYsI3oB6fDRYYuv4xUz9+CMWW07fE3EQ1IJ6nR18f8QumSPjqIj4nI+R6D fFZEV/CmoR9UoYRV0IWobz98k3Xtnf3LAoF9eSnuwk028TZOH3ZOETTE55L+1/zbSXD2q0+ BVzudeuYAbsqu9/8xW4dA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:+UheKvx6/u8=;VJzgZZAldj//ijjE9Stvg1vhIRL 7ZBi7uK1FrE6DcEbl55XiRIc7a5jVuk/C+QLeFtNiu7i7tiqONn3wAIBqO5kinspVcG2qyoyI JYayK1RnEP5n9MEoSXwMa6EKBWNzMVdv1HUnrZ0WdEY4sO/xAS7ndJ94xrdEhImrqQUoUkTsf ScoJOtEzYspGJHNqKIGVH8vobUKqlRKYjbgMmFmD/7ar4h1YV4KcX36VDB++A7pwNsPWbiZcL PGrj8iKvTEEge/VzuAUfvu/eviajt2FOwTMesntu+aOuh0bKnqiR2kaoUQxVrcJp5pn7Wcomd PjdvcD0LBtrDNNZO5SzpDmf7GDT5tJeY77gdBMyJtaYyltmPULp0twLnqVJlTyczK982RAm1C /gl15kQ/AJ78MtVesyM0g34hhpxcyZ5ssVNDJjIioIy1JnhFa6QE9YYm5y1MIbj3n82sXqrlP 1tOQUNRDTt8qb1Iu283XHiqOYy8DVpGe4Y1D8NK1W5+5Usqx2deShvrWlU1tWEibUGs4iJP5C kSjjzdavhf+E+ckkXIzgMK9x6luX2pxRdORYV9XhH+mP1oSMAT47cF9rKBL2/QsD4xatvQytj aP65q1r008kioid3BkawCMu28NdBL/3ka9MYGAbXUB3z343aZVqms1eB3GuJiJB4Ok90R6SyS yVuzJ/8fwOlDpyy0Vm1IzdXLe9I5LTBWov2DeYNjHDqFR28Zh4T+3ZgKcghNpUzJorjHgk1OY se+Y1QGnvJX6LfLVvhTYFKmBYhJ3WR3PkK1el7U4C5Dmm76kvDrCZd3T4L6OBiSKRi5wL8Ik1 iIb8dEKOjo58Wcw9ny3dAivjdK0wCY23v0rw7Av5Go62HQxpxN29M5agSIL3NpWwWD0yKsis/ pfYXSo70J0Y9VW8Akrfb5DKabNt7a4pLOMKrWECCKtv+GaXfX2pwjyFPAXMInKqbgkZW+iIpd SftpFxv7VmYb5SiAVutcYKXIyYs= Subject: Re: [LibreQoS] [Rpm] net neutrality back in the news X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Sep 2023 19:31:28 -0000 > On Sep 28, 2023, at 18:38, Dave Taht wrote: >=20 > On Wed, Sep 27, 2023 at 11:25=E2=80=AFPM Sebastian Moeller = wrote: >>=20 >> Hi Dave, >>=20 >> please excuse a number of tangents below ;) >=20 > It would be nice, if as a (dis)organisation... the bufferbloat team > could focus on somehow getting both sides of the network neutrality > debate deeplying understanding the technological problem their > pre-conceptions face, and the (now readily available and inexpensive) > solutions that could be deployed, by most ISPs, over a weekend. We are > regularly bringing up a few thousand people a week on libreqos (that > we know of), and then of course, there are all the home routers and > CPE that are increasingly capable of doing the right thing. >=20 > The time to try and shift the memes in in the coming days, and weeks. >=20 > So ya'all being distracting below... aggh... ok, I'll bite. [SM] Sorry to drag you into the weeds... >=20 >>=20 >>=20 >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm = wrote: >>>=20 >>> Jason just did a beautiful thread as to what was the original source >>> of the network neutrality >>> bittorrent vs voip bufferbloat blowup. >>>=20 >>> https://twitter.com/jlivingood/status/1707078242857849244 >>=20 >> But the core issue IMHO really was an economic one, the = over-subscription ratios that worked before torrenting simply did not = cut it any more in an environment when customers actually tried to use = their contracted access rates "quantitatively". In my outsider = perspective an ISP owes its customers the contracted rates (with some = slack*) and any sort of over-subscription is a valid economic = optimization an ISP might take, IFF that ISP is prepared to rapidly = increase segment capacity (or down-grade customer plans)) if the = deployed over-subscription rate proves to have been too optimistic. Mind = you, most ISPs market plans by access speed (and charge more for higher = speeds) and hence are somewhat responsible to actually deliver (again = with some slack) these speeds. >=20 > The average use at peak hours for home broadband is below 5mbits per > subscriber regardless of on a 25mbit or gbit plan. Remarkably, > business plans are less (but more bursty). Oversubscription within a > set of parameters is needed because, in part because upstream > bandwidth to the internet remains expensive (although I hope that cost > continues to drop rapidly), and in part, because it is needlessly > expensive to provision exactly the contracted rate to the customer. As > one example, I use the inverse square law as a rule of thumb for > wireless deployments - the range you can get from a 25/10 deployment > is geometrically better than 100/25, and the average bandwidth usage > nearly the same. [SM] +1; hence my argument is not oversubscription per se is = bad, but that oversubscription needs to be managed and the level of = oversubscription needs to to be adapted ot he actual usage patterns. I = am sure however that ISPs already actively do that (I assume most ISPs = want happy customers). > Agreeing on industry standards for what the "slack" should be, might = be of help. [SM] Not being part of that industry at all I would love to see = that as well, but can not contribute to defining that in any way. >=20 >>=20 >>=20 >> *) Claiming "Up to", only carries that far, if you sell me X and I = mostly get Y (with Y close to X) and occasionally Z (with Z << X), that = is acceptable unless occasionally is "at every late afternoon and = evening" prime-time. >=20 > We have discussed elsewhere, the idea of a minimum contracted rate > (down to), which is easier to reason about. [SM] Indeed, the German regulator (and I am not saying this is = the only or best option) decided to require ISPs to give 6 numbers = pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) = rate, a minimal (net) rate, and a usually achievable (net) rate, all = rates were defined as IP/TCP goodput to make verification easier on = end-users. The regulator also defined a measurement regime that = end-users can follow to check whether the ISP is fulfilling the contract = and the law gives remedies if ISPs do not deliver (either the right to = immediately cancel the contract and/or the right to adapt the monthly = payments to the actually delivered ratio of the contracted rates*). I = think I need to add, that ISPs can set these numbers freely, but are = only allowed to advertise with the maximum rate. But if we talk about a single number per direction, minimal rate = is probably easiest, or something like a "usually achievable rate" (that = needs to be met or exceeded in say 90% of >=3D 20 measurements or so). *) In a ironic twist however the regulator so far has not explained how = deviations in each of the 6 numbers above should be aggregated to get = one single contract rate achievement ratio..., most ISPs took measures = into their own hands and simply offer customers typically a permanent = rebate of 5 EUR or immediate cancelation what ever the customer = prefers.... >=20 >>=20 >>>=20 >>> Seeing all the political activity tied onto it since (and now again) >>> reminds of two families at war about an incident that had happened >>> generations and generations before, where the two sides no longer >>> remembered why they hated each other so, but just went on hating, = and >>> not forgiving, and not moving on. >>>=20 >>> Yes, there are entirely separate and additional NN issues, but the >>> technical problem of providing common carriage between two very >>> different network application types (voip/gaming vs file transfer) = is >>> thoroughly solved now, >>=20 >> I am not sure this was at the core of the problem, my take is = really that "always-on" and relative upload-heavy torrent simply = demonstrated painfully for all involved that the old oversubscription = ratios were not suitable for the changed traffic profiles. I have some = sympathy for the ISPs here, they certainly did not try to sell their = customers short (good ISPs try to retain their customers and that works = best when customers are happy with the service) and having this problem = appear on many segments at the same time is not a fun place to be, and = upload was/is often (too) low in DOCSIS segments anyway; but this is why = e.g. my bit torrent could affect your VoIP, simply by pushing the whole = segment into upload capacity congestion... (ISPs in theory could fix = this by plain old QoS engineering, but the issue at hand was with a = non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as = these packets need to be identified in a way that is not game-able**) [SM] See my later mail to Jason, I will not claim I know = Comcast's issues better than him and will accept that self-congestion = also played a major role in the initial problem. Over here in Europe the = net neutrality debate was kicked of less by an unfortunate confluence of = new usage profiles and traditional oversibscription ratios and less than = ideal packet scheduling, but rather by a series of pretty apparent = attempts at restricting consumer choice, see eg. = https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_E= nding_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted = slightly biased position: " =E2=80=A2 Blocking of applications and services: In order to = maximise profits, some ISPs =E2=80=93 that also offer their own services = and applications online =E2=80=93 exclude certain services and = applications of competing market players. The most prominent case of = this form of network discrimination is European mobile providers (like = Deutsche Telekom) blocking or restricting the use of Voice over IP = (VoIP) services (like Skype and Viber) for their customers.20 =E2=80=A2 Slowing or =E2=80=9Cthrottling=E2=80=9D internet = speeds: Some ISPs slow down specific services (like YouTube) and = applications (like Skype), or ask users to pay an extra fee to have = access to these internet platforms. Given the high latency (delay) = sensitivity of many applications, ISPs are able to compromise the = correct functioning of these services by slowing them down, preventing = the services from running properly. Often ISPs =E2=80=93 especially = telecommunication companies =E2=80=93 do this to favour their own voice = calling services over VoIP services, thereby crushing competition. =E2=80=A2 Blocking websites: ISPs often block websites for a = number of reasons =E2=80=93 to secure their network, or to avoid = competition, and sometimes for social, public relations or political = reasons. In the UK, for instance, Orange Telecom blocked the French = digital rights advocacy group, La Quadrature du Net=E2=80=99s website on = pre-paid mobile accounts.21 =E2=80=A2 Preferential treatment of services and platforms: ISPs = can also impose data caps on internet access contracts while granting = data allowance exceptions to a company=E2=80=99s own proprietary = streaming services (like Deutsche Telekom to its own = =E2=80=9CT-Entertain=E2=80=9D).22 They can (and do) also grant = preferential treatment to select services =E2=80=93 such as Orange = France with the popular music streaming service Deezer23 =E2=80=93 ahead = of other competitors, effectively imposing anti-competitive limitations = on markets such as those for legal online music. Moreover, generally = only large, well-established companies can afford this preferential = treatment, resulting in a further stifling of innovation." These look like clearer attempts to monetize the ISP position as = gate-keeper to his customer's eye-balls (for content providers) as well = as gate-keeper to the internet for for paying customers. I find it much harder to have sympathy for the listed examples of ISP = behavior than the situation of rapidly changing usage pattern posed = where technical changes needed to be made, but where no attempt at = unfair monetization was evident, but maybe I am overly sensitive. These = are all examples that make me personally applaud the EU for its = intervention to make clear rules what "internet access providers" can = and can not do. (The EU also was quite flexible in applying/interpreting = its rules during the covid isolation periods, when it made it clear that = e.g. treating certain traffic classes e.g. streaming media to lower = priority than video conferences was within the neutrality framework as = long as it was based on traffic class and not on specific service = providers IIRC). >=20 > Torrent problem thoroughly solved with FQ on the path and backpressure > from the mac scheduler. >=20 > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf [SM] Thanks for the link. This is mainly for the self-congesrion = case? >=20 >> I agree that on a single link we mostly solved the problem (a = few corner cases are left on links with very low capacity, where = essentially we can only manage the pain, not remove it)... >>=20 >>=20 >> **) Which is not rocket science either, a VoIP stream takes ~100 = Kbps, so in theory an ISP might simply allow each customer say 5 VoIP = stream equivalents by allowing up to 500Kbps od traffic marked with a = specific DSCP as higher priority (with higher access probability for the = shared medium). I am not sayng this is my preferred solution, just = saying this is a solution that would have been available at the time if = I memorize my docsis capabilities correctly. >>=20 >>=20 >>> and if only all sides recognised at least this >>> much, and made peace over it, and worked together to deploy those >>> solutions, maybe, just maybe, we could find mutually satisfactory >>> solutions to the other problems that plague the internet today, like >>> security, and the ipv6 rollout. >>=20 >> +1. IPv6 is IMHO a prime example where the regulators should = stop talking softly and start showing the big stick they carry. >=20 > I really would like to see a push for IPv6 mandated as a part of the > BEAD programs. [SM] +1; I am not the biggest IPv6 fan, but that's what we have = and it is mostly serviceable so let's get this "party started" finally. >=20 >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 = addresses only.. (most mass market ISPs do much better in that regard, = but for the stragglers it would help if the regulator would demand IPv6 = with PD***). Regarding security, the easiest way to achieve that would = be to put some heavy requirements on IoT manufacturers and vendors (like = do what you please as long as you are local LAN only, but once you reach = out into the cloud you need to fulfill the following list of = requirements, with timely security updates over a reasonable long usage = period), however that might not be very attractive for = politicians/regulators to tackle (active regulatory acts tends to get = bad press unless something bad happened, but even then the press often = complains about the acts coming too late, but I digress****) >=20 > There is a separate NRPM going on for the new cybersecurity label at > the FCC. If I had time, and a partner, > we could rework the doc we did a few years ago. It is due oct 6. >=20 >>=20 >>=20 >> ***) Strictly speaking IPv6 is required, since "internet access" is = defined as reaching all of the internet (as far as in the ISPs power) = and IPv6-only sites are not reachable for the CG-NAT-only customers. But = so far the local regulator does not seem to enforce that requirement, or = hopefully is working on this quietly behind the curtains. >>=20 >> ****) This is not to diss the press, they are doing what they are = supposed to do, but it just shows that active regulation is a tricky = business, and a light touch typically "looks better" (even though I see = no real evidence it actually works better). >>=20 >>> If anyone here knows anyone more political, still vibrating with 10+ >>> years of outrage about NN on this fronts, on one side or the other, = if >>> you could sit them down, over a beer, and try to explain that at the >>> start it was a technical problem nobody understood at the time, = maybe >>> that would help. >>=20 >> So in the EU that debate is essentially settled, there is a EU = regulation that essentially spills out what ISPs owe their customers and = that has become the law of the land. The rationale for required = un-biased service and freedom to select terminal devices is well = justified by the market ideals of the EU, allowing ISPs to discriminate = packets or terminal devices restricts the market and will lead to = undesired outcomes. (Fun fact most big players in capitalist societies = argue for "free markets" but at the same time act to work-around the = market mechanism by trying to move the market into an oligo- or even = monopoly condition, which is why strong regulation is required*****). >=20 > Governments make safer markets feasible. [SM] Yes, I agree, both in markets are a pretty decent tool, but = need constant maintenance. Regards & Sorry for the tangent Sebastian >=20 >>=20 >> *****) This is akin to professional sports where the audience = generally accepts that referees are necessary and occasionally need to = make "painful" calls, as the alternative would be anarchy, but I = digress. >>=20 >> Regards >> Sebastian >>=20 >>=20 >>>=20 >>> -- >>> Oct 30: = https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html >>> Dave T=C3=A4ht CSO, LibreQos >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >>=20 >=20 >=20 > --=20 > Oct 30: = https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html > Dave T=C3=A4ht CSO, LibreQos > <2510.jpeg>