From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D67233B29E; Sat, 30 Sep 2023 11:20:40 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1696087238; x=1696692038; i=moeller0@gmx.de; bh=HW2k7K3LbOas9uLKTVmUzpM+S3dmjBz0EXSkQdUJbSA=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=n3yl+rGdG448sJT6ySiO1emNMeralMdSbZ8uKPbTVxqVW34RMn8AF1JBGgXYuqgrmKese+sCxoH FqCnDIHEdU1bYqEW8PXv6Bmm7LsFRxLfPeMGK9G/xAzNvQgOu9jxnCM4csJhqFQc5/Z571XRX4haM pkggNvnGvAgxVLYugh8aSIFkMUvsD5oCtaw7P2lDhm9o3JEEd2pcVjdqqZ5ZHV/aNT+8tylBCoAVD 0sPVKQkZv0xROdrsei6Ug6fx99ugH+Xg9TWjhLd4gz7y7HZRnelLQVw46r9SUeqLEhX8M9dAeiC5Q sQ27NijsKklTVc06hR62c0/VEe6fWVDqiVPA== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([95.112.94.87]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MO9zH-1qxvUl1ga1-00OZqL; Sat, 30 Sep 2023 17:20:38 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.4\)) From: Sebastian Moeller In-Reply-To: Date: Sat, 30 Sep 2023 17:20:37 +0200 Cc: libreqos , Rpm , bloat Content-Transfer-Encoding: quoted-printable Message-Id: <4129491C-FFC7-4E5E-A5A4-8CBE9B5C5255@gmx.de> References: <5so3r00n-31pn-14s7-7775-08731s3s551r@ynat.uz> <7508FE73-A154-4CBA-984C-748A80C5FFEC@gmail.com> <039490DA-48A7-4AE2-B00F-AA2A260FB747@gmail.com> <3F6D16C8-530E-4066-8B6A-C0644B3C2D2C@gmx.de> To: Mike Conlow X-Mailer: Apple Mail (2.3696.120.41.1.4) X-Provags-ID: V03:K1:seoaqeW0Svf1fMyDQATIGnxzzRS4+qoShTpfkiJyJjO4E/Z2ve9 Eydx3EAM+eMXIx0ucNBmt5Lfc+icKrPjr7vHDK5NMOBEXyeLqH8rq6Eoemaw91tisSFRJsx qamof6Gnh0a78vqyWCjU0qoRC8zLLrP38pbSvDmIHtBttY9u5xrSFN4EMpQIJ0UB+qrDl6i BeMpyVKrl+3zmb3GkZvXQ== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:lVkBmrDcFdY=;lP4mmG1Mjy93xYGT3/f61IHXgm8 gPPDEc0l8dW+yck0sq3LjptFXhmdFBHpOnyh3RQnmv5wHpDShwf8AFm9MkGj+BceAud2MhcAI ySuCzO02fIEcUtHYjc9DormBnrLAT+m9aICw5sSTazeZH6EkZU3gkag47XXlv7kjmXrA56iU0 9EALSpnORZ4gJPfPntZHvmVVjcB5sWa0zS/d4B6C7fuzzsFLi7dkFdb1fHbcBO+6fFzDKaHot JVmKWehzToNZ8ACWIdX4dNs/lZKWs9CC50MDUfEJekx5UlyRw8RK7z1XaRMFQlkpxrhcYLFKa vzc0bzraDYa/M2FQyR9rw/R2tMYFzBezsQj7UyLcbmRMh3eseK3IixspfibYxng+cP2iIfCN8 3S11zZE9qOK1sMMVQWUKOUH5gZNfyRoEK+FayJ/npnIemwuoSnh+1KjeGdsgow6rV/C3/3xQR CTI/n5jj6xlrMBnmMb13FBpnosfz48x6k0CUDkbX7AnIL5EWHOGQthQFF02WqqpdOFp/Uwrmi OHmWlE2djBYYGnOHCnI08OZi9WDPujWH4S6iX5SS1Ggkwan5rAdh7WvvxxM93XyeSeNKYKzpc rPXDrhpVJ/XtLJeUEOWJOo/yXgrhb0Mc9mjQ9pHq6eR1zBeY3FiL/ONlpjaH0z/0MlhnD5dWQ jgLZ2NIn9FQSEjNG3NQHH7D4ndwz+LQq1+4msxZqsYvpqd8UCwQDc/LpVZPwvWdwTYaXs1q9D wTLKA/bbrt7ezgwM4BOuSAGMN5YsThNC59ImO05I5NhbMcNFKcpcgTHPYHjkle4RA2wgTRlHY L0tUv8j27UVQmR4mo4kAbZV2FR6Zhbc6f9l7JJw9v19mDf+qo2AL6xSjZthD5IG7Lfm97WM5C kP7+141NEj2kxlYBDNUq6eknK6TiM3h5TqzZxh+topLZYvriGj7Q+1LSUPDDAKQ+NCxBcc/7x UF4aw2xcVtV+CM8dtuMHQDefLRY= Subject: Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Sep 2023 15:20:41 -0000 Hi Mike, [I took the liberty to remove some individual address from the Cc, as I = assume most/all already be covered by the lists] > On Sep 30, 2023, at 16:41, Mike Conlow wrote: >=20 > First, a thank you to Dave, and lots of you all, for longtime = shepherding of this community and efforts to make the Internet better.=20= >=20 > As I read this thread and think about the coming debate in the U.S., = two things come to mind: >=20 > 1. Ofcom is considering a net neutrality "clarification". The first = topic in the consultation is whether ISPs should be allowed to offer = "premium quality retail plans". It doesn't specify the technical = implementation, but there would be different plans for "users who mainly = stream" vs "people who use high quality virtual reality applications". = Apparently ISPs feel the existing NN rules are not clear on whether this = is allowed. [SM] Not sure this is not simply an attempt of using regulatory = divergence from the EU (IMHO for no good reason or outcome)... Also und = er the existing EU rules ISPs are arguably already free to treat the = whole class of latency sensitive VR to lower delay than bulk streaming = as long as they do son consistently and not based on commercial = relationships with the senders... During the covid19 lock downs the EU offered clarification on the = regulation that really drove home the do not discriminate inside of a = specific traffic class, and define classes by purely technical not = economical parameters. That said, I always like to look at data and = hence am happy to the the UK apparently prepping to run that experiment = (I am also happy not to live there right now not having to prticipate in = said experiment*). *) Other than that the british islands offer a lot of really great = places I certainly would like to live at, but I digress. >=20 > The question I'm thinking about is do we want an Internet where end = user plans are divided up this way? [SM] Personally, I consider internet access infrastructure and = do not think this looks like a good way forward. > And if not, is a NN regulation the right place to put those rules? [SM] Could well be, but depends on the framing, no? >=20 > 2. To the point in the PS of the below email, I would agree things are = mostly working in the EU, and in the US. But things are broken in = Germany to the point where consumers have a degraded Internet experience = because their ISP won't provision enough interconnection.=20 [SM] This a very peculiar case of the local incumbent Deutsche = Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship = in its network and tends to follow regulations to the letter (not = however necessarily to the intent, but they are not different from other = corporations of similar size)). DTAG is large enough to qualify as tier = 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody = for transit and peer with all other T1-ISPs, they also have a relative = large share of eye-balls in one of Europes larger and profitable = markets. They (as did AT&T and Verizon in the US and probably other ISPs = in similar positions as well) that most of their users traffic was = within network (e.g. from German companies hosted/homed by DTAG) or via = important partners like Google that have decent peering links (unclear = whether/if Google actually is charged for that) but that there is a = considerable number of services that reach DTAG eye-balls via their = transit, that is essentially via one of the other T1-ISPs (I simplify = here, I have no insight in the actual bisiness relationships between all = players). And now DTAG basically instructed its generally capable = network team to simply manage the size of the peering links with the big = transit-providers carefully so that they never fully clogg, but clearly = see increased packet loss and queueing delay during prime time. That in = turn is clearly a competition problem if streaming service A = judders/jitters/and buffers jumps between quality tiers while streaming = service B smoothly and boringly just streams at the desired resolution. = Now Telekom is happy to offer service A a product they call "internet = transit" but that is priced pretty high (I have seen some comparative = numbers for transit pricing in Germany I am not permitted to share or = reveal more about) so high in fact that no content provider that can = afford more than a single transit provider would use for anything but = reaching DTAG eye-balls or closely related ones (like in the past = SwissCom). This behaviour is not s secret but evades regulatory action, because it = does not openly violate the EU regulation which in the BEREC = interpretation does not really cover the interconnection side. DTAG is = careful enough to not purposefully target specific potential customers = but simply treats all traffic coming in/out via "other transit than its = own" as "has to tolerate overheated links during primetime".=20 > Are NN rules the right place to address this [SM] They could well be the actual text of the 2015/2120 does = not make a distinction between access and interconnection. But this is a = tricky field and will directly affect parts of larger ISP's core = business so I do not see this happen in the EU anytime soon, unless ISPs = like Telekom clearly abuses this in a way that is too obvious... ATM it = is mostly telecom, but I believe any of the big old monopoly incumbents = likely is big enough in its home market to pull of a stunt like this, so = there is the potential of someone over doing it... > and make sure it doesn't happen in the US? Or is one bad actor across = the EU and US the cost of doing business and the Internet ecosystem and = "market" are mostly solving the issue? [SM] As happy as I am to diss DTAG for that behavior (I am also = happy to praise it in ears where it shows exemplary behavior) DTAG is = not alone in that business acumen, I think that some of the big US = telco's dod/do exactly the same, but unlike telekom I have no evidence. Regards Sebastian P.S.: I was a customer at DTAG for several years and I did not notice = the conscious under-peering with the other T1 ISPs in my day to day = usage, so while the issue clearly and measurably exists it is not an = issue that normal users will encounter often and are also unlikely to = properly root-cause (the blame will likely land by my example service A = above). >=20 >=20 >=20 > On Sat, Sep 30, 2023 at 8:19=E2=80=AFAM Sebastian Moeller via Rpm = wrote: > Hi Frantisek, >=20 > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm = wrote: > >=20 > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a = great body of work (IMHO) done on this subject by Martin Geddes: > > https://www.martingeddes.com/1261-2/ > >=20 > > But let's pick one report written by his colleagues and published by = Ofcom (UK telecoms regulator): > >=20 > > =E2=80=A2 You cannot conflate =E2=80=98equality of [packet] = treatment=E2=80=99 with delivering equality of [user application] = outcomes. Only the latter matters, as ordinary users don=E2=80=99t care = what happened to the packets in transit. Yet the relevant academic = literature fixates on the local operation of the mechanisms (including = Traffic Management), not their global aggregate effect. >=20 > [SM] The EU laid out pretty clear why they mandated the NN = regulations in eu regulation 2015/2120: >=20 > [...] > (8) > When providing internet access services, providers of those services = should treat all traffic equally, without discrimination, restriction or = interference, independently of its sender or receiver, content, = application or service, or terminal equipment. According to general = principles of Union law and settled case-law, comparable situations = should not be treated differently and different situations should not be = treated in the same way unless such treatment is objectively justified. > (9) > The objective of reasonable traffic management is to contribute to an = efficient use of network resources and to an optimisation of overall = transmission quality responding to the objectively different technical = quality of service requirements of specific categories of traffic, and = thus of the content, applications and services transmitted. Reasonable = traffic management measures applied by providers of internet access = services should be transparent, non-discriminatory and proportionate, = and should not be based on commercial considerations. The requirement = for traffic management measures to be non-discriminatory does not = preclude providers of internet access services from implementing, in = order to optimise the overall transmission quality, traffic management = measures which differentiate between objectively different categories of = traffic. Any such differentiation should, in order to optimise overall = quality and user experience, be permitted only on the basis of = objectively different technical quality of service requirements (for = example, in terms of latency, jitter, packet loss, and bandwidth) of the = specific categories of traffic, and not on the basis of commercial = considerations. Such differentiating measures should be proportionate in = relation to the purpose of overall quality optimisation and should treat = equivalent traffic equally. Such measures should not be maintained for = longer than necessary. > (10) > Reasonable traffic management does not require techniques which = monitor the specific content of data traffic transmitted via the = internet access service. > (11) > Any traffic management practices which go beyond such reasonable = traffic management measures, by blocking, slowing down, altering, = restricting, interfering with, degrading or discriminating between = specific content, applications or services, or specific categories of = content, applications or services, should be prohibited, subject to the = justified and defined exceptions laid down in this Regulation. Those = exceptions should be subject to strict interpretation and to = proportionality requirements. Specific content, applications and = services, as well as specific categories thereof, should be protected = because of the negative impact on end-user choice and innovation of = blocking, or of other restrictive measures not falling within the = justified exceptions. Rules against altering content, applications or = services refer to a modification of the content of the communication, = but do not ban non-discriminatory data compression techniques which = reduce the size of a data file without any modification of the content. = Such compression enables a more efficient use of scarce resources and = serves the end-users=E2=80=99 interests by reducing data volumes, = increasing speed and enhancing the experience of using the content, = applications or services concerned. > (12) > Traffic management measures that go beyond such reasonable traffic = management measures may only be applied as necessary and for as long as = necessary to comply with the three justified exceptions laid down in = this Regulation. > [...] >=20 > There really is little IMHO that can be brought against these, all = pretty fair and reasonable. What it does is accept that internet access = is essential infrastructure and that hence access needs to be as well = regulated as access to water, electricity, gas, streets, ... . Yes this = has some consequences of what ISPs can and can not do. But this is = normal "cost of business". I for one am quite happy about this = regulation existing as locally it did away with some (not all) = shenanigans of some ISPs that were clearly not operating in the interest = of their paying eye-balls. >=20 > There is a whole cottage industry of consultants that decry the EU's = decision and try to lobby against it, but honestly reading these mostly = makes me think harsher regulation might be required (on consultans about = how much they are allowed to massage the facts ;) ).=20 >=20 > Regards > Sebastian >=20 > P.S.: Of course if we look close enough we surely can find = corner-cases where either the EU regulations or the translation into = national law result in less desirable outcomes, but "nothing is perfect" = and all in all the regulations seem to be "good enough". With the caveat = that explicitly excluding ISP interconnect from the regulations BEREC = essentially pointed the way for ISPs wanting to monetize their eye-balls = twice to do so via interconnects, but that only works for the 800 pound = gorillas and generally is not a game smaller ISPs can play. I do = understand why BEREC wants to stay out of the interconnection issue, as = this is rather complicated and the market seems to generally work = okay-ish (that is not badly enough to make intervention a hot-button = issue for voters and hence politicians). >=20 >=20 >=20 > >=20 > > All the best, > >=20 > > Frank > >=20 > > Frantisek (Frank) Borsik > >=20 > > =20 > >=20 > > https://www.linkedin.com/in/frantisekborsik > >=20 > > Signal, Telegram, WhatsApp: +421919416714=20 > >=20 > > iMessage, mobile: +420775230885 > >=20 > > Skype: casioa5302ca > >=20 > > frantisek.borsik@gmail.com > >=20 > >=20 > >=20 > > On Fri, Sep 29, 2023 at 6:15=E2=80=AFPM dan via Rpm = wrote: > > ok, lots and lots of great comments here for sure. =20 > >=20 > > bandwidth abundance: Not for most people and ISPs. The 'carriers' = aren't carrying to many places at affordable rates. I've pulled quotes = from Lumen and Zayo at over $5k/month/gig. We typically pay 900-1400 = for a gig of service. This isn't abundance and it's widespread and it = leaves only major providers that can afford/amortize out 100G transits = etc. > > My answer to this is one that Dave and I have bounced back and forth = is the idea of micro IXs in every municipality and having that somehow = tied into access to the ROW in counties etc. Not fully hashed out, but = the fiber is in the ground, it could be sold, but the carriers are not = well incentivised to sell it. It takes the better part of a year to get = a DIA within 100ft of a Lumen hut sometimes... Heck, it could even be a = government program to get an =CE=BCIX with x feet of every school, city = hall, and library. I don't care how it's done but this would get = abundance NEAR end users and open up essentially every town to = competition. > >=20 > > monopoly. This is a historical thing for most cable and DSL = incumbents. They have enjoyed virtual monopolies with cable owning = population centers and DSL owning the outskirts and there is no product = darwinism here where customer satisfaction is a pressure. That may not = be the future but it definitely is the past. These companies may have = to shift into customer satisfaction as a major part instead of a minor = part of their corporate culture to fend off fttx and ultra-modern wisps. > >=20 > > Starlink is not offering significant competition to major carriers. = Starlink's 1.5 million customers are at LEAST 90% pulled from other = satellite services and small ISPs. Spectrum and Comcast's losses to = starlink are measured in decimal points. > >=20 > > Only fttx and ultra-modern wireless tech really threatens these = incumbents. Typical wisps aren't putting a dent in these guys, just = scraping the paint off their bumper. We're pulling customers at the = scale of 'dozens' for example. Spectrum's management doesn't know we = exist we're such a small threat to them. =20 > >=20 > > But to further the point here, these fttx and ultra-modern wisps can = only exist in places where there is adequate carrier services to start = with. In areas where they spend the money and do the build but there = aren't good carrier services, those fiber services suck and the wISPs = start to claw back even with inferior technology. We've pulled quite a = few customers off fttx deployments because of this sort of situation. > >=20 > >=20 > > On Fri, Sep 29, 2023 at 7:28=E2=80=AFAM Rich Brown = wrote: > > Thank you Jonathan for this clear description of the issues and = their history. I wonder if there's a fourth one - privacy.=20 > >=20 > > Rosenworcel's talk = https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out = that ISPs might want to monetize our traffic patterns and location data. = (This is less of an issue in the EU, but the US remains a Wild West in = this regard.)=20 > >=20 > > I am hopeful that the FCC will include this in their NPRM (which = must be available by now but I haven't looked...) > >=20 > > - Rich Brown > >=20 > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm = wrote: > > >=20 > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat = wrote: > > >>=20 > > >> Dave T called out earlier that the rise of bittorrent was a large = part of the inital NN discussion here in the US. But a second large = portion was a money grab from ISPs thinking that they could hold up = large paid websites (netflix for example) for additional fees by = threatening to make their service less useful to their users (viewing = their users as an asset to be marketed to the websites rather than = customers to be satisfied by providing them access to the websites) > > >>=20 > > >> I don't know if a new round of "it's not fair that Netflix = doesn't pay us for the bandwidth to service them" would fall flat at = this point or not. > > >=20 > > > I think there were three more-or-less separate concerns which = have, over time, fallen under the same umbrella: > > >=20 > > >=20 > > > 1: Capacity-seeking flows tend to interfere with = latency-sensitive flows, and the "induced demand" phenomenon means that = increases in link rate do not in themselves solve this problem, even = though they may be sold as doing so. > > >=20 > > > This is directly addressed by properly-sized buffers and/or AQM, = and even better by FQ and SQM. It's a solved problem, so long as the = solutions are deployed. It's not usually necessary, for example, to = specifically enhance service for latency-sensitive traffic, if FQ does a = sufficiently good job. An increased link rate *does* enhance service = quality for both latency-sensitive and capacity-seeking traffic, = provided FQ is in use. > > >=20 > > >=20 > > > 2: Swarm traffic tends to drown out conventional traffic, due to = congestion control algorithms which try to be more-or-less fair on a = per-flow basis, and the substantially larger number of parallel flows = used by swarm traffic. This also caused subscribers using swarm traffic = to impair the service of subscribers who had nothing to do with it. > > >=20 > > > FQ on a per-flow basis (see problem 1) actually amplifies this = effect, and I think it was occasionally used as an argument for *not* = deploying FQ. ISPs' initial response was to outright block swarm = traffic where they could identify it, which was then softened to merely = throttling it heavily, before NN regulations intervened. Usage quotas = also showed up around this time, and were probably related to this = problem. > > >=20 > > > This has since been addressed by several means. ISPs may use FQ = on a per-subscriber basis to prevent one subscriber's heavy traffic from = degrading service for another. Swarm applications nowadays tend to = employ altruistic congestion control which deliberately compensates for = the large number of flows, and/or mark them with one or more of the = Least Effort class DSCPs. Hence, swarm applications are no longer as = damaging to service quality as they used to be. Usage quotas, however, = still remain in use as a profit centre, to the point where an = "unlimited" service is a rare and precious specimen in many = jurisdictions. > > >=20 > > >=20 > > > 3: ISPs merged with media distribution companies, creating a = conflict of interest in which the media side of the business wanted the = internet side to actively favour "their own" media traffic at the = expense of "the competition". Some ISPs began to actively degrade = Netflix traffic, in particular by refusing to provision adequate peering = capacity at the nodes through which Netflix traffic predominated, or by = zero-rating (for the purpose of usage quotas) traffic from their own = media empire while refusing to do the same for Netflix traffic. > > >=20 > > > **THIS** was the true core of Net Neutrality. NN regulations = forced ISPs to carry Netflix traffic with reasonable levels of service, = even though they didn't want to for purely selfish and greedy commercial = reasons. NN succeeded in curbing an anti-competitive and = consumer-hostile practice, which I am perfectly sure would resume just = as soon as NN regulations were repealed. > > >=20 > > > And this type of practice is just the sort of thing that = technologies like L4S are designed to support. The ISPs behind L4S = actively do not want a technology that works end-to-end over the general = Internet. They want something that can provide a domination service = within their own walled gardens. That's why L4S is a NN hazard, and why = they actively resisted all attempts to displace it with SCE. > > >=20 > > >=20 > > > All of the above were made more difficult to solve by the = monopolistic nature of the Internet service industry. It is actively = difficult for Internet users to move to a truly different service, = especially one based on a different link technology. When attempts are = made to increase competition, for example by deploying a publicly-funded = network, the incumbents actively sabotage those attempts by any means = they can. Monopolies are inherently customer-hostile, and arguments = based on market forces fail in their presence. > > >=20 > > > - Jonathan Morton > > >=20 > > > _______________________________________________ > > > Rpm mailing list > > > Rpm@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > >=20 > > _______________________________________________ > > Rpm mailing list > > Rpm@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/rpm > > _______________________________________________ > > Rpm mailing list > > Rpm@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/rpm >=20 > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm