From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 0D2543B29E; Sat, 30 Sep 2023 11:23:19 -0400 (EDT) Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-578b407045bso11766145a12.0; Sat, 30 Sep 2023 08:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696087399; x=1696692199; darn=lists.bufferbloat.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=yxVfVyEesKJQPeW3FhkAIITEfQqEIZxKmcXfzNK7CPY=; b=WXL9IVLJXOEwqX7ZQ4luERksC3eiWk3hCXA1foevQ8qHBGNJ/ul1vzn4LSMIlcn8ik B8TPs9b+IrvKpIvrqlS9ZNUh43E7kPtUYDsbM/XVE3EP+xvWSsJ4bzs7EBECBHrKb/dR lnGsLqazdu7C5RRKTpMK9PYmWVBmFhNv0sLyE7nS+jWzLqsS0d/0sAwJp3RopvRtFsbI /m16ZKwT9qlPHbsd9PpgWugjOEnLL3TzSo0Z6UPba2fOajWpmLALKRmJKtkifG5JbO5M pQqqGoZvvhd8sOzPomXMV1munKtN5KjfjhcmBSWgYeeIugurzIesvpuSXSRPtVKANQQD LnpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696087399; x=1696692199; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yxVfVyEesKJQPeW3FhkAIITEfQqEIZxKmcXfzNK7CPY=; b=e+iyYyGFiCBl/J79nWKK/YhfZE2KX+MaXhrYGIij7S8XE4YCrS7UZpsbiU9cmuLS49 jHzMkhHfxUNyqgNFr7Ar+ASgu1uLG2lDTPC1Jbzb2VBHZ+2iYUKydJ6A11UASN3P7mjL t8Rlm9QJh5xK8EFkHfJpwJmfzr2jdWar6pex9LA/uLcT7kZt4EJ0HSJ+EXe+CeWLVWts 1awiNWRpSgj+ipQCBhY8LDHA1SckDjJH544sZsC+xMNtJLgE+FZfGbv7QplceNfm6X2Q l0ixwpOT0xK57YqVF3ALKPkH5J+UUxViBwS/0r4f/cdXAow9yPeDPcXSBalWW+qc5BsM PQPQ== X-Gm-Message-State: AOJu0YxYSOPAM/EMxUzHx4sSmMA1Z2oRJfxBB3w8feLGGbakbezZaJPK nNoghKwtNomtZH5HDCq2ILETAXZnd3yWMJxX19E= X-Google-Smtp-Source: AGHT+IGuHwete4EneKtziO4JW0WfURVgNggZM8ayClulsajSv6f6vR0PHdNTSXND6WBNrYBvBSZuf6SpZ7ENBRaThIc= X-Received: by 2002:a05:6a21:3e0a:b0:162:c4db:1dd4 with SMTP id bk10-20020a056a213e0a00b00162c4db1dd4mr6491883pzc.26.1696087398497; Sat, 30 Sep 2023 08:23:18 -0700 (PDT) MIME-Version: 1.0 References: <5so3r00n-31pn-14s7-7775-08731s3s551r@ynat.uz> <7508FE73-A154-4CBA-984C-748A80C5FFEC@gmail.com> <039490DA-48A7-4AE2-B00F-AA2A260FB747@gmail.com> <3F6D16C8-530E-4066-8B6A-C0644B3C2D2C@gmx.de> <4129491C-FFC7-4E5E-A5A4-8CBE9B5C5255@gmx.de> In-Reply-To: <4129491C-FFC7-4E5E-A5A4-8CBE9B5C5255@gmx.de> From: Dave Taht Date: Sat, 30 Sep 2023 08:23:06 -0700 Message-ID: To: Sebastian Moeller Cc: Mike Conlow , Rpm , bloat , Dave Taht via Starlink Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Sep 2023 15:23:20 -0000 The starlink list was not originally cc=C2=B4d and yet since I think this debate concerns that also, I have added the cc back. Carry on! On Sat, Sep 30, 2023 at 8:20=E2=80=AFAM Sebastian Moeller via LibreQoS wrote: > > Hi Mike, > > [I took the liberty to remove some individual address from the Cc, as I a= ssume most/all already be covered by the lists] > > > > On Sep 30, 2023, at 16:41, Mike Conlow wrote: > > > > First, a thank you to Dave, and lots of you all, for longtime shepherdi= ng of this community and efforts to make the Internet better. > > > > As I read this thread and think about the coming debate in the U.S., tw= o things come to mind: > > > > 1. Ofcom is considering a net neutrality "clarification". The first top= ic in the consultation is whether ISPs should be allowed to offer "premium = quality retail plans". It doesn't specify the technical implementation, but= there would be different plans for "users who mainly stream" vs "people wh= o use high quality virtual reality applications". Apparently ISPs feel the = existing NN rules are not clear on whether this is allowed. > > [SM] Not sure this is not simply an attempt of using regulatory d= ivergence from the EU (IMHO for no good reason or outcome)... Also und er t= he existing EU rules ISPs are arguably already free to treat the whole clas= s of latency sensitive VR to lower delay than bulk streaming as long as the= y do son consistently and not based on commercial relationships with the se= nders... > During the covid19 lock downs the EU offered clarification on the regulat= ion that really drove home the do not discriminate inside of a specific tra= ffic class, and define classes by purely technical not economical parameter= s. That said, I always like to look at data and hence am happy to the the U= K apparently prepping to run that experiment (I am also happy not to live t= here right now not having to prticipate in said experiment*). > > > *) Other than that the british islands offer a lot of really great places= I certainly would like to live at, but I digress. > > > > > The question I'm thinking about is do we want an Internet where end use= r plans are divided up this way? > > [SM] Personally, I consider internet access infrastructure and do= not think this looks like a good way forward. > > > And if not, is a NN regulation the right place to put those rules? > > [SM] Could well be, but depends on the framing, no? > > > > > 2. To the point in the PS of the below email, I would agree things are = mostly working in the EU, and in the US. But things are broken in Germany t= o the point where consumers have a degraded Internet experience because the= ir ISP won't provision enough interconnection. > > [SM] This a very peculiar case of the local incumbent Deutsche Te= lekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in i= ts network and tends to follow regulations to the letter (not however neces= sarily to the intent, but they are not different from other corporations of= similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is= , to the best of our knowledge they do not pay anybody for transit and peer= with all other T1-ISPs, they also have a relative large share of eye-balls= in one of Europes larger and profitable markets. They (as did AT&T and Ver= izon in the US and probably other ISPs in similar positions as well) that m= ost of their users traffic was within network (e.g. from German companies h= osted/homed by DTAG) or via important partners like Google that have decent= peering links (unclear whether/if Google actually is charged for that) but= that there is a considerable number of services that reach DTAG eye-balls = via their transit, that is essentially via one of the other T1-ISPs (I simp= lify here, I have no insight in the actual bisiness relationships between a= ll players). And now DTAG basically instructed its generally capable networ= k team to simply manage the size of the peering links with the big transit-= providers carefully so that they never fully clogg, but clearly see increas= ed packet loss and queueing delay during prime time. That in turn is clearl= y a competition problem if streaming service A judders/jitters/and buffers = jumps between quality tiers while streaming service B smoothly and boringly= just streams at the desired resolution. Now Telekom is happy to offer serv= ice A a product they call "internet transit" but that is priced pretty high= (I have seen some comparative numbers for transit pricing in Germany I am = not permitted to share or reveal more about) so high in fact that no conten= t provider that can afford more than a single transit provider would use fo= r anything but reaching DTAG eye-balls or closely related ones (like in the= past SwissCom). > This behaviour is not s secret but evades regulatory action, because it d= oes not openly violate the EU regulation which in the BEREC interpretation = does not really cover the interconnection side. DTAG is careful enough to n= ot purposefully target specific potential customers but simply treats all t= raffic coming in/out via "other transit than its own" as "has to tolerate o= verheated links during primetime". > > > > Are NN rules the right place to address this > > [SM] They could well be the actual text of the 2015/2120 does not= make a distinction between access and interconnection. But this is a trick= y field and will directly affect parts of larger ISP's core business so I d= o not see this happen in the EU anytime soon, unless ISPs like Telekom clea= rly abuses this in a way that is too obvious... ATM it is mostly telecom, b= ut I believe any of the big old monopoly incumbents likely is big enough in= its home market to pull of a stunt like this, so there is the potential of= someone over doing it... > > > and make sure it doesn't happen in the US? Or is one bad actor across t= he EU and US the cost of doing business and the Internet ecosystem and "mar= ket" are mostly solving the issue? > > [SM] As happy as I am to diss DTAG for that behavior (I am also h= appy to praise it in ears where it shows exemplary behavior) DTAG is not al= one in that business acumen, I think that some of the big US telco's dod/do= exactly the same, but unlike telekom I have no evidence. > > > Regards > Sebastian > > P.S.: I was a customer at DTAG for several years and I did not notice the= conscious under-peering with the other T1 ISPs in my day to day usage, so = while the issue clearly and measurably exists it is not an issue that norma= l users will encounter often and are also unlikely to properly root-cause (= the blame will likely land by my example service A above). > > > > > > > > > > On Sat, Sep 30, 2023 at 8:19=E2=80=AFAM Sebastian Moeller via Rpm wrote: > > Hi Frantisek, > > > > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm wrote: > > > > > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a g= reat body of work (IMHO) done on this subject by Martin Geddes: > > > https://www.martingeddes.com/1261-2/ > > > > > > But let's pick one report written by his colleagues and published by = Ofcom (UK telecoms regulator): > > > > > > =E2=80=A2 You cannot conflate =E2=80=98equality of [packet] tre= atment=E2=80=99 with delivering equality of [user application] outcomes. On= ly the latter matters, as ordinary users don=E2=80=99t care what happened t= o the packets in transit. Yet the relevant academic literature fixates on t= he local operation of the mechanisms (including Traffic Management), not th= eir global aggregate effect. > > > > [SM] The EU laid out pretty clear why they mandated the NN regu= lations in eu regulation 2015/2120: > > > > [...] > > (8) > > When providing internet access services, providers of those services sh= ould treat all traffic equally, without discrimination, restriction or inte= rference, independently of its sender or receiver, content, application or = service, or terminal equipment. According to general principles of Union la= w and settled case-law, comparable situations should not be treated differe= ntly and different situations should not be treated in the same way unless = such treatment is objectively justified. > > (9) > > The objective of reasonable traffic management is to contribute to an e= fficient use of network resources and to an optimisation of overall transmi= ssion quality responding to the objectively different technical quality of = service requirements of specific categories of traffic, and thus of the con= tent, applications and services transmitted. Reasonable traffic management = measures applied by providers of internet access services should be transpa= rent, non-discriminatory and proportionate, and should not be based on comm= ercial considerations. The requirement for traffic management measures to b= e non-discriminatory does not preclude providers of internet access service= s from implementing, in order to optimise the overall transmission quality,= traffic management measures which differentiate between objectively differ= ent categories of traffic. Any such differentiation should, in order to opt= imise overall quality and user experience, be permitted only on the basis o= f objectively different technical quality of service requirements (for exam= ple, in terms of latency, jitter, packet loss, and bandwidth) of the specif= ic categories of traffic, and not on the basis of commercial considerations= . Such differentiating measures should be proportionate in relation to the = purpose of overall quality optimisation and should treat equivalent traffic= equally. Such measures should not be maintained for longer than necessary. > > (10) > > Reasonable traffic management does not require techniques which monitor= the specific content of data traffic transmitted via the internet access s= ervice. > > (11) > > Any traffic management practices which go beyond such reasonable traffi= c management measures, by blocking, slowing down, altering, restricting, in= terfering with, degrading or discriminating between specific content, appli= cations or services, or specific categories of content, applications or ser= vices, should be prohibited, subject to the justified and defined exception= s laid down in this Regulation. Those exceptions should be subject to stric= t interpretation and to proportionality requirements. Specific content, app= lications and services, as well as specific categories thereof, should be p= rotected because of the negative impact on end-user choice and innovation o= f blocking, or of other restrictive measures not falling within the justifi= ed exceptions. Rules against altering content, applications or services ref= er to a modification of the content of the communication, but do not ban no= n-discriminatory data compression techniques which reduce the size of a dat= a file without any modification of the content. Such compression enables a = more efficient use of scarce resources and serves the end-users=E2=80=99 in= terests by reducing data volumes, increasing speed and enhancing the experi= ence of using the content, applications or services concerned. > > (12) > > Traffic management measures that go beyond such reasonable traffic mana= gement measures may only be applied as necessary and for as long as necessa= ry to comply with the three justified exceptions laid down in this Regulati= on. > > [...] > > > > There really is little IMHO that can be brought against these, all pret= ty fair and reasonable. What it does is accept that internet access is esse= ntial infrastructure and that hence access needs to be as well regulated as= access to water, electricity, gas, streets, ... . Yes this has some conseq= uences of what ISPs can and can not do. But this is normal "cost of busines= s". I for one am quite happy about this regulation existing as locally it d= id away with some (not all) shenanigans of some ISPs that were clearly not = operating in the interest of their paying eye-balls. > > > > There is a whole cottage industry of consultants that decry the EU's de= cision and try to lobby against it, but honestly reading these mostly makes= me think harsher regulation might be required (on consultans about how muc= h they are allowed to massage the facts ;) ). > > > > Regards > > Sebastian > > > > P.S.: Of course if we look close enough we surely can find corner-cases= where either the EU regulations or the translation into national law resul= t in less desirable outcomes, but "nothing is perfect" and all in all the r= egulations seem to be "good enough". With the caveat that explicitly exclud= ing ISP interconnect from the regulations BEREC essentially pointed the way= for ISPs wanting to monetize their eye-balls twice to do so via interconne= cts, but that only works for the 800 pound gorillas and generally is not a = game smaller ISPs can play. I do understand why BEREC wants to stay out of = the interconnection issue, as this is rather complicated and the market see= ms to generally work okay-ish (that is not badly enough to make interventio= n a hot-button issue for voters and hence politicians). > > > > > > > > > > > > All the best, > > > > > > Frank > > > > > > Frantisek (Frank) Borsik > > > > > > > > > > > > https://www.linkedin.com/in/frantisekborsik > > > > > > Signal, Telegram, WhatsApp: +421919416714 > > > > > > iMessage, mobile: +420775230885 > > > > > > Skype: casioa5302ca > > > > > > frantisek.borsik@gmail.com > > > > > > > > > > > > On Fri, Sep 29, 2023 at 6:15=E2=80=AFPM dan via Rpm wrote: > > > ok, lots and lots of great comments here for sure. > > > > > > bandwidth abundance: Not for most people and ISPs. The 'carriers' a= ren't carrying to many places at affordable rates. I've pulled quotes from= Lumen and Zayo at over $5k/month/gig. We typically pay 900-1400 for a gig= of service. This isn't abundance and it's widespread and it leaves only m= ajor providers that can afford/amortize out 100G transits etc. > > > My answer to this is one that Dave and I have bounced back and forth = is the idea of micro IXs in every municipality and having that somehow tied= into access to the ROW in counties etc. Not fully hashed out, but the fib= er is in the ground, it could be sold, but the carriers are not well incent= ivised to sell it. It takes the better part of a year to get a DIA within = 100ft of a Lumen hut sometimes... Heck, it could even be a government prog= ram to get an =CE=BCIX with x feet of every school, city hall, and library.= I don't care how it's done but this would get abundance NEAR end users an= d open up essentially every town to competition. > > > > > > monopoly. This is a historical thing for most cable and DSL incumben= ts. They have enjoyed virtual monopolies with cable owning population cent= ers and DSL owning the outskirts and there is no product darwinism here whe= re customer satisfaction is a pressure. That may not be the future but it = definitely is the past. These companies may have to shift into customer sa= tisfaction as a major part instead of a minor part of their corporate cultu= re to fend off fttx and ultra-modern wisps. > > > > > > Starlink is not offering significant competition to major carriers. = Starlink's 1.5 million customers are at LEAST 90% pulled from other satel= lite services and small ISPs. Spectrum and Comcast's losses to starlink ar= e measured in decimal points. > > > > > > Only fttx and ultra-modern wireless tech really threatens these incum= bents. Typical wisps aren't putting a dent in these guys, just scraping th= e paint off their bumper. We're pulling customers at the scale of 'dozens'= for example. Spectrum's management doesn't know we exist we're such a sma= ll threat to them. > > > > > > But to further the point here, these fttx and ultra-modern wisps can = only exist in places where there is adequate carrier services to start with= . In areas where they spend the money and do the build but there aren't go= od carrier services, those fiber services suck and the wISPs start to claw = back even with inferior technology. We've pulled quite a few customers off= fttx deployments because of this sort of situation. > > > > > > > > > On Fri, Sep 29, 2023 at 7:28=E2=80=AFAM Rich Brown wrote: > > > Thank you Jonathan for this clear description of the issues and their= history. I wonder if there's a fourth one - privacy. > > > > > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257= A1.pdf also points out that ISPs might want to monetize our traffic pattern= s and location data. (This is less of an issue in the EU, but the US remain= s a Wild West in this regard.) > > > > > > I am hopeful that the FCC will include this in their NPRM (which must= be available by now but I haven't looked...) > > > > > > - Rich Brown > > > > > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm wrote: > > > > > > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat wrote: > > > >> > > > >> Dave T called out earlier that the rise of bittorrent was a large = part of the inital NN discussion here in the US. But a second large portion= was a money grab from ISPs thinking that they could hold up large paid web= sites (netflix for example) for additional fees by threatening to make thei= r service less useful to their users (viewing their users as an asset to be= marketed to the websites rather than customers to be satisfied by providin= g them access to the websites) > > > >> > > > >> I don't know if a new round of "it's not fair that Netflix doesn't= pay us for the bandwidth to service them" would fall flat at this point or= not. > > > > > > > > I think there were three more-or-less separate concerns which have,= over time, fallen under the same umbrella: > > > > > > > > > > > > 1: Capacity-seeking flows tend to interfere with latency-sensitive= flows, and the "induced demand" phenomenon means that increases in link ra= te do not in themselves solve this problem, even though they may be sold as= doing so. > > > > > > > > This is directly addressed by properly-sized buffers and/or AQM, an= d even better by FQ and SQM. It's a solved problem, so long as the solutio= ns are deployed. It's not usually necessary, for example, to specifically = enhance service for latency-sensitive traffic, if FQ does a sufficiently go= od job. An increased link rate *does* enhance service quality for both lat= ency-sensitive and capacity-seeking traffic, provided FQ is in use. > > > > > > > > > > > > 2: Swarm traffic tends to drown out conventional traffic, due to c= ongestion control algorithms which try to be more-or-less fair on a per-flo= w basis, and the substantially larger number of parallel flows used by swar= m traffic. This also caused subscribers using swarm traffic to impair the = service of subscribers who had nothing to do with it. > > > > > > > > FQ on a per-flow basis (see problem 1) actually amplifies this effe= ct, and I think it was occasionally used as an argument for *not* deploying= FQ. ISPs' initial response was to outright block swarm traffic where they= could identify it, which was then softened to merely throttling it heavily= , before NN regulations intervened. Usage quotas also showed up around thi= s time, and were probably related to this problem. > > > > > > > > This has since been addressed by several means. ISPs may use FQ on= a per-subscriber basis to prevent one subscriber's heavy traffic from degr= ading service for another. Swarm applications nowadays tend to employ altr= uistic congestion control which deliberately compensates for the large numb= er of flows, and/or mark them with one or more of the Least Effort class DS= CPs. Hence, swarm applications are no longer as damaging to service qualit= y as they used to be. Usage quotas, however, still remain in use as a prof= it centre, to the point where an "unlimited" service is a rare and precious= specimen in many jurisdictions. > > > > > > > > > > > > 3: ISPs merged with media distribution companies, creating a confl= ict of interest in which the media side of the business wanted the internet= side to actively favour "their own" media traffic at the expense of "the c= ompetition". Some ISPs began to actively degrade Netflix traffic, in parti= cular by refusing to provision adequate peering capacity at the nodes throu= gh which Netflix traffic predominated, or by zero-rating (for the purpose o= f usage quotas) traffic from their own media empire while refusing to do th= e same for Netflix traffic. > > > > > > > > **THIS** was the true core of Net Neutrality. NN regulations force= d ISPs to carry Netflix traffic with reasonable levels of service, even tho= ugh they didn't want to for purely selfish and greedy commercial reasons. = NN succeeded in curbing an anti-competitive and consumer-hostile practice, = which I am perfectly sure would resume just as soon as NN regulations were = repealed. > > > > > > > > And this type of practice is just the sort of thing that technologi= es like L4S are designed to support. The ISPs behind L4S actively do not w= ant a technology that works end-to-end over the general Internet. They wan= t something that can provide a domination service within their own walled g= ardens. That's why L4S is a NN hazard, and why they actively resisted all = attempts to displace it with SCE. > > > > > > > > > > > > All of the above were made more difficult to solve by the monopolis= tic nature of the Internet service industry. It is actively difficult for = Internet users to move to a truly different service, especially one based o= n a different link technology. When attempts are made to increase competit= ion, for example by deploying a publicly-funded network, the incumbents act= ively sabotage those attempts by any means they can. Monopolies are inhere= ntly customer-hostile, and arguments based on market forces fail in their p= resence. > > > > > > > > - Jonathan Morton > > > > > > > > _______________________________________________ > > > > Rpm mailing list > > > > Rpm@lists.bufferbloat.net > > > > https://lists.bufferbloat.net/listinfo/rpm > > > > > > _______________________________________________ > > > Rpm mailing list > > > Rpm@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > > > _______________________________________________ > > > Rpm mailing list > > > Rpm@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > > > > _______________________________________________ > > Rpm mailing list > > Rpm@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/rpm > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos --=20 Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.htm= l Dave T=C3=A4ht CSO, LibreQos