[Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Mike,

[I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]


> On Sep 30, 2023, at 16:41, Mike Conlow <mconlow at cloudflare.com> wrote:
> 
> First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better. 
> 
> As I read this thread and think about the coming debate in the U.S., two things come to mind:
> 
> 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.

	[SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).


*) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.

> 
> The question I'm thinking about is do we want an Internet where end user plans are divided up this way?

	[SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.

> And if not, is a NN regulation the right place to put those rules?

	[SM] Could well be, but depends on the framing, no?

> 
> 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection. 

	[SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime". 


> Are NN rules the right place to address this

	[SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...

> and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?

	[SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.


Regards
	Sebastian

P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).


> 
> 
> 
> On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm at lists.bufferbloat.net> wrote:
> Hi Frantisek,
> 
> > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm at lists.bufferbloat.net> wrote:
> > 
> > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > https://www.martingeddes.com/1261-2/
> > 
> > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > 
> >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> 
>         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> 
> [...]
> (8)
> When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> (9)
> The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> (10)
> Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> (11)
> Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> (12)
> Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> [...]
> 
> There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> 
> There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 
> 
> Regards
>         Sebastian
> 
> P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> 
> 
> 
> > 
> > All the best,
> > 
> > Frank
> > 
> > Frantisek (Frank) Borsik
> > 
> >  
> > 
> > https://www.linkedin.com/in/frantisekborsik
> > 
> > Signal, Telegram, WhatsApp: +421919416714 
> > 
> > iMessage, mobile: +420775230885
> > 
> > Skype: casioa5302ca
> > 
> > frantisek.borsik at gmail.com
> > 
> > 
> > 
> > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm at lists.bufferbloat.net> wrote:
> > ok, lots and lots of great comments here for sure.  
> > 
> > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > 
> > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > 
> > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > 
> > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> > 
> > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > 
> > 
> > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover at gmail.com> wrote:
> > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> > 
> > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> > 
> > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > 
> > - Rich Brown
> > 
> > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm at lists.bufferbloat.net> wrote:
> > > 
> > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat at lists.bufferbloat.net> wrote:
> > >> 
> > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > >> 
> > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > 
> > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > 
> > > 
> > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > 
> > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > 
> > > 
> > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > 
> > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > 
> > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > 
> > > 
> > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > 
> > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > 
> > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > 
> > > 
> > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > 
> > > - Jonathan Morton
> > > 
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm at lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> > _______________________________________________
> > Rpm mailing list
> > Rpm at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Rpm mailing list
> Rpm at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm