Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Livingood, Jason]

On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:chromatix99@gmail.com>> wrote:
> Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated

That is not true and really not worth re-litigating here. 

> NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons. 

NN regulations played no role whatsoever in the resolution of that conflict - a business arrangement was reached, just as it was in the SK Telecom example recently: https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to 

> ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet. 

That's simply not true. As someone running an L4S field trial right now - we want the technology to get the widest possible deployment and be fully end-to-end. Why else would there be so much effort to ensure that ECN and DSCP marks can traverse network domain boundaries for example? Why else would there be strong app developer interest? What evidence do you have to show that anyone working on L4S want to create a walled garden? If anything, it seems the opposite of 5G network slicing, which seems to me personally to be another 3GPP run at walled garden stuff (like IMS). Ultimately it is like a lot of other IETF work -- it is an interesting technology and we'll have to see whether it gets good adoption - the 'market' will decide. 

> They want something that can provide a domination service within their own walled gardens. 

Also not correct. And last time I checked the balance sheets of companies in these sectors - video streaming services were losing money while provision of internet services were financially healthy. 

JL

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 2563 bytes --]

On Fri, Sep 29, 2023 at 7:17 AM Livingood, Jason via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:
> chromatix99@gmail.com>> wrote:
> > Some ISPs began to actively degrade Netflix traffic, in particular by
> refusing to provision adequate peering capacity at the nodes through which
> Netflix traffic predominated
>
> That is not true and really not worth re-litigating here.
>
> > NN regulations forced ISPs to carry Netflix traffic with reasonable
> levels of service, even though they didn't want to for purely selfish and
> greedy commercial reasons.
>
> NN regulations played no role whatsoever in the resolution of that
> conflict - a business arrangement was reached, just as it was in the SK
> Telecom example recently:
> https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
>
> > ISPs behind L4S actively do not want a technology that works end-to-end
> over the general Internet.
>
> That's simply not true. As someone running an L4S field trial right now -
> we want the technology to get the widest possible deployment and be fully
> end-to-end. Why else would there be so much effort to ensure that ECN and
> DSCP marks can traverse network domain boundaries for example? Why else
> would there be strong app developer interest? What evidence do you have to
> show that anyone working on L4S want to create a walled garden? If
> anything, it seems the opposite of 5G network slicing, which seems to me
> personally to be another 3GPP run at walled garden stuff (like IMS).
> Ultimately it is like a lot of other IETF work -- it is an interesting
> technology and we'll have to see whether it gets good adoption - the
> 'market' will decide.
>
> > They want something that can provide a domination service within their
> own walled gardens.
>
> Also not correct. And last time I checked the balance sheets of companies
> in these sectors - video streaming services were losing money while
> provision of internet services were financially healthy.
>
> JL
>
>
>
I think this stuff degrades into conspiracy theory often enough.  While I
don't discount the possibility of collusion, I don't give these
people/groups credit enough to pull of a mass scale conspiracy either....
If netflix is jammed down to small of a pipe at an ISP, that's more likely
(IMO...) disorganization or incompetence or disinterest over conspiracy.
 I feel the same about government in general...

[-- Attachment #2: Type: text/html, Size: 3210 bytes --]

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 3560 bytes --]

>
> > They want something that can provide a domination service within their
> own walled gardens.
> Also not correct. And last time I checked the balance sheets of companies
> in these sectors - *video streaming services were losing money while
> provision of internet services were financially healthy. *


Indeed, Jason:
https://www.vulture.com/2023/06/streaming-industry-netflix-max-disney-hulu-apple-tv-prime-video-peacock-paramount.html


All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Fri, Sep 29, 2023 at 5:53 PM dan via Rpm <rpm@lists.bufferbloat.net>
wrote:

>
>
> On Fri, Sep 29, 2023 at 7:17 AM Livingood, Jason via LibreQoS <
> libreqos@lists.bufferbloat.net> wrote:
>
>> On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:
>> chromatix99@gmail.com>> wrote:
>> > Some ISPs began to actively degrade Netflix traffic, in particular by
>> refusing to provision adequate peering capacity at the nodes through which
>> Netflix traffic predominated
>>
>> That is not true and really not worth re-litigating here.
>>
>> > NN regulations forced ISPs to carry Netflix traffic with reasonable
>> levels of service, even though they didn't want to for purely selfish and
>> greedy commercial reasons.
>>
>> NN regulations played no role whatsoever in the resolution of that
>> conflict - a business arrangement was reached, just as it was in the SK
>> Telecom example recently:
>> https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
>>
>> > ISPs behind L4S actively do not want a technology that works end-to-end
>> over the general Internet.
>>
>> That's simply not true. As someone running an L4S field trial right now -
>> we want the technology to get the widest possible deployment and be fully
>> end-to-end. Why else would there be so much effort to ensure that ECN and
>> DSCP marks can traverse network domain boundaries for example? Why else
>> would there be strong app developer interest? What evidence do you have to
>> show that anyone working on L4S want to create a walled garden? If
>> anything, it seems the opposite of 5G network slicing, which seems to me
>> personally to be another 3GPP run at walled garden stuff (like IMS).
>> Ultimately it is like a lot of other IETF work -- it is an interesting
>> technology and we'll have to see whether it gets good adoption - the
>> 'market' will decide.
>>
>> > They want something that can provide a domination service within their
>> own walled gardens.
>>
>> Also not correct. And last time I checked the balance sheets of companies
>> in these sectors - video streaming services were losing money while
>> provision of internet services were financially healthy.
>>
>> JL
>>
>>
>>
> I think this stuff degrades into conspiracy theory often enough.  While I
> don't discount the possibility of collusion, I don't give these
> people/groups credit enough to pull of a mass scale conspiracy either....
> If netflix is jammed down to small of a pipe at an ISP, that's more likely
> (IMO...) disorganization or incompetence or disinterest over conspiracy.
>  I feel the same about government in general...
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
>

[-- Attachment #2: Type: text/html, Size: 6299 bytes --]

[Rpm] net neutrality back in the news

[Dave Taht]

Jason just did a beautiful thread as to what was the original source
of the network neutrality
bittorrent vs voip bufferbloat blowup.

https://twitter.com/jlivingood/status/1707078242857849244

Seeing all the political activity tied onto it since (and now again)
reminds of two families at war about an incident that had happened
generations and generations before, where the two sides no longer
remembered why they hated each other so, but just went on hating, and
not forgiving, and not moving on.

Yes, there are entirely separate and additional NN issues, but the
technical problem of providing common carriage between two very
different network application types (voip/gaming vs file transfer) is
thoroughly solved now, and if only all sides recognised at least this
much, and made peace over it, and worked together to deploy those
solutions, maybe, just maybe, we could find mutually satisfactory
solutions to the other problems that plague the internet today, like
security, and the ipv6 rollout.

If anyone here knows anyone more political, still vibrating with 10+
years of outrage about NN on this fronts, on one side or the other, if
you could sit them down, over a beer, and try to explain that at the
start it was a technical problem nobody understood at the time, maybe
that would help.

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] net neutrality back in the news

[Sebastian Moeller]

Hi Dave,

please excuse a number of tangents below ;)


> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Jason just did a beautiful thread as to what was the original source
> of the network neutrality
> bittorrent vs voip bufferbloat blowup.
> 
> https://twitter.com/jlivingood/status/1707078242857849244

	But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.


*) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.


> 
> Seeing all the political activity tied onto it since (and now again)
> reminds of two families at war about an incident that had happened
> generations and generations before, where the two sides no longer
> remembered why they hated each other so, but just went on hating, and
> not forgiving, and not moving on.
> 
> Yes, there are entirely separate and additional NN issues, but the
> technical problem of providing common carriage between two very
> different network application types (voip/gaming vs file transfer) is
> thoroughly solved now,

	I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
	I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...


**) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.


> and if only all sides recognised at least this
> much, and made peace over it, and worked together to deploy those
> solutions, maybe, just maybe, we could find mutually satisfactory
> solutions to the other problems that plague the internet today, like
> security, and the ipv6 rollout.

	+1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry. Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****) 


***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.

****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).

> If anyone here knows anyone more political, still vibrating with 10+
> years of outrage about NN on this fronts, on one side or the other, if
> you could sit them down, over a beer, and try to explain that at the
> start it was a technical problem nobody understood at the time, maybe
> that would help.

	So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).


*****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.

Regards
	Sebastian


> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] net neutrality back in the news

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 9345 bytes --]

On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> please excuse a number of tangents below ;)

It would be nice, if as a (dis)organisation... the bufferbloat team
could focus on somehow getting both sides of the network neutrality
debate deeplying understanding the technological problem their
pre-conceptions face, and the (now readily available and inexpensive)
solutions that could be deployed, by most ISPs, over a weekend. We are
regularly bringing up a few thousand people a week on libreqos (that
we know of), and then of course, there are all the home routers and
CPE that are increasingly capable of doing the right thing.

The time to try and shift the memes in in the coming days, and weeks.

So ya'all being distracting below... aggh... ok, I'll bite.

>
>
> > On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >
> > Jason just did a beautiful thread as to what was the original source
> > of the network neutrality
> > bittorrent vs voip bufferbloat blowup.
> >
> > https://twitter.com/jlivingood/status/1707078242857849244
>
>         But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.

The average use at peak hours for home broadband is below 5mbits per
subscriber regardless of on a 25mbit or gbit plan. Remarkably,
business plans are less (but more bursty). Oversubscription within a
set of parameters is needed because, in part because upstream
bandwidth to the internet remains expensive (although I hope that cost
continues to drop rapidly), and in part, because it is needlessly
expensive to provision exactly the contracted rate to the customer. As
one example, I use the inverse square law as a rule of thumb for
wireless deployments - the range you can get from a 25/10 deployment
is geometrically better than 100/25, and the average bandwidth usage
nearly the same.

Agreeing on industry standards for what the "slack" should be, might be of help.

>
>
> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.

We have discussed elsewhere, the idea of a minimum contracted rate
(down to), which is easier to reason about.

>
> >
> > Seeing all the political activity tied onto it since (and now again)
> > reminds of two families at war about an incident that had happened
> > generations and generations before, where the two sides no longer
> > remembered why they hated each other so, but just went on hating, and
> > not forgiving, and not moving on.
> >
> > Yes, there are entirely separate and additional NN issues, but the
> > technical problem of providing common carriage between two very
> > different network application types (voip/gaming vs file transfer) is
> > thoroughly solved now,
>
>         I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

Torrent problem thoroughly solved with FQ on the path and backpressure
from the mac scheduler.

https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

>         I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>
>
> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>
>
> > and if only all sides recognised at least this
> > much, and made peace over it, and worked together to deploy those
> > solutions, maybe, just maybe, we could find mutually satisfactory
> > solutions to the other problems that plague the internet today, like
> > security, and the ipv6 rollout.
>
>         +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.

I really would like to see a push for IPv6 mandated as a part of the
BEAD programs.

> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)

There is a separate NRPM going on for the new cybersecurity label at
the FCC. If I had time, and a partner,
we could rework the doc we did a few years ago. It is due oct 6.

>
>
> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>
> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>
> > If anyone here knows anyone more political, still vibrating with 10+
> > years of outrage about NN on this fronts, on one side or the other, if
> > you could sit them down, over a beer, and try to explain that at the
> > start it was a technical problem nobody understood at the time, maybe
> > that would help.
>
>         So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).

Governments make safer markets feasible.

>
> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>
> Regards
>         Sebastian
>
>
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: 2510.jpeg --]
[-- Type: image/jpeg, Size: 66803 bytes --]

Re: [Rpm] net neutrality back in the news

[Sebastian Moeller]

> On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>> please excuse a number of tangents below ;)
> 
> It would be nice, if as a (dis)organisation... the bufferbloat team
> could focus on somehow getting both sides of the network neutrality
> debate deeplying understanding the technological problem their
> pre-conceptions face, and the (now readily available and inexpensive)
> solutions that could be deployed, by most ISPs, over a weekend. We are
> regularly bringing up a few thousand people a week on libreqos (that
> we know of), and then of course, there are all the home routers and
> CPE that are increasingly capable of doing the right thing.
> 
> The time to try and shift the memes in in the coming days, and weeks.
> 
> So ya'all being distracting below... aggh... ok, I'll bite.

	[SM] Sorry to drag you into the weeds...


> 
>> 
>> 
>>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Jason just did a beautiful thread as to what was the original source
>>> of the network neutrality
>>> bittorrent vs voip bufferbloat blowup.
>>> 
>>> https://twitter.com/jlivingood/status/1707078242857849244
>> 
>>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> 
> The average use at peak hours for home broadband is below 5mbits per
> subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> business plans are less (but more bursty). Oversubscription within a
> set of parameters is needed because, in part because upstream
> bandwidth to the internet remains expensive (although I hope that cost
> continues to drop rapidly), and in part, because it is needlessly
> expensive to provision exactly the contracted rate to the customer. As
> one example, I use the inverse square law as a rule of thumb for
> wireless deployments - the range you can get from a 25/10 deployment
> is geometrically better than 100/25, and the average bandwidth usage
> nearly the same.

	[SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).


> Agreeing on industry standards for what the "slack" should be, might be of help.

	[SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.


> 
>> 
>> 
>> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> 
> We have discussed elsewhere, the idea of a minimum contracted rate
> (down to), which is easier to reason about.

	[SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
	But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).


*) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....


> 
>> 
>>> 
>>> Seeing all the political activity tied onto it since (and now again)
>>> reminds of two families at war about an incident that had happened
>>> generations and generations before, where the two sides no longer
>>> remembered why they hated each other so, but just went on hating, and
>>> not forgiving, and not moving on.
>>> 
>>> Yes, there are entirely separate and additional NN issues, but the
>>> technical problem of providing common carriage between two very
>>> different network application types (voip/gaming vs file transfer) is
>>> thoroughly solved now,
>> 
>>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

	[SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:

"	• Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20

	• Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.


	• Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21

	• Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."

These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).



> 
> Torrent problem thoroughly solved with FQ on the path and backpressure
> from the mac scheduler.
> 
> https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

	[SM] Thanks for the link. This is mainly for the self-congesrion case?


> 
>>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>> 
>> 
>> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>> 
>> 
>>> and if only all sides recognised at least this
>>> much, and made peace over it, and worked together to deploy those
>>> solutions, maybe, just maybe, we could find mutually satisfactory
>>> solutions to the other problems that plague the internet today, like
>>> security, and the ipv6 rollout.
>> 
>>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> 
> I really would like to see a push for IPv6 mandated as a part of the
> BEAD programs.

	[SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.

> 
>> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> 
> There is a separate NRPM going on for the new cybersecurity label at
> the FCC. If I had time, and a partner,
> we could rework the doc we did a few years ago. It is due oct 6.
> 
>> 
>> 
>> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>> 
>> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>> 
>>> If anyone here knows anyone more political, still vibrating with 10+
>>> years of outrage about NN on this fronts, on one side or the other, if
>>> you could sit them down, over a beer, and try to explain that at the
>>> start it was a technical problem nobody understood at the time, maybe
>>> that would help.
>> 
>>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> 
> Governments make safer markets feasible.

	[SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.

Regards & Sorry for the tangent
	Sebastian

> 
>> 
>> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>> 
>> Regards
>>        Sebastian
>> 
>> 
>>> 
>>> --
>>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> <2510.jpeg>

Re: [Rpm] net neutrality back in the news

[Dave Taht]

@Sebastian: This is a really great list of what the issues were in the
EU, and if y'all can repost there, rather than here, perhaps more
light will be generated outside our circles.

https://news.ycombinator.com/item?id=37694306#37694307

On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
>
> > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Dave,
> >>
> >> please excuse a number of tangents below ;)
> >
> > It would be nice, if as a (dis)organisation... the bufferbloat team
> > could focus on somehow getting both sides of the network neutrality
> > debate deeplying understanding the technological problem their
> > pre-conceptions face, and the (now readily available and inexpensive)
> > solutions that could be deployed, by most ISPs, over a weekend. We are
> > regularly bringing up a few thousand people a week on libreqos (that
> > we know of), and then of course, there are all the home routers and
> > CPE that are increasingly capable of doing the right thing.
> >
> > The time to try and shift the memes in in the coming days, and weeks.
> >
> > So ya'all being distracting below... aggh... ok, I'll bite.
>
>         [SM] Sorry to drag you into the weeds...
>
>
> >
> >>
> >>
> >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >>>
> >>> Jason just did a beautiful thread as to what was the original source
> >>> of the network neutrality
> >>> bittorrent vs voip bufferbloat blowup.
> >>>
> >>> https://twitter.com/jlivingood/status/1707078242857849244
> >>
> >>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> >
> > The average use at peak hours for home broadband is below 5mbits per
> > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > business plans are less (but more bursty). Oversubscription within a
> > set of parameters is needed because, in part because upstream
> > bandwidth to the internet remains expensive (although I hope that cost
> > continues to drop rapidly), and in part, because it is needlessly
> > expensive to provision exactly the contracted rate to the customer. As
> > one example, I use the inverse square law as a rule of thumb for
> > wireless deployments - the range you can get from a 25/10 deployment
> > is geometrically better than 100/25, and the average bandwidth usage
> > nearly the same.
>
>         [SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).
>
>
> > Agreeing on industry standards for what the "slack" should be, might be of help.
>
>         [SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.
>
>
> >
> >>
> >>
> >> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> >
> > We have discussed elsewhere, the idea of a minimum contracted rate
> > (down to), which is easier to reason about.
>
>         [SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
>         But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).
>
>
> *) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....
>
>
> >
> >>
> >>>
> >>> Seeing all the political activity tied onto it since (and now again)
> >>> reminds of two families at war about an incident that had happened
> >>> generations and generations before, where the two sides no longer
> >>> remembered why they hated each other so, but just went on hating, and
> >>> not forgiving, and not moving on.
> >>>
> >>> Yes, there are entirely separate and additional NN issues, but the
> >>> technical problem of providing common carriage between two very
> >>> different network application types (voip/gaming vs file transfer) is
> >>> thoroughly solved now,
> >>
> >>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
>
>         [SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:
>
> "       • Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20
>
>         • Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.
>
>
>         • Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21
>
>         • Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."
>
> These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
> I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).
>
>
>
> >
> > Torrent problem thoroughly solved with FQ on the path and backpressure
> > from the mac scheduler.
> >
> > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
>
>         [SM] Thanks for the link. This is mainly for the self-congesrion case?
>
>
> >
> >>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
> >>
> >>
> >> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
> >>
> >>
> >>> and if only all sides recognised at least this
> >>> much, and made peace over it, and worked together to deploy those
> >>> solutions, maybe, just maybe, we could find mutually satisfactory
> >>> solutions to the other problems that plague the internet today, like
> >>> security, and the ipv6 rollout.
> >>
> >>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> >
> > I really would like to see a push for IPv6 mandated as a part of the
> > BEAD programs.
>
>         [SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.
>
> >
> >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> >
> > There is a separate NRPM going on for the new cybersecurity label at
> > the FCC. If I had time, and a partner,
> > we could rework the doc we did a few years ago. It is due oct 6.
> >
> >>
> >>
> >> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
> >>
> >> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
> >>
> >>> If anyone here knows anyone more political, still vibrating with 10+
> >>> years of outrage about NN on this fronts, on one side or the other, if
> >>> you could sit them down, over a beer, and try to explain that at the
> >>> start it was a technical problem nobody understood at the time, maybe
> >>> that would help.
> >>
> >>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> >
> > Governments make safer markets feasible.
>
>         [SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.
>
> Regards & Sorry for the tangent
>         Sebastian
>
> >
> >>
> >> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
> >>
> >> Regards
> >>        Sebastian
> >>
> >>
> >>>
> >>> --
> >>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> >>> Dave Täht CSO, LibreQos
> >>> _______________________________________________
> >>> Rpm mailing list
> >>> Rpm@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/rpm
> >>
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > <2510.jpeg>
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [LibreQoS] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 19919 bytes --]

"(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most'
in terms of customers being served by puts the Spectrums and Comcasts in
the mix (in the US) and they don't care about happy customers they care
about defacto monopolies in markets so that they don't have to care about
happy customers.  That might not be their motivation, but 30 years of their
behavior makes it clear that it's the case.


From my perspective, there should be a clear separation between carriers
and last mile delivery..  Even if it's the same company, the rules defining
each really should be different.

Common Carriers or rather, carrier class services for 'internet', should be
completely neutral.  Packets are packets.  However, I think it's important
to carve out methods to have dedicated links for real time flows at the
carrier level.  I don't know what that model looks like exactly, but being
too stubborn about purist NN principals could really hurt VoIP services if
there aren't methods to handle that.  I guess I really am describing
'internet fast lanes' for certain classes of services that we deem
important enough as a whole.  not individual ISPs deciding, but rather 'the
will of the people' saying VoIP is more important than netflix, you can
carve out dedicated capacity for that.

For the last mile, I'm actually less concerned with pure NN and more
concerned with no-blocking or 'brand' prioritization and required/label
transparency.  ie, if an ISP wants to sell a DPI's QoE'd service that says
'videos at 1080p' and that is on the product label clearly and not in small
print, then that's fine so long as 'videos' is agnostic of the vendor
INCLUDING the ISP's own video products.  "100Mbps internet services with
1080P video $65" for example.  "Upgrade to our 100M with 4k video for
+$10/m".  With that level of transparency I think that consumer protections
are well handled.  And, because of that transparency, normal market
capitalism mechanisms work.  I can say "100Mbps for $65 and full resolution
video" for example.   Or "100Mbps Net Neutral service".   The secret DPI's
QoE shaping is my main concern here, and where I think consumer protection
needs to be pursued.

Again however, I think that ISPs should be able to offer dedicated paths or
bandwidth for specific services like VoIP.  Services listed in a publicly
determined products list.


On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> @Sebastian: This is a really great list of what the issues were in the
> EU, and if y'all can repost there, rather than here, perhaps more
> light will be generated outside our circles.
>
> https://news.ycombinator.com/item?id=37694306#37694307
>
> On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> >
> >
> >
> > > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> please excuse a number of tangents below ;)
> > >
> > > It would be nice, if as a (dis)organisation... the bufferbloat team
> > > could focus on somehow getting both sides of the network neutrality
> > > debate deeplying understanding the technological problem their
> > > pre-conceptions face, and the (now readily available and inexpensive)
> > > solutions that could be deployed, by most ISPs, over a weekend. We are
> > > regularly bringing up a few thousand people a week on libreqos (that
> > > we know of), and then of course, there are all the home routers and
> > > CPE that are increasingly capable of doing the right thing.
> > >
> > > The time to try and shift the memes in in the coming days, and weeks.
> > >
> > > So ya'all being distracting below... aggh... ok, I'll bite.
> >
> >         [SM] Sorry to drag you into the weeds...
> >
> >
> > >
> > >>
> > >>
> > >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> > >>>
> > >>> Jason just did a beautiful thread as to what was the original source
> > >>> of the network neutrality
> > >>> bittorrent vs voip bufferbloat blowup.
> > >>>
> > >>> https://twitter.com/jlivingood/status/1707078242857849244
> > >>
> > >>        But the core issue IMHO really was an economic one, the
> over-subscription ratios that worked before torrenting simply did not cut
> it any more in an environment when customers actually tried to use their
> contracted access rates "quantitatively". In my outsider perspective an ISP
> owes its customers the contracted rates (with some slack*) and any sort of
> over-subscription is a valid economic optimization an ISP might take, IFF
> that ISP is prepared to rapidly increase segment capacity (or down-grade
> customer plans)) if the deployed over-subscription rate proves to have been
> too optimistic. Mind you, most ISPs market plans by access speed (and
> charge more for higher speeds) and hence are somewhat responsible to
> actually deliver (again with some slack) these speeds.
> > >
> > > The average use at peak hours for home broadband is below 5mbits per
> > > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > > business plans are less (but more bursty). Oversubscription within a
> > > set of parameters is needed because, in part because upstream
> > > bandwidth to the internet remains expensive (although I hope that cost
> > > continues to drop rapidly), and in part, because it is needlessly
> > > expensive to provision exactly the contracted rate to the customer. As
> > > one example, I use the inverse square law as a rule of thumb for
> > > wireless deployments - the range you can get from a 25/10 deployment
> > > is geometrically better than 100/25, and the average bandwidth usage
> > > nearly the same.
> >
> >         [SM] +1; hence my argument is not oversubscription per se is
> bad, but that oversubscription needs to be managed and the level of
> oversubscription needs to to be adapted ot he actual usage patterns. I am
> sure however that ISPs already actively do that (I assume most ISPs want
> happy customers).
> >
> >
> > > Agreeing on industry standards for what the "slack" should be, might
> be of help.
> >
> >         [SM] Not being part of that industry at all I would love to see
> that as well, but can not contribute to defining that in any way.
> >
> >
> > >
> > >>
> > >>
> > >> *) Claiming "Up to", only carries that far, if you sell me X and I
> mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is
> acceptable unless occasionally is "at every late afternoon and evening"
> prime-time.
> > >
> > > We have discussed elsewhere, the idea of a minimum contracted rate
> > > (down to), which is easier to reason about.
> >
> >         [SM] Indeed, the German regulator (and I am not saying this is
> the only or best option) decided to require ISPs to give 6 numbers
> pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate,
> a minimal (net) rate, and a usually achievable (net) rate, all rates were
> defined as IP/TCP goodput to make verification easier on end-users. The
> regulator also defined a measurement regime that end-users can follow to
> check whether the ISP is fulfilling the contract and the law gives remedies
> if ISPs do not deliver (either the right to immediately cancel the contract
> and/or the right to adapt the monthly payments to the actually delivered
> ratio of the contracted rates*). I think I need to add, that ISPs can set
> these numbers freely, but are only allowed to advertise with the maximum
> rate.
> >         But if we talk about a single number per direction, minimal rate
> is probably easiest, or something like a "usually achievable rate" (that
> needs to be met or exceeded in say 90% of >= 20 measurements or so).
> >
> >
> > *) In a ironic twist however the regulator so far has not explained how
> deviations in each of the 6 numbers above should be aggregated to get one
> single contract rate achievement ratio..., most ISPs took measures into
> their own hands and simply offer customers typically a permanent rebate of
> 5 EUR or immediate cancelation what ever the customer prefers....
> >
> >
> > >
> > >>
> > >>>
> > >>> Seeing all the political activity tied onto it since (and now again)
> > >>> reminds of two families at war about an incident that had happened
> > >>> generations and generations before, where the two sides no longer
> > >>> remembered why they hated each other so, but just went on hating, and
> > >>> not forgiving, and not moving on.
> > >>>
> > >>> Yes, there are entirely separate and additional NN issues, but the
> > >>> technical problem of providing common carriage between two very
> > >>> different network application types (voip/gaming vs file transfer) is
> > >>> thoroughly solved now,
> > >>
> > >>        I am not sure this was at the core of the problem, my take is
> really that "always-on" and relative upload-heavy torrent simply
> demonstrated painfully for all involved that the old oversubscription
> ratios were not suitable for the changed traffic profiles. I have some
> sympathy for the ISPs here, they certainly did not try to sell their
> customers short (good ISPs try to retain their customers and that works
> best when customers are happy with the service) and having this problem
> appear on many segments at the same time is not a fun place to be, and
> upload was/is often (too) low in DOCSIS segments anyway; but this is why
> e.g. my bit torrent could affect your VoIP, simply by pushing the whole
> segment into upload capacity congestion... (ISPs in theory could fix this
> by plain old QoS engineering, but the issue at hand was with a non-ISP
> VoIP/SIP service and there QoS becomes tricky if the ISP as these packets
> need to be identified in a way that is not game-able**)
> >
> >         [SM] See my later mail to Jason, I will not claim I know
> Comcast's issues better than him and will accept that self-congestion also
> played a major role in the initial problem. Over here in Europe the net
> neutrality debate was kicked of less by an unfortunate confluence of new
> usage profiles and traditional oversibscription ratios and less than ideal
> packet scheduling, but rather by a series of pretty apparent attempts at
> restricting consumer choice, see eg.
> https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf
> for an admitted slightly biased position:
> >
> > "       • Blocking of applications and services: In order to maximise
> profits, some ISPs – that also offer their own services and applications
> online – exclude certain services and applications of competing market
> players. The most prominent case of this form of network discrimination is
> European mobile providers (like Deutsche Telekom) blocking or restricting
> the use of Voice over IP (VoIP) services (like Skype and Viber) for their
> customers.20
> >
> >         • Slowing or “throttling” internet speeds: Some ISPs slow down
> specific services (like YouTube) and applications (like Skype), or ask
> users to pay an extra fee to have access to these internet platforms. Given
> the high latency (delay) sensitivity of many applications, ISPs are able to
> compromise the correct functioning of these services by slowing them down,
> preventing the services from running properly. Often ISPs – especially
> telecommunication companies – do this to favour their own voice calling
> services over VoIP services, thereby crushing competition.
> >
> >
> >         • Blocking websites: ISPs often block websites for a number of
> reasons – to secure their network, or to avoid competition, and sometimes
> for social, public relations or political reasons. In the UK, for instance,
> Orange Telecom blocked the French digital rights advocacy group, La
> Quadrature du Net’s website on pre-paid mobile accounts.21
> >
> >         • Preferential treatment of services and platforms: ISPs can
> also impose data caps on internet access contracts while granting data
> allowance exceptions to a company’s own proprietary streaming services
> (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also
> grant preferential treatment to select services – such as Orange France
> with the popular music streaming service Deezer23 – ahead of other
> competitors, effectively imposing anti-competitive limitations on markets
> such as those for legal online music. Moreover, generally only large,
> well-established companies can afford this preferential treatment,
> resulting in a further stifling of innovation."
> >
> > These look like clearer attempts to monetize the ISP position as
> gate-keeper to his customer's eye-balls (for content providers) as well as
> gate-keeper to the internet for for paying customers.
> > I find it much harder to have sympathy for the listed examples of ISP
> behavior than the situation of rapidly changing usage pattern posed where
> technical changes needed to be made, but where no attempt at unfair
> monetization was evident, but maybe I am overly sensitive. These are all
> examples that make me personally applaud the EU for its intervention to
> make clear rules what "internet access providers" can and can not do. (The
> EU also was quite flexible in applying/interpreting its rules during the
> covid isolation periods, when it made it clear that e.g. treating certain
> traffic classes e.g. streaming media to lower priority than video
> conferences was within the neutrality framework as long as it was based on
> traffic class and not on specific service providers IIRC).
> >
> >
> >
> > >
> > > Torrent problem thoroughly solved with FQ on the path and backpressure
> > > from the mac scheduler.
> > >
> > > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
> >
> >         [SM] Thanks for the link. This is mainly for the self-congesrion
> case?
> >
> >
> > >
> > >>        I agree that on a single link we mostly solved the problem (a
> few corner cases are left on links with very low capacity, where
> essentially we can only manage the pain, not remove it)...
> > >>
> > >>
> > >> **) Which is not rocket science either, a VoIP stream takes ~100
> Kbps, so in theory an ISP might simply allow each customer say 5 VoIP
> stream equivalents by allowing up to 500Kbps od traffic marked with a
> specific DSCP as higher priority (with higher access probability for the
> shared medium). I am not sayng this is my preferred solution, just saying
> this is a solution that would have been available at the time if I memorize
> my docsis capabilities correctly.
> > >>
> > >>
> > >>> and if only all sides recognised at least this
> > >>> much, and made peace over it, and worked together to deploy those
> > >>> solutions, maybe, just maybe, we could find mutually satisfactory
> > >>> solutions to the other problems that plague the internet today, like
> > >>> security, and the ipv6 rollout.
> > >>
> > >>        +1. IPv6 is IMHO a prime example where the regulators should
> stop talking softly and start showing the big stick they carry.
> > >
> > > I really would like to see a push for IPv6 mandated as a part of the
> > > BEAD programs.
> >
> >         [SM] +1; I am not the biggest IPv6 fan, but that's what we have
> and it is mostly serviceable so let's get this "party started" finally.
> >
> > >
> > >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4
> addresses only.. (most mass market ISPs do much better in that regard, but
> for the stragglers it would help if the regulator would demand IPv6 with
> PD***). Regarding security, the easiest way to achieve that would be to put
> some heavy requirements on IoT manufacturers and vendors (like do what you
> please as long as you are local LAN only, but once you reach out into the
> cloud you need to fulfill the following list of requirements, with timely
> security updates over a reasonable long usage period), however that might
> not be very attractive for politicians/regulators to tackle (active
> regulatory acts tends to get bad press unless something bad happened, but
> even then the press often complains about the acts coming too late, but I
> digress****)
> > >
> > > There is a separate NRPM going on for the new cybersecurity label at
> > > the FCC. If I had time, and a partner,
> > > we could rework the doc we did a few years ago. It is due oct 6.
> > >
> > >>
> > >>
> > >> ***) Strictly speaking IPv6 is required, since "internet access" is
> defined as reaching all of the internet (as far as in the ISPs power) and
> IPv6-only sites are not reachable for the CG-NAT-only customers. But so far
> the local regulator does not seem to enforce that requirement, or hopefully
> is working on this quietly behind the curtains.
> > >>
> > >> ****) This is not to diss the press, they are doing what they are
> supposed to do, but it just shows that active regulation is a tricky
> business, and a light touch typically "looks better" (even though I see no
> real evidence it actually works better).
> > >>
> > >>> If anyone here knows anyone more political, still vibrating with 10+
> > >>> years of outrage about NN on this fronts, on one side or the other,
> if
> > >>> you could sit them down, over a beer, and try to explain that at the
> > >>> start it was a technical problem nobody understood at the time, maybe
> > >>> that would help.
> > >>
> > >>        So in the EU that debate is essentially settled, there is a EU
> regulation that essentially spills out what ISPs owe their customers and
> that has become the law of the land. The rationale for required un-biased
> service and freedom to select terminal devices is well justified by the
> market ideals of the EU, allowing ISPs to discriminate packets or terminal
> devices restricts the market and will lead to undesired outcomes. (Fun fact
> most big players in capitalist societies argue for "free markets" but at
> the same time act to work-around the market mechanism by trying to move the
> market into an oligo- or even monopoly condition, which is why strong
> regulation is required*****).
> > >
> > > Governments make safer markets feasible.
> >
> >         [SM] Yes, I agree, both in markets are a pretty decent tool, but
> need constant maintenance.
> >
> > Regards & Sorry for the tangent
> >         Sebastian
> >
> > >
> > >>
> > >> *****) This is akin to professional sports where the audience
> generally accepts that referees are necessary and occasionally need to make
> "painful" calls, as the alternative would be anarchy, but I digress.
> > >>
> > >> Regards
> > >>        Sebastian
> > >>
> > >>
> > >>>
> > >>> --
> > >>> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > >>> Dave Täht CSO, LibreQos
> > >>> _______________________________________________
> > >>> Rpm mailing list
> > >>> Rpm@lists.bufferbloat.net
> > >>> https://lists.bufferbloat.net/listinfo/rpm
> > >>
> > >
> > >
> > > --
> > > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > > Dave Täht CSO, LibreQos
> > > <2510.jpeg>
> >
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 23355 bytes --]

Re: [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[Livingood, Jason]

> dan <dandenson@gmail.com> wrote: 

> "(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers.  

Corporations are motivated to generate returns for investors. In that context, happy customers stay longer (less churn) and spend more (upgrades, multiple services). And unhappy customers generate costs via disconnects (loss of revenue, costs to replace them with a new customer to just stay at the same subscriber levels), and costs via customer contacts (call center staff). So, IMO on a purely financial basis, public companies have significant motivation to retain customers and keep them happy. This typically follows through to staff members having part of their variable compensation based on things like NPS scores, contact rates, etc. And specifically in relation to Comcast, the company recently has 4 new wireless competitors: three 5G FWA and one LEO (more coming) - and those are posing significant competitive risks (and taking customers). 

> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...

The two thoughts your comments (thanks for the response BTW!) trigger are:
1 - Often regulation looks to the past - in this case maybe an era of bandwidth scarcity where prioritization may have mattered. I think we're in the midst of a shift into bandwidth abundance where priority does not matter. What will is latency/responsiveness, content/compute localization, reliability, consistency, security, etc. 
2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care (contact) costs and would see people start to immediately shift to competitors (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like something that could realistically happen any longer in the US.

JL

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 3155 bytes --]

On Thu, 28 Sep 2023, Livingood, Jason via Bloat wrote:

> Date: Thu, 28 Sep 2023 20:48:58 +0000
> From: "Livingood, Jason via Bloat" <bloat@lists.bufferbloat.net>
> Reply-To: "Livingood, Jason" <Jason_Livingood@comcast.com>
> To: dan <dandenson@gmail.com>, Dave Taht <dave.taht@gmail.com>
> Cc: Rpm <rpm@lists.bufferbloat.net>,
>     Dave Taht via Starlink <starlink@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>,
>     libreqos <libreqos@lists.bufferbloat.net>,
>     Jamal Hadi Salim <jhs@mojatatu.com>
> Subject: Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the
>     news
> 
>> dan <dandenson@gmail.com> wrote:
>
>> "(I assume most ISPs want happy customers)."

>> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' 
>> in terms of customers being served by puts the Spectrums and Comcasts in the 
>> mix (in the US) and they don't care about happy customers they care about 
>> defacto monopolies in markets so that they don't have to care about happy 
>> customers. 
>
> In that context, happy customers stay longer (less churn) and spend more 
> (upgrades, multiple services). And unhappy customers generate costs via 
> disconnects (loss of revenue, costs to replace them with a new customer to 
> just stay at the same subscriber levels), and costs via customer contacts 
> (call center staff).

Except when you have a monopoly in an area, at which point the ability of 
customers to leave is minimal, and years of bad customer service means that 
people don't bother complaining, so the call center staffing costs are lower 
than they should be.

>> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...
>
> The two thoughts your comments (thanks for the response BTW!) trigger are:

> 1 - Often regulation looks to the past - in this case maybe an era of 
> bandwidth scarcity where prioritization may have mattered. I think we're in 
> the midst of a shift into bandwidth abundance where priority does not matter. 
> What will is latency/responsiveness, content/compute localization, 
> reliability, consistency, security, etc.

> 2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care 
> (contact) costs and would see people start to immediately shift to competitors 
> (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like 
> something that could realistically happen any longer in the US.

Dave T called out earlier that the rise of bittorrent was a large part of the 
inital NN discussion here in the US. But a second large portion was a money grab 
from ISPs thinking that they could hold up large paid websites (netflix for 
example) for additional fees by threatening to make their service less useful to 
their users (viewing their users as an asset to be marketed to the websites 
rather than customers to be satisfied by providing them access to the websites)

I don't know if a new round of "it's not fair that Netflix doesn't pay us for 
the bandwidth to service them" would fall flat at this point or not.

David Lang

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Jonathan Morton]

> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> 
> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.

I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:


1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.

This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.


2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.

FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.

This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.


3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.

**THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.

And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.


All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.

 - Jonathan Morton

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Rich Brown]

Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 

Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 

I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)

- Rich Brown

> On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
>> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
>> 
>> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> 
> I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> 
> 
> 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> 
> This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> 
> 
> 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> 
> FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> 
> This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> 
> 
> 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> 
> **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> 
> And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> 
> 
> All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> 
> - Jonathan Morton
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 8530 bytes --]

ok, lots and lots of great comments here for sure.

bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't
carrying to many places at affordable rates.  I've pulled quotes from Lumen
and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of
service.  This isn't abundance and it's widespread and it leaves only
major providers that can afford/amortize out 100G transits etc.
My answer to this is one that Dave and I have bounced back and forth is the
idea of micro IXs in every municipality and having that somehow tied into
access to the ROW in counties etc.  Not fully hashed out, but the fiber is
in the ground, it could be sold, but the carriers are not well incentivised
to sell it.  It takes the better part of a year to get a DIA within 100ft
of a Lumen hut sometimes...  Heck, it could even be a government program to
get an μ*IX* with x feet of every school, city hall, and library.  I don't
care how it's done but this would get abundance NEAR end users and open up
essentially every town to competition.

monopoly.  This is a historical thing for most cable and DSL incumbents.
They have enjoyed virtual monopolies with cable owning population centers
and DSL owning the outskirts and there is no product darwinism here where
customer satisfaction is a pressure.  That may not be the future but it
definitely is the past.  These companies may have to shift into customer
satisfaction as a major part instead of a minor part of their corporate
culture to fend off fttx and ultra-modern wisps.

Starlink is not offering significant competition to major carriers.
Starlink's 1.5 million customers are at LEAST 90% pulled from other
satellite services and small ISPs.  Spectrum and Comcast's losses to
starlink are measured in decimal points.

Only fttx and ultra-modern wireless tech really threatens these
incumbents.  Typical wisps aren't putting a dent in these guys, just
scraping the paint off their bumper.  We're pulling customers at the scale
of 'dozens' for example.  Spectrum's management doesn't know we exist we're
such a small threat to them.

But to further the point here, these fttx and ultra-modern wisps can only
exist in places where there is adequate carrier services to start with.  In
areas where they spend the money and do the build but there aren't good
carrier services, those fiber services suck and the wISPs start to claw
back even with inferior technology.  We've pulled quite a few customers off
fttx deployments because of this sort of situation.


On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:

> Thank you Jonathan for this clear description of the issues and their
> history. I wonder if there's a fourth one - privacy.
>
> Rosenworcel's talk
> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
> that ISPs might want to monetize our traffic patterns and location data.
> (This is less of an issue in the EU, but the US remains a Wild West in this
> regard.)
>
> I am hopeful that the FCC will include this in their NPRM (which must be
> available by now but I haven't looked...)
>
> - Rich Brown
>
> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> >
> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> >>
> >> Dave T called out earlier that the rise of bittorrent was a large part
> of the inital NN discussion here in the US. But a second large portion was
> a money grab from ISPs thinking that they could hold up large paid websites
> (netflix for example) for additional fees by threatening to make their
> service less useful to their users (viewing their users as an asset to be
> marketed to the websites rather than customers to be satisfied by providing
> them access to the websites)
> >>
> >> I don't know if a new round of "it's not fair that Netflix doesn't pay
> us for the bandwidth to service them" would fall flat at this point or not.
> >
> > I think there were three more-or-less separate concerns which have, over
> time, fallen under the same umbrella:
> >
> >
> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
> flows, and the "induced demand" phenomenon means that increases in link
> rate do not in themselves solve this problem, even though they may be sold
> as doing so.
> >
> > This is directly addressed by properly-sized buffers and/or AQM, and
> even better by FQ and SQM.  It's a solved problem, so long as the solutions
> are deployed.  It's not usually necessary, for example, to specifically
> enhance service for latency-sensitive traffic, if FQ does a sufficiently
> good job.  An increased link rate *does* enhance service quality for both
> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> >
> >
> > 2:  Swarm traffic tends to drown out conventional traffic, due to
> congestion control algorithms which try to be more-or-less fair on a
> per-flow basis, and the substantially larger number of parallel flows used
> by swarm traffic.  This also caused subscribers using swarm traffic to
> impair the service of subscribers who had nothing to do with it.
> >
> > FQ on a per-flow basis (see problem 1) actually amplifies this effect,
> and I think it was occasionally used as an argument for *not* deploying
> FQ.  ISPs' initial response was to outright block swarm traffic where they
> could identify it, which was then softened to merely throttling it heavily,
> before NN regulations intervened.  Usage quotas also showed up around this
> time, and were probably related to this problem.
> >
> > This has since been addressed by several means.  ISPs may use FQ on a
> per-subscriber basis to prevent one subscriber's heavy traffic from
> degrading service for another.  Swarm applications nowadays tend to employ
> altruistic congestion control which deliberately compensates for the large
> number of flows, and/or mark them with one or more of the Least Effort
> class DSCPs.  Hence, swarm applications are no longer as damaging to
> service quality as they used to be.  Usage quotas, however, still remain in
> use as a profit centre, to the point where an "unlimited" service is a rare
> and precious specimen in many jurisdictions.
> >
> >
> > 3:  ISPs merged with media distribution companies, creating a conflict
> of interest in which the media side of the business wanted the internet
> side to actively favour "their own" media traffic at the expense of "the
> competition".  Some ISPs began to actively degrade Netflix traffic, in
> particular by refusing to provision adequate peering capacity at the nodes
> through which Netflix traffic predominated, or by zero-rating (for the
> purpose of usage quotas) traffic from their own media empire while refusing
> to do the same for Netflix traffic.
> >
> > **THIS** was the true core of Net Neutrality.  NN regulations forced
> ISPs to carry Netflix traffic with reasonable levels of service, even
> though they didn't want to for purely selfish and greedy commercial
> reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile
> practice, which I am perfectly sure would resume just as soon as NN
> regulations were repealed.
> >
> > And this type of practice is just the sort of thing that technologies
> like L4S are designed to support.  The ISPs behind L4S actively do not want
> a technology that works end-to-end over the general Internet.  They want
> something that can provide a domination service within their own walled
> gardens.  That's why L4S is a NN hazard, and why they actively resisted all
> attempts to displace it with SCE.
> >
> >
> > All of the above were made more difficult to solve by the monopolistic
> nature of the Internet service industry.  It is actively difficult for
> Internet users to move to a truly different service, especially one based
> on a different link technology.  When attempts are made to increase
> competition, for example by deploying a publicly-funded network, the
> incumbents actively sabotage those attempts by any means they can.
> Monopolies are inherently customer-hostile, and arguments based on market
> forces fail in their presence.
> >
> > - Jonathan Morton
> >
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>
>

[-- Attachment #2: Type: text/html, Size: 9601 bytes --]

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 9972 bytes --]

Back then in 2015, when NN was enacted by Wheeler & CO, there was a great
body of work (IMHO) done on this subject by Martin Geddes:
https://www.martingeddes.com/1261-2/

But let's pick *one
<https://www.martingeddes.com/ofcom-publishes-pnsol-scientific-report-on-net-neutrality/>
*report written
by his colleagues and published by Ofcom (UK telecoms regulator):


   - *You cannot conflate ‘equality of [packet] treatment’ with delivering
   equality of [user application] outcomes.* Only the latter matters, as
   ordinary users don’t care what happened to the packets in transit. Yet the
   relevant academic literature fixates on the local operation of the
   mechanisms (including Traffic Management), not their global aggregate
   effect.


All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net>
wrote:

> ok, lots and lots of great comments here for sure.
>
> bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't
> carrying to many places at affordable rates.  I've pulled quotes from Lumen
> and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of
> service.  This isn't abundance and it's widespread and it leaves only
> major providers that can afford/amortize out 100G transits etc.
> My answer to this is one that Dave and I have bounced back and forth is
> the idea of micro IXs in every municipality and having that somehow tied
> into access to the ROW in counties etc.  Not fully hashed out, but the
> fiber is in the ground, it could be sold, but the carriers are not well
> incentivised to sell it.  It takes the better part of a year to get a DIA
> within 100ft of a Lumen hut sometimes...  Heck, it could even be a
> government program to get an μ*IX* with x feet of every school, city
> hall, and library.  I don't care how it's done but this would get abundance
> NEAR end users and open up essentially every town to competition.
>
> monopoly.  This is a historical thing for most cable and DSL incumbents.
> They have enjoyed virtual monopolies with cable owning population centers
> and DSL owning the outskirts and there is no product darwinism here where
> customer satisfaction is a pressure.  That may not be the future but it
> definitely is the past.  These companies may have to shift into customer
> satisfaction as a major part instead of a minor part of their corporate
> culture to fend off fttx and ultra-modern wisps.
>
> Starlink is not offering significant competition to major carriers.
> Starlink's 1.5 million customers are at LEAST 90% pulled from other
> satellite services and small ISPs.  Spectrum and Comcast's losses to
> starlink are measured in decimal points.
>
> Only fttx and ultra-modern wireless tech really threatens these
> incumbents.  Typical wisps aren't putting a dent in these guys, just
> scraping the paint off their bumper.  We're pulling customers at the scale
> of 'dozens' for example.  Spectrum's management doesn't know we exist we're
> such a small threat to them.
>
> But to further the point here, these fttx and ultra-modern wisps can only
> exist in places where there is adequate carrier services to start with.  In
> areas where they spend the money and do the build but there aren't good
> carrier services, those fiber services suck and the wISPs start to claw
> back even with inferior technology.  We've pulled quite a few customers off
> fttx deployments because of this sort of situation.
>
>
> On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com>
> wrote:
>
>> Thank you Jonathan for this clear description of the issues and their
>> history. I wonder if there's a fourth one - privacy.
>>
>> Rosenworcel's talk
>> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
>> that ISPs might want to monetize our traffic patterns and location data.
>> (This is less of an issue in the EU, but the US remains a Wild West in this
>> regard.)
>>
>> I am hopeful that the FCC will include this in their NPRM (which must be
>> available by now but I haven't looked...)
>>
>> - Rich Brown
>>
>> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
>> rpm@lists.bufferbloat.net> wrote:
>> >
>> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
>> bloat@lists.bufferbloat.net> wrote:
>> >>
>> >> Dave T called out earlier that the rise of bittorrent was a large part
>> of the inital NN discussion here in the US. But a second large portion was
>> a money grab from ISPs thinking that they could hold up large paid websites
>> (netflix for example) for additional fees by threatening to make their
>> service less useful to their users (viewing their users as an asset to be
>> marketed to the websites rather than customers to be satisfied by providing
>> them access to the websites)
>> >>
>> >> I don't know if a new round of "it's not fair that Netflix doesn't pay
>> us for the bandwidth to service them" would fall flat at this point or not.
>> >
>> > I think there were three more-or-less separate concerns which have,
>> over time, fallen under the same umbrella:
>> >
>> >
>> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
>> flows, and the "induced demand" phenomenon means that increases in link
>> rate do not in themselves solve this problem, even though they may be sold
>> as doing so.
>> >
>> > This is directly addressed by properly-sized buffers and/or AQM, and
>> even better by FQ and SQM.  It's a solved problem, so long as the solutions
>> are deployed.  It's not usually necessary, for example, to specifically
>> enhance service for latency-sensitive traffic, if FQ does a sufficiently
>> good job.  An increased link rate *does* enhance service quality for both
>> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
>> >
>> >
>> > 2:  Swarm traffic tends to drown out conventional traffic, due to
>> congestion control algorithms which try to be more-or-less fair on a
>> per-flow basis, and the substantially larger number of parallel flows used
>> by swarm traffic.  This also caused subscribers using swarm traffic to
>> impair the service of subscribers who had nothing to do with it.
>> >
>> > FQ on a per-flow basis (see problem 1) actually amplifies this effect,
>> and I think it was occasionally used as an argument for *not* deploying
>> FQ.  ISPs' initial response was to outright block swarm traffic where they
>> could identify it, which was then softened to merely throttling it heavily,
>> before NN regulations intervened.  Usage quotas also showed up around this
>> time, and were probably related to this problem.
>> >
>> > This has since been addressed by several means.  ISPs may use FQ on a
>> per-subscriber basis to prevent one subscriber's heavy traffic from
>> degrading service for another.  Swarm applications nowadays tend to employ
>> altruistic congestion control which deliberately compensates for the large
>> number of flows, and/or mark them with one or more of the Least Effort
>> class DSCPs.  Hence, swarm applications are no longer as damaging to
>> service quality as they used to be.  Usage quotas, however, still remain in
>> use as a profit centre, to the point where an "unlimited" service is a rare
>> and precious specimen in many jurisdictions.
>> >
>> >
>> > 3:  ISPs merged with media distribution companies, creating a conflict
>> of interest in which the media side of the business wanted the internet
>> side to actively favour "their own" media traffic at the expense of "the
>> competition".  Some ISPs began to actively degrade Netflix traffic, in
>> particular by refusing to provision adequate peering capacity at the nodes
>> through which Netflix traffic predominated, or by zero-rating (for the
>> purpose of usage quotas) traffic from their own media empire while refusing
>> to do the same for Netflix traffic.
>> >
>> > **THIS** was the true core of Net Neutrality.  NN regulations forced
>> ISPs to carry Netflix traffic with reasonable levels of service, even
>> though they didn't want to for purely selfish and greedy commercial
>> reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile
>> practice, which I am perfectly sure would resume just as soon as NN
>> regulations were repealed.
>> >
>> > And this type of practice is just the sort of thing that technologies
>> like L4S are designed to support.  The ISPs behind L4S actively do not want
>> a technology that works end-to-end over the general Internet.  They want
>> something that can provide a domination service within their own walled
>> gardens.  That's why L4S is a NN hazard, and why they actively resisted all
>> attempts to displace it with SCE.
>> >
>> >
>> > All of the above were made more difficult to solve by the monopolistic
>> nature of the Internet service industry.  It is actively difficult for
>> Internet users to move to a truly different service, especially one based
>> on a different link technology.  When attempts are made to increase
>> competition, for example by deploying a publicly-funded network, the
>> incumbents actively sabotage those attempts by any means they can.
>> Monopolies are inherently customer-hostile, and arguments based on market
>> forces fail in their presence.
>> >
>> > - Jonathan Morton
>> >
>> > _______________________________________________
>> > Rpm mailing list
>> > Rpm@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/rpm
>>
>> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
>

[-- Attachment #2: Type: text/html, Size: 12855 bytes --]

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Frantisek,

> On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> https://www.martingeddes.com/1261-2/
> 
> But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> 
> 	• You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.

	[SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:

[...]
(8)
When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
(9)
The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
(10)
Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
(11)
Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
(12)
Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
[...]

There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.

There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 

Regards
	Sebastian

P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).



> 
> All the best,
> 
> Frank
> 
> Frantisek (Frank) Borsik
> 
>  
> 
> https://www.linkedin.com/in/frantisekborsik
> 
> Signal, Telegram, WhatsApp: +421919416714 
> 
> iMessage, mobile: +420775230885
> 
> Skype: casioa5302ca
> 
> frantisek.borsik@gmail.com
> 
> 
> 
> On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> ok, lots and lots of great comments here for sure.  
> 
> bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> 
> monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> 
> Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> 
> Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> 
> But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> 
> 
> On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> 
> Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> 
> I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> 
> - Rich Brown
> 
> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > 
> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> >> 
> >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> >> 
> >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > 
> > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > 
> > 
> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > 
> > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > 
> > 
> > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > 
> > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > 
> > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > 
> > 
> > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > 
> > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > 
> > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > 
> > 
> > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > 
> > - Jonathan Morton
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Mike,

[I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]


> On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> 
> First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better. 
> 
> As I read this thread and think about the coming debate in the U.S., two things come to mind:
> 
> 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.

	[SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).


*) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.

> 
> The question I'm thinking about is do we want an Internet where end user plans are divided up this way?

	[SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.

> And if not, is a NN regulation the right place to put those rules?

	[SM] Could well be, but depends on the framing, no?

> 
> 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection. 

	[SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime". 


> Are NN rules the right place to address this

	[SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...

> and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?

	[SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.


Regards
	Sebastian

P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).


> 
> 
> 
> On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> Hi Frantisek,
> 
> > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > 
> > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > https://www.martingeddes.com/1261-2/
> > 
> > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > 
> >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> 
>         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> 
> [...]
> (8)
> When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> (9)
> The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> (10)
> Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> (11)
> Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> (12)
> Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> [...]
> 
> There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> 
> There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 
> 
> Regards
>         Sebastian
> 
> P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> 
> 
> 
> > 
> > All the best,
> > 
> > Frank
> > 
> > Frantisek (Frank) Borsik
> > 
> >  
> > 
> > https://www.linkedin.com/in/frantisekborsik
> > 
> > Signal, Telegram, WhatsApp: +421919416714 
> > 
> > iMessage, mobile: +420775230885
> > 
> > Skype: casioa5302ca
> > 
> > frantisek.borsik@gmail.com
> > 
> > 
> > 
> > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > ok, lots and lots of great comments here for sure.  
> > 
> > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > 
> > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > 
> > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > 
> > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> > 
> > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > 
> > 
> > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> > 
> > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> > 
> > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > 
> > - Rich Brown
> > 
> > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > 
> > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > >> 
> > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > >> 
> > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > 
> > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > 
> > > 
> > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > 
> > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > 
> > > 
> > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > 
> > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > 
> > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > 
> > > 
> > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > 
> > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > 
> > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > 
> > > 
> > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > 
> > > - Jonathan Morton
> > > 
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[Dave Taht]

The starlink list was not originally cc´d and yet since I think this
debate concerns that also, I have added the cc back. Carry on!

On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS
<libreqos@lists.bufferbloat.net> wrote:
>
> Hi Mike,
>
> [I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]
>
>
> > On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> >
> > First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better.
> >
> > As I read this thread and think about the coming debate in the U.S., two things come to mind:
> >
> > 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.
>
>         [SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
> During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).
>
>
> *) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.
>
> >
> > The question I'm thinking about is do we want an Internet where end user plans are divided up this way?
>
>         [SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.
>
> > And if not, is a NN regulation the right place to put those rules?
>
>         [SM] Could well be, but depends on the framing, no?
>
> >
> > 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection.
>
>         [SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
> This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime".
>
>
> > Are NN rules the right place to address this
>
>         [SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...
>
> > and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?
>
>         [SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.
>
>
> Regards
>         Sebastian
>
> P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).
>
>
> >
> >
> >
> > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> > Hi Frantisek,
> >
> > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > >
> > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > > https://www.martingeddes.com/1261-2/
> > >
> > > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > >
> > >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> >
> >         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> >
> > [...]
> > (8)
> > When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> > (9)
> > The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> > (10)
> > Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> > (11)
> > Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> > (12)
> > Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> > [...]
> >
> > There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> >
> > There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ).
> >
> > Regards
> >         Sebastian
> >
> > P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> >
> >
> >
> > >
> > > All the best,
> > >
> > > Frank
> > >
> > > Frantisek (Frank) Borsik
> > >
> > >
> > >
> > > https://www.linkedin.com/in/frantisekborsik
> > >
> > > Signal, Telegram, WhatsApp: +421919416714
> > >
> > > iMessage, mobile: +420775230885
> > >
> > > Skype: casioa5302ca
> > >
> > > frantisek.borsik@gmail.com
> > >
> > >
> > >
> > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > ok, lots and lots of great comments here for sure.
> > >
> > > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > >
> > > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > >
> > > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > >
> > > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.
> > >
> > > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > >
> > >
> > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy.
> > >
> > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.)
> > >
> > > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > >
> > > - Rich Brown
> > >
> > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > >
> > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > > >>
> > > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > > >>
> > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > >
> > > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > >
> > > >
> > > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > >
> > > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > >
> > > >
> > > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > >
> > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > >
> > > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > >
> > > >
> > > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > >
> > > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > >
> > > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > >
> > > >
> > > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > >
> > > > - Jonathan Morton
> > > >
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > >
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[Dave Taht]

Despite the sturm und drang here if you google for network neutrality
there was a lot of press coverage 4 days ago.

... and mostly, silence, on the twitters at least. How is mastodon or
other social media?

I couldn´t help but notice that this was essentially, Diane
Feinstein´s last press release (she died yesterday):

https://www.feinstein.senate.gov/public/index.cfm/press-releases?id=C6FFF484-16F1-4CC9-9836-F36446C3B33D


On Sat, Sep 30, 2023 at 8:23 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> The starlink list was not originally cc´d and yet since I think this
> debate concerns that also, I have added the cc back. Carry on!
>
> On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS
> <libreqos@lists.bufferbloat.net> wrote:
> >
> > Hi Mike,
> >
> > [I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]
> >
> >
> > > On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> > >
> > > First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better.
> > >
> > > As I read this thread and think about the coming debate in the U.S., two things come to mind:
> > >
> > > 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.
> >
> >         [SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
> > During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).
> >
> >
> > *) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.
> >
> > >
> > > The question I'm thinking about is do we want an Internet where end user plans are divided up this way?
> >
> >         [SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.
> >
> > > And if not, is a NN regulation the right place to put those rules?
> >
> >         [SM] Could well be, but depends on the framing, no?
> >
> > >
> > > 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection.
> >
> >         [SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
> > This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime".
> >
> >
> > > Are NN rules the right place to address this
> >
> >         [SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...
> >
> > > and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?
> >
> >         [SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.
> >
> >
> > Regards
> >         Sebastian
> >
> > P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).
> >
> >
> > >
> > >
> > >
> > > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > Hi Frantisek,
> > >
> > > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > >
> > > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > > > https://www.martingeddes.com/1261-2/
> > > >
> > > > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > > >
> > > >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> > >
> > >         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> > >
> > > [...]
> > > (8)
> > > When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> > > (9)
> > > The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> > > (10)
> > > Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> > > (11)
> > > Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> > > (12)
> > > Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> > > [...]
> > >
> > > There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> > >
> > > There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ).
> > >
> > > Regards
> > >         Sebastian
> > >
> > > P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> > >
> > >
> > >
> > > >
> > > > All the best,
> > > >
> > > > Frank
> > > >
> > > > Frantisek (Frank) Borsik
> > > >
> > > >
> > > >
> > > > https://www.linkedin.com/in/frantisekborsik
> > > >
> > > > Signal, Telegram, WhatsApp: +421919416714
> > > >
> > > > iMessage, mobile: +420775230885
> > > >
> > > > Skype: casioa5302ca
> > > >
> > > > frantisek.borsik@gmail.com
> > > >
> > > >
> > > >
> > > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > ok, lots and lots of great comments here for sure.
> > > >
> > > > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > > > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > > >
> > > > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > > >
> > > > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > > >
> > > > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.
> > > >
> > > > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > > >
> > > >
> > > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > > > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy.
> > > >
> > > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.)
> > > >
> > > > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > > >
> > > > - Rich Brown
> > > >
> > > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > >
> > > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > > > >>
> > > > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > > > >>
> > > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > > >
> > > > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > > >
> > > > >
> > > > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > > >
> > > > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > > >
> > > > >
> > > > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > > >
> > > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > > >
> > > > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > > >
> > > > >
> > > > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > > >
> > > > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > > >
> > > > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > > >
> > > > >
> > > > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > > >
> > > > > - Jonathan Morton
> > > > >
> > > > > _______________________________________________
> > > > > Rpm mailing list
> > > > > Rpm@lists.bufferbloat.net
> > > > > https://lists.bufferbloat.net/listinfo/rpm
> > > >
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > >
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > LibreQoS mailing list
> > LibreQoS@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/libreqos
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos



--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos