Re: [LibreQoS] [Bloat] [Starlink] [Rpm] net neutrality back in the news

Re: [LibreQoS] [Bloat] [Starlink] [Rpm] net neutrality back in the news

[Livingood, Jason]

On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:chromatix99@gmail.com>> wrote:
> Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated

That is not true and really not worth re-litigating here. 

> NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons. 

NN regulations played no role whatsoever in the resolution of that conflict - a business arrangement was reached, just as it was in the SK Telecom example recently: https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to 

> ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet. 

That's simply not true. As someone running an L4S field trial right now - we want the technology to get the widest possible deployment and be fully end-to-end. Why else would there be so much effort to ensure that ECN and DSCP marks can traverse network domain boundaries for example? Why else would there be strong app developer interest? What evidence do you have to show that anyone working on L4S want to create a walled garden? If anything, it seems the opposite of 5G network slicing, which seems to me personally to be another 3GPP run at walled garden stuff (like IMS). Ultimately it is like a lot of other IETF work -- it is an interesting technology and we'll have to see whether it gets good adoption - the 'market' will decide. 

> They want something that can provide a domination service within their own walled gardens. 

Also not correct. And last time I checked the balance sheets of companies in these sectors - video streaming services were losing money while provision of internet services were financially healthy. 

JL

Re: [LibreQoS] [Bloat] [Starlink] [Rpm] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 2563 bytes --]

On Fri, Sep 29, 2023 at 7:17 AM Livingood, Jason via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:
> chromatix99@gmail.com>> wrote:
> > Some ISPs began to actively degrade Netflix traffic, in particular by
> refusing to provision adequate peering capacity at the nodes through which
> Netflix traffic predominated
>
> That is not true and really not worth re-litigating here.
>
> > NN regulations forced ISPs to carry Netflix traffic with reasonable
> levels of service, even though they didn't want to for purely selfish and
> greedy commercial reasons.
>
> NN regulations played no role whatsoever in the resolution of that
> conflict - a business arrangement was reached, just as it was in the SK
> Telecom example recently:
> https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
>
> > ISPs behind L4S actively do not want a technology that works end-to-end
> over the general Internet.
>
> That's simply not true. As someone running an L4S field trial right now -
> we want the technology to get the widest possible deployment and be fully
> end-to-end. Why else would there be so much effort to ensure that ECN and
> DSCP marks can traverse network domain boundaries for example? Why else
> would there be strong app developer interest? What evidence do you have to
> show that anyone working on L4S want to create a walled garden? If
> anything, it seems the opposite of 5G network slicing, which seems to me
> personally to be another 3GPP run at walled garden stuff (like IMS).
> Ultimately it is like a lot of other IETF work -- it is an interesting
> technology and we'll have to see whether it gets good adoption - the
> 'market' will decide.
>
> > They want something that can provide a domination service within their
> own walled gardens.
>
> Also not correct. And last time I checked the balance sheets of companies
> in these sectors - video streaming services were losing money while
> provision of internet services were financially healthy.
>
> JL
>
>
>
I think this stuff degrades into conspiracy theory often enough.  While I
don't discount the possibility of collusion, I don't give these
people/groups credit enough to pull of a mass scale conspiracy either....
If netflix is jammed down to small of a pipe at an ISP, that's more likely
(IMO...) disorganization or incompetence or disinterest over conspiracy.
 I feel the same about government in general...

[-- Attachment #2: Type: text/html, Size: 3210 bytes --]

Re: [LibreQoS] [Rpm] [Bloat] [Starlink] net neutrality back in the news

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 3560 bytes --]

>
> > They want something that can provide a domination service within their
> own walled gardens.
> Also not correct. And last time I checked the balance sheets of companies
> in these sectors - *video streaming services were losing money while
> provision of internet services were financially healthy. *


Indeed, Jason:
https://www.vulture.com/2023/06/streaming-industry-netflix-max-disney-hulu-apple-tv-prime-video-peacock-paramount.html


All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Fri, Sep 29, 2023 at 5:53 PM dan via Rpm <rpm@lists.bufferbloat.net>
wrote:

>
>
> On Fri, Sep 29, 2023 at 7:17 AM Livingood, Jason via LibreQoS <
> libreqos@lists.bufferbloat.net> wrote:
>
>> On 9/29/23, 00:54, "Jonathan Morton" <chromatix99@gmail.com <mailto:
>> chromatix99@gmail.com>> wrote:
>> > Some ISPs began to actively degrade Netflix traffic, in particular by
>> refusing to provision adequate peering capacity at the nodes through which
>> Netflix traffic predominated
>>
>> That is not true and really not worth re-litigating here.
>>
>> > NN regulations forced ISPs to carry Netflix traffic with reasonable
>> levels of service, even though they didn't want to for purely selfish and
>> greedy commercial reasons.
>>
>> NN regulations played no role whatsoever in the resolution of that
>> conflict - a business arrangement was reached, just as it was in the SK
>> Telecom example recently:
>> https://about.netflix.com/en/news/sk-telecom-sk-broadband-and-netflix-establish-strategic-partnership-to
>>
>> > ISPs behind L4S actively do not want a technology that works end-to-end
>> over the general Internet.
>>
>> That's simply not true. As someone running an L4S field trial right now -
>> we want the technology to get the widest possible deployment and be fully
>> end-to-end. Why else would there be so much effort to ensure that ECN and
>> DSCP marks can traverse network domain boundaries for example? Why else
>> would there be strong app developer interest? What evidence do you have to
>> show that anyone working on L4S want to create a walled garden? If
>> anything, it seems the opposite of 5G network slicing, which seems to me
>> personally to be another 3GPP run at walled garden stuff (like IMS).
>> Ultimately it is like a lot of other IETF work -- it is an interesting
>> technology and we'll have to see whether it gets good adoption - the
>> 'market' will decide.
>>
>> > They want something that can provide a domination service within their
>> own walled gardens.
>>
>> Also not correct. And last time I checked the balance sheets of companies
>> in these sectors - video streaming services were losing money while
>> provision of internet services were financially healthy.
>>
>> JL
>>
>>
>>
> I think this stuff degrades into conspiracy theory often enough.  While I
> don't discount the possibility of collusion, I don't give these
> people/groups credit enough to pull of a mass scale conspiracy either....
> If netflix is jammed down to small of a pipe at an ISP, that's more likely
> (IMO...) disorganization or incompetence or disinterest over conspiracy.
>  I feel the same about government in general...
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
>

[-- Attachment #2: Type: text/html, Size: 6299 bytes --]

Re: [LibreQoS] [Rpm] net neutrality back in the news

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 9345 bytes --]

On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> please excuse a number of tangents below ;)

It would be nice, if as a (dis)organisation... the bufferbloat team
could focus on somehow getting both sides of the network neutrality
debate deeplying understanding the technological problem their
pre-conceptions face, and the (now readily available and inexpensive)
solutions that could be deployed, by most ISPs, over a weekend. We are
regularly bringing up a few thousand people a week on libreqos (that
we know of), and then of course, there are all the home routers and
CPE that are increasingly capable of doing the right thing.

The time to try and shift the memes in in the coming days, and weeks.

So ya'all being distracting below... aggh... ok, I'll bite.

>
>
> > On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >
> > Jason just did a beautiful thread as to what was the original source
> > of the network neutrality
> > bittorrent vs voip bufferbloat blowup.
> >
> > https://twitter.com/jlivingood/status/1707078242857849244
>
>         But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.

The average use at peak hours for home broadband is below 5mbits per
subscriber regardless of on a 25mbit or gbit plan. Remarkably,
business plans are less (but more bursty). Oversubscription within a
set of parameters is needed because, in part because upstream
bandwidth to the internet remains expensive (although I hope that cost
continues to drop rapidly), and in part, because it is needlessly
expensive to provision exactly the contracted rate to the customer. As
one example, I use the inverse square law as a rule of thumb for
wireless deployments - the range you can get from a 25/10 deployment
is geometrically better than 100/25, and the average bandwidth usage
nearly the same.

Agreeing on industry standards for what the "slack" should be, might be of help.

>
>
> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.

We have discussed elsewhere, the idea of a minimum contracted rate
(down to), which is easier to reason about.

>
> >
> > Seeing all the political activity tied onto it since (and now again)
> > reminds of two families at war about an incident that had happened
> > generations and generations before, where the two sides no longer
> > remembered why they hated each other so, but just went on hating, and
> > not forgiving, and not moving on.
> >
> > Yes, there are entirely separate and additional NN issues, but the
> > technical problem of providing common carriage between two very
> > different network application types (voip/gaming vs file transfer) is
> > thoroughly solved now,
>
>         I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

Torrent problem thoroughly solved with FQ on the path and backpressure
from the mac scheduler.

https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

>         I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>
>
> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>
>
> > and if only all sides recognised at least this
> > much, and made peace over it, and worked together to deploy those
> > solutions, maybe, just maybe, we could find mutually satisfactory
> > solutions to the other problems that plague the internet today, like
> > security, and the ipv6 rollout.
>
>         +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.

I really would like to see a push for IPv6 mandated as a part of the
BEAD programs.

> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)

There is a separate NRPM going on for the new cybersecurity label at
the FCC. If I had time, and a partner,
we could rework the doc we did a few years ago. It is due oct 6.

>
>
> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>
> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>
> > If anyone here knows anyone more political, still vibrating with 10+
> > years of outrage about NN on this fronts, on one side or the other, if
> > you could sit them down, over a beer, and try to explain that at the
> > start it was a technical problem nobody understood at the time, maybe
> > that would help.
>
>         So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).

Governments make safer markets feasible.

>
> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>
> Regards
>         Sebastian
>
>
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: 2510.jpeg --]
[-- Type: image/jpeg, Size: 66803 bytes --]

Re: [LibreQoS] [Rpm] net neutrality back in the news

[Sebastian Moeller]

> On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>> please excuse a number of tangents below ;)
> 
> It would be nice, if as a (dis)organisation... the bufferbloat team
> could focus on somehow getting both sides of the network neutrality
> debate deeplying understanding the technological problem their
> pre-conceptions face, and the (now readily available and inexpensive)
> solutions that could be deployed, by most ISPs, over a weekend. We are
> regularly bringing up a few thousand people a week on libreqos (that
> we know of), and then of course, there are all the home routers and
> CPE that are increasingly capable of doing the right thing.
> 
> The time to try and shift the memes in in the coming days, and weeks.
> 
> So ya'all being distracting below... aggh... ok, I'll bite.

	[SM] Sorry to drag you into the weeds...


> 
>> 
>> 
>>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Jason just did a beautiful thread as to what was the original source
>>> of the network neutrality
>>> bittorrent vs voip bufferbloat blowup.
>>> 
>>> https://twitter.com/jlivingood/status/1707078242857849244
>> 
>>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> 
> The average use at peak hours for home broadband is below 5mbits per
> subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> business plans are less (but more bursty). Oversubscription within a
> set of parameters is needed because, in part because upstream
> bandwidth to the internet remains expensive (although I hope that cost
> continues to drop rapidly), and in part, because it is needlessly
> expensive to provision exactly the contracted rate to the customer. As
> one example, I use the inverse square law as a rule of thumb for
> wireless deployments - the range you can get from a 25/10 deployment
> is geometrically better than 100/25, and the average bandwidth usage
> nearly the same.

	[SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).


> Agreeing on industry standards for what the "slack" should be, might be of help.

	[SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.


> 
>> 
>> 
>> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> 
> We have discussed elsewhere, the idea of a minimum contracted rate
> (down to), which is easier to reason about.

	[SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
	But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).


*) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....


> 
>> 
>>> 
>>> Seeing all the political activity tied onto it since (and now again)
>>> reminds of two families at war about an incident that had happened
>>> generations and generations before, where the two sides no longer
>>> remembered why they hated each other so, but just went on hating, and
>>> not forgiving, and not moving on.
>>> 
>>> Yes, there are entirely separate and additional NN issues, but the
>>> technical problem of providing common carriage between two very
>>> different network application types (voip/gaming vs file transfer) is
>>> thoroughly solved now,
>> 
>>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

	[SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:

"	• Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20

	• Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.


	• Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21

	• Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."

These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).



> 
> Torrent problem thoroughly solved with FQ on the path and backpressure
> from the mac scheduler.
> 
> https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

	[SM] Thanks for the link. This is mainly for the self-congesrion case?


> 
>>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>> 
>> 
>> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>> 
>> 
>>> and if only all sides recognised at least this
>>> much, and made peace over it, and worked together to deploy those
>>> solutions, maybe, just maybe, we could find mutually satisfactory
>>> solutions to the other problems that plague the internet today, like
>>> security, and the ipv6 rollout.
>> 
>>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> 
> I really would like to see a push for IPv6 mandated as a part of the
> BEAD programs.

	[SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.

> 
>> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> 
> There is a separate NRPM going on for the new cybersecurity label at
> the FCC. If I had time, and a partner,
> we could rework the doc we did a few years ago. It is due oct 6.
> 
>> 
>> 
>> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>> 
>> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>> 
>>> If anyone here knows anyone more political, still vibrating with 10+
>>> years of outrage about NN on this fronts, on one side or the other, if
>>> you could sit them down, over a beer, and try to explain that at the
>>> start it was a technical problem nobody understood at the time, maybe
>>> that would help.
>> 
>>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> 
> Governments make safer markets feasible.

	[SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.

Regards & Sorry for the tangent
	Sebastian

> 
>> 
>> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>> 
>> Regards
>>        Sebastian
>> 
>> 
>>> 
>>> --
>>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> <2510.jpeg>

Re: [LibreQoS] [Rpm] net neutrality back in the news

[Dave Taht]

@Sebastian: This is a really great list of what the issues were in the
EU, and if y'all can repost there, rather than here, perhaps more
light will be generated outside our circles.

https://news.ycombinator.com/item?id=37694306#37694307

On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
>
> > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Dave,
> >>
> >> please excuse a number of tangents below ;)
> >
> > It would be nice, if as a (dis)organisation... the bufferbloat team
> > could focus on somehow getting both sides of the network neutrality
> > debate deeplying understanding the technological problem their
> > pre-conceptions face, and the (now readily available and inexpensive)
> > solutions that could be deployed, by most ISPs, over a weekend. We are
> > regularly bringing up a few thousand people a week on libreqos (that
> > we know of), and then of course, there are all the home routers and
> > CPE that are increasingly capable of doing the right thing.
> >
> > The time to try and shift the memes in in the coming days, and weeks.
> >
> > So ya'all being distracting below... aggh... ok, I'll bite.
>
>         [SM] Sorry to drag you into the weeds...
>
>
> >
> >>
> >>
> >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >>>
> >>> Jason just did a beautiful thread as to what was the original source
> >>> of the network neutrality
> >>> bittorrent vs voip bufferbloat blowup.
> >>>
> >>> https://twitter.com/jlivingood/status/1707078242857849244
> >>
> >>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> >
> > The average use at peak hours for home broadband is below 5mbits per
> > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > business plans are less (but more bursty). Oversubscription within a
> > set of parameters is needed because, in part because upstream
> > bandwidth to the internet remains expensive (although I hope that cost
> > continues to drop rapidly), and in part, because it is needlessly
> > expensive to provision exactly the contracted rate to the customer. As
> > one example, I use the inverse square law as a rule of thumb for
> > wireless deployments - the range you can get from a 25/10 deployment
> > is geometrically better than 100/25, and the average bandwidth usage
> > nearly the same.
>
>         [SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).
>
>
> > Agreeing on industry standards for what the "slack" should be, might be of help.
>
>         [SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.
>
>
> >
> >>
> >>
> >> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> >
> > We have discussed elsewhere, the idea of a minimum contracted rate
> > (down to), which is easier to reason about.
>
>         [SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
>         But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).
>
>
> *) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....
>
>
> >
> >>
> >>>
> >>> Seeing all the political activity tied onto it since (and now again)
> >>> reminds of two families at war about an incident that had happened
> >>> generations and generations before, where the two sides no longer
> >>> remembered why they hated each other so, but just went on hating, and
> >>> not forgiving, and not moving on.
> >>>
> >>> Yes, there are entirely separate and additional NN issues, but the
> >>> technical problem of providing common carriage between two very
> >>> different network application types (voip/gaming vs file transfer) is
> >>> thoroughly solved now,
> >>
> >>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
>
>         [SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:
>
> "       • Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20
>
>         • Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.
>
>
>         • Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21
>
>         • Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."
>
> These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
> I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).
>
>
>
> >
> > Torrent problem thoroughly solved with FQ on the path and backpressure
> > from the mac scheduler.
> >
> > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
>
>         [SM] Thanks for the link. This is mainly for the self-congesrion case?
>
>
> >
> >>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
> >>
> >>
> >> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
> >>
> >>
> >>> and if only all sides recognised at least this
> >>> much, and made peace over it, and worked together to deploy those
> >>> solutions, maybe, just maybe, we could find mutually satisfactory
> >>> solutions to the other problems that plague the internet today, like
> >>> security, and the ipv6 rollout.
> >>
> >>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> >
> > I really would like to see a push for IPv6 mandated as a part of the
> > BEAD programs.
>
>         [SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.
>
> >
> >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> >
> > There is a separate NRPM going on for the new cybersecurity label at
> > the FCC. If I had time, and a partner,
> > we could rework the doc we did a few years ago. It is due oct 6.
> >
> >>
> >>
> >> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
> >>
> >> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
> >>
> >>> If anyone here knows anyone more political, still vibrating with 10+
> >>> years of outrage about NN on this fronts, on one side or the other, if
> >>> you could sit them down, over a beer, and try to explain that at the
> >>> start it was a technical problem nobody understood at the time, maybe
> >>> that would help.
> >>
> >>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> >
> > Governments make safer markets feasible.
>
>         [SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.
>
> Regards & Sorry for the tangent
>         Sebastian
>
> >
> >>
> >> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
> >>
> >> Regards
> >>        Sebastian
> >>
> >>
> >>>
> >>> --
> >>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> >>> Dave Täht CSO, LibreQos
> >>> _______________________________________________
> >>> Rpm mailing list
> >>> Rpm@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/rpm
> >>
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > <2510.jpeg>
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [LibreQoS] [Rpm] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 19919 bytes --]

"(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most'
in terms of customers being served by puts the Spectrums and Comcasts in
the mix (in the US) and they don't care about happy customers they care
about defacto monopolies in markets so that they don't have to care about
happy customers.  That might not be their motivation, but 30 years of their
behavior makes it clear that it's the case.


From my perspective, there should be a clear separation between carriers
and last mile delivery..  Even if it's the same company, the rules defining
each really should be different.

Common Carriers or rather, carrier class services for 'internet', should be
completely neutral.  Packets are packets.  However, I think it's important
to carve out methods to have dedicated links for real time flows at the
carrier level.  I don't know what that model looks like exactly, but being
too stubborn about purist NN principals could really hurt VoIP services if
there aren't methods to handle that.  I guess I really am describing
'internet fast lanes' for certain classes of services that we deem
important enough as a whole.  not individual ISPs deciding, but rather 'the
will of the people' saying VoIP is more important than netflix, you can
carve out dedicated capacity for that.

For the last mile, I'm actually less concerned with pure NN and more
concerned with no-blocking or 'brand' prioritization and required/label
transparency.  ie, if an ISP wants to sell a DPI's QoE'd service that says
'videos at 1080p' and that is on the product label clearly and not in small
print, then that's fine so long as 'videos' is agnostic of the vendor
INCLUDING the ISP's own video products.  "100Mbps internet services with
1080P video $65" for example.  "Upgrade to our 100M with 4k video for
+$10/m".  With that level of transparency I think that consumer protections
are well handled.  And, because of that transparency, normal market
capitalism mechanisms work.  I can say "100Mbps for $65 and full resolution
video" for example.   Or "100Mbps Net Neutral service".   The secret DPI's
QoE shaping is my main concern here, and where I think consumer protection
needs to be pursued.

Again however, I think that ISPs should be able to offer dedicated paths or
bandwidth for specific services like VoIP.  Services listed in a publicly
determined products list.


On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> @Sebastian: This is a really great list of what the issues were in the
> EU, and if y'all can repost there, rather than here, perhaps more
> light will be generated outside our circles.
>
> https://news.ycombinator.com/item?id=37694306#37694307
>
> On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> >
> >
> >
> > > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> please excuse a number of tangents below ;)
> > >
> > > It would be nice, if as a (dis)organisation... the bufferbloat team
> > > could focus on somehow getting both sides of the network neutrality
> > > debate deeplying understanding the technological problem their
> > > pre-conceptions face, and the (now readily available and inexpensive)
> > > solutions that could be deployed, by most ISPs, over a weekend. We are
> > > regularly bringing up a few thousand people a week on libreqos (that
> > > we know of), and then of course, there are all the home routers and
> > > CPE that are increasingly capable of doing the right thing.
> > >
> > > The time to try and shift the memes in in the coming days, and weeks.
> > >
> > > So ya'all being distracting below... aggh... ok, I'll bite.
> >
> >         [SM] Sorry to drag you into the weeds...
> >
> >
> > >
> > >>
> > >>
> > >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> > >>>
> > >>> Jason just did a beautiful thread as to what was the original source
> > >>> of the network neutrality
> > >>> bittorrent vs voip bufferbloat blowup.
> > >>>
> > >>> https://twitter.com/jlivingood/status/1707078242857849244
> > >>
> > >>        But the core issue IMHO really was an economic one, the
> over-subscription ratios that worked before torrenting simply did not cut
> it any more in an environment when customers actually tried to use their
> contracted access rates "quantitatively". In my outsider perspective an ISP
> owes its customers the contracted rates (with some slack*) and any sort of
> over-subscription is a valid economic optimization an ISP might take, IFF
> that ISP is prepared to rapidly increase segment capacity (or down-grade
> customer plans)) if the deployed over-subscription rate proves to have been
> too optimistic. Mind you, most ISPs market plans by access speed (and
> charge more for higher speeds) and hence are somewhat responsible to
> actually deliver (again with some slack) these speeds.
> > >
> > > The average use at peak hours for home broadband is below 5mbits per
> > > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > > business plans are less (but more bursty). Oversubscription within a
> > > set of parameters is needed because, in part because upstream
> > > bandwidth to the internet remains expensive (although I hope that cost
> > > continues to drop rapidly), and in part, because it is needlessly
> > > expensive to provision exactly the contracted rate to the customer. As
> > > one example, I use the inverse square law as a rule of thumb for
> > > wireless deployments - the range you can get from a 25/10 deployment
> > > is geometrically better than 100/25, and the average bandwidth usage
> > > nearly the same.
> >
> >         [SM] +1; hence my argument is not oversubscription per se is
> bad, but that oversubscription needs to be managed and the level of
> oversubscription needs to to be adapted ot he actual usage patterns. I am
> sure however that ISPs already actively do that (I assume most ISPs want
> happy customers).
> >
> >
> > > Agreeing on industry standards for what the "slack" should be, might
> be of help.
> >
> >         [SM] Not being part of that industry at all I would love to see
> that as well, but can not contribute to defining that in any way.
> >
> >
> > >
> > >>
> > >>
> > >> *) Claiming "Up to", only carries that far, if you sell me X and I
> mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is
> acceptable unless occasionally is "at every late afternoon and evening"
> prime-time.
> > >
> > > We have discussed elsewhere, the idea of a minimum contracted rate
> > > (down to), which is easier to reason about.
> >
> >         [SM] Indeed, the German regulator (and I am not saying this is
> the only or best option) decided to require ISPs to give 6 numbers
> pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate,
> a minimal (net) rate, and a usually achievable (net) rate, all rates were
> defined as IP/TCP goodput to make verification easier on end-users. The
> regulator also defined a measurement regime that end-users can follow to
> check whether the ISP is fulfilling the contract and the law gives remedies
> if ISPs do not deliver (either the right to immediately cancel the contract
> and/or the right to adapt the monthly payments to the actually delivered
> ratio of the contracted rates*). I think I need to add, that ISPs can set
> these numbers freely, but are only allowed to advertise with the maximum
> rate.
> >         But if we talk about a single number per direction, minimal rate
> is probably easiest, or something like a "usually achievable rate" (that
> needs to be met or exceeded in say 90% of >= 20 measurements or so).
> >
> >
> > *) In a ironic twist however the regulator so far has not explained how
> deviations in each of the 6 numbers above should be aggregated to get one
> single contract rate achievement ratio..., most ISPs took measures into
> their own hands and simply offer customers typically a permanent rebate of
> 5 EUR or immediate cancelation what ever the customer prefers....
> >
> >
> > >
> > >>
> > >>>
> > >>> Seeing all the political activity tied onto it since (and now again)
> > >>> reminds of two families at war about an incident that had happened
> > >>> generations and generations before, where the two sides no longer
> > >>> remembered why they hated each other so, but just went on hating, and
> > >>> not forgiving, and not moving on.
> > >>>
> > >>> Yes, there are entirely separate and additional NN issues, but the
> > >>> technical problem of providing common carriage between two very
> > >>> different network application types (voip/gaming vs file transfer) is
> > >>> thoroughly solved now,
> > >>
> > >>        I am not sure this was at the core of the problem, my take is
> really that "always-on" and relative upload-heavy torrent simply
> demonstrated painfully for all involved that the old oversubscription
> ratios were not suitable for the changed traffic profiles. I have some
> sympathy for the ISPs here, they certainly did not try to sell their
> customers short (good ISPs try to retain their customers and that works
> best when customers are happy with the service) and having this problem
> appear on many segments at the same time is not a fun place to be, and
> upload was/is often (too) low in DOCSIS segments anyway; but this is why
> e.g. my bit torrent could affect your VoIP, simply by pushing the whole
> segment into upload capacity congestion... (ISPs in theory could fix this
> by plain old QoS engineering, but the issue at hand was with a non-ISP
> VoIP/SIP service and there QoS becomes tricky if the ISP as these packets
> need to be identified in a way that is not game-able**)
> >
> >         [SM] See my later mail to Jason, I will not claim I know
> Comcast's issues better than him and will accept that self-congestion also
> played a major role in the initial problem. Over here in Europe the net
> neutrality debate was kicked of less by an unfortunate confluence of new
> usage profiles and traditional oversibscription ratios and less than ideal
> packet scheduling, but rather by a series of pretty apparent attempts at
> restricting consumer choice, see eg.
> https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf
> for an admitted slightly biased position:
> >
> > "       • Blocking of applications and services: In order to maximise
> profits, some ISPs – that also offer their own services and applications
> online – exclude certain services and applications of competing market
> players. The most prominent case of this form of network discrimination is
> European mobile providers (like Deutsche Telekom) blocking or restricting
> the use of Voice over IP (VoIP) services (like Skype and Viber) for their
> customers.20
> >
> >         • Slowing or “throttling” internet speeds: Some ISPs slow down
> specific services (like YouTube) and applications (like Skype), or ask
> users to pay an extra fee to have access to these internet platforms. Given
> the high latency (delay) sensitivity of many applications, ISPs are able to
> compromise the correct functioning of these services by slowing them down,
> preventing the services from running properly. Often ISPs – especially
> telecommunication companies – do this to favour their own voice calling
> services over VoIP services, thereby crushing competition.
> >
> >
> >         • Blocking websites: ISPs often block websites for a number of
> reasons – to secure their network, or to avoid competition, and sometimes
> for social, public relations or political reasons. In the UK, for instance,
> Orange Telecom blocked the French digital rights advocacy group, La
> Quadrature du Net’s website on pre-paid mobile accounts.21
> >
> >         • Preferential treatment of services and platforms: ISPs can
> also impose data caps on internet access contracts while granting data
> allowance exceptions to a company’s own proprietary streaming services
> (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also
> grant preferential treatment to select services – such as Orange France
> with the popular music streaming service Deezer23 – ahead of other
> competitors, effectively imposing anti-competitive limitations on markets
> such as those for legal online music. Moreover, generally only large,
> well-established companies can afford this preferential treatment,
> resulting in a further stifling of innovation."
> >
> > These look like clearer attempts to monetize the ISP position as
> gate-keeper to his customer's eye-balls (for content providers) as well as
> gate-keeper to the internet for for paying customers.
> > I find it much harder to have sympathy for the listed examples of ISP
> behavior than the situation of rapidly changing usage pattern posed where
> technical changes needed to be made, but where no attempt at unfair
> monetization was evident, but maybe I am overly sensitive. These are all
> examples that make me personally applaud the EU for its intervention to
> make clear rules what "internet access providers" can and can not do. (The
> EU also was quite flexible in applying/interpreting its rules during the
> covid isolation periods, when it made it clear that e.g. treating certain
> traffic classes e.g. streaming media to lower priority than video
> conferences was within the neutrality framework as long as it was based on
> traffic class and not on specific service providers IIRC).
> >
> >
> >
> > >
> > > Torrent problem thoroughly solved with FQ on the path and backpressure
> > > from the mac scheduler.
> > >
> > > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
> >
> >         [SM] Thanks for the link. This is mainly for the self-congesrion
> case?
> >
> >
> > >
> > >>        I agree that on a single link we mostly solved the problem (a
> few corner cases are left on links with very low capacity, where
> essentially we can only manage the pain, not remove it)...
> > >>
> > >>
> > >> **) Which is not rocket science either, a VoIP stream takes ~100
> Kbps, so in theory an ISP might simply allow each customer say 5 VoIP
> stream equivalents by allowing up to 500Kbps od traffic marked with a
> specific DSCP as higher priority (with higher access probability for the
> shared medium). I am not sayng this is my preferred solution, just saying
> this is a solution that would have been available at the time if I memorize
> my docsis capabilities correctly.
> > >>
> > >>
> > >>> and if only all sides recognised at least this
> > >>> much, and made peace over it, and worked together to deploy those
> > >>> solutions, maybe, just maybe, we could find mutually satisfactory
> > >>> solutions to the other problems that plague the internet today, like
> > >>> security, and the ipv6 rollout.
> > >>
> > >>        +1. IPv6 is IMHO a prime example where the regulators should
> stop talking softly and start showing the big stick they carry.
> > >
> > > I really would like to see a push for IPv6 mandated as a part of the
> > > BEAD programs.
> >
> >         [SM] +1; I am not the biggest IPv6 fan, but that's what we have
> and it is mostly serviceable so let's get this "party started" finally.
> >
> > >
> > >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4
> addresses only.. (most mass market ISPs do much better in that regard, but
> for the stragglers it would help if the regulator would demand IPv6 with
> PD***). Regarding security, the easiest way to achieve that would be to put
> some heavy requirements on IoT manufacturers and vendors (like do what you
> please as long as you are local LAN only, but once you reach out into the
> cloud you need to fulfill the following list of requirements, with timely
> security updates over a reasonable long usage period), however that might
> not be very attractive for politicians/regulators to tackle (active
> regulatory acts tends to get bad press unless something bad happened, but
> even then the press often complains about the acts coming too late, but I
> digress****)
> > >
> > > There is a separate NRPM going on for the new cybersecurity label at
> > > the FCC. If I had time, and a partner,
> > > we could rework the doc we did a few years ago. It is due oct 6.
> > >
> > >>
> > >>
> > >> ***) Strictly speaking IPv6 is required, since "internet access" is
> defined as reaching all of the internet (as far as in the ISPs power) and
> IPv6-only sites are not reachable for the CG-NAT-only customers. But so far
> the local regulator does not seem to enforce that requirement, or hopefully
> is working on this quietly behind the curtains.
> > >>
> > >> ****) This is not to diss the press, they are doing what they are
> supposed to do, but it just shows that active regulation is a tricky
> business, and a light touch typically "looks better" (even though I see no
> real evidence it actually works better).
> > >>
> > >>> If anyone here knows anyone more political, still vibrating with 10+
> > >>> years of outrage about NN on this fronts, on one side or the other,
> if
> > >>> you could sit them down, over a beer, and try to explain that at the
> > >>> start it was a technical problem nobody understood at the time, maybe
> > >>> that would help.
> > >>
> > >>        So in the EU that debate is essentially settled, there is a EU
> regulation that essentially spills out what ISPs owe their customers and
> that has become the law of the land. The rationale for required un-biased
> service and freedom to select terminal devices is well justified by the
> market ideals of the EU, allowing ISPs to discriminate packets or terminal
> devices restricts the market and will lead to undesired outcomes. (Fun fact
> most big players in capitalist societies argue for "free markets" but at
> the same time act to work-around the market mechanism by trying to move the
> market into an oligo- or even monopoly condition, which is why strong
> regulation is required*****).
> > >
> > > Governments make safer markets feasible.
> >
> >         [SM] Yes, I agree, both in markets are a pretty decent tool, but
> need constant maintenance.
> >
> > Regards & Sorry for the tangent
> >         Sebastian
> >
> > >
> > >>
> > >> *****) This is akin to professional sports where the audience
> generally accepts that referees are necessary and occasionally need to make
> "painful" calls, as the alternative would be anarchy, but I digress.
> > >>
> > >> Regards
> > >>        Sebastian
> > >>
> > >>
> > >>>
> > >>> --
> > >>> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > >>> Dave Täht CSO, LibreQos
> > >>> _______________________________________________
> > >>> Rpm mailing list
> > >>> Rpm@lists.bufferbloat.net
> > >>> https://lists.bufferbloat.net/listinfo/rpm
> > >>
> > >
> > >
> > > --
> > > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > > Dave Täht CSO, LibreQos
> > > <2510.jpeg>
> >
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 23355 bytes --]

Re: [LibreQoS] [Starlink] [Rpm] net neutrality back in the news

[Livingood, Jason]

> dan <dandenson@gmail.com> wrote: 

> "(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers.  

Corporations are motivated to generate returns for investors. In that context, happy customers stay longer (less churn) and spend more (upgrades, multiple services). And unhappy customers generate costs via disconnects (loss of revenue, costs to replace them with a new customer to just stay at the same subscriber levels), and costs via customer contacts (call center staff). So, IMO on a purely financial basis, public companies have significant motivation to retain customers and keep them happy. This typically follows through to staff members having part of their variable compensation based on things like NPS scores, contact rates, etc. And specifically in relation to Comcast, the company recently has 4 new wireless competitors: three 5G FWA and one LEO (more coming) - and those are posing significant competitive risks (and taking customers). 

> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...

The two thoughts your comments (thanks for the response BTW!) trigger are:
1 - Often regulation looks to the past - in this case maybe an era of bandwidth scarcity where prioritization may have mattered. I think we're in the midst of a shift into bandwidth abundance where priority does not matter. What will is latency/responsiveness, content/compute localization, reliability, consistency, security, etc. 
2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care (contact) costs and would see people start to immediately shift to competitors (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like something that could realistically happen any longer in the US.

JL

Re: [LibreQoS] [Bloat] [Starlink] [Rpm] net neutrality back in the news

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 3155 bytes --]

On Thu, 28 Sep 2023, Livingood, Jason via Bloat wrote:

> Date: Thu, 28 Sep 2023 20:48:58 +0000
> From: "Livingood, Jason via Bloat" <bloat@lists.bufferbloat.net>
> Reply-To: "Livingood, Jason" <Jason_Livingood@comcast.com>
> To: dan <dandenson@gmail.com>, Dave Taht <dave.taht@gmail.com>
> Cc: Rpm <rpm@lists.bufferbloat.net>,
>     Dave Taht via Starlink <starlink@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>,
>     libreqos <libreqos@lists.bufferbloat.net>,
>     Jamal Hadi Salim <jhs@mojatatu.com>
> Subject: Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the
>     news
> 
>> dan <dandenson@gmail.com> wrote:
>
>> "(I assume most ISPs want happy customers)."

>> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' 
>> in terms of customers being served by puts the Spectrums and Comcasts in the 
>> mix (in the US) and they don't care about happy customers they care about 
>> defacto monopolies in markets so that they don't have to care about happy 
>> customers. 
>
> In that context, happy customers stay longer (less churn) and spend more 
> (upgrades, multiple services). And unhappy customers generate costs via 
> disconnects (loss of revenue, costs to replace them with a new customer to 
> just stay at the same subscriber levels), and costs via customer contacts 
> (call center staff).

Except when you have a monopoly in an area, at which point the ability of 
customers to leave is minimal, and years of bad customer service means that 
people don't bother complaining, so the call center staffing costs are lower 
than they should be.

>> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...
>
> The two thoughts your comments (thanks for the response BTW!) trigger are:

> 1 - Often regulation looks to the past - in this case maybe an era of 
> bandwidth scarcity where prioritization may have mattered. I think we're in 
> the midst of a shift into bandwidth abundance where priority does not matter. 
> What will is latency/responsiveness, content/compute localization, 
> reliability, consistency, security, etc.

> 2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care 
> (contact) costs and would see people start to immediately shift to competitors 
> (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like 
> something that could realistically happen any longer in the US.

Dave T called out earlier that the rise of bittorrent was a large part of the 
inital NN discussion here in the US. But a second large portion was a money grab 
from ISPs thinking that they could hold up large paid websites (netflix for 
example) for additional fees by threatening to make their service less useful to 
their users (viewing their users as an asset to be marketed to the websites 
rather than customers to be satisfied by providing them access to the websites)

I don't know if a new round of "it's not fair that Netflix doesn't pay us for 
the bandwidth to service them" would fall flat at this point or not.

David Lang

Re: [LibreQoS] [Bloat] [Starlink] [Rpm] net neutrality back in the news

[Jonathan Morton]

> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> 
> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.

I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:


1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.

This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.


2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.

FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.

This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.


3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.

**THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.

And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.


All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.

 - Jonathan Morton