[Rpm] net neutrality back in the news

[Rpm] net neutrality back in the news

[Dave Taht]

Jason just did a beautiful thread as to what was the original source
of the network neutrality
bittorrent vs voip bufferbloat blowup.

https://twitter.com/jlivingood/status/1707078242857849244

Seeing all the political activity tied onto it since (and now again)
reminds of two families at war about an incident that had happened
generations and generations before, where the two sides no longer
remembered why they hated each other so, but just went on hating, and
not forgiving, and not moving on.

Yes, there are entirely separate and additional NN issues, but the
technical problem of providing common carriage between two very
different network application types (voip/gaming vs file transfer) is
thoroughly solved now, and if only all sides recognised at least this
much, and made peace over it, and worked together to deploy those
solutions, maybe, just maybe, we could find mutually satisfactory
solutions to the other problems that plague the internet today, like
security, and the ipv6 rollout.

If anyone here knows anyone more political, still vibrating with 10+
years of outrage about NN on this fronts, on one side or the other, if
you could sit them down, over a beer, and try to explain that at the
start it was a technical problem nobody understood at the time, maybe
that would help.

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] net neutrality back in the news

[rjmcmahon]

Common Carriage goes way beyond our lifes. Eli Noam's write up in 1994 
is a good one.

http://www.columbia.edu/dlc/wp/citi/citinoam11.html

Beyond Liberalization II:
The Impending Doom of Common Carriage
Eli M. Noam
Professor of Finance and Economics
Columbia University, Graduate School of Business
March 15, 1994

I. Introduction 1

This article argues that the institution of common carriage, 
historically the foundation of the way telecommunications are delivered, 
will not survive. To clarify: "common carriers" (the misnomer often used 
to refer to telephone companies) will continue to exist, but the status 
under which they operate -- offering service on a non-discriminatory 
basis, neutral as to use and user -- will not.

...

VII. A Contract-Carrier Based Telecommunications System?

The conclusion of the analysis has been that common carriage will erode 
in time, and that a hybrid co-existence will not be stable. This is not 
to say that the common carriers qua carriers will become extinct; many 
of them will remain significant players, but they will conduct their 
business as contract carriers. But common carriage as such will 
disappear. This will not happen overnight, of course. Intermediate 
arrangements can buy several decades of transition time. But the basic 
dynamics will eventually assert themselves.

This conclusion is reached with much regret, because the socially 
positive aspects of common carriage are strong, and because the absence 
to common carriage often means gatekeeper power. But we should not let 
preferences obscure the clarity of analysis.

Bob
> Jason just did a beautiful thread as to what was the original source
> of the network neutrality
> bittorrent vs voip bufferbloat blowup.
> 
> https://twitter.com/jlivingood/status/1707078242857849244
> 
> Seeing all the political activity tied onto it since (and now again)
> reminds of two families at war about an incident that had happened
> generations and generations before, where the two sides no longer
> remembered why they hated each other so, but just went on hating, and
> not forgiving, and not moving on.
> 
> Yes, there are entirely separate and additional NN issues, but the
> technical problem of providing common carriage between two very
> different network application types (voip/gaming vs file transfer) is
> thoroughly solved now, and if only all sides recognised at least this
> much, and made peace over it, and worked together to deploy those
> solutions, maybe, just maybe, we could find mutually satisfactory
> solutions to the other problems that plague the internet today, like
> security, and the ipv6 rollout.
> 
> If anyone here knows anyone more political, still vibrating with 10+
> years of outrage about NN on this fronts, on one side or the other, if
> you could sit them down, over a beer, and try to explain that at the
> start it was a technical problem nobody understood at the time, maybe
> that would help.

Re: [Rpm] net neutrality back in the news

[Dave Taht]

Dear Bob:

I have always thought the long standing dispute over moving flow
queuing (FQ) into the network has always been about enabling common
carriage for the vast majority of possible use cases for the internet
or not, going all the way back to the fights over it in 1989, and many
discussions prior to that. I haven´t talked about it this way much,
preferring to enable the users first, to make their own choices in
spite of their service providers. I felt showing the general benefit
for making voip, gaming, videoconferencing, web traffic, and bulk
traffic, all co-exist in harmony, and trying to solve the thorny
problems extant in all the things that I had source code to, like
WiFi, was needed, first. Making them these algorithms, so commonly
available, and so obvious, and inevitable, seemed best, rather than
duking it out directly in the political arena, for which I do not have
the stomach or time. I also felt that in a competitive market, fq
would win.

A few days ago John Nagel talked to this point, somewhat. From hackernews:

Animats 3 days ago | root | parent | next [–]

> If everyone played nice, the exponential backoff timers would work as expected,

"As I wrote in 1985, in [1], under "Game Theoretic Aspects of Network
Congestion" and "Fairness in Packet Switching Systems", about fair
queuing, We would like to protect the network from hosts that are not
well-behaved. More specifically, we would like, in the presence of
both well-behaved and badly-behaved hosts, to insure that well-behaved
hosts receive better service than badly-behaved hosts. We have devised
a means of achieving this. The goal of fair queuing is not to improve
network performance overall. The goal of fair queuing is to reward
well-behaved hosts over badly-behaved hosts. If everyone is
well-behaved, the queue lengths are the same, usually 1 or 0, and fair
queuing does little.

There is an inherent conflict between this goal and achieving maximum
data transfer rates. If you try for near 100% utilization, the
problems become much worse. You can run comfortably at maybe 70%. This
was an accepted tradeoff for DoD systems. DoD wants things to keep
working in a crisis, even if normal operation is a bit slower. This is
why I'm not a big fan of HTTP/3. It's a attempt to get about 10% more
performance in the good case, at the cost of considerable extra
complexity and less immunity to gaming the system.

I never wrote about that much at the time, because if I had, people
would have realized earlier that traffic shaping is possible, which
implies that you can sell and bill for bandwidth and quality of
service. We might have ended up with pay per packet."

As for whether this realization has bubbled up as far as the FCC or
not, or the machinations to not deploy our fixes to bufferbloat
without retaining some sort of control as a MITM were conscious by
those fighting it so strongly, I do not know.

I do see the initial reactions to

https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf

seem rather canned so far, and yet reading that post over feels good...

... and I hope that perhaps tomorrow will be a brighter day for the
whole internet, than it is had in a long time.

On Wed, Sep 27, 2023 at 8:33 PM rjmcmahon <rjmcmahon@rjmcmahon.com> wrote:
>
> Common Carriage goes way beyond our lifes. Eli Noam's write up in 1994
> is a good one.
>
> http://www.columbia.edu/dlc/wp/citi/citinoam11.html
>
> Beyond Liberalization II:
> The Impending Doom of Common Carriage
> Eli M. Noam
> Professor of Finance and Economics
> Columbia University, Graduate School of Business
> March 15, 1994
>
> I. Introduction 1
>
> This article argues that the institution of common carriage,
> historically the foundation of the way telecommunications are delivered,
> will not survive. To clarify: "common carriers" (the misnomer often used
> to refer to telephone companies) will continue to exist, but the status
> under which they operate -- offering service on a non-discriminatory
> basis, neutral as to use and user -- will not.
>
> ...
>
> VII. A Contract-Carrier Based Telecommunications System?
>
> The conclusion of the analysis has been that common carriage will erode
> in time, and that a hybrid co-existence will not be stable. This is not
> to say that the common carriers qua carriers will become extinct; many
> of them will remain significant players, but they will conduct their
> business as contract carriers. But common carriage as such will
> disappear. This will not happen overnight, of course. Intermediate
> arrangements can buy several decades of transition time. But the basic
> dynamics will eventually assert themselves.
>
> This conclusion is reached with much regret, because the socially
> positive aspects of common carriage are strong, and because the absence
> to common carriage often means gatekeeper power. But we should not let
> preferences obscure the clarity of analysis.
>
> Bob
> > Jason just did a beautiful thread as to what was the original source
> > of the network neutrality
> > bittorrent vs voip bufferbloat blowup.
> >
> > https://twitter.com/jlivingood/status/1707078242857849244
> >
> > Seeing all the political activity tied onto it since (and now again)
> > reminds of two families at war about an incident that had happened
> > generations and generations before, where the two sides no longer
> > remembered why they hated each other so, but just went on hating, and
> > not forgiving, and not moving on.
> >
> > Yes, there are entirely separate and additional NN issues, but the
> > technical problem of providing common carriage between two very
> > different network application types (voip/gaming vs file transfer) is
> > thoroughly solved now, and if only all sides recognised at least this
> > much, and made peace over it, and worked together to deploy those
> > solutions, maybe, just maybe, we could find mutually satisfactory
> > solutions to the other problems that plague the internet today, like
> > security, and the ipv6 rollout.
> >
> > If anyone here knows anyone more political, still vibrating with 10+
> > years of outrage about NN on this fronts, on one side or the other, if
> > you could sit them down, over a beer, and try to explain that at the
> > start it was a technical problem nobody understood at the time, maybe
> > that would help.



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [Starlink] net neutrality back in the news

[Sebastian Moeller]

Hi Dave,

I really like this argument by John Nagle. It is IMHO complementary (and stronger) to what keep arguing, that far queueing is not most optimal but least pessimal (and close to the best an intermediary node can do with the limited information at hand, anything else needs additional information). What I think is elegant about flow-scheduling is that it taps into essential information for digital communications and hence is hard to exploit (hard, not impossible).


> On Sep 28, 2023, at 08:06, Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
> 
> Dear Bob:
> 
> I have always thought the long standing dispute over moving flow
> queuing (FQ) into the network has always been about enabling common
> carriage for the vast majority of possible use cases for the internet
> or not, going all the way back to the fights over it in 1989, and many
> discussions prior to that. I haven´t talked about it this way much,
> preferring to enable the users first, to make their own choices in
> spite of their service providers. I felt showing the general benefit
> for making voip, gaming, videoconferencing, web traffic, and bulk
> traffic, all co-exist in harmony, and trying to solve the thorny
> problems extant in all the things that I had source code to, like
> WiFi, was needed, first. Making them these algorithms, so commonly
> available, and so obvious, and inevitable, seemed best, rather than
> duking it out directly in the political arena, for which I do not have
> the stomach or time. I also felt that in a competitive market, fq
> would win.
> 
> A few days ago John Nagel talked to this point, somewhat. From hackernews:
> 
> Animats 3 days ago | root | parent | next [–]
> 
>> If everyone played nice, the exponential backoff timers would work as expected,
> 
> "As I wrote in 1985, in [1], under "Game Theoretic Aspects of Network
> Congestion" and "Fairness in Packet Switching Systems", about fair
> queuing, We would like to protect the network from hosts that are not
> well-behaved. More specifically, we would like, in the presence of
> both well-behaved and badly-behaved hosts, to insure that well-behaved
> hosts receive better service than badly-behaved hosts. We have devised
> a means of achieving this. The goal of fair queuing is not to improve
> network performance overall. The goal of fair queuing is to reward
> well-behaved hosts over badly-behaved hosts. If everyone is
> well-behaved, the queue lengths are the same, usually 1 or 0, and fair
> queuing does little.
> 
> There is an inherent conflict between this goal and achieving maximum
> data transfer rates. If you try for near 100% utilization, the
> problems become much worse. You can run comfortably at maybe 70%. This
> was an accepted tradeoff for DoD systems. DoD wants things to keep
> working in a crisis, even if normal operation is a bit slower. This is
> why I'm not a big fan of HTTP/3. It's a attempt to get about 10% more
> performance in the good case, at the cost of considerable extra
> complexity and less immunity to gaming the system.
> 
> I never wrote about that much at the time, because if I had, people
> would have realized earlier that traffic shaping is possible, which
> implies that you can sell and bill for bandwidth and quality of
> service. We might have ended up with pay per packet."
> 
> As for whether this realization has bubbled up as far as the FCC or
> not, or the machinations to not deploy our fixes to bufferbloat
> without retaining some sort of control as a MITM were conscious by
> those fighting it so strongly, I do not know.
> 
> I do see the initial reactions to
> 
> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf
> 
> seem rather canned so far, and yet reading that post over feels good...
> 
> ... and I hope that perhaps tomorrow will be a brighter day for the
> whole internet, than it is had in a long time.
> 
> On Wed, Sep 27, 2023 at 8:33 PM rjmcmahon <rjmcmahon@rjmcmahon.com> wrote:
>> 
>> Common Carriage goes way beyond our lifes. Eli Noam's write up in 1994
>> is a good one.
>> 
>> http://www.columbia.edu/dlc/wp/citi/citinoam11.html
>> 
>> Beyond Liberalization II:
>> The Impending Doom of Common Carriage
>> Eli M. Noam
>> Professor of Finance and Economics
>> Columbia University, Graduate School of Business
>> March 15, 1994
>> 
>> I. Introduction 1
>> 
>> This article argues that the institution of common carriage,
>> historically the foundation of the way telecommunications are delivered,
>> will not survive. To clarify: "common carriers" (the misnomer often used
>> to refer to telephone companies) will continue to exist, but the status
>> under which they operate -- offering service on a non-discriminatory
>> basis, neutral as to use and user -- will not.
>> 
>> ...
>> 
>> VII. A Contract-Carrier Based Telecommunications System?
>> 
>> The conclusion of the analysis has been that common carriage will erode
>> in time, and that a hybrid co-existence will not be stable. This is not
>> to say that the common carriers qua carriers will become extinct; many
>> of them will remain significant players, but they will conduct their
>> business as contract carriers. But common carriage as such will
>> disappear. This will not happen overnight, of course. Intermediate
>> arrangements can buy several decades of transition time. But the basic
>> dynamics will eventually assert themselves.
>> 
>> This conclusion is reached with much regret, because the socially
>> positive aspects of common carriage are strong, and because the absence
>> to common carriage often means gatekeeper power. But we should not let
>> preferences obscure the clarity of analysis.
>> 
>> Bob
>>> Jason just did a beautiful thread as to what was the original source
>>> of the network neutrality
>>> bittorrent vs voip bufferbloat blowup.
>>> 
>>> https://twitter.com/jlivingood/status/1707078242857849244
>>> 
>>> Seeing all the political activity tied onto it since (and now again)
>>> reminds of two families at war about an incident that had happened
>>> generations and generations before, where the two sides no longer
>>> remembered why they hated each other so, but just went on hating, and
>>> not forgiving, and not moving on.
>>> 
>>> Yes, there are entirely separate and additional NN issues, but the
>>> technical problem of providing common carriage between two very
>>> different network application types (voip/gaming vs file transfer) is
>>> thoroughly solved now, and if only all sides recognised at least this
>>> much, and made peace over it, and worked together to deploy those
>>> solutions, maybe, just maybe, we could find mutually satisfactory
>>> solutions to the other problems that plague the internet today, like
>>> security, and the ipv6 rollout.
>>> 
>>> If anyone here knows anyone more political, still vibrating with 10+
>>> years of outrage about NN on this fronts, on one side or the other, if
>>> you could sit them down, over a beer, and try to explain that at the
>>> start it was a technical problem nobody understood at the time, maybe
>>> that would help.
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Rpm] [Starlink] net neutrality back in the news

[Dave Taht]

Can we please move this discussion off and *only to* the new nnagain
mailing list?
This will be my last cross post.

I also agree strongly with nagle in that flow queuing is least
pessimal.I also do not think we have a universally agreed upon
definition of it. Mine has been:

"When a flow's arrival rate is less than all the other flows departure
rates, it goes out first, otherwise, it is more or less evenly mixed
with the other flows."

This sidesteps bytes vs packets, and is a little inaccurate, but
delightfully short. Fair queuing is just the latter part of the
phrase.

On Sun, Oct 1, 2023 at 10:08 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> I really like this argument by John Nagle. It is IMHO complementary (and stronger) to what keep arguing, that far queueing is not most optimal but least pessimal (and close to the best an intermediary node can do with the limited information at hand, anything else needs additional information). What I think is elegant about flow-scheduling is that it taps into essential information for digital communications and hence is hard to exploit (hard, not impossible).
>
>
> > On Sep 28, 2023, at 08:06, Dave Taht via Starlink <starlink@lists.bufferbloat.net> wrote:
> >
> > Dear Bob:
> >
> > I have always thought the long standing dispute over moving flow
> > queuing (FQ) into the network has always been about enabling common
> > carriage for the vast majority of possible use cases for the internet
> > or not, going all the way back to the fights over it in 1989, and many
> > discussions prior to that. I haven´t talked about it this way much,
> > preferring to enable the users first, to make their own choices in
> > spite of their service providers. I felt showing the general benefit
> > for making voip, gaming, videoconferencing, web traffic, and bulk
> > traffic, all co-exist in harmony, and trying to solve the thorny
> > problems extant in all the things that I had source code to, like
> > WiFi, was needed, first. Making them these algorithms, so commonly
> > available, and so obvious, and inevitable, seemed best, rather than
> > duking it out directly in the political arena, for which I do not have
> > the stomach or time. I also felt that in a competitive market, fq
> > would win.
> >
> > A few days ago John Nagel talked to this point, somewhat. From hackernews:
> >
> > Animats 3 days ago | root | parent | next [–]
> >
> >> If everyone played nice, the exponential backoff timers would work as expected,
> >
> > "As I wrote in 1985, in [1], under "Game Theoretic Aspects of Network
> > Congestion" and "Fairness in Packet Switching Systems", about fair
> > queuing, We would like to protect the network from hosts that are not
> > well-behaved. More specifically, we would like, in the presence of
> > both well-behaved and badly-behaved hosts, to insure that well-behaved
> > hosts receive better service than badly-behaved hosts. We have devised
> > a means of achieving this. The goal of fair queuing is not to improve
> > network performance overall. The goal of fair queuing is to reward
> > well-behaved hosts over badly-behaved hosts. If everyone is
> > well-behaved, the queue lengths are the same, usually 1 or 0, and fair
> > queuing does little.
> >
> > There is an inherent conflict between this goal and achieving maximum
> > data transfer rates. If you try for near 100% utilization, the
> > problems become much worse. You can run comfortably at maybe 70%. This
> > was an accepted tradeoff for DoD systems. DoD wants things to keep
> > working in a crisis, even if normal operation is a bit slower. This is
> > why I'm not a big fan of HTTP/3. It's a attempt to get about 10% more
> > performance in the good case, at the cost of considerable extra
> > complexity and less immunity to gaming the system.
> >
> > I never wrote about that much at the time, because if I had, people
> > would have realized earlier that traffic shaping is possible, which
> > implies that you can sell and bill for bandwidth and quality of
> > service. We might have ended up with pay per packet."
> >
> > As for whether this realization has bubbled up as far as the FCC or
> > not, or the machinations to not deploy our fixes to bufferbloat
> > without retaining some sort of control as a MITM were conscious by
> > those fighting it so strongly, I do not know.
> >
> > I do see the initial reactions to
> >
> > https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf
> >
> > seem rather canned so far, and yet reading that post over feels good...
> >
> > ... and I hope that perhaps tomorrow will be a brighter day for the
> > whole internet, than it is had in a long time.
> >
> > On Wed, Sep 27, 2023 at 8:33 PM rjmcmahon <rjmcmahon@rjmcmahon.com> wrote:
> >>
> >> Common Carriage goes way beyond our lifes. Eli Noam's write up in 1994
> >> is a good one.
> >>
> >> http://www.columbia.edu/dlc/wp/citi/citinoam11.html
> >>
> >> Beyond Liberalization II:
> >> The Impending Doom of Common Carriage
> >> Eli M. Noam
> >> Professor of Finance and Economics
> >> Columbia University, Graduate School of Business
> >> March 15, 1994
> >>
> >> I. Introduction 1
> >>
> >> This article argues that the institution of common carriage,
> >> historically the foundation of the way telecommunications are delivered,
> >> will not survive. To clarify: "common carriers" (the misnomer often used
> >> to refer to telephone companies) will continue to exist, but the status
> >> under which they operate -- offering service on a non-discriminatory
> >> basis, neutral as to use and user -- will not.
> >>
> >> ...
> >>
> >> VII. A Contract-Carrier Based Telecommunications System?
> >>
> >> The conclusion of the analysis has been that common carriage will erode
> >> in time, and that a hybrid co-existence will not be stable. This is not
> >> to say that the common carriers qua carriers will become extinct; many
> >> of them will remain significant players, but they will conduct their
> >> business as contract carriers. But common carriage as such will
> >> disappear. This will not happen overnight, of course. Intermediate
> >> arrangements can buy several decades of transition time. But the basic
> >> dynamics will eventually assert themselves.
> >>
> >> This conclusion is reached with much regret, because the socially
> >> positive aspects of common carriage are strong, and because the absence
> >> to common carriage often means gatekeeper power. But we should not let
> >> preferences obscure the clarity of analysis.
> >>
> >> Bob
> >>> Jason just did a beautiful thread as to what was the original source
> >>> of the network neutrality
> >>> bittorrent vs voip bufferbloat blowup.
> >>>
> >>> https://twitter.com/jlivingood/status/1707078242857849244
> >>>
> >>> Seeing all the political activity tied onto it since (and now again)
> >>> reminds of two families at war about an incident that had happened
> >>> generations and generations before, where the two sides no longer
> >>> remembered why they hated each other so, but just went on hating, and
> >>> not forgiving, and not moving on.
> >>>
> >>> Yes, there are entirely separate and additional NN issues, but the
> >>> technical problem of providing common carriage between two very
> >>> different network application types (voip/gaming vs file transfer) is
> >>> thoroughly solved now, and if only all sides recognised at least this
> >>> much, and made peace over it, and worked together to deploy those
> >>> solutions, maybe, just maybe, we could find mutually satisfactory
> >>> solutions to the other problems that plague the internet today, like
> >>> security, and the ipv6 rollout.
> >>>
> >>> If anyone here knows anyone more political, still vibrating with 10+
> >>> years of outrage about NN on this fronts, on one side or the other, if
> >>> you could sit them down, over a beer, and try to explain that at the
> >>> start it was a technical problem nobody understood at the time, maybe
> >>> that would help.
> >
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] net neutrality back in the news

[Sebastian Moeller]

Hi Dave,

please excuse a number of tangents below ;)


> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Jason just did a beautiful thread as to what was the original source
> of the network neutrality
> bittorrent vs voip bufferbloat blowup.
> 
> https://twitter.com/jlivingood/status/1707078242857849244

	But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.


*) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.


> 
> Seeing all the political activity tied onto it since (and now again)
> reminds of two families at war about an incident that had happened
> generations and generations before, where the two sides no longer
> remembered why they hated each other so, but just went on hating, and
> not forgiving, and not moving on.
> 
> Yes, there are entirely separate and additional NN issues, but the
> technical problem of providing common carriage between two very
> different network application types (voip/gaming vs file transfer) is
> thoroughly solved now,

	I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
	I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...


**) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.


> and if only all sides recognised at least this
> much, and made peace over it, and worked together to deploy those
> solutions, maybe, just maybe, we could find mutually satisfactory
> solutions to the other problems that plague the internet today, like
> security, and the ipv6 rollout.

	+1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry. Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****) 


***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.

****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).

> If anyone here knows anyone more political, still vibrating with 10+
> years of outrage about NN on this fronts, on one side or the other, if
> you could sit them down, over a beer, and try to explain that at the
> start it was a technical problem nobody understood at the time, maybe
> that would help.

	So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).


*****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.

Regards
	Sebastian


> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Starlink] net neutrality back in the news

[Gert Doering]

Hi,

On Thu, Sep 28, 2023 at 08:25:31AM +0200, Sebastian Moeller via Starlink wrote:
> ***) Strictly speaking IPv6 is required, since "internet access"
> is defined as reaching all of the internet (as far as in the ISPs
> power) and IPv6-only sites are not reachable for the CG-NAT-only
> customers. But so far the local regulator does not seem to enforce
> that requirement, or hopefully is working on this quietly behind
> the curtains.

I still maintain that a better google ranking for "content with IPv6"
would bring us 90%+ IPv6 availability on the content side in a very
short time (look at the craziness some of these SEOs are willing to
endure, just to get a better ranking)...

(For Germany, I do not see the lone acces ISP without IPv6 as "the most
sore spot", as most of the large ISPs and all of the mobile carriers 
include IPv6 nowadays - but content is still lacking)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

Re: [Rpm] [Starlink] net neutrality back in the news

[Sebastian Moeller]

Hi Gert,


> On Sep 28, 2023, at 08:36, Gert Doering <gert@space.net> wrote:
> 
> Hi,
> 
> On Thu, Sep 28, 2023 at 08:25:31AM +0200, Sebastian Moeller via Starlink wrote:
>> ***) Strictly speaking IPv6 is required, since "internet access"
>> is defined as reaching all of the internet (as far as in the ISPs
>> power) and IPv6-only sites are not reachable for the CG-NAT-only
>> customers. But so far the local regulator does not seem to enforce
>> that requirement, or hopefully is working on this quietly behind
>> the curtains.
> 
> I still maintain that a better google ranking for "content with IPv6"
> would bring us 90%+ IPv6 availability on the content side in a very
> short time (look at the craziness some of these SEOs are willing to
> endure, just to get a better ranking)...

	[SM] This is pretty convincing, especially given your perspective (I only view these things from my pure end-user vantage point).

> (For Germany, I do not see the lone acces ISP without IPv6 as "the most
> sore spot", as most of the large ISPs and all of the mobile carriers 
> include IPv6 nowadays - but content is still lacking)

	[SM] Has Vodafone started using IPv6 for their DSL-users yet*? About the content that is an interesting question, I will try to measure in my home network what IPv4/IPv6 traffic ration I actually see.

Regards
	Sebastian


*) According to https://de.statista.com/statistik/daten/studie/203020/umfrage/anzahl-der-dsl-kunden-von-vodafone-in-deutschland/ Vodafone Germany had around 2.5 Million DSL customers end on 2022 this is not a lot (there are roughly 40 Million access links in Germany) but it is also not "nothing".


> 
> Gert Doering
>        -- NetMaster
> -- 
> have you enabled IPv6 on something today...?
> 
> SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
> Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
> D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
> Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

Re: [Rpm] [Starlink] net neutrality back in the news

[Gert Doering]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

Hi,

On Thu, Sep 28, 2023 at 09:14:27AM +0200, Sebastian Moeller wrote:
> 	[SM] Has Vodafone started using IPv6 for their DSL-users yet*? About the content that is an interesting question, I will try to measure in my home network what IPv4/IPv6 traffic ration I actually see.

Many high-volume sides do offer IPv6, since years (all google infra,
youtube, facebook, etc.) - so depending on the type of end-users, you
can see well over 50% IPv6 traffic.  Think "student dormitories" and
"youtube" ;-)

But this really depends on the nature of your traffic, as many "news
outlet" types are still missing (stern.de, spiegel.de, bild.de, ...)
(and those would be prone to SEO coaxing ;-) ).

What irks me quite a bit is that there's still many content offerings
that use CDNs (Akamai, Cloudflare, etc.) who can do IPv6 perfectly fine,
you just have to check the "[X] yes, IPv6!" checkbox... so we have,
www.ard.de who are behind Akamai, but have no v6...


> [Vodafone DSL]
> *) According to https://de.statista.com/statistik/daten/studie/203020/umfrage/anzahl-der-dsl-kunden-von-vodafone-in-deutschland/ Vodafone Germany had around 2.5 Million DSL customers end on 2022 this is not a lot (there are roughly 40 Million access links in Germany) but it is also not "nothing".

I wasn't aware that Vodafone DSL is lacking in that regards - as they have
IPv6 on their cable and mobile offerings (arguably "different networks, 
before VF bought them and put the VF label on top").  But you're right,
there is work to do.  Thanks for the insight.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Rpm] [Starlink] net neutrality back in the news

[Sebastian Moeller]

Hi Gert,


> On Sep 28, 2023, at 09:33, Gert Doering <gert@space.net> wrote:
> 
> Hi,
> 
> On Thu, Sep 28, 2023 at 09:14:27AM +0200, Sebastian Moeller wrote:
>> 	[SM] Has Vodafone started using IPv6 for their DSL-users yet*? About the content that is an interesting question, I will try to measure in my home network what IPv4/IPv6 traffic ration I actually see.
> 
> Many high-volume sides do offer IPv6, since years (all google infra,
> youtube, facebook, etc.) - so depending on the type of end-users, you
> can see well over 50% IPv6 traffic.  Think "student dormitories" and
> "youtube" ;-)

	[SM] I have some youtube-affine kifs^W erm, "users" on my network that might push the ratio towards IPv6  then ;)


> But this really depends on the nature of your traffic, as many "news
> outlet" types are still missing (stern.de, spiegel.de, bild.de, ...)
> (and those would be prone to SEO coaxing ;-) ).

	[SM] True!


> What irks me quite a bit is that there's still many content offerings
> that use CDNs (Akamai, Cloudflare, etc.) who can do IPv6 perfectly fine,
> you just have to check the "[X] yes, IPv6!" checkbox... so we have,
> www.ard.de who are behind Akamai, but have no v6...

	[SM] I also seem to recall that the playstation and xbox networks operate mostly IPv4, valve probably as well.

> 
>> [Vodafone DSL]
>> *) According to https://de.statista.com/statistik/daten/studie/203020/umfrage/anzahl-der-dsl-kunden-von-vodafone-in-deutschland/ Vodafone Germany had around 2.5 Million DSL customers end on 2022 this is not a lot (there are roughly 40 Million access links in Germany) but it is also not "nothing".
> 
> I wasn't aware that Vodafone DSL is lacking in that regards - as they have
> IPv6 on their cable and mobile offerings (arguably "different networks, 
> before VF bought them and put the VF label on top").  But you're right,
> there is work to do.  Thanks for the insight.

	[SM] I am sure Vodafone DSL used to have that issue, but honestly I do not know about the current state (so maybe I should have reasearched that first before posting), I would hope for their users that they activated IPv6.

Besten Gruss
	Sebastian



> 
> Gert Doering
>        -- NetMaster
> -- 
> have you enabled IPv6 on something today...?
> 
> SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
> Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
> D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
> Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

Re: [Rpm] net neutrality back in the news

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 9345 bytes --]

On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> please excuse a number of tangents below ;)

It would be nice, if as a (dis)organisation... the bufferbloat team
could focus on somehow getting both sides of the network neutrality
debate deeplying understanding the technological problem their
pre-conceptions face, and the (now readily available and inexpensive)
solutions that could be deployed, by most ISPs, over a weekend. We are
regularly bringing up a few thousand people a week on libreqos (that
we know of), and then of course, there are all the home routers and
CPE that are increasingly capable of doing the right thing.

The time to try and shift the memes in in the coming days, and weeks.

So ya'all being distracting below... aggh... ok, I'll bite.

>
>
> > On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >
> > Jason just did a beautiful thread as to what was the original source
> > of the network neutrality
> > bittorrent vs voip bufferbloat blowup.
> >
> > https://twitter.com/jlivingood/status/1707078242857849244
>
>         But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.

The average use at peak hours for home broadband is below 5mbits per
subscriber regardless of on a 25mbit or gbit plan. Remarkably,
business plans are less (but more bursty). Oversubscription within a
set of parameters is needed because, in part because upstream
bandwidth to the internet remains expensive (although I hope that cost
continues to drop rapidly), and in part, because it is needlessly
expensive to provision exactly the contracted rate to the customer. As
one example, I use the inverse square law as a rule of thumb for
wireless deployments - the range you can get from a 25/10 deployment
is geometrically better than 100/25, and the average bandwidth usage
nearly the same.

Agreeing on industry standards for what the "slack" should be, might be of help.

>
>
> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.

We have discussed elsewhere, the idea of a minimum contracted rate
(down to), which is easier to reason about.

>
> >
> > Seeing all the political activity tied onto it since (and now again)
> > reminds of two families at war about an incident that had happened
> > generations and generations before, where the two sides no longer
> > remembered why they hated each other so, but just went on hating, and
> > not forgiving, and not moving on.
> >
> > Yes, there are entirely separate and additional NN issues, but the
> > technical problem of providing common carriage between two very
> > different network application types (voip/gaming vs file transfer) is
> > thoroughly solved now,
>
>         I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

Torrent problem thoroughly solved with FQ on the path and backpressure
from the mac scheduler.

https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

>         I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>
>
> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>
>
> > and if only all sides recognised at least this
> > much, and made peace over it, and worked together to deploy those
> > solutions, maybe, just maybe, we could find mutually satisfactory
> > solutions to the other problems that plague the internet today, like
> > security, and the ipv6 rollout.
>
>         +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.

I really would like to see a push for IPv6 mandated as a part of the
BEAD programs.

> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)

There is a separate NRPM going on for the new cybersecurity label at
the FCC. If I had time, and a partner,
we could rework the doc we did a few years ago. It is due oct 6.

>
>
> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>
> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>
> > If anyone here knows anyone more political, still vibrating with 10+
> > years of outrage about NN on this fronts, on one side or the other, if
> > you could sit them down, over a beer, and try to explain that at the
> > start it was a technical problem nobody understood at the time, maybe
> > that would help.
>
>         So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).

Governments make safer markets feasible.

>
> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>
> Regards
>         Sebastian
>
>
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[-- Attachment #2: 2510.jpeg --]
[-- Type: image/jpeg, Size: 66803 bytes --]

Re: [Rpm] [Starlink] net neutrality back in the news

[Livingood, Jason]

On 9/28/23, 12:45, "Starlink on behalf of Dave Taht via Starlink" <starlink-bounces@lists.bufferbloat.net <mailto:starlink-bounces@lists.bufferbloat.net> on behalf of starlink@lists.bufferbloat.net <mailto:starlink@lists.bufferbloat.net>> wrote:
> It would be nice, if as a (dis)organisation... the bufferbloat team
could focus on somehow getting both sides of the network neutrality
debate deeplying understanding the technological problem their
pre-conceptions face, and the (now readily available and inexpensive)
solutions that could be deployed, by most ISPs, over a weekend. We are
regularly bringing up a few thousand people a week on libreqos (that
we know of), and then of course, there are all the home routers and
CPE that are increasingly capable of doing the right thing.

[JL] The FCC will soon (maybe today) open a notice of proposed rulemaking - aka NPRM. That process provides an opportunity for anyone to file and filings from technical experts are always highly valued. 

Re: [Rpm] [Starlink] net neutrality back in the news

[rjmcmahon]

Here's is the point for TLDR by Noam. Neutral traffic acceptance is not 
no priorities. We want traffic priorities despite all the b.s. that 
they're unfair.

"All of common carriages free-flow, goals of low transaction cost, and 
no-liability goals are thus preserved by a system of (a) non-exclusive 
interconnection (b) neutral traffic acceptance."

Back to TLDR per Noam. This is the pertinent part. First, few in the 
U.S. want the IAPs to be common carriers. It would really bad.

The following factors are important in determining common carriage:
...
law and regulations define the responsibilities of the parties.

For contract carriers, on the other hand:
...
contracts define parties' responsibilities.

And then, the issue isn't so much about CPE side but peering or 
interconnection of networks.

Interconnectivity is critical to the future network system. Yet 
interconnectivity does not happen by itself; that is the lesson of 
decades of American experience. Open network architecture, comparably 
efficient interconnection, and collocation are part of this evolution.

Such interconnection arrangements do not depend on common carriage, 
though they are inspired by it. Therefore, its is possible,

Then Noam's suggestions on how to go forward to protect common carriage 
principals with contract carriage operators through "neutral" 
interconnections. Notice there is no mandate of equal traffic priority 
only neutral access to the network. Priorities can be negotiated per 
business contracts e.g. peering agreements.

VIII. What for the Future?

...

This suggests that new policy instruments will have to be found to deal 
with the negatives effect on information diversity and flow.

A way to do so is by replacing the principle of common carriage by a new 
principle of neutral interconnection. A carrier can elect to be private 
by running its own self-contained infrastructure, and having full 
control over its content, use and access. But if it interconnects into 
other networks and accepts transmission traffic from them, it cannot 
pick some bits over other bits. This means that while a private carrier 
can be selective in its direct customers, whether they are end-users or 
content providers, it cannot be selective in what it accepts from 
another interconnected carrier.

Among interconnected carriers, no carrier can transmit selectively 
traffic passed on to it by another carrier, based on content, uses, or 
usage, or refuse interconnection on these grounds. Any carrier offering 
interconnection to some carriers must offer it to other carriers, too, 
within technical constraints.

This does not require interconnection on equal terms, as in the case of 
common carriage. But it establishes the possibility of arbitrage if 
differentiated pricing occurs. All of common carriages free-flow, goals 
of low transaction cost, and no-liability goals are thus preserved by a 
system of (a) non-exclusive interconnection (b) neutral traffic 
acceptance.

Bob
> On 9/28/23, 12:45, "Starlink on behalf of Dave Taht via Starlink"
> <starlink-bounces@lists.bufferbloat.net
> <mailto:starlink-bounces@lists.bufferbloat.net> on behalf of
> starlink@lists.bufferbloat.net
> <mailto:starlink@lists.bufferbloat.net>> wrote:
>> It would be nice, if as a (dis)organisation... the bufferbloat team
> could focus on somehow getting both sides of the network neutrality
> debate deeplying understanding the technological problem their
> pre-conceptions face, and the (now readily available and inexpensive)
> solutions that could be deployed, by most ISPs, over a weekend. We are
> regularly bringing up a few thousand people a week on libreqos (that
> we know of), and then of course, there are all the home routers and
> CPE that are increasingly capable of doing the right thing.
> 
> [JL] The FCC will soon (maybe today) open a notice of proposed
> rulemaking - aka NPRM. That process provides an opportunity for anyone
> to file and filings from technical experts are always highly valued.
> 
> 
> 
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] net neutrality back in the news

[Sebastian Moeller]

> On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>> please excuse a number of tangents below ;)
> 
> It would be nice, if as a (dis)organisation... the bufferbloat team
> could focus on somehow getting both sides of the network neutrality
> debate deeplying understanding the technological problem their
> pre-conceptions face, and the (now readily available and inexpensive)
> solutions that could be deployed, by most ISPs, over a weekend. We are
> regularly bringing up a few thousand people a week on libreqos (that
> we know of), and then of course, there are all the home routers and
> CPE that are increasingly capable of doing the right thing.
> 
> The time to try and shift the memes in in the coming days, and weeks.
> 
> So ya'all being distracting below... aggh... ok, I'll bite.

	[SM] Sorry to drag you into the weeds...


> 
>> 
>> 
>>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>>> 
>>> Jason just did a beautiful thread as to what was the original source
>>> of the network neutrality
>>> bittorrent vs voip bufferbloat blowup.
>>> 
>>> https://twitter.com/jlivingood/status/1707078242857849244
>> 
>>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> 
> The average use at peak hours for home broadband is below 5mbits per
> subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> business plans are less (but more bursty). Oversubscription within a
> set of parameters is needed because, in part because upstream
> bandwidth to the internet remains expensive (although I hope that cost
> continues to drop rapidly), and in part, because it is needlessly
> expensive to provision exactly the contracted rate to the customer. As
> one example, I use the inverse square law as a rule of thumb for
> wireless deployments - the range you can get from a 25/10 deployment
> is geometrically better than 100/25, and the average bandwidth usage
> nearly the same.

	[SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).


> Agreeing on industry standards for what the "slack" should be, might be of help.

	[SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.


> 
>> 
>> 
>> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> 
> We have discussed elsewhere, the idea of a minimum contracted rate
> (down to), which is easier to reason about.

	[SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
	But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).


*) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....


> 
>> 
>>> 
>>> Seeing all the political activity tied onto it since (and now again)
>>> reminds of two families at war about an incident that had happened
>>> generations and generations before, where the two sides no longer
>>> remembered why they hated each other so, but just went on hating, and
>>> not forgiving, and not moving on.
>>> 
>>> Yes, there are entirely separate and additional NN issues, but the
>>> technical problem of providing common carriage between two very
>>> different network application types (voip/gaming vs file transfer) is
>>> thoroughly solved now,
>> 
>>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

	[SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:

"	• Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20

	• Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.


	• Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21

	• Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."

These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).



> 
> Torrent problem thoroughly solved with FQ on the path and backpressure
> from the mac scheduler.
> 
> https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf

	[SM] Thanks for the link. This is mainly for the self-congesrion case?


> 
>>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>> 
>> 
>> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>> 
>> 
>>> and if only all sides recognised at least this
>>> much, and made peace over it, and worked together to deploy those
>>> solutions, maybe, just maybe, we could find mutually satisfactory
>>> solutions to the other problems that plague the internet today, like
>>> security, and the ipv6 rollout.
>> 
>>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> 
> I really would like to see a push for IPv6 mandated as a part of the
> BEAD programs.

	[SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.

> 
>> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> 
> There is a separate NRPM going on for the new cybersecurity label at
> the FCC. If I had time, and a partner,
> we could rework the doc we did a few years ago. It is due oct 6.
> 
>> 
>> 
>> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>> 
>> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>> 
>>> If anyone here knows anyone more political, still vibrating with 10+
>>> years of outrage about NN on this fronts, on one side or the other, if
>>> you could sit them down, over a beer, and try to explain that at the
>>> start it was a technical problem nobody understood at the time, maybe
>>> that would help.
>> 
>>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> 
> Governments make safer markets feasible.

	[SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.

Regards & Sorry for the tangent
	Sebastian

> 
>> 
>> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>> 
>> Regards
>>        Sebastian
>> 
>> 
>>> 
>>> --
>>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>> _______________________________________________
>>> Rpm mailing list
>>> Rpm@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/rpm
>> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> <2510.jpeg>

Re: [Rpm] net neutrality back in the news

[Dave Taht]

@Sebastian: This is a really great list of what the issues were in the
EU, and if y'all can repost there, rather than here, perhaps more
light will be generated outside our circles.

https://news.ycombinator.com/item?id=37694306#37694307

On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
>
> > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi Dave,
> >>
> >> please excuse a number of tangents below ;)
> >
> > It would be nice, if as a (dis)organisation... the bufferbloat team
> > could focus on somehow getting both sides of the network neutrality
> > debate deeplying understanding the technological problem their
> > pre-conceptions face, and the (now readily available and inexpensive)
> > solutions that could be deployed, by most ISPs, over a weekend. We are
> > regularly bringing up a few thousand people a week on libreqos (that
> > we know of), and then of course, there are all the home routers and
> > CPE that are increasingly capable of doing the right thing.
> >
> > The time to try and shift the memes in in the coming days, and weeks.
> >
> > So ya'all being distracting below... aggh... ok, I'll bite.
>
>         [SM] Sorry to drag you into the weeds...
>
>
> >
> >>
> >>
> >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
> >>>
> >>> Jason just did a beautiful thread as to what was the original source
> >>> of the network neutrality
> >>> bittorrent vs voip bufferbloat blowup.
> >>>
> >>> https://twitter.com/jlivingood/status/1707078242857849244
> >>
> >>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
> >
> > The average use at peak hours for home broadband is below 5mbits per
> > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > business plans are less (but more bursty). Oversubscription within a
> > set of parameters is needed because, in part because upstream
> > bandwidth to the internet remains expensive (although I hope that cost
> > continues to drop rapidly), and in part, because it is needlessly
> > expensive to provision exactly the contracted rate to the customer. As
> > one example, I use the inverse square law as a rule of thumb for
> > wireless deployments - the range you can get from a 25/10 deployment
> > is geometrically better than 100/25, and the average bandwidth usage
> > nearly the same.
>
>         [SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).
>
>
> > Agreeing on industry standards for what the "slack" should be, might be of help.
>
>         [SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.
>
>
> >
> >>
> >>
> >> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
> >
> > We have discussed elsewhere, the idea of a minimum contracted rate
> > (down to), which is easier to reason about.
>
>         [SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
>         But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).
>
>
> *) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....
>
>
> >
> >>
> >>>
> >>> Seeing all the political activity tied onto it since (and now again)
> >>> reminds of two families at war about an incident that had happened
> >>> generations and generations before, where the two sides no longer
> >>> remembered why they hated each other so, but just went on hating, and
> >>> not forgiving, and not moving on.
> >>>
> >>> Yes, there are entirely separate and additional NN issues, but the
> >>> technical problem of providing common carriage between two very
> >>> different network application types (voip/gaming vs file transfer) is
> >>> thoroughly solved now,
> >>
> >>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
>
>         [SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:
>
> "       • Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20
>
>         • Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.
>
>
>         • Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21
>
>         • Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."
>
> These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
> I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).
>
>
>
> >
> > Torrent problem thoroughly solved with FQ on the path and backpressure
> > from the mac scheduler.
> >
> > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
>
>         [SM] Thanks for the link. This is mainly for the self-congesrion case?
>
>
> >
> >>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
> >>
> >>
> >> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
> >>
> >>
> >>> and if only all sides recognised at least this
> >>> much, and made peace over it, and worked together to deploy those
> >>> solutions, maybe, just maybe, we could find mutually satisfactory
> >>> solutions to the other problems that plague the internet today, like
> >>> security, and the ipv6 rollout.
> >>
> >>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
> >
> > I really would like to see a push for IPv6 mandated as a part of the
> > BEAD programs.
>
>         [SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.
>
> >
> >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
> >
> > There is a separate NRPM going on for the new cybersecurity label at
> > the FCC. If I had time, and a partner,
> > we could rework the doc we did a few years ago. It is due oct 6.
> >
> >>
> >>
> >> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
> >>
> >> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
> >>
> >>> If anyone here knows anyone more political, still vibrating with 10+
> >>> years of outrage about NN on this fronts, on one side or the other, if
> >>> you could sit them down, over a beer, and try to explain that at the
> >>> start it was a technical problem nobody understood at the time, maybe
> >>> that would help.
> >>
> >>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
> >
> > Governments make safer markets feasible.
>
>         [SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.
>
> Regards & Sorry for the tangent
>         Sebastian
>
> >
> >>
> >> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
> >>
> >> Regards
> >>        Sebastian
> >>
> >>
> >>>
> >>> --
> >>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> >>> Dave Täht CSO, LibreQos
> >>> _______________________________________________
> >>> Rpm mailing list
> >>> Rpm@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/rpm
> >>
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
> > <2510.jpeg>
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [LibreQoS] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 19919 bytes --]

"(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most'
in terms of customers being served by puts the Spectrums and Comcasts in
the mix (in the US) and they don't care about happy customers they care
about defacto monopolies in markets so that they don't have to care about
happy customers.  That might not be their motivation, but 30 years of their
behavior makes it clear that it's the case.


From my perspective, there should be a clear separation between carriers
and last mile delivery..  Even if it's the same company, the rules defining
each really should be different.

Common Carriers or rather, carrier class services for 'internet', should be
completely neutral.  Packets are packets.  However, I think it's important
to carve out methods to have dedicated links for real time flows at the
carrier level.  I don't know what that model looks like exactly, but being
too stubborn about purist NN principals could really hurt VoIP services if
there aren't methods to handle that.  I guess I really am describing
'internet fast lanes' for certain classes of services that we deem
important enough as a whole.  not individual ISPs deciding, but rather 'the
will of the people' saying VoIP is more important than netflix, you can
carve out dedicated capacity for that.

For the last mile, I'm actually less concerned with pure NN and more
concerned with no-blocking or 'brand' prioritization and required/label
transparency.  ie, if an ISP wants to sell a DPI's QoE'd service that says
'videos at 1080p' and that is on the product label clearly and not in small
print, then that's fine so long as 'videos' is agnostic of the vendor
INCLUDING the ISP's own video products.  "100Mbps internet services with
1080P video $65" for example.  "Upgrade to our 100M with 4k video for
+$10/m".  With that level of transparency I think that consumer protections
are well handled.  And, because of that transparency, normal market
capitalism mechanisms work.  I can say "100Mbps for $65 and full resolution
video" for example.   Or "100Mbps Net Neutral service".   The secret DPI's
QoE shaping is my main concern here, and where I think consumer protection
needs to be pursued.

Again however, I think that ISPs should be able to offer dedicated paths or
bandwidth for specific services like VoIP.  Services listed in a publicly
determined products list.


On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> @Sebastian: This is a really great list of what the issues were in the
> EU, and if y'all can repost there, rather than here, perhaps more
> light will be generated outside our circles.
>
> https://news.ycombinator.com/item?id=37694306#37694307
>
> On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> >
> >
> >
> > > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de>
> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> please excuse a number of tangents below ;)
> > >
> > > It would be nice, if as a (dis)organisation... the bufferbloat team
> > > could focus on somehow getting both sides of the network neutrality
> > > debate deeplying understanding the technological problem their
> > > pre-conceptions face, and the (now readily available and inexpensive)
> > > solutions that could be deployed, by most ISPs, over a weekend. We are
> > > regularly bringing up a few thousand people a week on libreqos (that
> > > we know of), and then of course, there are all the home routers and
> > > CPE that are increasingly capable of doing the right thing.
> > >
> > > The time to try and shift the memes in in the coming days, and weeks.
> > >
> > > So ya'all being distracting below... aggh... ok, I'll bite.
> >
> >         [SM] Sorry to drag you into the weeds...
> >
> >
> > >
> > >>
> > >>
> > >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> > >>>
> > >>> Jason just did a beautiful thread as to what was the original source
> > >>> of the network neutrality
> > >>> bittorrent vs voip bufferbloat blowup.
> > >>>
> > >>> https://twitter.com/jlivingood/status/1707078242857849244
> > >>
> > >>        But the core issue IMHO really was an economic one, the
> over-subscription ratios that worked before torrenting simply did not cut
> it any more in an environment when customers actually tried to use their
> contracted access rates "quantitatively". In my outsider perspective an ISP
> owes its customers the contracted rates (with some slack*) and any sort of
> over-subscription is a valid economic optimization an ISP might take, IFF
> that ISP is prepared to rapidly increase segment capacity (or down-grade
> customer plans)) if the deployed over-subscription rate proves to have been
> too optimistic. Mind you, most ISPs market plans by access speed (and
> charge more for higher speeds) and hence are somewhat responsible to
> actually deliver (again with some slack) these speeds.
> > >
> > > The average use at peak hours for home broadband is below 5mbits per
> > > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
> > > business plans are less (but more bursty). Oversubscription within a
> > > set of parameters is needed because, in part because upstream
> > > bandwidth to the internet remains expensive (although I hope that cost
> > > continues to drop rapidly), and in part, because it is needlessly
> > > expensive to provision exactly the contracted rate to the customer. As
> > > one example, I use the inverse square law as a rule of thumb for
> > > wireless deployments - the range you can get from a 25/10 deployment
> > > is geometrically better than 100/25, and the average bandwidth usage
> > > nearly the same.
> >
> >         [SM] +1; hence my argument is not oversubscription per se is
> bad, but that oversubscription needs to be managed and the level of
> oversubscription needs to to be adapted ot he actual usage patterns. I am
> sure however that ISPs already actively do that (I assume most ISPs want
> happy customers).
> >
> >
> > > Agreeing on industry standards for what the "slack" should be, might
> be of help.
> >
> >         [SM] Not being part of that industry at all I would love to see
> that as well, but can not contribute to defining that in any way.
> >
> >
> > >
> > >>
> > >>
> > >> *) Claiming "Up to", only carries that far, if you sell me X and I
> mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is
> acceptable unless occasionally is "at every late afternoon and evening"
> prime-time.
> > >
> > > We have discussed elsewhere, the idea of a minimum contracted rate
> > > (down to), which is easier to reason about.
> >
> >         [SM] Indeed, the German regulator (and I am not saying this is
> the only or best option) decided to require ISPs to give 6 numbers
> pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate,
> a minimal (net) rate, and a usually achievable (net) rate, all rates were
> defined as IP/TCP goodput to make verification easier on end-users. The
> regulator also defined a measurement regime that end-users can follow to
> check whether the ISP is fulfilling the contract and the law gives remedies
> if ISPs do not deliver (either the right to immediately cancel the contract
> and/or the right to adapt the monthly payments to the actually delivered
> ratio of the contracted rates*). I think I need to add, that ISPs can set
> these numbers freely, but are only allowed to advertise with the maximum
> rate.
> >         But if we talk about a single number per direction, minimal rate
> is probably easiest, or something like a "usually achievable rate" (that
> needs to be met or exceeded in say 90% of >= 20 measurements or so).
> >
> >
> > *) In a ironic twist however the regulator so far has not explained how
> deviations in each of the 6 numbers above should be aggregated to get one
> single contract rate achievement ratio..., most ISPs took measures into
> their own hands and simply offer customers typically a permanent rebate of
> 5 EUR or immediate cancelation what ever the customer prefers....
> >
> >
> > >
> > >>
> > >>>
> > >>> Seeing all the political activity tied onto it since (and now again)
> > >>> reminds of two families at war about an incident that had happened
> > >>> generations and generations before, where the two sides no longer
> > >>> remembered why they hated each other so, but just went on hating, and
> > >>> not forgiving, and not moving on.
> > >>>
> > >>> Yes, there are entirely separate and additional NN issues, but the
> > >>> technical problem of providing common carriage between two very
> > >>> different network application types (voip/gaming vs file transfer) is
> > >>> thoroughly solved now,
> > >>
> > >>        I am not sure this was at the core of the problem, my take is
> really that "always-on" and relative upload-heavy torrent simply
> demonstrated painfully for all involved that the old oversubscription
> ratios were not suitable for the changed traffic profiles. I have some
> sympathy for the ISPs here, they certainly did not try to sell their
> customers short (good ISPs try to retain their customers and that works
> best when customers are happy with the service) and having this problem
> appear on many segments at the same time is not a fun place to be, and
> upload was/is often (too) low in DOCSIS segments anyway; but this is why
> e.g. my bit torrent could affect your VoIP, simply by pushing the whole
> segment into upload capacity congestion... (ISPs in theory could fix this
> by plain old QoS engineering, but the issue at hand was with a non-ISP
> VoIP/SIP service and there QoS becomes tricky if the ISP as these packets
> need to be identified in a way that is not game-able**)
> >
> >         [SM] See my later mail to Jason, I will not claim I know
> Comcast's issues better than him and will accept that self-congestion also
> played a major role in the initial problem. Over here in Europe the net
> neutrality debate was kicked of less by an unfortunate confluence of new
> usage profiles and traditional oversibscription ratios and less than ideal
> packet scheduling, but rather by a series of pretty apparent attempts at
> restricting consumer choice, see eg.
> https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf
> for an admitted slightly biased position:
> >
> > "       • Blocking of applications and services: In order to maximise
> profits, some ISPs – that also offer their own services and applications
> online – exclude certain services and applications of competing market
> players. The most prominent case of this form of network discrimination is
> European mobile providers (like Deutsche Telekom) blocking or restricting
> the use of Voice over IP (VoIP) services (like Skype and Viber) for their
> customers.20
> >
> >         • Slowing or “throttling” internet speeds: Some ISPs slow down
> specific services (like YouTube) and applications (like Skype), or ask
> users to pay an extra fee to have access to these internet platforms. Given
> the high latency (delay) sensitivity of many applications, ISPs are able to
> compromise the correct functioning of these services by slowing them down,
> preventing the services from running properly. Often ISPs – especially
> telecommunication companies – do this to favour their own voice calling
> services over VoIP services, thereby crushing competition.
> >
> >
> >         • Blocking websites: ISPs often block websites for a number of
> reasons – to secure their network, or to avoid competition, and sometimes
> for social, public relations or political reasons. In the UK, for instance,
> Orange Telecom blocked the French digital rights advocacy group, La
> Quadrature du Net’s website on pre-paid mobile accounts.21
> >
> >         • Preferential treatment of services and platforms: ISPs can
> also impose data caps on internet access contracts while granting data
> allowance exceptions to a company’s own proprietary streaming services
> (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also
> grant preferential treatment to select services – such as Orange France
> with the popular music streaming service Deezer23 – ahead of other
> competitors, effectively imposing anti-competitive limitations on markets
> such as those for legal online music. Moreover, generally only large,
> well-established companies can afford this preferential treatment,
> resulting in a further stifling of innovation."
> >
> > These look like clearer attempts to monetize the ISP position as
> gate-keeper to his customer's eye-balls (for content providers) as well as
> gate-keeper to the internet for for paying customers.
> > I find it much harder to have sympathy for the listed examples of ISP
> behavior than the situation of rapidly changing usage pattern posed where
> technical changes needed to be made, but where no attempt at unfair
> monetization was evident, but maybe I am overly sensitive. These are all
> examples that make me personally applaud the EU for its intervention to
> make clear rules what "internet access providers" can and can not do. (The
> EU also was quite flexible in applying/interpreting its rules during the
> covid isolation periods, when it made it clear that e.g. treating certain
> traffic classes e.g. streaming media to lower priority than video
> conferences was within the neutrality framework as long as it was based on
> traffic class and not on specific service providers IIRC).
> >
> >
> >
> > >
> > > Torrent problem thoroughly solved with FQ on the path and backpressure
> > > from the mac scheduler.
> > >
> > > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
> >
> >         [SM] Thanks for the link. This is mainly for the self-congesrion
> case?
> >
> >
> > >
> > >>        I agree that on a single link we mostly solved the problem (a
> few corner cases are left on links with very low capacity, where
> essentially we can only manage the pain, not remove it)...
> > >>
> > >>
> > >> **) Which is not rocket science either, a VoIP stream takes ~100
> Kbps, so in theory an ISP might simply allow each customer say 5 VoIP
> stream equivalents by allowing up to 500Kbps od traffic marked with a
> specific DSCP as higher priority (with higher access probability for the
> shared medium). I am not sayng this is my preferred solution, just saying
> this is a solution that would have been available at the time if I memorize
> my docsis capabilities correctly.
> > >>
> > >>
> > >>> and if only all sides recognised at least this
> > >>> much, and made peace over it, and worked together to deploy those
> > >>> solutions, maybe, just maybe, we could find mutually satisfactory
> > >>> solutions to the other problems that plague the internet today, like
> > >>> security, and the ipv6 rollout.
> > >>
> > >>        +1. IPv6 is IMHO a prime example where the regulators should
> stop talking softly and start showing the big stick they carry.
> > >
> > > I really would like to see a push for IPv6 mandated as a part of the
> > > BEAD programs.
> >
> >         [SM] +1; I am not the biggest IPv6 fan, but that's what we have
> and it is mostly serviceable so let's get this "party started" finally.
> >
> > >
> > >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4
> addresses only.. (most mass market ISPs do much better in that regard, but
> for the stragglers it would help if the regulator would demand IPv6 with
> PD***). Regarding security, the easiest way to achieve that would be to put
> some heavy requirements on IoT manufacturers and vendors (like do what you
> please as long as you are local LAN only, but once you reach out into the
> cloud you need to fulfill the following list of requirements, with timely
> security updates over a reasonable long usage period), however that might
> not be very attractive for politicians/regulators to tackle (active
> regulatory acts tends to get bad press unless something bad happened, but
> even then the press often complains about the acts coming too late, but I
> digress****)
> > >
> > > There is a separate NRPM going on for the new cybersecurity label at
> > > the FCC. If I had time, and a partner,
> > > we could rework the doc we did a few years ago. It is due oct 6.
> > >
> > >>
> > >>
> > >> ***) Strictly speaking IPv6 is required, since "internet access" is
> defined as reaching all of the internet (as far as in the ISPs power) and
> IPv6-only sites are not reachable for the CG-NAT-only customers. But so far
> the local regulator does not seem to enforce that requirement, or hopefully
> is working on this quietly behind the curtains.
> > >>
> > >> ****) This is not to diss the press, they are doing what they are
> supposed to do, but it just shows that active regulation is a tricky
> business, and a light touch typically "looks better" (even though I see no
> real evidence it actually works better).
> > >>
> > >>> If anyone here knows anyone more political, still vibrating with 10+
> > >>> years of outrage about NN on this fronts, on one side or the other,
> if
> > >>> you could sit them down, over a beer, and try to explain that at the
> > >>> start it was a technical problem nobody understood at the time, maybe
> > >>> that would help.
> > >>
> > >>        So in the EU that debate is essentially settled, there is a EU
> regulation that essentially spills out what ISPs owe their customers and
> that has become the law of the land. The rationale for required un-biased
> service and freedom to select terminal devices is well justified by the
> market ideals of the EU, allowing ISPs to discriminate packets or terminal
> devices restricts the market and will lead to undesired outcomes. (Fun fact
> most big players in capitalist societies argue for "free markets" but at
> the same time act to work-around the market mechanism by trying to move the
> market into an oligo- or even monopoly condition, which is why strong
> regulation is required*****).
> > >
> > > Governments make safer markets feasible.
> >
> >         [SM] Yes, I agree, both in markets are a pretty decent tool, but
> need constant maintenance.
> >
> > Regards & Sorry for the tangent
> >         Sebastian
> >
> > >
> > >>
> > >> *****) This is akin to professional sports where the audience
> generally accepts that referees are necessary and occasionally need to make
> "painful" calls, as the alternative would be anarchy, but I digress.
> > >>
> > >> Regards
> > >>        Sebastian
> > >>
> > >>
> > >>>
> > >>> --
> > >>> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > >>> Dave Täht CSO, LibreQos
> > >>> _______________________________________________
> > >>> Rpm mailing list
> > >>> Rpm@lists.bufferbloat.net
> > >>> https://lists.bufferbloat.net/listinfo/rpm
> > >>
> > >
> > >
> > > --
> > > Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > > Dave Täht CSO, LibreQos
> > > <2510.jpeg>
> >
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>

[-- Attachment #2: Type: text/html, Size: 23355 bytes --]

Re: [Rpm] [LibreQoS] net neutrality back in the news

[Dave Taht]

I love that there are oh, 700+ people on these mailing lists, but we
have zero visibility due to google not indexing them, where hackernews
does. This is going to be an issue dominating the web (again, sadly)
for a few weeks at least, and it would really help to be doing it
there, rather than here:

https://news.ycombinator.com/item?id=37694306#37694307

As for me, up until tuesday I was trying to finish some technical
stuff, and I have a proposal due sept 30 that I have not started yet.
I regret now, even mentioning it on these lists. Please go forth and
find a forum with new folk to debate with?

I do look forward to whatever is supposed to be announced today(?),
and reading it thoroughly, and I hope that instead of starting with
decade old positions, that you assemble your thoughts, as to whatever
the FCC is actually proposing moving forward. Ideally, in order to
participate in the legal process, we would need to file a NPRM, which
I am willing to co-ordinate.

On Thu, Sep 28, 2023 at 1:09 PM dan <dandenson@gmail.com> wrote:
>
> "(I assume most ISPs want happy customers)."
> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers.  That might not be their motivation, but 30 years of their behavior makes it clear that it's the case.
>
>
> From my perspective, there should be a clear separation between carriers and last mile delivery..  Even if it's the same company, the rules defining each really should be different.
>
> Common Carriers or rather, carrier class services for 'internet', should be completely neutral.  Packets are packets.  However, I think it's important to carve out methods to have dedicated links for real time flows at the carrier level.  I don't know what that model looks like exactly, but being too stubborn about purist NN principals could really hurt VoIP services if there aren't methods to handle that.  I guess I really am describing 'internet fast lanes' for certain classes of services that we deem important enough as a whole.  not individual ISPs deciding, but rather 'the will of the people' saying VoIP is more important than netflix, you can carve out dedicated capacity for that.
>
> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency.  ie, if an ISP wants to sell a DPI's QoE'd service that says 'videos at 1080p' and that is on the product label clearly and not in small print, then that's fine so long as 'videos' is agnostic of the vendor INCLUDING the ISP's own video products.  "100Mbps internet services with 1080P video $65" for example.  "Upgrade to our 100M with 4k video for +$10/m".  With that level of transparency I think that consumer protections are well handled.  And, because of that transparency, normal market capitalism mechanisms work.  I can say "100Mbps for $65 and full resolution video" for example.   Or "100Mbps Net Neutral service".   The secret DPI's QoE shaping is my main concern here, and where I think consumer protection needs to be pursued.
>
> Again however, I think that ISPs should be able to offer dedicated paths or bandwidth for specific services like VoIP.  Services listed in a publicly determined products list.
>
>
> On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net> wrote:
>>
>> @Sebastian: This is a really great list of what the issues were in the
>> EU, and if y'all can repost there, rather than here, perhaps more
>> light will be generated outside our circles.
>>
>> https://news.ycombinator.com/item?id=37694306#37694307
>>
>> On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> >
>> >
>> >
>> > > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
>> > >
>> > > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de> wrote:
>> > >>
>> > >> Hi Dave,
>> > >>
>> > >> please excuse a number of tangents below ;)
>> > >
>> > > It would be nice, if as a (dis)organisation... the bufferbloat team
>> > > could focus on somehow getting both sides of the network neutrality
>> > > debate deeplying understanding the technological problem their
>> > > pre-conceptions face, and the (now readily available and inexpensive)
>> > > solutions that could be deployed, by most ISPs, over a weekend. We are
>> > > regularly bringing up a few thousand people a week on libreqos (that
>> > > we know of), and then of course, there are all the home routers and
>> > > CPE that are increasingly capable of doing the right thing.
>> > >
>> > > The time to try and shift the memes in in the coming days, and weeks.
>> > >
>> > > So ya'all being distracting below... aggh... ok, I'll bite.
>> >
>> >         [SM] Sorry to drag you into the weeds...
>> >
>> >
>> > >
>> > >>
>> > >>
>> > >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <rpm@lists.bufferbloat.net> wrote:
>> > >>>
>> > >>> Jason just did a beautiful thread as to what was the original source
>> > >>> of the network neutrality
>> > >>> bittorrent vs voip bufferbloat blowup.
>> > >>>
>> > >>> https://twitter.com/jlivingood/status/1707078242857849244
>> > >>
>> > >>        But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.
>> > >
>> > > The average use at peak hours for home broadband is below 5mbits per
>> > > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
>> > > business plans are less (but more bursty). Oversubscription within a
>> > > set of parameters is needed because, in part because upstream
>> > > bandwidth to the internet remains expensive (although I hope that cost
>> > > continues to drop rapidly), and in part, because it is needlessly
>> > > expensive to provision exactly the contracted rate to the customer. As
>> > > one example, I use the inverse square law as a rule of thumb for
>> > > wireless deployments - the range you can get from a 25/10 deployment
>> > > is geometrically better than 100/25, and the average bandwidth usage
>> > > nearly the same.
>> >
>> >         [SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).
>> >
>> >
>> > > Agreeing on industry standards for what the "slack" should be, might be of help.
>> >
>> >         [SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.
>> >
>> >
>> > >
>> > >>
>> > >>
>> > >> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.
>> > >
>> > > We have discussed elsewhere, the idea of a minimum contracted rate
>> > > (down to), which is easier to reason about.
>> >
>> >         [SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.
>> >         But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).
>> >
>> >
>> > *) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....
>> >
>> >
>> > >
>> > >>
>> > >>>
>> > >>> Seeing all the political activity tied onto it since (and now again)
>> > >>> reminds of two families at war about an incident that had happened
>> > >>> generations and generations before, where the two sides no longer
>> > >>> remembered why they hated each other so, but just went on hating, and
>> > >>> not forgiving, and not moving on.
>> > >>>
>> > >>> Yes, there are entirely separate and additional NN issues, but the
>> > >>> technical problem of providing common carriage between two very
>> > >>> different network application types (voip/gaming vs file transfer) is
>> > >>> thoroughly solved now,
>> > >>
>> > >>        I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
>> >
>> >         [SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf for an admitted slightly biased position:
>> >
>> > "       • Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20
>> >
>> >         • Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.
>> >
>> >
>> >         • Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21
>> >
>> >         • Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."
>> >
>> > These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.
>> > I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).
>> >
>> >
>> >
>> > >
>> > > Torrent problem thoroughly solved with FQ on the path and backpressure
>> > > from the mac scheduler.
>> > >
>> > > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
>> >
>> >         [SM] Thanks for the link. This is mainly for the self-congesrion case?
>> >
>> >
>> > >
>> > >>        I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...
>> > >>
>> > >>
>> > >> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.
>> > >>
>> > >>
>> > >>> and if only all sides recognised at least this
>> > >>> much, and made peace over it, and worked together to deploy those
>> > >>> solutions, maybe, just maybe, we could find mutually satisfactory
>> > >>> solutions to the other problems that plague the internet today, like
>> > >>> security, and the ipv6 rollout.
>> > >>
>> > >>        +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.
>> > >
>> > > I really would like to see a push for IPv6 mandated as a part of the
>> > > BEAD programs.
>> >
>> >         [SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.
>> >
>> > >
>> > >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)
>> > >
>> > > There is a separate NRPM going on for the new cybersecurity label at
>> > > the FCC. If I had time, and a partner,
>> > > we could rework the doc we did a few years ago. It is due oct 6.
>> > >
>> > >>
>> > >>
>> > >> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.
>> > >>
>> > >> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>> > >>
>> > >>> If anyone here knows anyone more political, still vibrating with 10+
>> > >>> years of outrage about NN on this fronts, on one side or the other, if
>> > >>> you could sit them down, over a beer, and try to explain that at the
>> > >>> start it was a technical problem nobody understood at the time, maybe
>> > >>> that would help.
>> > >>
>> > >>        So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).
>> > >
>> > > Governments make safer markets feasible.
>> >
>> >         [SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.
>> >
>> > Regards & Sorry for the tangent
>> >         Sebastian
>> >
>> > >
>> > >>
>> > >> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.
>> > >>
>> > >> Regards
>> > >>        Sebastian
>> > >>
>> > >>
>> > >>>
>> > >>> --
>> > >>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > >>> Dave Täht CSO, LibreQos
>> > >>> _______________________________________________
>> > >>> Rpm mailing list
>> > >>> Rpm@lists.bufferbloat.net
>> > >>> https://lists.bufferbloat.net/listinfo/rpm
>> > >>
>> > >
>> > >
>> > > --
>> > > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > > Dave Täht CSO, LibreQos
>> > > <2510.jpeg>
>> >
>>
>>
>> --
>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[Rpm] Getting Google to index. was:Re: [Starlink] [LibreQoS] net neutrality back in the news

[Rodney W. Grimes]

> I love that there are oh, 700+ people on these mailing lists, but we
> have zero visibility due to google not indexing them, where hackernews
> does. This is going to be an issue dominating the web (again, sadly)
> for a few weeks at least, and it would really help to be doing it
> there, rather than here:

You can correct the lack of indexing by google if you put a
properly built web interface to the archives of the list.

 [snip]
> Dave T?ht CSO, LibreQos

--
Rod Grimes                                                 rgrimes@freebsd.org

Re: [Rpm] [LibreQoS] net neutrality back in the news

[Jeremy Austin]

[-- Attachment #1: Type: text/plain, Size: 22185 bytes --]

On Thu, Sep 28, 2023 at 12:09 PM dan via LibreQoS <
libreqos@lists.bufferbloat.net> wrote:

> "(I assume most ISPs want happy customers)."
> made me laugh a little.  'Most' by quantity of businesses maybe, but
> 'most' in terms of customers being served by puts the Spectrums and
> Comcasts in the mix (in the US) and they don't care about happy customers
> they care about defacto monopolies in markets so that they don't have to
> care about happy customers.  That might not be their motivation, but 30
> years of their behavior makes it clear that it's the case.
>
>
> From my perspective, there should be a clear separation between carriers
> and last mile delivery..  Even if it's the same company, the rules defining
> each really should be different.
>
> Common Carriers or rather, carrier class services for 'internet', should
> be completely neutral.  Packets are packets.  However, I think it's
> important to carve out methods to have dedicated links for real time flows
> at the carrier level.  I don't know what that model looks like exactly, but
> being too stubborn about purist NN principals could really hurt VoIP
> services if there aren't methods to handle that.  I guess I really am
> describing 'internet fast lanes' for certain classes of services that we
> deem important enough as a whole.  not individual ISPs deciding, but rather
> 'the will of the people' saying VoIP is more important than netflix, you
> can carve out dedicated capacity for that.
>
>
Hoo boy. I'm interested in seeing how one can enforce the 'will of the
people' -- the application vendors (who are doing everything in their power
to prevent ISPs identifying *anything* about the traffic) will certainly
not obey such a will, and the ISP can in good faith implement such a
prioritization service but have no power to cryptographically authenticate
such traffic. This seems a dead end where no one is willing to bear the
cost.

For the last mile, I'm actually less concerned with pure NN and more
> concerned with no-blocking or 'brand' prioritization and required/label
> transparency.  ie, if an ISP wants to sell a DPI's QoE'd service that says
> 'videos at 1080p' and that is on the product label clearly and not in small
> print, then that's fine so long as 'videos' is agnostic of the vendor
> INCLUDING the ISP's own video products.  "100Mbps internet services with
> 1080P video $65" for example.  "Upgrade to our 100M with 4k video for
> +$10/m".  With that level of transparency I think that consumer protections
> are well handled.  And, because of that transparency, normal market
> capitalism mechanisms work.  I can say "100Mbps for $65 and full resolution
> video" for example.   Or "100Mbps Net Neutral service".   The secret DPI's
> QoE shaping is my main concern here, and where I think consumer protection
> needs to be pursued.
>

I don't disagree that consumer protection is vital, but competition might
be an easier remedy — today, for example, mobile providers aren't common
carriers for data, and users flock to WiFi which is cheaper per bit and
typically has less fudgery.

The above sentiment (we need service specification transparency) is
welcome, and largely aligned with my own personal thoughts — but very very
hard to prove that an ISP is complying with their own labels. NTIA doesn't
appear up to it. FCC/FTC certainly aren't at the current level of
specification/requirements.


> Again however, I think that ISPs should be able to offer dedicated paths
> or bandwidth for specific services like VoIP.  Services listed in a
> publicly determined products list.
>

That's entirely possible today, but those are carrier services, not BIAS.
Consumers don't want them. How are you going to sell them?

My $.02 not necessarily reflective of my employer,
Jeremy


>
>
> On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <
> libreqos@lists.bufferbloat.net> wrote:
>
>> @Sebastian: This is a really great list of what the issues were in the
>> EU, and if y'all can repost there, rather than here, perhaps more
>> light will be generated outside our circles.
>>
>> https://news.ycombinator.com/item?id=37694306#37694307
>>
>> On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <moeller0@gmx.de>
>> wrote:
>> >
>> >
>> >
>> > > On Sep 28, 2023, at 18:38, Dave Taht <dave.taht@gmail.com> wrote:
>> > >
>> > > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <moeller0@gmx.de>
>> wrote:
>> > >>
>> > >> Hi Dave,
>> > >>
>> > >> please excuse a number of tangents below ;)
>> > >
>> > > It would be nice, if as a (dis)organisation... the bufferbloat team
>> > > could focus on somehow getting both sides of the network neutrality
>> > > debate deeplying understanding the technological problem their
>> > > pre-conceptions face, and the (now readily available and inexpensive)
>> > > solutions that could be deployed, by most ISPs, over a weekend. We are
>> > > regularly bringing up a few thousand people a week on libreqos (that
>> > > we know of), and then of course, there are all the home routers and
>> > > CPE that are increasingly capable of doing the right thing.
>> > >
>> > > The time to try and shift the memes in in the coming days, and weeks.
>> > >
>> > > So ya'all being distracting below... aggh... ok, I'll bite.
>> >
>> >         [SM] Sorry to drag you into the weeds...
>> >
>> >
>> > >
>> > >>
>> > >>
>> > >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <
>> rpm@lists.bufferbloat.net> wrote:
>> > >>>
>> > >>> Jason just did a beautiful thread as to what was the original source
>> > >>> of the network neutrality
>> > >>> bittorrent vs voip bufferbloat blowup.
>> > >>>
>> > >>> https://twitter.com/jlivingood/status/1707078242857849244
>> > >>
>> > >>        But the core issue IMHO really was an economic one, the
>> over-subscription ratios that worked before torrenting simply did not cut
>> it any more in an environment when customers actually tried to use their
>> contracted access rates "quantitatively". In my outsider perspective an ISP
>> owes its customers the contracted rates (with some slack*) and any sort of
>> over-subscription is a valid economic optimization an ISP might take, IFF
>> that ISP is prepared to rapidly increase segment capacity (or down-grade
>> customer plans)) if the deployed over-subscription rate proves to have been
>> too optimistic. Mind you, most ISPs market plans by access speed (and
>> charge more for higher speeds) and hence are somewhat responsible to
>> actually deliver (again with some slack) these speeds.
>> > >
>> > > The average use at peak hours for home broadband is below 5mbits per
>> > > subscriber regardless of on a 25mbit or gbit plan. Remarkably,
>> > > business plans are less (but more bursty). Oversubscription within a
>> > > set of parameters is needed because, in part because upstream
>> > > bandwidth to the internet remains expensive (although I hope that cost
>> > > continues to drop rapidly), and in part, because it is needlessly
>> > > expensive to provision exactly the contracted rate to the customer. As
>> > > one example, I use the inverse square law as a rule of thumb for
>> > > wireless deployments - the range you can get from a 25/10 deployment
>> > > is geometrically better than 100/25, and the average bandwidth usage
>> > > nearly the same.
>> >
>> >         [SM] +1; hence my argument is not oversubscription per se is
>> bad, but that oversubscription needs to be managed and the level of
>> oversubscription needs to to be adapted ot he actual usage patterns. I am
>> sure however that ISPs already actively do that (I assume most ISPs want
>> happy customers).
>> >
>> >
>> > > Agreeing on industry standards for what the "slack" should be, might
>> be of help.
>> >
>> >         [SM] Not being part of that industry at all I would love to see
>> that as well, but can not contribute to defining that in any way.
>> >
>> >
>> > >
>> > >>
>> > >>
>> > >> *) Claiming "Up to", only carries that far, if you sell me X and I
>> mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is
>> acceptable unless occasionally is "at every late afternoon and evening"
>> prime-time.
>> > >
>> > > We have discussed elsewhere, the idea of a minimum contracted rate
>> > > (down to), which is easier to reason about.
>> >
>> >         [SM] Indeed, the German regulator (and I am not saying this is
>> the only or best option) decided to require ISPs to give 6 numbers
>> pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate,
>> a minimal (net) rate, and a usually achievable (net) rate, all rates were
>> defined as IP/TCP goodput to make verification easier on end-users. The
>> regulator also defined a measurement regime that end-users can follow to
>> check whether the ISP is fulfilling the contract and the law gives remedies
>> if ISPs do not deliver (either the right to immediately cancel the contract
>> and/or the right to adapt the monthly payments to the actually delivered
>> ratio of the contracted rates*). I think I need to add, that ISPs can set
>> these numbers freely, but are only allowed to advertise with the maximum
>> rate.
>> >         But if we talk about a single number per direction, minimal
>> rate is probably easiest, or something like a "usually achievable rate"
>> (that needs to be met or exceeded in say 90% of >= 20 measurements or so).
>> >
>> >
>> > *) In a ironic twist however the regulator so far has not explained how
>> deviations in each of the 6 numbers above should be aggregated to get one
>> single contract rate achievement ratio..., most ISPs took measures into
>> their own hands and simply offer customers typically a permanent rebate of
>> 5 EUR or immediate cancelation what ever the customer prefers....
>> >
>> >
>> > >
>> > >>
>> > >>>
>> > >>> Seeing all the political activity tied onto it since (and now again)
>> > >>> reminds of two families at war about an incident that had happened
>> > >>> generations and generations before, where the two sides no longer
>> > >>> remembered why they hated each other so, but just went on hating,
>> and
>> > >>> not forgiving, and not moving on.
>> > >>>
>> > >>> Yes, there are entirely separate and additional NN issues, but the
>> > >>> technical problem of providing common carriage between two very
>> > >>> different network application types (voip/gaming vs file transfer)
>> is
>> > >>> thoroughly solved now,
>> > >>
>> > >>        I am not sure this was at the core of the problem, my take is
>> really that "always-on" and relative upload-heavy torrent simply
>> demonstrated painfully for all involved that the old oversubscription
>> ratios were not suitable for the changed traffic profiles. I have some
>> sympathy for the ISPs here, they certainly did not try to sell their
>> customers short (good ISPs try to retain their customers and that works
>> best when customers are happy with the service) and having this problem
>> appear on many segments at the same time is not a fun place to be, and
>> upload was/is often (too) low in DOCSIS segments anyway; but this is why
>> e.g. my bit torrent could affect your VoIP, simply by pushing the whole
>> segment into upload capacity congestion... (ISPs in theory could fix this
>> by plain old QoS engineering, but the issue at hand was with a non-ISP
>> VoIP/SIP service and there QoS becomes tricky if the ISP as these packets
>> need to be identified in a way that is not game-able**)
>> >
>> >         [SM] See my later mail to Jason, I will not claim I know
>> Comcast's issues better than him and will accept that self-congestion also
>> played a major role in the initial problem. Over here in Europe the net
>> neutrality debate was kicked of less by an unfortunate confluence of new
>> usage profiles and traditional oversibscription ratios and less than ideal
>> packet scheduling, but rather by a series of pretty apparent attempts at
>> restricting consumer choice, see eg.
>> https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf
>> for an admitted slightly biased position:
>> >
>> > "       • Blocking of applications and services: In order to maximise
>> profits, some ISPs – that also offer their own services and applications
>> online – exclude certain services and applications of competing market
>> players. The most prominent case of this form of network discrimination is
>> European mobile providers (like Deutsche Telekom) blocking or restricting
>> the use of Voice over IP (VoIP) services (like Skype and Viber) for their
>> customers.20
>> >
>> >         • Slowing or “throttling” internet speeds: Some ISPs slow down
>> specific services (like YouTube) and applications (like Skype), or ask
>> users to pay an extra fee to have access to these internet platforms. Given
>> the high latency (delay) sensitivity of many applications, ISPs are able to
>> compromise the correct functioning of these services by slowing them down,
>> preventing the services from running properly. Often ISPs – especially
>> telecommunication companies – do this to favour their own voice calling
>> services over VoIP services, thereby crushing competition.
>> >
>> >
>> >         • Blocking websites: ISPs often block websites for a number of
>> reasons – to secure their network, or to avoid competition, and sometimes
>> for social, public relations or political reasons. In the UK, for instance,
>> Orange Telecom blocked the French digital rights advocacy group, La
>> Quadrature du Net’s website on pre-paid mobile accounts.21
>> >
>> >         • Preferential treatment of services and platforms: ISPs can
>> also impose data caps on internet access contracts while granting data
>> allowance exceptions to a company’s own proprietary streaming services
>> (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also
>> grant preferential treatment to select services – such as Orange France
>> with the popular music streaming service Deezer23 – ahead of other
>> competitors, effectively imposing anti-competitive limitations on markets
>> such as those for legal online music. Moreover, generally only large,
>> well-established companies can afford this preferential treatment,
>> resulting in a further stifling of innovation."
>> >
>> > These look like clearer attempts to monetize the ISP position as
>> gate-keeper to his customer's eye-balls (for content providers) as well as
>> gate-keeper to the internet for for paying customers.
>> > I find it much harder to have sympathy for the listed examples of ISP
>> behavior than the situation of rapidly changing usage pattern posed where
>> technical changes needed to be made, but where no attempt at unfair
>> monetization was evident, but maybe I am overly sensitive. These are all
>> examples that make me personally applaud the EU for its intervention to
>> make clear rules what "internet access providers" can and can not do. (The
>> EU also was quite flexible in applying/interpreting its rules during the
>> covid isolation periods, when it made it clear that e.g. treating certain
>> traffic classes e.g. streaming media to lower priority than video
>> conferences was within the neutrality framework as long as it was based on
>> traffic class and not on specific service providers IIRC).
>> >
>> >
>> >
>> > >
>> > > Torrent problem thoroughly solved with FQ on the path and backpressure
>> > > from the mac scheduler.
>> > >
>> > > https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf
>> >
>> >         [SM] Thanks for the link. This is mainly for the
>> self-congesrion case?
>> >
>> >
>> > >
>> > >>        I agree that on a single link we mostly solved the problem (a
>> few corner cases are left on links with very low capacity, where
>> essentially we can only manage the pain, not remove it)...
>> > >>
>> > >>
>> > >> **) Which is not rocket science either, a VoIP stream takes ~100
>> Kbps, so in theory an ISP might simply allow each customer say 5 VoIP
>> stream equivalents by allowing up to 500Kbps od traffic marked with a
>> specific DSCP as higher priority (with higher access probability for the
>> shared medium). I am not sayng this is my preferred solution, just saying
>> this is a solution that would have been available at the time if I memorize
>> my docsis capabilities correctly.
>> > >>
>> > >>
>> > >>> and if only all sides recognised at least this
>> > >>> much, and made peace over it, and worked together to deploy those
>> > >>> solutions, maybe, just maybe, we could find mutually satisfactory
>> > >>> solutions to the other problems that plague the internet today, like
>> > >>> security, and the ipv6 rollout.
>> > >>
>> > >>        +1. IPv6 is IMHO a prime example where the regulators should
>> stop talking softly and start showing the big stick they carry.
>> > >
>> > > I really would like to see a push for IPv6 mandated as a part of the
>> > > BEAD programs.
>> >
>> >         [SM] +1; I am not the biggest IPv6 fan, but that's what we have
>> and it is mostly serviceable so let's get this "party started" finally.
>> >
>> > >
>> > >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4
>> addresses only.. (most mass market ISPs do much better in that regard, but
>> for the stragglers it would help if the regulator would demand IPv6 with
>> PD***). Regarding security, the easiest way to achieve that would be to put
>> some heavy requirements on IoT manufacturers and vendors (like do what you
>> please as long as you are local LAN only, but once you reach out into the
>> cloud you need to fulfill the following list of requirements, with timely
>> security updates over a reasonable long usage period), however that might
>> not be very attractive for politicians/regulators to tackle (active
>> regulatory acts tends to get bad press unless something bad happened, but
>> even then the press often complains about the acts coming too late, but I
>> digress****)
>> > >
>> > > There is a separate NRPM going on for the new cybersecurity label at
>> > > the FCC. If I had time, and a partner,
>> > > we could rework the doc we did a few years ago. It is due oct 6.
>> > >
>> > >>
>> > >>
>> > >> ***) Strictly speaking IPv6 is required, since "internet access" is
>> defined as reaching all of the internet (as far as in the ISPs power) and
>> IPv6-only sites are not reachable for the CG-NAT-only customers. But so far
>> the local regulator does not seem to enforce that requirement, or hopefully
>> is working on this quietly behind the curtains.
>> > >>
>> > >> ****) This is not to diss the press, they are doing what they are
>> supposed to do, but it just shows that active regulation is a tricky
>> business, and a light touch typically "looks better" (even though I see no
>> real evidence it actually works better).
>> > >>
>> > >>> If anyone here knows anyone more political, still vibrating with 10+
>> > >>> years of outrage about NN on this fronts, on one side or the other,
>> if
>> > >>> you could sit them down, over a beer, and try to explain that at the
>> > >>> start it was a technical problem nobody understood at the time,
>> maybe
>> > >>> that would help.
>> > >>
>> > >>        So in the EU that debate is essentially settled, there is a
>> EU regulation that essentially spills out what ISPs owe their customers and
>> that has become the law of the land. The rationale for required un-biased
>> service and freedom to select terminal devices is well justified by the
>> market ideals of the EU, allowing ISPs to discriminate packets or terminal
>> devices restricts the market and will lead to undesired outcomes. (Fun fact
>> most big players in capitalist societies argue for "free markets" but at
>> the same time act to work-around the market mechanism by trying to move the
>> market into an oligo- or even monopoly condition, which is why strong
>> regulation is required*****).
>> > >
>> > > Governments make safer markets feasible.
>> >
>> >         [SM] Yes, I agree, both in markets are a pretty decent tool,
>> but need constant maintenance.
>> >
>> > Regards & Sorry for the tangent
>> >         Sebastian
>> >
>> > >
>> > >>
>> > >> *****) This is akin to professional sports where the audience
>> generally accepts that referees are necessary and occasionally need to make
>> "painful" calls, as the alternative would be anarchy, but I digress.
>> > >>
>> > >> Regards
>> > >>        Sebastian
>> > >>
>> > >>
>> > >>>
>> > >>> --
>> > >>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > >>> Dave Täht CSO, LibreQos
>> > >>> _______________________________________________
>> > >>> Rpm mailing list
>> > >>> Rpm@lists.bufferbloat.net
>> > >>> https://lists.bufferbloat.net/listinfo/rpm
>> > >>
>> > >
>> > >
>> > > --
>> > > Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > > Dave Täht CSO, LibreQos
>> > > <2510.jpeg>
>> >
>>
>>
>> --
>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> LibreQoS mailing list
>> LibreQoS@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/libreqos
>>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos
>


-- 
--
Jeremy Austin
Sr. Product Manager
Preseem | Aterlo Networks
preseem.com

Book a Call: https://app.hubspot.com/meetings/jeremy548
Phone: 1-833-733-7336 x718
Email: jeremy@preseem.com

Stay Connected with Newsletters & More:
*https://preseem.com/stay-connected/* <https://preseem.com/stay-connected/>

[-- Attachment #2: Type: text/html, Size: 27315 bytes --]

Re: [Rpm] [Bloat] [LibreQoS] net neutrality back in the news

[Livingood, Jason]

> From: Bloat <bloat-bounces@lists.bufferbloat.net> on behalf of Jeremy Austin via Bloat <bloat@lists.bufferbloat.net>

> I'm interested in seeing how one can enforce the 'will of the people' -- the application vendors (who are doing everything in their power to prevent ISPs identifying *anything* about the traffic) will certainly not obey such a will, and the ISP can in good faith implement such a prioritization service but have no power to cryptographically authenticate such traffic. This seems a dead end where no one is willing to bear the cost.

Great point! Back in the old days, sure you could use DPI to identify flow X as streaming video, flow Y as web traffic, and flow Z as email. No more - based on 3rd party stats somewhere between 90% - 95% of traffic is encrypted (increasing all the time) -- so no more ability to accurately infer application type and trigger a network response based on that. 

Great doc on the shift to pervasive encryption: https://www.rfc-editor.org/rfc/rfc9446.html "Reflections on Ten Years Past the Snowden Revelations"

JL 

Re: [Rpm] [Starlink] [LibreQoS] net neutrality back in the news

[Livingood, Jason]

> dan <dandenson@gmail.com> wrote: 

> "(I assume most ISPs want happy customers)."
made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers.  

Corporations are motivated to generate returns for investors. In that context, happy customers stay longer (less churn) and spend more (upgrades, multiple services). And unhappy customers generate costs via disconnects (loss of revenue, costs to replace them with a new customer to just stay at the same subscriber levels), and costs via customer contacts (call center staff). So, IMO on a purely financial basis, public companies have significant motivation to retain customers and keep them happy. This typically follows through to staff members having part of their variable compensation based on things like NPS scores, contact rates, etc. And specifically in relation to Comcast, the company recently has 4 new wireless competitors: three 5G FWA and one LEO (more coming) - and those are posing significant competitive risks (and taking customers). 

> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...

The two thoughts your comments (thanks for the response BTW!) trigger are:
1 - Often regulation looks to the past - in this case maybe an era of bandwidth scarcity where prioritization may have mattered. I think we're in the midst of a shift into bandwidth abundance where priority does not matter. What will is latency/responsiveness, content/compute localization, reliability, consistency, security, etc. 
2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care (contact) costs and would see people start to immediately shift to competitors (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like something that could realistically happen any longer in the US.

JL

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 3155 bytes --]

On Thu, 28 Sep 2023, Livingood, Jason via Bloat wrote:

> Date: Thu, 28 Sep 2023 20:48:58 +0000
> From: "Livingood, Jason via Bloat" <bloat@lists.bufferbloat.net>
> Reply-To: "Livingood, Jason" <Jason_Livingood@comcast.com>
> To: dan <dandenson@gmail.com>, Dave Taht <dave.taht@gmail.com>
> Cc: Rpm <rpm@lists.bufferbloat.net>,
>     Dave Taht via Starlink <starlink@lists.bufferbloat.net>,
>     bloat <bloat@lists.bufferbloat.net>,
>     libreqos <libreqos@lists.bufferbloat.net>,
>     Jamal Hadi Salim <jhs@mojatatu.com>
> Subject: Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the
>     news
> 
>> dan <dandenson@gmail.com> wrote:
>
>> "(I assume most ISPs want happy customers)."

>> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' 
>> in terms of customers being served by puts the Spectrums and Comcasts in the 
>> mix (in the US) and they don't care about happy customers they care about 
>> defacto monopolies in markets so that they don't have to care about happy 
>> customers. 
>
> In that context, happy customers stay longer (less churn) and spend more 
> (upgrades, multiple services). And unhappy customers generate costs via 
> disconnects (loss of revenue, costs to replace them with a new customer to 
> just stay at the same subscriber levels), and costs via customer contacts 
> (call center staff).

Except when you have a monopoly in an area, at which point the ability of 
customers to leave is minimal, and years of bad customer service means that 
people don't bother complaining, so the call center staffing costs are lower 
than they should be.

>> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...
>
> The two thoughts your comments (thanks for the response BTW!) trigger are:

> 1 - Often regulation looks to the past - in this case maybe an era of 
> bandwidth scarcity where prioritization may have mattered. I think we're in 
> the midst of a shift into bandwidth abundance where priority does not matter. 
> What will is latency/responsiveness, content/compute localization, 
> reliability, consistency, security, etc.

> 2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care 
> (contact) costs and would see people start to immediately shift to competitors 
> (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like 
> something that could realistically happen any longer in the US.

Dave T called out earlier that the rise of bittorrent was a large part of the 
inital NN discussion here in the US. But a second large portion was a money grab 
from ISPs thinking that they could hold up large paid websites (netflix for 
example) for additional fees by threatening to make their service less useful to 
their users (viewing their users as an asset to be marketed to the websites 
rather than customers to be satisfied by providing them access to the websites)

I don't know if a new round of "it's not fair that Netflix doesn't pay us for 
the bandwidth to service them" would fall flat at this point or not.

David Lang

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Jonathan Morton]

> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> 
> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.

I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:


1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.

This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.


2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.

FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.

This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.


3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.

**THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.

And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.


All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.

 - Jonathan Morton

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Rich Brown]

Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 

Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 

I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)

- Rich Brown

> On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
>> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
>> 
>> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> 
> I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> 
> 
> 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> 
> This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> 
> 
> 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> 
> FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> 
> This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> 
> 
> 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> 
> **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> 
> And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> 
> 
> All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> 
> - Jonathan Morton
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[dan]

[-- Attachment #1: Type: text/plain, Size: 8530 bytes --]

ok, lots and lots of great comments here for sure.

bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't
carrying to many places at affordable rates.  I've pulled quotes from Lumen
and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of
service.  This isn't abundance and it's widespread and it leaves only
major providers that can afford/amortize out 100G transits etc.
My answer to this is one that Dave and I have bounced back and forth is the
idea of micro IXs in every municipality and having that somehow tied into
access to the ROW in counties etc.  Not fully hashed out, but the fiber is
in the ground, it could be sold, but the carriers are not well incentivised
to sell it.  It takes the better part of a year to get a DIA within 100ft
of a Lumen hut sometimes...  Heck, it could even be a government program to
get an μ*IX* with x feet of every school, city hall, and library.  I don't
care how it's done but this would get abundance NEAR end users and open up
essentially every town to competition.

monopoly.  This is a historical thing for most cable and DSL incumbents.
They have enjoyed virtual monopolies with cable owning population centers
and DSL owning the outskirts and there is no product darwinism here where
customer satisfaction is a pressure.  That may not be the future but it
definitely is the past.  These companies may have to shift into customer
satisfaction as a major part instead of a minor part of their corporate
culture to fend off fttx and ultra-modern wisps.

Starlink is not offering significant competition to major carriers.
Starlink's 1.5 million customers are at LEAST 90% pulled from other
satellite services and small ISPs.  Spectrum and Comcast's losses to
starlink are measured in decimal points.

Only fttx and ultra-modern wireless tech really threatens these
incumbents.  Typical wisps aren't putting a dent in these guys, just
scraping the paint off their bumper.  We're pulling customers at the scale
of 'dozens' for example.  Spectrum's management doesn't know we exist we're
such a small threat to them.

But to further the point here, these fttx and ultra-modern wisps can only
exist in places where there is adequate carrier services to start with.  In
areas where they spend the money and do the build but there aren't good
carrier services, those fiber services suck and the wISPs start to claw
back even with inferior technology.  We've pulled quite a few customers off
fttx deployments because of this sort of situation.


On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:

> Thank you Jonathan for this clear description of the issues and their
> history. I wonder if there's a fourth one - privacy.
>
> Rosenworcel's talk
> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
> that ISPs might want to monetize our traffic patterns and location data.
> (This is less of an issue in the EU, but the US remains a Wild West in this
> regard.)
>
> I am hopeful that the FCC will include this in their NPRM (which must be
> available by now but I haven't looked...)
>
> - Rich Brown
>
> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> >
> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> >>
> >> Dave T called out earlier that the rise of bittorrent was a large part
> of the inital NN discussion here in the US. But a second large portion was
> a money grab from ISPs thinking that they could hold up large paid websites
> (netflix for example) for additional fees by threatening to make their
> service less useful to their users (viewing their users as an asset to be
> marketed to the websites rather than customers to be satisfied by providing
> them access to the websites)
> >>
> >> I don't know if a new round of "it's not fair that Netflix doesn't pay
> us for the bandwidth to service them" would fall flat at this point or not.
> >
> > I think there were three more-or-less separate concerns which have, over
> time, fallen under the same umbrella:
> >
> >
> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
> flows, and the "induced demand" phenomenon means that increases in link
> rate do not in themselves solve this problem, even though they may be sold
> as doing so.
> >
> > This is directly addressed by properly-sized buffers and/or AQM, and
> even better by FQ and SQM.  It's a solved problem, so long as the solutions
> are deployed.  It's not usually necessary, for example, to specifically
> enhance service for latency-sensitive traffic, if FQ does a sufficiently
> good job.  An increased link rate *does* enhance service quality for both
> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> >
> >
> > 2:  Swarm traffic tends to drown out conventional traffic, due to
> congestion control algorithms which try to be more-or-less fair on a
> per-flow basis, and the substantially larger number of parallel flows used
> by swarm traffic.  This also caused subscribers using swarm traffic to
> impair the service of subscribers who had nothing to do with it.
> >
> > FQ on a per-flow basis (see problem 1) actually amplifies this effect,
> and I think it was occasionally used as an argument for *not* deploying
> FQ.  ISPs' initial response was to outright block swarm traffic where they
> could identify it, which was then softened to merely throttling it heavily,
> before NN regulations intervened.  Usage quotas also showed up around this
> time, and were probably related to this problem.
> >
> > This has since been addressed by several means.  ISPs may use FQ on a
> per-subscriber basis to prevent one subscriber's heavy traffic from
> degrading service for another.  Swarm applications nowadays tend to employ
> altruistic congestion control which deliberately compensates for the large
> number of flows, and/or mark them with one or more of the Least Effort
> class DSCPs.  Hence, swarm applications are no longer as damaging to
> service quality as they used to be.  Usage quotas, however, still remain in
> use as a profit centre, to the point where an "unlimited" service is a rare
> and precious specimen in many jurisdictions.
> >
> >
> > 3:  ISPs merged with media distribution companies, creating a conflict
> of interest in which the media side of the business wanted the internet
> side to actively favour "their own" media traffic at the expense of "the
> competition".  Some ISPs began to actively degrade Netflix traffic, in
> particular by refusing to provision adequate peering capacity at the nodes
> through which Netflix traffic predominated, or by zero-rating (for the
> purpose of usage quotas) traffic from their own media empire while refusing
> to do the same for Netflix traffic.
> >
> > **THIS** was the true core of Net Neutrality.  NN regulations forced
> ISPs to carry Netflix traffic with reasonable levels of service, even
> though they didn't want to for purely selfish and greedy commercial
> reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile
> practice, which I am perfectly sure would resume just as soon as NN
> regulations were repealed.
> >
> > And this type of practice is just the sort of thing that technologies
> like L4S are designed to support.  The ISPs behind L4S actively do not want
> a technology that works end-to-end over the general Internet.  They want
> something that can provide a domination service within their own walled
> gardens.  That's why L4S is a NN hazard, and why they actively resisted all
> attempts to displace it with SCE.
> >
> >
> > All of the above were made more difficult to solve by the monopolistic
> nature of the Internet service industry.  It is actively difficult for
> Internet users to move to a truly different service, especially one based
> on a different link technology.  When attempts are made to increase
> competition, for example by deploying a publicly-funded network, the
> incumbents actively sabotage those attempts by any means they can.
> Monopolies are inherently customer-hostile, and arguments based on market
> forces fail in their presence.
> >
> > - Jonathan Morton
> >
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>
>

[-- Attachment #2: Type: text/html, Size: 9601 bytes --]

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 9972 bytes --]

Back then in 2015, when NN was enacted by Wheeler & CO, there was a great
body of work (IMHO) done on this subject by Martin Geddes:
https://www.martingeddes.com/1261-2/

But let's pick *one
<https://www.martingeddes.com/ofcom-publishes-pnsol-scientific-report-on-net-neutrality/>
*report written
by his colleagues and published by Ofcom (UK telecoms regulator):


   - *You cannot conflate ‘equality of [packet] treatment’ with delivering
   equality of [user application] outcomes.* Only the latter matters, as
   ordinary users don’t care what happened to the packets in transit. Yet the
   relevant academic literature fixates on the local operation of the
   mechanisms (including Traffic Management), not their global aggregate
   effect.


All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net>
wrote:

> ok, lots and lots of great comments here for sure.
>
> bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't
> carrying to many places at affordable rates.  I've pulled quotes from Lumen
> and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of
> service.  This isn't abundance and it's widespread and it leaves only
> major providers that can afford/amortize out 100G transits etc.
> My answer to this is one that Dave and I have bounced back and forth is
> the idea of micro IXs in every municipality and having that somehow tied
> into access to the ROW in counties etc.  Not fully hashed out, but the
> fiber is in the ground, it could be sold, but the carriers are not well
> incentivised to sell it.  It takes the better part of a year to get a DIA
> within 100ft of a Lumen hut sometimes...  Heck, it could even be a
> government program to get an μ*IX* with x feet of every school, city
> hall, and library.  I don't care how it's done but this would get abundance
> NEAR end users and open up essentially every town to competition.
>
> monopoly.  This is a historical thing for most cable and DSL incumbents.
> They have enjoyed virtual monopolies with cable owning population centers
> and DSL owning the outskirts and there is no product darwinism here where
> customer satisfaction is a pressure.  That may not be the future but it
> definitely is the past.  These companies may have to shift into customer
> satisfaction as a major part instead of a minor part of their corporate
> culture to fend off fttx and ultra-modern wisps.
>
> Starlink is not offering significant competition to major carriers.
> Starlink's 1.5 million customers are at LEAST 90% pulled from other
> satellite services and small ISPs.  Spectrum and Comcast's losses to
> starlink are measured in decimal points.
>
> Only fttx and ultra-modern wireless tech really threatens these
> incumbents.  Typical wisps aren't putting a dent in these guys, just
> scraping the paint off their bumper.  We're pulling customers at the scale
> of 'dozens' for example.  Spectrum's management doesn't know we exist we're
> such a small threat to them.
>
> But to further the point here, these fttx and ultra-modern wisps can only
> exist in places where there is adequate carrier services to start with.  In
> areas where they spend the money and do the build but there aren't good
> carrier services, those fiber services suck and the wISPs start to claw
> back even with inferior technology.  We've pulled quite a few customers off
> fttx deployments because of this sort of situation.
>
>
> On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com>
> wrote:
>
>> Thank you Jonathan for this clear description of the issues and their
>> history. I wonder if there's a fourth one - privacy.
>>
>> Rosenworcel's talk
>> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
>> that ISPs might want to monetize our traffic patterns and location data.
>> (This is less of an issue in the EU, but the US remains a Wild West in this
>> regard.)
>>
>> I am hopeful that the FCC will include this in their NPRM (which must be
>> available by now but I haven't looked...)
>>
>> - Rich Brown
>>
>> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
>> rpm@lists.bufferbloat.net> wrote:
>> >
>> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
>> bloat@lists.bufferbloat.net> wrote:
>> >>
>> >> Dave T called out earlier that the rise of bittorrent was a large part
>> of the inital NN discussion here in the US. But a second large portion was
>> a money grab from ISPs thinking that they could hold up large paid websites
>> (netflix for example) for additional fees by threatening to make their
>> service less useful to their users (viewing their users as an asset to be
>> marketed to the websites rather than customers to be satisfied by providing
>> them access to the websites)
>> >>
>> >> I don't know if a new round of "it's not fair that Netflix doesn't pay
>> us for the bandwidth to service them" would fall flat at this point or not.
>> >
>> > I think there were three more-or-less separate concerns which have,
>> over time, fallen under the same umbrella:
>> >
>> >
>> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
>> flows, and the "induced demand" phenomenon means that increases in link
>> rate do not in themselves solve this problem, even though they may be sold
>> as doing so.
>> >
>> > This is directly addressed by properly-sized buffers and/or AQM, and
>> even better by FQ and SQM.  It's a solved problem, so long as the solutions
>> are deployed.  It's not usually necessary, for example, to specifically
>> enhance service for latency-sensitive traffic, if FQ does a sufficiently
>> good job.  An increased link rate *does* enhance service quality for both
>> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
>> >
>> >
>> > 2:  Swarm traffic tends to drown out conventional traffic, due to
>> congestion control algorithms which try to be more-or-less fair on a
>> per-flow basis, and the substantially larger number of parallel flows used
>> by swarm traffic.  This also caused subscribers using swarm traffic to
>> impair the service of subscribers who had nothing to do with it.
>> >
>> > FQ on a per-flow basis (see problem 1) actually amplifies this effect,
>> and I think it was occasionally used as an argument for *not* deploying
>> FQ.  ISPs' initial response was to outright block swarm traffic where they
>> could identify it, which was then softened to merely throttling it heavily,
>> before NN regulations intervened.  Usage quotas also showed up around this
>> time, and were probably related to this problem.
>> >
>> > This has since been addressed by several means.  ISPs may use FQ on a
>> per-subscriber basis to prevent one subscriber's heavy traffic from
>> degrading service for another.  Swarm applications nowadays tend to employ
>> altruistic congestion control which deliberately compensates for the large
>> number of flows, and/or mark them with one or more of the Least Effort
>> class DSCPs.  Hence, swarm applications are no longer as damaging to
>> service quality as they used to be.  Usage quotas, however, still remain in
>> use as a profit centre, to the point where an "unlimited" service is a rare
>> and precious specimen in many jurisdictions.
>> >
>> >
>> > 3:  ISPs merged with media distribution companies, creating a conflict
>> of interest in which the media side of the business wanted the internet
>> side to actively favour "their own" media traffic at the expense of "the
>> competition".  Some ISPs began to actively degrade Netflix traffic, in
>> particular by refusing to provision adequate peering capacity at the nodes
>> through which Netflix traffic predominated, or by zero-rating (for the
>> purpose of usage quotas) traffic from their own media empire while refusing
>> to do the same for Netflix traffic.
>> >
>> > **THIS** was the true core of Net Neutrality.  NN regulations forced
>> ISPs to carry Netflix traffic with reasonable levels of service, even
>> though they didn't want to for purely selfish and greedy commercial
>> reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile
>> practice, which I am perfectly sure would resume just as soon as NN
>> regulations were repealed.
>> >
>> > And this type of practice is just the sort of thing that technologies
>> like L4S are designed to support.  The ISPs behind L4S actively do not want
>> a technology that works end-to-end over the general Internet.  They want
>> something that can provide a domination service within their own walled
>> gardens.  That's why L4S is a NN hazard, and why they actively resisted all
>> attempts to displace it with SCE.
>> >
>> >
>> > All of the above were made more difficult to solve by the monopolistic
>> nature of the Internet service industry.  It is actively difficult for
>> Internet users to move to a truly different service, especially one based
>> on a different link technology.  When attempts are made to increase
>> competition, for example by deploying a publicly-funded network, the
>> incumbents actively sabotage those attempts by any means they can.
>> Monopolies are inherently customer-hostile, and arguments based on market
>> forces fail in their presence.
>> >
>> > - Jonathan Morton
>> >
>> > _______________________________________________
>> > Rpm mailing list
>> > Rpm@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/rpm
>>
>> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
>

[-- Attachment #2: Type: text/html, Size: 12855 bytes --]

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Frantisek,

> On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> https://www.martingeddes.com/1261-2/
> 
> But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> 
> 	• You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.

	[SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:

[...]
(8)
When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
(9)
The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
(10)
Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
(11)
Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
(12)
Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
[...]

There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.

There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 

Regards
	Sebastian

P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).



> 
> All the best,
> 
> Frank
> 
> Frantisek (Frank) Borsik
> 
>  
> 
> https://www.linkedin.com/in/frantisekborsik
> 
> Signal, Telegram, WhatsApp: +421919416714 
> 
> iMessage, mobile: +420775230885
> 
> Skype: casioa5302ca
> 
> frantisek.borsik@gmail.com
> 
> 
> 
> On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> ok, lots and lots of great comments here for sure.  
> 
> bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> 
> monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> 
> Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> 
> Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> 
> But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> 
> 
> On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> 
> Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> 
> I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> 
> - Rich Brown
> 
> > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > 
> >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> >> 
> >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> >> 
> >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > 
> > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > 
> > 
> > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > 
> > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > 
> > 
> > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > 
> > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > 
> > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > 
> > 
> > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > 
> > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > 
> > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > 
> > 
> > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > 
> > - Jonathan Morton
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Starlink] [Bloat] [LibreQoS] net neutrality back in the news

[Vint Cerf]

[-- Attachment #1.1: Type: text/plain, Size: 17087 bytes --]

the phrase "treat equally" can (maybe should?) be interpreted as offering
the same options for traffic handling to all parties on the same terms and
conditions. If there is only one class of service, then equally is the only
option. If there are multiple classes of service, then these could
(should?) be available to all customers indiscriminately. For example,
there might be several distinct services with different maximum bit rates;
the higher rates possibly available for a higher charge. If there is
discrimination, it should be on the basis of customer choice and not
dictated by the provider.

Is that consistent with the European interpretation?

v


On Sat, Sep 30, 2023 at 1:19 PM Sebastian Moeller via Starlink <
starlink@lists.bufferbloat.net> wrote:

> Hi Frantisek,
>
> > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> >
> > Back then in 2015, when NN was enacted by Wheeler & CO, there was a
> great body of work (IMHO) done on this subject by Martin Geddes:
> > https://www.martingeddes.com/1261-2/
> >
> > But let's pick one report written by his colleagues and published by
> Ofcom (UK telecoms regulator):
> >
> >       • You cannot conflate ‘equality of [packet] treatment’ with
> delivering equality of [user application] outcomes. Only the latter
> matters, as ordinary users don’t care what happened to the packets in
> transit. Yet the relevant academic literature fixates on the local
> operation of the mechanisms (including Traffic Management), not their
> global aggregate effect.
>
>         [SM] The EU laid out pretty clear why they mandated the NN
> regulations in eu regulation 2015/2120:
>
> [...]
> (8)
> When providing internet access services, providers of those services
> should treat all traffic equally, without discrimination, restriction or
> interference, independently of its sender or receiver, content, application
> or service, or terminal equipment. According to general principles of Union
> law and settled case-law, comparable situations should not be treated
> differently and different situations should not be treated in the same way
> unless such treatment is objectively justified.
> (9)
> The objective of reasonable traffic management is to contribute to an
> efficient use of network resources and to an optimisation of overall
> transmission quality responding to the objectively different technical
> quality of service requirements of specific categories of traffic, and thus
> of the content, applications and services transmitted. Reasonable traffic
> management measures applied by providers of internet access services should
> be transparent, non-discriminatory and proportionate, and should not be
> based on commercial considerations. The requirement for traffic management
> measures to be non-discriminatory does not preclude providers of internet
> access services from implementing, in order to optimise the overall
> transmission quality, traffic management measures which differentiate
> between objectively different categories of traffic. Any such
> differentiation should, in order to optimise overall quality and user
> experience, be permitted only on the basis of objectively different
> technical quality of service requirements (for example, in terms of
> latency, jitter, packet loss, and bandwidth) of the specific categories of
> traffic, and not on the basis of commercial considerations. Such
> differentiating measures should be proportionate in relation to the purpose
> of overall quality optimisation and should treat equivalent traffic
> equally. Such measures should not be maintained for longer than necessary.
> (10)
> Reasonable traffic management does not require techniques which monitor
> the specific content of data traffic transmitted via the internet access
> service.
> (11)
> Any traffic management practices which go beyond such reasonable traffic
> management measures, by blocking, slowing down, altering, restricting,
> interfering with, degrading or discriminating between specific content,
> applications or services, or specific categories of content, applications
> or services, should be prohibited, subject to the justified and defined
> exceptions laid down in this Regulation. Those exceptions should be subject
> to strict interpretation and to proportionality requirements. Specific
> content, applications and services, as well as specific categories thereof,
> should be protected because of the negative impact on end-user choice and
> innovation of blocking, or of other restrictive measures not falling within
> the justified exceptions. Rules against altering content, applications or
> services refer to a modification of the content of the communication, but
> do not ban non-discriminatory data compression techniques which reduce the
> size of a data file without any modification of the content. Such
> compression enables a more efficient use of scarce resources and serves the
> end-users’ interests by reducing data volumes, increasing speed and
> enhancing the experience of using the content, applications or services
> concerned.
> (12)
> Traffic management measures that go beyond such reasonable traffic
> management measures may only be applied as necessary and for as long as
> necessary to comply with the three justified exceptions laid down in this
> Regulation.
> [...]
>
> There really is little IMHO that can be brought against these, all pretty
> fair and reasonable. What it does is accept that internet access is
> essential infrastructure and that hence access needs to be as well
> regulated as access to water, electricity, gas, streets, ... . Yes this has
> some consequences of what ISPs can and can not do. But this is normal "cost
> of business". I for one am quite happy about this regulation existing as
> locally it did away with some (not all) shenanigans of some ISPs that were
> clearly not operating in the interest of their paying eye-balls.
>
> There is a whole cottage industry of consultants that decry the EU's
> decision and try to lobby against it, but honestly reading these mostly
> makes me think harsher regulation might be required (on consultans about
> how much they are allowed to massage the facts ;) ).
>
> Regards
>         Sebastian
>
> P.S.: Of course if we look close enough we surely can find corner-cases
> where either the EU regulations or the translation into national law result
> in less desirable outcomes, but "nothing is perfect" and all in all the
> regulations seem to be "good enough". With the caveat that explicitly
> excluding ISP interconnect from the regulations BEREC essentially pointed
> the way for ISPs wanting to monetize their eye-balls twice to do so via
> interconnects, but that only works for the 800 pound gorillas and generally
> is not a game smaller ISPs can play. I do understand why BEREC wants to
> stay out of the interconnection issue, as this is rather complicated and
> the market seems to generally work okay-ish (that is not badly enough to
> make intervention a hot-button issue for voters and hence politicians).
>
>
>
> >
> > All the best,
> >
> > Frank
> >
> > Frantisek (Frank) Borsik
> >
> >
> >
> > https://www.linkedin.com/in/frantisekborsik
> >
> > Signal, Telegram, WhatsApp: +421919416714 <+421%20919%20416%20714>
> >
> > iMessage, mobile: +420775230885 <+420%20775%20230%20885>
> >
> > Skype: casioa5302ca
> >
> > frantisek.borsik@gmail.com
> >
> >
> >
> > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net>
> wrote:
> > ok, lots and lots of great comments here for sure.
> >
> > bandwidth abundance:  Not for most people and ISPs.  The 'carriers'
> aren't carrying to many places at affordable rates.  I've pulled quotes
> from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a
> gig of service.  This isn't abundance and it's widespread and it leaves
> only major providers that can afford/amortize out 100G transits etc.
> > My answer to this is one that Dave and I have bounced back and forth is
> the idea of micro IXs in every municipality and having that somehow tied
> into access to the ROW in counties etc.  Not fully hashed out, but the
> fiber is in the ground, it could be sold, but the carriers are not well
> incentivised to sell it.  It takes the better part of a year to get a DIA
> within 100ft of a Lumen hut sometimes...  Heck, it could even be a
> government program to get an μIX with x feet of every school, city hall,
> and library.  I don't care how it's done but this would get abundance NEAR
> end users and open up essentially every town to competition.
> >
> > monopoly.  This is a historical thing for most cable and DSL
> incumbents.  They have enjoyed virtual monopolies with cable owning
> population centers and DSL owning the outskirts and there is no product
> darwinism here where customer satisfaction is a pressure.  That may not be
> the future but it definitely is the past.  These companies may have to
> shift into customer satisfaction as a major part instead of a minor part of
> their corporate culture to fend off fttx and ultra-modern wisps.
> >
> > Starlink is not offering significant competition to major carriers.
> Starlink's 1.5 million customers are at LEAST 90% pulled from other
> satellite services and small ISPs.  Spectrum and Comcast's losses to
> starlink are measured in decimal points.
> >
> > Only fttx and ultra-modern wireless tech really threatens these
> incumbents.  Typical wisps aren't putting a dent in these guys, just
> scraping the paint off their bumper.  We're pulling customers at the scale
> of 'dozens' for example.  Spectrum's management doesn't know we exist we're
> such a small threat to them.
> >
> > But to further the point here, these fttx and ultra-modern wisps can
> only exist in places where there is adequate carrier services to start
> with.  In areas where they spend the money and do the build but there
> aren't good carrier services, those fiber services suck and the wISPs start
> to claw back even with inferior technology.  We've pulled quite a few
> customers off fttx deployments because of this sort of situation.
> >
> >
> > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com>
> wrote:
> > Thank you Jonathan for this clear description of the issues and their
> history. I wonder if there's a fourth one - privacy.
> >
> > Rosenworcel's talk
> https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out
> that ISPs might want to monetize our traffic patterns and location data.
> (This is less of an issue in the EU, but the US remains a Wild West in this
> regard.)
> >
> > I am hopeful that the FCC will include this in their NPRM (which must be
> available by now but I haven't looked...)
> >
> > - Rich Brown
> >
> > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <
> rpm@lists.bufferbloat.net> wrote:
> > >
> > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <
> bloat@lists.bufferbloat.net> wrote:
> > >>
> > >> Dave T called out earlier that the rise of bittorrent was a large
> part of the inital NN discussion here in the US. But a second large portion
> was a money grab from ISPs thinking that they could hold up large paid
> websites (netflix for example) for additional fees by threatening to make
> their service less useful to their users (viewing their users as an asset
> to be marketed to the websites rather than customers to be satisfied by
> providing them access to the websites)
> > >>
> > >> I don't know if a new round of "it's not fair that Netflix doesn't
> pay us for the bandwidth to service them" would fall flat at this point or
> not.
> > >
> > > I think there were three more-or-less separate concerns which have,
> over time, fallen under the same umbrella:
> > >
> > >
> > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive
> flows, and the "induced demand" phenomenon means that increases in link
> rate do not in themselves solve this problem, even though they may be sold
> as doing so.
> > >
> > > This is directly addressed by properly-sized buffers and/or AQM, and
> even better by FQ and SQM.  It's a solved problem, so long as the solutions
> are deployed.  It's not usually necessary, for example, to specifically
> enhance service for latency-sensitive traffic, if FQ does a sufficiently
> good job.  An increased link rate *does* enhance service quality for both
> latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > >
> > >
> > > 2:  Swarm traffic tends to drown out conventional traffic, due to
> congestion control algorithms which try to be more-or-less fair on a
> per-flow basis, and the substantially larger number of parallel flows used
> by swarm traffic.  This also caused subscribers using swarm traffic to
> impair the service of subscribers who had nothing to do with it.
> > >
> > > FQ on a per-flow basis (see problem 1) actually amplifies this effect,
> and I think it was occasionally used as an argument for *not* deploying
> FQ.  ISPs' initial response was to outright block swarm traffic where they
> could identify it, which was then softened to merely throttling it heavily,
> before NN regulations intervened.  Usage quotas also showed up around this
> time, and were probably related to this problem.
> > >
> > > This has since been addressed by several means.  ISPs may use FQ on a
> per-subscriber basis to prevent one subscriber's heavy traffic from
> degrading service for another.  Swarm applications nowadays tend to employ
> altruistic congestion control which deliberately compensates for the large
> number of flows, and/or mark them with one or more of the Least Effort
> class DSCPs.  Hence, swarm applications are no longer as damaging to
> service quality as they used to be.  Usage quotas, however, still remain in
> use as a profit centre, to the point where an "unlimited" service is a rare
> and precious specimen in many jurisdictions.
> > >
> > >
> > > 3:  ISPs merged with media distribution companies, creating a conflict
> of interest in which the media side of the business wanted the internet
> side to actively favour "their own" media traffic at the expense of "the
> competition".  Some ISPs began to actively degrade Netflix traffic, in
> particular by refusing to provision adequate peering capacity at the nodes
> through which Netflix traffic predominated, or by zero-rating (for the
> purpose of usage quotas) traffic from their own media empire while refusing
> to do the same for Netflix traffic.
> > >
> > > **THIS** was the true core of Net Neutrality.  NN regulations forced
> ISPs to carry Netflix traffic with reasonable levels of service, even
> though they didn't want to for purely selfish and greedy commercial
> reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile
> practice, which I am perfectly sure would resume just as soon as NN
> regulations were repealed.
> > >
> > > And this type of practice is just the sort of thing that technologies
> like L4S are designed to support.  The ISPs behind L4S actively do not want
> a technology that works end-to-end over the general Internet.  They want
> something that can provide a domination service within their own walled
> gardens.  That's why L4S is a NN hazard, and why they actively resisted all
> attempts to displace it with SCE.
> > >
> > >
> > > All of the above were made more difficult to solve by the monopolistic
> nature of the Internet service industry.  It is actively difficult for
> Internet users to move to a truly different service, especially one based
> on a different link technology.  When attempts are made to increase
> competition, for example by deploying a publicly-funded network, the
> incumbents actively sabotage those attempts by any means they can.
> Monopolies are inherently customer-hostile, and arguments based on market
> forces fail in their presence.
> > >
> > > - Jonathan Morton
> > >
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Please send any postal/overnight deliveries to:
Vint Cerf
Google, LLC
1900 Reston Metro Plaza, 16th Floor
Reston, VA 20190
+1 (571) 213 1346


until further notice

[-- Attachment #1.2: Type: text/html, Size: 19686 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3995 bytes --]

Re: [Rpm] [Starlink] [Bloat] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

So, let me start wit a big caveat:
I am just an internet end-user and have no insight into the ISP side of things.
Nor am I a lawyer and hence might moss some of the subtleties of the regulation and their translation into law in each member state.



> On Sep 30, 2023, at 14:42, Vint Cerf <vint@google.com> wrote:
> 
> the phrase "treat equally" can (maybe should?) be interpreted as offering the same options for traffic handling to all parties on the same terms and conditions.

	[SM] Let me start with noting that, clearly selling internet access links with different maximum capacity is established practice in the EU (as in most/all other markets), and has been before and after the regulation was published and converted to national law in the member states*. The EU regulation contains some rules about how ISPs are allowed to market these speed/capacity numbers and which remedies end customers have when the ISP under-delivers (as well as making the national regulator responsible to create methods to actually assess achievable capacity). So on the consumer side this is pretty clear (and the regulations encompasses all parties buying access service as end-customers, so a content provider buying internet access will covered just as a privat hoisehold.

*) There are however unicorn ISPs like Switzerland's Init7 that offered 1Gbps, 10 Gbps or 25 Gbps symmetric internet access links for the same monthly price (they recently seemed to have dropped the 1 Gbps tier, but 10 or 25 still cost the same relative low CHF 64.75/month); the main difference is the initial cost differs (probably to cover the cost of the more expensive optics at the optical switch in the CO). But again that is clearly NOT the norm ;) and as much as would wish otherwise Switzerland is not a member of the EU... 



Here is the first clause of the preamble (see https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32015R2120 ):
(1)
This Regulation aims to establish common rules to safeguard equal and non-discriminatory treatment of traffic in the provision of internet access services and related end-users’ rights. It aims to protect end-users and simultaneously to guarantee the continued functioning of the internet ecosystem as an engine of innovation. Reforms in the field of roaming should give end-users the confidence to stay connected when they travel within the Union, and should, over time, become a driver of convergent pricing and other conditions in the Union.

making it clear that this mainly focusses on internet access (and mobile roaming, but that seems less relevant for this thread).

> If there is only one class of service, then equally is the only option.

This would likely be other/non-internet accesss or specialized services in the parlance of the regulation:

(16)
There is demand on the part of providers of content, applications and services to be able to provide electronic communication services other than internet access services, for which specific levels of quality, that are not assured by internet access services, are necessary. Such specific levels of quality are, for instance, required by some services responding to a public interest or by some new machine-to-machine communications services. Providers of electronic communications to the public, including providers of internet access services, and providers of content, applications and services should therefore be free to offer services which are not internet access services and which are optimised for specific content, applications or services, or a combination thereof, where the optimisation is necessary in order to meet the requirements of the content, applications or services for a specific level of quality. National regulatory authorities should verify whether and to what extent such optimisation is objectively necessary to ensure one or more specific and key features of the content, applications or services and to enable a corresponding quality assurance to be given to end-users, rather than simply granting general priority over comparable content, applications or services available via the internet access service and thereby circumventing the provisions regarding traffic management measures applicable to the internet access services.
(17)
In order to avoid the provision of such other services having a negative impact on the availability or general quality of internet access services for end-users, sufficient capacity needs to be ensured. Providers of electronic communications to the public, including providers of internet access services, should, therefore, offer such other services, or conclude corresponding agreements with providers of content, applications or services facilitating such other services, only if the network capacity is sufficient for their provision in addition to any internet access services provided. The provisions of this Regulation on the safeguarding of open internet access should not be circumvented by means of other services usable or offered as a replacement for internet access services. However, the mere fact that corporate services such as virtual private networks might also give access to the internet should not result in them being considered to be a replacement of the internet access services, provided that the provision of such access to the internet by a provider of electronic communications to the public complies with Article 3(1) to (4) of this Regulation, and therefore cannot be considered to be a circumvention of those provisions. The provision of such services other than internet access services should not be to the detriment of the availability and general quality of internet access services for end-users. In mobile networks, traffic volumes in a given radio cell are more difficult to anticipate due to the varying number of active end-users, and for this reason an impact on the quality of internet access services for end-users might occur in unforeseeable circumstances. In mobile networks, the general quality of internet access services for end-users should not be deemed to incur a detriment where the aggregate negative impact of services other than internet access services is unavoidable, minimal and limited to a short duration. National regulatory authorities should ensure that providers of electronic communications to the public comply with that requirement. In this respect, national regulatory authorities should assess the impact on the availability and general quality of internet access services by analysing, inter alia, quality of service parameters (such as latency, jitter, packet loss), the levels and effects of congestion in the network, actual versus advertised speeds, the performance of internet access services as compared with services other than internet access services, and quality as perceived by end-users.


As an non-jurist I would gather that (17) above can be a bit tricky to police, but as long as an ISP is not actively using such services to monetize giving users a remedy for their own congested normal internet access offering something like this is not an issue. Also while the regulation does not mention interconnection at all, the interpretation of the EU's regulatory agencies (BEREC) explicitly treats it as something different:
https://www.berec.europa.eu/sites/default/files/files/document_register_store/2022/6/BoR_%2822%29_81_Update_to_the_BEREC_Guidelines_on_the_Implementation_of_the_Open_Internet_Regulation.pdf
"	• CAPs are protected as end-users under the Regulation in so far as CAPs use an IAS to reach other end-users. However, some CAPs may also operate their own networks and, as part of that, have interconnection agreements with ISPs; the provision of interconnection is a distinct service from the provision of IAS.

	• NRAs may take into account the interconnection policies and practices of ISPs in so far as they have the effect of limiting the exercise of end-user rights under Article 3(1). For example, this may be relevant in some cases, such as if the interconnection is implemented in a way which seeks to circumvent the Regulation.9"

As long as ISP do not use interconnects to blatently violate the articles of the EU directive they are out of scope for this regulation. I believe the assessment is that there is a healthy enough market to self-control. I lack real data to be able to assess the validity of that judgement, but I am willing to accept it as probably correct ;).


> If there are multiple classes of service, then these could (should?) be available to all customers indiscriminately. For example, there might be several distinct services with different maximum bit rates;

	[SM] As I mention above, different access rates are fine with the regulation, as are different prices, and ISPs are also free (at least from the perspective of the 2015/2120) to set any prices and rates for their plans as they see fit (resulting in clearly different prices for similar conditions between different european countries, this is also not a problem for the regulation*)


*) This and other regulations however expect more or less that any union citizen will be able to get the same price from the same ISP in the same country, so TelakomA might charge me X in Germany and Y in Austria, both need to charge the same whether a customer is from Germany or Austria as long as the customer books in the same country. The EU hopes that prices will converge inside the union, but it does not enforce this.


> the higher rates possibly available for a higher charge. If there is discrimination, it should be on the basis of customer choice and not dictated by the provider. 
> 
> Is that consistent with the European interpretation?

	[SM] For end-customers plans, yes this is fully aligned with the directive, the idea, to put it bluntly, seems not have been to completely disrupt the existing market, but merely prohibit attempts of ISPs to unduly exploit their often near-monopoly on customer eye-balls*. And I would say this all in all has worked pretty well (again, maybe not perfectly, but I am fine with "good enough").

If you have time the whole thing is worth a read as are the implementation guidelines, independent on whether you finally agree or disagree with the position the EU has taken on this matter. It will however cost some time and is written in the kind of language that quickly makes me drowsy.

Regards
	Sebastian

*) And there was precedence of unsavory behavior by some ISPs so this thing was not coming out of the blue. But on the other hand the regulation has not magically stopped al unsavory ISP behavior either. I happen to think that all in all this is a decent piece of regulation not perfect, but plenty "good enough" solving a real emerging problem in a way that seems just to all sides.


P.S.: The regulation also contains a section about transparency (essentially ISPs are free what rates to offer, but once they offer a rate they are responsible to actually deliver on their offer, doing away with a lot of the "up to X" shenanigans some ISPs had been up to before) that initially caused ISP backlash when it was introduced in German law, but since then things have been pretty quiet, ISP simply got better at making sure they deliver or alternatively allow under-served customers to cancel contracts without penalty. A lot of bruhaha about nothing....

P.P.S.: The last case where the 2015/2120 was used was to prohibit mobile operators from using zero-rating, that is not accounting some traffic against the mobile data caps. But I am sure I am not doing justice to the subtlety of that case with a single sentence ;)

P.P.P.S.: The same EU having made a rather reasonable piece of regulation can at the same time come up with hare-brained ideas about having content providers pay for the network build-out in the EU. While I have sympathy for taxing big technology companies more similar to other companies and individuals in the individual countries they generate revenue**, I do not think that forcing "big tech" to fill the coffers of "old telco" is an endeavor anybody but "big telco" shareholders should fathom acceptable.

**) Without strong evidence, I believe this is currently not the case


> 
> v
> 
> 
> On Sat, Sep 30, 2023 at 1:19 PM Sebastian Moeller via Starlink <starlink@lists.bufferbloat.net> wrote:
> Hi Frantisek,
> 
> > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > 
> > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > https://www.martingeddes.com/1261-2/
> > 
> > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > 
> >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> 
>         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> 
> [...]
> (8)
> When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> (9)
> The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> (10)
> Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> (11)
> Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> (12)
> Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> [...]
> 
> There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> 
> There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 
> 
> Regards
>         Sebastian
> 
> P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> 
> 
> 
> > 
> > All the best,
> > 
> > Frank
> > 
> > Frantisek (Frank) Borsik
> > 
> >  
> > 
> > https://www.linkedin.com/in/frantisekborsik
> > 
> > Signal, Telegram, WhatsApp: +421919416714 
> > 
> > iMessage, mobile: +420775230885
> > 
> > Skype: casioa5302ca
> > 
> > frantisek.borsik@gmail.com
> > 
> > 
> > 
> > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > ok, lots and lots of great comments here for sure.  
> > 
> > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > 
> > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > 
> > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > 
> > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> > 
> > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > 
> > 
> > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> > 
> > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> > 
> > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > 
> > - Rich Brown
> > 
> > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > 
> > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > >> 
> > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > >> 
> > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > 
> > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > 
> > > 
> > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > 
> > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > 
> > > 
> > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > 
> > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > 
> > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > 
> > > 
> > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > 
> > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > 
> > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > 
> > > 
> > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > 
> > > - Jonathan Morton
> > > 
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> -- 
> Please send any postal/overnight deliveries to:
> Vint Cerf
> Google, LLC
> 1900 Reston Metro Plaza, 16th Floor
> Reston, VA 20190
> +1 (571) 213 1346
> 
> 
> until further notice
> 
> 
> 

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Mike,

[I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]


> On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> 
> First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better. 
> 
> As I read this thread and think about the coming debate in the U.S., two things come to mind:
> 
> 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.

	[SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).


*) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.

> 
> The question I'm thinking about is do we want an Internet where end user plans are divided up this way?

	[SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.

> And if not, is a NN regulation the right place to put those rules?

	[SM] Could well be, but depends on the framing, no?

> 
> 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection. 

	[SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime". 


> Are NN rules the right place to address this

	[SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...

> and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?

	[SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.


Regards
	Sebastian

P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).


> 
> 
> 
> On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> Hi Frantisek,
> 
> > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > 
> > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > https://www.martingeddes.com/1261-2/
> > 
> > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > 
> >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> 
>         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> 
> [...]
> (8)
> When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> (9)
> The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> (10)
> Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> (11)
> Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> (12)
> Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> [...]
> 
> There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> 
> There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ). 
> 
> Regards
>         Sebastian
> 
> P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> 
> 
> 
> > 
> > All the best,
> > 
> > Frank
> > 
> > Frantisek (Frank) Borsik
> > 
> >  
> > 
> > https://www.linkedin.com/in/frantisekborsik
> > 
> > Signal, Telegram, WhatsApp: +421919416714 
> > 
> > iMessage, mobile: +420775230885
> > 
> > Skype: casioa5302ca
> > 
> > frantisek.borsik@gmail.com
> > 
> > 
> > 
> > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > ok, lots and lots of great comments here for sure.  
> > 
> > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > 
> > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > 
> > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > 
> > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.   
> > 
> > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > 
> > 
> > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy. 
> > 
> > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.) 
> > 
> > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > 
> > - Rich Brown
> > 
> > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > 
> > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > >> 
> > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > >> 
> > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > 
> > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > 
> > > 
> > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > 
> > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > 
> > > 
> > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > 
> > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > 
> > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > 
> > > 
> > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > 
> > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > 
> > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > 
> > > 
> > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > 
> > > - Jonathan Morton
> > > 
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > 
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
> 
> _______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[Dave Taht]

The starlink list was not originally cc´d and yet since I think this
debate concerns that also, I have added the cc back. Carry on!

On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS
<libreqos@lists.bufferbloat.net> wrote:
>
> Hi Mike,
>
> [I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]
>
>
> > On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> >
> > First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better.
> >
> > As I read this thread and think about the coming debate in the U.S., two things come to mind:
> >
> > 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.
>
>         [SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
> During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).
>
>
> *) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.
>
> >
> > The question I'm thinking about is do we want an Internet where end user plans are divided up this way?
>
>         [SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.
>
> > And if not, is a NN regulation the right place to put those rules?
>
>         [SM] Could well be, but depends on the framing, no?
>
> >
> > 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection.
>
>         [SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
> This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime".
>
>
> > Are NN rules the right place to address this
>
>         [SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...
>
> > and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?
>
>         [SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.
>
>
> Regards
>         Sebastian
>
> P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).
>
>
> >
> >
> >
> > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> > Hi Frantisek,
> >
> > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > >
> > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > > https://www.martingeddes.com/1261-2/
> > >
> > > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > >
> > >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> >
> >         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> >
> > [...]
> > (8)
> > When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> > (9)
> > The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> > (10)
> > Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> > (11)
> > Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> > (12)
> > Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> > [...]
> >
> > There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> >
> > There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ).
> >
> > Regards
> >         Sebastian
> >
> > P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> >
> >
> >
> > >
> > > All the best,
> > >
> > > Frank
> > >
> > > Frantisek (Frank) Borsik
> > >
> > >
> > >
> > > https://www.linkedin.com/in/frantisekborsik
> > >
> > > Signal, Telegram, WhatsApp: +421919416714
> > >
> > > iMessage, mobile: +420775230885
> > >
> > > Skype: casioa5302ca
> > >
> > > frantisek.borsik@gmail.com
> > >
> > >
> > >
> > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > ok, lots and lots of great comments here for sure.
> > >
> > > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > >
> > > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > >
> > > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > >
> > > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.
> > >
> > > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > >
> > >
> > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy.
> > >
> > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.)
> > >
> > > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > >
> > > - Rich Brown
> > >
> > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > >
> > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > > >>
> > > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > > >>
> > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > >
> > > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > >
> > > >
> > > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > >
> > > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > >
> > > >
> > > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > >
> > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > >
> > > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > >
> > > >
> > > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > >
> > > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > >
> > > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > >
> > > >
> > > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > >
> > > > - Jonathan Morton
> > > >
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > >
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > Rpm mailing list
> > Rpm@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/rpm
>
> _______________________________________________
> LibreQoS mailing list
> LibreQoS@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/libreqos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [LibreQoS] [Bloat] [Starlink] net neutrality back in the news

[Dave Taht]

Despite the sturm und drang here if you google for network neutrality
there was a lot of press coverage 4 days ago.

... and mostly, silence, on the twitters at least. How is mastodon or
other social media?

I couldn´t help but notice that this was essentially, Diane
Feinstein´s last press release (she died yesterday):

https://www.feinstein.senate.gov/public/index.cfm/press-releases?id=C6FFF484-16F1-4CC9-9836-F36446C3B33D


On Sat, Sep 30, 2023 at 8:23 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> The starlink list was not originally cc´d and yet since I think this
> debate concerns that also, I have added the cc back. Carry on!
>
> On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS
> <libreqos@lists.bufferbloat.net> wrote:
> >
> > Hi Mike,
> >
> > [I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]
> >
> >
> > > On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> > >
> > > First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better.
> > >
> > > As I read this thread and think about the coming debate in the U.S., two things come to mind:
> > >
> > > 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.
> >
> >         [SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
> > During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).
> >
> >
> > *) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.
> >
> > >
> > > The question I'm thinking about is do we want an Internet where end user plans are divided up this way?
> >
> >         [SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.
> >
> > > And if not, is a NN regulation the right place to put those rules?
> >
> >         [SM] Could well be, but depends on the framing, no?
> >
> > >
> > > 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection.
> >
> >         [SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
> > This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime".
> >
> >
> > > Are NN rules the right place to address this
> >
> >         [SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...
> >
> > > and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?
> >
> >         [SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.
> >
> >
> > Regards
> >         Sebastian
> >
> > P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).
> >
> >
> > >
> > >
> > >
> > > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > Hi Frantisek,
> > >
> > > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > >
> > > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > > > https://www.martingeddes.com/1261-2/
> > > >
> > > > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > > >
> > > >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> > >
> > >         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> > >
> > > [...]
> > > (8)
> > > When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> > > (9)
> > > The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> > > (10)
> > > Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> > > (11)
> > > Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> > > (12)
> > > Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> > > [...]
> > >
> > > There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> > >
> > > There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ).
> > >
> > > Regards
> > >         Sebastian
> > >
> > > P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> > >
> > >
> > >
> > > >
> > > > All the best,
> > > >
> > > > Frank
> > > >
> > > > Frantisek (Frank) Borsik
> > > >
> > > >
> > > >
> > > > https://www.linkedin.com/in/frantisekborsik
> > > >
> > > > Signal, Telegram, WhatsApp: +421919416714
> > > >
> > > > iMessage, mobile: +420775230885
> > > >
> > > > Skype: casioa5302ca
> > > >
> > > > frantisek.borsik@gmail.com
> > > >
> > > >
> > > >
> > > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > ok, lots and lots of great comments here for sure.
> > > >
> > > > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > > > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > > >
> > > > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > > >
> > > > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > > >
> > > > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.
> > > >
> > > > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > > >
> > > >
> > > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > > > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy.
> > > >
> > > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.)
> > > >
> > > > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > > >
> > > > - Rich Brown
> > > >
> > > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > >
> > > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > > > >>
> > > > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > > > >>
> > > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > > >
> > > > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > > >
> > > > >
> > > > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > > >
> > > > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > > >
> > > > >
> > > > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > > >
> > > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > > >
> > > > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > > >
> > > > >
> > > > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > > >
> > > > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > > >
> > > > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > > >
> > > > >
> > > > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > > >
> > > > > - Jonathan Morton
> > > > >
> > > > > _______________________________________________
> > > > > Rpm mailing list
> > > > > Rpm@lists.bufferbloat.net
> > > > > https://lists.bufferbloat.net/listinfo/rpm
> > > >
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > >
> > > _______________________________________________
> > > Rpm mailing list
> > > Rpm@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/rpm
> >
> > _______________________________________________
> > LibreQoS mailing list
> > LibreQoS@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/libreqos
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos



--
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

[Rpm] New email list: NNagain for network neutrality

[Dave Taht]

I have created a new mailing list to discuss title ii and nn
regulation, and called it NNagain,
partially because Ronald Reagan´s old phrase "There you go again",
transliterated, has been an earworm in my head for a week now. (this
is not a + or - for title ii and nn, it´s just my earworm! now,
possibly yours)

I just issued everyone an invite to join.  While I suspect that many
users of my mailing lists are interested in this broad topic, I felt
that continuing to flood so many mailboxes with it to be a bit much,
and it is going to drag on for a long time.

I am interested in the broadest possible and *civil* discussion with
members of all internet-using professions. It would be nice if we had
a resident regulatory lawyer or three, from all over the world, we had
people building or managing networks, and those making software,
government folk, NGOs, name the speciality! all in the loop.

I imagine others will establish slack channels, or matrix, or
mastodon, and so forth, but in my mind, email remains the bedrock of
complex communications. I for one would like to really go in deep on
some complex regulations, and also into the history of what worked,
and what didn´t, reflect on a future in space, and internet expansion
throughout the world, debunk claims made by all sides, and try to come
up with a blueprint for a future internet better for everyone.

Somehow.

Please pass out the invite also especially to folk with a different
perspective or skillset you can *enjoy* constructively disagreeing
with - and possibly mutual enlightenment will happen!

The list url is: https://lists.bufferbloat.net/listinfo/nnagain

Thank you all for being such a great crowd to hang with.

On Sat, Sep 30, 2023 at 10:35 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> Despite the sturm und drang here if you google for network neutrality
> there was a lot of press coverage 4 days ago.
>
> ... and mostly, silence, on the twitters at least. How is mastodon or
> other social media?
>
> I couldn´t help but notice that this was essentially, Diane
> Feinstein´s last press release (she died yesterday):
>
> https://www.feinstein.senate.gov/public/index.cfm/press-releases?id=C6FFF484-16F1-4CC9-9836-F36446C3B33D
>
>
> On Sat, Sep 30, 2023 at 8:23 AM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > The starlink list was not originally cc´d and yet since I think this
> > debate concerns that also, I have added the cc back. Carry on!
> >
> > On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS
> > <libreqos@lists.bufferbloat.net> wrote:
> > >
> > > Hi Mike,
> > >
> > > [I took the liberty to remove some individual address from the Cc, as I assume most/all already be covered by the lists]
> > >
> > >
> > > > On Sep 30, 2023, at 16:41, Mike Conlow <mconlow@cloudflare.com> wrote:
> > > >
> > > > First, a thank you to Dave, and lots of you all, for longtime shepherding of this community and efforts to make the Internet better.
> > > >
> > > > As I read this thread and think about the coming debate in the U.S., two things come to mind:
> > > >
> > > > 1. Ofcom is considering a net neutrality "clarification". The first topic in the consultation is whether ISPs should be allowed to offer "premium quality retail plans". It doesn't specify the technical implementation, but there would be different plans for "users who mainly stream" vs "people who use high quality virtual reality applications". Apparently ISPs feel the existing NN rules are not clear on whether this is allowed.
> > >
> > >         [SM] Not sure this is not simply an attempt of using regulatory divergence from the EU (IMHO for no good reason or outcome)... Also und er the existing EU rules ISPs are arguably already free to treat the whole class of latency sensitive VR to lower delay than bulk streaming as long as they do son consistently and not based on commercial relationships with the senders...
> > > During the covid19 lock downs the EU offered clarification on the regulation that really drove home the do not discriminate inside of a specific traffic class, and define classes by purely technical not economical parameters. That said, I always like to look at data and hence am happy to the the UK apparently prepping to run that experiment (I am also happy not to live there right now not having to prticipate in said experiment*).
> > >
> > >
> > > *) Other than that the british islands offer a lot of really great places I certainly would like to live at, but I digress.
> > >
> > > >
> > > > The question I'm thinking about is do we want an Internet where end user plans are divided up this way?
> > >
> > >         [SM] Personally, I consider internet access infrastructure and do not think this looks like a good way forward.
> > >
> > > > And if not, is a NN regulation the right place to put those rules?
> > >
> > >         [SM] Could well be, but depends on the framing, no?
> > >
> > > >
> > > > 2. To the point in the PS of the below email, I would agree things are mostly working in the EU, and in the US. But things are broken in Germany to the point where consumers have a degraded Internet experience because their ISP won't provision enough interconnection.
> > >
> > >         [SM] This a very peculiar case of the local incumbent Deutsche Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in its network and tends to follow regulations to the letter (not however necessarily to the intent, but they are not different from other corporations of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that is, to the best of our knowledge they do not pay anybody for transit and peer with all other T1-ISPs, they also have a relative large share of eye-balls in one of Europes larger and profitable markets. They (as did AT&T and Verizon in the US and probably other ISPs in similar positions as well) that most of their users traffic was within network (e.g. from German companies hosted/homed by DTAG) or via important partners like Google that have decent peering links (unclear whether/if Google actually is charged for that) but that there is a considerable number of services that reach DTAG eye-balls via their transit, that is essentially via one of the other T1-ISPs (I simplify here, I have no insight in the actual bisiness relationships between all players). And now DTAG basically instructed its generally capable network team to simply manage the size of the peering links with the big transit-providers carefully so that they never fully clogg, but clearly see increased packet loss and queueing delay during prime time. That in turn is clearly a competition problem if streaming service A judders/jitters/and buffers jumps between quality tiers while streaming service B smoothly and boringly just streams at the desired resolution. Now Telekom is happy to offer service A a product they call "internet transit" but that is priced pretty high (I have seen some comparative numbers for transit pricing in Germany I am not permitted to share or reveal more about) so high in fact that no content provider that can afford more than a single transit provider would use for anything but reaching DTAG eye-balls or closely related ones (like in the past SwissCom).
> > > This behaviour is not s secret but evades regulatory action, because it does not openly violate the EU regulation which in the BEREC interpretation does not really cover the interconnection side. DTAG is careful enough to not purposefully target specific potential customers but simply treats all traffic coming in/out via "other transit than its own" as "has to tolerate overheated links during primetime".
> > >
> > >
> > > > Are NN rules the right place to address this
> > >
> > >         [SM] They could well be the actual text of the 2015/2120 does not make a distinction between access and interconnection. But this is a tricky field and will directly affect parts of larger ISP's core business so I do not see this happen in the EU anytime soon, unless ISPs like Telekom clearly abuses this in a way that is too obvious... ATM it is mostly telecom, but I believe any of the big old monopoly incumbents likely is big enough in its home market to pull of a stunt like this, so there is the potential of someone over doing it...
> > >
> > > > and make sure it doesn't happen in the US? Or is one bad actor across the EU and US the cost of doing business and the Internet ecosystem and "market" are mostly solving the issue?
> > >
> > >         [SM] As happy as I am to diss DTAG for that behavior (I am also happy to praise it in ears where it shows exemplary behavior) DTAG is not alone in that business acumen, I think that some of the big US telco's dod/do exactly the same, but unlike telekom I have no evidence.
> > >
> > >
> > > Regards
> > >         Sebastian
> > >
> > > P.S.: I was a customer at DTAG for several years and I did not notice the conscious under-peering with the other T1 ISPs in my day to day usage, so while the issue clearly and measurably exists it is not an issue that normal users will encounter often and are also unlikely to properly root-cause (the blame will likely land by my example service A above).
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > Hi Frantisek,
> > > >
> > > > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > >
> > > > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great body of work (IMHO) done on this subject by Martin Geddes:
> > > > > https://www.martingeddes.com/1261-2/
> > > > >
> > > > > But let's pick one report written by his colleagues and published by Ofcom (UK telecoms regulator):
> > > > >
> > > > >       • You cannot conflate ‘equality of [packet] treatment’ with delivering equality of [user application] outcomes. Only the latter matters, as ordinary users don’t care what happened to the packets in transit. Yet the relevant academic literature fixates on the local operation of the mechanisms (including Traffic Management), not their global aggregate effect.
> > > >
> > > >         [SM] The EU laid out pretty clear why they mandated the NN regulations in eu regulation 2015/2120:
> > > >
> > > > [...]
> > > > (8)
> > > > When providing internet access services, providers of those services should treat all traffic equally, without discrimination, restriction or interference, independently of its sender or receiver, content, application or service, or terminal equipment. According to general principles of Union law and settled case-law, comparable situations should not be treated differently and different situations should not be treated in the same way unless such treatment is objectively justified.
> > > > (9)
> > > > The objective of reasonable traffic management is to contribute to an efficient use of network resources and to an optimisation of overall transmission quality responding to the objectively different technical quality of service requirements of specific categories of traffic, and thus of the content, applications and services transmitted. Reasonable traffic management measures applied by providers of internet access services should be transparent, non-discriminatory and proportionate, and should not be based on commercial considerations. The requirement for traffic management measures to be non-discriminatory does not preclude providers of internet access services from implementing, in order to optimise the overall transmission quality, traffic management measures which differentiate between objectively different categories of traffic. Any such differentiation should, in order to optimise overall quality and user experience, be permitted only on the basis of objectively different technical quality of service requirements (for example, in terms of latency, jitter, packet loss, and bandwidth) of the specific categories of traffic, and not on the basis of commercial considerations. Such differentiating measures should be proportionate in relation to the purpose of overall quality optimisation and should treat equivalent traffic equally. Such measures should not be maintained for longer than necessary.
> > > > (10)
> > > > Reasonable traffic management does not require techniques which monitor the specific content of data traffic transmitted via the internet access service.
> > > > (11)
> > > > Any traffic management practices which go beyond such reasonable traffic management measures, by blocking, slowing down, altering, restricting, interfering with, degrading or discriminating between specific content, applications or services, or specific categories of content, applications or services, should be prohibited, subject to the justified and defined exceptions laid down in this Regulation. Those exceptions should be subject to strict interpretation and to proportionality requirements. Specific content, applications and services, as well as specific categories thereof, should be protected because of the negative impact on end-user choice and innovation of blocking, or of other restrictive measures not falling within the justified exceptions. Rules against altering content, applications or services refer to a modification of the content of the communication, but do not ban non-discriminatory data compression techniques which reduce the size of a data file without any modification of the content. Such compression enables a more efficient use of scarce resources and serves the end-users’ interests by reducing data volumes, increasing speed and enhancing the experience of using the content, applications or services concerned.
> > > > (12)
> > > > Traffic management measures that go beyond such reasonable traffic management measures may only be applied as necessary and for as long as necessary to comply with the three justified exceptions laid down in this Regulation.
> > > > [...]
> > > >
> > > > There really is little IMHO that can be brought against these, all pretty fair and reasonable. What it does is accept that internet access is essential infrastructure and that hence access needs to be as well regulated as access to water, electricity, gas, streets, ... . Yes this has some consequences of what ISPs can and can not do. But this is normal "cost of business". I for one am quite happy about this regulation existing as locally it did away with some (not all) shenanigans of some ISPs that were clearly not operating in the interest of their paying eye-balls.
> > > >
> > > > There is a whole cottage industry of consultants that decry the EU's decision and try to lobby against it, but honestly reading these mostly makes me think harsher regulation might be required (on consultans about how much they are allowed to massage the facts ;) ).
> > > >
> > > > Regards
> > > >         Sebastian
> > > >
> > > > P.S.: Of course if we look close enough we surely can find corner-cases where either the EU regulations or the translation into national law result in less desirable outcomes, but "nothing is perfect" and all in all the regulations seem to be "good enough". With the caveat that explicitly excluding ISP interconnect from the regulations BEREC essentially pointed the way for ISPs wanting to monetize their eye-balls twice to do so via interconnects, but that only works for the 800 pound gorillas and generally is not a game smaller ISPs can play. I do understand why BEREC wants to stay out of the interconnection issue, as this is rather complicated and the market seems to generally work okay-ish (that is not badly enough to make intervention a hot-button issue for voters and hence politicians).
> > > >
> > > >
> > > >
> > > > >
> > > > > All the best,
> > > > >
> > > > > Frank
> > > > >
> > > > > Frantisek (Frank) Borsik
> > > > >
> > > > >
> > > > >
> > > > > https://www.linkedin.com/in/frantisekborsik
> > > > >
> > > > > Signal, Telegram, WhatsApp: +421919416714
> > > > >
> > > > > iMessage, mobile: +420775230885
> > > > >
> > > > > Skype: casioa5302ca
> > > > >
> > > > > frantisek.borsik@gmail.com
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > > ok, lots and lots of great comments here for sure.
> > > > >
> > > > > bandwidth abundance:  Not for most people and ISPs.  The 'carriers' aren't carrying to many places at affordable rates.  I've pulled quotes from Lumen and Zayo at over $5k/month/gig.  We typically pay 900-1400 for a gig of service.  This isn't abundance and it's widespread and it leaves only major providers that can afford/amortize out 100G transits etc.
> > > > > My answer to this is one that Dave and I have bounced back and forth is the idea of micro IXs in every municipality and having that somehow tied into access to the ROW in counties etc.  Not fully hashed out, but the fiber is in the ground, it could be sold, but the carriers are not well incentivised to sell it.  It takes the better part of a year to get a DIA within 100ft of a Lumen hut sometimes...  Heck, it could even be a government program to get an μIX with x feet of every school, city hall, and library.  I don't care how it's done but this would get abundance NEAR end users and open up essentially every town to competition.
> > > > >
> > > > > monopoly.  This is a historical thing for most cable and DSL incumbents.  They have enjoyed virtual monopolies with cable owning population centers and DSL owning the outskirts and there is no product darwinism here where customer satisfaction is a pressure.  That may not be the future but it definitely is the past.  These companies may have to shift into customer satisfaction as a major part instead of a minor part of their corporate culture to fend off fttx and ultra-modern wisps.
> > > > >
> > > > > Starlink is not offering significant competition to major carriers.    Starlink's 1.5 million customers are at LEAST 90% pulled from other satellite services and small ISPs.  Spectrum and Comcast's losses to starlink are measured in decimal points.
> > > > >
> > > > > Only fttx and ultra-modern wireless tech really threatens these incumbents.  Typical wisps aren't putting a dent in these guys, just scraping the paint off their bumper.  We're pulling customers at the scale of 'dozens' for example.  Spectrum's management doesn't know we exist we're such a small threat to them.
> > > > >
> > > > > But to further the point here, these fttx and ultra-modern wisps can only exist in places where there is adequate carrier services to start with.  In areas where they spend the money and do the build but there aren't good carrier services, those fiber services suck and the wISPs start to claw back even with inferior technology.  We've pulled quite a few customers off fttx deployments because of this sort of situation.
> > > > >
> > > > >
> > > > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hanover@gmail.com> wrote:
> > > > > Thank you Jonathan for this clear description of the issues and their history. I wonder if there's a fourth one - privacy.
> > > > >
> > > > > Rosenworcel's talk https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out that ISPs might want to monetize our traffic patterns and location data. (This is less of an issue in the EU, but the US remains a Wild West in this regard.)
> > > > >
> > > > > I am hopeful that the FCC will include this in their NPRM (which must be available by now but I haven't looked...)
> > > > >
> > > > > - Rich Brown
> > > > >
> > > > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm <rpm@lists.bufferbloat.net> wrote:
> > > > > >
> > > > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat <bloat@lists.bufferbloat.net> wrote:
> > > > > >>
> > > > > >> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> > > > > >>
> > > > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.
> > > > > >
> > > > > > I think there were three more-or-less separate concerns which have, over time, fallen under the same umbrella:
> > > > > >
> > > > > >
> > > > > > 1:  Capacity-seeking flows tend to interfere with latency-sensitive flows, and the "induced demand" phenomenon means that increases in link rate do not in themselves solve this problem, even though they may be sold as doing so.
> > > > > >
> > > > > > This is directly addressed by properly-sized buffers and/or AQM, and even better by FQ and SQM.  It's a solved problem, so long as the solutions are deployed.  It's not usually necessary, for example, to specifically enhance service for latency-sensitive traffic, if FQ does a sufficiently good job.  An increased link rate *does* enhance service quality for both latency-sensitive and capacity-seeking traffic, provided FQ is in use.
> > > > > >
> > > > > >
> > > > > > 2:  Swarm traffic tends to drown out conventional traffic, due to congestion control algorithms which try to be more-or-less fair on a per-flow basis, and the substantially larger number of parallel flows used by swarm traffic.  This also caused subscribers using swarm traffic to impair the service of subscribers who had nothing to do with it.
> > > > > >
> > > > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, and I think it was occasionally used as an argument for *not* deploying FQ.  ISPs' initial response was to outright block swarm traffic where they could identify it, which was then softened to merely throttling it heavily, before NN regulations intervened.  Usage quotas also showed up around this time, and were probably related to this problem.
> > > > > >
> > > > > > This has since been addressed by several means.  ISPs may use FQ on a per-subscriber basis to prevent one subscriber's heavy traffic from degrading service for another.  Swarm applications nowadays tend to employ altruistic congestion control which deliberately compensates for the large number of flows, and/or mark them with one or more of the Least Effort class DSCPs.  Hence, swarm applications are no longer as damaging to service quality as they used to be.  Usage quotas, however, still remain in use as a profit centre, to the point where an "unlimited" service is a rare and precious specimen in many jurisdictions.
> > > > > >
> > > > > >
> > > > > > 3:  ISPs merged with media distribution companies, creating a conflict of interest in which the media side of the business wanted the internet side to actively favour "their own" media traffic at the expense of "the competition".  Some ISPs began to actively degrade Netflix traffic, in particular by refusing to provision adequate peering capacity at the nodes through which Netflix traffic predominated, or by zero-rating (for the purpose of usage quotas) traffic from their own media empire while refusing to do the same for Netflix traffic.
> > > > > >
> > > > > > **THIS** was the true core of Net Neutrality.  NN regulations forced ISPs to carry Netflix traffic with reasonable levels of service, even though they didn't want to for purely selfish and greedy commercial reasons.  NN succeeded in curbing an anti-competitive and consumer-hostile practice, which I am perfectly sure would resume just as soon as NN regulations were repealed.
> > > > > >
> > > > > > And this type of practice is just the sort of thing that technologies like L4S are designed to support.  The ISPs behind L4S actively do not want a technology that works end-to-end over the general Internet.  They want something that can provide a domination service within their own walled gardens.  That's why L4S is a NN hazard, and why they actively resisted all attempts to displace it with SCE.
> > > > > >
> > > > > >
> > > > > > All of the above were made more difficult to solve by the monopolistic nature of the Internet service industry.  It is actively difficult for Internet users to move to a truly different service, especially one based on a different link technology.  When attempts are made to increase competition, for example by deploying a publicly-funded network, the incumbents actively sabotage those attempts by any means they can.  Monopolies are inherently customer-hostile, and arguments based on market forces fail in their presence.
> > > > > >
> > > > > > - Jonathan Morton
> > > > > >
> > > > > > _______________________________________________
> > > > > > Rpm mailing list
> > > > > > Rpm@lists.bufferbloat.net
> > > > > > https://lists.bufferbloat.net/listinfo/rpm
> > > > >
> > > > > _______________________________________________
> > > > > Rpm mailing list
> > > > > Rpm@lists.bufferbloat.net
> > > > > https://lists.bufferbloat.net/listinfo/rpm
> > > > > _______________________________________________
> > > > > Rpm mailing list
> > > > > Rpm@lists.bufferbloat.net
> > > > > https://lists.bufferbloat.net/listinfo/rpm
> > > >
> > > > _______________________________________________
> > > > Rpm mailing list
> > > > Rpm@lists.bufferbloat.net
> > > > https://lists.bufferbloat.net/listinfo/rpm
> > >
> > > _______________________________________________
> > > LibreQoS mailing list
> > > LibreQoS@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/libreqos
> >
> >
> >
> > --
> > Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> > Dave Täht CSO, LibreQos
>
>
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]


Mike Conlow via Bloat <bloat@lists.bufferbloat.net> wrote:
    > As I read this thread and think about the coming debate in the U.S., two
    > things come to mind:

    > 1. Ofcom is considering
    > <https://www.ofcom.org.uk/consultations-and-statements/category-1/net-neutrality-review>
    > a net neutrality "clarification". The first topic in the consultation is
    > whether ISPs should be allowed to offer "premium quality retail plans". It
    > doesn't specify the technical implementation, but there would be different
    > plans for "users who mainly stream" vs "people who use high quality
    > virtual reality applications". Apparently ISPs feel the existing NN rules
    > are not clear on whether this is allowed.

    > The question I'm thinking about is do we want an Internet where end user
    > plans are divided up this way? And if not, is a NN regulation the right
    > place to put those rules?

Network Neutrality means that all senders are treated the same by the *ISP*
The ISP doesn't get to decide to prefer some peers over others.

It doesn't mean that the customer can't be given controls to determine what
traffic they want, and what priority they want to give it.

I think those two categories are totally bonkers.  I would never want to
subscribe to either service plan, because clearly the ISP thinks they can
just offload bufferbloat.   We've had protocols to classify traffic for
decades, but ISPs couldn't be bothered to figure out how to sell that.

    > 2. To the point in the PS of the below email, I would agree things are
    > mostly working in the EU, and in the US. But things
    > <https://twitter.com/j0xaf/status/850081406277619712> are

What's twitter?

    > Are NN rules the right place to address this and make sure it doesn't
    > happen in the US? Or is one bad actor across the EU and US the cost of
    > doing business and the Internet ecosystem and "market" are *mostly* solving
    > the issue?

The EU bureaucrats are mostly lost in some fantasy land.
I don't think it will end well.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: [Rpm] [Bloat] [Starlink] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi David,


> On Sep 29, 2023, at 00:19, David Lang via Rpm <rpm@lists.bufferbloat.net> wrote:
> 
> On Thu, 28 Sep 2023, Livingood, Jason via Bloat wrote:
> 
>> Date: Thu, 28 Sep 2023 20:48:58 +0000
>> From: "Livingood, Jason via Bloat" <bloat@lists.bufferbloat.net>
>> Reply-To: "Livingood, Jason" <Jason_Livingood@comcast.com>
>> To: dan <dandenson@gmail.com>, Dave Taht <dave.taht@gmail.com>
>> Cc: Rpm <rpm@lists.bufferbloat.net>,
>>    Dave Taht via Starlink <starlink@lists.bufferbloat.net>,
>>    bloat <bloat@lists.bufferbloat.net>,
>>    libreqos <libreqos@lists.bufferbloat.net>,
>>    Jamal Hadi Salim <jhs@mojatatu.com>
>> Subject: Re: [Bloat] [Starlink] [LibreQoS] [Rpm] net neutrality back in the
>>    news
>>> dan <dandenson@gmail.com> wrote:
>> 
>>> "(I assume most ISPs want happy customers)."
> 
>>> made me laugh a little.  'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers. 
>> 
>> In that context, happy customers stay longer (less churn) and spend more (upgrades, multiple services). And unhappy customers generate costs via disconnects (loss of revenue, costs to replace them with a new customer to just stay at the same subscriber levels), and costs via customer contacts (call center staff).
> 
> Except when you have a monopoly in an area, at which point the ability of customers to leave is minimal, and years of bad customer service means that people don't bother complaining, so the call center staffing costs are lower than they should be.
> 
>>> For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency...
>> 
>> The two thoughts your comments (thanks for the response BTW!) trigger are:
> 
>> 1 - Often regulation looks to the past - in this case maybe an era of bandwidth scarcity where prioritization may have mattered. I think we're in the midst of a shift into bandwidth abundance where priority does not matter. What will is latency/responsiveness, content/compute localization, reliability, consistency, security, etc.
> 
>> 2 - If an ISP blocked YouTube or Netflix, they'd incur huge customer care (contact) costs and would see people start to immediately shift to competitors (5G FWA, FTTP or DOCSIS, WISP, Starlink/LEO, etc.). It just does not seem like something that could realistically happen any longer in the US.
> 
> Dave T called out earlier that the rise of bittorrent was a large part of the inital NN discussion here in the US. But a second large portion was a money grab from ISPs thinking that they could hold up large paid websites (netflix for example) for additional fees by threatening to make their service less useful to their users (viewing their users as an asset to be marketed to the websites rather than customers to be satisfied by providing them access to the websites)
> 
> I don't know if a new round of "it's not fair that Netflix doesn't pay us for the bandwidth to service them" would fall flat at this point or not.

	[SM] In the EU we have this as a continuous lobbying effort by big incumbent ISPs (a move to have the large content providers (CAPs) shoulder their "fair" share of the cost of modernizing the networks*), why this flys with at least some EU politicians is that the intended payees of this scheme are all located outside the EU and hence will have little support by the EU citizenry... (The latter is IMHO not fully undeserved either, the days of "do no evil" are long behind us and big tech often forgets that we are all in this together, but I digress). In the EU one of these days such an effort might actually succeed, as much as I dislike this.




*) This argument about fairness is indeed made by the same ISPs that already charge their eye-ball customers for the same capacity they say they need to built with particpatoin of the CAPs


> 
> David Lang_______________________________________________
> Rpm mailing list
> Rpm@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/rpm

Re: [Rpm] [Starlink] [Bloat] [LibreQoS] net neutrality back in the news

[Gert Doering]

Hi,

On Fri, Sep 29, 2023 at 08:24:13AM +0200, Sebastian Moeller via Starlink wrote:
> 	[SM] In the EU we have this as a continuous lobbying effort by big incumbent ISPs (a move to have the large content providers (CAPs) shoulder their "fair" share of the cost of modernizing the networks*), why this flys with at least some EU politicians is that the intended payees of this scheme are all located outside the EU and hence will have little support by the EU citizenry... (The latter is IMHO not fully undeserved either, the days of "do no evil" are long behind us and big tech often forgets that we are all in this together, but I digress). In the EU one of these days such an effort might actually succeed, as much as I dislike this.

And then the local incumbent uses that line of argument to arm-twist
all the smaller ISPs to pay them for traffic into their network...
(and calling up fees well above normal market rates for "transit").

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

Re: [Rpm] [Starlink] [Bloat] [LibreQoS] net neutrality back in the news

[Sebastian Moeller]

Hi Gert,


> On Sep 29, 2023, at 08:31, Gert Doering <gert@space.net> wrote:
> 
> Hi,
> 
> On Fri, Sep 29, 2023 at 08:24:13AM +0200, Sebastian Moeller via Starlink wrote:
>> 	[SM] In the EU we have this as a continuous lobbying effort by big incumbent ISPs (a move to have the large content providers (CAPs) shoulder their "fair" share of the cost of modernizing the networks*), why this flys with at least some EU politicians is that the intended payees of this scheme are all located outside the EU and hence will have little support by the EU citizenry... (The latter is IMHO not fully undeserved either, the days of "do no evil" are long behind us and big tech often forgets that we are all in this together, but I digress). In the EU one of these days such an effort might actually succeed, as much as I dislike this.
> 
> And then the local incumbent uses that line of argument to arm-twist
> all the smaller ISPs to pay them for traffic into their network...
> (and calling up fees well above normal market rates for "transit").

	Indeed, but that only flies because the regulators so far only feel responsible for the end-customer to internet access provider links, and explicitly exempt AS interconnect from their regulatory efforts. Given how complicated this can become I have some sympathy for their position, the national incumbent however plays a somewhat dangerous game, if he makes things too obvious it will likely result in regulatory interventions. This is also why the product sold is not "access to our eye-balls" but access "to the whole internet, including our eye-balls" yet at a cost that nobody is likely to use to access anything but that ISPs eye-balls. As much as it pains me that is behavior not untypical for large corporations these days...

Regards
	Sebastian


> 
> Gert Doering
>        -- NetMaster
> -- 
> have you enabled IPv6 on something today...?
> 
> SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
> Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
> D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
> Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

Re: [Rpm] [Bloat] [LibreQoS] net neutrality back in the news

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

On Thu, 28 Sep 2023, dan via Bloat wrote:

> Common Carriers or rather, carrier class services for 'internet', should be
> completely neutral.  Packets are packets.  However, I think it's important
> to carve out methods to have dedicated links for real time flows at the
> carrier level.  I don't know what that model looks like exactly, but being
> too stubborn about purist NN principals could really hurt VoIP services if
> there aren't methods to handle that.  I guess I really am describing
> 'internet fast lanes' for certain classes of services that we deem
> important enough as a whole.  not individual ISPs deciding, but rather 'the
> will of the people' saying VoIP is more important than netflix, you can
> carve out dedicated capacity for that.

the fq_codel/cake approach violates the strictest interpretation of 'packets are 
packets' but diffentiates between well behaved and short flows and ill-behaved 
bulk flows. That is content and destination neutral, but prioritizing for a fair 
experience to all.

In theory, 'fast lanes' and QoS priorizations can make VoIP and similar work, in 
practice there are too many different apps behaving in too many different ways 
for anyone to fix the problem with static rules and prioritization.

make sure that you don't through out cake-like content neutral improvements in 
your quest for 'a packet is a packet' (and remember, some of the people 
interpreting/implementing your rules will have it in their interest to make it 
as painful for users as possible to be able to blame you for the problem, so 
you can't count on 'reasonable interpretation' of the rules)

David Lang

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Rpm] [Bloat] net neutrality back in the news

[Livingood, Jason]

On 9/28/23, 02:25, "Bloat on behalf of Sebastian Moeller via Bloat" <bloat-bounces@lists.bufferbloat.net <mailto:bloat-bounces@lists.bufferbloat.net> on behalf of bloat@lists.bufferbloat.net <mailto:bloat@lists.bufferbloat.net>> wrote:
> But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". 

It was more complicated than that. At least in cable networks it came at the end of single channel DOCSIS 2.0 devices, as cable upgraded to channel bonding in DOCSIS 3.0 - which to your point brought dramatic increases in capacity. That was happening at the same time P2P was becoming popular. On the other hand, customers were congesting their own connections pretty pervasively - trying to use VoIP while their P2P client was active. At the time the P2P protocol tended to be pretty aggressive and smothered real-time apps (if you gave the client 1 Mbps US it would use it, if 10 Mbps - used that 100% as well, 25 Mbps - same, etc.). 

The answer ended up being a mix of more capacity, apps being more responsive to other LAN demands, and then advancements in congestion control & queuing.  But there were many customers who were basically self-congesting w/P2P and VoIP running in their home. 

> but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)

Sure - for their own (nascent & small scale) digital voice products. But to 3rd party VoIP - no one then was really doing inter-domain DSCP marking end-to-end (still are not). The non-ISP offering here was the big driver of consumer complaints.

> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).

It's not made easier in the US where one can strongly support formal national NN rules but be against putting it under Title-II regulation vs. Title-I. IANAL so cannot properly explain the nuances. Ultimately legislation is the best solution, as this will just be in the courts for years, but our US Congress is not the most functional body these days and people have tried to do legislation for many years. 

Jason

Re: [Rpm] [Bloat] net neutrality back in the news

[Dave Taht]

I put it on hackernews: https://news.ycombinator.com/item?id=37694306

I too strongly support formal NN rules, and am in general, against
some but certainly not all of the title II regulation, and unlike
jason, perhaps, tend to want to supercede lawyers´ claims that it can
only be solved via legal means with somehow providing sufficient
technical advice and technology to make the truly needed things happen
faster. I disagree thoroughly with the call for 100/20 nationwide, as
one example that can be better addressed and more quickly, by fixing
bufferbloat, and shoring up those with 10/1 service or none, to a
level of 25/10.

It is hard to understand the renewed fuss and furor, when law meets
technical reality. "For a successful technology, reality must take
precedence over public relations, for Nature cannot be fooled" -
Richard Feynman. The only cynical observation I have, is that the
entire mess generates billable hours and clickbait.

https://www.vox.com/technology/2023/9/28/23893138/fcc-net-neutrality-returns

vs

https://www.lightreading.com/regulatory-politics/at-mwc-fcc-s-carr-denounces-net-neutrality-revival-as-a-dead-end

I really enjoyed reading "Jessica Rosenworcel´s" down to earth and
practical comments on tuesday:

https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf

particularly her admonishment for "no death threats this time¨, and
calls for moderation such as:

"They say Title II is heavy-handed. And if we were seeking comment on
applying Title II
to broadband in its entirety, they might have a point. But instead, we
are proposing a light touch.
Back in 2015, when the FCC last had net neutrality rules upheld by the
courts, the FCC chose to
forbear from 27 provisions of Title II of the Communications Act and
over 700 agency
regulations for broadband and broadband providers. We are sticking
with that approach.
They say this is a stalking horse for rate regulation. Nope. No how,
no way. We know
competition is the best way to bring down rates for consumers. And
approaches like the
Affordable Connectivity Program are the best bet for making sure
service is affordable for all.
They say nothing bad has happened. Again, states stepped into the void
created by the
FCC retreating from net neutrality. I think it is time for Washington
to step back in with a
national policy to make sure internet access is fast, open, and fair."


On Thu, Sep 28, 2023 at 10:10 AM Livingood, Jason
<Jason_Livingood@comcast.com> wrote:
>
> On 9/28/23, 02:25, "Bloat on behalf of Sebastian Moeller via Bloat" <bloat-bounces@lists.bufferbloat.net <mailto:bloat-bounces@lists.bufferbloat.net> on behalf of bloat@lists.bufferbloat.net <mailto:bloat@lists.bufferbloat.net>> wrote:
> > But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively".
>
> It was more complicated than that. At least in cable networks it came at the end of single channel DOCSIS 2.0 devices, as cable upgraded to channel bonding in DOCSIS 3.0 - which to your point brought dramatic increases in capacity. That was happening at the same time P2P was becoming popular. On the other hand, customers were congesting their own connections pretty pervasively - trying to use VoIP while their P2P client was active. At the time the P2P protocol tended to be pretty aggressive and smothered real-time apps (if you gave the client 1 Mbps US it would use it, if 10 Mbps - used that 100% as well, 25 Mbps - same, etc.).
>
> The answer ended up being a mix of more capacity, apps being more responsive to other LAN demands, and then advancements in congestion control & queuing.  But there were many customers who were basically self-congesting w/P2P and VoIP running in their home.
>
> > but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
>
> Sure - for their own (nascent & small scale) digital voice products. But to 3rd party VoIP - no one then was really doing inter-domain DSCP marking end-to-end (still are not). The non-ISP offering here was the big driver of consumer complaints.
>
> > ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
>
> It's not made easier in the US where one can strongly support formal national NN rules but be against putting it under Title-II regulation vs. Title-I. IANAL so cannot properly explain the nuances. Ultimately legislation is the best solution, as this will just be in the courts for years, but our US Congress is not the most functional body these days and people have tried to do legislation for many years.
>
> Jason
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [Rpm] [Bloat] net neutrality back in the news

[Sebastian Moeller]

Hi Jason,

thanks for giving some perspective.

> On Sep 28, 2023, at 19:10, Livingood, Jason <Jason_Livingood@comcast.com> wrote:
> 
> On 9/28/23, 02:25, "Bloat on behalf of Sebastian Moeller via Bloat" <bloat-bounces@lists.bufferbloat.net <mailto:bloat-bounces@lists.bufferbloat.net> on behalf of bloat@lists.bufferbloat.net <mailto:bloat@lists.bufferbloat.net>> wrote:
>> But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". 
> 
> It was more complicated than that. At least in cable networks it came at the end of single channel DOCSIS 2.0 devices, as cable upgraded to channel bonding in DOCSIS 3.0 - which to your point brought dramatic increases in capacity. That was happening at the same time P2P was becoming popular. On the other hand, customers were congesting their own connections pretty pervasively - trying to use VoIP while their P2P client was active. At the time the P2P protocol tended to be pretty aggressive and smothered real-time apps (if you gave the client 1 Mbps US it would use it, if 10 Mbps - used that 100% as well, 25 Mbps - same, etc.). 

	[SM] Thanks! I will adjust my position ("mainly too much oversubscription") to give at least equal weight to self-congesation by overly aggressive new application types, simply because you have likely seen internal data demonstrating that and I fully believe your summary.


> The answer ended up being a mix of more capacity, apps being more responsive to other LAN demands, and then advancements in congestion control & queuing.  But there were many customers who were basically self-congesting w/P2P and VoIP running in their home. 

	[SM] So is it fair to say that the problam was also caused by a rapidly change in usage profiles or was that the smaller problem, if at all a problem?


> 
>> but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)
> 
> Sure - for their own (nascent & small scale) digital voice products. But to 3rd party VoIP - no one then was really doing inter-domain DSCP marking end-to-end (still are not). The non-ISP offering here was the big driver of consumer complaints.

	[SM] I accept that this was not a clear and simple solution, just that it would have been technically possible (just infeasible as it likely would have required multiple SLA's between all pairs of partner's to "secure" the full path?


>> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).
> 
> It's not made easier in the US where one can strongly support formal national NN rules but be against putting it under Title-II regulation vs. Title-I. IANAL so cannot properly explain the nuances.

	[SM] Same here, the legal nuances are lost on me as well, especially about noticeably different legal systems like the US' compared to home (were I am also not a legal expert, but were I have read a few laws and regulations). Also I assume that the whole issue would look quite different from an ISP perspective, nobody likes having their option space restricted for the acts of a few bad apples...


> Ultimately legislation is the best solution, as this will just be in the courts for years, but our US Congress is not the most functional body these days and people have tried to do legislation for many years. 

	[SM] Proper legislation has the advantage of offering a fair playing field for all ISPs, and after the unavoidable initial tire kicking by the courts will hopefully be robust and reliable. 
> 
> Jason
> 

Re: [Rpm] [EXTERNAL] Re: [Bloat] net neutrality back in the news

[Livingood, Jason]

On 9/28/23, 16:06, "Sebastian Moeller" <moeller0@gmx.de <mailto:moeller0@gmx.de>> wrote:
>> The answer ended up being a mix of more capacity, apps being more responsive to other LAN demands, and then advancements in congestion control & queuing. But there were many customers who were basically self-congesting w/P2P and VoIP running in their home. 

> [SM] So is it fair to say that the problam was also caused by a rapidly change in usage profiles or was that the smaller problem, if at all a problem?

It was certainly a problem. Like I said, a combination of factors. Certainly though, P2P (and it's associated aggressive underlying protocol) was huge - estimated at 70% of traffic volume in 2006. https://www.plagiarismtoday.com/2017/06/01/the-long-slow-decline-of-bittorrent/

Side note - a lot of IETF work flowed from the P2P workshop the IETF held in response to all this, including new WGs - LEDBAT, ALTO, CDNI.

JL

PS - some of the P2P workshop papers would be interesting to read again if we could find them:

[1] Nick Weaver - The case for "Ugly Now" User Fairness

[2] Paul Jessop - Position paper of the RIAA

[3] Nikloaos Laotaris, Pablo Rodriguez, Laurent Massoulie - ECHOES:
Edge Capacity Hosting Overlays of Nano Data Centers

[4] Bruce Davie, Stefano Previdi, Jan Medved, Albert Tian - Peer
Selection Guidance

[5] Marie-Jose Montpetit - Community Networks: getting P2P out of prison
- the next steps

[6] D. Bryan, S. Dawkins, B. Lowekamp, E. Shim - Infrastructure-related
Attributes of App Scenarios for P2PSIP

[7] Jiang XingFeng - Analysis of the Service Discovery in DHT network

[8] R. Penno - P2P Status and Requirements

[9] Patrick Crowley and Shakir James - Symbiotic P2P: Resolving the
conflict between ISPs and BitTorrent through mutual cooperation

[10] Robb Topolski  - Framing Peer to Peer File Sharing

[11] M. Stiemerling, S. Niccolini, S. Kiesel, J. Seedorf - A Network
Cooperative Overlay System

[12] Y. Wang, S. Tan, R. Grove - Traffic Localization with Multi-Layer,
Tracker-Based Peer-to-Peer Content Distribution Architecture

[13] Haiyong Xie, Y. Richard Yang, Avi Silberschatz, Arvind
Krishnamurthy, Laird Popkin - P4P: Provider Portal for P2P Applications

[14] Michael Merritt, Doug Pasko, Laird Popkin - Network-Friendly
Peer-to-Peer Services

[15] Camiant (Jackson) - Camiant Submission

[16] Jason Livingood, Rich Woundy - Comcast Submission

[17] Benny Rodrig - Enterprise IP Networks and the P2P Traffic Load
Impact

[18] Ted Hardie - Peer-to-Peer traffic and "Unattended Consequences"

[19] Jiang XingFeng, Ning Zong - Content Replication for Internet P2P
Applications

[20] Sandvine (Dundas) - Analysis of Traffic Demographics in Broadbank
networks

[21] Sandvine (Dundas) - Traffic  Management in a World with Network
Neutrality

[22] Stanislav Shalunov - Users want P2P, we make it work

[23] R. Cuevas, A. Cuevas, I. Martinez-Yelmo, C. Guerrero - Internet
scale mobility service: a case study on building a DHT based service for
ISPs

[24] M. Barnes, B. McCormick - Peer to Peer Infrastructure
Considerations

[25] Henning Schulzrinne - Encouraging Bandwidth Efficiency for
Peer-to-Peer Applications

[26] Damien Saucez, Benoit Donnet, Olivier Bonaventure, Dimitri
Papdimitriou - Towards an Open Path Selection Architecture

[27] Eric Rescorla - Notes on P2P Blocking and Evasion

[28] Vinay Aggrawal, Anja Feldmann - ISP-Aided Neighbor Selection in P2P
Systems

[29] Enrico Marocco, Vijay K. Gurbani, Volker Hilt, Ivica Rimac, Marco
Tomsu - Peer-to-Peer Infrastructure: A Survey of Research on the
Application-Layer Traffic Optimization Problem and the Need for Layer
Cooperation

[30] Tony Moncaster, Bob Briscoe, Louise Burness - Is There a Problem
With Peer-to-peer Traffic?

[31] David Sohn, Alissa Cooper - Peer-to-Peer Infrastructure
Considerations

[32] Bob Briscoe, Lou Burness, Tony Moncaster, Phil Eardley - Solving
this traffic management problem... and the next, and the next

Re: [Rpm] [EXTERNAL] Re: [Bloat] net neutrality back in the news

[Livingood, Jason]

I forgot to add - the workshop has a great summary at https://www.rfc-editor.org/rfc/rfc5594.html

On 9/28/23, 17:07, "Livingood, Jason" <Jason_Livingood@cable.comcast.com <mailto:Jason_Livingood@cable.comcast.com>> wrote:


On 9/28/23, 16:06, "Sebastian Moeller" <moeller0@gmx.de <mailto:moeller0@gmx.de> <mailto:moeller0@gmx.de <mailto:moeller0@gmx.de>>> wrote:
>> The answer ended up being a mix of more capacity, apps being more responsive to other LAN demands, and then advancements in congestion control & queuing. But there were many customers who were basically self-congesting w/P2P and VoIP running in their home. 


> [SM] So is it fair to say that the problam was also caused by a rapidly change in usage profiles or was that the smaller problem, if at all a problem?


It was certainly a problem. Like I said, a combination of factors. Certainly though, P2P (and it's associated aggressive underlying protocol) was huge - estimated at 70% of traffic volume in 2006. https://www.plagiarismtoday.com/2017/06/01/the-long-slow-decline-of-bittorrent/ <https://www.plagiarismtoday.com/2017/06/01/the-long-slow-decline-of-bittorrent/>


Side note - a lot of IETF work flowed from the P2P workshop the IETF held in response to all this, including new WGs - LEDBAT, ALTO, CDNI.


JL


PS - some of the P2P workshop papers would be interesting to read again if we could find them:


[1] Nick Weaver - The case for "Ugly Now" User Fairness


[2] Paul Jessop - Position paper of the RIAA


[3] Nikloaos Laotaris, Pablo Rodriguez, Laurent Massoulie - ECHOES:
Edge Capacity Hosting Overlays of Nano Data Centers


[4] Bruce Davie, Stefano Previdi, Jan Medved, Albert Tian - Peer
Selection Guidance


[5] Marie-Jose Montpetit - Community Networks: getting P2P out of prison
- the next steps


[6] D. Bryan, S. Dawkins, B. Lowekamp, E. Shim - Infrastructure-related
Attributes of App Scenarios for P2PSIP


[7] Jiang XingFeng - Analysis of the Service Discovery in DHT network


[8] R. Penno - P2P Status and Requirements


[9] Patrick Crowley and Shakir James - Symbiotic P2P: Resolving the
conflict between ISPs and BitTorrent through mutual cooperation


[10] Robb Topolski - Framing Peer to Peer File Sharing


[11] M. Stiemerling, S. Niccolini, S. Kiesel, J. Seedorf - A Network
Cooperative Overlay System


[12] Y. Wang, S. Tan, R. Grove - Traffic Localization with Multi-Layer,
Tracker-Based Peer-to-Peer Content Distribution Architecture


[13] Haiyong Xie, Y. Richard Yang, Avi Silberschatz, Arvind
Krishnamurthy, Laird Popkin - P4P: Provider Portal for P2P Applications


[14] Michael Merritt, Doug Pasko, Laird Popkin - Network-Friendly
Peer-to-Peer Services


[15] Camiant (Jackson) - Camiant Submission


[16] Jason Livingood, Rich Woundy - Comcast Submission


[17] Benny Rodrig - Enterprise IP Networks and the P2P Traffic Load
Impact


[18] Ted Hardie - Peer-to-Peer traffic and "Unattended Consequences"


[19] Jiang XingFeng, Ning Zong - Content Replication for Internet P2P
Applications


[20] Sandvine (Dundas) - Analysis of Traffic Demographics in Broadbank
networks


[21] Sandvine (Dundas) - Traffic Management in a World with Network
Neutrality


[22] Stanislav Shalunov - Users want P2P, we make it work


[23] R. Cuevas, A. Cuevas, I. Martinez-Yelmo, C. Guerrero - Internet
scale mobility service: a case study on building a DHT based service for
ISPs


[24] M. Barnes, B. McCormick - Peer to Peer Infrastructure
Considerations


[25] Henning Schulzrinne - Encouraging Bandwidth Efficiency for
Peer-to-Peer Applications


[26] Damien Saucez, Benoit Donnet, Olivier Bonaventure, Dimitri
Papdimitriou - Towards an Open Path Selection Architecture


[27] Eric Rescorla - Notes on P2P Blocking and Evasion


[28] Vinay Aggrawal, Anja Feldmann - ISP-Aided Neighbor Selection in P2P
Systems


[29] Enrico Marocco, Vijay K. Gurbani, Volker Hilt, Ivica Rimac, Marco
Tomsu - Peer-to-Peer Infrastructure: A Survey of Research on the
Application-Layer Traffic Optimization Problem and the Need for Layer
Cooperation


[30] Tony Moncaster, Bob Briscoe, Louise Burness - Is There a Problem
With Peer-to-peer Traffic?


[31] David Sohn, Alissa Cooper - Peer-to-Peer Infrastructure
Considerations


[32] Bob Briscoe, Lou Burness, Tony Moncaster, Phil Eardley - Solving
this traffic management problem... and the next, and the next

Re: [Rpm] [Bloat] [EXTERNAL] Re: net neutrality back in the news

[Erik Auerswald]

Hi,

On Thu, Sep 28, 2023 at 09:07:23PM +0000, Livingood, Jason via Bloat wrote:
> […]
> Side note - a lot of IETF work flowed from the P2P workshop the IETF
> held in response to all this, including new WGs - LEDBAT, ALTO, CDNI.
> 
> JL
> 
> PS - some of the P2P workshop papers would be interesting to read again if we could find them:

The wiki link in RFC 5594 <https://www.rfc-editor.org/rfc/rfc5594.html> seems to have been lost, but archive.org has
at least some of the contents, see, e.g.,
<https://web.archive.org/web/20170101000000*/http://trac.tools.ietf.org/area/rai/trac/wiki/PeerToPeerInfrastructure>.

> [1] Nick Weaver - The case for "Ugly Now" User Fairness

https://web.archive.org/web/20090721015803/http://trac.tools.ietf.org/area/rai/trac/attachment/wiki/PeerToPeerInfrastructure/1%20p2pi-weaver.txt

> [2] Paul Jessop - Position paper of the RIAA

That's missing from archive.org.

> [3] Nikloaos Laotaris, Pablo Rodriguez, Laurent Massoulie - ECHOES:
> Edge Capacity Hosting Overlays of Nano Data Centers
> […]

I did not look for the other position papers.

Cheers,
Erik
-- 
If things aren’t simple, they won’t work.
                        -- Radia Perlman