<div dir="ltr">"(I assume most ISPs want happy customers)."<br>made me laugh a little. 'Most' by quantity of businesses maybe, but 'most' in terms of customers being served by puts the Spectrums and Comcasts in the mix (in the US) and they don't care about happy customers they care about defacto monopolies in markets so that they don't have to care about happy customers. That might not be their motivation, but 30 years of their behavior makes it clear that it's the case.<br><br><br>From my perspective, there should be a clear separation between carriers and last mile delivery.. Even if it's the same company, the rules defining each really should be different.<br><br>Common Carriers or rather, carrier class services for 'internet', should be completely neutral. Packets are packets. However, I think it's important to carve out methods to have dedicated links for real time flows at the carrier level. I don't know what that model looks like exactly, but being too stubborn about purist NN principals could really hurt VoIP services if there aren't methods to handle that. I guess I really am describing 'internet fast lanes' for certain classes of services that we deem important enough as a whole. not individual ISPs deciding, but rather 'the will of the people' saying VoIP is more important than netflix, you can carve out dedicated capacity for that.<br><br>For the last mile, I'm actually less concerned with pure NN and more concerned with no-blocking or 'brand' prioritization and required/label transparency. ie, if an ISP wants to sell a DPI's QoE'd service that says 'videos at 1080p' and that is on the product label clearly and not in small print, then that's fine so long as 'videos' is agnostic of the vendor INCLUDING the ISP's own video products. "100Mbps internet services with 1080P video $65" for example. "Upgrade to our 100M with 4k video for +$10/m". With that level of transparency I think that consumer protections are well handled. And, because of that transparency, normal market capitalism mechanisms work. I can say "100Mbps for $65 and full resolution video" for example. Or "100Mbps Net Neutral service". The secret DPI's QoE shaping is my main concern here, and where I think consumer protection needs to be pursued.<br><br>Again however, I think that ISPs should be able to offer dedicated paths or bandwidth for specific services like VoIP. Services listed in a publicly determined products list. <div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 28, 2023 at 1:40 PM Dave Taht via LibreQoS <<a href="mailto:libreqos@lists.bufferbloat.net">libreqos@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">@Sebastian: This is a really great list of what the issues were in the<br>
EU, and if y'all can repost there, rather than here, perhaps more<br>
light will be generated outside our circles.<br>
<br>
<a href="https://news.ycombinator.com/item?id=37694306#37694307" rel="noreferrer" target="_blank">https://news.ycombinator.com/item?id=37694306#37694307</a><br>
<br>
On Thu, Sep 28, 2023 at 12:31 PM Sebastian Moeller <<a href="mailto:moeller0@gmx.de" target="_blank">moeller0@gmx.de</a>> wrote:<br>
><br>
><br>
><br>
> > On Sep 28, 2023, at 18:38, Dave Taht <<a href="mailto:dave.taht@gmail.com" target="_blank">dave.taht@gmail.com</a>> wrote:<br>
> ><br>
> > On Wed, Sep 27, 2023 at 11:25 PM Sebastian Moeller <<a href="mailto:moeller0@gmx.de" target="_blank">moeller0@gmx.de</a>> wrote:<br>
> >><br>
> >> Hi Dave,<br>
> >><br>
> >> please excuse a number of tangents below ;)<br>
> ><br>
> > It would be nice, if as a (dis)organisation... the bufferbloat team<br>
> > could focus on somehow getting both sides of the network neutrality<br>
> > debate deeplying understanding the technological problem their<br>
> > pre-conceptions face, and the (now readily available and inexpensive)<br>
> > solutions that could be deployed, by most ISPs, over a weekend. We are<br>
> > regularly bringing up a few thousand people a week on libreqos (that<br>
> > we know of), and then of course, there are all the home routers and<br>
> > CPE that are increasingly capable of doing the right thing.<br>
> ><br>
> > The time to try and shift the memes in in the coming days, and weeks.<br>
> ><br>
> > So ya'all being distracting below... aggh... ok, I'll bite.<br>
><br>
> [SM] Sorry to drag you into the weeds...<br>
><br>
><br>
> ><br>
> >><br>
> >><br>
> >>> On Sep 27, 2023, at 20:21, Dave Taht via Rpm <<a href="mailto:rpm@lists.bufferbloat.net" target="_blank">rpm@lists.bufferbloat.net</a>> wrote:<br>
> >>><br>
> >>> Jason just did a beautiful thread as to what was the original source<br>
> >>> of the network neutrality<br>
> >>> bittorrent vs voip bufferbloat blowup.<br>
> >>><br>
> >>> <a href="https://twitter.com/jlivingood/status/1707078242857849244" rel="noreferrer" target="_blank">https://twitter.com/jlivingood/status/1707078242857849244</a><br>
> >><br>
> >> But the core issue IMHO really was an economic one, the over-subscription ratios that worked before torrenting simply did not cut it any more in an environment when customers actually tried to use their contracted access rates "quantitatively". In my outsider perspective an ISP owes its customers the contracted rates (with some slack*) and any sort of over-subscription is a valid economic optimization an ISP might take, IFF that ISP is prepared to rapidly increase segment capacity (or down-grade customer plans)) if the deployed over-subscription rate proves to have been too optimistic. Mind you, most ISPs market plans by access speed (and charge more for higher speeds) and hence are somewhat responsible to actually deliver (again with some slack) these speeds.<br>
> ><br>
> > The average use at peak hours for home broadband is below 5mbits per<br>
> > subscriber regardless of on a 25mbit or gbit plan. Remarkably,<br>
> > business plans are less (but more bursty). Oversubscription within a<br>
> > set of parameters is needed because, in part because upstream<br>
> > bandwidth to the internet remains expensive (although I hope that cost<br>
> > continues to drop rapidly), and in part, because it is needlessly<br>
> > expensive to provision exactly the contracted rate to the customer. As<br>
> > one example, I use the inverse square law as a rule of thumb for<br>
> > wireless deployments - the range you can get from a 25/10 deployment<br>
> > is geometrically better than 100/25, and the average bandwidth usage<br>
> > nearly the same.<br>
><br>
> [SM] +1; hence my argument is not oversubscription per se is bad, but that oversubscription needs to be managed and the level of oversubscription needs to to be adapted ot he actual usage patterns. I am sure however that ISPs already actively do that (I assume most ISPs want happy customers).<br>
><br>
><br>
> > Agreeing on industry standards for what the "slack" should be, might be of help.<br>
><br>
> [SM] Not being part of that industry at all I would love to see that as well, but can not contribute to defining that in any way.<br>
><br>
><br>
> ><br>
> >><br>
> >><br>
> >> *) Claiming "Up to", only carries that far, if you sell me X and I mostly get Y (with Y close to X) and occasionally Z (with Z << X), that is acceptable unless occasionally is "at every late afternoon and evening" prime-time.<br>
> ><br>
> > We have discussed elsewhere, the idea of a minimum contracted rate<br>
> > (down to), which is easier to reason about.<br>
><br>
> [SM] Indeed, the German regulator (and I am not saying this is the only or best option) decided to require ISPs to give 6 numbers pre-contract signing: 3 for up- and 3 for downstream: a maximal (net) rate, a minimal (net) rate, and a usually achievable (net) rate, all rates were defined as IP/TCP goodput to make verification easier on end-users. The regulator also defined a measurement regime that end-users can follow to check whether the ISP is fulfilling the contract and the law gives remedies if ISPs do not deliver (either the right to immediately cancel the contract and/or the right to adapt the monthly payments to the actually delivered ratio of the contracted rates*). I think I need to add, that ISPs can set these numbers freely, but are only allowed to advertise with the maximum rate.<br>
> But if we talk about a single number per direction, minimal rate is probably easiest, or something like a "usually achievable rate" (that needs to be met or exceeded in say 90% of >= 20 measurements or so).<br>
><br>
><br>
> *) In a ironic twist however the regulator so far has not explained how deviations in each of the 6 numbers above should be aggregated to get one single contract rate achievement ratio..., most ISPs took measures into their own hands and simply offer customers typically a permanent rebate of 5 EUR or immediate cancelation what ever the customer prefers....<br>
><br>
><br>
> ><br>
> >><br>
> >>><br>
> >>> Seeing all the political activity tied onto it since (and now again)<br>
> >>> reminds of two families at war about an incident that had happened<br>
> >>> generations and generations before, where the two sides no longer<br>
> >>> remembered why they hated each other so, but just went on hating, and<br>
> >>> not forgiving, and not moving on.<br>
> >>><br>
> >>> Yes, there are entirely separate and additional NN issues, but the<br>
> >>> technical problem of providing common carriage between two very<br>
> >>> different network application types (voip/gaming vs file transfer) is<br>
> >>> thoroughly solved now,<br>
> >><br>
> >> I am not sure this was at the core of the problem, my take is really that "always-on" and relative upload-heavy torrent simply demonstrated painfully for all involved that the old oversubscription ratios were not suitable for the changed traffic profiles. I have some sympathy for the ISPs here, they certainly did not try to sell their customers short (good ISPs try to retain their customers and that works best when customers are happy with the service) and having this problem appear on many segments at the same time is not a fun place to be, and upload was/is often (too) low in DOCSIS segments anyway; but this is why e.g. my bit torrent could affect your VoIP, simply by pushing the whole segment into upload capacity congestion... (ISPs in theory could fix this by plain old QoS engineering, but the issue at hand was with a non-ISP VoIP/SIP service and there QoS becomes tricky if the ISP as these packets need to be identified in a way that is not game-able**)<br>
><br>
> [SM] See my later mail to Jason, I will not claim I know Comcast's issues better than him and will accept that self-congestion also played a major role in the initial problem. Over here in Europe the net neutrality debate was kicked of less by an unfortunate confluence of new usage profiles and traditional oversibscription ratios and less than ideal packet scheduling, but rather by a series of pretty apparent attempts at restricting consumer choice, see eg. <a href="https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf" rel="noreferrer" target="_blank">https://www.accessnow.org/wp-content/uploads/archive/docs/Net_Neutrality_Ending_Network_Discrimination_in_Europe_Access_v3.pdf</a> for an admitted slightly biased position:<br>
><br>
> " • Blocking of applications and services: In order to maximise profits, some ISPs – that also offer their own services and applications online – exclude certain services and applications of competing market players. The most prominent case of this form of network discrimination is European mobile providers (like Deutsche Telekom) blocking or restricting the use of Voice over IP (VoIP) services (like Skype and Viber) for their customers.20<br>
><br>
> • Slowing or “throttling” internet speeds: Some ISPs slow down specific services (like YouTube) and applications (like Skype), or ask users to pay an extra fee to have access to these internet platforms. Given the high latency (delay) sensitivity of many applications, ISPs are able to compromise the correct functioning of these services by slowing them down, preventing the services from running properly. Often ISPs – especially telecommunication companies – do this to favour their own voice calling services over VoIP services, thereby crushing competition.<br>
><br>
><br>
> • Blocking websites: ISPs often block websites for a number of reasons – to secure their network, or to avoid competition, and sometimes for social, public relations or political reasons. In the UK, for instance, Orange Telecom blocked the French digital rights advocacy group, La Quadrature du Net’s website on pre-paid mobile accounts.21<br>
><br>
> • Preferential treatment of services and platforms: ISPs can also impose data caps on internet access contracts while granting data allowance exceptions to a company’s own proprietary streaming services (like Deutsche Telekom to its own “T-Entertain”).22 They can (and do) also grant preferential treatment to select services – such as Orange France with the popular music streaming service Deezer23 – ahead of other competitors, effectively imposing anti-competitive limitations on markets such as those for legal online music. Moreover, generally only large, well-established companies can afford this preferential treatment, resulting in a further stifling of innovation."<br>
><br>
> These look like clearer attempts to monetize the ISP position as gate-keeper to his customer's eye-balls (for content providers) as well as gate-keeper to the internet for for paying customers.<br>
> I find it much harder to have sympathy for the listed examples of ISP behavior than the situation of rapidly changing usage pattern posed where technical changes needed to be made, but where no attempt at unfair monetization was evident, but maybe I am overly sensitive. These are all examples that make me personally applaud the EU for its intervention to make clear rules what "internet access providers" can and can not do. (The EU also was quite flexible in applying/interpreting its rules during the covid isolation periods, when it made it clear that e.g. treating certain traffic classes e.g. streaming media to lower priority than video conferences was within the neutrality framework as long as it was based on traffic class and not on specific service providers IIRC).<br>
><br>
><br>
><br>
> ><br>
> > Torrent problem thoroughly solved with FQ on the path and backpressure<br>
> > from the mac scheduler.<br>
> ><br>
> > <a href="https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf" rel="noreferrer" target="_blank">https://perso.telecom-paristech.fr/drossi/paper/rossi14comnet-b.pdf</a><br>
><br>
> [SM] Thanks for the link. This is mainly for the self-congesrion case?<br>
><br>
><br>
> ><br>
> >> I agree that on a single link we mostly solved the problem (a few corner cases are left on links with very low capacity, where essentially we can only manage the pain, not remove it)...<br>
> >><br>
> >><br>
> >> **) Which is not rocket science either, a VoIP stream takes ~100 Kbps, so in theory an ISP might simply allow each customer say 5 VoIP stream equivalents by allowing up to 500Kbps od traffic marked with a specific DSCP as higher priority (with higher access probability for the shared medium). I am not sayng this is my preferred solution, just saying this is a solution that would have been available at the time if I memorize my docsis capabilities correctly.<br>
> >><br>
> >><br>
> >>> and if only all sides recognised at least this<br>
> >>> much, and made peace over it, and worked together to deploy those<br>
> >>> solutions, maybe, just maybe, we could find mutually satisfactory<br>
> >>> solutions to the other problems that plague the internet today, like<br>
> >>> security, and the ipv6 rollout.<br>
> >><br>
> >> +1. IPv6 is IMHO a prime example where the regulators should stop talking softly and start showing the big stick they carry.<br>
> ><br>
> > I really would like to see a push for IPv6 mandated as a part of the<br>
> > BEAD programs.<br>
><br>
> [SM] +1; I am not the biggest IPv6 fan, but that's what we have and it is mostly serviceable so let's get this "party started" finally.<br>
><br>
> ><br>
> >> Heck in Germany we have ISPs that still only supply CG-NATed IPv4 addresses only.. (most mass market ISPs do much better in that regard, but for the stragglers it would help if the regulator would demand IPv6 with PD***). Regarding security, the easiest way to achieve that would be to put some heavy requirements on IoT manufacturers and vendors (like do what you please as long as you are local LAN only, but once you reach out into the cloud you need to fulfill the following list of requirements, with timely security updates over a reasonable long usage period), however that might not be very attractive for politicians/regulators to tackle (active regulatory acts tends to get bad press unless something bad happened, but even then the press often complains about the acts coming too late, but I digress****)<br>
> ><br>
> > There is a separate NRPM going on for the new cybersecurity label at<br>
> > the FCC. If I had time, and a partner,<br>
> > we could rework the doc we did a few years ago. It is due oct 6.<br>
> ><br>
> >><br>
> >><br>
> >> ***) Strictly speaking IPv6 is required, since "internet access" is defined as reaching all of the internet (as far as in the ISPs power) and IPv6-only sites are not reachable for the CG-NAT-only customers. But so far the local regulator does not seem to enforce that requirement, or hopefully is working on this quietly behind the curtains.<br>
> >><br>
> >> ****) This is not to diss the press, they are doing what they are supposed to do, but it just shows that active regulation is a tricky business, and a light touch typically "looks better" (even though I see no real evidence it actually works better).<br>
> >><br>
> >>> If anyone here knows anyone more political, still vibrating with 10+<br>
> >>> years of outrage about NN on this fronts, on one side or the other, if<br>
> >>> you could sit them down, over a beer, and try to explain that at the<br>
> >>> start it was a technical problem nobody understood at the time, maybe<br>
> >>> that would help.<br>
> >><br>
> >> So in the EU that debate is essentially settled, there is a EU regulation that essentially spills out what ISPs owe their customers and that has become the law of the land. The rationale for required un-biased service and freedom to select terminal devices is well justified by the market ideals of the EU, allowing ISPs to discriminate packets or terminal devices restricts the market and will lead to undesired outcomes. (Fun fact most big players in capitalist societies argue for "free markets" but at the same time act to work-around the market mechanism by trying to move the market into an oligo- or even monopoly condition, which is why strong regulation is required*****).<br>
> ><br>
> > Governments make safer markets feasible.<br>
><br>
> [SM] Yes, I agree, both in markets are a pretty decent tool, but need constant maintenance.<br>
><br>
> Regards & Sorry for the tangent<br>
> Sebastian<br>
><br>
> ><br>
> >><br>
> >> *****) This is akin to professional sports where the audience generally accepts that referees are necessary and occasionally need to make "painful" calls, as the alternative would be anarchy, but I digress.<br>
> >><br>
> >> Regards<br>
> >> Sebastian<br>
> >><br>
> >><br>
> >>><br>
> >>> --<br>
> >>> Oct 30: <a href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html" rel="noreferrer" target="_blank">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a><br>
> >>> Dave Täht CSO, LibreQos<br>
> >>> _______________________________________________<br>
> >>> Rpm mailing list<br>
> >>> <a href="mailto:Rpm@lists.bufferbloat.net" target="_blank">Rpm@lists.bufferbloat.net</a><br>
> >>> <a href="https://lists.bufferbloat.net/listinfo/rpm" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/rpm</a><br>
> >><br>
> ><br>
> ><br>
> > --<br>
> > Oct 30: <a href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html" rel="noreferrer" target="_blank">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a><br>
> > Dave Täht CSO, LibreQos<br>
> > <2510.jpeg><br>
><br>
<br>
<br>
-- <br>
Oct 30: <a href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html" rel="noreferrer" target="_blank">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a><br>
Dave Täht CSO, LibreQos<br>
_______________________________________________<br>
LibreQoS mailing list<br>
<a href="mailto:LibreQoS@lists.bufferbloat.net" target="_blank">LibreQoS@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/libreqos" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/libreqos</a><br>
</blockquote></div>