[Bloat] Fwd: [iccrg] Musings on the future of Internet Congestion Control

[Bloat] Fwd: [iccrg] Musings on the future of Internet Congestion Control

[Dave Taht]

---------- Forwarded message ---------
From: Michael Welzl <michawe@ifi.uio.no>
Date: Wed, Jun 15, 2022 at 1:02 AM
Subject: [iccrg] Musings on the future of Internet Congestion Control
To: <iccrg@irtf.org>
Cc: Peyman Teymoori <peymant@ifi.uio.no>, Md Safiqul Islam
<safiquli@ifi.uio.no>, Hutchison, David <d.hutchison@lancaster.ac.uk>,
Stein Gjessing <steing@ifi.uio.no>


Dear ICCRGers,

We just got a paper accepted that I wanted to share:
Michael Welzl, Peyman Teymoori, Safiqul Islam, David Hutchison, Stein
Gjessing: "Future Internet Congestion Control: The Diminishing
Feedback Problem", accepted for publication in IEEE Communications
Magazine, 2022.

The preprint is available at:
https://arxiv.org/abs/2206.06642
I thought that it could provoke an interesting discussion in this group.

Figures 4 and 5 in this paper show that, across the world, network
links do not just become "faster”: the range between the low end and
the high end grows too.
This, I think, is problematic for a global end-to-end standard - e.g.,
it means that we cannot simply keep scaling IW along forever (or, if
we do, utilization will decline more and more).

So, we ask: what is the way ahead?  Should congestion control really
stay end-to-end?

Cheers,
Michael

_______________________________________________
iccrg mailing list
iccrg@irtf.org
https://www.irtf.org/mailman/listinfo/iccrg


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

I might be out to lunch here, but why not accept a "total" speed limit per TCP flow and simply expect bulk transfers to employ more parallel streams; which is what I think download manager apps are already doing for a long time?

And if we accept an upper ceiling per TCP flow we should be able to select a reasonable upper bound for the initial window as well, no?



> On Jun 15, 2022, at 19:49, Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> ---------- Forwarded message ---------
> From: Michael Welzl <michawe@ifi.uio.no>
> Date: Wed, Jun 15, 2022 at 1:02 AM
> Subject: [iccrg] Musings on the future of Internet Congestion Control
> To: <iccrg@irtf.org>
> Cc: Peyman Teymoori <peymant@ifi.uio.no>, Md Safiqul Islam
> <safiquli@ifi.uio.no>, Hutchison, David <d.hutchison@lancaster.ac.uk>,
> Stein Gjessing <steing@ifi.uio.no>
> 
> 
> Dear ICCRGers,
> 
> We just got a paper accepted that I wanted to share:
> Michael Welzl, Peyman Teymoori, Safiqul Islam, David Hutchison, Stein
> Gjessing: "Future Internet Congestion Control: The Diminishing
> Feedback Problem", accepted for publication in IEEE Communications
> Magazine, 2022.
> 
> The preprint is available at:
> https://arxiv.org/abs/2206.06642
> I thought that it could provoke an interesting discussion in this group.
> 
> Figures 4 and 5 in this paper show that, across the world, network
> links do not just become "faster”: the range between the low end and
> the high end grows too.
> This, I think, is problematic for a global end-to-end standard - e.g.,
> it means that we cannot simply keep scaling IW along forever (or, if
> we do, utilization will decline more and more).
> 
> So, we ask: what is the way ahead?  Should congestion control really
> stay end-to-end?

	Do we really have any other option? It is the sender that decides how much to dup into the network after all. Sure the network could help by giving some information back as a hint (say a 4bit value encoding the maximum relative queue-fill level measured along the full one-way path) but in the end, unless the network is willing to police its idea about acceptable send behavior it is still the sender's decision what tho send when, no?
Given the discussion about L4S and FQ it seems clear that the "network" is not prepared to implement anything close to what is required to move congestion control into the network... I have a feeling though that I am missing your point and am barking up the wrong tree ;)

Regards
	Sebastian


> 
> Cheers,
> Michael
> 
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg
> 
> 
> -- 
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 3674 bytes --]



> On Jun 19, 2022, at 6:53 PM, Sebastian Moeller via Bloat <bloat@lists.bufferbloat.net> wrote:
> 
> I might be out to lunch here, but why not accept a "total" speed limit per TCP flow and simply expect bulk transfers to employ more parallel streams; which is what I think download manager apps are already doing for a long time?
> 
> And if we accept an upper ceiling per TCP flow we should be able to select a reasonable upper bound for the initial window as well, no?

Using multiple flows is a way to do it, albeit not a very good way (better to use a better congestion control than just run multiple instances - but of course, one works with what one can - a download manager is on the receiver side and can achieve this there). This is not related to the IW issue which is relevant for short flows, which are the most common type of traffic by far (a point that our paper makes, along with many prior publications).


>> On Jun 15, 2022, at 19:49, Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> ---------- Forwarded message ---------
>> From: Michael Welzl <michawe@ifi.uio.no>
>> Date: Wed, Jun 15, 2022 at 1:02 AM
>> Subject: [iccrg] Musings on the future of Internet Congestion Control
>> To: <iccrg@irtf.org>
>> Cc: Peyman Teymoori <peymant@ifi.uio.no>, Md Safiqul Islam
>> <safiquli@ifi.uio.no>, Hutchison, David <d.hutchison@lancaster.ac.uk>,
>> Stein Gjessing <steing@ifi.uio.no>
>> 
>> 
>> Dear ICCRGers,
>> 
>> We just got a paper accepted that I wanted to share:
>> Michael Welzl, Peyman Teymoori, Safiqul Islam, David Hutchison, Stein
>> Gjessing: "Future Internet Congestion Control: The Diminishing
>> Feedback Problem", accepted for publication in IEEE Communications
>> Magazine, 2022.
>> 
>> The preprint is available at:
>> https://arxiv.org/abs/2206.06642
>> I thought that it could provoke an interesting discussion in this group.
>> 
>> Figures 4 and 5 in this paper show that, across the world, network
>> links do not just become "faster”: the range between the low end and
>> the high end grows too.
>> This, I think, is problematic for a global end-to-end standard - e.g.,
>> it means that we cannot simply keep scaling IW along forever (or, if
>> we do, utilization will decline more and more).
>> 
>> So, we ask: what is the way ahead? Should congestion control really
>> stay end-to-end?
> 
> 	Do we really have any other option? It is the sender that decides how much to dup into the network after all. Sure the network could help by giving some information back as a hint (say a 4bit value encoding the maximum relative queue-fill level measured along the full one-way path) but in the end, unless the network is willing to police its idea about acceptable send behavior it is still the sender's decision what tho send when, no?

In a scenario where a connection-splitting PEP is installed before a lower-capacity downstream path segment, this PEP can already ask for more data today.  It’s doing it in an ugly way, by “cheating” TCP, which yields various disadvantages… so I’d say that this is part of the problem. PEPs exist, yet have to do things poorly because they are treated as if they shouldn’t exist, and so they become unpopular for, well, having done things poorly...


> Given the discussion about L4S and FQ it seems clear that the "network" is not prepared to implement anything close to what is required to move congestion control into the network... I have a feeling though that I am missing your point and am barking up the wrong tree ;)

I guess you are. This is about middleboxes doing much “heavier” stuff.

Cheers,
Michael


[-- Attachment #2: Type: text/html, Size: 9434 bytes --]

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi Michael,

so I reread your paper and stewed a bit on it.


I believe that I do not buy some of your premises.

e.g. you write:

"We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."

And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes. I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?

And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy). The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between. I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.

I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance. But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.

Regards
	Sebastian


> On Jun 20, 2022, at 14:58, Michael Welzl <michawe@ifi.uio.no> wrote:
> 
> 
> 
>> On Jun 19, 2022, at 6:53 PM, Sebastian Moeller via Bloat <bloat@lists.bufferbloat.net> wrote:
>> 
>> I might be out to lunch here, but why not accept a "total" speed limit per TCP flow and simply expect bulk transfers to employ more parallel streams; which is what I think download manager apps are already doing for a long time?
>> 
>> And if we accept an upper ceiling per TCP flow we should be able to select a reasonable upper bound for the initial window as well, no?
> 
> Using multiple flows is a way to do it, albeit not a very good way (better to use a better congestion control than just run multiple instances - but of course, one works with what one can - a download manager is on the receiver side and can achieve this there). This is not related to the IW issue which is relevant for short flows, which are the most common type of traffic by far (a point that our paper makes, along with many prior publications).
> 
> 
>>> On Jun 15, 2022, at 19:49, Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote:
>>> 
>>> ---------- Forwarded message ---------
>>> From: Michael Welzl <michawe@ifi.uio.no>
>>> Date: Wed, Jun 15, 2022 at 1:02 AM
>>> Subject: [iccrg] Musings on the future of Internet Congestion Control
>>> To: <iccrg@irtf.org>
>>> Cc: Peyman Teymoori <peymant@ifi.uio.no>, Md Safiqul Islam
>>> <safiquli@ifi.uio.no>, Hutchison, David <d.hutchison@lancaster.ac.uk>,
>>> Stein Gjessing <steing@ifi.uio.no>
>>> 
>>> 
>>> Dear ICCRGers,
>>> 
>>> We just got a paper accepted that I wanted to share:
>>> Michael Welzl, Peyman Teymoori, Safiqul Islam, David Hutchison, Stein
>>> Gjessing: "Future Internet Congestion Control: The Diminishing
>>> Feedback Problem", accepted for publication in IEEE Communications
>>> Magazine, 2022.
>>> 
>>> The preprint is available at:
>>> https://arxiv.org/abs/2206.06642
>>> I thought that it could provoke an interesting discussion in this group.
>>> 
>>> Figures 4 and 5 in this paper show that, across the world, network
>>> links do not just become "faster”: the range between the low end and
>>> the high end grows too.
>>> This, I think, is problematic for a global end-to-end standard - e.g.,
>>> it means that we cannot simply keep scaling IW along forever (or, if
>>> we do, utilization will decline more and more).
>>> 
>>> So, we ask: what is the way ahead? Should congestion control really
>>> stay end-to-end?
>> 
>> 	Do we really have any other option? It is the sender that decides how much to dup into the network after all. Sure the network could help by giving some information back as a hint (say a 4bit value encoding the maximum relative queue-fill level measured along the full one-way path) but in the end, unless the network is willing to police its idea about acceptable send behavior it is still the sender's decision what tho send when, no?
> 
> In a scenario where a connection-splitting PEP is installed before a lower-capacity downstream path segment, this PEP can already ask for more data today.  It’s doing it in an ugly way, by “cheating” TCP, which yields various disadvantages… so I’d say that this is part of the problem. PEPs exist, yet have to do things poorly because they are treated as if they shouldn’t exist, and so they become unpopular for, well, having done things poorly...
> 
> 
>> Given the discussion about L4S and FQ it seems clear that the "network" is not prepared to implement anything close to what is required to move congestion control into the network... I have a feeling though that I am missing your point and am barking up the wrong tree ;)
> 
> I guess you are. This is about middleboxes doing much “heavier” stuff.
> 
> Cheers,
> Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Michael Welzl]

Hi !


> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Michael,
> 
> so I reread your paper and stewed a bit on it.

Many thanks for doing that!  :)


> I believe that I do not buy some of your premises.

you say so, but I don’t really see much disagreement here. Let’s see:


> e.g. you write:
> 
> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
> 
> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.

You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.


> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?

There are various ways to do this; one is to cache information and re-use it, assuming that - at least sometimes - new flows will see the same path again. Another is to let parallel flows share information. Yet another is to just be blindly more aggressive. Yet another, chirping.


> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).

You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place? Why do people buy these boxes?


> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.

Truth is, PEPs are used a lot: at cellular edges, at satellite links…   because the network is *not* always doing reasonably well without them.


> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.

That’s to do with scalability, which depends on how close to the network’s edge one is.


> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.

That could well be one way to go. Nice if we provoked you to think!


> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.

Yeah you do  :-)

Cheers,
Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi Michael,


> On Jul 10, 2022, at 22:01, Michael Welzl <michawe@ifi.uio.no> wrote:
> 
> Hi !
> 
> 
>> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Michael,
>> 
>> so I reread your paper and stewed a bit on it.
> 
> Many thanks for doing that! :)
> 
> 
>> I believe that I do not buy some of your premises.
> 
> you say so, but I don’t really see much disagreement here. Let’s see:
> 
> 
>> e.g. you write:
>> 
>> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
>> 
>> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.
> 
> You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.

	[SM] No IMHO the underutilization is the direct consequence of requiring a gradual filling of the "pipes" to probe he available capacity. I see no way how this could be done differently with the traffic sources/sinks being uncoordinated entities at the edge, and I see no way of coordinating all end points and handle all paths. In other words, we can fine tune a parameters to tweak the probing a bit, make it more or less aggressive/fast, but the fact that we need to probe capacity somehow means underutilization can not be avoided unless we find a way of coordinating all of the sinks and sources. But being sufficiently dumb, all I can come up with is an all-knowing oracle or faster than light communication, and neither strikes me to be realistic ;)


> 
> 
>> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?
> 
> There are various ways to do this; one is to cache information and re-use it, assuming that - at least sometimes - new flows will see the same path again.

	[SM] And equally important, that a flow's capacity share along a path did not change be other flows appearing on the same path. This is a case of speculation which depending on link and path type will work out well more or less often, the question then becomes is the improvement on successful speculation worth the cost of unsuccessful speculation (mostly the case where the estimate is wildly above the path capacity). Personally I think that having each flow start searching achievable capacity from the "bottom" seems more robust and reliable. I would agree though that better managing the typical overshoot of slow start is a worthy goal (one if tackled successfully might allow a faster capacity search approach).

> Another is to let parallel flows share information.

	[SM] Sounds sweet, but since not even two back-to-back packets send over the internet from A to B are guaranteed to take exactly the same path, confirming that flows actually share a sufficiently similar path seems tricky. Also stipulating two flows actually share a common path say over a capacity limiting node we have say 99 other flows and our parallel already established flow in equilibrium, now our new flow probably could start with an CWND close to the established flows. But if the bottleneck is fully occupied with our parallel established flow the limit would be 50% of the existing flow's rate, but only if that flow actually has enough time to give way...

Could you elaborate how that could work, please?

> Yet another is to just be blindly more aggressive.

	[SM] Sure, works if the "cost" of admitting too much data is acceptable, alas from an end users perspective I know I have flows where I do not care much if the overcommit and start throttling themselves (think background bulk transfer) but where I would get unhappy if their over aggression would interfere with other more important to me flows (that is a part why i am a happy flow queueing user, FQ helps a lot in isolating the fall-out from overly aggressive flows mainly to themselves).


> Yet another, chirping.

	[SM] I would love for that to work, but I have seen no convincing data yet demonstrating that over the existing internet, however we know already from other papers, that inter-packet delay is a somewhat unreliable estimator for capacity, so using that, even in clever ways requires some accumulation and smoothing, so I wonder how much faster/better this is actually going compared to existing slow start with a sufficiently high starting IW? In a meta criticism way, I am somewhat surprised how little splash paced chirping seems to be making given how positive its inventors presented it, might be the typical inertia of the field or an indication that PC might not yet be ready for show-time. However, if you think of somethnig else than paced chirping here, could you share a reference, please?


> 
> 
>> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).
> 
> You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place?

	[SM] I would guess because "marketing" was unhappy with "engineering" emphasizing the side-effects/potential problems and focussed in the best-case scenario? ;)

> Why do people buy these boxes?

	[SM] Because e.g. for GEO links, latency is in a range where default unadulterated TCP will likely choke on itself, and when faced with requiring customers to change/tune TCPs or having "PEP" fudge it, ease of use of fudging won the day. That is a generous explanation (as this fudging is beneficial to both the operator and most end-users), I can come up with less charitable theories if you want ;) .

>> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.
> 
> Truth is, PEPs are used a lot: at cellular edges, at satellite links… because the network is *not* always doing reasonably well without them.

	[SM] Fair enough, I accept that there are use cases for those, but again, only if the actually enhance the "experience" will users be happy to accept them. The goals of the operators and the paying customers are not always aligned here, a PEP might be advantageous more to the operator than the end-user (theoretically also the other direction, but since operators pay for PEPs they are unlikely to deploy those) think mandatory image recompression or forced video quality downscaling.... (and sure these are not as clear as I pitched them, if after an emergency a PEP allows most/all users in a cell to still send somewhat degraded images that is better than the network choking itself with a few high quality images, assuming images from the emergency are somewhat useful).

>> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.
> 
> That’s to do with scalability, which depends on how close to the network’s edge one is.

	[SM] I have heard the alternative that it has to do with what operators of core-links request from their vendors and what features they are willing to pay for... but this is very anecdotal as I have little insight into big-iron vendors or core-link operators. 

>> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.
> 
> That could well be one way to go. Nice if we provoked you to think!

	[SM] You mostly made me realize what the recent increases in IW actually aim to accomplish ;) and that current slow start seems actually better than its reputation; it solves a hard problem surprisingly well. The max(pat_queue%) idea has been kicking around in my head ever since reading a paper about storing queue occupancy into packets to help CC along (sorry, do not recall the authors or the title right now) so that is not even my own original idea, but simply something I borrowed from smarter engineers simply because I found the data convincing and the theory sane. (Also because I grudgingly accept that latency increases measured over the internet are a tad too noisy to be easily useful* and too noisy for a meaningful controller based on the latency rate of change**)

>> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.
> 
> Yeah you do :-)

	[SM] Less than you let on ;). If L4S gets ratified (increasingly likely, mostly for political*** reasons) it gets considerably harder to get yet another queue size related bits into the IP header...


Regards
	Sebastian

*) Participating in discussions about using active latency measurements to adapt traffic shapers for variable rate links which exposes quite a number of latency and throughput related issues, albeit for me on an amateurs level of understanding: https://github.com/lynxthecat/CAKE-autorate

**) I naively think that to make slow-start exist gracefully we need a quick and reliable measure for pre-congestion, and latency increases are so noisy that neither quick nor reliable can be achieved, let alone both at the same time.


***) Well aware that "political" is a "problematic" word in view of the IETF, but L4S certainly will not be ratified on its merits, because these have not (yet?) been conclusively demonstrated; not ruling out the merits can not be realized, just that currently there is not sufficient hard data to make a reasonable prediction.



> 
> Cheers,
> Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Michael Welzl]

Hi Sebastian,

Neither our paper nor me are advocating one particular solution - we point at a problem and suggest that research on ways to solve the under-utilization problem might be worthwhile.
Jumping from this to discussing the pro’s and con’s of a potential concrete solution is quite a leap…

More below:


> On Jul 10, 2022, at 11:29 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> Hi Michael,
> 
> 
>> On Jul 10, 2022, at 22:01, Michael Welzl <michawe@ifi.uio.no> wrote:
>> 
>> Hi !
>> 
>> 
>>> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> so I reread your paper and stewed a bit on it.
>> 
>> Many thanks for doing that! :)
>> 
>> 
>>> I believe that I do not buy some of your premises.
>> 
>> you say so, but I don’t really see much disagreement here. Let’s see:
>> 
>> 
>>> e.g. you write:
>>> 
>>> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
>>> 
>>> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.
>> 
>> You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.
> 
> 	[SM] No IMHO the underutilization is the direct consequence of requiring a gradual filling of the "pipes" to probe he available capacity. I see no way how this could be done differently with the traffic sources/sinks being uncoordinated entities at the edge, and I see no way of coordinating all end points and handle all paths. In other words, we can fine tune a parameters to tweak the probing a bit, make it more or less aggressive/fast, but the fact that we need to probe capacity somehow means underutilization can not be avoided unless we find a way of coordinating all of the sinks and sources. But being sufficiently dumb, all I can come up with is an all-knowing oracle or faster than light communication, and neither strikes me to be realistic ;)

There’s quite a spectrum of possibilities between an oracle or “coordinating all of the sinks and sources” on one hand, and quite “blindly” probing from a constant IW on the other. The “fine tuning” that you mention is interesting research, IMO!


>>> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?
>> 
>> There are various ways to do this

[snip: a couple of concrete suggestions from me, and answers about what problems they might have, with requests for references from you]

I’m sorry, but I wasn’t really going to have a discussion about these particular possibilities. My point was only that many possible directions exist - being completely “blind” isn’t the only possible approach.
Instead of answering your comments to my suggestions, let me give you one single concrete piece here: our reference 6, as one example of the kind of resesarch that we consider worthwhile for the future:

"X. Nie, Y. Zhao, Z. Li, G. Chen, K. Sui, J. Zhang, Z. Ye, and D. Pei, “Dynamic TCP initial windows and congestion control schemes through reinforcement learning,” IEEE JSAC, vol. 37, no. 6, 2019.”
https://1989chenguo.github.io/Publications/TCP-RL-JSAC19.pdf

This work learns a useful value of IW over time, rather than using a constant. One author works at Baidu, the paper uses data from Baidu, and it says:
"TCP-RL has been deployed in one of the top global search engines for more than a year. Our online and testbed experiments show that for short flow transmission, compared with the common practice of IW = 10, TCP-RL can reduce the average transmission time by 23% to 29%.”

- so it’s probably fair to assume that this was (and perhaps still is) active in Baidu.


>>> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).
>> 
>> You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place?
> 
> 	[SM] I would guess because "marketing" was unhappy with "engineering" emphasizing the side-effects/potential problems and focussed in the best-case scenario? ;)

It appears that you want to just ill-talk PEPs. There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification, etc.  PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.

You’re bound to ask me for concrete things next, and if I give you something concrete (e.g., a paper on PEPs), you’ll find something bad about it - but this is not a constructive direction of this conversation. Please note that I’m not saying “PEPs are always good”: I only say that, in my personal opinion, they’re a worthwhile direction of future research. That’s a very different statement.


>> Why do people buy these boxes?
> 
> 	[SM] Because e.g. for GEO links, latency is in a range where default unadulterated TCP will likely choke on itself, and when faced with requiring customers to change/tune TCPs or having "PEP" fudge it, ease of use of fudging won the day. That is a generous explanation (as this fudging is beneficial to both the operator and most end-users), I can come up with less charitable theories if you want ;) .
> 
>>> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.
>> 
>> Truth is, PEPs are used a lot: at cellular edges, at satellite links… because the network is *not* always doing reasonably well without them.
> 
> 	[SM] Fair enough, I accept that there are use cases for those, but again, only if the actually enhance the "experience" will users be happy to accept them.

… and that’s the only reason to deploy them, given that (as the name suggests) they’re meant to increase performance. I’d be happy to learn more about why you appear to hate them so much (even just anecdotal).


> The goals of the operators and the paying customers are not always aligned here, a PEP might be advantageous more to the operator than the end-user (theoretically also the other direction, but since operators pay for PEPs they are unlikely to deploy those) think mandatory image recompression or forced video quality downscaling.... (and sure these are not as clear as I pitched them, if after an emergency a PEP allows most/all users in a cell to still send somewhat degraded images that is better than the network choking itself with a few high quality images, assuming images from the emergency are somewhat useful).

What is this, are you inventing a (too me, frankly, strange) scenario where PEPs do some evil for customers yet help operators, or is there an anecdote here?


>>> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.
>> 
>> That’s to do with scalability, which depends on how close to the network’s edge one is.
> 
> 	[SM] I have heard the alternative that it has to do with what operators of core-links request from their vendors and what features they are willing to pay for... but this is very anecdotal as I have little insight into big-iron vendors or core-link operators. 
> 
>>> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.
>> 
>> That could well be one way to go. Nice if we provoked you to think!
> 
> 	[SM] You mostly made me realize what the recent increases in IW actually aim to accomplish ;)

That’s fine!  Increasing IW is surely a part of the solution space - though I advocate doing something else (as in the example above) than just to increase the constant in a worldwide standard.


> and that current slow start seems actually better than its reputation; it solves a hard problem surprisingly well.

Actually, given that the large majority of flows end somewhere in slow start, what makes you say that it solves it “well”?


> The max(pat_queue%) idea has been kicking around in my head ever since reading a paper about storing queue occupancy into packets to help CC along (sorry, do not recall the authors or the title right now) so that is not even my own original idea, but simply something I borrowed from smarter engineers simply because I found the data convincing and the theory sane. (Also because I grudgingly accept that latency increases measured over the internet are a tad too noisy to be easily useful* and too noisy for a meaningful controller based on the latency rate of change**)
> 
>>> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.
>> 
>> Yeah you do :-)
> 
> 	[SM] Less than you let on ;). If L4S gets ratified

[snip]

I’m really not interested in an L4S debate.

Cheers,
Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

HI Michael,


> On Jul 11, 2022, at 08:24, Michael Welzl <michawe@ifi.uio.no> wrote:
> 
> Hi Sebastian,
> 
> Neither our paper nor me are advocating one particular solution - we point at a problem and suggest that research on ways to solve the under-utilization problem might be worthwhile.

	[SM2] That is easy to agree upon, as is agreeing on improving slow start and trying to reduce underutilization, but actually doing is hard; personally I am more interested in the hard part, so I might have misunderstood the gist of the discussion you want to start with that publication.

> Jumping from this to discussing the pro’s and con’s of a potential concrete solution is quite a leap…
> 
> More below:
> 
> 
>> On Jul 10, 2022, at 11:29 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Michael,
>> 
>> 
>>> On Jul 10, 2022, at 22:01, Michael Welzl <michawe@ifi.uio.no> wrote:
>>> 
>>> Hi !
>>> 
>>> 
>>>> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> so I reread your paper and stewed a bit on it.
>>> 
>>> Many thanks for doing that! :)
>>> 
>>> 
>>>> I believe that I do not buy some of your premises.
>>> 
>>> you say so, but I don’t really see much disagreement here. Let’s see:
>>> 
>>> 
>>>> e.g. you write:
>>>> 
>>>> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
>>>> 
>>>> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.
>>> 
>>> You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.
>> 
>> 	[SM] No IMHO the underutilization is the direct consequence of requiring a gradual filling of the "pipes" to probe he available capacity. I see no way how this could be done differently with the traffic sources/sinks being uncoordinated entities at the edge, and I see no way of coordinating all end points and handle all paths. In other words, we can fine tune a parameters to tweak the probing a bit, make it more or less aggressive/fast, but the fact that we need to probe capacity somehow means underutilization can not be avoided unless we find a way of coordinating all of the sinks and sources. But being sufficiently dumb, all I can come up with is an all-knowing oracle or faster than light communication, and neither strikes me to be realistic ;)
> 
> There’s quite a spectrum of possibilities between an oracle or “coordinating all of the sinks and sources” on one hand, and quite “blindly” probing from a constant IW on the other.

	[SM] You say "blindly" I say "starting from a conservative but reliable prior"... And what I see is that qualitatively significantly better approaches are not really possible, so we need to discuss small quantitative changes.


> The “fine tuning” that you mention is interesting research, IMO!

	[SM] The paper did not read that you were soliciting ideas for small gradual improvements to me.

>>>> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?
>>> 
>>> There are various ways to do this
> 
> [snip: a couple of concrete suggestions from me, and answers about what problems they might have, with requests for references from you]
> 
> I’m sorry, but I wasn’t really going to have a discussion about these particular possibilities. My point was only that many possible directions exist - being completely “blind” isn’t the only possible approach.

	[SM] Again I do not consider "blind" to be an appropriate qualification here.


> Instead of answering your comments to my suggestions, let me give you one single concrete piece here: our reference 6, as one example of the kind of resesarch that we consider worthwhile for the future:
> 
> "X. Nie, Y. Zhao, Z. Li, G. Chen, K. Sui, J. Zhang, Z. Ye, and D. Pei, “Dynamic TCP initial windows and congestion control schemes through reinforcement learning,” IEEE JSAC, vol. 37, no. 6, 2019.”
> https://1989chenguo.github.io/Publications/TCP-RL-JSAC19.pdf

	[SM] From the title I predict that this is going to lean into the "cache" idea trying to improve the average hit rate of said cache...

> This work learns a useful value of IW over time, rather than using a constant. One author works at Baidu, the paper uses data from Baidu, and it says:
> "TCP-RL has been deployed in one of the top global search engines for more than a year. Our online and testbed experiments show that for short flow transmission, compared with the common practice of IW = 10, TCP-RL can reduce the average transmission time by 23% to 29%.”
> 
> - so it’s probably fair to assume that this was (and perhaps still is) active in Baidu.

	[SM] This seems to confirm my prediction... however the paper seems to be written pretty exclusively from the view of an operator of server farms, not sure this approach will actually do any good for leaf end-points in e.g. home networks (that is for their sending behavior). I tend to prefer symmetric solutions, but if data center traffic can reach higher utilization without compromising end-user quality of experience and fairness, what is not to like about this. It is however fully within the existing slow-start framework, no?



> 
> 
>>>> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).
>>> 
>>> You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place?
>> 
>> 	[SM] I would guess because "marketing" was unhappy with "engineering" emphasizing the side-effects/potential problems and focussed in the best-case scenario? ;)
> 
> It appears that you want to just ill-talk PEPs.

	[SM] Not really, I just wanted to point out that I expect the term PEP to come from entities selling those products and in our current environment it is clear that products are named and promoted emphasizing the potential benefit they can bring and not by the additional risks they might carry (e.g. fission power plants were sold on the idea of essentially unlimited cheap emission free energy, and not on the concurrent problem with waste disposal over time frames in the order of the aggregate human civilisation from the bronze age). I have no beef with that, but I do not think that taking the "positive" name as a sign that PEPs are generally liked or live up to their name (note I am also not saying that they do not, just that the name PEP is a rather unreliable predictor here).


> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,

	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?

> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.

	[SM] Again, I agree it could in theory especially if well-architected. 

> You’re bound to ask me for concrete things next, and if I give you something concrete (e.g., a paper on PEPs), you’ll find something bad about it

	[SM] Them are the rules of the game... however if we should play the game that way, I will come out of it having learned something new and potentially changing my opinion.

> - but this is not a constructive direction of this conversation. Please note that I’m not saying “PEPs are always good”: I only say that, in my personal opinion, they’re a worthwhile direction of future research. That’s a very different statement.

	[SM] Fair enough. I am less optimistic, but happy to be disappointed in my pessimism.

> 
>>> Why do people buy these boxes?
>> 
>> 	[SM] Because e.g. for GEO links, latency is in a range where default unadulterated TCP will likely choke on itself, and when faced with requiring customers to change/tune TCPs or having "PEP" fudge it, ease of use of fudging won the day. That is a generous explanation (as this fudging is beneficial to both the operator and most end-users), I can come up with less charitable theories if you want ;) .
>> 
>>>> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.
>>> 
>>> Truth is, PEPs are used a lot: at cellular edges, at satellite links… because the network is *not* always doing reasonably well without them.
>> 
>> 	[SM] Fair enough, I accept that there are use cases for those, but again, only if the actually enhance the "experience" will users be happy to accept them.
> 
> … and that’s the only reason to deploy them, given that (as the name suggests) they’re meant to increase performance. I’d be happy to learn more about why you appear to hate them so much (even just anecdotal).
> 
>> The goals of the operators and the paying customers are not always aligned here, a PEP might be advantageous more to the operator than the end-user (theoretically also the other direction, but since operators pay for PEPs they are unlikely to deploy those) think mandatory image recompression or forced video quality downscaling.... (and sure these are not as clear as I pitched them, if after an emergency a PEP allows most/all users in a cell to still send somewhat degraded images that is better than the network choking itself with a few high quality images, assuming images from the emergency are somewhat useful).
> 
> What is this, are you inventing a (too me, frankly, strange) scenario where PEPs do some evil for customers yet help operators,

	[SM] This is no invention, but how capitalism works, sorry. The party paying for the PEP decides on using it based on the advantages it offers for them. E.g. a mobile carrier that (in the past) forcible managed to downgrade the quality of streaming video over mobile links without giving the paying end-user an option to use either choppy high resolution or smooth low resolution video. By the way, that does not make the operator evil, it is just that operator and paying customers goals and desires are not all that well aligned (e.g. the operator wants to maximize revenue, the customer to minimize cost).

> or is there an anecdote here?

	[SM] I think the video downscaling thing actually happened in the German market, but I am not sure on the exact details, so I might misinterpret things a bit here. However the observation about alignment of goals I believe to be universally true.


> 
>>>> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.
>>> 
>>> That’s to do with scalability, which depends on how close to the network’s edge one is.
>> 
>> 	[SM] I have heard the alternative that it has to do with what operators of core-links request from their vendors and what features they are willing to pay for... but this is very anecdotal as I have little insight into big-iron vendors or core-link operators. 
>> 
>>>> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.
>>> 
>>> That could well be one way to go. Nice if we provoked you to think!
>> 
>> 	[SM] You mostly made me realize what the recent increases in IW actually aim to accomplish ;)
> 
> That’s fine! Increasing IW is surely a part of the solution space - though I advocate doing something else (as in the example above) than just to increase the constant in a worldwide standard.

	[SM] Happy to agree, I am not saying I think increasing IW is something I unconditionally support, just that I see what it offers.


>> and that current slow start seems actually better than its reputation; it solves a hard problem surprisingly well.
> 
> Actually, given that the large majority of flows end somewhere in slow start, what makes you say that it solves it “well”?

	[SM] As I said, I accepted that there is no silver bullet, and hence some gradual probing with increasing CWND/rate is unavoidable which immediately implies that some flows will end before reaching capacity. So the fact that flows end in slow-start is not a problem but part of the solution. I see no way of ever having all flows immediately start at their "stable" long-term capacity share (something that does not exist in the first place in environments with un-correlated and unpredictable cross traffic). But short of that almost all flows will need more round trips to finish that theoretically minimally possible. I tried to make that point before, and I am not saying current slow-start is 100% perfect, but I do not expect the possible fine-tuning to get us close enough to the theoretical performance of an "oracle" solution to count as "revolutionary" improvement.


>> The max(pat_queue%) idea has been kicking around in my head ever since reading a paper about storing queue occupancy into packets to help CC along (sorry, do not recall the authors or the title right now) so that is not even my own original idea, but simply something I borrowed from smarter engineers simply because I found the data convincing and the theory sane. (Also because I grudgingly accept that latency increases measured over the internet are a tad too noisy to be easily useful* and too noisy for a meaningful controller based on the latency rate of change**)
>> 
>>>> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.
>>> 
>>> Yeah you do :-)
>> 
>> 	[SM] Less than you let on ;). If L4S gets ratified
> 
> [snip]
> 
> I’m really not interested in an L4S debate.

	[SM] I understand, however I see clear reasons why L4S is detrimental to your stated goals as it will getting more information from the network less likely. I also tried to explain, why I believe that to be a theoretically viable way forwards to improve slow-start dynamics. Maybe show why my proposal is bunk while completely ignoring L4S? Or is that the kind of "particular solution" you do not want to discuss at the current stage?

Anyway, thanks for your time. I fear I have made my points in the last mail already and are mostly repeating myself, so I would not feel offended in any way if you let this sub-discussion sleep and wait for more topical discussion entries. 


Regards
	Sebastian


> 
> Cheers,
> Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 21632 bytes --]

Hi !

A few answers below -


> On Jul 11, 2022, at 9:33 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> HI Michael,
> 
> 
>> On Jul 11, 2022, at 08:24, Michael Welzl <michawe@ifi.uio.no <mailto:michawe@ifi.uio.no>> wrote:
>> 
>> Hi Sebastian,
>> 
>> Neither our paper nor me are advocating one particular solution - we point at a problem and suggest that research on ways to solve the under-utilization problem might be worthwhile.
> 
> 	[SM2] That is easy to agree upon, as is agreeing on improving slow start and trying to reduce underutilization, but actually doing is hard; personally I am more interested in the hard part, so I might have misunderstood the gist of the discussion you want to start with that publication.

What you’re doing is jumping ahead. I suggest doing this with research rather than an email discussion, but that’s what we’re now already into.


>> Jumping from this to discussing the pro’s and con’s of a potential concrete solution is quite a leap…
>> 
>> More below:
>> 
>> 
>>> On Jul 10, 2022, at 11:29 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>> Hi Michael,
>>> 
>>> 
>>>> On Jul 10, 2022, at 22:01, Michael Welzl <michawe@ifi.uio.no> wrote:
>>>> 
>>>> Hi !
>>>> 
>>>> 
>>>>> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>>> 
>>>>> Hi Michael,
>>>>> 
>>>>> so I reread your paper and stewed a bit on it.
>>>> 
>>>> Many thanks for doing that! :)
>>>> 
>>>> 
>>>>> I believe that I do not buy some of your premises.
>>>> 
>>>> you say so, but I don’t really see much disagreement here. Let’s see:
>>>> 
>>>> 
>>>>> e.g. you write:
>>>>> 
>>>>> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
>>>>> 
>>>>> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.
>>>> 
>>>> You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.
>>> 
>>> 	[SM] No IMHO the underutilization is the direct consequence of requiring a gradual filling of the "pipes" to probe he available capacity. I see no way how this could be done differently with the traffic sources/sinks being uncoordinated entities at the edge, and I see no way of coordinating all end points and handle all paths. In other words, we can fine tune a parameters to tweak the probing a bit, make it more or less aggressive/fast, but the fact that we need to probe capacity somehow means underutilization can not be avoided unless we find a way of coordinating all of the sinks and sources. But being sufficiently dumb, all I can come up with is an all-knowing oracle or faster than light communication, and neither strikes me to be realistic ;)
>> 
>> There’s quite a spectrum of possibilities between an oracle or “coordinating all of the sinks and sources” on one hand, and quite “blindly” probing from a constant IW on the other.
> 
> 	[SM] You say "blindly" I say "starting from a conservative but reliable prior"... And what I see is that qualitatively significantly better approaches are not really possible, so we need to discuss small quantitative changes.

More about the term “blind” below:


>> The “fine tuning” that you mention is interesting research, IMO!
> 
> 	[SM] The paper did not read that you were soliciting ideas for small gradual improvements to me.

It calls for being drastic in the way we think about things, because it makes the argument that PEPs (different kinds of them!) might in fact be the right approach - but it doesn’t say that “only drastic solutions are good solutions”. Our “The Way Forward” section has 3 subsections; one of them is on end-to-end approaches, where we call out the RL-IW approach I mention below as one good way ahead. I would categorize this as “small and gradual”.


>>>>> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?
>>>> 
>>>> There are various ways to do this
>> 
>> [snip: a couple of concrete suggestions from me, and answers about what problems they might have, with requests for references from you]
>> 
>> I’m sorry, but I wasn’t really going to have a discussion about these particular possibilities. My point was only that many possible directions exist - being completely “blind” isn’t the only possible approach.
> 
> 	[SM] Again I do not consider "blind" to be an appropriate qualification here.

IW is a global constant (not truly configured the same everywhere, most probably for good reason!  but the standard suggests a globally unique value).
From then on, the cwnd is doubled a couple of times. No feedback about the path’s capacity exists - and then, the connection is over.
Okay, there is ONE thing that such a flow gets: the RTT. “Blind except for RTT measurements”, then.

Importantly, such a flow never learns how large its cwnd *could* have become without ever causing a problem. Perhaps 10 times more? 100 times?


>> Instead of answering your comments to my suggestions, let me give you one single concrete piece here: our reference 6, as one example of the kind of resesarch that we consider worthwhile for the future:
>> 
>> "X. Nie, Y. Zhao, Z. Li, G. Chen, K. Sui, J. Zhang, Z. Ye, and D. Pei, “Dynamic TCP initial windows and congestion control schemes through reinforcement learning,” IEEE JSAC, vol. 37, no. 6, 2019.”
>> https://1989chenguo.github.io/Publications/TCP-RL-JSAC19.pdf <https://1989chenguo.github.io/Publications/TCP-RL-JSAC19.pdf>
> 
> 	[SM] From the title I predict that this is going to lean into the "cache" idea trying to improve the average hit rate of said cache...
> 
>> This work learns a useful value of IW over time, rather than using a constant. One author works at Baidu, the paper uses data from Baidu, and it says:
>> "TCP-RL has been deployed in one of the top global search engines for more than a year. Our online and testbed experiments show that for short flow transmission, compared with the common practice of IW = 10, TCP-RL can reduce the average transmission time by 23% to 29%.”
>> 
>> - so it’s probably fair to assume that this was (and perhaps still is) active in Baidu.
> 
> 	[SM] This seems to confirm my prediction... however the paper seems to be written pretty exclusively from the view of an operator of server farms, not sure this approach will actually do any good for leaf end-points in e.g. home networks (that is for their sending behavior). I tend to prefer symmetric solutions, but if data center traffic can reach higher utilization without compromising end-user quality of experience and fairness, what is not to like about this. It is however fully within the existing slow-start framework, no?

Yes!


>>>>> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).
>>>> 
>>>> You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place?
>>> 
>>> 	[SM] I would guess because "marketing" was unhappy with "engineering" emphasizing the side-effects/potential problems and focussed in the best-case scenario? ;)
>> 
>> It appears that you want to just ill-talk PEPs.
> 
> 	[SM] Not really, I just wanted to point out that I expect the term PEP to come from entities selling those products and in our current environment it is clear that products are named and promoted emphasizing the potential benefit they can bring and not by the additional risks they might carry (e.g. fission power plants were sold on the idea of essentially unlimited cheap emission free energy, and not on the concurrent problem with waste disposal over time frames in the order of the aggregate human civilisation from the bronze age). I have no beef with that, but I do not think that taking the "positive" name as a sign that PEPs are generally liked or live up to their name (note I am also not saying that they do not, just that the name PEP is a rather unreliable predictor here).

I don’t even think that this name has that kind of history. My point was that they’re called PEPs because they’re *meant* to improve performance; that’s what they’re designed for. You describe “a PEP that does not enhance performance”, which, to me, is like talking about a web server that doesn’t serve web pages. Sure, not all PEPs may always work well, but they should - that’s their raison d’être.


>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
> 
> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
> 
>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
> 
> 	[SM] Again, I agree it could in theory especially if well-architected. 

That’s what I’m advocating.


>> You’re bound to ask me for concrete things next, and if I give you something concrete (e.g., a paper on PEPs), you’ll find something bad about it
> 
> 	[SM] Them are the rules of the game... however if we should play the game that way, I will come out of it having learned something new and potentially changing my opinion.
> 
>> - but this is not a constructive direction of this conversation. Please note that I’m not saying “PEPs are always good”: I only say that, in my personal opinion, they’re a worthwhile direction of future research. That’s a very different statement.
> 
> 	[SM] Fair enough. I am less optimistic, but happy to be disappointed in my pessimism.
> 
>> 
>>>> Why do people buy these boxes?
>>> 
>>> 	[SM] Because e.g. for GEO links, latency is in a range where default unadulterated TCP will likely choke on itself, and when faced with requiring customers to change/tune TCPs or having "PEP" fudge it, ease of use of fudging won the day. That is a generous explanation (as this fudging is beneficial to both the operator and most end-users), I can come up with less charitable theories if you want ;) .
>>> 
>>>>> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.
>>>> 
>>>> Truth is, PEPs are used a lot: at cellular edges, at satellite links… because the network is *not* always doing reasonably well without them.
>>> 
>>> 	[SM] Fair enough, I accept that there are use cases for those, but again, only if the actually enhance the "experience" will users be happy to accept them.
>> 
>> … and that’s the only reason to deploy them, given that (as the name suggests) they’re meant to increase performance. I’d be happy to learn more about why you appear to hate them so much (even just anecdotal).
>> 
>>> The goals of the operators and the paying customers are not always aligned here, a PEP might be advantageous more to the operator than the end-user (theoretically also the other direction, but since operators pay for PEPs they are unlikely to deploy those) think mandatory image recompression or forced video quality downscaling.... (and sure these are not as clear as I pitched them, if after an emergency a PEP allows most/all users in a cell to still send somewhat degraded images that is better than the network choking itself with a few high quality images, assuming images from the emergency are somewhat useful).
>> 
>> What is this, are you inventing a (too me, frankly, strange) scenario where PEPs do some evil for customers yet help operators,
> 
> 	[SM] This is no invention, but how capitalism works, sorry. The party paying for the PEP decides on using it based on the advantages it offers for them. E.g. a mobile carrier that (in the past) forcible managed to downgrade the quality of streaming video over mobile links without giving the paying end-user an option to use either choppy high resolution or smooth low resolution video. By the way, that does not make the operator evil, it is just that operator and paying customers goals and desires are not all that well aligned (e.g. the operator wants to maximize revenue, the customer to minimize cost).

You claim that these goals and desires are not well aligned (and a PEP is then an instrument in this evil) - do you have any proof, or even anecdotes, to support that claim?
I would think that operators generally try to make their customers happy (or they would switch to different operators).  Yes there may be some misalignments in incentives, but I believe that these are more subtle points. E.g., who wants a choppy high resolution video? Do such users really exist?


>> or is there an anecdote here?
> 
> 	[SM] I think the video downscaling thing actually happened in the German market, but I am not sure on the exact details, so I might misinterpret things a bit here. However the observation about alignment of goals I believe to be universally true.

I’d be interested in hearing more. Was there an outcry of customers who wanted their choppy high resolution video back?   :-)    :-)


>>>>> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.
>>>> 
>>>> That’s to do with scalability, which depends on how close to the network’s edge one is.
>>> 
>>> 	[SM] I have heard the alternative that it has to do with what operators of core-links request from their vendors and what features they are willing to pay for... but this is very anecdotal as I have little insight into big-iron vendors or core-link operators. 
>>> 
>>>>> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.
>>>> 
>>>> That could well be one way to go. Nice if we provoked you to think!
>>> 
>>> 	[SM] You mostly made me realize what the recent increases in IW actually aim to accomplish ;)
>> 
>> That’s fine! Increasing IW is surely a part of the solution space - though I advocate doing something else (as in the example above) than just to increase the constant in a worldwide standard.
> 
> 	[SM] Happy to agree, I am not saying I think increasing IW is something I unconditionally support, just that I see what it offers.
> 
> 
>>> and that current slow start seems actually better than its reputation; it solves a hard problem surprisingly well.
>> 
>> Actually, given that the large majority of flows end somewhere in slow start, what makes you say that it solves it “well”?
> 
> 	[SM] As I said, I accepted that there is no silver bullet, and hence some gradual probing with increasing CWND/rate is unavoidable which immediately implies that some flows will end before reaching capacity.

You say “some” but data says “the large majority”.


> So the fact that flows end in slow-start is not a problem but part of the solution. I see no way of ever having all flows immediately start at their "stable" long-term capacity share (something that does not exist in the first place in environments with un-correlated and unpredictable cross traffic). But short of that almost all flows will need more round trips to finish that theoretically minimally possible. I tried to make that point before, and I am not saying current slow-start is 100% perfect, but I do not expect the possible fine-tuning to get us close enough to the theoretical performance of an "oracle" solution to count as "revolutionary" improvement.

It doesn’t need to be revolutionary; I think that ways to learn / cache the IW are already quite useful.

Now, you repeatedly mentioned that caching may not work because flows don’t always traverse the same path. True … but then, what about all the flows that do traverse the same bottleneck (to the same receiver, or set of receivers in a home), which is usually at the edge? That bottleneck may often be the same. Now, if we just had an in-network device that could divide the path into a “core” segment where it’s safe to use a pretty large IW value, and a downstream segment where the IW value may need be smaller, but a certain workable range might be known to the device, because that devices sits right at the edge…


>>> The max(pat_queue%) idea has been kicking around in my head ever since reading a paper about storing queue occupancy into packets to help CC along (sorry, do not recall the authors or the title right now) so that is not even my own original idea, but simply something I borrowed from smarter engineers simply because I found the data convincing and the theory sane. (Also because I grudgingly accept that latency increases measured over the internet are a tad too noisy to be easily useful* and too noisy for a meaningful controller based on the latency rate of change**)
>>> 
>>>>> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.
>>>> 
>>>> Yeah you do :-)
>>> 
>>> 	[SM] Less than you let on ;). If L4S gets ratified
>> 
>> [snip]
>> 
>> I’m really not interested in an L4S debate.
> 
> 	[SM] I understand, however I see clear reasons why L4S is detrimental to your stated goals as it will getting more information from the network less likely. I also tried to explain, why I believe that to be a theoretically viable way forwards to improve slow-start dynamics. Maybe show why my proposal is bunk while completely ignoring L4S? Or is that the kind of "particular solution" you do not want to discuss at the current stage?

I’d say the latter. We could spend weeks of time and tonds of emails discussing explicit-feedback based schemes…  instead, if you think your idea is good, why not build it, test it, and evaluate its trade-offs?

I don’t see L4S as being *detrimental* to our stated goals, BTW - but, as it stands, I see limitations in its usefulness because TCP Prague (AFAIK) only changes Congestion Avoidance, at least up to now. I’m getting the impression that Congestion Avoidance with a greedy sender is a rare animal. Non-greedy (i.e., the sender takes a break) is a different thing again - various implementations exist, as do proposals for how to handle this … a flow with pauses is not too different from multiple consecutive short flows. Well, it always uses the same 5-tuple, which makes caching strategies more likely to succeed.


> Anyway, thanks for your time. I fear I have made my points in the last mail already and are mostly repeating myself, so I would not feel offended in any way if you let this sub-discussion sleep and wait for more topical discussion entries. 
> 
> 
> Regards
> 	Sebastian

Cheers,
Michael


[-- Attachment #2: Type: text/html, Size: 62121 bytes --]

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi Michael,


> On Jul 11, 2022, at 10:49, Michael Welzl <michawe@ifi.uio.no> wrote:
> 
> Hi !
> 
> A few answers below -
> 
> 
>> On Jul 11, 2022, at 9:33 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> HI Michael,
>> 
>> 
>>> On Jul 11, 2022, at 08:24, Michael Welzl <michawe@ifi.uio.no> wrote:
>>> 
>>> Hi Sebastian,
>>> 
>>> Neither our paper nor me are advocating one particular solution - we point at a problem and suggest that research on ways to solve the under-utilization problem might be worthwhile.
>> 
>> 	[SM2] That is easy to agree upon, as is agreeing on improving slow start and trying to reduce underutilization, but actually doing is hard; personally I am more interested in the hard part, so I might have misunderstood the gist of the discussion you want to start with that publication.
> 
> What you’re doing is jumping ahead. I suggest doing this with research rather than an email discussion, but that’s what we’re now already into.

	[SM] Sure, except my day job is in a completely different field (wetware instead of hard- or software), it is very unlikely that I will be able to contribute meaningfully to the kind of research. I am not saying all I can contribute is just talk; I helped out with a few obscure details in the bufferbloat movement (better anti-bufferbloat movement) and I did contribute a bit to sqm and the earlier cited autorate approach (which is topical because it is concerned with capacity detection), but I will not be conducting nor publishing any CS-grade paper on CC and/or slow start. Maybe a reason for me to respectfully bow out of this discussion as talk is cheap and easy to come by even without me helping.

>>> Jumping from this to discussing the pro’s and con’s of a potential concrete solution is quite a leap…
>>> 
>>> More below:
>>> 
>>> 
>>>> On Jul 10, 2022, at 11:29 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>> Hi Michael,
>>>> 
>>>> 
>>>>> On Jul 10, 2022, at 22:01, Michael Welzl <michawe@ifi.uio.no> wrote:
>>>>> 
>>>>> Hi !
>>>>> 
>>>>> 
>>>>>> On Jul 10, 2022, at 7:27 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>>>> 
>>>>>> Hi Michael,
>>>>>> 
>>>>>> so I reread your paper and stewed a bit on it.
>>>>> 
>>>>> Many thanks for doing that! :)
>>>>> 
>>>>> 
>>>>>> I believe that I do not buy some of your premises.
>>>>> 
>>>>> you say so, but I don’t really see much disagreement here. Let’s see:
>>>>> 
>>>>> 
>>>>>> e.g. you write:
>>>>>> 
>>>>>> "We will now examine two factors that make the the present situation particularly worrisome. First, the way the infrastructure has been evolving gives TCP an increasingly large operational space in which it does not see any feedback at all. Second, most TCP connections are extremely short. As a result, it is quite rare for a TCP connection to even see a single congestion notification during its lifetime."
>>>>>> 
>>>>>> And seem to see a problem that flows might be able to finish their data transfer business while still in slow start. I see the same data, but see no problem. Unless we have an oracle that tells each sender (over a shared bottleneck) exactly how much to send at any given time point, different control loops will interact on those intermediary nodes.
>>>>> 
>>>>> You really say that you don’t see the solution. The problem is that capacities are underutilized, which means that flows take longer (sometimes, much longer!) to finish than they theoretically could, if we had a better solution.
>>>> 
>>>> 	[SM] No IMHO the underutilization is the direct consequence of requiring a gradual filling of the "pipes" to probe he available capacity. I see no way how this could be done differently with the traffic sources/sinks being uncoordinated entities at the edge, and I see no way of coordinating all end points and handle all paths. In other words, we can fine tune a parameters to tweak the probing a bit, make it more or less aggressive/fast, but the fact that we need to probe capacity somehow means underutilization can not be avoided unless we find a way of coordinating all of the sinks and sources. But being sufficiently dumb, all I can come up with is an all-knowing oracle or faster than light communication, and neither strikes me to be realistic ;)
>>> 
>>> There’s quite a spectrum of possibilities between an oracle or “coordinating all of the sinks and sources” on one hand, and quite “blindly” probing from a constant IW on the other.
>> 
>> 	[SM] You say "blindly" I say "starting from a conservative but reliable prior"... And what I see is that qualitatively significantly better approaches are not really possible, so we need to discuss small quantitative changes.
> 
> More about the term “blind” below:
> 
> 
>>> The “fine tuning” that you mention is interesting research, IMO!
>> 
>> 	[SM] The paper did not read that you were soliciting ideas for small gradual improvements to me.
> 
> It calls for being drastic in the way we think about things, because it makes the argument that PEPs (different kinds of them!) might in fact be the right approach - but it doesn’t say that “only drastic solutions are good solutions”. Our “The Way Forward” section has 3 subsections; one of them is on end-to-end approaches, where we call out the RL-IW approach I mention below as one good way ahead. I would categorize this as “small and gradual”.

	[SM] Having contact with machine learning and reenforcement learning in my dayjob, I am at best cautiously optimistic for such a solution to end up with an improvement across the board, but again happy to be shown wrong. I prefer solutions with algorithms that are easier to interpret than what machine learning (especially with deep networks) ends up with.


> 
> 
>>>>>> I might be limited in my depth of thought here, but having each flow probing for capacity seems exactly the right approach... and doubling CWND or rate every RTT is pretty aggressive already (making slow start shorter by reaching capacity faster within the slow-start framework requires either to start with a higher initial value (what increasing IW tries to achieve?) or use a larger increase factor than 2 per RTT). I consider increased IW a milder approach than the alternative. And once one accepts that a gradual rate increasing is the way forward it falls out logically that some flows will finish before they reach steady state capacity especially if that flows available capacity is large. So what exactly is the problem with short flows not reaching capacity and what alternative exists that does not lead to carnage if more-aggressive start-up phases drive the bottleneck load into emergency drop territory?
>>>>> 
>>>>> There are various ways to do this
>>> 
>>> [snip: a couple of concrete suggestions from me, and answers about what problems they might have, with requests for references from you]
>>> 
>>> I’m sorry, but I wasn’t really going to have a discussion about these particular possibilities. My point was only that many possible directions exist - being completely “blind” isn’t the only possible approach.
>> 
>> 	[SM] Again I do not consider "blind" to be an appropriate qualification here.
> 
> IW is a global constant (not truly configured the same everywhere, most probably for good reason!

	[SM] Arguable how "good" these reasons are. It is already noticeable that a number of CDNs use higher than standard IWs and extract more than their expected fair share of a link's capacity. That is certainly in the interest of the CDN abnd the CDN's paying customers, but not necessarily in the interest of the end-user accessing the data on the CDN. (Personally I do not care too much about this, since I use an fq-scheduler at my ingress and egress, so high IW flows finish faster if spare capacity is available and only cause queue build up for them selves if there is no spare capacity).

>  but the standard suggests a globally unique value).

	[SM] One thing I learned about IETF standards in the last years is that not all of them are of the same quality, I will look carefully at each standards before accepting its recommendation as "gospel". (IMHO the IETF process is very welcome but assumes good faith all around, and hence seems easily gamed).

> From then on, the cwnd is doubled a couple of times. No feedback about the path’s capacity exists - and then, the connection is over.

	[SM] I disagree, the flow very much knows that the reached CWND/sending rate was <= than the capacity available for that flow, that is feed-back IMHO. If you are a gambling type you could try to speculatively re-use that information.

> Okay, there is ONE thing that such a flow gets: the RTT. “Blind except for RTT measurements”, then.

	[SM] I guess your point is "flow does not know the maximal capacity it could have gotten"?

> Importantly, such a flow never learns how large its cwnd *could* have become without ever causing a problem. Perhaps 10 times more? 100 times?

	[SM] Sure. ATM the only way to learn a path's capacity is actually to saturate the path*, but if a flow is done with its data transfer, having if exchange dummy data just to probe capacity seems like a waste all around. I guess what I want to ask is, how would knowing how much available but untapped capacity was available at one point help?


*) stuff like deducing capacity from packet pair interval at the receiver (assuming packets sent back to back) is notoriously imprecise, so unless "chirping" overcomes that imprecision without costing too many round trips worth of noise supression measuring capacity by causing congestion is the only way. Not great.


> 
> 
>>> Instead of answering your comments to my suggestions, let me give you one single concrete piece here: our reference 6, as one example of the kind of resesarch that we consider worthwhile for the future:
>>> 
>>> "X. Nie, Y. Zhao, Z. Li, G. Chen, K. Sui, J. Zhang, Z. Ye, and D. Pei, “Dynamic TCP initial windows and congestion control schemes through reinforcement learning,” IEEE JSAC, vol. 37, no. 6, 2019.”
>>> https://1989chenguo.github.io/Publications/TCP-RL-JSAC19.pdf
>> 
>> 	[SM] From the title I predict that this is going to lean into the "cache" idea trying to improve the average hit rate of said cache...
>> 
>>> This work learns a useful value of IW over time, rather than using a constant. One author works at Baidu, the paper uses data from Baidu, and it says:
>>> "TCP-RL has been deployed in one of the top global search engines for more than a year. Our online and testbed experiments show that for short flow transmission, compared with the common practice of IW = 10, TCP-RL can reduce the average transmission time by 23% to 29%.”
>>> 
>>> - so it’s probably fair to assume that this was (and perhaps still is) active in Baidu.
>> 
>> 	[SM] This seems to confirm my prediction... however the paper seems to be written pretty exclusively from the view of an operator of server farms, not sure this approach will actually do any good for leaf end-points in e.g. home networks (that is for their sending behavior). I tend to prefer symmetric solutions, but if data center traffic can reach higher utilization without compromising end-user quality of experience and fairness, what is not to like about this. It is however fully within the existing slow-start framework, no?
> 
> Yes!
> 
> 
>>>>>> And as an aside, a PEP (performance enhancing proxy) that does not enhance performance is useless at best and likely harmful (rather a PDP, performance degrading proxy).
>>>>> 
>>>>> You’ve made it sound worse by changing the term, for whatever that’s worth. If they never help, why has anyone ever called them PEPs in the first place?
>>>> 
>>>> 	[SM] I would guess because "marketing" was unhappy with "engineering" emphasizing the side-effects/potential problems and focussed in the best-case scenario? ;)
>>> 
>>> It appears that you want to just ill-talk PEPs.
>> 
>> 	[SM] Not really, I just wanted to point out that I expect the term PEP to come from entities selling those products and in our current environment it is clear that products are named and promoted emphasizing the potential benefit they can bring and not by the additional risks they might carry (e.g. fission power plants were sold on the idea of essentially unlimited cheap emission free energy, and not on the concurrent problem with waste disposal over time frames in the order of the aggregate human civilisation from the bronze age). I have no beef with that, but I do not think that taking the "positive" name as a sign that PEPs are generally liked or live up to their name (note I am also not saying that they do not, just that the name PEP is a rather unreliable predictor here).
> 
> I don’t even think that this name has that kind of history. My point was that they’re called PEPs because they’re *meant* to improve performance;

	[SM] That is not really how our economic system works... products are primarily intended to generate more revenue than cost, it helps if they offer something to the customer, but that is really just a means to extract revenue...


> that’s what they’re designed for. You describe “a PEP that does not enhance performance”, which, to me, is like talking about a web server that doesn’t serve web pages. Sure, not all PEPs may always work well, but they should - that’s their raison d’être.

	[SM] That is a very optimistic view, I would love to be able to share.

> 
>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>> 
>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>> 
>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>> 
>> 	[SM] Again, I agree it could in theory especially if well-architected. 
> 
> That’s what I’m advocating.

	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?

> 
>>> You’re bound to ask me for concrete things next, and if I give you something concrete (e.g., a paper on PEPs), you’ll find something bad about it
>> 
>> 	[SM] Them are the rules of the game... however if we should play the game that way, I will come out of it having learned something new and potentially changing my opinion.
>> 
>>> - but this is not a constructive direction of this conversation. Please note that I’m not saying “PEPs are always good”: I only say that, in my personal opinion, they’re a worthwhile direction of future research. That’s a very different statement.
>> 
>> 	[SM] Fair enough. I am less optimistic, but happy to be disappointed in my pessimism.
>> 
>>> 
>>>>> Why do people buy these boxes?
>>>> 
>>>> 	[SM] Because e.g. for GEO links, latency is in a range where default unadulterated TCP will likely choke on itself, and when faced with requiring customers to change/tune TCPs or having "PEP" fudge it, ease of use of fudging won the day. That is a generous explanation (as this fudging is beneficial to both the operator and most end-users), I can come up with less charitable theories if you want ;) .
>>>> 
>>>>>> The network so far has been doing reasonably well with putting more protocol smarts at the ends than in the parts in between.
>>>>> 
>>>>> Truth is, PEPs are used a lot: at cellular edges, at satellite links… because the network is *not* always doing reasonably well without them.
>>>> 
>>>> 	[SM] Fair enough, I accept that there are use cases for those, but again, only if the actually enhance the "experience" will users be happy to accept them.
>>> 
>>> … and that’s the only reason to deploy them, given that (as the name suggests) they’re meant to increase performance. I’d be happy to learn more about why you appear to hate them so much (even just anecdotal).
>>> 
>>>> The goals of the operators and the paying customers are not always aligned here, a PEP might be advantageous more to the operator than the end-user (theoretically also the other direction, but since operators pay for PEPs they are unlikely to deploy those) think mandatory image recompression or forced video quality downscaling.... (and sure these are not as clear as I pitched them, if after an emergency a PEP allows most/all users in a cell to still send somewhat degraded images that is better than the network choking itself with a few high quality images, assuming images from the emergency are somewhat useful).
>>> 
>>> What is this, are you inventing a (too me, frankly, strange) scenario where PEPs do some evil for customers yet help operators,
>> 
>> 	[SM] This is no invention, but how capitalism works, sorry. The party paying for the PEP decides on using it based on the advantages it offers for them. E.g. a mobile carrier that (in the past) forcible managed to downgrade the quality of streaming video over mobile links without giving the paying end-user an option to use either choppy high resolution or smooth low resolution video. By the way, that does not make the operator evil, it is just that operator and paying customers goals and desires are not all that well aligned (e.g. the operator wants to maximize revenue, the customer to minimize cost).
> 
> You claim that these goals and desires are not well aligned (and a PEP is then an instrument in this evil)

	[SM] Again this is expected behavior in our economic system, I have not and will not classify that as "evil", but I will also not start believing that companies offer products just to get a warm and fuzzy feeling. It is part of the principle of how a market economy works that the goals of the entities involved are opposite of each other, that is how a market is supposed to optimize resource allocation.

> - do you have any proof, or even anecdotes, to support that claim?

	[SM] The claim that sellers want the highest revenue/cost ratio while buyers want the lowest cost/utility seems hardly controversial or requiring a citation.


> I would think that operators generally try to make their customers happy (or they would switch to different operators).  Yes there may be some misalignments in incentives, but I believe that these are more subtle points. E.g., who wants a choppy high resolution video? Do such users really exist?

	[SM] I might be able to buffer that choppy video well enough to allow smooth playback at the desired higher resolution/quality (or I might be happy with a few seconds to compare quality of displays), given that I essentially buy internet access from my mobile carrier that carrier should get out of my way. (However if the carrier also offers "video-optimization" as an opt-in feature end-users can toggle at will that is a different kettle of fish and something I would consider good service). IIRC a German carrier was simply downforcing quality for all video streaming at all time, mostly to minimize cost and bandwidth usage, which pretty much looks like an exercise to minimize operational cost and not to increase customer satisfaction. So yes there are "misalignments in incentives" that are inherent and structural to the way we organize our society. (I am sort of okay living with that, but I will not sugar coat it).


>>> or is there an anecdote here?
>> 
>> 	[SM] I think the video downscaling thing actually happened in the German market, but I am not sure on the exact details, so I might misinterpret things a bit here. However the observation about alignment of goals I believe to be universally true.
> 
> I’d be interested in hearing more. Was there an outcry of customers who wanted their choppy high resolution video back?   :-)    :-)

	[SM] The ISP downscaled not only when it reduced "choppy" play but at all times to minimize bandwidth use and increase available "free cell capacity" which IMHO is an economic decision to minimize costs and not one to improve customer experience. 


>>>>>> I have witnessed the arguments in the "L4S wars" about how little processing one can ask the more central network nodes perform, e.g. flow queueing which would solve a lot of the issues (e.g. a hyper aggressive slow-start flow would mostly hurt itself if it overshoots its capacity) seems to be a complete no-go.
>>>>> 
>>>>> That’s to do with scalability, which depends on how close to the network’s edge one is.
>>>> 
>>>> 	[SM] I have heard the alternative that it has to do with what operators of core-links request from their vendors and what features they are willing to pay for... but this is very anecdotal as I have little insight into big-iron vendors or core-link operators. 
>>>> 
>>>>>> I personally think what we should do is have the network supply more information to the end points to control their behavior better. E.g. if we would mandate a max_queue-fill-percentage field in a protocol header and have each node write max(current_value_of_the_field, queue-filling_percentage_of_the_current_node) in every packet, end points could estimate how close to congestion the path is (e.g. by looking at the rate of %queueing changes) and tailor their growth/shrinkage rates accordingly, both during slow-start and during congestion avoidance.
>>>>> 
>>>>> That could well be one way to go. Nice if we provoked you to think!
>>>> 
>>>> 	[SM] You mostly made me realize what the recent increases in IW actually aim to accomplish ;)
>>> 
>>> That’s fine! Increasing IW is surely a part of the solution space - though I advocate doing something else (as in the example above) than just to increase the constant in a worldwide standard.
>> 
>> 	[SM] Happy to agree, I am not saying I think increasing IW is something I unconditionally support, just that I see what it offers.
>> 
>> 
>>>> and that current slow start seems actually better than its reputation; it solves a hard problem surprisingly well.
>>> 
>>> Actually, given that the large majority of flows end somewhere in slow start, what makes you say that it solves it “well”?
>> 
>> 	[SM] As I said, I accepted that there is no silver bullet, and hence some gradual probing with increasing CWND/rate is unavoidable which immediately implies that some flows will end before reaching capacity.
> 
> You say “some” but data says “the large majority”.

	[SM] Well, I am fine with that as I see no real viable alternative. Again we might be able to tweak the capacity search parameters a bit but that the ramp up takes time is a feature not a bug (so even if we would no the immeiate available capacity, we should not jump there ib one fell swoop, unless we are big fans of non-dampened oscillations), this is how multiple concurrent not synchronized flows can coexist next to each other in a reasonable fashion without requiring too much smarts in the network.


>> So the fact that flows end in slow-start is not a problem but part of the solution. I see no way of ever having all flows immediately start at their "stable" long-term capacity share (something that does not exist in the first place in environments with un-correlated and unpredictable cross traffic). But short of that almost all flows will need more round trips to finish that theoretically minimally possible. I tried to make that point before, and I am not saying current slow-start is 100% perfect, but I do not expect the possible fine-tuning to get us close enough to the theoretical performance of an "oracle" solution to count as "revolutionary" improvement.
> 
> It doesn’t need to be revolutionary; I think that ways to learn / cache the IW are already quite useful.

	[SM] As all speculation that is a "gamble", where the risk and cost of mis-prediction need to be weighted against the advantage of getting it right. But I guess I agree that making IW more of a free parameter might be interesting and a way forward. My gut feeling tells me however we might want to look at pacing these out a bit, I have no data to back this up though.

> 
> Now, you repeatedly mentioned that caching may not work because flows don’t always traverse the same path. True … but then, what about all the flows that do traverse the same bottleneck (to the same receiver, or set of receivers in a home), which is usually at the edge? That bottleneck may often be the same.

	[SM] Yes, "may" is the operational word here... but the point about IW is that it would need to scale inversely with the expected flow length (in packets or bytes) for a long running bulk flow current start-up should just in the noise regarding completion time for a very short flow however changing IW will be quite noticeable. But we do not know this about flows in advance often, no? Not sure that the optimal IW is actually determined by the bottleneck (I really do not know and have not thought about this so I might well be wrong here).

> Now, if we just had an in-network device that could divide the path into a “core” segment where it’s safe to use a pretty large IW value, and a downstream segment where the IW value may need be smaller, but a certain workable range might be known to the device, because that devices sits right at the edge…

	[SM] This seems to be problematic if end-to-end encryption is desired, no? But essentially this also seems to be implemented already, except that we call these things CDNs instead of proxies ;) (kidding!)

>>>> The max(pat_queue%) idea has been kicking around in my head ever since reading a paper about storing queue occupancy into packets to help CC along (sorry, do not recall the authors or the title right now) so that is not even my own original idea, but simply something I borrowed from smarter engineers simply because I found the data convincing and the theory sane. (Also because I grudgingly accept that latency increases measured over the internet are a tad too noisy to be easily useful* and too noisy for a meaningful controller based on the latency rate of change**)
>>>> 
>>>>>> But alas we seem to go the path of a relative dumb 1 bit signal giving us an under-defined queue filling state instead and to estimate relative queue filling dynamics from that we need many samples (so literally too little too late, or L3T2), but I digress.
>>>>> 
>>>>> Yeah you do :-)
>>>> 
>>>> 	[SM] Less than you let on ;). If L4S gets ratified
>>> 
>>> [snip]
>>> 
>>> I’m really not interested in an L4S debate.
>> 
>> 	[SM] I understand, however I see clear reasons why L4S is detrimental to your stated goals as it will getting more information from the network less likely. I also tried to explain, why I believe that to be a theoretically viable way forwards to improve slow-start dynamics. Maybe show why my proposal is bunk while completely ignoring L4S? Or is that the kind of "particular solution" you do not want to discuss at the current stage?
> 
> I’d say the latter. We could spend weeks of time and tonds of emails discussing explicit-feedback based schemes…  instead, if you think your idea is good, why not build it, test it, and evaluate its trade-offs?

	[SM] In all honesty, because my day-job is in a pretty different field and I do not believe I can or even would want to perform publishable CS research after hours (let alone find a journal accepting layman submissions without any relevant affiliation).

> I don’t see L4S as being *detrimental* to our stated goals, BTW - but, as it stands, I see limitations in its usefulness because TCP Prague (AFAIK) only changes Congestion Avoidance, at least up to now. I’m getting the impression that Congestion Avoidance with a greedy sender is a rare animal.

	[SM] Erm, DASH-type traffic seems quite common, no? There the individual segments transmitted can be large enough to reach (close to) capacity?


> Non-greedy (i.e., the sender takes a break) is a different thing again - various implementations exist, as do proposals for how to handle this … a flow with pauses is not too different from multiple consecutive short flows. Well, it always uses the same 5-tuple, which makes caching strategies more likely to succeed.

	[SM] Not having looked at that, but I would have assumed that TCP will not start from scratch if there was a short break in transmission due the the sender having nothing to send for a while? However, my playing around with shaper tracking on variable rate links like LTE or starlink (or WiFi with mobile clients in motion) indicates that state caching should probably be sufficiently short duration so it will not result in too much traffic when the path capacity dropped in the rest period. Always starting from a low baseline, as TCP does, is clearly the conservative and prudent thing to do, unless one values reducing e.g. average completion time over say a high quantile.



Regards
	Sebastian


>> Anyway, thanks for your time. I fear I have made my points in the last mail already and are mostly repeating myself, so I would not feel offended in any way if you let this sub-discussion sleep and wait for more topical discussion entries. 
>> 
>> 
>> Regards
>> 	Sebastian
> 
> Cheers,
> Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

On Tue, 12 Jul 2022, Sebastian Moeller via Bloat wrote:

>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>
>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>> 
>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>
>>> 	[SM] Again, I agree it could in theory especially if well-architected. 
>> 
>> That’s what I’m advocating.
>
> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?

the windows protocols work very poorly over high latency links (i.e. long 
distance links) and the PEPs that short circuit those protocols make life much 
nicer for users as well as reducing network traffic.

it's a nasty protocol to start with, but it's the reality on the ground and 
proxies do help a lot.

David Lang

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi David,

Thanks!

> On Jul 12, 2022, at 19:56, David Lang <david@lang.hm> wrote:
> 
> On Tue, 12 Jul 2022, Sebastian Moeller via Bloat wrote:
> 
>>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>> 
>>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>> 
>>>> 	[SM] Again, I agree it could in theory especially if well-architected. 
>>> That’s what I’m advocating.
>> 
>> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?
> 
> the windows protocols work very poorly over high latency links (i.e. long distance links) and the PEPs that short circuit those protocols make life much nicer for users as well as reducing network traffic.

	[SM] Windows protocols, like in microsoft's server message block (smb) protocol or as in "protocols using data windows", like TCP's congestion and receive window?

> it's a nasty protocol to start with, but it's the reality on the ground and proxies do help a lot.

	[SM] Are such proxies located in third party middle boxes/proxies or are these part of microsoft's software suite for enterprises (assuming the first as answer to my question)?

> David Lang


Regards
	Sebastian

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]

On Tue, 12 Jul 2022, Sebastian Moeller wrote:

> Hi David,
>
> Thanks!
>
>> On Jul 12, 2022, at 19:56, David Lang <david@lang.hm> wrote:
>>
>> On Tue, 12 Jul 2022, Sebastian Moeller via Bloat wrote:
>>
>>>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>>>
>>>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>>>
>>>>> 	[SM] Again, I agree it could in theory especially if well-architected.
>>>> That’s what I’m advocating.
>>>
>>> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?
>>
>> the windows protocols work very poorly over high latency links (i.e. long distance links) and the PEPs that short circuit those protocols make life much nicer for users as well as reducing network traffic.
>
> 	[SM] Windows protocols, like in microsoft's server message block (smb) protocol or as in "protocols using data windows", like TCP's congestion and receive window?

microsoft windows smb

>> it's a nasty protocol to start with, but it's the reality on the ground and proxies do help a lot.
>
> 	[SM] Are such proxies located in third party middle boxes/proxies or are these part of microsoft's software suite for enterprises (assuming the first as answer to my question)?

third party middle boxes that you put in each office as a proxy.

David Lang

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Michael Welzl]

I’ll cut some clutter:


>> What you’re doing is jumping ahead. I suggest doing this with research rather than an email discussion, but that’s what we’re now already into.
> 
> 	[SM] Sure, except my day job
(snip)

that’s ok - I wasn’t trying to tell you off for discussing ideas!   - I was just trying to clarify:
1. we agree there’s a problem;
2. our paper made the point that research is needed;
3. now you’re discussing possible research ideas, which is the step that comes after.

… and trying to discuss 3. in depth easily gets hand-wavy, unless one actually does the research first. My So all I say is that this discussion is a bit pointless  (I’m trying to end it - please!  :-)  )


> Maybe a reason for me to respectfully bow out of this discussion as talk is cheap and easy to come by even without me helping.

Well, let’s at least try to keep this short…


>> Okay, there is ONE thing that such a flow gets: the RTT. “Blind except for RTT measurements”, then.
> 
> 	[SM] I guess your point is "flow does not know the maximal capacity it could have gotten"?

Yes.


>> Importantly, such a flow never learns how large its cwnd *could* have become without ever causing a problem. Perhaps 10 times more? 100 times?
> 
> 	[SM] Sure. ATM the only way to learn a path's capacity is actually to saturate the path*, but if a flow is done with its data transfer, having if exchange dummy data just to probe capacity seems like a waste all around.

That’s an example of details of a possible future mechanism. And yes, I wouldn’t advocate this particular one.


> I guess what I want to ask is, how would knowing how much available but untapped capacity was available at one point help?
> 
> 
> *) stuff like deducing capacity from packet pair interval at the receiver (assuming packets sent back to back) is notoriously imprecise, so unless "chirping" overcomes that imprecision without costing too many round trips worth of noise supression measuring capacity by causing congestion is the only way. Not great.

We’re dealing with a world of heuristics here; nothing is ever 100% known beforehand about Internet transfers in general. So, something like your aside here would also never be a binary 100% bad case - how bad it would be depends on many parameters which are worth investigating, per envisioned mechanism - and then we’re doing research. So “I imagine mechanism X, but this will never work” is exactly the kind of discussion that's a waste of time, IMO.


>> I don’t even think that this name has that kind of history. My point was that they’re called PEPs because they’re *meant* to improve performance;
> 
> 	[SM] That is not really how our economic system works... products are primarily intended to generate more revenue than cost, it helps if they offer something to the customer, but that is really just a means to extract revenue...
> 
> 
>> that’s what they’re designed for. You describe “a PEP that does not enhance performance”, which, to me, is like talking about a web server that doesn’t serve web pages. Sure, not all PEPs may always work well, but they should - that’s their raison d’être.
> 
> 	[SM] That is a very optimistic view, I would love to be able to share.

This is not about optimism. If a device doesn’t improve performance, it shouldn’t be called a PEP. If someone does it nevertheless, okay, but then that’s not the device we’re discussing here.
I.e.: I say “but we’re discussing a web server, not a DNS server” and you say “this is optimistic”. That’s just weird.


>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>> 
>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>> 
>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>> 
>>> 	[SM] Again, I agree it could in theory especially if well-architected. 
>> 
>> That’s what I’m advocating.
> 
> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?

It doesn’t really exist yet - else I wouldn’t need to advocate it  :-)   but: e.g., for QUIC, one could extend MASQUE proxies with performance-enhancing functions. I believe that colleagues from Ericsson are advocating that. For TCP, RFC 8803 sets a very good example, IMO.


>>> 	[SM] This is no invention, but how capitalism works, sorry. The party paying for the PEP decides on using it based on the advantages it offers for them. E.g. a mobile carrier that (in the past) forcible managed to downgrade the quality of streaming video over mobile links without giving the paying end-user an option to use either choppy high resolution or smooth low resolution video. By the way, that does not make the operator evil, it is just that operator and paying customers goals and desires are not all that well aligned (e.g. the operator wants to maximize revenue, the customer to minimize cost).
>> 
>> You claim that these goals and desires are not well aligned (and a PEP is then an instrument in this evil)
> 
> 	[SM] Again this is expected behavior in our economic system, I have not and will not classify that as "evil", but I will also not start believing that companies offer products just to get a warm and fuzzy feeling. It is part of the principle of how a market economy works that the goals of the entities involved are opposite of each other, that is how a market is supposed to optimize resource allocation.
> 
>> - do you have any proof, or even anecdotes, to support that claim?
> 
> 	[SM] The claim that sellers want the highest revenue/cost ratio while buyers want the lowest cost/utility seems hardly controversial or requiring a citation.

Aha. No, I understand you now. But it *is* more subtle than that, because the market will also make unhappy customers switch to a different company, except when they have no choice (monopoly).


>> I would think that operators generally try to make their customers happy (or they would switch to different operators).  Yes there may be some misalignments in incentives, but I believe that these are more subtle points. E.g., who wants a choppy high resolution video? Do such users really exist?
> 
> 	[SM] I might be able to buffer that choppy video well enough to allow smooth playback at the desired higher resolution/quality (or I might be happy with a few seconds to compare quality of displays), given that I essentially buy internet access from my mobile carrier that carrier should get out of my way. (However if the carrier also offers "video-optimization" as an opt-in feature end-users can toggle at will that is a different kettle of fish and something I would consider good service). IIRC a German carrier was simply downforcing quality for all video streaming at all time, mostly to minimize cost and bandwidth usage, which pretty much looks like an exercise to minimize operational cost and not to increase customer satisfaction. So yes there are "misalignments in incentives" that are inherent and structural to the way we organize our society. (I am sort of okay living with that, but I will not sugar coat it).

You say “you” might be able to buffer - but was this about a specific application?
Anyway, let’s say a provider downgrades Netflix and instead tries people to opt in to their own, costly service instead. I suspect that this would make most customers want to switch their provider.
So - in this way, the market does enforce a certain alignment of interests nevertheless.


>> Now, if we just had an in-network device that could divide the path into a “core” segment where it’s safe to use a pretty large IW value, and a downstream segment where the IW value may need be smaller, but a certain workable range might be known to the device, because that devices sits right at the edge…
> 
> 	[SM] This seems to be problematic if end-to-end encryption is desired, no? But essentially this also seems to be implemented already, except that we call these things CDNs instead of proxies ;) (kidding!)

Potentially solvable with MASQUE proxies (see above).


>>> 	[SM] I understand, however I see clear reasons why L4S is detrimental to your stated goals as it will getting more information from the network less likely. I also tried to explain, why I believe that to be a theoretically viable way forwards to improve slow-start dynamics. Maybe show why my proposal is bunk while completely ignoring L4S? Or is that the kind of "particular solution" you do not want to discuss at the current stage?
>> 
>> I’d say the latter. We could spend weeks of time and tonds of emails discussing explicit-feedback based schemes…  instead, if you think your idea is good, why not build it, test it, and evaluate its trade-offs?
> 
> 	[SM] In all honesty, because my day-job is in a pretty different field and I do not believe I can or even would want to perform publishable CS research after hours (let alone find a journal accepting layman submissions without any relevant affiliation).

That’s totally fine, but then it’s not worth the time to discuss the idea, frankly.


>> I don’t see L4S as being *detrimental* to our stated goals, BTW - but, as it stands, I see limitations in its usefulness because TCP Prague (AFAIK) only changes Congestion Avoidance, at least up to now. I’m getting the impression that Congestion Avoidance with a greedy sender is a rare animal.
> 
> 	[SM] Erm, DASH-type traffic seems quite common, no? There the individual segments transmitted can be large enough to reach (close to) capacity?

Here I don’t have enough data (though there are some papers we could dig into)…  but I suspect that the answer is: no, they are probably not large enough to reach capacity (and "close to”: who knows?).


Cheers,
Michael

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi Michael,

On 13 July 2022 00:27:54 CEST, Michael Welzl <michawe@ifi.uio.no> wrote:
>I’ll cut some clutter:
>
>
>>> What you’re doing is jumping ahead. I suggest doing this with research rather than an email discussion, but that’s what we’re now already into.
>> 
>> 	[SM] Sure, except my day job
>(snip)
>
>that’s ok - I wasn’t trying to tell you off for discussing ideas!   - I was just trying to clarify:
>1. we agree there’s a problem;

Apparently I was not clear enough; no, I do not see current slow start as a big problem. As I said some quantitative fine tuning might be possible/in order. But I do not see a qualitative change as a realistic possibility. Yes, as a researcher in the field I might see things your way bit as the amateur I am, I disagree. But no matter the perceived magnitude of possible improvements research in this field is interesting and worthwhile, I think.


>2. our paper made the point that research is needed;
>3. now you’re discussing possible research ideas, which is the step that comes after.

True, but that is because that is what interested me on your paper, and I am confident as an outsider unlikely to contribute actual research my opinion on whether research is needed would be of interest to no one.


>
>… and trying to discuss 3. in depth easily gets hand-wavy, unless one actually does the research first. My So all I say is that this discussion is a bit pointless  (I’m trying to end it - please!  :-)  )

Fair enough, thanks for the friendly and interesting discussion.

Regards
        Sebastian


>
>
>> Maybe a reason for me to respectfully bow out of this discussion as talk is cheap and easy to come by even without me helping.
>
>Well, let’s at least try to keep this short…
>
>
>>> Okay, there is ONE thing that such a flow gets: the RTT. “Blind except for RTT measurements”, then.
>> 
>> 	[SM] I guess your point is "flow does not know the maximal capacity it could have gotten"?
>
>Yes.
>
>
>>> Importantly, such a flow never learns how large its cwnd *could* have become without ever causing a problem. Perhaps 10 times more? 100 times?
>> 
>> 	[SM] Sure. ATM the only way to learn a path's capacity is actually to saturate the path*, but if a flow is done with its data transfer, having if exchange dummy data just to probe capacity seems like a waste all around.
>
>That’s an example of details of a possible future mechanism. And yes, I wouldn’t advocate this particular one.
>
>
>> I guess what I want to ask is, how would knowing how much available but untapped capacity was available at one point help?
>> 
>> 
>> *) stuff like deducing capacity from packet pair interval at the receiver (assuming packets sent back to back) is notoriously imprecise, so unless "chirping" overcomes that imprecision without costing too many round trips worth of noise supression measuring capacity by causing congestion is the only way. Not great.
>
>We’re dealing with a world of heuristics here; nothing is ever 100% known beforehand about Internet transfers in general. So, something like your aside here would also never be a binary 100% bad case - how bad it would be depends on many parameters which are worth investigating, per envisioned mechanism - and then we’re doing research. So “I imagine mechanism X, but this will never work” is exactly the kind of discussion that's a waste of time, IMO.
>
>
>>> I don’t even think that this name has that kind of history. My point was that they’re called PEPs because they’re *meant* to improve performance;
>> 
>> 	[SM] That is not really how our economic system works... products are primarily intended to generate more revenue than cost, it helps if they offer something to the customer, but that is really just a means to extract revenue...
>> 
>> 
>>> that’s what they’re designed for. You describe “a PEP that does not enhance performance”, which, to me, is like talking about a web server that doesn’t serve web pages. Sure, not all PEPs may always work well, but they should - that’s their raison d’être.
>> 
>> 	[SM] That is a very optimistic view, I would love to be able to share.
>
>This is not about optimism. If a device doesn’t improve performance, it shouldn’t be called a PEP. If someone does it nevertheless, okay, but then that’s not the device we’re discussing here.
>I.e.: I say “but we’re discussing a web server, not a DNS server” and you say “this is optimistic”. That’s just weird.
>
>
>>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>> 
>>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>>> 
>>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>> 
>>>> 	[SM] Again, I agree it could in theory especially if well-architected. 
>>> 
>>> That’s what I’m advocating.
>> 
>> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?
>
>It doesn’t really exist yet - else I wouldn’t need to advocate it  :-)   but: e.g., for QUIC, one could extend MASQUE proxies with performance-enhancing functions. I believe that colleagues from Ericsson are advocating that. For TCP, RFC 8803 sets a very good example, IMO.
>
>
>>>> 	[SM] This is no invention, but how capitalism works, sorry. The party paying for the PEP decides on using it based on the advantages it offers for them. E.g. a mobile carrier that (in the past) forcible managed to downgrade the quality of streaming video over mobile links without giving the paying end-user an option to use either choppy high resolution or smooth low resolution video. By the way, that does not make the operator evil, it is just that operator and paying customers goals and desires are not all that well aligned (e.g. the operator wants to maximize revenue, the customer to minimize cost).
>>> 
>>> You claim that these goals and desires are not well aligned (and a PEP is then an instrument in this evil)
>> 
>> 	[SM] Again this is expected behavior in our economic system, I have not and will not classify that as "evil", but I will also not start believing that companies offer products just to get a warm and fuzzy feeling. It is part of the principle of how a market economy works that the goals of the entities involved are opposite of each other, that is how a market is supposed to optimize resource allocation.
>> 
>>> - do you have any proof, or even anecdotes, to support that claim?
>> 
>> 	[SM] The claim that sellers want the highest revenue/cost ratio while buyers want the lowest cost/utility seems hardly controversial or requiring a citation.
>
>Aha. No, I understand you now. But it *is* more subtle than that, because the market will also make unhappy customers switch to a different company, except when they have no choice (monopoly).
>
>
>>> I would think that operators generally try to make their customers happy (or they would switch to different operators).  Yes there may be some misalignments in incentives, but I believe that these are more subtle points. E.g., who wants a choppy high resolution video? Do such users really exist?
>> 
>> 	[SM] I might be able to buffer that choppy video well enough to allow smooth playback at the desired higher resolution/quality (or I might be happy with a few seconds to compare quality of displays), given that I essentially buy internet access from my mobile carrier that carrier should get out of my way. (However if the carrier also offers "video-optimization" as an opt-in feature end-users can toggle at will that is a different kettle of fish and something I would consider good service). IIRC a German carrier was simply downforcing quality for all video streaming at all time, mostly to minimize cost and bandwidth usage, which pretty much looks like an exercise to minimize operational cost and not to increase customer satisfaction. So yes there are "misalignments in incentives" that are inherent and structural to the way we organize our society. (I am sort of okay living with that, but I will not sugar coat it).
>
>You say “you” might be able to buffer - but was this about a specific application?
>Anyway, let’s say a provider downgrades Netflix and instead tries people to opt in to their own, costly service instead. I suspect that this would make most customers want to switch their provider.
>So - in this way, the market does enforce a certain alignment of interests nevertheless.
>
>
>>> Now, if we just had an in-network device that could divide the path into a “core” segment where it’s safe to use a pretty large IW value, and a downstream segment where the IW value may need be smaller, but a certain workable range might be known to the device, because that devices sits right at the edge…
>> 
>> 	[SM] This seems to be problematic if end-to-end encryption is desired, no? But essentially this also seems to be implemented already, except that we call these things CDNs instead of proxies ;) (kidding!)
>
>Potentially solvable with MASQUE proxies (see above).
>
>
>>>> 	[SM] I understand, however I see clear reasons why L4S is detrimental to your stated goals as it will getting more information from the network less likely. I also tried to explain, why I believe that to be a theoretically viable way forwards to improve slow-start dynamics. Maybe show why my proposal is bunk while completely ignoring L4S? Or is that the kind of "particular solution" you do not want to discuss at the current stage?
>>> 
>>> I’d say the latter. We could spend weeks of time and tonds of emails discussing explicit-feedback based schemes…  instead, if you think your idea is good, why not build it, test it, and evaluate its trade-offs?
>> 
>> 	[SM] In all honesty, because my day-job is in a pretty different field and I do not believe I can or even would want to perform publishable CS research after hours (let alone find a journal accepting layman submissions without any relevant affiliation).
>
>That’s totally fine, but then it’s not worth the time to discuss the idea, frankly.
>
>
>>> I don’t see L4S as being *detrimental* to our stated goals, BTW - but, as it stands, I see limitations in its usefulness because TCP Prague (AFAIK) only changes Congestion Avoidance, at least up to now. I’m getting the impression that Congestion Avoidance with a greedy sender is a rare animal.
>> 
>> 	[SM] Erm, DASH-type traffic seems quite common, no? There the individual segments transmitted can be large enough to reach (close to) capacity?
>
>Here I don’t have enough data (though there are some papers we could dig into)…  but I suspect that the answer is: no, they are probably not large enough to reach capacity (and "close to”: who knows?).
>
>
>Cheers,
>Michael
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[Sebastian Moeller]

Hi David,


> On Jul 12, 2022, at 21:22, David Lang <david@lang.hm> wrote:
> 
> On Tue, 12 Jul 2022, Sebastian Moeller wrote:
> 
>> Hi David,
>> 
>> Thanks!
>> 
>>> On Jul 12, 2022, at 19:56, David Lang <david@lang.hm> wrote:
>>> 
>>> On Tue, 12 Jul 2022, Sebastian Moeller via Bloat wrote:
>>> 
>>>>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>>>> 
>>>>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>>>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>>>> 
>>>>>> 	[SM] Again, I agree it could in theory especially if well-architected.
>>>>> That’s what I’m advocating.
>>>> 
>>>> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?
>>> 
>>> the windows protocols work very poorly over high latency links (i.e. long distance links) and the PEPs that short circuit those protocols make life much nicer for users as well as reducing network traffic.
>> 
>> 	[SM] Windows protocols, like in microsoft's server message block (smb) protocol or as in "protocols using data windows", like TCP's congestion and receive window?
> 
> microsoft windows smb

	[SM2] Thanks!


> 
>>> it's a nasty protocol to start with, but it's the reality on the ground and proxies do help a lot.
>> 
>> 	[SM] Are such proxies located in third party middle boxes/proxies or are these part of microsoft's software suite for enterprises (assuming the first as answer to my question)?
> 
> third party middle boxes that you put in each office as a proxy.
> 
> David Lang


[SM2] Interesting, I had actually noted that accessing files via my work VPN is a pain (in both windows and macos, as the servers use SMB). My work around was to use microsofts remote desktop (which on my access link feels reasonably snappy) to do most work remotely and also offers to transfer files, so I did all the heavy processing remotely and only exchanged either initial input or final output files, essentially working around the fact that SMB is less than impressive once the RTT goes into the milliseconds range... (However I wonder, with a filesystem essentially being a general purpose database designed for arbitrarily large blobs, how much of that issue is inherent to the problem and how much avoidable pain did microsoft add when designing their protocol?)

Kind Regards
	Sebastian

Re: [Bloat] [iccrg] Musings on the future of Internet Congestion Control

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 4605 bytes --]

On Wed, 13 Jul 2022, Sebastian Moeller wrote:

> Hi David,
>
>
>> On Jul 12, 2022, at 21:22, David Lang <david@lang.hm> wrote:
>>
>> On Tue, 12 Jul 2022, Sebastian Moeller wrote:
>>
>>> Hi David,
>>>
>>> Thanks!
>>>
>>>> On Jul 12, 2022, at 19:56, David Lang <david@lang.hm> wrote:
>>>>
>>>> On Tue, 12 Jul 2022, Sebastian Moeller via Bloat wrote:
>>>>
>>>>>>>> There are plenty of useful things that they can do and yes, I personally think they’re the way of the future - but **not** in their current form, where they must “lie” to TCP, cause ossification,
>>>>>>>
>>>>>>> 	[SM] Here I happily agree, if we can get the nagative side-effects removed that would be great, however is that actually feasible or just desirable?
>>>>>>>> etc. PEPs have never been considered as part of the congestion control design - when they came on the scene, in the IETF, they were despised for breaking the architecture, and then all the trouble with how they need to play tricks was discovered (spoofing IP addresses, making assumptions about header fields, and whatnot). That doesn’t mean that a very different kind of PEP - one which is authenticated and speaks an agreed-upon protocol - couldn’t be a good solution.
>>>>>>>
>>>>>>> 	[SM] Again, I agree it could in theory especially if well-architected.
>>>>>> That’s what I’m advocating.
>>>>>
>>>>> 	[SM] Well, can you give an example of an existing well-architected PEP as proof of principle?
>>>>
>>>> the windows protocols work very poorly over high latency links (i.e. long distance links) and the PEPs that short circuit those protocols make life much nicer for users as well as reducing network traffic.
>>>
>>> 	[SM] Windows protocols, like in microsoft's server message block (smb) protocol or as in "protocols using data windows", like TCP's congestion and receive window?
>>
>> microsoft windows smb
>
> 	[SM2] Thanks!
>
>
>>
>>>> it's a nasty protocol to start with, but it's the reality on the ground and proxies do help a lot.
>>>
>>> 	[SM] Are such proxies located in third party middle boxes/proxies or are these part of microsoft's software suite for enterprises (assuming the first as answer to my question)?
>>
>> third party middle boxes that you put in each office as a proxy.
>>
>> David Lang
>
>
> [SM2] Interesting, I had actually noted that accessing files via my work VPN is a pain (in both windows and macos, as the servers use SMB). My work around was to use microsofts remote desktop (which on my access link feels reasonably snappy) to do most work remotely and also offers to transfer files, so I did all the heavy processing remotely and only exchanged either initial input or final output files, essentially working around the fact that SMB is less than impressive once the RTT goes into the milliseconds range... (However I wonder, with a filesystem essentially being a general purpose database designed for arbitrarily large blobs, how much of that issue is inherent to the problem and how much avoidable pain did microsoft add when designing their protocol?)

It's very much a microsoft protocol issue. It's very chatty, and so very 
sensitive to high latency. the microsoft proxies implement a combination of 
caching and bulk queries to eliminate round trips for the user.

Local HTTP and DNS proxy servers do the same thing with their protocols, cache 
the data that doesn't change and only send the queries that they must over the 
networks.

Google is experimenting with similar things for mobile devices where something 
at the datacenter (with high speed links and lots of caching available) can 
assemble all the elements of a web page and send the combined result to the 
user.

With Bufferbloat, we spend a lot of time focusing on eliminating unneccessary 
latency, but the speed of light imposes minimum latency for connections, and 
there are benefits to the users to be able to further eliminate those round 
trips where possible (and where you can't eliminate them, look at shortening the 
distances)

good performance proxies are not re-coding images or things like that most of 
the time, that takes significant CPU and requires that the proxy receive the 
entire image to re-process it, instead they are protocol-aware systems that work 
to eliminate round trips to the server.

There are cases where the endpoints is expected to be low performance where it 
can make sense to re-code things from what the server provides to something that 
the client can handle with hardware acceleration, or to something that requires 
less bandwidth, but those are special cases, not the common case.

David Lang