Re: [Starlink] Starlink Digest, Vol 18, Issue 1

Re: [Starlink] Starlink Digest, Vol 18, Issue 1

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 7900 bytes --]


Hi Sebastian -
regarding slow start and video, I was just thinking and I came up with a possible answer regarding the slow start issue compatible with the Internet architecture. Read the whole piece below, I hope it makes sense.
Since the "bottleneck link" in Starlink will almost certainly be the downlink shared among multiple dishys, which to me is like the CMTS in a cable modem system, I think the reasoning is more general than Starlink.  Oddly, :-) the idea has to do with AQM and TCP flow control. Our common interest.
 
It's a concept, not a full implementation. It doesn't require massive rethinking of the whole Internet. Which means, unlike the QUIC project, it can be experimented with in just a part of the Internet, but would require a video-server partner and a bottleneck-link operator partner.
 
Please steal and improve if you like it.
 
- David
 
> Date: Thu, 1 Sep 2022 09:58:16 +0200

> From: Sebastian Moeller <moeller0@gmx.de>
> To: Ulrich Speidel <u.speidel@auckland.ac.nz>
> Cc: David Lang <david@lang.hm>, Ulrich Speidel via Starlink
> <starlink@lists.bufferbloat.net>
> Subject: Re: [Starlink] Starlink "beam spread"
 
<snip>
> I am prepared to eat crow on this in the future, but I am highly skeptical about
> CDNs in space (in spite of it being a cool project from the technological side).
>
You and me both...
> 
> *) As it looks slow start is getting a bad rep from multiple sides, but I see not
> better alternative out there that solves the challenge slow-start tackles in a
> better way. Namely gradual ramping and probing of sending rates/congestion windows
> to avoid collapse, this in turn means that short flows will never reach capacity,
> the solution to which might well be, use longer flows then...


I'm old enough to remember how the TCP slow start problem was first dealt with in HTTP pretty well (good enough for the time, and focused on the end-to-end picture of the WWW architecture (REST).
My friend Jim Gettys was involved, as were a couple of others - I think Jeff Mogul, too.
 
The answer was HTTP 1.1. That is, using a single TCP flow for all HTTP requests to a server. That would increase the flow to get through slow start more quickly (aggregating many flows) and holding the flow open, assuming (almost always correctly) that between the user's clicks and the multiple parts of content, this would keep the flow out of slow start. (HTTP 1.1's multiplexing of requests onto a single TCP flow had other advantages, too - OS Internet stacks couldn't handle many simultaneous HTTP connections, adding more delay when a web page was sourced from many servers).
 
As far as "pretty good" solution Akamai also solved the bulk of the Web's  problem - NOT by caching at all, but by letting ISPs *purchase* server capacity closer to the bulk of their consumers, and letting that *purchased* capacity pay for Akamai's costs. This moved the function of what to cache "out of the network" to the edge, and didn't require ANY changes to HTTP (in fact, it tended to concentrate more traffic into a single HTTP 1.1 flow, to a shared Akamai server.
 
You may not see where I'm going with this yet - but actually it's an End-to-End Argument against putting "function into the network" when the function is about the end points. With HTTP 1.1, a small change *at the endpoints of the Web* was easier and simpler than some hypothetical "intelligence" in the network that would obviate "slow start". With Akamai, a small change that allowed heavily trafficed web services to "buy server capacity" in Akamai's server fabric, which was "at the endpoints of the Internet" itself just distributed around the edges of the Internet was good enough.
 
So, let's look at video traffic. Well, first of all, any response time needs it has is mostly fixed by buffering at the receiver. Not needing any fixes "in the network" - you want to receive video, allocate a big buffer and fill it at the start. (This is where slow start gets in the way, perhaps).
I'm pretty sure *most* video watching doesn't go over RTP anymore - it goes over TCP now. [Dropped frames are interpolated, but instead retransmitted, because the buffer is sufficient in all video watching gear at the viewing end. RTP is useful for two-way videoconferencing, but that's because conferencing has the 100 msec. need for RTT, unless you are conferencing with Mars or the Moon.]
 
So "slow start" of some sort is the current solution that deals with the congestion burst that would disrupt others early in a video watching session.
 
But I think you are right - there are better ways to deal with the shared bottleneck link that a new video session might encounter. - something like slow start is needed ONLY if the bottleneck is *shared* among multiple users.
 
The problem is the potential of "starvation" of other flows at the bottleneck link along the video's path.
 
And this is a problem we now have one partial solution for, but it's not deployed where it needs to be. That is, in a word - "fair" queuing plus aggressive packet dropping (and maybe aggressive ECN marking). Notice I say it needs to be at the *bottleneck link*, not the home router, which isn't the bottleneck in practice.
Now "fair" queueing's most important property is that it eliminates starvation or (near starvation) caused by one flow against another.
 
And the key about putting it in the bottleneck link (with packet drops being the default means of signalling incipient congestion) is that the bottleneck link is the only place in the network where any computer knows what the competing traffic is. You need to somehow get both the new flow and the competing flows to negotiate a new "fair" balance of usage, without building up unacceptable queueing delay.
 
So where is the "bottleneck link" today?  Well, in cable Internet, it's the CMTS almost all the time. That is, not at the premises of the watcher. This is where the congestion needs to be signalled to ALL users from. Some need to slow down (including the new video watching stream, which has to slow down relative to what it is asking for which is to fill up the buffer as fast as possible when you start watching).
 
The problem with video watching is that when you tune to a new video, you want the buffer to fill "as fast as possible", but the others sharing the Internet resources don't want to be "starved". They want "fair" queuing (not Harrison Bergeron fairness - look that up), but a "best efforts" (to use the Cerf and Kahn phrase) fairness, which is non-starvation.
 
But fair queueing only works if the sources of the traffic cut their rate as a result of congestion signalling.
The real "fairness" requires pushing the problem to the endpoints. It's an end-to-end argument. That's why we need to replace "slow start" gradually with something better.
 
I don't know what the better thing is, but it needs to trigger the AQM mechanism more quickly than does slow start (including causing early signals to the sharers of the bottleneck link).
If we assume that packet drops are the only effective signals (because ECN just has failed to be sufficiently widely deployed - a problem that really, really bothers me), that means that during the initial phase of a TCP session you need some packets to get dropped between source and sink, and also some packets on the competing flows to get dropped. But you don't want too much multiplicative decrease to be triggered in the AIMD control loop of the TCPs involved (or in the QUIC control loops whatever they may be).

So ideally, a single very high-speed burst of noise packets sent over a period of say 10msec. This would probably only cause a single round of Multiplicative Decrease. These "noise packets" could be all duplicates of the same packet, and would thus not require retransmission. But they'd trigger a brief moment of congestion.

[-- Attachment #2: Type: text/html, Size: 13368 bytes --]

Re: [Starlink] Starlink Digest, Vol 18, Issue 1

[Dave Täht]

I have been meaning to reply to this for a while, but
I needed to find cites, and sorry for the top post.

IF (and that's a big IF) every bottleneck link was sufficiently FQ'd,
then there was an argument that slow start was not needed so much, that
you could just "do yourself in". I think that was one of
luca and jame's roberts papers, but I can't remember the exact one,
this one comes close, where he cites Dave Clark's "Design philosophy of
internet protocols"

https://team.inria.fr/rap/files/2013/12/KMOR05a.pdf

...

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.178.8468&rep=rep1&type=pdf

In observing how a few ISPs do rate shaping by HTB, they *are* essentially
doing FQ between customers, and it does turn out that doing FQ+AQM
for a customer "just works" - we just got libreqos.io past 10Gbit on cheap
hw. The cost of "cake" hardly enters into it, we're totally bottlenecked
on reads, not on writes.

I really like your "emit a burst of "noise" idea. It reminds me of:

"Fight fire with fire" Eliminating standing queues with large UDP packet floods"

https://core.ac.uk/download/pdf/30904825.pdf



On Thu, Sep 01, 2022 at 01:08:48PM -0400, David P. Reed via Starlink wrote:
> 
> Hi Sebastian -
> regarding slow start and video, I was just thinking and I came up with a possible answer regarding the slow start issue compatible with the Internet architecture. Read the whole piece below, I hope it makes sense.
> Since the "bottleneck link" in Starlink will almost certainly be the downlink shared among multiple dishys, which to me is like the CMTS in a cable modem system, I think the reasoning is more general than Starlink.  Oddly, :-) the idea has to do with AQM and TCP flow control. Our common interest.
>  
> It's a concept, not a full implementation. It doesn't require massive rethinking of the whole Internet. Which means, unlike the QUIC project, it can be experimented with in just a part of the Internet, but would require a video-server partner and a bottleneck-link operator partner.
>  
> Please steal and improve if you like it.
>  
> - David
>  
> > Date: Thu, 1 Sep 2022 09:58:16 +0200
> 
> > From: Sebastian Moeller <moeller0@gmx.de>
> > To: Ulrich Speidel <u.speidel@auckland.ac.nz>
> > Cc: David Lang <david@lang.hm>, Ulrich Speidel via Starlink
> > <starlink@lists.bufferbloat.net>
> > Subject: Re: [Starlink] Starlink "beam spread"
>  
> <snip>
> > I am prepared to eat crow on this in the future, but I am highly skeptical about
> > CDNs in space (in spite of it being a cool project from the technological side).
> >
> You and me both...
> > 
> > *) As it looks slow start is getting a bad rep from multiple sides, but I see not
> > better alternative out there that solves the challenge slow-start tackles in a
> > better way. Namely gradual ramping and probing of sending rates/congestion windows
> > to avoid collapse, this in turn means that short flows will never reach capacity,
> > the solution to which might well be, use longer flows then...
> 
> 
> I'm old enough to remember how the TCP slow start problem was first dealt with in HTTP pretty well (good enough for the time, and focused on the end-to-end picture of the WWW architecture (REST).
> My friend Jim Gettys was involved, as were a couple of others - I think Jeff Mogul, too.
>  
> The answer was HTTP 1.1. That is, using a single TCP flow for all HTTP requests to a server. That would increase the flow to get through slow start more quickly (aggregating many flows) and holding the flow open, assuming (almost always correctly) that between the user's clicks and the multiple parts of content, this would keep the flow out of slow start. (HTTP 1.1's multiplexing of requests onto a single TCP flow had other advantages, too - OS Internet stacks couldn't handle many simultaneous HTTP connections, adding more delay when a web page was sourced from many servers).
>  
> As far as "pretty good" solution Akamai also solved the bulk of the Web's  problem - NOT by caching at all, but by letting ISPs *purchase* server capacity closer to the bulk of their consumers, and letting that *purchased* capacity pay for Akamai's costs. This moved the function of what to cache "out of the network" to the edge, and didn't require ANY changes to HTTP (in fact, it tended to concentrate more traffic into a single HTTP 1.1 flow, to a shared Akamai server.
>  
> You may not see where I'm going with this yet - but actually it's an End-to-End Argument against putting "function into the network" when the function is about the end points. With HTTP 1.1, a small change *at the endpoints of the Web* was easier and simpler than some hypothetical "intelligence" in the network that would obviate "slow start". With Akamai, a small change that allowed heavily trafficed web services to "buy server capacity" in Akamai's server fabric, which was "at the endpoints of the Internet" itself just distributed around the edges of the Internet was good enough.
>  
> So, let's look at video traffic. Well, first of all, any response time needs it has is mostly fixed by buffering at the receiver. Not needing any fixes "in the network" - you want to receive video, allocate a big buffer and fill it at the start. (This is where slow start gets in the way, perhaps).
> I'm pretty sure *most* video watching doesn't go over RTP anymore - it goes over TCP now. [Dropped frames are interpolated, but instead retransmitted, because the buffer is sufficient in all video watching gear at the viewing end. RTP is useful for two-way videoconferencing, but that's because conferencing has the 100 msec. need for RTT, unless you are conferencing with Mars or the Moon.]
>  
> So "slow start" of some sort is the current solution that deals with the congestion burst that would disrupt others early in a video watching session.
>  
> But I think you are right - there are better ways to deal with the shared bottleneck link that a new video session might encounter. - something like slow start is needed ONLY if the bottleneck is *shared* among multiple users.
>  
> The problem is the potential of "starvation" of other flows at the bottleneck link along the video's path.
>  
> And this is a problem we now have one partial solution for, but it's not deployed where it needs to be. That is, in a word - "fair" queuing plus aggressive packet dropping (and maybe aggressive ECN marking). Notice I say it needs to be at the *bottleneck link*, not the home router, which isn't the bottleneck in practice.
> Now "fair" queueing's most important property is that it eliminates starvation or (near starvation) caused by one flow against another.
>  
> And the key about putting it in the bottleneck link (with packet drops being the default means of signalling incipient congestion) is that the bottleneck link is the only place in the network where any computer knows what the competing traffic is. You need to somehow get both the new flow and the competing flows to negotiate a new "fair" balance of usage, without building up unacceptable queueing delay.
>  
> So where is the "bottleneck link" today?  Well, in cable Internet, it's the CMTS almost all the time. That is, not at the premises of the watcher. This is where the congestion needs to be signalled to ALL users from. Some need to slow down (including the new video watching stream, which has to slow down relative to what it is asking for which is to fill up the buffer as fast as possible when you start watching).
>  
> The problem with video watching is that when you tune to a new video, you want the buffer to fill "as fast as possible", but the others sharing the Internet resources don't want to be "starved". They want "fair" queuing (not Harrison Bergeron fairness - look that up), but a "best efforts" (to use the Cerf and Kahn phrase) fairness, which is non-starvation.
>  
> But fair queueing only works if the sources of the traffic cut their rate as a result of congestion signalling.
> The real "fairness" requires pushing the problem to the endpoints. It's an end-to-end argument. That's why we need to replace "slow start" gradually with something better.
>  
> I don't know what the better thing is, but it needs to trigger the AQM mechanism more quickly than does slow start (including causing early signals to the sharers of the bottleneck link).
> If we assume that packet drops are the only effective signals (because ECN just has failed to be sufficiently widely deployed - a problem that really, really bothers me), that means that during the initial phase of a TCP session you need some packets to get dropped between source and sink, and also some packets on the competing flows to get dropped. But you don't want too much multiplicative decrease to be triggered in the AIMD control loop of the TCPs involved (or in the QUIC control loops whatever they may be).
> 
> So ideally, a single very high-speed burst of noise packets sent over a period of say 10msec. This would probably only cause a single round of Multiplicative Decrease. These "noise packets" could be all duplicates of the same packet, and would thus not require retransmission. But they'd trigger a brief moment of congestion.

> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink


-- 
Fixing the Internet on a too-tight budget: https://patreon.com/dtaht
Dave Taht, Chief bottlewasher, TekLibre