[Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Dave Taht]

Occasionally I pass along a recent paper that I don't understand in
the hope that someone can enlighten me.
This is one of those occasions, where I am trying to leverage what I
understand of existing FQ-codel behaviors against real traffic.

https://www.hindawi.com/journals/mpe/2022/4539940/

Compared to the previous study on finite-buffer M/M/1 priority queues
with time and space priority, where service times are identical and
exponentially distributed for both types of traffic, in our model we
assume that service times are different and are generally distributed
for different types of traffic. As a result, our model is more
suitable for the performance analysis of communication systems
accommodating multiple types of traffic with different service-time
distributions. For the proposed queueing model, we derive the
queue-length distributions, loss probabilities, and mean waiting times
of both types of traffic, as well as the push-out probability of
delay-sensitive traffic.

--
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

On Wed 27 Jul 2022 at 17:34, Dave Taht <dave.taht@gmail.com> wrote:

> Occasionally I pass along a recent paper that I don't understand in
> the hope that someone can enlighten me.
> This is one of those occasions, where I am trying to leverage what I
> understand of existing FQ-codel behaviors against real traffic.



I’m not sure how realistic the model is. The delay-sensitive class gets
non-preemptive service prioritization over the loss-sensitive class (
dequeue time) So far so good.
The loss-sensitive class can take advantage of the presence of
delay-sensitive packets in the queue at enqueue time if the buffer is full
by dropping delay sensitive traffic.

I don’t think it models anything useful for fq-codel.
It might be a model for loss-less networks  like fiber channel or things
like NVMe over  Fabric with Ethernet as fabric.


>
> https://www.hindawi.com/journals/mpe/2022/4539940/
>
> Compared to the previous study on finite-buffer M/M/1 priority queues
> with time and space priority, where service times are identical and
> exponentially distributed for both types of traffic, in our model we
> assume that service times are different and are generally distributed
> for different types of traffic. As a result, our model is more
> suitable for the performance analysis of communication systems
> accommodating multiple types of traffic with different service-time
> distributions. For the proposed queueing model, we derive the
> queue-length distributions, loss probabilities, and mean waiting times
> of both types of traffic, as well as the push-out probability of
> delay-sensitive traffic.
>
> --
> FQ World Domination pending:
> https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC
>

[-- Attachment #2: Type: text/html, Size: 2710 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Bless, Roland (TM)]

Hi Dave,

IMHO the problem w.r.t the applicability of most models from
queueing theory is that they only work for load < 1, whereas
we are using the network with load values ~1 (i.e., around one) due to 
congestion control feedback loops that drive the bottleneck link
to saturation (unless you consider application limited traffic sources).
So I'm not sure that you can expect interesting results from these
kind of models to understand FQ-codel 's behavior w.r.t. to "real
traffic". So it also depends on how accurately one can model the real
traffic source aggregate that is actually a mixture of short-lived and
longer-lived flows with a CC feedback loop.

Regards,
  Roland

On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
> Occasionally I pass along a recent paper that I don't understand in
> the hope that someone can enlighten me.
> This is one of those occasions, where I am trying to leverage what I
> understand of existing FQ-codel behaviors against real traffic.
> 
> https://www.hindawi.com/journals/mpe/2022/4539940/
> 
> Compared to the previous study on finite-buffer M/M/1 priority queues
> with time and space priority, where service times are identical and
> exponentially distributed for both types of traffic, in our model we
> assume that service times are different and are generally distributed
> for different types of traffic. As a result, our model is more
> suitable for the performance analysis of communication systems
> accommodating multiple types of traffic with different service-time
> distributions. For the proposed queueing model, we derive the
> queue-length distributions, loss probabilities, and mean waiting times
> of both types of traffic, as well as the push-out probability of
> delay-sensitive traffic.

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Bjørn Ivar Teigen]

[-- Attachment #1: Type: text/plain, Size: 3363 bytes --]

Hi everyone,

Interesting paper Dave, I've got a few thoughts:

I like the split into delay-sensitive and loss-sensitive data. Different
applications can have different needs and this split allows a queuing
algorithm to take those differences into account. Not the first time I've
seen this kind of split, but the other one I've seen used M/M/1/k queues
(document here:
https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch
)

That said, the performance metrics are derived from the embedded Markov
chain of the queuing system. This means the metrics are averages over *all
of time*, and thus there can be shorter periods (seconds, minutes, hours)
of much worse than average performance. Therefore the conclusions of the
paper should be taken with a grain of salt in my opinion.

On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <
starlink@lists.bufferbloat.net> wrote:

> Hi Dave,
>
> IMHO the problem w.r.t the applicability of most models from
> queueing theory is that they only work for load < 1, whereas
> we are using the network with load values ~1 (i.e., around one) due to
> congestion control feedback loops that drive the bottleneck link
> to saturation (unless you consider application limited traffic sources).
>

To be fair there are queuing theory models that include packet loss (which
is the case for the paper Dave is asking about here), and these can work
perfectly well for load > 1. Agree about the CC feedback loops affecting
the results though. Even if the distributions are general in the paper,
they still assume samples are IID which is not true for real networks.
Feedback loops make real traffic self-correlated, which makes the short
periods of worse than average performance worse and more frequent than IID
models might suggest.

Regards,
Bjørn Ivar


>
> Regards,
>   Roland
>
> On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
> > Occasionally I pass along a recent paper that I don't understand in
> > the hope that someone can enlighten me.
> > This is one of those occasions, where I am trying to leverage what I
> > understand of existing FQ-codel behaviors against real traffic.
> >
> > https://www.hindawi.com/journals/mpe/2022/4539940/
> >
> > Compared to the previous study on finite-buffer M/M/1 priority queues
> > with time and space priority, where service times are identical and
> > exponentially distributed for both types of traffic, in our model we
> > assume that service times are different and are generally distributed
> > for different types of traffic. As a result, our model is more
> > suitable for the performance analysis of communication systems
> > accommodating multiple types of traffic with different service-time
> > distributions. For the proposed queueing model, we derive the
> > queue-length distributions, loss probabilities, and mean waiting times
> > of both types of traffic, as well as the push-out probability of
> > delay-sensitive traffic.
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Bjørn Ivar Teigen
Head of Research
+47 47335952 | bjorn@domos.no <name@domos.no> | www.domos.no

[-- Attachment #2: Type: text/html, Size: 6017 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Sebastian Moeller]

Hi all,


> On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
> 
> Hi everyone,
> 
> Interesting paper Dave, I've got a few thoughts:
> 
> I like the split into delay-sensitive and loss-sensitive data.

However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).

About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.

Regards
	Sebastian



> Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch) 
> 
> That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
> 
> On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
> Hi Dave,
> 
> IMHO the problem w.r.t the applicability of most models from
> queueing theory is that they only work for load < 1, whereas
> we are using the network with load values ~1 (i.e., around one) due to 
> congestion control feedback loops that drive the bottleneck link
> to saturation (unless you consider application limited traffic sources).
> 
> To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
> 
> Regards,
> Bjørn Ivar
>  
> 
> Regards,
>   Roland
> 
> On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
> > Occasionally I pass along a recent paper that I don't understand in
> > the hope that someone can enlighten me.
> > This is one of those occasions, where I am trying to leverage what I
> > understand of existing FQ-codel behaviors against real traffic.
> > 
> > https://www.hindawi.com/journals/mpe/2022/4539940/
> > 
> > Compared to the previous study on finite-buffer M/M/1 priority queues
> > with time and space priority, where service times are identical and
> > exponentially distributed for both types of traffic, in our model we
> > assume that service times are different and are generally distributed
> > for different types of traffic. As a result, our model is more
> > suitable for the performance analysis of communication systems
> > accommodating multiple types of traffic with different service-time
> > distributions. For the proposed queueing model, we derive the
> > queue-length distributions, loss probabilities, and mean waiting times
> > of both types of traffic, as well as the push-out probability of
> > delay-sensitive traffic.
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> -- 
> Bjørn Ivar Teigen
> Head of Research
> +47 47335952 | bjorn@domos.no | www.domos.no
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Bjørn Ivar Teigen]

[-- Attachment #1: Type: text/plain, Size: 5519 bytes --]

Very good point. Perhaps we can think of it as "at what point does delay
equal loss?". As you say, too much latency (causing reordering for
instance, or triggering an algorithm to smooth over missing data), is
functionally equivalent to loss, and therefore buffering beyond that point
is making things worse by delaying more traffic. The point at which this
kicks in varies a lot between applications though, so some kind of
classification might still make sense.

In a way, I think FQ_Codel does this classification implicitly by treating
sparse and non-sparse flows differently.

- Bjørn Ivar

On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi all,
>
>
> > On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <
> starlink@lists.bufferbloat.net> wrote:
> >
> > Hi everyone,
> >
> > Interesting paper Dave, I've got a few thoughts:
> >
> > I like the split into delay-sensitive and loss-sensitive data.
>
> However often real data is slightly different (e.g. not nicely either
> delay- or loss-sensitive)... e.g. for "real-time" games you have both delay
> and loss sensitivity (similarly for VoIP), however both can deal with
> occasional lost or delayed packets (if the delay is large enough to say be
> re-ordered with the temporally next data packet (voice sample in VoIP,
> server-tick update in games), that packet's data will likely not be
> evaluated at all). And large scale bulk downloads are both tolerant to
> delay and occasional loss. So if we think about a state space spanned by a
> delay and a loss-sensitivity axis, I predict most real traffic types will
> cluster somewhat around the diagonal (more or less closely).
>
> About the rest of the paper I have nothing to contribute, since I did not
> spend the time to work though it.
>
> Regards
>         Sebastian
>
>
>
> > Different applications can have different needs and this split allows a
> queuing algorithm to take those differences into account. Not the first
> time I've seen this kind of split, but the other one I've seen used M/M/1/k
> queues (document here:
> https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>
> >
> > That said, the performance metrics are derived from the embedded Markov
> chain of the queuing system. This means the metrics are averages over *all
> of time*, and thus there can be shorter periods (seconds, minutes, hours)
> of much worse than average performance. Therefore the conclusions of the
> paper should be taken with a grain of salt in my opinion.
> >
> > On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <
> starlink@lists.bufferbloat.net> wrote:
> > Hi Dave,
> >
> > IMHO the problem w.r.t the applicability of most models from
> > queueing theory is that they only work for load < 1, whereas
> > we are using the network with load values ~1 (i.e., around one) due to
> > congestion control feedback loops that drive the bottleneck link
> > to saturation (unless you consider application limited traffic sources).
> >
> > To be fair there are queuing theory models that include packet loss
> (which is the case for the paper Dave is asking about here), and these can
> work perfectly well for load > 1. Agree about the CC feedback loops
> affecting the results though. Even if the distributions are general in the
> paper, they still assume samples are IID which is not true for real
> networks. Feedback loops make real traffic self-correlated, which makes the
> short periods of worse than average performance worse and more frequent
> than IID models might suggest.
> >
> > Regards,
> > Bjørn Ivar
> >
> >
> > Regards,
> >   Roland
> >
> > On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
> > > Occasionally I pass along a recent paper that I don't understand in
> > > the hope that someone can enlighten me.
> > > This is one of those occasions, where I am trying to leverage what I
> > > understand of existing FQ-codel behaviors against real traffic.
> > >
> > > https://www.hindawi.com/journals/mpe/2022/4539940/
> > >
> > > Compared to the previous study on finite-buffer M/M/1 priority queues
> > > with time and space priority, where service times are identical and
> > > exponentially distributed for both types of traffic, in our model we
> > > assume that service times are different and are generally distributed
> > > for different types of traffic. As a result, our model is more
> > > suitable for the performance analysis of communication systems
> > > accommodating multiple types of traffic with different service-time
> > > distributions. For the proposed queueing model, we derive the
> > > queue-length distributions, loss probabilities, and mean waiting times
> > > of both types of traffic, as well as the push-out probability of
> > > delay-sensitive traffic.
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> >
> >
> > --
> > Bjørn Ivar Teigen
> > Head of Research
> > +47 47335952 | bjorn@domos.no | www.domos.no
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>
>

-- 
Bjørn Ivar Teigen
Head of Research
+47 47335952 | bjorn@domos.no <name@domos.no> | www.domos.no

[-- Attachment #2: Type: text/html, Size: 8600 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Dave Taht]

thx for the comments everyone!

On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
<starlink@lists.bufferbloat.net> wrote:
>
> Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
>
> In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.

the implicit flow analysis of fq_codel paper toke did is here:
http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
It's a really nice feature!, and helps a lot when also applied to wifi
station scheduling.

I have sometimes thought that increasing to quantum to account for two
paced packets in a row (at high rates) was a good idea,
other times having paced transports analyze the "beat frequency" of
sending packets through fq_codel vs a vs the ack flow characteristics
(for example, filtering) might be useful.

Imagine that instead of sending packets on a fixed but increasing
pacing schedule within an RTT thusly

PPPPPPPPPP # IW10 burst
PP      PP      PP     PP    PP # often about 24 packets in what we
think the RTT is

PP  PP  PP  PP PP PP PP

PPPPPPPPPPPPPPPPPP

PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
yes this is inaccurate a model in a zillion ways, forgive me for
purposes of extrapolation in ascii text)

If instead...

You broke up the pacing within an RTT on an actual curve, selecting
some random segment out of PI as your actual starting point, say, at
3.14596 here.

PPPPPP PPPPP PPP
PPPPP PPPPPPPP
PPPPPPPPP PPP PP

3.14159265358979323846264338327950288419716939937510
  58209749445923078164062862089986280348253421170679
  82148086513282306647093844609550582231725359408128
  48111745028410270193852110555964462294895493038196
  44288109756659334461284756482337867831652712019091
  45648566923460348610454326648213393607260249141273
  72458700660631558817488152092096282925409171536436
  78925903600113305305488204665213841469519415116094
  33057270365759591953092186117381932611793105118548
  07446237996274956735188575272489122793818301194912
  98336733624406566430860213949463952247371907021798
  60943702770539217176293176752384674818467669405132
  00056812714526356082778577134275778960917363717872
  14684409012249534301465495853710507922796892589235
  42019956112129021960864034418159813629774771309960
  51870721134999999837297804995105973173281609631859
  50244594553469083026425223082533446850352619311881
  71010003137838752886587533208381420617177669147303
  59825349042875546873115956286388235378759375195778
  18577805321712268066130019278766111959092164201989

what could you learn?


> - Bjørn Ivar
>
> On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> Hi all,
>>
>>
>> > On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >
>> > Hi everyone,
>> >
>> > Interesting paper Dave, I've got a few thoughts:
>> >
>> > I like the split into delay-sensitive and loss-sensitive data.
>>
>> However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
>>
>> About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
>>
>> Regards
>>         Sebastian
>>
>>
>>
>> > Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>> >
>> > That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
>> >
>> > On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
>> > Hi Dave,
>> >
>> > IMHO the problem w.r.t the applicability of most models from
>> > queueing theory is that they only work for load < 1, whereas
>> > we are using the network with load values ~1 (i.e., around one) due to
>> > congestion control feedback loops that drive the bottleneck link
>> > to saturation (unless you consider application limited traffic sources).
>> >
>> > To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
>> >
>> > Regards,
>> > Bjørn Ivar
>> >
>> >
>> > Regards,
>> >   Roland
>> >
>> > On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
>> > > Occasionally I pass along a recent paper that I don't understand in
>> > > the hope that someone can enlighten me.
>> > > This is one of those occasions, where I am trying to leverage what I
>> > > understand of existing FQ-codel behaviors against real traffic.
>> > >
>> > > https://www.hindawi.com/journals/mpe/2022/4539940/
>> > >
>> > > Compared to the previous study on finite-buffer M/M/1 priority queues
>> > > with time and space priority, where service times are identical and
>> > > exponentially distributed for both types of traffic, in our model we
>> > > assume that service times are different and are generally distributed
>> > > for different types of traffic. As a result, our model is more
>> > > suitable for the performance analysis of communication systems
>> > > accommodating multiple types of traffic with different service-time
>> > > distributions. For the proposed queueing model, we derive the
>> > > queue-length distributions, loss probabilities, and mean waiting times
>> > > of both types of traffic, as well as the push-out probability of
>> > > delay-sensitive traffic.
>> > _______________________________________________
>> > Starlink mailing list
>> > Starlink@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/starlink
>> >
>> >
>> > --
>> > Bjørn Ivar Teigen
>> > Head of Research
>> > +47 47335952 | bjorn@domos.no | www.domos.no
>> > _______________________________________________
>> > Starlink mailing list
>> > Starlink@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/starlink
>>
>
>
> --
> Bjørn Ivar Teigen
> Head of Research
> +47 47335952 | bjorn@domos.no | www.domos.no
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Dave Taht]

To give credit where credit is due, "packet chirping" had been
explored before in the context
of the l4s early marking ecn effort:

https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf

It died here: https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf
For now.

On Thu, Jul 28, 2022 at 6:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> thx for the comments everyone!
>
> On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
> <starlink@lists.bufferbloat.net> wrote:
> >
> > Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
> >
> > In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.
>
> the implicit flow analysis of fq_codel paper toke did is here:
> http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
> It's a really nice feature!, and helps a lot when also applied to wifi
> station scheduling.
>
> I have sometimes thought that increasing to quantum to account for two
> paced packets in a row (at high rates) was a good idea,
> other times having paced transports analyze the "beat frequency" of
> sending packets through fq_codel vs a vs the ack flow characteristics
> (for example, filtering) might be useful.
>
> Imagine that instead of sending packets on a fixed but increasing
> pacing schedule within an RTT thusly
>
> PPPPPPPPPP # IW10 burst
> PP      PP      PP     PP    PP # often about 24 packets in what we
> think the RTT is
>
> PP  PP  PP  PP PP PP PP
>
> PPPPPPPPPPPPPPPPPP
>
> PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
> yes this is inaccurate a model in a zillion ways, forgive me for
> purposes of extrapolation in ascii text)
>
> If instead...
>
> You broke up the pacing within an RTT on an actual curve, selecting
> some random segment out of PI as your actual starting point, say, at
> 3.14596 here.
>
> PPPPPP PPPPP PPP
> PPPPP PPPPPPPP
> PPPPPPPPP PPP PP
>
> 3.14159265358979323846264338327950288419716939937510
>   58209749445923078164062862089986280348253421170679
>   82148086513282306647093844609550582231725359408128
>   48111745028410270193852110555964462294895493038196
>   44288109756659334461284756482337867831652712019091
>   45648566923460348610454326648213393607260249141273
>   72458700660631558817488152092096282925409171536436
>   78925903600113305305488204665213841469519415116094
>   33057270365759591953092186117381932611793105118548
>   07446237996274956735188575272489122793818301194912
>   98336733624406566430860213949463952247371907021798
>   60943702770539217176293176752384674818467669405132
>   00056812714526356082778577134275778960917363717872
>   14684409012249534301465495853710507922796892589235
>   42019956112129021960864034418159813629774771309960
>   51870721134999999837297804995105973173281609631859
>   50244594553469083026425223082533446850352619311881
>   71010003137838752886587533208381420617177669147303
>   59825349042875546873115956286388235378759375195778
>   18577805321712268066130019278766111959092164201989
>
> what could you learn?
>
>
> > - Bjørn Ivar
> >
> > On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> Hi all,
> >>
> >>
> >> > On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
> >> >
> >> > Hi everyone,
> >> >
> >> > Interesting paper Dave, I've got a few thoughts:
> >> >
> >> > I like the split into delay-sensitive and loss-sensitive data.
> >>
> >> However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
> >>
> >> About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
> >>
> >> Regards
> >>         Sebastian
> >>
> >>
> >>
> >> > Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
> >> >
> >> > That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
> >> >
> >> > On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
> >> > Hi Dave,
> >> >
> >> > IMHO the problem w.r.t the applicability of most models from
> >> > queueing theory is that they only work for load < 1, whereas
> >> > we are using the network with load values ~1 (i.e., around one) due to
> >> > congestion control feedback loops that drive the bottleneck link
> >> > to saturation (unless you consider application limited traffic sources).
> >> >
> >> > To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
> >> >
> >> > Regards,
> >> > Bjørn Ivar
> >> >
> >> >
> >> > Regards,
> >> >   Roland
> >> >
> >> > On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
> >> > > Occasionally I pass along a recent paper that I don't understand in
> >> > > the hope that someone can enlighten me.
> >> > > This is one of those occasions, where I am trying to leverage what I
> >> > > understand of existing FQ-codel behaviors against real traffic.
> >> > >
> >> > > https://www.hindawi.com/journals/mpe/2022/4539940/
> >> > >
> >> > > Compared to the previous study on finite-buffer M/M/1 priority queues
> >> > > with time and space priority, where service times are identical and
> >> > > exponentially distributed for both types of traffic, in our model we
> >> > > assume that service times are different and are generally distributed
> >> > > for different types of traffic. As a result, our model is more
> >> > > suitable for the performance analysis of communication systems
> >> > > accommodating multiple types of traffic with different service-time
> >> > > distributions. For the proposed queueing model, we derive the
> >> > > queue-length distributions, loss probabilities, and mean waiting times
> >> > > of both types of traffic, as well as the push-out probability of
> >> > > delay-sensitive traffic.
> >> > _______________________________________________
> >> > Starlink mailing list
> >> > Starlink@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/starlink
> >> >
> >> >
> >> > --
> >> > Bjørn Ivar Teigen
> >> > Head of Research
> >> > +47 47335952 | bjorn@domos.no | www.domos.no
> >> > _______________________________________________
> >> > Starlink mailing list
> >> > Starlink@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/starlink
> >>
> >
> >
> > --
> > Bjørn Ivar Teigen
> > Head of Research
> > +47 47335952 | bjorn@domos.no | www.domos.no
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
>
>
>
> --
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 9922 bytes --]

Hi Dave,

I thought it was accepted knowledge that inter-packet delays across the internet are not reliable? Why are paced chirps immune from that problem? Or asked differently after all noise filtering required for robust and reliable operation over the existing internet, is this still going to be noticeably faster than current slow-start? In spite of the slow in the name doubling every RTT is IMHO already a pretty aggressive growth function....

Sidenote, you really think the second paper nailed PCs coffin shut?

On 29 July 2022 16:55:42 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>To give credit where credit is due, "packet chirping" had been
>explored before in the context
>of the l4s early marking ecn effort:
>
>https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf
>
>It died here: https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf
>For now.
>
>On Thu, Jul 28, 2022 at 6:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> thx for the comments everyone!
>>
>> On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
>> <starlink@lists.bufferbloat.net> wrote:
>> >
>> > Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
>> >
>> > In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.
>>
>> the implicit flow analysis of fq_codel paper toke did is here:
>> http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
>> It's a really nice feature!, and helps a lot when also applied to wifi
>> station scheduling.
>>
>> I have sometimes thought that increasing to quantum to account for two
>> paced packets in a row (at high rates) was a good idea,
>> other times having paced transports analyze the "beat frequency" of
>> sending packets through fq_codel vs a vs the ack flow characteristics
>> (for example, filtering) might be useful.
>>
>> Imagine that instead of sending packets on a fixed but increasing
>> pacing schedule within an RTT thusly
>>
>> PPPPPPPPPP # IW10 burst
>> PP      PP      PP     PP    PP # often about 24 packets in what we
>> think the RTT is
>>
>> PP  PP  PP  PP PP PP PP
>>
>> PPPPPPPPPPPPPPPPPP
>>
>> PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
>> yes this is inaccurate a model in a zillion ways, forgive me for
>> purposes of extrapolation in ascii text)
>>
>> If instead...
>>
>> You broke up the pacing within an RTT on an actual curve, selecting
>> some random segment out of PI as your actual starting point, say, at
>> 3.14596 here.
>>
>> PPPPPP PPPPP PPP
>> PPPPP PPPPPPPP
>> PPPPPPPPP PPP PP
>>
>> 3.14159265358979323846264338327950288419716939937510
>>   58209749445923078164062862089986280348253421170679
>>   82148086513282306647093844609550582231725359408128
>>   48111745028410270193852110555964462294895493038196
>>   44288109756659334461284756482337867831652712019091
>>   45648566923460348610454326648213393607260249141273
>>   72458700660631558817488152092096282925409171536436
>>   78925903600113305305488204665213841469519415116094
>>   33057270365759591953092186117381932611793105118548
>>   07446237996274956735188575272489122793818301194912
>>   98336733624406566430860213949463952247371907021798
>>   60943702770539217176293176752384674818467669405132
>>   00056812714526356082778577134275778960917363717872
>>   14684409012249534301465495853710507922796892589235
>>   42019956112129021960864034418159813629774771309960
>>   51870721134999999837297804995105973173281609631859
>>   50244594553469083026425223082533446850352619311881
>>   71010003137838752886587533208381420617177669147303
>>   59825349042875546873115956286388235378759375195778
>>   18577805321712268066130019278766111959092164201989
>>
>> what could you learn?
>>
>>
>> > - Bjørn Ivar
>> >
>> > On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
>> >>
>> >> Hi all,
>> >>
>> >>
>> >> > On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >> >
>> >> > Hi everyone,
>> >> >
>> >> > Interesting paper Dave, I've got a few thoughts:
>> >> >
>> >> > I like the split into delay-sensitive and loss-sensitive data.
>> >>
>> >> However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
>> >>
>> >> About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
>> >>
>> >> Regards
>> >>         Sebastian
>> >>
>> >>
>> >>
>> >> > Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>> >> >
>> >> > That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
>> >> >
>> >> > On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >> > Hi Dave,
>> >> >
>> >> > IMHO the problem w.r.t the applicability of most models from
>> >> > queueing theory is that they only work for load < 1, whereas
>> >> > we are using the network with load values ~1 (i.e., around one) due to
>> >> > congestion control feedback loops that drive the bottleneck link
>> >> > to saturation (unless you consider application limited traffic sources).
>> >> >
>> >> > To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
>> >> >
>> >> > Regards,
>> >> > Bjørn Ivar
>> >> >
>> >> >
>> >> > Regards,
>> >> >   Roland
>> >> >
>> >> > On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
>> >> > > Occasionally I pass along a recent paper that I don't understand in
>> >> > > the hope that someone can enlighten me.
>> >> > > This is one of those occasions, where I am trying to leverage what I
>> >> > > understand of existing FQ-codel behaviors against real traffic.
>> >> > >
>> >> > > https://www.hindawi.com/journals/mpe/2022/4539940/
>> >> > >
>> >> > > Compared to the previous study on finite-buffer M/M/1 priority queues
>> >> > > with time and space priority, where service times are identical and
>> >> > > exponentially distributed for both types of traffic, in our model we
>> >> > > assume that service times are different and are generally distributed
>> >> > > for different types of traffic. As a result, our model is more
>> >> > > suitable for the performance analysis of communication systems
>> >> > > accommodating multiple types of traffic with different service-time
>> >> > > distributions. For the proposed queueing model, we derive the
>> >> > > queue-length distributions, loss probabilities, and mean waiting times
>> >> > > of both types of traffic, as well as the push-out probability of
>> >> > > delay-sensitive traffic.
>> >> > _______________________________________________
>> >> > Starlink mailing list
>> >> > Starlink@lists.bufferbloat.net
>> >> > https://lists.bufferbloat.net/listinfo/starlink
>> >> >
>> >> >
>> >> > --
>> >> > Bjørn Ivar Teigen
>> >> > Head of Research
>> >> > +47 47335952 | bjorn@domos.no | www.domos.no
>> >> > _______________________________________________
>> >> > Starlink mailing list
>> >> > Starlink@lists.bufferbloat.net
>> >> > https://lists.bufferbloat.net/listinfo/starlink
>> >>
>> >
>> >
>> > --
>> > Bjørn Ivar Teigen
>> > Head of Research
>> > +47 47335952 | bjorn@domos.no | www.domos.no
>> > _______________________________________________
>> > Starlink mailing list
>> > Starlink@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/starlink
>>
>>
>>
>> --
>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>> Dave Täht CEO, TekLibre, LLC
>
>
>
>-- 
>FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>Dave Täht CEO, TekLibre, LLC

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 11474 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Dave Taht]

On Fri, Jul 29, 2022 at 8:29 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>
> Hi Dave,
>
> I thought it was accepted knowledge that inter-packet delays across the internet are not reliable? Why are paced chirps immune from that problem? Or asked differently after all noise filtering required for robust and reliable operation over the existing internet, is this still going to be noticeably faster than current slow-start? In spite of the slow in the name doubling every RTT is IMHO already a pretty aggressive growth function....

I would like to be able to slow "slow start" based on interpacket arrival times.

Elsewhere cloudflare is enabling packet pacing at IW1000....
https://blog.cloudflare.com/when-the-window-is-not-fully-open-your-tcp-stack-is-doing-more-than-you-think/
Increasingly it feels like there is going to be one network for the
data center, and another for everyone else.

> Sidenote, you really think the second paper nailed PCs coffin shut?

No. It did outline the problems well.

With a shift in perspective, instead of trying to smooth over the
difference in behaviors in the underlying network to fit into an 90s
isochronous packet switched network model, in the face of overwhelming
differences in real world (overwhemingly wireless) behaviors...

... It seems easier to improve the transports than fix the routers at
this point. Much as bbr models policers today perhaps it's best to try
and model more network types.

>
> On 29 July 2022 16:55:42 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> To give credit where credit is due, "packet chirping" had been
>> explored before in the context
>> of the l4s early marking ecn effort:
>>
>> https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf
>>
>> It died here: https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf
>> For now.
>>
>> On Thu, Jul 28, 2022 at 6:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>>
>>>  thx for the comments everyone!
>>>
>>>  On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
>>>  <starlink@lists.bufferbloat.net> wrote:
>>>>
>>>>
>>>>  Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
>>>>
>>>>  In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.
>>>
>>>
>>>  the implicit flow analysis of fq_codel paper toke did is here:
>>>  http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
>>>  It's a really nice feature!, and helps a lot when also applied to wifi
>>>  station scheduling.
>>>
>>>  I have sometimes thought that increasing to quantum to account for two
>>>  paced packets in a row (at high rates) was a good idea,
>>>  other times having paced transports analyze the "beat frequency" of
>>>  sending packets through fq_codel vs a vs the ack flow characteristics
>>>  (for example, filtering) might be useful.
>>>
>>>  Imagine that instead of sending packets on a fixed but increasing
>>>  pacing schedule within an RTT thusly
>>>
>>>  PPPPPPPPPP # IW10 burst
>>>  PP      PP      PP     PP    PP # often about 24 packets in what we
>>>  think the RTT is
>>>
>>>  PP  PP  PP  PP PP PP PP
>>>
>>>  PPPPPPPPPPPPPPPPPP
>>>
>>>  PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
>>>  yes this is inaccurate a model in a zillion ways, forgive me for
>>>  purposes of extrapolation in ascii text)
>>>
>>>  If instead...
>>>
>>>  You broke up the pacing within an RTT on an actual curve, selecting
>>>  some random segment out of PI as your actual starting point, say, at
>>>  3.14596 here.
>>>
>>>  PPPPPP PPPPP PPP
>>>  PPPPP PPPPPPPP
>>>  PPPPPPPPP PPP PP
>>>
>>>  3.14159265358979323846264338327950288419716939937510
>>>    58209749445923078164062862089986280348253421170679
>>>    82148086513282306647093844609550582231725359408128
>>>    48111745028410270193852110555964462294895493038196
>>>    44288109756659334461284756482337867831652712019091
>>>    45648566923460348610454326648213393607260249141273
>>>    72458700660631558817488152092096282925409171536436
>>>    78925903600113305305488204665213841469519415116094
>>>    33057270365759591953092186117381932611793105118548
>>>    07446237996274956735188575272489122793818301194912
>>>    98336733624406566430860213949463952247371907021798
>>>    60943702770539217176293176752384674818467669405132
>>>    00056812714526356082778577134275778960917363717872
>>>    14684409012249534301465495853710507922796892589235
>>>    42019956112129021960864034418159813629774771309960
>>>    51870721134999999837297804995105973173281609631859
>>>    50244594553469083026425223082533446850352619311881
>>>    71010003137838752886587533208381420617177669147303
>>>    59825349042875546873115956286388235378759375195778
>>>    18577805321712268066130019278766111959092164201989
>>>
>>>  what could you learn?
>>>
>>>
>>>>  - Bjørn Ivar
>>>>
>>>>  On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>>>
>>>>>
>>>>>  Hi all,
>>>>>
>>>>>
>>>>>>  On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
>>>>>>
>>>>>>  Hi everyone,
>>>>>>
>>>>>>  Interesting paper Dave, I've got a few thoughts:
>>>>>>
>>>>>>  I like the split into delay-sensitive and loss-sensitive data.
>>>>>
>>>>>
>>>>>  However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
>>>>>
>>>>>  About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
>>>>>
>>>>>  Regards
>>>>>          Sebastian
>>>>>
>>>>>
>>>>>
>>>>>>  Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>>>>>>
>>>>>>  That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
>>>>>>
>>>>>>  On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
>>>>>>  Hi Dave,
>>>>>>
>>>>>>  IMHO the problem w.r.t the applicability of most models from
>>>>>>  queueing theory is that they only work for load < 1, whereas
>>>>>>  we are using the network with load values ~1 (i.e., around one) due to
>>>>>>  congestion control feedback loops that drive the bottleneck link
>>>>>>  to saturation (unless you consider application limited traffic sources).
>>>>>>
>>>>>>  To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
>>>>>>
>>>>>>  Regards,
>>>>>>  Bjørn Ivar
>>>>>>
>>>>>>
>>>>>>  Regards,
>>>>>>    Roland
>>>>>>
>>>>>>  On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
>>>>>>>
>>>>>>>  Occasionally I pass along a recent paper that I don't understand in
>>>>>>>  the hope that someone can enlighten me.
>>>>>>>  This is one of those occasions, where I am trying to leverage what I
>>>>>>>  understand of existing FQ-codel behaviors against real traffic.
>>>>>>>
>>>>>>>  https://www.hindawi.com/journals/mpe/2022/4539940/
>>>>>>>
>>>>>>>  Compared to the previous study on finite-buffer M/M/1 priority queues
>>>>>>>  with time and space priority, where service times are identical and
>>>>>>>  exponentially distributed for both types of traffic, in our model we
>>>>>>>  assume that service times are different and are generally distributed
>>>>>>>  for different types of traffic. As a result, our model is more
>>>>>>>  suitable for the performance analysis of communication systems
>>>>>>>  accommodating multiple types of traffic with different service-time
>>>>>>>  distributions. For the proposed queueing model, we derive the
>>>>>>>  queue-length distributions, loss probabilities, and mean waiting times
>>>>>>>  of both types of traffic, as well as the push-out probability of
>>>>>>>  delay-sensitive traffic.
>>>>>>
>>>>>> ________________________________
>>>>>>  Starlink mailing list
>>>>>>  Starlink@lists.bufferbloat.net
>>>>>>  https://lists.bufferbloat.net/listinfo/starlink
>>>>>>
>>>>>>
>>>>>>  --
>>>>>>  Bjørn Ivar Teigen
>>>>>>  Head of Research
>>>>>>  +47 47335952 | bjorn@domos.no | www.domos.no
>>>>>> ________________________________
>>>>>>  Starlink mailing list
>>>>>>  Starlink@lists.bufferbloat.net
>>>>>>  https://lists.bufferbloat.net/listinfo/starlink
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>>  Bjørn Ivar Teigen
>>>>  Head of Research
>>>>  +47 47335952 | bjorn@domos.no | www.domos.no
>>>> ________________________________
>>>>  Starlink mailing list
>>>>  Starlink@lists.bufferbloat.net
>>>>  https://lists.bufferbloat.net/listinfo/starlink
>>>
>>>
>>>
>>>
>>>  --
>>>  FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>>>  Dave Täht CEO, TekLibre, LLC
>>
>>
>>
>>
>> --
>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>> Dave Täht CEO, TekLibre, LLC
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Sebastian Moeller]

Hi Dave,


> On Jul 29, 2022, at 19:08, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Fri, Jul 29, 2022 at 8:29 AM Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> Hi Dave,
>> 
>> I thought it was accepted knowledge that inter-packet delays across the internet are not reliable? Why are paced chirps immune from that problem? Or asked differently after all noise filtering required for robust and reliable operation over the existing internet, is this still going to be noticeably faster than current slow-start? In spite of the slow in the name doubling every RTT is IMHO already a pretty aggressive growth function....
> 
> I would like to be able to slow "slow start" based on interpacket arrival times.

	But that still tries to reason based on shitty^W noisy data... packet inter-arrival times are not good clean actionable data points, Cleaning them up is possible, sure, but will cost time, hence my question how much faster would a robust and reliable paced-chirping slow start be over "exponential growths"?
But I like your idea of "sith pacing", always in tight pairs to tickle out ACKs faster.
Side-note one way of sanity checking capacity estimates from packet pairs is to also send packets of different sizes but that still leaves the "it takes time to extract a decent estimate of the true capacity out of the junk data" problem pure packet inter-arrival times analysis has maybe reducing the time a bit. In the context of TCP it would probably also require deep surgery to release smaller than MSS segments, let alone what happens if carefully crafted packets hit GRO/GSO and friends.



> 
> Elsewhere cloudflare is enabling packet pacing at IW1000....
> https://blog.cloudflare.com/when-the-window-is-not-fully-open-your-tcp-stack-is-doing-more-than-you-think/

	Holy beverage, that is a lot, at that point they might consider increasing the gain factor instead of the offset from which to grow, so instead doubling evert RTT starting from 1000, start from 20 but grow by a factor of 4... I bet they are too chicken for that ;) (looking at coarse plots it looks like it takes only a few rounds ~ 7 for IW10_quadruple to leave IW1000_double in the dust)...

> Increasingly it feels like there is going to be one network for the
> data center, and another for everyone else.

	That happens if you hand data center guys the key to the internet, I guess...


>> Sidenote, you really think the second paper nailed PCs coffin shut?
> 
> No. It did outline the problems well.

	Knowing the sources, I have my reservations, they seems typically strong on the potential upsides and a bit dismissive/cavalier on the negative side-effects part. Also very little reception for paced chirping so far, but that can just mean that things are a bit slower in reaching production code and I got spoiled how fast codel/fq_codel made it into the kernel (both probably considerably less challenging that modifying the TCP state machine).

> With a shift in perspective, instead of trying to smooth over the
> difference in behaviors in the underlying network to fit into an 90s
> isochronous packet switched network model, in the face of overwhelming
> differences in real world (overwhemingly wireless) behaviors...

	Not convinced that is the case. I really believe the slow start is too slow argument is showing unsustainable expectations, as I said exponential window growths is pretty aggressive... the goal that every flow independent of size should achieve speeds close to true capacity simply seems to require an oracle scheduler on all senders injecting into the same path. Color me sceptical...

> ... It seems easier to improve the transports than fix the routers at
> this point. Much as bbr models policers today perhaps it's best to try
> and model more network types.

	+1; that seems preferable to all the approaches of putting more smarts into the network itself...

Thanks for the discussion ;)


Regards
	Sebastian


> 
>> 
>> On 29 July 2022 16:55:42 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> To give credit where credit is due, "packet chirping" had been
>>> explored before in the context
>>> of the l4s early marking ecn effort:
>>> 
>>> https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf
>>> 
>>> It died here: https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf
>>> For now.
>>> 
>>> On Thu, Jul 28, 2022 at 6:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>>>> 
>>>> 
>>>> thx for the comments everyone!
>>>> 
>>>> On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
>>>> <starlink@lists.bufferbloat.net> wrote:
>>>>> 
>>>>> 
>>>>> Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
>>>>> 
>>>>> In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.
>>>> 
>>>> 
>>>> the implicit flow analysis of fq_codel paper toke did is here:
>>>> http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
>>>> It's a really nice feature!, and helps a lot when also applied to wifi
>>>> station scheduling.
>>>> 
>>>> I have sometimes thought that increasing to quantum to account for two
>>>> paced packets in a row (at high rates) was a good idea,
>>>> other times having paced transports analyze the "beat frequency" of
>>>> sending packets through fq_codel vs a vs the ack flow characteristics
>>>> (for example, filtering) might be useful.
>>>> 
>>>> Imagine that instead of sending packets on a fixed but increasing
>>>> pacing schedule within an RTT thusly
>>>> 
>>>> PPPPPPPPPP # IW10 burst
>>>> PP      PP      PP     PP    PP # often about 24 packets in what we
>>>> think the RTT is
>>>> 
>>>> PP  PP  PP  PP PP PP PP
>>>> 
>>>> PPPPPPPPPPPPPPPPPP
>>>> 
>>>> PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
>>>> yes this is inaccurate a model in a zillion ways, forgive me for
>>>> purposes of extrapolation in ascii text)
>>>> 
>>>> If instead...
>>>> 
>>>> You broke up the pacing within an RTT on an actual curve, selecting
>>>> some random segment out of PI as your actual starting point, say, at
>>>> 3.14596 here.
>>>> 
>>>> PPPPPP PPPPP PPP
>>>> PPPPP PPPPPPPP
>>>> PPPPPPPPP PPP PP
>>>> 
>>>> 3.14159265358979323846264338327950288419716939937510
>>>>   58209749445923078164062862089986280348253421170679
>>>>   82148086513282306647093844609550582231725359408128
>>>>   48111745028410270193852110555964462294895493038196
>>>>   44288109756659334461284756482337867831652712019091
>>>>   45648566923460348610454326648213393607260249141273
>>>>   72458700660631558817488152092096282925409171536436
>>>>   78925903600113305305488204665213841469519415116094
>>>>   33057270365759591953092186117381932611793105118548
>>>>   07446237996274956735188575272489122793818301194912
>>>>   98336733624406566430860213949463952247371907021798
>>>>   60943702770539217176293176752384674818467669405132
>>>>   00056812714526356082778577134275778960917363717872
>>>>   14684409012249534301465495853710507922796892589235
>>>>   42019956112129021960864034418159813629774771309960
>>>>   51870721134999999837297804995105973173281609631859
>>>>   50244594553469083026425223082533446850352619311881
>>>>   71010003137838752886587533208381420617177669147303
>>>>   59825349042875546873115956286388235378759375195778
>>>>   18577805321712268066130019278766111959092164201989
>>>> 
>>>> what could you learn?
>>>> 
>>>> 
>>>>> - Bjørn Ivar
>>>>> 
>>>>> On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>>>> 
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> 
>>>>>>> On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
>>>>>>> 
>>>>>>> Hi everyone,
>>>>>>> 
>>>>>>> Interesting paper Dave, I've got a few thoughts:
>>>>>>> 
>>>>>>> I like the split into delay-sensitive and loss-sensitive data.
>>>>>> 
>>>>>> 
>>>>>> However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
>>>>>> 
>>>>>> About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
>>>>>> 
>>>>>> Regards
>>>>>>         Sebastian
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>>>>>>> 
>>>>>>> That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
>>>>>>> 
>>>>>>> On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
>>>>>>> Hi Dave,
>>>>>>> 
>>>>>>> IMHO the problem w.r.t the applicability of most models from
>>>>>>> queueing theory is that they only work for load < 1, whereas
>>>>>>> we are using the network with load values ~1 (i.e., around one) due to
>>>>>>> congestion control feedback loops that drive the bottleneck link
>>>>>>> to saturation (unless you consider application limited traffic sources).
>>>>>>> 
>>>>>>> To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> Bjørn Ivar
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>>   Roland
>>>>>>> 
>>>>>>> On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
>>>>>>>> 
>>>>>>>> Occasionally I pass along a recent paper that I don't understand in
>>>>>>>> the hope that someone can enlighten me.
>>>>>>>> This is one of those occasions, where I am trying to leverage what I
>>>>>>>> understand of existing FQ-codel behaviors against real traffic.
>>>>>>>> 
>>>>>>>> https://www.hindawi.com/journals/mpe/2022/4539940/
>>>>>>>> 
>>>>>>>> Compared to the previous study on finite-buffer M/M/1 priority queues
>>>>>>>> with time and space priority, where service times are identical and
>>>>>>>> exponentially distributed for both types of traffic, in our model we
>>>>>>>> assume that service times are different and are generally distributed
>>>>>>>> for different types of traffic. As a result, our model is more
>>>>>>>> suitable for the performance analysis of communication systems
>>>>>>>> accommodating multiple types of traffic with different service-time
>>>>>>>> distributions. For the proposed queueing model, we derive the
>>>>>>>> queue-length distributions, loss probabilities, and mean waiting times
>>>>>>>> of both types of traffic, as well as the push-out probability of
>>>>>>>> delay-sensitive traffic.
>>>>>>> 
>>>>>>> ________________________________
>>>>>>> Starlink mailing list
>>>>>>> Starlink@lists.bufferbloat.net
>>>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Bjørn Ivar Teigen
>>>>>>> Head of Research
>>>>>>> +47 47335952 | bjorn@domos.no | www.domos.no
>>>>>>> ________________________________
>>>>>>> Starlink mailing list
>>>>>>> Starlink@lists.bufferbloat.net
>>>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Bjørn Ivar Teigen
>>>>> Head of Research
>>>>> +47 47335952 | bjorn@domos.no | www.domos.no
>>>>> ________________________________
>>>>> Starlink mailing list
>>>>> Starlink@lists.bufferbloat.net
>>>>> https://lists.bufferbloat.net/listinfo/starlink
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>>>> Dave Täht CEO, TekLibre, LLC
>>> 
>>> 
>>> 
>>> 
>>> --
>>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>>> Dave Täht CEO, TekLibre, LLC
>> 
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 
> 
> 
> -- 
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 9922 bytes --]

Hi Dave,

I thought it was accepted knowledge that inter-packet delays across the internet are not reliable? Why are paced chirps immune from that problem? Or asked differently after all noise filtering required for robust and reliable operation over the existing internet, is this still going to be noticeably faster than current slow-start? In spite of the slow in the name doubling every RTT is IMHO already a pretty aggressive growth function....

Sidenote, you really think the second paper nailed PCs coffin shut?

On 29 July 2022 16:55:42 CEST, Dave Taht <dave.taht@gmail.com> wrote:
>To give credit where credit is due, "packet chirping" had been
>explored before in the context
>of the l4s early marking ecn effort:
>
>https://www.bobbriscoe.net/presents/1802paced-chirps/1802paced-chirps.pdf
>
>It died here: https://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf
>For now.
>
>On Thu, Jul 28, 2022 at 6:50 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> thx for the comments everyone!
>>
>> On Thu, Jul 28, 2022 at 3:16 AM Bjørn Ivar Teigen via Starlink
>> <starlink@lists.bufferbloat.net> wrote:
>> >
>> > Very good point. Perhaps we can think of it as "at what point does delay equal loss?". As you say, too much latency (causing reordering for instance, or triggering an algorithm to smooth over missing data), is functionally equivalent to loss, and therefore buffering beyond that point is making things worse by delaying more traffic. The point at which this kicks in varies a lot between applications though, so some kind of classification might still make sense.
>> >
>> > In a way, I think FQ_Codel does this classification implicitly by treating sparse and non-sparse flows differently.
>>
>> the implicit flow analysis of fq_codel paper toke did is here:
>> http://www.diva-portal.org/smash/get/diva2:1251687/FULLTEXT01.pdf
>> It's a really nice feature!, and helps a lot when also applied to wifi
>> station scheduling.
>>
>> I have sometimes thought that increasing to quantum to account for two
>> paced packets in a row (at high rates) was a good idea,
>> other times having paced transports analyze the "beat frequency" of
>> sending packets through fq_codel vs a vs the ack flow characteristics
>> (for example, filtering) might be useful.
>>
>> Imagine that instead of sending packets on a fixed but increasing
>> pacing schedule within an RTT thusly
>>
>> PPPPPPPPPP # IW10 burst
>> PP      PP      PP     PP    PP # often about 24 packets in what we
>> think the RTT is
>>
>> PP  PP  PP  PP PP PP PP
>>
>> PPPPPPPPPPPPPPPPPP
>>
>> PPPPPPPPPPPPPPPPPPPPPPP stready buffering and ultimately a drop (and
>> yes this is inaccurate a model in a zillion ways, forgive me for
>> purposes of extrapolation in ascii text)
>>
>> If instead...
>>
>> You broke up the pacing within an RTT on an actual curve, selecting
>> some random segment out of PI as your actual starting point, say, at
>> 3.14596 here.
>>
>> PPPPPP PPPPP PPP
>> PPPPP PPPPPPPP
>> PPPPPPPPP PPP PP
>>
>> 3.14159265358979323846264338327950288419716939937510
>>   58209749445923078164062862089986280348253421170679
>>   82148086513282306647093844609550582231725359408128
>>   48111745028410270193852110555964462294895493038196
>>   44288109756659334461284756482337867831652712019091
>>   45648566923460348610454326648213393607260249141273
>>   72458700660631558817488152092096282925409171536436
>>   78925903600113305305488204665213841469519415116094
>>   33057270365759591953092186117381932611793105118548
>>   07446237996274956735188575272489122793818301194912
>>   98336733624406566430860213949463952247371907021798
>>   60943702770539217176293176752384674818467669405132
>>   00056812714526356082778577134275778960917363717872
>>   14684409012249534301465495853710507922796892589235
>>   42019956112129021960864034418159813629774771309960
>>   51870721134999999837297804995105973173281609631859
>>   50244594553469083026425223082533446850352619311881
>>   71010003137838752886587533208381420617177669147303
>>   59825349042875546873115956286388235378759375195778
>>   18577805321712268066130019278766111959092164201989
>>
>> what could you learn?
>>
>>
>> > - Bjørn Ivar
>> >
>> > On Thu, 28 Jul 2022 at 11:55, Sebastian Moeller <moeller0@gmx.de> wrote:
>> >>
>> >> Hi all,
>> >>
>> >>
>> >> > On Jul 28, 2022, at 11:26, Bjørn Ivar Teigen via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >> >
>> >> > Hi everyone,
>> >> >
>> >> > Interesting paper Dave, I've got a few thoughts:
>> >> >
>> >> > I like the split into delay-sensitive and loss-sensitive data.
>> >>
>> >> However often real data is slightly different (e.g. not nicely either delay- or loss-sensitive)... e.g. for "real-time" games you have both delay and loss sensitivity (similarly for VoIP), however both can deal with occasional lost or delayed packets (if the delay is large enough to say be re-ordered with the temporally next data packet (voice sample in VoIP, server-tick update in games), that packet's data will likely not be evaluated at all). And large scale bulk downloads are both tolerant to delay and occasional loss. So if we think about a state space spanned by a delay and a loss-sensitivity axis, I predict most real traffic types will cluster somewhat around the diagonal (more or less closely).
>> >>
>> >> About the rest of the paper I have nothing to contribute, since I did not spend the time to work though it.
>> >>
>> >> Regards
>> >>         Sebastian
>> >>
>> >>
>> >>
>> >> > Different applications can have different needs and this split allows a queuing algorithm to take those differences into account. Not the first time I've seen this kind of split, but the other one I've seen used M/M/1/k queues (document here: https://www.researchgate.net/publication/2452029_A_Queueing_Theory_Model_that_Enables_Control_of_Loss_and_Delay_of_Traffic_at_a_Network_Switch)
>> >> >
>> >> > That said, the performance metrics are derived from the embedded Markov chain of the queuing system. This means the metrics are averages over *all of time*, and thus there can be shorter periods (seconds, minutes, hours) of much worse than average performance. Therefore the conclusions of the paper should be taken with a grain of salt in my opinion.
>> >> >
>> >> > On Thu, 28 Jul 2022 at 10:45, Bless, Roland (TM) via Starlink <starlink@lists.bufferbloat.net> wrote:
>> >> > Hi Dave,
>> >> >
>> >> > IMHO the problem w.r.t the applicability of most models from
>> >> > queueing theory is that they only work for load < 1, whereas
>> >> > we are using the network with load values ~1 (i.e., around one) due to
>> >> > congestion control feedback loops that drive the bottleneck link
>> >> > to saturation (unless you consider application limited traffic sources).
>> >> >
>> >> > To be fair there are queuing theory models that include packet loss (which is the case for the paper Dave is asking about here), and these can work perfectly well for load > 1. Agree about the CC feedback loops affecting the results though. Even if the distributions are general in the paper, they still assume samples are IID which is not true for real networks. Feedback loops make real traffic self-correlated, which makes the short periods of worse than average performance worse and more frequent than IID models might suggest.
>> >> >
>> >> > Regards,
>> >> > Bjørn Ivar
>> >> >
>> >> >
>> >> > Regards,
>> >> >   Roland
>> >> >
>> >> > On 27.07.22 at 17:34 Dave Taht via Starlink wrote:
>> >> > > Occasionally I pass along a recent paper that I don't understand in
>> >> > > the hope that someone can enlighten me.
>> >> > > This is one of those occasions, where I am trying to leverage what I
>> >> > > understand of existing FQ-codel behaviors against real traffic.
>> >> > >
>> >> > > https://www.hindawi.com/journals/mpe/2022/4539940/
>> >> > >
>> >> > > Compared to the previous study on finite-buffer M/M/1 priority queues
>> >> > > with time and space priority, where service times are identical and
>> >> > > exponentially distributed for both types of traffic, in our model we
>> >> > > assume that service times are different and are generally distributed
>> >> > > for different types of traffic. As a result, our model is more
>> >> > > suitable for the performance analysis of communication systems
>> >> > > accommodating multiple types of traffic with different service-time
>> >> > > distributions. For the proposed queueing model, we derive the
>> >> > > queue-length distributions, loss probabilities, and mean waiting times
>> >> > > of both types of traffic, as well as the push-out probability of
>> >> > > delay-sensitive traffic.
>> >> > _______________________________________________
>> >> > Starlink mailing list
>> >> > Starlink@lists.bufferbloat.net
>> >> > https://lists.bufferbloat.net/listinfo/starlink
>> >> >
>> >> >
>> >> > --
>> >> > Bjørn Ivar Teigen
>> >> > Head of Research
>> >> > +47 47335952 | bjorn@domos.no | www.domos.no
>> >> > _______________________________________________
>> >> > Starlink mailing list
>> >> > Starlink@lists.bufferbloat.net
>> >> > https://lists.bufferbloat.net/listinfo/starlink
>> >>
>> >
>> >
>> > --
>> > Bjørn Ivar Teigen
>> > Head of Research
>> > +47 47335952 | bjorn@domos.no | www.domos.no
>> > _______________________________________________
>> > Starlink mailing list
>> > Starlink@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/starlink
>>
>>
>>
>> --
>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>> Dave Täht CEO, TekLibre, LLC
>
>
>
>-- 
>FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>Dave Täht CEO, TekLibre, LLC

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 11626 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 6421 bytes --]


From: "Bless, Roland (TM)" <roland.bless@kit.edu> 
models fromqueueing theory is that they only work for load < 1, whereaswe are using the network with load values ~1 (i.e., around one) due tocongestion control feedback loops that drive the bottleneck linkto saturation (unless you consider application limited traffic sources).
 
Let me remind people here that there is some kind of really weird thinking going on here about what should be typical behavior in the Intenet when it is working well.
 
No, the goal of the Internet is not to saturate all bottlenecks at maximum capacity. That is the opposite of the goal, and it is the opposite of a sane operating point.
 
Every user seeks low response time, typically a response time on the order of the unloaded delay in the network, for ALL traffic. (whether it's the response to a file transfer or a voice frame or a WWW request). * 
 
Queueing is always suboptimal, if you can achieve goodput without introducing any queueing delay. Because a queue built up at any link delays *all* traffic sharing that link, so the overall cost to all users goes up radically when multiple streams share a link, because the queueing *delay* gets multiplied by the number of flows affected!
 
So the most desirable operating point (which Kleinrock and his students recently demonstrated with his "power metric") is to have each queue in every link average < 1 packet in length. (big or small packets, doesn't matter, actually).
 
Now the bigger issue is that this is unachievable when the flows in the network are bursty. Poisson being the least bursty, and easiest to analyze of the random processes generating flows. Typical Internet usage is incredibly bursty at all time scales, though - the burstiness is fractal when observed for real (at least if you look at time scales from 1 ms. to 1 day as your unit of analysis).  Fractal random processes of this sort are not Poisson at all.
 
So what is the best one ought to try to do?
 
Well, "keeping utilization at 100%" is never what real network operators seek. Never, ever. Instead, congestion control is focused on latency control, not optimizing utilization.
 
The only folks who seem to focus on utilization is the bean counting fraternity, because they seem to think the only cost is the wires, so you want the wires to be full. That, in my opinion, and even in most accounting systems that consider the whole enterprise rather than the wires/fibers/airtime alone, is IGNORANT and STUPID.
 
However, academics and vendors of switches care nothing about latency at network scale. They focus on wirespeed as the only metric.
 
Well, in the old Bell Telephone days, the metric of the Bell System that really mattered was not utilization on every day. Instead it was avoiding outages due to peak load. That often was "Mother's Day" - a few hours out of one day once a year. Because an outage on Mother's day (busy signals) meant major frustration!
 
Why am I talking about this?
 
Because I have been trying for decades (and I am not alone) to apply a "Clue-by-Four" to the thick skulls of folks who don't think about the Internet at scale, or even won't think about an Enterprise Internet at scale (or Starlink at scale). And it doesn't sink in.
 
Andrew Odlyzko, a brilliant mathematician at Bell Labs for most of his career also tried to point out that the utilization of the "bottleneck links" in any enterprise, up to the size of ATT in the old days, was typically tuned to < 10% of saturation at almost any time. Why? Because the CEO freaked out at the quality of service of this critical infrastructure (which means completing tasks quickly, when load is unusual) and fired people.
 
And in fact, the wires are the cheapest resource - the computers and people connected by those resources that can't do work while waiting for queueing delay are vastly more expensive to leave idle. Networks don't do "work" that matters. Queueing isn't "efficient". It's evil.
 
Which is why dropping packets rather then queueing them is *good*, if the sender will slow down and can resend them. Intentially dropped packets should be nonzero under load, if an outsider is observing for measruing quality.
 
I call this brain-miswiring about optimizing throughput to fill a bottleneck link the Hotrodder Fallacy. That's the idea that one should optimize like a drag racer optimizes his car - to burn up the tires and the engine to meet an irrelevant metric for automobiles. A nice hobby that has never improved any actual vehicle. (Even F1 racing is far more realistic, given you want your cars to last for the lifetime of the race).
 
A problem with much of the "network research" community is that it never has actually looked at what networks are used for and tried to solve those problems. Instead, they define irrelevant problems and encourage all students and professors to pursue irrelevancy.
 
Now let's look at RRUL. While it nicely looks at latency for small packets under load, it actually disregards the performance of the load streams, which are only used to "fill the pipe". Fortunately, they are TCP, so they rate limit themselves by window adjustment. But they are speed unlimited TCP streams that are meaningless.
 
Actual situations (like what happens when someone starts using BitTorrent while another in the same household is playing a twitch Multi-user FPS) don't actually look like RRUL. Because in fact the big load is ALSO fractal. Bittorrent demand isn't constant over time - far from it. It's bursty.
 
Everything is bursty at different timescales in the Internet. There are no CBR flows.
 
So if we want to address the real congestion problems, we need realistic thinking about what the real problem is.
 
Unfortunately this is not achieved by the kind of thinking that created diffserv, sadly. Because everything is bursty, just with different timescales in some cases. Even "flash override" priority traffic is incredibly bursty.
 
Coming back to Starlink - Starlink apparently is being designed by folks who really do not understand these fundamental ideas. Instead, they probably all worked in researchy environments where the practical realities of being part of a worldwide public Internet were ignored.
 (The FCC folks are almost as bad. I have found no-one at FCC engineering who understands fractal burstiness - even w.'t. the old Bell System).
 
 
 

[-- Attachment #2: Type: text/html, Size: 30096 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Bjørn Ivar Teigen]

[-- Attachment #1: Type: text/plain, Size: 8589 bytes --]

Hi David,

I agree with much of this.
The transients, what happens immediately after a traffic burst or a change
in link capacity, matter a lot because that is where the latency spikes (or
losses) are. Because of the burstiness of traffic, and I would add the
variability of links to this as well, transients are the norm, not the
exception. Almost none of the work on congestion control even comments on
performance during transient periods, or worse, disingenuously cites 99th
percentile latency when the test duration is long enough to make that
meaningless.
Don't agree that queueing is evil though. What any end-to-end connection
wants is to complete some kind of back-and-forth interaction sequence
within some kind of deadline. Sometimes queuing is better than dropping to
achieve this goal, though obviously within limits.

I've got a pet theory that the burstiness of internet traffic is an
artifact of the fact that academics and marketing departments, in a weird
kind of unholy alliance, keep focusing on peak throughput. That spawns
offloads and other devices to cram maximum bytes into minimal timeslots,
often adding queues as well to make sure all timeslots are filled to the
brim. It is not at all obvious to me that internet traffic HAS to be
bursty, it's just that we (accidentally) made it that way.

- Bjørn

On Fri, 29 Jul 2022 at 21:38, David P. Reed via Starlink <
starlink@lists.bufferbloat.net> wrote:

> From: "Bless, Roland (TM)" <roland.bless@kit.edu>
>
> models from
> queueing theory is that they only work for load < 1, whereas
> we are using the network with load values ~1 (i.e., around one) due to
> congestion control feedback loops that drive the bottleneck link
> to saturation (unless you consider application limited traffic sources).
>
>
>
> Let me remind people here that there is some kind of really weird thinking
> going on here about what should be typical behavior in the Intenet when it
> is working well.
>
>
>
> No, the goal of the Internet is not to saturate all bottlenecks at maximum
> capacity. That is the opposite of the goal, and it is the opposite of a
> sane operating point.
>
>
>
> Every user seeks low response time, typically a response time on the order
> of the unloaded delay in the network, for ALL traffic. (whether it's the
> response to a file transfer or a voice frame or a WWW request). *
>
>
>
> Queueing is always suboptimal, if you can achieve goodput without
> introducing any queueing delay. Because a queue built up at any link delays
> *all* traffic sharing that link, so the overall cost to all users goes up
> radically when multiple streams share a link, because the queueing *delay*
> gets multiplied by the number of flows affected!
>
>
>
> So the most desirable operating point (which Kleinrock and his students
> recently demonstrated with his "power metric") is to have each queue in
> every link average < 1 packet in length. (big or small packets, doesn't
> matter, actually).
>
>
>
> Now the bigger issue is that this is unachievable when the flows in the
> network are bursty. Poisson being the least bursty, and easiest to analyze
> of the random processes generating flows. Typical Internet usage is
> incredibly bursty at all time scales, though - the burstiness is fractal
> when observed for real (at least if you look at time scales from 1 ms. to 1
> day as your unit of analysis).  Fractal random processes of this sort are
> not Poisson at all.
>
>
>
> So what is the best one ought to try to do?
>
>
>
> Well, "keeping utilization at 100%" is never what real network operators
> seek. Never, ever. Instead, congestion control is focused on latency
> control, not optimizing utilization.
>
>
>
> The only folks who seem to focus on utilization is the bean counting
> fraternity, because they seem to think the only cost is the wires, so you
> want the wires to be full. That, in my opinion, and even in most accounting
> systems that consider the whole enterprise rather than the
> wires/fibers/airtime alone, is IGNORANT and STUPID.
>
>
>
> However, academics and vendors of switches care nothing about latency at
> network scale. They focus on wirespeed as the only metric.
>
>
>
> Well, in the old Bell Telephone days, the metric of the Bell System that
> really mattered was not utilization on every day. Instead it was avoiding
> outages due to peak load. That often was "Mother's Day" - a few hours out
> of one day once a year. Because an outage on Mother's day (busy signals)
> meant major frustration!
>
>
>
> Why am I talking about this?
>
>
>
> Because I have been trying for decades (and I am not alone) to apply a
> "Clue-by-Four" to the thick skulls of folks who don't think about the
> Internet at scale, or even won't think about an Enterprise Internet at
> scale (or Starlink at scale). And it doesn't sink in.
>
>
>
> Andrew Odlyzko, a brilliant mathematician at Bell Labs for most of his
> career also tried to point out that the utilization of the "bottleneck
> links" in any enterprise, up to the size of ATT in the old days, was
> typically tuned to < 10% of saturation at almost any time. Why? Because the
> CEO freaked out at the quality of service of this critical infrastructure
> (which means completing tasks quickly, when load is unusual) and fired
> people.
>
>
>
> And in fact, the wires are the cheapest resource - the computers and
> people connected by those resources that can't do work while waiting for
> queueing delay are vastly more expensive to leave idle. Networks don't do
> "work" that matters. Queueing isn't "efficient". It's evil.
>
>
>
> Which is why dropping packets rather then queueing them is *good*, if the
> sender will slow down and can resend them. Intentially dropped packets
> should be nonzero under load, if an outsider is observing for measruing
> quality.
>
>
>
> I call this brain-miswiring about optimizing throughput to fill a
> bottleneck link the Hotrodder Fallacy. That's the idea that one should
> optimize like a drag racer optimizes his car - to burn up the tires and the
> engine to meet an irrelevant metric for automobiles. A nice hobby that has
> never improved any actual vehicle. (Even F1 racing is far more realistic,
> given you want your cars to last for the lifetime of the race).
>
>
>
> A problem with much of the "network research" community is that it never
> has actually looked at what networks are used for and tried to solve those
> problems. Instead, they define irrelevant problems and encourage all
> students and professors to pursue irrelevancy.
>
>
>
> Now let's look at RRUL. While it nicely looks at latency for small packets
> under load, it actually disregards the performance of the load streams,
> which are only used to "fill the pipe". Fortunately, they are TCP, so they
> rate limit themselves by window adjustment. But they are speed unlimited
> TCP streams that are meaningless.
>
>
>
> Actual situations (like what happens when someone starts using BitTorrent
> while another in the same household is playing a twitch Multi-user FPS)
> don't actually look like RRUL. Because in fact the big load is ALSO
> fractal. Bittorrent demand isn't constant over time - far from it. It's
> bursty.
>
>
>
> Everything is bursty at different timescales in the Internet. There are no
> CBR flows.
>
>
>
> So if we want to address the real congestion problems, we need realistic
> thinking about what the real problem is.
>
>
>
> Unfortunately this is not achieved by the kind of thinking that created
> diffserv, sadly. Because everything is bursty, just with different
> timescales in some cases. Even "flash override" priority traffic is
> incredibly bursty.
>
>
>
> Coming back to Starlink - Starlink apparently is being designed by folks
> who really do not understand these fundamental ideas. Instead, they
> probably all worked in researchy environments where the practical realities
> of being part of a worldwide public Internet were ignored.
>
>  (The FCC folks are almost as bad. I have found no-one at FCC engineering
> who understands fractal burstiness - even w.'t. the old Bell System).
>
>
>
>
>
>
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
>


-- 
Bjørn Ivar Teigen
Head of Research
+47 47335952 | bjorn@domos.no <name@domos.no> | www.domos.no

[-- Attachment #2: Type: text/html, Size: 28162 bytes --]

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Juliusz Chroboczek]

> I've got a pet theory that the burstiness of internet traffic is an artifact of
> the fact that academics and marketing departments, in a weird kind of unholy
> alliance, keep focusing on peak throughput.

Hmm.

Packet pacing is counterintuitive and tricky to get right.  It's highly
counter-intuitive to most people that delaying packet transmission will
end up reducing latency.  It's tricky to decide how to pace your packets
when you don't know (1) how your congestion window is going to evolve and
(2) how the application demand is going to evolve in the near future.  Let
alone that it reduces cache locality and requires fine-grained timers,
both of which are costly if implemented in software on a general-purpose
OS.

As to academics having sold out to marketing departments -- it seems I've
missed out, where do I collect my share?

-- Juliusz

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[David Lang]

I don't think anyone sold out, I think they have just fallen into the trap of 
measuring what's easy to measure.

David Lang

On Sat, 30 Jul 2022, Juliusz Chroboczek via Starlink wrote:

> As to academics having sold out to marketing departments -- it seems I've
> missed out, where do I collect my share?

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Bless, Roland (TM)]

Hi David,

don't get me wrong, I just wanted to point out that the applicability
of some queueing models is questionable. I also prefer low delay over 
utilization and we did research in that direction with TCP LoLa
https://ieeexplore.ieee.org/document/8109356 for which low latency
is more important than throughput (whereas BBRs priority is the
opposite).

For capacity seeking flows, LoLa creates a "small" limited queue
at the bottleneck to let them detect that they have reached the 
bottleneck rate.
If your flow transmits at a far lower rate, its flow completion
time increases unnecessarily and resources are wasted. However,
link utilizations of 98% should also be fine but that is hard
to detect for end-to-end flows without explicit feedback from
the network.
So for links with a fixed outgoing rate I agree that you don't
need a queue in order to fully utilize the link. For links with
a variable output rate (e.g., wireless) though, _small_ queues may
have positive effects on throughput in case the output rate increases
again and the sender couldn't react so fast.

More comments inline...

On 29.07.22 at 21:38 David P. Reed via Starlink wrote:
> From: "Bless, Roland (TM)" <roland.bless@kit.edu>
> 
> models from
> queueing theory is that they only work for load < 1, whereas
> we are using the network with load values ~1 (i.e., around one) due to
> congestion control feedback loops that drive the bottleneck link
> to saturation (unless you consider application limited traffic sources).
> 
> Let me remind people here that there is some kind of really weird 
> thinking going on here about what should be typical behavior in the 
> Intenet when it is working well.
> 
> No, the goal of the Internet is not to saturate all bottlenecks at 
> maximum capacity. That is the opposite of the goal, and it is the 
> opposite of a sane operating point.
> 
> Every user seeks low response time, typically a response time on the 
> order of the unloaded delay in the network, for ALL traffic. (whether 
> it's the response to a file transfer or a voice frame or a WWW request). *

Yep, however, flow completion time also increases in case you don't
use the available bottleneck capacity efficiently.

> Queueing is always suboptimal, if you can achieve goodput without 
> introducing any queueing delay. Because a queue built up at any link 
> delays *all* traffic sharing that link, so the overall cost to all users 
> goes up radically when multiple streams share a link, because the 
> queueing *delay* gets multiplied by the number of flows affected!

Yep, no queue is nearly always better, however, queueing delay
might be still acceptable when the queueing delay is small.

> So the most desirable operating point (which Kleinrock and his students 
> recently demonstrated with his "power metric") is to have each queue in 
> every link average < 1 packet in length. (big or small packets, doesn't 
> matter, actually).
> 
> Now the bigger issue is that this is unachievable when the flows in the 
> network are bursty. Poisson being the least bursty, and easiest to 
> analyze of the random processes generating flows. Typical Internet usage 
> is incredibly bursty at all time scales, though - the burstiness is 
> fractal when observed for real (at least if you look at time scales from 
> 1 ms. to 1 day as your unit of analysis).  Fractal random processes of 
> this sort are not Poisson at all.
> 
> So what is the best one ought to try to do?
> 
> Well, "keeping utilization at 100%" is never what real network operators 
> seek. Never, ever. Instead, congestion control is focused on latency 
> control, not optimizing utilization.

Yes, see above.

> The only folks who seem to focus on utilization is the bean counting 
> fraternity, because they seem to think the only cost is the wires, so 
> you want the wires to be full. That, in my opinion, and even in most 
> accounting systems that consider the whole enterprise rather than the 
> wires/fibers/airtime alone, is IGNORANT and STUPID.

Yep.

> However, academics and vendors of switches care nothing about latency at 
> network scale. They focus on wirespeed as the only metric.

Nope, there have been a lots of research efforts in that direction in 
the last years to reduce latency.

> Well, in the old Bell Telephone days, the metric of the Bell System that 
> really mattered was not utilization on every day. Instead it was 
> avoiding outages due to peak load. That often was "Mother's Day" - a few 
> hours out of one day once a year. Because an outage on Mother's day 
> (busy signals) meant major frustration!
> 
> Why am I talking about this?
> 
> Because I have been trying for decades (and I am not alone) to apply a 
> "Clue-by-Four" to the thick skulls of folks who don't think about the 
> Internet at scale, or even won't think about an Enterprise Internet at 
> scale (or Starlink at scale). And it doesn't sink in.
> 
> Andrew Odlyzko, a brilliant mathematician at Bell Labs for most of his 
> career also tried to point out that the utilization of the "bottleneck 
> links" in any enterprise, up to the size of ATT in the old days, was 
> typically tuned to < 10% of saturation at almost any time. Why? Because 
> the CEO freaked out at the quality of service of this critical 
> infrastructure (which means completing tasks quickly, when load is 
> unusual) and fired people.
> 
> And in fact, the wires are the cheapest resource - the computers and 
> people connected by those resources that can't do work while waiting for 
> queueing delay are vastly more expensive to leave idle. Networks don't 
> do "work" that matters. Queueing isn't "efficient". It's evil.

Yep.

> Which is why dropping packets rather then queueing them is *good*, if 
> the sender will slow down and can resend them. Intentially dropped 
> packets should be nonzero under load, if an outsider is observing for 
> measruing quality.
> 
> I call this brain-miswiring about optimizing throughput to fill a 
> bottleneck link the Hotrodder Fallacy. That's the idea that one should 
> optimize like a drag racer optimizes his car - to burn up the tires and 
> the engine to meet an irrelevant metric for automobiles. A nice hobby 
> that has never improved any actual vehicle. (Even F1 racing is far more 
> realistic, given you want your cars to last for the lifetime of the race).
> 
> A problem with much of the "network research" community is that it never 
> has actually looked at what networks are used for and tried to solve 
> those problems. Instead, they define irrelevant problems and encourage 
> all students and professors to pursue irrelevancy.

I don't like this kind of generalization. But in one respect you might
be right: it is difficult to get insight on real network traffic for 
academics and there are only incomplete measurement data sources 
available. Occasionally, large providers provide data to some of
the academics in case there is some close collaboration. So getting
better measurement data of real networks would always be a win.

> Now let's look at RRUL. While it nicely looks at latency for small 
> packets under load, it actually disregards the performance of the load 
> streams, which are only used to "fill the pipe". Fortunately, they are 
> TCP, so they rate limit themselves by window adjustment. But they are 
> speed unlimited TCP streams that are meaningless.
> 
> Actual situations (like what happens when someone starts using 
> BitTorrent while another in the same household is playing a twitch 
> Multi-user FPS) don't actually look like RRUL. Because in fact the big 
> load is ALSO fractal. Bittorrent demand isn't constant over time - far 
> from it. It's bursty.
> 
> Everything is bursty at different timescales in the Internet. There are 
> no CBR flows.
> 
> So if we want to address the real congestion problems, we need realistic 
> thinking about what the real problem is.
> 
> Unfortunately this is not achieved by the kind of thinking that created 
> diffserv, sadly. Because everything is bursty, just with different 
> timescales in some cases. Even "flash override" priority traffic is 
> incredibly bursty.
> 
> Coming back to Starlink - Starlink apparently is being designed by folks 
> who really do not understand these fundamental ideas. Instead, they 
> probably all worked in researchy environments where the practical 
> realities of being part of a worldwide public Internet were ignored.
> 
>   (The FCC folks are almost as bad. I have found no-one at FCC 
> engineering who understands fractal burstiness - even w.'t. the old Bell 
> System).

I'm not sure, but Starlink could theoretically offer significant
latency advantages between certain destinations, but they are moot
if queueing delay is not controlled inside the Starlink network.

Regards,
  Roland

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[Sebastian Moeller]

Hi David,

interesting food for thought...


> On Jul 29, 2022, at 21:38, David P. Reed via Starlink <starlink@lists.bufferbloat.net> wrote:
> 
> From: "Bless, Roland (TM)" <roland.bless@kit.edu>
> models from
> queueing theory is that they only work for load < 1, whereas
> we are using the network with load values ~1 (i.e., around one) due to
> congestion control feedback loops that drive the bottleneck link
> to saturation (unless you consider application limited traffic sources).
>  
> Let me remind people here that there is some kind of really weird thinking going on here about what should be typical behavior in the Intenet when it is working well.
>  
> No, the goal of the Internet is not to saturate all bottlenecks at maximum capacity. That is the opposite of the goal, and it is the opposite of a sane operating point.
>  
> Every user seeks low response time, typically a response time on the order of the unloaded delay in the network, for ALL traffic. (whether it's the response to a file transfer or a voice frame or a WWW request). * 
>  
> Queueing is always suboptimal, if you can achieve goodput without introducing any queueing delay. Because a queue built up at any link delays *all* traffic sharing that link, so the overall cost to all users goes up radically when multiple streams share a link, because the queueing *delay* gets multiplied by the number of flows affected!
>  
> So the most desirable operating point (which Kleinrock and his students recently demonstrated with his "power metric") is to have each queue in every link average < 1 packet in length. (big or small packets, doesn't matter, actually).
>  
> Now the bigger issue is that this is unachievable when the flows in the network are bursty. Poisson being the least bursty, and easiest to analyze of the random processes generating flows. Typical Internet usage is incredibly bursty at all time scales, though - the burstiness is fractal when observed for real (at least if you look at time scales from 1 ms. to 1 day as your unit of analysis).  Fractal random processes of this sort are not Poisson at all.

	[SM] In this context I like the framing from the CoDel ACM paper, with the queue acting as shock absorber for burst, as you indicate bursts are unavoidable in a network with unsynchronized senders. So it seems prudent to engineer with bursts as use-case (how ever undesirable) in mind (compared to simply declaring bursts undesirable and require endpoints not to be bursty, as L4S seems to do*).


> So what is the best one ought to try to do?
>  
> Well, "keeping utilization at 100%" is never what real network operators seek. Never, ever. Instead, congestion control is focused on latency control, not optimizing utilization.

	[SM] I thought that these are not orthogonal goals and one needs to pick an operating point in the throughput<->latency gradient somehow? This becomes relevant for smaller links like internet access links more than for back bone links. It is relatively easy to drive my 100/40 link into saturation by normal usage, so I have a clear goal of keeping latency acceptable under saturating loads.


> The only folks who seem to focus on utilization is the bean counting fraternity, because they seem to think the only cost is the wires, so you want the wires to be full.

	[SM] Pithy, yet I am sure the bean counters also account for the cost of ports/interfaces ;)

> That, in my opinion, and even in most accounting systems that consider the whole enterprise rather than the wires/fibers/airtime alone, is IGNORANT and STUPID.
>  
> However, academics and vendors of switches care nothing about latency at network scale. They focus on wirespeed as the only metric.
>  
> Well, in the old Bell Telephone days, the metric of the Bell System that really mattered was not utilization on every day. Instead it was avoiding outages due to peak load. That often was "Mother's Day" - a few hours out of one day once a year. Because an outage on Mother's day (busy signals) meant major frustration!

	[SM] If one designs for a (rare) worst-case scenario, one is in the clear most of the time. I wish that was possible with my internet access link though... I get a sync of 116.7/37.0 Mbps which I shape own to a gross 105.0/36.0 it turns out it is not that hard to saturate that link occasionally with just normal usage by a family of five, so I clearly am far away from 90% reserve capacity, and I have little change of expanding the capacity by a factr of 10 within my budget...


> Why am I talking about this?
>  
> Because I have been trying for decades (and I am not alone) to apply a "Clue-by-Four" to the thick skulls of folks who don't think about the Internet at scale, or even won't think about an Enterprise Internet at scale (or Starlink at scale). And it doesn't sink in.
>  
> Andrew Odlyzko, a brilliant mathematician at Bell Labs for most of his career also tried to point out that the utilization of the "bottleneck links" in any enterprise, up to the size of ATT in the old days, was typically tuned to < 10% of saturation at almost any time. Why? Because the CEO freaked out at the quality of service of this critical infrastructure (which means completing tasks quickly, when load is unusual) and fired people.
>  
> And in fact, the wires are the cheapest resource - the computers and people connected by those resources that can't do work while waiting for queueing delay are vastly more expensive to leave idle. Networks don't do "work" that matters. Queueing isn't "efficient". It's evil.
>  
> Which is why dropping packets rather then queueing them is *good*, if the sender will slow down and can resend them. Intentially dropped packets should be nonzero under load, if an outsider is observing for measruing quality.
>  
> I call this brain-miswiring about optimizing throughput to fill a bottleneck link the Hotrodder Fallacy. That's the idea that one should optimize like a drag racer optimizes his car - to burn up the tires and the engine to meet an irrelevant metric for automobiles. A nice hobby that has never improved any actual vehicle. (Even F1 racing is far more realistic, given you want your cars to last for the lifetime of the race).
>  
> A problem with much of the "network research" community is that it never has actually looked at what networks are used for and tried to solve those problems. Instead, they define irrelevant problems and encourage all students and professors to pursue irrelevancy.
>  
> Now let's look at RRUL. While it nicely looks at latency for small packets under load, it actually disregards the performance of the load streams, which are only used to "fill the pipe".

	[SM] I respectfully disagree. They are used to simulate those "fill the pipe" flows that do happen in edge networks... think multiple machines downloading multi-gigabyte update packages (OS, games, software, ...) when ever they feel like it. The sparse latency measurement flows simulate low rate/sparse interactive traffic...
But note that depending on the situation a nominally sparse flaw can use up quite some capacity, I talked to a games who observed in riot games valorant in a mylti-player online game with 10-20 players traffic at 20 Mbps with cyclic burst 128 times a second. On a slow link that becomes a noticeable capacity hog.

> Fortunately, they are TCP, so they rate limit themselves by window adjustment. But they are speed unlimited TCP streams that are meaningless.

	[SM] Flent will however present information about those flows if instructed to do so (IIRC by the --socket-stats argument):


                                                 avg       median       99th %          # data pts
 Ping (ms) ICMP 1.1.1.1 (extra)       :        13.26        11.70        29.30 ms             1393
 Ping (ms) avg                        :        32.17          N/A          N/A ms             1607
 Ping (ms)::ICMP                      :        32.76        30.60        48.02 ms             1395
 Ping (ms)::UDP 0 (0)                 :        32.64        30.52        46.55 ms             1607
 Ping (ms)::UDP 1 (0)                 :        31.39        29.90        45.98 ms             1607
 Ping (ms)::UDP 2 (0)                 :        32.85        30.82        47.04 ms             1607
 Ping (ms)::UDP 3 (0)                 :        31.72        30.25        46.49 ms             1607
 Ping (ms)::UDP 4 (0)                 :        31.37        29.78        45.61 ms             1607
 Ping (ms)::UDP 5 (0)                 :        31.36        29.74        45.13 ms             1607
 Ping (ms)::UDP 6 (0)                 :        32.85        30.71        47.34 ms             1607
 Ping (ms)::UDP 7 (0)                 :        33.16        31.08        47.93 ms             1607
 TCP download avg                     :         7.82          N/A          N/A Mbits/s        1607
 TCP download sum                     :        62.55          N/A          N/A Mbits/s        1607
 TCP download::0 (0)                  :         7.86         7.28        13.81 Mbits/s        1607
 TCP download::1 (0)                  :         8.18         7.88        13.98 Mbits/s        1607
 TCP download::2 (0)                  :         7.62         7.05        13.81 Mbits/s        1607
 TCP download::3 (0)                  :         7.73         7.37        13.23 Mbits/s        1607
 TCP download::4 (0)                  :         7.58         7.07        13.51 Mbits/s        1607
 TCP download::5 (0)                  :         7.92         7.37        14.03 Mbits/s        1607
 TCP download::6 (0)                  :         8.07         7.58        14.33 Mbits/s        1607
 TCP download::7 (0)                  :         7.59         6.96        13.94 Mbits/s        1607
 TCP totals                           :        93.20          N/A          N/A Mbits/s        1607
 TCP upload avg                       :         3.83          N/A          N/A Mbits/s        1607
 TCP upload sum                       :        30.65          N/A          N/A Mbits/s        1607
 TCP upload::0 (0)                    :         3.82         3.86         9.57 Mbits/s        1607
 TCP upload::0 (0)::tcp_cwnd          :        14.31        14.00        23.00                 856
 TCP upload::0 (0)::tcp_delivery_rate :         3.67         3.81         4.95                 855
 TCP upload::0 (0)::tcp_pacing_rate   :         4.72         4.85         6.93                 855
 TCP upload::0 (0)::tcp_rtt           :        42.48        41.36        65.32                 851
 TCP upload::0 (0)::tcp_rtt_var       :         2.83         2.38         9.90                 851
 TCP upload::1 (0)                    :         3.90         3.94        16.49 Mbits/s        1607
 TCP upload::1 (0)::tcp_cwnd          :        14.46        14.00        23.00                 857
 TCP upload::1 (0)::tcp_delivery_rate :         3.75         3.83         5.74                 856
 TCP upload::1 (0)::tcp_pacing_rate   :         4.81         4.89         8.15                 856
 TCP upload::1 (0)::tcp_rtt           :        42.12        41.07        63.10                 852
 TCP upload::1 (0)::tcp_rtt_var       :         2.74         2.36         8.36                 852
 TCP upload::2 (0)                    :         3.85         3.96         5.11 Mbits/s        1607
 TCP upload::2 (0)::tcp_cwnd          :        14.15        14.00        22.00                 852
 TCP upload::2 (0)::tcp_delivery_rate :         3.69         3.81         4.93                 851
 TCP upload::2 (0)::tcp_pacing_rate   :         4.73         4.91         6.55                 851
 TCP upload::2 (0)::tcp_rtt           :        41.73        41.09        56.97                 851
 TCP upload::2 (0)::tcp_rtt_var       :         2.59         2.29         7.71                 851
 TCP upload::3 (0)                    :         3.81         3.95         5.32 Mbits/s        1607
 TCP upload::3 (0)::tcp_cwnd          :        13.90        14.00        21.00                 851
 TCP upload::3 (0)::tcp_delivery_rate :         3.66         3.82         4.89                 851
 TCP upload::3 (0)::tcp_pacing_rate   :         4.67         4.87         6.36                 851
 TCP upload::3 (0)::tcp_rtt           :        41.44        41.09        56.46                 847
 TCP upload::3 (0)::tcp_rtt_var       :         2.74         2.46         8.27                 847
 TCP upload::4 (0)                    :         3.77         3.88         5.35 Mbits/s        1607
 TCP upload::4 (0)::tcp_cwnd          :        13.86        14.00        21.00                 852
 TCP upload::4 (0)::tcp_delivery_rate :         3.61         3.75         4.87                 852
 TCP upload::4 (0)::tcp_pacing_rate   :         4.63         4.83         6.46                 852
 TCP upload::4 (0)::tcp_rtt           :        41.74        41.18        57.27                 850
 TCP upload::4 (0)::tcp_rtt_var       :         2.73         2.45         8.38                 850
 TCP upload::5 (0)                    :         3.83         3.93         5.60 Mbits/s        1607
 TCP upload::5 (0)::tcp_cwnd          :        13.98        14.00        22.00                 851
 TCP upload::5 (0)::tcp_delivery_rate :         3.68         3.80         5.05                 851
 TCP upload::5 (0)::tcp_pacing_rate   :         4.69         4.82         6.65                 851
 TCP upload::5 (0)::tcp_rtt           :        41.50        40.91        56.42                 847
 TCP upload::5 (0)::tcp_rtt_var       :         2.68         2.34         8.24                 847
 TCP upload::6 (0)                    :         3.86         3.97         5.60 Mbits/s        1607
 TCP upload::6 (0)::tcp_cwnd          :        14.27        14.00        22.00                 850
 TCP upload::6 (0)::tcp_delivery_rate :         3.71         3.83         5.07                 850
 TCP upload::6 (0)::tcp_pacing_rate   :         4.74         4.90         6.77                 850
 TCP upload::6 (0)::tcp_rtt           :        42.03        41.66        55.81                 850
 TCP upload::6 (0)::tcp_rtt_var       :         2.71         2.49         7.85                 850
 TCP upload::7 (0)                    :         3.81         3.92         5.18 Mbits/s        1607
 TCP upload::7 (0)::tcp_cwnd          :        14.01        14.00        22.00                 850
 TCP upload::7 (0)::tcp_delivery_rate :         3.67         3.82         4.94                 849
 TCP upload::7 (0)::tcp_pacing_rate   :         4.57         4.69         6.52                 850
 TCP upload::7 (0)::tcp_rtt           :        42.62        42.16        56.20                 847
 TCP upload::7 (0)::tcp_rtt_var       :         2.50         2.19         8.02                 847
 cpu_stats_root@192.168.42.1::load    :         0.31         0.30         0.75                1286


While the tcp_rtt is smoothed, it still tells something about the latency of the load bearing flows.


>  
> Actual situations (like what happens when someone starts using BitTorrent while another in the same household is playing a twitch Multi-user FPS) don't actually look like RRUL. Because in fact the big load is ALSO fractal. Bittorrent demand isn't constant over time - far from it. It's bursty.

	[SM] And this is where having an FQ scheduler for ingress and egress really helps, it can isolate most of the fall-out from bursty traffic onto the bursty traffic itself. However, occasionally a user actually evaluates the bursty traffic as more important than the rest (my example from above with bursty real-time traffic of a game) in which case FQ tends to result in unhappiness if the capacity share of the affected flow is such that the bursts are partly dropped (and even if they are just spread out in time too much).

> Everything is bursty at different timescales in the Internet. There are no CBR flows.

	[SM] Probably true, but I think on the scale of a few seconds/minutes things can be "constant" enough, no? 

> So if we want to address the real congestion problems, we need realistic thinking about what the real problem is.
>  
> Unfortunately this is not achieved by the kind of thinking that created diffserv, sadly. Because everything is bursty, just with different timescales in some cases. Even "flash override" priority traffic is incredibly bursty.

	[SM] I thought the rationale for "flash override" is not that its traffic pattern is any different (smoother) from other traffic classes, but simply that delivery of such marked packets has highest priority and the network should do what it can to expedite such packets if at the cost of other packets, so be it... (some link technologies even allow to pre-empt packets already in transfer to expedite higher priority packets). Personally, I like strict precedence, it is both unforgiving and easy to predict, and pretty much useless for a shared  medium like the internet, at least as an end 2 end policy.


> Coming back to Starlink - Starlink apparently is being designed by folks who really do not understand these fundamental ideas. Instead, they probably all worked in researchy environments where the practical realities of being part of a worldwide public Internet were ignored.

	[SM] Also in a world where use-facing tests and evaluations will emphasize maximal throughput rates a lot, as these are easy to measure and follow the simple "larger is better" principle consumers are trained to understand.


>  (The FCC folks are almost as bad. I have found no-one at FCC engineering who understands fractal burstiness - even w.'t. the old Bell System).




*) It might appear that I have a bone to pick with L4S (which I have), but it really is just a great example of engineering malpractice, especially not designing for the existing internet, but assuming one can simply "require" a more L4S compatible internet though the power of IETF drafts. Case in point, L4S wants to bound the maximum bursts duration for compliant senders, which even if it worked, still leaves the problem, that unsynchronized senders can and will still occasionally add up to extended periods at line rate.



> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink

Re: [Starlink] Finite-Buffer M/G/1 Queues with Time and Space Priorities

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 20798 bytes --]


Sebastian - of course we agree far more than we disagree, and this seems like healthy debate, focused on actual user benefit at scale, which is where I hope the Internet focuses. The "bufferbloat" community has been really kicking it for users, in my opinion.
 
[Off topic: I don't have time for IETF's process, now that it is such a parody of captured bureaucratic process, rather than "rough consensus and working code", now missing for at least 2 decades of corporate control of IAB. None of us "old farts" ever though emulating ITU style bureaucracy would solve any important problem the Internet was trying to solve.]
 
On Sunday, July 31, 2022 7:58am, "Sebastian Moeller" <moeller0@gmx.de> said:



> Hi David,
> 
> interesting food for thought...
> 
> 
> > On Jul 29, 2022, at 21:38, David P. Reed via Starlink
> <starlink@lists.bufferbloat.net> wrote:
> >
> > From: "Bless, Roland (TM)" <roland.bless@kit.edu>
> > models from
> > queueing theory is that they only work for load < 1, whereas
> > we are using the network with load values ~1 (i.e., around one) due to
> > congestion control feedback loops that drive the bottleneck link
> > to saturation (unless you consider application limited traffic sources).
> >
> > Let me remind people here that there is some kind of really weird thinking
> going on here about what should be typical behavior in the Intenet when it is
> working well.
> >
> > No, the goal of the Internet is not to saturate all bottlenecks at maximum
> capacity. That is the opposite of the goal, and it is the opposite of a sane
> operating point.
> >
> > Every user seeks low response time, typically a response time on the order of
> the unloaded delay in the network, for ALL traffic. (whether it's the response to
> a file transfer or a voice frame or a WWW request). *
> >
> > Queueing is always suboptimal, if you can achieve goodput without introducing
> any queueing delay. Because a queue built up at any link delays *all* traffic
> sharing that link, so the overall cost to all users goes up radically when
> multiple streams share a link, because the queueing *delay* gets multiplied by the
> number of flows affected!
> >
> > So the most desirable operating point (which Kleinrock and his students
> recently demonstrated with his "power metric") is to have each queue in every link
> average < 1 packet in length. (big or small packets, doesn't matter,
> actually).
> >
> > Now the bigger issue is that this is unachievable when the flows in the
> network are bursty. Poisson being the least bursty, and easiest to analyze of the
> random processes generating flows. Typical Internet usage is incredibly bursty at
> all time scales, though - the burstiness is fractal when observed for real (at
> least if you look at time scales from 1 ms. to 1 day as your unit of analysis). 
> Fractal random processes of this sort are not Poisson at all.
> 
> [SM] In this context I like the framing from the CoDel ACM paper, with the queue
> acting as shock absorber for burst, as you indicate bursts are unavoidable in a
> network with unsynchronized senders. So it seems prudent to engineer with bursts
> as use-case (how ever undesirable) in mind (compared to simply declaring bursts
> undesirable and require endpoints not to be bursty, as L4S seems to do*).
> 
> 
> > So what is the best one ought to try to do?
> >
> > Well, "keeping utilization at 100%" is never what real network operators
> seek. Never, ever. Instead, congestion control is focused on latency control, not
> optimizing utilization.
> 
> [SM] I thought that these are not orthogonal goals and one needs to pick an
> operating point in the throughput<->latency gradient somehow? This becomes
> relevant for smaller links like internet access links more than for back bone
> links. It is relatively easy to drive my 100/40 link into saturation by normal
> usage, so I have a clear goal of keeping latency acceptable under saturating
> loads.
> 
> 
> > The only folks who seem to focus on utilization is the bean counting
> fraternity, because they seem to think the only cost is the wires, so you want the
> wires to be full.
> 
> [SM] Pithy, yet I am sure the bean counters also account for the cost of
> ports/interfaces ;)
> 
> > That, in my opinion, and even in most accounting systems that consider the
> whole enterprise rather than the wires/fibers/airtime alone, is IGNORANT and
> STUPID.
> >
> > However, academics and vendors of switches care nothing about latency at
> network scale. They focus on wirespeed as the only metric.
> >
> > Well, in the old Bell Telephone days, the metric of the Bell System that
> really mattered was not utilization on every day. Instead it was avoiding outages
> due to peak load. That often was "Mother's Day" - a few hours out of one day once
> a year. Because an outage on Mother's day (busy signals) meant major frustration!
> 
> [SM] If one designs for a (rare) worst-case scenario, one is in the clear most of
> the time. I wish that was possible with my internet access link though... I get a
> sync of 116.7/37.0 Mbps which I shape own to a gross 105.0/36.0 it turns out it
> is not that hard to saturate that link occasionally with just normal usage by a
> family of five, so I clearly am far away from 90% reserve capacity, and I have
> little change of expanding the capacity by a factr of 10 within my budget...
> 
> 
> > Why am I talking about this?
> >
> > Because I have been trying for decades (and I am not alone) to apply a
> "Clue-by-Four" to the thick skulls of folks who don't think about the Internet at
> scale, or even won't think about an Enterprise Internet at scale (or Starlink at
> scale). And it doesn't sink in.
> >
> > Andrew Odlyzko, a brilliant mathematician at Bell Labs for most of his career
> also tried to point out that the utilization of the "bottleneck links" in any
> enterprise, up to the size of ATT in the old days, was typically tuned to < 10%
> of saturation at almost any time. Why? Because the CEO freaked out at the quality
> of service of this critical infrastructure (which means completing tasks quickly,
> when load is unusual) and fired people.
> >
> > And in fact, the wires are the cheapest resource - the computers and people
> connected by those resources that can't do work while waiting for queueing delay
> are vastly more expensive to leave idle. Networks don't do "work" that matters.
> Queueing isn't "efficient". It's evil.
> >
> > Which is why dropping packets rather then queueing them is *good*, if the
> sender will slow down and can resend them. Intentially dropped packets should be
> nonzero under load, if an outsider is observing for measruing quality.
> >
> > I call this brain-miswiring about optimizing throughput to fill a bottleneck
> link the Hotrodder Fallacy. That's the idea that one should optimize like a drag
> racer optimizes his car - to burn up the tires and the engine to meet an
> irrelevant metric for automobiles. A nice hobby that has never improved any actual
> vehicle. (Even F1 racing is far more realistic, given you want your cars to last
> for the lifetime of the race).
> >
> > A problem with much of the "network research" community is that it never has
> actually looked at what networks are used for and tried to solve those problems.
> Instead, they define irrelevant problems and encourage all students and professors
> to pursue irrelevancy.
> >
> > Now let's look at RRUL. While it nicely looks at latency for small packets
> under load, it actually disregards the performance of the load streams, which are
> only used to "fill the pipe".
> 
> [SM] I respectfully disagree. They are used to simulate those "fill the pipe"
> flows that do happen in edge networks... think multiple machines downloading
> multi-gigabyte update packages (OS, games, software, ...) when ever they feel
> like it. The sparse latency measurement flows simulate low rate/sparse
> interactive traffic...
> But note that depending on the situation a nominally sparse flaw can use up quite
> some capacity, I talked to a games who observed in riot games valorant in a
> mylti-player online game with 10-20 players traffic at 20 Mbps with cyclic burst
> 128 times a second. On a slow link that becomes a noticeable capacity hog.
> 
> > Fortunately, they are TCP, so they rate limit themselves by window
> adjustment. But they are speed unlimited TCP streams that are meaningless.
> 
> [SM] Flent will however present information about those flows if instructed to do
> so (IIRC by the --socket-stats argument):
> 
> 
> avg median 99th % 
> # data pts
> Ping (ms) ICMP 1.1.1.1 (extra) : 13.26 11.70 29.30 ms 
> 1393
> Ping (ms) avg : 32.17 N/A N/A ms 
> 1607
> Ping (ms)::ICMP : 32.76 30.60 48.02 ms 
> 1395
> Ping (ms)::UDP 0 (0) : 32.64 30.52 46.55 ms 
> 1607
> Ping (ms)::UDP 1 (0) : 31.39 29.90 45.98 ms 
> 1607
> Ping (ms)::UDP 2 (0) : 32.85 30.82 47.04 ms 
> 1607
> Ping (ms)::UDP 3 (0) : 31.72 30.25 46.49 ms 
> 1607
> Ping (ms)::UDP 4 (0) : 31.37 29.78 45.61 ms 
> 1607
> Ping (ms)::UDP 5 (0) : 31.36 29.74 45.13 ms 
> 1607
> Ping (ms)::UDP 6 (0) : 32.85 30.71 47.34 ms 
> 1607
> Ping (ms)::UDP 7 (0) : 33.16 31.08 47.93 ms 
> 1607
> TCP download avg : 7.82 N/A N/A
> Mbits/s 1607
> TCP download sum : 62.55 N/A N/A
> Mbits/s 1607
> TCP download::0 (0) : 7.86 7.28 13.81
> Mbits/s 1607
> TCP download::1 (0) : 8.18 7.88 13.98
> Mbits/s 1607
> TCP download::2 (0) : 7.62 7.05 13.81
> Mbits/s 1607
> TCP download::3 (0) : 7.73 7.37 13.23
> Mbits/s 1607
> TCP download::4 (0) : 7.58 7.07 13.51
> Mbits/s 1607
> TCP download::5 (0) : 7.92 7.37 14.03
> Mbits/s 1607
> TCP download::6 (0) : 8.07 7.58 14.33
> Mbits/s 1607
> TCP download::7 (0) : 7.59 6.96 13.94
> Mbits/s 1607
> TCP totals : 93.20 N/A N/A
> Mbits/s 1607
> TCP upload avg : 3.83 N/A N/A
> Mbits/s 1607
> TCP upload sum : 30.65 N/A N/A
> Mbits/s 1607
> TCP upload::0 (0) : 3.82 3.86 9.57
> Mbits/s 1607
> TCP upload::0 (0)::tcp_cwnd : 14.31 14.00 23.00 
> 856
> TCP upload::0 (0)::tcp_delivery_rate : 3.67 3.81 4.95 
> 855
> TCP upload::0 (0)::tcp_pacing_rate : 4.72 4.85 6.93 
> 855
> TCP upload::0 (0)::tcp_rtt : 42.48 41.36 65.32 
> 851
> TCP upload::0 (0)::tcp_rtt_var : 2.83 2.38 9.90 
> 851
> TCP upload::1 (0) : 3.90 3.94 16.49
> Mbits/s 1607
> TCP upload::1 (0)::tcp_cwnd : 14.46 14.00 23.00 
> 857
> TCP upload::1 (0)::tcp_delivery_rate : 3.75 3.83 5.74 
> 856
> TCP upload::1 (0)::tcp_pacing_rate : 4.81 4.89 8.15 
> 856
> TCP upload::1 (0)::tcp_rtt : 42.12 41.07 63.10 
> 852
> TCP upload::1 (0)::tcp_rtt_var : 2.74 2.36 8.36 
> 852
> TCP upload::2 (0) : 3.85 3.96 5.11
> Mbits/s 1607
> TCP upload::2 (0)::tcp_cwnd : 14.15 14.00 22.00 
> 852
> TCP upload::2 (0)::tcp_delivery_rate : 3.69 3.81 4.93 
> 851
> TCP upload::2 (0)::tcp_pacing_rate : 4.73 4.91 6.55 
> 851
> TCP upload::2 (0)::tcp_rtt : 41.73 41.09 56.97 
> 851
> TCP upload::2 (0)::tcp_rtt_var : 2.59 2.29 7.71 
> 851
> TCP upload::3 (0) : 3.81 3.95 5.32
> Mbits/s 1607
> TCP upload::3 (0)::tcp_cwnd : 13.90 14.00 21.00 
> 851
> TCP upload::3 (0)::tcp_delivery_rate : 3.66 3.82 4.89 
> 851
> TCP upload::3 (0)::tcp_pacing_rate : 4.67 4.87 6.36 
> 851
> TCP upload::3 (0)::tcp_rtt : 41.44 41.09 56.46 
> 847
> TCP upload::3 (0)::tcp_rtt_var : 2.74 2.46 8.27 
> 847
> TCP upload::4 (0) : 3.77 3.88 5.35
> Mbits/s 1607
> TCP upload::4 (0)::tcp_cwnd : 13.86 14.00 21.00 
> 852
> TCP upload::4 (0)::tcp_delivery_rate : 3.61 3.75 4.87 
> 852
> TCP upload::4 (0)::tcp_pacing_rate : 4.63 4.83 6.46 
> 852
> TCP upload::4 (0)::tcp_rtt : 41.74 41.18 57.27 
> 850
> TCP upload::4 (0)::tcp_rtt_var : 2.73 2.45 8.38 
> 850
> TCP upload::5 (0) : 3.83 3.93 5.60
> Mbits/s 1607
> TCP upload::5 (0)::tcp_cwnd : 13.98 14.00 22.00 
> 851
> TCP upload::5 (0)::tcp_delivery_rate : 3.68 3.80 5.05 
> 851
> TCP upload::5 (0)::tcp_pacing_rate : 4.69 4.82 6.65 
> 851
> TCP upload::5 (0)::tcp_rtt : 41.50 40.91 56.42 
> 847
> TCP upload::5 (0)::tcp_rtt_var : 2.68 2.34 8.24 
> 847
> TCP upload::6 (0) : 3.86 3.97 5.60
> Mbits/s 1607
> TCP upload::6 (0)::tcp_cwnd : 14.27 14.00 22.00 
> 850
> TCP upload::6 (0)::tcp_delivery_rate : 3.71 3.83 5.07 
> 850
> TCP upload::6 (0)::tcp_pacing_rate : 4.74 4.90 6.77 
> 850
> TCP upload::6 (0)::tcp_rtt : 42.03 41.66 55.81 
> 850
> TCP upload::6 (0)::tcp_rtt_var : 2.71 2.49 7.85 
> 850
> TCP upload::7 (0) : 3.81 3.92 5.18
> Mbits/s 1607
> TCP upload::7 (0)::tcp_cwnd : 14.01 14.00 22.00 
> 850
> TCP upload::7 (0)::tcp_delivery_rate : 3.67 3.82 4.94 
> 849
> TCP upload::7 (0)::tcp_pacing_rate : 4.57 4.69 6.52 
> 850
> TCP upload::7 (0)::tcp_rtt : 42.62 42.16 56.20 
> 847
> TCP upload::7 (0)::tcp_rtt_var : 2.50 2.19 8.02 
> 847
> cpu_stats_root@192.168.42.1::load : 0.31 0.30 0.75 
> 1286
> 
> 
> While the tcp_rtt is smoothed, it still tells something about the latency of the
> load bearing flows.
>
 
I agree. I have used flent enough to have poked around at its options, and yes, the data is there.
But RRUL's assumption that there is always "more" to send on the load generating TCP flows explores only one kind of case. Suppose you have three streaming video watchers at HD rates, and they don't quite fill up the downlink, yet they actually are "buffering" sufficiently most of the time. How well do they share the downlink so that none of them pause unnecessarily? Maybe FQ_codel works, maybe it doesn't work so well. If you want a variant, imagine a small business office with 20 staff doing Zoom conferences with customers. Zoom actually on the uplink side is bursty to some extent. It can tolerate sub-100 msec. outages. 
My point is that there are real kinds of burstiness besides "click driven", and they have different time-scales of variability. Studying these seems vastly more useful than one more paper based on RRUL as the only point of study. (and even more vastly better than just measuring the pipe's max throughput and utilization).
 
> 
> >
> > Actual situations (like what happens when someone starts using BitTorrent
> while another in the same household is playing a twitch Multi-user FPS) don't
> actually look like RRUL. Because in fact the big load is ALSO fractal. Bittorrent
> demand isn't constant over time - far from it. It's bursty.
> 
> [SM] And this is where having an FQ scheduler for ingress and egress really
> helps,
I think FQ_codel is great! An insight that I get from it is that "flow fairness" plus dropping works pretty well for today's Internet traffic to provide very good responsiveness that the user sees.
However, I think QUIC, which lacks "flows" that are visible at the queue manager, will become problematic. Not necessarily at the "Home Router" end - but at the cloud endpoints that serve many, many users.
 
 
> it can isolate most of the fall-out from bursty traffic onto the bursty
> traffic itself. However, occasionally a user actually evaluates the bursty
> traffic as more important than the rest (my example from above with bursty
> real-time traffic of a game) in which case FQ tends to result in unhappiness if
> the capacity share of the affected flow is such that the bursts are partly
> dropped (and even if they are just spread out in time too much).
> 
> > Everything is bursty at different timescales in the Internet. There are no
> CBR flows.
> 
> [SM] Probably true, but I think on the scale of a few seconds/minutes things can
> be "constant" enough, no?
Even your case of your family's single edge connection, your peak-average over intervals of one hour or more, and probably minute-to-minute as well, show burstiness. There may be a few times each day where the "Mother's Day" event happens, but I bet your average usage in every hour, and probably every minute is < 10%.  What happens when (if your family works like many) you sit down to dinner? And then get back to work?
 
I bet you buy the fastest "up-to" speed you can afford, but not because your average is very high at all.
Right?

> 
> > So if we want to address the real congestion problems, we need realistic
> thinking about what the real problem is.
> >
> > Unfortunately this is not achieved by the kind of thinking that created
> diffserv, sadly. Because everything is bursty, just with different timescales in
> some cases. Even "flash override" priority traffic is incredibly bursty.
> 
> [SM] I thought the rationale for "flash override" is not that its traffic pattern
> is any different (smoother) from other traffic classes, but simply that delivery
> of such marked packets has highest priority and the network should do what it can
> to expedite such packets if at the cost of other packets, so be it... (some link
> technologies even allow to pre-empt packets already in transfer to expedite
> higher priority packets). Personally, I like strict precedence, it is both
> unforgiving and easy to predict, and pretty much useless for a shared medium
> like the internet, at least as an end 2 end policy.
>
Actually, if one uses EDF scheduling, it is provably optimal in a particular sense. That is, if it is possible to meet all the deadlines with a particular schedule, then EDF schedule will achieve all the deadlines.
That is from the early 1970's work on "real-time scheduling" disciplines.
"Strict precedence" is the case of EDF when the deadline for ALL packets is the send time + a network-wide "latency bound". If you used EDF with a latency bound of, say, 100 msec for all packets, and each packet was sequenced in deadline-order, the network would be VERY predictable and responsive.
 
The imagined need for "flash override" priority would go away, unless the emergency somehow required sub-100msec. latency, if all flows backed off using TCP AIMD backoff, and queues in bottleneck links were dropped.
 
No one's actually tried this at scale, but theory all suggests it would be brilliantly stable and predictable.
(How it would work if the network is constantly being DDoSed everywhere isn't  a fair question - no scheduling algorithm can work in constant DDoS, you need meta-network policing to find culprits and shut them off ASAP).
 
> 
> > Coming back to Starlink - Starlink apparently is being designed by folks who
> really do not understand these fundamental ideas. Instead, they probably all
> worked in researchy environments where the practical realities of being part of a
> worldwide public Internet were ignored.
> 
> [SM] Also in a world where use-facing tests and evaluations will emphasize
> maximal throughput rates a lot, as these are easy to measure and follow the
> simple "larger is better" principle consumers are trained to understand.
> 
> 
> > (The FCC folks are almost as bad. I have found no-one at FCC engineering who
> understands fractal burstiness - even w.'t. the old Bell System).
> 
> 
> 
> 
> *) It might appear that I have a bone to pick with L4S (which I have), but it
> really is just a great example of engineering malpractice, especially not
> designing for the existing internet, but assuming one can simply "require" a more
> L4S compatible internet though the power of IETF drafts. Case in point, L4S wants
> to bound the maximum bursts duration for compliant senders, which even if it
> worked, still leaves the problem, that unsynchronized senders can and will still
> occasionally add up to extended periods at line rate.
 
I totally agree with the L4S bias you have. It seems wrong-headed to require every participant in the Internet to behave when you don't even tell them why they have to behave or what behaving means. My concern about IETF bureaucracy emulation applies here, as well.
 
[If BT wants to try L4S across all of BT's customers and take the flack when it fails miserably, it becomes "working code" when it finally works. Then they can get a rough consensus, rather than they and others "dictating" L4S must be.
 
They have not simulated actual usage (I admit that simulating "actual usage" is hard to do, when you don't even know what your users are actually doing today, as I've mentioned above.)  That suggests a "pilot" experimental process. Even ECN, RED, diffserv and MBONE were pilots. Which is where it was learned that they don't work at scale. Which is why no one seriously deploys them, to this day, because if they actually worked, there would be a race among users to demand them]
> 
> 
> 
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> 
> 

[-- Attachment #2: Type: text/html, Size: 27489 bytes --]