[Bloat] uplink bufferbloat and scheduling problems

[Bloat] uplink bufferbloat and scheduling problems

[Dave Taht]

Money quote: "Figure 2a is a good argument to focus latency
research work on downlink bufferbloat."

It peaked at 1.6s in their test:
https://hal.archives-ouvertes.fr/hal-03420681/document

-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]


What's the difference between uplink and downlink?  In DOCSIS the rate asymmetry was the issue. But in WiFi, the air interface is completely symmetric (802.11ax, though, maybe not because of centrally polling).
 
In any CSMA link (WiFi), there is no "up" or "down". There is only sender and receiver, and each station and the AP are always doing both.
 
The problem with shared media links is that the "waiting queue" is distributed, so to manage queue depth, ALL of the potential senders must respond aggressively to excess packets.
 
This is why a lot (maybe all) of the silicon vendors are making really bad choices w.r.t. bufferbloat by adding buffering in the transmitter chip itself, and not discarding or marking when queues build up. It's the same thing that constantly leads hardware guys to think that more memory for buffers improves throughput, and only advertising throughput.
 
To say it again: More memory *doesn't* improve throughput when the queue depths exceed one packet on average, and it degrades "goodput" at higher levels by causing the ultimate sender to "give up" due to long latency. (at the extreme, users will just click again on a slow URL, causing all the throughput to be "badput", because they force the system to transmit it again, while leaving packets clogging the queues.
 
So, if you want good performance on a shared radio medium, you need to squish each flow's queue depth down from sender to receiver to "average < 1 in queue", and also drop packets when there are too many simultaneous flows competing for airtime. And if your source process can't schedule itself frequently enough, don't expect the network to replace buffering at the TCP source and destination - it is not intended to be a storage system.
 
 
 
On Tuesday, November 30, 2021 7:13pm, "Dave Taht" <dave.taht@gmail.com> said:



> Money quote: "Figure 2a is a good argument to focus latency
> research work on downlink bufferbloat."
> 
> It peaked at 1.6s in their test:
> https://hal.archives-ouvertes.fr/hal-03420681/document
> 
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> 
> Dave Täht CEO, TekLibre, LLC
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[David Lang]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2670 bytes --]

with 802.11ac, the difference between uplink and downlink is that the AP can 
transmit to multiple users at the same time (multiple signals spacially 
multiplexed), but the users transmit back one at a time.

David Lang

On Wed, 1 Dec 2021, David P. Reed wrote:

> What's the difference between uplink and downlink?  In DOCSIS the rate asymmetry was the issue. But in WiFi, the air interface is completely symmetric (802.11ax, though, maybe not because of centrally polling).
> 
> In any CSMA link (WiFi), there is no "up" or "down". There is only sender and receiver, and each station and the AP are always doing both.
> 
> The problem with shared media links is that the "waiting queue" is distributed, so to manage queue depth, ALL of the potential senders must respond aggressively to excess packets.
> 
> This is why a lot (maybe all) of the silicon vendors are making really bad choices w.r.t. bufferbloat by adding buffering in the transmitter chip itself, and not discarding or marking when queues build up. It's the same thing that constantly leads hardware guys to think that more memory for buffers improves throughput, and only advertising throughput.
> 
> To say it again: More memory *doesn't* improve throughput when the queue depths exceed one packet on average, and it degrades "goodput" at higher levels by causing the ultimate sender to "give up" due to long latency. (at the extreme, users will just click again on a slow URL, causing all the throughput to be "badput", because they force the system to transmit it again, while leaving packets clogging the queues.
> 
> So, if you want good performance on a shared radio medium, you need to squish each flow's queue depth down from sender to receiver to "average < 1 in queue", and also drop packets when there are too many simultaneous flows competing for airtime. And if your source process can't schedule itself frequently enough, don't expect the network to replace buffering at the TCP source and destination - it is not intended to be a storage system.
> 
> 
> 
> On Tuesday, November 30, 2021 7:13pm, "Dave Taht" <dave.taht@gmail.com> said:
>
>
>
>> Money quote: "Figure 2a is a good argument to focus latency
>> research work on downlink bufferbloat."
>> 
>> It peaked at 1.6s in their test:
>> https://hal.archives-ouvertes.fr/hal-03420681/document
>> 
>> --
>> I tried to build a better future, a few times:
>> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>> 
>> Dave Täht CEO, TekLibre, LLC
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[David Lang]

On Wed, 1 Dec 2021, David P. Reed wrote:

> To say it again: More memory *doesn't* improve throughput when the queue 
> depths exceed one packet on average

slight disagreement here. the buffer improves throughput up to the point where 
it handles one burst of packets. When packets are transmitted individually, 
that's about one packet (insert hand waving about scheduling delays, etc). but 
with wifi where you can transmit multiple packets in one airtime slot, you need 
enough buffer to handle the entire burst.

David Lang

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[Valdis Klētnieks]

On Wed, 01 Dec 2021 13:09:46 -0800, David Lang said:

> with wifi where you can transmit multiple packets in one airtime slot, you need 
> enough buffer to handle the entire burst.

OK, I'll bite... roughly how many min-sized or max-sized packets can you fit
into one slot?

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[Toke Høiland-Jørgensen]

"Valdis Klētnieks" <valdis.kletnieks@vt.edu> writes:

> On Wed, 01 Dec 2021 13:09:46 -0800, David Lang said:
>
>> with wifi where you can transmit multiple packets in one airtime slot, you need 
>> enough buffer to handle the entire burst.
>
> OK, I'll bite... roughly how many min-sized or max-sized packets can you fit
> into one slot?

On 802.11n, 64kB; on 802.11ac, 4MB(!); on 802.11ax, no idea - the same as 802.11ac?

-Toke

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[Jan Ceuleers]

On 01/12/2021 21:26, David P. Reed wrote:
> In any CSMA link (WiFi), there is no "up" or "down". There is only
> sender and receiver, and each station and the AP are always doing both.

Generally though the silicon used in WiFi APs is much more capable than
that used in STAs. The capabilities of (particularly STA-side) silicon
are different as they relate to uplink and downlink.

Of course for a given AP-STA link a common subset of capabilities needs
to be used, but that subset might be different in either direction.

The antennas (number and gain), receiver sensitivity and transmit power
are also different.

A simple example of a reason why uplink and downlink might be different:
the achievable bitrate on the AP-to-STA downlink might be higher than
the uplink because the AP's transmit power is higher. In order to be
able to achieve the same bitrate if the STA's transmit power is lower
the AP's receiver sensitivity would have to be that much better.

Generalising to other shared-media technologies: whenever the UL/DL link
budgets are different the link is going to by asymmetric.

Jan

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[Sebastian Moeller]

Hi David,


you probably had noticed that the cited paper was about LTE/(5G) where the base station operates the scheduler that arbitrates both up- and downstream transmissions. And according to the paper that ends up in bursting on the upstream (I wonder how L4S with its increases burst sensitivity is going to deal with that*)...
I found that paper really nice, short and concise with a simple overview of the used grant request system....



*) My bet is that they are going to claim LTE/5G will have to change their ways....


> On Dec 1, 2021, at 21:26, David P. Reed <dpreed@deepplum.com> wrote:
> 
> What's the difference between uplink and downlink?  In DOCSIS the rate asymmetry was the issue. But in WiFi, the air interface is completely symmetric (802.11ax, though, maybe not because of centrally polling).
>  
> In any CSMA link (WiFi), there is no "up" or "down". There is only sender and receiver, and each station and the AP are always doing both.
>  
> The problem with shared media links is that the "waiting queue" is distributed, so to manage queue depth, ALL of the potential senders must respond aggressively to excess packets.
>  
> This is why a lot (maybe all) of the silicon vendors are making really bad choices w.r.t. bufferbloat by adding buffering in the transmitter chip itself, and not discarding or marking when queues build up. It's the same thing that constantly leads hardware guys to think that more memory for buffers improves throughput, and only advertising throughput.
>  
> To say it again: More memory *doesn't* improve throughput when the queue depths exceed one packet on average, and it degrades "goodput" at higher levels by causing the ultimate sender to "give up" due to long latency. (at the extreme, users will just click again on a slow URL, causing all the throughput to be "badput", because they force the system to transmit it again, while leaving packets clogging the queues.
>  
> So, if you want good performance on a shared radio medium, you need to squish each flow's queue depth down from sender to receiver to "average < 1 in queue", and also drop packets when there are too many simultaneous flows competing for airtime. And if your source process can't schedule itself frequently enough, don't expect the network to replace buffering at the TCP source and destination - it is not intended to be a storage system.

	With a variable rate link like LTE or WiFi some buffering above 1 in queue seems unavoidable, e.g. even if steady state traffic at X Mbps converged to 1-in-queue if the rate the drops to say x/10 Mbps all th packets in flight will hit the buffers at the upstream end of the bottleneck link, no? If rate changes happen rarely, I guess the "average" will still be meaningful, but what if rate changes happen often?

Regards
	Sebastian



>  
>  
>  
> On Tuesday, November 30, 2021 7:13pm, "Dave Taht" <dave.taht@gmail.com> said:
> 
> > Money quote: "Figure 2a is a good argument to focus latency
> > research work on downlink bufferbloat."
> > 
> > It peaked at 1.6s in their test:
> > https://hal.archives-ouvertes.fr/hal-03420681/document
> > 
> > --
> > I tried to build a better future, a few times:
> > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> > 
> > Dave Täht CEO, TekLibre, LLC
> > _______________________________________________
> > Cerowrt-devel mailing list
> > Cerowrt-devel@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
> >
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Thu, 2 Dec 2021, Toke Høiland-Jørgensen wrote:

> "Valdis Klētnieks" <valdis.kletnieks@vt.edu> writes:
>
>> On Wed, 01 Dec 2021 13:09:46 -0800, David Lang said:
>>
>>> with wifi where you can transmit multiple packets in one airtime slot, you need
>>> enough buffer to handle the entire burst.
>>
>> OK, I'll bite... roughly how many min-sized or max-sized packets can you fit
>> into one slot?
>
> On 802.11n, 64kB; on 802.11ac, 4MB(!); on 802.11ax, no idea - the same as 802.11ac?

As I understnad it, 802.11ax can do 16MB (4MB to each of 4 different endpoints)

This is made significantly messier because the headers for each transmission are 
sent at FAR slower rates than the data can be, so if you send a single 64 byte 
packet in a timeslot that could send 4/16MB, it's not a matter of taking 
1/128,000 of the time (the ratio of the data), it's more like 1/2 of the 
time.

So it's really valuable for overall throughput to fill those transmit slots 
rather than having the data trickle out over many slots.

David Lang

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[Dave Collier-Brown]

On 12/1/21 16:09, David Lang wrote:

> On Wed, 1 Dec 2021, David P. Reed wrote:
>
>> To say it again: More memory *doesn't* improve throughput when the
>> queue depths exceed one packet on average
>
> slight disagreement here. the buffer improves throughput up to the
> point where it handles one burst of packets. When packets are
> transmitted individually, that's about one packet (insert hand waving
> about scheduling delays, etc). but with wifi where you can transmit
> multiple packets in one airtime slot, you need enough buffer to handle
> the entire burst.


A different hand-wave: what about "packet trains"? They make using
queuing networks mis-estimate, do they come close together enough that
routers need to be sensitive to them, and affect the number of packets
they need to buffer?

--dave

--
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com |              -- Mark Twain


CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including any and all attachments, contains confidential information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or disclosure is strictly prohibited and is not a waiver of confidentiality. If you have received this telecommunication in error, please notify the sender immediately by return electronic mail and delete the message from your inbox and deleted items folders. This telecommunication does not constitute an express or implied agreement to conduct transactions by electronic means, nor does it constitute a contract offer, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication are subject to legal review and the completion of formal documentation and are not binding until same is confirmed in writing and has been signed by an authorized signatory.

Re: [Bloat] [Cerowrt-devel] uplink bufferbloat and scheduling problems

[David Lang]

On Thu, 2 Dec 2021, Dave Collier-Brown wrote:

> A different hand-wave: what about "packet trains"? They make using
> queuing networks mis-estimate, do they come close together enough that
> routers need to be sensitive to them, and affect the number of packets
> they need to buffer?

I think that's going to depend on how big the burst is, and how sensitive it is 
to packet loss. If it's not big, and is sensitive to packet loss, then it's not 
a big deal to buffer it. But if it's 'too big', it will interfere with others.

since it's impossible to know how big they are going to be, especially compared 
to others, it seems to work pretty well to just not try to account for them.

I believe that current best options do give new flows a bit more leeway before 
throttling them, so if it is a small burst at the start of a flow, it will go 
through quickly, and only start getting throttled if the volume remains high.

Remember, perfect is the enemy of better. You can always craft a case where 
perfect knowledge of future traffic would let you optimize in a different way, 
but since such knowledge doesn't exist in the real world, attempts to account 
for everything are doomed to failure.

David Lang