[Bloat] Trying to *really* understand Linux pacing

[Bloat] Trying to *really* understand Linux pacing

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

Dear de-bloaters of buffers,
Esteemed experts of low delay and pacing!

I have no longer been satisfied with high-level descriptions of how pacing works in Linux, and how it interacts with TSQ (I’ve seen some, in various papers, over the years) - but I wanted to REALLY understand it. So, I have dug through the code.

I documented this experience here:
https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing <https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing>
but it has some holes and may have mistakes.

Actually, my main problem is that I don’t really know what goes on when I configure a larger IW… things seem to get quite “off” there. Why? Anyone up for solving that riddle?  ;-)
(see the tests I documented towards the end of the document)

Generally, if someone who has their hands on files such as tcp_output.c all the time could take a look, and perhaps “fill” my holes, or improve anything that might be wrong, that would be fantastic!   I think that anyone should be allowed to comment and make suggestions in this doc.

MANY thanks to whoever finds the time to take a look !

Cheers,
Michael


[-- Attachment #2: Type: text/html, Size: 1876 bytes --]

Re: [Bloat] Trying to *really* understand Linux pacing

[Dave Taht]

Dear Michael:

Thank you for digging deeply into packet pacing, TSQ, etc. I think
there are some interesting new possibilities in probing (especially
during slow start) that could make the core idea even more effective
than it is. I also tend to think that attempting it in various cloudy
environments and virtualization schemes, and with certain drivers, the
side effects are not as well understood as I would like. For example,
AWS's nitro lacks BQL as does virtio-net.

I think the netdev community, now cc'd, would be interested in your
document and explorations so far, below. I hope for more
enlightenment.

On Wed, Feb 7, 2024 at 6:57 AM Michael Welzl via Bloat
<bloat@lists.bufferbloat.net> wrote:
>
> Dear de-bloaters of buffers,
> Esteemed experts of low delay and pacing!
>
> I have no longer been satisfied with high-level descriptions of how pacing works in Linux, and how it interacts with TSQ (I’ve seen some, in various papers, over the years) - but I wanted to REALLY understand it. So, I have dug through the code.
>
> I documented this experience here:
> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
> but it has some holes and may have mistakes.
>
> Actually, my main problem is that I don’t really know what goes on when I configure a larger IW… things seem to get quite “off” there. Why? Anyone up for solving that riddle?  ;-)
> (see the tests I documented towards the end of the document)
>
> Generally, if someone who has their hands on files such as tcp_output.c all the time could take a look, and perhaps “fill” my holes, or improve anything that might be wrong, that would be fantastic!   I think that anyone should be allowed to comment and make suggestions in this doc.
>
> MANY thanks to whoever finds the time to take a look !
>
> Cheers,
> Michael
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: [Bloat] Trying to *really* understand Linux pacing

[Michael Welzl]

Whoa… and now I’m even more afraid   :-)

My sincere apologies to anyone whose code I may have mis-represented!  I just tried to get it…  sorry if there are silly mistakes in there!


> On 7 Feb 2024, at 13:05, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Dear Michael:
> 
> Thank you for digging deeply into packet pacing, TSQ, etc. I think
> there are some interesting new possibilities in probing (especially
> during slow start) that could make the core idea even more effective
> than it is. I also tend to think that attempting it in various cloudy
> environments and virtualization schemes, and with certain drivers, the
> side effects are not as well understood as I would like. For example,
> AWS's nitro lacks BQL as does virtio-net.
> 
> I think the netdev community, now cc'd, would be interested in your
> document and explorations so far, below. I hope for more
> enlightenment.
> 
> On Wed, Feb 7, 2024 at 6:57 AM Michael Welzl via Bloat
> <bloat@lists.bufferbloat.net> wrote:
>> 
>> Dear de-bloaters of buffers,
>> Esteemed experts of low delay and pacing!
>> 
>> I have no longer been satisfied with high-level descriptions of how pacing works in Linux, and how it interacts with TSQ (I’ve seen some, in various papers, over the years) - but I wanted to REALLY understand it. So, I have dug through the code.
>> 
>> I documented this experience here:
>> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
>> but it has some holes and may have mistakes.
>> 
>> Actually, my main problem is that I don’t really know what goes on when I configure a larger IW… things seem to get quite “off” there. Why? Anyone up for solving that riddle?  ;-)
>> (see the tests I documented towards the end of the document)
>> 
>> Generally, if someone who has their hands on files such as tcp_output.c all the time could take a look, and perhaps “fill” my holes, or improve anything that might be wrong, that would be fantastic!   I think that anyone should be allowed to comment and make suggestions in this doc.
>> 
>> MANY thanks to whoever finds the time to take a look !
>> 
>> Cheers,
>> Michael
>> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> 
> 
> -- 
> 40 years of net history, a couple songs:
> https://www.youtube.com/watch?v=D9RGX6QFm5E
> Dave Täht CSO, LibreQos

Re: [Bloat] Trying to *really* understand Linux pacing

[Dave Taht]

On Wed, Feb 7, 2024 at 7:08 AM Michael Welzl <michawe@ifi.uio.no> wrote:
>
> Whoa… and now I’m even more afraid   :-)
>
> My sincere apologies to anyone whose code I may have mis-represented!  I just tried to get it…  sorry if there are silly mistakes in there!

I think his invention of packet pacing alone, of Eric Dumazet's many
inventions, deserves him a place in the Internet Hall of Fame. It make
short buffered switches saner, allows interactive traffic to fit in
between other traffic (essentially host based FQ), and as the Linux
default, has helped power the expansion of ever more bandwidth to ever
more people without overloading the edges far more, than his fq_codel
did. I also love, that by making it just work, he thoroughly disproved
a formerly influential paper on it, circa 2004 or so.

And I do not understand the implementation either, and have been
trying to come up with ways of improving slow start by varying it. I
hope more enlightenment spreads, and your attempt to document it
improves.

https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit#heading=h.7624xn94jzf6

>
>
> > On 7 Feb 2024, at 13:05, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > Dear Michael:
> >
> > Thank you for digging deeply into packet pacing, TSQ, etc. I think
> > there are some interesting new possibilities in probing (especially
> > during slow start) that could make the core idea even more effective
> > than it is. I also tend to think that attempting it in various cloudy
> > environments and virtualization schemes, and with certain drivers, the
> > side effects are not as well understood as I would like. For example,
> > AWS's nitro lacks BQL as does virtio-net.
> >
> > I think the netdev community, now cc'd, would be interested in your
> > document and explorations so far, below. I hope for more
> > enlightenment.
> >
> > On Wed, Feb 7, 2024 at 6:57 AM Michael Welzl via Bloat
> > <bloat@lists.bufferbloat.net> wrote:
> >>
> >> Dear de-bloaters of buffers,
> >> Esteemed experts of low delay and pacing!
> >>
> >> I have no longer been satisfied with high-level descriptions of how pacing works in Linux, and how it interacts with TSQ (I’ve seen some, in various papers, over the years) - but I wanted to REALLY understand it. So, I have dug through the code.
> >>
> >> I documented this experience here:
> >> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
> >> but it has some holes and may have mistakes.
> >>
> >> Actually, my main problem is that I don’t really know what goes on when I configure a larger IW… things seem to get quite “off” there. Why? Anyone up for solving that riddle?  ;-)
> >> (see the tests I documented towards the end of the document)
> >>
> >> Generally, if someone who has their hands on files such as tcp_output.c all the time could take a look, and perhaps “fill” my holes, or improve anything that might be wrong, that would be fantastic!   I think that anyone should be allowed to comment and make suggestions in this doc.
> >>
> >> MANY thanks to whoever finds the time to take a look !
> >>
> >> Cheers,
> >> Michael
> >>
> >> _______________________________________________
> >> Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> > 40 years of net history, a couple songs:
> > https://www.youtube.com/watch?v=D9RGX6QFm5E
> > Dave Täht CSO, LibreQos
>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: [Bloat] Trying to *really* understand Linux pacing

[Stephen Hemminger]

On Wed, 7 Feb 2024 07:05:27 -0500
Dave Taht via Bloat <bloat@lists.bufferbloat.net> wrote:

> I also tend to think that attempting it in various cloudy
> environments and virtualization schemes, and with certain drivers, the
> side effects are not as well understood as I would like. For example,
> AWS's nitro lacks BQL as does virtio-net.

FYI - Microsoft Azure drivers don't do BQL either.
Both netvsc and mana.

The problem is that implementing BQL is not as easy as it looks
and has possibility of introducing bugs. One big customer with a stuck
connection totally negates any benefit!

For virtio, there have been several attempts at BQL and they never worked
perfectly.

Re: [Bloat] Trying to *really* understand Linux pacing

[Neal Cardwell]

[-- Attachment #1: Type: text/plain, Size: 138 bytes --]

Thanks, Michael, for the nice doc! This is really nice for the Linux
networking community to have. I posted a few comments.

thanks,
neal

[-- Attachment #2: Type: text/html, Size: 215 bytes --]

Re: [Bloat] Trying to *really* understand Linux pacing

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

Dear all,

I’m now finally done updating the document, based on inputs from various folks in this round - most notably Neal, who went to great lengths to help me understand what I saw in my tests (thank you!).  Now the description should be mostly correct (I hope) and pretty complete, and it also includes TSO / GSO.

Comments are still welcome if anyone sees a mistake or something:
https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing <https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing>

Several people have in the meanwhile told me that this is useful for the community. I’m glad to hear that!  I really only started this for myself, just to understand what’s going on. Now I believe I do… but hey, if this helps others too, this is great!   So, please feel free to forward this.

Cheers,
Michael


> On 7 Feb 2024, at 21:13, Neal Cardwell <ncardwell@google.com> wrote:
> 
> Thanks, Michael, for the nice doc! This is really nice for the Linux networking community to have. I posted a few comments.
> 
> thanks,
> neal
> 
> 


[-- Attachment #2: Type: text/html, Size: 1946 bytes --]

Re: [Bloat] Trying to *really* understand Linux pacing

[Dave Taht]

also your investigation showed me a probably bug in cake's
gso-splitting. thanks!

However, after capturing the how as you just have, deeper
understanding the effects on the network itself of this stuff would be
great.

On Mon, Feb 19, 2024 at 8:54 AM Michael Welzl via Bloat
<bloat@lists.bufferbloat.net> wrote:
>
> Dear all,
>
> I’m now finally done updating the document, based on inputs from various folks in this round - most notably Neal, who went to great lengths to help me understand what I saw in my tests (thank you!).  Now the description should be mostly correct (I hope) and pretty complete, and it also includes TSO / GSO.
>
> Comments are still welcome if anyone sees a mistake or something:
> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
>
> Several people have in the meanwhile told me that this is useful for the community. I’m glad to hear that!  I really only started this for myself, just to understand what’s going on. Now I believe I do… but hey, if this helps others too, this is great!   So, please feel free to forward this.
>
> Cheers,
> Michael
>
>
> On 7 Feb 2024, at 21:13, Neal Cardwell <ncardwell@google.com> wrote:
>
> Thanks, Michael, for the nice doc! This is really nice for the Linux networking community to have. I posted a few comments.
>
> thanks,
> neal
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos