[Cake] Advantages to tightly tuning latency

[Cake] Advantages to tightly tuning latency

[Justin Kilpatrick]

I have a frequently changing link I'm using automated tools to monitor and tune using Cake. Currently I'm only tuning bandwidth parameter using latency and packet loss data.

My reading of the codel RFC seems to say that trying to tune the 'interval' value using known path and link latency won't provide any advantages over just tuning the bandwidth parameter.

Obviously codel is just one part of the Cake setup and I'm wondering if there are any advantages I'm missing by not providing this extra input using data I already gather. 

Thanks!

-- 
  Justin Kilpatrick
  justin@althea.net

Re: [Cake] Advantages to tightly tuning latency

[Jonathan Morton]

> On 21 Apr, 2020, at 9:22 pm, Justin Kilpatrick <justin@althea.net> wrote:
> 
> I have a frequently changing link I'm using automated tools to monitor and tune using Cake. Currently I'm only tuning bandwidth parameter using latency and packet loss data.
> 
> My reading of the codel RFC seems to say that trying to tune the 'interval' value using known path and link latency won't provide any advantages over just tuning the bandwidth parameter.
> 
> Obviously codel is just one part of the Cake setup and I'm wondering if there are any advantages I'm missing by not providing this extra input using data I already gather. 

The default latency parameters are tuned well for general Internet paths.  The median path length on the public Internet is about 80ms, for which the default interval of 100ms and target of 5ms works well.  Codel is also designed to accommodate a significant deviation from the expected path length without too much difficulty.

I think it's only worth trying to adjust this if your typical path is substantially different from that norm.  If all your traffic goes over a satellite link, for example, the default parameters might be too tight.  If the vast majority of it goes to a local CDN, you could try the "metro" keyword to tighten things up a bit.  Otherwise, you'll be fine.

Also, most protocols are actually not very sensitive to how tight the AQM is set in the first place.  Either they don't really care about latency at all (eg. bulk downloads) or they are latency-sensitive but also sparse (eg. DNS, NTP, VoIP).  So they are more interested in being isolated from the influence of other flows, which Cake does pretty well regardless of the AQM settings.

It's *considerably* more important to ensure that your shaper is configured correctly.  That means setting not only the bandwidth parameter, but the overhead parameters as well.  A bad shaper setting could result in some or all of your traffic not seeing Cake as the effective bottleneck, and thus not receiving its care.  This can be an orders-of-magnitude effect, depending on just how bloated the underlying hardware is.

 - Jonathan Morton

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

It has always been my dream, that at least for outbound, there would
be sufficient backpressure from the driver
to not have to shape at all, or monitor the link. We have that now in
BQL and AQL. free.fr's dsl driver "does the right thing" - no other
dsl driver does. Nor usb network devices. I hope more folk roll up
their sleeves and test the ath10k some, it's looking lovely from here.

https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/

next up either the new mediatek chip or intel..

On Tue, Apr 21, 2020 at 11:40 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 21 Apr, 2020, at 9:22 pm, Justin Kilpatrick <justin@althea.net> wrote:
> >
> > I have a frequently changing link I'm using automated tools to monitor and tune using Cake. Currently I'm only tuning bandwidth parameter using latency and packet loss data.
> >
> > My reading of the codel RFC seems to say that trying to tune the 'interval' value using known path and link latency won't provide any advantages over just tuning the bandwidth parameter.
> >
> > Obviously codel is just one part of the Cake setup and I'm wondering if there are any advantages I'm missing by not providing this extra input using data I already gather.
>
> The default latency parameters are tuned well for general Internet paths.  The median path length on the public Internet is about 80ms, for which the default interval of 100ms and target of 5ms works well.  Codel is also designed to accommodate a significant deviation from the expected path length without too much difficulty.
>
> I think it's only worth trying to adjust this if your typical path is substantially different from that norm.  If all your traffic goes over a satellite link, for example, the default parameters might be too tight.  If the vast majority of it goes to a local CDN, you could try the "metro" keyword to tighten things up a bit.  Otherwise, you'll be fine.
>
> Also, most protocols are actually not very sensitive to how tight the AQM is set in the first place.  Either they don't really care about latency at all (eg. bulk downloads) or they are latency-sensitive but also sparse (eg. DNS, NTP, VoIP).  So they are more interested in being isolated from the influence of other flows, which Cake does pretty well regardless of the AQM settings.
>
> It's *considerably* more important to ensure that your shaper is configured correctly.  That means setting not only the bandwidth parameter, but the overhead parameters as well.  A bad shaper setting could result in some or all of your traffic not seeing Cake as the effective bottleneck, and thus not receiving its care.  This can be an orders-of-magnitude effect, depending on just how bloated the underlying hardware is.
>
>  - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Advantages to tightly tuning latency

[Thibaut]

[-- Attachment #1: Type: text/plain, Size: 3454 bytes --]

Hi,

> Le 21 avr. 2020 à 20:44, Dave Taht <dave.taht@gmail.com> a écrit :
> 
> It has always been my dream, that at least for outbound, there would
> be sufficient backpressure from the driver
> to not have to shape at all, or monitor the link. We have that now in
> BQL and AQL. free.fr's dsl driver "does the right thing" - no other
> dsl driver does.

My curiosity is piqued. Can you elaborate on this? What does free.fr <http://free.fr/> do?

Thanks,
Thibaut

> Nor usb network devices. I hope more folk roll up
> their sleeves and test the ath10k some, it's looking lovely from here.
> 
> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/
> 
> next up either the new mediatek chip or intel..
> 
> On Tue, Apr 21, 2020 at 11:40 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> On 21 Apr, 2020, at 9:22 pm, Justin Kilpatrick <justin@althea.net> wrote:
>>> 
>>> I have a frequently changing link I'm using automated tools to monitor and tune using Cake. Currently I'm only tuning bandwidth parameter using latency and packet loss data.
>>> 
>>> My reading of the codel RFC seems to say that trying to tune the 'interval' value using known path and link latency won't provide any advantages over just tuning the bandwidth parameter.
>>> 
>>> Obviously codel is just one part of the Cake setup and I'm wondering if there are any advantages I'm missing by not providing this extra input using data I already gather.
>> 
>> The default latency parameters are tuned well for general Internet paths.  The median path length on the public Internet is about 80ms, for which the default interval of 100ms and target of 5ms works well.  Codel is also designed to accommodate a significant deviation from the expected path length without too much difficulty.
>> 
>> I think it's only worth trying to adjust this if your typical path is substantially different from that norm.  If all your traffic goes over a satellite link, for example, the default parameters might be too tight.  If the vast majority of it goes to a local CDN, you could try the "metro" keyword to tighten things up a bit.  Otherwise, you'll be fine.
>> 
>> Also, most protocols are actually not very sensitive to how tight the AQM is set in the first place.  Either they don't really care about latency at all (eg. bulk downloads) or they are latency-sensitive but also sparse (eg. DNS, NTP, VoIP).  So they are more interested in being isolated from the influence of other flows, which Cake does pretty well regardless of the AQM settings.
>> 
>> It's *considerably* more important to ensure that your shaper is configured correctly.  That means setting not only the bandwidth parameter, but the overhead parameters as well.  A bad shaper setting could result in some or all of your traffic not seeing Cake as the effective bottleneck, and thus not receiving its care.  This can be an orders-of-magnitude effect, depending on just how bloated the underlying hardware is.
>> 
>> - Jonathan Morton
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> Make Music, Not War
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


[-- Attachment #2: Type: text/html, Size: 4760 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[Jonathan Morton]

> On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
> 
> My curiosity is piqued. Can you elaborate on this? What does free.fr do?

They're a large French ISP.  They made their own CPE devices, and debloated both them and their network quite a while ago.  In that sense, at least, they're a model for others to follow - but few have.

 - Jonathan Morton

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
> >
> > My curiosity is piqued. Can you elaborate on this? What does free.fr do?
>
> They're a large French ISP.  They made their own CPE devices, and debloated both them and their network quite a while ago.  In that sense, at least, they're a model for others to follow - but few have.
>
>  - Jonathan Morton

they are one of the few ISPs that insisted on getting full source code
to their DSL stack, and retained the chops to be able to modify it. I
really admire their revolution v6 product. First introduced in 2010,
it's been continuously updated, did ipv6 at the outset, got fq_codel
when it first came out, and they update the kernel regularly. All
kinds of great features on it, and ecn is enabled by default for those
also (things like samba). over 3 million boxes now I hear....

with <1ms of delay in the dsl driver, they don't need to shape, they
just run at line rate using three tiers of DRR that look a lot like
cake. They shared their config with me, and before I lost heart for
future internet drafts, I'd stuck it here:

https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd

Occasionally they share some data with me. Sometimes I wish I lived in
paris just so I could have good internet! (their fiber offering is
reasonably buffered (not fq_codeled) and the wifi... maybe I can get
them to talk about what they did)

When free.fr shipped fq_codel 2 months after we finalized it, I
figured the rest of the world was only months behind. How hard is it
to add 50 lines of BQL oriented code to a DSL firmware?





-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

I have spent WAY too much time on reddit this week.

https://www.reddit.com/r/HomeNetworking/comments/g4upyw/how_to_determine_if_a_wireless_router_or_access/

Off to rebuild openwrt! Jon, how's SCE looking? ready for a backport yet?

Re: [Cake] Advantages to tightly tuning latency

[Jonathan Morton]

> On 22 Apr, 2020, at 1:50 am, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Jon, how's SCE looking? ready for a backport yet?

We can't do any sort of wide deployment of SCE until it's been approved as an Internet experiment by the IETF.  Interested individuals can already compile the SCE-enabled kernel and jump through the hoops to try things out - carefully.  There's a bit of infrastructure code to go with the new TCP algorithms and qdiscs, so I'm not certain how easy a backport would be; better to just build the (relatively) current code for now.

IETF TSVWG interim meeting next week (the second of two replacing planned in-person sessions at Vancouver) will discuss the big ECT(1) issue, which is hotly disputed between SCE and L4S.  The key question is whether ECT(1) should become a classifier input to the network (orthogonal to Diffserv but with some of the same basic problems), or an additional congestion signal output from the network (indicating a lesser degree of congestion, to which a smaller and more nuanced response is desired).  It's anyone's guess how that will turn out, but the technical merit is on our side and that really should count for something.

If you're keeping an eye on the TSVWG list, expect a major bombshell to drop there in the next few days.

 - Jonathan Morton

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

On Tue, Apr 21, 2020 at 4:07 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 22 Apr, 2020, at 1:50 am, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > Jon, how's SCE looking? ready for a backport yet?
>
> We can't do any sort of wide deployment of SCE until it's been approved as an Internet experiment by the IETF.

If the present algorithm on the qdisc is stable, I'd like to try it on wifi.

> Interested individuals can already compile the SCE-enabled kernel and jump through the hoops to try things out - carefully.  There's a bit of infrastructure code to go with the new TCP algorithms and qdiscs, so I'm not certain how easy a backport would be; better to just build the (relatively) current code for now.

Not on openwrt. It seemed easy to backport just cake there, and the
rest of the code on dedicated servers and clients. Similarly I'd try
for the wifi attempt also.

I care about the cpu impact a lot. Also a recent string of postings on
netdev that I have had too much PTSD to reply to seem to indicate that
accecn *requires* that the tcp offload engine be disabled, which is
difficult to swallow. Can SCE work with tcp offloads enabled (on the
server, client and qdisc?)?

The same post claimed that apple proved we could "just turn ecn on",
and to explore that claim I updated my osx to the latest only to
immediate find apple's heuristics *disabled* attempts at ecn
negotiation on the second of two rrul tests, and I'd also poked into a
worldwide dataset that showed WAY less ecn attempts making it from
apple gear to the test server.

Another post (which I have not responded to either) pointed to an
improvement in the 3WHS that may or may not be genuinely useful, but
at that point, I went back to fixing wifi with what I knew worked.

The code itself, was not bad. Perhaps some review of that set of
patches and thread is needed by some others with stronger stomachs.

https://www.spinics.net/lists/netdev/msg638882.html


>
> IETF TSVWG interim meeting next week (the second of two replacing planned in-person sessions at Vancouver) will discuss the big ECT(1) issue, which is hotly disputed between SCE and L4S.  The key question is whether ECT(1) should become a classifier input to the network (orthogonal to Diffserv but with some of the same basic problems), or an additional congestion signal output from the network (indicating a lesser degree of congestion, to which a smaller and more nuanced response is desired).  It's anyone's guess how that will turn out, but the technical merit is on our side and that really should count for something.
>
> If you're keeping an eye on the TSVWG list, expect a major bombshell to drop there in the next few days.

I do. I wish more did. Beverage in hand. :)

>  - Jonathan Morton
>


-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Advantages to tightly tuning latency

[Thibaut]

[-- Attachment #1: Type: text/plain, Size: 2889 bytes --]



> Le 22 avr. 2020 à 00:44, Dave Taht <dave.taht@gmail.com> a écrit :
> 
> On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
>>> 
>>> My curiosity is piqued. Can you elaborate on this? What does free.fr do?
>> 
>> They're a large French ISP.  They made their own CPE devices, and debloated both them and their network quite a while ago.  In that sense, at least, they're a model for others to follow - but few have.
>> 
>> - Jonathan Morton
> 
> they are one of the few ISPs that insisted on getting full source code
> to their DSL stack, and retained the chops to be able to modify it. I
> really admire their revolution v6 product. First introduced in 2010,
> it's been continuously updated, did ipv6 at the outset, got fq_codel
> when it first came out, and they update the kernel regularly. All
> kinds of great features on it, and ecn is enabled by default for those
> also (things like samba). over 3 million boxes now I hear....
> 
> with <1ms of delay in the dsl driver, they don't need to shape, they
> just run at line rate using three tiers of DRR that look a lot like
> cake. They shared their config with me, and before I lost heart for
> future internet drafts, I'd stuck it here:
> 
> https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd <https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd>

Very interesting, thanks. I wonder if they trickled down these improvements to the older V5: I had to plug my old V5 back after my DSLAM was moved to native IPV6 (I couldn’t find how to talk to it over VDSL2 modem, as I did before the switch), and though there was a massive drop in uplink bandwidth (from 10Mbps VDSL2 to 1Mbps ADSL), I noticed that I no longer needed cake on the router wan interface. Latency remained very well controlled without having to do anything special. In fact, enabling cake with the previous settings was wrecking havoc! (Maybe it interfered with whatever Free is doing in the box).

> Occasionally they share some data with me. Sometimes I wish I lived in
> paris just so I could have good internet! (their fiber offering is
> reasonably buffered (not fq_codeled) and the wifi... maybe I can get
> them to talk about what they did)

You don’t have to live in Paris to enjoy good internet: I’m currently stranded in the countryside and I enjoy a better connection than many a Parisian, thanks to a public/private FTTH network that appears to be very well handled by my current ISP (K-Net) :)

> When free.fr shipped fq_codel 2 months after we finalized it, I
> figured the rest of the world was only months behind. How hard is it
> to add 50 lines of BQL oriented code to a DSL firmware?

Heh.

Cheers,
Thibaut

[-- Attachment #2: Type: text/html, Size: 4220 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]

On Wed, Apr 22, 2020 at 12:44 AM Dave Taht <dave.taht@gmail.com> wrote:

> On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> > > On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
> > >
> > > My curiosity is piqued. Can you elaborate on this? What does free.fr
> do?
> >
> > They're a large French ISP.  They made their own CPE devices, and
> debloated both them and their network quite a while ago.  In that sense, at
> least, they're a model for others to follow - but few have.
> >
> >  - Jonathan Morton
>
> they are one of the few ISPs that insisted on getting full source code
> to their DSL stack, and retained the chops to be able to modify it. I
> really admire their revolution v6 product. First introduced in 2010,
> it's been continuously updated, did ipv6 at the outset, got fq_codel
> when it first came out, and they update the kernel regularly. All
> kinds of great features on it, and ecn is enabled by default for those
> also (things like samba). over 3 million boxes now I hear....
>
> with <1ms of delay in the dsl driver, they don't need to shape, they
> just run at line rate using three tiers of DRR that look a lot like
> cake. They shared their config with me, and before I lost heart for
> future internet drafts, I'd stuck it here:
>
>
> https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd
>
> Occasionally they share some data with me. Sometimes I wish I lived in
> paris just so I could have good internet! (their fiber offering is
> reasonably buffered (not fq_codeled) and the wifi... maybe I can get
> them to talk about what they did)
>
> When free.fr shipped fq_codel 2 months after we finalized it, I
> figured the rest of the world was only months behind. How hard is it
> to add 50 lines of BQL oriented code to a DSL firmware?
>
>
Free has been using SFQ since 2005 (if I remember well).
They announced the wide deployment of SFQ in the free.fr newsgroup.
Wi-Fi in the free.fr router was not as good though.

In Paris there is a lot of GPON now that is replacing DSL. But there is
a nation-wide effort funded by local administrations to get fiber
everywhere. There are small towns in the countryside with fiber.
Public money has made, and is making that possible.
There is still a little of Euro-DOCSIS, but frankly compared to fiber
it has no chance to survive.

I currently have 2Gbps/600Mbps access with orange.fr and free.fr has a
subscription
at 10Gbps GPON. I won't tell you the price because you may feel depressed
compared to other countries where prices are much higher.

The challenge becomes to keep up with these link rates in software
as there is a lot of hardware offloading.

As soon as 802.11ax becomes the norm, software scheduling will become
a challenge.

Luca

[-- Attachment #2: Type: text/html, Size: 5457 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

On Wed, Apr 22, 2020 at 2:04 AM Luca Muscariello <muscariello@ieee.org> wrote:
>
>
>
> On Wed, Apr 22, 2020 at 12:44 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>> >
>> > > On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
>> > >
>> > > My curiosity is piqued. Can you elaborate on this? What does free.fr do?
>> >
>> > They're a large French ISP.  They made their own CPE devices, and debloated both them and their network quite a while ago.  In that sense, at least, they're a model for others to follow - but few have.
>> >
>> >  - Jonathan Morton
>>
>> they are one of the few ISPs that insisted on getting full source code
>> to their DSL stack, and retained the chops to be able to modify it. I
>> really admire their revolution v6 product. First introduced in 2010,
>> it's been continuously updated, did ipv6 at the outset, got fq_codel
>> when it first came out, and they update the kernel regularly. All
>> kinds of great features on it, and ecn is enabled by default for those
>> also (things like samba). over 3 million boxes now I hear....
>>
>> with <1ms of delay in the dsl driver, they don't need to shape, they
>> just run at line rate using three tiers of DRR that look a lot like
>> cake. They shared their config with me, and before I lost heart for
>> future internet drafts, I'd stuck it here:
>>
>> https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd
>>
>> Occasionally they share some data with me. Sometimes I wish I lived in
>> paris just so I could have good internet! (their fiber offering is
>> reasonably buffered (not fq_codeled) and the wifi... maybe I can get
>> them to talk about what they did)
>>
>> When free.fr shipped fq_codel 2 months after we finalized it, I
>> figured the rest of the world was only months behind. How hard is it
>> to add 50 lines of BQL oriented code to a DSL firmware?
>>
>
> Free has been using SFQ since 2005 (if I remember well).
> They announced the wide deployment of SFQ in the free.fr newsgroup.
> Wi-Fi in the free.fr router was not as good though.

They're working on it. :)

> In Paris there is a lot of GPON now that is replacing DSL. But there is
> a nation-wide effort funded by local administrations to get fiber
> everywhere. There are small towns in the countryside with fiber.
> Public money has made, and is making that possible.
> There is still a little of Euro-DOCSIS, but frankly compared to fiber
> it has no chance to survive.

I am very, very happy for y'all. Fiber has always been the sanest
thing. Is there
a SPF+ gpon card yet I can plug into a convention open source router yet?

>
> I currently have 2Gbps/600Mbps access with orange.fr and free.fr has a subscription
> at 10Gbps GPON. I won't tell you the price because you may feel depressed
> compared to other countries where prices are much higher.

I'd emigrate!!!

> The challenge becomes to keep up with these link rates in software
> as there is a lot of hardware offloading.

At this point, I kind of buy the stanford sqrt(bdp) argument. All you
really need for gigE+ fiber access to work well
for most modern traffic is a fairly short fifo (say, 20ms). Any form
of FQ would help but be hardly noticible. I think
there needs to be work on the hop between the internet and the subscriber...

Web traffic is dominated by RTT above 40mbit (presently).
streaming video traffic - is no more than 20Mbit, and your occasional
big download is a dozen big streams that would
bounce off a short fifo well.
gbit access to the home is (admittedly glorious, wonderful!) overkill
for all present forms of traffic.

I'm pretty sure if I had gig fiber I could come up with a way to use
it up (exiting the cloud entirely comes to mind), but
lacking new applications that demand that much bandwidth...

I of course, would like to see lola ( https://lola.conts.it/ ) finally
work, and videoconferencing and game stream with high rates and faster
(even raw) encoding also has potential to reduce e2e latencies
enormously at that layer.

>
> As soon as 802.11ax becomes the norm, software scheduling will become
> a challenge.

Do you mean in fiber or wireless? wireless is really problematic at ANY speed.

at gfiber, the buffering moved to the wifi, and there are other
problems that really impact achievable bandwidth. When I was last in
paris, I could "hear" 300+ access points from my apt, and could only
get 100-200kbit per second out of the wireless n ap I had, unless I
cheated and stuck my traffic in the VI queue. A friend of mine there,
couldn't even get wifi across the room! Beacons ate into a lot of the
available
bandwidth. Since 5ghz (and soon 6ghz - is 6E a thing in france) is
shorter range I'm hoping that's got better, but with
802.11ac and ax peeing on half the wifi spectrum by default, I imagine
achievable rates in high density locations with many APs will be very
low... and very jittery... and thus still require good ATF, fq, and
aqm technologies.

I have high hopes for OFDMA and DU but thus far haven't found an AP
doing it. I'm not sure what to do about the beaconing problem except
offer a free tradein to all my neighbors still emitting G style
frames....

And in looking over some preliminary code for the mt76 ax chip, I
worry about both bad design of the firmware, and
insufficient resources on-chip to manage well.

How is the 5G rollout going in france?

I recently learned that much of japan is... wait for it... wimax.

>
> Luca



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Advantages to tightly tuning latency

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 6687 bytes --]

On Wed, Apr 22, 2020 at 4:48 PM Dave Taht <dave.taht@gmail.com> wrote:

> On Wed, Apr 22, 2020 at 2:04 AM Luca Muscariello <muscariello@ieee.org>
> wrote:
> >
> >
> >
> > On Wed, Apr 22, 2020 at 12:44 AM Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >> >
> >> > > On 22 Apr, 2020, at 1:25 am, Thibaut <hacks@slashdirt.org> wrote:
> >> > >
> >> > > My curiosity is piqued. Can you elaborate on this? What does
> free.fr do?
> >> >
> >> > They're a large French ISP.  They made their own CPE devices, and
> debloated both them and their network quite a while ago.  In that sense, at
> least, they're a model for others to follow - but few have.
> >> >
> >> >  - Jonathan Morton
> >>
> >> they are one of the few ISPs that insisted on getting full source code
> >> to their DSL stack, and retained the chops to be able to modify it. I
> >> really admire their revolution v6 product. First introduced in 2010,
> >> it's been continuously updated, did ipv6 at the outset, got fq_codel
> >> when it first came out, and they update the kernel regularly. All
> >> kinds of great features on it, and ecn is enabled by default for those
> >> also (things like samba). over 3 million boxes now I hear....
> >>
> >> with <1ms of delay in the dsl driver, they don't need to shape, they
> >> just run at line rate using three tiers of DRR that look a lot like
> >> cake. They shared their config with me, and before I lost heart for
> >> future internet drafts, I'd stuck it here:
> >>
> >>
> https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd
> >>
> >> Occasionally they share some data with me. Sometimes I wish I lived in
> >> paris just so I could have good internet! (their fiber offering is
> >> reasonably buffered (not fq_codeled) and the wifi... maybe I can get
> >> them to talk about what they did)
> >>
> >> When free.fr shipped fq_codel 2 months after we finalized it, I
> >> figured the rest of the world was only months behind. How hard is it
> >> to add 50 lines of BQL oriented code to a DSL firmware?
> >>
> >
> > Free has been using SFQ since 2005 (if I remember well).
> > They announced the wide deployment of SFQ in the free.fr newsgroup.
> > Wi-Fi in the free.fr router was not as good though.
>
> They're working on it. :)
>
> > In Paris there is a lot of GPON now that is replacing DSL. But there is
> > a nation-wide effort funded by local administrations to get fiber
> > everywhere. There are small towns in the countryside with fiber.
> > Public money has made, and is making that possible.
> > There is still a little of Euro-DOCSIS, but frankly compared to fiber
> > it has no chance to survive.
>
> I am very, very happy for y'all. Fiber has always been the sanest
> thing. Is there
> a SPF+ gpon card yet I can plug into a convention open source router yet?
>
> >
> > I currently have 2Gbps/600Mbps access with orange.fr and free.fr has a
> subscription
> > at 10Gbps GPON. I won't tell you the price because you may feel depressed
> > compared to other countries where prices are much higher.
>
> I'd emigrate!!!
>
> > The challenge becomes to keep up with these link rates in software
> > as there is a lot of hardware offloading.
>


I just meant that these routers tend to use HW offloading
and kernel qdiscs may be bypassed.



>
> At this point, I kind of buy the stanford sqrt(bdp) argument. All you
> really need for gigE+ fiber access to work well
> for most modern traffic is a fairly short fifo (say, 20ms). Any form
> of FQ would help but be hardly noticible. I think
> there needs to be work on the hop between the internet and the
> subscriber...
>
> Web traffic is dominated by RTT above 40mbit (presently).
> streaming video traffic - is no more than 20Mbit, and your occasional
> big download is a dozen big streams that would
> bounce off a short fifo well.
> gbit access to the home is (admittedly glorious, wonderful!) overkill
> for all present forms of traffic.
>
> I'm pretty sure if I had gig fiber I could come up with a way to use
> it up (exiting the cloud entirely comes to mind), but
> lacking new applications that demand that much bandwidth...
>
> I of course, would like to see lola ( https://lola.conts.it/ ) finally
> work, and videoconferencing and game stream with high rates and faster
> (even raw) encoding also has potential to reduce e2e latencies
> enormously at that layer.
>
> >
> > As soon as 802.11ax becomes the norm, software scheduling will become
> > a challenge.
>
> Do you mean in fiber or wireless? wireless is really problematic at ANY
> speed.
>

I meant that software scheduling becomes a challenge for the same
reason as above. Increase in total throughput of the box
will call for hardware offloading and kernel qdisc may be bypassed.

It is not a challenge per se, it is a challenge because traffic
may not be managed by the kernel.




>
> at gfiber, the buffering moved to the wifi, and there are other
> problems that really impact achievable bandwidth. When I was last in
> paris, I could "hear" 300+ access points from my apt, and could only
> get 100-200kbit per second out of the wireless n ap I had, unless I
> cheated and stuck my traffic in the VI queue. A friend of mine there,
> couldn't even get wifi across the room! Beacons ate into a lot of the
> available
> bandwidth. Since 5ghz (and soon 6ghz - is 6E a thing in france) is
> shorter range I'm hoping that's got better, but with
> 802.11ac and ax peeing on half the wifi spectrum by default, I imagine
> achievable rates in high density locations with many APs will be very
> low... and very jittery... and thus still require good ATF, fq, and
> aqm technologies.
>
> I have high hopes for OFDMA and DU but thus far haven't found an AP
> doing it. I'm not sure what to do about the beaconing problem except
> offer a free tradein to all my neighbors still emitting G style
> frames....
>
> And in looking over some preliminary code for the mt76 ax chip, I
> worry about both bad design of the firmware, and
> insufficient resources on-chip to manage well.
>
> How is the 5G rollout going in france?
>

Good question. I've just seen a speed test at Gbps on a phone
which can drain your battery in less than 5 minutes. Amazing tech!



>
> I recently learned that much of japan is... wait for it... wimax.
>
> >
> > Luca
>
>
>
> --
> Make Music, Not War
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
>

[-- Attachment #2: Type: text/html, Size: 9795 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 8216 bytes --]


Having asymmetric gigabit cable modem service (1 Gb/s down) and very short latencies (5 ms.) to many servers of interest that can source 1 Gb/s), I would just comment that I find it very, very useful for "normal" use.
 
Perhaps my point is this: "normal" isn't a narrow gaussian distribution of performance needs. It's what might be called a time-varying long tailed distribution.
 
I pay for 1 gb/sec because it is "worth it" to download from, say, github cloning or a docker container image in under 1 second.
 
To think that isn't valuable is to miss the point that the Internet's performance isn't about isochronous flows or slow FTPs - it's not about throughput. It's about service delay.
 
And congestion control is about mitigating service delays under load, by eliminating sustained queueing delays that build up due to multiplexed use otherwise.
 
To talk about one use at a time, and treat an average throughput as the goal metric is to miss the entire point.
 
A home access connection is frequently multiplexed over unrelated uses. If you are single, live in your own apartment, ... you have a very, very warped idea of real usage.
On Wednesday, April 22, 2020 11:28am, "Luca Muscariello" <muscariello@ieee.org> said:







On Wed, Apr 22, 2020 at 4:48 PM Dave Taht <[ dave.taht@gmail.com ]( mailto:dave.taht@gmail.com )> wrote:On Wed, Apr 22, 2020 at 2:04 AM Luca Muscariello <[ muscariello@ieee.org ]( mailto:muscariello@ieee.org )> wrote:
 >
 >
 >
 > On Wed, Apr 22, 2020 at 12:44 AM Dave Taht <[ dave.taht@gmail.com ]( mailto:dave.taht@gmail.com )> wrote:
 >>
 >> On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <[ chromatix99@gmail.com ]( mailto:chromatix99@gmail.com )> wrote:
 >> >
 >> > > On 22 Apr, 2020, at 1:25 am, Thibaut <[ hacks@slashdirt.org ]( mailto:hacks@slashdirt.org )> wrote:
 >> > >
 >> > > My curiosity is piqued. Can you elaborate on this? What does [ free.fr ]( http://free.fr ) do?
 >> >
 >> > They're a large French ISP.  They made their own CPE devices, and debloated both them and their network quite a while ago.  In that sense, at least, they're a model for others to follow - but few have.
 >> >
 >> >  - Jonathan Morton
 >>
 >> they are one of the few ISPs that insisted on getting full source code
 >> to their DSL stack, and retained the chops to be able to modify it. I
 >> really admire their revolution v6 product. First introduced in 2010,
 >> it's been continuously updated, did ipv6 at the outset, got fq_codel
 >> when it first came out, and they update the kernel regularly. All
 >> kinds of great features on it, and ecn is enabled by default for those
 >> also (things like samba). over 3 million boxes now I hear....
 >>
 >> with <1ms of delay in the dsl driver, they don't need to shape, they
 >> just run at line rate using three tiers of DRR that look a lot like
 >> cake. They shared their config with me, and before I lost heart for
 >> future internet drafts, I'd stuck it here:
 >>
 >> [ https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd ]( https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.mkd )
 >>
 >> Occasionally they share some data with me. Sometimes I wish I lived in
 >> paris just so I could have good internet! (their fiber offering is
 >> reasonably buffered (not fq_codeled) and the wifi... maybe I can get
 >> them to talk about what they did)
 >>
 >> When [ free.fr ]( http://free.fr ) shipped fq_codel 2 months after we finalized it, I
 >> figured the rest of the world was only months behind. How hard is it
 >> to add 50 lines of BQL oriented code to a DSL firmware?
 >>
 >
 > Free has been using SFQ since 2005 (if I remember well).
 > They announced the wide deployment of SFQ in the [ free.fr ]( http://free.fr ) newsgroup.
 > Wi-Fi in the [ free.fr ]( http://free.fr ) router was not as good though.

 They're working on it. :)

 > In Paris there is a lot of GPON now that is replacing DSL. But there is
 > a nation-wide effort funded by local administrations to get fiber
 > everywhere. There are small towns in the countryside with fiber.
 > Public money has made, and is making that possible.
 > There is still a little of Euro-DOCSIS, but frankly compared to fiber
 > it has no chance to survive.

 I am very, very happy for y'all. Fiber has always been the sanest
 thing. Is there
 a SPF+ gpon card yet I can plug into a convention open source router yet?

 >
 > I currently have 2Gbps/600Mbps access with [ orange.fr ]( http://orange.fr ) and [ free.fr ]( http://free.fr ) has a subscription
 > at 10Gbps GPON. I won't tell you the price because you may feel depressed
 > compared to other countries where prices are much higher.

 I'd emigrate!!!

 > The challenge becomes to keep up with these link rates in software
 > as there is a lot of hardware offloading.

I just meant that these routers tend to use HW offloading 
and kernel qdiscs may be bypassed.
 
 At this point, I kind of buy the stanford sqrt(bdp) argument. All you
 really need for gigE+ fiber access to work well
 for most modern traffic is a fairly short fifo (say, 20ms). Any form
 of FQ would help but be hardly noticible. I think
 there needs to be work on the hop between the internet and the subscriber...

 Web traffic is dominated by RTT above 40mbit (presently).
 streaming video traffic - is no more than 20Mbit, and your occasional
 big download is a dozen big streams that would
 bounce off a short fifo well.
 gbit access to the home is (admittedly glorious, wonderful!) overkill
 for all present forms of traffic.

 I'm pretty sure if I had gig fiber I could come up with a way to use
 it up (exiting the cloud entirely comes to mind), but
 lacking new applications that demand that much bandwidth...

 I of course, would like to see lola ( [ https://lola.conts.it/ ]( https://lola.conts.it/ ) ) finally
 work, and videoconferencing and game stream with high rates and faster
 (even raw) encoding also has potential to reduce e2e latencies
 enormously at that layer.

 >
 > As soon as 802.11ax becomes the norm, software scheduling will become
 > a challenge.

 Do you mean in fiber or wireless? wireless is really problematic at ANY speed.

I meant that software scheduling becomes a challenge for the same
reason as above. Increase in total throughput of the box
will call for hardware offloading and kernel qdisc may be bypassed.
It is not a challenge per se, it is a challenge because traffic
may not be managed by the kernel.
 
 at gfiber, the buffering moved to the wifi, and there are other
 problems that really impact achievable bandwidth. When I was last in
 paris, I could "hear" 300+ access points from my apt, and could only
 get 100-200kbit per second out of the wireless n ap I had, unless I
 cheated and stuck my traffic in the VI queue. A friend of mine there,
 couldn't even get wifi across the room! Beacons ate into a lot of the
 available
 bandwidth. Since 5ghz (and soon 6ghz - is 6E a thing in france) is
 shorter range I'm hoping that's got better, but with
 802.11ac and ax peeing on half the wifi spectrum by default, I imagine
 achievable rates in high density locations with many APs will be very
 low... and very jittery... and thus still require good ATF, fq, and
 aqm technologies.

 I have high hopes for OFDMA and DU but thus far haven't found an AP
 doing it. I'm not sure what to do about the beaconing problem except
 offer a free tradein to all my neighbors still emitting G style
 frames....

 And in looking over some preliminary code for the mt76 ax chip, I
 worry about both bad design of the firmware, and
 insufficient resources on-chip to manage well.

 How is the 5G rollout going in france?

Good question. I've just seen a speed test at Gbps on a phone
which can drain your battery in less than 5 minutes. Amazing tech!
 
 I recently learned that much of japan is... wait for it... wimax.

 >
 > Luca



 -- 
 Make Music, Not War

 Dave Täht
 CTO, TekLibre, LLC
[ http://www.teklibre.com ]( http://www.teklibre.com )
 Tel: 1-831-435-0729

[-- Attachment #2: Type: text/html, Size: 12406 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote:

Hello,

> > Free has been using SFQ since 2005 (if I remember well).
> > They announced the wide deployment of SFQ in the free.fr newsgroup.
> > Wi-Fi in the free.fr router was not as good though.
> 
> They're working on it. :)

yes indeed.

Switching to softmac approach, so now mac80211 will do rate control
and scheduling (using wake_tx_queue model).

for 5ghz, we use ath10k

> I am very, very happy for y'all. Fiber has always been the sanest
> thing. Is there a SPF+ gpon card yet I can plug into a convention
> open source router yet?

FYI Free.fr uses 10G-EPON, not GPON.

Also most deployments are using an additionnal terminal equipement at
called "ONT" or "ONU" that handle the PON part and exposes an ethernet
port where the operator CPE is plugged. So we are back to the early
days of DSL, where the hardest part (scheduling) is done inside a
black box. That makes it easier to replace the operator CPE with your
own standard ethernet router though.

At least SOCs with integrated PON (supporting all flavours
GPON/EPON/..)  are starting to be deployed. Nothing available in
opensource.

Also note that it's not just kernel drivers, you also need some higher
OAM stack to make that work, and there are a lot of existing
standards, DPOE (EPON), OMCI (GPON)... all with interop challenges.

> > The challenge becomes to keep up with these link rates in software
> > as there is a lot of hardware offloading.

Yes that's our pain point, because that's what the SOCs vendors
deliver and you need to use that because there is no alternative.

It's not entierely the SOCs vendors fault though.

15 years ago, your average SOC's CPU would be something like 200Mhz
MIPS, Linux standard forwarding path (softirq => routing+netfilter =>
qdisc) was too slow for this, too much cache footprint/overhead. So
every vendor started building alternatives forwarding path in their
hardware and never looked back.

Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which
with a non crappy network driver and internal fabric should do be able
to route 1Gbit/s with out-of-the-box kernel forwarding.

But that's too late. SOC vendors compete against each others, and the
big telcos need a way to tell which SOC is better to make a buying
decision. So synthetic benchmarks have become the norm, and since
everybody was able to do fill their pipe with 1500 bytes packets,
benchmarks have moved to unrealistic 64 bytes packets (so called
wirespeed)

If you don't have hardware acceleration for forwarding, you don't
exist in those benchmarks and will not sell your chipset. Also they
invested so much in their alternative network stack that it's
difficult to stop (huge R&D teams). That being said, they do have a
point, when speed go above 1Gbit/s, the kernel becomes the bottleneck.

For Free.fr 10Gbit/s offer, we had to develop an alternative
(software) forwarding path using polling mode model (DPDK style),
otherwise our albeit powerful ARM Cortex A72@2Ghz could not forward
more than 2Gbit/s.

And going multicore/RSS does not fly when the test case is single
stream TCP session, which is what most speedtest application do (ookla
only recently added multi-connections test).

-- 
Maxime

Re: [Cake] Advantages to tightly tuning latency

[Toke Høiland-Jørgensen]

Maxime Bizon <mbizon@freebox.fr> writes:

> On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote:
>
> Hello,
>
>> > Free has been using SFQ since 2005 (if I remember well).
>> > They announced the wide deployment of SFQ in the free.fr newsgroup.
>> > Wi-Fi in the free.fr router was not as good though.
>> 
>> They're working on it. :)
>
> yes indeed.
>
> Switching to softmac approach, so now mac80211 will do rate control
> and scheduling (using wake_tx_queue model).
>
> for 5ghz, we use ath10k

That is awesome! Please make sure you include the AQL patch for ath10k,
it really works wonders, as Dave demonstrated:

https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.html

>> I am very, very happy for y'all. Fiber has always been the sanest
>> thing. Is there a SPF+ gpon card yet I can plug into a convention
>> open source router yet?
>
> FYI Free.fr uses 10G-EPON, not GPON.
>
> Also most deployments are using an additionnal terminal equipement at
> called "ONT" or "ONU" that handle the PON part and exposes an ethernet
> port where the operator CPE is plugged. So we are back to the early
> days of DSL, where the hardest part (scheduling) is done inside a
> black box. That makes it easier to replace the operator CPE with your
> own standard ethernet router though.
>
> At least SOCs with integrated PON (supporting all flavours
> GPON/EPON/..)  are starting to be deployed. Nothing available in
> opensource.
>
> Also note that it's not just kernel drivers, you also need some higher
> OAM stack to make that work, and there are a lot of existing
> standards, DPOE (EPON), OMCI (GPON)... all with interop challenges.

It always bugged me that there was no open source support for these
esoteric protocols and standards. It would seem like an obvious place to
pool resources, but I guess proprietary vendors are going to keep doing
their thing :/

>> > The challenge becomes to keep up with these link rates in software
>> > as there is a lot of hardware offloading.
>
> Yes that's our pain point, because that's what the SOCs vendors
> deliver and you need to use that because there is no alternative.
>
> It's not entierely the SOCs vendors fault though.
>
> 15 years ago, your average SOC's CPU would be something like 200Mhz
> MIPS, Linux standard forwarding path (softirq => routing+netfilter =>
> qdisc) was too slow for this, too much cache footprint/overhead. So
> every vendor started building alternatives forwarding path in their
> hardware and never looked back.
>
> Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which
> with a non crappy network driver and internal fabric should do be able
> to route 1Gbit/s with out-of-the-box kernel forwarding.
>
> But that's too late. SOC vendors compete against each others, and the
> big telcos need a way to tell which SOC is better to make a buying
> decision. So synthetic benchmarks have become the norm, and since
> everybody was able to do fill their pipe with 1500 bytes packets,
> benchmarks have moved to unrealistic 64 bytes packets (so called
> wirespeed)
>
> If you don't have hardware acceleration for forwarding, you don't
> exist in those benchmarks and will not sell your chipset. Also they
> invested so much in their alternative network stack that it's
> difficult to stop (huge R&D teams). That being said, they do have a
> point, when speed go above 1Gbit/s, the kernel becomes the bottleneck.
>
> For Free.fr 10Gbit/s offer, we had to develop an alternative
> (software) forwarding path using polling mode model (DPDK style),
> otherwise our albeit powerful ARM Cortex A72@2Ghz could not forward
> more than 2Gbit/s.

We're working on that in kernel land - ever heard of XDP? On big-iron
servers we have no issues pushing 10s and 100s of Gbps in software
(well, the latter only given enough cores to throw at the problem :)).
There's not a lot of embedded platforms support as of yet, but we do
have some people in the ARM world working on that.

Personally, I do see embedded platforms as an important (future) use
case for XDP, though, in particular for CPEs. So I would be very
interested in hearing details about your particular platform, and your
DPDK solution, so we can think about what it will take to achieve the
same with XDP. If you're interested in this, please feel free to reach
out :)

> And going multicore/RSS does not fly when the test case is single
> stream TCP session, which is what most speedtest application do (ookla
> only recently added multi-connections test).

Setting aside the fact that those single-stream tests ought to die a
horrible death, I do wonder if it would be feasible to do a bit of
'optimising for the test'? With XDP we do have the ability to steer
packets between CPUs based on arbitrary criteria, and while it is not as
efficient as hardware-based RSS it may be enough to achieve line rate
for a single TCP flow?

-Toke

Re: [Cake] Advantages to tightly tuning latency

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 6004 bytes --]

On Thu, Apr 23, 2020 at 1:57 PM Toke Høiland-Jørgensen <toke@redhat.com>
wrote:

> Maxime Bizon <mbizon@freebox.fr> writes:
>
> > On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote:
> >
> > Hello,
> >
> >> > Free has been using SFQ since 2005 (if I remember well).
> >> > They announced the wide deployment of SFQ in the free.fr newsgroup.
> >> > Wi-Fi in the free.fr router was not as good though.
> >>
> >> They're working on it. :)
> >
> > yes indeed.
> >
> > Switching to softmac approach, so now mac80211 will do rate control
> > and scheduling (using wake_tx_queue model).
> >
> > for 5ghz, we use ath10k
>
> That is awesome! Please make sure you include the AQL patch for ath10k,
> it really works wonders, as Dave demonstrated:
>
>
> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.html
>
> >> I am very, very happy for y'all. Fiber has always been the sanest
> >> thing. Is there a SPF+ gpon card yet I can plug into a convention
> >> open source router yet?
> >
> > FYI Free.fr uses 10G-EPON, not GPON.
> >
> > Also most deployments are using an additionnal terminal equipement at
> > called "ONT" or "ONU" that handle the PON part and exposes an ethernet
> > port where the operator CPE is plugged. So we are back to the early
> > days of DSL, where the hardest part (scheduling) is done inside a
> > black box. That makes it easier to replace the operator CPE with your
> > own standard ethernet router though.
> >
> > At least SOCs with integrated PON (supporting all flavours
> > GPON/EPON/..)  are starting to be deployed. Nothing available in
> > opensource.
> >
> > Also note that it's not just kernel drivers, you also need some higher
> > OAM stack to make that work, and there are a lot of existing
> > standards, DPOE (EPON), OMCI (GPON)... all with interop challenges.
>
> It always bugged me that there was no open source support for these
> esoteric protocols and standards. It would seem like an obvious place to
> pool resources, but I guess proprietary vendors are going to keep doing
> their thing :/
>
> >> > The challenge becomes to keep up with these link rates in software
> >> > as there is a lot of hardware offloading.
> >
> > Yes that's our pain point, because that's what the SOCs vendors
> > deliver and you need to use that because there is no alternative.
> >
> > It's not entierely the SOCs vendors fault though.
> >
> > 15 years ago, your average SOC's CPU would be something like 200Mhz
> > MIPS, Linux standard forwarding path (softirq => routing+netfilter =>
> > qdisc) was too slow for this, too much cache footprint/overhead. So
> > every vendor started building alternatives forwarding path in their
> > hardware and never looked back.
> >
> > Nowdays, the baseline SOC CPU would be ARM Cortex A53@~1Ghz, which
> > with a non crappy network driver and internal fabric should do be able
> > to route 1Gbit/s with out-of-the-box kernel forwarding.
> >
> > But that's too late. SOC vendors compete against each others, and the
> > big telcos need a way to tell which SOC is better to make a buying
> > decision. So synthetic benchmarks have become the norm, and since
> > everybody was able to do fill their pipe with 1500 bytes packets,
> > benchmarks have moved to unrealistic 64 bytes packets (so called
> > wirespeed)
>

Yes, I'm not working anymore on these kinds of platforms
but I do remember the pain.
Hardware offloading may also have unexpected behaviours
for stateful offloads. A flow starts in a slow path and
then it moves to the fast path in hardware.
Out of order at this stage can be nasty for a TCP connection.
Worse a packet loss.



> >
> > If you don't have hardware acceleration for forwarding, you don't
> > exist in those benchmarks and will not sell your chipset. Also they
> > invested so much in their alternative network stack that it's
> > difficult to stop (huge R&D teams). That being said, they do have a
> > point, when speed go above 1Gbit/s, the kernel becomes the bottleneck.
> >
> > For Free.fr 10Gbit/s offer, we had to develop an alternative
> > (software) forwarding path using polling mode model (DPDK style),
> > otherwise our albeit powerful ARM Cortex A72@2Ghz could not forward
> > more than 2Gbit/s.
>
> We're working on that in kernel land - ever heard of XDP? On big-iron
> servers we have no issues pushing 10s and 100s of Gbps in software
> (well, the latter only given enough cores to throw at the problem :)).
> There's not a lot of embedded platforms support as of yet, but we do
> have some people in the ARM world working on that.
>
> Personally, I do see embedded platforms as an important (future) use
> case for XDP, though, in particular for CPEs. So I would be very
> interested in hearing details about your particular platform, and your
> DPDK solution, so we can think about what it will take to achieve the
> same with XDP. If you're interested in this, please feel free to reach
> out :)
>
> > And going multicore/RSS does not fly when the test case is single
> > stream TCP session, which is what most speedtest application do (ookla
> > only recently added multi-connections test).
>
> Setting aside the fact that those single-stream tests ought to die a
> horrible death, I do wonder if it would be feasible to do a bit of
> 'optimising for the test'? With XDP we do have the ability to steer
> packets between CPUs based on arbitrary criteria, and while it is not as
> efficient as hardware-based RSS it may be enough to achieve line rate
> for a single TCP flow?
>


Toke yes I was implicitly thinking about XDP but I did
not read yet any experience in CPEs using that.

DPDK, netmap and kernel bypass may be an option but
you lose all qdiscs.




>
> -Toke
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 8769 bytes --]

Re: [Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Thursday 23 Apr 2020 à 13:57:25 (+0200), Toke Høiland-Jørgensen wrote:

Hello Toke,

> That is awesome! Please make sure you include the AQL patch for ath10k,
> it really works wonders, as Dave demonstrated:
> 
> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.html

Was it in 5.4 ? we try to stick to LTS kernel

> We're working on that in kernel land - ever heard of XDP? On big-iron
> servers we have no issues pushing 10s and 100s of Gbps in software
> (well, the latter only given enough cores to throw at the problem :)).
> There's not a lot of embedded platforms support as of yet, but we do
> have some people in the ARM world working on that.
> 
> Personally, I do see embedded platforms as an important (future) use
> case for XDP, though, in particular for CPEs. So I would be very
> interested in hearing details about your particular platform, and your
> DPDK solution, so we can think about what it will take to achieve the
> same with XDP. If you're interested in this, please feel free to reach
> out :)

Last time I looked at XDP, its primary use cases were "early drop" /
"anti ddos".

In our case, each packet has to be routed+NAT, we have VLAN tags, we
also have MAP-E for IPv4 traffic. So in the vanilla forwading path,
this does multiple rounds of RX/TX because of tunneling.

TBH, the hard work in our optimized forwarding code is figuring out
what modifications to apply to each packets. Now whether modifications
and tx would be done by XDP or by hand written C code in the kernel is
more of a detail, even though using XDP is much cleaner of course.

What the kernel always lacked is what DaveM called once the "grand
unified flow cache", the ability to do a single lookup and be able to
decide what to do with the packet. Instead we have the bridge
forwarding table, the ip routing table (used to be a cache), the
netfilter conntrack lookup, and multiple round of those if you do
tunneling.

Once you have this "flow table" infrastructure, it becomes easy to
offload forwarding, either to real hardware, or software (for example,
dedicate a CPU core in polling mode)

The good news is that it seems nftables is building this:

https://wiki.nftables.org/wiki-nftables/index.php/Flowtable

I'm still using iptables, but it seems that the features I was missing
like TCPMSS are now in nft also, so I will have a look.


> Setting aside the fact that those single-stream tests ought to die a
> horrible death, I do wonder if it would be feasible to do a bit of
> 'optimising for the test'? With XDP we do have the ability to steer
> packets between CPUs based on arbitrary criteria, and while it is not as
> efficient as hardware-based RSS it may be enough to achieve line rate
> for a single TCP flow?

You cannot do steering for a single TCP flow at those rates because
you will get out-of-order packets and kill TCP performance.

I do not consider those single-stream tests to be unrealistic, this is
exactly what happen if say you buy a game on Steam and download it.

-- 
Maxime

Re: [Cake] Advantages to tightly tuning latency

[Toke Høiland-Jørgensen]

Maxime Bizon <mbizon@freebox.fr> writes:

> On Thursday 23 Apr 2020 à 13:57:25 (+0200), Toke Høiland-Jørgensen wrote:
>
> Hello Toke,
>
>> That is awesome! Please make sure you include the AQL patch for ath10k,
>> it really works wonders, as Dave demonstrated:
>> 
>> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.html
>
> Was it in 5.4 ? we try to stick to LTS kernel

Didn't make it in until 5.5, unfortunately... :(

I can try to produce a patch that you can manually apply on top of 5.4
if you're interested?

>> We're working on that in kernel land - ever heard of XDP? On big-iron
>> servers we have no issues pushing 10s and 100s of Gbps in software
>> (well, the latter only given enough cores to throw at the problem :)).
>> There's not a lot of embedded platforms support as of yet, but we do
>> have some people in the ARM world working on that.
>> 
>> Personally, I do see embedded platforms as an important (future) use
>> case for XDP, though, in particular for CPEs. So I would be very
>> interested in hearing details about your particular platform, and your
>> DPDK solution, so we can think about what it will take to achieve the
>> same with XDP. If you're interested in this, please feel free to reach
>> out :)
>
> Last time I looked at XDP, its primary use cases were "early drop" /
> "anti ddos".

Yeah, that's the obvious use case (i.e., easiest to implement). But we
really want it to be a general purpose acceleration layer where you can
selectively use only the kernel facilities you need for your use case -
or even skip some of them entirely and reimplement an optimised subset
fitting your use case.

> In our case, each packet has to be routed+NAT, we have VLAN tags, we
> also have MAP-E for IPv4 traffic. So in the vanilla forwading path,
> this does multiple rounds of RX/TX because of tunneling.
>
> TBH, the hard work in our optimized forwarding code is figuring out
> what modifications to apply to each packets. Now whether modifications
> and tx would be done by XDP or by hand written C code in the kernel is
> more of a detail, even though using XDP is much cleaner of course.
>
> What the kernel always lacked is what DaveM called once the "grand
> unified flow cache", the ability to do a single lookup and be able to
> decide what to do with the packet. Instead we have the bridge
> forwarding table, the ip routing table (used to be a cache), the
> netfilter conntrack lookup, and multiple round of those if you do
> tunneling.
>
> Once you have this "flow table" infrastructure, it becomes easy to
> offload forwarding, either to real hardware, or software (for example,
> dedicate a CPU core in polling mode)
>
> The good news is that it seems nftables is building this:
>
> https://wiki.nftables.org/wiki-nftables/index.php/Flowtable
>
> I'm still using iptables, but it seems that the features I was missing
> like TCPMSS are now in nft also, so I will have a look.


I find it useful to think of XDP as a 'software offload' - i.e. a fast
path where you implement the most common functionality as efficiently as
possible and dynamically fall back to the full stack for the edge cases.
Enabling lookups in the flow table from XDP would be an obvious thing to
do, for instance. There were some patches going by to enable some kind
of lookup into conntrack at some point, but I don't recall the details.

Anyhow, my larger point was that we really do want to enable such use
cases for XDP; but we are lacking the details of what exactly is missing
before we can get to something that's useful / deployable. So any
details you could share about what feature set you are supporting in
your own 'fast path' implementation would be really helpful. As would
details about the hardware platform you are using. You can send them
off-list if you don't want to make it public, of course :)

>> Setting aside the fact that those single-stream tests ought to die a
>> horrible death, I do wonder if it would be feasible to do a bit of
>> 'optimising for the test'? With XDP we do have the ability to steer
>> packets between CPUs based on arbitrary criteria, and while it is not as
>> efficient as hardware-based RSS it may be enough to achieve line rate
>> for a single TCP flow?
>
> You cannot do steering for a single TCP flow at those rates because
> you will get out-of-order packets and kill TCP performance.

Depends on the TCP stack (I think). 

> I do not consider those single-stream tests to be unrealistic, this is
> exactly what happen if say you buy a game on Steam and download it.

Steam is perhaps a bad example as that is doing something very much like
bittorrent AFAIK; but point taken, people do occasionally run
single-stream downloads and want them to be fast. I'm just annoyed that
this becomes the *one* benchmark people run, to the exclusion of
everything else that has a much larger impact on the overall user
experience :/

-Toke

Re: [Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Thursday 23 Apr 2020 à 18:42:11 (+0200), Toke Høiland-Jørgensen wrote:

> Didn't make it in until 5.5, unfortunately... :(
> 
> I can try to produce a patch that you can manually apply on top of 5.4
> if you're interested?

I could do it, but the thing I'm more worried about is the lack of
test coverage from everyone else.

> Anyhow, my larger point was that we really do want to enable such use
> cases for XDP; but we are lacking the details of what exactly is missing
> before we can get to something that's useful / deployable. So any
> details you could share about what feature set you are supporting in
> your own 'fast path' implementation would be really helpful. As would
> details about the hardware platform you are using. You can send them
> off-list if you don't want to make it public, of course :)

there is no hardware specific feature used, it's all software

imagine this "simple" setup, pretty much what anyone's home router is
doing:

<br0> with <eth0> + <wlan0> inside, private IPv4 address
<wan0.vlan> with IPv6, vlan interface over <wan0>
<map0> with IPv4, MAP-E tunnel over <wan0.vlan>

then:
  - IPv6 routing between <br0> and <wan0.vlan>
  - IPv4 routing + NAT between <br0> and <map0>

iptables would be filled with usual rules, per interface ALLOW rules
in FORWARD chain, DNAT rules in PREROUTING to access LAN from WAN...

and then you want this to be fast :)

What we do is build a "flow" table on top of conntrack, so with a
single lookup we find the flow, the destination interface, and what
modifications to apply to the packet (L3 address to change, encap to
add/remove, etc etc)

Then we do this lookup more or less early in RX path, on our oldest
platform we even had to do this from the ethernet driver, and do TX
from there too, skipping qdisc layer and allowing cache maintenance
hacks (partial invalidation and wback)


nftable with flowtables seems to be have developped something that
could replace our flow cache, but I'm not sure if it can handle our
tunneling scheme yet. It even has a notion of offloaded flow for
hardware that can support it.

If you add an XDP offload to it, with an option to do the
lookup/modification/tx at the layer you want, depending on the
performance you need, whether you want qdisc.. that you'd give you
pretty much the same thing we use today, but with a cleaner design.


> Depends on the TCP stack (I think). 

I guess Linux deals with OFO better, but unfortunately that's not the
main OS used by our subscribers...

> Steam is perhaps a bad example as that is doing something very much like
> bittorrent AFAIK; but point taken, people do occasionally run
> single-stream downloads and want them to be fast. I'm just annoyed that
> this becomes the *one* benchmark people run, to the exclusion of
> everything else that has a much larger impact on the overall user
> experience :/

that one is easy

convince ookla to add some kind of "latency under load" metric, and
have them report it as a big red flag when too high, and even better
add scary messages like "this connection is not suitable for online
gaming".

subscribers will bug telco, then telco will bug SOCs vendors

-- 
Maxime

Re: [Cake] Advantages to tightly tuning latency

[Sebastian Moeller]

Hi Maxime,

if I might ask you a tangential question, did you also look at MAP-T (translation) and if so, what made you choose MAP-E (encapsulation)?

Best Regards
	Sebastian


> On Apr 23, 2020, at 19:31, Maxime Bizon <mbizon@freebox.fr> wrote:
> 
> 
> On Thursday 23 Apr 2020 à 18:42:11 (+0200), Toke Høiland-Jørgensen wrote:
> 
>> Didn't make it in until 5.5, unfortunately... :(
>> 
>> I can try to produce a patch that you can manually apply on top of 5.4
>> if you're interested?
> 
> I could do it, but the thing I'm more worried about is the lack of
> test coverage from everyone else.
> 
>> Anyhow, my larger point was that we really do want to enable such use
>> cases for XDP; but we are lacking the details of what exactly is missing
>> before we can get to something that's useful / deployable. So any
>> details you could share about what feature set you are supporting in
>> your own 'fast path' implementation would be really helpful. As would
>> details about the hardware platform you are using. You can send them
>> off-list if you don't want to make it public, of course :)
> 
> there is no hardware specific feature used, it's all software
> 
> imagine this "simple" setup, pretty much what anyone's home router is
> doing:
> 
> <br0> with <eth0> + <wlan0> inside, private IPv4 address
> <wan0.vlan> with IPv6, vlan interface over <wan0>
> <map0> with IPv4, MAP-E tunnel over <wan0.vlan>
> 
> then:
>  - IPv6 routing between <br0> and <wan0.vlan>
>  - IPv4 routing + NAT between <br0> and <map0>
> 
> iptables would be filled with usual rules, per interface ALLOW rules
> in FORWARD chain, DNAT rules in PREROUTING to access LAN from WAN...
> 
> and then you want this to be fast :)
> 
> What we do is build a "flow" table on top of conntrack, so with a
> single lookup we find the flow, the destination interface, and what
> modifications to apply to the packet (L3 address to change, encap to
> add/remove, etc etc)
> 
> Then we do this lookup more or less early in RX path, on our oldest
> platform we even had to do this from the ethernet driver, and do TX
> from there too, skipping qdisc layer and allowing cache maintenance
> hacks (partial invalidation and wback)
> 
> 
> nftable with flowtables seems to be have developped something that
> could replace our flow cache, but I'm not sure if it can handle our
> tunneling scheme yet. It even has a notion of offloaded flow for
> hardware that can support it.
> 
> If you add an XDP offload to it, with an option to do the
> lookup/modification/tx at the layer you want, depending on the
> performance you need, whether you want qdisc.. that you'd give you
> pretty much the same thing we use today, but with a cleaner design.
> 
> 
>> Depends on the TCP stack (I think). 
> 
> I guess Linux deals with OFO better, but unfortunately that's not the
> main OS used by our subscribers...
> 
>> Steam is perhaps a bad example as that is doing something very much like
>> bittorrent AFAIK; but point taken, people do occasionally run
>> single-stream downloads and want them to be fast. I'm just annoyed that
>> this becomes the *one* benchmark people run, to the exclusion of
>> everything else that has a much larger impact on the overall user
>> experience :/
> 
> that one is easy
> 
> convince ookla to add some kind of "latency under load" metric, and
> have them report it as a big red flag when too high, and even better
> add scary messages like "this connection is not suitable for online
> gaming".
> 
> subscribers will bug telco, then telco will bug SOCs vendors
> 
> -- 
> Maxime
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Thursday 23 Apr 2020 à 20:30:09 (+0200), Sebastian Moeller wrote:

Hello,

> if I might ask you a tangential question, did you also look at MAP-T
> (translation) and if so, what made you choose MAP-E (encapsulation)?

decision was made a long time ago

I remember it had something to do with the fact that, at that time,
Cisco only supported MAP-E on the switches we planned to use for this,
so we did not even look at MAP-T

-- 
Maxime

Re: [Cake] Advantages to tightly tuning latency

[Toke Høiland-Jørgensen]

Maxime Bizon <mbizon@freebox.fr> writes:

> On Thursday 23 Apr 2020 à 18:42:11 (+0200), Toke Høiland-Jørgensen wrote:
>
>> Didn't make it in until 5.5, unfortunately... :(
>> 
>> I can try to produce a patch that you can manually apply on top of 5.4
>> if you're interested?
>
> I could do it, but the thing I'm more worried about is the lack of
> test coverage from everyone else.

Yeah, I guess you'd be on the hook for backporting any follow-ups
yourself if you do that; maybe better to wait for the next longterm
kernel release, then...

>> Anyhow, my larger point was that we really do want to enable such use
>> cases for XDP; but we are lacking the details of what exactly is missing
>> before we can get to something that's useful / deployable. So any
>> details you could share about what feature set you are supporting in
>> your own 'fast path' implementation would be really helpful. As would
>> details about the hardware platform you are using. You can send them
>> off-list if you don't want to make it public, of course :)
>
> there is no hardware specific feature used, it's all software

I meant more details of your SOC platform. You already said it's
ARM-based, so I guess the most important missing piece is which (Linux)
driver does the Ethernet device(s) use?

> imagine this "simple" setup, pretty much what anyone's home router is
> doing:
>
> <br0> with <eth0> + <wlan0> inside, private IPv4 address
> <wan0.vlan> with IPv6, vlan interface over <wan0>
> <map0> with IPv4, MAP-E tunnel over <wan0.vlan>
>
> then:
>   - IPv6 routing between <br0> and <wan0.vlan>
>   - IPv4 routing + NAT between <br0> and <map0>
>
> iptables would be filled with usual rules, per interface ALLOW rules
> in FORWARD chain, DNAT rules in PREROUTING to access LAN from WAN...
>
> and then you want this to be fast :)
>
> What we do is build a "flow" table on top of conntrack, so with a
> single lookup we find the flow, the destination interface, and what
> modifications to apply to the packet (L3 address to change, encap to
> add/remove, etc etc)
>
> Then we do this lookup more or less early in RX path, on our oldest
> platform we even had to do this from the ethernet driver, and do TX
> from there too, skipping qdisc layer and allowing cache maintenance
> hacks (partial invalidation and wback)

This sounds pretty much what you'd do with an XDP program: Packet comes
in -> XDP program runs, parses the headers, does a flow lookup, modifies
the packet and redirects it out the egress interface. All in one go,
kernel never even builds an skb for the packet.

You can build most of that with XDP today, but you'd need to implement
all the lookups yourself using BPF maps; having a hook into the kernel
conntrack / flow tables would help with that. I guess I should look into
what happened with that hook.

Oh, and we also need to solve queueing in XDP; it's all line rate ATM,
which is obviously not ideal for a CPE :)

> nftable with flowtables seems to be have developped something that
> could replace our flow cache, but I'm not sure if it can handle our
> tunneling scheme yet. It even has a notion of offloaded flow for
> hardware that can support it.

Well, the nice thing about XDP is that you can just implement any custom
encapsulation that is not supported by the kernel yourself :)

> If you add an XDP offload to it, with an option to do the
> lookup/modification/tx at the layer you want, depending on the
> performance you need, whether you want qdisc.. that you'd give you
> pretty much the same thing we use today, but with a cleaner design.

Yup, I think so. What does your current solution do with packets that
are destined for the WiFi interface, BTW? Just punt them to the regular
kernel path?

>> Depends on the TCP stack (I think). 
>
> I guess Linux deals with OFO better, but unfortunately that's not the
> main OS used by our subscribers...

Yeah, you really should do something about that ;)

>> Steam is perhaps a bad example as that is doing something very much like
>> bittorrent AFAIK; but point taken, people do occasionally run
>> single-stream downloads and want them to be fast. I'm just annoyed that
>> this becomes the *one* benchmark people run, to the exclusion of
>> everything else that has a much larger impact on the overall user
>> experience :/
>
> that one is easy
>
> convince ookla to add some kind of "latency under load" metric, and
> have them report it as a big red flag when too high, and even better
> add scary messages like "this connection is not suitable for online
> gaming".
>
> subscribers will bug telco, then telco will bug SOCs vendors

Heh. Easy in theory, yeah. I do believe people on this list have tried
to convince them; no luck thus far :/

-Toke

Re: [Cake] Advantages to tightly tuning latency

[Maxime Bizon]

On Thursday 23 Apr 2020 à 20:35:15 (+0200), Toke Høiland-Jørgensen wrote:

> I meant more details of your SOC platform. You already said it's
> ARM-based, so I guess the most important missing piece is which (Linux)
> driver does the Ethernet device(s) use?

- Marvell Kirkwood, mv643xx_eth driver
- Marvell A8k, mvpp2 driver

> Yup, I think so. What does your current solution do with packets that
> are destined for the WiFi interface, BTW? Just punt them to the regular
> kernel path?

yes, but that won't fly anymore for 11ax rates

-- 
Maxime

Re: [Cake] Advantages to tightly tuning latency

[Toke Høiland-Jørgensen]

Maxime Bizon <mbizon@freebox.fr> writes:

> On Thursday 23 Apr 2020 à 20:35:15 (+0200), Toke Høiland-Jørgensen wrote:
>
>> I meant more details of your SOC platform. You already said it's
>> ARM-based, so I guess the most important missing piece is which (Linux)
>> driver does the Ethernet device(s) use?
>
> - Marvell Kirkwood, mv643xx_eth driver
> - Marvell A8k, mvpp2 driver

No native XDP support in any of those, unfortunately :(

>> Yup, I think so. What does your current solution do with packets that
>> are destined for the WiFi interface, BTW? Just punt them to the regular
>> kernel path?
>
> yes, but that won't fly anymore for 11ax rates

Indeed, that was partly why I asked :)
Got any plans?

-Toke

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

On Thu, Apr 23, 2020 at 4:05 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Maxime Bizon <mbizon@freebox.fr> writes:
>
> > On Thursday 23 Apr 2020 à 20:35:15 (+0200), Toke Høiland-Jørgensen wrote:
> >
> >> I meant more details of your SOC platform. You already said it's
> >> ARM-based, so I guess the most important missing piece is which (Linux)
> >> driver does the Ethernet device(s) use?
> >
> > - Marvell Kirkwood, mv643xx_eth driver
> > - Marvell A8k, mvpp2 driver
>
> No native XDP support in any of those, unfortunately :(
>
> >> Yup, I think so. What does your current solution do with packets that
> >> are destined for the WiFi interface, BTW? Just punt them to the regular
> >> kernel path?
> >
> > yes, but that won't fly anymore for 11ax rates
>
> Indeed, that was partly why I asked :)
> Got any plans?

I would really love to get something going out of this initiative:

https://pointer.ngi.eu/pages/ngi-pointer-opencalls

... either with marvell wifi, 802.11ax, ath11k, mt76... gpon onts,
just something, anything fq-ing at the very least, over fiber of any
sort, would be a start.

is there anyone out there with some spare time and needs some eu dollars?

I tend to disagree that we need massive offloading on 802.11ax. We
need firmware on the chip that does per station scheduling, for the
rest of the queue management a smarter host cpu in the a72 class more
than suffices, especially if the packets are arriving gso'd.

>
> -Toke

>


--
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Thu, Apr 23, 2020 at 4:57 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Maxime Bizon <mbizon@freebox.fr> writes:
>
> > On Wednesday 22 Apr 2020 à 07:48:43 (-0700), Dave Taht wrote:
> >
> > Hello,
> >
> >> > Free has been using SFQ since 2005 (if I remember well).
> >> > They announced the wide deployment of SFQ in the free.fr newsgroup.
> >> > Wi-Fi in the free.fr router was not as good though.
> >>
> >> They're working on it. :)
> >
> > yes indeed.
> >
> > Switching to softmac approach, so now mac80211 will do rate control
> > and scheduling (using wake_tx_queue model).
> >
> > for 5ghz, we use ath10k
>
> That is awesome! Please make sure you include the AQL patch for ath10k,
> it really works wonders, as Dave demonstrated:
>
> https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-March/002721.html

and THIS patch is looking lovely at the higher rates I'm testing at.
Yea, it needs to be more robust, but I'm seeing tcp rtt inflation
of exactly the right buffer size (15ms), on an HT20 or VHT80 link,
where before I'd see 30+ms... (it's nice that flent can plot both tcp
rtt and ping rtt now),
ecn "just works",

[-- Attachment #2: 982-do-codel-more-right.patch --]
[-- Type: text/x-patch, Size: 1767 bytes --]

diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index c431722..92ba09b 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -476,10 +476,11 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,
 	sta->sta.max_rc_amsdu_len = IEEE80211_MAX_MPDU_LEN_HT_BA;
 
 	sta->cparams.ce_threshold = CODEL_DISABLED_THRESHOLD;
-	sta->cparams.target = MS2TIME(20);
+	sta->cparams.target = MS2TIME(5);
 	sta->cparams.interval = MS2TIME(100);
 	sta->cparams.ecn = true;
-
+	sta_dbg(sdata, "Codel target, interval %d, %d\n", sta->cparams.target,
+		sta->cparams.interval);
 	sta_dbg(sdata, "Allocated STA %pM\n", sta->sta.addr);
 
 	return sta;
@@ -2468,15 +2469,9 @@ static void sta_update_codel_params(struct sta_info *sta, u32 thr)
 	if (!sta->sdata->local->ops->wake_tx_queue)
 		return;
 
-	if (thr && thr < STA_SLOW_THRESHOLD * sta->local->num_sta) {
-		sta->cparams.target = MS2TIME(50);
-		sta->cparams.interval = MS2TIME(300);
-		sta->cparams.ecn = false;
-	} else {
-		sta->cparams.target = MS2TIME(20);
-		sta->cparams.interval = MS2TIME(100);
-		sta->cparams.ecn = true;
-	}
+	sta->cparams.target = MS2TIME(5);
+	sta->cparams.interval = MS2TIME(100);
+	sta->cparams.ecn = true;
 }
 
 void ieee80211_sta_set_expected_throughput(struct ieee80211_sta *pubsta,
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index 535911b..ca50d0a 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -1551,7 +1551,7 @@ int ieee80211_txq_setup_flows(struct ieee80211_local *local)
 
 	codel_params_init(&local->cparams);
 	local->cparams.interval = MS2TIME(100);
-	local->cparams.target = MS2TIME(20);
+	local->cparams.target = MS2TIME(5);
 	local->cparams.ecn = true;
 
 	local->cvars = kcalloc(fq->flows_cnt, sizeof(local->cvars[0]),

Re: [Cake] Advantages to tightly tuning latency

[Justin Kilpatrick]

On Tue, Apr 21, 2020, at 2:44 PM, Dave Taht wrote:
> It has always been my dream, that at least for outbound, there would
> be sufficient backpressure from the driver
> to not have to shape at all, or monitor the link. We have that now in
> BQL and AQL. free.fr's dsl driver "does the right thing" - no other
> dsl driver does. Nor usb network devices. I hope more folk roll up
> their sleeves and test the ath10k some, it's looking lovely from here.
> 
> https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/
> 
> next up either the new mediatek chip or intel..

I'm curious if you have any opinions about the WiFi stacks for the Marvel Armada and Qualcomm IPQ40xx. Any trees I should be barking up for better performance? We have had some complaints in higher interference areas...

These devices have the best WireGuard performance per dollar for Althea's use case so we're deploying them pretty heavily.

-- 
  Justin Kilpatrick
  justin@althea.net

Re: [Cake] Advantages to tightly tuning latency

[Dave Taht]

On Tue, Apr 21, 2020 at 4:07 PM Justin Kilpatrick <justin@althea.net> wrote:
>
> On Tue, Apr 21, 2020, at 2:44 PM, Dave Taht wrote:
> > It has always been my dream, that at least for outbound, there would
> > be sufficient backpressure from the driver
> > to not have to shape at all, or monitor the link. We have that now in
> > BQL and AQL. free.fr's dsl driver "does the right thing" - no other
> > dsl driver does. Nor usb network devices. I hope more folk roll up
> > their sleeves and test the ath10k some, it's looking lovely from here.
> >
> > https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/
> >
> > next up either the new mediatek chip or intel..
>
> I'm curious if you have any opinions about the WiFi stacks for the Marvel Armada

Marvell's wifi is currently "not horrible", but certainly overbuffered.

>and Qualcomm IPQ40xx.

I was under the impression this looked like an ath10k to the world. Am
I wrong? What products is it in these days?

>Any trees I should be barking up for better performance? We have had some complaints in higher interference areas...
>
> These devices have the best WireGuard performance per dollar for Althea's use case so we're deploying them pretty heavily.
>
> --
>   Justin Kilpatrick
>   justin@althea.net



-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729