[Cerowrt-devel] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Cerowrt-devel] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Dave Taht]

It's been a while since I did a talk, this is the shuttleworth flash
grants conference... feel free to hang out with me at:

https://crocus.irif.fr:8443/group/bufferbloat

---------- Forwarded message ---------
From: Juliusz Chroboczek <jch@irif.fr>
Date: Tue, Feb 23, 2021 at 9:37 AM
Subject: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23
To: <galene@lists.galene.org>


Slightly off topic, Dave Taht will be giving a talk about bufferbloat and
jitter in the Internet at 8pm CET today, Tuesday 23 February 2021:

  https://ffwd.flashgrants.org/calendar.html#event-16/

I haven't seen his talk yet, but Dave is usually a good speaker.
_______________________________________________
Galene mailing list -- galene@lists.galene.org
To unsubscribe send an email to galene-leave@lists.galene.org


-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

Thanks for the talk Dave and it was nice meeting you all!

Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still

Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.

Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...

Best Regards
Nils

Re: [Cerowrt-devel] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Dave Taht]

wow, that is (predictably) miserable, even with cake. The only
solution that is going to
work is to somehow actively monitor your link quality and adjust cake
to suit. Or we can start trying to use kathie's passive ping tools.

On Tue, Feb 23, 2021 at 1:30 PM Nils Andreas Svee <me@lochnair.net> wrote:
>
> Thanks for the talk Dave and it was nice meeting you all!
>
> Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still
>
> Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.
>
> Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
> Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...
>
> Best Regards
> Nils



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Stephen Hemminger]

On Tue, 23 Feb 2021 17:14:31 -0800
Dave Taht <dave.taht@gmail.com> wrote:

> wow, that is (predictably) miserable, even with cake. The only
> solution that is going to
> work is to somehow actively monitor your link quality and adjust cake
> to suit. Or we can start trying to use kathie's passive ping tools.
> 
> On Tue, Feb 23, 2021 at 1:30 PM Nils Andreas Svee <me@lochnair.net> wrote:
> >
> > Thanks for the talk Dave and it was nice meeting you all!
> >
> > Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still
> >
> > Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.
> >
> > Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
> > Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...
> >
> > Best Regards
> > Nils  
> 
> 
> 

Because cable was down, used a burner SIM and a spare phone over LTE for a day.
And agree it is awful. Couldn't bring myself to run Flent over this pig with wings.

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> wow, that is (predictably) miserable, even with cake. The only
> solution that is going to
> work is to somehow actively monitor your link quality and adjust cake
> to suit. Or we can start trying to use kathie's passive ping tools.

We have a PhD student working on a BPF-based implementation of pping:
https://github.com/xdp-project/bpf-examples/tree/master/pping

My hope is that this can end up being an always-on thing that runs on
the router and can be used to adjust the CAKE parameters as latency
spikes.

There are still a few rough edges on the implementation (most notably
the data output can become quite high), but it should otherwise be
usable, so feel free to take it for a spin. Needs a fairly recent LLVM
(10+ IIRC) to compile the BPF parts.

-Toke

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Taraldsen Erik]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]

Disclamer: I'm working on the Fixed Wireless products for Telenor (Zyxel NR7101 outdoor wall mounted unit).  Not the Mobile Broadband products.  We are working with Zyxel and Qualcom to try and implement an upstream queue which adapts to available radio resources. To much NDA so can't really disclose anything useful.  Lets just say we are aware of the issues and are actively working to try and improve the situation - but don't hold your breath for a sollution.




What sort of HW are you running your LTE on?


Do you have a subscription with rate limitations?  The PGW (router which enforces the limit) is a lot more latency friendly than if you are radio limited.  So it may be beneficial to have a "slow" subscription rather than "free speed" then it comes to latency.  Slow meaning lower subscrption rate than radio rate.


Also would be interesting to see separate tests for upstream and downstream, as the issues and implementations are different in each direction.




-Erik



________________________________
Fra: Bloat <bloat-bounces@lists.bufferbloat.net> på vegne av Nils Andreas Svee <me@lochnair.net>
Sendt: tirsdag 23. februar 2021 22.30.33
Til: Dave Taht; bloat; cerowrt-devel; Make-Wifi-fast; Toke Høiland-Jørgensen via Cake
Emne: Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

Thanks for the talk Dave and it was nice meeting you all!

Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still

Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.

Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...

Best Regards
Nils
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: Type: text/html, Size: 3569 bytes --]

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Taraldsen Erik <erik.taraldsen@telenor.no> writes:

> Disclamer: I'm working on the Fixed Wireless products for Telenor
> (Zyxel NR7101 outdoor wall mounted unit). Not the Mobile Broadband
> products. We are working with Zyxel and Qualcom to try and implement
> an upstream queue which adapts to available radio resources. To much
> NDA so can't really disclose anything useful. Lets just say we are
> aware of the issues and are actively working to try and improve the
> situation - but don't hold your breath for a sollution.
>
> What sort of HW are you running your LTE on?
>
> Do you have a subscription with rate limitations? The PGW (router
> which enforces the limit) is a lot more latency friendly than if you
> are radio limited. So it may be beneficial to have a "slow"
> subscription rather than "free speed" then it comes to latency. Slow
> meaning lower subscrption rate than radio rate.

Ah, this is lovely! "How do I get my internet to be faster?" "Just buy a
slower connection!" :D

-Toke

Re: [Cerowrt-devel] [Make-wifi-fast] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Jonathan Morton]

> On 24 Feb, 2021, at 5:19 pm, Taraldsen Erik <erik.taraldsen@telenor.no> wrote:
> 
> Do you have a subscription with rate limitations?  The PGW (router which enforces the limit) is a lot more latency friendly than if you are radio limited.  So it may be beneficial to have a "slow" subscription rather than "free speed" then it comes to latency.  Slow meaning lower subscrption rate than radio rate.

This is actually something I've noticed in Finland with DNA.  The provisioning shaper they use for the "poverty tariff" is quite well debloated (which was very much not the case some years ago).  However, there's no tariff at any convenient level between 1Mbps (poverty tariff) and 50Mbps (probably radio limited on a single carrier).

 - Jonathan Morton

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

I'll look into pping. Admittedly I'm quite ignorant about BPF, so I'll likely blunder about for a bit, but hey, got it to compile - *and* run, but I didn't get any output other than the messages from clean_map. Dunno if I did something wrong, I'll look at it again tomorrow.

Best Regards
Nils

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

On 24 February 2021 23:49:48 CET, Nils Andreas Svee <me@lochnair.net> wrote:
>I'll look into pping. Admittedly I'm quite ignorant about BPF, so I'll
>likely blunder about for a bit, but hey, got it to compile - *and* run,
>but I didn't get any output other than the messages from clean_map.
>Dunno if I did something wrong, I'll look at it again tomorrow.


It monitors TCP traffic, so it'll only show something if you have TCP flows going while you run it...

-Toke

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Ah, yeah it's fixed wireless I meant. Didn't really know how to say it right in English.

We've got the Huawei B818-260 with an EMCOM XPOL-2 4G/5G on the wall.

Yes, we've got a 30 Mbit/sec subscription. In practice we usually see ~30 Mbit downstream and 10-15 upstream, and I believe when we first got the B818 and antenna hooked up I measured ~70 Mbit downstream with a subscription that didn't have any rate limitations, so I assumed we should have a good amount of leeway if something affected the signal.

Sure, I can run some more tests tomorrow. Could also grab some signal stats from the B818 if those are of interest.

By the way, I forgot to mention it when I posted yesterdays tests, but those were conducted over a WireGuard tunnel with CAKE for the downstream running on the other side. Doing that was the only way to get the ADSL subscription we had to behave decently, it simply couldn't handle things like Steam downloads with CAKE on a IFB device in ingress mode, and shaping downstream this way kinda stuck.

Best Regards
Nils

[-- Attachment #2: Type: text/html, Size: 1643 bytes --]

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

I ran it on my router though, which has a decent amount of TCP flows running at any given time.
It's all going over a wg tunnel though, that might be wonky for all I know.

libbpf doesn't like it if I try to disable the VPN and run it on the WAN interface at least, because it's a virtio interface.
> libbpf: Kernel error message: virtio_net: Too few free TX rings available

Best Regards
Nils

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

"Nils Andreas Svee" <me@lochnair.net> writes:

> I ran it on my router though, which has a decent amount of TCP flows running at any given time.
> It's all going over a wg tunnel though, that might be wonky for all I
> know.

Ah, wireguard doesn't have XDP support, so that's likely not going to
work; and if you run it on the physical interface, even if you didn't
get driver errors, the tool would just see the encrypted packets which
is not terribly helpful (it parses TCP timestamps to match
incoming/outgoing packets and compute the RTT).

I guess we should be more flexible about which hooks we support, so it
can also be used on devices with no XDP support. Adding in Simon, who is
writing the code; I think he is focused on getting a couple of other
features done first, but this could go on the TODO list :)

-Toke

Re: [Cerowrt-devel] [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

On 2/25/21 11:30 AM, Toke Høiland-Jørgensen wrote:

> Ah, wireguard doesn't have XDP support, so that's likely not going to
> work; and if you run it on the physical interface, even if you didn't
> get driver errors, the tool would just see the encrypted packets which
> is not terribly helpful (it parses TCP timestamps to match
> incoming/outgoing packets and compute the RTT).

I figured that might be the case. Yes I would've disabled the VPN if I 
didn't get driver errors.
I changed the network interface to use an emulated Intel E1000 tonight, 
and if I bypass the VPN it works as it should.

> I guess we should be more flexible about which hooks we support, so it
> can also be used on devices with no XDP support. Adding in Simon, who is
> writing the code; I think he is focused on getting a couple of other
> features done first, but this could go on the TODO list :)

It's not like I'm in a hurry, and I'd probably need some time to figure 
out how to tweak the CAKE parameters correctly with this anyway ;)

Speaking of, isn't one of the challenges with solutions like these that 
it's hard to tell when conditions have improved and allow for more 
throughput? At least that's what I remember being the issue when I 
tested CAKE's autorate-ingress back in the day.


- Nils

>> -Toke

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Nils Andreas Svee <me@lochnair.net> writes:

> On 2/25/21 11:30 AM, Toke Høiland-Jørgensen wrote:
>
>> Ah, wireguard doesn't have XDP support, so that's likely not going to
>> work; and if you run it on the physical interface, even if you didn't
>> get driver errors, the tool would just see the encrypted packets which
>> is not terribly helpful (it parses TCP timestamps to match
>> incoming/outgoing packets and compute the RTT).
>
> I figured that might be the case. Yes I would've disabled the VPN if I 
> didn't get driver errors.
> I changed the network interface to use an emulated Intel E1000 tonight, 
> and if I bypass the VPN it works as it should.

Right, awesome!

>> I guess we should be more flexible about which hooks we support, so it
>> can also be used on devices with no XDP support. Adding in Simon, who is
>> writing the code; I think he is focused on getting a couple of other
>> features done first, but this could go on the TODO list :)
>
> It's not like I'm in a hurry, and I'd probably need some time to figure 
> out how to tweak the CAKE parameters correctly with this anyway ;)
>
> Speaking of, isn't one of the challenges with solutions like these that 
> it's hard to tell when conditions have improved and allow for more 
> throughput? At least that's what I remember being the issue when I 
> tested CAKE's autorate-ingress back in the day.

Yeah, there would have to be some kind of probing to discover when the
bandwidth goes up (maybe something like what BBR does?). Working out the
details of this is still in the future, this is all just loose plans
that I'll try to get back to once we have the measurement tool working
reasonably well. Input and experiments welcome, of course!

-Toke