[Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Dave Taht]

It's been a while since I did a talk, this is the shuttleworth flash
grants conference... feel free to hang out with me at:

https://crocus.irif.fr:8443/group/bufferbloat

---------- Forwarded message ---------
From: Juliusz Chroboczek <jch@irif.fr>
Date: Tue, Feb 23, 2021 at 9:37 AM
Subject: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23
To: <galene@lists.galene.org>


Slightly off topic, Dave Taht will be giving a talk about bufferbloat and
jitter in the Internet at 8pm CET today, Tuesday 23 February 2021:

  https://ffwd.flashgrants.org/calendar.html#event-16/

I haven't seen his talk yet, but Dave is usually a good speaker.
_______________________________________________
Galene mailing list -- galene@lists.galene.org
To unsubscribe send an email to galene-leave@lists.galene.org


-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

Thanks for the talk Dave and it was nice meeting you all!

Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still

Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.

Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...

Best Regards
Nils

Re: [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Dave Taht]

wow, that is (predictably) miserable, even with cake. The only
solution that is going to
work is to somehow actively monitor your link quality and adjust cake
to suit. Or we can start trying to use kathie's passive ping tools.

On Tue, Feb 23, 2021 at 1:30 PM Nils Andreas Svee <me@lochnair.net> wrote:
>
> Thanks for the talk Dave and it was nice meeting you all!
>
> Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still
>
> Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.
>
> Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
> Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...
>
> Best Regards
> Nils



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Stephen Hemminger]

On Tue, 23 Feb 2021 17:14:31 -0800
Dave Taht <dave.taht@gmail.com> wrote:

> wow, that is (predictably) miserable, even with cake. The only
> solution that is going to
> work is to somehow actively monitor your link quality and adjust cake
> to suit. Or we can start trying to use kathie's passive ping tools.
> 
> On Tue, Feb 23, 2021 at 1:30 PM Nils Andreas Svee <me@lochnair.net> wrote:
> >
> > Thanks for the talk Dave and it was nice meeting you all!
> >
> > Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still
> >
> > Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.
> >
> > Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
> > Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...
> >
> > Best Regards
> > Nils  
> 
> 
> 

Because cable was down, used a burner SIM and a spare phone over LTE for a day.
And agree it is awful. Couldn't bring myself to run Flent over this pig with wings.

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> wow, that is (predictably) miserable, even with cake. The only
> solution that is going to
> work is to somehow actively monitor your link quality and adjust cake
> to suit. Or we can start trying to use kathie's passive ping tools.

We have a PhD student working on a BPF-based implementation of pping:
https://github.com/xdp-project/bpf-examples/tree/master/pping

My hope is that this can end up being an always-on thing that runs on
the router and can be used to adjust the CAKE parameters as latency
spikes.

There are still a few rough edges on the implementation (most notably
the data output can become quite high), but it should otherwise be
usable, so feel free to take it for a spin. Needs a fairly recent LLVM
(10+ IIRC) to compile the BPF parts.

-Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Taraldsen Erik]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]

Disclamer: I'm working on the Fixed Wireless products for Telenor (Zyxel NR7101 outdoor wall mounted unit).  Not the Mobile Broadband products.  We are working with Zyxel and Qualcom to try and implement an upstream queue which adapts to available radio resources. To much NDA so can't really disclose anything useful.  Lets just say we are aware of the issues and are actively working to try and improve the situation - but don't hold your breath for a sollution.




What sort of HW are you running your LTE on?


Do you have a subscription with rate limitations?  The PGW (router which enforces the limit) is a lot more latency friendly than if you are radio limited.  So it may be beneficial to have a "slow" subscription rather than "free speed" then it comes to latency.  Slow meaning lower subscrption rate than radio rate.


Also would be interesting to see separate tests for upstream and downstream, as the issues and implementations are different in each direction.




-Erik



________________________________
Fra: Bloat <bloat-bounces@lists.bufferbloat.net> på vegne av Nils Andreas Svee <me@lochnair.net>
Sendt: tirsdag 23. februar 2021 22.30.33
Til: Dave Taht; bloat; cerowrt-devel; Make-Wifi-fast; Toke Høiland-Jørgensen via Cake
Emne: Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

Thanks for the talk Dave and it was nice meeting you all!

Never really did much in the way of Flent tests after moving from ADSL to Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G sucks indeed., not as bad as DSL without SQM mind, but still

Avg. latency without SQM at some points close to 800 ms or above. Had to sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.

Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
Oh btw I promise I'll try to not break things when you need to access something on that box again Dave...

Best Regards
Nils
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[-- Attachment #2: Type: text/html, Size: 3569 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Taraldsen Erik <erik.taraldsen@telenor.no> writes:

> Disclamer: I'm working on the Fixed Wireless products for Telenor
> (Zyxel NR7101 outdoor wall mounted unit). Not the Mobile Broadband
> products. We are working with Zyxel and Qualcom to try and implement
> an upstream queue which adapts to available radio resources. To much
> NDA so can't really disclose anything useful. Lets just say we are
> aware of the issues and are actively working to try and improve the
> situation - but don't hold your breath for a sollution.
>
> What sort of HW are you running your LTE on?
>
> Do you have a subscription with rate limitations? The PGW (router
> which enforces the limit) is a lot more latency friendly than if you
> are radio limited. So it may be beneficial to have a "slow"
> subscription rather than "free speed" then it comes to latency. Slow
> meaning lower subscrption rate than radio rate.

Ah, this is lovely! "How do I get my internet to be faster?" "Just buy a
slower connection!" :D

-Toke

Re: [Cake] [Make-wifi-fast] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Jonathan Morton]

> On 24 Feb, 2021, at 5:19 pm, Taraldsen Erik <erik.taraldsen@telenor.no> wrote:
> 
> Do you have a subscription with rate limitations?  The PGW (router which enforces the limit) is a lot more latency friendly than if you are radio limited.  So it may be beneficial to have a "slow" subscription rather than "free speed" then it comes to latency.  Slow meaning lower subscrption rate than radio rate.

This is actually something I've noticed in Finland with DNA.  The provisioning shaper they use for the "poverty tariff" is quite well debloated (which was very much not the case some years ago).  However, there's no tariff at any convenient level between 1Mbps (poverty tariff) and 50Mbps (probably radio limited on a single carrier).

 - Jonathan Morton

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

I'll look into pping. Admittedly I'm quite ignorant about BPF, so I'll likely blunder about for a bit, but hey, got it to compile - *and* run, but I didn't get any output other than the messages from clean_map. Dunno if I did something wrong, I'll look at it again tomorrow.

Best Regards
Nils

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

On 24 February 2021 23:49:48 CET, Nils Andreas Svee <me@lochnair.net> wrote:
>I'll look into pping. Admittedly I'm quite ignorant about BPF, so I'll
>likely blunder about for a bit, but hey, got it to compile - *and* run,
>but I didn't get any output other than the messages from clean_map.
>Dunno if I did something wrong, I'll look at it again tomorrow.


It monitors TCP traffic, so it'll only show something if you have TCP flows going while you run it...

-Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Ah, yeah it's fixed wireless I meant. Didn't really know how to say it right in English.

We've got the Huawei B818-260 with an EMCOM XPOL-2 4G/5G on the wall.

Yes, we've got a 30 Mbit/sec subscription. In practice we usually see ~30 Mbit downstream and 10-15 upstream, and I believe when we first got the B818 and antenna hooked up I measured ~70 Mbit downstream with a subscription that didn't have any rate limitations, so I assumed we should have a good amount of leeway if something affected the signal.

Sure, I can run some more tests tomorrow. Could also grab some signal stats from the B818 if those are of interest.

By the way, I forgot to mention it when I posted yesterdays tests, but those were conducted over a WireGuard tunnel with CAKE for the downstream running on the other side. Doing that was the only way to get the ADSL subscription we had to behave decently, it simply couldn't handle things like Steam downloads with CAKE on a IFB device in ingress mode, and shaping downstream this way kinda stuck.

Best Regards
Nils

[-- Attachment #2: Type: text/html, Size: 1643 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

I ran it on my router though, which has a decent amount of TCP flows running at any given time.
It's all going over a wg tunnel though, that might be wonky for all I know.

libbpf doesn't like it if I try to disable the VPN and run it on the WAN interface at least, because it's a virtio interface.
> libbpf: Kernel error message: virtio_net: Too few free TX rings available

Best Regards
Nils

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Taraldsen Erik]

[-- Attachment #1: Type: text/plain, Size: 2759 bytes --]

This is getting rather rather Telenor internal and probably is not true for other ISP's, but here we go.  Mobile Broad Band (MBB) is handled by Telenor's Mobile division. Fixed Wireless Access (FWA) is handled by Telenors Fixed division (the same group who does DSL, DOCSIS and GPON).  To further complicate matters, Telenor Group (holding company for Telenor Fixed and Telenor Mobile) deiced to launch FWA before the new value chain was ready.  So we have two versions of FWA - internally called FWA1 and FWA2.  FWA1 (which you have Nils) is not fully compliant with the regulatory definition of a fixed access and has very limited management support (technical monitoring, firmware upgrades etc).  I'm working on the FWA2 value chain and devices.


I assume you are doing the tests on Ethernet, not wifi?  I know the B818 has some wifi issues as well.  To isolate the LTE access I mean.


Yes interesting to see the signal stats from the device, but not needed.  More for my curiosity to compare with the Zyxel devices.


So with subscription limited to 30Mbit and probably radio head room up to ~70Mbit downstream and 10Mbit upstream.  My guess is that you will see more latency in the upstream direction.  I believe the B818 has the very typical "1000 packets fifo, never drop" implementation.



-Erik






________________________________
Fra: Cake <cake-bounces@lists.bufferbloat.net> på vegne av Nils Andreas Svee <me@lochnair.net>
Sendt: torsdag 25. februar 2021 00.27
Til: Taraldsen Erik; Dave Taht; bloat; cerowrt-devel; Make-Wifi-fast; Toke Høiland-Jørgensen via Cake
Emne: Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

Ah, yeah it's fixed wireless I meant. Didn't really know how to say it right in English.

We've got the Huawei B818-260 with an EMCOM XPOL-2 4G/5G on the wall.

Yes, we've got a 30 Mbit/sec subscription. In practice we usually see ~30 Mbit downstream and 10-15 upstream, and I believe when we first got the B818 and antenna hooked up I measured ~70 Mbit downstream with a subscription that didn't have any rate limitations, so I assumed we should have a good amount of leeway if something affected the signal.

Sure, I can run some more tests tomorrow. Could also grab some signal stats from the B818 if those are of interest.

By the way, I forgot to mention it when I posted yesterdays tests, but those were conducted over a WireGuard tunnel with CAKE for the downstream running on the other side. Doing that was the only way to get the ADSL subscription we had to behave decently, it simply couldn't handle things like Steam downloads with CAKE on a IFB device in ingress mode, and shaping downstream this way kinda stuck.

Best Regards
Nils


[-- Attachment #2: Type: text/html, Size: 4255 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

"Nils Andreas Svee" <me@lochnair.net> writes:

> I ran it on my router though, which has a decent amount of TCP flows running at any given time.
> It's all going over a wg tunnel though, that might be wonky for all I
> know.

Ah, wireguard doesn't have XDP support, so that's likely not going to
work; and if you run it on the physical interface, even if you didn't
get driver errors, the tool would just see the encrypted packets which
is not terribly helpful (it parses TCP timestamps to match
incoming/outgoing packets and compute the RTT).

I guess we should be more flexible about which hooks we support, so it
can also be used on devices with no XDP support. Adding in Simon, who is
writing the code; I think he is focused on getting a couple of other
features done first, but this could go on the TODO list :)

-Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 4353 bytes --]

On 2/25/21 9:18 AM, Taraldsen Erik wrote:

> This is getting rather rather Telenor internal and probably is not 
> true for other ISP's, but here we go.  Mobile Broad Band (MBB) is 
> handled by Telenor's Mobile division. Fixed Wireless Access (FWA) is 
> handled by Telenors Fixed division (the same group who does DSL, 
> DOCSIS and GPON).  To further complicate matters, Telenor Group 
> (holding company for Telenor Fixed and Telenor Mobile) deiced to 
> launch FWA before the new value chain was ready.  So we have two 
> versions of FWA - internally called FWA1 and FWA2.  FWA1 (which you 
> have Nils) is not fully compliant with the regulatory definition of a 
> fixed access and has very limited management support (technical 
> monitoring, firmware upgrades etc).  I'm working on the FWA2 value 
> chain and devices.
>
I'll admit it doesn't surprise me that something like that was going on, 
after we got the B818 I can't recall seeing a single new FW revision.
>
> I assume you are doing the tests on Ethernet, not wifi?  I know the 
> B818 has some wifi issues as well.  To isolate the LTE access I mean.
>
I am indeed running them on Ethernet. I don't actually use the B818 for 
anything else than as a LTE modem, so I wouldn't know, if I could get 
the thing to bridge I would. Or replace it with something else entirely 
that I can control, but that doesn't seem to be an option on FWA. That 
said the Zyxel looks like a better option since I assume it acts like a 
bridge by default.
>
> Yes interesting to see the signal stats from the device, but not 
> needed.  More for my curiosity to compare with the Zyxel devices.
>
I dumped the raw signal stats the web interface grabs in an XML file 
together with the Flent tests. Also did some upload only tests tonight 
at different speeds (no VPN in play this time).
>
> So with subscription limited to 30Mbit and probably radio head room up 
> to ~70Mbit downstream and 10Mbit upstream.  My guess is that you will 
> see more latency in the upstream direction.  I believe the B818 has 
> the very typical "1000 packets fifo, never drop" implementation.
>
Most likely yes. That's been my observation as well, that it generally 
acts up the worst when somethings using the upstream. Not entirely sure 
what I can do about that, seeing as I had to shape at 5Mbit to get rid 
of the worst spikes (but not all).


For all I know there might be something to gain by adjusting the antenna 
slightly, it's pointed in the general direction of the cell tower, but 
we didn't fine tune it (I seem to recall it was a cold winter evening).


On that point, I would've liked to collect signal stats over time, but 
the B818 seems to insist on chucking me out after being idle for a few 
minutes, better known as scraping the stats with cURL


- Nils.

>
> -Erik
>
>
>
>
>
>
>
> ------------------------------------------------------------------------
> *Fra:* Cake <cake-bounces@lists.bufferbloat.net> på vegne av Nils 
> Andreas Svee <me@lochnair.net>
> *Sendt:* torsdag 25. februar 2021 00.27
> *Til:* Taraldsen Erik; Dave Taht; bloat; cerowrt-devel; 
> Make-Wifi-fast; Toke Høiland-Jørgensen via Cake
> *Emne:* Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and 
> jitter at 8pm CET Tuesday 23
> Ah, yeah it's fixed wireless I meant. Didn't really know how to say it 
> right in English.
>
> We've got the Huawei B818-260 with an EMCOM XPOL-2 4G/5G on the wall.
>
> Yes, we've got a 30 Mbit/sec subscription. In practice we usually see 
> ~30 Mbit downstream and 10-15 upstream, and I believe when we first 
> got the B818 and antenna hooked up I measured ~70 Mbit downstream with 
> a subscription that didn't have any rate limitations, so I assumed we 
> should have a good amount of leeway if something affected the signal.
>
> Sure, I can run some more tests tomorrow. Could also grab some signal 
> stats from the B818 if those are of interest.
>
> By the way, I forgot to mention it when I posted yesterdays tests, but 
> those were conducted over a WireGuard tunnel with CAKE for the 
> downstream running on the other side. Doing that was the only way to 
> get the ADSL subscription we had to behave decently, it simply 
> couldn't handle things like Steam downloads with CAKE on a IFB device 
> in ingress mode, and shaping downstream this way kinda stuck.
>
> Best Regards
> Nils
>

[-- Attachment #2: Type: text/html, Size: 9182 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

On 2/25/21 11:30 AM, Toke Høiland-Jørgensen wrote:

> Ah, wireguard doesn't have XDP support, so that's likely not going to
> work; and if you run it on the physical interface, even if you didn't
> get driver errors, the tool would just see the encrypted packets which
> is not terribly helpful (it parses TCP timestamps to match
> incoming/outgoing packets and compute the RTT).

I figured that might be the case. Yes I would've disabled the VPN if I 
didn't get driver errors.
I changed the network interface to use an emulated Intel E1000 tonight, 
and if I bypass the VPN it works as it should.

> I guess we should be more flexible about which hooks we support, so it
> can also be used on devices with no XDP support. Adding in Simon, who is
> writing the code; I think he is focused on getting a couple of other
> features done first, but this could go on the TODO list :)

It's not like I'm in a hurry, and I'd probably need some time to figure 
out how to tweak the CAKE parameters correctly with this anyway ;)

Speaking of, isn't one of the challenges with solutions like these that 
it's hard to tell when conditions have improved and allow for more 
throughput? At least that's what I remember being the issue when I 
tested CAKE's autorate-ingress back in the day.


- Nils

>> -Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Taraldsen Erik]

[-- Attachment #1: Type: text/plain, Size: 2635 bytes --]

Fra: Nils Andreas Svee <me@lochnair.net>

I am indeed running them on Ethernet. I don't actually use the B818 for anything else than as a LTE modem, so I wouldn't know, if I could get the thing to bridge I would. Or replace it with something else entirely that I can control, but that doesn't seem to be an option on FWA. That said the Zyxel looks like a better option since I assume it acts like a bridge by default.
The Zyxel device indeed acts as a bridge, or at least as close approximation as we can get it.  The PDP addressing protocol in mobile networks requres the address termination to happen where the SIM card resides.  So the device does some trickery with brctl, routing and iptables to simulate a bridge setup.


I dumped the raw signal stats the web interface grabs in an XML file together with the Flent tests. Also did some upload only tests tonight at different speeds (no VPN in play this time).
rsrp is good and rsrq is great at your location.  However you have ended up on the 800MHz band.  That is intended for coverage, not capacity.  It uses only 10MHz bandwitdh and is shared with a lot more customers.  You probably should be able to get an 1800MHz frequency which has 20MHz and is shared among fewer customers.

Most likely yes. That's been my observation as well, that it generally acts up the worst when somethings using the upstream. Not entirely sure what I can do about that, seeing as I had to shape at 5Mbit to get rid of the worst spikes (but not all).

This is tricky.  You don't have a static set of resources.  You request resources "as needed".  The "as needed" amongst other things reads the buffer back pressure.  So if you shape to far down the LTE device will not request enough resources.  Shape to high and there will not be enough resources available to share.  And available resources vary with number of subscribers on that cell, weather, the subscribers usage and interference from other cell towers.  To get a proper solution to this I don't see a way around getting the chipset manufacturers on board.



On that point, I would've liked to collect signal stats over time, but the B818 seems to insist on chucking me out after being idle for a few minutes, better known as scraping the stats with cURL


Have you tried to use the telnet service port (20249) on the B818?  Not all variants have that open but you could give it a shot.  You also may need to acquire an datalock code for the "at^datalock=" command.

telnet LAN_IP 20249



This is getting LTE/5G spesific.  Not sure if it belongs on the list.  Let us know if we are generating noise.


-Erik

[-- Attachment #2: Type: text/html, Size: 4035 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Nils Andreas Svee <me@lochnair.net> writes:

> On 2/25/21 11:30 AM, Toke Høiland-Jørgensen wrote:
>
>> Ah, wireguard doesn't have XDP support, so that's likely not going to
>> work; and if you run it on the physical interface, even if you didn't
>> get driver errors, the tool would just see the encrypted packets which
>> is not terribly helpful (it parses TCP timestamps to match
>> incoming/outgoing packets and compute the RTT).
>
> I figured that might be the case. Yes I would've disabled the VPN if I 
> didn't get driver errors.
> I changed the network interface to use an emulated Intel E1000 tonight, 
> and if I bypass the VPN it works as it should.

Right, awesome!

>> I guess we should be more flexible about which hooks we support, so it
>> can also be used on devices with no XDP support. Adding in Simon, who is
>> writing the code; I think he is focused on getting a couple of other
>> features done first, but this could go on the TODO list :)
>
> It's not like I'm in a hurry, and I'd probably need some time to figure 
> out how to tweak the CAKE parameters correctly with this anyway ;)
>
> Speaking of, isn't one of the challenges with solutions like these that 
> it's hard to tell when conditions have improved and allow for more 
> throughput? At least that's what I remember being the issue when I 
> tested CAKE's autorate-ingress back in the day.

Yeah, there would have to be some kind of probing to discover when the
bandwidth goes up (maybe something like what BBR does?). Working out the
details of this is still in the future, this is all just loose plans
that I'll try to get back to once we have the measurement tool working
reasonably well. Input and experiments welcome, of course!

-Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Toke Høiland-Jørgensen]

Taraldsen Erik <erik.taraldsen@telenor.no> writes:

> This is getting LTE/5G spesific.  Not sure if it belongs on the list.
> Let us know if we are generating noise.

I for one think it's fascinating - carry on! :)

-Toke

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 4228 bytes --]


On 2/26/21 8:26 AM, Taraldsen Erik wrote:
> *Fra:* Nils Andreas Svee <me@lochnair.net>
>
>     I am indeed running them on Ethernet. I don't actually use the
>     B818 for anything else than as a LTE modem, so I wouldn't know, if
>     I could get the thing to bridge I would. Or replace it with
>     something else entirely that I can control, but that doesn't seem
>     to be an option on FWA. That said the Zyxel looks like a better
>     option since I assume it acts like a bridge by default.
>
> The Zyxel device indeed acts as a bridge, or at least as close 
> approximation as we can get it.  The PDP addressing protocol in mobile 
> networks requres the address termination to happen where the SIM card 
> resides.  So the device does some trickery with brctl, routing and 
> iptables to simulate a bridge setup.
>
Oh nice. Sure it's not a "true" bridge, but for all intents and purposes 
it gets the job done. Kinda tempting to get one of those now, if only to 
avoid the whole using DMZ mode on the B818 to pass everything through.
>
>     I dumped the raw signal stats the web interface grabs in an XML
>     file together with the Flent tests. Also did some upload only
>     tests tonight at different speeds (no VPN in play this time).
>
> rsrp is good and rsrq is great at your location.  However you have 
> ended up on the 800MHz band.  That is intended for coverage, not 
> capacity.  It uses only 10MHz bandwitdh and is shared with a lot more 
> customers.  You probably should be able to get an 1800MHz frequency 
> which has 20MHz and is shared among fewer customers.
>
Yeah I thought it should be pretty good. I don't really know how to 
interpret those numbers, but Teltonika's guidelines seemed to indicate 
that it should be good.

Good point about the bands. For the first months after we signed up it 
tended to switch between 800Mhz and 1800Mhz and mostly stay on 1800Mhz, 
but now it's been stuck on 800Mhz for quite some time.

I wanted to try and lock it to 1800Mhz, but there's no option for that 
exposed in the GUI that I can find.

>     Most likely yes. That's been my observation as well, that it
>     generally acts up the worst when somethings using the upstream.
>     Not entirely sure what I can do about that, seeing as I had to
>     shape at 5Mbit to get rid of the worst spikes (but not all). 
>
> This is tricky.  You don't have a static set of resources.  You 
> request resources "as needed".  The "as needed" amongst other things 
> reads the buffer back pressure.  So if you shape to far down the LTE 
> device will not request enough resources.  Shape to high and there 
> will not be enough resources available to share.  And available 
> resources vary with number of subscribers on that cell, weather, the 
> subscribers usage and interference from other cell towers.  To get a 
> proper solution to this I don't see a way around getting the chipset 
> manufacturers on board.
>
>
Downside of shared mediums of course. So basically my best bet is to 
find somewhere in the middle to shape on, or use the pping stuff to 
somehow dynamically configure it. Of course, but we don't really have a 
voice with them, so the best we can do is support your efforts however 
we can.
>
>
>     On that point, I would've liked to collect signal stats over time,
>     but the B818 seems to insist on chucking me out after being idle
>     for a few minutes, better known as scraping the stats with cURL
>
>
> Have you tried to use the telnet service port (20249) on the B818?  
> Not all variants have that open but you could give it a shot.  You 
> also may need to acquire an datalock code for the "at^datalock=" command.
>
> telnet LAN_IP 20249
>
I had not, I didn't scan higher ports. It *is* open though, so I was 
able to connect to it. It only yells at me if I try to run any of the 
few AT commands I know (except /at)/, not sure if that's because of the 
datalock code. If that's what I'm missing, how does one go about getting 
a hold of one of those? When I looked it up I only found some sites I 
have concern about the legitimacy of.
>
>
> This is getting LTE/5G spesific.  Not sure if it belongs on the list.  
> Let us know if we are generating noise.
>
>
> -Erik

[-- Attachment #2: Type: text/html, Size: 8934 bytes --]

[Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Taraldsen Erik]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

Fra: Nils Andreas Svee <me@lochnair.net>


I wanted to try and lock it to 1800Mhz, but there's no option for that exposed in the GUI that I can find.


Generally few devices allow for that.  I do know the Zyxel devices we use allow for that in the engineering version we have for lab purposes.  I'm not sure what is available in the commercial version.  I do know however that we completely lock down the local interfaces of the Telenor branded version of the Zyxel device.


I had not, I didn't scan higher ports. It *is* open though, so I was able to connect to it. It only yells at me if I try to run any of the few AT commands I know (except at), not sure if that's because of the datalock code. If that's what I'm missing, how does one go about getting a hold of one of those? When I looked it up I only found some sites I have concern about the legitimacy of.
The datalock algorithm has been broken and is available through some of those sketchy sites.  After that Huawei updated their algorithm.  I do believe it follows device, not sw.  So if your B818 was produced before the change you can use those sites.  Spin up a VM or something to try them out.

The proper procedure it to contact Huawei and get them to give you the code. I believe it is intended only for trusted partners, but you can give it a shot to try and reach out to them.



-Erik

[-- Attachment #2: Type: text/html, Size: 2370 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Nils Andreas Svee]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

On 2/26/21 12:47 PM, Toke Høiland-Jørgensen wrote:

> Yeah, there would have to be some kind of probing to discover when the
> bandwidth goes up (maybe something like what BBR does?). Working out the
> details of this is still in the future, this is all just loose plans
> that I'll try to get back to once we have the measurement tool working
> reasonably well. Input and experiments welcome, of course!

I've kept to maintaining CAKE binaries for the EdgeRouters, so I have a 
lot to read up on if I'm gonna take a stab at this, should be fun though :)

I'll have a look at how BBR does it, and see if I can't figure out how 
that works at least.


[-- Attachment #2: Type: text/html, Size: 997 bytes --]

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Bless, Roland (TM)]

Hi,

On 26.02.21 at 16:27 Nils Andreas Svee wrote:
> On 2/26/21 12:47 PM, Toke Høiland-Jørgensen wrote:
> 
>> Yeah, there would have to be some kind of probing to discover when the
>> bandwidth goes up (maybe something like what BBR does?). Working out the
>> details of this is still in the future, this is all just loose plans
>> that I'll try to get back to once we have the measurement tool working
>> reasonably well. Input and experiments welcome, of course!
> 
> I've kept to maintaining CAKE binaries for the EdgeRouters, so I have a 
> lot to read up on if I'm gonna take a stab at this, should be fun though :)
> 
> I'll have a look at how BBR does it, and see if I can't figure out how 
> that works at least.
BBR increases its sending rate and looks whether the delivery rate
increases. It assumes that the bottleneck limit hasn't been reached
in case the delivery rate still increases. This is certainly true when
it is the only flow at the bottleneck, but not true when multiple
flows are present (the probing flow may simply steal other flows'
shares adn thus get a higher delivery rate nevertheless).
BBRv2 at least checks for packet loss and ECN
signals and detects when it hits a hard limit. One could try to
detect a correlated increase in RTT when probing for more bandwidth
and then stop, because it seems that the queue is filled by the
increased probing rate. However, getting that reliably detected out of
the RTT measurement noise is sometimes difficult.

Regards,
  Roland

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Sebastian Moeller]

Hi Roland,

are you sure that BBRv2 actually evaluates CE marks and responds to them? As far as I understood, BBR is simply not rfc3168 compliant, there was some talk about teaching BBRv2+ some still to be defined high frequency (low amplitude) ECN signaling.

The thing I fail to understand is, why BBR with its cavalier approach to drops did not grow ECN support on day one. While a drop could have a lot of reasons, including transient/stray wifi losses, a CE mark is less ambiguous.

Best Regards
	Sebastian

> On Feb 26, 2021, at 17:59, Bless, Roland (TM) <roland.bless@kit.edu> wrote:
> 
> Hi,
> 
> On 26.02.21 at 16:27 Nils Andreas Svee wrote:
>> On 2/26/21 12:47 PM, Toke Høiland-Jørgensen wrote:
>>> Yeah, there would have to be some kind of probing to discover when the
>>> bandwidth goes up (maybe something like what BBR does?). Working out the
>>> details of this is still in the future, this is all just loose plans
>>> that I'll try to get back to once we have the measurement tool working
>>> reasonably well. Input and experiments welcome, of course!
>> I've kept to maintaining CAKE binaries for the EdgeRouters, so I have a lot to read up on if I'm gonna take a stab at this, should be fun though :)
>> I'll have a look at how BBR does it, and see if I can't figure out how that works at least.
> BBR increases its sending rate and looks whether the delivery rate
> increases. It assumes that the bottleneck limit hasn't been reached
> in case the delivery rate still increases. This is certainly true when
> it is the only flow at the bottleneck, but not true when multiple
> flows are present (the probing flow may simply steal other flows'
> shares adn thus get a higher delivery rate nevertheless).
> BBRv2 at least checks for packet loss and ECN
> signals and detects when it hits a hard limit. One could try to
> detect a correlated increase in RTT when probing for more bandwidth
> and then stop, because it seems that the queue is filled by the
> increased probing rate. However, getting that reliably detected out of
> the RTT measurement noise is sometimes difficult.
> 
> Regards,
> Roland
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Cake] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Bless, Roland (TM)]

Hi Sebastian,

On 26.02.21 at 18:14 Sebastian Moeller wrote:
> are you sure that BBRv2 actually evaluates CE marks and responds to them? As far as I understood, BBR is simply not rfc3168 compliant, there was some talk about teaching BBRv2+ some still to be defined high frequency (low amplitude) ECN signaling.
AFAIK, BBRv2 reacts not in an RFC3168 compliant (classic) manner, but 
more in a DCTCP style manner.
> The thing I fail to understand is, why BBR with its cavalier approach to drops did not grow ECN support on day one. While a drop could have a lot of reasons, including transient/stray wifi losses, a CE mark is less ambiguous.
Yes, it is. But as I understand the motivation for BBR's design,
they do not want to sacrifice throughput in case of non-congestion losses.
If you read the original BBR paper, one of its motivations was its use 
in B4,
where Google operates shallow-buffered switches. Because of their small 
buffer size,
they will cause occasional packet losses due to occurring short-term 
traffic
aggregation effects, which are different from persistent congestion.

A traditional RED-based AQM may also generate CE marks in this
case and the "strong" backoff may cost too much utilization in the end
(esp. if you consider 100+Gbit/s links), so the DCTCP style is much more
scalable in this respect.

Regards,
  Roland

>> On Feb 26, 2021, at 17:59, Bless, Roland (TM) <roland.bless@kit.edu> wrote:
>>
>> Hi,
>>
>> On 26.02.21 at 16:27 Nils Andreas Svee wrote:
>>> On 2/26/21 12:47 PM, Toke Høiland-Jørgensen wrote:
>>>> Yeah, there would have to be some kind of probing to discover when the
>>>> bandwidth goes up (maybe something like what BBR does?). Working out the
>>>> details of this is still in the future, this is all just loose plans
>>>> that I'll try to get back to once we have the measurement tool working
>>>> reasonably well. Input and experiments welcome, of course!
>>> I've kept to maintaining CAKE binaries for the EdgeRouters, so I have a lot to read up on if I'm gonna take a stab at this, should be fun though :)
>>> I'll have a look at how BBR does it, and see if I can't figure out how that works at least.
>> BBR increases its sending rate and looks whether the delivery rate
>> increases. It assumes that the bottleneck limit hasn't been reached
>> in case the delivery rate still increases. This is certainly true when
>> it is the only flow at the bottleneck, but not true when multiple
>> flows are present (the probing flow may simply steal other flows'
>> shares adn thus get a higher delivery rate nevertheless).
>> BBRv2 at least checks for packet loss and ECN
>> signals and detects when it hits a hard limit. One could try to
>> detect a correlated increase in RTT when probing for more bandwidth
>> and then stop, because it seems that the queue is filled by the
>> increased probing rate. However, getting that reliably detected out of
>> the RTT measurement noise is sometimes difficult.
>>
>> Regards,
>> Roland
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat