[Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Arie]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Recently I accidentally discovered something quite odd about my home DOCSIS
connection (shaped with cake) while running a flent rrul test.
During the flent rrul test, another machine in my home had started a Steam
game update, this opened somewhere between 8-12 connections and saturated
my downstream.

The interesting part was the resulting rrul graph, showing perfect latency
control during the test, but 100-150ms of bufferbloat at the very start and
end of the rrul graph (when rrul is idle, just pinging).

Instead of a Steam download I started an iperf3 client on a remote machine
that opened 16 connections to a LAN machine. Same behavior as the Steam
download.

At first I thought this was a bug with cake, but no matter how I tuned cake
(limiting my 400/40 connection to 50/20 for example), the behavior was
similar.
I tried fq_codel+htb, and got similar results to cake, high latency before
the rrul test kicked off, great latency while rrul was doing its up and
downloads.

So I swapped out my self-compile LEDE router with an Edgerouter ER-X with
both stock and LEDE firmware, both with fq_codel and cake (on both stock
and LEDE firmware). Same results.
Even without any SQM active, the rrul test still improved the latency of my
connection when it's also downloading from 8+ connections at the same time.

I managed to take the flent rrul test out of the equation by starting a
hping3 "flood" to the hop next of my cable modem.

hping3 -2 -d 0 -s 10080 -k -p 80 -i u100 first-hop-or-ISP-gateway-goes-here
>

This also fixed the latency while many downloads were running by sending
about 300KB/s of empty UDP packets to some unresponsive port on my ISPs
local gateway.

I'm puzzled by this result, somehow sending 300KB/s of empty packets
upstream drastically improves the latency of my connection when it's
receiving many downloads.

Equipment used:
Cisco EPC3212 cable modem (8x4 EuroDOCSIS 3.0)
Linksys WRT1900ACS router with LEDE
Ubiquiti Edgerouter ER-X with both EdgeOS (stock) and LEDE

[-- Attachment #2: Type: text/html, Size: 2496 bytes --]

Re: [Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Dave Taht]

Power save?

On Tue, Jan 16, 2018 at 11:36 AM, Arie <nospam@ariekanarie.nl> wrote:
> Recently I accidentally discovered something quite odd about my home DOCSIS
> connection (shaped with cake) while running a flent rrul test.
> During the flent rrul test, another machine in my home had started a Steam
> game update, this opened somewhere between 8-12 connections and saturated my
> downstream.
>
> The interesting part was the resulting rrul graph, showing perfect latency
> control during the test, but 100-150ms of bufferbloat at the very start and
> end of the rrul graph (when rrul is idle, just pinging).
>
> Instead of a Steam download I started an iperf3 client on a remote machine
> that opened 16 connections to a LAN machine. Same behavior as the Steam
> download.
>
> At first I thought this was a bug with cake, but no matter how I tuned cake
> (limiting my 400/40 connection to 50/20 for example), the behavior was
> similar.
> I tried fq_codel+htb, and got similar results to cake, high latency before
> the rrul test kicked off, great latency while rrul was doing its up and
> downloads.
>
> So I swapped out my self-compile LEDE router with an Edgerouter ER-X with
> both stock and LEDE firmware, both with fq_codel and cake (on both stock and
> LEDE firmware). Same results.
> Even without any SQM active, the rrul test still improved the latency of my
> connection when it's also downloading from 8+ connections at the same time.
>
> I managed to take the flent rrul test out of the equation by starting a
> hping3 "flood" to the hop next of my cable modem.
>
>> hping3 -2 -d 0 -s 10080 -k -p 80 -i u100
>> first-hop-or-ISP-gateway-goes-here
>
>
> This also fixed the latency while many downloads were running by sending
> about 300KB/s of empty UDP packets to some unresponsive port on my ISPs
> local gateway.
>
> I'm puzzled by this result, somehow sending 300KB/s of empty packets
> upstream drastically improves the latency of my connection when it's
> receiving many downloads.
>
> Equipment used:
> Cisco EPC3212 cable modem (8x4 EuroDOCSIS 3.0)
> Linksys WRT1900ACS router with LEDE
> Ubiquiti Edgerouter ER-X with both EdgeOS (stock) and LEDE
>
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Arie]

[-- Attachment #1: Type: text/plain, Size: 2848 bytes --]

Not powersave of any component in my control. Powersave of the CMTS? I'd
expect high latency for idle pings in that case too.



On 16 January 2018 at 20:52, Dave Taht <dave.taht@gmail.com> wrote:

> Power save?
>
> On Tue, Jan 16, 2018 at 11:36 AM, Arie <nospam@ariekanarie.nl> wrote:
> > Recently I accidentally discovered something quite odd about my home
> DOCSIS
> > connection (shaped with cake) while running a flent rrul test.
> > During the flent rrul test, another machine in my home had started a
> Steam
> > game update, this opened somewhere between 8-12 connections and
> saturated my
> > downstream.
> >
> > The interesting part was the resulting rrul graph, showing perfect
> latency
> > control during the test, but 100-150ms of bufferbloat at the very start
> and
> > end of the rrul graph (when rrul is idle, just pinging).
> >
> > Instead of a Steam download I started an iperf3 client on a remote
> machine
> > that opened 16 connections to a LAN machine. Same behavior as the Steam
> > download.
> >
> > At first I thought this was a bug with cake, but no matter how I tuned
> cake
> > (limiting my 400/40 connection to 50/20 for example), the behavior was
> > similar.
> > I tried fq_codel+htb, and got similar results to cake, high latency
> before
> > the rrul test kicked off, great latency while rrul was doing its up and
> > downloads.
> >
> > So I swapped out my self-compile LEDE router with an Edgerouter ER-X with
> > both stock and LEDE firmware, both with fq_codel and cake (on both stock
> and
> > LEDE firmware). Same results.
> > Even without any SQM active, the rrul test still improved the latency of
> my
> > connection when it's also downloading from 8+ connections at the same
> time.
> >
> > I managed to take the flent rrul test out of the equation by starting a
> > hping3 "flood" to the hop next of my cable modem.
> >
> >> hping3 -2 -d 0 -s 10080 -k -p 80 -i u100
> >> first-hop-or-ISP-gateway-goes-here
> >
> >
> > This also fixed the latency while many downloads were running by sending
> > about 300KB/s of empty UDP packets to some unresponsive port on my ISPs
> > local gateway.
> >
> > I'm puzzled by this result, somehow sending 300KB/s of empty packets
> > upstream drastically improves the latency of my connection when it's
> > receiving many downloads.
> >
> > Equipment used:
> > Cisco EPC3212 cable modem (8x4 EuroDOCSIS 3.0)
> > Linksys WRT1900ACS router with LEDE
> > Ubiquiti Edgerouter ER-X with both EdgeOS (stock) and LEDE
> >
> >
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 3883 bytes --]

Re: [Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]


Arie <nospam@ariekanarie.nl> wrote:
    > I'm puzzled by this result, somehow sending 300KB/s of empty packets
    > upstream drastically improves the latency of my connection when it's
    > receiving many downloads.

Maybe not exactly power save, but perhaps asking for the upstream bandwidth
gets the CMTS to notice you more often and schedule a dump of it's queue of
traffic for you.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Bloat] Curious bufferbloat on DOCSIS connection with many downloads

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]

Along what Michael Richardson said, with GPON I have seen where idle
latency was ~1.5ms, but under load latency was about 0.2ms. I wouldn't be
surprised if TDMA scheduling playing a factor. I don't know what kind of
latency SLAs DOCSIS can or typically have. GPON has scheduling latency all
the way down to 0.5ms.

On Tue, Jan 16, 2018 at 3:34 PM, Michael Richardson <mcr@sandelman.ca>
wrote:

>
> Arie <nospam@ariekanarie.nl> wrote:
>     > I'm puzzled by this result, somehow sending 300KB/s of empty packets
>     > upstream drastically improves the latency of my connection when it's
>     > receiving many downloads.
>
> Maybe not exactly power save, but perhaps asking for the upstream bandwidth
> gets the CMTS to notice you more often and schedule a dump of it's queue of
> traffic for you.
>
> --
> ]               Never tell me the odds!                 | ipv6 mesh
> networks [
> ]   Michael Richardson, Sandelman Software Works        | network
> architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on
> rails    [
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

[-- Attachment #2: Type: text/html, Size: 2001 bytes --]