[Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2813 bytes --]

This is something I noticed years ago, inspired me to think about a
"bobbie" policer, and I decided I could live with, and never poked
into further. After our successes with shaping inbound cable at
then-typical 20mbit rates, I was happy, although never satisifed with
the 85% magic number we use to come up with a set rate for inbound
shaping.

Simultaneously with all that work we did on sqm, linux added tcp tsq,
pacing, etc, etc and in general inbound shaping cable seems to work ok
against one or more linux tcp flows. We end up with some persistent
queuing delay (at the cmts in the 5-15ms range) which I've generally
assumed was unavoidable that fq cannot cut through (oh, I dream of FQ
at the CMTS!)

BUT: In testing fast.com's test, I see it fail to shape it to anything
sane, with spikes of up to 160ms.

So, anyway, I've seen this pathology before with netflix flows. What I
didn't realize then was that it was independent of the shaped rate.
(see plots) It's independent of the rtt setting in cake too (at least,
down to 25ms). My assumption is the netflix (freebsd) traffic + the
cable is so bursty as to not let codel keep improving its drop rate.

I'm curious if it's also the case on other link layer techs (dsl, fiber)?

The structure of my test is simple: shape inbound to 80% of the
cablemodem rate, setup irtt (not strictly needed), start this,

root # flent -H flent-fremont.bufferbloat.net -t 'your_location' -s .02 ping

, let it run a few sec, then run the fast.com test.

I tried to verify this using linux reno on a recent kernel, but that
seemed healthy. Assuming it actually got reno. Or that this weirdness
is a function of the RTT.

1) Can someone else on a cablemodem (even without the latest cake,
this happens to me on older cake and fq_codel) try this test?
2) Can someone with a dsl or fiber device try this test?
3) Is there a freebsd box "out on the net", 45ms or so from me, we can
setup netperf/irtt/etc on to run flent with? (I can donate a linode in
LA I but we'd need someone that can setup freebsd)

Some pics attached, flent data files at:
http://www.taht.net/~d/fast_vs_cable.tgz

PS I also have two other issues going on. This is the first time I've
been using irtt with a 20ms interval, and I regularly see single 50+ms
spikes (in both ping and irtt) data and also see irtt stop
transmitting. On this front, it could merely be that my (not tested in
months!) test cablemodem setup is malfunctioning also! Or we're
hitting power save. Or (finally) seeing request-grant delays. Or
scheduling delay somewhere in the net-next kernel I'm using... Or....
(regardless, this seems independent of my main issue, and I've not had
such high res data before)
-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com

[-- Attachment #2: ping_-_flent-newark-reno.png --]
[-- Type: image/png, Size: 134097 bytes --]

[-- Attachment #3: ping_-_shape_fail_60mbit.png --]
[-- Type: image/png, Size: 51478 bytes --]

[-- Attachment #4: ping_-_shape_fail_40mbit.png --]
[-- Type: image/png, Size: 61707 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> 
> 1) Can someone else on a cablemodem (even without the latest cake,
> this happens to me on older cake and fq_codel) try this test?
> 

I just tried this on my cable comcast connection. I set ingress to ~80%
of what fast.com reports when no shaper is in place.

#tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
dsthost docsis ingress
#tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
srchost nat docsis ack-filter

I got the same result as you. This is using latest cake.

Georgios

[-- Attachment #2: ping_-_Maryland.png --]
[-- Type: image/png, Size: 89791 bytes --]

[-- Attachment #3: ping-2018-07-21T131541.170952.Maryland.flent.gz --]
[-- Type: application/vnd.flent.data.gzip, Size: 173721 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Jonathan Morton]

> On 21 Jul, 2018, at 8:20 pm, Georgios Amanakis <gamanakis@gmail.com> wrote:
> 
> I got the same result as you. This is using latest cake.

I'd like to see a tcptrace of what's going on here.  A packet capture with snaplen 100 should allow me to generate one.

 - Jonathan Morton

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

On Sat, 2018-07-21 at 20:23 +0300, Jonathan Morton wrote:
> 
> I'd like to see a tcptrace of what's going on here.  A packet capture
> with snaplen 100 should allow me to generate one.
> 

I ran it again, with net.ipv4.tcp_congestion_control=reno.
Same settings as before. 'tcpdump -s 100' ran on the host (not the
router).

Georgios



[-- Attachment #2: ping-2018-07-21T133340.777464.Maryland.flent.gz --]
[-- Type: application/vnd.flent.data.gzip, Size: 176276 bytes --]

[-- Attachment #3: ping_-_Maryland-2.png --]
[-- Type: image/png, Size: 61612 bytes --]

[-- Attachment #4: host.pcap.xz --]
[-- Type: application/x-xz, Size: 834284 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

for reference can you do a download and capture against flent-newark,
while using the ping test?
On Sat, Jul 21, 2018 at 10:44 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> On Sat, 2018-07-21 at 20:23 +0300, Jonathan Morton wrote:
> >
> > I'd like to see a tcptrace of what's going on here.  A packet capture
> > with snaplen 100 should allow me to generate one.
> >
>
> I ran it again, with net.ipv4.tcp_congestion_control=reno.
> Same settings as before. 'tcpdump -s 100' ran on the host (not the
> router).
>
> Georgios
>
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Sat, 2018-07-21 at 10:47 -0700, Dave Taht wrote:
> for reference can you do a download and capture against flent-newark,
> while using the ping test?
> 

New data, this is what I did:

1) Started a ping test using the flent-fremont server.
2) Started a tcp_8down test (for 15 seconds) using the flent-newark
server. I chose tcp_8down since fast.com was also using 8 flows.
3) Captured on the host where the above tests ran.

It seems to be working as expected here.

Georgios

[-- Attachment #2: ping_-_Maryland.png --]
[-- Type: image/png, Size: 59518 bytes --]

[-- Attachment #3: ping-2018-07-21T141033.963131.Maryland.flent.gz --]
[-- Type: application/vnd.flent.data.gzip, Size: 57558 bytes --]

[-- Attachment #4: tcp_8down-2018-07-21T141046.492857.Maryland_Newark.flent.gz --]
[-- Type: application/vnd.flent.data.gzip, Size: 11777 bytes --]

[-- Attachment #5: host.newark.pcap.xz --]
[-- Type: application/x-xz, Size: 295604 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

hmm? you only have 15mbits down?
On Sat, Jul 21, 2018 at 11:18 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> On Sat, 2018-07-21 at 10:47 -0700, Dave Taht wrote:
> > for reference can you do a download and capture against flent-newark,
> > while using the ping test?
> >
>
> New data, this is what I did:
>
> 1) Started a ping test using the flent-fremont server.
> 2) Started a tcp_8down test (for 15 seconds) using the flent-newark
> server. I chose tcp_8down since fast.com was also using 8 flows.
> 3) Captured on the host where the above tests ran.
>
> It seems to be working as expected here.
>
> Georgios



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

Yes. Unshaped it is ~20mbit, the tests were ran with cake shaping at 80%.

On Sat, Jul 21, 2018, 2:20 PM Dave Taht <dave.taht@gmail.com> wrote:

> hmm? you only have 15mbits down?
> On Sat, Jul 21, 2018 at 11:18 AM Georgios Amanakis <gamanakis@gmail.com>
> wrote:
> >
> > On Sat, 2018-07-21 at 10:47 -0700, Dave Taht wrote:
> > > for reference can you do a download and capture against flent-newark,
> > > while using the ping test?
> > >
> >
> > New data, this is what I did:
> >
> > 1) Started a ping test using the flent-fremont server.
> > 2) Started a tcp_8down test (for 15 seconds) using the flent-newark
> > server. I chose tcp_8down since fast.com was also using 8 flows.
> > 3) Captured on the host where the above tests ran.
> >
> > It seems to be working as expected here.
> >
> > Georgios
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 1518 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

To summarize:

A) I think the magic 85% figure only applies at lower bandwidths.
B) We are at least partially in a pathological situation where

CMTS = 380ms of buffering, token bucket fifo at 100mbit
Cakebox: AQMing and trying to shape below 85mbit, gradually ramping up
the signalling once per 100ms downwards.

The cmts buffer fills more rapidly, particularly in slow start, while
presenting packets to the inbound shaper at 100mbit. cake starts
signalling, late, trying to achieve  but at that point the apparent
RTTs are still growing rapidly (because of the buffer building up in
the cmts inflicting that RTT), so as fast as we signal, we've got such
a big buffer built up in the CMTS that tcp only sees one signal per
RTT which is mismatched against what cake is trying to thwart. The
pathology persists.

the idea for bobbie was that the goal for codel is wrong for inbound
shaping, that instead of aiming for a rate, we needed to sum all the
overage over our rate and reduce that until it all drains from cmts
shaper. So, lets say (waving hands a bit here)

we get 160mbits/sec for 8 seconds with an outbound shaped rate of 100.
That 480mbits (independent of any signalling we did to try to reduce
it) "stuck" up there. We're trying to gradually get it to 85mbits/sec,
but the signalling is now so far behind the tcp's now observed actual
rtt that it takes forever to get anywhere and we end up in steady
state.

The more, aggressive, flows you have, the worst this disparity gets.

using perhaps cake's ingress estimator it seems possible to "bob" the
rate down until it drains, or to work on more aggressively drain the
built up queue than the gentle approach fq_codel uses, policer style.
On Sat, Jul 21, 2018 at 11:18 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> On Sat, 2018-07-21 at 10:47 -0700, Dave Taht wrote:
> > for reference can you do a download and capture against flent-newark,
> > while using the ping test?
> >
>
> New data, this is what I did:
>
> 1) Started a ping test using the flent-fremont server.
> 2) Started a tcp_8down test (for 15 seconds) using the flent-newark
> server. I chose tcp_8down since fast.com was also using 8 flows.
> 3) Captured on the host where the above tests ran.
>
> It seems to be working as expected here.
>
> Georgios



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Jonathan Morton]

> On 21 Jul, 2018, at 11:01 pm, Dave Taht <dave.taht@gmail.com> wrote:
> 
> The cmts buffer fills more rapidly, particularly in slow start, while
> presenting packets to the inbound shaper at 100mbit. cake starts
> signalling, late, trying to achieve  but at that point the apparent
> RTTs are still growing rapidly (because of the buffer building up in
> the cmts inflicting that RTT), so as fast as we signal, we've got such
> a big buffer built up in the CMTS that tcp only sees one signal per
> RTT which is mismatched against what cake is trying to thwart. The
> pathology persists.
> 
> the idea for bobbie was that the goal for codel is wrong for inbound
> shaping, that instead of aiming for a rate, we needed to sum all the
> overage over our rate and reduce that until it all drains from cmts
> shaper.

Another possibility, which I've previously mentioned but haven't got around to implementing, is to give ECN more flexibility in signalling - so that it can indicate impending congestion as well as actual congestion.

That is, as well as the present CE mark meaning "back off now", there may be softer signals carried on the dual encodings of ECT, meaning "ramp down now", "don't ramp up", and "ramp up only with caution".  These signals can be given without delay, according to instantaneous conditions at the bottleneck, without needing to estimate path RTT.  You could think of it as a version of DCTCP that can actually be deployed in the internet, because it doesn't destroy the existing meaning of CE.

The main problem is with getting the endpoints (both receiver and sender) to recognise these new signals and react appropriately to them.  Producing these signals at the AQM is relatively easy.  I think I worked out a way to do it with the two padding bytes that normally accompany the Timestamp option in TCP - this requires replacing the Timestamp option with one that has the same semantics, but also carries the extra data about recent ECT marks, and doesn't require padding to be naturally aligned in the packet.

This would give a way to halt slow-start when it reaches roughly the correct window size, instead of having it overshoot first.  It would also give a way to gently control the cwnd to the ideal value while in steady-state, instead of oscillating around it.

 - Jonathan Morton

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 148 bytes --]

This is my "inbound trying to shape a cable connection" smoking gun.
The delay curve is the same
shaping the 110mbit cmts down to 85mbit OR 55mbit.

[-- Attachment #2: inbound_shaping_smoking_gun.png --]
[-- Type: image/png, Size: 130658 bytes --]

[-- Attachment #3: inbound_smoking_gun.tgz --]
[-- Type: application/x-compressed-tar, Size: 1245294 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

I had a similar issue with my previous cable modem, whatever I shaped to
didn't matter, I still had long delays. I "fixed" it by continuously
sending a stream of empty UDP packets upstream:

hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 IP-OF-FIRST-OUTSIDE-CABLE-HOP-HERE

On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:

> This is my "inbound trying to shape a cable connection" smoking gun.
> The delay curve is the same
> shaping the 110mbit cmts down to 85mbit OR 55mbit.
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #2: Type: text/html, Size: 1162 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

or, another way we might look at it is there is very little we can do
as the cmts has to have data in it in order to burst schedule the mac
for the next string of packets, much like how fq_codel for wifi has
"one in the hardware, one ready to go", a cmts has at least one in the
hardware (per channel? a multiple? what?).

or I could be on drugs entirely. And this thread did start with
fast.com misbehaving badly regardless of the shaper in place or not,
which is not what I'm looking at now.  I need to setup a 45ms rtt
test...

anyway, as per your suggestion, the latency gets MUCH better with your
hping3 idea running, which implies that we've  been fooled all along
by the rrul test. On the other hand, I think this will hurt other
cable modems on the same wire. On the gripping hand, I'm happier
knowing that with a busier network, docsis cable, when shaped, gets
better, and that I should junk my existing test cablemodem due to the
persistent spikes I see.

I wonder if it's the sent path or the return path shattering latency
so well? I wonder if hping3 would count against your badwidth cap?

going back to trying to figure out why fast.com is so gnarly
On Sat, Jul 21, 2018 at 2:18 PM Arie <nospam@ariekanarie.nl> wrote:
>
> I had a similar issue with my previous cable modem, whatever I shaped to didn't matter, I still had long delays. I "fixed" it by continuously sending a stream of empty UDP packets upstream:
>
> hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 IP-OF-FIRST-OUTSIDE-CABLE-HOP-HERE
>
> On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> This is my "inbound trying to shape a cable connection" smoking gun.
>> The delay curve is the same
>> shaping the 110mbit cmts down to 85mbit OR 55mbit.
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #2: hping3.png --]
[-- Type: image/png, Size: 46361 bytes --]

[-- Attachment #3: hping3_cdf.png --]
[-- Type: image/png, Size: 48578 bytes --]

[-- Attachment #4: 4-8-16-hping.png --]
[-- Type: image/png, Size: 60631 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

wow. That is the best dslreports test result for cable I have ever had.

with hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 96.120.89.153

http://www.dslreports.com/speedtest/36209937

Without:

http://www.dslreports.com/speedtest/36210095

You say a different cablemodem does better without hping3 running? which? :)

Most of my production gear is based on an older arris modem (which is
quite good), most of my test gear is a bunch of netgear (free) modems
and service I got free from my time working for comcast.

I haven't got around to springing for a docsis 3.1 modem yet (they are
awfully pricy).
On Sat, Jul 21, 2018 at 2:37 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> or, another way we might look at it is there is very little we can do
> as the cmts has to have data in it in order to burst schedule the mac
> for the next string of packets, much like how fq_codel for wifi has
> "one in the hardware, one ready to go", a cmts has at least one in the
> hardware (per channel? a multiple? what?).
>
> or I could be on drugs entirely. And this thread did start with
> fast.com misbehaving badly regardless of the shaper in place or not,
> which is not what I'm looking at now.  I need to setup a 45ms rtt
> test...
>
> anyway, as per your suggestion, the latency gets MUCH better with your
> hping3 idea running, which implies that we've  been fooled all along
> by the rrul test. On the other hand, I think this will hurt other
> cable modems on the same wire. On the gripping hand, I'm happier
> knowing that with a busier network, docsis cable, when shaped, gets
> better, and that I should junk my existing test cablemodem due to the
> persistent spikes I see.
>
> I wonder if it's the sent path or the return path shattering latency
> so well? I wonder if hping3 would count against your badwidth cap?
>
> going back to trying to figure out why fast.com is so gnarly
> On Sat, Jul 21, 2018 at 2:18 PM Arie <nospam@ariekanarie.nl> wrote:
> >
> > I had a similar issue with my previous cable modem, whatever I shaped to didn't matter, I still had long delays. I "fixed" it by continuously sending a stream of empty UDP packets upstream:
> >
> > hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 IP-OF-FIRST-OUTSIDE-CABLE-HOP-HERE
> >
> > On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> This is my "inbound trying to shape a cable connection" smoking gun.
> >> The delay curve is the same
> >> shaping the 110mbit cmts down to 85mbit OR 55mbit.
> >>
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
> >>
> >
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

hmmm... And wow, does this modem have difficulty with udp

Summary of ping test run 'hping3-cake-docsis' (at 2018-07-21 22:22:14.758201):

                           avg       median     # data pts
 Ping (ms) ICMP :        26.50        23.20 ms        2395
 Ping (ms) UDP  :         1.02        44.21 ms          47 # That's
like a bazillion percent packet loss and explains why irtt is having
such difficulty.

Thank you very much for clues as to what's being going south in my
lab. I had no idea this problem at this low layer was so common:

http://badmodems.com/Forum/app.php/badmodems

On Sat, Jul 21, 2018 at 3:13 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> wow. That is the best dslreports test result for cable I have ever had.
>
> with hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 96.120.89.153
>
> http://www.dslreports.com/speedtest/36209937
>
> Without:
>
> http://www.dslreports.com/speedtest/36210095
>
> You say a different cablemodem does better without hping3 running? which? :)
>
> Most of my production gear is based on an older arris modem (which is
> quite good), most of my test gear is a bunch of netgear (free) modems
> and service I got free from my time working for comcast.
>
> I haven't got around to springing for a docsis 3.1 modem yet (they are
> awfully pricy).
> On Sat, Jul 21, 2018 at 2:37 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > or, another way we might look at it is there is very little we can do
> > as the cmts has to have data in it in order to burst schedule the mac
> > for the next string of packets, much like how fq_codel for wifi has
> > "one in the hardware, one ready to go", a cmts has at least one in the
> > hardware (per channel? a multiple? what?).
> >
> > or I could be on drugs entirely. And this thread did start with
> > fast.com misbehaving badly regardless of the shaper in place or not,
> > which is not what I'm looking at now.  I need to setup a 45ms rtt
> > test...
> >
> > anyway, as per your suggestion, the latency gets MUCH better with your
> > hping3 idea running, which implies that we've  been fooled all along
> > by the rrul test. On the other hand, I think this will hurt other
> > cable modems on the same wire. On the gripping hand, I'm happier
> > knowing that with a busier network, docsis cable, when shaped, gets
> > better, and that I should junk my existing test cablemodem due to the
> > persistent spikes I see.
> >
> > I wonder if it's the sent path or the return path shattering latency
> > so well? I wonder if hping3 would count against your badwidth cap?
> >
> > going back to trying to figure out why fast.com is so gnarly
> > On Sat, Jul 21, 2018 at 2:18 PM Arie <nospam@ariekanarie.nl> wrote:
> > >
> > > I had a similar issue with my previous cable modem, whatever I shaped to didn't matter, I still had long delays. I "fixed" it by continuously sending a stream of empty UDP packets upstream:
> > >
> > > hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 IP-OF-FIRST-OUTSIDE-CABLE-HOP-HERE
> > >
> > > On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:
> > >>
> > >> This is my "inbound trying to shape a cable connection" smoking gun.
> > >> The delay curve is the same
> > >> shaping the 110mbit cmts down to 85mbit OR 55mbit.
> > >>
> > >> _______________________________________________
> > >> Cake mailing list
> > >> Cake@lists.bufferbloat.net
> > >> https://lists.bufferbloat.net/listinfo/cake
> > >>
> > >
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1: Type: text/plain, Size: 5455 bytes --]

 You're now the third person where I've seen the hping3 trick work, I'm
pretty pleased it's with Mr Bufferbloat himself ;)

I figured this out by accident some time ago. I used fq_codel and later
cake to keep people happy at a small LAN party where 10 people shared a
40Mbit DOCSIS connection. Year after year we kept renting the same vacation
homes, with the same old Cisco EPC3295 cable modems. We used to bring some
weird pfsense box with tons of custom game-based rules to keep latencies
under control, but we just replaced that with an openwrt box running
fq_codel one day, way simpler and better latencies (thank you and the other
bufferbloat people!)

Suddenly one year, my openwrt/cake box was no longer able to keep the
latency under control and people started complaining. I noticed that while
an upload was running, the latency was fine (despite someone hogging the
downstream with big downloads). As soon as the upload stopped, the big
download started to cause ping spikes again.
After some testing, I was able to use the hping3 trick to send the minimum
needed upstream traffic to keep pings low, LAN party saved.

Meanwhile on my home connection I used a similar DOCSIS modem. I'd always
been able to just shape my connection close to the advertised rates. One
day, latencies (and DSLReports bufferbloat score) got bad. Interestingly,
flent RRUL results reported lower latencies during the test run than during
the idle period before and after the test. Again I could use the same
hping3 trick to "fix" it. I've asked the bufferbloat mailing list to see if
anyone knew what was going on, but nothing came of it.
My ISP kept pushing new DOCSIS modems, so I took my chances despite it
using a puma 6 chipset (TG2492LG). This one is fine without the hping
trick, just like my old modem used to be.

Here's what I learned about some cable modems with my particular ISP
(Ziggo, the Netherlands) in my specific region:
Cisco EPC3212 (DOCSIS 3.0 8x4), used to work fine, now gets big latency
spikes regardless of the shaped rate.
Technicolor 7200 (DOCSIS 3.0 8x4), still works fine.
Arris  TG2492LG (DOCSIS 3.0 24x8), shaping works just fine, latency is
under control, but it has a puma 6 chip which causes latency spikes in TCP
and ICMP packets. UDP does not seem to be affected.


On 22 July 2018 at 00:13, Dave Taht <dave.taht@gmail.com> wrote:

> wow. That is the best dslreports test result for cable I have ever had.
>
> with hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 96.120.89.153
>
> http://www.dslreports.com/speedtest/36209937
>
> Without:
>
> http://www.dslreports.com/speedtest/36210095
>
> You say a different cablemodem does better without hping3 running? which?
> :)
>
> Most of my production gear is based on an older arris modem (which is
> quite good), most of my test gear is a bunch of netgear (free) modems
> and service I got free from my time working for comcast.
>
> I haven't got around to springing for a docsis 3.1 modem yet (they are
> awfully pricy).
> On Sat, Jul 21, 2018 at 2:37 PM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > or, another way we might look at it is there is very little we can do
> > as the cmts has to have data in it in order to burst schedule the mac
> > for the next string of packets, much like how fq_codel for wifi has
> > "one in the hardware, one ready to go", a cmts has at least one in the
> > hardware (per channel? a multiple? what?).
> >
> > or I could be on drugs entirely. And this thread did start with
> > fast.com misbehaving badly regardless of the shaper in place or not,
> > which is not what I'm looking at now.  I need to setup a 45ms rtt
> > test...
> >
> > anyway, as per your suggestion, the latency gets MUCH better with your
> > hping3 idea running, which implies that we've  been fooled all along
> > by the rrul test. On the other hand, I think this will hurt other
> > cable modems on the same wire. On the gripping hand, I'm happier
> > knowing that with a busier network, docsis cable, when shaped, gets
> > better, and that I should junk my existing test cablemodem due to the
> > persistent spikes I see.
> >
> > I wonder if it's the sent path or the return path shattering latency
> > so well? I wonder if hping3 would count against your badwidth cap?
> >
> > going back to trying to figure out why fast.com is so gnarly
> > On Sat, Jul 21, 2018 at 2:18 PM Arie <nospam@ariekanarie.nl> wrote:
> > >
> > > I had a similar issue with my previous cable modem, whatever I shaped
> to didn't matter, I still had long delays. I "fixed" it by continuously
> sending a stream of empty UDP packets upstream:
> > >
> > > hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 IP-OF-FIRST-OUTSIDE-CABLE-HOP-
> HERE
> > >
> > > On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:
> > >>
> > >> This is my "inbound trying to shape a cable connection" smoking gun.
> > >> The delay curve is the same
> > >> shaping the 110mbit cmts down to 85mbit OR 55mbit.
> > >>
> > >> _______________________________________________
> > >> Cake mailing list
> > >> Cake@lists.bufferbloat.net
> > >> https://lists.bufferbloat.net/listinfo/cake
> > >>
> > >
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 8910 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Ryan Mounce]

On 22 July 2018 at 08:40, Arie <nospam@ariekanarie.nl> wrote:
> You're now the third person where I've seen the hping3 trick work, I'm
> pretty pleased it's with Mr Bufferbloat himself ;)
>
> I figured this out by accident some time ago. I used fq_codel and later cake
> to keep people happy at a small LAN party where 10 people shared a 40Mbit
> DOCSIS connection. Year after year we kept renting the same vacation homes,
> with the same old Cisco EPC3295 cable modems. We used to bring some weird
> pfsense box with tons of custom game-based rules to keep latencies under
> control, but we just replaced that with an openwrt box running fq_codel one
> day, way simpler and better latencies (thank you and the other bufferbloat
> people!)
>
> Suddenly one year, my openwrt/cake box was no longer able to keep the
> latency under control and people started complaining. I noticed that while
> an upload was running, the latency was fine (despite someone hogging the
> downstream with big downloads). As soon as the upload stopped, the big
> download started to cause ping spikes again.
> After some testing, I was able to use the hping3 trick to send the minimum
> needed upstream traffic to keep pings low, LAN party saved.
>
> Meanwhile on my home connection I used a similar DOCSIS modem. I'd always
> been able to just shape my connection close to the advertised rates. One
> day, latencies (and DSLReports bufferbloat score) got bad. Interestingly,
> flent RRUL results reported lower latencies during the test run than during
> the idle period before and after the test. Again I could use the same hping3
> trick to "fix" it. I've asked the bufferbloat mailing list to see if anyone
> knew what was going on, but nothing came of it.
> My ISP kept pushing new DOCSIS modems, so I took my chances despite it using
> a puma 6 chipset (TG2492LG). This one is fine without the hping trick, just
> like my old modem used to be.
>
> Here's what I learned about some cable modems with my particular ISP (Ziggo,
> the Netherlands) in my specific region:
> Cisco EPC3212 (DOCSIS 3.0 8x4), used to work fine, now gets big latency
> spikes regardless of the shaped rate.
> Technicolor 7200 (DOCSIS 3.0 8x4), still works fine.
> Arris  TG2492LG (DOCSIS 3.0 24x8), shaping works just fine, latency is under
> control, but it has a puma 6 chip which causes latency spikes in TCP and
> ICMP packets. UDP does not seem to be affected.
>
>
> On 22 July 2018 at 00:13, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> wow. That is the best dslreports test result for cable I have ever had.
>>
>> with hping3 -2 -d 0 -s 10080 -k -p 80 -i u150 96.120.89.153
>>
>> http://www.dslreports.com/speedtest/36209937
>>
>> Without:
>>
>> http://www.dslreports.com/speedtest/36210095
>>
>> You say a different cablemodem does better without hping3 running? which?
>> :)
>>
>> Most of my production gear is based on an older arris modem (which is
>> quite good), most of my test gear is a bunch of netgear (free) modems
>> and service I got free from my time working for comcast.
>>
>> I haven't got around to springing for a docsis 3.1 modem yet (they are
>> awfully pricy).
>> On Sat, Jul 21, 2018 at 2:37 PM Dave Taht <dave.taht@gmail.com> wrote:
>> >
>> > or, another way we might look at it is there is very little we can do
>> > as the cmts has to have data in it in order to burst schedule the mac
>> > for the next string of packets, much like how fq_codel for wifi has
>> > "one in the hardware, one ready to go", a cmts has at least one in the
>> > hardware (per channel? a multiple? what?).
>> >
>> > or I could be on drugs entirely. And this thread did start with
>> > fast.com misbehaving badly regardless of the shaper in place or not,
>> > which is not what I'm looking at now.  I need to setup a 45ms rtt
>> > test...
>> >
>> > anyway, as per your suggestion, the latency gets MUCH better with your
>> > hping3 idea running, which implies that we've  been fooled all along
>> > by the rrul test. On the other hand, I think this will hurt other
>> > cable modems on the same wire. On the gripping hand, I'm happier
>> > knowing that with a busier network, docsis cable, when shaped, gets
>> > better, and that I should junk my existing test cablemodem due to the
>> > persistent spikes I see.
>> >
>> > I wonder if it's the sent path or the return path shattering latency
>> > so well? I wonder if hping3 would count against your badwidth cap?
>> >
>> > going back to trying to figure out why fast.com is so gnarly
>> > On Sat, Jul 21, 2018 at 2:18 PM Arie <nospam@ariekanarie.nl> wrote:
>> > >
>> > > I had a similar issue with my previous cable modem, whatever I shaped
>> > > to didn't matter, I still had long delays. I "fixed" it by continuously
>> > > sending a stream of empty UDP packets upstream:
>> > >
>> > > hping3 -2 -d 0 -s 10080 -k -p 80 -i u150
>> > > IP-OF-FIRST-OUTSIDE-CABLE-HOP-HERE
>> > >
>> > > On 21 July 2018 at 22:36, Dave Taht <dave.taht@gmail.com> wrote:
>> > >>
>> > >> This is my "inbound trying to shape a cable connection" smoking gun.
>> > >> The delay curve is the same
>> > >> shaping the 110mbit cmts down to 85mbit OR 55mbit.
>> > >>
>> > >> _______________________________________________
>> > >> Cake mailing list
>> > >> Cake@lists.bufferbloat.net
>> > >> https://lists.bufferbloat.net/listinfo/cake
>> > >>
>> > >
>> >
>> >
>> > --
>> >
>> > Dave Täht
>> > CEO, TekLibre, LLC
>> > http://www.teklibre.com
>> > Tel: 1-669-226-2619
>>
>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

I can independently confirm the "hping trick" for DOCSIS. In my case
this wasn't a latency concern, but an accidental workaround for
crippling packet loss on a remote connection. Also I used fping :)

Warning: history lesson / rant.

In Australia we had the very peculiar circumstance of the privatised
incumbent telco Telstra having overbuilt both its own copper telephone
network and its competitor's once threating HFC network (Optus) with
their own HFC to protect their landline telephone profits. Now two
decades later, the government has established a new monopoly wholesale
network operator (NBN Co) who after a subsequent change of governent
have effectively agreed to acquire all three of the above networks
from the existing operators.

Cable TV never gained much traction in Australia for a number of
reasons, mostly due to a healthy free to air broadcast industry, and
the rapid advance in satellite TV aroundthe same time that the HFC
networks were being rolled out. The monopoly pay TV retailer Foxtel
prefers to install a satellite dish than run a coax drop to access
their leased spectrum on each of the HFC networks. So only about 1/3
of premises in Australia are passed by HFC, only about 2/3 of those
are connected, and of those only about half had an active service of
any kind over the coax.

The proud new owners of two HFC networks and one copper network
decided that each area will get a different form of access depending
on the existing networks, the so called "multi-technology mix" / MTM.
The Optus HFC network was found to have very large segments, and given
that its coverage is almost completely duplicated by the other HFC
network, it has been abandoned. The Telstra HFC network mentioned
above with ~1/3 of covered premises having an active service is to
replace the copper telephone network where it exists. And where there
is only copper, most will get vectored VDSL2 deployed as FTTN. A lucky
few have an overbuilt GPON FTTP network from the previous government's
rollout (aborted due to cost except for new communities), and weird
gaps between these technology boundaries are now being plugged with
GPON-backhauled unvectored VDSL2 FTTC with reverse-powered DPUs. Oh,
and apartments get vectored VDSL2 FTTB. And in regional areas they've
built a TD-LTE network for fixed wireless. And they deployed two
geostationary bent-pipe satellites and a network of ground stations
for rural areas. Connections via this alphabet soup of technologies
are all supposed to be unified, wholesaled to retail ISPs as a single
product construct (except for satellite, and now due to capacity
constraints fixed wireless also has extra pricing asterisks). So this
is a project with unprecedented scope and complexity within Australia,
and is unsurprisingly running behind schedule and over budget.

Back to HFC/DOCSIS. For the most part, Australia conforms to European
standards which means a 65MHz upstream bandwidth split for HFC, much
more generous than the 42MHz split in North America. The incumbent
operator uses EuroDOCSIS 3.0, with 4 upstream channels fitting
comfortably in the top, clean part of the upstream spectrum. The new
operator taking over the network has installed their CMTS in parallel,
operating on the lower half of the usable upstream spectrum with, as I
recall, 3 upstream channels. The incumbent operator offers a maximum
speed tier of 120/5.5. The new operator offers 100/40, with less
upstream bandwidth available, occupying noisiest end of the spectrum.
They are also expanding the network to 100% coverage, installing drops
to existing premises on an opt-out basis so that they can decommission
the copper telephone network as soon as possible. And of course, they
are aggressively onboarding customers with migration agreements in
place as the government-sanctioned monopoly fixed line network
operator. In order to get the target speeds out of the small amount of
noisy spectrum, they are being very aggressive with modulation etc.

The problem is, they are moving too fast and tripping over themselves.
With all the new drops and new subscribers, the noise floor in the
upstream is rising rapidly. In the early stages of the rollout this
was managed acceptably, an audit was undertaken, modelling performed,
nodes split, the plant swept for noise ingress, all before releasing
the 'new' network to subscribers and around the same time that the
bulk of new drops were being added. Then the bean counters demanded
more revenue sooner, and they got greedy by rapidly releasing the
network in a large number of areas that were not foreen to require
much more work to connect the missing premises. Literally the only
work performed on the plant prior to releasing most of these areas was
to replace the existing node with a segmentable node (but not actually
segmented). Technicians were sent out to perform every installation,
even where the drop already existed and only a new modem was needed
(mostly to babysit IT issues provisioning the modems, as they are
using BSoD/MPLS L2VPN to deliver their wholesale-only service). This
scheme quickly caught up with them, and a brewing PR nightmare
culminated in the indefinite suspension of all new connections to the
network and now nearly a year later they have only re-released a small
amount of the network. They are back to plan A and properly auditing
and upgrading the plant *before* activating new subscribers on their
noise-sensitive spectrum. Once the upgrades are complete, the
incumbent operator has vacated, and DOCSIS 3.1 is enabled across the
footprint (they have been installing D3.1 modems and are using D3.1
capable CMTS chassis) they will be left with a very clean and
performant network after a routh transition.


TL;DR

D3.0 modem with upstream channel bonding, remote location without
ability to power cycle. A  noise ingress fault develops on the primary
upstream channel. The CMTS is aware of the fault and the network is
now operating in DOCSIS 3.0 Partial Service mode, however CMs
connected before the fault are still impacted. Very high (don't
remember, something like 20%) packet loss when the link is unloaded,
however under load packet loss almost disappears. Upstream data
requests are sent on the primary upstream channel, and are getting
lost. As the CMTS is aware of the fault, it is not issuing data grants
on the noisy channel so the actual packet transmission is successful
if it gets that far.

If there is remaining data in the CM's upstream queue when an upstream
packet is sent, a data grant "piggy-backs" the data to avoid a
separate grant. By sending a constrant stream of packets upstream on
an interval shorter than the request-grant cycle, all of these
requests will be piggy-backed with your dummy data on the clean
channel :)

I wish I had done some more measurements and more accurately qualified
the minimum possible packet sizes and longest interval to achieve this
request. I was also deprioritising the dummy packets with DSCP/cake so
there was no impact to my actual traffic, although of course the dummy
data is still going upstream over the coax which is the most
congestion-prone link as far as my operator is concerned. It would
also be nice to qualify the impact of this trick on RTT/jitter,
although for my case I don't really care so long as it's <10ms.
(reordering on upstream bonded DOCSIS 3.0 is an isssue, however).

Myself, I am now on a shiny new 100/40 FTTdp/C connection which is
flawless in terms of latency, loss, jitter, reordering. The network
operator / wholesaler polices traffic, my retailer shapes with a
somewhat reasonably sized FIFO in the downstream, and I shape ingress
w/ cake to 99% link capacity to keep steady-state latency under
control during downloads. Not much I can realistically do about
downstream latency spikes, unless Cisco have some FQ that I can
persuade my retailer to implement on their ASR9k in place of the FIFO.

Regards,
Ryan

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

Great info, thx. Using this opportunity to rant about city-wid
networks, I'd have done something so different
than what the governments and ISPs have inflicted on us, substituting
redundancy for reliability.

I'd have used bog standard ethernet over fiber instead of gpon. The
only advantages to gpon were that it was a standard normal folk
(still) can't use, it offered encryption to the co-lo, and the gpon
splitter at the neighborhood cabinett could be unpowered, and a
telco-like design could be made telco-level reliable.Theoretically. In
reality it constrains the market and raises the price of gpon capable
gear enormously, thus creating a cost for the isp and a healthy profit
margin for the fiber vendor.

Neighborhood cabinets would be cross connected north, east, west,
south, uplink1, uplink2, thus rendering the entire network immune to
fiber cuts and other disruptions of service and allowing competition
for service from multiple isps. Fiber or copper or wireless (cell) to
the building from there. Your neighbor would be one hop away. Local
cellular or wifi would spring out of smaller towers distributed above
those cabinets.

Lest you think I'm entirely insane, that's how amsterdam's network was
built out 10+ years ago.
https://arstechnica.com/tech-policy/2010/03/how-amsterdam-was-wired-for-open-access-fiber/

I'd have avoided MPLS, and gone with something like 64xlat to deal
with the ipv4 distribution problem. There'd be a secure routing
protocol holding the city-wide network together. And there'd be
ponies. Lots of ponies.

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Jonas Mårtensson]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

On Mon, Jul 23, 2018 at 4:56 PM Dave Taht <dave.taht@gmail.com> wrote:

> Great info, thx. Using this opportunity to rant about city-wid
> networks, I'd have done something so different
> than what the governments and ISPs have inflicted on us, substituting
> redundancy for reliability.
>
> I'd have used bog standard ethernet over fiber instead of gpon.
>

Well, in Sweden 100% of ftth connections are standard active ethernet
instead of gpon (and more than 60% of fixed connections are ftth).

[-- Attachment #2: Type: text/html, Size: 792 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

On Mon, Jul 23, 2018 at 8:26 AM Jonas Mårtensson
<martensson.jonas@gmail.com> wrote:
>
>
>
> On Mon, Jul 23, 2018 at 4:56 PM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Great info, thx. Using this opportunity to rant about city-wid
>> networks, I'd have done something so different
>> than what the governments and ISPs have inflicted on us, substituting
>> redundancy for reliability.
>>
>> I'd have used bog standard ethernet over fiber instead of gpon.
>
>
> Well, in Sweden 100% of ftth connections are standard active ethernet instead of gpon (and more than 60% of fixed connections are ftth).

The internet was amazing in sweden when I lived there. The winter
darkness wasn't. I think a goodly percentage of my continued
involvement in the bufferbloat project is due to my quest for *both*
good internet and sunshine.



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Mon, 23 Jul 2018 at 17:26 Jonas Mårtensson <martensson.jonas@gmail.com>
wrote:

> On Mon, Jul 23, 2018 at 4:56 PM Dave Taht <dave.taht@gmail.com> wrote:
>
>> Great info, thx. Using this opportunity to rant about city-wid
>> networks, I'd have done something so different
>> than what the governments and ISPs have inflicted on us, substituting
>> redundancy for reliability.
>>
>> I'd have used bog standard ethernet over fiber instead of gpon.
>>
>
> Well, in Sweden 100% of ftth connections are standard active ethernet
> instead of gpon (and more than 60% of fixed connections are ftth).
>

Here in South Africa, we have one provider (Vumatel) deploying AE, and
everyone else doing GPON :/

[-- Attachment #2: Type: text/html, Size: 1304 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

I wish vumatel the best of luck. One thing that still will help the
fiber networks is applying fq_* derived algoritms.

This is cake vs sonic (gpon) fiber in san francisco. They have a
totally reasonable (60ms) buffer in their ONT, cut to... well... the
result I get is even less latency loaded than unloaded:

http://www.taht.net/~d/sonic_106_cake_vs_default.png

In my mind fq techniques merely on the up on fiber onts will make a
huge difference in perceived QoE, and the marketing department can
keep claiming it's fiber bandwidth that counts to consumers.

FIOS uses actiontec. I haven't picked apart what kernels the use in a
couple years.

http://opensource.actiontec.com/

(hey verizon FIOS? gfiber? you listening?)
On Mon, Jul 23, 2018 at 9:45 AM Tristan Seligmann
<mithrandi@mithrandi.net> wrote:
>
> On Mon, 23 Jul 2018 at 17:26 Jonas Mårtensson <martensson.jonas@gmail.com> wrote:
>>
>> On Mon, Jul 23, 2018 at 4:56 PM Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> Great info, thx. Using this opportunity to rant about city-wid
>>> networks, I'd have done something so different
>>> than what the governments and ISPs have inflicted on us, substituting
>>> redundancy for reliability.
>>>
>>> I'd have used bog standard ethernet over fiber instead of gpon.
>>
>>
>> Well, in Sweden 100% of ftth connections are standard active ethernet instead of gpon (and more than 60% of fixed connections are ftth).
>
>
> Here in South Africa, we have one provider (Vumatel) deploying AE, and everyone else doing GPON :/



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Sebastian Moeller]

Hi Dave,

To follow you on this tangent ;)

I have come to the conclusion that the question about whether PtP is better or GPON heavily depends on who is paying.
For a local community like Amsterdam the goal probably was to be able to accommodate as many possible network topologies as possible so that the ISPs that were supposed rent the "dark fiber" can implement any topology they desire. And since it is trivial to use a passive splitter in the CO building to reduce PtP to PtMP, PtP is the way to go. 
For an (incumbent) ISP however, especially a heavily regulated one, the goal is slightly different: build the cheapest network allowing to sell the desired speed-grades while makeing sharing as hard as possible.
The cost of the build-out becomes more important (ISPs are typically publicly traded and have much shorter amortization horizons than a city government (I blame the short-sightedness of investors and the way management pay typically is linked to short-term stock performance, but I digress)) and I have seen numbers that show that GPON gear really is cheaper than AON gear for the same number of customers (it also is less capable, but that is beside the point).
But the kicker, at least in Germany seems to be that the incumbent really really wants to avoid to end in a situation where it can be forced to hand over physical last-mile links to the competition, exactly what is possible with a PtP set-up (the competitor puts its own fiber ethernet switch into the CO, and the ISP that build the network only gets a tiny fixed "rent" per month). 
Finally the GPON standards are made by ITU and hence by the telco community while ethernet is in the hands of the ieee and telco's have less standing there to get their wishes, but I am not sure how important that point really is.

My conclusion is, that it seems immensely sub-optimal to have incumbent ISPs do the local network build-out as what they will build is not ideally suited to what the community served actually wants. This conclusion is also backed by the number of state laws that have cropped up over time making community-initiated and financed network build-out as hard as possible, indicating that the telco lobbyists share my assessment about which network ownership (and topology) actually serves the end-customers best ;)



Best Regards
	Sebastian


> On Jul 23, 2018, at 16:56, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Great info, thx. Using this opportunity to rant about city-wid
> networks, I'd have done something so different
> than what the governments and ISPs have inflicted on us, substituting
> redundancy for reliability.
> 
> I'd have used bog standard ethernet over fiber instead of gpon. The
> only advantages to gpon were that it was a standard normal folk
> (still) can't use, it offered encryption to the co-lo, and the gpon
> splitter at the neighborhood cabinett could be unpowered, and a
> telco-like design could be made telco-level reliable.Theoretically. In
> reality it constrains the market and raises the price of gpon capable
> gear enormously, thus creating a cost for the isp and a healthy profit
> margin for the fiber vendor.
> 
> Neighborhood cabinets would be cross connected north, east, west,
> south, uplink1, uplink2, thus rendering the entire network immune to
> fiber cuts and other disruptions of service and allowing competition
> for service from multiple isps. Fiber or copper or wireless (cell) to
> the building from there. Your neighbor would be one hop away. Local
> cellular or wifi would spring out of smaller towers distributed above
> those cabinets.
> 
> Lest you think I'm entirely insane, that's how amsterdam's network was
> built out 10+ years ago.
> https://arstechnica.com/tech-policy/2010/03/how-amsterdam-was-wired-for-open-access-fiber/
> 
> I'd have avoided MPLS, and gone with something like 64xlat to deal
> with the ipv4 distribution problem. There'd be a secure routing
> protocol holding the city-wide network together. And there'd be
> ponies. Lots of ponies.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 3779 bytes --]

I don't know if this is possible for higher density cities, but the fiber
ISP here uses P2P fiber ring from the house all the way back to the CO.
It's only at the CO that they aggregate to the GPON port. This means I do
not share any field fiber with anyone else and the ring design allows for a
single fiber cut to not take out my Internet.

It seems " and allowing competition for service from multiple isps" is the
main point for your described setup. My current setup is fine for my single
ISP, but they don't have to share with anyone else. I have heard of
alternative setups where the CO was not owned by an ISP, but where all the
ISPs hooked into the fiber network. North West East South redundancy sound
overly redundant, but I guess I would not complain. I assume it means
powered equipment in the field, unless there's a way to passively do that
without dropping the signal strength. I would prefer a two point redundant
system that was passive over an active 4 way redundant system that could
have power failures, which are more common than fiber cuts around here. My
firewall has nearly 450 days of uptime, not many power outages either.

I am already one hop away from everyone in the city, including my employer.
The ISP uses a flat network model, everything plugs into the core. The core
router has many ports with a minimum rates of 100Gb. The GPON units plug
directly into the core, and they're only used as layer 2 devices. The GPON
units have 1 or more 100gb or 400gb uplinks. The network is provisioned to
be fully non-blocking. It can handle all customers at 100% of their
provisioned rates at the same time. Other than for redundancy, there's
little reason to have routing/forwarding being done in the field. A "hub"
pattern is fine and scales just fine, and less complex.

I'm not sure one hop away if useful in a multi-ISP shared network since
your packets need to go to your ISP in order to get routed back to your
neighbor.

On Mon, Jul 23, 2018 at 9:56 AM Dave Taht <dave.taht@gmail.com> wrote:

> Great info, thx. Using this opportunity to rant about city-wid
> networks, I'd have done something so different
> than what the governments and ISPs have inflicted on us, substituting
> redundancy for reliability.
>
> I'd have used bog standard ethernet over fiber instead of gpon. The
> only advantages to gpon were that it was a standard normal folk
> (still) can't use, it offered encryption to the co-lo, and the gpon
> splitter at the neighborhood cabinett could be unpowered, and a
> telco-like design could be made telco-level reliable.Theoretically. In
> reality it constrains the market and raises the price of gpon capable
> gear enormously, thus creating a cost for the isp and a healthy profit
> margin for the fiber vendor.
>
> Neighborhood cabinets would be cross connected north, east, west,
> south, uplink1, uplink2, thus rendering the entire network immune to
> fiber cuts and other disruptions of service and allowing competition
> for service from multiple isps. Fiber or copper or wireless (cell) to
> the building from there. Your neighbor would be one hop away. Local
> cellular or wifi would spring out of smaller towers distributed above
> those cabinets.
>
> Lest you think I'm entirely insane, that's how amsterdam's network was
> built out 10+ years ago.
>
> https://arstechnica.com/tech-policy/2010/03/how-amsterdam-was-wired-for-open-access-fiber/
>
> I'd have avoided MPLS, and gone with something like 64xlat to deal
> with the ipv4 distribution problem. There'd be a secure routing
> protocol holding the city-wide network together. And there'd be
> ponies. Lots of ponies.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 5517 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Ryan Mounce]

Unloaded latency is 5ms. My retailer has a 100Mbps FIFO shaper that
will queue up to 100ms worth of full MTU packets.

Without ingress shaping, loaded latency builds up to about 95ms and
then stays there for the remainder of the test.

With ingress shaped to 99Mbps with cake, fast.com loaded latency peaks
around 40ms, then quickly settles down to 8ms. On multiple runs of the
test, the worst latency peak I saw was 60ms.

So even shaping ingress close to link capacity is a huge win against a FIFO.
Regards,
Ryan Mounce

ryan@mounce.com.au
0415 799 929


On 24 July 2018 at 00:14, Dave Taht <dave.taht@gmail.com> wrote:
> what does the fast.com test do to you with and without inbound shaping
> on your link?

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Ryan Mounce]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

> On 24 July 2018 at 00:14, Dave Taht <dave.taht@gmail.com> wrote:
>> what does the fast.com test do to you with and without inbound shaping
>> on your link?

Sorry, brain dead today and didn't properly read your original email.
I have now got flent up and running on macOS, see attached unshaped +
80% ingress shaped results.

I will be able to get results in a couple of days time from an oddball
DOCSIS 3.0 connection. Despite the different last mile access
technology it is via the same wholesaler as my FTTdp connection. They
police traffic on their aggregation switches rather than shaping on
the CMTS, while the retailer shapes traffic on their BNG.

Ryan.

[-- Attachment #2: ping-2018-07-24T113209.008003.gpon-vdsl2-unshaped.png --]
[-- Type: image/png, Size: 50783 bytes --]

[-- Attachment #3: ping-2018-07-24T113504.189142.gpon-vdsl2-80pct.png --]
[-- Type: image/png, Size: 64906 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

On Mon, Jul 23, 2018 at 7:17 PM Ryan Mounce <ryan@mounce.com.au> wrote:
>
> > On 24 July 2018 at 00:14, Dave Taht <dave.taht@gmail.com> wrote:
> >> what does the fast.com test do to you with and without inbound shaping
> >> on your link?
>
> Sorry, brain dead today and didn't properly read your original email.
> I have now got flent up and running on macOS, see attached unshaped +
> 80% ingress shaped results.

Nice! thx. Do you remember what fast.com reported for these two tests?

I think it's P75 latency. P98 would be better.

Also, do you know of a good hosting facility we could use to dump a
flent server in australia?

> I will be able to get results in a couple of days time from an oddball
> DOCSIS 3.0 connection. Despite the different last mile access
> technology it is via the same wholesaler as my FTTdp connection. They
> police traffic on their aggregation switches rather than shaping on
> the CMTS, while the retailer shapes traffic on their BNG.
>
> Ryan.



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Ryan Mounce]

[-- Attachment #1: Type: text/plain, Size: 2064 bytes --]

On 24 July 2018 at 11:59, Dave Taht <dave.taht@gmail.com> wrote:
> On Mon, Jul 23, 2018 at 7:17 PM Ryan Mounce <ryan@mounce.com.au> wrote:
>>
>> > On 24 July 2018 at 00:14, Dave Taht <dave.taht@gmail.com> wrote:
>> >> what does the fast.com test do to you with and without inbound shaping
>> >> on your link?
>>
>> Sorry, brain dead today and didn't properly read your original email.
>> I have now got flent up and running on macOS, see attached unshaped +
>> 80% ingress shaped results.
>
> Nice! thx. Do you remember what fast.com reported for these two tests?

Unloaded 5ms. 80% shaped was ~8ms loaded throughout the whole test.
Unshaped was ~95ms.

Also attached another test for this connection, with my usual
configuration shaping ingress at 99% link capacity.

>
> I think it's P75 latency. P98 would be better.
>
> Also, do you know of a good hosting facility we could use to dump a
> flent server in australia?

Hosting and bandwidth remain relatively expensive in Australia,
although things have improved as the cost of, and dependency upon
international traffic has dropped. This does not change the fact that
Australia is an expensive country in general. Sydney still remains the
centre of the Australian internet in terms of peering and
international connectivity.

Personally I have have had nothing but good experience with Ransom IT
for virtual servers. They do not run their own network, but they have
good upstreams in each location.
https://www.ransomit.com.au/australian-vps/

Vultr is a cheaper option, can't vouch for them though.

>
>> I will be able to get results in a couple of days time from an oddball
>> DOCSIS 3.0 connection. Despite the different last mile access
>> technology it is via the same wholesaler as my FTTdp connection. They
>> police traffic on their aggregation switches rather than shaping on
>> the CMTS, while the retailer shapes traffic on their BNG.
>>
>> Ryan.
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619

[-- Attachment #2: ping-2018-07-24T114804.289079.gpon-vdsl2-99pct.png --]
[-- Type: image/png, Size: 58050 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 603 bytes --]



> On 24 Jul 2018, at 03:50, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
<snip>

Just ‘cos everyone else is doing it *and* to prove I’ve got flent running after installing homebrew python3 ……
Result of Dave’s test with in & out shaped at 99% VDSL 80/20 (UK, Sky using incumbent BT Openreach infra)

Fast.com reported Latency 6ms unloaded 13ms loaded. 72Mbit down, 18Mbit up.  The really ‘fun’ bit is that they report servers as London, Leeds & tadaaa Basingstoke where I actually live.  Where?!!!!!


Cheers,

Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

[-- Attachment #1.2.1: Type: text/html, Size: 1300 bytes --]

[-- Attachment #1.2.2: ping_-_Basingstoke99.png --]
[-- Type: image/png, Size: 105831 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1.1.1: Type: text/plain, Size: 962 bytes --]

Now that you can join the party, I note there IS a flent server in england
with irtt on it.

*flent-*london.bufferbloat.net



On Tue, Jul 24, 2018 at 1:15 AM Kevin Darbyshire-Bryant <
kevin@darbyshire-bryant.me.uk> wrote:

>
>
> On 24 Jul 2018, at 03:50, Ryan Mounce <ryan@mounce.com.au> wrote:
>
> <snip>
>
> Just ‘cos everyone else is doing it *and* to prove I’ve got flent running
> after installing homebrew python3 ……
> Result of Dave’s test with in & out shaped at 99% VDSL 80/20 (UK, Sky
> using incumbent BT Openreach infra)
>
> Fast.com reported Latency 6ms unloaded 13ms loaded. 72Mbit down, 18Mbit
> up.  The really ‘fun’ bit is that they report servers as London, Leeds &
> tadaaa Basingstoke where I actually live.  Where?!!!!!
>
>
> Cheers,
>
> Kevin D-B
>
> 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #1.1.2: Type: text/html, Size: 2349 bytes --]

[-- Attachment #1.2: ping_-_Basingstoke99.png --]
[-- Type: image/png, Size: 105831 bytes --]

[-- Attachment #2: ping_-_Basingstoke99.png --]
[-- Type: image/png, Size: 105831 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Kevin Darbyshire-Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 780 bytes --]



> On 24 Jul 2018, at 14:51, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Now that you can join the party, I note there IS a flent server in england with irtt on it.
> 
> flent-london.bufferbloat.net

Ok, well if you’re desperately interested….. :-)

Plot I did running to the london server, one shaped at 99% downstream, the other at 98%…. and I’ve decided to stick at 98% as a result - plot is a little bit hard to distinguish but I’d say there’s around 8ms avg less latency-ish between the two.  And an order of magnitude between using london v fremont :-)

The upstream barely registers

# tc -s qdisc show dev ifb4eth0
qdisc cake 801a: root refcnt 2 bandwidth 78400Kbit diffserv3 dual-dsthost nat ingress split-gso rtt 100.0ms ptm overhead 26



[-- Attachment #1.2.1: Type: text/html, Size: 1533 bytes --]

[-- Attachment #1.2.2: ping-basingstoke-london-98v99.png --]
[-- Type: image/png, Size: 127599 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1.1.1: Type: text/plain, Size: 1919 bytes --]

cdfs help. I also try to encourage sending the flent.gz files too. :)

Outbound, to 99% or more of the rate, *with perfect framing* looks great so
far, 'cept on crappy cablemodems.

I am concerned about recommending values as high as 98% for inbound
shaping. We are engineering to
the test here (2? 3 flows? on a very short rtt), and need to leave *some*
headroom for multiple flows to enter in slow start and get kicked out of it.

I'll buy that the old 85% figure made sense in the sub 20Mbit era, and that
we only need enough headroom to allow X flows to enter based on the
characteristics of the link and typical traffic - 15 new flows per second *
per active user/10 ? and cake's response to slow start is more agressive...

try a:

flent -H flent-london.bufferbloat.net -s .02 --te=download_streams=32 -t
'98%' tcp_ndown

with htb+fq_codel and cake to see that spike more clearly.


On Tue, Jul 24, 2018 at 7:54 AM Kevin Darbyshire-Bryant <
kevin@darbyshire-bryant.me.uk> wrote:

>
>
> On 24 Jul 2018, at 14:51, Dave Taht <dave.taht@gmail.com> wrote:
>
> Now that you can join the party, I note there IS a flent server in england
> with irtt on it.
>
> flent-london.bufferbloat.net
>
>
> Ok, well if you’re desperately interested….. :-)
>
> Plot I did running to the london server, one shaped at 99% downstream, the
> other at 98%…. and I’ve decided to stick at 98% as a result - plot is a
> little bit hard to distinguish but I’d say there’s around 8ms avg less
> latency-ish between the two.  And an order of magnitude between using
> london v fremont :-)
>
> The upstream barely registers
>
> # tc -s qdisc show dev ifb4eth0
> qdisc cake 801a: root refcnt 2 bandwidth 78400Kbit diffserv3 dual-dsthost
> nat ingress split-gso rtt 100.0ms ptm overhead 26
>
>
>

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #1.1.2: Type: text/html, Size: 3015 bytes --]

[-- Attachment #1.2: ping-basingstoke-london-98v99.png --]
[-- Type: image/png, Size: 127599 bytes --]

[-- Attachment #2: ping-basingstoke-london-98v99.png --]
[-- Type: image/png, Size: 127599 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Tristan Seligmann]

[-- Attachment #1.1: Type: text/plain, Size: 2801 bytes --]

Here's my results from South Africa. Ping test while running the fast.com
test, and the tcp_ndown test, running Cake on an EdgeRouter X. I get 140 /
140 Mbps on the fast.com test with 1ms unloaded and 6ms loaded.

Not sure why throughput is so low, but I often see something like this on
artificial bandwidth tests; it peaks close to 200, but then drops back down
toward the end of the test. I'm shaping to 200 / 200 Mbps with cake which
is my sold connection speed; the actual link is 1 Gbps active ethernet over
fibre with the ISP limiting to about 210-220 Mbps.

On Tue, 24 Jul 2018 at 17:19 Dave Taht <dave.taht@gmail.com> wrote:

> cdfs help. I also try to encourage sending the flent.gz files too. :)
>
> Outbound, to 99% or more of the rate, *with perfect framing* looks great
> so far, 'cept on crappy cablemodems.
>
> I am concerned about recommending values as high as 98% for inbound
> shaping. We are engineering to
> the test here (2? 3 flows? on a very short rtt), and need to leave *some*
> headroom for multiple flows to enter in slow start and get kicked out of it.
>
> I'll buy that the old 85% figure made sense in the sub 20Mbit era, and
> that we only need enough headroom to allow X flows to enter based on the
> characteristics of the link and typical traffic - 15 new flows per second
> * per active user/10 ? and cake's response to slow start is more
> agressive...
>
> try a:
>
> flent -H flent-london.bufferbloat.net -s .02 --te=download_streams=32 -t
> '98%' tcp_ndown
>
> with htb+fq_codel and cake to see that spike more clearly.
>
>
> On Tue, Jul 24, 2018 at 7:54 AM Kevin Darbyshire-Bryant <
> kevin@darbyshire-bryant.me.uk> wrote:
>
>>
>>
>> On 24 Jul 2018, at 14:51, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Now that you can join the party, I note there IS a flent server in
>> england with irtt on it.
>>
>> flent-london.bufferbloat.net
>>
>>
>> Ok, well if you’re desperately interested….. :-)
>>
>> Plot I did running to the london server, one shaped at 99% downstream,
>> the other at 98%…. and I’ve decided to stick at 98% as a result - plot is a
>> little bit hard to distinguish but I’d say there’s around 8ms avg less
>> latency-ish between the two.  And an order of magnitude between using
>> london v fremont :-)
>>
>> The upstream barely registers
>>
>> # tc -s qdisc show dev ifb4eth0
>> qdisc cake 801a: root refcnt 2 bandwidth 78400Kbit diffserv3 dual-dsthost
>> nat ingress split-gso rtt 100.0ms ptm overhead 26
>>
>>
>>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #1.2: Type: text/html, Size: 4298 bytes --]

[-- Attachment #2: ping-2018-07-24T195552.487731.mithrandi_net.flent.gz --]
[-- Type: application/gzip, Size: 27247 bytes --]

[-- Attachment #3: tcp_ndown-2018-07-24T200049.454211.98_.flent.gz --]
[-- Type: application/gzip, Size: 1246335 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Tristan Seligmann]

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]

I should note that I think the ER-X is running out of CPU at times in this
configuration, so that may contribute to the weirdness somewhat.

On Tue, 24 Jul 2018 at 20:05 Tristan Seligmann <mithrandi@mithrandi.net>
wrote:

> Here's my results from South Africa. Ping test while running the fast.com
> test, and the tcp_ndown test, running Cake on an EdgeRouter X. I get 140 /
> 140 Mbps on the fast.com test with 1ms unloaded and 6ms loaded.
>
> Not sure why throughput is so low, but I often see something like this on
> artificial bandwidth tests; it peaks close to 200, but then drops back down
> toward the end of the test. I'm shaping to 200 / 200 Mbps with cake which
> is my sold connection speed; the actual link is 1 Gbps active ethernet over
> fibre with the ISP limiting to about 210-220 Mbps.
>
>
> On Tue, 24 Jul 2018 at 17:19 Dave Taht <dave.taht@gmail.com> wrote:
>
>> cdfs help. I also try to encourage sending the flent.gz files too. :)
>>
>> Outbound, to 99% or more of the rate, *with perfect framing* looks great
>> so far, 'cept on crappy cablemodems.
>>
>> I am concerned about recommending values as high as 98% for inbound
>> shaping. We are engineering to
>> the test here (2? 3 flows? on a very short rtt), and need to leave *some*
>> headroom for multiple flows to enter in slow start and get kicked out of it.
>>
>> I'll buy that the old 85% figure made sense in the sub 20Mbit era, and
>> that we only need enough headroom to allow X flows to enter based on the
>> characteristics of the link and typical traffic - 15 new flows per second
>> * per active user/10 ? and cake's response to slow start is more
>> agressive...
>>
>> try a:
>>
>> flent -H flent-london.bufferbloat.net -s .02 --te=download_streams=32 -t
>> '98%' tcp_ndown
>>
>> with htb+fq_codel and cake to see that spike more clearly.
>>
>>
>> On Tue, Jul 24, 2018 at 7:54 AM Kevin Darbyshire-Bryant <
>> kevin@darbyshire-bryant.me.uk> wrote:
>>
>>>
>>>
>>> On 24 Jul 2018, at 14:51, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> Now that you can join the party, I note there IS a flent server in
>>> england with irtt on it.
>>>
>>> flent-london.bufferbloat.net
>>>
>>>
>>> Ok, well if you’re desperately interested….. :-)
>>>
>>> Plot I did running to the london server, one shaped at 99% downstream,
>>> the other at 98%…. and I’ve decided to stick at 98% as a result - plot is a
>>> little bit hard to distinguish but I’d say there’s around 8ms avg less
>>> latency-ish between the two.  And an order of magnitude between using
>>> london v fremont :-)
>>>
>>> The upstream barely registers
>>>
>>> # tc -s qdisc show dev ifb4eth0
>>> qdisc cake 801a: root refcnt 2 bandwidth 78400Kbit diffserv3
>>> dual-dsthost nat ingress split-gso rtt 100.0ms ptm overhead 26
>>>
>>>
>>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>

[-- Attachment #2: Type: text/html, Size: 4885 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1112 bytes --]

Hi Kevin, 

Maybe don't use the ptm keyword, but rather reduce the shaper rates to at most 98.46 percent of the sync rate...

On July 24, 2018 4:54:01 PM GMT+02:00, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
>
>
>> On 24 Jul 2018, at 14:51, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> Now that you can join the party, I note there IS a flent server in
>england with irtt on it.
>> 
>> flent-london.bufferbloat.net
>
>Ok, well if you’re desperately interested….. :-)
>
>Plot I did running to the london server, one shaped at 99% downstream,
>the other at 98%…. and I’ve decided to stick at 98% as a result - plot
>is a little bit hard to distinguish but I’d say there’s around 8ms avg
>less latency-ish between the two.  And an order of magnitude between
>using london v fremont :-)
>
>The upstream barely registers
>
># tc -s qdisc show dev ifb4eth0
>qdisc cake 801a: root refcnt 2 bandwidth 78400Kbit diffserv3
>dual-dsthost nat ingress split-gso rtt 100.0ms ptm overhead 26

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 2057 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]


> On Jul 24, 2018, at 3:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Now that you can join the party, I note there IS a flent server in england with irtt on it.
> 
> flent-london.bufferbloat.net <http://london.bufferbloat.net/>

Hmm, anything I can look at on the London server?

$ irtt client flent-london.bufferbloat.net
[Connecting] connecting to flent-london.bufferbloat.net
Error: read udp4 a.b.c.d:59071->176.58.107.8:2112: read: connection refused

$ irtt client flent-fremont.bufferbloat.net
[Connecting] connecting to flent-fremont.bufferbloat.net
[23.239.20.41:2112] [Connected] connection established
seq=0 rtt=159.2ms rd=80.83ms sd=78.36ms ipdv=n/a
^Cinterrupt


[-- Attachment #2: Type: text/html, Size: 2151 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

It died for some reason. restarted, logging now.
On Tue, Jul 24, 2018 at 12:38 PM Pete Heist <pete@heistp.net> wrote:
>
>
> On Jul 24, 2018, at 3:51 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> Now that you can join the party, I note there IS a flent server in england with irtt on it.
>
> flent-london.bufferbloat.net
>
>
> Hmm, anything I can look at on the London server?
>
> $ irtt client flent-london.bufferbloat.net
> [Connecting] connecting to flent-london.bufferbloat.net
> Error: read udp4 a.b.c.d:59071->176.58.107.8:2112: read: connection refused
>
> $ irtt client flent-fremont.bufferbloat.net
> [Connecting] connecting to flent-fremont.bufferbloat.net
> [23.239.20.41:2112] [Connected] connection established
> seq=0 rtt=159.2ms rd=80.83ms sd=78.36ms ipdv=n/a
> ^Cinterrupt
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]


> On Jul 24, 2018, at 10:44 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> It died for some reason. restarted, logging now.

Ok, when it happens again, also useful would be output from ‘irtt version’, whether it came from source or the Debian package, and any args the server is run with. Maybe submit an issue to reduce noise on this thread (https://github.com/heistp/irtt/issues <https://github.com/heistp/irtt/issues>). Thanks much!

Also presenting the plots no-one asked for or probably needs, fast.net <http://fast.net/> vs 3km NLOS point-to-point WiFi, both without shaping and with Cake using a queueing strategy I’ll document later. Effectively for this test egress and ingress are both rate limited to around 80-85%.

https://www.drhleny.cz/bufferbloat/fast_vs_p2pwifi.tar <https://www.drhleny.cz/bufferbloat/fast_vs_p2pwifi.tar>




[-- Attachment #2.1: Type: text/html, Size: 1858 bytes --]

[-- Attachment #2.2: freenet_no_shaping.png --]
[-- Type: image/png, Size: 73797 bytes --]

[-- Attachment #2.3: freenet_cake_hdd_25mbit.png --]
[-- Type: image/png, Size: 87232 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

[-- Attachment #1.1.1: Type: text/plain, Size: 1028 bytes --]

On Tue, Jul 24, 2018 at 3:23 PM Pete Heist <pete@heistp.net> wrote:

>
> On Jul 24, 2018, at 10:44 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> It died for some reason. restarted, logging now.
>
>
> Ok, when it happens again, also useful would be output from ‘irtt
> version’, whether it came from source or the Debian package, and any args
> the server is run with. Maybe submit an issue to reduce noise on this
> thread (https://github.com/heistp/irtt/issues). Thanks much!
>
> Also presenting the plots no-one asked for or probably needs, fast.net vs
> 3km NLOS point-to-point WiFi, both without shaping and with Cake using a
> queueing strategy I’ll document later. Effectively for this test egress and
> ingress are both rate limited to around 80-85%.
>

Heh. Feeling the love here... Our usual 10x+ reduction in latency. :yawn:.
:)


>
> https://www.drhleny.cz/bufferbloat/fast_vs_p2pwifi.tar
>
>
>

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[-- Attachment #1.1.2: Type: text/html, Size: 2446 bytes --]

[-- Attachment #1.2: freenet_no_shaping.png --]
[-- Type: image/png, Size: 73797 bytes --]

[-- Attachment #1.3: freenet_cake_hdd_25mbit.png --]
[-- Type: image/png, Size: 87232 bytes --]

[-- Attachment #2: freenet_no_shaping.png --]
[-- Type: image/png, Size: 73797 bytes --]

[-- Attachment #3: freenet_cake_hdd_25mbit.png --]
[-- Type: image/png, Size: 87232 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]


> On Jul 25, 2018, at 12:29 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Tue, Jul 24, 2018 at 3:23 PM Pete Heist <pete@heistp.net <mailto:pete@heistp.net>> wrote:
> 
> Also presenting the plots no-one asked for or probably needs, fast.net <http://fast.net/> vs 3km NLOS point-to-point WiFi, both without shaping and with Cake using a queueing strategy I’ll document later. Effectively for this test egress and ingress are both rate limited to around 80-85%.
> 
> Heh. Feeling the love here... Our usual 10x+ reduction in latency. :yawn:. :)

Yep, it’s dramatic. Thanks again Cake. Well, if nothing else, we might generate funding selling a textbook full of examples of "order of magnitude"… ;)


[-- Attachment #2: Type: text/html, Size: 1789 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1.1: Type: text/plain, Size: 1176 bytes --]

Two more data points. Shaped my connection to 250Mbit out of the advertised
250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This
is a pre-linux-net-next cake running on an Edgemax ER4 with
kernel 3.10.107-UBNT.

The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the
cable modem. UDP is not affected.

On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:

> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >
> > 1) Can someone else on a cablemodem (even without the latest cake,
> > this happens to me on older cake and fq_codel) try this test?
> >
>
> I just tried this on my cable comcast connection. I set ingress to ~80%
> of what fast.com reports when no shaper is in place.
>
> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> dsthost docsis ingress
> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> srchost nat docsis ack-filter
>
> I got the same result as you. This is using latest cake.
>
> Georgios
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>

[-- Attachment #1.2: Type: text/html, Size: 1869 bytes --]

[-- Attachment #2: fast-com-shaped-rate-200-of-250.png --]
[-- Type: image/png, Size: 122500 bytes --]

[-- Attachment #3: fast-com-shaped-rate-250-of-250.png --]
[-- Type: image/png, Size: 128466 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

Yours is not as horrific as mine in either case.

Can you provide an unshaped result as well?


On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
>
> Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
>
> The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
>
> On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
>>
>> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
>> >
>> > 1) Can someone else on a cablemodem (even without the latest cake,
>> > this happens to me on older cake and fq_codel) try this test?
>> >
>>
>> I just tried this on my cable comcast connection. I set ingress to ~80%
>> of what fast.com reports when no shaper is in place.
>>
>> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
>> dsthost docsis ingress
>> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
>> srchost nat docsis ack-filter
>>
>> I got the same result as you. This is using latest cake.
>>
>> Georgios
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1.1: Type: text/plain, Size: 1855 bytes --]

Unshaped reno attached.

On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:

> Yours is not as horrific as mine in either case.
>
> Can you provide an unshaped result as well?
>
>
> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> >
> > Two more data points. Shaped my connection to 250Mbit out of the
> advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the
> 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with
> kernel 3.10.107-UBNT.
> >
> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in
> the cable modem. UDP is not affected.
> >
> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
> >>
> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >> >
> >> > 1) Can someone else on a cablemodem (even without the latest cake,
> >> > this happens to me on older cake and fq_codel) try this test?
> >> >
> >>
> >> I just tried this on my cable comcast connection. I set ingress to ~80%
> >> of what fast.com reports when no shaper is in place.
> >>
> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> >> dsthost docsis ingress
> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> >> srchost nat docsis ack-filter
> >>
> >> I got the same result as you. This is using latest cake.
> >>
> >> Georgios
> >>
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
> >>
> >
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #1.2: Type: text/html, Size: 3010 bytes --]

[-- Attachment #2: unshaped_reno.png --]
[-- Type: image/png, Size: 116463 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

Thank you. At least for your ISP (?), at 250mbit, you can hold the
damage down to something reasonable, and certainly you are winning big
on the upload.

Good to know the ER4 can keep up, also.
On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
>
> Unshaped reno attached.
>
> On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Yours is not as horrific as mine in either case.
>>
>> Can you provide an unshaped result as well?
>>
>>
>> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
>> >
>> > Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
>> >
>> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
>> >
>> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
>> >>
>> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
>> >> >
>> >> > 1) Can someone else on a cablemodem (even without the latest cake,
>> >> > this happens to me on older cake and fq_codel) try this test?
>> >> >
>> >>
>> >> I just tried this on my cable comcast connection. I set ingress to ~80%
>> >> of what fast.com reports when no shaper is in place.
>> >>
>> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
>> >> dsthost docsis ingress
>> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
>> >> srchost nat docsis ack-filter
>> >>
>> >> I got the same result as you. This is using latest cake.
>> >>
>> >> Georgios
>> >>
>> >> _______________________________________________
>> >> Cake mailing list
>> >> Cake@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/cake
>> >>
>> >
>> > _______________________________________________
>> > Cake mailing list
>> > Cake@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cake
>>
>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1.1: Type: text/plain, Size: 2775 bytes --]

It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in the
fast.com test. Both the site and the flent ping test report about the same
60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so I
guess those stress my connection more than fast.com


On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:

> Thank you. At least for your ISP (?), at 250mbit, you can hold the
> damage down to something reasonable, and certainly you are winning big
> on the upload.
>
> Good to know the ER4 can keep up, also.
> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
> >
> > Unshaped reno attached.
> >
> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> Yours is not as horrific as mine in either case.
> >>
> >> Can you provide an unshaped result as well?
> >>
> >>
> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >
> >> > Two more data points. Shaped my connection to 250Mbit out of the
> advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the
> 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with
> kernel 3.10.107-UBNT.
> >> >
> >> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset
> in the cable modem. UDP is not affected.
> >> >
> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com>
> wrote:
> >> >>
> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >> >> >
> >> >> > 1) Can someone else on a cablemodem (even without the latest cake,
> >> >> > this happens to me on older cake and fq_codel) try this test?
> >> >> >
> >> >>
> >> >> I just tried this on my cable comcast connection. I set ingress to
> ~80%
> >> >> of what fast.com reports when no shaper is in place.
> >> >>
> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit
> dual-
> >> >> dsthost docsis ingress
> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> >> >> srchost nat docsis ack-filter
> >> >>
> >> >> I got the same result as you. This is using latest cake.
> >> >>
> >> >> Georgios
> >> >>
> >> >> _______________________________________________
> >> >> Cake mailing list
> >> >> Cake@lists.bufferbloat.net
> >> >> https://lists.bufferbloat.net/listinfo/cake
> >> >>
> >> >
> >> > _______________________________________________
> >> > Cake mailing list
> >> > Cake@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/cake
> >>
> >>
> >>
> >> --
> >>
> >> Dave Täht
> >> CEO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-669-226-2619
> >
> >
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #1.2: Type: text/html, Size: 4523 bytes --]

[-- Attachment #2: ziggo_shaped_245M.jpg --]
[-- Type: image/jpeg, Size: 149010 bytes --]

[-- Attachment #3: ziggo_unshaped.jpg --]
[-- Type: image/jpeg, Size: 150723 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

In other news:

https://github.com/tohojo/sqm-scripts/issues/68

does the ER4 have act_police?

On Sat, Jul 21, 2018 at 10:55 AM Arie <nospam@ariekanarie.nl> wrote:
>
> It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in the fast.com test. Both the site and the flent ping test report about the same 60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so I guess those stress my connection more than fast.com
>
>
> On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Thank you. At least for your ISP (?), at 250mbit, you can hold the
>> damage down to something reasonable, and certainly you are winning big
>> on the upload.
>>
>> Good to know the ER4 can keep up, also.
>> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
>> >
>> > Unshaped reno attached.
>> >
>> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
>> >>
>> >> Yours is not as horrific as mine in either case.
>> >>
>> >> Can you provide an unshaped result as well?
>> >>
>> >>
>> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
>> >> >
>> >> > Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
>> >> >
>> >> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
>> >> >
>> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
>> >> >>
>> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
>> >> >> >
>> >> >> > 1) Can someone else on a cablemodem (even without the latest cake,
>> >> >> > this happens to me on older cake and fq_codel) try this test?
>> >> >> >
>> >> >>
>> >> >> I just tried this on my cable comcast connection. I set ingress to ~80%
>> >> >> of what fast.com reports when no shaper is in place.
>> >> >>
>> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
>> >> >> dsthost docsis ingress
>> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
>> >> >> srchost nat docsis ack-filter
>> >> >>
>> >> >> I got the same result as you. This is using latest cake.
>> >> >>
>> >> >> Georgios
>> >> >>
>> >> >> _______________________________________________
>> >> >> Cake mailing list
>> >> >> Cake@lists.bufferbloat.net
>> >> >> https://lists.bufferbloat.net/listinfo/cake
>> >> >>
>> >> >
>> >> > _______________________________________________
>> >> > Cake mailing list
>> >> > Cake@lists.bufferbloat.net
>> >> > https://lists.bufferbloat.net/listinfo/cake
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Dave Täht
>> >> CEO, TekLibre, LLC
>> >> http://www.teklibre.com
>> >> Tel: 1-669-226-2619
>> >
>> >
>>
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>
>


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Arie]

[-- Attachment #1: Type: text/plain, Size: 3397 bytes --]

It does.

On 21 July 2018 at 20:02, Dave Taht <dave.taht@gmail.com> wrote:

> In other news:
>
> https://github.com/tohojo/sqm-scripts/issues/68
>
> does the ER4 have act_police?
>
> On Sat, Jul 21, 2018 at 10:55 AM Arie <nospam@ariekanarie.nl> wrote:
> >
> > It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in
> the fast.com test. Both the site and the flent ping test report about the
> same 60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so
> I guess those stress my connection more than fast.com
> >
> >
> > On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> Thank you. At least for your ISP (?), at 250mbit, you can hold the
> >> damage down to something reasonable, and certainly you are winning big
> >> on the upload.
> >>
> >> Good to know the ER4 can keep up, also.
> >> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >
> >> > Unshaped reno attached.
> >> >
> >> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
> >> >>
> >> >> Yours is not as horrific as mine in either case.
> >> >>
> >> >> Can you provide an unshaped result as well?
> >> >>
> >> >>
> >> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >> >
> >> >> > Two more data points. Shaped my connection to 250Mbit out of the
> advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the
> 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with
> kernel 3.10.107-UBNT.
> >> >> >
> >> >> > The regular spikes in ICMP ping is due to the crappy Puma 6
> chipset in the cable modem. UDP is not affected.
> >> >> >
> >> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com>
> wrote:
> >> >> >>
> >> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >> >> >> >
> >> >> >> > 1) Can someone else on a cablemodem (even without the latest
> cake,
> >> >> >> > this happens to me on older cake and fq_codel) try this test?
> >> >> >> >
> >> >> >>
> >> >> >> I just tried this on my cable comcast connection. I set ingress
> to ~80%
> >> >> >> of what fast.com reports when no shaper is in place.
> >> >> >>
> >> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit
> dual-
> >> >> >> dsthost docsis ingress
> >> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit
> dual-
> >> >> >> srchost nat docsis ack-filter
> >> >> >>
> >> >> >> I got the same result as you. This is using latest cake.
> >> >> >>
> >> >> >> Georgios
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> Cake mailing list
> >> >> >> Cake@lists.bufferbloat.net
> >> >> >> https://lists.bufferbloat.net/listinfo/cake
> >> >> >>
> >> >> >
> >> >> > _______________________________________________
> >> >> > Cake mailing list
> >> >> > Cake@lists.bufferbloat.net
> >> >> > https://lists.bufferbloat.net/listinfo/cake
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Dave Täht
> >> >> CEO, TekLibre, LLC
> >> >> http://www.teklibre.com
> >> >> Tel: 1-669-226-2619
> >> >
> >> >
> >>
> >>
> >> --
> >>
> >> Dave Täht
> >> CEO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-669-226-2619
> >
> >
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
>

[-- Attachment #2: Type: text/html, Size: 5954 bytes --]

[Cake] policers, finally

[Dave Taht]

On Sat, Jul 21, 2018 at 11:37 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sat, Jul 21, 2018 at 11:23 AM Arie <nospam@ariekanarie.nl> wrote:
> >
> > It does.
>
> well, one approach seems to be to police as per that bug report at say
> 90-95% of the actual inbound rate, while shaping at 85%... or some
> combination thereof.
>
> figuring out the "burst" value is tricky, of course, but I'd much
> rather let the queues build up in cake than the cmts, and putting a
> brick wall there at this point in the internet's evolution seems like
> a start.
>
> this morning I spent watching ietf preso after preso "accept" that
> 100ms of queuing delay and pdv was acceptible, then showing tests with
> 50 packet buffers showing that things like BBR worked ok at 100mbit,
> when... well... here I am at hundred mbit, with 300+ms of inherent
> queuing delay on the cable downlink, that shapes down nicely using
> cake vs cubic to 5-40, looking at the carnage. I *don't like* policers
> (at least, not what's deployed today), and the BBR folk don't like
> them either... but they seem necessary if overly aggressive transports
> (as netflix's appears to be, also) exist.
>
>
> >
> > On 21 July 2018 at 20:02, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> In other news:
> >>
> >> https://github.com/tohojo/sqm-scripts/issues/68
> >>
> >> does the ER4 have act_police?
> >>
> >> On Sat, Jul 21, 2018 at 10:55 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >
> >> > It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in the fast.com test. Both the site and the flent ping test report about the same 60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so I guess those stress my connection more than fast.com
> >> >
> >> >
> >> > On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:
> >> >>
> >> >> Thank you. At least for your ISP (?), at 250mbit, you can hold the
> >> >> damage down to something reasonable, and certainly you are winning big
> >> >> on the upload.
> >> >>
> >> >> Good to know the ER4 can keep up, also.
> >> >> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >> >
> >> >> > Unshaped reno attached.
> >> >> >
> >> >> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
> >> >> >>
> >> >> >> Yours is not as horrific as mine in either case.
> >> >> >>
> >> >> >> Can you provide an unshaped result as well?
> >> >> >>
> >> >> >>
> >> >> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> >> >> >> >
> >> >> >> > Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
> >> >> >> >
> >> >> >> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
> >> >> >> >
> >> >> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
> >> >> >> >>
> >> >> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >> >> >> >> >
> >> >> >> >> > 1) Can someone else on a cablemodem (even without the latest cake,
> >> >> >> >> > this happens to me on older cake and fq_codel) try this test?
> >> >> >> >> >
> >> >> >> >>
> >> >> >> >> I just tried this on my cable comcast connection. I set ingress to ~80%
> >> >> >> >> of what fast.com reports when no shaper is in place.
> >> >> >> >>
> >> >> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> >> >> >> >> dsthost docsis ingress
> >> >> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> >> >> >> >> srchost nat docsis ack-filter
> >> >> >> >>
> >> >> >> >> I got the same result as you. This is using latest cake.
> >> >> >> >>
> >> >> >> >> Georgios
> >> >> >> >>
> >> >> >> >> _______________________________________________
> >> >> >> >> Cake mailing list
> >> >> >> >> Cake@lists.bufferbloat.net
> >> >> >> >> https://lists.bufferbloat.net/listinfo/cake
> >> >> >> >>
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Cake mailing list
> >> >> >> > Cake@lists.bufferbloat.net
> >> >> >> > https://lists.bufferbloat.net/listinfo/cake
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >>
> >> >> >> Dave Täht
> >> >> >> CEO, TekLibre, LLC
> >> >> >> http://www.teklibre.com
> >> >> >> Tel: 1-669-226-2619
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Dave Täht
> >> >> CEO, TekLibre, LLC
> >> >> http://www.teklibre.com
> >> >> Tel: 1-669-226-2619
> >> >
> >> >
> >>
> >>
> >> --
> >>
> >> Dave Täht
> >> CEO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-669-226-2619
> >
> >
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] policers, finally

[Dave Taht]

I was off in the cheap seats watching,

https://www.youtube.com/watch?v=LdjavTiMrs0&feature=youtu.be&t=1h10m3s

saying "yes, for ghu's sake, back off half on ecn - or more! And if
you are going to try defeating policers, despite the desperate reasons
for them to exist... it's time to come up with a policer that will
fool your algorithm enough so the gamer downstairs stops screaming in
frustration!"

and then there was the aqm talk where they tried to move the setpoint
so the link was always buffered, rather than trying to hit the
artificially small buffer size and back off... (actually I liked this
talk because you can also try configuring the setpoint to where it was
below capacity)
On Sat, Jul 21, 2018 at 11:38 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sat, Jul 21, 2018 at 11:37 AM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Sat, Jul 21, 2018 at 11:23 AM Arie <nospam@ariekanarie.nl> wrote:
> > >
> > > It does.
> >
> > well, one approach seems to be to police as per that bug report at say
> > 90-95% of the actual inbound rate, while shaping at 85%... or some
> > combination thereof.
> >
> > figuring out the "burst" value is tricky, of course, but I'd much
> > rather let the queues build up in cake than the cmts, and putting a
> > brick wall there at this point in the internet's evolution seems like
> > a start.
> >
> > this morning I spent watching ietf preso after preso "accept" that
> > 100ms of queuing delay and pdv was acceptible, then showing tests with
> > 50 packet buffers showing that things like BBR worked ok at 100mbit,
> > when... well... here I am at hundred mbit, with 300+ms of inherent
> > queuing delay on the cable downlink, that shapes down nicely using
> > cake vs cubic to 5-40, looking at the carnage. I *don't like* policers
> > (at least, not what's deployed today), and the BBR folk don't like
> > them either... but they seem necessary if overly aggressive transports
> > (as netflix's appears to be, also) exist.
> >
> >
> > >
> > > On 21 July 2018 at 20:02, Dave Taht <dave.taht@gmail.com> wrote:
> > >>
> > >> In other news:
> > >>
> > >> https://github.com/tohojo/sqm-scripts/issues/68
> > >>
> > >> does the ER4 have act_police?
> > >>
> > >> On Sat, Jul 21, 2018 at 10:55 AM Arie <nospam@ariekanarie.nl> wrote:
> > >> >
> > >> > It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in the fast.com test. Both the site and the flent ping test report about the same 60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so I guess those stress my connection more than fast.com
> > >> >
> > >> >
> > >> > On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:
> > >> >>
> > >> >> Thank you. At least for your ISP (?), at 250mbit, you can hold the
> > >> >> damage down to something reasonable, and certainly you are winning big
> > >> >> on the upload.
> > >> >>
> > >> >> Good to know the ER4 can keep up, also.
> > >> >> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
> > >> >> >
> > >> >> > Unshaped reno attached.
> > >> >> >
> > >> >> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
> > >> >> >>
> > >> >> >> Yours is not as horrific as mine in either case.
> > >> >> >>
> > >> >> >> Can you provide an unshaped result as well?
> > >> >> >>
> > >> >> >>
> > >> >> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> > >> >> >> >
> > >> >> >> > Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
> > >> >> >> >
> > >> >> >> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
> > >> >> >> >
> > >> >> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
> > >> >> >> >>
> > >> >> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> > >> >> >> >> >
> > >> >> >> >> > 1) Can someone else on a cablemodem (even without the latest cake,
> > >> >> >> >> > this happens to me on older cake and fq_codel) try this test?
> > >> >> >> >> >
> > >> >> >> >>
> > >> >> >> >> I just tried this on my cable comcast connection. I set ingress to ~80%
> > >> >> >> >> of what fast.com reports when no shaper is in place.
> > >> >> >> >>
> > >> >> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> > >> >> >> >> dsthost docsis ingress
> > >> >> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> > >> >> >> >> srchost nat docsis ack-filter
> > >> >> >> >>
> > >> >> >> >> I got the same result as you. This is using latest cake.
> > >> >> >> >>
> > >> >> >> >> Georgios
> > >> >> >> >>
> > >> >> >> >> _______________________________________________
> > >> >> >> >> Cake mailing list
> > >> >> >> >> Cake@lists.bufferbloat.net
> > >> >> >> >> https://lists.bufferbloat.net/listinfo/cake
> > >> >> >> >>
> > >> >> >> >
> > >> >> >> > _______________________________________________
> > >> >> >> > Cake mailing list
> > >> >> >> > Cake@lists.bufferbloat.net
> > >> >> >> > https://lists.bufferbloat.net/listinfo/cake
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> --
> > >> >> >>
> > >> >> >> Dave Täht
> > >> >> >> CEO, TekLibre, LLC
> > >> >> >> http://www.teklibre.com
> > >> >> >> Tel: 1-669-226-2619
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >> --
> > >> >>
> > >> >> Dave Täht
> > >> >> CEO, TekLibre, LLC
> > >> >> http://www.teklibre.com
> > >> >> Tel: 1-669-226-2619
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Dave Täht
> > >> CEO, TekLibre, LLC
> > >> http://www.teklibre.com
> > >> Tel: 1-669-226-2619
> > >
> > >
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

[Cake] Policers

[Dave Taht]

https://www.nanog.org/sites/default/files/20170201_Flach_An_Internet-Wide_Analysis_v1.pdf

On Sat, Jul 21, 2018 at 11:45 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> I was off in the cheap seats watching,
>
> https://www.youtube.com/watch?v=LdjavTiMrs0&feature=youtu.be&t=1h10m3s
>
> saying "yes, for ghu's sake, back off half on ecn - or more! And if
> you are going to try defeating policers, despite the desperate reasons
> for them to exist... it's time to come up with a policer that will
> fool your algorithm enough so the gamer downstairs stops screaming in
> frustration!"
>
> and then there was the aqm talk where they tried to move the setpoint
> so the link was always buffered, rather than trying to hit the
> artificially small buffer size and back off... (actually I liked this
> talk because you can also try configuring the setpoint to where it was
> below capacity)
> On Sat, Jul 21, 2018 at 11:38 AM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > On Sat, Jul 21, 2018 at 11:37 AM Dave Taht <dave.taht@gmail.com> wrote:
> > >
> > > On Sat, Jul 21, 2018 at 11:23 AM Arie <nospam@ariekanarie.nl> wrote:
> > > >
> > > > It does.
> > >
> > > well, one approach seems to be to police as per that bug report at say
> > > 90-95% of the actual inbound rate, while shaping at 85%... or some
> > > combination thereof.
> > >
> > > figuring out the "burst" value is tricky, of course, but I'd much
> > > rather let the queues build up in cake than the cmts, and putting a
> > > brick wall there at this point in the internet's evolution seems like
> > > a start.
> > >
> > > this morning I spent watching ietf preso after preso "accept" that
> > > 100ms of queuing delay and pdv was acceptible, then showing tests with
> > > 50 packet buffers showing that things like BBR worked ok at 100mbit,
> > > when... well... here I am at hundred mbit, with 300+ms of inherent
> > > queuing delay on the cable downlink, that shapes down nicely using
> > > cake vs cubic to 5-40, looking at the carnage. I *don't like* policers
> > > (at least, not what's deployed today), and the BBR folk don't like
> > > them either... but they seem necessary if overly aggressive transports
> > > (as netflix's appears to be, also) exist.
> > >
> > >
> > > >
> > > > On 21 July 2018 at 20:02, Dave Taht <dave.taht@gmail.com> wrote:
> > > >>
> > > >> In other news:
> > > >>
> > > >> https://github.com/tohojo/sqm-scripts/issues/68
> > > >>
> > > >> does the ER4 have act_police?
> > > >>
> > > >> On Sat, Jul 21, 2018 at 10:55 AM Arie <nospam@ariekanarie.nl> wrote:
> > > >> >
> > > >> > It's Ziggo NL/Liberty Global. I'm surprised by the reported latency in the fast.com test. Both the site and the flent ping test report about the same 60-ish ms.  DSLReports and flent RRUL report way more bufferbloat, so I guess those stress my connection more than fast.com
> > > >> >
> > > >> >
> > > >> > On 21 July 2018 at 19:45, Dave Taht <dave.taht@gmail.com> wrote:
> > > >> >>
> > > >> >> Thank you. At least for your ISP (?), at 250mbit, you can hold the
> > > >> >> damage down to something reasonable, and certainly you are winning big
> > > >> >> on the upload.
> > > >> >>
> > > >> >> Good to know the ER4 can keep up, also.
> > > >> >> On Sat, Jul 21, 2018 at 10:36 AM Arie <nospam@ariekanarie.nl> wrote:
> > > >> >> >
> > > >> >> > Unshaped reno attached.
> > > >> >> >
> > > >> >> > On 21 July 2018 at 19:27, Dave Taht <dave.taht@gmail.com> wrote:
> > > >> >> >>
> > > >> >> >> Yours is not as horrific as mine in either case.
> > > >> >> >>
> > > >> >> >> Can you provide an unshaped result as well?
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On Sat, Jul 21, 2018 at 10:24 AM Arie <nospam@ariekanarie.nl> wrote:
> > > >> >> >> >
> > > >> >> >> > Two more data points. Shaped my connection to 250Mbit out of the advertised 250Mbit (my usual setting) and shaped to 200Mbit out of the 250Mbit. This is a pre-linux-net-next cake running on an Edgemax ER4 with kernel 3.10.107-UBNT.
> > > >> >> >> >
> > > >> >> >> > The regular spikes in ICMP ping is due to the crappy Puma 6 chipset in the cable modem. UDP is not affected.
> > > >> >> >> >
> > > >> >> >> > On 21 July 2018 at 19:20, Georgios Amanakis <gamanakis@gmail.com> wrote:
> > > >> >> >> >>
> > > >> >> >> >> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> > > >> >> >> >> >
> > > >> >> >> >> > 1) Can someone else on a cablemodem (even without the latest cake,
> > > >> >> >> >> > this happens to me on older cake and fq_codel) try this test?
> > > >> >> >> >> >
> > > >> >> >> >>
> > > >> >> >> >> I just tried this on my cable comcast connection. I set ingress to ~80%
> > > >> >> >> >> of what fast.com reports when no shaper is in place.
> > > >> >> >> >>
> > > >> >> >> >> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> > > >> >> >> >> dsthost docsis ingress
> > > >> >> >> >> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> > > >> >> >> >> srchost nat docsis ack-filter
> > > >> >> >> >>
> > > >> >> >> >> I got the same result as you. This is using latest cake.
> > > >> >> >> >>
> > > >> >> >> >> Georgios
> > > >> >> >> >>
> > > >> >> >> >> _______________________________________________
> > > >> >> >> >> Cake mailing list
> > > >> >> >> >> Cake@lists.bufferbloat.net
> > > >> >> >> >> https://lists.bufferbloat.net/listinfo/cake
> > > >> >> >> >>
> > > >> >> >> >
> > > >> >> >> > _______________________________________________
> > > >> >> >> > Cake mailing list
> > > >> >> >> > Cake@lists.bufferbloat.net
> > > >> >> >> > https://lists.bufferbloat.net/listinfo/cake
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> --
> > > >> >> >>
> > > >> >> >> Dave Täht
> > > >> >> >> CEO, TekLibre, LLC
> > > >> >> >> http://www.teklibre.com
> > > >> >> >> Tel: 1-669-226-2619
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >> --
> > > >> >>
> > > >> >> Dave Täht
> > > >> >> CEO, TekLibre, LLC
> > > >> >> http://www.teklibre.com
> > > >> >> Tel: 1-669-226-2619
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Dave Täht
> > > >> CEO, TekLibre, LLC
> > > >> http://www.teklibre.com
> > > >> Tel: 1-669-226-2619
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Dave Täht
> > > CEO, TekLibre, LLC
> > > http://www.teklibre.com
> > > Tel: 1-669-226-2619
> >
> >
> >
> > --
> >
> > Dave Täht
> > CEO, TekLibre, LLC
> > http://www.teklibre.com
> > Tel: 1-669-226-2619
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

The previous one was with:
net.ipv4.tcp_congestion_control=cubic

I retried with:
net.ipv4.tcp_congestion_control=reno

Georgios

[-- Attachment #2: ping-2018-07-21T132454.572444.Maryland.flent.gz --]
[-- Type: application/vnd.flent.data.gzip, Size: 175075 bytes --]

[-- Attachment #3: ping_-_Maryland-1.png --]
[-- Type: image/png, Size: 55670 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

On Sat, Jul 21, 2018 at 10:28 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> The previous one was with:
> net.ipv4.tcp_congestion_control=cubic
>
> I retried with:
> net.ipv4.tcp_congestion_control=reno
>
> Georgios

In the fast test this has no effect on the remote server's tcp, it's
always going to be reno.

Trying to cross-check behavior using our tests...

There isn't a specific reno setting test in flent for tcp_download as
best as I recall, so I was just calling netperf -H wherever -l 60 --
-K reno,reno

then running the flent ping test as previous mentioned.

(flent-fremont.bufferbloat.net and flent-newark both support reno bbr
and cubic, I haven't checked the others)

PS A side note is that we are not fully successfully moving the
inbound bottleneck to cake (at least in the cable case), as we do get
quite a bit of queuing delay even with linux tcp driving the tests.
I'd long written this off as inevitable, due to the bursty cable mac
but I'm grumpy this morning. 0 delay via fq would be better than even
the 15-40ms I'm getting now with linux flows.....

reno bbr cubic


-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] [Cerowrt-devel] inbound cake or fq_codel shaping fails on cable on netflix reno

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> On Sat, Jul 21, 2018 at 10:28 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>>
>> The previous one was with:
>> net.ipv4.tcp_congestion_control=cubic
>>
>> I retried with:
>> net.ipv4.tcp_congestion_control=reno
>>
>> Georgios
>
> In the fast test this has no effect on the remote server's tcp, it's
> always going to be reno.
>
> Trying to cross-check behavior using our tests...
>
> There isn't a specific reno setting test in flent for tcp_download as
> best as I recall, so I was just calling netperf -H wherever -l 60 --
> -K reno,reno

--test-parameter tcp_cong_control=reno should work for all tests...

-Toke

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Dave Taht]

George does your result mean you also have a crappy cablemodem?
On Sat, Jul 21, 2018 at 10:20 AM Georgios Amanakis <gamanakis@gmail.com> wrote:
>
> On Sat, 2018-07-21 at 09:09 -0700, Dave Taht wrote:
> >
> > 1) Can someone else on a cablemodem (even without the latest cake,
> > this happens to me on older cake and fq_codel) try this test?
> >
>
> I just tried this on my cable comcast connection. I set ingress to ~80%
> of what fast.com reports when no shaper is in place.
>
> #tc qdisc add dev ens4 root handle 8011 cake bandwidth 16000kbit dual-
> dsthost docsis ingress
> #tc qdisc add dev ens3 root handle 8012 cake bandwidth 2500kbit dual-
> srchost nat docsis ack-filter
>
> I got the same result as you. This is using latest cake.
>
> Georgios



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Georgios Amanakis]

On Mon, 2018-07-23 at 19:36 -0700, Dave Taht wrote:
> George does your result mean you also have a crappy cablemodem?
> 

Yes, I think so. It's a Linksys DPC3008 DOCSIS 3.0. Also, I cannot get
it to behave any differently with hping3 as Arie suggested.

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2742 bytes --]

> On Jul 21, 2018, at 6:09 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> PS I also have two other issues going on. This is the first time I've
> been using irtt with a 20ms interval, and I regularly see single 50+ms
> spikes (in both ping and irtt) data and also see irtt stop
> transmitting.

irtt should keep sending for the duration of the test. I noticed that it looks like irtt was actually used in only one of these initial tests: ping-2018-07-21T082842.445812.flent-newark-reno.flent. In the rest, netperf UDP_RR was used, which can stop sending upon packet loss.

If irtt was configured but didn’t run, that may be because flent does a connectivity check to the server with “irtt client -n”, where it sends three requests within 900ms (200ms timeout, then 300ms then 400ms) and if it doesn’t receive a reply, it falls back to netperf UDP_RR. Do you think that’s what happened here?

> On this front, it could merely be that my (not tested in
> months!) test cablemodem setup is malfunctioning also! Or we're
> hitting power save. Or (finally) seeing request-grant delays. Or
> scheduling delay somewhere in the net-next kernel I'm using... Or....
> (regardless, this seems independent of my main issue, and I've not had
> such high res data before)

Regarding the spikes both you and Arie you’re seeing, I also saw in one of your later emails "0 delay via fq would be better than even
the 15-40ms I'm getting now with linux flows”. Those numbers struck me as similar to an issue I’m still dealing with:

https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spikes-about-once-per-second-even-just-to/m-p/2359800/highlight/true#M119202 <https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spikes-about-once-per-second-even-just-to/m-p/2359800/highlight/true#M119202>

To summarize, with airOS on the NanoStation M5, there are isochronous pauses around once per second in the processing of all packets, not just for the WiFi device but Ethernet also. Packets are not lost, but queued for either 20ms, if one Ethernet port is connected, or 40ms, if both are connected. This behavior is exactly described by the ar7240sw_phy_poll_reset function in ag71xx_ar7240.c, so it looks to me like the ar7240 internal switch is being reset once per second for no apparent reason. So far I’ve gotten crickets in response.

The cable modem you’re using doesn’t happen to have built-in WiFi and an ar7240, does it? If it does and has the same or a similar driver problem, you could just do a low-interval ping straight to its Ethernet adapter and check for isochronous spikes, something like:

sudo ping -l 100 -c 5000 -i 0.001 cablemodem

Now, back to vacation :)

[-- Attachment #2: Type: text/html, Size: 3583 bytes --]

Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno

[Sebastian Moeller]

I believe that cable modems all default to 192.168.100.1, this seems to be backed by "Cable Modem Operations Support System Interface Specification", CM-SP-CM-OSSIv3.1-I04-150611:

"	• The CM MUST support 192.168.100.1, as the well-known diagnostic IP address accessible only from the CMCI interfaces. The CM MUST support the well-known diagnostic IP address, 192.168.100.1, on all physical interfaces associated with the CMCI. The CM MUST drop SNMP requests coming from the RF interface targeting the well-known IP address."

There might be exceptions to this, but I would be amazed if these would be common...

so:

sudo ping -l 100 -c 5000 -i 0.001 192.168.100.1

should work on all/most docsis setups.


> On Jul 22, 2018, at 11:57, Pete Heist <pete@heistp.net> wrote:
> 
>> On Jul 21, 2018, at 6:09 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> PS I also have two other issues going on. This is the first time I've
>> been using irtt with a 20ms interval, and I regularly see single 50+ms
>> spikes (in both ping and irtt) data and also see irtt stop
>> transmitting.
> 
> irtt should keep sending for the duration of the test. I noticed that it looks like irtt was actually used in only one of these initial tests: ping-2018-07-21T082842.445812.flent-newark-reno.flent. In the rest, netperf UDP_RR was used, which can stop sending upon packet loss.
> 
> If irtt was configured but didn’t run, that may be because flent does a connectivity check to the server with “irtt client -n”, where it sends three requests within 900ms (200ms timeout, then 300ms then 400ms) and if it doesn’t receive a reply, it falls back to netperf UDP_RR. Do you think that’s what happened here?
> 
>> On this front, it could merely be that my (not tested in
>> months!) test cablemodem setup is malfunctioning also! Or we're
>> hitting power save. Or (finally) seeing request-grant delays. Or
>> scheduling delay somewhere in the net-next kernel I'm using... Or....
>> (regardless, this seems independent of my main issue, and I've not had
>> such high res data before)
> 
> Regarding the spikes both you and Arie you’re seeing, I also saw in one of your later emails "0 delay via fq would be better than even
> the 15-40ms I'm getting now with linux flows”. Those numbers struck me as similar to an issue I’m still dealing with:
> 
> https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spikes-about-once-per-second-even-just-to/m-p/2359800/highlight/true#M119202
> 
> To summarize, with airOS on the NanoStation M5, there are isochronous pauses around once per second in the processing of all packets, not just for the WiFi device but Ethernet also. Packets are not lost, but queued for either 20ms, if one Ethernet port is connected, or 40ms, if both are connected. This behavior is exactly described by the ar7240sw_phy_poll_reset function in ag71xx_ar7240.c, so it looks to me like the ar7240 internal switch is being reset once per second for no apparent reason. So far I’ve gotten crickets in response.
> 
> The cable modem you’re using doesn’t happen to have built-in WiFi and an ar7240, does it? If it does and has the same or a similar driver problem, you could just do a low-interval ping straight to its Ethernet adapter and check for isochronous spikes, something like:
> 
> sudo ping -l 100 -c 5000 -i 0.001 cablemodem
> 
> Now, back to vacation :)
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake