[Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Michel Blais]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

Hi all,

Is it possible while using srchost, or dual-srchost, and dsthost, or
dual-dsthost, to do a bandwidth limitation by host instead of global ? From
what I read, seem like not.

I saw several times on messages on mailling list that some option could be
usefull for ISP and dual-srchost and dual-dsthost would seem something
usefull for small ISP, an alternative to Mikrotik PCQ with all the
advantage of cake like flow fairness and latency.

Thanks
---
Cordialement,  /  With regards,

Michel Blais

[-- Attachment #2: Type: text/html, Size: 871 bytes --]

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Dave Taht]

On Mon, Jun 11, 2018 at 8:43 AM, Michel Blais <michel@targointernet.com> wrote:
> Hi all,
>
> Is it possible while using srchost, or dual-srchost, and dsthost, or
> dual-dsthost, to do a bandwidth limitation by host instead of global ? From
> what I read, seem like not.

No. It's been one of those things where custom non-linux hardware is
used on the head end (CMTSes, DSLAMs),
where even trying seems futile. If there was a head-end maker that
wanted to try (and sponsor the effort), that would be cool.

One way to maybe get there without custom hw would be to insert a fat
x86 or arm64 cake box in front of a CMTS or dslam as a transparent
bridge (much like we see with certain dpi products) to do one veth per
subscriber. We'd coalesce all the IPs a subscriber has into a cake
instance

# eth0 takes all traffic, eth1 outputs the reshaped traffic
# bridge veth0,veth1 to eth1
# Route customer 1
ip route add 1.1.1.1 dev veth0
ip route add 2001::1:x:y:z/64 dev veth0
# Route customer 2
ip route add 1.1.1.2 dev veth1
ip route add 2001::2:x:y:z/64 dev veth1

tc qdisc add dev veth0 cake bandwidth 100mbit
tc qdisc add dev veth1 cake bandwidth 20mbit

Some comms from the head-end would be helpful to be monitoring the
achieved rate (line noise issues) and global bandwidth available
and tweaking each cake instance to suit every few seconds (tc qdisc
change dev veth0 cake bandwidth 80mbit).

Given that the latest cake is cracking 40gbit on a single interface,
it might be interesting to see how hard we could push, say, 10,000
instances like the above through a big box to see what happens. But I
tend to think the work should be layered onto a line card rather than
a separate box.

This is why we try so hard to make the code dual licensed and publish
papers documenting the algoriths. 7 years after this effort started
cable head-ends still use 2sec of FIFO buffering. I'd had high hopes
we'd see *something* from arris by now that did more of the right
thing.

they tried sfq way back when

https://pdfs.semanticscholar.org/c577/0612bfaa1dff4daf2b0cfe56b79627dddc9c.pdf

and sfq is in one of their fiber products.

I keep hoping we'll make some onts and gpon hardware soon, with bql +
fq_codel at least.

> I saw several times on messages on mailling list that some option could be
> usefull for ISP and dual-srchost and dual-dsthost would seem something
> usefull for small ISP, an alternative to Mikrotik PCQ with all the advantage
> of cake like flow fairness and latency.

Keep hoping that mikrotik will get on the ball. Many fq_codel requests
on their forums. ubnt is all over this stuff.

> Thanks
> ---
> Cordialement,  /  With regards,
>
> Michel Blais
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Toke Høiland-Jørgensen]

Michel Blais <michel@targointernet.com> writes:

> Hi all,
>
> Is it possible while using srchost, or dual-srchost, and dsthost, or
> dual-dsthost, to do a bandwidth limitation by host instead of global ? From
> what I read, seem like not.

No, not currently (as Dave said). I do believe it would be possible to
extend the current architecture to support this, though. Basically,
making the number of tins configurable to an arbitrary number and making
it possible to set individual bandwidths on them should do the trick. A
TC filter could then be used to assign traffic to the right tin based on
whatever information is available.

However, I don't think the current way of finding the next tin to
service is going to scale to an arbitrary number of tins, so this
probably needs to be changed before such a configuration is really
feasible...

-Toke

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Stephen Hemminger]

On Mon, 11 Jun 2018 21:44:25 +0200
Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Michel Blais <michel@targointernet.com> writes:
> 
> > Hi all,
> >
> > Is it possible while using srchost, or dual-srchost, and dsthost, or
> > dual-dsthost, to do a bandwidth limitation by host instead of global ? From
> > what I read, seem like not.  
> 
> No, not currently (as Dave said). I do believe it would be possible to
> extend the current architecture to support this, though. Basically,
> making the number of tins configurable to an arbitrary number and making
> it possible to set individual bandwidths on them should do the trick. A
> TC filter could then be used to assign traffic to the right tin based on
> whatever information is available.
> 
> However, I don't think the current way of finding the next tin to
> service is going to scale to an arbitrary number of tins, so this
> probably needs to be changed before such a configuration is really
> feasible...
> 
> -Toke

Making cake into a version of HTB is not necessarily a good idea

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Dave Taht]

On Mon, Jun 11, 2018 at 12:54 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Mon, 11 Jun 2018 21:44:25 +0200
> Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
>> Michel Blais <michel@targointernet.com> writes:
>>
>> > Hi all,
>> >
>> > Is it possible while using srchost, or dual-srchost, and dsthost, or
>> > dual-dsthost, to do a bandwidth limitation by host instead of global ? From
>> > what I read, seem like not.
>>
>> No, not currently (as Dave said). I do believe it would be possible to
>> extend the current architecture to support this, though. Basically,
>> making the number of tins configurable to an arbitrary number and making
>> it possible to set individual bandwidths on them should do the trick. A
>> TC filter could then be used to assign traffic to the right tin based on
>> whatever information is available.
>>
>> However, I don't think the current way of finding the next tin to
>> service is going to scale to an arbitrary number of tins, so this
>> probably needs to be changed before such a configuration is really
>> feasible...
>>
>> -Toke
>
> Making cake into a version of HTB is not necessarily a good idea

Not the plan. But arguably, we'd need a list of requirements from a
major head-end
vendor or DPI co (like sandvine) to even try to address the ISP side
of the universe.

I long ago ran out of motivation and money to continue working on
bufferbloat. It's fixed enough,
for those that care. It's on enough devices (more every day) now to
give those that succeed some market advantage.

Heaving cake over the transom is the "last" thing. And after 18
versions in the last round, toke's fried, I'm fried, everybody's
fried. I'm very happy it started cracking 40gbits, stumped at the bug
stopping us.

I do long for the day where I could buy a cable modem, dsl device, can
bus, ont, gpon, wireless router, enode-b, etc, etc off the shelf,
clearly  marketed with bufferbloat-related fixes - and ISPs will
supply default gear that worked "right", etc, etc that all did the
right thing... but I figure that's going to take explicit market
demand, regulation, or a miracle - or all three. And a decade. Or
more.

I *am* going to washington DC week after next for the lanman2018
presentation of cake, and perhaps I'll find a way to raise some hell
with the FCC, congress, or the FTC (suggestions wanted), but...




-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]


> On Jun 11, 2018, at 10:17 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> I long ago ran out of motivation and money to continue working on
> bufferbloat. It's fixed enough,
> for those that care. It's on enough devices (more every day) now to
> give those that succeed some market advantage.

Any thoughts on worthy work beyond bloat?

> Heaving cake over the transom is the "last" thing. And after 18
> versions in the last round, toke's fried, I'm fried, everybody's
> fried. I'm very happy it started cracking 40gbits, stumped at the bug
> stopping us.

I wish I could even reproduce that, then I’d at least take a caveman approach to solving it.

> I *am* going to washington DC week after next for the lanman2018
> presentation of cake, and perhaps I'll find a way to raise some hell
> with the FCC, congress, or the FTC (suggestions wanted), but...

Please do.

I feel we need a re-decentralization of the Internet’s infrastructure and services, and the FCC’s actions are not helping. I’m not sure how to get that across when the only language spoken is money.

[-- Attachment #2: Type: text/html, Size: 8890 bytes --]

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Dave Taht]

On Mon, Jun 11, 2018 at 11:55 PM, Pete Heist <pete@heistp.net> wrote:
>
> On Jun 11, 2018, at 10:17 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> I long ago ran out of motivation and money to continue working on
> bufferbloat. It's fixed enough,
> for those that care. It's on enough devices (more every day) now to
> give those that succeed some market advantage.
>
>
> Any thoughts on worthy work beyond bloat?

I gave into despair when the shuttleworth grant didn't come through
last year. I'm glad y'all keep slogging on.

I started at this as a third world problem. Living in the first world
is thus far proving too expensive.
Unless I get my act together on $dayjob soon even moving back to
Nicaragua will be impossible. (and Nica is currently very dicy
politically)

I'm still primarily motivated by fixing the internet outside the usa.
My daydream, once I had
enough cash in the bank or some (say, 50k) maintenence level of
income,  was to sail
from pacific island to pacific island, updating uplinks along the way,
finishing up writing a few papers,
contemplating some other currently insoluable networking problems -
like routing.

These days... I don't know. I'm at a cusp. If I keep failing at dayjob
for another month I'll try to find something
outside this field entirely. Last year, being dead broke, I did
construction for a while. It was good for me.

> Heaving cake over the transom is the "last" thing. And after 18
> versions in the last round, toke's fried, I'm fried, everybody's
> fried. I'm very happy it started cracking 40gbits, stumped at the bug
> stopping us.
>
>
> I wish I could even reproduce that, then I’d at least take a caveman
> approach to solving it.
>
> I *am* going to washington DC week after next for the lanman2018
>
> presentation of cake, and perhaps I'll find a way to raise some hell
> with the FCC, congress, or the FTC (suggestions wanted), but...
>
>
> Please do.

The only idea I have at the moment is to stand outside the ftc's offices in
a zoot suit with guitar and empty violin case.

>
> I feel we need a re-decentralization of the Internet’s infrastructure and
> services, and the FCC’s actions are not helping. I’m not sure how to get
> that across when the only language spoken is money.

Obama's FTC seemed like a better place than the FCC for many issues.



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Michel Blais]

[-- Attachment #1: Type: text/plain, Size: 4294 bytes --]

Thanks for the answers.

@ Toke

I was not clear on my first message but what I have in mind isn't
individual shaping but more like a bandwidth limit identical for every
dst-host, something like 50 mb/s by dst-host for every host instead of a
global 1 Gb/s or 10 Gb/s without limit for everyone.

@Dave

Put a linux box before telecommunication hardware is what I had in mind. I
will give a try with veth interfaces by customer.

As for network device, I personnaly think the objectif should then be to
convice network chipset maker like Microchip, Cavium, Freescale, Broadcom,
etc.

Ubnt all over this stuff ? Don't want to burst your bubble but airmax (test
on M serie) use HTB + SFQ if traffic shaping is configured but default
pfifo_fast if no traffic shaping is configured. On gen 2 AC, still pfifo
without traffic shaping but don't know when traffic shaping is configured
but I'm pretty sure it would be the same. I pretty sure airfiber line is
also fifo only. I know codel is support in the edgemax router line but not
compatible with cpu offload so won't be use by most except on X serie (line
without cpu offload) that is often use as home/small office router. I doubt
any WISP (Ubnt main market) using Edgerouter for routing would activate
codel since it would hurt badly forwarding performance. Anyway, just like
for CMTS, WISP bottleneck is the wireless radio, not the router.
---
Cordialement,  /  With regards,

Michel Blais

2018-06-12 14:55 GMT-04:00 Dave Taht <dave.taht@gmail.com>:

> On Mon, Jun 11, 2018 at 11:55 PM, Pete Heist <pete@heistp.net> wrote:
> >
> > On Jun 11, 2018, at 10:17 PM, Dave Taht <dave.taht@gmail.com> wrote:
> >
> > I long ago ran out of motivation and money to continue working on
> > bufferbloat. It's fixed enough,
> > for those that care. It's on enough devices (more every day) now to
> > give those that succeed some market advantage.
> >
> >
> > Any thoughts on worthy work beyond bloat?
>
> I gave into despair when the shuttleworth grant didn't come through
> last year. I'm glad y'all keep slogging on.
>
> I started at this as a third world problem. Living in the first world
> is thus far proving too expensive.
> Unless I get my act together on $dayjob soon even moving back to
> Nicaragua will be impossible. (and Nica is currently very dicy
> politically)
>
> I'm still primarily motivated by fixing the internet outside the usa.
> My daydream, once I had
> enough cash in the bank or some (say, 50k) maintenence level of
> income,  was to sail
> from pacific island to pacific island, updating uplinks along the way,
> finishing up writing a few papers,
> contemplating some other currently insoluable networking problems -
> like routing.
>
> These days... I don't know. I'm at a cusp. If I keep failing at dayjob
> for another month I'll try to find something
> outside this field entirely. Last year, being dead broke, I did
> construction for a while. It was good for me.
>
> > Heaving cake over the transom is the "last" thing. And after 18
> > versions in the last round, toke's fried, I'm fried, everybody's
> > fried. I'm very happy it started cracking 40gbits, stumped at the bug
> > stopping us.
> >
> >
> > I wish I could even reproduce that, then I’d at least take a caveman
> > approach to solving it.
> >
> > I *am* going to washington DC week after next for the lanman2018
> >
> > presentation of cake, and perhaps I'll find a way to raise some hell
> > with the FCC, congress, or the FTC (suggestions wanted), but...
> >
> >
> > Please do.
>
> The only idea I have at the moment is to stand outside the ftc's offices in
> a zoot suit with guitar and empty violin case.
>
> >
> > I feel we need a re-decentralization of the Internet’s infrastructure and
> > services, and the FCC’s actions are not helping. I’m not sure how to get
> > that across when the only language spoken is money.
>
> Obama's FTC seemed like a better place than the FCC for many issues.
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 6448 bytes --]

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]


> On Jun 16, 2018, at 5:33 PM, Michel Blais <michel@targointernet.com> wrote:
> 
> I doubt any WISP (Ubnt main market) using Edgerouter for routing would activate codel since it would hurt badly forwarding performance. Anyway, just like for CMTS, WISP bottleneck is the wireless radio, not the router.


Hi Michel- badly hurt forwarding performance in what way?

The bottleneck is not always the wireless radio, particularly when the link is not over-provisioned. I’m working with a WISP that uses Ubiquiti gear and APUs in their backhaul that do soft rate limiting (on their stable links) with HTB and SFQ. In my lab tests using NSM5s and APUs, both Cake and fq_codel outperform SFQ in both inter-flow and (especially) intra-flow latency under load, so that’s why we’re working on switching to one or the other. HTB is needed though for customized fairness, as you noted in your original question.

Out-of-the-box performance of NSM5s with airMAX enabled leaves something left to be desired as far as latency under load goes, so one should either over-provision or do some kind of shaping. Compare:

airMAX enabled, default queueing:
https://www.drhleny.cz/nsm5-airmax/default_rrulbe/ <https://www.drhleny.cz/nsm5-airmax/default_rrulbe/>

airMAX disabled, cake queueing (enabling airMAX here hurts inter-flow latency):
https://www.drhleny.cz/nsm5/eg_csrt_rrulbe_eg_cake_40mbit/ <https://www.drhleny.cz/nsm5/eg_csrt_rrulbe_eg_cake_40mbit/>

By the way, if you note the isochronous spikes in the output, that appears to be due to a bug in the NSM5’s Ethernet or internal switch driver that has been fixed in LEDE/OpenWRT. If you or anyone else happens to have an NSM5, I’d appreciate your testing and adding to the thread, where I didn’t yet manage to convince Ubiquiti that it’s real and an issue:

https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spikes-about-once-per-second-even-just-to/td-p/2358704/page/2 <https://community.ubnt.com/t5/airMAX-Installation/NanoStation-M5-ping-spikes-about-once-per-second-even-just-to/td-p/2358704/page/2>

I’m actively writing an article on this topic and will post when finished...


[-- Attachment #2: Type: text/html, Size: 3247 bytes --]

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Pete Heist]

> On Jun 16, 2018, at 9:20 PM, Pete Heist <pete@heistp.net> wrote:
> 
> The bottleneck is not always the wireless radio, particularly when the link is not over-provisioned.

Sorry, I wrote this incorrectly. What I mean is that when the bottleneck _is_ the wireless radio (i.e. it’s not over-provisioned), soft rate limiting (and also CoDel) can still be used to manage the queue and improve latency under load.

Re: [Cake] Bandwidith rate by host instead of global while using [dual-]srchost and [dual-]dsthost

[Toke Høiland-Jørgensen]

Michel Blais <michel@targointernet.com> writes:

> Thanks for the answers.
>
> @ Toke
>
> I was not clear on my first message but what I have in mind isn't
> individual shaping but more like a bandwidth limit identical for every
> dst-host, something like 50 mb/s by dst-host for every host instead of
> a global 1 Gb/s or 10 Gb/s without limit for everyone.

Whether or not the limit is the same for every host or not doesn't
really matter, does it? It's still per-host (or per-customer in the case
where the IP-to-customer mapping is not 1-to-1) shaping with an
arbitrary number of hosts?

-Toke