[Cake] cake separate qos for lan

[Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

Hi ,
I'm experimenting in replacing a mikrotik router with plain linux. by
following the instructions on the cake page, i have setup the following
line in the /etc/ppp/ip-ip script so that the user will be limited to
bandwidth using cake.


/usr/sbin/tc qdisc add dev $pppdev root cake bandwidth ${BURST_DOWN}bit

but i have certain lan traffic available to the customer which should be
available at higher speed, for eg. cache traffic and i want to set that
speed to 20mbit default .

if i understand correctly i will have to mark traffic coming in from that
lan source using iptables, can someone guide me how to set bandwidth only
for that source to be higher.




-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 971 bytes --]

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 26 Mar, 2016, at 17:14, Allan Pinto <allan316@gmail.com> wrote:
> 
> I'm experimenting in replacing a mikrotik router with plain linux. by following the instructions on the cake page, i have setup the following line in the /etc/ppp/ip-ip script so that the user will be limited to bandwidth using cake.
> 
> /usr/sbin/tc qdisc add dev $pppdev root cake bandwidth ${BURST_DOWN}bit
> 
> but i have certain lan traffic available to the customer which should be available at higher speed, for eg. cache traffic and i want to set that speed to 20mbit default .
> 
> if i understand correctly i will have to mark traffic coming in from that lan source using iptables, can someone guide me how to set bandwidth only for that source to be higher.

I’m not certain what your topology is here.  Is the cache inside or outside the point where the router is fitted, or is it within the router, or off to the side on a separate port?

Also, the command shown will apply a limit to *egress* traffic on the given port.  If you need to do *ingress* shaping, there’s a different sequence of commands.

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

> Is the cache inside or outside the point where the router is fitted
 outside..

Cache-Server
       |
internet Gateway ---> L2 switch   --> LInux router with cake - - [ pppoe
connection ]  --> customer



>Also, the command shown will apply a limit to *egress* traffic on the
given port.  If you need to do *ingress* shaping, there’s a different
sequence of commands
i have put the ingress commands, i did not mention them as i did not have
any need of changes there.



On Sun, Mar 27, 2016 at 3:44 AM, Jonathan Morton <chromatix99@gmail.com>
wrote:

>
> > On 26 Mar, 2016, at 17:14, Allan Pinto <allan316@gmail.com> wrote:
> >
> > I'm experimenting in replacing a mikrotik router with plain linux. by
> following the instructions on the cake page, i have setup the following
> line in the /etc/ppp/ip-ip script so that the user will be limited to
> bandwidth using cake.
> >
> > /usr/sbin/tc qdisc add dev $pppdev root cake bandwidth ${BURST_DOWN}bit
> >
> > but i have certain lan traffic available to the customer which should be
> available at higher speed, for eg. cache traffic and i want to set that
> speed to 20mbit default .
> >
> > if i understand correctly i will have to mark traffic coming in from
> that lan source using iptables, can someone guide me how to set bandwidth
> only for that source to be higher.
>
> I’m not certain what your topology is here.  Is the cache inside or
> outside the point where the router is fitted, or is it within the router,
> or off to the side on a separate port?
>
> Also, the command shown will apply a limit to *egress* traffic on the
> given port.  If you need to do *ingress* shaping, there’s a different
> sequence of commands.
>
>  - Jonathan Morton
>
>


-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 3080 bytes --]

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 27 Mar, 2016, at 08:31, Allan Pinto <allan316@gmail.com> wrote:
> 
> Cache-Server
>        |
> internet Gateway ---> L2 switch   --> LInux router with cake - - [ pppoe connection ]  --> customer

Aha - that is a different topology than we usually assume.  So the egress side of the port is the right one to consider.  It’s nice to see Cake being considered for the provider's side of the link.

Interesting.

Cake doesn’t have its own facilities to do the sort of specific discrimination you want.  All the mechanisms it has are geared to sharing a fixed capacity as equitably as is feasible.  So you will need to divide the traffic using some other mechanism, and pass it through two separate instances of Cake.

Ideally you want one instance set for the capacity of the physical link, with all traffic passing through it, and the second instance set for the allocation for non-cache traffic, with the cache traffic bypassing it.  The customer will then get full link capacity when accessing the cache and nothing else, and latency will still be controlled well with a complex mix of traffic.

I recommend you use the IMQ mechanism (http://lartc.org/howto/lartc.imq.html) to achieve the ideal configuration above:

ip link set imq0 up

tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth $FULL_RATE triple-isolate

tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE flows

iptables -t mangle -A PREROUTING -o ppp0 -s $CACHE_IP -j IMQ —todev 0

That reminds me - we need to update the documentation to properly describe the overhead and triple-isolation keywords.  You might need a different overhead setting than “pppoe-vcmux” depending on the details of your link.

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 27 Mar, 2016, at 10:35, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> iptables -t mangle -A PREROUTING -o ppp0 -s $CACHE_IP -j IMQ —todev 0

Correction:

iptables -t mangle -A PREROUTING -o ppp0 ! -s $CACHE_IP -j IMQ --todev 0

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Allan,

> On Mar 27, 2016, at 09:35 , Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> 
>> On 27 Mar, 2016, at 08:31, Allan Pinto <allan316@gmail.com> wrote:
>> 
>> Cache-Server
>>       |
>> internet Gateway ---> L2 switch   --> LInux router with cake - - [ pppoe connection ]  --> customer
> 
> Aha - that is a different topology than we usually assume.  So the egress side of the port is the right one to consider.  It’s nice to see Cake being considered for the provider's side of the link.
> 
> Interesting.
> 
> Cake doesn’t have its own facilities to do the sort of specific discrimination you want.  All the mechanisms it has are geared to sharing a fixed capacity as equitably as is feasible.  So you will need to divide the traffic using some other mechanism, and pass it through two separate instances of Cake.
> 
> Ideally you want one instance set for the capacity of the physical link, with all traffic passing through it, and the second instance set for the allocation for non-cache traffic, with the cache traffic bypassing it.  The customer will then get full link capacity when accessing the cache and nothing else, and latency will still be controlled well with a complex mix of traffic.
> 
> I recommend you use the IMQ mechanism (http://lartc.org/howto/lartc.imq.html) to achieve the ideal configuration above:

	While IMQ seems more complete than IFB, ifb made it into the main-line kernel, so IMQ will be staying out in the cold, so it might be more future-proof to just use IFBs from the get-go; unless you are happy to keep compiling out of tree modules (which you wll have to do anyways for cake, but cake hopefully will be included/offered upstream).

> 
> ip link set imq0 up
> 
> tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth $FULL_RATE triple-isolate

	I would respectfully recommend to avoid the symbolic overhead parameters and use “overhead NN” instead (with NN being the number in bytes), as the current state of the symbolic parameters is under-documented and inconsistent. 

> 
> tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE flows
> 
> iptables -t mangle -A PREROUTING -o ppp0 -s $CACHE_IP -j IMQ —todev 0
> 
> That reminds me - we need to update the documentation to properly describe the overhead and triple-isolation keywords.  

	We first need to fix the overhead keywords I believe, having some of them re-set the overhead to a fixed number and some being modifiers that can be reasonably added to other overhead keywords with no distinction of this fact in either the parameter name or documentation is sub-optimal… The current state where one can supply multiple keywords on the commend line with some being additive and some resetting the the overhead to a fixed value is highly surprising. 
	And IMHO just documenting this inconsistency better is not the right solution. I would vote for making all of the additive and document that well; this will allow to do the right thing easily but will also allow for the unforeseeable in the future (and I actually had a patch ages ago that pushed the behaviour further in the “right” direction).

Best Regards
	Sebastian

> You might need a different overhead setting than “pppoe-vcmux” depending on the details of your link.
> 
> - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi Jonathan,
Thank you for the response I will check this and get back.
Wishing you all a happy Easter.

-----Original Message-----
From: "Jonathan Morton" <chromatix99@gmail.com>
Sent: ‎27-‎03-‎2016 13:12
To: "Allan Pinto" <allan316@gmail.com>
Cc: "cake@lists.bufferbloat.net" <cake@lists.bufferbloat.net>
Subject: Re: [Cake] cake separate qos for lan


> On 27 Mar, 2016, at 10:35, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> iptables -t mangle -A PREROUTING -o ppp0 -s $CACHE_IP -j IMQ —todev 0

Correction:

iptables -t mangle -A PREROUTING -o ppp0 ! -s $CACHE_IP -j IMQ --todev 0

 - Jonathan Morton


[-- Attachment #2: Type: text/html, Size: 1732 bytes --]

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
> 
> it might be more future-proof to just use IFBs from the get-go

For this particular use-case, it seems to be more complicated to use IFB than IMQ, largely because there is no iptables rule to divert packets through an IFB device, and unlike iptables, the CBQ filter mechanism doesn’t directly support negative matches of any kind.

However, I think this would work - though it’s completely untested:

ip link set ifb0 up

tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth $FULL_RATE triple-isolate

tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE flows

tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src $CACHE_IP/32

tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred egress redirect dev ifb0

The logic of the above is that a positive match is made on the cache traffic, but no action is taken.  This terminates filter processing for that traffic.  The remaining traffic is redirected unconditionally to the IFB device by the second filter rule.

One thing I’m not entirely certain of is whether traffic that has been through an IFB device is then requeued in the normal way on the original device.  I’d appreciate feedback on whether this system does in fact work.

> I would respectfully recommend to avoid the symbolic overhead parameters

Even if I change their underlying behaviour in the future, it’ll be in a way that retains backwards compatibility with all the examples I’ve given for the current scheme.  I mostly wanted to raise awareness that the overhead compensation system exists for use on encapsulated links.

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 2151 bytes --]

>tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE
flows
here imq0 should be replaced by ifb0 right?.
i will be testing this tonight and reply with results.

On Mon, Mar 28, 2016 at 4:01 PM, Jonathan Morton <chromatix99@gmail.com>
wrote:

>
> > On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
> >
> > it might be more future-proof to just use IFBs from the get-go
>
> For this particular use-case, it seems to be more complicated to use IFB
> than IMQ, largely because there is no iptables rule to divert packets
> through an IFB device, and unlike iptables, the CBQ filter mechanism
> doesn’t directly support negative matches of any kind.
>
> However, I think this would work - though it’s completely untested:
>
> ip link set ifb0 up
>
> tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth
> $FULL_RATE triple-isolate
>
> tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE
> flows
>
> tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src
> $CACHE_IP/32
>
> tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred
> egress redirect dev ifb0
>
> The logic of the above is that a positive match is made on the cache
> traffic, but no action is taken.  This terminates filter processing for
> that traffic.  The remaining traffic is redirected unconditionally to the
> IFB device by the second filter rule.
>
> One thing I’m not entirely certain of is whether traffic that has been
> through an IFB device is then requeued in the normal way on the original
> device.  I’d appreciate feedback on whether this system does in fact work.
>
> > I would respectfully recommend to avoid the symbolic overhead parameters
>
> Even if I change their underlying behaviour in the future, it’ll be in a
> way that retains backwards compatibility with all the examples I’ve given
> for the current scheme.  I mostly wanted to raise awareness that the
> overhead compensation system exists for use on encapsulated links.
>
>  - Jonathan Morton
>
>


-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 2963 bytes --]

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Allan,

> On Mar 27, 2016, at 07:31 , Allan Pinto <allan316@gmail.com> wrote:
> 
> > Is the cache inside or outside the point where the router is fitted
>  outside..
> 
> Cache-Server
>        |
> internet Gateway —> L2 switch   --> LInux router with cake - - [ pppoe connection ]  --> customer

	Looking at this I would assume you can skip ingress shaping completely, just shape on the pppoe devices egress (for customer ingress) as well as on egress to the L2-switch for customer egress. This should allow to completely avoid IMQ/IFB and the associated computation cost. On the other hand I am not sure how well this will scale with its two cake instances per customer…

> 
> 
> 
> >Also, the command shown will apply a limit to *egress* traffic on the given port.  If you need to do *ingress* shaping, there’s a different sequence of commands
> i have put the ingress commands, i did not mention them as i did not have any need of changes there.
>                          
> 
> 
> On Sun, Mar 27, 2016 at 3:44 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> > On 26 Mar, 2016, at 17:14, Allan Pinto <allan316@gmail.com> wrote:
> >
> > I'm experimenting in replacing a mikrotik router with plain linux. by following the instructions on the cake page, i have setup the following line in the /etc/ppp/ip-ip script so that the user will be limited to bandwidth using cake.
> >
> > /usr/sbin/tc qdisc add dev $pppdev root cake bandwidth ${BURST_DOWN}bit
> >
> > but i have certain lan traffic available to the customer which should be available at higher speed, for eg. cache traffic and i want to set that speed to 20mbit default .
> >
> > if i understand correctly i will have to mark traffic coming in from that lan source using iptables, can someone guide me how to set bandwidth only for that source to be higher.
> 
> I’m not certain what your topology is here.  Is the cache inside or outside the point where the router is fitted, or is it within the router, or off to the side on a separate port?
> 
> Also, the command shown will apply a limit to *egress* traffic on the given port.  If you need to do *ingress* shaping, there’s a different sequence of commands.
> 
>  - Jonathan Morton
> 
> 
> 
> 
> -- 
> Thanx and regd's.
> 
> Allan.
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Jonathan,

> On Mar 28, 2016, at 12:31 , Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> 
>> On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
>> 
>> it might be more future-proof to just use IFBs from the get-go
> 
> For this particular use-case, it seems to be more complicated to use IFB than IMQ, largely because there is no iptables rule to divert packets through an IFB device, and unlike iptables, the CBQ filter mechanism doesn’t directly support negative matches of any kind.

	As I just wrote, can’t we completely avoid the IMQ/IFB here and use dual egress shaping instead (once on the pppoe device and once on the interface connected to the switch, which effectively should shape both directions)?

> 
> However, I think this would work - though it’s completely untested:
> 
> ip link set ifb0 up
> 
> tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth $FULL_RATE triple-isolate

	I wonder how you came up with pppoe-vcmux, I have not seen any information about the link technology in Allan’s post. As far as I know a number of (mislead) ISPs use PPPoE even on fiber links. @Allan, what is the link technology you use?

> 
> tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE flows

	I believe this might work as egress on the interface facing the L2-switch…

> 
> tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src $CACHE_IP/32
> 
> tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred egress redirect dev ifb0
> 
> The logic of the above is that a positive match is made on the cache traffic, but no action is taken.  This terminates filter processing for that traffic.  The remaining traffic is redirected unconditionally to the IFB device by the second filter rule.
> 
> One thing I’m not entirely certain of is whether traffic that has been through an IFB device is then requeued in the normal way on the original device.  

	It should, but only on egress…


> I’d appreciate feedback on whether this system does in fact work.
> 
>> I would respectfully recommend to avoid the symbolic overhead parameters
> 
> Even if I change their underlying behaviour in the future, it’ll be in a way that retains backwards compatibility with all the examples I’ve given for the current scheme.  I mostly wanted to raise awareness that the overhead compensation system exists for use on encapsulated links.

	All fair points!

> 
> - Jonathan Morton
> 


Best Regards
	Sebastian

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 4357 bytes --]

Hi Sebastian,
I should have made this more clear please see below topology with added
comments. the customers connecting to the linux router can be in range from
100 to 2000, so shaping on the switch is not really a option. I am right
now testing on a i3 machine, but for actual live testing am planning to
test with i7 or a xeon.

                  Cache-Server [ connected to internet gateway , traffic
can be sent to it via wccp or policy based routing ]
                           |
  internet---->internet Gateway —> L2 switch [ MEN network on fiber ]   -->
LInux router with cake[ includes a pppoe server which authenticates with
radius ] - - [ pppoe connection over a fiber men network ]  --> customer [
customers can be 100 to 2000 ]

basically the customer will create a broadband connection on his pc to
connect.

. > @Allan, what is the link technology you use?
fiber 1g/10g/last hop cat5e

> As I just wrote, can’t we completely avoid the IMQ/IFB here and use dual
egress shaping instead (once on the pppoe device and once on the interface
connected to the switch, which effectively should shape both directions)?

i may be wrong here, but i think jonathan is advising the use of IMQ/IFB to
provide two different shaping scenarios on egress itself. not ingress. as i
need cache traffic to have higher bandwidth on the egress towards customer
but non- cache traffic [ pure internet ] to remain within the bandwidth
limits purchased by the customer.



On Mon, Mar 28, 2016 at 5:39 PM, moeller0 <moeller0@gmx.de> wrote:

> Hi Jonathan,
>
> > On Mar 28, 2016, at 12:31 , Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> >
> >> On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
> >>
> >> it might be more future-proof to just use IFBs from the get-go
> >
> > For this particular use-case, it seems to be more complicated to use IFB
> than IMQ, largely because there is no iptables rule to divert packets
> through an IFB device, and unlike iptables, the CBQ filter mechanism
> doesn’t directly support negative matches of any kind.
>
>         As I just wrote, can’t we completely avoid the IMQ/IFB here and
> use dual egress shaping instead (once on the pppoe device and once on the
> interface connected to the switch, which effectively should shape both
> directions)?
>
> >
> > However, I think this would work - though it’s completely untested:
> >
> > ip link set ifb0 up
> >
> > tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth
> $FULL_RATE triple-isolate
>
>         I wonder how you came up with pppoe-vcmux, I have not seen any
> information about the link technology in Allan’s post. As far as I know a
> number of (mislead) ISPs use PPPoE even on fiber links. @Allan, what is the
> link technology you use?
>
> >
> > tc qdisc replace dev imq0 root handle 2: cake raw bandwidth
> $NONCACHE_RATE flows
>
>         I believe this might work as egress on the interface facing the
> L2-switch…
>
> >
> > tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src
> $CACHE_IP/32
> >
> > tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action
> mirred egress redirect dev ifb0
> >
> > The logic of the above is that a positive match is made on the cache
> traffic, but no action is taken.  This terminates filter processing for
> that traffic.  The remaining traffic is redirected unconditionally to the
> IFB device by the second filter rule.
> >
> > One thing I’m not entirely certain of is whether traffic that has been
> through an IFB device is then requeued in the normal way on the original
> device.
>
>         It should, but only on egress…
>
>
> > I’d appreciate feedback on whether this system does in fact work.
> >
> >> I would respectfully recommend to avoid the symbolic overhead parameters
> >
> > Even if I change their underlying behaviour in the future, it’ll be in a
> way that retains backwards compatibility with all the examples I’ve given
> for the current scheme.  I mostly wanted to raise awareness that the
> overhead compensation system exists for use on encapsulated links.
>
>         All fair points!
>
> >
> > - Jonathan Morton
> >
>
>
> Best Regards
>         Sebastian




-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 5941 bytes --]

Re: [Cake] cake separate qos for lan

[moeller0]

Hi Allen,

> On Mar 28, 2016, at 14:25 , Allan Pinto <allan316@gmail.com> wrote:
> 
> Hi Sebastian, 
> I should have made this more clear please see below topology with added comments. the customers connecting to the linux router can be in range from 100 to 2000, so shaping on the switch is not really a option.

	Oh, I did not advocate shaping on the switch, just on the Linux router’s interface connecting it to that switch, but it seems i misunderstood your issue, I assumed you want to shape both directions…. 

> I am right now testing on a i3 machine, but for actual live testing am planning to test with i7 or a xeon.
> 
>                   Cache-Server [ connected to internet gateway , traffic can be sent to it via wccp or policy based routing ]
>                            |
>   internet---->internet Gateway —> L2 switch [ MEN network on fiber ]   --> LInux router with cake[ includes a pppoe server which authenticates with radius ] - - [ pppoe connection over a fiber men network ]  --> customer [ customers can be 100 to 2000 ]
> 
> basically the customer will create a broadband connection on his pc to connect.
> 
> . > @Allan, what is the link technology you use?
> fiber 1g/10g/last hop cat5e

	Nice, that means you can certainly skip using pppoe-vcmux, as that is ATM/AAl5 specific, I would assume that using “overhead NN” should be more effective. Since you run the show you will know exactly which overhead to account for (keep in mind that Linux will occasionally add 14 bytes for parts of the ethernet header). It might make sense to include preamble and inter-frame gap into the per packet overhead as effectively you are shaping on ethernet IIUC.

> 
> > As I just wrote, can’t we completely avoid the IMQ/IFB here and use dual egress shaping instead (once on the pppoe device and once on the interface connected to the switch, which effectively should shape both directions)?
> 
> i may be wrong here, but i think jonathan is advising the use of IMQ/IFB to provide two different shaping scenarios on egress itself. not ingress. as i need cache traffic to have higher bandwidth on the egress towards customer but non- cache traffic [ pure internet ] to remain within the bandwidth limits purchased by the customer.

	Ah, you are right, I have not fully thought through your requirements then. I am quite curios to learn how this will work out ;) Especially since you will need to run (100 to 2000) * 2 cake instances on the router if you go for a “two shaper per customer” approach. 

Best Regards
	Sebastian

> 
> 
> 
> On Mon, Mar 28, 2016 at 5:39 PM, moeller0 <moeller0@gmx.de> wrote:
> Hi Jonathan,
> 
> > On Mar 28, 2016, at 12:31 , Jonathan Morton <chromatix99@gmail.com> wrote:
> >
> >
> >> On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
> >>
> >> it might be more future-proof to just use IFBs from the get-go
> >
> > For this particular use-case, it seems to be more complicated to use IFB than IMQ, largely because there is no iptables rule to divert packets through an IFB device, and unlike iptables, the CBQ filter mechanism doesn’t directly support negative matches of any kind.
> 
>         As I just wrote, can’t we completely avoid the IMQ/IFB here and use dual egress shaping instead (once on the pppoe device and once on the interface connected to the switch, which effectively should shape both directions)?
> 
> >
> > However, I think this would work - though it’s completely untested:
> >
> > ip link set ifb0 up
> >
> > tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth $FULL_RATE triple-isolate
> 
>         I wonder how you came up with pppoe-vcmux, I have not seen any information about the link technology in Allan’s post. As far as I know a number of (mislead) ISPs use PPPoE even on fiber links. @Allan, what is the link technology you use?
> 
> >
> > tc qdisc replace dev imq0 root handle 2: cake raw bandwidth $NONCACHE_RATE flows
> 
>         I believe this might work as egress on the interface facing the L2-switch…
> 
> >
> > tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src $CACHE_IP/32
> >
> > tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred egress redirect dev ifb0
> >
> > The logic of the above is that a positive match is made on the cache traffic, but no action is taken.  This terminates filter processing for that traffic.  The remaining traffic is redirected unconditionally to the IFB device by the second filter rule.
> >
> > One thing I’m not entirely certain of is whether traffic that has been through an IFB device is then requeued in the normal way on the original device.
> 
>         It should, but only on egress…
> 
> 
> > I’d appreciate feedback on whether this system does in fact work.
> >
> >> I would respectfully recommend to avoid the symbolic overhead parameters
> >
> > Even if I change their underlying behaviour in the future, it’ll be in a way that retains backwards compatibility with all the examples I’ve given for the current scheme.  I mostly wanted to raise awareness that the overhead compensation system exists for use on encapsulated links.
> 
>         All fair points!
> 
> >
> > - Jonathan Morton
> >
> 
> 
> Best Regards
>         Sebastian
> 
> 
> 
> -- 
> Thanx and regd's.
> 
> Allan.
> 

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 6145 bytes --]

getting a illegal filter id for these two commands,

>tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src
$CACHE_IP/32
> tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred
egress redirect dev ifb0

will check further and reply.

On Mon, Mar 28, 2016 at 6:36 PM, moeller0 <moeller0@gmx.de> wrote:

> Hi Allen,
>
> > On Mar 28, 2016, at 14:25 , Allan Pinto <allan316@gmail.com> wrote:
> >
> > Hi Sebastian,
> > I should have made this more clear please see below topology with added
> comments. the customers connecting to the linux router can be in range from
> 100 to 2000, so shaping on the switch is not really a option.
>
>         Oh, I did not advocate shaping on the switch, just on the Linux
> router’s interface connecting it to that switch, but it seems i
> misunderstood your issue, I assumed you want to shape both directions….
>
> > I am right now testing on a i3 machine, but for actual live testing am
> planning to test with i7 or a xeon.
> >
> >                   Cache-Server [ connected to internet gateway , traffic
> can be sent to it via wccp or policy based routing ]
> >                            |
> >   internet---->internet Gateway —> L2 switch [ MEN network on fiber ]
>  --> LInux router with cake[ includes a pppoe server which authenticates
> with radius ] - - [ pppoe connection over a fiber men network ]  -->
> customer [ customers can be 100 to 2000 ]
> >
> > basically the customer will create a broadband connection on his pc to
> connect.
> >
> > . > @Allan, what is the link technology you use?
> > fiber 1g/10g/last hop cat5e
>
>         Nice, that means you can certainly skip using pppoe-vcmux, as that
> is ATM/AAl5 specific, I would assume that using “overhead NN” should be
> more effective. Since you run the show you will know exactly which overhead
> to account for (keep in mind that Linux will occasionally add 14 bytes for
> parts of the ethernet header). It might make sense to include preamble and
> inter-frame gap into the per packet overhead as effectively you are shaping
> on ethernet IIUC.
>
> >
> > > As I just wrote, can’t we completely avoid the IMQ/IFB here and use
> dual egress shaping instead (once on the pppoe device and once on the
> interface connected to the switch, which effectively should shape both
> directions)?
> >
> > i may be wrong here, but i think jonathan is advising the use of IMQ/IFB
> to provide two different shaping scenarios on egress itself. not ingress.
> as i need cache traffic to have higher bandwidth on the egress towards
> customer but non- cache traffic [ pure internet ] to remain within the
> bandwidth limits purchased by the customer.
>
>         Ah, you are right, I have not fully thought through your
> requirements then. I am quite curios to learn how this will work out ;)
> Especially since you will need to run (100 to 2000) * 2 cake instances on
> the router if you go for a “two shaper per customer” approach.
>
> Best Regards
>         Sebastian
>
> >
> >
> >
> > On Mon, Mar 28, 2016 at 5:39 PM, moeller0 <moeller0@gmx.de> wrote:
> > Hi Jonathan,
> >
> > > On Mar 28, 2016, at 12:31 , Jonathan Morton <chromatix99@gmail.com>
> wrote:
> > >
> > >
> > >> On 27 Mar, 2016, at 11:20, moeller0 <moeller0@gmx.de> wrote:
> > >>
> > >> it might be more future-proof to just use IFBs from the get-go
> > >
> > > For this particular use-case, it seems to be more complicated to use
> IFB than IMQ, largely because there is no iptables rule to divert packets
> through an IFB device, and unlike iptables, the CBQ filter mechanism
> doesn’t directly support negative matches of any kind.
> >
> >         As I just wrote, can’t we completely avoid the IMQ/IFB here and
> use dual egress shaping instead (once on the pppoe device and once on the
> interface connected to the switch, which effectively should shape both
> directions)?
> >
> > >
> > > However, I think this would work - though it’s completely untested:
> > >
> > > ip link set ifb0 up
> > >
> > > tc qdisc replace dev ppp0 root handle 1: cake pppoe-vcmux bandwidth
> $FULL_RATE triple-isolate
> >
> >         I wonder how you came up with pppoe-vcmux, I have not seen any
> information about the link technology in Allan’s post. As far as I know a
> number of (mislead) ISPs use PPPoE even on fiber links. @Allan, what is the
> link technology you use?
> >
> > >
> > > tc qdisc replace dev imq0 root handle 2: cake raw bandwidth
> $NONCACHE_RATE flows
> >
> >         I believe this might work as egress on the interface facing the
> L2-switch…
> >
> > >
> > > tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip
> src $CACHE_IP/32
> > >
> > > tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action
> mirred egress redirect dev ifb0
> > >
> > > The logic of the above is that a positive match is made on the cache
> traffic, but no action is taken.  This terminates filter processing for
> that traffic.  The remaining traffic is redirected unconditionally to the
> IFB device by the second filter rule.
> > >
> > > One thing I’m not entirely certain of is whether traffic that has been
> through an IFB device is then requeued in the normal way on the original
> device.
> >
> >         It should, but only on egress…
> >
> >
> > > I’d appreciate feedback on whether this system does in fact work.
> > >
> > >> I would respectfully recommend to avoid the symbolic overhead
> parameters
> > >
> > > Even if I change their underlying behaviour in the future, it’ll be in
> a way that retains backwards compatibility with all the examples I’ve given
> for the current scheme.  I mostly wanted to raise awareness that the
> overhead compensation system exists for use on encapsulated links.
> >
> >         All fair points!
> >
> > >
> > > - Jonathan Morton
> > >
> >
> >
> > Best Regards
> >         Sebastian
> >
> >
> >
> > --
> > Thanx and regd's.
> >
> > Allan.
> >
>
>


-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 7622 bytes --]

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 28 Mar, 2016, at 15:25, Allan Pinto <allan316@gmail.com> wrote:
> 
> I should have made this more clear please see below topology with added comments. the customers connecting to the linux router can be in range from 100 to 2000, so shaping on the switch is not really a option. I am right now testing on a i3 machine, but for actual live testing am planning to test with i7 or a xeon.
> 
>                   Cache-Server [ connected to internet gateway , traffic can be sent to it via wccp or policy based routing ]
>                            |
>   internet---->internet Gateway —> L2 switch [ MEN network on fiber ]   --> LInux router with cake[ includes a pppoe server which authenticates with radius ] - - [ pppoe connection over a fiber men network ]  --> customer [ customers can be 100 to 2000 ]

I see - so you are doing something like FTTP to a block of flats, where each flat is potentially a separate customer.  That also means you have lots of virtual interfaces (each carrying one PPPoE session) over one physical interface.

> getting a illegal filter id for these two commands, 
> 
> >tc filter replace dev ppp0 protocol ip prio 1 handle 11 u32 match ip src $CACHE_IP/32
> >tc filter replace dev ppp0 protocol ip prio 2 handle 12 u32 action mirred egress redirect dev ifb0

Hmm.  Apparently filters need to be attached to a classful qdisc as a parent, which cake is not - I had hoped that the root class would act as a surrogate.  This filter system has a lot of under-documented and counter-intuitive behaviour.

Did you try the IMQ alternative?  I don’t see any reason why it shouldn’t work, as long as you can build a kernel with IMQ support.

But since you have lots of customers per router, there may still be a sensible way to do it without IMQ.  Here we step back from the idea of attaching cake instances directly to the virtual interfaces, and instead construct a complete shaping system using a classful structure on the physical device.  Each customer gets a class (and thus a cake instance which you can set the bandwidth of individually), and the cache gets its own separate class (and a cake instance).  This reduces the number of cake instances required considerably.

I believe cake has been tested as capable of handling 10Gb/sec traffic on commodity hardware.  Certainly I have personally had a very modest machine (an AMD E-450) shaping accurately at essentially GigE line rate, that being the fastest network hardware I own.  I don’t think this will be materially affected by the number of cake instances present, but as always, hard data from real experiments trumps theory.

However, by the time traffic hits the physical egress interface, it has already been encapsulated in PPPoE, making it much harder to write a filter to identify which customer’s traffic it is, or whether it came from the cache.  So we need to do the shaping on an IFB device, with the traffic redirected from the internal ingress device, where it has not yet been encapsulated.  For sanity’s sake, we should also put a basic cake instance on the physical egress port.

The following is thus a combination of ingress filtering and a hashed filter, to handle up to 1022 customers with consecutive IPv4 addresses.  Adjust as required.  Beware of the leopard.

tc qdisc replace dev $EGRESS root handle 1: cake bandwidth $LOTS besteffort flows

ip link set ifb0 up

tc qdisc replace dev $INGRESS handle ffff: ingress

tc filter replace dev $INGRESS parent ffff: protocol all u32 action mirred egress redirect dev ifb0

tc filter replace dev ifb0 parent 2:0 prio 5 protocol ip u32

tc filter add dev ifb0 parent 2:0 prio 6 handle 3: protocol ip u32 match ip src $CACHEIP flowid 2:3ff

tc qdisc replace dev ifb0 parent 2:3ff handle 5:3ff cake bandwidth $PLENTY besteffort triple-isolate

tc filter add dev ifb0 parent 2:0 prio 5 handle 4: protocol ip u32 divisor 1024

for each customer, with $HEXID in [1..3fe]:

	tc filter add dev ifb0 protocol ip parent 2:0 prio 5 u32 ht 4:${HEXID}: match ip dst $CUSTIP flowid 2:${HEXID}

	tc qdisc replace dev ifb0 parent 2:${HEXID} handle 5:${HEXID} cake bandwidth $CUSTRATE triple-isolate

tc filter add dev ifb0 protocol ip parent 2:0 prio 5 u32 ht 800:: match ip dst ${CUSTNET}/22 hashkey mask 0x000003ff at 16 link 4:

Honestly, I think the IMQ version is simpler and easier to understand, as well as catching all customer traffic.  The above mess doesn’t even handle IPv6...

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[Stephen Hemminger]

On Mon, 28 Mar 2016 22:20:58 +0300
Jonathan Morton <chromatix99@gmail.com> wrote:

> Did you try the IMQ alternative?  I don’t see any reason why it shouldn’t work, as long as you can build a kernel with IMQ support.

IMQ was abandoned by original author because there was no way to make it reliable and SMP safe.

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 29 Mar, 2016, at 00:01, Stephen Hemminger <stephen@networkplumber.org> wrote:
> 
> IMQ was abandoned by original author because there was no way to make it reliable and SMP safe.

That’s a shame, because for this use-case, it’s a heck of a lot easier to get it doing the right thing - simply because iptables is that much more flexible than the u32 filter.

But maybe u32, in the hashing configuration, scales better.

 - Jonathan Morton

Re: [Cake] cake separate qos for lan

[Toke Høiland-Jørgensen]

Jonathan Morton <chromatix99@gmail.com> writes:

> But maybe u32, in the hashing configuration, scales better.

I did a similar setup for a small ISP quite some time ago that used HTB
as the classful qdisc and a hashing u32 filter to divide traffic. This
was pre-FQ-CoDel, so it used SFQ on the leaves, but there's no reason
you couldn't just stick Cake on there.

Unfortunately I don't have the sources lying around anymore (and
probably couldn't share it if I did).

-Toke

Re: [Cake] cake separate qos for lan

[Dave Taht]

An overall ISP in tc need exposed by this discussion is some means of
mapping multiple ipv4 and ipv6 addresses and netmasks into something
that will return a (key,value) pair. This would work something like
ipset does, although what you would return is not "present or not" but
present and a value

insert customers 1.1.1.1,1
insert customers 2001:db1::/64,1
insert customers 2.2.2.2,2
insert customers 2001:db2::/64,1

then on the relevant path you'd set up the qdisc hierarchy and do a
lookup into that to get the right number to go to the right cake
instance. You'd also have to do a longest prefix match into the above,
so a 1x1 hash won't do.

the massive tc filter option discussed already does not scale to a
random number of customers with randomly different numbers of ip
addresses, types, and netmasks. Code like this must exist in dedicated
devices already, CMTSes, BRASes, deep packet inspection devices, etc.

Secondly, in the case of cake the hierarchy could just be a bunch of
somewhat unassociated queues, not htb or drr, letting cake do the
scheduling of deliveries.

Regrettably I know of few ISPs that are actively using linux in any
way they have sources to. I suspect a few dslams are linux based, but
nobody's talking.

...

Another way to maybe get there is to use the ip route functionality
instead and send stuff to virtual devices layered on top of the real
device.

ip route add from :: to 1.1.1.0/24 dev dev1
ip -6 route add from :: to 2001:db1::/64 dev dev1
ip -6 route add from :: to 2001:db2::/64 dev dev2
ip route add from :: to 2.2.2.0/24 dev dev2

Then the reverse would be out one of two devices, one device
configured for the "fast, local cache server", the other for the
regular internet.

solution too long to fit in the margins of this email.

Re: [Cake] cake separate qos for lan

[Dave Taht]

I gave this a shot, it doesn't route the packets back trhough the
underlying interface...
perhaps policy routing?

TC=/home/d/git/tc-adv/tc/tc
IFACE=eno1

F=2601:640:4103:56c0:2ab4:103a:39c5:a43a
S=2601:640:4103:56c0:120d:7fff:0:647

ip link add $IFACE name fast type dummy # maybe ifb?
ip link add $IFACE name slow type dummy # maybe ifb?

ip link set fast up
ip link set slow up

$TC qdisc add dev fast root cake bandwidth 20Mbit
$TC qdisc add dev slow root cake bandwidth 5Mbit
$TC qdisc add dev $IFACE root cake bandwidth 20Mbit

ip route add $F dev fast metric 1
ip route add $S dev slow metric 1
Dave Täht
Let's go make home routers and wifi faster! With better software!
https://www.gofundme.com/savewifi


On Tue, Mar 29, 2016 at 4:31 PM, Dave Taht <dave.taht@gmail.com> wrote:
> An overall ISP in tc need exposed by this discussion is some means of
> mapping multiple ipv4 and ipv6 addresses and netmasks into something
> that will return a (key,value) pair. This would work something like
> ipset does, although what you would return is not "present or not" but
> present and a value
>
> insert customers 1.1.1.1,1
> insert customers 2001:db1::/64,1
> insert customers 2.2.2.2,2
> insert customers 2001:db2::/64,1
>
> then on the relevant path you'd set up the qdisc hierarchy and do a
> lookup into that to get the right number to go to the right cake
> instance. You'd also have to do a longest prefix match into the above,
> so a 1x1 hash won't do.
>
> the massive tc filter option discussed already does not scale to a
> random number of customers with randomly different numbers of ip
> addresses, types, and netmasks. Code like this must exist in dedicated
> devices already, CMTSes, BRASes, deep packet inspection devices, etc.
>
> Secondly, in the case of cake the hierarchy could just be a bunch of
> somewhat unassociated queues, not htb or drr, letting cake do the
> scheduling of deliveries.
>
> Regrettably I know of few ISPs that are actively using linux in any
> way they have sources to. I suspect a few dslams are linux based, but
> nobody's talking.
>
> ...
>
> Another way to maybe get there is to use the ip route functionality
> instead and send stuff to virtual devices layered on top of the real
> device.
>
> ip route add from :: to 1.1.1.0/24 dev dev1
> ip -6 route add from :: to 2001:db1::/64 dev dev1
> ip -6 route add from :: to 2001:db2::/64 dev dev2
> ip route add from :: to 2.2.2.0/24 dev dev2
>
> Then the reverse would be out one of two devices, one device
> configured for the "fast, local cache server", the other for the
> regular internet.
>
> solution too long to fit in the margins of this email.

Re: [Cake] cake separate qos for lan

[Allan Pinto]

[-- Attachment #1: Type: text/plain, Size: 3884 bytes --]

>I gave this a shot, it doesn't route the packets back trhough the
>underlying interface...
>perhaps policy routing?

tried it with policy routing too, packets get dropped when i point them to
the ifb interfaces fast/slow. tried both dummy and ifb.

ip route add $custip dev fast table 10
ip route add $custip dev slow table 11
ip rule add from $cacheip to $custip lookup 10 priority 10
ip rule add from all to $custip lookup 11 priority 11

im not sure if we can route it to ifb this way,( maybe only works with tc
redirection?) as i feel that i have just set up a routing loop.

i did not have time to recompile the kernel for imq, as jonathan mentioned
but will try that too,
did not try the ifb method mentioned by jonathan yet.



On Wed, Mar 30, 2016 at 5:46 AM, Dave Taht <dave.taht@gmail.com> wrote:

> I gave this a shot, it doesn't route the packets back trhough the
> underlying interface...
> perhaps policy routing?
>
> TC=/home/d/git/tc-adv/tc/tc
> IFACE=eno1
>
> F=2601:640:4103:56c0:2ab4:103a:39c5:a43a
> S=2601:640:4103:56c0:120d:7fff:0:647
>
> ip link add $IFACE name fast type dummy # maybe ifb?
> ip link add $IFACE name slow type dummy # maybe ifb?
>
> ip link set fast up
> ip link set slow up
>
> $TC qdisc add dev fast root cake bandwidth 20Mbit
> $TC qdisc add dev slow root cake bandwidth 5Mbit
> $TC qdisc add dev $IFACE root cake bandwidth 20Mbit
>
> ip route add $F dev fast metric 1
> ip route add $S dev slow metric 1
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> https://www.gofundme.com/savewifi
>
>
> On Tue, Mar 29, 2016 at 4:31 PM, Dave Taht <dave.taht@gmail.com> wrote:
> > An overall ISP in tc need exposed by this discussion is some means of
> > mapping multiple ipv4 and ipv6 addresses and netmasks into something
> > that will return a (key,value) pair. This would work something like
> > ipset does, although what you would return is not "present or not" but
> > present and a value
> >
> > insert customers 1.1.1.1,1
> > insert customers 2001:db1::/64,1
> > insert customers 2.2.2.2,2
> > insert customers 2001:db2::/64,1
> >
> > then on the relevant path you'd set up the qdisc hierarchy and do a
> > lookup into that to get the right number to go to the right cake
> > instance. You'd also have to do a longest prefix match into the above,
> > so a 1x1 hash won't do.
> >
> > the massive tc filter option discussed already does not scale to a
> > random number of customers with randomly different numbers of ip
> > addresses, types, and netmasks. Code like this must exist in dedicated
> > devices already, CMTSes, BRASes, deep packet inspection devices, etc.
> >
> > Secondly, in the case of cake the hierarchy could just be a bunch of
> > somewhat unassociated queues, not htb or drr, letting cake do the
> > scheduling of deliveries.
> >
> > Regrettably I know of few ISPs that are actively using linux in any
> > way they have sources to. I suspect a few dslams are linux based, but
> > nobody's talking.
> >
> > ...
> >
> > Another way to maybe get there is to use the ip route functionality
> > instead and send stuff to virtual devices layered on top of the real
> > device.
> >
> > ip route add from :: to 1.1.1.0/24 dev dev1
> > ip -6 route add from :: to 2001:db1::/64 dev dev1
> > ip -6 route add from :: to 2001:db2::/64 dev dev2
> > ip route add from :: to 2.2.2.0/24 dev dev2
> >
> > Then the reverse would be out one of two devices, one device
> > configured for the "fast, local cache server", the other for the
> > regular internet.
> >
> > solution too long to fit in the margins of this email.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 
Thanx and regd's.

Allan.

[-- Attachment #2: Type: text/html, Size: 5238 bytes --]

Re: [Cake] cake separate qos for lan

[Jonathan Morton]

> On 31 Mar, 2016, at 14:49, Allan Pinto <allan316@gmail.com> wrote:
> 
> im not sure if we can route it to ifb this way,( maybe only works with tc redirection?) as i feel that i have just set up a routing loop.

Both IFB and IMQ use the original routing information to direct the packet to the original device after passing through.  There is thus no second pass through the routing tables (or the firewall).

I think to use routing tables in any way for this purpose, you would need to use a virtual-ethernet-pair device, not IFB or IMQ.  Consider that a fallback option if we can’t get anything else to work.

I’m also thinking of cleaner ways to do this by means of adding code, rather than masses of configuration.  It really shouldn’t be this hard to produce a working setup.

 - Jonathan Morton