[Cake] act_connmark + dscp

[Kevin Darbyshire-Bryant]

> On 9 Mar 2019, at 14:08, Toke Høiland-Jørgensen <toke at redhat.com> wrote:
> 
> Kevin Darbyshire-Bryant <kevin at darbyshire-bryant.me.uk> writes:
> 
>> OK, what I am trying to do is classify incoming connections into
>> relevant cake tins to impose some bandwidth fairness.  e.g. classify
>> bittorrent & things that are downloads into the Bulk tin, and
>> prioritise stuff that is streaming video into the Video tin. Incoming
>> DSCP has a) been washed and b) is unreliable anyway so is unhelpful in
>> this case.  iptables runs too late, so having rules to classify
>> incoming stuff is pointless.
> 
> Right, I see.
> 
> [... snip .. ]
> 
>> Then I recently discovered act_connmark
>> (http://linux-ip.net/gl/tc-filters/tc-filters-node2.html) - the
>> thinking being I could use iptables on egress to set fwmarks to
>> classify a connection and have the ingress packets magically follow.
>> This worked but still required 3 tc filter actions to cope with 4
>> tins:
>> 
>> $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x01 fw action skbedit priority ${MAJOR}1
>> $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x03 fw action skbedit priority ${MAJOR}3
>> $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x04 fw action skbedit priority ${MAJOR}4
> 
> Right, so this can be replaced with the fwmark action we already added
> (and that I just pushed an update to so it supports masking the value
> before selecting a tin).

Yes.  I’d point out the (hopefully obvious) that the flag mask needs to be one bit bigger than you might immediately think.  e.g. diffserv4 needs to store 5 values (0-4), 3 bits. 0 is being used as an implied ’tin is not set, fall back to DSCP’.  One could store DSCP+1 of course and use the same logic.

> 
>> The overriding (if required) of DSCP could be done in iptables and to
>> avoid going through the iptables DSCP decision/mangling for every
>> packet I could use a flag within the fwmark to indicate the decision
>> had previously been made and stored for this connection.
> 
> [ ... ]
> 
>> I’m doing 2 things.
>> 
>> 1) Classifying traffic into tins on ingress based on the egress DSCP
>> values contained in fwmarks.
>> 
>> 2) Basing the fwmark contained DSCP on the initial packet of the
>> connection, possibly after being modified once by iptables rules.
> 
> So I tried prototyping what it would actually look like to do all this
> in iptables. The result is below (in iptables-restore format). I haven't
> tested it, but I believe something along the lines of this will work,
> when used along with the CAKE fwmark support (setting a mask of 0xFF
> when configuring CAKE).
> 
> Now, the obvious eyesore on this is the need to replicate CAKEs diffserv
> mappings in iptables rules (21 rules in this case, for the diffserv4
> mapping). As long as this only runs once per connection I don't actually
> think it's much of a performance issue for normal use, but obviously
> there could be pathological cases, and it's also annoying to have to do
> that.
> 
> So, first question becomes: Do you agree that the firewall rules below
> would solve your use case (ignoring the ugliness of having to replicate
> the diffserv parsing in iptables)? Or am I missing something?

I’ve had a quick look over it and think it would work.

The ugliness of doing the diffserv parsing is what prompted the idea of storing the DSCP directly and I felt the stored tin selection was effectively abstracting the diffserv field anyway.  Storing the DSCP is more compatible with differing egress v ingress mappings (eg. egress diffserv4, ingress diffserv3 though I can’t really think of a use case for that)

Of course using fwmark as tin number selector in cake doesn’t preclude some other mechanism of storing & recovering DSCP to/from firewall mark e.g. the previously discussed act-connmark+dscp which would help anyone who wanted to do such ‘link traversing’ DSCP shenanigans.  That of course makes you happier since cake doesn’t embed itself further into conntrack.

> 
> -Toke
> 
> 
> 
> *mangle
> :PREROUTING ACCEPT [0:0]
> :APPLY-MARKS - [0:0]
> :MARK-DSCP - [0:0]
> :MARK-POLICY - [0:0]
> 
> # Run on inside iface - eth0 in this case
> -A PREROUTING -i eth0 -J APPLY-MARKS
> 
> # Make sure we have the marks from conntrack
> -A APPLY-MARKS -J CONNMARK --restore-mark --nfmask 0xFF --ctmask 0xFF
> 
> # Abort if our "already set" bit is set
> -A APPLY-MARKS -m mark --mark 0x80/0x80 -j RETURN
> 
> # If a DSCP value is set, use DSCP-based marking
> -A APPLY-MARKS -m dscp ! --dscp 0 -J MARK-DSCP
> 
> # Otherwise, set our own policy
> -A APPLY-MARKS -m dscp --dscp 0 -J MARK-POLICY
> 
> # Set our "already set" bit, and store things back into conntrack
> -A APPLY-MARKS -J MARK --set-mark 0x80/0x80
> -A APPLY-MARKS -J CONNMARK --save-mark --nfmask 0xFF --ctmask 0xFF
> 
> # DSCP-based fwmark setting
> # add whatever DSCP coverage you want here; the below is CAKE's diffserv4
> # fwmark offsets into CAKE's tin_order, and are 1-indexed, so tin selection
> # is offset by 1, and bulk and besteffort are swapped compared to the array
> # in sch_cake.c
> # We assume CAKE is configured with an fwmark mask of 0x7F
> 
> -A MARK-DSCP -m dscp --dscp 1 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 4 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 8 -J MARK --set-mark 0x1/0x7F
> -A MARK-DSCP -m dscp --dscp 16 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 18 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 20 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 22 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 24 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 26 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 28 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 30 -J MARK --set-mark 0x3/0x7F
> -A MARK-DSCP -m dscp --dscp 34 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 36 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 38 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 32 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 40 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 44 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 46 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 48 -J MARK --set-mark 0x4/0x7F
> -A MARK-DSCP -m dscp --dscp 56 -J MARK --set-mark 0x4/0x7F
> # default best effort
> -A MARK-DSCP -m mark --mark 0x0/0x7F -J --set-mark 0x2/0x7F
> 
> 
> # Policy-based fwmark setting
> -A MARK-POLICY -p tcp -s 192.168.219.5 -m comment --comment "Skybox Bulk" -j MARK --set-mark 0x1/0x7F
> # add more policy rules here; anything not marked will use the DSCP bits
> # of each packet; optional catch-all to avoid that and make everything BE:
> #-A MARK-DSCP -m mark --mark 0x0/0x7F -J --set-mark 0x2/0x7F
>