[Cake] Ingress classification

Kevin Darbyshire-Bryant kevin at darbyshire-bryant.me.uk
Tue Feb 5 07:08:30 EST 2019

Hi Experts!

After a long self imposed absence I’m back…and as confused as ever.  I’m trying to use cake’s diffserv/dscp interpretation abilities with my Internet facing wan link on openwrt.

The chances of getting DSCP across the Internet unmolested are nill, but I can use dscp codes within my own LAN/Router to select relevant tins and hence bandwidth/latency limits within cake.  e.g. to de-prioritise certain host/port combinations (bittorrent into bulk)

egress traffic is ‘easy’, simply use an iptables rule e.g. as produced by fw3 on openwrt:


config rule                                      
        option name 'Bittorrent server DSCP BULK'
        option target 'dscp'                     
        option src 'lan'                         
        option family 'ipv4'                     
        option src_ip ''           
        option dest '*'                          
        option set_dscp 'CS1'                    
        option src_port '6989'                   
        option proto 'tcp udp'


iptables -t mangle -A PREROUTING -p tcp -s -m tcp --sport 6989 -m comment --comment "!fw3: Bittorrent server DSCP BULK" -j DSCP --set-dscp 0x08
iptables -t mangle -A PREROUTING -p udp -s -m udp --sport 6989 -m comment --comment "!fw3: Bittorrent server DSCP BULK" -j DSCP --set-dscp 0x08

root at Router:~# iptables -t mangle -L
target     prot opt source               destination         
DSCP       tcp  --  Waldorf.lan.darbyshire-bryant.me.uk  anywhere             tcp spt:6989 /* !fw3: Bittorrent server DSCP BULK */ DSCP set 0x08
DSCP       udp  --  Waldorf.lan.darbyshire-bryant.me.uk  anywhere             udp spt:6989 /* !fw3: Bittorrent server DSCP BULK */ DSCP set 0x08

So that’s the egress (upload) sorted out ‘easy’.  Of course the tcp ack packets on the return (ingress) path aren’t classified as BULK nor is any bittorrent download traffic.

Ingress is harder and my knowledge goes flaky at this point too.  The qdisc is running at a point pre de-NAT, this was the ‘flaw’ with host fairness in the NAT world before the ’nat’ keyword.  Similarly I also think that any rules to change DSCP at this point have also happened too late for cake to notice. Then Toke shone a light and gave Cake tc filter knowledge and so for a while I had the following in a tweaked ‘layer_cake.qos’ script:

MAJOR=$( tc qdisc show dev $DEV | head -1 | awk '{print $3}' )
# catch bittorrent incoming
    $TC filter add dev $DEV parent $MAJOR protocol ip u32 \
      match ip dport 6989 0xffff action skbedit priority ${MAJOR}1

This catches any traffic destined to port 6989 and tells cake to put it into the Bulk Tin.  So I have a solution that is putting traffic coming from my bittorrent host on port 6989 or going to port 6989 into the bulk tin.  It’s not exactly scaleable though.

Then I saw someone using connmark [1] and I’ve subsequently tweaked that into my own version and it appears to work but… I don’t really know/understand why - hence this email!

One of the things that openwrt’s sqm-scripts does to enable ingress QoS is to redirect ingress traffic on the WAN interface to an ifb interface…and you put the qdisc on the egress of the ifb to effectively create an ingress qdisc.  The mirror/redirect function is implemented by ‘action mirred redirect dev target_dev’.  Linux/TC also allows you to restore connection tracking marks at this stage as well, hence the following line:

# redirect all IP packets arriving in $IFACE to ifb0                                      
# and restore connmark                                                        
$TC filter add dev $IFACE parent ffff: protocol all prio 10 u32 \                         
    match u32 0 0 flowid 1:1 action connmark action mirred egress redirect dev $DEV 

With the connmarks restored TC filters can be told to ‘do stuff’ based on those marks, in this case edit the skb’s priority field so now Cake stuffs the packet into the selected tin(1/3):

MAJOR=$( tc qdisc show dev $DEV | head -1 | awk '{print $3}' )                            
$TC filter add dev $DEV parent $MAJOR protocol ip  \                                      
        handle 0x01 fw action skbedit priority ${MAJOR}1 #handle mark 0x01                          
$TC filter add dev $DEV parent $MAJOR protocol ip  \                                      
        handle 0x03 fw action skbedit priority ${MAJOR}3 #handle mark 0x03

As I understand it, this isn’t changing the DSCP field on the packet itself but just an skb priority that Cake understands (something to do with Major numbers matching)

So now all we have to do is mark our connections as required.  And here’s where my knowledge trouble really starts:

# Configure iptables chain to mark packets                                                                
    ipt -t mangle -N QOS_MARK_${IFACE}                                                        
# If packets have DSCP 8 then mark that connection so the response packets are also sent to bulk
    ipt -t mangle -A QOS_MARK_${IFACE} -p tcp -m dscp --dscp 8 -j MARK --set-mark 0x01/0xff         
    ipt -t mangle -A QOS_MARK_${IFACE} -p udp -m dscp --dscp 8 -j MARK --set-mark 0x01/0xff         

#kdb high priority marking
    ipt -t mangle -A QOS_MARK_${IFACE} -p tcp -m dscp --dscp 0x30 -j MARK --set-mark 0x03/0xff      
    ipt -t mangle -A QOS_MARK_${IFACE} -p udp -m dscp --dscp 0x30 -j MARK --set-mark 0x03/0xff   

This makes sense so far, basically anything that has a matching dscp already, set the corresponding connmark.

Then the original script does this:

# You can add more rules here. For example, to mark incomming connections on port 9999 tcp/udp  
    # ipt -t mangle -A QOS_MARK_${IFACE} -i $IFACE -p tcp --dport 9999 -j MARK --set-mark 0x01/0xff 
    # ipt -t mangle -A QOS_MARK_${IFACE} -i $IFACE -p udp --dport 9999 -j MARK --set-mark 0x01/0xff

I’m confused at this point - see later.

# save the connmark - will be 0x00 unless any of the above changed it                                                                            
    ipt -t mangle -A QOS_MARK_${IFACE} -j CONNMARK  --save-mark

That makes sense, I get that.

And finally after all that effort of making the chain, we’d better actually use it:

# Send unmarked connections to the marking chain
    ipt -t mangle -A PREROUTING  -i $IFACE -m mark --mark 0x00/0xff -g QOS_MARK_${IFACE}
    ipt -t mangle -A POSTROUTING -o $IFACE -m mark --mark 0x00/0xff -g QOS_MARK_${IFACE}

And so, finally, we come to my point of confusion and my question:  How/Does setting a firewall mark on ingress actually work because I didn’t think the prerouting mangle table would have run before tc’s action connmark action mirred.

Somehow it does appear to be working but I genuinely don’t understand how say an unsolicited ingress UDP packet (from a bittorrent) would have been marked.

Sorry for the ‘War & Peace’ length.  I’ve tried reading iptables tutorials and a tc filters tutorial [2] but I can’t quite put it all together.

[1] https://github.com/dtaht/sch_cake/issues/97#issuecomment-412248518
[2] http://linux-ip.net/gl/tc-filters/tc-filters.html


Kevin D-B

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

More information about the Cake mailing list