[Cake] Ingress classification
John Sager
john at sager.me.uk
Tue Feb 5 08:38:09 EST 2019
As you say, an unsolicited incoming packet doesn't get marked. However it
creates a conntrack record with zero mark. What you then do is to mark the
conntrack record later so that all subsequent packets on that connection get
marked by 'action connmark'. So the first packet gets classified on ifb to
some low priority queue, but subsequent ones go where they should.
I do this for incoming ssh and VPN connections, though I'm using
htb/fq_codel rather than cake at the moment.
John
On 05/02/2019 12:08, Kevin Darbyshire-Bryant wrote:
> Hi Experts!
>
> After a long self imposed absence I’m back…and as confused as ever. I’m trying to use cake’s diffserv/dscp interpretation abilities with my Internet facing wan link on openwrt.
>
> The chances of getting DSCP across the Internet unmolested are nill, but I can use dscp codes within my own LAN/Router to select relevant tins and hence bandwidth/latency limits within cake. e.g. to de-prioritise certain host/port combinations (bittorrent into bulk)
>
> egress traffic is ‘easy’, simply use an iptables rule e.g. as produced by fw3 on openwrt:
>
> /etc/config/firewall:
>
> config rule
> option name 'Bittorrent server DSCP BULK'
> option target 'dscp'
> option src 'lan'
> option family 'ipv4'
> option src_ip '192.168.218.12'
> option dest '*'
> option set_dscp 'CS1'
> option src_port '6989'
> option proto 'tcp udp'
>
> Produces:
>
> iptables -t mangle -A PREROUTING -p tcp -s 192.168.218.12/255.255.255.255 -m tcp --sport 6989 -m comment --comment "!fw3: Bittorrent server DSCP BULK" -j DSCP --set-dscp 0x08
> iptables -t mangle -A PREROUTING -p udp -s 192.168.218.12/255.255.255.255 -m udp --sport 6989 -m comment --comment "!fw3: Bittorrent server DSCP BULK" -j DSCP --set-dscp 0x08
>
> root at Router:~# iptables -t mangle -L
> Chain PREROUTING (policy ACCEPT)
> target prot opt source destination
> DSCP tcp -- Waldorf.lan.darbyshire-bryant.me.uk anywhere tcp spt:6989 /* !fw3: Bittorrent server DSCP BULK */ DSCP set 0x08
> DSCP udp -- Waldorf.lan.darbyshire-bryant.me.uk anywhere udp spt:6989 /* !fw3: Bittorrent server DSCP BULK */ DSCP set 0x08
>
> So that’s the egress (upload) sorted out ‘easy’. Of course the tcp ack packets on the return (ingress) path aren’t classified as BULK nor is any bittorrent download traffic.
>
>
> Ingress is harder and my knowledge goes flaky at this point too. The qdisc is running at a point pre de-NAT, this was the ‘flaw’ with host fairness in the NAT world before the ’nat’ keyword. Similarly I also think that any rules to change DSCP at this point have also happened too late for cake to notice. Then Toke shone a light and gave Cake tc filter knowledge and so for a while I had the following in a tweaked ‘layer_cake.qos’ script:
>
> MAJOR=$( tc qdisc show dev $DEV | head -1 | awk '{print $3}' )
> # catch bittorrent incoming
> $TC filter add dev $DEV parent $MAJOR protocol ip u32 \
> match ip dport 6989 0xffff action skbedit priority ${MAJOR}1
>
> This catches any traffic destined to port 6989 and tells cake to put it into the Bulk Tin. So I have a solution that is putting traffic coming from my bittorrent host on port 6989 or going to port 6989 into the bulk tin. It’s not exactly scaleable though.
>
> Then I saw someone using connmark [1] and I’ve subsequently tweaked that into my own version and it appears to work but… I don’t really know/understand why - hence this email!
>
> One of the things that openwrt’s sqm-scripts does to enable ingress QoS is to redirect ingress traffic on the WAN interface to an ifb interface…and you put the qdisc on the egress of the ifb to effectively create an ingress qdisc. The mirror/redirect function is implemented by ‘action mirred redirect dev target_dev’. Linux/TC also allows you to restore connection tracking marks at this stage as well, hence the following line:
>
> # redirect all IP packets arriving in $IFACE to ifb0
> # and restore connmark
> $TC filter add dev $IFACE parent ffff: protocol all prio 10 u32 \
> match u32 0 0 flowid 1:1 action connmark action mirred egress redirect dev $DEV
>
> With the connmarks restored TC filters can be told to ‘do stuff’ based on those marks, in this case edit the skb’s priority field so now Cake stuffs the packet into the selected tin(1/3):
>
> MAJOR=$( tc qdisc show dev $DEV | head -1 | awk '{print $3}' )
> $TC filter add dev $DEV parent $MAJOR protocol ip \
> handle 0x01 fw action skbedit priority ${MAJOR}1 #handle mark 0x01
> $TC filter add dev $DEV parent $MAJOR protocol ip \
> handle 0x03 fw action skbedit priority ${MAJOR}3 #handle mark 0x03
>
> As I understand it, this isn’t changing the DSCP field on the packet itself but just an skb priority that Cake understands (something to do with Major numbers matching)
>
>
> So now all we have to do is mark our connections as required. And here’s where my knowledge trouble really starts:
>
> # Configure iptables chain to mark packets
> ipt -t mangle -N QOS_MARK_${IFACE}
>
> # If packets have DSCP 8 then mark that connection so the response packets are also sent to bulk
> ipt -t mangle -A QOS_MARK_${IFACE} -p tcp -m dscp --dscp 8 -j MARK --set-mark 0x01/0xff
> ipt -t mangle -A QOS_MARK_${IFACE} -p udp -m dscp --dscp 8 -j MARK --set-mark 0x01/0xff
>
> #kdb high priority marking
> ipt -t mangle -A QOS_MARK_${IFACE} -p tcp -m dscp --dscp 0x30 -j MARK --set-mark 0x03/0xff
> ipt -t mangle -A QOS_MARK_${IFACE} -p udp -m dscp --dscp 0x30 -j MARK --set-mark 0x03/0xff
>
> This makes sense so far, basically anything that has a matching dscp already, set the corresponding connmark.
>
> Then the original script does this:
>
> # You can add more rules here. For example, to mark incomming connections on port 9999 tcp/udp
> # ipt -t mangle -A QOS_MARK_${IFACE} -i $IFACE -p tcp --dport 9999 -j MARK --set-mark 0x01/0xff
> # ipt -t mangle -A QOS_MARK_${IFACE} -i $IFACE -p udp --dport 9999 -j MARK --set-mark 0x01/0xff
>
> I’m confused at this point - see later.
>
> # save the connmark - will be 0x00 unless any of the above changed it
> ipt -t mangle -A QOS_MARK_${IFACE} -j CONNMARK --save-mark
>
> That makes sense, I get that.
>
> And finally after all that effort of making the chain, we’d better actually use it:
>
> # Send unmarked connections to the marking chain
> ipt -t mangle -A PREROUTING -i $IFACE -m mark --mark 0x00/0xff -g QOS_MARK_${IFACE}
> ipt -t mangle -A POSTROUTING -o $IFACE -m mark --mark 0x00/0xff -g QOS_MARK_${IFACE}
>
>
> And so, finally, we come to my point of confusion and my question: How/Does setting a firewall mark on ingress actually work because I didn’t think the prerouting mangle table would have run before tc’s action connmark action mirred.
>
> Somehow it does appear to be working but I genuinely don’t understand how say an unsolicited ingress UDP packet (from a bittorrent) would have been marked.
>
> Sorry for the ‘War & Peace’ length. I’ve tried reading iptables tutorials and a tc filters tutorial [2] but I can’t quite put it all together.
>
>
> [1] https://github.com/dtaht/sch_cake/issues/97#issuecomment-412248518
> [2] http://linux-ip.net/gl/tc-filters/tc-filters.html
>
> Cheers,
>
> Kevin D-B
>
> 012C ACB2 28C6 C53E 9775 9123 B3A2 389B 9DE2 334A
>
> _______________________________________________
> Cake mailing list
> Cake at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
More information about the Cake
mailing list