From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E90BB3BA8E for ; Sat, 9 Mar 2019 09:08:29 -0500 (EST) Received: by mail-ed1-f54.google.com with SMTP id d12so403378edp.0 for ; Sat, 09 Mar 2019 06:08:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=0XRxnoX1o41wNagkZ7uIj9FBryMFQXkWgws0rYZqzEE=; b=kvAIzIFHfgp5KnR5wFczd/fX/CB8brIXWiS+gXtmPuJVZlxZsD06Mu+20Cr2VzlJ0b LaXgFpSYymOWUEC4+92pcswOXjc3U9LnR5lNSAF7qli0k082kIyDrW1pQKYWboKtZvCn 9w2nPE2ht2noCucM/WlsrpIxUmcFScyboPy45eUje6gXb5zuaMg6OqF6ZGYmktTiju9Q ajEyArzU2TIDy1spejSHZTnBYqwlDkRCZ01Rv+ceLur933u35lkcZyiKaIBnq6z4iPG0 mzlyz1vKvSG3HVdJoiOJoyBq5HBvaXXoK35KMN8ravsBiq6/+hWVFCWg/dBl3H9LzAnQ 6n3Q== X-Gm-Message-State: APjAAAX8gdAzAB7E2YZFnyZvyR2TTtmEDSdkLM4dLQ/lc6fMdAWFfB2s 8PVRYGHI7YWEfYGIqXGT27+h/w== X-Google-Smtp-Source: APXvYqw65t+yXxowjI4zYb7G3KyvVOQIZg5yzNKN8X993Jn/5DJbbADcBmKCb3+J/U5oeXdS/Z+nGQ== X-Received: by 2002:a17:906:e56:: with SMTP id q22mr15729639eji.132.1552140508850; Sat, 09 Mar 2019 06:08:28 -0800 (PST) Received: from alrua-x1.borgediget.toke.dk (borgediget.toke.dk. [85.204.121.218]) by smtp.gmail.com with ESMTPSA id y45sm2994427edd.33.2019.03.09.06.08.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 09 Mar 2019 06:08:28 -0800 (PST) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id BABFC1804A0; Sat, 9 Mar 2019 15:08:27 +0100 (CET) From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= To: Kevin Darbyshire-Bryant Cc: "cake\@lists.bufferbloat.net" In-Reply-To: <7FFADD0C-591A-4BB4-B96C-C0157963E1EB@darbyshire-bryant.me.uk> References: <875zsw110r.fsf@toke.dk> <6B530473-971A-4265-B94B-3595D39D57AF@darbyshire-bryant.me.uk> <87r2bjyoyn.fsf@toke.dk> <4505E3A0-6AE2-4C0B-960D-B1EDB616F0CA@darbyshire-bryant.me.uk> <878sxq1t3e.fsf@toke.dk> <00E839ED-7FA4-4577-838F-775EC9A90608@darbyshire-bryant.me.uk> <871s3h7ghi.fsf@toke.dk> <7FFADD0C-591A-4BB4-B96C-C0157963E1EB@darbyshire-bryant.me.uk> X-Clacks-Overhead: GNU Terry Pratchett Date: Sat, 09 Mar 2019 15:08:27 +0100 Message-ID: <87ef7g5eec.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] act_connmark + dscp X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2019 14:08:30 -0000 Kevin Darbyshire-Bryant writes: > OK, what I am trying to do is classify incoming connections into > relevant cake tins to impose some bandwidth fairness. e.g. classify > bittorrent & things that are downloads into the Bulk tin, and > prioritise stuff that is streaming video into the Video tin. Incoming > DSCP has a) been washed and b) is unreliable anyway so is unhelpful in > this case. iptables runs too late, so having rules to classify > incoming stuff is pointless. Right, I see. [... snip .. ] > Then I recently discovered act_connmark > (http://linux-ip.net/gl/tc-filters/tc-filters-node2.html) - the > thinking being I could use iptables on egress to set fwmarks to > classify a connection and have the ingress packets magically follow. > This worked but still required 3 tc filter actions to cope with 4 > tins: > > $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x01 fw action= skbedit priority ${MAJOR}1 > $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x03 fw action= skbedit priority ${MAJOR}3 > $TC filter add dev $IFACE parent $MAJOR protocol ip handle 0x04 fw action= skbedit priority ${MAJOR}4 Right, so this can be replaced with the fwmark action we already added (and that I just pushed an update to so it supports masking the value before selecting a tin). > The overriding (if required) of DSCP could be done in iptables and to > avoid going through the iptables DSCP decision/mangling for every > packet I could use a flag within the fwmark to indicate the decision > had previously been made and stored for this connection. [ ... ] > I=E2=80=99m doing 2 things. > > 1) Classifying traffic into tins on ingress based on the egress DSCP > values contained in fwmarks. > > 2) Basing the fwmark contained DSCP on the initial packet of the > connection, possibly after being modified once by iptables rules. So I tried prototyping what it would actually look like to do all this in iptables. The result is below (in iptables-restore format). I haven't tested it, but I believe something along the lines of this will work, when used along with the CAKE fwmark support (setting a mask of 0xFF when configuring CAKE). Now, the obvious eyesore on this is the need to replicate CAKEs diffserv mappings in iptables rules (21 rules in this case, for the diffserv4 mapping). As long as this only runs once per connection I don't actually think it's much of a performance issue for normal use, but obviously there could be pathological cases, and it's also annoying to have to do that. So, first question becomes: Do you agree that the firewall rules below would solve your use case (ignoring the ugliness of having to replicate the diffserv parsing in iptables)? Or am I missing something? -Toke *mangle :PREROUTING ACCEPT [0:0] :APPLY-MARKS - [0:0] :MARK-DSCP - [0:0] :MARK-POLICY - [0:0] # Run on inside iface - eth0 in this case -A PREROUTING -i eth0 -J APPLY-MARKS # Make sure we have the marks from conntrack -A APPLY-MARKS -J CONNMARK --restore-mark --nfmask 0xFF --ctmask 0xFF # Abort if our "already set" bit is set -A APPLY-MARKS -m mark --mark 0x80/0x80 -j RETURN # If a DSCP value is set, use DSCP-based marking -A APPLY-MARKS -m dscp ! --dscp 0 -J MARK-DSCP # Otherwise, set our own policy -A APPLY-MARKS -m dscp --dscp 0 -J MARK-POLICY # Set our "already set" bit, and store things back into conntrack -A APPLY-MARKS -J MARK --set-mark 0x80/0x80 -A APPLY-MARKS -J CONNMARK --save-mark --nfmask 0xFF --ctmask 0xFF # DSCP-based fwmark setting # add whatever DSCP coverage you want here; the below is CAKE's diffserv4 # fwmark offsets into CAKE's tin_order, and are 1-indexed, so tin selection # is offset by 1, and bulk and besteffort are swapped compared to the array # in sch_cake.c # We assume CAKE is configured with an fwmark mask of 0x7F -A MARK-DSCP -m dscp --dscp 1 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 4 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 8 -J MARK --set-mark 0x1/0x7F -A MARK-DSCP -m dscp --dscp 16 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 18 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 20 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 22 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 24 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 26 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 28 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 30 -J MARK --set-mark 0x3/0x7F -A MARK-DSCP -m dscp --dscp 34 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 36 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 38 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 32 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 40 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 44 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 46 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 48 -J MARK --set-mark 0x4/0x7F -A MARK-DSCP -m dscp --dscp 56 -J MARK --set-mark 0x4/0x7F # default best effort -A MARK-DSCP -m mark --mark 0x0/0x7F -J --set-mark 0x2/0x7F # Policy-based fwmark setting -A MARK-POLICY -p tcp -s 192.168.219.5 -m comment --comment "Skybox Bulk" -= j MARK --set-mark 0x1/0x7F # add more policy rules here; anything not marked will use the DSCP bits # of each packet; optional catch-all to avoid that and make everything BE: #-A MARK-DSCP -m mark --mark 0x0/0x7F -J --set-mark 0x2/0x7F