From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E8ADD3B2A3 for ; Fri, 7 Apr 2017 07:13:17 -0400 (EDT) Received: from [172.17.3.29] ([134.76.241.253]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0M9K5G-1coAdO0xcE-00CfYd; Fri, 07 Apr 2017 13:13:16 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) From: Sebastian Moeller In-Reply-To: Date: Fri, 7 Apr 2017 13:13:15 +0200 Cc: Jonathan Morton , Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <193C2284-31C2-4FDC-AA2D-DD19BE7E9E22@gmx.de> References: <2FD59D30-3102-4A3E-A38E-050E438DABF0@gmail.com> <6F118C46-16DB-48AC-A90D-7E6D44B6D069@gmail.com> <1E4563E2-63E2-419D-AFDD-8CD74F22539B@gmail.com> To: Pete Heist X-Mailer: Apple Mail (2.3273) X-Provags-ID: V03:K0:Fur5CUEpHT+xWyEC41HuiRGIdtVFgDyhy2L6EGfMBNf3VpJ/UlW 7hJ8jVGWeX6gOZ8XmhbRQs5aRuRmMSQCZd+EjsOnoPTjIrj66/Kkmm4ny+3gOv6AlZwM8z0 h/lbgfMofw6WjxvMD8mdx5iVl6XJ3e9x1RUtP4/y+HMf++sJ3KObdxnRRnHK/J0DqnlNskh 51cvUhumTcI4P3yxnV6ww== X-UI-Out-Filterresults: notjunk:1;V01:K0:KJUZWJRwsHQ=:0AX7IWO4xZee9XVz7Sg/VK XonMe+gsr88etRkw94dnX/Swp+9trFnHGeIauU2Dljdy9a5cntr1UmcpS+lHvBdMxKnU9iLO1 MDQfSlu6aMAlPm0pXzvcLwCbjrOrmBrVpFXahtZa93W13PhG21ZvF1a9dP11FMNcgn/IpniM9 5DyCwVBBBNDkaqwQqK1Q83ptadBa5Ihdtql6/H+jbvokncLxA4rqPYYRSmvEenaeiEStu+52+ SQGDaPEeka0vdRPi2cgQIbv/gexXxrIxEMOZEmNb0QvbdXvo8CPrrx0Cpiq0iD4OpnfD7YUmy XTH43D2YvsaZha2+3C4jsV4FY34fBKlxVa+4bqAxxdZOq8ixiX48jlE6Ip5NfyqPZIj88cFuj 444jLY1fAL+rTL3MQ6kCKF7YMM63Cy+CEXzlwkXuGqO9HiJA/Q4sUtR7991FdezdItl6/xnFr +/DMRIAAFOMVrBYQWjrEdUQ1VTXoEdjUq4DSmNwAfISWEmTgdKFJ+lT3qVwtLhlKLlH6zj7L4 yaGk1YmyJZScQzDGtTHi1wHjPOyZpP/hf88xbGxzKpVOyoqsDbNdufIja75gi0HO87vFSmurV rRyUoKdlrkz+tgf+ywKP9QmpqBj/GeuHqdEUUOWB0g0YnfHkpYHtt+Z6xgS3hSlTV1xuNCPOu LaYYpwpGvi34RSPWQnw3XF0JWwdXOxRYeZaNjB3y3eZn46l//4+oyjUZu3Dc1Bt6aYADk7qe+ w80951Dh6xAnPq3Odxl84z3vlFvRqcwb3ZkNqXKixhQjXM68Ym+sHl86Z0X2eEsaxKxFcHwoY Xvr1rmW Subject: Re: [Cake] flow isolation for ISPs X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Apr 2017 11:13:18 -0000 Hi Peter, > On Apr 7, 2017, at 11:37, Pete Heist wrote: >=20 >>=20 >> On Apr 7, 2017, at 10:28 AM, Jonathan Morton wrote: >>>=20 >>> On 7 Apr, 2017, at 11:13, Pete Heist wrote: >>>=20 >>>> On Apr 6, 2017, at 11:26 AM, Pete Heist = wrote: >>>>=20 >>>>> On Apr 6, 2017, at 11:11 AM, Jonathan Morton = wrote: >>>>>=20 >>>>> On 6 Apr, 2017, at 11:27, Pete Heist wrote: >>>>>>=20 >>>>>> There is a table of member ID to a list of MAC addresses for the = member, so if there could somehow be fairness based on that table and by = MAC address, that could solve it, but I don=E2=80=99t see how it could = be implemented. >>>>>=20 >>>>> One option would be to use HTB with FLOWER filters to sort out the = subscribers into classes, and use Cake or fq_codel as a child qdisc per = class. Remember that Cake can be used in =E2=80=9Cunlimited=E2=80=9D = mode to rely on an external shaping source. >>>=20 >>> One more thought, would it be possible for Cake to optionally = include the packet=E2=80=99s mark in the hash? >>>=20 >>> I know it=E2=80=99s additional functionality, and another keyword, = but it could get you out of the business of the myriad of ways people = might want to do flow isolation, and you=E2=80=99d still have a = catch-all answer for such cases. >>>=20 >>> There could be a keyword =E2=80=98hash-mark=E2=80=99, let=E2=80=99s = say, which first includes the mark in the hash, then does on to deal = with any other flow isolation keywords as usual. So for example if I = have =E2=80=98hash-mark=E2=80=99 and =E2=80=98dual-srchost=E2=80=99, the = hash is first on the mark, then by source host, then by flow. I could = set the mark to be the member number with iptables. >>=20 >> That isn=E2=80=99t really how hashing works; there is no =E2=80=9Cfirst= , second, third=E2=80=9D structure, just an accumulation of entropy = which is all mashed together. In order to run the triple-isolation = algorithm at all, I have to take separate hashes of the relevant host = addresses, alongside the general 5-tuple hash. >>=20 >> However, it would be possible to use the =E2=80=9Cmark=E2=80=9D = directly as one of the host identifiers which triple-isolate operates on = to provide that layer of fairness. That=E2=80=99s probably what you = meant. >>=20 >> Since this wouldn=E2=80=99t unduly complicate the configuration = interface, it could be a feasible way of adding this functionality for = modest installations, up to a strict maximum of 1024 subscribers (and a = recommended maximum somewhat below that). >=20 > Ok, I=E2=80=99m still getting familiar with how triple-isolate is = implemented. For example, I was surprised in my test setup that no = fairness is enforced when four client IPs connect to a single server IP, = but I understand from this discussion = (https://github.com/dtaht/sch_cake/issues/46) that that is actually what = is expected. We would probably use dual-srchost and dual-dsthost in the = backhaul, which seems to work very well, and in the backhaul we have the = information to specify that in both directions. (Also, there is no NAT = to deal with at this level.) >=20 > Just to see if I understand the marking proposal, here's the behavior = I would expect: if there are two TCP flows (on egress) with mark 1 and = one with mark 2, that together saturate the link, the measured rate of = the two flows with mark 1 will add up to the rate of the single flow = with mark 2. Is that right? And would you still add a keyword to specify = that the mark should be used at all? >=20 > I=E2=80=99m not sure where the 1024 limit comes from, but it would = probably be fine in our case as of now, with 800 members. Even in the = future, I don=E2=80=99t think occasional collisions would be a big = problem, and I think there are things we could do to minimize them. Seeing your 800 members I remember a discussion over at the lede forum, = https://forum.lede-project.org/t/lede-as-a-dedicated-qos-bufferbloat-appli= ance/1861/27?u=3Dmoeller0 where orangetek, used cake on a wired backhaul = for approximately 600 end users. He reported for number of concurrent = flows: =E2=80=9CAs far as i can tell, around 25k-30k during busy = hours.=E2=80=9D=20 He also increased the number of CAKE_BINs in the code to 64k. So = depending on your user=E2=80=99s 1024 might be a bit tight, given that = you still ideally want flows to not share bins if possible (sure cake is = great in avoiding sharing unless impossible, but with enough flows you = might want/need to simply hard code your cake instances for higher = limits). Best Regards >=20 >>> It looks like the mark could be obtained from the =E2=80=98mark' = field of the sk_buff struct, but I don=E2=80=99t know the validity of = the field in various cases. For example, I don=E2=80=99t think I can set = the mark on ingress before it reaches a qdisc on an IFB device. >>=20 >> It has been suggested, in the context of using the =E2=80=9Cmark=E2=80=9D= for Diffserv purposes, that Linux=E2=80=99 conntrack facility could = preserve the mark between directions of flow. Cake can already query = conntrack for NAT awareness. >=20 > That would be nice for the future, but for now I guess this wouldn=E2=80= =99t work on ingress. It shouldn=E2=80=99t be much of a problem in the = backhaul though, because we=E2=80=99re the ones sending the downstream = traffic, and we can set the marks on that. >=20 > Overall, I think this could be a nice feature. Let me know if I can = help in some way and thank you for your feedback. :) >=20 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake