From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 1CF9F3B29E for ; Fri, 27 Jul 2018 14:58:44 -0400 (EDT) Received: by mail-lj1-x22d.google.com with SMTP id p10-v6so5276004ljg.2 for ; Fri, 27 Jul 2018 11:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NFWbI21Q0gZNRb1VmacfNxLRXhkfdUit8E71QcFX//E=; b=jkHwx2cAwRtjdniSAwRnK43jj2syJY1BinjEKiaJWsLpCQatjpJxG6GRSrS4zvOozv zO8cBCSTFPYLDBmlfjT+GM1CxXXB6B9s5cK6720baH+5PzQ5R0VfPUSOCMdH8txpipWv 9qnjnFN63u45vQNm+rPJRHuA40Vq8xOzCfbUXruHMnxdp06qGXGZdWbNCuglRWxeTQAZ qibxHVYypgaeAzSc71wtSqidZD4Iop/DxkfGC7YdpxXjZY52Zc4O1fHp2FdHlXR3H7xH uyu5j4YYdRdEKLAy0scg8GEDsi8qXMgUTy4C/RH3zXGjQiukez/aI16CXodfEFKb+bZx 6WrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NFWbI21Q0gZNRb1VmacfNxLRXhkfdUit8E71QcFX//E=; b=MdPYucE+aILn3rW2+Y3EccVneky8es4O9OOCDgxEXOEYFYk65O4UAElOdW/aHxZ4NX 5PVfe+1XiPmmH9CtjsXLsYxZnYlO1Gkimw0/1gcGKkjwCkDKDyHjebLWYdobaZqend9s mvDpF2ilALl4BvglIpBG9it/3XuDTEB4SuJi35L4und05mFyWvI021yTB98hBKdR+ic0 cWr5asl5FYlxGWwR/CHHXdmZfuqrbWSspsnEdBxWiB3Dj1zAPAN36zy5xzvXjaMWfFWf yo5RVEGGsac820DZtTHC2GthHaML9g6bztKvS+7YCT375/zxUB3Q8h00v2ZhKPIMH2sB x3QQ== X-Gm-Message-State: AOUpUlGgs6JhH0q10IH4sFbTSkmxlnLAm2Cs+VvmLjpoMPy9llQ+focc Yry6BrpnekKSKq/ZHbWFpcE= X-Google-Smtp-Source: AAOMgpe7SwcC30y+yY6aS0qemDZXHtXhP8UmdP0oshyvyHyee6h6c3D2c+T1sPzyunMz8vOptyk6Vg== X-Received: by 2002:a2e:5012:: with SMTP id e18-v6mr5665172ljb.22.1532717923009; Fri, 27 Jul 2018 11:58:43 -0700 (PDT) Received: from [192.168.239.216] (83-245-232-26-nat-p.elisa-mobile.fi. [83.245.232.26]) by smtp.gmail.com with ESMTPSA id w12-v6sm798135ljw.60.2018.07.27.11.58.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Jul 2018 11:58:42 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Jonathan Morton In-Reply-To: Date: Fri, 27 Jul 2018 21:58:40 +0300 Cc: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Dave Taht , Cake List Content-Transfer-Encoding: quoted-printable Message-Id: References: <1357421162.31089.1531812291583@webmail.strato.de> <1c323544b3076c0ab31b887d6113f25f572e41ae.camel@coverfire.com> <87woth28rw.fsf@toke.dk> <87tvol1z6h.fsf@toke.dk> To: Dan Siemon X-Mailer: Apple Mail (2.3445.9.1) Subject: Re: [Cake] =?utf-8?q?Using_cake_to_shape_1000=E2=80=99s_of_users=2E?= X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2018 18:58:44 -0000 > On 27 Jul, 2018, at 5:04 pm, Dan Siemon wrote: >=20 > Obviously I can't speak for other potential users but we follow the > upstream kernel very aggressively and have no interest in porting > something like this to older kernels. That's a useful data point. Honestly it shouldn't be too difficult to = install the latest kernels on a dedicated box in general. > We have some deployments with multiple access technologies (eg DOCSIS, > DSL and wireless) behind the same box so per customer overhead would = be > useful. The design I presently have in mind would allow setting the overhead per = speed tier. Thus you could have "10Mbit DOCSIS" and "10Mbit ADSL" as = separate tiers. That is, there's likely to be far fewer unique overhead = settings than customers. > I am interested in the diffserv class ideas in Cake but need to do = some > experimentation to see if that helps in this environment. It would be > interesting to be able to target sub-classes (not sure what the proper > Cake terminology is) based on a eBPF classifier. The trick with Diffserv is classifying the traffic correctly, given that = most traffic in the wild is not properly marked, and some actively tries = to hide its nature. As an ISP, applying your own rules could be seen as = questionable according to some interpretations of Net Neutrality. With = Cake that's not an issue by default, since it relies entirely on the = DSCP on the packet itself, but it does mean that a lot of traffic which = *should* probably be classified other than best-effort is not. Cake does cope well with unclassified latency-sensitive traffic, due to = DRR++, so it's not actually necessary to specifically identify such = applications. What it has trouble with, in common with all other = flow-isolating qdiscs, is applications which open many parallel = connections for bulk transfer (eg. BitTorrent, which is used by many = software-update mechanisms as well as file-sharing). Classifying this = traffic as Bulk (CS1 DSCP) gets it out of the way of other applications = while still permitting them to use available capacity. I have some small hope that wider deployment of sensible, basic Diffserv = at bottlenecks will encourage applications to start using it as = intended, thereby solving a decades-long chicken-egg problem. Presently = Cake has such an implementation by default, but is not yet widely = deployed enough to have any visible impact. This is something we should probably discuss in more detail later. > Re accounting, we currently count per-IP via eBPF not via qdisc = counts. Fine, that's a feature I'm happy to leave out. > Each subscriber can have several IPs in which case the traffic for > those IPs needs to go to a single class so that their entire traffic > envelope is shaped to the desired plan rate. At present we do this via > eBPF since it is essentially a map lookup operation. >=20 > A few of the above points touch on something that may be somewhat > unique service provider deployments vs homes, there are many = situations > where the classification logic has to be a map lookup and cannot be > done just by looking at a given packet. Yes, eBPF does seem to be a good fit for that. So in summary, the logical flow of a packet should be: 1: Map dst or src IP to subscriber (eBPF). 2: Map subscriber to speed/overhead tier (eBPF). 3: (optional) Classify Diffserv (???). 4: Enqueue per flow, handle global queue overflow (rare!) by dropping = from head of longest queue (like Cake). --- enqueue/dequeue split --- 5: Wait for shaper on earliest-scheduled subscriber link with waiting = traffic (borrow sch_fq's rbtree?). 6: Wait for shaper on aggregate backhaul link (congestion can happen = here too). 7: Choose one of subscriber's queues, apply AQM and deliver a packet = (mostly like Cake does). If that seems reasonable, we can treat it as a baseline spec for others' = input. - Jonathan Morton