From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 55C3B3BA8E for ; Tue, 27 Nov 2018 15:42:39 -0500 (EST) Received: by mail-qt1-x829.google.com with SMTP id y20so23377397qtm.13 for ; Tue, 27 Nov 2018 12:42:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=k8HVPKMJmu8QYCB4ClyZhnJIA1dcSovzY15dt0EEIN4=; b=n94qq86fk/OcWfZBTS4AEMML2tZRNP28zAR5uWPtYNlvwvaNCdNIJoeOA6Yso6STGP PnKe/N5fWFhk/De3WDngNCCKr702Jusy7w/MEoJ3QlTx89oK9YwmscUv+Dd2ay3aPZVg sigPLxunittkToc5UjOIMlZOqLVfWZJsGm67VYpwwPjoqA1acM0ejoGbuzyKJRElKtRk x2VB5sBRUTbzHmCFmjCk2CcmxDc3cpL9IYW0NZHRtKJeNad//sbxGxrgmCOVPdzOvUo5 iqV2sORJcUw268HA1ipbDJl3ZcaxbvxMqj7wMnkFSLxVr7UX+MNJlNaRiKkZ0ec94THO 7Cqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=k8HVPKMJmu8QYCB4ClyZhnJIA1dcSovzY15dt0EEIN4=; b=rqyca7OBqe1YadqjRn8PPidwB/pElZgQfoWqH2k+uApS4/IvnOjWqlK0yO3b/zrUmc ZC6aknCU7bYE/6GppJ1Y003+hzxbJdX9q3gMhQ1u37gVgbQH8BSHl5ZDmuGaZRGd4XZs 755nvrT3aTsZqUwMQNMNkZ70ph4qlFOfHx1Y4lzmg/UyX13rn3V5Bo6n0b9rpu6N47SK 4RBZNGu2u/rdIJudkeOi3Y0pRNkgOgOgWfXI9N9FoSvBSH5X2D3jVkOQ2fnyws8WWSoG bKmYSufuVrp9qWZjErBdhvpffwrV/xbAQACvFcukcPs8PuYUaPrBUaaRKemrKzVVrXnA vSWA== X-Gm-Message-State: AA+aEWblhiIyK02NCTbnxU56mkQI3d1JwZoqZHDN57+dFAkE04J+uU+1 vRx53CBqlk/4CniNdjLOCcPH2Oo0zyQjTKLhwNw= X-Google-Smtp-Source: AFSGD/X/eSdK3C4Srqmy44yCpmrYqaqNu99jq3jO7XkhLNcr0WUTEZskMrXN3Isu1WdzgQM494eQT8O0inzzIJEDEBY= X-Received: by 2002:a0c:afd1:: with SMTP id t17mr32297180qvc.93.1543351358649; Tue, 27 Nov 2018 12:42:38 -0800 (PST) MIME-Version: 1.0 References: <65EAC6C1-4688-46B6-A575-A6C7F2C066C5@heistp.net> In-Reply-To: From: Dave Taht Date: Tue, 27 Nov 2018 12:42:26 -0800 Message-ID: To: Neal Cardwell Cc: Pete Heist , bloat Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] when does the CoDel part of fq_codel help in the real world? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2018 20:42:39 -0000 On Mon, Nov 26, 2018 at 11:28 AM Neal Cardwell wrote= : > > I believe Dave Taht has pointed out, essentially, that the "codel" part o= f fq_codel can be useful in cases where the definition of "flow" is not vis= ible to fq_codel, so that "fq" part is inactive. For example, if there is V= PN traffic, where the individual flows are not separable by fq_codel, then = it can help to have "codel" AQM for the aggregate of encrypted VPN traffic.= I imagine this rationale could apply where there is any kind of encapsulat= ion or encryption that makes the notion of "flow" opaque to fq_codel. Yep. This should end up a bullet point somewhere. The bsd implementation of fq_codel does not do anywhere near the dpi that linux does to extract the relevant fields. That's a truly amazing amount of sub-protocol hashing, sometimes I sit back and have to both admire and shudder at this: https://elixir.bootlin.com/linux/v4.18.6/source/net/core/flow_dissector.c#L= 578 and it's even more frightful if you look at the list of ethernet types and protocols that are not dissected. Codeling opaque flows is useful. We will always have opaque flows. Some protocol types cannot be fq'd due to design - babel for example mixes up hellos with route transfers. I've done things like measure induced latency on wireguard streams of late and codel keeps it sane. still, wireguard internally is optimized for single flow "dragster" performance, and I'd like it to gain the same fq_codel optimization that did such nice things for multiple flows terminating on the router for ipsec. The before/after on that was marvelous. Another problem people are perpetually pointing out is hashing is expensive, and me, saying it's "free" with hardware support for it, with examples like most modern 10GigE ethernet cards doing it on-board, hardware assists for sha and references to the FPGA lit, such as: https://zistvan.github.io/doc/trets15-istvan.pdf still... hashing is sw expensive. Recently I went on a quest for a better hash than jenkins which is used throughout the kernel today. cityhash and murmur for example. > > neal > > > On Mon, Nov 26, 2018 at 2:08 PM Pete Heist wrote: >> >> In Toke=E2=80=99s thesis defense, there was an interesting exchange with= examination committee member Michael (apologies for not catching the last = name) regarding how the CoDel part of fq_codel helps in the real world: >> >> https://www.youtube.com/watch?v=3Dupvx6rpSLSw&t=3D2h16m20s >> >> My attempt at a transcript is at the end of this message. (I probably wo= n=E2=80=99t attempt a full defense transcript, but if someone wants more of= a particular section I can try. :) >> >> So I just thought to continue the discussion- when does the CoDel part o= f fq_codel actually help in the real world? I=E2=80=99ll speculate with a f= ew possibilities: >> >> 1) Multiplexed HTTP/2.0 requests containing both a saturating stream and= interactive traffic. For example, a game that uses HTTP/2.0 to download ne= w map data while position updates or chat happen at the same time. Standalo= ne programs could use HTTP/2.0 this way, or for web apps, the browser may m= ultiplex concurrent uses of XHR over a single TCP connection. I don=E2=80= =99t know of any examples. >> >> 2) SSH with port forwarding while using an interactive terminal together= with a bulk transfer? >> >> 3) Does CoDel help the TCP protocol itself somehow? For example, does it= speed up the round-trip time when acknowledging data segments, improving b= ehavior on lossy links? Similarly, does it speed up the TCP close sequence = for saturating flows? >> >> Pete >> >> --- >> >> M: In fq_codel what is really the point of CoDel? >> T: Yeah, uh, a bit better intra-flow latency... >> M: Right, who cares about that? >> T: Apparently some people do. >> M: No I mean specifically, what types of flows care about that? >> T: Yeah, so, um, flows that are TCP based or have some kind of- like, el= astic flows that still want low latency. >> M: Elastic flows that are TCP based that want low latency... >> T: Things where you want to discover the- like, you want to utilize the = full link and sort of probe the bandwidth, but you still want low latency. >> M: Can you be more concrete what kind of application is that? >> T: I, yeah, I=E2=80=A6 >> M: Give me any application example that=E2=80=99s gonna benefit from the= CoDel part- CoDel bits in fq_codel? Because I have problems with this. >> T: I, I do too... So like, you can implement things this way but equival= ently if you have something like fq_codel you could, like, if you have a vi= deo streaming application that interleaves control=E2=80=A6 >> M: that runs on UDP often. >> T: Yeah, but I, Netflix=E2=80=A6 >> M: Ok that=E2=80=99s a long way=E2=80=A6 >> T: No, I tend to agree with you that, um=E2=80=A6 >> M: Because the biggest issue in my opinion is, is web traffic- for web t= raffic, just giving it a huge queue makes the chance bigger that uh, so you may end up with a (higher) fast= er completion time by buffering a lot. Uh, you=E2=80=99re not benefitting a= t all by keeping the queue very small, you are simply Right, yo= u=E2=80=99re benefitting altogether by just which is what the q= ueue does with this nice sparse flow, uh=E2=80=A6 >> T: You have the infinite buffers in the for that to work, ri= ght. One benefit you get from CoDel is that - you screw with things like - = you have to drop eventually. >> M: You should at some point. The chances are bigger that the small flow = succeeds (if given a huge queue). And, in web surfing, why does that, uh(?) >> T: Yeah, mmm... >> M: Because that would be an example of something where I care about late= ncy but I care about low completion. Other things where I care about latenc= y they often don=E2=80=99t send very much. bursts, you have = to accommodate them basically. Or you have interactive traffic which is UDP= and tries to, often react from queueing delay . I=E2=80=99m beg= inning to suspect that fq minus CoDel is really the best out th= ere. >> T: But if, yeah, if you have enough buffer. >> M: Well, the more the better. >> T: Yeah, well. >> M: Haha, I got you to say yes. [laughter :] That goes in history. I said= the more the better and you said yeah. >> T: No but like, it goes back to good-queue bad-queue, like, buffering in= itself has value, you just need to manage it. >> M: Ok. >> T: Which is also the reason why just having a small queue doesn=E2=80=99= t help in itself. >> M: Right yeah. Uh, I have a silly question about fq_codel, a very silly = one and there may be something I missed in the papers, probably I did, but = I'm I was just wondering I mean first of all this is also a bit silly in th= at it=E2=80=99s a security thing, and I think that=E2=80=99s ki= nd of a package by itself silly because fq_codel often probably = just in principle, is that something I could easily attack by creating new= flows for every packet? >> T: No because, they, you will=E2=80=A6 >> M: With the sparse flows, and it=E2=80=99s gonna=E2=80=A6 >> T: Yeah, but at some point you=E2=80=99re going to go over the threshold= , I, you could, there there=E2=80=99s this thing where the flow goes in, it= =E2=80=99s sparse, it empties out and then you put it on the normal round r= obin implementation before you queue And if you don=E2=80=99t d= o that than you can have, you could time packets so that they get priority = just at the right time and you could have lockout. >> M: Yes. >> T: But now you will just fall back to fq. >> M: Ok, it was just a curiousity, it=E2=80=99s probably in the paper. >> T: I think we added that in the RFC, um, you really need to, like, this = part is important. >> >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740