From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 302743BA8E for ; Mon, 27 Nov 2017 06:04:23 -0500 (EST) Received: by mail-wr0-x236.google.com with SMTP id o14so25991149wrf.9 for ; Mon, 27 Nov 2017 03:04:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:message-id:date:to:mime-version; bh=czAPEcutDto/w3ijqor5suwTRxyefRzwPFnuYL5fADQ=; b=D3e8UWLxYrA0X9AcR0gvy8cW5aBBD8aPDEiNsBIyzB2qX8xWueuofYujy35RcuUwUC iVDCaH8i9SlRcxCTiQQgvRixP51nX6WMNe7Eh2twRH2Qi+BDFfsx401LlJNlTW8vQsOx tax0GTf3tLnD2YHu9AnM014jEDcartXkRIPuwluNl1rHcao6nQEdjGJFEg7FL8TfluA2 oA9jQNsYOJgd+L7cKIo65ekYkj5aaLFiaaXNK1vyR7hIPwIFqHOlIazAqsDve7KbcFaH NI8Dc3VIgXLtMH6UvsFQdAU84kixfDyszauK7/D2RgimocjBU8HafkiPncgCXixTKLb7 6Itg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:message-id:date:to:mime-version; bh=czAPEcutDto/w3ijqor5suwTRxyefRzwPFnuYL5fADQ=; b=fxCjuP4Cz0btJ9sLzgTJWas/QJ/By/zEDLDxEoR2G9a+0G4X1zco9KVI4W/gJWEcna 1YwIrwaw2ZL0yiMYrYVDeG2eRcqYoibSQi3TEU0uIWwOVvCsBOFO+OIr75CB/SJ8NujR l/qSVXLCo7SrnnCUAAZdB9ajWIZkNHwNWwCSaieMTmSqbhPfc5wpMTLuYcxsiFD5mCl/ tnhLe2lUdto8R2CZw8k/YCXq+Dx/ZCHAFtHtPr2g03Kw0pf2nBy1NPWmd8Bmr48+BHFv 6jRYwhP1RrmByEBjOSSmtYyxjDniXT2ceZfg+hEqtXJKR/ZYyjQ2Qhd2vRZQpGK8MJ4I tBzA== X-Gm-Message-State: AJaThX42sW7N8Jg3B5dZgmiSrQ/qOquyDtW3Ye+6Chlen4Wan+RGecpA lLl14QgJ/M7g1u2f7Wl9I2r0DdpW X-Google-Smtp-Source: AGs4zMahXHN3D0jMuNkvb4qT+Q+oEUd+YL8lucff1oQoOA9mAtt5Hpv/bNDtfInDVCfpm8BCxlbxBg== X-Received: by 10.223.166.235 with SMTP id t98mr31346068wrc.251.1511780661692; Mon, 27 Nov 2017 03:04:21 -0800 (PST) Received: from [10.72.0.130] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id p42sm50862742wrb.28.2017.11.27.03.04.20 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 27 Nov 2017 03:04:20 -0800 (PST) From: Pete Heist Content-Type: multipart/alternative; boundary="Apple-Mail=_89852D60-7F70-45D4-A770-A966C4D98DB2" Message-Id: <85E1A7B2-8AA7-418A-BE43-209A1EC8881A@gmail.com> Date: Mon, 27 Nov 2017 12:04:19 +0100 To: cake@lists.bufferbloat.net Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) Subject: [Cake] cake flenter results round 1 X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 11:04:23 -0000 --Apple-Mail=_89852D60-7F70-45D4-A770-A966C4D98DB2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 http://www.drhleny.cz/bufferbloat/cake/round1/ = Round 1 Tarball: http://www.drhleny.cz/bufferbloat/cake/round1.tgz = Round 0 Tarball (previous run): = http://www.drhleny.cz/bufferbloat/cake/round0.tgz = *** Notes/Analysis *** * New bql tests show the effectiveness of cake=E2=80=99s TSO/GSO/GRO = =E2=80=9Cpeeling=E2=80=9D vs fq_codel? Or am I seeing an mq artifact on = my 4-queue device? = http://www.drhleny.cz/bufferbloat/cake/round1/bql_csrt_rrulbe_eg_fq_codel_= nolimit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/bql_csrt_rrulbe_eg_cakeeth_n= olimit/index.html = * Cake holds TCP RTT to half that of fq_codel at 10mbit bandwidth. I = like to call this technique of rate limiting well below the = interface=E2=80=99s maximum "over-limiting=E2=80=9D, which seems to work = well with stable point-to-point WiFi connections. (Of course, = point-to-multipoint or unstable rates requires the new ath9k/10k driver = changes as limiting in this way would not be effective, well explained = here- https://www.youtube.com/watch?v=3DRb-UnHDw02o = ): = http://www.drhleny.cz/bufferbloat/cake/round1/eg_csrt_rrulbe_eg_sfq_10.0mb= it/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/eg_csrt_rrulbe_eg_fq_codel_1= 0.0mbit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/eg_csrt_rrulbe_eg_cakeeth_10= .0mbit/index.html = * Cake at 950mbit performed just as well as fq_codel, vs the round0 runs = where fq_codel had a bit of an advantage. Perhaps the addition of the = =E2=80=9Cethernet=E2=80=9D keyword did this? = http://www.drhleny.cz/bufferbloat/cake/round1/eg_csrt_rrulbe_eg_fq_codel_9= 50mbit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/eg_csrt_rrulbe_eg_cakeeth_95= 0mbit/index.html = ** I=E2=80=99m finding the "32 Flows, RRUL Best-Effort=E2=80=9D tests = fascinating to look at. It might be possible to spot implementation = differences between fq_codel and cake from these. * At 10mbit, cake and fq_codel are better at most things than sfq by an = order of magnitude or more. But interestingly, at this bandwidth = fq_codel=E2=80=99s results look a bit better than cake, where total = bandwidth for fq_codel is higher (4.78/9.12mbit for fq_codel and = 3.91/8.63mbit for cake) and ping latency a bit lower (1.79ms vs 1.92ms), = and TCP RTT significantly better (~30ms vs ~45 ms). Maybe cake's = =E2=80=9Cethernet=E2=80=9D keyword at these low bandwidths affects a = test like this disproportionally? = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_sfq_10.0mbit/inde= x.html = = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_fq_codel_10.0mbit= /index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_cakeeth_10.0mbit/= index.html = * At 100mbit, the situation reverses, with fq_codel TCP RTT above 10ms = and cake around 4.75ms. = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_fq_codel_100mbit/= index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_cakeeth_100mbit/i= ndex.html = * And then above 200mbit, fq_codel performs considerably better than = cake at the 32/32 flow tests. At 900mbit, UDP/ping is 1.1ms for fq_codel = and 10ms for cake. TCP RTT is ~6.5ms for fq_codel and ~12ms for cake. = Dave=E2=80=99s earlier explanation probably applies here: "Since = fq_codel supports superpackets and cake peels them, we have a cpu and = latency hit that originates from that. Also the code derived algorithm = in cake differs quite significantly from mainline codel, and my = principal gripe about it has been that it has not been extensively = tested against higher delays." = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_fq_codel_900mbit/= index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/32flows_eg_cakeeth_900mbit/i= ndex.html = * On the Cake RTT tests, we take about a 15% hit in total TCP throughput = at rtt 1ms vs rtt 10ms (1454mbit vs 1700mbit), and a 55% hit at rtt = 100us (which is why you=E2=80=99d probably only consider using that on = 10gbit links). If we don=E2=80=99t remove the =E2=80=98ethernet=E2=80=99 = keyword altogether, I guess I=E2=80=99d like to see it at least be 10ms, = as TCP RTT only goes from around 0.8ms to 1.8ms, which I don=E2=80=99t = think makes a huge latency difference in real world terms. Or it might = be another argument for removing datacentre, ethernet and metro = altogether, because there are tradeoffs to decide about. = http://www.drhleny.cz/bufferbloat/cake/round1/cake_rtt_10ms_rrulbe_eg_cake= _900mbit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/cake_rtt_1ms_rrulbe_eg_cake_= 900mbit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/cake_rtt_100us_rrulbe_eg_cak= e_900mbit/index.html = * I wonder if the UDP flood tests really work at 900mbit: = http://www.drhleny.cz/bufferbloat/cake/round1/udpflood_eg_fq_codel_900mbit= /index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/udpflood_eg_cakeeth_900mbit/= index.html = * Again as before, I=E2=80=99m surprised that srchost/dsthost is much = more fair. Numbers that follow are 1-flow/12-flow throughput. For = srchost/dsthost, it=E2=80=99s 413/439mbit up, 413/447 down and for = dual-srchost/dual-dsthost it=E2=80=99s 126/647mbit up, 77/749mbit down. = Rampant speculation: does this have to do with the =E2=80=9Cpeeling=E2=80=9D= ? And should we / do we even do peeling with soft rate limiting? I think = I saw it help with bql(?), but I=E2=80=99m not sure I=E2=80=99ve seen it = help when rate limited below the interface=E2=80=99s rate. = http://www.drhleny.cz/bufferbloat/cake/round1/hostiso_eg_cake_src_cake_dst= _900mbit/index.html = = http://www.drhleny.cz/bufferbloat/cake/round1/hostiso_eg_cake_dsrc_cake_dd= st_900mbit/index.html = * I still need a better understanding of what triple-isolate does. It = isn=E2=80=99t clear to me from the man page. Results here are similar to = dual-srchost/dual-dsthost: = http://www.drhleny.cz/bufferbloat/cake/round1/hostiso_eg_cake_dsrc_cake_dd= st_900mbit/index.html = *** Round 2 Plans *** - Add bql tests to anywhere rate limiting is used - Add ethernet keyword to host isolation tests - Add ethtool output to host info - Remove or improve flow isolation tests - Add host isolation tests with rtt variation (to look again at problem = I reported in an earlier thread) *** Future Plans *** - Use netem to make a spread of rtts and bandwidths - Add VoIP tests (I hope to do this with irtt) - Add ack filtering tests - Test BBR - Use qemu to test other archs (I may never get to this, honestly) --Apple-Mail=_89852D60-7F70-45D4-A770-A966C4D98DB2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 http://www.drhleny.cz/bufferbloat/cake/round1/


Round 0 Tarball = (previous run): http://www.drhleny.cz/bufferbloat/cake/round0.tgz

*** = Notes/Analysis ***

* New bql tests show the effectiveness of = cake=E2=80=99s TSO/GSO/GRO =E2=80=9Cpeeling=E2=80=9D vs fq_codel? = Or am I seeing an mq artifact on my 4-queue device?


* Cake holds TCP RTT to half that of = fq_codel at 10mbit bandwidth. I like to call this technique of rate = limiting well below the interface=E2=80=99s maximum "over-limiting=E2=80=9D= , which seems to work well with stable point-to-point WiFi connections. = (Of course, point-to-multipoint or unstable rates requires the new = ath9k/10k driver changes as limiting in this way would not be effective, = well explained here- https://www.youtube.com/watch?v=3DRb-UnHDw02o):


* Cake at 950mbit performed just as = well as fq_codel, vs the round0 runs where fq_codel had a bit of an = advantage. Perhaps the addition of the =E2=80=9Cethernet=E2=80=9D = keyword did this?


** I=E2=80=99m finding the "32 Flows, = RRUL Best-Effort=E2=80=9D tests fascinating to look at. It might be = possible to spot implementation differences between fq_codel and cake = from these.

* = At 10mbit, cake and fq_codel are better at most things than sfq by an = order of magnitude or more. But interestingly, at this bandwidth = fq_codel=E2=80=99s results look a bit better than cake, where total = bandwidth for fq_codel is higher (4.78/9.12mbit for fq_codel and = 3.91/8.63mbit for cake) and ping latency a bit lower (1.79ms vs 1.92ms), = and TCP RTT significantly better (~30ms vs ~45 ms). Maybe cake's = =E2=80=9Cethernet=E2=80=9D keyword at these low bandwidths affects a = test like this disproportionally?


* At 100mbit, the situation reverses, with fq_codel TCP RTT = above 10ms and cake around 4.75ms.


* And then above 200mbit, fq_codel performs considerably = better than cake at the 32/32 flow tests. At 900mbit, UDP/ping is 1.1ms = for fq_codel and 10ms for cake. TCP RTT is ~6.5ms for fq_codel and ~12ms = for cake. Dave=E2=80=99s earlier explanation probably applies here: = "Since fq_codel supports superpackets and cake peels them, we have a cpu = and latency hit that originates from that. Also the code derived = algorithm in cake differs quite significantly from mainline codel, and = my principal gripe about it has been that it has not been extensively = tested against higher delays."


* On the Cake RTT tests, we take about a 15% hit in total TCP = throughput at rtt 1ms vs rtt 10ms (1454mbit vs 1700mbit), and a 55% hit = at rtt 100us (which is why you=E2=80=99d probably only consider using = that on 10gbit links). If we don=E2=80=99t remove the =E2=80=98ethernet=E2= =80=99 keyword altogether, I guess I=E2=80=99d like to see it at least = be 10ms, as TCP RTT only goes from around 0.8ms to 1.8ms, which I = don=E2=80=99t think makes a huge latency difference in real world terms. = Or it might be another argument for removing datacentre, ethernet and = metro altogether, because there are tradeoffs to decide about.


* I wonder if the UDP flood tests = really work at 900mbit:


* Again as before, I=E2=80=99m surprised that srchost/dsthost = is much more fair. Numbers that follow are 1-flow/12-flow throughput. = For srchost/dsthost, it=E2=80=99s 413/439mbit up, 413/447 down and for = dual-srchost/dual-dsthost it=E2=80=99s 126/647mbit up, 77/749mbit down. = Rampant speculation: does this have to do with the =E2=80=9Cpeeling=E2=80=9D= ? And should we / do we even do peeling with soft rate limiting? I think = I saw it help with bql(?), but I=E2=80=99m not sure I=E2=80=99ve seen it = help when rate limited below the interface=E2=80=99s rate.


* I still need a better understanding = of what triple-isolate does. It isn=E2=80=99t clear to me from the man = page. Results here are similar to dual-srchost/dual-dsthost:



*** = Round 2 Plans ***

- Add bql tests to anywhere rate limiting is used
- Add ethernet keyword to host isolation tests
- Add ethtool output to host info
- = Remove or improve flow isolation tests
- Add host isolation tests with rtt variation = (to look again at problem I reported in an earlier thread)

*** = Future Plans ***

- Use netem to make a spread of rtts and bandwidths
- Add VoIP tests (I hope to do this with = irtt)
- Add ack filtering tests
- Test BBR
- Use = qemu to test other archs (I may never get to this, honestly)

= --Apple-Mail=_89852D60-7F70-45D4-A770-A966C4D98DB2--