From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B5C303B2A4 for ; Mon, 27 Nov 2017 12:50:42 -0500 (EST) Received: by mail-wr0-x236.google.com with SMTP id z34so3989857wrz.10 for ; Mon, 27 Nov 2017 09:50:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=j/DOfCGrpMQ8dCo3Hj1RPAXw57u0ylpEl6MWvO95aWU=; b=MlWTDDxwVzwzU5Z2P/ArS772be0Fh/oOcoSyKPwKvKt52N60JVJtbzAGn3OszBdR+x Ipanp9n7zWnlSTc3e4kN2e4q6yVToQC5+tMoC6hVQcdnSqnabep71QWBhBMlysd7NtBj yJghw8+SEaLHqTV22v6T/6ZntwsB78YJaFtBok9d3ZBGuBKwSk6GZdCrc09/rgiRfVQs h+sBvNlCKzc/pMDUUM7livX6rY81SX6mjczI4CNUvN7N24UReAdJGvGBUdby31v83TaP /EMVPqcs7z6bxeuTniLVfd0B4+qD/ltSPEuJmVWpwqCjkRVzxOh4b0y5/DWMRvSkRR6X E43g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=j/DOfCGrpMQ8dCo3Hj1RPAXw57u0ylpEl6MWvO95aWU=; b=YryhDkjS4Y8QEO6U0/3LbJLqmHDT24E5WTDUB2RnLOfXmnISqboTYnjQXwDH9L2kFA mvUCawI/QdrhSTnMoKS0rZTJyj+98GVYIymbimiZzB5oKqKAKGu+7lL9qj937qiHPAbn IqaxXjm7mAAYaQont+xML9Youa2Y+OkXZfT33KttGaMFFdne4GQVYvuaWarLEaKL7jqW juGgRJ1WlfMW+au1AglawW1iGp3GpAjvk4nVHQTtysSbmyrmPZMP4cUVjYO0fOG3tGsp YOwMYV/TzqZzZep85iSh5TVCDAI1F+XqxZ26WqLNK3163C3rZAojwteDSGajXLRk4a37 3ebw== X-Gm-Message-State: AJaThX6Xavp3ZkTbRR42SAfxKWEa7buyYCVYJm6lcQHdk1s5Rxl6n2x+ KK6ql7qiTLGxLv8EE7fslNc= X-Google-Smtp-Source: AGs4zMZi8zpG4ZH+FLsE+Hcwbb01u4x2ttwKFD5/pjpXnXUK5Z5frnI1DEvuKp3+rpXOXQqXigYBMg== X-Received: by 10.223.148.199 with SMTP id 65mr34237241wrr.14.1511805041776; Mon, 27 Nov 2017 09:50:41 -0800 (PST) Received: from [10.72.0.130] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id d23sm12688523wma.48.2017.11.27.09.50.40 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 27 Nov 2017 09:50:40 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_49B47EEE-66CE-4552-9B21-E4A74847C5D3" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: Date: Mon, 27 Nov 2017 18:50:39 +0100 Cc: Sebastian Moeller , Cake List Message-Id: References: <85E1A7B2-8AA7-418A-BE43-209A1EC8881A@gmail.com> <87d1447z9w.fsf@toke.dk> <27F95EB1-490B-404C-8F77-98646B6159E7@gmail.com> <1C937A63-CEC1-4173-8812-EA2A85972B73@gmail.com> <20D304DC-494E-4A00-9B39-1E9F4B0F0CB6@gmail.com> <67B1612D-895D-4E3A-8CBD-21580B470696@gmail.com> <2A5F940F-F713-4578-8123-5CAD98A9C4C3@gmx.de> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] cake flenter results round 1 X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 17:50:43 -0000 --Apple-Mail=_49B47EEE-66CE-4552-9B21-E4A74847C5D3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I=E2=80=99m also finding out how simple it is to miss one little thing = when looking at hundreds of test runs. Finding the =E2=80=9Cgod = metric=E2=80=9D for rrul would make life easier... > On Nov 27, 2017, at 6:38 PM, Dave Taht wrote: >=20 > On Mon, Nov 27, 2017 at 9:34 AM, Sebastian Moeller > wrote: >> But 444.35 + 443.65 =3D 888, no? >=20 > My bad. I miss-read the test setup. Pre-coffee here, though, that > caused an adrenalin spike. >=20 > Yea! per host fairness 1v12! and correct bandwidth on this cpu. :whew: >=20 >>=20 >>> On Nov 27, 2017, at 18:33, Dave Taht wrote: >>>=20 >>> georgios >>>=20 >>> the result you got was "fair", but you shoul have seen something >>> closer to 900mbit than 400. >>>=20 >>> On Mon, Nov 27, 2017 at 8:17 AM, Georgios Amanakis = wrote: >>>> Dear Pete, >>>>=20 >>>> I am trying to replicate the unfair behaviour you are seeing with >>>> dual-{src,dst}host, albeit on different hardware and I am getting a = fair >>>> distribution. Hardware are Xeon E3-1220Lv2 (router), = i3-3110M(Clients). All >>>> running Archlinux, latest cake and patched iproute2-4.14.1, = connected with >>>> Gbit ethernet, TSO/GSO/GRO enabled. >>>>=20 >>>> Qdisc setup: >>>> ---------------- >>>> Router: >>>> qdisc cake 8003: dev ens4 root refcnt 2 bandwidth 900Mbit diffserv3 >>>> dual-dsthost rtt 100.0ms raw >>>>=20 >>>> Client A(kernel default): >>>> qdisc fq_codel 0: dev eno2 root refcnt 2 limit 10240p flows 1024 = quantum >>>> 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn >>>>=20 >>>> Client B (kernel default): >>>> qdisc fq_codel 0: dev enp1s0 root refcnt 2 limit 10240p flows 1024 = quantum >>>> 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn >>>> ---------------- >>>>=20 >>>>=20 >>>> Cli: >>>> ---------------- >>>> Router: >>>> netserver & >>>>=20 >>>> Client A: >>>> flent tcp_1down -H router >>>>=20 >>>> Client B: >>>> flent tcp_12down -H router >>>> ---------------- >>>>=20 >>>>=20 >>>> Results: >>>> ---------------- >>>> Router: >>>> qdisc cake 8003: root refcnt 2 bandwidth 900Mbit diffserv3 = dual-dsthost rtt >>>> 100.0ms raw >>>> Sent 7126680117 bytes 4725904 pkt (dropped 10, overlimits 4439745 = requeues >>>> 0) >>>> backlog 0b 0p requeues 0 >>>> memory used: 1224872b of 15140Kb >>>> capacity estimate: 900Mbit >>>> Bulk Best Effort Voice >>>> thresh 56250Kbit 900Mbit 225Mbit >>>> target 5.0ms 5.0ms 5.0ms >>>> interval 100.0ms 100.0ms 100.0ms >>>> pk_delay 14us 751us 7us >>>> av_delay 2us 642us 1us >>>> sp_delay 1us 1us 1us >>>> pkts 109948 4601651 14315 >>>> bytes 160183242 6964893773 1618242 >>>> way_inds 0 21009 0 >>>> way_miss 160 188 5 >>>> way_cols 0 0 0 >>>> drops 0 10 0 >>>> marks 0 0 0 >>>> ack_drop 0 0 0 >>>> sp_flows 0 1 1 >>>> bk_flows 1 0 0 >>>> un_flows 0 0 0 >>>> max_len 7570 68130 1022 >>>>=20 >>>>=20 >>>> Client A: >>>> avg median # data pts >>>> Ping (ms) ICMP : 0.11 0.08 ms 350 >>>> TCP download : 443.65 430.38 Mbits/s 301 >>>>=20 >>>>=20 >>>> Client B: >>>> avg median # data pts >>>> Ping (ms) ICMP : 0.09 0.06 ms 350 >>>> TCP download avg : 37.03 35.87 Mbits/s 301 >>>> TCP download sum : 444.35 430.40 Mbits/s 301 >>>> TCP download::1 : 37.00 35.87 Mbits/s 301 >>>> TCP download::10 : 37.01 35.87 Mbits/s 301 >>>> TCP download::11 : 37.02 35.87 Mbits/s 301 >>>> TCP download::12 : 37.00 35.87 Mbits/s 301 >>>> TCP download::2 : 37.03 35.87 Mbits/s 301 >>>> TCP download::3 : 36.99 35.87 Mbits/s 301 >>>> TCP download::4 : 37.03 35.87 Mbits/s 301 >>>> TCP download::5 : 37.07 35.87 Mbits/s 301 >>>> TCP download::6 : 37.00 35.87 Mbits/s 301 >>>> TCP download::7 : 37.12 35.87 Mbits/s 301 >>>> TCP download::8 : 37.05 35.87 Mbits/s 301 >>>> TCP download::9 : 37.03 35.87 Mbits/s 301 >>>> ---------------- >>>>=20 >>>> Does this suggest that it is indeed a problem of an underpowered = CPU in your >>>> case? >>>>=20 >>>> George >>>>=20 >>>>=20 >>>> On Mon, Nov 27, 2017 at 10:53 AM, Pete Heist = wrote: >>>>>=20 >>>>>=20 >>>>> On Nov 27, 2017, at 3:48 PM, Jonathan Morton = >>>>> wrote: >>>>>=20 >>>>> It's not at all obvious how we'd detect that. Packets are staying = in the >>>>> queue for less time than the codel target, which is exactly what = you'd get >>>>> if you weren't saturated at all. >>>>>=20 >>>>> That makes complete sense when you put it that way. Cake has no = way of >>>>> knowing why the input rate is lower than expected, even if it=E2=80=99= s part of the >>>>> cause. >>>>>=20 >>>>> I don=E2=80=99t think flent can know this either. It can=E2=80=99t = easily know the cause >>>>> for its total output to be lower than expected. >>>>>=20 >>>>> All I know is, this is a common problem in deployments, = particularly on >>>>> low-end hardware like ER-Xs, that can be tricky for users to = figure out. >>>>>=20 >>>>> I don=E2=80=99t even think monitoring CPU in general would work. = The CPU could be >>>>> high because it=E2=80=99s doing other calculations, but there=E2=80=99= s still enough for >>>>> cake at a low rate, and there=E2=80=99s no need to warn in that = case. I=E2=80=99d be >>>>> interested in any ideas on how to know this is happening in the = system as a >>>>> whole. So far, there are just various clues that one needs to = piece together >>>>> (no or few drops or marks, less total throughput that expected, = high cpu >>>>> without other external usage, etc). Then it needs to be proven = with a test. >>>>>=20 >>>>> Anyway thanks, your clue was what I needed! I need to remember to = review >>>>> the qdisc stats when something unexpected happens. >>>>>=20 >>>>> _______________________________________________ >>>>> Cake mailing list >>>>> Cake@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/cake >>>>>=20 >>>>=20 >>>>=20 >>>> _______________________________________________ >>>> Cake mailing list >>>> Cake@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/cake >>>>=20 >>>=20 >>>=20 >>>=20 >>> -- >>>=20 >>> Dave T=C3=A4ht >>> CEO, TekLibre, LLC >>> http://www.teklibre.com >>> Tel: 1-669-226-2619 >>> _______________________________________________ >>> Cake mailing list >>> Cake@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cake >>=20 >=20 >=20 >=20 > --=20 >=20 > Dave T=C3=A4ht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake = --Apple-Mail=_49B47EEE-66CE-4552-9B21-E4A74847C5D3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
I=E2=80=99m also finding out how simple it is = to miss one little thing when looking at hundreds of test runs. Finding = the =E2=80=9Cgod metric=E2=80=9D for rrul would make life = easier...

On Nov 27, 2017, at 6:38 PM, Dave Taht <dave.taht@gmail.com>= wrote:

On Mon, Nov 27, 2017 at 9:34 AM, Sebastian = Moeller <moeller0@gmx.de> wrote:
But 444.35 + 443.65 =3D 888, no?

My bad. I miss-read the test setup. Pre-coffee = here, though, that
caused an adrenalin spike.

Yea! per host fairness 1v12! and correct = bandwidth on this cpu. :whew:


On Nov 27, 2017, at = 18:33, Dave Taht <dave.taht@gmail.com> wrote:

georgios

the result you got was = "fair", but you shoul have seen something
closer to = 900mbit than 400.

On Mon, Nov 27, 2017 at = 8:17 AM, Georgios Amanakis <gamanakis@gmail.com> wrote:
Dear Pete,

I am = trying to replicate the unfair behaviour you are seeing with
dual-{src,dst}host, albeit on different hardware and I am = getting a fair
distribution. Hardware are Xeon E3-1220Lv2 = (router), i3-3110M(Clients). All
running Archlinux, latest = cake and patched iproute2-4.14.1, connected with
Gbit = ethernet, TSO/GSO/GRO enabled.

Qdisc = setup:
----------------
Router:
qdisc cake 8003: dev ens4 root refcnt 2 bandwidth 900Mbit = diffserv3
dual-dsthost rtt 100.0ms raw

Client A(kernel default):
qdisc fq_codel 0: dev = eno2 root refcnt 2 limit 10240p flows 1024 quantum
1514 = target 5.0ms interval 100.0ms memory_limit 32Mb ecn

Client B (kernel default):
qdisc fq_codel 0: = dev enp1s0 root refcnt 2 limit 10240p flows 1024 quantum
1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
----------------


Cli:
----------------
Router:
netserver &

Client A:
flent tcp_1down -H router

Client = B:
flent tcp_12down -H router
----------------


Results:
----------------
Router:
qdisc cake 8003: root refcnt 2 = bandwidth 900Mbit diffserv3 dual-dsthost rtt
100.0ms = raw
Sent 7126680117 bytes 4725904 pkt (dropped 10, = overlimits 4439745 requeues
0)
backlog 0b 0p = requeues 0
memory used: 1224872b of 15140Kb
capacity estimate: 900Mbit
          &nb= sp;    Bulk   Best Effort =      Voice
thresh =     56250Kbit     900Mbit =     225Mbit
target =         5.0ms =       5.0ms =       5.0ms
interval =     100.0ms     100.0ms =     100.0ms
pk_delay =        14us =       751us =         7us
av_delay=         2us =       642us =         1us
sp_delay=         1us =         1us =         1us
pkts =          109948 =     4601651 =       14315
bytes =      160183242  6964893773 =     1618242
way_inds =           0 =       21009 =           0
way_miss         160 =         188 =           5
way_cols =           0 =           0 =           0
drops =             &n= bsp;0          10 =           0
marks =             &n= bsp;0           0 =           0
ack_drop =           0 =           0 =           0
sp_flows =           0 =           1 =           1
bk_flows =           1 =           0 =           0
un_flows =           0 =           0 =           0
max_len         7570 =       68130 =        1022


Client A:
          &nb= sp;            = ;  avg       median =          # data pts
Ping (ms) ICMP : =         0.11 =         0.08 ms =             &n= bsp;350
TCP download   : =       443.65 =       430.38 Mbits/s =         301


Client B:
          &nb= sp;            = ;    avg       median =          # data pts
Ping (ms) ICMP   : =         0.09 =         0.06 ms =             &n= bsp;350
TCP download avg : =        37.03 =        35.87 Mbits/s =         301
TCP = download sum :       444.35 =       430.40 Mbits/s =         301
TCP = download::1  :        37.00 =        35.87 Mbits/s =         301
TCP = download::10 :        37.01 =        35.87 Mbits/s =         301
TCP = download::11 :        37.02 =        35.87 Mbits/s =         301
TCP = download::12 :        37.00 =        35.87 Mbits/s =         301
TCP = download::2  :        37.03 =        35.87 Mbits/s =         301
TCP = download::3  :        36.99 =        35.87 Mbits/s =         301
TCP = download::4  :        37.03 =        35.87 Mbits/s =         301
TCP = download::5  :        37.07 =        35.87 Mbits/s =         301
TCP = download::6  :        37.00 =        35.87 Mbits/s =         301
TCP = download::7  :        37.12 =        35.87 Mbits/s =         301
TCP = download::8  :        37.05 =        35.87 Mbits/s =         301
TCP = download::9  :        37.03 =        35.87 Mbits/s =         301
----------------

Does this = suggest that it is indeed a problem of an underpowered CPU in your
case?

George


On Mon, Nov 27, 2017 at 10:53 AM, Pete Heist = <peteheist@gmail.com> wrote:


On Nov 27, 2017, = at 3:48 PM, Jonathan Morton <chromatix99@gmail.com>
wrote:

It's not at all obvious how we'd detect that. =  Packets are staying in the
queue for less time than = the codel target, which is exactly what you'd get
if you = weren't saturated at all.

That makes = complete sense when you put it that way. Cake has no way of
knowing why the input rate is lower than expected, even if = it=E2=80=99s part of the
cause.

I don=E2=80=99t think flent can know this either. It can=E2=80=99= t easily know the cause
for its total output to be lower = than expected.

All I know is, this is a = common problem in deployments, particularly on
low-end = hardware like ER-Xs, that can be tricky for users to figure out.

I don=E2=80=99t even think monitoring CPU in = general would work. The CPU could be
high because it=E2=80=99= s doing other calculations, but there=E2=80=99s still enough for
cake at a low rate, and there=E2=80=99s no need to warn in = that case. I=E2=80=99d be
interested in any ideas on how = to know this is happening in the system as a
whole. So = far, there are just various clues that one needs to piece together
(no or few drops or marks, less total throughput that = expected, high cpu
without other external usage, etc). = Then it needs to be proven with a test.

Anyway thanks, your clue was what I needed! I need to = remember to review
the qdisc stats when something = unexpected happens.

_______________________________________________
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake



_______________________________________________
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake




--
Dave T=C3=A4ht
CEO, TekLibre, = LLC
http://www.teklibre.com
Tel: = 1-669-226-2619
_______________________________________________
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake




-- 

Dave T=C3=A4ht
CEO, TekLibre, = LLC
http://www.teklibre.com
Tel: 1-669-226-2619
_______________________________________________
Cake mailing = list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

= --Apple-Mail=_49B47EEE-66CE-4552-9B21-E4A74847C5D3--