From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id B1A963B2A4 for ; Mon, 27 Nov 2017 12:32:42 -0500 (EST) Received: by mail-wm0-x234.google.com with SMTP id u83so35944884wmb.5 for ; Mon, 27 Nov 2017 09:32:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=F2rJ2cAlc9YKqrCcl/csOsjudldUdf1ybvQ3306/aac=; b=ap+tWcq5hFtgIxjHkrUp9+/jWl574rxbTOkBEUyEFxELo7cVQ2sJLLAC7xxIZa9/h0 Ct/1WTptt75HOQKEGmXdAk4k0jTYahdEsQGMCCa0JiR3VqokoK1K0qCvPM2JD4gf3nrY n1g4En8GLHhq881l/9iWm+FdiBSGvYVxIV0OHIzJVw2hNPnalFYN1Y4lFh9mFJLg8DIV EAhzBYVHL1IVr6voa14g6OFhngzDxJfFxgDncsRd6vWydOPb5Li0MrnI4Hg1S97JIZCK Y7+Ws+7w/CuCRMjOtTwBpMOiHt66Pny3lBdko4xvQ4Q7JtZ8uH2wHRM5C4u7G6CwpnLP M+NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=F2rJ2cAlc9YKqrCcl/csOsjudldUdf1ybvQ3306/aac=; b=a3q86N9P/bD0knmoMkEk8sU38ZcTPHQaBR61cX3g21IYKW/D1STq1/yiBqL6jEumd0 K/wkrqMf39WHvU5QNTuPmJWHNmrUsyB82z4nmzkzWsjTo0tCdIbM76Z7NdQmDJ09duQW /XkwS5UUOn2Z13nZ4LpjFFhnakizLJpFhN9HHIL21QlUtNE6viu7hWfVcqtiBdpXoLCS qRVnmm0QCGsFzJvDm1ntOtaO1iI7nP5XLyUH8ds29pWXU8SyiSUI7sBPEP7oRVBzAJrv bX9mGsctaw91x4D7qgipUOLd/WJZxvnBbX6WdeC2Nd76IimXtdH88cd1lV1YSSLOJ6fx XwtA== X-Gm-Message-State: AJaThX6M44e44gwdsYBuNTrIviuCnTM7na2LCeIsQXTPEFwssOx+YhuR ifysx/U/TNVyTaE8aM4HlVI= X-Google-Smtp-Source: AGs4zMaBPVep7rLRojD9HCZvbPVl75pVaXrGLQzlaKgBNqZYPX+FAHUaFdtoMFrfGAG40n2JEGiHYw== X-Received: by 10.28.59.69 with SMTP id i66mr17873376wma.130.1511803961743; Mon, 27 Nov 2017 09:32:41 -0800 (PST) Received: from [10.72.0.130] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id s195sm21790255wmb.33.2017.11.27.09.32.40 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 27 Nov 2017 09:32:41 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_09FA1009-A0CE-4AB3-8868-73CEC3C4529A" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: Date: Mon, 27 Nov 2017 18:32:40 +0100 Cc: Cake List Message-Id: References: <85E1A7B2-8AA7-418A-BE43-209A1EC8881A@gmail.com> <87d1447z9w.fsf@toke.dk> <27F95EB1-490B-404C-8F77-98646B6159E7@gmail.com> <1C937A63-CEC1-4173-8812-EA2A85972B73@gmail.com> <20D304DC-494E-4A00-9B39-1E9F4B0F0CB6@gmail.com> <67B1612D-895D-4E3A-8CBD-21580B470696@gmail.com> To: Georgios Amanakis X-Mailer: Apple Mail (2.3124) Subject: Re: [Cake] cake flenter results round 1 X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 17:32:43 -0000 --Apple-Mail=_09FA1009-A0CE-4AB3-8868-73CEC3C4529A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Yes, especially since you=E2=80=99ve got higher-end hardware than I. = 443.65mbit vs 444.35mbit looks pretty fair. :) Thanks for reproducing it. I=E2=80=99m going to have to review some of = my flenter tests in light of this. I=E2=80=99m getting a handle on the = limitations of the APU2 hardware. It=E2=80=99s quite good especially for = the price, but has limits on what it can do at Gbit rates. It can = actually be useful to test what happens when the CPU is overburdened, = only I need to avoid being fooled by it. You could also add the =E2=80=98ethernet=E2=80=99 keyword, which I=E2=80=99= m going to add for this test in my next round, although that wouldn=E2=80=99= t have fixed what I was seeing anyway... Pete > On Nov 27, 2017, at 5:17 PM, Georgios Amanakis = wrote: >=20 > Dear Pete, >=20 > I am trying to replicate the unfair behaviour you are seeing with = dual-{src,dst}host, albeit on different hardware and I am getting a fair = distribution. Hardware are Xeon E3-1220Lv2 (router), i3-3110M(Clients). = All running Archlinux, latest cake and patched iproute2-4.14.1, = connected with Gbit ethernet, TSO/GSO/GRO enabled. >=20 > Qdisc setup: > ---------------- > Router:=20 > qdisc cake 8003: dev ens4 root refcnt 2 bandwidth 900Mbit diffserv3 = dual-dsthost rtt 100.0ms raw >=20 > Client A(kernel default): > qdisc fq_codel 0: dev eno2 root refcnt 2 limit 10240p flows 1024 = quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn >=20 > Client B (kernel default):=20 > qdisc fq_codel 0: dev enp1s0 root refcnt 2 limit 10240p flows 1024 = quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn > ---------------- >=20 >=20 > Cli: > ---------------- > Router: > netserver & >=20 > Client A: > flent tcp_1down -H router >=20 > Client B: > flent tcp_12down -H router > ---------------- >=20 >=20 > Results: > ---------------- > Router: > qdisc cake 8003: root refcnt 2 bandwidth 900Mbit diffserv3 = dual-dsthost rtt 100.0ms raw > Sent 7126680117 bytes 4725904 pkt (dropped 10, overlimits 4439745 = requeues 0) > backlog 0b 0p requeues 0 > memory used: 1224872b of 15140Kb > capacity estimate: 900Mbit > Bulk Best Effort Voice > thresh 56250Kbit 900Mbit 225Mbit > target 5.0ms 5.0ms 5.0ms > interval 100.0ms 100.0ms 100.0ms > pk_delay 14us 751us 7us > av_delay 2us 642us 1us > sp_delay 1us 1us 1us > pkts 109948 4601651 14315 > bytes 160183242 6964893773 1618242 > way_inds 0 21009 0 > way_miss 160 188 5 > way_cols 0 0 0 > drops 0 10 0 > marks 0 0 0 > ack_drop 0 0 0 > sp_flows 0 1 1 > bk_flows 1 0 0 > un_flows 0 0 0 > max_len 7570 68130 1022 >=20 >=20 > Client A: > avg median # data pts > Ping (ms) ICMP : 0.11 0.08 ms 350 > TCP download : 443.65 430.38 Mbits/s 301 >=20 >=20 > Client B: > avg median # data pts > Ping (ms) ICMP : 0.09 0.06 ms 350 > TCP download avg : 37.03 35.87 Mbits/s 301 > TCP download sum : 444.35 430.40 Mbits/s 301 > TCP download::1 : 37.00 35.87 Mbits/s 301 > TCP download::10 : 37.01 35.87 Mbits/s 301 > TCP download::11 : 37.02 35.87 Mbits/s 301 > TCP download::12 : 37.00 35.87 Mbits/s 301 > TCP download::2 : 37.03 35.87 Mbits/s 301 > TCP download::3 : 36.99 35.87 Mbits/s 301 > TCP download::4 : 37.03 35.87 Mbits/s 301 > TCP download::5 : 37.07 35.87 Mbits/s 301 > TCP download::6 : 37.00 35.87 Mbits/s 301 > TCP download::7 : 37.12 35.87 Mbits/s 301 > TCP download::8 : 37.05 35.87 Mbits/s 301 > TCP download::9 : 37.03 35.87 Mbits/s 301 > ---------------- >=20 > Does this suggest that it is indeed a problem of an underpowered CPU = in your case? >=20 > George >=20 >=20 > On Mon, Nov 27, 2017 at 10:53 AM, Pete Heist > wrote: >=20 >> On Nov 27, 2017, at 3:48 PM, Jonathan Morton > wrote: >> It's not at all obvious how we'd detect that. Packets are staying in = the queue for less time than the codel target, which is exactly what = you'd get if you weren't saturated at all. >>=20 >=20 > That makes complete sense when you put it that way. Cake has no way of = knowing why the input rate is lower than expected, even if it=E2=80=99s = part of the cause. >=20 > I don=E2=80=99t think flent can know this either. It can=E2=80=99t = easily know the cause for its total output to be lower than expected. >=20 > All I know is, this is a common problem in deployments, particularly = on low-end hardware like ER-Xs, that can be tricky for users to figure = out. >=20 > I don=E2=80=99t even think monitoring CPU in general would work. The = CPU could be high because it=E2=80=99s doing other calculations, but = there=E2=80=99s still enough for cake at a low rate, and there=E2=80=99s = no need to warn in that case. I=E2=80=99d be interested in any ideas on = how to know this is happening in the system as a whole. So far, there = are just various clues that one needs to piece together (no or few drops = or marks, less total throughput that expected, high cpu without other = external usage, etc). Then it needs to be proven with a test. >=20 > Anyway thanks, your clue was what I needed! I need to remember to = review the qdisc stats when something unexpected happens. >=20 > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake = >=20 >=20 --Apple-Mail=_09FA1009-A0CE-4AB3-8868-73CEC3C4529A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Yes, especially since you=E2=80=99ve got = higher-end hardware than I. 443.65mbit vs 444.35mbit looks pretty fair. = :)

Thanks for = reproducing it. I=E2=80=99m going to have to review some of my flenter = tests in light of this. I=E2=80=99m getting a handle on the limitations = of the APU2 hardware. It=E2=80=99s quite good especially for the price, = but has limits on what it can do at Gbit rates. It can actually be = useful to test what happens when the CPU is overburdened, only I need to = avoid being fooled by it.

You could also add the =E2=80=98ethernet=E2=80=99 keyword, = which I=E2=80=99m going to add for this test in my next round, although = that wouldn=E2=80=99t have fixed what I was seeing anyway...

Pete

On = Nov 27, 2017, at 5:17 PM, Georgios Amanakis <gamanakis@gmail.com>= wrote:

Dear Pete,

I am trying to replicate the unfair = behaviour you are seeing with dual-{src,dst}host, albeit on different = hardware and I am getting a fair distribution. Hardware are Xeon = E3-1220Lv2 (router), i3-3110M(Clients). All running Archlinux, latest = cake and patched iproute2-4.14.1, connected with Gbit ethernet, = TSO/GSO/GRO enabled.

Qdisc setup:
----------------
Router:
qdisc cake = 8003: dev ens4 root refcnt 2 bandwidth 900Mbit diffserv3 dual-dsthost = rtt 100.0ms raw

Client A(kernel default):
qdisc fq_codel = 0: dev eno2 root refcnt 2 limit 10240p flows 1024 quantum 1514 target = 5.0ms interval 100.0ms memory_limit 32Mb ecn

Client B (kernel default):
qdisc fq_codel 0: = dev enp1s0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target = 5.0ms interval 100.0ms memory_limit 32Mb ecn
----------------


Cli:
----------------
Router:
netserver &

Client A:
flent tcp_1down -H router

Client B:
flent tcp_12down -H router
----------------


Results:
----------------
Router:
qdisc cake 8003: root refcnt 2 = bandwidth 900Mbit diffserv3 dual-dsthost rtt 100.0ms raw
 Sent 7126680117 bytes 4725904 pkt (dropped 10, = overlimits 4439745 requeues 0)
 backlog 0b 0p = requeues 0
 memory used: 1224872b of 15140Kb
 capacity estimate: 900Mbit
          &nb= sp;      Bulk   Best = Effort      Voice
  = thresh     56250Kbit     = 900Mbit     225Mbit
  = target         = 5.0ms       = 5.0ms       5.0ms
  = interval     100.0ms     = 100.0ms     100.0ms
  = pk_delay        = 14us       = 751us         7us
  = av_delay         = 2us       = 642us         1us
  = sp_delay         = 1us         = 1us         1us
  = pkts          = 109948     = 4601651       14315
  = bytes      160183242  = 6964893773     1618242
  = way_inds           = 0       = 21009           0
  = way_miss         = 160         = 188           5
  = way_cols           = 0           = 0           0
  = drops           &nb= sp;  0          = 10           0
  = marks           &nb= sp;  0           = 0           0
  = ack_drop           = 0           = 0           0
  = sp_flows           = 0           = 1           1
  = bk_flows           = 1           = 0           0
  = un_flows           = 0           = 0           0
  = max_len         = 7570       = 68130        1022


Client = A:
          &nb= sp;            = ;    avg       = median          # data = pts
 Ping (ms) ICMP = :         = 0.11         0.08 = ms            =   350
 TCP download   = :       = 443.65       430.38 = Mbits/s         301


Client = B:
          &nb= sp;            = ;      avg       = median          # data = pts
 Ping (ms) ICMP   = :         = 0.09         0.06 = ms            =   350
 TCP download avg = :        = 37.03        35.87 = Mbits/s         301
 TCP download sum :       = 444.35       430.40 = Mbits/s         301
 TCP download::1  = :        = 37.00        35.87 = Mbits/s         301
 TCP download::10 = :        = 37.01        35.87 = Mbits/s         301
 TCP download::11 = :        = 37.02        35.87 = Mbits/s         301
 TCP download::12 = :        = 37.00        35.87 = Mbits/s         301
 TCP download::2  = :        = 37.03        35.87 = Mbits/s         301
 TCP download::3  = :        = 36.99        35.87 = Mbits/s         301
 TCP download::4  = :        = 37.03        35.87 = Mbits/s         301
 TCP download::5  = :        = 37.07        35.87 = Mbits/s         301
 TCP download::6  = :        = 37.00        35.87 = Mbits/s         301
 TCP download::7  = :        = 37.12        35.87 = Mbits/s         301
 TCP download::8  = :        = 37.05        35.87 = Mbits/s         301
 TCP download::9  = :        = 37.03        35.87 = Mbits/s         301
----------------

Does this suggest that it is indeed a problem of an = underpowered CPU in your case?

George


On Mon, Nov 27, 2017 at 10:53 AM, = Pete Heist <peteheist@gmail.com> wrote:

On Nov 27, 2017, at 3:48 PM, Jonathan Morton <chromatix99@gmail.com> wrote:

It's not at all obvious how we'd detect = that.  Packets are staying in the queue for less time than the = codel target, which is exactly what you'd get if you weren't saturated = at all.

That makes = complete sense when you put it that way. Cake has no way of knowing why = the input rate is lower than expected, even if it=E2=80=99s part of the = cause.

I = don=E2=80=99t think flent can know this either. It can=E2=80=99t easily = know the cause for its total output to be lower than expected.

All I know is, this is a = common problem in deployments, particularly on low-end hardware like = ER-Xs, that can be tricky for users to figure out.

I don=E2=80=99t even = think monitoring CPU in general would work. The CPU could be high = because it=E2=80=99s doing other calculations, but there=E2=80=99s still = enough for cake at a low rate, and there=E2=80=99s no need to warn in = that case. I=E2=80=99d be interested in any ideas on how to know this is = happening in the system as a whole. So far, there are just various clues = that one needs to piece together (no or few drops or marks, less total = throughput that expected, high cpu without other external usage, etc). = Then it needs to be proven with a test.

Anyway thanks, your clue was what I = needed! I need to remember to review the qdisc stats when something = unexpected happens.

_______________________________________________
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake



= --Apple-Mail=_09FA1009-A0CE-4AB3-8868-73CEC3C4529A--