From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x236.google.com (mail-wr0-x236.google.com [IPv6:2a00:1450:400c:c0c::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BA23B3BA8E for ; Tue, 3 Jul 2018 03:35:22 -0400 (EDT) Received: by mail-wr0-x236.google.com with SMTP id h40-v6so838625wrh.2 for ; Tue, 03 Jul 2018 00:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=nrw3smtWBTfUN9iZPOgJT+PZge+7VNO9Jm88tz5UtWQ=; b=hKJvGXSYrhfx464nwoikJ1n39ChuCMjxxEEfWMZzyGsI+5nlwtigp+L27J/ggpVwor afelGM1x6Jo9tS+oeFBk/S9mkxU8yW9P/GjJyRUlgu8/MTu0B1mKVmVcw7V56xrc+ZBe VbjGMxOsCY57ndmdzS2SXM4Rfkk/w5xlK37VmAx73HbpG/PUibfdghFKuhRXKyMc97lp WDCRsmDiChrwTF8deS2aaabZAm/M6tqbJO5dgqWG5ommsy5wO9Ps7dwNu/ssUHpgW0mC DT2MEaPMu4FKM4tltuC5eDAXzz5ceNbCj5aR2sQv+z1HUuItjqUZaqgFRCtgQYwhyT2E Y87g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=nrw3smtWBTfUN9iZPOgJT+PZge+7VNO9Jm88tz5UtWQ=; b=FnJGp1eElzZ86Fl136WbGHwC9l0KBJoGKbPjrTRL1j0/0QbWlR0VW0v9YhYSlD2PfX N3kU6V3VZ3xH8SlmdJGyf2W1163pcVkVhDx/O6BU1gdgn60CEnn3UXLq5PE4O4SqgIqN 6c/jT+sL18Mh1yWqKx+llGNou75DXOnFdUiWZtAnL4T0bYgbWsWj3v00aOOOhzf34veh VHpT1iJqUObxOJrpDKq41Mbb6QI+75Xm5terkUw5vHOswPnEKEvYDhg8FB98LvnC0ule ElTI1RZBhPKd/N5oZnCYzOxc+5hvliVHgrY9tShRaA5JoCBLcGM3cmtTdUbHpxNeerXl cyeg== X-Gm-Message-State: APt69E1GqsyJgsAxDXAnaeClLU4VzTg1ik+hVxiF6dMKegNbVgRqm7i9 AYLaBuYxTSgITrgnFIi9YKdCAA== X-Google-Smtp-Source: AAOMgpe+1YXuYiN8vJmo5DOgCEknl28ZnUtXrci5Dwex5wT1Gomeiv5qT1S6/ilf/3lk6YGzPNA52w== X-Received: by 2002:adf:b002:: with SMTP id f2-v6mr18871912wra.75.1530603321635; Tue, 03 Jul 2018 00:35:21 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id z18-v6sm357345wrr.76.2018.07.03.00.35.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Jul 2018 00:35:21 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.4 \(3445.8.2\)) From: Pete Heist In-Reply-To: <48ECB6C8-5D22-4785-A6CE-696D87EC5496@toke.dk> Date: Tue, 3 Jul 2018 09:35:19 +0200 Cc: Georgios Amanakis , Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <7D3ED7F4-586C-43F8-BCF2-8540E3FE1B83@heistp.net> References: <94C9790F-E9BC-4D59-9845-17C305E4B910@darbyshire-bryant.me.uk> <17AF79A0-0213-44E3-95B9-62795A644A47@heistp.net> <87lgatj13k.fsf@toke.dk> <87fu11ipir.fsf@toke.dk> <871scligay.fsf@toke.dk> <2AE036E5-BD3D-4176-9476-9EC824EC1D18@darbyshire-bryant.me.uk> <87r2klh1fz.fsf@toke.dk> <87lgath01v.fsf@toke.dk> <52B2B44D-4382-404C-8F6D-03F12A72B11F@heistp.net> <31667353-48F2-4FAB-AC05-163680451719@toke.dk> <48ECB6C8-5D22-4785-A6CE-696D87EC5496@toke.dk> To: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: Apple Mail (2.3445.8.2) Subject: Re: [Cake] cake at 60gbit X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2018 07:35:23 -0000 > On Jul 3, 2018, at 12:23 AM, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >=20 > My hunch is that this has something to do with the way mlx5 uses = multiple receive queues (and thus multiple CPUs). Which is probably = different from veth... It doesn=E2=80=99t reproduce between between 2x APU2c4 (4 hardware = queues and igb driver w/ mq support), but that=E2=80=99s 1Gbit, plus = this Mellanox thing appears to have all sorts of hardware whatnots if = this is the right device: = http://www.mellanox.com/related-docs/prod_silicon/PB_ConnectX-4_EN_IC.pdf I noticed it has support for some link-layer congestion notification = I=E2=80=99d not heard of until now (802.1Qau, = https://www.ietf.org/proceedings/67/slides/tsvarea-2.pdf).