From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BD7E33BA8E for ; Tue, 3 Jul 2018 05:57:24 -0400 (EDT) Received: by mail-wm0-x241.google.com with SMTP id z137-v6so1608831wmc.0 for ; Tue, 03 Jul 2018 02:57:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fOuP7BlRJhj0RT7h1f9atEWB0yyTFIXHANk153bUQao=; b=BRI4zOn2tTfQckmx5fLxaBgvyRo3/gqJRAVcOPBvESGq9x5d7oWKyRgMJoszARVSq5 U7m29a4wzORQy36FBba3yfTXTD0JtK8n1zIS/M/lc3Kp6igA78QuodQxBO+2IC9VYVsz Fvql5ECvV3W4cjaFs3SfEYisTvVft0hNHPd6nKHyENVr3xZAomfcHxqTYZg5c3EiZdy5 7NgfJgyG+GqTPDALYoCx4oO7UABcQjk7HJ+Nh5/MuqGL+hALFpa6+rLVCUyMy2DUivQc QN56hx23lNB7MonrjFFHH+a9khVjuQnaBpyYRyvT8VZngxYaf/9r+vwSNA1xy2damxIa GYww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=fOuP7BlRJhj0RT7h1f9atEWB0yyTFIXHANk153bUQao=; b=BugX/G+Fngu5IigAXAgF+4dVF+WCAuhm3iFiHRXb1qUpsFuF/WGwLrtv+Zw2KesH+k e+btYA0o+J5euGx/5HRWIvYklUAOZwb8eau4YegmHoMO7vNG/hoeSz0HdMo8M2JL8dJj QXO8qcHsaj9AqdroVMClwt+Zump9ni/vCQExFSPR1889Ge8cdiHUjPrchdipRAspZwuV NohHKk+Tb+hm3xdJ+FHbCCUIp99JIzqVtvubXIWB95P0a4GE4zdN55ZXCWPhd4GncMJl 090PZWw6YVZTepiiVMpTUrOf34VwnCVK9hh8lt7uGZultMP7z1Egt60x9rgKuciDlAyK 9VCg== X-Gm-Message-State: APt69E399rW9W/zOfBKq9RUvdfDpCbdfW8hJ+ApML3jsTHtNfNMdtSTq zH4t9o0PC0bMotjJjA7mVFDDvA== X-Google-Smtp-Source: AAOMgpfTnVq6k3C/9fznnvf6JBClPwuC3shUR2T7rolKfzB6mk2Pyh5EUuI2oq5GrFrD+Gw0KIIl0g== X-Received: by 2002:a1c:3a8f:: with SMTP id h137-v6mr3495255wma.41.1530611843810; Tue, 03 Jul 2018 02:57:23 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id r123-v6sm1348518wmb.21.2018.07.03.02.57.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Jul 2018 02:57:23 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.4 \(3445.8.2\)) From: Pete Heist In-Reply-To: <73DD74AD-C2E7-4A12-AE49-C06D4486660E@gmail.com> Date: Tue, 3 Jul 2018 11:57:22 +0200 Cc: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= , Cake List Content-Transfer-Encoding: quoted-printable Message-Id: <6EB2B347-7434-4DE9-8FD3-E162BC40E14D@heistp.net> References: <94C9790F-E9BC-4D59-9845-17C305E4B910@darbyshire-bryant.me.uk> <17AF79A0-0213-44E3-95B9-62795A644A47@heistp.net> <87lgatj13k.fsf@toke.dk> <87fu11ipir.fsf@toke.dk> <871scligay.fsf@toke.dk> <2AE036E5-BD3D-4176-9476-9EC824EC1D18@darbyshire-bryant.me.uk> <87r2klh1fz.fsf@toke.dk> <87lgath01v.fsf@toke.dk> <52B2B44D-4382-404C-8F6D-03F12A72B11F@heistp.net> <31667353-48F2-4FAB-AC05-163680451719@toke.dk> <48ECB6C8-5D22-4785-A6CE-696D87EC5496@toke.dk> <73DD74AD-C2E7-4A12-AE49-C06D4486660E@gmail.com> To: Jonathan Morton X-Mailer: Apple Mail (2.3445.8.2) Subject: Re: [Cake] cake at 60gbit X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2018 09:57:25 -0000 > On Jul 3, 2018, at 11:18 AM, Jonathan Morton = wrote: >=20 >> On 3 Jul, 2018, at 1:23 am, Toke H=C3=B8iland-J=C3=B8rgensen = wrote: >>=20 >> My hunch is that this has something to do with the way mlx5 uses = multiple receive queues (and thus multiple CPUs). Which is probably = different from veth... >=20 > At this stage I'm pretty confident it has nothing to do with Cake, and = everything to do with the Mellanox hardware and driver. It does strike = me that Linux' default handling of multiqueue hardware doesn't map very = well to the qdisc interface. That=E2=80=99s looking highly likely to me too. I=E2=80=99ll stop trying = to repro this unless we come up with something new to try. I wonder if enabling the =E2=80=9Clockless qdisc=E2=80=9D support = (https://lwn.net/Articles/698135/) we discussed late last year would = change anything. It should work either way, even with a single qdisc = lock, but just speculating, at those speeds with multiple cores = competing for a single qdisc lock I wonder if we=E2=80=99re exposing a = problem in the driver or somewhere else.=