From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <toke@toke.dk>
Received: from mail.toke.dk (mail.toke.dk [IPv6:2001:470:dc45:1000::1])
 (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by lists.bufferbloat.net (Postfix) with ESMTPS id 54F923B2A4
 for <cake@lists.bufferbloat.net>; Fri,  6 Jul 2018 07:33:23 -0400 (EDT)
From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= <toke@toke.dk>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023;
 t=1530876801; bh=7TtONkJeIYG1/2ju5ICbeFwd41SNyJfUPJVm6ruULQE=;
 h=From:To:Cc:Subject:In-Reply-To:References:Date:From;
 b=JrAKGmEakPT0wOZZDiBV5wj8p1FyWJhdk3exa46gdIooEVaLf/glLW14/8lu2dvAx
 pWkT4+nSulbX9Oxwe3fK+tAoC8xiLyKqgPDuxHHLgCxpTGuZnQjXN7qGhzkZg0jNnw
 Y3huX8Do9nw3JwlhDSF7yAMlhBVl1Ob7IaMcNYMxyLjtP+gPhNq/UJs3ob5ApO3lip
 v1uWw2h1UM7YsUWK/bVvVHG59lNUiDfy1VZB4RGHP7LPmjASysq+BK2yZa7lpK+9Nc
 821zwkoLKnkJpSuUxctQsnMytiCRN5tFKl9LbkBzQoH2E+dlQ/3UNAIJ8A4Wrjng8b
 ENK9atkqU4EPw==
To: Pete Heist <pete@heistp.net>
Cc: Jonathan Morton <chromatix99@gmail.com>,
 Cake List <cake@lists.bufferbloat.net>
In-Reply-To: <87lgaobq0m.fsf@toke.dk>
References: <mailman.381.1530347349.3512.cake@lists.bufferbloat.net>
 <D43CE745-E5AC-4AC1-879A-2975545BC8B8@heistp.net> <87fu11ipir.fsf@toke.dk>
 <mailman.407.1530550780.3512.cake@lists.bufferbloat.net>
 <A17BCC62-D57E-44AC-8D75-82EF278E745A@darbyshire-bryant.me.uk>
 <871scligay.fsf@toke.dk>
 <2AE036E5-BD3D-4176-9476-9EC824EC1D18@darbyshire-bryant.me.uk>
 <87r2klh1fz.fsf@toke.dk>
 <CAA93jw4kzYqr2b0hLwpBZVGp202ZzJ7s3a0XkgbADB0tiFenqA@mail.gmail.com>
 <87lgath01v.fsf@toke.dk> <52B2B44D-4382-404C-8F6D-03F12A72B11F@heistp.net>
 <31667353-48F2-4FAB-AC05-163680451719@toke.dk>
 <CACvFP_g+4VbfqdUu32=KMD_0b-fSjaKaktu+nHu89MtyAUW=iw@mail.gmail.com>
 <48ECB6C8-5D22-4785-A6CE-696D87EC5496@toke.dk>
 <73DD74AD-C2E7-4A12-AE49-C06D4486660E@gmail.com> <87fu10haw7.fsf@toke.dk>
 <8736wxco28.fsf@toke.dk> <BFEB4B8B-A705-407C-B4D1-D6DAF374C3C4@heistp.net>
 <87o9fkbtky.fsf@toke.dk> <E4F0A8C1-DA88-420C-B8EF-02D6A4F20972@heistp.net>
 <87lgaobq0m.fsf@toke.dk>
Date: Fri, 06 Jul 2018 13:33:19 +0200
X-Clacks-Overhead: GNU Terry Pratchett
Message-ID: <87in5sbnuo.fsf@toke.dk>
MIME-Version: 1.0
Content-Type: text/plain
Subject: Re: [Cake] cake at 60gbit
X-BeenThere: cake@lists.bufferbloat.net
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Cake - FQ_codel the next generation <cake.lists.bufferbloat.net>
List-Unsubscribe: <https://lists.bufferbloat.net/options/cake>,
 <mailto:cake-request@lists.bufferbloat.net?subject=unsubscribe>
List-Archive: <https://lists.bufferbloat.net/pipermail/cake>
List-Post: <mailto:cake@lists.bufferbloat.net>
List-Help: <mailto:cake-request@lists.bufferbloat.net?subject=help>
List-Subscribe: <https://lists.bufferbloat.net/listinfo/cake>,
 <mailto:cake-request@lists.bufferbloat.net?subject=subscribe>
X-List-Received-Date: Fri, 06 Jul 2018 11:33:23 -0000

AHA! Found the culprit!

The bulk dequeue mechanism in sch_generic.c will dequeue a bunch of
packets at once, then check if they belong on the same hardware txq. If
they don't, they will be put back on a separate queue in the qdisc
structure (sch->skb_bad_txq), and the qlen will be increased, without
telling the qdisc about it.

This obviously only happens on hardware with multiple TXQs, which is why
the bug doesn't happen on veth.

Now, to figure out what to do about it...

-Toke