From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-x241.google.com (mail-qk0-x241.google.com [IPv6:2607:f8b0:400d:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 227DA3BA8E for ; Sat, 2 Jun 2018 14:55:52 -0400 (EDT) Received: by mail-qk0-x241.google.com with SMTP id r66-v6so22377205qkr.7 for ; Sat, 02 Jun 2018 11:55:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=XQiM/zdfajevGW08xcD3WIpCfOFaGP068nEaCgNYaCg=; b=pa5+BMCxDyx/T1JSQlm4lk4ANJ87ayd18BL4unSHP3h+NJ1Sdir17rs++vT1T6yOz+ 9lXca8/rmfnhxKiHFzLFvP+bGCSmgIbn7g0ls0DQC7K2WAWtCCJ82FQ8XRQCnt6Amype ySGVFe1oWppXjnuZIu9ioVanm4M0G4ClUcC4+rJ+so1EftCTcN8CR/LKFtHv6PI1dIdk OqMkFEtwFNZTXPWH56QwSi+GJXOfPe6ULNBBUzsgqQHU0SnJ4BIuhs1evH+hPIai/QfB xlCAPfVypjrNVgh0tf3Nwl4Q7fKsU6YD2KvROnyfQhJBy4cxvTK7/jbuPlI14XaLUZy8 z2Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XQiM/zdfajevGW08xcD3WIpCfOFaGP068nEaCgNYaCg=; b=laZ386jI96SBzHXVG0Z14xG/lXBF4cHnNsfw1uW71GPfSwBda+S0snAyCaUwBExZk7 olwcVJLuy05uKyt66ruTir6bukJnDxOGeN2O3u5RUW4ljhgGabI95tiBGS86SkJtKQZA NQMGvSv+hdF6oUJ7mePJ4xsqeTlhFmMfeFeZUPlQefDauFFKQwuhN8DKes6Y0i2xFdwb 0Rj5Xdi0tv1dVOyuImT06dL/EzsHY2xnbM/hEld3HGLAf6LRbcTPHU9ksZglHU1SZxgH 6lJs4SDuZ8vKxwvdH6iaQ3czhFo+qSikcrrfrO6PIfK05phRk/xaFPykczFO49+NMU5c F5bg== X-Gm-Message-State: APt69E23DRUQdITEBRmD/KVtBZ6GeRSFKzoTSxswLVVwK+o4OYGbakz0 WVOx8F0490VmjhYdHXN3InUXeK7G/BwwuQ8VUVE= X-Google-Smtp-Source: ADUXVKIjQL3ObkQnq59FuEw5JVRGF2jTWdZitLpNXZFwgH8GH4dgUvml0WCEYf1AzRwS3hliBXHUeCC1sm5dBxCvqvE= X-Received: by 2002:a37:2801:: with SMTP id o1-v6mr14052078qkh.319.1527965751697; Sat, 02 Jun 2018 11:55:51 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:aed:24f0:0:0:0:0:0 with HTTP; Sat, 2 Jun 2018 11:55:50 -0700 (PDT) In-Reply-To: <87a7sdcan1.fsf@toke.dk> References: <87muwe9z7j.fsf@toke.dk> <87k1ri9yve.fsf@toke.dk> <877enimhuh.fsf@toke.dk> <9A273CA6-FDF6-485D-B466-8B01D4573BD9@gmail.com> <871sdqmg8l.fsf@toke.dk> <87a7sdcan1.fsf@toke.dk> From: Dave Taht Date: Sat, 2 Jun 2018 11:55:50 -0700 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Jonathan Morton , Cake List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Cake] Lockup at high speeds X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Jun 2018 18:55:52 -0000 On Sat, Jun 2, 2018 at 11:21 AM, Toke H=C3=B8iland-J=C3=B8rgensen wrote: > Dave Taht writes: > >> How on earth are you getting these speeds? I'm stuck at a pathetic *4* >> gbit here on the admittedly ancient 12 core boxes I'd got. >> (I'll go rework my veth topology) > > Well, having actual hardware do the packet forwarding, and only having > each machine process each packet once probably helps... > >> rcu stuff makes me nervous, what happens if that nat stuff is disabled e= ntirely. >> >> What if you made q.len and sparse/bullk flow counts atomic ops? > > The whole qdisc dequeue op is running while holding a global lock. The > freezes I'm seeing are happening because the other cores get stuck > waiting for the cake dequeue operation to complete (and thus release the > qdisc lock). Which is why I suspect an infinite loop somewhere... Am I wrong in remembering that some qdisc stats are multi-core (like qlen?)= , and summed at some other step? If you pin the card or qdisc to a single core what happens? (Besides it getting slower) > >> I tend to lean towards an overflow post 40gbit also. > > An overflow of what? 40 Gbps is 5 GBps, which is the unit that rates are > being measured in. So there's no obvious reason to expect that... Which > is not to say it couldn't happen, of course... > > -Toke --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619