From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-x244.google.com (mail-qk0-x244.google.com [IPv6:2607:f8b0:400d:c09::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5E4DF3B29E for ; Thu, 23 Aug 2018 14:20:03 -0400 (EDT) Received: by mail-qk0-x244.google.com with SMTP id 89-v6so4253947qkp.2 for ; Thu, 23 Aug 2018 11:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5QLg+u7+mVnnt0dXrC4sOcJiCckoAPgjLIfK2wJXM/k=; b=dPq2VSk7ukKkHOxpbkB/XnJtOXTkLuh4g3gP+KnjAouMhW/WF0mp9QnsX6a4ljC6fm wIooOPGtF9LuotMTRxk5KBA4O1PuxFLz71Mi7lVemTvOVyy0NUwlDctdL2LWOOZpjWpK RXpiteGqQ0NSlRJ4Hu62swK6tQGgqTXlQs6uXpBkjPtcbgiT8V5aGdW5cVs2fYEbsC/t rfvOPhGOgH6Ol9XlFcOHYxAp1vey/3qynnny78in1jl+dW2yCfvgJeCQS86z9PD0QeYG tvzcDrKUAc8JpsGLWUP7W8l0f4nh+wTbdAtmbr+m5X7wPCjxq456NYQLoLgoBamZqfYA rV1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5QLg+u7+mVnnt0dXrC4sOcJiCckoAPgjLIfK2wJXM/k=; b=sbC6KXE0GquIbs0QCzWkrWvb3eMNrKQmo+HWVAQ2WTLB3DV4AlgGBtSxI4/1QnL/Lg hqdAOKQXZCwZaNJUCTinTkyJQ8EwRNxDGx6Zo4QZTfZac1STw+jLj4icrEYjj7uXkbWD 3tYVqOIqim51vv1wsmhLa7HRhyhgP2EgJZIlRbgRX6+Tf7n1dXsZpAgOyXtyfwFm1Cck fauTMieHpEWDTgbegS/BFiVy1F8fhpS/eBflPhpXUVeDa7wCTFxSBK/Ils/78JqRViyv Ks/Jbzuj4LCvNAQSkM9mj9urJ74djxG8u+XuUnc3BERDZGSGL4IpUdtwHlMV0S2Xr8So AwfQ== X-Gm-Message-State: APzg51Bq7HjVV9CgW61Lr95ciyeMV0yhxaQUbLWpJ7i+Q3edWrbWk6PR YMlO3re6zKJ4p8gD6swWlm4aotH5W5r1ooNkMdk= X-Google-Smtp-Source: ANB0VdbXTdwLzN7ypYUbVi+6+aNHvcS/j4c7fggV+vmXwpo4XxQJ8FmcopAZJTsMgx4PSbWIVKUitpsDhyOu4CGllaA= X-Received: by 2002:a37:4f4b:: with SMTP id d72-v6mr16894257qkb.17.1535048402885; Thu, 23 Aug 2018 11:20:02 -0700 (PDT) MIME-Version: 1.0 References: <66e2374b-f998-b132-410e-46c9089bb06b@gmail.com> <360212B1-8411-4ED0-877A-92E59070F518@gmx.de> In-Reply-To: From: Dave Taht Date: Thu, 23 Aug 2018 11:21:33 -0700 Message-ID: To: Mikael Abrahamsson Cc: Rosen Penev , bloat Content-Type: text/plain; charset="UTF-8" Subject: Re: [Bloat] [Cerowrt-devel] beating the drum for BQL X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 18:20:03 -0000 One of the things not readily evident in trying to scale up, is the cost of even the most basic routing table lookup. A lot of good work in this area landed in linux 4.1 and 4.2 (see a couple posts here: https://vincent.bernat.im/en/blog/2017-performance-progression-ipv4-route-lookup-linux ) Lookup time for even the smallest number of routes is absolutely miserable for IPv6 - https://vincent.bernat.im/en/blog/2017-ipv6-route-lookup-linux I think one of the biggest driving factors of the whole TSO/GRO thing is due to trying to get smaller packets through this phase of the kernel, and not that they are so much more efficient at the card itself. Given the kerfuffle over here ( https://github.com/systemd/systemd/issues/9725 ) I'd actually like to come up with a way to move the linux application socket buffers to the post-lookup side of the routing table. We spend a lot of extra time bloating up superpackets just so they are cheaper to route. TCAMs are expensive as hell, but the addition of even a small one, readily accessible to userspace or from the kernel, might help in the general case. I've actually oft wished to be able to offload these sort of lookups into higher level algorithms and languages like python, as a general purpose facility. Hey, if we can have giant GPUs, why can't our cpus have tcams? programmable TCAM support got enabled in a recent (mellonox?) product. Can't find the link at the moment TCAMs of course, is where big fat dedicated routers and switches shine, over linux - and even arp table lookups are expensive in linux, though I'm not sure if anyone has looked lately.