From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4DA073B29D; Sat, 7 Jan 2023 09:53:18 -0500 (EST) Received: by mail-wm1-x334.google.com with SMTP id bi26-20020a05600c3d9a00b003d3404a89faso4897956wmb.1; Sat, 07 Jan 2023 06:53:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nmyS76K6NYQi3sV32j3pSGsXgwf+jDwxrvap97glzVQ=; b=M8LbGPYJzO0gEi1rp2S8JXfJHLQcLmF9NB5laVr8o7Er+GC7cmC/WhY62vr+VflNJE ul4oYe476vMf+gU5xohI9OXSeHYV4O7+783BpbFn7w6Fjm4z5jhUwn/UgpQrCkLZgNL4 ywjbBhktnhW+p1OXKtsNU0GE8cw93DKkFWiq9VQFhEADmN6BmPDB6TTBZ5WdKY8sM+go ctQXR35DfI7SKyU6R9SDQIZU+VMbElYUUOL+1v5riCtSWWzl5PJ1mWacsVU7qhycQq+l Mlx2gnwISM/zT4VnlmeEgzjdKunZEpmLOH0tyWAJHrcEpjoQlWoGXsGBiRmF+sMz7GPC pMPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nmyS76K6NYQi3sV32j3pSGsXgwf+jDwxrvap97glzVQ=; b=xECNaMUJlDPNGSve4pOAz7YaY9PZq84P9hItLAccH1ObDAWndhnUBim7DInRSf3QWX cietukVKYk5Li1TNLQVG/pYuAj9YpYS2nB0c+TV43CZY2oDsnVGCyaGy3fu6YZDjzwsA bIeAA9LDmqxYbc7gEfQRttPP1hPjeIoPFLLJVzima8VbAIbAlWY0xUOvkwnpwAuawmuM tFTeKNDIkYi90E+GfZKgjRjb6bD4m0DYoIJ9dACNMze5/8l+7IWZ6XmJfW2PsrL44/07 WX6Ogym+l8K0GVKF1slZVfZiid2drS4AlhQZdwpnPen89385yVqAvwwy+JScE+F9DU7g J1cw== X-Gm-Message-State: AFqh2krNQtb+ukyiEbH8FDrIQ+LuWZUjangSOVneBZLaCzmJtIs5OVbC OzT3X2XP8F7DtCJ7xNyq0cvNbHS9cSv7rYfs56w= X-Google-Smtp-Source: AMrXdXt4pNY13CcrrME5ebFZJTOBygVSrShZkIFL3xFRPPiG4MQB8rNjhY6/mTO05+IeR/ZYR0MmXMp6kPQp4hI1IE0= X-Received: by 2002:a05:600c:2309:b0:3d5:f77e:40b6 with SMTP id 9-20020a05600c230900b003d5f77e40b6mr1943899wmo.206.1673103196649; Sat, 07 Jan 2023 06:53:16 -0800 (PST) MIME-Version: 1.0 References: <714A007E-1589-45D1-BBD7-5A45CFCB9D08@gmail.com> In-Reply-To: <714A007E-1589-45D1-BBD7-5A45CFCB9D08@gmail.com> From: Dave Taht Date: Sat, 7 Jan 2023 06:52:54 -0800 Message-ID: To: Rich Brown Cc: bloat@lists.bufferbloat.net, libreqos Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] really lovely live plots of bandwidth, queue depth, marks and drops in libreqos X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Jan 2023 14:53:18 -0000 On Sat, Jan 7, 2023 at 4:57 AM Rich Brown via Bloat wrote: > > > > On Jan 6, 2023, at 11:14 PM, bloat-request@lists.bufferbloat.net wrote: > > Wearing my theorist hat... and looking at all the ISP plans > https://payne.taht.net is now emulating.... (click on bandwidth test, > then a plan) > > > This is really cool! I once wrote how, when I first worked on what became wifi, how hard it was to imagine gear that once took two refrigerator-sized wireless devices PLUS a refrigerator - to something that took up a tiny piece of a single chip, in only 20 years. Same goes here - I had until now, figured that we'd first see a traffic conditioner like this appear in an FPGA for some high end offloaded ethernet card, and then move down into logic, not be doable with a cheap off the shelf whitebox. My hat is off to robert chacon for getting something that worked, and herbert, for pushing it even further, and the half dozen early adoptor ISPs (that we know of) that are willing to deploy even beta versions on their customer bases (after having heard their support lines go silent) > I see the "Bandwidth Test", but don't see a place to look at a Plan... click on it, the tests will start to run, it will pick up the (presently) 20 plans we are are emulating... successfully show the RTTs are in line with codel theory (you'll see the top 10 worst RTTs end up ordered by 10/10 25/3 and so on) - then click on the link for the plan for those, and more detail comes to light. (feedback on what should be plotted and how, deeply desired) The testbed is from 3 hefty bare metal servers (temporarily) donated by equinix (thank you equinix!), the work, mostly volunteer, with a few dollars thrown in so far by the libreqos-users of the world (the biggest live libreqos site we know of so far is 10k subscribers pushing 11gbit). This is the first bufferbloat.net supported project that does have some chance at an actual business model - the going rate is 50 cents/subscriber/month from the commercial QoE vendors like Preseem, Paraqum, Cambium, etc - but as usual, to get the science right, it's open sourced (But gplv2, and I'm considering writing my bits of it as GPLv3). I think the QoE market is big enough for both closed and open versions of middleboxes like these to exist, and oy! have I now learned of all the other problems ISPs have in doing this, and deploying IPv6, and how much (at the higher rates) the real problems move to the customer wifi. We haven't automated outputting the flent results of those tests, and at the moment I'm mostly interested in validating results of the all the new tests like goresponsiveness, speedtest, etc. I hope to line up the authors of those tools to help. There's also emulations of a fifo, sfq, etc we can do fairly easily instead of just cake and fq_codel. Adding in proper delays to emulate technologies like DSL (20ms baseline latency), cable (10), fiber and fixed wireless (2) is fairly straightforward, some emulation of wifi's behaviors also. I made an attempt to correctly emulate 10k IP addresses (with those delays AND a routing protocol) and quickly ran the sole client box out of cpu - I'd had grand plans (see https://github.com/LibreQoE/LibreQoS/tree/main/sim) - but it's looking like we'll have to spin up a ton more client boxes, or adopt the same XDP approaches we're using in the middlebox, to emulate that many customers. I note that despite doing these emulations, and now having such a cool demo - the real purpose of the work was to improve the next release of libreqos (v1.4, targetted for mid-march) so it could have 6 sigmas of reliability, and could hopefully crack 50Gbit on 16 cores, perhaps even 100Gbit on 64 cores, in the real world. It is not, unfortunately, scaling linearly presently, and we keep putting in features and seeing performance get thwacked - I'd really wanted to sample stats at 10ms, not 50ms (nyquist theorem), no can do without a lot of new rust that would access the tc stats in binary rather than json. Rust eludes me... the lead dev is the author of a great book on rust (which still eludes me), There's been a couple other nice diversions along the way, presently the testbed is running with a single network port with in/out over two vlans. More github sponsors gladly accepted - I really wish those endpoints and edges that would like to build a metaverse would chip in, to me, this work is a win - win - win - or a business model, suggested. It's not just the enormous up-front cost of perfecting this that need covering, but six sigmas is *hard*.... comprehensive analytics, also... It's the most fun I've had on a project in a long time. If anyone would like to tag along the developer channel is #libreqos:matrix.org where I have become @dtaht:matrix.org. PS The names of the machines, btw, are based on starcraft - the client boxes and emulations are "zergXX", the middlebox "Payne", and the test drivers protossts. > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC