From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 991FF3B29E for ; Wed, 24 May 2023 11:31:59 -0400 (EDT) Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-3f42ba32e24so9607995e9.3 for ; Wed, 24 May 2023 08:31:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684942318; x=1687534318; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=21Dyyah6H8eNJcD6SPL1HNazojiwuKltTpyVz5Qqao0=; b=rn4ra9YKev8nN4ub4rae44T9aXSK/mSYLIOGnHmKeHNrKLMuGSEFxCLktn5zi1xwgf FErdUABaewYineGwWjf7uNcXNqMM/MGB8noUd1xHpiGA/B4SzKJ/iICAl4zA02GeEaij IVYH1ayjWvCLEtXTbPghXF/6kWAB32wk9aDXHZJNdEkdgVK5hilufwtlT2fX0aqlagHR Z30eMCDP34chnHoYTWjnX8Hafwhq3XD/ABWk9IBZ3/vVostEjIwML/+iCUIxu2cPGygE +qndZk72PzPhEN8kdOtbq4gj//t0DvRtc2LgP4OxG1DLSkoeJebMKqyTQLQtNlYQOLV0 81RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684942318; x=1687534318; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=21Dyyah6H8eNJcD6SPL1HNazojiwuKltTpyVz5Qqao0=; b=ZYBqgAyd4ymgW9xWwPmi314OwQzgWChxv394GH78sbrMNHmXyAbEv4pmEJo+Wn5xXP NVKpLAYHMtX1K2aBB5kWpL6gnSYgApXW3q4vRGSHm1ldABy319pTz568/bWfQ1RloWAw BdmIwwevbMEM9sKDYAVTK6ye/zKYasuhL+VnIuqsHT6H7Jg4CkznjDUDbYIflB+048ob gayJfz+KO+FjvtYxcB5p6ac8BX1ZS+zp10HOV/UbBXDJGHdPfVaOShEsiGkmUjr3Lkt4 ZkkxzxQKelkUjrBxiNKvMCT+49ESn30KOx7ZdzYvc1jZP/BbOS4/7hbsK10RRdyh4C46 24cw== X-Gm-Message-State: AC+VfDygh4kz7kmtUrch3N5u2Qj/cN3+afYf09BsTLhV3puMDDSKNsRS u0evPPv50LHHRFwDrAPdpM1fQvJLekiAjqxw1JQRaXzds8c= X-Google-Smtp-Source: ACHHUZ4WJW8ANXMbEWNjJu2dwa3B+sDu/GyapQ3mzyJcUOKaEdd8ewW0pCTjoj9QOQQrNd4tCT+OO29F5th5PROW1Qo= X-Received: by 2002:a05:600c:248:b0:3f4:fffc:cd74 with SMTP id 8-20020a05600c024800b003f4fffccd74mr132893wmj.16.1684942318023; Wed, 24 May 2023 08:31:58 -0700 (PDT) MIME-Version: 1.0 References: <0no84q43-s4n6-45n8-50or-12o3rq104n99@ynat.uz> <48b00469-0dbb-54c4-bedb-3aecbf714a1a@auckland.ac.nz> <728orr66-1432-751p-263q-sqopr12s20sq@ynat.uz> <077e6ad1-d7cc-2d57-39f8-e9646bea32a5@auckland.ac.nz> <09552rq0-0n24-0pqo-4085-n918r0n71138@ynat.uz> <9q29o7n2-69rs-3os1-s93q-0795262qs3o1@ynat.uz> <1c9df33b-e964-f531-7326-1a11b159e6a7@auckland.ac.nz> <5307.1684939754@localhost> In-Reply-To: <5307.1684939754@localhost> From: Dave Taht Date: Wed, 24 May 2023 09:31:45 -0600 Message-ID: To: Michael Richardson Cc: Ulrich Speidel , "starlink@lists.bufferbloat.net" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2023 15:31:59 -0000 On Wed, May 24, 2023 at 8:49=E2=80=AFAM Michael Richardson wrote: > > > Dave Taht via Starlink wrote: > > These are the biggest reliability reasons why I think FQ is *necess= ary* > > across the edges of the internet. > > It saved your bacon, but yeah, like all other resilient protocols (DNS, > Happy Eyeballs) tends to hide when one option is failing :-) > > > pure AQM, in the case above, since that flood was uncontrollable, w= ould > > have resulted in a 99.99% or so drop rate for all other traffic. Wh= ile > > that would have been easier to diagnose I suppose, the near term > > outcome would have been quite damaging. > > What this says is that fq_codel doesn't have enough management reporting > interfaces. Going back 25 years, this has always been a problem with ho= me > routers: ntop3 is great, but it's not easy to use, and it's not that > accessible, and it often can't see things that move around. > > > I always try to make a clear distinction between FQ and AQM techniq= ues. > > Both are useful and needed, for different reasons (but in the gener= al > > case, I think the DRR++ derived FQ in fq_codel is the cats pajamas,= and > > far more important than any form of AQM) > > Could fq_codel emit flow statistics as a side-effect of it's classificati= ons? It does. It always has. "tc -s class show" gives details of each queue. it is a 5 tuple hash by default. This can of course be overridden via filters to use another classification method.There are a few on-router tools that do process this and provide a nice dashboard. This could be better, in trying to identify problematic flows, but would require more in kernel (ebpf?) processing than we have yet attempted on a home router. AI is also on our minds. Most of my focus for the past year has been in getting cake to scale as an ISP middlebox, in Libreqos. For example in LibreQos we are presently very successful in sampling cake queue data at my preferred sample rate (10ms), in production, for up to about 1k subscribers. However, in production, with 10k subs (11gbit), sampling at 1s rates is where we are now. (that is 40 million queues sampled once per second). I am sure we can improve the sample rates at high levels of subs further... compress reporting, etc, but until now, most ISPs only had 5 minute averages to look at. There are some really cool things you can do at high sample rates. Here is a live/realtime movie of what netflix actually looks like: https://www.youtube.com/watch?v=3DC-2oSBr2200 (also) Another thing is that real traffic, displayed as we do it now, is kind of mesmerizing, and looks very different from what we generate via flent, on the testbed. Anyway, on the libreqos front now we have over 30 ISPs and 98 folk participating in the chat room, please feel free to hang out with us: https://app.element.io/#/room/#libreqos:matrix.org - ask questions, propose tests and plots.... I return y=C3=A1ll now to starlink (which could really use this stuff!) > -- > ] Never tell me the odds! | ipv6 mesh netwo= rks [ > ] Michael Richardson, Sandelman Software Works | IoT architec= t [ > ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails= [ > --=20 Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227= 111937/ Dave T=C3=A4ht CSO, LibreQos