From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 470313CB37 for ; Mon, 31 Oct 2022 23:32:08 -0400 (EDT) Received: by mail-wr1-x42b.google.com with SMTP id z14so18489270wrn.7 for ; Mon, 31 Oct 2022 20:32:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ub9KDeJicn08sAfyHsODWyG6PrzXO1ETerEGWG+PeRM=; b=av9iGX5LB+8gla4/UAApM77OOwbPBNjzVPqE1LIwi6HoVVR/xVBNkM4wIUeMu4oPFY 6Mj14nKa4oqmSMShIVpKE3lGE3J4ncKIMTmXkQMUK3PlOY+ISvewKqAG7hk6M+Lf1URF 2c15kuzuPAfwyCtqcSAPOiUqByT8AQMx64AzonLD/6m7NEqd9GN0Ldr1+faDqOgFC8bH i3wMR5R/GqkDq37WybrZzS/dbTfQyhColkBCKA66OWfhlKRMDsksk232Q3bhlYpT5wcn VNSp/w95W9QTUmOwKxHCX6K/oCOcJmOY0CR/YYlyxPiAaYsXJnetOm0MvIZ8FrSgUUwD UHkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ub9KDeJicn08sAfyHsODWyG6PrzXO1ETerEGWG+PeRM=; b=27oJxWqXn5lheXPNhEZsHYxomt1HL9HopIAAtLX/LrZTLUmaTDomjXbMHXmJpImhzu PL2jXM48kXMMW+XLlO3hbfVdiZqhRoZ9mzphmdLdLeiW1ZGQHzMMLebKcic53AMFGljz ovArTSV8eHVhp90KqI9e8XoA53+mrwTAowHuOrt02fqRNji4emLufU8NFlm7r/udT06H FmQc1uWuC2kId06EDWNrfWOzkPWhQqIIEFHbNUKKMEJIvqefneCZQFKjIdGSKlSF5wbE Puc3h7/YIFmuNyxxJf5SHmHJKAASSa12kDfCxzEEayEJtWhLte6LxUgC7dxx1sa5EeTB L8fg== X-Gm-Message-State: ACrzQf2nV3QypMKd9A4M3Myp76j42eF6IaTDQqOJwalJSpg/jvGIlzkb TclPGkkpkr+56HteIjdVH+gEFLuuBpIcPMHAluI= X-Google-Smtp-Source: AMsMyM4Sv629h8mYA8R3js7YhUj8Qi+cXkNvDq6jDDYU3G6mn/bxZyBkQpOHNa5JSHvBOymdqQjjgCTxU23ypW+NLp0= X-Received: by 2002:adf:f352:0:b0:236:ba3a:d58e with SMTP id e18-20020adff352000000b00236ba3ad58emr8047805wrp.430.1667273526784; Mon, 31 Oct 2022 20:32:06 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Mon, 31 Oct 2022 20:31:54 -0700 Message-ID: To: Herbert Wolverson Cc: libreqos Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [LibreQoS] Integration system, aka fun with graph theory X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2022 03:32:08 -0000 Calling rdtsc directly used to be even faster than gettimeofday https://github.com/dtaht/libv6/blob/master/erm/includes/get_cycles.h On Mon, Oct 31, 2022 at 2:20 PM Herbert Wolverson via LibreQoS wrote: > > I'd agree with color coding (when it exists - no rush, IMO) being configu= rable. > > From the "how much delay are we adding" discussion earlier, I thought I'd= do a little bit of profiling of the BPF programs themselves. This is with = the latest round of performance updates (https://github.com/thebracket/cpum= ap-pping/issues/2), so it's not measuring anything in production. I simply = added a call to get the clock at the start, and again at the end - and log = the difference. Measuring both XDP and TC BPF programs. (Execution goes (pa= cket arrives)->(XDP cpumap sends it to the right CPU)->(egress)->(TC sends = it to the right classifier, on the correct CPU and measures RTT latency). T= his is adding about two clock checks and a debug log entry to execution tim= e, so measuring it is slowing it down. > > The results are interesting, and mostly tell me to try a different measur= ement system. I'm seeing a pretty wide variance. Hammering it with an iperf= session and a queue capped at 5 gbit/s: most of the TC timings were 40 nan= oseconds - not a packet that requires extra tracking, already in cache, so = proceed. When the TCP RTT tracker fired and recorded a performance event, i= t peaked at 5,900 nanoseconds. So the tc xdp program seems to be adding a w= orst-case of 0.0059 ms to packet times. The XDP side of things is typically= in the 300-400 nanosecond range, I saw a handful of worst-case numbers in = the 3400 nanosecond range. So the XDP side is adding 0.00349 ms. So - assum= ing worst case (and keeping the overhead added by the not-so-great monitori= ng), we're adding 0.0093 ms to packet transit time with the BPF programs. > > With a much more sedate queue (ceiling 500 mbit/s), I saw much more consi= stent numbers. The vast majority of XDP timings were in the 75-150 nanoseco= nd range, and TC was a consistent 50-55 nanoseconds when it didn't have an = update to perform - peaking very occasionally at 1500 nanoseconds. Only add= ing 0.00155 ms to packet times is pretty good. > > It definitely performs best on long streams, probably because the previou= s lookups are all in cache. This is also making me question the answer I fo= und to "how long does it take to read the clock?" I'd seen ballpark estimat= es of 53 nanoseconds. Given that this reads the clock twice, that can't be = right. (I'm *really* not sure how to measure that one) > > Again - not a great test (I'll have to learn the perf system to do this p= roperly - which in turn opens up the potential for flame graphs and some pr= oper tracing). Interesting ballpark, though. > > On Mon, Oct 31, 2022 at 10:56 AM dan wrote: >> >> >> >> On Sun, Oct 30, 2022 at 8:21 PM Dave Taht via LibreQoS wrote: >>> >>> How about the idea of "metaverse-ready" metrics, with one table that is= preseem-like and another that's >>> >>> blue =3D < 8ms >>> green =3D < 20ms >>> yellow =3D < 50ms >>> orange =3D < 70ms >>> red =3D > 70ms >> >> >> These need configurable. There are a lot of wisps that would have every= thing orange/red. We're considering anything under 100ms good on the rural= plans. Also keep in mind that if you're tracking latence via pping etc, = then you need some buffer in there for the internet at large. <70ms to Ama= zon is one thing, they're very well connected, but <70ms to most of the int= ernet isn't probably very realistic and would make most charts look like po= op. > > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC