From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 334CD3CB37 for ; Mon, 31 Oct 2022 17:57:57 -0400 (EDT) Received: by mail-ej1-x633.google.com with SMTP id t25so32825640ejb.8 for ; Mon, 31 Oct 2022 14:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jackrabbitwireless.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=N7qYFgVGKhlbisvvIXtPzsaGHLTYQvrCQRqg1QLlS78=; b=3i5l5rKoU0Tx49U6YHgf2N3vI5re6sH5pW81u13UdOd1mcvxGQoB0hZ92EEqLGJOdv 3F9NGeYwJyh+JlUhKsi3Ky+6q0whXw7WcDfiTMCq5lmYwoxLKhxuvehpWOWXzoPDamhL UAKlPGXZ45sFHq1sRf7YagPHjAAAHj8WrkVEY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=N7qYFgVGKhlbisvvIXtPzsaGHLTYQvrCQRqg1QLlS78=; b=vzcskhO4iJ73PHHTd9P2hszm8sjPVYtFgDtLhmSFMtBhG4F+jyVF4Kb0ye0fpjSp8H 6qHVHPaGBFp4ndTdai1YwgGTXK4Sv5g8uTzL2WTVUJtpDG+yv35NIf61AjOGWvxJ0dAd FDJwNKXQdZr7MOMboImxVB8hNyM1KDD/HNqT+DrWlBUKhPg2tFDi1ZOzl3M2vzFhWgNF 8xnAPa84i/tzUCWX4lFCpmIjeUYbXTa93vGnoM8clBMambiEVlgE1BldtIZNXgxqSgPK oHXm2n0q9dAfPywrfcCnmTJuSTvJpT0mhtzu1T5tmZwFA+JHfTfOeNPvhib+ov1wwOao PJAA== X-Gm-Message-State: ACrzQf1i8UnRlT2vz1ovoWeBcgztZN6OzbZlcO5DDSUst601YhonH4ps 1ixjjYowpv2dsT9Pjbu8RYx2c8n8+5FG/mD1Fc5Kfw== X-Google-Smtp-Source: AMsMyM4j9Xs10KFJyIILiP/ukbD9zMkx7bm3XZpNIFHmQVtTV80Tz7QFHXXpaaWNCjmEr6RRaUBqPKZ/cuqWaK0QmSU= X-Received: by 2002:a17:906:d9ce:b0:78d:dddb:3974 with SMTP id qk14-20020a170906d9ce00b0078ddddb3974mr12646ejb.411.1667253476092; Mon, 31 Oct 2022 14:57:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Robert_Chac=C3=B3n?= Date: Mon, 31 Oct 2022 15:57:44 -0600 Message-ID: To: Herbert Wolverson Cc: libreqos Content-Type: multipart/alternative; boundary="00000000000031875705ec5bb3f4" Subject: Re: [LibreQoS] Integration system, aka fun with graph theory X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Oct 2022 21:57:57 -0000 --00000000000031875705ec5bb3f4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > I'd agree with color coding (when it exists - no rush, IMO) being configurable. Thankfully it will be configurable, and easily, through the InfluxDB interface. Any operator will be able to click the Gear icon above the tables and set the thresholds to whatever is desired. I've set it to include both a standard table and "metaverse-ready" table based on Dave's threshold recommendations. - Standard (Preseem like) - green =3D < 75 ms - yellow =3D < 100 ms - red =3D > 100 ms - Metaverse-Ready - blue =3D < 8ms - green =3D < 20ms - yellow =3D < 50ms - orange =3D < 70ms - red =3D > 70ms Are the defaults here reasonable at least? Should we change the Standard table thresholds a bit? > Only adding 0.00155 ms to packet times is pretty good. Agreed! That's excellent. Great work on this so far it's looking like you're making tremendous progress. On Mon, Oct 31, 2022 at 3:20 PM Herbert Wolverson via LibreQoS < libreqos@lists.bufferbloat.net> wrote: > I'd agree with color coding (when it exists - no rush, IMO) being > configurable. > > From the "how much delay are we adding" discussion earlier, I thought I'd > do a little bit of profiling of the BPF programs themselves. This is with > the latest round of performance updates ( > https://github.com/thebracket/cpumap-pping/issues/2), so it's not > measuring anything in production. I simply added a call to get the clock = at > the start, and again at the end - and log the difference. Measuring both > XDP and TC BPF programs. (Execution goes (packet arrives)->(XDP cpumap > sends it to the right CPU)->(egress)->(TC sends it to the right classifie= r, > on the correct CPU and measures RTT latency). This is adding about two > clock checks and a debug log entry to execution time, so measuring it is > slowing it down. > > The results are interesting, and mostly tell me to try a different > measurement system. I'm seeing a pretty wide variance. Hammering it with = an > iperf session and a queue capped at 5 gbit/s: most of the TC timings were > 40 nanoseconds - not a packet that requires extra tracking, already in > cache, so proceed. When the TCP RTT tracker fired and recorded a > performance event, it peaked at 5,900 nanoseconds. So the tc xdp program > seems to be adding a worst-case of 0.0059 ms to packet times. The XDP sid= e > of things is typically in the 300-400 nanosecond range, I saw a handful o= f > worst-case numbers in the 3400 nanosecond range. So the XDP side is addin= g > 0.00349 ms. So - assuming worst case (and keeping the overhead added by t= he > not-so-great monitoring), we're adding *0.0093 ms* to packet transit time > with the BPF programs. > > With a much more sedate queue (ceiling 500 mbit/s), I saw much more > consistent numbers. The vast majority of XDP timings were in the 75-150 > nanosecond range, and TC was a consistent 50-55 nanoseconds when it didn'= t > have an update to perform - peaking very occasionally at 1500 nanoseconds= . > Only adding 0.00155 ms to packet times is pretty good. > > It definitely performs best on long streams, probably because the previou= s > lookups are all in cache. This is also making me question the answer I > found to "how long does it take to read the clock?" I'd seen ballpark > estimates of 53 nanoseconds. Given that this reads the clock twice, that > can't be right. (I'm *really* not sure how to measure that one) > > Again - not a great test (I'll have to learn the perf system to do this > properly - which in turn opens up the potential for flame graphs and some > proper tracing). Interesting ballpark, though. > > On Mon, Oct 31, 2022 at 10:56 AM dan wrote: > >> >> >> On Sun, Oct 30, 2022 at 8:21 PM Dave Taht via LibreQoS < >> libreqos@lists.bufferbloat.net> wrote: >> >>> How about the idea of "metaverse-ready" metrics, with one table that is >>> preseem-like and another that's >>> >>> blue =3D < 8ms >>> green =3D < 20ms >>> yellow =3D < 50ms >>> orange =3D < 70ms >>> red =3D > 70ms >>> >> >> These need configurable. There are a lot of wisps that would have >> everything orange/red. We're considering anything under 100ms good on t= he >> rural plans. Also keep in mind that if you're tracking latence via ppi= ng >> etc, then you need some buffer in there for the internet at large. <70m= s >> to Amazon is one thing, they're very well connected, but <70ms to most o= f >> the internet isn't probably very realistic and would make most charts lo= ok >> like poop. >> > _______________________________________________ > LibreQoS mailing list > LibreQoS@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos > --=20 Robert Chac=C3=B3n CEO | JackRabbit Wireless LLC Dev | LibreQoS.io --00000000000031875705ec5bb3f4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> I'd agree with color coding (when it exists = - no rush, IMO) being configurable.

Thankfully it = will be configurable, and easily, through the InfluxDB interface.
Any operator will be able to click the Gear icon above the tables and set the thresholds to whatever is desired.
I've set it to includ= e both a standard table and "metaverse-ready" table based on Dave= 's threshold recommendations.
  • Standard (Preseem like)=
    • green =3D < 75 ms
    • yellow =3D < 100 ms
    • red =3D > 100 ms
  • Metaverse-Ready
      • blue =3D =C2=A0< 8ms
    • green =3D < 20ms
    • yellow =3D &l= t; 50ms
    • orange =C2=A0=3D < 70ms
    • red =3D > 70ms
Are the defaults here reasonable at least? Sh= ould we change the Standard table thresholds a bit?

> Only adding 0.00155 ms to packet times is pretty good.
Agreed! That's excellent. Great work on this so far it'= ;s looking like you're making tremendous progress.

<= div class=3D"gmail_quote">
On Mon, Oct= 31, 2022 at 3:20 PM Herbert Wolverson via LibreQoS <libreqos@lists.bufferbloat.net> wrote= :
I'd agree with color coding (when it exists - no rush, IMO) bei= ng configurable.

From the "how much delay= are we adding" discussion earlier, I thought I'd do a little bit = of profiling of the BPF programs themselves. This is with the latest round = of performance updates (https://github.com/thebracket/cpumap-pping/i= ssues/2), so it's not measuring anything in production. I simply ad= ded a call to get the clock at the start, and again at the end - and log th= e difference. Measuring both XDP and TC BPF programs. (Execution goes (pack= et arrives)->(XDP cpumap sends it to the right CPU)->(egress)->(TC= sends it to the right classifier, on the correct CPU and measures RTT late= ncy). This is adding about two clock checks and a debug log entry to execut= ion time, so measuring it is slowing it down.

= The results are interesting, and mostly tell me to try a different measurem= ent system. I'm seeing a pretty wide variance. Hammering it with an ipe= rf session and a queue capped at 5 gbit/s: most of the TC timings were 40 n= anoseconds - not a packet that requires extra tracking, already in cache, s= o proceed. When the TCP RTT tracker fired and recorded a performance event,= it peaked at 5,900 nanoseconds. So the tc xdp program seems to be adding a= worst-case of 0.0059 ms to packet times. The XDP side of things is typical= ly in the 300-400 nanosecond range, I saw a handful of worst-case numbers i= n the 3400 nanosecond range. So the XDP side is adding 0.00349 ms. So - ass= uming worst case (and keeping the overhead added by the not-so-great monito= ring), we're adding 0.0093 ms to packet transit time with the BP= F programs.

With a much more sedate queue (ceiling= 500 mbit/s), I saw much more consistent numbers. The vast majority of XDP = timings were in the 75-150 nanosecond range, and TC was a consistent 50-55 = nanoseconds when it didn't have an update to perform - peaking very occ= asionally at 1500 nanoseconds. Only adding 0.00155 ms to packet times is pr= etty good.

It definitely performs best on long str= eams, probably because the previous lookups are all in cache. This is also = making me question the answer I found to "how long does it take to rea= d the clock?" I'd seen ballpark estimates of 53 nanoseconds. Given= that this reads the clock twice, that can't be right. (I'm *really= * not sure how to measure that one)

Again - no= t a great test (I'll have to learn the perf system to do this properly = - which in turn opens up the potential for flame graphs and some proper tra= cing). Interesting ballpark, though.

On Mon, Oct 31, 2022 at 10:56= AM dan <danden= son@gmail.com> wrote:


On Sun, Oct 30, 2022 at = 8:21 PM Dave Taht via LibreQoS <libreqos@lists.bufferbloat.net> wrote:
=
How about the idea of "metaverse-ready" metrics, with one ta= ble that is preseem-like and another that's

blue =3D=C2=A0 < 8ms
green =3D < 20ms
yellow = =3D < 50ms
orange=C2=A0 =3D < 70ms
red =3D > 7= 0ms

These need configurable.=C2=A0 There a= re a lot of wisps that would have everything orange/red.=C2=A0 We're co= nsidering anything under 100ms good on the rural plans.=C2=A0 =C2=A0Also ke= ep in mind that if you're tracking latence via pping etc, then you need= some buffer in there for the internet at large.=C2=A0 <70ms to Amazon i= s one thing, they're very well connected, but <70ms to most of the i= nternet isn't probably very realistic and would make most charts look l= ike poop.=C2=A0
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos

--
Robert Chac=C3=B3n

--00000000000031875705ec5bb3f4--