From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CFC753B29E for ; Tue, 18 Oct 2022 14:01:23 -0400 (EDT) Received: by mail-pl1-x635.google.com with SMTP id l1so14546674pld.13 for ; Tue, 18 Oct 2022 11:01:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=9B1pMueFFeJKxuEQpwH12yCTz9T7OoWb/AMbcr9Df4E=; b=dnr9qCP3YdRY585ENhPSOHss0VYTtJNpggOZVk+fITGEtI2gY5ZZQHdZYxFp+5EutM 54ZQJedlS7ag6fKOyg24VIIK8Jvc+eLyKhBetxG5GEaQwTrmjCUcHWeLB2OwL56Cz1NR wo/y2K+p6Tfksw1vSHgAmNwnne07aACUVX1r1zzCejTRXAF/U90zZi56DzsY8jotsbAp 9X+v/hDDTQjYG0l2ZF1mMewHltb45le58ABTIDs4K84kLnpxjOD8Ti54KEJ9eG60l7Ep Rr+e8uHML0iSkOHGA9K3V0Hcc52PMBuTLPo7j8cQ5A6JvCyL0vBgibigmDMLjHM0HhJ5 s0NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9B1pMueFFeJKxuEQpwH12yCTz9T7OoWb/AMbcr9Df4E=; b=P1xgSLQM11PI5GLRt5I+DuX2Nd/MlCc0CKJ9NBpS3NkRqH9qLmOIO0kDrl2NHkrKDG LndMf/D7CmQKzjt9TOnFv4j3Pbc4+4qH9zseTXpxzu0nRWRv1eFNc42lzZbmHswIHSXN YnVixFkM6SvKtOVEXPIglhMecOSfTTMzAp6YySu5ZBnqFwlhjwOzbTDuo3WeZ/rT4TZa PFPYs4BPa2ZhtKTU1Z0hWnnys+qamJhwVVGdctEe03Rz62um3ahkG9GvnWcSw2wj/kbm FCsCnMCi2wh3FMoz5BMGg3Vfe8nbcQgvyI7PvnmfnG3toSCbs7SwaNfTm1iScFrxlabY mYpw== X-Gm-Message-State: ACrzQf1pT2I0XsrQ1+3EO1H6c5uW/XlV6Ykmo9RgR4uFxIHqb4ZChfhv BugF/3ODSaGC+flyPLDe5tDBTCMrJyS7YA7Ctb+0i1oRVrM= X-Google-Smtp-Source: AMsMyM7nO+2EBMWIl9LD2siBiSq6dIERiZZFccRxf8eMJoWldBV/s2Q/q3ZuUDEMu5goTbSeCgNSMBnc46eNzcFvPxI= X-Received: by 2002:a17:902:dad2:b0:17f:7439:20f2 with SMTP id q18-20020a170902dad200b0017f743920f2mr4256374plx.29.1666116082209; Tue, 18 Oct 2022 11:01:22 -0700 (PDT) MIME-Version: 1.0 References: <87bkqatu61.fsf@toke.dk> <759c25c6fd54dceccc00eada5ccf5358d2d1c20c.camel@kau.se> In-Reply-To: From: Herbert Wolverson Date: Tue, 18 Oct 2022 13:01:11 -0500 Message-ID: Cc: "libreqos@lists.bufferbloat.net" Content-Type: multipart/alternative; boundary="0000000000003c1b6605eb52e179" Subject: Re: [LibreQoS] In BPF pping - so far X-BeenThere: libreqos@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Many ISPs need the kinds of quality shaping cake can do List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Oct 2022 18:01:24 -0000 --0000000000003c1b6605eb52e179 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It's probably not entirely thread-safe right now (ran into some issues reading per_cpu maps back from userspace; hopefully, I'll get that figured out) - but the commits I just pushed have it basically working on single-stream testing. :-) Setup cpumap as usual, and periodically run xdp-pping. This gives you per-connection RTT information in JSON: [ {"tc":"1:5", "avg" : 5, "min" : 5, "max" : 5, "samples" : 1}, {}] (With the extra {} because I'm not tracking the tail and haven't done comma removal). The tool also empties the various maps used to gather data, acting as a "reset" point. There's a max of 60 samples per queue, in a ringbuffer setup (so newest will start to overwrite the oldest). I'll start trying to test on a larger scale now. On Mon, Oct 17, 2022 at 3:34 PM Robert Chac=C3=B3n < robert.chacon@jackrabbitwireless.com> wrote: > Hey Herbert, > > Fantastic work! Super exciting to see this coming together, especially so > quickly. > I'll test it soon. > I understand and agree with your decision to omit certain features (ICMP > tracking,DNS tracking, etc) to optimize performance for our use case. Lik= e > you said, in order to merge the functionality without a performance hit, > merging them is sort of the only way right now. Otherwise there would be = a > lot of redundancy and lost throughput for an ISP's use. Though hopefully > long term there will be a way to keep all projects working independently > but interoperably with a plugin system of some kind. > > By the way, I'm making some headway on LibreQoS v1.3. Focusing on > optimizations for high sub counts (8000+ subs) as well as stateful change= s > to the queue structure. > I'm working to set up a physical lab to test high throughput and high > client count scenarios. > When testing beyond ~32,000 filters we get "no space left on device" from > xdp-cpumap-tc, which I think relates to the bpf map size limitation you > mentioned. Maybe in the coming months we can take a look at that. > > Anyway great work on the cpumap-pping program! Excited to see more on thi= s. > > Thanks, > Robert > > On Mon, Oct 17, 2022 at 12:45 PM Herbert Wolverson via LibreQoS < > libreqos@lists.bufferbloat.net> wrote: > >> Hey, >> >> My current (unfinished) progress on this is now available here: >> https://github.com/thebracket/cpumap-pping-hackjob >> >> I mean it about the warnings, this isn't at all stable, debugged - and >> can't promise that it won't unleash the nasal demons >> (to use a popular C++ phrase). The name is descriptive! ;-) >> >> With that said, I'm pretty happy so far: >> >> * It runs only on the classifier - which xdp-cpumap-tc has nicely shunte= d >> onto a dedicated CPU. It has to run on both >> the inbound and outbound classifiers, since otherwise it would only se= e >> half the conversation. >> * It does assume that your ingress and egress CPUs are mapped to the sam= e >> interface; I do that anyway in BracketQoS. Not doing >> that opens up a potential world of pain, since writes to the shared >> maps would require a locking scheme. Too much locking, and you lose all = of >> the benefit of using multiple CPUs to begin with. >> * It is pretty wasteful of RAM, but most of the shaper systems I've >> worked with have lots of it. >> * I've been gradually removing features that I don't want for BracketQoS= . >> A hypothetical future "useful to everyone" version wouldn't do that. >> * Rate limiting is working, but I removed the requirement for a shared >> configuration provided from userland - so right now it's always set to >> report at 1 second intervals per stream. >> >> My testbed is currently 3 Hyper-V VMs - a simple "client" and "world", >> and a "shaper" VM in between running a slightly hacked-up LibreQoS. >> iperf from "client" to "world" (with Libre set to allow 10gbit/s max, vi= a >> a cake/HTB queue setup) is around 5 gbit/s at present, on my >> test PC (the host is a core i7, 12th gen, 12 cores - 64gb RAM and fast >> SSDs) >> >> Output currently consists of debug messages reading: >> cpumap/0/map:4-1371 [000] D..2. 515.399222: bpf_trace_printk: (tc= ) >> Flow open event >> cpumap/0/map:4-1371 [000] D..2. 515.399239: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 374696 >> cpumap/0/map:4-1371 [000] D..2. 515.399466: bpf_trace_printk: (tc= ) >> Flow open event >> cpumap/0/map:4-1371 [000] D..2. 515.399475: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 247069 >> cpumap/0/map:4-1371 [000] D..2. 516.405151: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 5217155 >> cpumap/0/map:4-1371 [000] D..2. 517.405248: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4515394 >> cpumap/0/map:4-1371 [000] D..2. 518.406117: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4481289 >> cpumap/0/map:4-1371 [000] D..2. 519.406255: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4255268 >> cpumap/0/map:4-1371 [000] D..2. 520.407864: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 5249493 >> cpumap/0/map:4-1371 [000] D..2. 521.406664: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 3795993 >> cpumap/0/map:4-1371 [000] D..2. 522.407469: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 3949519 >> cpumap/0/map:4-1371 [000] D..2. 523.408126: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4365335 >> cpumap/0/map:4-1371 [000] D..2. 524.408929: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4154910 >> cpumap/0/map:4-1371 [000] D..2. 525.410048: bpf_trace_printk: (tc= ) >> Send performance event (5,1), 4405582 >> cpumap/0/map:4-1371 [000] D..2. 525.434080: bpf_trace_printk: (tc= ) >> Send flow event >> cpumap/0/map:4-1371 [000] D..2. 525.482714: bpf_trace_printk: (tc= ) >> Send flow event >> >> The times haven't been tweaked yet. The (5,1) is tc handle major/minor, >> allocated by the xdp-cpumap parent. >> I get pretty low latency between VMs; I'll set up a test with some >> real-world data very soon. >> >> I plan to keep hacking away, but feel free to take a peek. >> >> Thanks, >> Herbert >> >> On Mon, Oct 17, 2022 at 10:14 AM Simon Sundberg >> wrote: >> >>> Hi, thanks for adding me to the conversation. Just a couple of quick >>> notes. >>> >>> On Mon, 2022-10-17 at 16:13 +0200, Toke H=C3=B8iland-J=C3=B8rgensen wro= te: >>> > [ Adding Simon to Cc ] >>> > >>> > Herbert Wolverson via LibreQoS >>> writes: >>> > >>> > > Hey, >>> > > >>> > > I've had some pretty good success with merging xdp-pping ( >>> > > >>> https://github.com/xdp-project/bpf-examples/blob/master/pping/pping.h ) >>> > > into xdp-cpumap-tc ( https://github.com/xdp-project/xdp-cpumap-tc )= . >>> > > >>> > > I ported over most of the xdp-pping code, and then changed the entr= y >>> point >>> > > and packet parsing code to make use of the work already done in >>> > > xdp-cpumap-tc (it's already parsed a big chunk of the packet, no >>> need to do >>> > > it twice). Then I switched the maps to per-cpu maps, and had to pin >>> them - >>> > > otherwise the two tc instances don't properly share data. >>> > > >>> >>> I guess the xdp-cpumap-tc ensures that the same flow is processed on >>> the same CPU core at both ingress or egress. Otherwise, if a flow may >>> be processed by different cores on ingress and egress the per-CPU maps >>> will not really work reliably as each core will have a different view >>> on the state of the flow, if there's been a previous packet with a >>> certain TSval from that flow etc. >>> >>> Furthermore, if a flow is always processed on the same core (on both >>> ingress and egress) I think per-CPU maps may be a bit wasteful on >>> memory. From my understanding the keys for per-CPU maps are still >>> shared across all CPUs, it's just that each CPU gets its own value. So >>> all CPUs will then have their own data for each flow, but it's only the >>> CPU processing the flow that will have any relevant data for the flow >>> while the remaining CPUs will just have an empty state for that flow. >>> Under the same assumption that packets within the same flow are always >>> processed on the same core there should generally not be any >>> concurrency issues with having a global (non-per-CPU) either as packets >>> from the same flow cannot be processed concurrently then (and thus no >>> concurrent access to the same value in the map). I am however still >>> very unclear on if there's any considerable performance impact between >>> global and per-CPU map versions if the same key is not accessed >>> concurrently. >>> >>> > > Right now, output >>> > > is just stubbed - I've still got to port the perfmap output code. >>> Instead, >>> > > I'm dumping a bunch of extra data to the kernel debug pipe, so I ca= n >>> see >>> > > roughly what the output would look like. >>> > > >>> > > With debug enabled and just logging I'm now getting about 4.9 >>> Gbits/sec on >>> > > single-stream iperf between two VMs (with a shaper VM in the >>> middle). :-) >>> > >>> > Just FYI, that "just logging" is probably the biggest source of >>> > overhead, then. What Simon found was that sending the data from kerne= l >>> > to userspace is one of the most expensive bits of epping, at least wh= en >>> > the number of data points goes up (which is does as additional flows >>> are >>> > added). >>> >>> Yhea, reporting individual RTTs when there's lots of them (you may get >>> upwards of 1000 RTTs/s per flow) is not only problematic in terms of >>> direct overhead from the tool itself, but also becomes demanding for >>> whatever you use all those RTT samples for (i.e. need to log, parse, >>> analyze etc. a very large amount of RTTs). One way to deal with that is >>> of course to just apply some sort of sampling (the -r/--rate-limit and >>> -R/--rtt-rate >>> > >>> > > So my question: how would you prefer to receive this data? I'll hav= e >>> to >>> > > write a daemon that provides userspace control (periodic cleanup as >>> well as >>> > > reading the performance stream), so the world's kinda our oyster. I >>> can >>> > > stick to Kathie's original format (and dump it to a named pipe, >>> perhaps?), >>> > > a condensed format that only shows what you want to use, an efficie= nt >>> > > binary format if you feel like parsing that... >>> > >>> > It would be great if we could combine efforts a bit here so we don't >>> > fork the codebase more than we have to. I.e., if "upstream" epping an= d >>> > whatever daemon you end up writing can agree on data format etc that >>> > would be fantastic! Added Simon to Cc to facilitate this :) >>> > >>> > Briefly what I've discussed before with Simon was to have the ability >>> to >>> > aggregate the metrics in the kernel (WiP PR [0]) and have a userspace >>> > utility periodically pull them out. What we discussed was doing this >>> > using an LPM map (which is not in that PR yet). The idea would be tha= t >>> > userspace would populate the LPM map with the keys (prefixes) they >>> > wanted statistics for (in LibreQOS context that could be one key per >>> > customer, for instance). Epping would then do a map lookup into the >>> LPM, >>> > and if it gets a match it would update the statistics in that map ent= ry >>> > (keeping a histogram of latency values seen, basically). Simon's PR >>> > below uses this technique where userspace will "reset" the histogram >>> > every time it loads it by swapping out two different map entries when >>> it >>> > does a read; this allows you to control the sampling rate from >>> > userspace, and you'll just get the data since the last time you polle= d. >>> >>> Thank's Toke for summarzing both the current state and the plan going >>> forward. I will just note that this PR (and all my other work with >>> ePPing/BPF-PPing/XDP-PPing/I-suck-at-names-PPing) will be more or less >>> on hold for a couple of weeks right now as I'm trying to finish up a >>> paper. >>> >>> > I was thinking that if we all can agree on the map format, then your >>> > polling daemon could be one userspace "client" for that, and the eppi= ng >>> > binary itself could be another; but we could keep compatibility betwe= en >>> > the two, so we don't duplicate effort. >>> > >>> > Similarly, refactoring of the epping code itself so it can be plugged >>> > into the cpumap-tc code would be a good goal... >>> >>> Should probably do that...at some point. In general I think it's a bit >>> of an interesting problem to think about how to chain multiple XDP/tc >>> programs together in an efficent way. Most XDP and tc programs will do >>> some amount of packet parsing and when you have many chained programs >>> parsing the same packets this obviously becomes a bit wasteful. In the >>> same time it would be nice if one didn't need to manually merge >>> multiple programs together into a single one like this to get rid of >>> this duplicated parsing, or at least make that process of merging those >>> programs as simple as possible. >>> >>> >>> > -Toke >>> > >>> > [0] https://github.com/xdp-project/bpf-examples/pull/59 >>> >>> N=C3=A4r du skickar e-post till Karlstads universitet behandlar vi dina >>> personuppgifter. >>> When you send an e-mail to Karlstad University, we will process your >>> personal data. >>> >> _______________________________________________ >> LibreQoS mailing list >> LibreQoS@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/libreqos >> > > > -- > Robert Chac=C3=B3n > CEO | JackRabbit Wireless LLC > --0000000000003c1b6605eb52e179 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
It's probably not entirely thread-safe right now = (ran into some issues reading per_cpu maps back from userspace; hopefully, = I'll get that figured out) - but the commits I just pushed have it basi= cally working on single-stream testing. :-)

Setup = cpumap as usual, and periodically run xdp-pping. This gives you per-connection RTT information in JSON:

[
{"t= c":"1:5", "avg" : 5, "min" : 5, "ma= x" : 5, "samples" : 1},
{}]

(With the extra {} because I'm not tracking the tail and haven't= done comma removal). The tool also empties the various maps used to gather= data, acting as a "reset" point. There's a max of 60 samples= per queue, in a ringbuffer setup (so newest will start to overwrite the ol= dest).

I'll start trying to test on a larger s= cale now.

On Mon, Oct 17, 2022 at 3:34 PM Robert Chac=C3=B3n <<= a href=3D"mailto:robert.chacon@jackrabbitwireless.com">robert.chacon@jackra= bbitwireless.com> wrote:
Hey Herbert,

Fantastic work! Super exciting to see this coming together, especially so = quickly.
I'll test it soon.
I understand and agree = with your decision to omit certain features (ICMP tracking,DNS tracking, et= c) to optimize performance for our use case. Like you said, in order to mer= ge the functionality without a performance hit, merging them is sort of the= only way right now. Otherwise there would be a lot of redundancy and lost = throughput for an ISP's use. Though hopefully long term there will be a= way to keep all projects working independently but interoperably with a pl= ugin system of some kind.

By the way, I'm= making some headway on LibreQoS v1.3. Focusing on optimizations for high s= ub counts (8000+ subs) as well as stateful changes to the queue structure.<= /div>
I'm working to set up a physical lab to test high throughput = and high client count scenarios.
When testing beyond ~32,000 filt= ers we get "no space left on device" from xdp-cpumap-tc, which I = think relates to the bpf map size limitation you mentioned. Maybe in the co= ming months we can take a look at that.

Anyway gre= at work on the cpumap-pping program! Excited to see more on this.

Thanks,
Robert

On Mon, Oct 17, 2022 at 1= 2:45 PM Herbert Wolverson via LibreQoS <libreqos@lists.bufferbloat.net> = wrote:
Hey,

My current (unfinished) progres= s on this is now available here: https://github.com/thebracket/cpumap= -pping-hackjob

I mean it about the warnings, t= his isn't at all stable, debugged - and can't promise that it won&#= 39;t unleash the nasal demons
(to use a popular C++ phrase). The = name is descriptive! ;-)

With that said, I'm p= retty happy so far:

* It runs only on the classifi= er - which xdp-cpumap-tc has nicely shunted onto a dedicated CPU. It has to= run on both
=C2=A0 the inbound and outbound classifiers, since o= therwise it would only see half the conversation.
* It does a= ssume that your ingress and egress CPUs are mapped to the same interface; I= do that anyway in BracketQoS. Not doing
=C2=A0 that opens up a p= otential world of pain, since writes to the shared maps would require a loc= king scheme. Too much locking, and you lose all of the benefit of using mul= tiple CPUs to begin with.
* It is pretty wasteful of RAM, but mos= t of the shaper systems I've worked with have lots of it.
* I= 've been gradually removing features that I don't want for BracketQ= oS. A hypothetical future "useful to everyone" version wouldn'= ;t do that.
* Rate limiting is working, but I removed the require= ment for a shared configuration provided from userland - so right now it= 9;s always set to report at 1 second intervals per stream.
My testbed is currently 3 Hyper-V VMs - a simple "client&= quot; and "world", and a "shaper" VM in between running= a slightly hacked-up LibreQoS.
iperf from "client"= ; to "world" (with Libre set to allow 10gbit/s max, via a cake/HT= B queue setup) is around 5 gbit/s at present, on my
test PC = (the host is a core i7, 12th gen, 12 cores - 64gb RAM and fast SSDs)
<= div>
Output currently consists of debug messages reading:
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 515.39922= 2: bpf_trace_printk: (tc) Flow open event
=C2=A0 cpumap/0/map:4-1371 =C2= =A0 =C2=A0[000] D..2. =C2=A0 515.399239: bpf_trace_printk: (tc) Send perfor= mance event (5,1), 374696
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] = D..2. =C2=A0 515.399466: bpf_trace_printk: (tc) Flow open event
=C2=A0 c= pumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 515.399475: bpf_trace_pr= intk: (tc) Send performance event (5,1), 247069
=C2=A0 cpumap/0/map:4-13= 71 =C2=A0 =C2=A0[000] D..2. =C2=A0 516.405151: bpf_trace_printk: (tc) Send = performance event (5,1), 5217155
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2= =A0[000] D..2. =C2=A0 517.405248: bpf_trace_printk: (tc) Send performance e= vent (5,1), 4515394
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. = =C2=A0 518.406117: bpf_trace_printk: (tc) Send performance event (5,1), 448= 1289
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 519.4062= 55: bpf_trace_printk: (tc) Send performance event (5,1), 4255268
=C2=A0 = cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 520.407864: bpf_trace_p= rintk: (tc) Send performance event (5,1), 5249493
=C2=A0 cpumap/0/map:4-= 1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 521.406664: bpf_trace_printk: (tc) Sen= d performance event (5,1), 3795993
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2= =A0[000] D..2. =C2=A0 522.407469: bpf_trace_printk: (tc) Send performance e= vent (5,1), 3949519
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. = =C2=A0 523.408126: bpf_trace_printk: (tc) Send performance event (5,1), 436= 5335
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 524.4089= 29: bpf_trace_printk: (tc) Send performance event (5,1), 4154910
=C2=A0 = cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 525.410048: bpf_trace_p= rintk: (tc) Send performance event (5,1), 4405582
=C2=A0 cpumap/0/map:4-= 1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 525.434080: bpf_trace_printk: (tc) Sen= d flow event
=C2=A0 cpumap/0/map:4-1371 =C2=A0 =C2=A0[000] D..2. =C2=A0 = 525.482714: bpf_trace_printk: (tc) Send flow event

The times haven't been tweaked yet. The (5,1) is tc handle major/minor= , allocated by the xdp-cpumap parent.
I get pretty low laten= cy between VMs; I'll set up a test with some real-world data very soon.=

I plan to keep hacking away, but feel free to= take a peek.

Thanks,
Herbert
<= /div>
O= n Mon, Oct 17, 2022 at 10:14 AM Simon Sundberg <Simon.Sundberg@kau.se> wrote:
=
Hi, thanks for addi= ng me to the conversation. Just a couple of quick
notes.

On Mon, 2022-10-17 at 16:13 +0200, Toke H=C3=B8iland-J=C3=B8rgensen wrote:<= br> > [ Adding Simon to Cc ]
>
> Herbert Wolverson via LibreQoS <libreqos@lists.bufferbloat.net> wri= tes:
>
> > Hey,
> >
> > I've had some pretty good success with merging xdp-pping ( > > https://github.com/xd= p-project/bpf-examples/blob/master/pping/pping.h )
> > into xdp-cpumap-tc ( https://github.com/xdp-pr= oject/xdp-cpumap-tc ).
> >
> > I ported over most of the xdp-pping code, and then changed the en= try point
> > and packet parsing code to make use of the work already done in > > xdp-cpumap-tc (it's already parsed a big chunk of the packet,= no need to do
> > it twice). Then I switched the maps to per-cpu maps, and had to p= in them -
> > otherwise the two tc instances don't properly share data.
> >

I guess the xdp-cpumap-tc ensures that the same flow is processed on
the same CPU core at both ingress or egress. Otherwise, if a flow may
be processed by different cores on ingress and egress the per-CPU maps
will not really work reliably as each core will have a different view
on the state of the flow, if there's been a previous packet with a
certain TSval from that flow etc.

Furthermore, if a flow is always processed on the same core (on both
ingress and egress) I think per-CPU maps may be a bit wasteful on
memory. From my understanding the keys for per-CPU maps are still
shared across all CPUs, it's just that each CPU gets its own value. So<= br> all CPUs will then have their own data for each flow, but it's only the=
CPU processing the flow that will have any relevant data for the flow
while the remaining CPUs will just have an empty state for that flow.
Under the same assumption that packets within the same flow are always
processed on the same core there should generally not be any
concurrency issues with having a global (non-per-CPU) either as packets
from the same flow cannot be processed concurrently then (and thus no
concurrent access to the same value in the map). I am however still
very unclear on if there's any considerable performance impact between<= br> global and per-CPU map versions if the same key is not accessed
concurrently.

> > Right now, output
> > is just stubbed - I've still got to port the perfmap output c= ode. Instead,
> > I'm dumping a bunch of extra data to the kernel debug pipe, s= o I can see
> > roughly what the output would look like.
> >
> > With debug enabled and just logging I'm now getting about 4.9= Gbits/sec on
> > single-stream iperf between two VMs (with a shaper VM in the midd= le). :-)
>
> Just FYI, that "just logging" is probably the biggest source= of
> overhead, then. What Simon found was that sending the data from kernel=
> to userspace is one of the most expensive bits of epping, at least whe= n
> the number of data points goes up (which is does as additional flows a= re
> added).

Yhea, reporting individual RTTs when there's lots of them (you may get<= br> upwards of 1000 RTTs/s per flow) is not only problematic in terms of
direct overhead from the tool itself, but also becomes demanding for
whatever you use all those RTT samples for (i.e. need to log, parse,
analyze etc. a very large amount of RTTs). One way to deal with that is
of course to just apply some sort of sampling (the -r/--rate-limit and
-R/--rtt-rate
>
> > So my question: how would you prefer to receive this data? I'= ll have to
> > write a daemon that provides userspace control (periodic cleanup = as well as
> > reading the performance stream), so the world's kinda our oys= ter. I can
> > stick to Kathie's original format (and dump it to a named pip= e, perhaps?),
> > a condensed format that only shows what you want to use, an effic= ient
> > binary format if you feel like parsing that...
>
> It would be great if we could combine efforts a bit here so we don'= ;t
> fork the codebase more than we have to. I.e., if "upstream" = epping and
> whatever daemon you end up writing can agree on data format etc that > would be fantastic! Added Simon to Cc to facilitate this :)
>
> Briefly what I've discussed before with Simon was to have the abil= ity to
> aggregate the metrics in the kernel (WiP PR [0]) and have a userspace<= br> > utility periodically pull them out. What we discussed was doing this > using an LPM map (which is not in that PR yet). The idea would be that=
> userspace would populate the LPM map with the keys (prefixes) they
> wanted statistics for (in LibreQOS context that could be one key per > customer, for instance). Epping would then do a map lookup into the LP= M,
> and if it gets a match it would update the statistics in that map entr= y
> (keeping a histogram of latency values seen, basically). Simon's P= R
> below uses this technique where userspace will "reset" the h= istogram
> every time it loads it by swapping out two different map entries when = it
> does a read; this allows you to control the sampling rate from
> userspace, and you'll just get the data since the last time you po= lled.

Thank's Toke for summarzing both the current state and the plan going forward. I will just note that this PR (and all my other work with
ePPing/BPF-PPing/XDP-PPing/I-suck-at-names-PPing) will be more or less
on hold for a couple of weeks right now as I'm trying to finish up a paper.

> I was thinking that if we all can agree on the map format, then your > polling daemon could be one userspace "client" for that, and= the epping
> binary itself could be another; but we could keep compatibility betwee= n
> the two, so we don't duplicate effort.
>
> Similarly, refactoring of the epping code itself so it can be plugged<= br> > into the cpumap-tc code would be a good goal...

Should probably do that...at some point. In general I think it's a bit<= br> of an interesting problem to think about how to chain multiple XDP/tc
programs together in an efficent way. Most XDP and tc programs will do
some amount of packet parsing and when you have many chained programs
parsing the same packets this obviously becomes a bit wasteful. In the
same time it would be nice if one didn't need to manually merge
multiple programs together into a single one like this to get rid of
this duplicated parsing, or at least make that process of merging those
programs as simple as possible.


> -Toke
>
> [0] https://github.com/xdp-project/bpf-examp= les/pull/59

N=C3=A4r du skickar e-post till Karlstads universitet behandlar vi dina per= sonuppgifter<https://www.kau.se/gdpr>.
When you send an e-mail to Karlstad University, we will process your person= al data<https://www.kau.se/en/gdpr>.
_______________________________________________
LibreQoS mailing list
LibreQo= S@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/libreqos


--
Robert Chac=C3=B3n
CEO | JackRabbit Wireless LLC
--0000000000003c1b6605eb52e179--