[Cake] Understanding the Achieved Rate Multiplication Effect in FlowQueue-based AQM Bottleneck

[David P. Reed]

There are lots of easy ways to (using Intel standard CPUs) get *submicrosecond* precision on clock times.
 
In TidalScale where I work, our hyperkernel (hypervisor) synchronizes the TSC (which has about 0.3 nsec resolution) between 10 GigE switch connected servers to an accuracy of maybe 100 nsec. We use a custom algorithm I designed, but the Precision Time Protocol standard easily gets sub-microsecond accurate measurements.
 
Now if you aren't measuring in a hypervisor, or very low level Linux kernel stack, I suggest that using DPDK (another Intel goodie that works on Linux) lets you do a lot in userspace code - including doing the whole IP stack in userspace.
 
I understand that a lot of Cake usage is about non-Intel, low-end consumer router processors, and in those it is really hard to time anything (and the scheduler itself is often struggling to schedule stuff predictably), but not every idea where measurement leads to significant latency improvement has to be initially explored on those low end consumer MIPS and ARM chips.
 
There are a number of single board computers that have at least two GigE ports and are Intel-64 Celerons or better that have good nanosecond TSC clocks, PCIe slots that support high-end wifi, etc.
 
I can recommend some...
 
 
On Saturday, December 4, 2021 8:24pm, "Dave Taht" <dave.taht at gmail.com> said:



> I too have been trying to get below 1ms (heck, 3ms) precision or at
> least, resolution. I came up with the most promising thing I can think
> of for
> interactions in a multithreaded environment yet, I think. glibc has
> long mapped the kernel clock page into process memory, so i was
> thinking
> (hoping) that mmaping that on top of itself a zillion times and using
> that as my test data source for writes out across the network I'd get
> some
> really fine grained insights.
> 
> Haven't got around to it yet.
> 
> On Sat, Dec 4, 2021 at 3:01 PM David P. Reed <dpreed at deepplum.com> wrote:
> >
> > I agree with your broad assessment, Jonathan.
> >
> >
> >
> > The self-interference problem within a host isn't just a network problem.
> It's a user-space scheduler problem as well.
> >
> >
> >
> > There are lots of interactions between user-space scheduler (in the case of
> Linux, the "Completely Fair Scheduler" and its quantum, which is set by the HZ
> variable at boot) and the network stack in the kernel. This interactions have
> non-trivial effects when mutliple flows are independently created by concurrent
> processes).
> >
> >
> >
> > Lately, I've been studying, for reasons related to my day job, the complex
> interactions of timing at sub-millisecond scale among threads and processes on a
> single system in Linux. I/O driven by threads become highly correlated, and so
> assuming "independence" among flow timing is just not a good assumption.
> >
> >
> >
> > The paper observes the results of "dependencies" that couple/resonate.
> >
> >
> >
> > On Friday, December 3, 2021 7:09pm, "Jonathan Morton"
> <chromatix99 at gmail.com> said:
> >
> > > > On 4 Dec, 2021, at 12:27 am, Dave Taht <dave.taht at gmail.com>
> wrote:
> > > >
> > > >
> > >
> https://jonathankua.github.io/preprints/jkua-ieeelcn2021_understanding_ar_preprint-20jul2021.pdf
> > > >
> > > > I would love it if somehow the measured effects of chunklets
> against cake's
> > > per-host/per flow fq was examined one day.
> > >
> > > I haven't actually measured it, but based on what the above paper says,
> I can make
> > > some firm predictions:
> > >
> > > 1: When competing against traffic to the same local host, the
> performance effects
> > > they describe will be present.
> > >
> > > 2: When competing against traffic to a different local-network host,
> the
> > > performance effects they describe will be attenuated or even entirely
> absent.
> > >
> > > 3: They noted one or two cases of observable effects of hash collisions
> in their
> > > tests with FQ-Codel. These will be greatly reduced in prevalence with
> Cake, due
> > > to the set-associative hash function which specifically addresses that
> phenomenon.
> > >
> > > - Jonathan Morton
> > > _______________________________________________
> > > Cake mailing list
> > > Cake at lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/cake
> > >
> 
> 
> 
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> 
> Dave Täht CEO, TekLibre, LLC
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/cake/attachments/20211207/d3cc1595/attachment.html>