From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DDE633B2A4 for ; Thu, 3 Nov 2022 07:48:40 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1667476115; bh=n//H1t1DFzfBYWgRnxlMhnsNa1bI3bZe9dMqrl7s7MY=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=NDFKcsEUVvODWoBhjJdvz0/sYNPNfCh9FQrZkV8AYKaAJOa7YaLrxFaSLJqfslbZ4 HUKg35FQsB6z9NDWIuAhVYU3kTTsYLcsz5oI9LnsBKuOXykz8pm4n6leVCYXWbh/B6 LaVwgTDOqgF9IlNQvrvVJ5KQmpQnagu9/Bih/2stHhsOMOeAuYHaJ4cxSf+Io3gUgg 1n+RQxTm9vOCsE8RypdW9AyLjYe/qhTEB5W7hEoMlBSxOCx43wsi5itr2Gn4K3kuGF c/5gDIwttqXMUPDyYrD4zaXV4SwGahSk2oyzfNcM0sCg2NNDWWCdHWUVNAhpNvjJ9+ RC4jXUvxoCVQQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N49lJ-1oyytG2b5d-01092w; Thu, 03 Nov 2022 12:48:35 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: Date: Thu, 3 Nov 2022 12:48:33 +0100 Cc: rjmcmahon , Rpm , IETF IPPM WG , aesomerville@gmail.com Content-Transfer-Encoding: quoted-printable Message-Id: <3ED5FA7B-C94E-4B04-A2DB-34DCD0F6642F@gmx.de> References: <0a8cc31c7077918bf84fddf9db50db02@rjmcmahon.com> <344f2a33b6bcae4ad4390dcb96f92589@rjmcmahon.com> <261B90F5-FD4E-46D5-BEFE-6BF12D249A28@gmx.de> <6DEE77BC-2E17-4192-9173-3E78D0164AD6@gmx.de> To: Ruediger.Geib@telekom.de X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:bPue4byPbCSzKIQCGHIfAvg4ac1qpLnSYG48Hj05mDyj1VsYUm/ 5evXIgG7/7JBQB52pq3qRmLmv1lFBKHIS8zLLUfecQv9BvZivnphGUXaG/k7fOMSm5YqGyN Y0GLNYPTZDU7CepY+3FiHbWnrXESFZWXnJiqV88lDtN59CmCSrf9VLB7Aaqbp/PEFHtpT8m gbq1pimLBWUCWxBEoKh4A== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:ZYhaGfxs9MA=;1CA2g3z4Oc0QF8P/53fVbgt0oj6 SOF7gEah3CpARS8c+i0ao/5gSFFhBZR+BY1sGMPltdC8a96DYSK0SDWRhzrml8/EQhq4WDZOp +CpjzOT8sZ9oSOSdmI5WlDxyC/l5TMIFFceTaFqAwCM4mwKkEw3OabHZCKa3a5je8OaJGdBGa a2fFdx0b3jRCT4/tAWWl6Ih8XUoqxtexrbamjt9aKABUcihuhPOkJSivjnuDpSNeR8XFCoSRE ziRvcepS+2EvMG+ZDdi5qqbWeGiTAIUHddXuQrlMl2P8usPtSMcIAvqTk87mqOVXIf1ZPIOMf g5PrkYqzMx00H6tfINJXOsj3VY7eT4QVzsebUOqEkuoQuqLSqEYqpEzs1SsWMdQFBQ59HyU7n aG6YhpW/H9F6HWbXNhKB611nL70X/8ToL55vFke0JV3R06+iReHE8podERqG7SMe7XrGIjUhe 6ccWw2u4l0DLz07e/k5b48G83CriFXse0LQHdvy5J5AcQS6S+OPIOy+cGpnaEbmwM3tSp8li1 lUgLXZ/fius5oPa1tV521RTC2ZZRtkEi73XywF5MsHLHfZfzoL6FbK5fiD3iyayrvC14eSpho AEUGr18cqbpLuaTT+9mYJvIE3PAjEKSlx6UgXAPRt/VwSIdjLS8bT2crmnp07dknyjtD5mIP5 xg53/iqgg/EUq18MLQbVXAB1AMPLTPx/rjwwW1ASwxRgQXTgnMQSKfX8RAxd1AmM192CyOjcD HAtuf2YRk99Dg85nlLFGXnJ5YtYg7HUURFWwrStbZFTw193eJdznlv1WAZMOpBEjgmm+sM/RR KdFiu4OjL9WM57MC2kghX29/nrBRk5Hr/nL4SNZiD7PJVc/7eMPst8QOfIOKd7AZ6ENSZ7Ydq XsRU75TB9HG8GdZ9W5XRpuxtaI46Dqq3x8R671/5tKA2vJZToViSmnYtw0MgmnGsCzlkvX0KO LAKgK10sCBJVFEw+FO1hf+AtnQM= Subject: Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Nov 2022 11:48:41 -0000 Hi Ruediger, many thanks for the interesting discussion. > On Nov 3, 2022, at 12:25, = wrote: >=20 > Hi Sebastian, >=20 > Quick feedback [RG2] >=20 > ----------------------------------------- > Hi Ruediger, >=20 >> On Nov 3, 2022, at 09:20, = wrote: >>=20 >> Hi Sebastian, >>=20 >> [SM] Pragmatically we work with delay increase over baseline, which = seems to work well enough to be useful, while it is unlikely to be = perfect. >>=20 >> RG: I agree. And I appreciate "well enough" - many factors may impact = delays along a measurement path, causing e.g. temporal noise or = permanent change. >=20 > [SM] Exactly! We do try to account for some of these factors to = some degree: > a) we use a set of diverse reflectors by default and use a voting = mechanism and declare "congested_state" only if W out of the last X = delay samples (from Y reflectors) were above threshold Z (all of W, X, = Y, and Z are configurable to tune the code for specific links, but the = defaults seem to already improve perceived latency-under-load noticeably = for many/most users that reported back to us). > [RG2] Allright, that approach is reasonable. >=20 > b) we keep individual "baseline" values for each reflector that = iintend to model the path-dependent delay component, we adjust the = baseline with two rules: > A) if a sample was larger that baseline, we feed it into an EWMA = to update/increase the baseline slightly (which will allow the baseline = to slowly grow to larger values if e.g. a path changed and now is = longer); the assumption here is that underestimating the "true" baseline = will make the controller more "trigger-happy" which will keep latency = low at the expense of potential throughput, and we prefer that transient = loss in potential throughput over a similarly transient decrease in = responsiveness. > B) if a sample is smaller than the baseline we immediately = update the baseline to that value. (One rationale for that is that = during prolonged congestion epochs the EWMA method in A) will have = artificially increased our base line estimate so we want to be quick to = correct it again if the congestion relaxes even for a short duration; = the other is to quickly adapt in case a path change results in a shorter = path delay; again the principle is that we value responsiveness over = throughput). >=20 > [RG2] That's a tricky and absolutely required part, differentiating a = path change from congestion. A comment on B) - please keep in mind, that = the baseline path may have a smaller delay, but immediately face = congestion. That may not be an issue, if the congestion (buffer-fill) to = be detected reliably adds latency in a range significantly above the = monitored physical delay. [SM2] Yes! We have to deal with a set of known unknowns here and = hence we need to employ a few heuristics. About the min-delay being = inflated already by persistent congestion, yes we are blind to that no = ifs not buts. Rule B) tries to keep such errors as short as possible in = that we reduce our baseline estimate immediately after we have evidence = that we over-estimated it. That does as you observed help little if = congestion stays persistently high. Personally I am lucky enough that I = rarelely encountered something like this on my access link, but that is = just that luck. Our main goal is to control a known bottleneck (like an access = link) and not the full internet path to a specific target location so = most relevant traffic should be self generated so we have some rough = estimate on the relative load of our link (but the available capacity = share might still depend on other users in a shared segment so our load = estimate is only approximate otherwise our job would be done already, if = we know what fraction of our available capacity we currentky use and our = current traffic rate we could directly calculate the shaper limit to = keep responsiveness high ;) ). Regards Sebastian > =20 >=20 > =20 >=20 >> Regards, >>=20 >> Ruediger >>=20 >>=20 >>=20 >> -----Urspr=C3=BCngliche Nachricht----- >> Von: Sebastian Moeller >> Gesendet: Mittwoch, 2. November 2022 22:41 >> An: Geib, R=C3=BCdiger >> Cc: rjmcmahon ; Rpm=20 >> ; ippm@ietf.org >> Betreff: Re: [ippm] [Rpm] lightweight active sensing of bandwidth and=20= >> buffering >>=20 >> Dear Ruediger, >>=20 >> thank you very much for your helpful information. I will chew over = this and see how/if I can exploit these "development of congestion = observations" somehow.=20 >> The goal of these plots is not primarily to detect congestion* (that = would be the core of autorate's functionality, detect increases in delay = and respond in reducing the shaper rate to counter act them), but more = to show how well this works (the current rationale is that compared to a = situation without traffic shaping the difference in high versus low-load = CDFs should be noticeably** smaller). >>=20 >> *) autorate will be in control of an artificial bottleneck and we do = measure the achieved throughput per direction, so we can reason about = "congestion" based on throughput and delay; the loading is organic in = that we simply measure the traffic volume per time of what travels over = the relevant interfaces, the delay measurements however are active, = which has its pros and cons... >> **) Maybe even run a few statistical tests, like = Mann-Withney-U/Wilcoxon ranksum test and then claim "significantly = smaller". I feel a parametric t-test might not be in order here, with = delay PDFs decidedly non-normal in shape (then again they likely are = mono-modal, so t-test would still work okayish in spite of its core = assumption being violated). >>=20 >>=20 >>> On Nov 2, 2022, at 10:41, = wrote: >>>=20 >>> Bob, Sebastian, >>>=20 >>> not being active on your topic, just to add what I observed on = congestion: >>=20 >> [SM] I will try to explain how/if we could exploit your = observations=20 >> for our controller >>=20 >>> - starts with an increase of jitter, but measured minimum delays = still remain constant. Technically, a queue builds up some of the time, = but it isn't present permanently. >>=20 >> [SM] So in that phase we would expect CDFs to have different = slopes, higher variance should result in shallower slope? As for using = this insight for the actual controller, I am not sure how that would = work; maybe maintaining a "jitter" base line per reflector and test = whether each new sample deviates significantly from that base line? That = is similar to the approach we are currently taking with delay/RTT. >>=20 >>> - buffer fill reaches a "steady state", called bufferbloat on access=20= >>> I think >>=20 >> [SM] I would call it buffer bloat if that steady-state results = in too high delays increases (which to a degree is a subjective = judgement). Although in accordance with the Nichols/Jacobsen analogy of = buffers/queues as shock absorbers a queue with with acceptable = steady-state induced delay might not work too well to even out = occasional bursts? >>=20 >>> ; technically, OWD increases also for the minimum delays, jitter now = decreases (what you've described that as "the delay magnitude" decreases = or "minimum CDF shift" respectively, if I'm correct). >>=20 >> [SM] That is somewhat unfortunate as it is harder to detect = quickly than something that simply increases and stays high (like RTT). >>=20 >>> I'd expect packet loss to occur, once the buffer fill is on steady = state, but loss might be randomly distributed and could be of a low = percentage. >>=20 >> [SM] Loss is mostly invisible to our controller (it would need = to affect our relatively thin active delay measurement traffic we have = no insight into the rest of the traffic), but more than that the = controller's goal is to avoid this situation so hopefully it will be = rare and transient. >>=20 >>> - a sudden rather long load burst may cause a jump-start to = "steady-state" buffer fill. >>=20 >> [SM] As would a rather steep drop in available capacity with = traffic in-flight sized to the previous larger capacity. This is e.g. = what can be observed over shared media like docsis/cable and GSM = successors. >>=20 >>=20 >>> The above holds for a slow but steady load increase (where the = measurement frequency determines the timescale qualifying "slow"). >>> - in the end, max-min delay or delay distribution/jitter likely = isn't an easy to handle single metric to identify congestion. >>=20 >> [SM] Pragmatically we work with delay increase over baseline, = which seems to work well enough to be useful, while it is unlikely to be = perfect. The CDFs I plotted are really just for making sense post hoc = out of the logged data... (cake-autorate is currently designed to = maintain a "flight-recorder" log buffer that can be extracted after = noticeable events, and I am trying to come up with how to slice and dice = the data to help explain "noticeable events" from the limited log data = we have). >>=20 >> Many Thanks & Kind Regards >> Sebastian >>=20 >>=20 >>>=20 >>> Regards, >>>=20 >>> Ruediger >>>=20 >>>=20 >>>> On Nov 2, 2022, at 00:39, rjmcmahon via Rpm = wrote: >>>>=20 >>>> Bufferbloat shifts the minimum of the latency or OWD CDF. >>>=20 >>> [SM] Thank you for spelling this out explicitly, I only worked = on a vage implicit assumption along those lines. However what I want to = avoid is using delay magnitude itself as classifier between high and low = load condition as that seems statistically uncouth to then show that the = delay differs between the two classes;).=20 >>> Yet, your comment convinced me that my current load threshold = (at least for the high load condition) probably is too small, exactly = because the "base" of the high-load CDFs coincides with the base of the = low-load CDFs implying that the high-load class contains too many = samples with decent delay (which after all is one of the goals of the = whole autorate endeavor). >>>=20 >>>=20 >>>> A suggestion is to disable x-axis auto-scaling and start from zero. >>>=20 >>> [SM] Will reconsider. I started with start at zero, end then=20 >>> switched to an x-range that starts with the delay corresponding to=20= >>> 0.01% for the reflector/condition with the lowest such value and=20 >>> stops at 97.5% for the reflector/condition with the highest delay=20 >>> value. My rationale is that the base delay/path delay of each=20 >>> reflector is not all that >>> informative* (and it can still be learned from reading the x-axis),=20= >>> the long tail > 50% however is where I expect most differences so I=20= >>> want to emphasize this and finally I wanted to avoid that the actual=20= >>> "curvy" part gets compressed so much that all lines more or less=20 >>> coincide. As I said, I will reconsider this >>>=20 >>>=20 >>> *) We also maintain individual baselines per reflector, so I could = just plot the differences from baseline, but that would essentially = equalize all reflectors, and I think having a plot that easily shows = reflectors with outlying base delay can be informative when selecting = reflector candidates. However once we actually switch to OWDs baseline = correction might be required anyways, as due to colck differences ICMP = type 13/14 data can have massive offsets that are mostly indicative of = un synched clocks**. >>>=20 >>> **) This is whyI would prefer to use NTP servers as reflectors with = NTP requests, my expectation is all of these should be reasonably synced = by default so that offsets should be in the sane range.... >>>=20 >>>=20 >>>>=20 >>>> Bob >>>>> For about 2 years now the cake w-adaptive bandwidth project has=20 >>>>> been exploring techniques to lightweightedly sense bandwidth and=20= >>>>> buffering problems. One of my favorites was their discovery that=20= >>>>> ICMP type 13 got them working OWD from millions of ipv4 devices! >>>>> They've also explored leveraging ntp and multiple other methods,=20= >>>>> and have scripts available that do a good job of compensating for=20= >>>>> 5g and starlink's misbehaviors. >>>>> They've also pioneered a whole bunch of new graphing techniques,=20= >>>>> which I do wish were used more than single number summaries=20 >>>>> especially in analyzing the behaviors of new metrics like rpm,=20 >>>>> samknows, ookla, and >>>>> RFC9097 - to see what is being missed. >>>>> There are thousands of posts about this research topic, a new post=20= >>>>> on OWD just went by here. >>>>> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793 >>>>> and of course, I love flent's enormous graphing toolset for=20 >>>>> simulating and analyzing complex network behaviors. >>>> _______________________________________________ >>>> Rpm mailing list >>>> Rpm@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/rpm >>>=20 >>> _______________________________________________ >>> ippm mailing list >>> ippm@ietf.org >>> https://www.ietf.org/mailman/listinfo/ippm