From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 29ACA3B29E for ; Tue, 24 Jul 2018 17:39:42 -0400 (EDT) Received: by mail-lf1-x132.google.com with SMTP id y200-v6so3998343lfd.7 for ; Tue, 24 Jul 2018 14:39:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=NSgt2cLnt+nN3lXG0mMYBV9r6GRkDazPTA9hwe61Coo=; b=U7tExT7E4u75wPbZGFNDfRYiXyHUz/4KWsdoSP95hP+MjF06GTQIwm8NJ+mS2Aovck eyTv+3z8G5zrxNlm9lXaZwHeqWolHrXUrsTxQIFKfmF/ad3AO+yPHUoBhW01+c1wiDxN 4E9dCkZFgcn7tlxf6mE/wGu8zZJfDcL3S1rD+AQvjolki14389XJ2NtFLMhTfF9I7R/O 7mv4po3OLDQTofxuS1mq1OKY6oB5hkGB/OSaJkUBjHhB5GTwxFZhjWUZUp4jwVoiPGMK iznPLQ7RGpD/1nFahg2+OjkOzDhzN0EtDA3IpeK7z4qNFsvFOWeimKRZTVKMlIWvXB6w 2DTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NSgt2cLnt+nN3lXG0mMYBV9r6GRkDazPTA9hwe61Coo=; b=iad65V6UyoM7REPjABqeBDpfp+cjlUeBVsGwYgOFpdvZfcU6fYpKKrEJNrtNtAHK2A gpNH5Lry+AQuiw+7MXgjCwLW+tlDcSQyT94po0cBd2Fa/T55qoOUK/mmB2bIhclA5Jd4 noTFKkJB9oCF+Z+JXosJ1LqjoOtQ1AO70zmSlepvg5lv9Ma/8nIezgGF5oSgI8aos3FC omolJMzVR+0+NHiZSnh9nOOu+9AqNHER1lqtDbv2ndIIorjvaYf82Pndilf+rG4cRTdz fAR7a7ieRt1Durzzd3AbqJBtLwNu4IhJJpYKcSfFoEcbLSL9cEBRuB7QNsH+GYxJ4Kn0 wGqA== X-Gm-Message-State: AOUpUlE8KUw+GPw4edUJFVZanZBfa0zUt1/t+PdytTTSR/DlFAjMgrZm BqKqyGzE97w6N5zChZrlTv/2nTImDJBx4MF0Byc= X-Google-Smtp-Source: AAOMgpd1ynD7Jl/2CK3+zRR9H5usOW7upbJWh5R795IGdEReDRjYBw7KEvin1rJfwFaNfozqN0+evpmlhZQoEIMYoZo= X-Received: by 2002:a19:ec04:: with SMTP id b4-v6mr10852174lfa.91.1532468380741; Tue, 24 Jul 2018 14:39:40 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Benjamin Cronce Date: Tue, 24 Jul 2018 16:39:29 -0500 Message-ID: To: Dave Taht Cc: bloat Content-Type: multipart/alternative; boundary="00000000000076b4ac0571c599cf" Subject: Re: [Bloat] No backpressure "shaper"+AQM X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2018 21:39:42 -0000 --00000000000076b4ac0571c599cf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jul 24, 2018 at 3:57 PM Dave Taht wrote: > On Tue, Jul 24, 2018 at 1:11 PM Benjamin Cronce wrote= : > > > > Maybe the Bobbie idea already would do this, but I did not see it > explicitly mentioned on its wiki. > > you are basically correct below. bobbie's core idea was a kinder, > gentler, deficit mode policer, with something fq_codel-like aimed at > reducing accumulated bytes to below the set rate. > > this is way different from conventional policers > > > The below is all about shaping ingress, not egress. > > > > My issue is that my ISP already does a good job with their AQM but > nothing is perfect and their implementation of rate limiting has a kind o= f > burst at the start. According to speedtests, my 250Mb connection starts o= ff > around 500Mb/s and slowly drops over the next few seconds until a near > perfect 250Mb/s steady state. > > ideally their htb shaper would use fq_codel as the underlying qdisc. > Or at least reduce their burst size to something saner. I hope it's > not a policer. > > You can usually "see" a policer in action. Long strings of packets are > dropped in a row. > I feel as if this new configuration is not quite a policer as it feels much less abrupt as the old configuration. It used to have massive loss spikes that wrecked havoc on other flows and make the fat TCP flows have a kind of rebound. Their newer setup seems to be gentler. While there is an increased rate of loss as it attempt to "slowly" settle at the provisioned rate, it's not like the cliff it used to be, it actually has a slope. > > > The burst at the beginning adds a certain amount of destabilization to > the TCP flows since the window quickly grows to 500Mb and then has to > backdown by dropping. If I add my own traffic shaping and AQM, I can redu= ce > the reported TCP retransmissions from ~3% to ~0.1%. > > sure. > > > > The problem that I'm getting is by adding my own shaping, a measurable > amount of the benefit of their AQM is lost. While I am limited to Codel, > HFSC+Codel, or FairQ+Codel for now, I am actually doing a worse job of > anti-bufferbloat than my ISP is. Fewer latency spices according to > DSLReports. > > ? measured how. > Just looking visual at the DSLReport graphs, I more normally see maybe a few 40ms-150ms ping spikes, while my own attempts to shape can get me several 300ms spikes. I would really need a lot more samples and actually run the numbers on them, but just causally looking at them, I get the sense that mine is worse. > > > > > This also does not touch on that the act of adding back-pressure in its > nature increases latency. I cannot say if it's a fundamental requirement = in > order to better my current situation, but I am curious if there's a bette= r > way. A thought that came to me is that like Bobbie, do a light touch as t= he > packets have already made their way and you don't want to aggressively dr= op > packets, but at the same time, I want the packets that already made the > journey to mostly unhindered enter my network. > > > > That's when I thought of a backpressure-less AQM. > > I like the restating of what policers do.... > I think I need to look at the definition of a policer. I always through them as a strict cut-off. I'm not talking about mass dropping packets beyond a rate, just doing something like Codel where a packet here and there get dropped at an increasing rate until the observed rate normalizes. > > >Instead of having backpressure and measuring sojourn time as a function > of how long it takes packets to get scheduled, predict an estimated sojou= rn > time based on the observed rate of flow, but allow packets to immediately > vacate the queue. The AQM would either mark ECN or drop the packet, but > never delay the packet. > > aqms don't delay packets. shapers do. > My described "AQM" is not a shaper in that it does not schedule packets(possibly FIFO and at line rate), but does understand bandwidth. It neither delays packets nor has a strict cut-off. It essentially would allow packets to flow through at line rate, but if the "average" rate gets too high, it may decide to drop/mark the next packet. It might be described as bufferless shaping where the goal is to minimize packet-loss. Shaping purely by a gentle rate of increasing loss. Of course this whole thought may be total rubbish, but I figured I'd throw it out there. > > > In summary, my ISP seems to have better latency with their AQM, but due > to their shaper, loss during the burst is much higher than desirable. > > > > Maybe this will be mostly moot once I get fq_codel going on pfSense, bu= t > I do find it an interesting issue. > > I thought it's been in there for a while? > Technically, but not practically. It should be easily available via the UI with 2.4.4 which is slowly nearing release. > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > > -- > > Dave T=C3=A4ht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 > --00000000000076b4ac0571c599cf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue= , Jul 24, 2018 at 3:57 PM Dave Taht <dave.taht@gmail.com> wrote:
On Tue, Jul 24, 2018 at 1:11 PM Benjamin Cronce <bcronce@gmail.com> wrote:
>
> Maybe the Bobbie idea already would do this, but I did not see it expl= icitly mentioned on its wiki.

you are basically correct below. bobbie's core idea was a kinder,
gentler, deficit mode policer, with something fq_codel-like aimed at
reducing accumulated bytes to below the set rate.

this is way different from conventional policers

> The below is all about shaping ingress, not egress.
>
> My issue is that my ISP already does a good job with their AQM but not= hing is perfect and their implementation of rate limiting has a kind of bur= st at the start. According to speedtests, my 250Mb connection starts off ar= ound 500Mb/s and slowly drops over the next few seconds until a near perfec= t 250Mb/s steady state.

ideally their htb shaper would use fq_codel as the underlying qdisc.
Or at least reduce their burst size to something saner. I hope it's
not a policer.

You can usually "see" a policer in action. Long strings of packet= s are
dropped in a row.

I feel as if this new= configuration is not quite a policer as it feels much less abrupt as the o= ld configuration. It used to have massive loss spikes that wrecked havoc on= other flows and make the fat TCP flows have a kind of rebound. Their newer= setup seems to be gentler. While there is an increased rate of loss as it = attempt to "slowly" settle at the provisioned rate, it's not = like the cliff it used to be, it actually has a slope.
=C2=A0

> The burst at the beginning adds a certain amount of destabilization to= the TCP flows since the window quickly grows to 500Mb and then has to back= down by dropping. If I add my own traffic shaping and AQM, I can reduce the= reported TCP retransmissions from ~3% to ~0.1%.

sure.
=C2=A0

>
> The problem that I'm getting is by adding my own shaping, a measur= able amount of the benefit of their AQM is lost. While I am limited to Code= l, HFSC+Codel, or FairQ+Codel for now, I am actually doing a worse job of a= nti-bufferbloat than my ISP is. Fewer latency spices according to DSLReport= s.

? measured how.
Just looking visual at the DSLReport g= raphs, I more normally see maybe a few 40ms-150ms ping spikes, while my own= attempts to shape can get me several 300ms spikes. I would really need a l= ot more samples and actually run the numbers on them, but just causally loo= king at them, I get the sense that mine is worse.

>
> This also does not touch on that the act of adding back-pressure in it= s nature increases latency. I cannot say if it's a fundamental requirem= ent in order to better my current situation, but I am curious if there'= s a better way. A thought that came to me is that like Bobbie, do a light t= ouch as the packets have already made their way and you don't want to a= ggressively drop packets, but at the same time, I want the packets that alr= eady made the journey to mostly unhindered enter my network.
>
> That's when I thought of a backpressure-less AQM.

I like the restating of what policers do....
I think I= need to look at the definition of a policer. I always through them as a st= rict cut-off. I'm not talking about mass dropping packets beyond a rate= , just doing something like Codel where a packet here and there get dropped= at an increasing rate until the observed rate normalizes.

>Instead of having backpressure and measuring sojourn time as a function= of how long it takes packets to get scheduled, predict an estimated sojour= n time based on the observed rate of flow, but allow packets to immediately= vacate the queue. The AQM would either mark ECN or drop the packet, but ne= ver delay the packet.

aqms don't delay packets. shapers do.
My described= "AQM" is not a shaper in that it does not schedule packets(possi= bly FIFO and at line rate), but does understand bandwidth. It neither delay= s packets nor has a strict cut-off. It essentially would allow packets to f= low through at line rate, but if the "average" rate gets too high= , it may decide to drop/mark the next packet. It might be described as buff= erless shaping where the goal is to minimize packet-loss. Shaping purely by= a gentle rate of increasing loss.

Of course this = whole thought may be total rubbish, but I figured I'd throw it out ther= e.

> In summary, my ISP seems to have better latency with their AQM, but du= e to their shaper, loss during the burst is much higher than desirable.
>
> Maybe this will be mostly moot once I get fq_codel going on pfSense, b= ut I do find it an interesting issue.

I thought it's been in there for a while?
Technica= lly, but not practically. It should be easily available via the UI with 2.4= .4 which is slowly nearing release.

> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


--

Dave T=C3=A4ht
CEO, TekLibre, LLC
ht= tp://www.teklibre.com
Tel: 1-669-226-2619
--00000000000076b4ac0571c599cf--