[Bloat] number of home routers with ingress AQM

[Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

Dear all,

I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?

I know in openwrt with sqm that is the default, but I have no idea about the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active?


Best Regards
	Sebastian

Re: [Bloat] number of home routers with ingress AQM

[Mikael Abrahamsson]

On Tue, 2 Apr 2019, Sebastian Moeller wrote:

> I just wondered if anybody has any reasonable estimate how many 
> end-users actually employ fair-queueing AQMs with active ECN-marking for 
> ingress traffic @home? I am trying to understand whether L4S approach to 
> simply declare these as insignificant in number is justifiable?

If more than 0.01% of HGWs did this I'd be extremely surprised.

> I know in openwrt with sqm that is the default, but I have no idea about

To configure ingress shaping you actually have to know the speed and 
configure it. It's not the default. Also, it's useless if the transport 
network queues the packets at lower rate than at what you receive it. When 
I used my DOCSIS connection it routinely forwarded packets at lower rates 
than what I bought (and had configured the ingress shaper for).

> the number of devices that actually use sqm in the field; @Jonathan: 
> does evenroute have numbers you are willing to share, like total numbers 
> or % of iqrouters with ecn-marking ingress routing active?

ISP networks typically looks like this in the ISP->HGW direction:

BNG->L2->L2->HGW

This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So 
shaping is done egress on BNG and it tries to send at lower rate than any 
of the L2 devices. Generally there is no ingress shaping of any kind on 
the HGW, it doesn't even know what speed the subscription is.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

HI Mikael,


> On Apr 2, 2019, at 14:10, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Tue, 2 Apr 2019, Sebastian Moeller wrote:
> 
>> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
> 
> If more than 0.01% of HGWs did this I'd be extremely surprised.

	Me too, but still I want to see numbers, plus those are likely those end-users that could be won-over by L4S as theses obvipusly would be receptive for the "low-latency" for all pitch of the L4S project. 

> 
>> I know in openwrt with sqm that is the default, but I have no idea about
> 
> To configure ingress shaping you actually have to know the speed and configure it.

	As a first approximation knowing your speed just requires to run a single (decent) speedtest, as the reported TCP/IPv[4|6]/HTTP[s]-Goodput numbers can be feed directly into any shaper (shapers typically desire gross rates, but since net < gross net rates will just do fine if the aim is to keep latency under control). Deutsche Telekom actually sends crude estimates of the achievable goodput via PPPoE-ACK messages that AVM's FritzBox routers (quite popular in Germany for those unhappy with the ISP supplied gear) evaluate to configure up- and downstream shaping. Which due to lack of proper documentation of the PPPoE-ACK message is a bit of a hit and miss job.

> It's not the default. Also, it's useless if the transport network queues the packets at lower rate than at what you receive it.

	Sure, that is not a solution for congestion outside of the access link, but that is an impossible problem to solve. But getting the access link debloated often is an important first step even if it does not fix the whole internet ;)

> When I used my DOCSIS connection it routinely forwarded packets at lower rates than what I bought (and had configured the ingress shaper for).

	Gotta love oversubscribed segments ;) Gargoyle firmware has an adaptive method that tries to tackl;e that poroblem, and eventouter, IIUC, actually runs multiple bandwidth measures per day to track the actually achievable bandwidth. But these are just work-arounds. Now if your cable ISP acctually had used a competent QoS system you might not have had to suffer bad latency with bad bandwidth. But I believe cablelabs has identified that as a problem and is actively working on it (and sooner or later their flow-queueing in disguise will do the right thing ;) ). Flow-queueing in disguise, I hear nobody ask, well sure they mandate a DualPI2 solution but also mandate in Annex P Queue Protection Algorithm (Normative) of CM-SP-MULPIv3.1-I17-190121 something that must come close to flow queuueing in cost, to cite from the source "The algorithm maintains per-Microflow state that holds a “queuing score” representing how much each Microflow was responsible for recent queuing." I asked about the cost but have not heard an answer yet.

> 
>> the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active?
> 
> ISP networks typically looks like this in the ISP->HGW direction:
> 
> BNG->L2->L2->HGW
> 
> This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So shaping is done egress on BNG and it tries to send at lower rate than any of the L2 devices.

	Yes, I know, and there are good reasons for doing it that way.

> Generally there is no ingress shaping of any kind on the HGW,

	That is part of my question....

> it doesn't even know what speed the subscription is.

	See above how Deutsche Telekom deals with that issue, at least in the German market.

Thanks for your insights, much appreciated

Best Regards
	Sebastian



> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Mikael Abrahamsson]

On Tue, 2 Apr 2019, Sebastian Moeller wrote:

> 	See above how Deutsche Telekom deals with that issue, at least in 
> the German market.

I've read rumours about some ISPs implementing interaction with the VDSL 
DSLAM where there is an estimation of the current link-speed for each 
individual customer and then it tries to set the BNG egress shaper 
appropriately.

I am very happy about my FTTH solution I have now since from what I can 
see the L2 network is almost never a limiting factor (much better than my 
DOCSIS connection) so my bidirectional SQM with CAKE seems to work very 
well.

Still, the HGW can never solve these problems properly, the egress shaping 
in the BNG needs to do a proper job. From what I have been told, there has 
been improvements here in the past few years.

What I am more worried about is the egress shaping from the HGW. I talked 
to several vendors at BBF and they talked about ingress policing being 
commonly used on the BNG. This means no ingress shaping at all (just 
packet drops if they exceed the configured rate). I don't know about 
buffering on the HGW though and how the policed rate is set compared to 
the L2 rate the HGW can send via.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

Hi Mikael,


> On Apr 2, 2019, at 15:04, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Tue, 2 Apr 2019, Sebastian Moeller wrote:
> 
>> 	See above how Deutsche Telekom deals with that issue, at least in the German market.
> 
> I've read rumours about some ISPs implementing interaction with the VDSL DSLAM where there is an estimation of the current link-speed for each individual customer and then it tries to set the BNG egress shaper appropriately.

	I believe this is what Deutsche Telekom actually does already.

> 
> I am very happy about my FTTH solution I have now since from what I can see the L2 network is almost never a limiting factor (much better than my DOCSIS connection) so my bidirectional SQM with CAKE seems to work very well.

	I envy you ;) that said, I have little issue with my VDSL2 link, but at 50/10 it is hardly pushing the limit. DOCSIS with its often huge segments is a mixed bag, I have head od segments with 1000 customers and no issue even saturating a DOCSIS 3.1 1000/50 plan, and also people with severe issues with more modes 100/20 plans.

> 
> Still, the HGW can never solve these problems properly,

	Assuming fixed bandwidth, it can do a pretty good approximation of properly though ;)

> the egress shaping in the BNG needs to do a proper job.

	I agree, but then my wishlist includes flow-queueing and then reality intervenes, and I keep having to do this on my side as BNGs do not offer fq for customer bandwidth shaping/policing, and might never do.


> From what I have been told, there has been improvements here in the past few years.

	I agree, when I started with Deutsche Telekom latency under saturating load spikes (in ICMP probes) were above 1000ms in 2015, but now with the Juniper BNG shaper solution I saw around 300ms (I switched ISP, but the cap is still around 300ms). IMHO this is much better, but also far away from good enough, so I keep shaping on my end to keep latency acceptable (with a 50/10 link saturating loads are common enough to justify the time spent for configuring the home ingress shaper IMHO).

> 
> What I am more worried about is the egress shaping from the HGW.

	If you ask me that is going to come, all shared media links (docsis, G-PON, ...) already need this to keep customer modems from shouting over each other.

> I talked to several vendors at BBF and they talked about ingress policing being commonly used on the BNG. This means no ingress shaping at all (just packet drops if they exceed the configured rate).

	I wonder how that rhymes with the 300-1000ms added latency I see under load, assuming the BNG-limiter to be the bottleneck.

> I don't know about buffering on the HGW though and how the policed rate is set compared to the L2 rate the HGW can send via.

	Typical DSL modem-routers and DOCSIS modems (presumably < 3.1) often show pretty manly buffering (that is to say probably a bit too much with too little brains attached ;) ). I note that the L4S-project already identified CPEs as the next target to get upstream queueing tackled, I see no chance for doing that effectively without getting the ISPs on board. I have no idea how well the heavy cable labs involvement will sit with the old telco incumbents and whether any non-ITU standard will fly. But let's see...

Best Regards
	Sebastian

> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Ryan Mounce]

On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> I've read rumours about some ISPs implementing interaction with the VDSL
> DSLAM where there is an estimation of the current link-speed for each
> individual customer and then it tries to set the BNG egress shaper
> appropriately.

NBN here in Australia do this for their wholesale FTTN/B VDSL2
product, injecting the downstream and upstream sync speed into
PPPoE/DHCP requests. This still depends on the retail service
providers to make use of these attributes from their BNG to configure
the shaper.

>
> I am very happy about my FTTH solution I have now since from what I can
> see the L2 network is almost never a limiting factor (much better than my
> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very
> well.
>
> Still, the HGW can never solve these problems properly, the egress shaping
> in the BNG needs to do a proper job. From what I have been told, there has
> been improvements here in the past few years.
>
> What I am more worried about is the egress shaping from the HGW. I talked
> to several vendors at BBF and they talked about ingress policing being
> commonly used on the BNG. This means no ingress shaping at all (just
> packet drops if they exceed the configured rate). I don't know about
> buffering on the HGW though and how the policed rate is set compared to
> the L2 rate the HGW can send via.

NBN is an example again. Their documented behaviour is to police
traffic in both directions. Most ISPs then shape in the downstream -
and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end
user to shape traffic in the upstream. Many - probably even most users
have no shaping whatsoever in the upstream, only a policer.

And then there is their new FTTdp product, where it is not currently
possible to determine the real VDSL2 sync speed. If there's a drop of
rain it will resync at a lower speed in the upstream, and then
everything ends up queuing inside the supplied modem...

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Foulkes]

Hi Ryan,

See below

> On Apr 2, 2019, at 9:33 AM, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> I've read rumours about some ISPs implementing interaction with the VDSL
>> DSLAM where there is an estimation of the current link-speed for each
>> individual customer and then it tries to set the BNG egress shaper
>> appropriately.
> 
> NBN here in Australia do this for their wholesale FTTN/B VDSL2
> product, injecting the downstream and upstream sync speed into
> PPPoE/DHCP requests. This still depends on the retail service
> providers to make use of these attributes from their BNG to configure
> the shaper.

Very interesting, and extremely useful that they surface the sync rate in the PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at this time?

FYI- The IQrouter supports the notion of a tightly coupled modem from which current sync rates (as well as other SNR / Atten stats) can be dynamically queried, which in turn are used in the AQM settings computations. Works wonders on flaky DSL lines that vary sync rate with time of day and weather. If that VDSL2 sync rate relayed is exposed in the system somehow, it would make dynamic adjustments possible.


> 
>> 
>> I am very happy about my FTTH solution I have now since from what I can
>> see the L2 network is almost never a limiting factor (much better than my
>> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very
>> well.
>> 
>> Still, the HGW can never solve these problems properly, the egress shaping
>> in the BNG needs to do a proper job. From what I have been told, there has
>> been improvements here in the past few years.
>> 
>> What I am more worried about is the egress shaping from the HGW. I talked
>> to several vendors at BBF and they talked about ingress policing being
>> commonly used on the BNG. This means no ingress shaping at all (just
>> packet drops if they exceed the configured rate). I don't know about
>> buffering on the HGW though and how the policed rate is set compared to
>> the L2 rate the HGW can send via.
> 
> NBN is an example again. Their documented behaviour is to police
> traffic in both directions. Most ISPs then shape in the downstream -
> and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end
> user to shape traffic in the upstream. Many - probably even most users
> have no shaping whatsoever in the upstream, only a policer.
> 
> And then there is their new FTTdp product, where it is not currently
> possible to determine the real VDSL2 sync speed. If there's a drop of
> rain it will resync at a lower speed in the upstream, and then
> everything ends up queuing inside the supplied modem…

Oh, so they regressed from their other offering, not exposing sync rates?

BTW- this hole notion of the BGM relaying the provisioned or current sync rates should be a mandated requirement, as it has huge benefits for AQMs. Not that it can be relied on 100%, but at least it makes a good starting point.


> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] number of home routers with ingress AQM

[Ryan Mounce]

On Wed, 3 Apr 2019 at 00:41, Jonathan Foulkes <jfoulkes@evenroute.com> wrote:
>
> Hi Ryan,
>
> See below
>
> > On Apr 2, 2019, at 9:33 AM, Ryan Mounce <ryan@mounce.com.au> wrote:
> >
> > On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >>
> >> I've read rumours about some ISPs implementing interaction with the VDSL
> >> DSLAM where there is an estimation of the current link-speed for each
> >> individual customer and then it tries to set the BNG egress shaper
> >> appropriately.
> >
> > NBN here in Australia do this for their wholesale FTTN/B VDSL2
> > product, injecting the downstream and upstream sync speed into
> > PPPoE/DHCP requests. This still depends on the retail service
> > providers to make use of these attributes from their BNG to configure
> > the shaper.
>
> Very interesting, and extremely useful that they surface the sync rate in the PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at this time?

These attributes are tacked onto the requests towards the ISP - so
unfortunately they're not of any use to configure a shaper on the home
gateway side.

> FYI- The IQrouter supports the notion of a tightly coupled modem from which current sync rates (as well as other SNR / Atten stats) can be dynamically queried, which in turn are used in the AQM settings computations. Works wonders on flaky DSL lines that vary sync rate with time of day and weather. If that VDSL2 sync rate relayed is exposed in the system somehow, it would make dynamic adjustments possible.

Yeah, I did something similar for a family member stuck on ADSL.
Scraped the sync speed from the cheap bridged TP-Link modem and used
it to configure cake - with periodic updates in case of a resync.

> > NBN is an example again. Their documented behaviour is to police
> > traffic in both directions. Most ISPs then shape in the downstream -
> > and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end
> > user to shape traffic in the upstream. Many - probably even most users
> > have no shaping whatsoever in the upstream, only a policer.
> >
> > And then there is their new FTTdp product, where it is not currently
> > possible to determine the real VDSL2 sync speed. If there's a drop of
> > rain it will resync at a lower speed in the upstream, and then
> > everything ends up queuing inside the supplied modem…
>
> Oh, so they regressed from their other offering, not exposing sync rates?

Yep. The wholesaler provides the modem (also integrating a reverse
power supply to the mini DSLAM / "DPU") so the customer demarcation
point is a gigabit ethernet port - abstracting the xDSL side of things
from the customer's equipment. Apparently they're working on making
real time sync speeds available ISPs that want to query them as part
of service diagnostics, and haven't even got that contrived solution
working yet. And then you still need to get that info from your ISP,
fortunately mine has just added the unique ability to trigger your own
service diagnostics and retrieve the results so it may be possible to
scrape it this way. Quite the kludge at that point.

Fortunately the mini DSLAM is typically less than 50m from your
property boundary and spliced directly into the lead-in cable, so it
is normally capable of syncing above the maximum 100/40 rate that's
offered and thus limited by the consistent policer rather than the
variable line rate. Unless it rains - then all bets are off. Isn't
copper great?

>
> BTW- this hole notion of the BGM relaying the provisioned or current sync rates should be a mandated requirement, as it has huge benefits for AQMs. Not that it can be relied on 100%, but at least it makes a good starting point.

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Foulkes]

Hi Ryan,

See below

> On Apr 2, 2019, at 9:33 AM, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> I've read rumours about some ISPs implementing interaction with the VDSL
>> DSLAM where there is an estimation of the current link-speed for each
>> individual customer and then it tries to set the BNG egress shaper
>> appropriately.
> 
> NBN here in Australia do this for their wholesale FTTN/B VDSL2
> product, injecting the downstream and upstream sync speed into
> PPPoE/DHCP requests. This still depends on the retail service
> providers to make use of these attributes from their BNG to configure
> the shaper.

Very interesting, and extremely useful that they surface the sync rate in the PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at this time?

FYI- The IQrouter supports the notion of a tightly coupled modem from which current sync rates (as well as other SNR / Atten stats) can be dynamically queried, which in turn are used in the AQM settings computations. Works wonders on flaky DSL lines that vary sync rate with time of day and weather. If that VDSL2 sync rate relayed is exposed in the system somehow, it would make dynamic adjustments possible.


> 
>> 
>> I am very happy about my FTTH solution I have now since from what I can
>> see the L2 network is almost never a limiting factor (much better than my
>> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very
>> well.
>> 
>> Still, the HGW can never solve these problems properly, the egress shaping
>> in the BNG needs to do a proper job. From what I have been told, there has
>> been improvements here in the past few years.
>> 
>> What I am more worried about is the egress shaping from the HGW. I talked
>> to several vendors at BBF and they talked about ingress policing being
>> commonly used on the BNG. This means no ingress shaping at all (just
>> packet drops if they exceed the configured rate). I don't know about
>> buffering on the HGW though and how the policed rate is set compared to
>> the L2 rate the HGW can send via.
> 
> NBN is an example again. Their documented behaviour is to police
> traffic in both directions. Most ISPs then shape in the downstream -
> and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end
> user to shape traffic in the upstream. Many - probably even most users
> have no shaping whatsoever in the upstream, only a policer.
> 
> And then there is their new FTTdp product, where it is not currently
> possible to determine the real VDSL2 sync speed. If there's a drop of
> rain it will resync at a lower speed in the upstream, and then
> everything ends up queuing inside the supplied modem…

Oh, so they regressed from their other offering, not exposing sync rates?

BTW- this hole notion of the BGM relaying the provisioned or current sync rates should be a mandated requirement, as it has huge benefits for AQMs. Not that it can be relied on 100%, but at least it makes a good starting point.


> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

> On Apr 2, 2019, at 16:14, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
> Hi Ryan,
> 
> See below
> 
>> On Apr 2, 2019, at 9:33 AM, Ryan Mounce <ryan@mounce.com.au> wrote:
>> 
>> On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>> 
>>> I've read rumours about some ISPs implementing interaction with the VDSL
>>> DSLAM where there is an estimation of the current link-speed for each
>>> individual customer and then it tries to set the BNG egress shaper
>>> appropriately.
>> 
>> NBN here in Australia do this for their wholesale FTTN/B VDSL2
>> product, injecting the downstream and upstream sync speed into
>> PPPoE/DHCP requests. This still depends on the retail service
>> providers to make use of these attributes from their BNG to configure
>> the shaper.
> 
> Very interesting, and extremely useful that they surface the sync rate in the PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at this time?
> 
> FYI- The IQrouter supports the notion of a tightly coupled modem from which current sync rates (as well as other SNR / Atten stats) can be dynamically queried, which in turn are used in the AQM settings computations. Works wonders on flaky DSL lines that vary sync rate with time of day and weather. If that VDSL2 sync rate relayed is exposed in the system somehow, it would make dynamic adjustments possible.

	IMHO the ISP should be mandated make  machine readable dynamically updated information about a links limits available. Grossrate, unavoidable overhead and potentially additional encodings like (ATM's 48/53, or PTM's 64/65). With BNG-shapers/policers the synch-rate alone does not help except for allowing an estimate of the upper bound. 

Best Regards
	Sebastian

> 
> 
>> 
>>> 
>>> I am very happy about my FTTH solution I have now since from what I can
>>> see the L2 network is almost never a limiting factor (much better than my
>>> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very
>>> well.
>>> 
>>> Still, the HGW can never solve these problems properly, the egress shaping
>>> in the BNG needs to do a proper job. From what I have been told, there has
>>> been improvements here in the past few years.
>>> 
>>> What I am more worried about is the egress shaping from the HGW. I talked
>>> to several vendors at BBF and they talked about ingress policing being
>>> commonly used on the BNG. This means no ingress shaping at all (just
>>> packet drops if they exceed the configured rate). I don't know about
>>> buffering on the HGW though and how the policed rate is set compared to
>>> the L2 rate the HGW can send via.
>> 
>> NBN is an example again. Their documented behaviour is to police
>> traffic in both directions. Most ISPs then shape in the downstream -
>> and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end
>> user to shape traffic in the upstream. Many - probably even most users
>> have no shaping whatsoever in the upstream, only a policer.
>> 
>> And then there is their new FTTdp product, where it is not currently
>> possible to determine the real VDSL2 sync speed. If there's a drop of
>> rain it will resync at a lower speed in the upstream, and then
>> everything ends up queuing inside the supplied modem…
> 
> Oh, so they regressed from their other offering, not exposing sync rates?
> 
> BTW- this hole notion of the BGM relaying the provisioned or current sync rates should be a mandated requirement, as it has huge benefits for AQMs. Not that it can be relied on 100%, but at least it makes a good starting point.
> 
> 
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Foulkes]

Responses below

> On Apr 2, 2019, at 8:10 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Tue, 2 Apr 2019, Sebastian Moeller wrote:
> 
>> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
> 
> If more than 0.01% of HGWs did this I'd be extremely surprised.

My observation is that the number is very small, even devices with SQM services, rarely see them enabled, and when they are, are set to sub-optimal values. 
I see Sebastian doing a valiant, even heroic effort at addressing technical users questions on forums, but even those users seem confused at times.

> 
>> I know in openwrt with sqm that is the default, but I have no idea about
> 
> To configure ingress shaping you actually have to know the speed and configure it. It's not the default. Also, it's useless if the transport network queues the packets at lower rate than at what you receive it. When I used my DOCSIS connection it routinely forwarded packets at lower rates than what I bought (and had configured the ingress shaper for).

As noted in other responses, the actual throughput needs to be measured and then monitored to ensure the ingress shaping is aligned with current capacity of the link. And not just the HGW to BNG, but just as importantly, account for any constraints in backhaul from the BNG.

> 
>> the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active?

@Sebastian, 100% of IQrouters running firmware 3.x (which uses Cake as the default AQM) respect/use ECN. This has been shipping since September, 2018. All existing v2 IQrouters (first ship January 2017) may upgrade to 3x (user initiated, but one-click).
As for split, 70% of deployed IQrouters are doing ECN today. As for count, well, that’s private. But the good new is we have ISP customers rolling them out at a good clip. 
Turns out that having a sane traffic manager at the HGW on every node of a DSLAM is very good for the DSLAM, the backhaul and the actual users, who quit screaming at the ISP ;-)

> 
> ISP networks typically looks like this in the ISP->HGW direction:
> 
> BNG->L2->L2->HGW
> 
> This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So shaping is done egress on BNG and it tries to send at lower rate than any of the L2 devices. Generally there is no ingress shaping of any kind on the HGW, it doesn't even know what speed the subscription is.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Foulkes]

Responses below

> On Apr 2, 2019, at 8:10 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
> On Tue, 2 Apr 2019, Sebastian Moeller wrote:
> 
>> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
> 
> If more than 0.01% of HGWs did this I'd be extremely surprised.

My observation is that the number is very small, even devices with SQM services, rarely see them enabled, and when they are, are set to sub-optimal values. 
I see Sebastian doing a valiant, even heroic effort at addressing technical users questions on forums, but even those users seem confused at times.

> 
>> I know in openwrt with sqm that is the default, but I have no idea about
> 
> To configure ingress shaping you actually have to know the speed and configure it. It's not the default. Also, it's useless if the transport network queues the packets at lower rate than at what you receive it. When I used my DOCSIS connection it routinely forwarded packets at lower rates than what I bought (and had configured the ingress shaper for).

As noted in other responses, the actual throughput needs to be measured and then monitored to ensure the ingress shaping is aligned with current capacity of the link. And not just the HGW to BNG, but just as importantly, account for any constraints in backhaul from the BNG.

> 
>> the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active?

@Sebastian, 100% of IQrouters running firmware 3.x (which uses Cake as the default AQM) respect/use ECN. This has been shipping since September, 2018. All existing v2 IQrouters (first ship January 2017) may upgrade to 3x (user initiated, but one-click).
As for split, 70% of deployed IQrouters are doing ECN today. As for count, well, that’s private. But the good new is we have ISP customers rolling them out at a good clip. 
Turns out that having a sane traffic manager at the HGW on every node of a DSLAM is very good for the DSLAM, the backhaul and the actual users, who quit screaming at the ISP ;-)

> 
> ISP networks typically looks like this in the ISP->HGW direction:
> 
> BNG->L2->L2->HGW
> 
> This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So shaping is done egress on BNG and it tries to send at lower rate than any of the L2 devices. Generally there is no ingress shaping of any kind on the HGW, it doesn't even know what speed the subscription is.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Morton]

> On 2 Apr, 2019, at 5:14 pm, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
> 
> But the good new is we have ISP customers rolling them out at a good clip.

> Turns out that having a sane traffic manager at the HGW on every node of a DSLAM is very good for the DSLAM, the backhaul and the actual users, who quit screaming at the ISP ;-)

This is very encouraging to hear, not least that there are measurable technical benefits to the ISP as well as (more obviously) to the immediate user.  How big are your marketing plans?  :-)

 - Jonathan Morton

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

Hi Jonathan,

Thanks for the data.

On April 2, 2019 4:14:34 PM GMT+02:00, Jonathan Foulkes <jf@jonathanfoulkes.com> wrote:
>Responses below
>
>> On Apr 2, 2019, at 8:10 AM, Mikael Abrahamsson <swmike@swm.pp.se>
>wrote:
>> 
>> On Tue, 2 Apr 2019, Sebastian Moeller wrote:
>> 
>>> I just wondered if anybody has any reasonable estimate how many
>end-users actually employ fair-queueing AQMs with active ECN-marking
>for ingress traffic @home? I am trying to understand whether L4S
>approach to simply declare these as insignificant in number is
>justifiable?
>> 
>> If more than 0.01% of HGWs did this I'd be extremely surprised.
>
>My observation is that the number is very small, even devices with SQM
>services, rarely see them enabled, and when they are, are set to
>sub-optimal values. 
>I see Sebastian doing a valiant, even heroic effort at addressing
>technical users questions on forums, but even those users seem confused
>at times.
>
>> 
>>> I know in openwrt with sqm that is the default, but I have no idea
>about
>> 
>> To configure ingress shaping you actually have to know the speed and
>configure it. It's not the default. Also, it's useless if the transport
>network queues the packets at lower rate than at what you receive it.
>When I used my DOCSIS connection it routinely forwarded packets at
>lower rates than what I bought (and had configured the ingress shaper
>for).
>
>As noted in other responses, the actual throughput needs to be measured
>and then monitored to ensure the ingress shaping is aligned with
>current capacity of the link. And not just the HGW to BNG, but just as
>importantly, account for any constraints in backhaul from the BNG.

Sure, but for the most part I have been limited either by the access link/BNG-shaper or limited peering/transit between my ISP and specific target servers, and in the second case I would hate to slow everything to a crawl just because my ISP and say YouTube try to Duke out who should pay for access, content to eyeballs or vice versa...

>
>> 
>>> the number of devices that actually use sqm in the field; @Jonathan:
>does evenroute have numbers you are willing to share, like total
>numbers or % of iqrouters with ecn-marking ingress routing active?
>
>@Sebastian, 100% of IQrouters running firmware 3.x (which uses Cake as
>the default AQM) respect/use ECN. This has been shipping since
>September, 2018. All existing v2 IQrouters (first ship January 2017)
>may upgrade to 3x (user initiated, but one-click).
>As for split, 70% of deployed IQrouters are doing ECN today. 

Excellent, thanks so most of them....


>As for count, well, that’s private.

I had a hunch, that would be the case ;) . Understandable, though.

>But the good new is we have ISP customers
>rolling them out at a good clip. 
>Turns out that having a sane traffic manager at the HGW on every node
>of a DSLAM is very good for the DSLAM, the backhaul and the actual
>users, who quit screaming at the ISP ;-)

Oh, nice, I fully agree upstream AQM is a case where the goals and incentives for end-users and ISPs seem well aligned.


>
>> 
>> ISP networks typically looks like this in the ISP->HGW direction:
>> 
>> BNG->L2->L2->HGW
>> 
>> This is the same regardless if it's DSL, DOCSIS, FTTH/PON or
>whatever. So shaping is done egress on BNG and it tries to send at
>lower rate than any of the L2 devices. Generally there is no ingress
>shaping of any kind on the HGW, it doesn't even know what speed the
>subscription is.
>> 
>> -- 
>> Mikael Abrahamsson    email: swmike@swm.pp.se
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Bloat] number of home routers with ingress AQM

[Ryan Mounce]

On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller <moeller0@gmx.de> wrote:
>
> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?

L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks
*without* fq. I don't think there would be any such ingress shapers
configured on home gateways. Certainly not by anyone on this list...
anyone running non-fq codel or flowblind cake for ingress shaping?

-Ryan

Re: [Bloat] number of home routers with ingress AQM

[Mikael Abrahamsson]

On Tue, 2 Apr 2019, Ryan Mounce wrote:

> L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks 
> *without* fq. I don't think there would be any such ingress shapers 
> configured on home gateways. Certainly not by anyone on this list... 
> anyone running non-fq codel or flowblind cake for ingress shaping?

What you described is probably on 95% or more of egress shaping on the BNG 
and on egress shaping on HGWs in the field.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Ryan Mounce]

On Wed, 3 Apr 2019 at 00:04, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> What you described is probably on 95% or more of egress shaping on the BNG
> and on egress shaping on HGWs in the field.

How many of these single queue deployments actually have ECN marking
enabled? This is the really dangerous case with L4S.

Re: [Bloat] number of home routers with ingress AQM

[Mikael Abrahamsson]

On Wed, 3 Apr 2019, Ryan Mounce wrote:

> On Wed, 3 Apr 2019 at 00:04, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> What you described is probably on 95% or more of egress shaping on the BNG
>> and on egress shaping on HGWs in the field.
>
> How many of these single queue deployments actually have ECN marking
> enabled? This is the really dangerous case with L4S.

I don't have numbers, but from looking at the platforms I have available 
to me, very few do ECN-marking.

Considering that Apple has been beating the ECN-enablement drum for a few 
years now, there might actually be ECN-marking deployment soon.

I'm in process of trying to figure out support for ingress/egress shaping 
and ECN marking on the most common BNG platforms, we'll see what that will 
result in.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

> On Apr 2, 2019, at 15:15, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
> 
> L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks
> *without* fq.

	I know, but I believe that they misunderstand the issues resulting from post-bottleneck shaping, like ingress shaping on the remote side of the true bottleneck. The idea seems that sending at too high a rate is unproblematic if the AQM can simply queue up these packets and delay them accordingly. But in the ingress shaper case these packets already traversed the bottleneck and aone has payed the bandwidth price to hoist them to the home, delaying or even dropping on the AQM side will not magically get the time back the packets took traversing the link.
	Why do I care, because that ingress shaping setup is what I use at home, and I have zero confidence that ISPs will come up with a solution to my latency desires that I am going to be happy with... And what I see from the L4S mixes light with a lot of shadows.


> I don't think there would be any such ingress shapers
> configured on home gateways. Certainly not by anyone on this list...
> anyone running non-fq codel or flowblind cake for ingress shaping?

	As stated above, I believe fq to not be a reliable safety valve for the ingress shaping case.

Best Regards
	Sebastian

> 
> -Ryan

Re: [Bloat] number of home routers with ingress AQM

[Ryan Mounce]

On Wed, 3 Apr 2019 at 00:05, Sebastian Moeller <moeller0@gmx.de> wrote:
>
>
>
> > On Apr 2, 2019, at 15:15, Ryan Mounce <ryan@mounce.com.au> wrote:
> >
> > On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller <moeller0@gmx.de> wrote:
> >>
> >> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
> >
> > L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks
> > *without* fq.
>
>         I know, but I believe that they misunderstand the issues resulting from post-bottleneck shaping, like ingress shaping on the remote side of the true bottleneck. The idea seems that sending at too high a rate is unproblematic if the AQM can simply queue up these packets and delay them accordingly. But in the ingress shaper case these packets already traversed the bottleneck and aone has payed the bandwidth price to hoist them to the home, delaying or even dropping on the AQM side will not magically get the time back the packets took traversing the link.

My understanding is that with an AQM performing RFC 3168 style ECN
marking, the L4S flows will build a standing queue within their flow
queue before the AQM starts ECN marking. At this point the L4S flow
will be less responsive to marks with a linear (?) decrease rather
than treating it like loss (multiplicative decrease), however the AQM
will just keep on marking to keep in in check. The fq shaper will
continue to isolate it from other flows at the bottleneck and enforce
fairness between flows (or in the case of an ingress shaper after the
true bottleneck, continue to selectively apply drops/marks that
effectively 'nudge' flows towards fairness).

If there's no fq and there is RFC 3168 style ECN marking, the L4S
flows assume that they're receiving aggressive DCTCP-style marking,
respond less aggressively to each mark, and will starve conventional
Reno friendly flows. L4S people are relying on there being almost no
such bottlenecks on the internet, and to be honest I think this is a
fair assumption. The most deployed single queue AQM may be PIE as
mandated by DOCSIS 3.1, which had ECN marking disabled before the
whole DualQ/L4S thing. Bob Briscoe thinks the main concern is people
that have found old Cisco routers with RED and re-commissioned them...

As L4S flows would still be still be getting ECN marked in either case,
there would be no dropping of packets that have already traversed the
bottleneck in order to signal congestion. So long as there is fq to
enforce fairness, I can't see any problem.

Sure, signalling congestion without loss doesn't mean that the packets
haven't already spent more than their fair share of time at the
previous bottleneck. This is a more general problem with shaping
ingress further downstream from the true bottleneck - and I don't see
that it's any worse here than with any other TCP overshooting during
slow start.

There are certainly more real threats such as Akamai's FAST TCP. It's
like BBR in that it attempts to detect and respond to congestion based
on queuing latency, and tries to ignore low levels of random packet
loss that don't occur due to congestion. Their implementation
definitely presents issues for ingress shaping:
https://forums.whirlpool.net.au/thread/2530363

I saw that thread years ago, and then eventually saw the issue for
myself after changing ISP. Suddenly Windows Update was downloading
from my ISP's embedded Akamai cache using FAST TCP, and was
effectively unresponsive to cake's ingress shaping, instead filling
the queue in my ISP's BNG.

Fact is... ingress shaping is a hack. The real problem needs to be
solved in the BNG, in the CMTS, in the DSLAM, etc. L4S is just one
proposal and of course it deserves scrutiny before gobbling up the
precious ECT(1) codepoint, however it seems to have some traction with
vendors and a chance at wide deployment with a view to address address
exactly this problem *at* the bottleneck. For that, it also deserves
consideration.

>         Why do I care, because that ingress shaping setup is what I use at home, and I have zero confidence that ISPs will come up with a solution to my latency desires that I am going to be happy with... And what I see from the L4S mixes light with a lot of shadows.
>
>
> > I don't think there would be any such ingress shapers
> > configured on home gateways. Certainly not by anyone on this list...
> > anyone running non-fq codel or flowblind cake for ingress shaping?
>
>         As stated above, I believe fq to not be a reliable safety valve for the ingress shaping case.

-Ryan

Re: [Bloat] number of home routers with ingress AQM

[Sebastian Moeller]

Hi Ryan,

thanks for your insights, more below in-line.

> On Apr 3, 2019, at 01:23, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> On Wed, 3 Apr 2019 at 00:05, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> 
>> 
>>> On Apr 2, 2019, at 15:15, Ryan Mounce <ryan@mounce.com.au> wrote:
>>> 
>>> On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>> I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable?
>>> 
>>> L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks
>>> *without* fq.
>> 
>>        I know, but I believe that they misunderstand the issues resulting from post-bottleneck shaping, like ingress shaping on the remote side of the true bottleneck. The idea seems that sending at too high a rate is unproblematic if the AQM can simply queue up these packets and delay them accordingly. But in the ingress shaper case these packets already traversed the bottleneck and aone has payed the bandwidth price to hoist them to the home, delaying or even dropping on the AQM side will not magically get the time back the packets took traversing the link.
> 
> My understanding is that with an AQM performing RFC 3168 style ECN
> marking, the L4S flows will build a standing queue within their flow
> queue before the AQM starts ECN marking.

	I agree, so far so normal.

> At this point the L4S flow
> will be less responsive to marks with a linear (?) decrease rather
> than treating it like loss (multiplicative decrease), however the AQM
> will just keep on marking to keep in in check.

	Again agreed, except an AQM expecting a quadratic response will not mark rapidly enough to slow down an L4S flow as fast as other flows.

> The fq shaper will
> continue to isolate it from other flows at the bottleneck and enforce
> fairness between flows (or in the case of an ingress shaper after the
> true bottleneck, continue to selectively apply drops/marks that
> effectively 'nudge' flows towards fairness).

	Again agreed, except the L4S flow will not respond as the AQM predicts/expects and will keep sending at a higher rate (at least the CE mark should stop the flow from increasing the tx bandwidth...) and hence will keep occupying the true bottleneck. In other words the fq-isolation will work considerably worse than the rash judgment "misunderstanding the meaning of a CE-mark does not matter for fq-AQMs" assumes.

> 
> If there's no fq and there is RFC 3168 style ECN marking, the L4S
> flows assume that they're receiving aggressive DCTCP-style marking,
> respond less aggressively to each mark, and will starve conventional
> Reno friendly flows.

	Yes, except in the above case effectively the same is going to happen, the L4S flow is still not throttling back hard enough and will hence occupy more bandwidth of the bottleneck than intended.

> L4S people are relying on there being almost no
> such bottlenecks on the internet, and to be honest I think this is a
> fair assumption.

	I disagree, on whether they a) have a robust and reliable estimate of the number of such beasts and b) I would like to see less assumption and more measurements here ;)

> The most deployed single queue AQM may be PIE as
> mandated by DOCSIS 3.1, which had ECN marking disabled before the
> whole DualQ/L4S thing. Bob Briscoe thinks the main concern is people
> that have found old Cisco routers with RED and re-commissioned them...

	Again thinks and assumes does not fill me with warm and fuzzy feelings, I want to see hard cold data.


> 
> As L4S flows would still be still be getting ECN marked in either case,
> there would be no dropping of packets that have already traversed the
> bottleneck in order to signal congestion. So long as there is fq to
> enforce fairness, I can't see any problem.

	The problem is that fairness is enforced too late here, as the under-responsive L4S flow already hogged more bottleneck bandwidth than intended.

> 
> Sure, signalling congestion without loss doesn't mean that the packets
> haven't already spent more than their fair share of time at the
> previous bottleneck.

	And that is my point.


> This is a more general problem with shaping
> ingress further downstream from the true bottleneck - and I don't see
> that it's any worse here than with any other TCP overshooting during
> slow start.

	Just because ingress shaping has challenges I see no reason to make these worse by design... The point I am making is neither against L4S in general or its quest for a different TCP-CC with better congestion signaling, my concern is about the half-backed attempt to press a single codepoint into service where actually two codepoints (or rather an independent full bit) are required. I detest the fact that the project intends to simply argue away the issues this incomplete isolation causes instead of figuring out how to solve the issue properly. IMHO that either means make it possible to share L4S and non-L4S flows _fairly_ on TCP-friendly ECN-marking AQMs, or find a way to properly identify the handiwork of an L4S-complient AQM from a TCP-friendly one. 

> 
> There are certainly more real threats such as Akamai's FAST TCP.

	I typically do not accept the argument, that one rule-violation justifies ore rule-violations ;)

> It's
> like BBR in that it attempts to detect and respond to congestion based
> on queuing latency,

	Somebody should give CC designers the Codel paper and make them understand that this approach will need to be sensitive to changes in the 5-10ms range to account for the presence of competent AQM in the path... At the current time I see no real justification for pushing this same broken idea time and time again, it never worked well, see TCP-Vegas, LEDBAT, ... 

> and tries to ignore low levels of random packet
> loss that don't occur due to congestion.

	Yes, that is just as unfortunate a misdesign, but it is encouraging that BBRv2 actually tries to look harder at this issue and come up with a "loss-model" to somehow disambiguate between different losses.

> Their implementation
> definitely presents issues for ingress shaping:
> https://forums.whirlpool.net.au/thread/2530363

	This is why I call this mis-designed, the underlaying model of the network in BBR obviously does not match reality, in my world that means the model needs changing ;)

> 
> I saw that thread years ago, and then eventually saw the issue for
> myself after changing ISP. Suddenly Windows Update was downloading
> from my ISP's embedded Akamai cache using FAST TCP, and was
> effectively unresponsive to cake's ingress shaping, instead filling
> the queue in my ISP's BNG.

	I note that FAST TCP in all likelihood is not TCP-friendly.


> 
> Fact is... ingress shaping is a hack.

	That is harsh, but I give you sub-optimal and approximate, it also works amazingly well given all theoretical reasons why it should not work.

> The real problem needs to be
> solved in the BNG, in the CMTS, in the DSLAM, etc.

	But realistically it is not going to be solved in the way end-users would like, simply due to the economic incentives not being aligned for ISPs and end-users. 
In addition the remote end might not even have enough information to perform the kind of shaping a user might desire, e.g. I use cake's isolation options to have approximate per-internal-IP-fairness, but for IPv4 that requires having access to the internal IP addresses, which simply are unkown at the upstream end.


> L4S is just one
> proposal and of course it deserves scrutiny before gobbling up the
> precious ECT(1) codepoint,

	I could live with it eating ECT(1), as much as I prefer SCE's conceptually purity and seeming simplicity, my issue is that it gobbles ECT(1) even though ECT(1) does not solve the issue L4S is using it for; and they know this. but try to just argue the issue away instead of at least trying to empirically quantify it.


> however it seems to have some traction with
> vendors and a chance at wide deployment with a view to address address
> exactly this problem *at* the bottleneck.

	Let's see how this plays out.

> For that, it also deserves
> consideration.
> 
>>        Why do I care, because that ingress shaping setup is what I use at home, and I have zero confidence that ISPs will come up with a solution to my latency desires that I am going to be happy with... And what I see from the L4S mixes light with a lot of shadows.
>> 
>> 
>>> I don't think there would be any such ingress shapers
>>> configured on home gateways. Certainly not by anyone on this list...
>>> anyone running non-fq codel or flowblind cake for ingress shaping?
>> 
>>        As stated above, I believe fq to not be a reliable safety valve for the ingress shaping case.
> 
> -Ryan

Re: [Bloat] number of home routers with ingress AQM

[Jonathan Morton]

>> Fact is... ingress shaping is a hack.
> 
> That is harsh, but I give you sub-optimal and approximate

As the designer of this particular hack, I would characterise it as a workaround.  It *does* work, within certain assumptions which are narrower than for a conventional before-the-bottleneck deployment.  Cake includes a specific shaper mode to help expand those constraints slightly.

One of the assumptions is indeed that the flows respond promptly to AQM, or are non-queue-building in the first place.  Without that, the ingress shaper cannot exercise control over the contents of the dumb bottleneck queue upstream of it.  There is no clear relationship between the fill levels of the two queues; the dumb one can be full but only trickling into the smart one downstream.

Conventional TCP traffic responds to a CE mark after one RTT and should clear its excess traffic from the queue well within two (assuming congestion-avoidance mode); if the AQM also has a response delay built in (as Codel does), that may result in 3 RTTs between congestion onset and establishment of control.  Slow start is another matter entirely, but one I haven't analysed very throughly in this context; it's safe to say however that draining a 2x overshoot will take longer than a few-segment overshoot.

SCE would potentially remove some of that delay, as it signals immediately a queue appears, but we're still talking about one-plus RTTs delay before the dumb queue is drained.  SCE also pulls the TCP out of slow-start soon enough to avoid a 2x overshoot (there's an interaction with sender-side pacing here; without pacing, slow-start is exited very early).  This is the best-case scenario until the true bottleneck learns AQM.

L4S' modification of the response to CE goes in the opposite direction, unless the AQM is modified to suit.  DCTCP stabilises at 2 CEs per RTT (each CE removes half a segment from the cwnd, while the Reno-linear growth function adds one segment per RTT).  Codel grows its signalling frequency linearly over time; if its interval is set close to the actual path RTT (as is recommended), it will take 3 intervals/RTTs to reach the stability criterion for DCTCP, and a further 3 RTTs to control it all the way down to the true BDP.

This is not too bad in an FQ context, but how long would it take to correct a 2x overshoot from slow-start?  DCTCP doesn't get a halving of cwnd per RTT until every single packet is CE-marked!

I think we need to set up a testbed with the new TCP Prague implementation to demonstrate these effects.

 - Jonathan Morton