[Cake] CAKE/SQM behind ISP CPE (bridge vs non-bridge, router vs inline)

[Cake] CAKE/SQM behind ISP CPE (bridge vs non-bridge, router vs inline)

[Gabriel Ulysee Cyril Rossetti]

Hi all,

I would appreciate some guidance on deployment patterns where we want to measure connection quality and, where possible, improve queue management using SQM (CAKE or FQ-CoDel).

There are two common starting situations:
   1. The ISP allows using your own router
   2. The ISP does not allow replacing their CPE, or their device must remain in place for some reason (for example telephony/VoIP dependencies)

In the first case, the happy path is to replace the ISP router with an OpenWRT router.

In the second case, the ISP router remains the demarcation point, and we are exploring whether to place either:
   * an OpenWRT router behind it, actively routing for the LAN, or
   * an inline / bridge-like device behind it, primarily for measurement (throughput, latency under load, bufferbloat tests)

Objective
   1.  The primary goal is to measure connection quality at or near the demarcation point, including throughput, latency under load, and bufferbloat behavior.
   2. Secondarily, where the inserted device is actually a router, we would like to improve queue management using SQM.

There are two independent axes in the constrained setup:

A) ISP router state
   * Bridge mode
   * Routing mode (no bridge)

B) Device placed behind ISP router
   * OpenWRT router, actively routing for the LAN, capable of running SQM
   * Inline / bridge-like device, primarily for measurement, not acting as the main router

Scenarios
Scenario A: ISP allows your own router

In this case, we replace the ISP device with an OpenWRT router.

My understanding is that this is the ideal case for SQM:
   * CAKE/FQ-CoDel can shape slightly below line rate
   * The bottleneck is effectively controlled on the OpenWRT router
   * Bufferbloat mitigation should be effective

Scenario B: ISP router remains, but is in bridge mode, downstream OpenWRT router

Here the ISP CPE still remains physically present, but the downstream OpenWRT router becomes the first routing hop and serves the LAN.

My understanding is that this should also be a good case for SQM:
   * The OpenWRT router can shape slightly below line rate
   * The effective bottleneck can be controlled there
   * Bufferbloat mitigation should be effective

Scenario C: ISP router remains in routing mode, downstream OpenWRT router (double NAT)

The ISP CPE remains in routing mode, and we add a second router behind it.

Questions:
   * Does SQM (CAKE/FQ-CoDel) still provide meaningful benefit in practice?
   * Is shaping slightly below the ISP rate sufficient to pull the bottleneck downstream?
   * To what extent do hidden queues in the ISP CPE limit effectiveness?

Scenario D: ISP router remains in routing mode, downstream inline / bridge-like device

Instead of adding a second router, we insert an inline / bridge-like device behind the ISP CPE, primarily for measurement.

Here I am less clear:
   * Does it make sense to run CAKE/FQ-CoDel on an inline / bridge-like device?
   * Or is SQM only really meaningful when the device is actively routing and controlling traffic?
   * If the device is mainly for measurement, is combining SQM and measurement in the same box useful, or conceptually the wrong approach?

My current understanding, which I would welcome corrections to, is:
   * Best case: own OpenWRT router replaces ISP router, or ISP router is bridged and OpenWRT becomes the real router
   * Constrained case: ISP router remains in routing mode, and SQM may still help if we shape low enough, but we remain partially constrained by queues inside the ISP CPE
   * Inline devices are useful for measurement, but it is unclear whether SQM is meaningful in that role

Any corrections, practical deployment patterns, or field experience would be very helpful.

Thanks in advance,
Gabriel

[Cake] Re: CAKE/SQM behind ISP CPE (bridge vs non-bridge, router vs inline)

[David Lang]

remember, you are not running cake on the ISP side of the link, so by 
definition, you do not have full control of your inbound traffic.

As a result, you have to shape to something below line rate as you are working 
with indirect methods that are far less precise than what would be ideal.

As such, having the cpe equipment in the loop probably does not change things 
noticably in terms of managing downstream traffic.

When you are managing the upstream traffic that you are directly controlling, 
you can shape it very close to line rate (just make it the WAN line rate, not 
the ethernet line rate to your cpe :-) )

there are things the cpe can do to hurt you, but if you can't replace it, you 
can't replace it. It won't be as bad as you are fearing.

David Lang