[Cerowrt-devel] SQM and PPPoE, more questions than answers...

[Sebastian Moeller]

Hi All,

some more testing:
On Oct 12, 2014, at 01:12 , Sebastian Moeller <moeller0 at gmx.de> wrote:

> Hi,
> 
> just to document my current understanding of using SQM on a router that also terminates a pppoe wan connection. We basically have two options either set up SQM on the real interface (let’s call it ge00 like cerowrt does) or on the associated pop device, pppoe-ge00. In theory both should produce the same results; in praxis current SQM has significant different results. Let me enumerate the main differences that show up when testing with netperf-wrapper’s RRUL test:
> 
> 1) SQM on ge00 does not show a working egress classification in the RRUL test (no visible “banding”/stratification of the 4 different priority TCP flows), while SQM on pppoe-ge00 does show this stratification.
> 
> 	Now the reason for this is quite obvious once we take into account that on ge00 the kernel sees a packet that already contains a PPP header between ethernet and IP header and has a different ether_type field, and our diffserv filters currently ignore everything except straight ipv4 and ipv6 packets, so due to the unexpected/un-handled PPP header everything lands in the default priority class and hence no stratification. If we shape on pppoe-ge00 the kernel seems to do all processing before encapsulating the data with PP so all filters just work. In theory that should be relatively easy to fix (at least for the specific PPPoE case, I am unsure about a generic solution) by using offsets to try to access the TOS bits in PPP-packets. Also most likely we face the same issue in other encapsulations that pass through cerowrt to some degree (except most of those will use an outer IP header from where we can scratch DSCPs…, but I digress)

	Usind tc filters u32 filter makes it possible to actually dive into PPPoE encapsulated ipv4 and ipv6 packets and perform classification on “pass-through” PPPoE packets (as encountered when starting SQM on ge00 instead of pppoe-ge00, if the latter actually handles the wan connection), so that one is solved (but see below).

> 
> 2) SQM on ge00 shows better latency under load (LUL), the LUL increases for ~2*fq_codels target so 10ms, while SQM on pppeo-ge00 shows a LUL-increase (LULI) roughly twice as large or around 20ms.
> 
> 	I have no idea why that is, if anybody has an idea please chime in.

	Once SQM on ge00 actually dives into the PPPoE packets and applies/tests u32 filters the LUL increases to be almost identical to pppoe-ge00’s if both ingress and egress classification are active and do work. So it looks like the u32 filters I naively set up are quite costly. Maybe there is a better way to set these up...

> 
> 3) SQM on pppoe-ge00 has a rough 20% higher egress rate than SQM on ge00 (with ingress more or less identical between the two). Also 2) and 3) do not seem to be coupled, artificially reducing the egress rate on pppoe-ge00 to yield the same egress rate as seen on ge00 does not reduce the LULI to the ge00 typical 10ms, but it stays at 20ms.
> 
> 	For this I also have no good hypothesis, any ideas?

	With classification fixed the difference in egress rate shrinks to ~10% instead of 20, so this partly seems related to the classification issue as well.

> 
> 
> So the current choice is either to accept a noticeable increase in LULI (but note some years ago even an average of 20ms most likely was rare in the real life) or a equally noticeable decrease in egress bandwidth… 

	I guess it is back to the drawing board to figure out how to speed up the classification… and then revisit the PPPoE question again…

Regards
	Sebastian

> 
> Best Regards
> 	Sebastian
> 
> P.S.: It turns out, at least on my link, that for shaping on pppoe-ge00 the kernel does not account for any header automatically, so I need to specify a per-packet-overhead (PPOH) of 40 bytes (an an ADSL2+ link with ATM linklayer); when shaping on ge00 however (with the kernel still terminating the PPPoE link to my ISP) I only need to specify an PPOH of 26 as the kernel already adds the 14 bytes for the ethernet header…
>