[Cerowrt-devel] SQM and PPPoE, more questions than answers...

[Sebastian Moeller]

Hi,

just to document my current understanding of using SQM on a router that also terminates a pppoe wan connection. We basically have two options either set up SQM on the real interface (let’s call it ge00 like cerowrt does) or on the associated pop device, pppoe-ge00. In theory both should produce the same results; in praxis current SQM has significant different results. Let me enumerate the main differences that show up when testing with netperf-wrapper’s RRUL test:

1) SQM on ge00 does not show a working egress classification in the RRUL test (no visible “banding”/stratification of the 4 different priority TCP flows), while SQM on pppoe-ge00 does show this stratification.

	Now the reason for this is quite obvious once we take into account that on ge00 the kernel sees a packet that already contains a PPP header between ethernet and IP header and has a different ether_type field, and our diffserv filters currently ignore everything except straight ipv4 and ipv6 packets, so due to the unexpected/un-handled PPP header everything lands in the default priority class and hence no stratification. If we shape on pppoe-ge00 the kernel seems to do all processing before encapsulating the data with PP so all filters just work. In theory that should be relatively easy to fix (at least for the specific PPPoE case, I am unsure about a generic solution) by using offsets to try to access the TOS bits in PPP-packets. Also most likely we face the same issue in other encapsulations that pass through cerowrt to some degree (except most of those will use an outer IP header from where we can scratch DSCPs…, but I digress)

2) SQM on ge00 shows better latency under load (LUL), the LUL increases for ~2*fq_codels target so 10ms, while SQM on pppeo-ge00 shows a LUL-increase (LULI) roughly twice as large or around 20ms.

	I have no idea why that is, if anybody has an idea please chime in.

3) SQM on pppoe-ge00 has a rough 20% higher egress rate than SQM on ge00 (with ingress more or less identical between the two). Also 2) and 3) do not seem to be coupled, artificially reducing the egress rate on pppoe-ge00 to yield the same egress rate as seen on ge00 does not reduce the LULI to the ge00 typical 10ms, but it stays at 20ms.

	For this I also have no good hypothesis, any ideas?


So the current choice is either to accept a noticeable increase in LULI (but note some years ago even an average of 20ms most likely was rare in the real life) or a equally noticeable decrease in egress bandwidth… 

Best Regards
	Sebastian

P.S.: It turns out, at least on my link, that for shaping on pppoe-ge00 the kernel does not account for any header automatically, so I need to specify a per-packet-overhead (PPOH) of 40 bytes (an an ADSL2+ link with ATM linklayer); when shaping on ge00 however (with the kernel still terminating the PPPoE link to my ISP) I only need to specify an PPOH of 26 as the kernel already adds the 14 bytes for the ethernet header…