* Re: [Cake] Frontier FIOS Framing
2019-09-23 4:26 ` Jonathan Morton
2019-09-23 6:57 ` Marco Belmonte
@ 2019-09-23 7:44 ` Sebastian Moeller
2019-09-23 11:53 ` Ryan Mounce
2 siblings, 0 replies; 5+ messages in thread
From: Sebastian Moeller @ 2019-09-23 7:44 UTC (permalink / raw)
To: Jonathan Morton; +Cc: marco, cake
> On Sep 23, 2019, at 06:26, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> I have searched every nook and cranny of the bloated internet looking for any information I can find on whether Frontier/Verizon FIOS (assuming the only difference between the service offered by both Frontier and Verizon is in name only) requires any special framing parameters passed on to sch_cake's overhead settings. Most mentions of cake/fq/scm/etc and FIOS are ether very dated and inconclusive or I find messages and forum posts asking questions a lot like this one.
>
> I don't know precisely what framing FIOS uses. However, most provisioning shapers used by cable/fibre ISPs operate on Ethernet frames, so if you use the "ethernet" keyword you should match what the shaper is doing. The proof of the pudding is in the eating, of course.
Ethernet will assume 38 bytes of overhead, including the "silent" inter-frame-gap overhead (which essentially is a gap in transmission sized so that one could transmit 12 octets instead), for real active ethernet over fiber that should be correct (maybe it needs another 4 bytes if a vlan tag is used).
But according to https://www.lightreading.com/gigabit/fttx/verizon-preps-next-major-broadband-upgrade/d/d-id/722062 verizon uses GPON, and all we know about GPON is that is uses a 5 byte GEM header which replaces parts of the ethernet overhead (IFG, Preamble, SFD, -> 12+7+1 = 20Bytes) for an estimated per packet overhead of 38-20+5 = 23 Bytes. But I do not know how much additional "hidden" overhead GPON adds to each packet. So "ethernet's" 38 bytes should be a decent safe bet (it is always better to over-estimate per-packet-overhead, as otherwise small packets will fool the shaper).
>
>> Currently this is what I have and am also curious if I should be using the "nat" keyword for both ingress and egress? I'm not entirely sure - see below:
>
> If your box is doing NAT *and* you are using a flow-mode that depends on accurate internal host information, then you should have the "nat" keyboard on in both directions. Otherwise it's more efficient to switch it off, though leaving it on does no harm otherwise.
>
> The default flow-mode is "triple-isolate", which does use internal host information. So do the "dual-srchost" and "dual-dsthost" modes, which are more precise but need you to specify which direction the traffic is flowing. The "besteffort" and "flows" modes do not, but you should only use those if you're deliberately experimenting with something.
>
>> In absence of framing compensation I figured I should just go extreme by reserving more bandwidth than the qdisc needs because I also read somewhere I think that mentioned that if you don't compensate and are incorrect everything stops working as opposed to if you over compensate you might lose out on bandwidth but you'll still win in the latency department.
>
> That's approximately correct, close enough for actual practice. It's also why we included the "conservative" keyword, which applies the maximum amount of framing compensation that is ever likely to be seen in the wild - much more than you'd expect to see on a cable/fibre link, but only slightly more than on most ADSL lines.
I am not 100% sure the "likely to be seen in the wild" assumption still holds, ds-lite will easily add another 40 bytes of overhead for IPv4 packets (as will other IPv4 in IPv6 encapsulations)...
>
> The overhead compensation matters more with small packets than with the larger ones used for bulk transfers; for the latter, reserving a little more bandwidth will appear to make everything work.
+1, for any given packet-size under-estimation of either gross-shaper-rate or per-packet-overhead can be compensated by over-estimating the other, and since most speedtests employ large packets that can lead to under-estimation of the overhead which will cause bufferbloat if there are enough small packets in flight (for which effective transmit duration estimation will be too small with too little overhead leading to observable bufferbloat).
> For fibre I would try "ethernet" and reserve about 1% bandwidth each way, then if possible test to see whether there is any bloat.
First the iteration, perform a speedtest and plug the resulting goodput numbers into cake as gross-shaper-rates, and then you can slowly increase these limits until you see latency under load increase more than you are willing to accept. I would first keep the ingress/download rate at goodput number and optimize egress/upload and then do the same for the ingress rate. Ideally use a bidirectionally saturating speedtest (like flent's RRUL or RRUL_CS8), as that will typically be more sensitive and show a higher magnitude of latency under load increase for the same settings than doing the up-/downloading tests sequentially.
Best Regards
Sebastian
>
> - Jonathan Morton
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Cake] Frontier FIOS Framing
2019-09-23 4:26 ` Jonathan Morton
2019-09-23 6:57 ` Marco Belmonte
2019-09-23 7:44 ` Sebastian Moeller
@ 2019-09-23 11:53 ` Ryan Mounce
2 siblings, 0 replies; 5+ messages in thread
From: Ryan Mounce @ 2019-09-23 11:53 UTC (permalink / raw)
To: Jonathan Morton; +Cc: marco, Cake List
On Mon, 23 Sep 2019 at 13:56, Jonathan Morton <chromatix99@gmail.com> wrote:
>
> The overhead compensation matters more with small packets than with the larger ones used for bulk transfers; for the latter, reserving a little more bandwidth will appear to make everything work. For fibre I would try "ethernet" and reserve about 1% bandwidth each way, then if possible test to see whether there is any bloat.
The "ethernet" keyword is perfect for shaping real line rate ethernet,
however it's more likely that the upstream ONU/OLT/BNG is
shaping/policing without regard for the 8 byte preamble + 12 byte
inter-frame gap. It's almost certainly blind to the native GEM framing
of GPON.
My starting point would be "mpu 64 overhead 18". There are possibly
VLAN tags in use behind the scenes, so try incrementing overhead by 4
from there.
For example, this is the simplified cake incantation for the upstream
of my 100/40 GPON connection:
tc qdisc replace dev eth2 root cake dual-srchost nat mpu 64 overhead
26 bandwidth 40Mbit
There are 2 VLAN tags added by the OLT before my upstream traffic is
policed by an aggregation switch, so 18 bytes ethernet (incl. FCS) + 4
bytes VLAN + 4 bytes VLAN = 26
Correspondingly in the downstream I use:
tc qdisc replace dev ifb2 root cake ingress dual-dsthost nat mpu 64
overhead 26 bandwidth 99Mbit
As alluded to by Jonathan, the 1% margin is required in order for cake
to reliably enforce host/flow fairness on downstream/"ingress"
traffic. So long as mpu/overhead/bandwidth are fine tuned, I haven't
needed to allow for any margin in the upstream.
-Ryan
^ permalink raw reply [flat|nested] 5+ messages in thread