Re: [Cake] openwrt build with latest cake and other qdiscs

Re: [Cake] openwrt build with latest cake and other qdiscs

[Alan Jenkins]

Hi

First comment on testing cake:

The usage in the wiki assumes you already know how to change qdiscs on 
the fly.  Here's what happened when I tried the command from the wiki:

root@mortar:~# tc qdisc add dev pppoe-wan cake
RTNETLINK answers: Invalid argument

Here's what doesn't get an error, maybe the wiki should show it instead:

root@mortar:~# tc qdisc replace dev pppoe-wan root cake

root@mortar:~# tc qdisc show dev pppoe-wan
qdisc cake 8004: root refcnt 2 unlimited diffserv4 flows

Regards
Alan

Re: [Cake] openwrt build with latest cake and other qdiscs

[Alan Jenkins]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hi again

I tested Dave's build of cake on wndr3800.  (Didn't try specifying 
cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" 
adjustment worked fine on my adsl.  I can't distinguish it from 
sqm-scripts with the same configuration.

SQM seems to win from having "overhead 40", which I can't configure in cake.


	

up / kbit/s

	

down / kbit/s

	

overhead / bytes

	

Median RRUL / ms

SQM

	

16000

	

950

	

0

	

74

Cake

	

17175

	

995

	

0

	

76


	


	


	


	


Cake

	

16000

	

950

	

0

	

74

SQM

	

17175

	

995

	

40

	

70


	


	


	


	


DSL sync rate

	

17619

	

1020

	


	

59


IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to 
default to 'overhead 44'.  (The highest overhead Seb reported; the 
original worst case overhead[1] plus a vlan header for IPTV).

Alan

[1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage

[-- Attachment #2: Type: text/html, Size: 14090 bytes --]

Re: [Cake] openwrt build with latest cake and other qdiscs

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

"cake" is the most current code. Please use that. I suspect you just ran
into the hard bandwidth/QoS implementation in the older code.

I had not got around to trying the "atm"  option in cake, which tries to
compensate for cell size also. So you should enable that - and I havent
added support for it to sqm either!

On Sun, May 10, 2015 at 10:29 AM, Alan Jenkins <
alan.christopher.jenkins@gmail.com> wrote:

>  Hi again
>
> I tested Dave's build of cake on wndr3800.  (Didn't try specifying
> cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment
> worked fine on my adsl.  I can't distinguish it from sqm-scripts with the
> same configuration.
>
> SQM seems to win from having "overhead 40", which I can't configure in
> cake.
>
>
>   up / kbit/s
>
> down / kbit/s
>
> overhead / bytes
>
> Median RRUL / ms
>
> SQM
>
> 16000
>
> 950
>
> 0
>
> 74
>
> Cake
>
> 17175
>
> 995
>
> 0
>
> 76
>
>
>
>
>
>
>    Cake
>
> 16000
>
> 950
>
> 0
>
> 74
>
> SQM
>
> 17175
>
> 995
>
> 40
>
> 70
>
>
>
>
>
>
>    DSL sync rate
>
> 17619
>
> 1020
>
>
>   59
>
> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to
> default to 'overhead 44'.  (The highest overhead Seb reported; the original
> worst case overhead[1] plus a vlan header for IPTV).
>
> Alan
>
> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>


-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

[-- Attachment #2: Type: text/html, Size: 13228 bytes --]

Re: [Cake] openwrt build with latest cake and other qdiscs

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

I wish we had a good way to monitor cpu usage over the course of a test
like this.

On Sun, May 10, 2015 at 1:37 PM, Dave Taht <dave.taht@gmail.com> wrote:

> "cake" is the most current code. Please use that. I suspect you just ran
> into the hard bandwidth/QoS implementation in the older code.
>
> I had not got around to trying the "atm"  option in cake, which tries to
> compensate for cell size also. So you should enable that - and I havent
> added support for it to sqm either!
>
> On Sun, May 10, 2015 at 10:29 AM, Alan Jenkins <
> alan.christopher.jenkins@gmail.com> wrote:
>
>>  Hi again
>>
>> I tested Dave's build of cake on wndr3800.  (Didn't try specifying
>> cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment
>> worked fine on my adsl.  I can't distinguish it from sqm-scripts with the
>> same configuration.
>>
>> SQM seems to win from having "overhead 40", which I can't configure in
>> cake.
>>
>>
>>   up / kbit/s
>>
>> down / kbit/s
>>
>> overhead / bytes
>>
>> Median RRUL / ms
>>
>> SQM
>>
>> 16000
>>
>> 950
>>
>> 0
>>
>> 74
>>
>> Cake
>>
>> 17175
>>
>> 995
>>
>> 0
>>
>> 76
>>
>>
>>
>>
>>
>>
>>    Cake
>>
>> 16000
>>
>> 950
>>
>> 0
>>
>> 74
>>
>> SQM
>>
>> 17175
>>
>> 995
>>
>> 40
>>
>> 70
>>
>>
>>
>>
>>
>>
>>    DSL sync rate
>>
>> 17619
>>
>> 1020
>>
>>
>>   59
>>
>> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to
>> default to 'overhead 44'.  (The highest overhead Seb reported; the original
>> worst case overhead[1] plus a vlan header for IPTV).
>>
>> Alan
>>
>> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>>
>
>
> --
> Dave Täht
> Open Networking needs **Open Source Hardware**
>
> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
>



-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

[-- Attachment #2: Type: text/html, Size: 14064 bytes --]

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Dave,
On May 10, 2015, at 22:38 , Dave Taht <dave.taht@gmail.com> wrote:

> I wish we had a good way to monitor cpu usage over the course of a test like this.

	I fully agree, I resort to ssh-ing into the router and run “top -d 1” on the shell to at least get some handle on the CPU use, to assess whether my poor wndr3700v2 runs out of steam? Just as an aside for lurkers on the list, the way to read the top output is looking at the idle percentage which should be > 0% (or looking at system user and sirq, as SIRQ typically is the highest, just looking at system and user will not give you the information you were looking for, if the question is can this router handle the offered load)

Best Regards
	Sebastian


> 
> On Sun, May 10, 2015 at 1:37 PM, Dave Taht <dave.taht@gmail.com> wrote:
> "cake" is the most current code. Please use that. I suspect you just ran into the hard bandwidth/QoS implementation in the older code. 
> 
> I had not got around to trying the "atm"  option in cake, which tries to compensate for cell size also. So you should enable that - and I havent added support for it to sqm either!
> 
> On Sun, May 10, 2015 at 10:29 AM, Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
> Hi again
> 
> I tested Dave's build of cake on wndr3800.  (Didn't try specifying cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment worked fine on my adsl.  I can't distinguish it from sqm-scripts with the same configuration.
> 
> SQM seems to win from having "overhead 40", which I can't configure in cake.
> 
> 
> up / kbit/s
> 
> down / kbit/s
> 
> overhead / bytes
> 
> Median RRUL / ms
> 
> SQM
> 
> 16000
> 
> 950
> 
> 0
> 
> 74
> 
> Cake
> 
> 17175
> 
> 995
> 
> 0
> 
> 76
> 
> 
> 
> 
> 
> 
> Cake
> 
> 16000
> 
> 950
> 
> 0
> 
> 74
> 
> SQM
> 
> 17175
> 
> 995
> 
> 40
> 
> 70
> 
> 
> 
> 
> 
> 
> DSL sync rate
> 
> 17619
> 
> 1020
> 
> 
> 59
> 
> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to default to 'overhead 44'.  (The highest overhead Seb reported; the original worst case overhead[1] plus a vlan header for IPTV).
> 
> Alan
> 
> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> 
> -- 
> Dave Täht
> Open Networking needs **Open Source Hardware**
> 
> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
> 
> 
> 
> -- 
> Dave Täht
> Open Networking needs **Open Source Hardware**
> 
> https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

HI Alan,

I believe we got side tracked while discussing cake’s atm adaptation layer, but with a bit of luck you could use tc’s stab option to add the overhead and leave the atm encapsulation to cake (which does a better job than stab). If you could try and report back that would be helpful. I am a bit in a hurry, but will have more time next week to help out if need be…


Best Regards
	Sebastian

On May 10, 2015, at 19:29 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> Hi again
> 
> I tested Dave's build of cake on wndr3800.  (Didn't try specifying cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment worked fine on my adsl.  I can't distinguish it from sqm-scripts with the same configuration.
> 
> SQM seems to win from having "overhead 40", which I can't configure in cake.
> 
> 
> up / kbit/s
> down / kbit/s
> overhead / bytes
> Median RRUL / ms
> SQM
> 16000
> 950
> 0
> 74
> Cake
> 17175
> 995
> 0
> 76
> 
> 
> 
> 
> 
> Cake
> 16000
> 950
> 0
> 74
> SQM
> 17175
> 995
> 40
> 70
> 
> 
> 
> 
> 
> DSL sync rate
> 17619
> 1020
> 
> 59
> 
> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to default to 'overhead 44'.  (The highest overhead Seb reported; the original worst case overhead[1] plus a vlan header for IPTV).
> 
> Alan
> 
> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] openwrt build with latest cake and other qdiscs

[Dave Taht]

no, that is not how cake works. I think.

On Sun, May 10, 2015 at 2:46 PM, Sebastian Moeller <moeller0@gmx.de> wrote:
> HI Alan,
>
> I believe we got side tracked while discussing cake’s atm adaptation layer, but with a bit of luck you could use tc’s stab option to add the overhead and leave the atm encapsulation to cake (which does a better job than stab). If you could try and report back that would be helpful. I am a bit in a hurry, but will have more time next week to help out if need be…
>
>
> Best Regards
>         Sebastian
>
> On May 10, 2015, at 19:29 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>
>> Hi again
>>
>> I tested Dave's build of cake on wndr3800.  (Didn't try specifying cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment worked fine on my adsl.  I can't distinguish it from sqm-scripts with the same configuration.
>>
>> SQM seems to win from having "overhead 40", which I can't configure in cake.
>>
>>
>> up / kbit/s
>> down / kbit/s
>> overhead / bytes
>> Median RRUL / ms
>> SQM
>> 16000
>> 950
>> 0
>> 74
>> Cake
>> 17175
>> 995
>> 0
>> 76
>>
>>
>>
>>
>>
>> Cake
>> 16000
>> 950
>> 0
>> 74
>> SQM
>> 17175
>> 995
>> 40
>> 70
>>
>>
>>
>>
>>
>> DSL sync rate
>> 17619
>> 1020
>>
>> 59
>>
>> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to default to 'overhead 44'.  (The highest overhead Seb reported; the original worst case overhead[1] plus a vlan header for IPTV).
>>
>> Alan
>>
>> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Alan,

On May 10, 2015, at 19:29 , Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:

> Hi again
> 
> I tested Dave's build of cake on wndr3800.  (Didn't try specifying cake2/cake3; I'm guessing tc cake means cake3 on there).  "atm" adjustment worked fine on my adsl.  I can't distinguish it from sqm-scripts with the same configuration.
> 
> SQM seems to win from having "overhead 40", which I can't configure in cake.

	As expected, specifying the overhead is quite important for ATM links, since it shifts the cell padding around (worst case without overhead accounting is to drag in an 47of48 byte payload cell that the accounting did not expect, causing almost 50% unaccounted overhead for small packets, no shaper is going to be happy with such precision ;)

> 
> [table snipped as it did not survive my mailer...]
> IMO it wouldn't hurt for 'tc atm', 'tc adsl', and everything else to default to 'overhead 44'.  (The highest overhead Seb reported; the original worst case overhead[1] plus a vlan header for IPTV).

	Actually there old worst case was 44, with added VLAN tagging this would end up at 48 bytes, but that would also include the ethernet frame check sequence, which I have never seen used over ATM in the real world (which sounds quite weird, without the FCS the payload is not checked for bit errors anymore, so we relay on a) the atm checksums and b) that neither the ethernet to ATM nor the ATM to ethernet repackaging introduces bit errors)… (but I do not claim that I have seen that much). Also I wonder what about double VLAN tagging as is sometimes used bit bitstream access, no idea where the 2nd VLAN gets terminated, one would hope way before the DSL link, but certainty would be nice. I would humbly suggest that we should just get a numerical option for cake to specify the overhead...

Best Regards
	Sebastian

> 
> Alan
> 
> [1] http://ace-host.stuart.id.au/russell/files/tc/tc-atm/#usage
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 11 May, 2015, at 09:50, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> I would humbly suggest that we should just get a numerical option for cake to specify the overhead...

That is in the plans, along with keywords in tc as mnemonics for the most common configurations.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

>> I would humbly suggest that we should just get a numerical option for cake to specify the overhead...
> 
> That is in the plans, along with keywords in tc as mnemonics for the most common configurations.

I’ve just pushed support for an overhead parameter; both cake itself and the iproute2 module.  I took the opportunity to put in a minor optimisation for the cell-framing compensation as well.

The iproute2 support isn’t as complete as I’d eventually like.  However, you can specify a numerical overhead (positive or negative, within sane limits) independently of whether ATM framing compensation is turned on or off.

There are also two “easy mode” keywords: “raw” is equivalent to “noatm overhead 0”, and “conservative” means “atm overhead 48”.  The latter will almost certainly overestimate the actual overhead.

The help text for cake also now indicates the default options more clearly.

Also pushed is a one-line build fix for iproute2.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Jonathan,


On May 13, 2015, at 08:43 , Jonathan Morton <chromatix99@gmail.com> wrote:

>>> I would humbly suggest that we should just get a numerical option for cake to specify the overhead...
>> 
>> That is in the plans, along with keywords in tc as mnemonics for the most common configurations.
> 
> I’ve just pushed support for an overhead parameter; both cake itself and the iproute2 module.  I took the opportunity to put in a minor optimisation for the cell-framing compensation as well.

	Great, thanks a lot. I have a question though: http://lxr.free-electrons.com/ident?i=psched_l2t_ns basically does the same operation, but slightly different:
               DIV_ROUND_UP instead of do_div((n+d-1), d)
What is the kernel policy here, reuse specialized macros or rather code more readable (with slight redundancy)?


Also from:
static inline u32 cake_overhead(struct cake_sched_data *q, u32 in)
{
        u32 out = in + q->rate_overhead;
        
        if(q->rate_flags & CAKE_FLAG_ATM) {
                out += 47;
                do_div(out, 48);
                out *= 53;
        }
        
        return out;
}       

It seems clear that cake does fully rely on the supplied overhead, unlike htb which will automatically add ethernet overhead and an estimate? of the additional header GRO packets drag in, see:
http://lxr.free-electrons.com/source/net/core/dev.c#L2744

I know we generally recommend not to use GRO oat least on slow links, but maybe cake should do the right thing even if encountering offloads?

I actually like that cake does not try to auto-adjust the overhead by itself, since the kernel does this automatically for an ethernet link, but not say for a PPPoE interface, making it a bit tricky to recommend the proper encapsulation to ATM users, “use 40 if you shape on the pppoe-wan interface but 26 if you shape on the wan interface directly is a sure way to confuse people”. That said autadjusting for GRO does seem to make some sense, even the light version of just getting the number of actual ethernet packets hiding in the GRO aggregate an multiplying the rate_overhead with that number, or probably better just multiply out by the number of packets in the aggregate (to make sure we honor AAL5’s nasty integer number of cells per packet rule). I am out of my league here, but I have a hunch this might be something useful for the cake audience and the goal to keep configuration simple.

> 
> The iproute2 support isn’t as complete as I’d eventually like.  However, you can specify a numerical overhead (positive or negative, within sane limits) independently of whether ATM framing compensation is turned on or off.

	Excellent, so this will also work for PTM encapsulation and additional ISP mandated overheads.

> 
> There are also two “easy mode” keywords: “raw” is equivalent to “noatm overhead 0”, and “conservative” means “atm overhead 48”.  The latter will almost certainly overestimate the actual overhead.

	Great, this is probably the most user friendly set of keywords to start out with, at worst overestimating the overhead can make the shaper assume a packet is double its actual wire size and hence wasting 50% of bandwidth, but the larger the packet the smaller the wasted percentage of bandwidth get, and users can iteratively decrease the overhead to find the sweet spot for their link. So I fully endorse these two settings (for what my endorsement is worth ;) )

> 
> The help text for cake also now indicates the default options more clearly.
> 
> Also pushed is a one-line build fix for iproute2.

	This is really looking great, I will need to switch my router over to this (or better get a new router so the old one can stay configured as is, as family friendly back-up ;) )

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

>> I’ve just pushed support for an overhead parameter; both cake itself and the iproute2 module.  I took the opportunity to put in a minor optimisation for the cell-framing compensation as well.
> 
> 	Great, thanks a lot. I have a question though: http://lxr.free-electrons.com/ident?i=psched_l2t_ns basically does the same operation, but slightly different:
>               DIV_ROUND_UP instead of do_div((n+d-1), d)
> What is the kernel policy here, reuse specialized macros or rather code more readable (with slight redundancy)?

It looks as though the DIV_ROUND_UP() expands to exactly the same code, except that a plain division is used instead of do_div().  The latter includes a conversion to multiplying by the inverse on ARM, when the divisor is a constant (which it is), since ARM doesn’t have a hardware integer divide.  (AArch64 does.)

With that said, I haven’t closely examined the resulting assembler.

I’m also not going to use psched_l2t_ns(), because I use the corrected packet length for other purposes than just time.  It also fails to support negative overheads, which can occasionally occur when using IPoA.

> It seems clear that cake does fully rely on the supplied overhead, unlike htb which will automatically add ethernet overhead and an estimate? of the additional header GRO packets drag in, see:
> http://lxr.free-electrons.com/source/net/core/dev.c#L2744

I can’t figure out the connection between HTB and that code.  Also, that appears to be GSO, not GRO.  I’m not precisely sure what the difference is, but I’d hazard a guess that GSO is outbound, GRO is inbound.

Frankly, I hate having to deal with packet aggregates in the core network stack.  Device drivers can aggregate if that makes sense for the hardware, but I’d much rather that was kept out of my qdisc.  Peeling is on the agenda; that’ll make sure we are dealing with actual, individual packets when we need to.  Certainly when dealing with cell-framing overhead, we *always* need to know individual packet sizes.

> I actually like that cake does not try to auto-adjust the overhead by itself, since the kernel does this automatically for an ethernet link, but not say for a PPPoE interface, making it a bit tricky to recommend the proper encapsulation to ATM users, “use 40 if you shape on the pppoe-wan interface but 26 if you shape on the wan interface directly is a sure way to confuse people”.

I consider that a user-interface problem, as well as reflecting a general problem with PPPoE.  Actually, PPPoE has *never* been user-friendly; it outright sucks in all respects.  I can’t think of a single reason to use PPPoE instead of PPPoA.  AFAIK, all Finnish and most British DSL ISPs use either PPPoA or bridging; I’ve only personally encountered PPPoE in the US.

To help reduce confusion, it would probably be best to offer consistent advice on which interface to shape and how much overhead to account for there.  I think shaping the traffic that actually goes over the link is more correct than shaping the traffic that goes to the modem (which might include some management traffic that doesn’t go on the wire).  So you should shape on the PPPoE interface and add the full 40 bytes there.  Happily, this advice is also safe if the user accidentally selects the wrong interface, since 40 bytes is conservative for the Ethernet interface.

Anyway, user-interface problems are best solved in userspace.  Cake’s internal implementation is thus kept simple and numerical.  The tc module now supports that directly, and more user-friendly support can be added either there or in external scripts, or some combination of the two.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Alan Jenkins]

On 14/05/15 11:24, Jonathan Morton wrote:
> I can’t figure out the connection between HTB and that code. Also,
> that appears to be GSO, not GRO. I’m not precisely sure what the
> difference is, but I’d hazard a guess that GSO is outbound, GRO is
> inbound.

I've just been reading thanks to Seb.  GSO is outbound.  GRO is inbound.

They appear to interact in routing: if you GRO packets, you end up with 
 >MTU skbs, which then *have* to be de-aggregated somehow before you can 
send them.

(And the popular reference on GRO is https://lwn.net/Articles/358910/).

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 14 May, 2015, at 13:33, Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
> 
> On 14/05/15 11:24, Jonathan Morton wrote:
>> I can’t figure out the connection between HTB and that code. Also,
>> that appears to be GSO, not GRO. I’m not precisely sure what the
>> difference is, but I’d hazard a guess that GSO is outbound, GRO is
>> inbound.
> 
> I've just been reading thanks to Seb.  GSO is outbound.  GRO is inbound.
> 
> They appear to interact in routing: if you GRO packets, you end up with >MTU skbs, which then *have* to be de-aggregated somehow before you can send them.
> 
> (And the popular reference on GRO is https://lwn.net/Articles/358910/).

Which says that GSO can resegment GRO chunks.  Hmm.

Are we typically getting GRO in CPE devices?

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Jonathan,


On May 14, 2015, at 12:24 , Jonathan Morton <chromatix99@gmail.com> wrote:

>>> I’ve just pushed support for an overhead parameter; both cake itself and the iproute2 module.  I took the opportunity to put in a minor optimisation for the cell-framing compensation as well.
>> 
>> 	Great, thanks a lot. I have a question though: http://lxr.free-electrons.com/ident?i=psched_l2t_ns basically does the same operation, but slightly different:
>>              DIV_ROUND_UP instead of do_div((n+d-1), d)
>> What is the kernel policy here, reuse specialized macros or rather code more readable (with slight redundancy)?
> 
> It looks as though the DIV_ROUND_UP() expands to exactly the same code, except that a plain division is used instead of do_div().  The latter includes a conversion to multiplying by the inverse on ARM, when the divisor is a constant (which it is), since ARM doesn’t have a hardware integer divide.  (AArch64 does.)
> 
> With that said, I haven’t closely examined the resulting assembler.

	I just noticed the difference and thought I’d bring it up, so I can understand the code better, that’s all.

> 
> I’m also not going to use psched_l2t_ns(), because I use the corrected packet length for other purposes than just time.

	Sure, HTB does its accounting in a weird way, and the different rate tables plainly confuse me. I was just referencing thgis code for the do_div vs DIV_ROUND_UP question.

>  It also fails to support negative overheads, which can occasionally occur when using IPoA.

	I know, that is why we default to “stab” in sqm scripts… and as far as I can see Alan tested whether stab works with cake and it seems it does. Still it is much better if cake controls both overhead and encapsulation, since stab’s encapsulation handling is not optimal.

> 
>> It seems clear that cake does fully rely on the supplied overhead, unlike htb which will automatically add ethernet overhead and an estimate? of the additional header GRO packets drag in, see:
>> http://lxr.free-electrons.com/source/net/core/dev.c#L2744
> 
> I can’t figure out the connection between HTB and that code.  

	Well, this function is called by __dev_xmit_skb (see http://lxr.free-electrons.com/source/net/core/dev.c#L2774 ) so it is not HTB specific, that is everyone looking in qdisc_skb_cb(skb)->pkt_len for the size seems to get that adjustment, only the following call to  qdisc_calculate_pkt_len(skb, q); unfortunately overrides skb->pkt_len with skb->len+overhead, but everybody else using pkt_len should get this size correction, I believe.




> Also, that appears to be GSO, not GRO.

	My bad, I was using GRO just as a moniker for packet aggregate processing in the network stack, without even thinking through the details.

>  I’m not precisely sure what the difference is, but I’d hazard a guess that GSO is outbound, GRO is inbound.

	No idea.

> 
> Frankly, I hate having to deal with packet aggregates in the core network stack.  

	But that ship has sailed, I fear, at high speeds the network stack profits noticeably by not going through the motions for each packet sequentially, but basically treating a batch of packets as one that the NIC will then segment out, so I have my doubts whether this is going away any time soon.

> Device drivers can aggregate if that makes sense for the hardware, but I’d much rather that was kept out of my qdisc.  Peeling is on the agenda; that’ll make sure we are dealing with actual, individual packets when we need to.  

	I agree, that sounds conceptually much cleaner, but peeling is going to be costlier than pushing the segmentation to the NIC, so bandwidth aficionados will not appreciate unconditional peeling, I would guess.

> Certainly when dealing with cell-framing overhead, we *always* need to know individual packet sizes.

	Well that or the sum for an aggregate as long as the sum takes all fancy “celling” into account, all we really need to know to how many bits the data expands on the wire.

> 
>> I actually like that cake does not try to auto-adjust the overhead by itself, since the kernel does this automatically for an ethernet link, but not say for a PPPoE interface, making it a bit tricky to recommend the proper encapsulation to ATM users, “use 40 if you shape on the pppoe-wan interface but 26 if you shape on the wan interface directly is a sure way to confuse people”.
> 
> I consider that a user-interface problem, as well as reflecting a general problem with PPPoE.  Actually, PPPoE has *never* been user-friendly; it outright sucks in all respects.  I can’t think of a single reason to use PPPoE instead of PPPoA.  AFAIK, all Finnish and most British DSL ISPs use either PPPoA or bridging; I’ve only personally encountered PPPoE in the US.

	Again, I agree, but say in Germany all big ISPs use PPPoE, even over fiber, so this is going to stay with us a bit longer. Since ATM is going to go the way of the dodo fast, PPPoA will not be an option for much longer, so dhcp would be nice to have (it is not like the ISP does not know which line it services anyway, so the billing and identification issue that is often brought up is a bit of a straw man; I believe they just stick to it because their billing back-end already knows how to handle this).

> 
> To help reduce confusion, it would probably be best to offer consistent advice on which interface to shape and how much overhead to account for there.  I think shaping the traffic that actually goes over the link is more correct than shaping the traffic that goes to the modem (which might include some management traffic that doesn’t go on the wire).  So you should shape on the PPPoE interface and add the full 40 bytes there.

	Well almost, this depends whether there is a BRAS throttle or not, the pppoe interface does not see or account for the PPPoE management packets, that without BRAS throttling will also eat up bits on the DSL link. I admit those packets are rare, but still… There should be no other important traffic to the modem heavy enough to be noticeable to the user. That said, I currently shape on pppoe-ge00, and it works well enough, I guess the PPPoE traffic simply squeezes into the small %age the shaper is reduced from line/throttled rate.


>  Happily, this advice is also safe if the user accidentally selects the wrong interface, since 40 bytes is conservative for the Ethernet interface.

	As seen from our latency focussed vantage point ;)

> 
> Anyway, user-interface problems are best solved in userspace.  Cake’s internal implementation is thus kept simple and numerical.  The tc module now supports that directly, and more user-friendly support can be added either there or in external scripts, or some combination of the two.

	Okay, sounds like a good division of labor between the kernel and tc ;)


Best Regards
	Sebastian

> - Jonathan Morton
> 

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 14 May, 2015, at 13:58, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> Peeling is on the agenda; that’ll make sure we are dealing with actual, individual packets when we need to.  
> 
> I agree, that sounds conceptually much cleaner, but peeling is going to be costlier than pushing the segmentation to the NIC, so bandwidth aficionados will not appreciate unconditional peeling, I would guess.
> 
>> Certainly when dealing with cell-framing overhead, we *always* need to know individual packet sizes.
> 
> Well that or the sum for an aggregate as long as the sum takes all fancy “celling” into account, all we really need to know to how many bits the data expands on the wire.

Since GRO and GSO are for Ethernet and thus *don’t* take cell-framing overhead into account, I really do have to break them up for that purpose if no other.

But beyond that, it can be conditional.  I propose that peeling be done if either:

1) the ATM flag is set, or

2) the aggregate would occupy more than 1ms on the wire (at the shaped rate).

This would imply that pairs of 1500-byte packets would be separated at shaping rates up to 24Mbps, but smaller aggregates of smaller packets could still pass unhindered.  Conveniently, 24Mbps is also the ADSL cap, although since that’s the downstream rate, it’s not quite so relevant.

Incidentally, GSO also interferes with my proposed ELR signalling, in a way that it doesn’t with ordinary ECN.  ELR needs to consider packets individually in order to apply the correct signalling pattern to them.  However, if the NIC did the signalling, that wouldn’t be an objection.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Jonathan,


On May 14, 2015, at 15:12 , Jonathan Morton <chromatix99@gmail.com> wrote:

> 
>> On 14 May, 2015, at 13:58, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>> Peeling is on the agenda; that’ll make sure we are dealing with actual, individual packets when we need to.  
>> 
>> I agree, that sounds conceptually much cleaner, but peeling is going to be costlier than pushing the segmentation to the NIC, so bandwidth aficionados will not appreciate unconditional peeling, I would guess.
>> 
>>> Certainly when dealing with cell-framing overhead, we *always* need to know individual packet sizes.
>> 
>> Well that or the sum for an aggregate as long as the sum takes all fancy “celling” into account, all we really need to know to how many bits the data expands on the wire.
> 
> Since GRO and GSO are for Ethernet and thus *don’t* take cell-framing overhead into account, I really do have to break them up for that purpose if no other.

	I am probably daft, but looking at the comment in front of skb_gso_network_seglen() (see http://lxr.free-electrons.com/ident?i=skb_gso_network_seglen ) makes me thick there is a way of getting the lengths of the individual segments of a GSO aggregate, and I assume one just adds the overhead to each and does the atm cell adjustments if need be, no need to touch the actual data or actually peel the geo. Then again I have not really followed the code and could easily be out to lunch. But sure peeling will be better fairness wise.

> 
> But beyond that, it can be conditional.  I propose that peeling be done if either:
> 
> 1) the ATM flag is set, or
> 
> 2) the aggregate would occupy more than 1ms on the wire (at the shaped rate).

	Oh, I like where you going with this a lot!

> 
> This would imply that pairs of 1500-byte packets would be separated at shaping rates up to 24Mbps, but smaller aggregates of smaller packets could still pass unhindered.  Conveniently, 24Mbps is also the ADSL cap, although since that’s the downstream rate, it’s not quite so relevant.

	Upstream the limit is more like 2.8Mbps (for AnnexJ)

> 
> Incidentally, GSO also interferes with my proposed ELR signalling, in a way that it doesn’t with ordinary ECN.  ELR needs to consider packets individually in order to apply the correct signalling pattern to them.  However, if the NIC did the signalling, that wouldn’t be an objection.

	Interesting thoughts.

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 14 May, 2015, at 17:57, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> I am probably daft, but looking at the comment in front of skb_gso_network_seglen() (see http://lxr.free-electrons.com/ident?i=skb_gso_network_seglen ) makes me thick there is a way of getting the lengths of the individual segments of a GSO aggregate

All the segments (except possibly the last) will be MTU sized, I suspect.  The necessary headers would be duplicated, with fresh (sequential) IP IDs, adjusted TCP sequence numbers and recalculated checksums.  The size of the last segment (and maybe the number of segments) would depend on the header size, since a bigger header (due eg. to TCP timestamps) reduces the payload per packet.

I don’t immediately see how to reliably calculate the sizes of the resulting packets.  So I’d rather split them up and be certain about it, in the cases where it matters most, and take the size of the aggregate as sufficiently reliable in other cases.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Sebastian Moeller]

Hi Jonathan,


On May 14, 2015, at 17:32 , Jonathan Morton <chromatix99@gmail.com> wrote:

> 
>> On 14 May, 2015, at 17:57, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>> I am probably daft, but looking at the comment in front of skb_gso_network_seglen() (see http://lxr.free-electrons.com/ident?i=skb_gso_network_seglen ) makes me thick there is a way of getting the lengths of the individual segments of a GSO aggregate
> 
> All the segments (except possibly the last) will be MTU sized, I suspect.  The necessary headers would be duplicated, with fresh (sequential) IP IDs, adjusted TCP sequence numbers and recalculated checksums.  The size of the last segment (and maybe the number of segments) would depend on the header size, since a bigger header (due eg. to TCP timestamps) reduces the payload per packet.

	As I said, I am out of my league here ;), I would not be amazed if there was a way to get to the segment lengths without going to the data (as I believe the GSO header has all the information to pick the individual payloads out of the continuous GSO skb, but I lack the skill to find the correct reference)

> 
> I don’t immediately see how to reliably calculate the sizes of the resulting packets.  So I’d rather split them up and be certain about it, in the cases where it matters most, and take the size of the aggregate as sufficiently reliable in other cases.

	I guess your idea of peeling for ATM carrier and for >1ms wire time aggregates sounds like a decent idea…

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 14 May, 2015, at 21:15, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
> I guess your idea of peeling for ATM carrier and for >1ms wire time aggregates sounds like a decent idea…

Code review time, then:

	/*
	 * We tolerate GSO aggregates if they occupy < 1ms of wire time
	 * AND we don't need to perform ATM cell-framing.  We're unlikely
	 * to need the latter above 24Mbps (best ADSL downlink), where
	 * handling individual packets is still cheap.
	 *
	 * But if those conditions aren't met, we need to split it.
	 */
	if(unlikely(len > (q->rate_bps >> 10) ||
			(q->rate_flags & CAKE_FLAG_ATM)) &&
			skb_is_gso(skb))
	{
		struct sk_buff *segs, *nskb;
		netdev_features_t features = netif_skb_features(skb);
		int ret = NET_XMIT_DROP;

		segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);

		if (IS_ERR_OR_NULL(segs))
			return qdisc_reshape_fail(skb, sch);

		while (segs) {
			nskb = segs->next;
			segs->next = NULL;
			qdisc_skb_cb(segs)->pkt_len = segs->len;

			switch(cake_enqueue(segs, sch)) {
				case NET_XMIT_CN:
					ret = NET_XMIT_CN;
					/* fall */

				case NET_XMIT_SUCCESS:
					if(ret == NET_XMIT_DROP)
						ret = NET_XMIT_SUCCESS;
					qdisc_tree_decrease_qlen(sch, -1);
					/* fall */

				default:;
			}
			segs = nskb;
		}

		qdisc_tree_decrease_qlen(sch, 1);
		consume_skb(skb);
		return ret;
	}

I’m thinking of inserting the above into the beginning of cake_enqueue().  This of course makes it a recursive function (but only by one level).

A bit of lateral thinking was necessary to convert the equivalent code in TBF (which relies on a child qdisc to do the actual queuing) into a form that cake could use, so I’m not pushing it to git until someone can spot what, if anything, I’ve got horribly wrong.  However, it does compile and doesn’t seem to crash when I run it on a box with GSO turned on, so it’s probably worth a cautious try.

One quirk is that, in its current form, it’ll *always* peel GSO aggregates in unlimited mode.  That’s because rate_bps is zero in that case.  I could clean up that test by having a separate “peeling threshold” calculated at configure time.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Dave Taht]

Well, there is a lot of redundant work being done in cake_enqueue in
this case - the diffserv lookup (which might not be redundant but
usually is), the hashing (which will be), grabbing the timestamp
(probably won't hurt to just grab it once for the whole aggregate),
and so on down through cake_enqueue

I do not have sufficient expertise to evaluate the rest. But would
certainly like to benchmark it as is.

On Thu, May 14, 2015 at 3:06 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 14 May, 2015, at 21:15, Sebastian Moeller <moeller0@gmx.de> wrote:
>>
>> I guess your idea of peeling for ATM carrier and for >1ms wire time aggregates sounds like a decent idea…
>
> Code review time, then:
>
>         /*
>          * We tolerate GSO aggregates if they occupy < 1ms of wire time
>          * AND we don't need to perform ATM cell-framing.  We're unlikely
>          * to need the latter above 24Mbps (best ADSL downlink), where
>          * handling individual packets is still cheap.
>          *
>          * But if those conditions aren't met, we need to split it.
>          */
>         if(unlikely(len > (q->rate_bps >> 10) ||
>                         (q->rate_flags & CAKE_FLAG_ATM)) &&
>                         skb_is_gso(skb))
>         {
>                 struct sk_buff *segs, *nskb;
>                 netdev_features_t features = netif_skb_features(skb);
>                 int ret = NET_XMIT_DROP;
>
>                 segs = skb_gso_segment(skb, features & ~NETIF_F_GSO_MASK);
>
>                 if (IS_ERR_OR_NULL(segs))
>                         return qdisc_reshape_fail(skb, sch);
>
>                 while (segs) {
>                         nskb = segs->next;
>                         segs->next = NULL;
>                         qdisc_skb_cb(segs)->pkt_len = segs->len;
>
>                         switch(cake_enqueue(segs, sch)) {
>                                 case NET_XMIT_CN:
>                                         ret = NET_XMIT_CN;
>                                         /* fall */
>
>                                 case NET_XMIT_SUCCESS:
>                                         if(ret == NET_XMIT_DROP)
>                                                 ret = NET_XMIT_SUCCESS;
>                                         qdisc_tree_decrease_qlen(sch, -1);
>                                         /* fall */
>
>                                 default:;
>                         }
>                         segs = nskb;
>                 }
>
>                 qdisc_tree_decrease_qlen(sch, 1);
>                 consume_skb(skb);
>                 return ret;
>         }
>
> I’m thinking of inserting the above into the beginning of cake_enqueue().  This of course makes it a recursive function (but only by one level).
>
> A bit of lateral thinking was necessary to convert the equivalent code in TBF (which relies on a child qdisc to do the actual queuing) into a form that cake could use, so I’m not pushing it to git until someone can spot what, if anything, I’ve got horribly wrong.  However, it does compile and doesn’t seem to crash when I run it on a box with GSO turned on, so it’s probably worth a cautious try.
>
> One quirk is that, in its current form, it’ll *always* peel GSO aggregates in unlimited mode.  That’s because rate_bps is zero in that case.  I could clean up that test by having a separate “peeling threshold” calculated at configure time.
>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

Re: [Cake] openwrt build with latest cake and other qdiscs

[Jonathan Morton]

> On 15 May, 2015, at 05:27, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Well, there is a lot of redundant work being done in cake_enqueue in
> this case - the diffserv lookup (which might not be redundant but
> usually is), the hashing (which will be), grabbing the timestamp
> (probably won't hurt to just grab it once for the whole aggregate),
> and so on down through cake_enqueue
> 
> I do not have sufficient expertise to evaluate the rest. But would
> certainly like to benchmark it as is.

Well, premature optimisation and all that.  I think I can see how to refactor it to take those possibilities into account, but I’d like to know whether it basically works as-is before I try that.

On the assumption that most people who are more interested in the overhead code have already grabbed it, I’ve pushed it to the sch_cake repo so that we’re on the same page.

 - Jonathan Morton

Re: [Cake] openwrt build with latest cake and other qdiscs

[Alan Jenkins]

On 10/05/2015, Alan Jenkins <alan.christopher.jenkins@gmail.com> wrote:
>
> SQM seems to win from having "overhead 40", which I can't configure in
> cake.

<snip HTML table where adding 'overhead 40' reduces latency by 6ms>.
(In less demanding cases there's little difference though).

In my continuing quest to learn tc syntax...

I report that 'tc stab overhead 40' works correctly with 'tc cake
atm'.  It reduces the latency in this case, no difference from SQM
with the same setting.  Can't ask for more than that :).

Assuming there's a new cake build at some point, I will happily try
'tc cake overhead 40'.

I'm not really watching the router CPU.  It seems to be running at
below 20% (all in softirq).

Alan


Also learnt netperf-wrapper --gui and --batch. Really nice tooling,
without it I'd be getting confused and tired by now.

[Cake] More overhead keywords

[Jonathan Morton]

This is entirely userspace (ie. iproute2) stuff:

The initial cake-overhead patch included only “raw” and “conservative” shortcut keywords, alongside the numeric “overhead” parameter for experts.  I’ve now worked out an extended set of keywords which, I think, takes care of all the normal cases.

There are eight new keywords which deal with the basic ADSL configurations.  These switch on ATM cell-framing compensation, and set the overhead based on the raw IP packet as a baseline.

ipoa-vcmux (8)
ipoa-llcsnap (16)
bridged-vcmux (24)
bridged-llcsnap (32)
pppoa-vcmux (10)
pppoa-llc (14)
pppoe-vcmux (32)
pppoe-llcsnap (40)

Note that “pppoa-llc” is not a typo - it really doesn’t involve SNAP, and is thus a little more compact than if it did.

Two more new keywords deal with the basic VDSL2 configurations.  Again, the overheads use IP as a baseline, but this time ATM cell-framing is turned off.  Apparently PTM does have a small additional overhead on the order of 1/128, due to HDLC framing which attaches special meaning to 0x7D and 0x7E bytes; I might need to add approximate handling for that, kernel-side.

pppoe-ptm (27)
bridged-ptm (19)

The final three keywords are not for standalone use, but act as modifiers to some previous keyword.  They can be specified more than once, which is probably only useful for “ether-vlan”.

via-ethernet (-14)
ether-fcs (+4)
ether-vlan (+4)

Does all this look reasonable?  In particular, have I missed anything regarding PTM/VDSL2?  I’m using http://www.ethernetinthefirstmile.com/articles/WTC2002.pdf as a reference, but it isn’t as clear as it might be.

 - Jonathan Morton

Re: [Cake] More overhead keywords

[Sebastian Moeller]

Hi Jonathan,

On May 18, 2015, at 02:49 , Jonathan Morton <chromatix99@gmail.com> wrote:

> This is entirely userspace (ie. iproute2) stuff:
> 
> The initial cake-overhead patch included only “raw” and “conservative” shortcut keywords, alongside the numeric “overhead” parameter for experts.  I’ve now worked out an extended set of keywords which, I think, takes care of all the normal cases.

	Great!

> 
> There are eight new keywords which deal with the basic ADSL configurations.  These switch on ATM cell-framing compensation, and set the overhead based on the raw IP packet as a baseline.
> 
> ipoa-vcmux (8)
> ipoa-llcsnap (16)
> bridged-vcmux (24)
> bridged-llcsnap (32)
> pppoa-vcmux (10)
> pppoa-llc (14)
> pppoe-vcmux (32)
> pppoe-llcsnap (40)
> 
> Note that “pppoa-llc” is not a typo - it really doesn’t involve SNAP, and is thus a little more compact than if it did.

	I am probably odd in my preferences, but to be able one needs to read a certain number of technical documents to figure out which encapsulation is used; these keywords woui;d make more sense sort of if they were something along the lines of: “BT_ADSL”, so that the user would just need to know what she signed up for. Then again no ISP I know guarantees any stability in the encapsulation or actually discloses what is in use...

> 
> Two more new keywords deal with the basic VDSL2 configurations.  Again, the overheads use IP as a baseline, but this time ATM cell-framing is turned off.  Apparently PTM does have a small additional overhead on the order of 1/128, due to HDLC framing which attaches special meaning to 0x7D and 0x7E bytes; I might need to add approximate handling for that, kernel-side.

	HDLC is only used for VDSL1 not for the relevant data bearer channels in VDSL2 (VDSL1 and VDSL2 are not as closely related as ADSL1 and ADSL2 are). The issue with HDLC, in my eyes is that 1/128 is just a stochastic approximation since it is payload dependent, think a packet with just 0x7D as payload which will I believe roughly double in wire size. So really one would need to read through the packet first (which would be what “deepest packet inspection”?), count the forbidden octets and adjust the skb->pkt_len (or cake’s overhead) for the real wire size, but that will not really work on puny routers that are already struggling with NAT/routing/firewalling/traffic-shaping. As far as I know VDSL1 is still in use, allegedly in South Korea, but in Europe VDSL2 seems to be more dominant, no idea about the U.S..

For what it is worth here are my notes about the VDSL2 overhead:

VDSL2 header:
Deutsche Telekom Overhead: 2 Byte PPP + 6 Byte PPPoE + 4 Byte VLAN = 10 Byte
VDSL (IEEE 802.3-2012 61.3 relevant for VDSL2):  1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 4 Byte
#FAST-ETHERNET: 7 Byte Preamble + 1 Byte start of frame delimiter (SFD) + 12 Byte inter #frame gap (IFG) = 20 Byte
COMMON: 4 Byte Frame Check Sequence (FCS) + 6 (dest MAC) + 6 (src MAC) + 2 (ethertype) = 18 byte 	(By now I think that the ethernet FCS is not transmitter over the VDSL segment, but recalculated at the other end, as was usual with ADSL)

maxMTU: 1500

postMTU header:
8 (PPP/PPPoE) = 1492
preMTU header:
VDSL2:
1 (S) + 1 (Ck) + 2 (PTM-FCS) = 4
ethernet
18 (common) + 4 (VLAN) = 22 (with FCS)
14 (common) + 4 (VLAN) = 18

payload:
maxMTU - postMTU_header = 1500 - 8 = 1492

fullFrameSize
maxMTU + preMTU_header = 1500 + 4 + 22 = 1526
maxMTU + preMTU_header = 1500 + 4 + 18 = 1522

And then there is the PTM framing (one octet in 65, the sync octet, does not carry user data and hence is overhead):
PTM 64/65 
framing factor: 64/65 = 0.984615384615

I am not sure whether this is already excluded from the sync rates reported to the user, but since this is independent of data/packet size and we always need to set the shaper lower than 100% anyways these 1.5% might be irrelevant.

I hope this is somewhat clear.

> 
> pppoe-ptm (27)
> bridged-ptm (19)
> 
> The final three keywords are not for standalone use, but act as modifiers to some previous keyword.  They can be specified more than once, which is probably only useful for “ether-vlan”.
> 
> via-ethernet (-14)
> ether-fcs (+4)
> ether-vlan (+4)

	I like this idea, but it might be a bit surprising that these behave differently from the others, maybe prefix them with "mod_" or so? Also if we really want to hold the users hand, only ether-vlan reasonably can appear more than once, so maybe just add a keyword 2nd-ether-vlan and disallow multiple instances of the same key-word? 

> 
> Does all this look reasonable?  In particular, have I missed anything regarding PTM/VDSL2?  I’m using http://www.ethernetinthefirstmile.com/articles/WTC2002.pdf as a reference, but it isn’t as clear as it might be.

I think that IEEE 802.3-2012 61.3 is more explicit and authoritative. But clear as in nice to read and understand this is not… ;) Maybe T-REC-G.993.2-201112-I!!PDF-E can also be of help...


Best Regards
	Sebastian

> 
> - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] More overhead keywords

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

Hmm. Looks like that document I found was really out of date, then. But it
got me fairly close.

For me, the overhead starts at the IP packet (so for your example, at the
1492 byte level), and excludes optional parts of the spec like FCS (since I
have a separate flag for that). So I need to take the 8 bytes for PPPoE,
the 14 for Ethernet and the 4 for PTM, total 26 - one less than my existing
figure.

Presumably the bridged version is also one byte smaller than my
calculation, 18 rather than 19. Is there also a version which transmits IP
without Ethernet framing?

You would then specify "pppoe-ptm ether-vlan via-ethernet" to set your
example connection up the friendly way, or just "overhead 16" for the terse
way (and you can already do that in the current version).

The 64/65 sync overhead is something we'll have to discover by experiment.
Luckily, it's pretty easy to tell whether we're filling up a dumb FIFO or
not.

I've gone for the technical labels for three reasons. First, it reflects
what's actually happening, which generally reduces confusion in the long
run. Second, you might underestimate the number of ADSL ISPs worldwide, as
well as the difficulty of keeping such a database up to date. Third, every
DSL modem and ISP I know of has made it reasonably easy to discover at
least the base encapsulations - autodetection of vcmux vs llc isn't
absolutely reliable, for example. They might be less forthcoming about vlan
and FCS, but one can make intelligent guesses here, based on whether it's a
converged services ISP.

Ideally, we could do with a tool (at dslreports?) which makes detecting the
actual overhead easier. This would be doable using small packets to magnify
the differences.

And if the user really can't work it out, they can always throw up their
hands and specify "conservative".

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 2063 bytes --]

Re: [Cake] More overhead keywords

[Sebastian Moeller]

Hi Jonathan,

On May 18, 2015, at 10:13 , Jonathan Morton <chromatix99@gmail.com> wrote:

> Hmm. Looks like that document I found was really out of date, then. But it got me fairly close.
> 
> For me, the overhead starts at the IP packet (so for your example, at the 1492 byte level), and excludes optional parts of the spec like FCS (since I have a separate flag for that). So I need to take the 8 bytes for PPPoE, the 14 for Ethernet and the 4 for PTM, total 26 - one less than my existing figure.
> 
> Presumably the bridged version is also one byte smaller than my calculation, 18 rather than 19. Is there also a version which transmits IP without Ethernet framing?

	Not as far as I know, but I do not claim to be an expert here.

> 
> You would then specify "pppoe-ptm ether-vlan via-ethernet" to set your example connection up the friendly way, or just "overhead 16" for the terse way (and you can already do that in the current version).
> 
> The 64/65 sync overhead is something we'll have to discover by experiment. Luckily, it's pretty easy to tell whether we're filling up a dumb FIFO or not.

	That is what I had thought as well, before realizing my ISP actually throttles my connection at the BRAS so that the VDSL link itself never becomes the bottleneck…

> 
> I've gone for the technical labels for three reasons. First, it reflects what's actually happening, which generally reduces confusion in the long run.

	I am not sure that wielding term as vcmix and llcsnap reduces confusion ;) just kidding.

> Second, you might underestimate the number of ADSL ISPs worldwide, as well as the difficulty of keeping such a database up to date.

	Fair enough.

> Third, every DSL modem and ISP I know of has made it reasonably easy to discover at least the base encapsulations - autodetection of vcmux vs llc isn't absolutely reliable, for example.

	Well ,not in Germany were it is all detective work and no clear documentation from the ISPs.

> They might be less forthcoming about vlan and FCS, but one can make intelligent guesses here, based on whether it's a converged services ISP.

	True

> 
> Ideally, we could do with a tool (at dslreports?) which makes detecting the actual overhead easier. This would be doable using small packets to magnify the differences.

	I happen to have a very hacky implementation of such a tool available, but that only works for ATM carriers, I  am not sure whether this really can be done for VDSL2 at all (for example, some ISPs limit the number of packets per second so one never can saturate the link with small packets). But for ATM it just takes a few hours of collecting data and a bit of processing ;) 

> 
> And if the user really can't work it out, they can always throw up their hands and specify "conservative".
> 
> - Jonathan Morton

Re: [Cake] More overhead keywords

[David Lang]

[-- Attachment #1: Type: TEXT/Plain, Size: 1174 bytes --]

On Mon, 18 May 2015, Jonathan Morton wrote:

> I've gone for the technical labels for three reasons. First, it reflects
> what's actually happening, which generally reduces confusion in the long
> run. Second, you might underestimate the number of ADSL ISPs worldwide, as
> well as the difficulty of keeping such a database up to date. Third, every
> DSL modem and ISP I know of has made it reasonably easy to discover at
> least the base encapsulations - autodetection of vcmux vs llc isn't
> absolutely reliable, for example. They might be less forthcoming about vlan
> and FCS, but one can make intelligent guesses here, based on whether it's a
> converged services ISP.

offer both the official labels and some easy-to-understand boxes to use

for example

underlying packet size (ATM = 48, Ethernet = 1500)

protocol overhead (various examples)

David Lang

> Ideally, we could do with a tool (at dslreports?) which makes detecting the
> actual overhead easier. This would be doable using small packets to magnify
> the differences.
>
> And if the user really can't work it out, they can always throw up their
> hands and specify "conservative".
>
> - Jonathan Morton
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 137 bytes --]

_______________________________________________
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] More overhead keywords

[Dave Taht]

anyone up for doing the man page or imrpoving this further?

http://www.bufferbloat.net/projects/codel/wiki/Cake

On Mon, May 18, 2015 at 12:41 PM, David Lang <david@lang.hm> wrote:
> On Mon, 18 May 2015, Jonathan Morton wrote:
>
>> I've gone for the technical labels for three reasons. First, it reflects
>> what's actually happening, which generally reduces confusion in the long
>> run. Second, you might underestimate the number of ADSL ISPs worldwide, as
>> well as the difficulty of keeping such a database up to date. Third, every
>> DSL modem and ISP I know of has made it reasonably easy to discover at
>> least the base encapsulations - autodetection of vcmux vs llc isn't
>> absolutely reliable, for example. They might be less forthcoming about
>> vlan
>> and FCS, but one can make intelligent guesses here, based on whether it's
>> a
>> converged services ISP.
>
>
> offer both the official labels and some easy-to-understand boxes to use
>
> for example
>
> underlying packet size (ATM = 48, Ethernet = 1500)
>
> protocol overhead (various examples)
>
> David Lang
>
>
>> Ideally, we could do with a tool (at dslreports?) which makes detecting
>> the
>> actual overhead easier. This would be doable using small packets to
>> magnify
>> the differences.
>>
>> And if the user really can't work it out, they can always throw up their
>> hands and specify "conservative".
>>
>> - Jonathan Morton
>
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>



-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

Re: [Cake] More overhead keywords

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

On 18/05/2015 20:57, Dave Taht wrote:
> anyone up for doing the man page or imrpoving this further?
>
> http://www.bufferbloat.net/projects/codel/wiki/Cake

Well I was going to fix a few typos and include a few pointers on including cake out of tree in OpenWrt but can't edit :-(

Have an account, logged in, see the bloat project with cake subproject....computer says no ;-)


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4791 bytes --]

Re: [Cake] More overhead keywords

[Dave Taht]

I had a thought. the flow_dissector stuff already has a ton of stuff
in it for breaking apart various frame types already, instead of
specifying (at least some of these) options, we could inspect the
traffic itself and set things more automagically?

Re: [Cake] More overhead keywords

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

That might work for the via-ethernet option, but the others inherently
refer to things that aren't visible at the interface we're actually
attached to.

I should push that update, and then someone can think about the
documentation.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 314 bytes --]