[Cake] overhead for double nat VDSL2 connection

Sebastian Moeller moeller0 at gmx.de
Sun Dec 17 17:30:50 EST 2017


HI Mark,


> On Dec 17, 2017, at 17:00, Mark Captur <mark.captur at gmail.com> wrote:
> 
> Thanks for the comprehensive and quick reply :) below are some further answers

	Always happy to help.

> 
> On 17 December 2017 at 16:23, Sebastian Moeller <moeller0 at gmx.de> wrote:
> Hi Mark,
> 
> > On Dec 17, 2017, at 11:45, Mark Captur <mark.captur at gmail.com> wrote:
> >
> > My setup is as follows
> >
> > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> >
> > Here is my current SQM config
> > config queue 'eth1'
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option qdisc 'cake'
> >         option qdisc_advanced '1'
> >         option ingress_ecn 'ECN'
> >         option egress_ecn 'NOECN'
> >         option qdisc_really_really_advanced '1'
> >         option script 'layer_cake.qos'
> >         option interface 'eth0.2'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
> >         option upload '2400'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >         option squash_dscp '1'
> >         option squash_ingress '1'
> >         option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
> >         option download '0'
> >
> > config queue
> >         option debug_logging '0'
> >         option verbosity '5'
> >         option download '0'
> >         option qdisc 'cake'
> >         option script 'layer_cake.qos'
> >         option qdisc_advanced '1'
> >         option squash_dscp '0'
> >         option squash_ingress '0'
> >         option ingress_ecn 'ECN'
> >         option qdisc_really_really_advanced '1'
> >         option egress_ecn 'ECN'
> >         option interface 'eth0.1'
> >         option enabled '1'
> >         option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
> >         option upload '30000'
> >         option linklayer 'ethernet'
> >         option overhead '8'
> >
> > Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).
> 
>         The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
> Done 
> 2) add "mpu 64" to the eqdisc/iqdisc fields
> Done 
> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
> Dne 
> 4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).
> Done  0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 for downstream as other wise ping go very high when line saturated

	Sure, the 98.46% of sync is a theoretical upper limit and not a really reachable for ingress (but you might want tto try the new "ingress" keyword on eth0.1)

>  
> 
> root at OpenWrt-Mi3G:~# cat /etc/config/sqm
> 
> config queue 'eth1'
>         option debug_logging '0'
>         option verbosity '5'
>         option qdisc 'cake'
>         option qdisc_advanced '1'
>         option ingress_ecn 'ECN'
>         option egress_ecn 'NOECN'
>         option qdisc_really_really_advanced '1'
>         option script 'layer_cake.qos'
>         option interface 'eth0.2'
>         option enabled '1'
>         option linklayer 'ethernet'
>         option squash_dscp '1'
>         option squash_ingress '1'
>         option download '0'
>         option upload '2440'
>         option overhead '30'
>         option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost'
>         option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost diffserv4'
> 
> config queue
>         option debug_logging '0'
>         option verbosity '5'
>         option download '0'
>         option qdisc 'cake'
>         option script 'layer_cake.qos'
>         option qdisc_advanced '1'
>         option squash_dscp '0'
>         option squash_ingress '0'
>         option ingress_ecn 'ECN'
>         option qdisc_really_really_advanced '1'
>         option egress_ecn 'ECN'
>         option interface 'eth0.1'
>         option enabled '1'
>         option linklayer 'ethernet'
>         option overhead '30'
>         option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost diffserv4'
>         option upload '30500'
> 
> 
> 4) tell us about your ISP and plan ;)
> ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i sync at 32/2.48 

	Thanks, I had a look, but i could not easily whether they use a VLAN on their vdsl2 link.

Best Regards
	Sebastian
	
> 
> 
> 
> > I tried using ATM_overhead_detector and get the following output
> 
>         Well, yes sorry, this will not work at all for links using PTM; in theory VDSL links could be using ATM and still be in compliance with the ITU specs (at lest that is my reading) but fortunately nobody seems to be doing that. (Also more unfortunately no ISP sseems to be using PTM encoding on ADSL links, but I digress).
> 
> >
> > 869001 lines parsed...
> > Found 144921 ping packets in /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt
> > Elapsed time is 972.739 seconds.
> > Minimum size of ping payload used: 16 bytes.
> > warning: division by zero
> > warning: called from
> >     ATM_overhead_detector at line 201 column 15
> > warning: legend: ignoring extra labels
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Saved figure (5) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png
> > lower bound estimate for one ATM cell RTT based of specified up and downlink is 0.052419 ms.
> > estimate for one ATM cell RTT based on linear fit of the ping sweep data is 0.052419 ms.
> > Starting brute-force search for optimal stair fit, might take a while...
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Best staircase fit cumulative difference is: 2.6952
> > Best linear fit cumulative difference is: 2.7314
> > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 linear fit 2.7314
> 
>         Yes, that difference is simply too small to be meaningful. I am currently collecting reference values for these for different technologies and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair residuals, but curently the code simply reports the smaller value as more likely...
> 
> 
> > remaining ATM cell length after ICMP header is 5 bytes.
> > ICMP RTT of a single ATM cell is 0.05871 ms.
> >
> > Estimated overhead preceding the IP header: 48 bytes
> > Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.png
> >
> > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
> > 48 bytes overhead indicate
> > Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes) overhead...
> > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at modem
> > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 36
> >
> > Add the following to both the egress root qdisc:
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Add the following to both the ingress root qdisc:
> >
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Elapsed time is 1042.06 seconds.
> > Done...
> > ans = [](0x0)
> >
> >
> > Could my VDSL2 be using ATM? so i have the wrong overhead set?
> 
>         Yes in theory it could, but I am almost 100% certain it is not doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I also believe you have the wrong overhead specified... (if you insist on the keywords try pppoe-ptm). Now, please keep in mind that you properly need to "model" the encapsulation on the bottleneck link, if your ISP uses a traffic shaper on the otherside of your link you need to actually model that shapers settings. As far as I can see, ISPs will model the per packet overhead roughly correctly, but will reduce the bandwidth from the sync values reported by the modem, this requires a few iterations of trial and error to get right. Also some ISPs also use VLAN tags on the bottleneck link...
> 
> Hope this helps...
> 
> Best Regards
>         Sebastian
> 
> Here is tc output
> 
> root at OpenWrt-Mi3G:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0)
>   maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc noqueue 0: dev br-lan root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit diffserv4 dual-dsthost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
>  Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 14112b of 4Mb
>  capacity estimate: 31500Kbit
>                    Bulk  Best Effort        Video        Voice
>   thresh       1968Kbit    31500Kbit    15750Kbit     7875Kbit
>   target          9.2ms        2.5ms        2.5ms        2.5ms
>   interval       56.7ms       50.0ms       50.0ms       50.0ms
>   pk_delay          7us        4.3ms        3.5ms        3.2ms
>   av_delay          0us        183us        173us        271us
>   sp_delay          0us          7us          8us          7us
>   pkts                4          408        20384        38841
>   bytes             717        64851     30954992     58820110
>   way_inds            0            0            0            0
>   way_miss            4           64            4           11
>   way_cols            0            0            0            0
>   drops               0            0            5            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            0            0            0
>   bk_flows            0            0            1            0
>   un_flows            0            0            0            0
>   max_len           189         1536         2988         2988
> 
> qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4 dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
>  Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues 0)
>  backlog 10860b 9p requeues 0
>  memory used: 116928b of 4Mb
>  capacity estimate: 2440Kbit
>                    Bulk  Best Effort        Video        Voice
>   thresh      152496bit     2440Kbit     1220Kbit      610Kbit
>   target        119.1ms        7.4ms       14.9ms       29.8ms
>   interval      238.3ms       54.9ms       62.4ms       77.3ms
>   pk_delay          0us      101.5ms        8.2ms          7us
>   av_delay          0us       49.7ms        2.6ms          0us
>   sp_delay          0us        6.8ms        245us          0us
>   pkts                0        22757        11178           10
>   bytes               0      3029896       951692         1096
>   way_inds            0            0            0            0
>   way_miss            0           61            3            3
>   way_cols            0            0            0            0
>   drops               0          130            0            0
>   marks               0            0            0            0
>   ack_drop            0            0            0            0
>   sp_flows            0            1            1            0
>   bk_flows            0            1            0            0
>   un_flows            0            0            0            0
>   max_len             0        11700          120          179
> 
> qdisc noqueue 0: dev wlan1 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc noqueue 0: dev wlan0 root refcnt 2
>  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
>  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> 
> Best regards,
> Mark 
> >
> > here is my tc -s qdisc output
> >
> > root at OpenWrt-Mi3G:~# tc -s qdisc
> > qdisc noqueue 0: dev lo root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
> >   maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> > qdisc noqueue 0: dev br-lan root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> >  Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 36684b of 4Mb
> >  capacity estimate: 30Mbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh       1875Kbit       30Mbit       15Mbit     7500Kbit
> >   target          9.7ms        2.5ms        2.5ms        2.5ms
> >   interval       57.2ms       50.0ms       50.0ms       50.0ms
> >   pk_delay          0us        5.6ms        855us         15us
> >   av_delay          0us        436us         28us         10us
> >   sp_delay          0us         27us          8us          6us
> >   pkts                0        33679          457         1346
> >   bytes               0     19869595        80517        69940
> >   way_inds            0          249            0            0
> >   way_miss            0         3099           19            7
> >   way_cols            0            0            0            0
> >   drops               0           10            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            1            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         7064         2982          350
> >
> > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4 dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> >  Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
> >  backlog 0b 0p requeues 0
> >  memory used: 94752b of 4Mb
> >  capacity estimate: 2400Kbit
> >                    Bulk  Best Effort        Video        Voice
> >   thresh        150Kbit     2400Kbit     1200Kbit      600Kbit
> >   target        121.1ms        7.6ms       15.1ms       30.3ms
> >   interval      242.2ms       55.1ms       62.6ms       77.8ms
> >   pk_delay          0us       29.5ms         19us        4.4ms
> >   av_delay          0us        3.8ms          1us        2.3ms
> >   sp_delay          0us         50us          1us          9us
> >   pkts                0        34979           31         1099
> >   bytes               0     23342354         2930       162457
> >   way_inds            0          177            0            0
> >   way_miss            0         2003           27           16
> >   way_cols            0            0            0            0
> >   drops               0          205            0            0
> >   marks               0            0            0            0
> >   ack_drop            0            0            0            0
> >   sp_flows            0            1            0            0
> >   bk_flows            0            1            0            0
> >   un_flows            0            0            0            0
> >   max_len             0         8342           98          598
> >
> > qdisc noqueue 0: dev wlan1 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc noqueue 0: dev wlan0 root refcnt 2
> >  Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> >  Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
> >   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> >   new_flows_len 0 old_flows_len 0
> >
> > Thanks,
> > Mark
> > _______________________________________________
> > Cake mailing list
> > Cake at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake



More information about the Cake mailing list