[Cake] overhead for double nat VDSL2 connection
Sebastian Moeller
moeller0 at gmx.de
Sun Dec 17 17:30:50 EST 2017
HI Mark,
> On Dec 17, 2017, at 17:00, Mark Captur <mark.captur at gmail.com> wrote:
>
> Thanks for the comprehensive and quick reply :) below are some further answers
Always happy to help.
>
> On 17 December 2017 at 16:23, Sebastian Moeller <moeller0 at gmx.de> wrote:
> Hi Mark,
>
> > On Dec 17, 2017, at 11:45, Mark Captur <mark.captur at gmail.com> wrote:
> >
> > My setup is as follows
> >
> > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x
> >
> > Here is my current SQM config
> > config queue 'eth1'
> > option debug_logging '0'
> > option verbosity '5'
> > option qdisc 'cake'
> > option qdisc_advanced '1'
> > option ingress_ecn 'ECN'
> > option egress_ecn 'NOECN'
> > option qdisc_really_really_advanced '1'
> > option script 'layer_cake.qos'
> > option interface 'eth0.2'
> > option enabled '1'
> > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost diffserv4'
> > option upload '2400'
> > option linklayer 'ethernet'
> > option overhead '8'
> > option squash_dscp '1'
> > option squash_ingress '1'
> > option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost'
> > option download '0'
> >
> > config queue
> > option debug_logging '0'
> > option verbosity '5'
> > option download '0'
> > option qdisc 'cake'
> > option script 'layer_cake.qos'
> > option qdisc_advanced '1'
> > option squash_dscp '0'
> > option squash_ingress '0'
> > option ingress_ecn 'ECN'
> > option qdisc_really_really_advanced '1'
> > option egress_ecn 'ECN'
> > option interface 'eth0.1'
> > option enabled '1'
> > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost diffserv4'
> > option upload '30000'
> > option linklayer 'ethernet'
> > option overhead '8'
> >
> > Is the overhead correct? should i use the bridged-ptm keyword (or should i use pppoe-ptm).
>
> The overhead certainly seems confusing. Personally, I dislike the overhead related compound keywords like *-ptm and would recommend the following:
> 1) remove the bridged-ptm from the eqdisc/iqdisc fields
> Done
> 2) add "mpu 64" to the eqdisc/iqdisc fields
> Done
> 3) set overhad to 8+18+4 = 30 bytes (you will need to account for everything added on the bottleneck, so your packets will be MTU 1492, to leave room for the PPPoE header that the modem adds).
> Dne
> 4) DO not set the ptm keyword at all, instead make sure to set the shaper bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring needless operations per packet).
> Done 0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 for downstream as other wise ping go very high when line saturated
Sure, the 98.46% of sync is a theoretical upper limit and not a really reachable for ingress (but you might want tto try the new "ingress" keyword on eth0.1)
>
>
> root at OpenWrt-Mi3G:~# cat /etc/config/sqm
>
> config queue 'eth1'
> option debug_logging '0'
> option verbosity '5'
> option qdisc 'cake'
> option qdisc_advanced '1'
> option ingress_ecn 'ECN'
> option egress_ecn 'NOECN'
> option qdisc_really_really_advanced '1'
> option script 'layer_cake.qos'
> option interface 'eth0.2'
> option enabled '1'
> option linklayer 'ethernet'
> option squash_dscp '1'
> option squash_ingress '1'
> option download '0'
> option upload '2440'
> option overhead '30'
> option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost'
> option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost diffserv4'
>
> config queue
> option debug_logging '0'
> option verbosity '5'
> option download '0'
> option qdisc 'cake'
> option script 'layer_cake.qos'
> option qdisc_advanced '1'
> option squash_dscp '0'
> option squash_ingress '0'
> option ingress_ecn 'ECN'
> option qdisc_really_really_advanced '1'
> option egress_ecn 'ECN'
> option interface 'eth0.1'
> option enabled '1'
> option linklayer 'ethernet'
> option overhead '30'
> option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost diffserv4'
> option upload '30500'
>
>
> 4) tell us about your ISP and plan ;)
> ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i sync at 32/2.48
Thanks, I had a look, but i could not easily whether they use a VLAN on their vdsl2 link.
Best Regards
Sebastian
>
>
>
> > I tried using ATM_overhead_detector and get the following output
>
> Well, yes sorry, this will not work at all for links using PTM; in theory VDSL links could be using ATM and still be in compliance with the ITU specs (at lest that is my reading) but fortunately nobody seems to be doing that. (Also more unfortunately no ISP sseems to be using PTM encoding on ADSL links, but I digress).
>
> >
> > 869001 lines parsed...
> > Found 144921 ping packets in /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt
> > Elapsed time is 972.739 seconds.
> > Minimum size of ping payload used: 16 bytes.
> > warning: division by zero
> > warning: called from
> > ATM_overhead_detector at line 201 column 15
> > warning: legend: ignoring extra labels
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Saved figure (5) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png
> > lower bound estimate for one ATM cell RTT based of specified up and downlink is 0.052419 ms.
> > estimate for one ATM cell RTT based on linear fit of the ping sweep data is 0.052419 ms.
> > Starting brute-force search for optimal stair fit, might take a while...
> > Unknown or ambiguous terminal name 'wxt'
> > Unknown or ambiguous terminal name 'wxt'
> > Best staircase fit cumulative difference is: 2.6952
> > Best linear fit cumulative difference is: 2.7314
> > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 linear fit 2.7314
>
> Yes, that difference is simply too small to be meaningful. I am currently collecting reference values for these for different technologies and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair residuals, but curently the code simply reports the smaller value as more likely...
>
>
> > remaining ATM cell length after ICMP header is 5 bytes.
> > ICMP RTT of a single ATM cell is 0.05871 ms.
> >
> > Estimated overhead preceding the IP header: 48 bytes
> > Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.png
> >
> > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/
> > 48 bytes overhead indicate
> > Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes) overhead...
> > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at modem
> > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM AAL5 SAR (8) : Total 36
> >
> > Add the following to both the egress root qdisc:
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Add the following to both the ingress root qdisc:
> >
> > A) Assuming the router connects over ethernet to the DSL-modem:
> > stab mtu 2048 tsize 128 overhead 48 linklayer atm
> >
> > Elapsed time is 1042.06 seconds.
> > Done...
> > ans = [](0x0)
> >
> >
> > Could my VDSL2 be using ATM? so i have the wrong overhead set?
>
> Yes in theory it could, but I am almost 100% certain it is not doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I also believe you have the wrong overhead specified... (if you insist on the keywords try pppoe-ptm). Now, please keep in mind that you properly need to "model" the encapsulation on the bottleneck link, if your ISP uses a traffic shaper on the otherside of your link you need to actually model that shapers settings. As far as I can see, ISPs will model the per packet overhead roughly correctly, but will reduce the bandwidth from the sync values reported by the modem, this requires a few iterations of trial and error to get right. Also some ISPs also use VLAN tags on the bottleneck link...
>
> Hope this helps...
>
> Best Regards
> Sebastian
>
> Here is tc output
>
> root at OpenWrt-Mi3G:~# tc -s qdisc
> qdisc noqueue 0: dev lo root refcnt 2
> Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0)
> maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0
> new_flows_len 0 old_flows_len 0
> qdisc noqueue 0: dev br-lan root refcnt 2
> Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit diffserv4 dual-dsthost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
> Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 14112b of 4Mb
> capacity estimate: 31500Kbit
> Bulk Best Effort Video Voice
> thresh 1968Kbit 31500Kbit 15750Kbit 7875Kbit
> target 9.2ms 2.5ms 2.5ms 2.5ms
> interval 56.7ms 50.0ms 50.0ms 50.0ms
> pk_delay 7us 4.3ms 3.5ms 3.2ms
> av_delay 0us 183us 173us 271us
> sp_delay 0us 7us 8us 7us
> pkts 4 408 20384 38841
> bytes 717 64851 30954992 58820110
> way_inds 0 0 0 0
> way_miss 4 64 4 11
> way_cols 0 0 0 0
> drops 0 0 5 0
> marks 0 0 0 0
> ack_drop 0 0 0 0
> sp_flows 0 0 0 0
> bk_flows 0 0 1 0
> un_flows 0 0 0 0
> max_len 189 1536 2988 2988
>
> qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4 dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64
> Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues 0)
> backlog 10860b 9p requeues 0
> memory used: 116928b of 4Mb
> capacity estimate: 2440Kbit
> Bulk Best Effort Video Voice
> thresh 152496bit 2440Kbit 1220Kbit 610Kbit
> target 119.1ms 7.4ms 14.9ms 29.8ms
> interval 238.3ms 54.9ms 62.4ms 77.3ms
> pk_delay 0us 101.5ms 8.2ms 7us
> av_delay 0us 49.7ms 2.6ms 0us
> sp_delay 0us 6.8ms 245us 0us
> pkts 0 22757 11178 10
> bytes 0 3029896 951692 1096
> way_inds 0 0 0 0
> way_miss 0 61 3 3
> way_cols 0 0 0 0
> drops 0 130 0 0
> marks 0 0 0 0
> ack_drop 0 0 0 0
> sp_flows 0 1 1 0
> bk_flows 0 1 0 0
> un_flows 0 0 0 0
> max_len 0 11700 120 179
>
> qdisc noqueue 0: dev wlan1 root refcnt 2
> Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc noqueue 0: dev wlan0 root refcnt 2
> Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> new_flows_len 0 old_flows_len 0
>
> Best regards,
> Mark
> >
> > here is my tc -s qdisc output
> >
> > root at OpenWrt-Mi3G:~# tc -s qdisc
> > qdisc noqueue 0: dev lo root refcnt 2
> > Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> > Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0)
> > maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0
> > new_flows_len 0 old_flows_len 0
> > qdisc noqueue 0: dev br-lan root refcnt 2
> > Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> > Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0)
> > backlog 0b 0p requeues 0
> > memory used: 36684b of 4Mb
> > capacity estimate: 30Mbit
> > Bulk Best Effort Video Voice
> > thresh 1875Kbit 30Mbit 15Mbit 7500Kbit
> > target 9.7ms 2.5ms 2.5ms 2.5ms
> > interval 57.2ms 50.0ms 50.0ms 50.0ms
> > pk_delay 0us 5.6ms 855us 15us
> > av_delay 0us 436us 28us 10us
> > sp_delay 0us 27us 8us 6us
> > pkts 0 33679 457 1346
> > bytes 0 19869595 80517 69940
> > way_inds 0 249 0 0
> > way_miss 0 3099 19 7
> > way_cols 0 0 0 0
> > drops 0 10 0 0
> > marks 0 0 0 0
> > ack_drop 0 0 0 0
> > sp_flows 0 1 1 0
> > bk_flows 0 1 0 0
> > un_flows 0 0 0 0
> > max_len 0 7064 2982 350
> >
> > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4 dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 hard_header_len 14
> > Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0)
> > backlog 0b 0p requeues 0
> > memory used: 94752b of 4Mb
> > capacity estimate: 2400Kbit
> > Bulk Best Effort Video Voice
> > thresh 150Kbit 2400Kbit 1200Kbit 600Kbit
> > target 121.1ms 7.6ms 15.1ms 30.3ms
> > interval 242.2ms 55.1ms 62.6ms 77.8ms
> > pk_delay 0us 29.5ms 19us 4.4ms
> > av_delay 0us 3.8ms 1us 2.3ms
> > sp_delay 0us 50us 1us 9us
> > pkts 0 34979 31 1099
> > bytes 0 23342354 2930 162457
> > way_inds 0 177 0 0
> > way_miss 0 2003 27 16
> > way_cols 0 0 0 0
> > drops 0 205 0 0
> > marks 0 0 0 0
> > ack_drop 0 0 0 0
> > sp_flows 0 1 0 0
> > bk_flows 0 1 0 0
> > un_flows 0 0 0 0
> > max_len 0 8342 98 598
> >
> > qdisc noqueue 0: dev wlan1 root refcnt 2
> > Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc noqueue 0: dev wlan0 root refcnt 2
> > Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
> > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
> > Sent 304 bytes 4 pkts (dropped 0, overlimits 0)
> > maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
> > new_flows_len 0 old_flows_len 0
> >
> > Thanks,
> > Mark
> > _______________________________________________
> > Cake mailing list
> > Cake at lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
More information about the Cake
mailing list