Thanks for the comprehensive and quick reply :) below are some further answers On 17 December 2017 at 16:23, Sebastian Moeller wrote: > Hi Mark, > > > On Dec 17, 2017, at 11:45, Mark Captur wrote: > > > > My setup is as follows > > > > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master eth0.2 > (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to 192.168.1.x > > > > Here is my current SQM config > > config queue 'eth1' > > option debug_logging '0' > > option verbosity '5' > > option qdisc 'cake' > > option qdisc_advanced '1' > > option ingress_ecn 'ECN' > > option egress_ecn 'NOECN' > > option qdisc_really_really_advanced '1' > > option script 'layer_cake.qos' > > option interface 'eth0.2' > > option enabled '1' > > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost > diffserv4' > > option upload '2400' > > option linklayer 'ethernet' > > option overhead '8' > > option squash_dscp '1' > > option squash_ingress '1' > > option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost' > > option download '0' > > > > config queue > > option debug_logging '0' > > option verbosity '5' > > option download '0' > > option qdisc 'cake' > > option script 'layer_cake.qos' > > option qdisc_advanced '1' > > option squash_dscp '0' > > option squash_ingress '0' > > option ingress_ecn 'ECN' > > option qdisc_really_really_advanced '1' > > option egress_ecn 'ECN' > > option interface 'eth0.1' > > option enabled '1' > > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost > diffserv4' > > option upload '30000' > > option linklayer 'ethernet' > > option overhead '8' > > > > Is the overhead correct? should i use the bridged-ptm keyword (or should > i use pppoe-ptm). > > The overhead certainly seems confusing. Personally, I dislike the > overhead related compound keywords like *-ptm and would recommend the > following: > 1) remove the bridged-ptm from the eqdisc/iqdisc fields > Done > 2) add "mpu 64" to the eqdisc/iqdisc fields > Done > 3) set overhad to 8+18+4 = 30 bytes (you will need to account for > everything added on the bottleneck, so your packets will be MTU 1492, to > leave room for the PPPoE header that the modem adds). > Dne > 4) DO not set the ptm keyword at all, instead make sure to set the shaper > bandwidth to <= sync bandwidth * 64/65 = sync bandwidth * 0.984615384615 > (to account for ptm's 64/65 encoding _without_ incurring needless > operations per packet). > Done 0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 for downstream as other wise ping go very high when line saturated root@OpenWrt-Mi3G:~# cat /etc/config/sqm config queue 'eth1' option debug_logging '0' option verbosity '5' option qdisc 'cake' option qdisc_advanced '1' option ingress_ecn 'ECN' option egress_ecn 'NOECN' option qdisc_really_really_advanced '1' option script 'layer_cake.qos' option interface 'eth0.2' option enabled '1' option linklayer 'ethernet' option squash_dscp '1' option squash_ingress '1' option download '0' option upload '2440' option overhead '30' option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost' option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost diffserv4' config queue option debug_logging '0' option verbosity '5' option download '0' option qdisc 'cake' option script 'layer_cake.qos' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option qdisc_really_really_advanced '1' option egress_ecn 'ECN' option interface 'eth0.1' option enabled '1' option linklayer 'ethernet' option overhead '30' option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost diffserv4' option upload '30500' 4) tell us about your ISP and plan ;) ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i sync at 32/2.48 > > > > > I tried using ATM_overhead_detector and get the following output > > Well, yes sorry, this will not work at all for links using PTM; in > theory VDSL links could be using ATM and still be in compliance with the > ITU specs (at lest that is my reading) but fortunately nobody seems to be > doing that. (Also more unfortunately no ISP sseems to be using PTM encoding > on ADSL links, but I digress). > > > > > 869001 lines parsed... > > Found 144921 ping packets in /home/mark/src/ATM_overhead_ > detector/ping_sweep__20171217_075202.txt > > Elapsed time is 972.739 seconds. > > Minimum size of ping payload used: 16 bytes. > > warning: division by zero > > warning: called from > > ATM_overhead_detector at line 201 column 15 > > warning: legend: ignoring extra labels > > Unknown or ambiguous terminal name 'wxt' > > Unknown or ambiguous terminal name 'wxt' > > Saved figure (5) to: /home/mark/src/ATM_overhead_ > detector/ping_sweep__20171217_075202_data.png > > lower bound estimate for one ATM cell RTT based of specified up and > downlink is 0.052419 ms. > > estimate for one ATM cell RTT based on linear fit of the ping sweep data > is 0.052419 ms. > > Starting brute-force search for optimal stair fit, might take a while... > > Unknown or ambiguous terminal name 'wxt' > > Unknown or ambiguous terminal name 'wxt' > > Best staircase fit cumulative difference is: 2.6952 > > Best linear fit cumulative difference is: 2.7314 > > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 > linear fit 2.7314 > > Yes, that difference is simply too small to be meaningful. I am > currently collecting reference values for these for different technologies > and it seems on real ATM/AAL5 links linear residuals are >= 2 * stair > residuals, but curently the code simply reports the smaller value as more > likely... > > > > remaining ATM cell length after ICMP header is 5 bytes. > > ICMP RTT of a single ATM cell is 0.05871 ms. > > > > Estimated overhead preceding the IP header: 48 bytes > > Saved figure (6) to: /home/mark/src/ATM_overhead_detector/ping_sweep__ > 20171217_075202_results.png > > > > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ > > 48 bytes overhead indicate > > Overhead of 0 bytes is not possible so assume 1 full packet (48 bytes) > overhead... > > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag terminated at > modem > > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD [8] > (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), ATM > AAL5 SAR (8) : Total 36 > > > > Add the following to both the egress root qdisc: > > A) Assuming the router connects over ethernet to the DSL-modem: > > stab mtu 2048 tsize 128 overhead 48 linklayer atm > > > > Add the following to both the ingress root qdisc: > > > > A) Assuming the router connects over ethernet to the DSL-modem: > > stab mtu 2048 tsize 128 overhead 48 linklayer atm > > > > Elapsed time is 1042.06 seconds. > > Done... > > ans = [](0x0) > > > > > > Could my VDSL2 be using ATM? so i have the wrong overhead set? > > Yes in theory it could, but I am almost 100% certain it is not > doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I > also believe you have the wrong overhead specified... (if you insist on the > keywords try pppoe-ptm). Now, please keep in mind that you properly need to > "model" the encapsulation on the bottleneck link, if your ISP uses a > traffic shaper on the otherside of your link you need to actually model > that shapers settings. As far as I can see, ISPs will model the per packet > overhead roughly correctly, but will reduce the bandwidth from the sync > values reported by the modem, this requires a few iterations of trial and > error to get right. Also some ISPs also use VLAN tags on the bottleneck > link... > > Hope this helps... > > Best Regards > Sebastian > > Here is tc output > root@OpenWrt-Mi3G:~# tc -s qdisc qdisc noqueue 0: dev lo root refcnt 2 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0) maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0 new_flows_len 0 old_flows_len 0 qdisc noqueue 0: dev br-lan root refcnt 2 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit diffserv4 dual-dsthost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64 Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues 0) backlog 0b 0p requeues 0 memory used: 14112b of 4Mb capacity estimate: 31500Kbit Bulk Best Effort Video Voice thresh 1968Kbit 31500Kbit 15750Kbit 7875Kbit target 9.2ms 2.5ms 2.5ms 2.5ms interval 56.7ms 50.0ms 50.0ms 50.0ms pk_delay 7us 4.3ms 3.5ms 3.2ms av_delay 0us 183us 173us 271us sp_delay 0us 7us 8us 7us pkts 4 408 20384 38841 bytes 717 64851 30954992 58820110 way_inds 0 0 0 0 way_miss 4 64 4 11 way_cols 0 0 0 0 drops 0 0 5 0 marks 0 0 0 0 ack_drop 0 0 0 0 sp_flows 0 0 0 0 bk_flows 0 0 1 0 un_flows 0 0 0 0 max_len 189 1536 2988 2988 qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4 dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu 64 Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues 0) backlog 10860b 9p requeues 0 memory used: 116928b of 4Mb capacity estimate: 2440Kbit Bulk Best Effort Video Voice thresh 152496bit 2440Kbit 1220Kbit 610Kbit target 119.1ms 7.4ms 14.9ms 29.8ms interval 238.3ms 54.9ms 62.4ms 77.3ms pk_delay 0us 101.5ms 8.2ms 7us av_delay 0us 49.7ms 2.6ms 0us sp_delay 0us 6.8ms 245us 0us pkts 0 22757 11178 10 bytes 0 3029896 951692 1096 way_inds 0 0 0 0 way_miss 0 61 3 3 way_cols 0 0 0 0 drops 0 130 0 0 marks 0 0 0 0 ack_drop 0 0 0 0 sp_flows 0 1 1 0 bk_flows 0 1 0 0 un_flows 0 0 0 0 max_len 0 11700 120 179 qdisc noqueue 0: dev wlan1 root refcnt 2 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc noqueue 0: dev wlan0 root refcnt 2 Sent 0 bytes 0 pkts (dropped 0, overlimits 0) qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn Sent 304 bytes 4 pkts (dropped 0, overlimits 0) maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 new_flows_len 0 old_flows_len 0 Best regards, Mark > > > > here is my tc -s qdisc output > > > > root@OpenWrt-Mi3G:~# tc -s qdisc > > qdisc noqueue 0: dev lo root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum > 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > > Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0) > > maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0 > > new_flows_len 0 old_flows_len 0 > > qdisc noqueue 0: dev br-lan root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 > dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 > hard_header_len 14 > > Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues 0) > > backlog 0b 0p requeues 0 > > memory used: 36684b of 4Mb > > capacity estimate: 30Mbit > > Bulk Best Effort Video Voice > > thresh 1875Kbit 30Mbit 15Mbit 7500Kbit > > target 9.7ms 2.5ms 2.5ms 2.5ms > > interval 57.2ms 50.0ms 50.0ms 50.0ms > > pk_delay 0us 5.6ms 855us 15us > > av_delay 0us 436us 28us 10us > > sp_delay 0us 27us 8us 6us > > pkts 0 33679 457 1346 > > bytes 0 19869595 80517 69940 > > way_inds 0 249 0 0 > > way_miss 0 3099 19 7 > > way_cols 0 0 0 0 > > drops 0 10 0 0 > > marks 0 0 0 0 > > ack_drop 0 0 0 0 > > sp_flows 0 1 1 0 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 0 7064 2982 350 > > > > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit diffserv4 > dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead 22 > hard_header_len 14 > > Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 requeues 0) > > backlog 0b 0p requeues 0 > > memory used: 94752b of 4Mb > > capacity estimate: 2400Kbit > > Bulk Best Effort Video Voice > > thresh 150Kbit 2400Kbit 1200Kbit 600Kbit > > target 121.1ms 7.6ms 15.1ms 30.3ms > > interval 242.2ms 55.1ms 62.6ms 77.8ms > > pk_delay 0us 29.5ms 19us 4.4ms > > av_delay 0us 3.8ms 1us 2.3ms > > sp_delay 0us 50us 1us 9us > > pkts 0 34979 31 1099 > > bytes 0 23342354 2930 162457 > > way_inds 0 177 0 0 > > way_miss 0 2003 27 16 > > way_cols 0 0 0 0 > > drops 0 205 0 0 > > marks 0 0 0 0 > > ack_drop 0 0 0 0 > > sp_flows 0 1 0 0 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 0 8342 98 598 > > > > qdisc noqueue 0: dev wlan1 root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc noqueue 0: dev wlan0 root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum > 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > > Sent 304 bytes 4 pkts (dropped 0, overlimits 0) > > maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > > new_flows_len 0 old_flows_len 0 > > > > Thanks, > > Mark > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > >