From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3E1CC3BA8E for ; Sun, 17 Dec 2017 17:30:53 -0500 (EST) Received: from hms-beagle2.lan ([79.210.203.204]) by mail.gmx.com (mrgmx002 [212.227.17.190]) with ESMTPSA (Nemesis) id 0Lk7T8-1f1QBN2EQM-00c67W; Sun, 17 Dec 2017 23:30:51 +0100 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) From: Sebastian Moeller In-Reply-To: Date: Sun, 17 Dec 2017 23:30:50 +0100 Cc: cake@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: References: <73C84EA7-2ADD-4914-BBDE-92E8408C106F@gmx.de> To: Mark Captur X-Mailer: Apple Mail (2.3273) X-Provags-ID: V03:K0:uJI4JGIfRzvHhXO94V1tEpSt1qPM4KYbNtgjoKOKDh1tjRoRRBy nAjFZQlfy2wzhtpOeqWd5aMJmHPfuloUESwvBTxun7bMMsveBPvIjCPTz/p5IjDyZuIIUBq DkQSnWUCNOcLW7gjF58BrW1sMe0J3M+upMwaVFYPCtHdHbvgzEBfCEWWFEOfawPbMfF20Px yXSIY9qyoKg4npTUvrwFw== X-UI-Out-Filterresults: notjunk:1;V01:K0:yTcLpguh/U8=:dosypUqE1RzE1wg3Hj234F raQ1cl/kjle97ONQ4Fe7C9RN+ZSgVo6RmHKcJIgXSxr3KAGnZVK2vnhBaqngrAFw0RTBXMcdK PjgcEiTC4xVwsxjxmQft07trQS6J+DudTgVhdgXJTSlvaM63NMCFLlJ5N5UfJNWmQ1gFzaRQC pEKWU65VvS57lyaalLEGlbjucYT3L5Cf4vcQmQAmcOfqjli0MSj8LC5fzDRxIRlhLs4u39jnA NaDOYjmi9PHV+SLH1g5mldY2f1s6CpZwBGYeAxgBSfSd7nyUf3JkbA4B19/m9VzWQqWD4/KNd S+4mDFyhjnGxXfvvTBcWx+TBwCL91YrbW4hpAEOjkaEaRVc6DkiAZFGHNrA4SF+tPAAXRz7jt Lm5eZfXCQyNzenKTs6D8/sXWv9J/OiwHAi4g79KoFMkFEo34HkhI1hl/BQwxadAtiTHybS64r /ajNQ16XIhYtbZHQkEKk5MSH0jcOOkcPIJ/hpz/k+omDL/LKRoiAk24SA15Jmrd20+qQcE2ih dsMvBsdy8oFvlX6IKoiET9G3LuSAMmuH9qzenf18Jdy7sjZSa5wH3sA0Pr3w1sIdXzJPlY1FE rZ0OAgJnB0Sorg7zCudmmD/SvsD5TFkinGthreHrVxrWmdFTjsew1qtBJ9Pif0AFGsbkLtVxh ThqEOlDNdSyKeT7iJvlWymql9z3Ts/S4tNQKVhF5gkcIwXvcEHaxAavEvoPtLyc2+NGw5aLP0 0o034e7C6Ob09VtFyW0XB0dRzJGN969fC0rZtpflvSyLNujc7YgNwkDj6hl1zA5/1VorJ7Myv Q5l3YT3gjebbbP2Gk/4TbI701sqIpkNybTY7dySaJZIejh39D4= Subject: Re: [Cake] overhead for double nat VDSL2 connection X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 22:30:53 -0000 HI Mark, > On Dec 17, 2017, at 17:00, Mark Captur wrote: >=20 > Thanks for the comprehensive and quick reply :) below are some further = answers Always happy to help. >=20 > On 17 December 2017 at 16:23, Sebastian Moeller = wrote: > Hi Mark, >=20 > > On Dec 17, 2017, at 11:45, Mark Captur = wrote: > > > > My setup is as follows > > > > vdsl2 modem doing pppoe itself and nat to 10.x.x.x -> lede master = eth0.2 (wan static ip in modem's DMZ) eth0.1 (lan) doing nat to = 192.168.1.x > > > > Here is my current SQM config > > config queue 'eth1' > > option debug_logging '0' > > option verbosity '5' > > option qdisc 'cake' > > option qdisc_advanced '1' > > option ingress_ecn 'ECN' > > option egress_ecn 'NOECN' > > option qdisc_really_really_advanced '1' > > option script 'layer_cake.qos' > > option interface 'eth0.2' > > option enabled '1' > > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-srchost = diffserv4' > > option upload '2400' > > option linklayer 'ethernet' > > option overhead '8' > > option squash_dscp '1' > > option squash_ingress '1' > > option iqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost' > > option download '0' > > > > config queue > > option debug_logging '0' > > option verbosity '5' > > option download '0' > > option qdisc 'cake' > > option script 'layer_cake.qos' > > option qdisc_advanced '1' > > option squash_dscp '0' > > option squash_ingress '0' > > option ingress_ecn 'ECN' > > option qdisc_really_really_advanced '1' > > option egress_ecn 'ECN' > > option interface 'eth0.1' > > option enabled '1' > > option eqdisc_opts 'nat rtt 50000 bridged-ptm dual-dsthost = diffserv4' > > option upload '30000' > > option linklayer 'ethernet' > > option overhead '8' > > > > Is the overhead correct? should i use the bridged-ptm keyword (or = should i use pppoe-ptm). >=20 > The overhead certainly seems confusing. Personally, I dislike = the overhead related compound keywords like *-ptm and would recommend = the following: > 1) remove the bridged-ptm from the eqdisc/iqdisc fields > Done=20 > 2) add "mpu 64" to the eqdisc/iqdisc fields > Done=20 > 3) set overhad to 8+18+4 =3D 30 bytes (you will need to account for = everything added on the bottleneck, so your packets will be MTU 1492, to = leave room for the PPPoE header that the modem adds). > Dne=20 > 4) DO not set the ptm keyword at all, instead make sure to set the = shaper bandwidth to <=3D sync bandwidth * 64/65 =3D sync bandwidth * = 0.984615384615 (to account for ptm's 64/65 encoding _without_ incurring = needless operations per packet). > Done 0.98 (2440 is .98 of 2480) for upstream but had to reduce to .95 = for downstream as other wise ping go very high when line saturated Sure, the 98.46% of sync is a theoretical upper limit and not a = really reachable for ingress (but you might want tto try the new = "ingress" keyword on eth0.1) > =20 >=20 > root@OpenWrt-Mi3G:~# cat /etc/config/sqm >=20 > config queue 'eth1' > option debug_logging '0' > option verbosity '5' > option qdisc 'cake' > option qdisc_advanced '1' > option ingress_ecn 'ECN' > option egress_ecn 'NOECN' > option qdisc_really_really_advanced '1' > option script 'layer_cake.qos' > option interface 'eth0.2' > option enabled '1' > option linklayer 'ethernet' > option squash_dscp '1' > option squash_ingress '1' > option download '0' > option upload '2440' > option overhead '30' > option iqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost' > option eqdisc_opts 'nat rtt 50000 mpu 64 dual-srchost = diffserv4' >=20 > config queue > option debug_logging '0' > option verbosity '5' > option download '0' > option qdisc 'cake' > option script 'layer_cake.qos' > option qdisc_advanced '1' > option squash_dscp '0' > option squash_ingress '0' > option ingress_ecn 'ECN' > option qdisc_really_really_advanced '1' > option egress_ecn 'ECN' > option interface 'eth0.1' > option enabled '1' > option linklayer 'ethernet' > option overhead '30' > option eqdisc_opts 'nat rtt 50000 mpu 64 dual-dsthost = diffserv4' > option upload '30500' >=20 >=20 > 4) tell us about your ISP and plan ;) > ISP is Go Malta VDSL2 70/10, however i'm very far from cabinet so i = sync at 32/2.48=20 Thanks, I had a look, but i could not easily whether they use a = VLAN on their vdsl2 link. Best Regards Sebastian =09 >=20 >=20 >=20 > > I tried using ATM_overhead_detector and get the following output >=20 > Well, yes sorry, this will not work at all for links using = PTM; in theory VDSL links could be using ATM and still be in compliance = with the ITU specs (at lest that is my reading) but fortunately nobody = seems to be doing that. (Also more unfortunately no ISP sseems to be = using PTM encoding on ADSL links, but I digress). >=20 > > > > 869001 lines parsed... > > Found 144921 ping packets in = /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202.txt > > Elapsed time is 972.739 seconds. > > Minimum size of ping payload used: 16 bytes. > > warning: division by zero > > warning: called from > > ATM_overhead_detector at line 201 column 15 > > warning: legend: ignoring extra labels > > Unknown or ambiguous terminal name 'wxt' > > Unknown or ambiguous terminal name 'wxt' > > Saved figure (5) to: = /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_data.png > > lower bound estimate for one ATM cell RTT based of specified up and = downlink is 0.052419 ms. > > estimate for one ATM cell RTT based on linear fit of the ping sweep = data is 0.052419 ms. > > Starting brute-force search for optimal stair fit, might take a = while... > > Unknown or ambiguous terminal name 'wxt' > > Unknown or ambiguous terminal name 'wxt' > > Best staircase fit cumulative difference is: 2.6952 > > Best linear fit cumulative difference is: 2.7314 > > Quantized ATM carrier LIKELY (cummulative residual: stair fit 2.6952 = linear fit 2.7314 >=20 > Yes, that difference is simply too small to be meaningful. I = am currently collecting reference values for these for different = technologies and it seems on real ATM/AAL5 links linear residuals are >=3D= 2 * stair residuals, but curently the code simply reports the smaller = value as more likely... >=20 >=20 > > remaining ATM cell length after ICMP header is 5 bytes. > > ICMP RTT of a single ATM cell is 0.05871 ms. > > > > Estimated overhead preceding the IP header: 48 bytes > > Saved figure (6) to: = /home/mark/src/ATM_overhead_detector/ping_sweep__20171217_075202_results.p= ng > > > > According to http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ > > 48 bytes overhead indicate > > Overhead of 0 bytes is not possible so assume 1 full packet (48 = bytes) overhead... > > Connection: Bridged, LLC/SNAP+FCS RFC-1483/2684 + VLAN tag = terminated at modem > > Protocol (bytes): Ethernet Header (14), VLAN tag (4), Ethernet PAD = [8] (0), Ethernet Checksum (4), ATM LLC (3), ATM SNAP (5), ATM pad (2), = ATM AAL5 SAR (8) : Total 36 > > > > Add the following to both the egress root qdisc: > > A) Assuming the router connects over ethernet to the DSL-modem: > > stab mtu 2048 tsize 128 overhead 48 linklayer atm > > > > Add the following to both the ingress root qdisc: > > > > A) Assuming the router connects over ethernet to the DSL-modem: > > stab mtu 2048 tsize 128 overhead 48 linklayer atm > > > > Elapsed time is 1042.06 seconds. > > Done... > > ans =3D [](0x0) > > > > > > Could my VDSL2 be using ATM? so i have the wrong overhead set? >=20 > Yes in theory it could, but I am almost 100% certain it is not = doing that (AFAICT ISPs try to get rid of all ATM equipment ASAP). But I = also believe you have the wrong overhead specified... (if you insist on = the keywords try pppoe-ptm). Now, please keep in mind that you properly = need to "model" the encapsulation on the bottleneck link, if your ISP = uses a traffic shaper on the otherside of your link you need to actually = model that shapers settings. As far as I can see, ISPs will model the = per packet overhead roughly correctly, but will reduce the bandwidth = from the sync values reported by the modem, this requires a few = iterations of trial and error to get right. Also some ISPs also use VLAN = tags on the bottleneck link... >=20 > Hope this helps... >=20 > Best Regards > Sebastian >=20 > Here is tc output >=20 > root@OpenWrt-Mi3G:~# tc -s qdisc > qdisc noqueue 0: dev lo root refcnt 2 > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 = quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > Sent 13974072314 bytes 14402910 pkts (dropped 0, overlimits 0) > maxpacket 17032 drop_overlimit 0 new_flow_count 132368 ecn_mark 0 > new_flows_len 0 old_flows_len 0 > qdisc noqueue 0: dev br-lan root refcnt 2 > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc cake 8069: dev eth0.1 root refcnt 2 bandwidth 31500Kbit = diffserv4 dual-dsthost nat rtt 50.0ms raw total_overhead 14 = hard_header_len 14 mpu 64 > Sent 89833020 bytes 59632 pkt (dropped 5, overlimits 44617 requeues = 0) > backlog 0b 0p requeues 0 > memory used: 14112b of 4Mb > capacity estimate: 31500Kbit > Bulk Best Effort Video Voice > thresh 1968Kbit 31500Kbit 15750Kbit 7875Kbit > target 9.2ms 2.5ms 2.5ms 2.5ms > interval 56.7ms 50.0ms 50.0ms 50.0ms > pk_delay 7us 4.3ms 3.5ms 3.2ms > av_delay 0us 183us 173us 271us > sp_delay 0us 7us 8us 7us > pkts 4 408 20384 38841 > bytes 717 64851 30954992 58820110 > way_inds 0 0 0 0 > way_miss 4 64 4 11 > way_cols 0 0 0 0 > drops 0 0 5 0 > marks 0 0 0 0 > ack_drop 0 0 0 0 > sp_flows 0 0 0 0 > bk_flows 0 0 1 0 > un_flows 0 0 0 0 > max_len 189 1536 2988 2988 >=20 > qdisc cake 8067: dev eth0.2 root refcnt 2 bandwidth 2440Kbit diffserv4 = dual-srchost nat rtt 50.0ms raw total_overhead 14 hard_header_len 14 mpu = 64 > Sent 3780967 bytes 33806 pkt (dropped 130, overlimits 12207 requeues = 0) > backlog 10860b 9p requeues 0 > memory used: 116928b of 4Mb > capacity estimate: 2440Kbit > Bulk Best Effort Video Voice > thresh 152496bit 2440Kbit 1220Kbit 610Kbit > target 119.1ms 7.4ms 14.9ms 29.8ms > interval 238.3ms 54.9ms 62.4ms 77.3ms > pk_delay 0us 101.5ms 8.2ms 7us > av_delay 0us 49.7ms 2.6ms 0us > sp_delay 0us 6.8ms 245us 0us > pkts 0 22757 11178 10 > bytes 0 3029896 951692 1096 > way_inds 0 0 0 0 > way_miss 0 61 3 3 > way_cols 0 0 0 0 > drops 0 130 0 0 > marks 0 0 0 0 > ack_drop 0 0 0 0 > sp_flows 0 1 1 0 > bk_flows 0 1 0 0 > un_flows 0 0 0 0 > max_len 0 11700 120 179 >=20 > qdisc noqueue 0: dev wlan1 root refcnt 2 > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc noqueue 0: dev wlan0 root refcnt 2 > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 = quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > Sent 304 bytes 4 pkts (dropped 0, overlimits 0) > maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > new_flows_len 0 old_flows_len 0 >=20 > Best regards, > Mark=20 > > > > here is my tc -s qdisc output > > > > root@OpenWrt-Mi3G:~# tc -s qdisc > > qdisc noqueue 0: dev lo root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 = quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > > Sent 3460986864 bytes 3651814 pkts (dropped 0, overlimits 0) > > maxpacket 17032 drop_overlimit 0 new_flow_count 29673 ecn_mark 0 > > new_flows_len 0 old_flows_len 0 > > qdisc noqueue 0: dev br-lan root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc cake 8055: dev eth0.1 root refcnt 2 bandwidth 30Mbit diffserv4 = dual-dsthost nat rtt 50.0ms ptm overhead 22 via-ethernet total_overhead = 22 hard_header_len 14 > > Sent 20012069 bytes 35472 pkt (dropped 10, overlimits 7587 requeues = 0) > > backlog 0b 0p requeues 0 > > memory used: 36684b of 4Mb > > capacity estimate: 30Mbit > > Bulk Best Effort Video Voice > > thresh 1875Kbit 30Mbit 15Mbit 7500Kbit > > target 9.7ms 2.5ms 2.5ms 2.5ms > > interval 57.2ms 50.0ms 50.0ms 50.0ms > > pk_delay 0us 5.6ms 855us 15us > > av_delay 0us 436us 28us 10us > > sp_delay 0us 27us 8us 6us > > pkts 0 33679 457 1346 > > bytes 0 19869595 80517 69940 > > way_inds 0 249 0 0 > > way_miss 0 3099 19 7 > > way_cols 0 0 0 0 > > drops 0 10 0 0 > > marks 0 0 0 0 > > ack_drop 0 0 0 0 > > sp_flows 0 1 1 0 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 0 7064 2982 350 > > > > qdisc cake 8053: dev eth0.2 root refcnt 2 bandwidth 2400Kbit = diffserv4 dual-srchost nat rtt 50.0ms ptm overhead 22 via-ethernet = total_overhead 22 hard_header_len 14 > > Sent 23218772 bytes 35904 pkt (dropped 205, overlimits 27550 = requeues 0) > > backlog 0b 0p requeues 0 > > memory used: 94752b of 4Mb > > capacity estimate: 2400Kbit > > Bulk Best Effort Video Voice > > thresh 150Kbit 2400Kbit 1200Kbit 600Kbit > > target 121.1ms 7.6ms 15.1ms 30.3ms > > interval 242.2ms 55.1ms 62.6ms 77.8ms > > pk_delay 0us 29.5ms 19us 4.4ms > > av_delay 0us 3.8ms 1us 2.3ms > > sp_delay 0us 50us 1us 9us > > pkts 0 34979 31 1099 > > bytes 0 23342354 2930 162457 > > way_inds 0 177 0 0 > > way_miss 0 2003 27 16 > > way_cols 0 0 0 0 > > drops 0 205 0 0 > > marks 0 0 0 0 > > ack_drop 0 0 0 0 > > sp_flows 0 1 0 0 > > bk_flows 0 1 0 0 > > un_flows 0 0 0 0 > > max_len 0 8342 98 598 > > > > qdisc noqueue 0: dev wlan1 root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc noqueue 0: dev wlan0 root refcnt 2 > > Sent 0 bytes 0 pkts (dropped 0, overlimits 0) > > qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 = quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn > > Sent 304 bytes 4 pkts (dropped 0, overlimits 0) > > maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > > new_flows_len 0 old_flows_len 0 > > > > Thanks, > > Mark > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake