From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 5719321F1AF for ; Thu, 8 Aug 2013 02:41:14 -0700 (PDT) Received: from due.zmbp.uni-tuebingen.de ([134.2.88.165]) by mail.gmx.com (mrgmx102) with ESMTPSA (Nemesis) id 0Mam2Q-1VQYIO3xd4-00KTpm for ; Thu, 08 Aug 2013 11:41:11 +0200 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) From: Sebastian Moeller In-Reply-To: <32269B74-55A6-4CFE-8372-9935325BA774@imap.cc> Date: Thu, 8 Aug 2013 11:41:11 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <46820144-E1DD-4858-A2B6-38F95D2894DB@imap.cc> <1AEAC257-ACFB-463E-B5CF-C7D086EE5B03@imap.cc> <32269B74-55A6-4CFE-8372-9935325BA774@imap.cc> To: Fred Stratton X-Mailer: Apple Mail (2.1508) X-Provags-ID: V03:K0:9oSNhGr7r3SDJF85ObFhfYP50TDJf1/c1kz7C6MJmVSPq9uOkUb /FJwzhyat12qmFkDm44LjnQk4nPKfZb3iUt/dv76BpAh1NmxbcA4I/73KuxuCU5YCcpb+x3 JMtFeUC/Vv3E184rqVCEATfudFFBdEqcAPoZZz4XcgGc3SnTo2rw3zJNywffntDlY95YpKB AXRxMLCUVaqATwKuZ7XtA== Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] cerowrt-3.10.2-1 dev release + owamp X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Aug 2013 09:41:14 -0000 Hi Fred, On Aug 8, 2013, at 01:21 , Fred Stratton wrote: >=20 > On 7 Aug 2013, at 14:38, Sebastian Moeller wrote: >=20 >> Hi Fred, >>=20 >> this got a bit longish so I took the liberty to reduce the quoted = text a bit >>=20 >>=20 >> On Aug 5, 2013, at 12:47 , Fred Stratton = wrote: >>=20 >> [snipp] >>=20 >>>>>> You are using 2 routers in series. I have disabled all routing = functions on the 2wire. It is transparent to the network. >>>>>=20 >>>>> Which is exactly the situation I faced with the cable modem = before; my cerowrt-router was provisioned with an IP address through the = bridged cable-modem via DHCP, but I still could access the modem's = 192.168.100.1 with out any configuration required. I know there is some = openwork information = (http://wiki.openwrt.org/doc/howto/access.modem.through.nat) that makes = it look like one needs to do more involved fiddling with the firewall, = but that turned out not to be required with cerowrt. I do not know how = that works if one runs a pppoe client on cerowrt though and I left = cerowrt's ip address assignment in place. (My hunch is that since = cerowrt leaves the typical 192.168.N.N ranges alone the whole issue gets = reduced to a simple routing issue=85 and since Dave takes care that cero = works well as secondary (test) router in a typical home situation, I = guess routing 192.168.N.N is well with in cerowrt's scope) >>>>> But, I guess you tried that already and it still does not work. = Would be interesting to learn why=85 >>>>=20 >>>> The difference is that you have the ISP gateway as a primary device = issuing a DHCP address to the cerowrt secondary router. The 2 devices = are then obviously on the same ipv4 subnet. >>>>=20 >>>> I use the 2700 transparently. DHCP is turned off. If I turn it on, = I have to use the device in DMZ mode with its firewall on, which I do = not want to do. >>=20 >> Sorry, to keep harping on this, but this is pretty close to what = I did with the cable modem. As I said I had it working with a similar = setup as you have, cerowrt was assigned a public IP (75.142.58.156) = address by the cable-ISPs dhcp server while the modems configuration = interface was running on the "private" 192.168.100.1. So the modem and = cero were decidedly not on the same IP subnet, but still I could connect = to it without needing to change anything. Initially, before I found out = that it works out of the box I had defined an alias IP address on the = wan interface of (WAN2CABLEMODEM ipv4-address: 192.168.100.2; = ipv4-netmask: 255.255.255.0). But it turned out that this was not = necessary as of cerowrt 3.3.8-17 I did not need to do this any more, = accessing the cable modem just worked by directing a browser to = 192.168.100.1. >>=20 >> So, have you tried to access the modem recently by simply = directing a browser to its address? And have you tried the same after = just configuring an alias as hinted above? If so what was the result? >>=20 >=20 > I configured an alias using uci at your prompting. It works. I can now = access the 2700. Excellent, that is solved then. On to the next issues... >=20 >=20 >>>>=20 >>>> Initially, I used the 2700 with the tomatoUSB router attached to = that, and then a router running openWRT. This setup allowed access to = the 2700, through a masquerade in tomatoUSB.=20 >>>>=20 >>>> Although ipv6 addresses were propagated throughout the network by = Barrier Breaker, ipv6 did not work, probably because of the way radvd = works in tomato. >>>>=20 >>>> I have never used the cerowrt as a secondary device because of = this. >>>>>=20 >>>>=20 >>=20 >> [snipp] >>=20 >>>> I do not want to use cable, which is expensive. The DOCSIS box - a = custom Netgear device - has a poor reputation. >>>>=20 >>>> I do not want to use fibre, again, because when it comes here, it = will be supplied by BT/, and is traffic shaped and capped. The BT web = site has 35 pages of price increases for this year. >>>>=20 >>>> I will continue with ADSL2+ >>=20 >> I fully agree that getting ADSL(2+) links debated and offering = low latency internet access is worthwhile as a considerable number of = people simply have no other choice available. And this is why your case = is so interesting! If you can improve the "interactivity" in your home = and document the required steps somewhere others will have an easier = time. >>=20 >> Yes. Hopefully others using ADSL will also participate. Okay, let's assume for a minute that the ATM link layer = adaptation mechanism might be busted. Since each package (and its = overhead) always consume an integer number of ATM cells and no ATM cells = are shared between packets, the worst case ATM overhead would be close = to 50% (a small packet with 48 + 1 byte length will take 2 full 48 byte = ATM cells, just looking at the payload here). For bigger packets the ATM = quantization overhead will be a smaller percentage. So if you shape down = to 50% of link rates (and do your testing not only with very small = packets) this should make the shaping robust even with a busted ATM link = layer adaptation mechanism. Assuming sane numbers of "channels" reserved = for FEC this should also nicely cover the useable link rate reduction = caused by these channels. So I would like to ask you to try shaping up and down rates to = 50% of the link rates and then see how the link behaves in face of = concurrent streaming and interactive traffic. In case you go that route, = please let us know whether this improves anything or not. Oh, and I = guess it would be a good idea to do these tests not over any tunnel by = by "plain" IP4 or IP6, depending on your actual connection to your ISP. One other thing to try would be to see whether latencies change = over the course of the day or not (if the uplink bottleneck should be = the uplink of the DSLAM itself shaping on your end will not help much) Best Sebastian >>=20 >> best >> Sebastian >>=20 >>=20 >>>>>=20 >>>>> Best >>>>> Sebastian >>>>>=20 >>>>>=20 >>>>>>>=20 >>>>>>> best regards >>>>>>> Sebastian >>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> On 3 Aug 2013, at 10:38, Sebastian Moeller = wrote: >>>>>>>>=20 >>>>>>>>> Hi Fred, >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> On Aug 1, 2013, at 00:35 , Fred Stratton = wrote: >>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On 31 Jul 2013, at 23:14, Sebastian Moeller = wrote: >>>>>>>>>>=20 >>>>>>>>>>> Hi Fred, >>>>>>>>>>>=20 >>>>>>>>>>> thanks a lot. >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> On Jul 31, 2013, at 23:37 , Fred Stratton = wrote: >>>>>>>>>>>=20 >>>>>>>>>>>> tc -s -d class show dev ge00 >>>>>>>>>>>>=20 >>>>>>>>>>>> class htb 1:10 parent 1:1 leaf 110: prio 0 quantum 1500 = rate 700000bit ceil 700000bit burst 1599b/1 mpu 0b overhead 0b cburst = 1599b/1 mpu 0b overhead 0b level 0=20 >>>>>>>>>>>> Sent 15809014 bytes 115190 pkt (dropped 4733, overlimits 0 = requeues 0)=20 >>>>>>>>>>>> rate 3616bit 3pps backlog 0b 0p requeues 0=20 >>>>>>>>>>>> lended: 115190 borrowed: 0 giants: 0 >>>>>>>>>>>> tokens: 263560 ctokens: 263560 >>>>>>>>>>>>=20 >>>>>>>>>>>> class htb 1:1 root rate 700000bit ceil 700000bit burst = 1599b/1 mpu 0b overhead 0b cburst 1599b/1 mpu 0b overhead 0b level 7=20 >>>>>>>>>>>> Sent 15809014 bytes 115190 pkt (dropped 0, overlimits 0 = requeues 0)=20 >>>>>>>>>>>> rate 3616bit 3pps backlog 0b 0p requeues 0=20 >>>>>>>>>>>> lended: 0 borrowed: 0 giants: 0 >>>>>>>>>>>> tokens: 263560 ctokens: 263560 >>>>>>>>>>>>=20 >>>>>>>>>>>> class fq_codel 110:1b8 parent 110:=20 >>>>>>>>>>>> (dropped 0, overlimits 0 requeues 0)=20 >>>>>>>>>>>> backlog 0b 0p requeues 0=20 >>>>>>>>>>>> deficit 84 count 0 lastcount 0 delay 10us >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> tc -s -d class show dev ifb0 >>>>>>>>>>>> class htb 1:10 parent 1:1 leaf 110: prio 0 quantum 1500 = rate 7000Kbit ceil 7000Kbit burst 1598b/1 mpu 0b overhead 0b cburst = 1598b/1 mpu 0b overhead 0b level 0=20 >>>>>>>>>>>> Sent 192992612 bytes 168503 pkt (dropped 0, overlimits 0 = requeues 0)=20 >>>>>>>>>>>> rate 17096bit 4pps backlog 0b 0p requeues 0=20 >>>>>>>>>>>> lended: 168503 borrowed: 0 giants: 0 >>>>>>>>>>>> tokens: 27454 ctokens: 27454 >>>>>>>>>>>>=20 >>>>>>>>>>>> class htb 1:1 root rate 7000Kbit ceil 7000Kbit burst = 1598b/1 mpu 0b overhead 0b cburst 1598b/1 mpu 0b overhead 0b level 7=20 >>>>>>>>>>>> Sent 192992612 bytes 168503 pkt (dropped 0, overlimits 0 = requeues 0)=20 >>>>>>>>>>>> rate 17096bit 4pps backlog 0b 0p requeues 0=20 >>>>>>>>>>>> lended: 0 borrowed: 0 giants: 0 >>>>>>>>>>>> tokens: 27454 ctokens: 27454 >>>>>>>>>>>>=20 >>>>>>>>>>>> class fq_codel 110:cc parent 110:=20 >>>>>>>>>>>> (dropped 10, overlimits 0 requeues 0)=20 >>>>>>>>>>>> backlog 0b 0p requeues 0=20 >>>>>>>>>>>> deficit -198 count 1 lastcount 1 ldelay 2.3ms >>>>>>>>>>>> class fq_codel 110:1d9 parent 110:=20 >>>>>>>>>>>> (dropped 0, overlimits 0 requeues 0)=20 >>>>>>>>>>>> backlog 0b 0p requeues 0=20 >>>>>>>>>>>> deficit 226 count 0 lastcount 0 ldelay 2us >>>>>>>>>>>> class fq_codel 110:1de parent 110:=20 >>>>>>>>>>>> (dropped 0, overlimits 0 requeues 0)=20 >>>>>>>>>>>> backlog 0b 0p requeues 0=20 >>>>>>>>>>>> deficit 238 count 0 lastcount 0 ldelay 10us >>>>>>>>>>>> class fq_codel 110:345 parent 110:=20 >>>>>>>>>>>> (dropped 0, overlimits 0 requeues 0)=20 >>>>>>>>>>>> backlog 0b 0p requeues 0=20 >>>>>>>>>>>> deficit 226 count 0 lastcount 0 delay 9us >>>>>>>>>>>>=20 >>>>>>>>>>>> I changed the hard coded values in = /usr/lib/aqm/functions.sh to arbitrary values, rebooted and obtained the = same results. Both reflect the 7000kbit/s down and 700kbit/s up I = entered in the window. >>>>>>>>>>>=20 >>>>>>>>>>> What is the line rate as read out from the del modem or = specified in your contract? >>>>>>>>>>=20 >>>>>>>>>> Speedtest.net shows the rate as circa 8.7 megabits/s down, 1 = megabit/s up. Line has radio frequency interference from unidentified = sources.. >>>>>>>>>=20 >>>>>>>>> So it looks like specify a generous reserve for the = shaper. Can you log into your modem and get the current line rates? The = rf interference, is it constant (if you can get nice SNR per sub carrier = or even ust bit loading per frequency plots) that is does it only affect = the same frequencies or does it change? (I ask, because temporary = interference might reduce the effective line rate, potentially moving = the buffer back into the del modem) >>>>>>>>>=20 >>>>>>>>>> Target snr upped to 12 deciBel. Line can sustain 10 = megabits/s with repeated loss of sync.at lower snr. Contract is for 'up = to 20megabits/s'. =20 >>>>>>>>>=20 >>>>>>>>> So ADSL2+ as you even mentioned it before. >>>>>>>>>=20 >>>>>>>>>> 850 metres from exchange. Line length circa 1.2km. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>> I ticked the adsl box. Altering the value in functions.sh = and unticking the box, with reboot, produced the same outcome. >>>>>>>>>>>=20 >>>>>>>>>>> This nicely shows I screwed up my testing (and or forgot = to reboot between changes). Or I did try too high a data rate (initially = 97% of the raw link rate) >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> traceroute google.com >>>>>>>>>>>> traceroute: Warning: google.com has multiple addresses; = using 173.194.41.128 >>>>>>>>>>>> traceroute to google.com (173.194.41.128), 64 hops max, 52 = byte packets >>>>>>>>>>>> 1 172.30.42.1 (172.30.42.1) 0.631 ms 0.323 ms 0.249 ms >>>>>>>>>>>> 2 * * * >>>>>>>>>>>> 3 10.1.3.234 (10.1.3.234) 22.596 ms 21.241 ms 22.392 ms >>>>>>>>>>>> 4 * 10.1.3.214 (10.1.3.214) 27.018 ms 26.703 ms >>>>>>>>>>>> 5 10.1.4.249 (10.1.4.249) 29.682 ms 28.923 ms 27.479 ms >>>>>>>>>>>> 6 * * * >>>>>>>>>>>> 7 * 209.85.252.186 (209.85.252.186) 30.379 ms * >>>>>>>>>>>> 8 72.14.238.55 (72.14.238.55) 25.745 ms 25.345 ms = 25.594 ms >>>>>>>>>>>> 9 lhr08s03-in-f0.1e100.net (173.194.41.128) 27.566 ms = 27.390 ms 27.663 ms >>>>>>>>>>>>=20 >>>>>>>>>>>> mtr shows packet losses at hops 2-5=20 >>>>>>>>>>>> 10.1.3.* are Internet Watch Foundation. >>>>>>>>>>>=20 >>>>>>>>>>> This looks pretty reasonable for an adsl link (could be = way worse with higher interleaving) >>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> Netalyzr was used. I appreciate it is an imperfect metric. >>>>>>>>>>=20 >>>>>>>>>> OK. Like the ping train idea. Cannot get netperf 2.6.0 to = build on Ubuntu 12.04 >>>>>>>>>=20 >>>>>>>>> So I typically run a 1000 count ping against the nearest = host that is on the other side of the DSL link that also gives = consistent ping RTTs without load. Then I start my test loads like = saturating the upload with a long runnig TCP transfer and opening 99 = media heavy tabs in a browser (I really should try the chrome benchmark = that Dave is using). And the I simply look through the ping statistic = results, typically I look at the maximum, and at the standard deviation = to get a handle on how tight the shaper held latency under control. (If = I should get netperf-wrapper to work under Macosx I will try to use that = for testing, but it does not even install, and if I get past that hurdle = I will have to adjust for the differences between Gnu ping and BSD = ping). >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Best Regards >>>>>>>>> Sebastian >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> Well, I ran into this issue before. In short netalyzr's = worst case delay numbers do not seem to reflect how an fq_codelled = connection feels. =20 >>>>>>>>>>> Netalyzr uses an unresponsive UDP probe to force the = bottleneck router's buffers to fill up; with unresponsiveness being a = property no sane flow over the intent should exhibit. Codel/fq_codel is = tailored for responsive flows and will only gradually increase its drop = frequency so responsive TCP flows will be controlled gently and keep = link utilization high. Given enough time codel will also rein in an = unresponsive flows. But netalyzr's probe duration is too short for that = to be happening during netalyzr's runtime. >>>>>>>>>>> Fq_codel in my experience does a decent job at keeping = interactivity high even with competing traffic like netalyzr (so turn a = ping train against say 10.1.3.234 while netalyzr runs or try = netperf-wrapper in addition).=20 >>>>>>>>>>> So netalyzr really probes the worst case buffer depth = against basically a "denial of service" type of load; I am not fully = sure what the expectancy on the disc here should be. >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> best >>>>>>>>>>> Sebastian >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> On 31 Jul 2013, at 21:38, Sebastian Moeller = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>>> tc -s -d class show dev ge00 >>>>>>>>>>>>=20 >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Cerowrt-devel mailing list >>>>>>>>>>>> Cerowrt-devel@lists.bufferbloat.net >>>>>>>>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> _______________________________________________ >>>>>>>>>> Cerowrt-devel mailing list >>>>>>>>>> Cerowrt-devel@lists.bufferbloat.net >>>>>>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>>>>>>=20 >>>>>>>>=20 >>>>>>>> _______________________________________________ >>>>>>>> Cerowrt-devel mailing list >>>>>>>> Cerowrt-devel@lists.bufferbloat.net >>>>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>>>>=20 >>>>>>=20 >>>>>> _______________________________________________ >>>>>> Cerowrt-devel mailing list >>>>>> Cerowrt-devel@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>>>>=20 >>>>=20 >>>=20 >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>=20 >=20 > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel