[Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.

Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.

Test Conditions:

- Running tests from my MacBook Pro, 10.9.2. 
- Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
- DSL service from Fairpoint, nominally "7 Mbps down/768kbps up". 
- DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
- Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
- To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).

The Tests: 

I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.

For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
For the Speedtest.net tests, I used their web GUI in the obvious way.

For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings. 

Test Procedure:

I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.

The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.

With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.

Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)

There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.

I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.

Conclusion:

I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.

We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.

The data follows...

Rich

================================================================

RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/

6089/737:
	Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77

6200/750:
	Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77

6400/800:
	Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78

6600/830:
	Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77

7000/800:
	Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
	Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)

7000/830:
	Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
	Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)

7600/850: 
	Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84

SQM Disabled:
	Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78

================================================================

Speedtest.net: First values are the reported down/up rates in the Speedtest GUI

6089/737:
	5.00/0.58 
	Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45

6200/750:
	5.08/0.58
	Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44

6400/800:
	5.24/0.60
	Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43

6600/830:
	5.41/0.65
	Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44

7000/800:
	5.73/0.62
	Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44

7000/830:
	5.74/0.65 (5.71/0.62 second run)
	Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
	Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
 
7600/850: 
	6.20/0.67
	Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48

SQM Disabled:
	6.46/0.73
	Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

Nice work!

I have a problem in that I can't remember if target autotuning made it
into that release or not.

Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
above and paste? I still think
target is still too low on the egress side with the current calculation.

Secondly, now that you have a setting you like, trying pie, codel, and
ns2_codel also would be interesting.

efq_codel is currently uninteresting. Wasn't clear if you were using
nfq_codel or fq_codel throughout.


On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>
> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>
> Test Conditions:
>
> - Running tests from my MacBook Pro, 10.9.2.
> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>
> The Tests:
>
> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>
> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
> For the Speedtest.net tests, I used their web GUI in the obvious way.
>
> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>
> Test Procedure:
>
> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>
> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>
> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>
> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>
> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>
> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>
> Conclusion:
>
> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>
> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>
> The data follows...
>
> Rich
>
> ================================================================
>
> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>
> 6089/737:
>         Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>
> 6200/750:
>         Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>
> 6400/800:
>         Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>
> 6600/830:
>         Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>
> 7000/800:
>         Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>         Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>
> 7000/830:
>         Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>         Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>
> 7600/850:
>         Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>
> SQM Disabled:
>         Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>
> ================================================================
>
> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>
> 6089/737:
>         5.00/0.58
>         Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>
> 6200/750:
>         5.08/0.58
>         Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>
> 6400/800:
>         5.24/0.60
>         Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>
> 6600/830:
>         5.41/0.65
>         Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>
> 7000/800:
>         5.73/0.62
>         Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>
> 7000/830:
>         5.74/0.65 (5.71/0.62 second run)
>         Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>         Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>
> 7600/850:
>         6.20/0.67
>         Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>
> SQM Disabled:
>         6.46/0.73
>         Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Sebastian Moeller]

Hi Dave,


On Mar 2, 2014, at 22:41 , Dave Taht <dave.taht@gmail.com> wrote:

> Nice work!
> 
> I have a problem in that I can't remember if target autotuning made it
> into that release or not.

	Currently there is no default auto tuning, the user needs to put the word auto into the "target" fields in the queue discipline tab in SQM (which is only exposed after selecting "show advanced configuration" and "show dangerous configuration", not very user friendly, at least it does not say 'Beware of the Leopard'...). So I am sure Rich would have noted that. I think we need a bit of testing of the auto-tuning feature and then make auto the default for ingest and egress. But for the current users I did not want to change the default without some testing (my track record of breaking things slightly comes to mind ;) )

Best Regards
	Sebastian

> 
> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
> above and paste? I still think
> target is still too low on the egress side with the current calculation.
> 
> Secondly, now that you have a setting you like, trying pie, codel, and
> ns2_codel also would be interesting.
> 
> efq_codel is currently uninteresting. Wasn't clear if you were using
> nfq_codel or fq_codel throughout.
> 
> 
> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>> 
>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>> 
>> Test Conditions:
>> 
>> - Running tests from my MacBook Pro, 10.9.2.
>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>> 
>> The Tests:
>> 
>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>> 
>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>> 
>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>> 
>> Test Procedure:
>> 
>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>> 
>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>> 
>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>> 
>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>> 
>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>> 
>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>> 
>> Conclusion:
>> 
>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>> 
>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>> 
>> The data follows...
>> 
>> Rich
>> 
>> ================================================================
>> 
>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>> 
>> 6089/737:
>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>> 
>> 6200/750:
>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>> 
>> 6400/800:
>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>> 
>> 6600/830:
>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>> 
>> 7000/800:
>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>> 
>> 7000/830:
>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>> 
>> 7600/850:
>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>> 
>> SQM Disabled:
>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>> 
>> ================================================================
>> 
>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>> 
>> 6089/737:
>>        5.00/0.58
>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>> 
>> 6200/750:
>>        5.08/0.58
>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>> 
>> 6400/800:
>>        5.24/0.60
>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>> 
>> 6600/830:
>>        5.41/0.65
>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>> 
>> 7000/800:
>>        5.73/0.62
>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>> 
>> 7000/830:
>>        5.74/0.65 (5.71/0.62 second run)
>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>> 
>> 7600/850:
>>        6.20/0.67
>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>> 
>> SQM Disabled:
>>        6.46/0.73
>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Aaron Wood]

Is there a writeup on each of the fq_codel variants?

-Aaron

Sent from my iPhone

> On Mar 2, 2014, at 22:41, Dave Taht <dave.taht@gmail.com> wrote:
> 
> Nice work!
> 
> I have a problem in that I can't remember if target autotuning made it
> into that release or not.
> 
> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
> above and paste? I still think
> target is still too low on the egress side with the current calculation.
> 
> Secondly, now that you have a setting you like, trying pie, codel, and
> ns2_codel also would be interesting.
> 
> efq_codel is currently uninteresting. Wasn't clear if you were using
> nfq_codel or fq_codel throughout.
> 
> 
>> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>> 
>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>> 
>> Test Conditions:
>> 
>> - Running tests from my MacBook Pro, 10.9.2.
>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>> 
>> The Tests:
>> 
>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>> 
>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>> 
>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>> 
>> Test Procedure:
>> 
>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>> 
>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>> 
>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>> 
>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>> 
>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>> 
>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>> 
>> Conclusion:
>> 
>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>> 
>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>> 
>> The data follows...
>> 
>> Rich
>> 
>> ================================================================
>> 
>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>> 
>> 6089/737:
>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>> 
>> 6200/750:
>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>> 
>> 6400/800:
>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>> 
>> 6600/830:
>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>> 
>> 7000/800:
>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>> 
>> 7000/830:
>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>> 
>> 7600/850:
>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>> 
>> SQM Disabled:
>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>> 
>> ================================================================
>> 
>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>> 
>> 6089/737:
>>        5.00/0.58
>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>> 
>> 6200/750:
>>        5.08/0.58
>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>> 
>> 6400/800:
>>        5.24/0.60
>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>> 
>> 6600/830:
>>        5.41/0.65
>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>> 
>> 7000/800:
>>        5.73/0.62
>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>> 
>> 7000/830:
>>        5.74/0.65 (5.71/0.62 second run)
>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>> 
>> 7600/850:
>>        6.20/0.67
>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>> 
>> SQM Disabled:
>>        6.46/0.73
>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

fq_codel is fq_codel from linux mainline, usually configured with a
300 byte quantum.

codel is linux codel

ns2_codel is derived from a smoother decay curve than codel from
patches posted on kathie's website.

nfq_codel is derived from ns2_codel, with an enhancement that rotates
the flow list on every packet,
thus behaving more like sfq_codel.

In at least one variant of cero a month or two back, I eliminated the
maxpacket check in nfq_codel
for which my hope was to not have to fiddle with the target. It helped
somewhat but also led to more
loss when I felt it shouldn't. I think I have a gentler version of
that but haven't implemented it. I would
really like to not have to fiddle with the target... but am growing
resigned to the idea that a real
solution incorporates the rate shaper directly into the Xfq_codel
algorithm so as to autotune it,
and perhaps reduce the number of flows under control.

(The original version of fq_codel (linux 3.5) suffered from the
horizontal standing queue problem,
where each flow at low speeds added more latency. (but low rate flows
were never dropped). I still
struggle with this tradeoff)

One thing to note in rich's graphs is flow preservation is better at
the lower speeds - you have more
of the measurement flows survive longer. This is something you'll also
see in nfq_codel.

I think rich's upstream and measurement stream results will be better
with target 20ms on egress. They will still plot -p all_scaled
badly, and to get a decent plot would require reprocessing with the -p
totals plot type.


On Sun, Mar 2, 2014 at 1:51 PM, Aaron Wood <woody77@gmail.com> wrote:
> Is there a writeup on each of the fq_codel variants?
>
> -Aaron
>
> Sent from my iPhone
>
>> On Mar 2, 2014, at 22:41, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> Nice work!
>>
>> I have a problem in that I can't remember if target autotuning made it
>> into that release or not.
>>
>> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
>> above and paste? I still think
>> target is still too low on the egress side with the current calculation.
>>
>> Secondly, now that you have a setting you like, trying pie, codel, and
>> ns2_codel also would be interesting.
>>
>> efq_codel is currently uninteresting. Wasn't clear if you were using
>> nfq_codel or fq_codel throughout.
>>
>>
>>> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>>>
>>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>>>
>>> Test Conditions:
>>>
>>> - Running tests from my MacBook Pro, 10.9.2.
>>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>>>
>>> The Tests:
>>>
>>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>>>
>>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>>>
>>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>>>
>>> Test Procedure:
>>>
>>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>>>
>>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>>>
>>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>>>
>>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>>>
>>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>>>
>>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>>>
>>> Conclusion:
>>>
>>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>>>
>>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>>>
>>> The data follows...
>>>
>>> Rich
>>>
>>> ================================================================
>>>
>>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>>>
>>> 6089/737:
>>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>>>
>>> 6200/750:
>>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>>>
>>> 6400/800:
>>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>>>
>>> 6600/830:
>>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>>>
>>> 7000/800:
>>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>>>
>>> 7000/830:
>>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>>>
>>> 7600/850:
>>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>>>
>>> SQM Disabled:
>>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>>>
>>> ================================================================
>>>
>>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>>>
>>> 6089/737:
>>>        5.00/0.58
>>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>>>
>>> 6200/750:
>>>        5.08/0.58
>>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>>>
>>> 6400/800:
>>>        5.24/0.60
>>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>>>
>>> 6600/830:
>>>        5.41/0.65
>>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>>>
>>> 7000/800:
>>>        5.73/0.62
>>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>>>
>>> 7000/830:
>>>        5.74/0.65 (5.71/0.62 second run)
>>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>>>
>>> 7600/850:
>>>        6.20/0.67
>>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>>>
>>> SQM Disabled:
>>>        6.46/0.73
>>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

On Mar 2, 2014, at 4:41 PM, Dave Taht <dave.taht@gmail.com> wrote:

> Nice work!
> 
> I have a problem in that I can't remember if target autotuning made it
> into that release or not.
> 
> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
> above and paste? I still think
> target is still too low on the egress side with the current calculation.

Pasted below. The 14.8 msec is the result of the calculation that comes from the current adapt_target_to_slow_link() function for 830kbps.

> Secondly, now that you have a setting you like, trying pie, codel, and
> ns2_codel also would be interesting.

Need to find another available time slot, but sure.

> efq_codel is currently uninteresting. Wasn't clear if you were using
> nfq_codel or fq_codel throughout.

nfq_codel used for all tests.

Rich

root@cerowrt:~# tc -s qdisc show dev ge00
qdisc htb 1: root refcnt 2 r2q 10 default 12 direct_packets_stat 0
 Sent 122649261 bytes 332175 pkt (dropped 0, overlimits 579184 requeues 0)
 backlog 0b 0p requeues 0
qdisc nfq_codel 110: parent 1:11 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc nfq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
 Sent 122649261 bytes 332175 pkt (dropped 129389, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 37646 ecn_mark 0
  new_flows_len 1 old_flows_len 2
qdisc nfq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 278307120 bytes 467944 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
> 
> 
> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>> 
>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>> 
>> Test Conditions:
>> 
>> - Running tests from my MacBook Pro, 10.9.2.
>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>> 
>> The Tests:
>> 
>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>> 
>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>> 
>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>> 
>> Test Procedure:
>> 
>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>> 
>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>> 
>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>> 
>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>> 
>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>> 
>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>> 
>> Conclusion:
>> 
>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>> 
>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>> 
>> The data follows...
>> 
>> Rich
>> 
>> ================================================================
>> 
>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>> 
>> 6089/737:
>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>> 
>> 6200/750:
>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>> 
>> 6400/800:
>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>> 
>> 6600/830:
>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>> 
>> 7000/800:
>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>> 
>> 7000/830:
>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>> 
>> 7600/850:
>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>> 
>> SQM Disabled:
>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>> 
>> ================================================================
>> 
>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>> 
>> 6089/737:
>>        5.00/0.58
>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>> 
>> 6200/750:
>>        5.08/0.58
>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>> 
>> 6400/800:
>>        5.24/0.60
>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>> 
>> 6600/830:
>>        5.41/0.65
>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>> 
>> 7000/800:
>>        5.73/0.62
>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>> 
>> 7000/830:
>>        5.74/0.65 (5.71/0.62 second run)
>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>> 
>> 7600/850:
>>        6.20/0.67
>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>> 
>> SQM Disabled:
>>        6.46/0.73
>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

Sebastian,

> 	Currently there is no default auto tuning, the user needs to put the word auto into the "target" fields in the queue discipline tab in SQM (which is only exposed after selecting "show advanced configuration" and "show dangerous configuration", not very user friendly, at least it does not say 'Beware of the Leopard'...). So I am sure Rich would have noted that. I think we need a bit of testing of the auto-tuning feature and then make auto the default for ingest and egress. But for the current users I did not want to change the default without some testing (my track record of breaking things slightly comes to mind ;) )

Once we get the calculation correct, I would vote for the default being "auto" (that is, the ingress and egress strings would be pre-configure to "auto"). It would be the right thing for virtually anyone, but people could change it in the name of research.

While I'm lobbying, I would still like to see the second item in the Link Layer choices be "Ethernet with overhead" (instead of simply "Ethernet') Doing this makes it easier to explain:

"None" means none. CeroWrt does not make any per-packet adjustments at all.

"Ethernet with overhead" means that CeroWrt assumes that each packet has an additional overhead of N bytes that needs to be taken into account.

"ATM: ..." means that CeroWrt accounts for the incredibly arcane (stupid?) vagaries of the ATM 48-in-53 byte framing.

Thanks.

Rich

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

Target 30ms would be interesting. 30+ % packet loss on this test (and
your link is still operable!)

At your speed range the behavior of the tcp upload is governed by the
initial window size more than anything else. I doubt the
mac adjusts that properly so it keeps hammering away at it... my
concern from looking at your upload graphs was that you were
actually getting hammered so hard you were actually going through RTO
(receive timeout) on the uploads.

On Sun, Mar 2, 2014 at 5:27 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>
> On Mar 2, 2014, at 4:41 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
>> Nice work!
>>
>> I have a problem in that I can't remember if target autotuning made it
>> into that release or not.
>>
>> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
>> above and paste? I still think
>> target is still too low on the egress side with the current calculation.
>
> Pasted below. The 14.8 msec is the result of the calculation that comes from the current adapt_target_to_slow_link() function for 830kbps.
>
>> Secondly, now that you have a setting you like, trying pie, codel, and
>> ns2_codel also would be interesting.
>
> Need to find another available time slot, but sure.
>
>> efq_codel is currently uninteresting. Wasn't clear if you were using
>> nfq_codel or fq_codel throughout.
>
> nfq_codel used for all tests.
>
> Rich
>
> root@cerowrt:~# tc -s qdisc show dev ge00
> qdisc htb 1: root refcnt 2 r2q 10 default 12 direct_packets_stat 0
>  Sent 122649261 bytes 332175 pkt (dropped 0, overlimits 579184 requeues 0)
>  backlog 0b 0p requeues 0
> qdisc nfq_codel 110: parent 1:11 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc nfq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>  Sent 122649261 bytes 332175 pkt (dropped 129389, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 37646 ecn_mark 0
>   new_flows_len 1 old_flows_len 2
> qdisc nfq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>   new_flows_len 0 old_flows_len 0
> qdisc ingress ffff: parent ffff:fff1 ----------------
>  Sent 278307120 bytes 467944 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
>>
>>
>> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>>>
>>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>>>
>>> Test Conditions:
>>>
>>> - Running tests from my MacBook Pro, 10.9.2.
>>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>>>
>>> The Tests:
>>>
>>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>>>
>>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>>>
>>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>>>
>>> Test Procedure:
>>>
>>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>>>
>>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>>>
>>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>>>
>>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>>>
>>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>>>
>>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>>>
>>> Conclusion:
>>>
>>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>>>
>>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>>>
>>> The data follows...
>>>
>>> Rich
>>>
>>> ================================================================
>>>
>>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>>>
>>> 6089/737:
>>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>>>
>>> 6200/750:
>>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>>>
>>> 6400/800:
>>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>>>
>>> 6600/830:
>>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>>>
>>> 7000/800:
>>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>>>
>>> 7000/830:
>>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>>>
>>> 7600/850:
>>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>>>
>>> SQM Disabled:
>>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>>>
>>> ================================================================
>>>
>>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>>>
>>> 6089/737:
>>>        5.00/0.58
>>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>>>
>>> 6200/750:
>>>        5.08/0.58
>>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>>>
>>> 6400/800:
>>>        5.24/0.60
>>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>>>
>>> 6600/830:
>>>        5.41/0.65
>>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>>>
>>> 7000/800:
>>>        5.73/0.62
>>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>>>
>>> 7000/830:
>>>        5.74/0.65 (5.71/0.62 second run)
>>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>>>
>>> 7600/850:
>>>        6.20/0.67
>>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>>>
>>> SQM Disabled:
>>>        6.46/0.73
>>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

I wouldn't mind a wireshark packet capture of a target 14.8ms and a
target 30ms test.

On Sun, Mar 2, 2014 at 5:41 PM, Dave Taht <dave.taht@gmail.com> wrote:
> Target 30ms would be interesting. 30+ % packet loss on this test (and
> your link is still operable!)
>
> At your speed range the behavior of the tcp upload is governed by the
> initial window size more than anything else. I doubt the
> mac adjusts that properly so it keeps hammering away at it... my
> concern from looking at your upload graphs was that you were
> actually getting hammered so hard you were actually going through RTO
> (receive timeout) on the uploads.
>
> On Sun, Mar 2, 2014 at 5:27 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>>
>> On Mar 2, 2014, at 4:41 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>>> Nice work!
>>>
>>> I have a problem in that I can't remember if target autotuning made it
>>> into that release or not.
>>>
>>> Coulde you do a tc -s qdisc show dev ge00 on your favorite of the
>>> above and paste? I still think
>>> target is still too low on the egress side with the current calculation.
>>
>> Pasted below. The 14.8 msec is the result of the calculation that comes from the current adapt_target_to_slow_link() function for 830kbps.
>>
>>> Secondly, now that you have a setting you like, trying pie, codel, and
>>> ns2_codel also would be interesting.
>>
>> Need to find another available time slot, but sure.
>>
>>> efq_codel is currently uninteresting. Wasn't clear if you were using
>>> nfq_codel or fq_codel throughout.
>>
>> nfq_codel used for all tests.
>>
>> Rich
>>
>> root@cerowrt:~# tc -s qdisc show dev ge00
>> qdisc htb 1: root refcnt 2 r2q 10 default 12 direct_packets_stat 0
>>  Sent 122649261 bytes 332175 pkt (dropped 0, overlimits 579184 requeues 0)
>>  backlog 0b 0p requeues 0
>> qdisc nfq_codel 110: parent 1:11 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>>   new_flows_len 0 old_flows_len 0
>> qdisc nfq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>>  Sent 122649261 bytes 332175 pkt (dropped 129389, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>>   maxpacket 0 drop_overlimit 0 new_flow_count 37646 ecn_mark 0
>>   new_flows_len 1 old_flows_len 2
>> qdisc nfq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 14.8ms interval 109.8ms
>>  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>>   maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
>>   new_flows_len 0 old_flows_len 0
>> qdisc ingress ffff: parent ffff:fff1 ----------------
>>  Sent 278307120 bytes 467944 pkt (dropped 0, overlimits 0 requeues 0)
>>  backlog 0b 0p requeues 0
>>>
>>>
>>> On Sun, Mar 2, 2014 at 1:18 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>>>> I took some time this weekend, and ran careful speed and latency tests on the CeroWrt 3.10.28-16 build. I have a much better understanding of how all this works, both in theory and in practice. Here's an executive summary of the overall test procedure with lots of details below.
>>>>
>>>> Adjusting CeroWrt's configured up- and download rates in the SQM page affects both the actual data transfer rates as well as the latency. If you set the values too low, CeroWrt will enforce that bottleneck, and the transfer rates will be lower than you could attain on your link. If you configure them too high, though, the transfer rates may look better, but the latency can go off the charts. Here's how I arrived at a good balance.
>>>>
>>>> Test Conditions:
>>>>
>>>> - Running tests from my MacBook Pro, 10.9.2.
>>>> - Wi-Fi off; ethernet cable direct to Netgear WNDR3700v2 with CeroWrt 3.10.28-16.
>>>> - DSL service from Fairpoint, nominally "7 Mbps down/768kbps up".
>>>> - DSL Modem sync rate (the actual rate that bits enter/leave my house) is 7616kbps down; 864kbps up. The line is apparently fairly clean, too.
>>>> - Base ping time to the nearest router at ISP (via traceroute) is 29-30 msec.
>>>> - To minimize other traffic, I turned off most of the computers at home, and also quit my mail client (which is surprisingly chatty).
>>>>
>>>> The Tests:
>>>>
>>>> I ran two different tests: netperf-wrapper with the RRUL test, and speedtest.net. These give very different views of performance. RRUL really stresses the line using multiple simultaneous up and download streams. Speedtest.net is a consumer test that only tests one direction at a time, and for a short time. We want to look good with both.
>>>>
>>>> For the RRUL tests, I invoked netperf-wrapper like this: netperf-wrapper rrul -p all_scaled -l 60 -H atl.richb-hanover.com -t text-shown-in-chart
>>>> For the Speedtest.net tests, I used their web GUI in the obvious way.
>>>>
>>>> For both tests, I used a script (pingstats.sh, see my next message) to collect the ping times and give min, max, average, median, and 10th and 90th percentile readings.
>>>>
>>>> Test Procedure:
>>>>
>>>> I ran a series of tests starting with the up/down link rates spelled out by Sebastian Moeller's amazingly detailed note last week. See https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-February/002375.html Read it carefully. There's a lot of insight available there.
>>>>
>>>> The initial configuration was 6089/737 down/up, with the (nearly default) values for Queue Discipline (nfq_codel, simple.qos, ECN on for ingress; NOECN for egress, auto for both ingress and egress latency targets), and ATM link layer with 44 bytes of overhead.
>>>>
>>>> With those initial configuration values, latency was good but the speeds were disappointing. I then re-ran the tests with CeroWrt configured for higher up/down link speeds to see where things broke.
>>>>
>>>> Things got better and better with increasing link rates until I hit 7600/850 - at that point, latency began to get quite large. (Of course, with SQM disabled, the latency got dreadful.)
>>>>
>>>> There was an anomaly at 7000/800 kbps. The 90th percentile and max numbers jumped up quite a lot, but went *down* for the next test in the sequence when I increased the upload speed to 7000/830. I ran the experiment twice to confirm that behavior.
>>>>
>>>> I should also note that in the course of the experiment, I re-ran many of these tests. Although I did not document each of the runs, the results (speedtest.net rates and the pingstats.sh values) were quite consistent and repeatable.
>>>>
>>>> Conclusion:
>>>>
>>>> I'm running with CeroWrt 3.10.28-16 configured for down/up 7000/830, (nearly) default Queue Discipline and ATM+44 bytes of overhead. With these configurations, latency is well in hand and my network is pretty speedy.
>>>>
>>>> We need to figure out how to explain to people what to expect re: the tradeoff between "faster speeds" that show up in Speedtest.net (with accompanying crappy performance) and slightly slower speeds with a *way* better experience.
>>>>
>>>> The data follows...
>>>>
>>>> Rich
>>>>
>>>> ================================================================
>>>>
>>>> RRUL Tests:  The charts associated with these RRUL runs are all available at http://richb-hanover.com/rrul-tests-cerowrt-3-10-28-16/
>>>>
>>>> 6089/737:
>>>>        Min: 28.936 10pct: 29.094 Avg: 40.529 Median: 37.961 90pct: 52.636 Max: 77.171 Num pings: 77
>>>>
>>>> 6200/750:
>>>>        Min: 28.715 10pct: 29.298 Avg: 41.805 Median: 39.826 90pct: 57.414 Max: 72.363 Num pings: 77
>>>>
>>>> 6400/800:
>>>>        Min: 28.706 10pct: 29.119 Avg: 39.598 Median: 38.428 90pct: 52.351 Max: 69.492 Num pings: 78
>>>>
>>>> 6600/830:
>>>>        Min: 28.485 10pct: 29.114 Avg: 41.708 Median: 39.753 90pct: 57.552 Max: 87.328 Num pings: 77
>>>>
>>>> 7000/800:
>>>>        Min: 28.570 10pct: 29.180 Avg: 46.245 Median: 42.684 90pct: 62.376 Max: 169.991 Num pings: 77
>>>>        Min: 28.775 10pct: 29.226 Avg: 43.628 Median: 40.446 90pct: 60.216 Max: 121.334 Num pings: 76 (2nd run)
>>>>
>>>> 7000/830:
>>>>        Min: 28.942 10pct: 29.285 Avg: 44.283 Median: 45.318 90pct: 58.002 Max: 85.035 Num pings: 78
>>>>        Min: 28.951 10pct: 29.479 Avg: 43.182 Median: 41.000 90pct: 57.570 Max: 74.964 Num pings: 76 (2nd run)
>>>>
>>>> 7600/850:
>>>>        Min: 28.756 10pct: 29.078 Avg: 55.426 Median: 46.063 90pct: 81.847 Max: 277.807 Num pings: 84
>>>>
>>>> SQM Disabled:
>>>>        Min: 28.665 10pct: 29.062 Avg: 1802.521 Median: 2051.276 90pct: 2762.941 Max: 4217.644 Num pings: 78
>>>>
>>>> ================================================================
>>>>
>>>> Speedtest.net: First values are the reported down/up rates in the Speedtest GUI
>>>>
>>>> 6089/737:
>>>>        5.00/0.58
>>>>        Min: 28.709 10pct: 28.935 Avg: 33.416 Median: 31.619 90pct: 38.608 Max: 49.193 Num pings: 45
>>>>
>>>> 6200/750:
>>>>        5.08/0.58
>>>>        Min: 28.759 10pct: 29.055 Avg: 33.974 Median: 32.584 90pct: 41.938 Max: 46.605 Num pings: 44
>>>>
>>>> 6400/800:
>>>>        5.24/0.60
>>>>        Min: 28.447 10pct: 28.826 Avg: 34.675 Median: 31.155 90pct: 41.285 Max: 81.503 Num pings: 43
>>>>
>>>> 6600/830:
>>>>        5.41/0.65
>>>>        Min: 28.868 10pct: 29.053 Avg: 35.158 Median: 32.928 90pct: 44.099 Max: 51.571 Num pings: 44
>>>>
>>>> 7000/800:
>>>>        5.73/0.62
>>>>        Min: 28.359 10pct: 28.841 Avg: 35.205 Median: 33.620 90pct: 43.735 Max: 54.812 Num pings: 44
>>>>
>>>> 7000/830:
>>>>        5.74/0.65 (5.71/0.62 second run)
>>>>        Min: 28.605 10pct: 29.055 Avg: 34.945 Median: 31.773 90pct: 42.645 Max: 54.077 Num pings: 44
>>>>        Min: 28.649 10pct: 28.820 Avg: 34.866 Median: 32.398 90pct: 43.533 Max: 69.288 Num pings: 56 (2nd run)
>>>>
>>>> 7600/850:
>>>>        6.20/0.67
>>>>        Min: 28.835 10pct: 28.963 Avg: 36.253 Median: 34.912 90pct: 44.659 Max: 54.023 Num pings: 48
>>>>
>>>> SQM Disabled:
>>>>        6.46/0.73
>>>>        Min: 28.452 10pct: 28.872 Avg: 303.754 Median: 173.498 90pct: 499.678 Max: 1799.814 Num pings: 45
>>>> _______________________________________________
>>>> Cerowrt-devel mailing list
>>>> Cerowrt-devel@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>>
>>>
>>>
>>> --
>>> Dave Täht
>>>
>>> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html
>>
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

Dave,

> Target 30ms would be interesting. 30+ % packet loss on this test (and
> your link is still operable!)

Wait. How can you detect the packet loss from the data I sent along?

> At your speed range the behavior of the tcp upload is governed by the
> initial window size more than anything else. I doubt the
> mac adjusts that properly so it keeps hammering away at it... my
> concern from looking at your upload graphs was that you were
> actually getting hammered so hard you were actually going through RTO
> (receive timeout) on the uploads.

Yuck. That would be bad.

re: Wireshark. I'll try to get it soon.

Thanks.

Rich

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

it's in tc's output

qdisc nfq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300
target 14.8ms interval 109.8ms
 Sent 122649261 bytes 332175 pkt (dropped 129389, overlimits 0 requeues 0)
                                                                  ^^^^^^
 backlog 0b 0p requeues 0.
  maxpacket 0 drop_overlimit 0 new_flow_count 37646 ecn_mark 0
  new_flows_len 1 old_flows_len 2

It's late in england. going to bed before ietf tomorrow/

(I don't have a mac anymore and never thought to do a caipture at this rate.

I think the mac is iw4, but as we have 4 uploads going we are
effectively iw16 at
rrul startup, and yet, here we are trying to cut latencies to 15ms.

what I'm interested in determining is if the mac flows are experiencing RTOs,
and/or falling back to iw2...

For way more detail on the joys of iwX at high and low bandwidths:

https://datatracker.ietf.org/doc/rfc6928/?include_text=1

JG also has a rant on iw10 considered harmful. This is iw4 considered harmful
at speeds below 2mbit....

I have been trying to come up with a way to for default gws to communicate
a good iw size for a while and/or have a plot of behaviors at this speed.

On Sun, Mar 2, 2014 at 5:49 PM, Rich Brown <richb.hanover@gmail.com> wrote:
> Dave,
>
>> Target 30ms would be interesting. 30+ % packet loss on this test (and
>> your link is still operable!)
>
> Wait. How can you detect the packet loss from the data I sent along?
>
>> At your speed range the behavior of the tcp upload is governed by the
>> initial window size more than anything else. I doubt the
>> mac adjusts that properly so it keeps hammering away at it... my
>> concern from looking at your upload graphs was that you were
>> actually getting hammered so hard you were actually going through RTO
>> (receive timeout) on the uploads.
>
> Yuck. That would be bad.
>
> re: Wireshark. I'll try to get it soon.
>
> Thanks.
>
> Rich



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

Dave,

I got the wireshark info. https://dl.dropboxusercontent.com/u/21403660/RRUL-Wireshark.zip

It was my intent to replicate the RRUL -p all_scaled test with both 'auto' and '30ms' for the egress target. I hope I got it correct - both the settings and the wireshark capture. 

Rich

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

Wow. Mac TCP is certainly getting stuffed at 768KB up.

using a wireshark filter of tcp.stream eq 18

it looks like you are iw3, with 8 bytes of overhead on the wire, and
although the MSS gets reduced slightly (not enough!)
the window never gets below 132480. The throughput graph is reasonable
regardless.... but I have to figure out a way to "see" if the RTO
timer is firing or not. Thankfully there's an expert handy this
afternoon.

I'll setup a duplicate test here on a different OS and see what I see...

supplying the rrul *json.gz files would let me do some totals plots....

On Sun, Mar 2, 2014 at 6:28 PM, Rich Brown <richb.hanover@gmail.com> wrote:
> Dave,
>
> I got the wireshark info. https://dl.dropboxusercontent.com/u/21403660/RRUL-Wireshark.zip
>
> It was my intent to replicate the RRUL -p all_scaled test with both 'auto' and '30ms' for the egress target. I hope I got it correct - both the settings and the wireshark capture.
>
> Rich
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1308 bytes --]

On Mon, Mar 3, 2014 at 12:51 AM, Dave Taht <dave.taht@gmail.com> wrote:
> Wow. Mac TCP is certainly getting stuffed at 768KB up.
>
> using a wireshark filter of tcp.stream eq 18
>
> it looks like you are iw3, with 8 bytes of overhead on the wire, and
> although the MSS gets reduced slightly (not enough!)
> the window never gets below 132480. The throughput graph is reasonable
> regardless.... but I have to figure out a way to "see" if the RTO
> timer is firing or not. Thankfully there's an expert handy this
> afternoon.
>
> I'll setup a duplicate test here on a different OS and see what I see...
>
> supplying the rrul *json.gz files would let me do some totals plots....
>
> On Sun, Mar 2, 2014 at 6:28 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> Dave,
>>
>> I got the wireshark info. https://dl.dropboxusercontent.com/u/21403660/RRUL-Wireshark.zip
>>
>> It was my intent to replicate the RRUL -p all_scaled test with both 'auto' and '30ms' for the egress target. I hope I got it correct - both the settings and the wireshark capture.
>>
>> Rich
>>
>
>
>
> --
> Dave Täht
>
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

[-- Attachment #2: richthroughput.png --]
[-- Type: image/png, Size: 247543 bytes --]

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Rich Brown]

Dave,

Thanks for looking at this.

> Wow. Mac TCP is certainly getting stuffed at 768KB up.

This sounds like an OSX problem (which I think we always suspected. *Every* network stack needs to have this worked on.)

Does anyone know how to approach Apple about this? I am a registered Apple developer (me, and my 100,000 closest friends). I don't think I have any connections there that could rise above the noise.

> using a wireshark filter of tcp.stream eq 18
> 
> it looks like you are iw3, with 8 bytes of overhead on the wire, and
> although the MSS gets reduced slightly (not enough!)
> the window never gets below 132480. The throughput graph is reasonable
> regardless.... but I have to figure out a way to "see" if the RTO
> timer is firing or not. Thankfully there's an expert handy this
> afternoon.
> 
> I'll setup a duplicate test here on a different OS and see what I see...

I will see if I can get netperf-wrapper installed on my Win7 laptop. That effort will compete for time with my work week, though.

> supplying the rrul *json.gz files would let me do some totals plots....

https://dl.dropboxusercontent.com/u/21403660/RRUL-json.zip

> 
> On Sun, Mar 2, 2014 at 6:28 PM, Rich Brown <richb.hanover@gmail.com> wrote:
>> Dave,
>> 
>> I got the wireshark info. https://dl.dropboxusercontent.com/u/21403660/RRUL-Wireshark.zip
>> 
>> It was my intent to replicate the RRUL -p all_scaled test with both 'auto' and '30ms' for the egress target. I hope I got it correct - both the settings and the wireshark capture.
>> 
>> Rich
>> 
> 
> 
> 
> -- 
> Dave Täht
> 
> Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

Rich Brown <richb.hanover@gmail.com> writes:

> I will see if I can get netperf-wrapper installed on my Win7 laptop.
> That effort will compete for time with my work week, though.

I think that it should run under cygwin; however, I seem to recall
someone reporting that the native cygwin 'ping' utility does not support
timestamping, and so can't be used. Installing fping should solve this,
but I don't think it's packaged for cygwin. If you're willing to look
into this, that would be great; I don't have a cygwin environment handy
to test it out myself... :)

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Cerowrt-devel] Better results from CeroWrt 3.10.28-16

[Sebastian Moeller]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

Hi Rich,


On Mar 3, 2014, at 02:38 , Rich Brown <richb.hanover@gmail.com> wrote:

> Sebastian,
> 
>> 	Currently there is no default auto tuning, the user needs to put the word auto into the "target" fields in the queue discipline tab in SQM (which is only exposed after selecting "show advanced configuration" and "show dangerous configuration", not very user friendly, at least it does not say 'Beware of the Leopard'...). So I am sure Rich would have noted that. I think we need a bit of testing of the auto-tuning feature and then make auto the default for ingest and egress. But for the current users I did not want to change the default without some testing (my track record of breaking things slightly comes to mind ;) )
> 
> Once we get the calculation correct, I would vote for the default being "auto" (that is, the ingress and egress strings would be pre-configure to "auto"). It would be the right thing for virtually anyone, but people could change it in the name of research.
> 
> While I'm lobbying, I would still like to see the second item in the Link Layer choices be "Ethernet with overhead" (instead of simply "Ethernet') Doing this makes it easier to explain:
> 
> "None" means none. CeroWrt does not make any per-packet adjustments at all.
> 
> "Ethernet with overhead" means that CeroWrt assumes that each packet has an additional overhead of N bytes that needs to be taken into account.
> 
> "ATM: ..." means that CeroWrt accounts for the incredibly arcane (stupid?) vagaries of the ATM 48-in-53 byte framing.



	Okay, changed. Any other changes to the GUI text you would like to see? Just let me know :) 

Best Regards
	sebastian

> 
> Thanks.
> 
> Rich


[-- Attachment #2.1: Type: text/html, Size: 2589 bytes --]

[-- Attachment #2.2: PastedGraphic-1.png --]
[-- Type: image/png, Size: 109245 bytes --]