[Bloat] [Cerowrt-devel] Comcast upped service levels -> WNDR3800 can't cope...

Thu Sep 4 10:05:04 EDT 2014

Dave-

Inline responses with bvs ---

Bill Ver Steeg

-----Original Message-----
From: Dave Taht [mailto:dave.taht at gmail.com] 
Sent: Wednesday, September 03, 2014 8:33 PM
To: Bill Ver Steeg (versteb)
Cc: Sebastian Moeller; cerowrt-devel at lists.bufferbloat.net; bloat
Subject: Re: [Bloat] [Cerowrt-devel] Comcast upped service levels -> WNDR3800 can't cope...

On Wed, Sep 3, 2014 at 4:17 PM, Bill Ver Steeg (versteb) <versteb at cisco.com> wrote:
> Speaking of IPv6 performance testing- In a recent FTTH field deployment, the network operator deployed an IPv6-only network and tunneled all subscriber IPv4 traffic over an IPv6 tunnel to the upstream network edge. It then unpacked the IPv4 traffic from the IPv6 tunnel and sent it on its merry way.

I tried to deploy ipv4 over ipv6 encapsulation when I was in Nicaragua
6 years back (the alternative was triple nat,
IPv4 addresses were really scarce on the ground), and got beat by the encapsulation overhead, performance, multiple bugs and bufferbloat, then. I figure that most of that has improved - in particular I imagine that their encapsulated traffic still has a 1500 MTU for the ipv4 traffic?

Bvs --- I do not recall the MTU, but it was almost certainly 1500+. The MTU was not an issue in any testing we did.

The original gear I'd had for the experiment could do a 2k MTU, which was very helpful in making the ipv4 encapsulation
 1500 bytes as much of the internet expects, but a later version couldn't quite get past 1540 bytes without problems.

> Long story short, the 4o6 tunneling code in the residential gateway was not nearly as performant as the IPv6 forwarding code. I actually got better IPv4 throughput running an IPv6 VPN on my end device, then sending my IPv4 traffic through that tunnel - thus avoiding the tunnel code on the gateway. If I recall correctly, the tunnel code capped out at about 20 Mbps and the IPv6 code went up to the 50Mbps SLA rate. I stumbled into this while running some IPTV video tests while running throughput benchmarks on my PC (with apparently pseudo-random results, until we figured out the various tunnels). Took me a while to figure it out. Delay also spiked when the gateway got bogged down......

I can believe it. I have seen many "bumps in the wire" do bad things when run past their limits. Notable were several PPPoe and PPPoA boxes. Older cablemodems, and last generation access points are going to all have similar problems when hooked up at these higher speeds. In the future, stuff that does this sort of tunneling or encapsulation, or while coverting from one media type to another, (say ethernet->cable,
ethernet->gpon, etc) may also run into it when the provider ups their
access speeds from one band to another, as both comcast and verizon have.

This is of course, both a problem and an opportunity. A problem because it will generate more support calls, and an opportunity to sell better gear into the marketplace as ISP speeds are upgraded.

Bvs --- speaking as an employee of a vendor of such equipment, problems in the field are never an "opportunity". If anything, they are an opportunity for the network operator to buy gear from somebody else 8-(.

Some enterprising manufacturer could make a point of pitching their
product(s) as actually capable of modern transfer speeds on modern ISPs, doing benchmarks, etc.

Given the mass delusional product naming in the home ap marketplace, where nearly every product is named and pitched over the base capability of the standards used, rather than the sordid reality, I don't think anything short of a consumer reports, or legal action, will result in sanity here.

Gigabit "routers", indeed, when only the switch is cable of that!
Nothing I've tried below 100 bucks can forward, well, at a gigabit, with a number of real-world firewall rules. Even using x86 gear is kind of problematic thus far.

http://snapon.lab.bufferbloat.net/~cero2/nuc-to-puck/results.html

> More capable gateways were deployed in the latter stages of the deployment, and they seemed to keep up with the 50 Mbps SLA rate.

What was the measured latency under load?

Bvs --- We saw what one would expect from a tail-drop based system. The delay was basically constant at some 10s of ms until we hit the upstream/downstream SLA rate, then it started to creep up. If I recall correctly, it actually did not creep up much - maybe by 40ms or so..... If I had to guess (and I am guessing because we did not provide this piece of equipment in this deployment - we just supplied the Set Top Box and the core network) I would say that the buffer sizes were set to a fixed size that was based on a range of downstream SLA rates. Because we were running at quite high speeds, the delay associated with a full buffer was fairly modest. Had we been running at a lower SLA rate, I suspect that our delay would have been much higher.

>
> Bill Ver Steeg
>
>
>
>
>
> -----Original Message-----
> From: bloat-bounces at lists.bufferbloat.net 
> [mailto:bloat-bounces at lists.bufferbloat.net] On Behalf Of Dave Taht
> Sent: Wednesday, September 03, 2014 3:31 PM
> To: Sebastian Moeller
> Cc: cerowrt-devel at lists.bufferbloat.net; bloat
> Subject: Re: [Bloat] [Cerowrt-devel] Comcast upped service levels -> WNDR3800 can't cope...
>
> On Wed, Sep 3, 2014 at 12:22 PM, Sebastian Moeller <moeller0 at gmx.de> wrote:
>> Hi Aaron,
>>
>>
>> On Sep 3, 2014, at 17:12 , Aaron Wood <woody77 at gmail.com> wrote:
>>
>>> On Wed, Sep 3, 2014 at 4:08 AM, Jonathan Morton <chromatix99 at gmail.com> wrote:
>>> Given that the CPU load is confirmed as high, the pcap probably isn't as useful.  The rest would be interesting to look at.
>>>
>>> Are you able to test with smaller packet sizes?  That might help to isolate packet-throughput (ie. connection tracking) versus byte-throughput problems.
>>>
>>>  - Jonathan Morton
>>>
>>> Doing another test setup will take a few days (maybe not until the weekend).  But I can get the data uploaded, and do some preliminary crunching on it.
>>
>>         So the current SQM system allows to shape on multiple interfaces, so you could set up the shaper on se00 and test between sw10 and se00 (should work if you reliably get fast enough wifi connection, something like combined shaped bandwidth <= 70% of wifi rate should work). That would avoid the whole firewall and connection tracking logic.
>>         My home wifi environment is quite variable/noisy and not 
>> well-suited for this test: with rrul_be I got stuck at around 70Mbps combined bandwidth, with different distributions of the up and down-leg for no-shaping, shaping to 50Mbps10Mbps, and shaping to 100Mbps50Mbps. SIRQ got pretty much pegged at 96-99% during all netperf-wrapper runs, so I assume this to be the bottleneck (the radio was in the > 200mbps range during the test with occasional drops to 150mbps). So my conclusion would: be it really is the shaping that is limited on my wndr3700v2 with cerowrt 3.10.50-1, again if I would be confident about the measurement which I am not (but EOUTOFTIME). That or my rf environment might only allow for roughly 70-80Mbps combined throughput. For what it is worth: test where performed between macbook running macosx 10.9.4 and hp proliant n54l running 64bit openSuse 13.1, kernel 3.11.10-17 (AMD turion with tg3 gbit ethernet adapter (BQL enabled), running fq_codel on eth0), with sha  ping on the se00 interface.
>
>
> A note on wifi throughput. CeroWrt routes, rather than bridges, between interfaces. So I would expect for simple benchmarks, openwrt (which bridges) might show much better wifi<-> ethernet behavior.
>
> We route, rather than bridge wifi, because of 1) it made it easier to debug it, and 2) the theory that multicast on busier networks messes up wifi far more than not-bridging slows it down. Have not accumulated a lot of proof of this, but this was kind of enlightening:
> http://tools.ietf.org/html/draft-desmouceaux-ipv6-mcast-wifi-power-usa
> ge-00
>
> I note that my regular benchmarking environment has mostly been 2 or more routers with nat and firewalling disabled.
>
> Given the trend towards looking at iptables and nat overhead on this thread, an ipv6 benchmark on this box might be revealing.
>
>> Best Regards
>>         Sebastian
>>
>>
>>>
>>> -Aaron
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel at lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
> --
> Dave Täht
>
> https://www.bufferbloat.net/projects/make-wifi-fast
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

--
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast