[Cerowrt-devel] great interview on the scale conference wifi successes

[Cerowrt-devel] great interview on the scale conference wifi successes

[Dave Taht]

https://www.youtube.com/watch?v=UXvGbEYeWp0&t=4795

(takes a bit to get to the wifi part)

H/T to david lang for getting it so right!

-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[David Lang]

On Tue, 10 Mar 2015, Dave Taht wrote:

> https://www.youtube.com/watch?v=UXvGbEYeWp0&t=4795
>
> (takes a bit to get to the wifi part)
>
> H/T to david lang for getting it so right!

Thanks, for those interested in what I did, you can see the presentation I gave 
on the topic at LISA '12 
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david_wireless 
and the ;login article I wrote as a follow-up 
https://www.usenix.org/publications/login/april-2013-volume-38-number-2/wireless-means-radio

This year I deployed 53 APs. I'll make an updated map showing where they were 
deployed.

David Lang

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[Dave Taht]

On Tue, Mar 10, 2015 at 1:44 PM, David Lang <david@lang.hm> wrote:
> On Tue, 10 Mar 2015, Dave Taht wrote:
>
>> https://www.youtube.com/watch?v=UXvGbEYeWp0&t=4795
>>
>> (takes a bit to get to the wifi part)
>>
>> H/T to david lang for getting it so right!
>
>
> Thanks, for those interested in what I did, you can see the presentation I
> gave on the topic at LISA '12
> https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david_wireless
> and the ;login article I wrote as a follow-up
> https://www.usenix.org/publications/login/april-2013-volume-38-number-2/wireless-means-radio
>
> This year I deployed 53 APs. I'll make an updated map showing where they
> were deployed.

So far as I know all the APs were fq-codeled, but the firewall/gw was
not. Had I not been too sick to make it to the conference, might have
got that done. As it was, david ran some netperf-wrapper tests on the
firewall on the last day, and, well, that data was miserable. Next
year! we'll get that right.

> David Lang



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[David Lang]

On Tue, 10 Mar 2015, Dave Taht wrote:

> On Tue, Mar 10, 2015 at 1:44 PM, David Lang <david@lang.hm> wrote:
>> On Tue, 10 Mar 2015, Dave Taht wrote:
>>
>>> https://www.youtube.com/watch?v=UXvGbEYeWp0&t=4795
>>>
>>> (takes a bit to get to the wifi part)
>>>
>>> H/T to david lang for getting it so right!
>>
>>
>> Thanks, for those interested in what I did, you can see the presentation I
>> gave on the topic at LISA '12
>> https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david_wireless
>> and the ;login article I wrote as a follow-up
>> https://www.usenix.org/publications/login/april-2013-volume-38-number-2/wireless-means-radio
>>
>> This year I deployed 53 APs. I'll make an updated map showing where they
>> were deployed.
>
> So far as I know all the APs were fq-codeled, but the firewall/gw was
> not. Had I not been too sick to make it to the conference, might have
> got that done. As it was, david ran some netperf-wrapper tests on the
> firewall on the last day, and, well, that data was miserable. Next
> year! we'll get that right.

Next year we are moving to a new location (arguably we outgrew the current 
location quite a while ago, and this year we had people sitting on the floor in 
rooms, hanging out at the doorway, and filling the overflow rooms where we 
live-streamed the talk as well)

the new location is considerably larger (~2x the space), so we will have to buy 
a lot more APs. The WNDR3800 is not available in these sorts of quantities, so 
we will have to buy something new. We are looking at two major options

1. going with something similar to what we've been running (say the 3700v4)

2. switching to ac/mimo APs

since we will be needing ~120 APs for next year, the cost of the different 
options will be significant. At this point we are just starting to look and 
figure out what's available, what sort of price difference there will be, and 
what the advantages of the better APs would be.

I'll probably purchase some samples to experiment over the next few months, but 
we won't make the mass purchase of whatever until late in the year.

David Lang

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[Richard Smith]

On 03/10/2015 05:12 PM, Dave Taht wrote:

>>
>> This year I deployed 53 APs. I'll make an updated map showing where they
>> were deployed.
>
> So far as I know all the APs were fq-codeled, but the firewall/gw was
> not.

How does this work?  I thought the AP's in this setup were run as bridges.

-- 
Richard A. Smith

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[dpreed]

On Thursday, March 12, 2015 11:31am, "Richard Smith" <smithbone@gmail.com> said:

> On 03/10/2015 05:12 PM, Dave Taht wrote:
> 
>>>
>>> This year I deployed 53 APs. I'll make an updated map showing where they
>>> were deployed.
>>
>> So far as I know all the APs were fq-codeled, but the firewall/gw was
>> not.
> 
> How does this work?  I thought the AP's in this setup were run as bridges.

Pretty good question! Of course if AP's running as Ethernet bridges are bloated (meaning their queues can grow quite large) that's yet another reason that we need to make WiFi fast (by putting codel into the bridge function).

Ethernet bridges should definitely manage their outbound queues to keep the queues in them on the average quite small (average < 2 frames in steady state). Otherwise, if the outbound queue runs at 802.11b rates, and the inbound queues run at 802.11ac rates, there will be a serious disaster.

Since you can't ECN generalized Ethernet packets, codel would have to drop packets. And this might have been what David Lang is doing. (of course, it's perfectly reasonable if you know that the LAN is transporting an IP datagram, to ECN-mark those datagrams.  This is what an Internet transport layer is allowed to do, which is why ECN is part of the envelope, not the contents of the end-to-end packet.

The same argument applies to packets held for retransmission over an 802.11 link. It's perfectly OK to hold a packet outside the outbound queue for retransmission when the conditions to the destination get better, but that packet should not block the next packet coming in going to a different destination.  The retransmission queue (which is there to improve reliability) is a different thing.  [However, my intuition suggests that only one packet per next hop should be in the retransmission queue, and it should not stay there very long - after a period of time, let the sender at the next layer up figure out what to do. Propagation changes in the 10's of millisecond time frame. It won't get better if you wait 1/2 second or more]

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[David Lang]

On Thu, 12 Mar 2015, dpreed@reed.com wrote:

> On Thursday, March 12, 2015 11:31am, "Richard Smith" <smithbone@gmail.com> said:
>
>> On 03/10/2015 05:12 PM, Dave Taht wrote:
>>
>>>>
>>>> This year I deployed 53 APs. I'll make an updated map showing where they
>>>> were deployed.
>>>
>>> So far as I know all the APs were fq-codeled, but the firewall/gw was
>>> not.
>>
>> How does this work?  I thought the AP's in this setup were run as bridges.
>
> Pretty good question! Of course if AP's running as Ethernet bridges are 
> bloated (meaning their queues can grow quite large) that's yet another reason 
> that we need to make WiFi fast (by putting codel into the bridge function).

Even when acting as a bridge, there is a need to queue packets that go in each 
direction, so the queuing discipline will come into play. Since this was built 
from (almost) the latest CC code (it was compiled based on the image the 
saturday before the conference, on monday they upgraded the kernel to 3.18, but 
I didn't have time to get a new image tested before tuesday evening when we 
started deployment)

> Ethernet bridges should definitely manage their outbound queues to keep the 
> queues in them on the average quite small (average < 2 frames in steady 
> state). Otherwise, if the outbound queue runs at 802.11b rates, and the 
> inbound queues run at 802.11ac rates, there will be a serious disaster.
>
> Since you can't ECN generalized Ethernet packets, codel would have to drop 
> packets. And this might have been what David Lang is doing. (of course, it's 
> perfectly reasonable if you know that the LAN is transporting an IP datagram, 
> to ECN-mark those datagrams.  This is what an Internet transport layer is 
> allowed to do, which is why ECN is part of the envelope, not the contents of 
> the end-to-end packet.

I just left this as the default OpenWRT CC settings, which are now fq_codel. We 
were not doing any explicit ECN marking or dropping.

David Lang

> The same argument applies to packets held for retransmission over an 802.11 
> link. It's perfectly OK to hold a packet outside the outbound queue for 
> retransmission when the conditions to the destination get better, but that 
> packet should not block the next packet coming in going to a different 
> destination.  The retransmission queue (which is there to improve reliability) 
> is a different thing.  [However, my intuition suggests that only one packet 
> per next hop should be in the retransmission queue, and it should not stay 
> there very long - after a period of time, let the sender at the next layer up 
> figure out what to do. Propagation changes in the 10's of millisecond time 
> frame. It won't get better if you wait 1/2 second or more]
>
>
>
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[Dave Taht]

so the wifi and ethernet queues are fq_codeled in this past SCALE.

One little understood fact about aggregated' wifi is that you CAN
actually FQ the faster rate ethernet queue with the theoretically
slower rate wifi queue, because a wifi aggregate is decoded at memory
speeds, not wire or wireless speeds, and thus, can fill the BQL
ethernet driver queue and have packets stuck at the qdisc layer, where
they can be handled more smartly.

That said, don't know if that ever happened at scale without data.

The wifi outgoing interfaces on each AP on the other hand, could very
well have had either the fq or the codel portions of the algorithm
engage... they certainly do on my tests here.

Although dlang did get quite a lot of data (which I have not yet seen,
or have I had time for - does anyone here have time to analyze it? He
got a ton of minstrel statistics in particular), but I am unsure if he
collected any of the tc -s qdisc data from the wifi interfaces. (?)

I also have to note that the ath9k driver has got so good now, that a
ton of the benefit at scale probably came from that, and a great deal
more, from dlang's excellent design for the network itself. It is sad,
of course, that useful functionality like sta-to-sta transfers has to
be disabled in such environments nowadays.
I am really sorry I couldn't go.

I am seeing 25% better forwarding rates on ethernet out of 3.18 vs
3.10 on the same wndr 3800 hardware, btw.


On Thu, Mar 12, 2015 at 4:58 PM, David Lang <david@lang.hm> wrote:
> On Thu, 12 Mar 2015, dpreed@reed.com wrote:
>
>> On Thursday, March 12, 2015 11:31am, "Richard Smith" <smithbone@gmail.com>
>> said:
>>
>>> On 03/10/2015 05:12 PM, Dave Taht wrote:
>>>
>>>>>
>>>>> This year I deployed 53 APs. I'll make an updated map showing where
>>>>> they
>>>>> were deployed.
>>>>
>>>>
>>>> So far as I know all the APs were fq-codeled, but the firewall/gw was
>>>> not.
>>>
>>>
>>> How does this work?  I thought the AP's in this setup were run as
>>> bridges.
>>
>>
>> Pretty good question! Of course if AP's running as Ethernet bridges are
>> bloated (meaning their queues can grow quite large) that's yet another
>> reason that we need to make WiFi fast (by putting codel into the bridge
>> function).
>
>
> Even when acting as a bridge, there is a need to queue packets that go in
> each direction, so the queuing discipline will come into play. Since this
> was built from (almost) the latest CC code (it was compiled based on the
> image the saturday before the conference, on monday they upgraded the kernel
> to 3.18, but I didn't have time to get a new image tested before tuesday
> evening when we started deployment)
>
>> Ethernet bridges should definitely manage their outbound queues to keep
>> the queues in them on the average quite small (average < 2 frames in steady
>> state). Otherwise, if the outbound queue runs at 802.11b rates, and the
>> inbound queues run at 802.11ac rates, there will be a serious disaster.
>>
>> Since you can't ECN generalized Ethernet packets, codel would have to drop
>> packets. And this might have been what David Lang is doing. (of course, it's
>> perfectly reasonable if you know that the LAN is transporting an IP
>> datagram, to ECN-mark those datagrams.  This is what an Internet transport
>> layer is allowed to do, which is why ECN is part of the envelope, not the
>> contents of the end-to-end packet.
>
>
> I just left this as the default OpenWRT CC settings, which are now fq_codel.
> We were not doing any explicit ECN marking or dropping.
>
> David Lang
>
>
>> The same argument applies to packets held for retransmission over an
>> 802.11 link. It's perfectly OK to hold a packet outside the outbound queue
>> for retransmission when the conditions to the destination get better, but
>> that packet should not block the next packet coming in going to a different
>> destination.  The retransmission queue (which is there to improve
>> reliability) is a different thing.  [However, my intuition suggests that
>> only one packet per next hop should be in the retransmission queue, and it
>> should not stay there very long - after a period of time, let the sender at
>> the next layer up figure out what to do. Propagation changes in the 10's of
>> millisecond time frame. It won't get better if you wait 1/2 second or more]
>>
>>
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: [Cerowrt-devel] [Bloat] great interview on the scale conference wifi successes

[David Lang]

On Thu, 12 Mar 2015, Dave Taht wrote:

> so the wifi and ethernet queues are fq_codeled in this past SCALE.
>
> One little understood fact about aggregated' wifi is that you CAN
> actually FQ the faster rate ethernet queue with the theoretically
> slower rate wifi queue, because a wifi aggregate is decoded at memory
> speeds, not wire or wireless speeds, and thus, can fill the BQL
> ethernet driver queue and have packets stuck at the qdisc layer, where
> they can be handled more smartly.
>
> That said, don't know if that ever happened at scale without data.
>
> The wifi outgoing interfaces on each AP on the other hand, could very
> well have had either the fq or the codel portions of the algorithm
> engage... they certainly do on my tests here.
>
> Although dlang did get quite a lot of data (which I have not yet seen,
> or have I had time for - does anyone here have time to analyze it? He
> got a ton of minstrel statistics in particular), but I am unsure if he
> collected any of the tc -s qdisc data from the wifi interfaces. (?)

no, I missed/forgot the request for that.

it's about 20G of data uncompressed, I'm running lzma on it now to see how small 
it will shrink.

> I also have to note that the ath9k driver has got so good now, that a
> ton of the benefit at scale probably came from that, and a great deal
> more, from dlang's excellent design for the network itself. It is sad,
> of course, that useful functionality like sta-to-sta transfers has to
> be disabled in such environments nowadays.
> I am really sorry I couldn't go.
>
> I am seeing 25% better forwarding rates on ethernet out of 3.18 vs
> 3.10 on the same wndr 3800 hardware, btw.

hmm, does this have a noticable effect on the HTB throttling performance?

David Lang