[Cake] faster scheduling, maybe

[Cake] faster scheduling, maybe

[Dave Taht]

http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cake] faster scheduling, maybe

[David Lang]

On Mon, 6 Jun 2016, Dave Taht wrote:

> http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf

I don't think so.

They don't even try for fairness between flows, they are just looking at 
fairness between different VMs. they tell a VM that it has complete access to 
the NIC for a time, then give another VM complete access to the NIC. At best 
they put each VMs traffic into a different hardware queue in the NIC.

This avoids all AQM decisions on the part of the host OS, because the packets 
never get to the host OS.

The speed improvement is by bypassing the host OS and just having the VMs 
deliver packets directly to the NIC. This speeds things up, but at the cost of 
any coordination across VMs. Each VM can run fq_codel but it's much corser 
timeslicing between VMs.

David Lang

Re: [Cake] faster scheduling, maybe

[Dave Taht]

On Mon, Jun 6, 2016 at 11:48 AM, David Lang <david@lang.hm> wrote:
> On Mon, 6 Jun 2016, Dave Taht wrote:
>
>> http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf
>
>
> I don't think so.
>
> They don't even try for fairness between flows, they are just looking at
> fairness between different VMs. they tell a VM that it has complete access
> to the NIC for a time, then give another VM complete access to the NIC. At
> best they put each VMs traffic into a different hardware queue in the NIC.
>
> This avoids all AQM decisions on the part of the host OS, because the
> packets never get to the host OS.
>
> The speed improvement is by bypassing the host OS and just having the VMs
> deliver packets directly to the NIC. This speeds things up, but at the cost
> of any coordination across VMs. Each VM can run fq_codel but it's much
> corser timeslicing between VMs.


Well, the principal things bugging me are:

* that we have multi-core on nearly all the new routers.
* Nearly all the ethernet devices themselves support hardware multiqueue.
* we take 6 locks on the qdiscs
* rx and tx ring cleanup are often combined in existing drivers in a
single thread.

The chance to rework the mac80211 layer on make-wifi-fast (where
manufacturers are also busy adding hardware mq), gives us a chance to
rethink how we process access to these queues.

>
> David Lang



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Cake] faster scheduling, maybe

[David Lang]

On Mon, 6 Jun 2016, Dave Taht wrote:

> On Mon, Jun 6, 2016 at 11:48 AM, David Lang <david@lang.hm> wrote:
>> On Mon, 6 Jun 2016, Dave Taht wrote:
>>
>>> http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf
>>
>>
>> I don't think so.
>>
>> They don't even try for fairness between flows, they are just looking at
>> fairness between different VMs. they tell a VM that it has complete access
>> to the NIC for a time, then give another VM complete access to the NIC. At
>> best they put each VMs traffic into a different hardware queue in the NIC.
>>
>> This avoids all AQM decisions on the part of the host OS, because the
>> packets never get to the host OS.
>>
>> The speed improvement is by bypassing the host OS and just having the VMs
>> deliver packets directly to the NIC. This speeds things up, but at the cost
>> of any coordination across VMs. Each VM can run fq_codel but it's much
>> corser timeslicing between VMs.
>
>
> Well, the principal things bugging me are:
>
> * that we have multi-core on nearly all the new routers.
> * Nearly all the ethernet devices themselves support hardware multiqueue.
> * we take 6 locks on the qdiscs
> * rx and tx ring cleanup are often combined in existing drivers in a
> single thread.

These are valid concerns. But this paper just arbitrated between multiple VMs 
accessing one hardware NIC. If you don't have multiple VMs in play, their 
approach has nothing to work with.

> The chance to rework the mac80211 layer on make-wifi-fast (where
> manufacturers are also busy adding hardware mq), gives us a chance to
> rethink how we process access to these queues.

Watching the discussion I see a few things.

1. if we can figure out exactly how much data the system is going to handle, we 
can fill each queue with what we want in the next aggregate to that destination.

2. with multiple queues/cores, when we have data from different sources, we can 
route it to different cores and split the work of sorting the data into 
different queues between the cores (each working on a different subset of 
queues, so not having to lock against other cores)

2a. balancing traffic across the cores/queuesets (or at least the output from 
each of them) would be tricky, but that's where thinking like this paper could 
possibly help.

3. as we are seeing in the MAC80211 work, qdiscs operate way to early in the 
process, so we need to eliminate them for wifi and doing the queueing closer to 
the hardware. On a network where packets to different destinaions can be freely 
mixed, qdiscs can continue to operate.

for drivers where the rx and tx ring cleanup is combined, are you talking about 
ones that already go BQL? or are these drivers that need BQL added as well? it 
may be that splitting this when BQL is added is the right thing to do.

David Lang

Re: [Cake] faster scheduling, maybe

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

Preliminary benchmarks of new network APIs like netmap are showing 20mpps
 between guest and host for untrusted guests and 70mpps to trusted guests,
and scales near linearly with more cores. With that many pps per guest, why
not let the host do an AQM? High end service NICs support multiple virtual
devices where you can tell the NIC to evenly distribute bandwidth among the
virtual devices. It's already mostly a solved problem, just some people
reinventing the wheel. I know FreeBSD is currently looking at adding netmap
to connect the guest to the host so the guests can do line-rate 10Gb and
almost 40Gb.

On Mon, Jun 6, 2016 at 1:48 PM, David Lang <david@lang.hm> wrote:

> On Mon, 6 Jun 2016, Dave Taht wrote:
>
> http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf
>>
>
> I don't think so.
>
> They don't even try for fairness between flows, they are just looking at
> fairness between different VMs. they tell a VM that it has complete access
> to the NIC for a time, then give another VM complete access to the NIC. At
> best they put each VMs traffic into a different hardware queue in the NIC.
>
> This avoids all AQM decisions on the part of the host OS, because the
> packets never get to the host OS.
>
> The speed improvement is by bypassing the host OS and just having the VMs
> deliver packets directly to the NIC. This speeds things up, but at the cost
> of any coordination across VMs. Each VM can run fq_codel but it's much
> corser timeslicing between VMs.
>
> David Lang
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 2496 bytes --]

Re: [Cake] faster scheduling, maybe

[David Lang]

That works as long as the right answer is to evenly share the bandwidth between 
the VMs.

But what if one VM is running a single elephant flow while another VM is a VoIP 
server with hundreds of connections? Is it still correct to split the bandwidth 
evenly between the two?

David Lang

On Mon, 6 Jun 2016, Benjamin Cronce wrote:

> Preliminary benchmarks of new network APIs like netmap are showing 20mpps
> between guest and host for untrusted guests and 70mpps to trusted guests,
> and scales near linearly with more cores. With that many pps per guest, why
> not let the host do an AQM? High end service NICs support multiple virtual
> devices where you can tell the NIC to evenly distribute bandwidth among the
> virtual devices. It's already mostly a solved problem, just some people
> reinventing the wheel. I know FreeBSD is currently looking at adding netmap
> to connect the guest to the host so the guests can do line-rate 10Gb and
> almost 40Gb.
>
> On Mon, Jun 6, 2016 at 1:48 PM, David Lang <david@lang.hm> wrote:
>
>> On Mon, 6 Jun 2016, Dave Taht wrote:
>>
>> http://info.iet.unipi.it/~luigi/papers/20160511-mysched-preprint.pdf
>>>
>>
>> I don't think so.
>>
>> They don't even try for fairness between flows, they are just looking at
>> fairness between different VMs. they tell a VM that it has complete access
>> to the NIC for a time, then give another VM complete access to the NIC. At
>> best they put each VMs traffic into a different hardware queue in the NIC.
>>
>> This avoids all AQM decisions on the part of the host OS, because the
>> packets never get to the host OS.
>>
>> The speed improvement is by bypassing the host OS and just having the VMs
>> deliver packets directly to the NIC. This speeds things up, but at the cost
>> of any coordination across VMs. Each VM can run fq_codel but it's much
>> corser timeslicing between VMs.
>>
>> David Lang
>>
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
>>
>