From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2.tohojo.dk (mail2.tohojo.dk [77.235.48.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 67B703B29F; Sat, 1 Oct 2016 07:51:54 -0400 (EDT) X-Virus-Scanned: amavisd-new at mail2.tohojo.dk DKIM-Filter: OpenDKIM Filter v2.10.3 mail2.tohojo.dk 433AD41622 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=201310; t=1475322711; bh=tf2QEvSxeY7vugFnMOolO/gBM5QA9dhKYZ2wIV2mbiI=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=JKTfAODPbrtgGssnWjf8mNhuvmgCcDFGfsFg9uIdJ0KsDALmN/DQavtXF/S/ePbAL dKZqk8VVIh8JjNk86RyLM1U6BAcC/YDbbgwzjDG4EC+0dZqi+CiIlc4IFty9fU/J3p 4mHk8QWyxzIlYfl1rEUlMqz8AvjPD4bdJfBl7l1Y= Sender: toke@toke.dk Received: by alrua-karlstad.karlstad.toke.dk (Postfix, from userid 1000) id 21D05869E97; Sat, 1 Oct 2016 13:51:50 +0200 (CEST) From: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: "Jason A. Donenfeld" Cc: cake@lists.bufferbloat.net, make-wifi-fast@lists.bufferbloat.net, WireGuard mailing list , Dave Taht References: Date: Sat, 01 Oct 2016 13:51:50 +0200 In-Reply-To: (Jason A. Donenfeld's message of "Sat, 1 Oct 2016 01:21:04 +0200") X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87twcw9tih.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Make-wifi-fast] [Cake] WireGuard Queuing, Bufferbloat, Performance, Latency, and related issues X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Oct 2016 11:51:54 -0000 "Jason A. Donenfeld" writes: > Hi all, > > On Fri, Sep 30, 2016 at 9:18 PM, Dave Taht wrote: >> All: I've always dreamed of a vpn that could fq and - when it was >> bottlenecking on cpu - throw away packets intelligently. Wireguard, >> which is what jason & co are working on, is a really simple, elegant >> set of newer vpn ideas that currently has a queuing model designed to >> optimize for multi-cpu encryption, and not, so much, for managing >> worst case network behaviors, or fairness, or working on lower end >> hardware. > > Would love any feedback and support for working on the queuing model > with WireGuard. I hear the bufferbloat folks are geniuses at that... Looking at your queueing structure description, I'd say the reusable FQ stuff is a pretty good fit. There's a lot of conceptual overlap between the mac80211 concept of having a queue per station, and yours of having a queue per peer. Have a look at include/net/fq.h and include/net/fq_impl.h - and see net/mac80211/tx.c for a usage example (grep for 'txq'). This can be further enhanced by using CoDel as the dequeue function (again, see mac80211 for an example). Basically, what this gives you is a set of per-flow queues that you can use to do fairness queueing between flows. These queues can then be partitioned into a number of 'tins', which in your use case would correspond to peers. This saves you from allocating a big set of queues for each peer, but you can still get flow-based FQ to that peer (including the nice latency prioritisation for sparse flows that FQ-CoDel also has). I think you could probably use the FQ structure as a drop-in replacement for the queue you have already in peer->tx_packet_queue. That would give you fairness between flows going to a peer, prioritisation of sparse flows to that peer, and CoDel to push back on flows when the VPN is the bottleneck. As far as the encryption step goes, I see two obvious ways of doing it: 1. Do what you are doing now, basically, i.e. just pull packets off the FQ, encrypt then, and send them out. This has the drawback of (potentially) being bursty, and it adds latency after the point where it can be controlled by CoDel. 2. Perform encryption on enqueue. Basically, submit a packet to encryption as soon as you get it, and when it finishes, stick it on the peer queue. Then move the dequeue operation to a separate step that just pulls from the queue. This would get you the benefit of having FQ and CoDel operate at the point right before the packets hit the wire. Option 2 is a bit more complicated in that it would need to deal with the case where no session is established. In this case, you could probably just stick the non-encrypted packets on the queue, then when the session is established, pull everything off the queue, encrypt it, and put it back. A flag somewhere could then keep track of whether packets on the queue are encrypted or not. You'd also need to deal with the fact that the FQ hashing doesn't work for encrypted packets. Probably this can be dealt with by hashing the packet before it is encrypted and storing the hash in an appropriate field somewhere (skb->cb?); then changing fq_impl.h to take a function pointer to override fq_flow_classify(). Another thing that ties into the above which I couldn't find in your description is how you schedule between peers? Is there a round-robin scheduler somewhere or something? -Toke