[Bloat] new network deploy

[Bloat] new network deploy

[Luca Dionisi]

Hi all

I am starting to deploy a local TCP/IP network in order to test a new
routing protocol.
Info on the routing protocol: www.netsukuku.org
Info on the network being deployed: pyntk.blogspot.com
It is separated from the Internet at the moment.

I ask for advices on how to make it as free as possible from the
bufferbloat problem.
I would also make some tests on this network and report here what I
find out about bufferbloat.

Further information:
 * every node active on the network will be a modern linux host.
 * all of the links between nodes will be on wifi or on cables owned
by the users. That is, no VPN links over the Internet.
   * if it is needed I can list the actual chipset of ethernet cards
and radio devices.
 * most of the wifi links will be managed by radio chipset installed
in linux hosts (mostly ad-hoc, no strange do-it-all out-of-control
embedded-router-bridge-repeater devices)

There is also a particular case. Two linux nodes are connected via a
VPN at layer 2 (I use tinc) over another network with static addresses
and routing. This network is formed by the 2 linux hosts and 2
embedded radio devices, from Ubiquity, which run AirOS, a derived of
OpenWRT.
In this particular case, where do I have to look for possible buffers?
Is it enough to look the tx-queue of the virtual nic or do I have to
check also the settings of the Ubiquity devices?

--Luca

Re: [Bloat] new network deploy

[Jim Gettys]

On 01/31/2011 04:52 AM, Luca Dionisi wrote:
> Hi all
>
> I am starting to deploy a local TCP/IP network in order to test a new
> routing protocol.
> Info on the routing protocol: www.netsukuku.org
> Info on the network being deployed: pyntk.blogspot.com
> It is separated from the Internet at the moment.
>
> I ask for advices on how to make it as free as possible from the
> bufferbloat problem.
> I would also make some tests on this network and report here what I
> find out about bufferbloat.
>
> Further information:
>   * every node active on the network will be a modern linux host.
>   * all of the links between nodes will be on wifi or on cables owned
> by the users. That is, no VPN links over the Internet.
>     * if it is needed I can list the actual chipset of ethernet cards
> and radio devices.
>   * most of the wifi links will be managed by radio chipset installed
> in linux hosts (mostly ad-hoc, no strange do-it-all out-of-control
> embedded-router-bridge-repeater devices)
>
> There is also a particular case. Two linux nodes are connected via a
> VPN at layer 2 (I use tinc) over another network with static addresses
> and routing. This network is formed by the 2 linux hosts and 2
> embedded radio devices, from Ubiquity, which run AirOS, a derived of
> OpenWRT.
> In this particular case, where do I have to look for possible buffers?
> Is it enough to look the tx-queue of the virtual nic or do I have to
> check also the settings of the Ubiquity devices?
>
>
Everywhere; here are places where we know buffers hide (on some hardware 
or an other).

o transmit queue
o ring buffers in the hardware

Additionally, using the OLPC wireless as an example, it also buffered:
o one packet in the device driver itself (aiding locking issues)
o four packets out in its mesh wireless module.

So I'd start by making sure I understood where buffering is in the 
system, and putting some reasonable upper bound on their size.

As to solving it for real, here you have a more difficult problem.  We 
don't have existing algorithms that "just work" for wireless.  RED won't 
suffice; the situation is too dynamic, and the dynamic range of the 
goodput on wireless, particularly in a mesh, is orders of magnitude, so 
any static upper bound, which may be much better than nothing, will 
still often be much too large.

There are two ways forward there toward real solutions.

1) is the unpublished nRED algorithm Van thinks will work. I will 
continue to nag Van Jacobson about finishing his paper up.

2) the following is running code that Van came across in December.

On 01/06/2011 01:50 PM, Van Jacobson wrote:
 > Jim,
 >
 > Here's the Doug Leith paper I mentioned. As I said on the phone I
 > think there's an easier, more robust way to accomplish the same
 > thing but they have running code and I don't. You can get their
 > mad-wifi implementation at
 > http://www.hamilton.ie/tianji_li/buffersizing.html
 >
 >   - van

Best regards,
			- Jim

Re: [Bloat] new network deploy

[Dave Täht]

Luca Dionisi <luca.dionisi@gmail.com> writes:

> Hi all
>
> I am starting to deploy a local TCP/IP network in order to test a new
> routing protocol.
> Info on the routing protocol: www.netsukuku.org
> Info on the network being deployed: pyntk.blogspot.com
> It is separated from the Internet at the moment.

Very cool!

>
> I ask for advices on how to make it as free as possible from the
> bufferbloat problem.
> I would also make some tests on this network and report here what I
> find out about bufferbloat.
>
> Further information:
>  * every node active on the network will be a modern linux host.

You are in for a world of bufferbloat.

>  * all of the links between nodes will be on wifi or on cables owned
> by the users. That is, no VPN links over the Internet.
>    * if it is needed I can list the actual chipset of ethernet cards
> and radio devices.

That would be very helpful!

>  * most of the wifi links will be managed by radio chipset installed
> in linux hosts (mostly ad-hoc, no strange do-it-all out-of-control
> embedded-router-bridge-repeater devices)

I note you can improve multicast performance by going with greenfield
settings.

>
> There is also a particular case. Two linux nodes are connected via a
> VPN at layer 2 (I use tinc) over another network with static addresses
> and routing. This network is formed by the 2 linux hosts and 2
> embedded radio devices, from Ubiquity, which run AirOS, a derived of
> OpenWRT.

I was using the ubiquity M5 radios in my mesh design (wisp6). 
The firmware Ubiquity was using used the 2.6.15 kernel (as best as I
recall) and I could gain no insight into the device drivers themselves
at the time. 

Their (deservedly lauded) traffic shaping solution had some really hairy
looking tc code, which I looked at only briefly, before discarding AirOs
in favor of Openwrt (because I needed IPv6)

And then I promptly shot myself in the foot, with bufferbloat.

> In this particular case, where do I have to look for possible buffers?
> Is it enough to look the tx-queue of the virtual nic or do I have to
> check also the settings of the Ubiquity devices?

Data accumulates at the point of the biggest buffer, under contention,
so I would check all of them! The interaction of virtual nics and real
nics has not been thoroughly explored.

I'm trying to work on clearly explaining how to test for bufferbloat in
the wiki, Jim Gettys' experiments on his blog are a good place to start,
however I use iperf and mtr rather than ntcp and ping because iperf is
in the openwrt repository.

http://gettys.wordpress.com/2010/11/29/home-router-puzzle-piece-one-fun-with-your-switch/

http://gettys.wordpress.com/2010/12/02/home-router-puzzle-piece-two-fun-with-wireless/

-- 
Dave Taht
http://nex-6.taht.net