Re: [Bloat] Analyzing wireless multiqueue behavior & bufferbloat

Re: [Bloat] Analyzing wireless multiqueue behavior & bufferbloat

[Jean Tourrilhes]

	Hi all,

	Dave Täht asked me to give my two cents about interraction
between Wireless and TCP/IP.

	My first reaction is "here we go, again". This subject has
been beaten to death by the experts in the past, for example the IETF
had a PILC subgroup, it generated a couple of intertesting documents
on the subject, and the mailing list has interesting discussion.
	The PILC workgroup :
		http://datatracker.ietf.org/wg/pilc/charter/
		http://www.isi.edu/pilc/
		http://tools.ietf.org/html/draft-ietf-pilc-error-06
	My humble contribution on the mailing list :
		http://www.ietf.org/mail-archive/web/pilc/current/msg00211.html
	I did not go deep in the Bloat discussion, I don't have time
to read the mailing list, so I don't know if there is something truly
new apart from a cute marketing name.

	My personal view on the subject of TCP/IP over Wireless is
that the characteristics of the physical layer will dictate what is
possible, and without extensive knowledge and characterisation of said
physical layer, there is no point having a discussion.
	The Wireless layer is very peculiar, and to use it efficiently
you need to be very intimate with it. I don't think it's a good idea
to make TCP/IP aware of things like RSSI and phy speed on every hop,
so the MAC has to encapsulate all that (the wireless hop may be in the
middle of a multi-hop link). Obviously, we would like the MAC layer to
help TCP/IP rather than get in the way. The main way to achieve that
is for the MAC timing to be way below the TCP/IP timings. Also, UDP
and TCP have different characteristics and both need to be supported.

	In my mind, the two main point is how should TCP/IP deal with
fading and bursty interferers. Forget about the other sideshow.
	A gross approximation of fading is that every 100ms the
characteristic of the link change drastically. Sometimes it is faster,
sometimes slower, sometime no packet can go through. If you link is
very poor for 100ms due to fading, what should TCP/IP do ?
	The 2.4 GHz and 5 GHz bands are unlicensed, which mean that
there can be quite a few potential interferers. I personally looked at
micorowave ovens and Bluetooth interference on 802.11, but you can
also have cordless phones, baby monitors, ZigBee and maybe wireless
USB. Very often, those interferers will result in bursty losses, that
the MAC cover by retrying various packet for a long time, what should
TCP/IP do ?

Dave Täht wrote :
> 
> I've put some thought into the history of my use of proxies over
> wireless - re-analyzing something that I'd incorporated into my gut back
> in 1996.
> 
> The "split TCP" smoothing effect on the last mile I'd noticed *then* was
> minor; the time savings were dominated by the latencies in the modem of
> 100ms or so. [2]
> 
> The mosquitonet research[3] had clearly shown the disastrous effects on
> un-error-corrected wireless on TCP.

	There are many papers dealing with the subject, see at the
end...

> Even then, we regarded 1-3% packet loss as acceptable.

	Is that loss above Phy, or loss above the MAC layer ?
	If it's above the Phy, that would be what I expect, but I'd
also say the Phy and the MAC not very agressive in trying to use the
medium. If it's above the MAC, then I would consider the MAC broken,
unless the link conditions are really bad.

> 1) Wireless Packet aggregation is now becoming more common, leading
> to bursty behavior for short packets. What sorts of packets are
> being aggregated? How much buffering is in place? I worry.

	There is many way to do Packet aggregation. Some add minimal
delay in presence of errors, some take time to detect errors.

> 2) I think I can theorize that starting 8 TCP connections in
> a browser, rather than 2, also "smooths" out bursty packet loss on
> the wireless last hop.

	I sincerly hope that your 802.11 link does not loose packets
and only adds trivial delay with a good ARQ mechanism.

> browsers still look for wpad

	Do they ? I always found support pretty lacking.

> I imagine, if, for example, apple would use these techniques on their
> tightly integrated wireless gateways and apple tv - that it would
> provide a competitive advantage. 
> 
> The same goes for other manufacturers.

	The problem is that there is no money to be made selling $50
wireless gateways and $15 Mini-PCI NIC, so I don't see who would be
driving innovation in Wireless LAN space.
	Note that one of the problem of those gateways is that they
are software devices, and therefore need buffers to handle the
inherent latencies between the NIC and the CPU. Real Time handling of
incomming network packet would be needed to decrease those buffers.

On Fri, Feb 11, 2011 at 08:23:11AM -0700, Dave Täht wrote:
> There also seems to be an impedance mismatch between the
> availability of various theoretical QoS mechanisms in the 802.11
> standards and their actual implementations in the networking
> stacks. We're treating wireless too much like ethernet to the
> detriment of the entire internet.

	Last time I checked (which is a long time ago), the Intel guys
(iwlwifi) were the most advances in term of QoS support in Wireless.

> High on my list is making TX_RETRY a tunable.

	Has always been available in Wireless Extensions since day
one, for obvious reasons. Not many driver support it.
	Note that the optimal retry mechanism is not based on number
of retry, but on elapsed time. Obviously, no driver that I know
implement that :-(

> And I'm deeply concerned about possible side-effects of packet
> aggregation in 802.11n.

	There is many type of packet aggregation in 802.11 and
proprietary schemes, and they should behave differently.

> I've also been unable to dig up the original mosquitonet paper - or
> something more current - that determined the need for link layer error
> correction/retransmits in the first place.

	The reference I find truly useful, I've added to the papers
I published on the subject :
	http://www.hpl.hp.com/personal/Jean_Tourrilhes/Papers/papers.html
	Those are old papers, but the references should be still relevant.

	A paper that I don't cite, but was a big hit in its time was
the Berkely Snoop Protocol :
		http://nms.lcs.mit.edu/~hari/papers/snoop.html
	Ironically, this is the Snoop protocol that convinced me that
the right way was to design the appropriate ARQ mechanisms in the link
layer, rather than using Split-TCP approaches, as long as the ARQ was
immediate so that its timing would not be perceived by TCP/IP.
	If you look at Snoop and understand what it does, your gut
reaction should be, yeah, we can do much simpler by doing that at the
MAC level.

	Personally, I think you really should read the following paper
which I consider one of the definitive answer on the subject :

[3] Hari Balakrishnan, Venkata N. Padmanabhan, Srinivasan Seshan and Randy H. Katz. A comparison of mechanisms for improving TCP performance over wireless links. Proc. ACM SIGCOMM Conference, Stanford, CA, USA, Aug 1996. 

	Remember that their timeout for LL are very coarse compared to
what a typical MAC would do, which explain why they have to play with
TCP acks. And I think they deal pretty adequately with all the Proxy
approaches ;-)

	Have fun...

	Jean

Re: [Bloat] Analyzing wireless multiqueue behavior & bufferbloat

[Jean Tourrilhes]

On Fri, Feb 11, 2011 at 09:55:18AM -0800, Jean Tourrilhes wrote:
> 
> 	My first reaction is "here we go, again". This subject has
> been beaten to death by the experts in the past, for example the IETF
> had a PILC subgroup, it generated a couple of intertesting documents
> on the subject, and the mailing list has interesting discussion.
> 	The PILC workgroup :
> 		http://datatracker.ietf.org/wg/pilc/charter/
> 		http://www.isi.edu/pilc/
> 		http://tools.ietf.org/html/draft-ietf-pilc-error-06
> 	My humble contribution on the mailing list :
> 		http://www.ietf.org/mail-archive/web/pilc/current/msg00211.html
> 	I did not go deep in the Bloat discussion, I don't have time
> to read the mailing list, so I don't know if there is something truly
> new apart from a cute marketing name.

	I forgot to mention that most of my comments were specific to
"Wireless over TCP/IP", and not the overall buffering issues.
	For buffering, see section 13 of this PILC document :
		http://tools.ietf.org/html/draft-ietf-pilc-link-design-15

	Regards,

	Jean

Re: [Bloat] Analyzing wireless multiqueue behavior & bufferbloat

[Dave Täht]

(I note that although my personal focus is on wireless, that bufferbloat
is a current problem in many/most other types of devices, and I do not
want the bufferbloat discussion dominated by wireless issues.)

I suppose it would help others a little bit if others knew the formerly
private email jean was responding to:

I'd written:

All:

I've (as yet) been unable to dig up anything that coherently explains
how the Linux mq and multiq qdiscs (if indeed, they are different) are
supposed to be hooked up to the wireless driver portion of the stack,
and managed. 

There also seems to be an impedance mismatch between the availability of
various theoretical QoS mechanisms in the 802.11 standards and their
actual implementations in the networking stacks. We're treating wireless
too much like ethernet to the detriment of the entire internet.

I have no insight at all as to how things are done on Apple products.

High on my list is making TX_RETRY a tunable.

And I'm deeply concerned about possible side-effects of packet
aggregation in 802.11n.

I've also been unable to dig up the original mosquitonet paper - or
something more current - that determined the need for link layer error
correction/retransmits in the first place.

Perhaps you guys could ask around for more/better information on these
issues and get back on them to the bloat list? [1]

-- 
Dave Taht
http://nex-6.taht.net