[Bloat] do you still exist?

[Jean Tourrilhes]

On Fri, Feb 25, 2011 at 09:49:31AM -0700, Dave Täht wrote:
> 
> I'm deeply concerned about the 10GigE switches coming out, too. The only
> one I looked at (the intel) seemed to be using absurd amounts of
> buffering. How is the HP gear in this respect?

	Actually, the switches themselves have very low latency, and
many are even cut-through. The ProCurve 5406zl switch, which I've
implemented OpenFlow on, is specified with a 2.1us latency for 64B
packets at 10GB/s (which is noise compared to the ping RTT measured
Linux->Linux). This switch implements WRED and should keep buffer
queues fairly small in presence of congestion.
	To me, the issue most often is in the NIC. Most NIC use large
transmit offload and large receive offload to cover the latency of the
NIC<->CPU channel. I had to tweak the NIC to hell to get acceptable
10GB/s performance with 1500 B packets (of course, much easier with
jumbo).
	Because the NIC is becoming so bursty, the switch need some
buffers to smooth out the bursts. If you look at the time picture, at
10 Gb/s, those buffers don't represent a huge amount of time,
especially compared to end to end RTT and TCP timeout. Lower cost
10Gb/s switches will likely have less buffers (for cost saving), and
remember that you also have to divide the buffer by the number of
ports to get an meaningful comparison.

	In my opinion, I think you should not be concerned about
switches, and more about residential gateways, and I'll explain why.
	There is a very healthy market for entreprise switches, with
many expensive and high end models. Because those high end entreprise
switches are expensive, a lot of R&D can be spent on optimising their
performance with TCP/IP traffic. Entreprise customers are
knowledgeable and demand great performance, and performance is one of
the main criteria, and the purchase cost of the device is somewhat
trivial compared to operational cost.
	Then, the technology developped for high end entreprise
switches naturally percolate to smaller switches and residential
switches. For example, the NetGear GS608 residential switch is
specified with 15us max latency, and 128k buffer (16k per port), but
I don't know if it implements RED or not.

	Residential gateway are a totally different market, there is
no money, it's all volume with razor thin margin, and the products are
renewed very frequently. There is no high end or entreprise version
driving the R&D. Consequently, the amount of engineering going into
those is limited, and this is highlighted by the fact that most
advanced users turn to third party firmware.
	Also, for those gateway, the number one priority is low cost,
followed by cheap price, and then maybe UI, security features and
power consumption. Nobody cares about performance and everybody will
blame to broadband provider anyway.
	Another issue is that those devices are implemented using
a CPU, and there are latency in the coupling of the NIC and the CPU,
due to the way the CPU and OS respond to I/Os. That's something
I alluded in an earlier e-mail.

	Have fun...

	Jean