[Bloat] Best practices for paced TCP on Linux?

[Dave Taht]

On Fri, Apr 6, 2012 at 2:37 PM, Steinar H. Gunderson
<sgunderson at bigfoot.com> wrote:
> Hi,
>
> This is only related to bloat, so bear with me if it's not 100% on-topic;
> I guess the list is about the best place on the Internet to get a reasonble
> answer for this anyway :-)

This is actually both very relevant and very hard to simulate. Having
someone willing
to try this in the real world would be excellent.

> Long story short, I have a Linux box (running 3.2.0 or so) with a 10Gbit/sec
> interface, streaming a large amount of video streams to external users,
> at 1Mbit/sec, 3Mbit/sec or 5Mbit/sec (different values). Unfortunately, even
> though there is no congestion in _our_ network (we have 190 Gbit/sec free!),
> some users are complaining that they can't keep the stream up.

Interesting. While there can be many causes I do tend to think that
10GigE is putting incredible pressure on downstream devices that
is very hard to observe, unless you are downstream...

>
> My guess is that this is because at 10Gbit/sec, we are crazy bursty, and
> somewhere along the line, there will be devices doing down conversion without
> enough buffers (for instance, I've seen drop behavior on Cisco 2960-S in a
> very real ISP network on 10->1 Gbit/sec down conversion, and I doubt it's the
> worst offender here).
>
> Is there anything I can do about this on my end? I looked around for paced
> TCP implementations, but couldn't find anything current. Can I somehow shape
> each TCP stream to 10Mbit/sec or so each with a combination of SFQ and TBF?
> (SFQRED?)

It would be best to get some packet captures from user's streams that are
complaining of the issue.

Most 10Gige network cards have the ability to have multiple hardware queues
that will do some FQ for you.

SFQ or QFQ will FQ the output streams. SFQRED or QFQ + something will be
able to do some level of queue management but not 'shaping', or 'packet pacing'

However in your environment you will need the beefed up SFQ that is in 3.3.
and BQL. If you are not saturating that 10GigE card, you can turn off TSO/GSO
as well.

You can reduce your tcp sent windows. There's a horde of other things you
can try, but it would be best to start of with a few measurements, both
on your side and at a customer's.


> I'm not very well versed in tc, so anything practical would be very much

I have a debloat script (with about 5 out of tree versions) that tries it's best
to do the right thing on home gear and takes a bit of the arcaneness out
of the mix. I have not got around to trying out a couple algos

> appreciated. Bonus points if we won't have to patch the kernel.

3.3.1 is out and in my environment is stable. YMMV.

> /* Steinar */
> --
> Homepage: http://www.sesse.net/
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net