General list for discussing Bufferbloat
 help / color / mirror / Atom feed
From: "George B." <georgeb@gmail.com>
To: Jonathan Morton <chromatix99@gmail.com>
Cc: "bloat@lists.bufferbloat.net" <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] philosophical question
Date: Tue, 31 May 2011 10:20:54 -0700	[thread overview]
Message-ID: <BANLkTikaiMmq+DZYa6ZKd0=t0YopVkCTLA@mail.gmail.com> (raw)
In-Reply-To: <0E55287D-41D8-4CE7-8AD8-493A874B80D7@gmail.com>

On Mon, May 30, 2011 at 8:57 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> If most of your clients are mobile, you should use a tcp congestion control algorithm such as Westwood+ which is designed for the task. This is designed to distinguish between congestion and random packet losses. It is much less aggressive at filling buffers than the default CUBIC.

Not only are they mobile, the behavior might be considered like that
of a "thin" client in the context of "tcp-thin"

http://www.mjmwired.net/kernel/Documentation/networking/tcp-thin.txt

So the servers have two different sorts of connections.  There would
be thousands of long-lived connections with only an occasional packet
going back and forth.  Those streams are on lossy mobile networks.
Then there are several hundred very "fat" and fast connections moving
a lot of data.  Sometimes a client might change from a "thin" stream
to a "thick" stream if it must collect a lot of content.

So maybe westwood+ along with the "tcp-thin" settings in 2.6.38 might
be a good idea.

Looking at one server this morning, I have about 27,000 TCP
connections in ESTABLISHED state.  Many (most) of these are "thin"
flows to devices that exchange a packet only occasionally.  The server
has 16 cores talking to 2 GigE NICs with 8 queues each.  There is
about 40 meg of traffic flowing into the server from the network and
the outbound bandwidth is about 8 meg/sec.  Much of that 8 meg (about
2.5 meg) is logging traffic going to a local log host via IPv6 + jumbo
frames.

The way this is configured is there are two NICs, eth0 and eth1 and
three vlans, lets call them vlan 2 (front end traffic) vlan 3 (logging
traffic) and vlan 4 (backend traffic).  VLAN 2 would be configured on
the NICs (eth0.2 and eth1.2) and then bonded using balance-xor using
the layer2-3 xmit hash.  This way a given flow should always hash to a
given vlan interface on a particular NIC.  So I have three bond
interfaces talking to a multiqueue aware vlan driver.  This allows one
processor to handle log traffic while a different processor handles
front-end traffic and another processor handing a backend transaction
at the same time.

The higher inbound to outbound ratio is backwards from your
traditional server profile but that is because the traffic that comes
in gets compressed and sent back out and it is mostly text and text
compresses nicely.

/proc/sys/net/ipv4/tcp_ecn is currently set to "2" meaning use ECN if
I see ECN set from the other end but don't initiate connections with
ECN set.   As I never initiate connections to the clients, that really
isn't an issue.  The client always initiates the connection so if I
see ECN, it will be used.

So the exercise I am going through is trying to determine the best
qdisc and where to put it.  On the bond interfaces, on the vlan
interfaces or on the NICs?  Something simple like SFQ would probably
work ok.  I just want to make sure a single packet to the client can
get through without a lot of delay in the face of a "fat" stream going
somewhere else.  Currently it is using the default mq qdisc:

root@foo:~>  tc -s qdisc show
qdisc mq 0: dev eth0 root
 Sent 5038416030858 bytes 1039959904 pkt (dropped 0, overlimits 0
requeues 24686)
 rate 0bit 0pps backlog 0b 0p requeues 24686
qdisc mq 0: dev eth1 root
 Sent 1380477553077 bytes 2131951760 pkt (dropped 0, overlimits 0 requeues 2934)
 rate 0bit 0pps backlog 0b 0p requeues 2934

So there have been no packets dropped and there is no backlog and the
path is clean all the way to the Internet without any congestion in my
network (the path is currently about 5 times bigger than current
bandwidth utilization and is 10GigE all the way from the switch to
which the server is connected all the way to the Internet).  Any
congestion would be somewhere upstream from me.

Suggestions?

George

  reply	other threads:[~2011-05-31 17:04 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-05-30  4:24 George B.
2011-05-30  7:53 ` Neil Davies
2011-05-30 12:25 ` Dave Taht
2011-05-30 15:29   ` George B.
2011-05-30 15:57     ` Jonathan Morton
2011-05-31 17:20       ` George B. [this message]
2011-05-31 21:40         ` Juliusz Chroboczek
2011-05-30 17:05     ` Dave Taht
2011-05-31 18:07 ` Bill Sommerfeld
2011-05-31 19:17   ` Rick Jones
2011-05-31 19:19     ` Jim Gettys
2011-05-31 19:28   ` George B.

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='BANLkTikaiMmq+DZYa6ZKd0=t0YopVkCTLA@mail.gmail.com' \
    --to=georgeb@gmail.com \
    --cc=bloat@lists.bufferbloat.net \
    --cc=chromatix99@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox