Discussion of explicit congestion notification's impact on the Internet
 help / color / mirror / Atom feed
From: Neal Cardwell <ncardwell@google.com>
To: Dave Taht <dave.taht@gmail.com>
Cc: ECN-Sane <ecn-sane@lists.bufferbloat.net>,
	 BBR Development <bbr-dev@googlegroups.com>,
	flent-users <flent-users@flent.org>
Subject: Re: [Ecn-sane] [bbr-dev] duplicating the BBRv2 tests at iccrg in flent?
Date: Fri, 5 Apr 2019 11:10:48 -0400	[thread overview]
Message-ID: <CADVnQy=DfST=dHFkZg9EeQRL0OH9HOqgRfJ0uWgUu4fBLD9tSA@mail.gmail.com> (raw)
In-Reply-To: <CAA93jw6F0QPzKE-M6H_WOQ1EfM+6L5XEL_a+uEMYdU+OByVUaQ@mail.gmail.com>


[-- Attachment #1.1: Type: text/plain, Size: 3542 bytes --]

On Fri, Apr 5, 2019 at 3:42 AM Dave Taht <dave.taht@gmail.com> wrote:

> I see from the iccrg preso at 7 minutes 55 s in, that there is a test
> described as:
>
> 20 BBRv2 flows
> starting each 100ms, 1G, 1ms
> Linux codel with ECN ce_threshold at 242us sojurn time.
>

Hi, Dave! Thanks for your e-mail.


> I interpret this as
>
> 20 flows, starting 100ms apart
> on a 1G link
> with a 1ms transit time
> and linux codel with ce_threshold 242us
>

Yes, except the 1ms is end-to-end two-way propagation time.


> 0) This is iperf? There is no crypto?
>

Each flow is a netperf TCP stream, with no crypto.


>
> 1) "sojourn time" not as as setting the codel target to 242us?
>
> I tend to mentally tie the concept of sojourn time to the target
> variable, not ce_threshold
>

Right. I didn't mean setting the codel target to 242us. Where the slide
says "Linux codel with ECN ce_threshold at 242us sojourn time" I literally
mean a Linux machine with a codel qdisc configured as:

  codel ce_threshold 242us

This is using the ce_threshold feature added in:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80ba92fa1a92dea1

... for which the commit message says:

"A DCTCP enabled egress port simply have a queue occupancy threshold
above which ECT packets get CE mark. In codel language this translates to a
sojourn time, so that one doesn't have to worry about bytes or bandwidth
but delays."

The 242us comes from the seriailization delay for 20 packets at 1Gbps.

2) In our current SCE work we have repurposed ce_threshold to do sce
> instead (to save on cpu and also to make it possible to fiddle without
> making a userspace api change). Should we instead create a separate
> sce_threshold option to allow for backward compatible usage?
>

Yes, you would need to maintain the semantics of ce_threshold for backwards
compatibility for users who are relying on the current semantics. IMHO your
suggestion to use a separate sce_threshold sounds like the way to go, if
adding SCE to qdiscs in Linux.


> 3) Transit time on your typical 1G link is actually 13us for a big
> packet, why 1ms?
>

The 1ms is the path two-way propagation delay ("min RTT"). We run a range
of RTTs in our tests, and the graph happens to be for an RTT of 1ms.


> is that 1ms from netem?
>

Yes.


> 4) What is the topology here?
>
> host -> qdisc -> wire -> host?
>
> host -> qdisc -> wire -> router -> host?
>

Those two won't work with Linux TCP, because putting the qdisc on the
sender pulls the qdisc delays inside the TSQ control loop, giving a
behavior very different from reality (even CUBIC won't bloat if the network
emulation qdiscs are on the sender host).

What we use for our testing is:

  host -> wire -> qdiscs -> host

Where "qdiscs" includes netem and whatever AQM is in use, if any.


> 5) What was the result with fq_codel instead?
>

With fq_codel and the same ECN marking threshold (fq_codel ce_threshold
242us), we see slightly smoother fairness properties (not surprising) but
with slightly higher latency.

The basic summary:

retransmits: 0
flow throughput: [46.77 .. 51.48]
RTT samples at various percentiles:
  %   | RTT (ms)
------+---------
   0    1.009
  50    1.334
  60    1.416
  70    1.493
  80    1.569
  90    1.655
  95    1.725
  99    1.902
  99.9  2.328
 100    6.414

Bandwidth share graphs are attached. (Hopefully the graphs will make it
through various lists; if not, you can check the bbr-dev group thread
<https://groups.google.com/d/msg/bbr-dev/34nWGWLUjp4/nBxQRV01BgAJ>.)

best,
neal

[-- Attachment #1.2: Type: text/html, Size: 7205 bytes --]

[-- Attachment #2: bbr-v2-ecn-fq_codel-bw-individual.png --]
[-- Type: image/png, Size: 26678 bytes --]

[-- Attachment #3: bbr-v2-ecn-fq_codel-bw-cum.png --]
[-- Type: image/png, Size: 81688 bytes --]

  reply	other threads:[~2019-04-05 15:11 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-05  7:42 [Ecn-sane] " Dave Taht
2019-04-05 15:10 ` Neal Cardwell [this message]
2019-04-05 15:51   ` [Ecn-sane] [bbr-dev] " Dave Taht
2019-04-05 16:58     ` Neal Cardwell
2019-04-05 16:20   ` Jonathan Morton
2019-04-06 11:56     ` Neal Cardwell
2019-04-06 14:37       ` [Ecn-sane] [Flent-users] " Sebastian Moeller
2019-04-09  1:33         ` Neal Cardwell
2019-04-09  2:09           ` Jonathan Morton
2019-04-09  6:30           ` Sebastian Moeller
2019-04-09 14:33             ` Neal Cardwell
2019-04-09 17:20               ` Sebastian Moeller
2019-04-06 11:49   ` [Ecn-sane] " Dave Taht
2019-04-06 12:31     ` Neal Cardwell

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.bufferbloat.net/postorius/lists/ecn-sane.lists.bufferbloat.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CADVnQy=DfST=dHFkZg9EeQRL0OH9HOqgRfJ0uWgUu4fBLD9tSA@mail.gmail.com' \
    --to=ncardwell@google.com \
    --cc=bbr-dev@googlegroups.com \
    --cc=dave.taht@gmail.com \
    --cc=ecn-sane@lists.bufferbloat.net \
    --cc=flent-users@flent.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox