Re: [Ecn-sane] [bbr-dev] duplicating the BBRv2 tests at iccrg in flent?

[Neal Cardwell <ncardwell@google.com>]

[-- Attachment #1: Type: text/plain, Size: 3236 bytes --]

On Sat, Apr 6, 2019 at 7:49 AM Dave Taht <dave.taht@gmail.com> wrote:

> > With fq_codel and the same ECN marking threshold (fq_codel ce_threshold
> 242us), we see slightly smoother fairness properties (not surprising) but
> with slightly higher latency.
> >
> > The basic summary:
> >
> > retransmits: 0
> > flow throughput: [46.77 .. 51.48]
> > RTT samples at various percentiles:
> >   %   | RTT (ms)
> > ------+---------
> >    0    1.009
> >   50    1.334
> >   60    1.416
> >   70    1.493
> >   80    1.569
> >   90    1.655
> >   95    1.725
> >   99    1.902
> >   99.9  2.328
> >  100    6.414
>
> I am still trying to decode the output of this tool. Perhaps you can
> check my math?
>
> At 0, I figure this is the bare minimum RTT latency measurement from
> the first flow started, basically a syn/syn ack pair, yielding 9us, of
> which gives a tiny baseline to/from the wire delay, the overhead of a
> tcp connection getting going, and the routing overheads (if you are
> using a route table?) of (last I looked) ~25ns per virtual hop (so *8)
> for ipv4 and something much larger if you are using ipv6. This is
> using ipv4?
>

This is using IPv6.


> A perfect level of interleave of 20 flows, 2 large packets each, would
> yield a RTT measurement of around 532 extra us, but you are only
> seeing that at the 80th percentile...
>

The flows would only get ~500us extra delay above the two-way propagation
delay if all of those packets ended up in the queue at the same time. But
the BDP here is 1Gbps*1ms = 82 packets, and there are 20 flows, so for
periods where the flows keep their steady-state inflight around 4 or
smaller, their aggregate inflight will be around 4*20 = 80, which is below
the BDP, so there is no queue. In a steady-state test like this the ECN
signal allows the flows to usually keep their inflight close to this level,
so that the higher queues and queuing delays only happen when some or all
of the flows are pushing up their inflight to probe for bandwidth around
the same time. For a scenario like this, those dynamics are not unique to
BBRv2; in a scenario like this, at a high level similar reasoning would
apply to DCTCP or TCP Prague.


> 100: The almost-worst case basic scenario is that 20 flows of 64k
> GRO/GSO bytes each are served in order. That's 13us * 42 * 20 =
> 10.920ms. It's potentially worse than that due to the laws of
> probability one flow could get scheduled more than once.
>

Keep in mind that there's no reason why the flows would use maximally-sized
64KByte GSO packets in this test. The typical pacing rate for the flows is
close to the throughput of 50Mbps, and  the TCP/BBR TSO autosizing code
will tend to choose GSO skbs with around 1ms of data (n the 24-540Mbps
range), and at at 50Mbps this 1ms allotment is about 4*MSS.


>
> 1) What is the qdisc on the server and client side? sch_fq? pfifo_fast?
>

Sender and receiver qdiscs are pfifo_fast


> 2) What happens when you run the same test with gso/gro disabled?
>

Disabling GSO and GRO is not a realistic config, and we have limited
developer resources on the BBR project, so I'm not going to have time to
run that kind of test (sorry!). Perhaps you can run that test after we
open-source BBRv2.

best,
neal

[-- Attachment #2: Type: text/html, Size: 4518 bytes --]