[Bloat] iperf3 and packet bursts

[Bloat] iperf3 and packet bursts

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

We were using iperf3 at work to test a network path (we wanted the
fixed-rate throttling that it can do).  And while TCP would fill the 50Mbps
link from the ISP, UDP couldn't.  UDP couldn't get over 8Mbps of goodput,
no matter what rate we specified on the command line.

We found a 100ms timer that's used to PWM the packet transmission to
perform the throttling.  Fine for TCP, fine where the end-to-end physical
links are the same rate.  But throw a 10:1 rate change through a switch
into that, and suddenly you find out that the switch isn't that bloated.

I modified iperf3 to use a 1ms timer, and was able to get things much
smoother.  I doubt it's as smooth as iperf3 gets on Linux when fq pacing is
used, but it's a big improvement vs. the nice small buffers in switches.

I put together a writeup with graphs on my blog:
http://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html

I have a forked version of iperf3 on github:
https://github.com/woody77/iperf

This uses the 1ms timer, and has a few other fixes, such as it resets the
throttling calculations at the start of each stats interval.  That change
stops iperf3 from transmitting at maximum rate after congestion has stopped
it from achieving the target rate.  There will be another writeup on that,
but I need to get some good sample data together for graphing.

-Aaron Wood

[-- Attachment #2: Type: text/html, Size: 1703 bytes --]

Re: [Bloat] iperf3 and packet bursts

[Dave Täht]

Groovy. I note that I am really fond of the linux "fdtimer" notion for
tickers, we use that throughout the high speed stats gathering code in
flent.

I'd really like a voip or ping tool that used those, and I've always
worried about iperf's internal notion of a sampling interval.

On 9/20/16 3:00 PM, Aaron Wood wrote:
> We were using iperf3 at work to test a network path (we wanted the
> fixed-rate throttling that it can do).  And while TCP would fill the
> 50Mbps link from the ISP, UDP couldn't.  UDP couldn't get over 8Mbps of
> goodput, no matter what rate we specified on the command line.
> 
> We found a 100ms timer that's used to PWM the packet transmission to
> perform the throttling.  Fine for TCP, fine where the end-to-end
> physical links are the same rate.  But throw a 10:1 rate change through
> a switch into that, and suddenly you find out that the switch isn't that
> bloated.
> 
> I modified iperf3 to use a 1ms timer, and was able to get things much
> smoother.  I doubt it's as smooth as iperf3 gets on Linux when fq pacing
> is used, but it's a big improvement vs. the nice small buffers in switches.
> 
> I put together a writeup with graphs on my blog:
> http://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html
> 
> I have a forked version of iperf3 on github:
> https://github.com/woody77/iperf
> 
> This uses the 1ms timer, and has a few other fixes, such as it resets
> the throttling calculations at the start of each stats interval.  That
> change stops iperf3 from transmitting at maximum rate after congestion
> has stopped it from achieving the target rate.  There will be another
> writeup on that, but I need to get some good sample data together for
> graphing.
> 
> -Aaron Wood
> 
> 
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 

Re: [Bloat] iperf3 and packet bursts

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Tue, Sep 20, 2016 at 3:03 PM, Dave Täht <dave@taht.net> wrote:

> Groovy. I note that I am really fond of the linux "fdtimer" notion for
> tickers, we use that throughout the high speed stats gathering code in
> flent.
>
> I'd really like a voip or ping tool that used those, and I've always
> worried about iperf's internal notion of a sampling interval.
>
> On 9/20/16 3:00 PM, Aaron Wood wrote:
> > I modified iperf3 to use a 1ms timer, and was able to get things much
> > smoother.  I doubt it's as smooth as iperf3 gets on Linux when fq pacing
> > is used, but it's a big improvement vs. the nice small buffers in
> switches.
>

Thanks!

For rates of <1000 packets per second, the 1ms timer I put in will give
something like that (it fires every ms, but that's just a check for sending
or not).  If you want to use it to model a 120pps flow of 64-byte packets:

iperf3 -c <host> -u -l 64 -b 61440

And then it will pace those out at roughly every 80ms (just verified this
on my box)

-Aaron

[-- Attachment #2: Type: text/html, Size: 1583 bytes --]

Re: [Bloat] iperf3 and packet bursts

[Aaron Wood]

[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]

And I just pushed a branch that calculates a timer interval based on the
throttle rate and the buffer size, with a cap on the timer frequency that
can be changed via the command-line.  It _seems_ to work well, and give
really smooth pacing.  Incredibly smooth pacing, actually (at least at 1ms
time bucket intervals).

The branch is here:  https://github.com/woody77/iperf/tree/pacing_timer

-Aaron

On Tue, Sep 20, 2016 at 4:32 PM, Aaron Wood <woody77@gmail.com> wrote:

> On Tue, Sep 20, 2016 at 3:03 PM, Dave Täht <dave@taht.net> wrote:
>
>> Groovy. I note that I am really fond of the linux "fdtimer" notion for
>> tickers, we use that throughout the high speed stats gathering code in
>> flent.
>>
>> I'd really like a voip or ping tool that used those, and I've always
>> worried about iperf's internal notion of a sampling interval.
>>
>> On 9/20/16 3:00 PM, Aaron Wood wrote:
>> > I modified iperf3 to use a 1ms timer, and was able to get things much
>> > smoother.  I doubt it's as smooth as iperf3 gets on Linux when fq pacing
>> > is used, but it's a big improvement vs. the nice small buffers in
>> switches.
>>
>
> Thanks!
>
> For rates of <1000 packets per second, the 1ms timer I put in will give
> something like that (it fires every ms, but that's just a check for sending
> or not).  If you want to use it to model a 120pps flow of 64-byte packets:
>
> iperf3 -c <host> -u -l 64 -b 61440
>
> And then it will pace those out at roughly every 80ms (just verified this
> on my box)
>
> -Aaron
>

[-- Attachment #1.2: Type: text/html, Size: 2591 bytes --]

[-- Attachment #2: sub-millisecond-pacing.png --]
[-- Type: image/png, Size: 41630 bytes --]