General list for discussing Bufferbloat
 help / color / mirror / Atom feed
From: Dave Taht <dave.taht@gmail.com>
To: Jerry Jongerius <jerryj@duckware.com>
Cc: bloat <bloat@lists.bufferbloat.net>
Subject: Re: [Bloat] The Dark Problem with AQM in the Internet?
Date: Mon, 1 Sep 2014 10:40:34 -0700	[thread overview]
Message-ID: <CAA93jw6vc2+GHrm7sRL51tmk4VXAzRBQo-iu6dn+qOwrefzbqQ@mail.gmail.com> (raw)
In-Reply-To: <000101cfc60a$61f19a70$25d4cf50$@duckware.com>

On Mon, Sep 1, 2014 at 10:30 AM, Jerry Jongerius <jerryj@duckware.com> wrote:
> Westwood+, as described in published researched papers, does not fully
> explain the graph that was seen.  However, Westwood+, as implemented in
> Linux, DOES fully explain the graph that was seen.  One place to review the
> source code is here:
>
> http://lxr.free-electrons.com/source/net/ipv4/tcp_westwood.c?v=3.2
>
> Some observations about this code:
>
> 1. The bandwidth estimate is run through a “(7×prev+new)/8” filter TWICE
> [see lines 93-94].
> 2. The units of time for all objects in the code (rtt, bwe, delta, etc) is
> ‘jiffies’, not milliseconds, nor microseconds [see line 108].
> 3. The bandwidth estimate is updated every “rtt” with the test in the code
> (line 139) essentially: delta>rtt.  However, “rtt” is the last unsmoothed
> rtt seen on the link (and increasing during bufferbloat).  When rtt
> increases, the frequency of bandwidth updates drops.
> 4. The server is Linux 3.2 with HZ=100 (meaning jiffies increases every
> 10ms).

Oy, this also means that there is no BQL on this server, and thus it's
TX ring can get quite filled. So I'd like to see what a BQL enabled server
does to westwood+ now.

https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

I imagine that tcp offloads are enabled, also? So much work went into fixing
things like tcp timestamps, etc, in the face of TSO, after 3.2.



>
> When you graph some of the raw data observed (see
> http://www.duckware.com/blog/the-dark-problem-with-aqm-in-the-internet/image
> s/chart.gif), the Westwood+ bandwidth estimate takes significant time to
> ramp up.

>
> For the first 0.84 seconds of the download, we expect the Westwood+ code to
> update the bandwidth estimate around 14 times, or once every 60ms or so.
> However, after this, we know there is a bufferbloat episode, with RTT times
> increasing (decreasing the frequency of bandwidth updates).  The red line in
> the graph above suggests that Westwood might have only updated the bandwidth
> estimate around 9-10 more times, before using it to set cwnd/ssthresh.
>
> - Jerry
>
>
>
>
> -----Original Message-----
> From: Jonathan Morton [mailto:chromatix99@gmail.com]
> Sent: Saturday, August 30, 2014 2:46 AM
> To: Stephen Hemminger
> Cc: Jerry Jongerius; bloat@lists.bufferbloat.net
> Subject: Re: [Bloat] The Dark Problem with AQM in the Internet?
>
>
> On 30 Aug, 2014, at 9:28 am, Stephen Hemminger wrote:
>
>> On Sat, 30 Aug 2014 09:05:58 +0300
>> Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>>
>>> On 29 Aug, 2014, at 5:37 pm, Jerry Jongerius wrote:
>>>
>>>>> did you check to see if packets were re-sent even if they weren't
>>>>> lost? on of the side effects of excessive buffering is that it's
>>>>> possible for a packet to be held in the buffer long enough that the
>>>>> sender thinks that it's been lost and retransmits it, so the packet
>>>>> is effectivly 'lost' even if it actually arrives at it's destination.
>>>>
>>>> Yes.  A duplicate packet for the missing packet is not seen.
>>>>
>>>> The receiver 'misses' a packet; starts sending out tons of dup acks
>>>> (for all packets in flight and queued up due to bufferbloat), and
>>>> then way later, the packet does come in (after the RTT caused by
>>>> bufferbloat; indicating it is the 'resent' packet).
>>>
>>> I think I've cracked this one - the cause, if not the solution.
>>>
>>> Let's assume, for the moment, that Jerry is correct and PowerBoost plays
> no part in this.  That implies that the flow is not using the full bandwidth
> after the loss, *and* that the additive increase of cwnd isn't sufficient to
> recover to that point within the test period.
>>>
>>> There *is* a sequence of events that can lead to that happening:
>>>
>>> 1) Packet is lost, at the tail end of the bottleneck queue.
>>>
>>> 2) Eventually, receiver sees the loss and starts sending duplicate acks
> (each triggering CA_EVENT_SLOW_ACK path in the sender).  Sender (running
> Westwood+) assumes that each of these represents a received, full-size
> packet, for bandwidth estimation purposes.
>>>
>>> 3) The receiver doesn't send, or the sender doesn't receive, a duplicate
> ack for every packet actually received.  Maybe some firewall sees a large
> number of identical packets arriving - without SACK or timestamps, they
> *would* be identical - and filters some of them.  The bandwidth estimate
> therefore becomes significantly lower than the true value, and additionally
> the RTO fires and causes the sender to reset cwnd to 1 (CA_EVENT_LOSS).
>>>
>>> 4) The retransmitted packet finally reaches the receiver, and the ack it
> sends includes all the data received in the meantime (about 3.5MB).  This is
> not sufficient to immediately reset the bandwidth estimate to the true
> value, because the BWE is sampled at RTT intervals, and also includes
> low-pass filtering.
>>>
>>> 5) This ends the recovery phase (CA_EVENT_CWR_COMPLETE), and the sender
> resets the slow-start threshold to correspond to the estimated
> delay-bandwidth product (MinRTT * BWE) at that moment.
>>>
>>> 6) This estimated DBP is lower than the true value, so the subsequent
> slow-start phase ends with the cwnd inadequately sized.  Additive increase
> would eventually correct that - but the key word is *eventually*.
>>>
>>> - Jonathan Morton
>>
>> Bandwidth estimates by ack RTT is fraught with problems. The returning
>> ACK can be delayed for any number of reasons such as other traffic or
>> aggregation. This kind of delay based congestion control suffers badly
> from any latency induced in the network.
>> So instead of causing bloat, it gets hit by bloat.
>
> In this case, the TCP is actually tracking RTT surprisingly well, but the
> bandwidth estimate goes wrong because the duplicate ACKs go missing.  Note
> that if the MinRTT was estimated too high (which is the only direction it
> could go), this would result in the slow-start threshold being *higher* than
> required, and the symptoms observed would not occur, since the cwnd would
> grow to the required value after recovery.
>
> This is the opposite effect from what happens to TCP Vegas in a bloated
> environment.  Vegas stops increasing cwnd when the estimated RTT is
> noticeably higher than MinRTT, but if the true MinRTT changes (or it has to
> compete with a non-Vegas TCP flow), it has trouble tracking that fact.
>
> There is another possibility:  that the assumption of non-queue RTT being
> constant against varying bandwidth is incorrect.  If that is the case, then
> the observed behaviour can be explained without recourse to lost duplicate
> ACKs - so Westwood+ is correctly tracking both MinRTT and BWE - but (MinRTT
> * BWE) turns out to be a poor estimate of the true BDP.  I think this still
> fails to explain why the cwnd is reset (which should occur only on RTO), but
> everything else potentially fits.
>
> I think we can distinguish the two theories by running tests against a
> server that supports SACK and timestamps, and where ideally we can capture
> packet traces at both ends.
>
> - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

  reply	other threads:[~2014-09-01 17:40 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-08-23 18:16 Jerry Jongerius
2014-08-23 19:30 ` Jonathan Morton
2014-08-23 20:01 ` Sebastian Moeller
2014-08-25 17:13   ` Greg White
2014-08-25 18:09     ` Jim Gettys
2014-08-25 19:12       ` Sebastian Moeller
2014-08-25 21:17         ` Bill Ver Steeg (versteb)
2014-08-25 21:20           ` Bill Ver Steeg (versteb)
2014-08-28 13:19     ` Jerry Jongerius
2014-08-28 14:07       ` Jonathan Morton
2014-08-28 17:20         ` Jerry Jongerius
2014-08-28 17:41           ` Dave Taht
2014-08-28 18:15           ` Jonathan Morton
2014-08-29 14:21             ` Jerry Jongerius
2014-08-29 16:31               ` Jonathan Morton
2014-08-29 16:54                 ` Jerry Jongerius
2014-08-28 18:59           ` Sebastian Moeller
2014-08-29 11:33             ` Jerry Jongerius
2014-08-29 12:18               ` Sebastian Moeller
2014-08-29 14:42               ` Dave Taht
2014-08-29  1:59           ` David Lang
2014-08-29 14:37             ` Jerry Jongerius
2014-08-30  6:05               ` Jonathan Morton
2014-08-30  6:28                 ` Stephen Hemminger
2014-08-30  6:45                   ` Jonathan Morton
2014-09-01 17:30                     ` Jerry Jongerius
2014-09-01 17:40                       ` Dave Taht [this message]
2014-08-28 14:39       ` Rich Brown
2014-08-28 16:20         ` Jerry Jongerius
2014-08-28 16:35           ` Fred Baker (fred)
2014-08-28 18:00             ` Jan Ceuleers
2014-08-28 18:13               ` Dave Taht
2014-08-29  1:57                 ` David Lang
2014-08-28 18:41               ` Kenyon Ralph
2014-08-28 19:04                 ` Dave Taht
2014-08-28 16:36           ` Greg White
2014-08-28 16:52             ` Bill Ver Steeg (versteb)
2014-09-01 11:47           ` Richard Scheffenegger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.bufferbloat.net/postorius/lists/bloat.lists.bufferbloat.net/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAA93jw6vc2+GHrm7sRL51tmk4VXAzRBQo-iu6dn+qOwrefzbqQ@mail.gmail.com \
    --to=dave.taht@gmail.com \
    --cc=bloat@lists.bufferbloat.net \
    --cc=jerryj@duckware.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox