From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id B3A3021F720 for ; Thu, 28 Aug 2014 10:41:38 -0700 (PDT) Received: by mail-oi0-f47.google.com with SMTP id x69so796612oia.34 for ; Thu, 28 Aug 2014 10:41:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=gTGr0EOb0VacnlR4HI7CzwTwQzeT2Di4Ckw3RLeYJLo=; b=cX1kMfDcfq8gYZ0RLEV6icNaymsKQcR+1z6lIO1ZnWNfi3YyKQ0L+2L1FQVbvRTcFB Ls+q1vCYZ1/wvccW4tKsVXSZcFP3sXWCMSC9RzsjhrFk/bVgEQVpwqHXbz3Jk5QmL5DN AmSbNUj5ffxGoH/+HIaC5OfV81VeW86/3iMtKGAKV7W/ZD/YRi/Pck/nbJQnOm0ddooD MIDYrGmFEdFPLopNcOELIHjBh09VSZURqQPBW5B3FmnLkLxHm/t5IdMzE0WXePtANzYV ZGGUOAlH0c4krCx+DlECXFpTOKFuozyVKlF1/lArYGZK6Yse6qJit1L4lGkFsPzh8pCn Gx2w== MIME-Version: 1.0 X-Received: by 10.182.73.231 with SMTP id o7mr5273571obv.56.1409247697709; Thu, 28 Aug 2014 10:41:37 -0700 (PDT) Received: by 10.202.93.69 with HTTP; Thu, 28 Aug 2014 10:41:37 -0700 (PDT) In-Reply-To: <002201cfc2e4$565c1100$03143300$@duckware.com> References: <000001cfbefe$69194c70$3b4be550$@duckware.com> <000901cfc2c2$c21ae460$4650ad20$@duckware.com> <4A89264B-36C5-4D1F-9E5E-33F2B42C364E@gmail.com> <002201cfc2e4$565c1100$03143300$@duckware.com> Date: Thu, 28 Aug 2014 10:41:37 -0700 Message-ID: From: Dave Taht To: Jerry Jongerius Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: bloat Subject: Re: [Bloat] The Dark Problem with AQM in the Internet? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Aug 2014 17:41:39 -0000 On Thu, Aug 28, 2014 at 10:20 AM, Jerry Jongerius wro= te: > Jonathan, > > Yes, WireShark shows that *only* one packet gets lost. Regardless of RWI= N > size. The RWIN size can be below the BDP (no measurable queuing within t= he > CMTS). Or, the RWIN size can be very large, causing significant queuing > within the CMTS. With a larger RWIN value, the single dropped packet > typically happens sooner in the download, rather than later. The fact th= ere > is no "burst loss" is a significant clue. > > The graph is fully explained by the Westwood+ algorithm that the server i= s > using. If you input the data observed into the Westwood+ bandwidth > estimator, you end up with the rate seen in the graph after the packet lo= ss > event. The reason the rate gets limited (no ramp up) is due to Westwood+ > behavior on a RTO. And the reason there is the RTO is due the bufferbloa= t, > and the timing of the lost packet in relation to when the bufferbloat > starts. When there is no RTO, I see the expected drop (to the Westwood+ > bandwidth estimate) and ramp back up. On a RTO, Westwood+ sets both > ssthresh and cwnd to its bandwidth estimate. On the same network, what does cubic do? > The PC does SACK, the server does not, so not used. Timestamps off. Timestamps are *critical* for good tcp performance above 5-10mbit on most cc algos. I note that the netperf-wrapper test has the ability to test multiple variants of TCP, if enabled on the server (basically you need to modprobe the needed algorithms, enable them in /proc/sys/net/ipv4/tcp_allowed_congestion_contro= l, and select them in the test tool (iperf and netperf have support)). Everyone here has installed netperf-wrapper already, yes? Very fast to generate a good test and a variety of plots like those shown here: http://burntchrome.blogspot.com/2014_05_01_archive.html (in reading that over, does anyone have any news on CMTS aqm or packet scheduling systems? It's the bulk of the problem there...) netperf-wrapper is easy to bring up on linux, on osx it needs macports, and the only way I've come up to test windows behavior is using windows as a netperf client rather than server. I haven't looked into westwood+'s behavior much of late, I will try to add = it and a few other tcps to some future tests. I do have some old plots showing it misbehaving relative to other TCPs, but that was before many fixes lande= d in the kernel. Note: I keep hoping to find a correctly working ledbat module, the one I have doesn't look correct (and needs to be updated to linux 3.15's change to us based timestamping.) > > - Jerry > > > -----Original Message----- > From: Jonathan Morton [mailto:chromatix99@gmail.com] > Sent: Thursday, August 28, 2014 10:08 AM > To: Jerry Jongerius > Cc: 'Greg White'; 'Sebastian Moeller'; bloat@lists.bufferbloat.net > Subject: Re: [Bloat] The Dark Problem with AQM in the Internet? > > > On 28 Aug, 2014, at 4:19 pm, Jerry Jongerius wrote: > >> AQM is a great solution for bufferbloat. End of story. But if you want > to track down which device in the network intentionally dropped a packet > (when many devices in the network path will be running AQM), how are you > going to do that? Or how do you propose to do that? > > We don't plan to do that. Not from the outside. Frankly, we can't relia= bly > tell which routers drop packets today, when AQM is not at all widely > deployed, so that's no great loss. > > But if ECN finally gets deployed, AQM can set the Congestion Experienced > flag instead of dropping packets, most of the time. You still don't get = to > see which router did it, but the packet still gets through and the TCP > session knows what to do about it. > >> The graph presented is caused the interaction of a single dropped packet= , > bufferbloat, and the Westwood+ congestion control algorithm - and not pow= er > boost. > > This surprises me somewhat - Westwood+ is supposed to be deliberately > tolerant of single packet losses, since it was designed explicitly to get > around the problem of slight random loss on wireless networks. > > I'd be surprised if, in fact, *only* one packet was lost. The more usual > case is of "burst loss", where several packets are lost in quick successi= on, > and not necessarily consecutive packets. This tends to happen repeatedly= on > dump drop-tail queues, unless the buffer is so large that it accommodates > the entire receive window (which, for modern OSes, is quite impressive in= a > dark sort of way). Burst loss is characteristic of congestion, whereas > random loss tends to lose isolated packets, so it would be much less > surprising for Westwood+ to react to it. > > The packets were lost in the first place because the queue became > chock-full, probably at just about the exact moment when the PowerBoost > allowance ran out and the bandwidth came down (which tends to cause the > buffer to fill rapidly), so you get the worst-case scenario: the buffer a= t > its fullest, and the bandwidth draining it at its minimum. This maximise= s > the time before your TCP gets to even notice the lost packet's nonexisten= ce, > during which the sender keeps the buffer full because it still thinks > everything's fine. > > What is probably happening is that the bottleneck queue, being so large, > delays the retransmission of the lost packet until the Retransmit Timer > expires. This will cause Reno-family TCPs to revert to slow-start, assum= ing > (rightly in this case) that the characteristics of the channel have chang= ed. > You can see that it takes most of the first second for the sender to ramp= up > to full speed, and nearly as long to ramp back up to the reduced speed, b= oth > of which are characteristic of slow-start at WAN latencies. NB: during > slow-start, the buffer remains empty as long as the incoming data rate is > less than the output capacity, so latency is at a minimum. > > Do you have TCP SACK and timestamps turned on? Those usually allow minor > losses like that to be handled more gracefully - the sending TCP gets a > better idea of the RTT (allowing it to set the Retransmit Timer more > intelligently), and would be able to see that progress is still being mad= e > with the backlog of buffered packets, even though the core TCP ACK is not > advancing. In the event of burst loss, it would also be able to retransm= it > the correct set of packets straight away. > > What AQM would do for you here - if your ISP implemented it properly - is= to > eliminate the negative effects of filling that massive buffer at your ISP= . > It would allow the sending TCP to detect and recover from any packet loss > more quickly, and with ECN turned on you probably wouldn't even get any > packet loss. > > What's also interesting is that, after recovering from the change in > bandwidth, you get smaller bursts of about 15-40KB arriving at roughly > half-second intervals, mixed in with the relatively steady 1-, 2- and > 3-packet stream. That is characteristic of low-level packet loss with a > low-latency recovery. > > This either implies that your ISP has stuck you on a much shorter buffer = for > the lower-bandwidth (non-PowerBoost) regime, *or* that the sender is > enforcing a smaller congestion window on you after having suffered a > slow-start recovery. The latter restricts your bandwidth to match the > delay-bandwidth product, but happily the "delay" in that equation is at a > minimum if it keeps your buffer empty. > > And frankly, you're still getting 45Mbps under those conditions. Many > people would kill for that sort of performance - although they'd probably > then want to kill everyone in the Comcast call centre later on. > > - Jonathan Morton > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_= indecent.article