From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-fr.alcatel-lucent.com (fr-hpida-esg-02.alcatel-lucent.com [135.245.210.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 43F243B2A3 for ; Wed, 2 Mar 2016 06:28:28 -0500 (EST) Received: from fr712umx4.dmz.alcatel-lucent.com (unknown [135.245.210.45]) by Websense Email Security Gateway with ESMTPS id 3836CD0E8237D; Wed, 2 Mar 2016 11:28:25 +0000 (GMT) Received: from fr712usmtp2.zeu.alcatel-lucent.com (fr712usmtp2.zeu.alcatel-lucent.com [135.239.2.42]) by fr712umx4.dmz.alcatel-lucent.com (GMO-o) with ESMTP id u22BSQcC015509 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 2 Mar 2016 11:28:26 GMT Received: from FR711WXCHHUB02.zeu.alcatel-lucent.com (fr711wxchhub02.zeu.alcatel-lucent.com [135.239.2.112]) by fr712usmtp2.zeu.alcatel-lucent.com (GMO) with ESMTP id u22BQKG1010048 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Wed, 2 Mar 2016 12:28:11 +0100 Received: from FR711WXCHMBA05.zeu.alcatel-lucent.com ([169.254.1.15]) by FR711WXCHHUB02.zeu.alcatel-lucent.com ([135.239.2.112]) with mapi id 14.03.0195.001; Wed, 2 Mar 2016 12:26:05 +0100 From: "De Schepper, Koen (Nokia - BE)" To: =?iso-8859-1?Q?EXT_Dave_T=E4ht?= , "aqm@ietf.org" , "bloat@lists.bufferbloat.net" Thread-Topic: [aqm] review: Deployment of RITE mechanisms, in use-case trial testbeds report part 1 Thread-Index: AQHRcZEvLyL6lympGUCsjn8iCQ7i+p9F61oA Date: Wed, 2 Mar 2016 11:26:03 +0000 Message-ID: References: <56BB8F05.2030006@mti-systems.com> <56D1F349.6040601@taht.net> In-Reply-To: <56D1F349.6040601@taht.net> Accept-Language: nl-BE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [135.239.27.41] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Bloat] [aqm] review: Deployment of RITE mechanisms, in use-case trial testbeds report part 1 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 11:28:28 -0000 Hi Dave, Thanks for reviewing our results. But probably I should not have referred t= o this document, as it contained indeed preliminary results, not showing al= l experiments. More elaborate results will be presented. =20 Just to be clear: we agree that FQ_ is currently the only network-"fix" to = make sure that every flow gets an equal and stable rate. But I think you ag= ree that FQ_ has some disadvantages too (mainly how to correctly identifyin= g "flows", requiring still a classic self-queue, and not allowing deviation= both up and down from the fair rate). The goal of L4S is to solve the problem in the end-system allowing a neutra= l, transport layer independent and simple network. And note that our rate f= airness benchmark is FQ_! If we can reach results that are close to FQ_ (di= sregarding its disadvantages here), we have achieved our goal. We don't exp= ect L4S to be better than the FQ_ for its default rate fairness, except whe= re FQ_ cannot fix the classic TCP problems in the end-system. =20 Related to the results in this document, the absolute quality selected is i= ndeed skewed by both the Windows CTCP and DCTCP achieving a higher throughp= ut than its Linux Reno and DCTCP counterparts. So the absolute values of qu= ality should not be compared, but rather the stability of the quality. In t= he border cases where the available bitrate is close to the switchover poin= ts, the quality of DCTCP HAS stays stable at the expected quality, the FQ a= nd PIE start fluctuating more heavily, PIE due to classic TCP rate variabil= ity and PIE's bigger queue, and FQ_ because the HAS flows cannot regain los= t scheduling opportunities during TCP time-outs and restarts (referred to a= s "the peeks are cut off"). =20 Of course feel free to reproduce our results, Windows server is free availa= ble for evaluation including the IIS server, Linux has DCTCP and the AQMs (= the immediate step function for DCTCP can be made with RED, so no need for = DualQ), and also other (open source) adaptive streaming servers can be used= . More inline too... Regards, Koen. > -----Original Message----- > From: aqm [mailto:aqm-bounces@ietf.org] On Behalf Of EXT Dave T=E4ht > Sent: zaterdag 27 februari 2016 20:05 > To: aqm@ietf.org; bloat@lists.bufferbloat.net > Subject: [aqm] review: Deployment of RITE mechanisms, in use-case trial > testbeds report part 1 >=20 >=20 >=20 >=20 > On 2/26/16 3:23 AM, De Schepper, Koen (Nokia - BE) wrote: > > Hi Wes, > > > > Just to let you know that we are still working on AQMs that support > scalable (L4S) TCPs. > > We could present some of our latest results (if there will be a > meeting in Buenos Aires, otherwise in Berlin?) > > > > * Performance of HTTP Adaptive Video Streaming (HAS) with different > TCP's and AQMs > > o HAS is currently ~30% of Internet traffic, but no AQM testing so > far has included it >=20 > I am aware of several unpublished studies. There was also something that > compared 1-3 HAS flows from several years back from stanford that I've > longed to be repeated against these aqm technologies. >=20 > https://reproducingnetworkresearch.wordpress.com/2014/06/03/cs244-14- > confused-timid-and-unstable-picking-a-video-streaming-rate-is-hard/ >=20 > > o the results are very poor with a particular popular AQM >=20 > Define "very poor". ? >=20 "Very poor" compared to what can be expected from a perfect per flow schedu= ler in the network ;-). Of course it is not necessarily due to FQ_'s perfor= mance, rather the combination of FQ_ and Classic TCP's shortcomings. If som= ething is not functioning well in the end-systems, it cannot always be fixe= d in the network only. So this is certainly not a criticism on the FQ_ impl= ementation. > > Presenter: Inton Tsang > > Duration: 10mins > > Draft: Comparative testing of draft-ietf-aqm-pie-01, draft-ietf-aqm- > fq-codel-04, draft-briscoe-aqm-dualq-coupled >=20 >=20 >=20 > > > > For experiment write-up, see Section 3 of > https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3- > public1.pdf >=20 > At the risk of sounding grumpy and pedantic, partially driven by trying > to read through 58 pages of text on a b/w kindle (before I switched to > something higher res) I appreciate your effort in reviewing the full document, but I mentioned se= ction 3 as the relevant section here. Note that this deliverable is not a like a peer reviewed publication, but a= write-up of research at a certain moment in time, and therefore is mostly = work in progress. It has been reviewed before by independent assigned exper= ts by the EU, and their some of their comments were in line with yours. >=20 > No source, can't reproduce. >=20 > The diagrams of the topology, etc, were quite helpful. Insight into the > actual complexity of BT's network was quite useful. I would love to know > the actual usage of the various QoS queues in terms of real traffic... >=20 > Has DCTP's not responding to drop properly problem been fixed in linux > yet? (sec 3.3.2). That remains a scary bug.... >=20 > More detailed comments: >=20 > The tests tested download traffic only and seem to have assumed that the > uplink would never have been a bottleneck. Home connections are > asymmetric and increasingly so - comcast, for example is now putting out > links with 75Mbit down, 5.5Mbit up. It is not clear what the uplink rate > of the emulated or real network is in this paper. >=20 > 2) Section 2 - would have been tons more interesting had it evaluated > the effects of other workloads against actual "twitch" games such as any > of the quake series, call of duty, or starcraft. The game chosen was > quite uninteresting from "needing a low latency and jitter to win" on, > front. >=20 > Applying interesting background network workloads while game bots > competed as in here http://sscaitournament.com/ would have been great > fun! - injecting random jitter and delay into the network stack has been > what Karl Auerbach has been doing to the competitors at the DARPA robot > competitions, and he's probably been the cause of quite a few of those > real-life robots falling over (I can hear his evil chuckles now). >=20 > Artificially applying a wifi loss rate of 1.5% is a far cry from what > actually happens on wifi, where what you see is more typically large > spikes in l2 delay, nowadays. >=20 > 3) As for section 3 - it uses a "bottleneck of 20Mbps and RTT of 20ms, > ten TCP long file download flows as background traffic and one HAS > request, which launch two TCP flows." I will try to restrain myself, > but: >=20 > A) In a cable modem scenario, the typical uplink varies from 2mbit to > 4mbit on a 20mbit link. I am not aware of many 20mbit symmetric links, > aside from FIOS. I would love symmetry on home links, is BT deploying > that?? We define these experiments with the purpose to analyze and improve, not ne= cessarily to resemble realistic scenario's. Of course we try to cover the r= ange of today's realistic cases and finally evaluation of realistic test ca= ses are needed as well. But when mixing too many mechanisms together we fin= d it too complex to analyze. The current Internet is working most of the ti= me correctly, it are only the short moments of coincidental conditions that= make users experience the Internet as unreliable. We tried to create these= conditions in our testcases frequently enough so we can evaluate the impac= t of different mechanisms (AQMs, schedulers and CCs). Also detecting and so= lving "none-used" problems allow new applications to become realistic... >=20 > B) this is a workload that has very little resemblance to any reality I > can think of and seems expressly designed to show the burst tolerance > and additional latency induced by pie and dualQ to an advantage, and > fq_codel at a local minimum. >=20 > For the sake of science, varying the number of long running downloads > from, say, 2,4,8 and 16, would have provided a more rounded picture. > Still I am really hard pressed to think of any case where home HAS > traffic will be competing with more than 2-3 long running full rate > downloads in your typical home. I HAVE seen 3 android devices attempt to > update themselves all at once, but that's it.... >=20 > Using bittorrent to generate 5-15 flows down, perhaps? Even then, most > real world torrents are not running very fast, there's utp based > congestion control, and torrenters spend way more time in the upload > phase than the download. >=20 > ... >=20 > Using web access bursts (10-100 flows lasting for 2-8 seconds), chock > full of dns accesses, seems to be a good test against HAS traffic. I see > there is a section doing that, but it is against 5 background flows, > also and I'm pretty sure there will be a pause between bursts, and > although the "The web requests followed an exponential arrival process, > while the size of the downloaded files were designed to represent actual > Internet web objects following a Pareto distribution with size between 1 > Kbyte and 1 Mbyte" is documented... >=20 > I could do the math assuming that distribution * the number of files... > but I would prefer the total size of the web transfer be documented, and > it's time to complete a load presented... nor is it clear if dns lookups > are enabled or counted. Unless I missed it somewhere? If you are going > to measure two loads - one HAS, one web, presenting the results for both > for contrast, seems logical. >=20 > "Figure 3.22 shows that HAS-DCTCP can maintain its performance even with > the additional 100 web traffic request per second as background traffic. > It consistently delivered segments of 2056 Kbps encoding > bit rate. In contrast, under the same network condition, the additional > web traffic deteriorate further the performance of HAS-CTCP as the > delivered segment alternated qualities between 688 Kbps and > 991 Kbps encoding rates. This had a lower overall quality when compared > with the previous experiment, which without the additional web traffic > could achieved a sustainable delivery of segment with 991 Kbps > encoding rates, as shown in Figure 3.18" >=20 > This kind of stuff really gets under my skin. What was the web page > completion time? Which matters more to the users of the household? a > slight decline in video quality, or a web page load? >=20 > Sacrificing all other network applications on the crucible of 4k video > is really not my goal in life, I would like to see videoconferencing, > gaming, web, in particular, work well in the presence of HAS. >=20 > Please, please, please, put web page completion times in this document? For our purpose, we prefer the statistical mix of downloads with different = sizes and the representation of completion times for these different sizes.= Again creating sporadic events frequently enough to study them. The repres= entation allows clearly to compare the impact of the different mechanisms, = as we repeat the same scenarios over the different combinations of AQMs, sc= hedulers and CCs. >=20 > ... >=20 > Also that sort of web traffic would probably cycle on a 30-60 second > basis at most - people just can't read that fast! Secondly, a search > engine query often precedes an actual web page lookup. >=20 > (I note that web page size growth has slowed dramatically since this wg > started, it is now well below the projections we had a few years ago > (7mbit by 20XX, more like 4 now, but I'd have to go redo the math - > growth went from exponential-looking in 2012 to linear now...). There's > also some data presented recently by some googlers at netconf 1.1 that > showed the impact and commonality of dns failures across their > subscriber base..... >=20 > I freely confess that maybe i'm out of touch, that maybe having perfect > television quality while waiting seconds for web pages to load, is what > most internet users want, and they deserve to get it, good and hard, > along with their supersized mcgreasy burgers and calming drugs delivered > to their door. >=20 > C) And no reverse traffic whatsoever in section three. >=20 > In your typical HAS video streaming home scenario, the worst behaviors I > can think of would be bittorrent generated *on the upload*, but the > simplest case of a single upload happening at all - tends towards > starving the ack streams and ramp up times on the video download flows. > Which was not tested. Sigh. How long have I banged on this drum? >=20 > D) It is unproven from a QOE perspective (I think) that having a video > stream change qualities on a "better average" basis is of benefit to the > user - seeing a stream never pause "for buffering", and consistent > quality, seems saner. I notice when the quality changes down, but rarely > up. "buffering" - that I seriously notice. >=20 > I would have liked reference data for drop tail vs the various > experiments in this paper. Typical amounts of head end buffering on > docsis 3.0 arris cmtss seem to be in the 800ms range at 20mbit. >=20 > E) A workload that I would like to see tested with some rigor is a > family of four - one doing a big upload (facebook/instagram), another > browsing the web, another doing a phone call or videoconference, and a > fourth attempting to watch a movie at the highest possible resolution. >=20 > The bar for the last has moved to SD or HD quality in the past few years > - 18mbits is a number I keep seeing more and more. Someone with a 20Mbit > link IS going to try for the highest quality, and showing the dynamic > range of that (1-18mbits) would be more interesting than 1mbit to 2, as > in in this paper. >=20 > I also would welcome 2-3 HAS testing on downloads against these AQMs, > attempting those rates, along the lines of the stanford thing I > mentioned first. >=20 > I petered out before reading sections 4 and 5. I will try to get to it > this week. >=20 > _______________________________________________ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm