From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 33F7C21F306; Fri, 12 Jun 2015 08:02:26 -0700 (PDT) Received: by qgf75 with SMTP id 75so12341102qgf.1; Fri, 12 Jun 2015 08:02:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=0nLlzhawY4Pi6+QfLEPt9VFpXUPLX58dB2eVvCI7MXc=; b=Z1SMPh5JuS4Mus5Md6mA3U+IB41XXFQT7IU4Z87moxkG5JudP4a5OqFcRgJq6tW+7O kKgbLh7bzNrIipM4NyzfXHmq6ibGHEyvCzwkhJcGDZSB7obeV6+b2w57gNCDdeTgkynY 4MrIUgEyq9/+zfht9oYWUeptORNYV/t5RVOSuDXemC+MNv7nBdqnKPC3ltHbaOBC9q44 BAsoNHVHKfMdvwMxy2HUxLXqz42fj0CQVRcv/2YfvY0TDsbRuMt2uwP+O8pyu9qdHoVN Yrl17p1utKY/yJ17aTZVvUADqAAru02EeG4rPOCFel60F81WTuS+eET3CV3DHVQaVxGr sXPw== MIME-Version: 1.0 X-Received: by 10.140.131.67 with SMTP id 64mr20094601qhd.57.1434121345522; Fri, 12 Jun 2015 08:02:25 -0700 (PDT) Received: by 10.140.105.6 with HTTP; Fri, 12 Jun 2015 08:02:25 -0700 (PDT) In-Reply-To: <6594DC36-738D-4E1D-857E-701B2B11388F@gmx.de> References: <6594DC36-738D-4E1D-857E-701B2B11388F@gmx.de> Date: Fri, 12 Jun 2015 08:02:25 -0700 Message-ID: From: Daniel Havey To: Sebastian Moeller Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Fri, 12 Jun 2015 10:11:58 -0700 Cc: cake@lists.bufferbloat.net, "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] [Cake] active sensing queue management X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 15:02:55 -0000 On Thu, Jun 11, 2015 at 12:27 AM, Sebastian Moeller wrote= : > Hi Daniel, > > thanks for the clarifications. > > On Jun 11, 2015, at 02:10 , Daniel Havey wrote: > >> Hmmm, maybe I can help clarify. Bufferbloat occurs in the slowest >> queue on the path. This is because the other queues are faster and >> will drain. AQM algorithms work only if they are placed where the >> packets pile up (e.g. the slowest queue in the path). This is >> documented in Kathy and Van's CoDel paper. > > I am with you so far. > >> >> This is usually all well and good because we know where the bottleneck >> (the slowest queue in the path) is. It is the IP layer in the modem >> where the ISP implements their rate limiter. That is why algorithms >> such as PIE and CoDel are implemented in the IP layer on the modem. > > Okay. > >> >> Suppose the full committed rate of the token bucket rate limiter is 8 >> Mbps. This means that the queue at the IP layer in the modem is >> capable of emitting packets at 8 Mbps sustained rate. The problem >> occurs during peak hours when the ISP is not providing the full >> committed rate of 8 Mbps or that some queue in the system (probably in >> the access link) is providing something less than 8 Mbps (say for sake >> of discussion that the number is 7.5 Mbps). >> >> We know that (see Kathy and Van's paper) that AQM algorithms only work >> when they are placed at the slowest queue. However, the AQM is placed >> at the queue that is capable of providing 8 Mbps and this is not the >> slowest queue. The AQM algorithm will not work in these conditions. >> >> This is what is shown in the paper where the CoDel and PIE performance >> goes to hell in a handbasket. The ASQM algorithm is designed to >> address this problem. > > Except that DOCSIS 3.1 pie in the modem does not work that way. A= s I understand http://www.cablelabs.com/wp-content/uploads/2014/06/DOCSIS-A= QM_May2014.pdf section 3.2 MAP PROCESSING REQUIREMENTS, cable-modem pie wil= l not stuff data into the lower layers until it has received a grant to act= ually send that data, hence no uncontrolled sub-IP layer buffer bloat possi= ble (unless there are severe RF issues that take out data during transmissi= on). So at least for the upstream direction docsis 3.1 pie will not suffer = from buffer displacement, if I understand the cable labs white paper correc= tly. Hmmm, interesting. Are you sure? I'm a CS not a EE so the PHY layer is like black magic to me. However, I still think (although I am willing to be convinced otherwise by someone with superior knowledge :)) that the IP layer puts packet into a MAC layer queue. Then the MAC layer makes a queue depth based request for bandwidth in order to serialize and send the data. If somebody really knows how this works, please help! :^) Is the upload of a docsis 3.1 modem really unbloatable below the IP layer? I just want to know for my own edification :) > Your solution still might be a valuable add-on to control the downstream = buffer bloat in addition. I agree! If that reading of the cablelabs paper is correct then this nicely solves the upload vs. download problem and we don't really need BQL either. If it is not true, then we use BQL on the egress to solve the upload bloat problem and ASQM to solve the download bloat problem. Perfect solution! I love it when a plan comes together! :^) > I also believe that free in france had a modified DSL driver for = their box that made sure sub-IP buffering was bound to a low number of pack= ets as well, so no displaced buffers there as well. Now it seems that this = solution for DSL was unique so far and has not caught on, but once docsis3.= 1 modems hit the market upstream PIE in the modems will be reality. freefrance? Dave isn't that your provider? I thought they were running fq_CoDel? In any case, just because PIE in the modems is a reality don't be tempted to declare the problem solved and go home. Never underestimate the ability of the ISPs to do the wrong thing for very good reasons :^) What happens if they don't turn it on? This is really what I was trying to solve with ASQM. What if your provider won't run CoDel or PIE for whatever incomprehensible reason? Then you run ASQM and be done with it. > >> >> >> >> >> >> On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller wro= te: >>> Hi Dave, >>> >>> >>> On Jun 10, 2015, at 21:53 , Dave Taht wrote: >>> >>>> http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf >>>> >>>> gargoyle's qos system follows a similar approach, using htb + sfq, and >>>> a short ttl udp flow. >>>> >>>> Doing this sort of measured, then floating the rate control with >>>> "cake" would be fairly easy (although it tends to be a bit more >>>> compute intensive not being on a fast path) >>>> >>>> What is sort of missing here is trying to figure out which side of the >>>> bottleneck is the bottleneck (up or down). >>> >> >> Yeah, we never did figure out how to separate the up from the >> downlink. However, we just consider the access link as a whole (up + >> down) and mark/drop according to ratios of queuing time. > > This is a bit sad; why reduce say the effective uplink bandwidth = if only the downstream is contended? Not that I have a good alternative sol= ution that will not require help from outside boxes. > >> Overall it >> seems to work well, but, we never did a mathematical analysis. Kind >> of like saying it's not a "bug", it is a feature. And it this case it >> is true since both sides can experience bloat. > > Yes, but you only want to throttle traffic on the congested leg o= f the link, otherwise bandwidth efficiency goes to hell if you look at bi-d= irection link-saturating traffic. > >> >> >>> Yeah, they relay on having a reliable packet reflector upstream = of the =E2=80=9Cbottleneck=E2=80=9D so they get their timestamped probe pac= kets returned. In the paper they used either uplink or downlink traffic so = figuring where the bottleneck was easy at least this is how I interpret =E2= =80=9CExperiments were performed in the upload (data flowing from the users= to the CDNs) as well as in the download direction.". At least this is what= I get from their short description in glossing over the paper. >>> Nice paper, but really not a full solution either. Unless the IS= Ps cooperate in supplying stable reflectors powerful enough to support all = downstream customers. But if the ISPs cooperate, I would guess, they could = eradicate downstream buffer bloat to begin with. Or the ISPs could have the= reflector also add its own UTC time stamp which would allow to dissect the= RTT into its constituting one-way delays to detect the currently bloated d= irection. (Think ICMP type 13/14 message pairs "on steroids", with higher r= esolution than milliseconds, but for buffer bloat detection ms resolution w= ould probably be sufficient anyways). Currently, I hear that ISP equipment = will not treat ICMP requests with priority though. >> >> Not exactly. We thought this through for some time and considered >> many angles. Each method has its advantages and disadvantages. >> >> We decided not to use ICMP at all because of the reasons you stated >> above. We also decided not to use a "reflector" although as you said >> it would allow us to separate upload queue time from download. We >> decided not to use this because it would be difficult to get ISPs to >> do this. >> >> Are final choice for the paper was "magic" IP packets. This consists >> of an IP packet header and the timestamp. The IP packet is "self >> addressed" and we trick the iptables to emit the packet on the correct >> interface. This packet will be returned to us as soon as it reaches >> another IP layer (typically at the CMTS). > > Ah, thanks; I did not get this from reading over your paper (but = that is probably caused by me being a layman and having read it very quickl= y). Question how large is that packet on-the-wire? IP header plus 8 byte ma= kes me assume 20+8 =3D 28, but that is missing the ethernet header, so rath= er 14+20+8 =3D 42, but isn=E2=80=99t the shorts ethernet frame 64bytes? > >> >> Here's a quick summary: >> ICMP -- Simple, but, needs the ISP's cooperation (good luck :) >> Reflector -- Separates upload queue time from download queue time, >> but, requires the ISP to cooperate and to build something for us. >> (good luck :) >> Magic IP packets -- Requires nothing from the ISP (YaY! We have a >> winner!), but, is a little more complex. > > At the cost that you only get RTT instead of two one-way delays a= s one ideally would like. But as stated above if you combine your method wi= th say docsis3.1 pie which promises to keep the upstream under tight contro= l, the any RTT changes should (mainly) be caused by downstream over-bufferi= ng (effectively allowing you use you method to control the downstream well)= . > >> >> >>> Also I am confused what they actually simulated: =E2=80=9CThe mo= dems and CMTS were equipped with ASQM, CoDel and PIE,=E2=80=9D and =E2=80= =9CHowever, the problem pop- ularly called bufferbloat can move about among= many queues some of which are resistant to traditional AQM such as Layer 2= MAC protocols used in cable/DSL links. We call this problem bufferbloat di= splacement.=E2=80=9D seem to be slightly at odds. If modems and CTMS have d= ecent AQMs all they need to do is not stuff their sub-IP layer queuesand be= done with it. The way I understood the cable labs PIE story, they intended= to do exactly that, so at least the =E2=80=9Cbuffer displacement=E2=80=9D = remedy by ASQM reads a bit like a straw man argument. But as I am a) not of= the cs field, and b) only glossed over the paper, most likely I am missing= something important that is clearly in the paper... >> >> Good point! However, once again it's not quite that simple. Queues >> are necessary to absorb short term variations in packet arrival rate >> (or bursts). The queue required for any flow is given by the >> bandwidth delay product. > > Not a CS person, but that does not ring fully true; this basicall= y assumes a physical medium that will dump all packets into the buffer at o= ne time point and send them out a full delay period later; I think in reali= ty packets will be serialized and hence some packet will most likely have l= eft the buffer already before all have arrived, so the BDP is an more estim= ate of an upper bound=E2=80=A6 not that there is anything wrong with design= ing solutions aim to handle the worst case well. > >> Since we don't know the delay we can't >> predict the queue size in advance. What I'm getting at is the >> equipment manufacturers aren't putting in humongous queues because >> they are stupid, they are putting them there because in some cases you >> might really need that large of a queue. > > I thought our current pet hypothesis is that they aim for BDP at = their highest rated speeds or so, and all customers running that (huh speed= capable) equipment at lower rates are out of luck. > >> >> Statically sizing the queues is not the answer. Managing the size of >> the queue with an algorithm is the answer. :) > > No disagreement here, we just discuss the how not the why ;) > > Best Regards > Sebastian > >> >> >> >>> >>> Best Regards >>> Sebastian >>> >>>> >>>> -- >>>> Dave T=C3=A4ht >>>> What will it take to vastly improve wifi for everyone? >>>> https://plus.google.com/u/0/explore/makewififast >>>> _______________________________________________ >>>> Cake mailing list >>>> Cake@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/cake >>> >