From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f98.google.com (mail-la0-f98.google.com [209.85.215.98]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id DB77421F299 for ; Mon, 4 May 2015 03:20:16 -0700 (PDT) Received: by labmn9 with SMTP id mn9so2803279lab.0 for ; Mon, 04 May 2015 03:20:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=RnBNWoBDSiHUOtWwoU1eLSyQPA1wXfPAO0EQpPMx5BI=; b=DLk/5DGXUt8Exd30lXuuecnpsYiEW/G1J4h7cPbNDoDJHDqh746gAP289AbxTDiHuV pQ71DF04S6PFEecvb5btN0hKcrDF6bikqlsD6BubBr4iID6yyaZ3LP0hFfxo2LEd+faB 5DdAfdeiHIyvga0alJ+eCkLkthlWzJliTd9ePaRMmylABtfL9SZmGtq58T/VmGdQKc3E AVyWmfNJLJs4jtigtqOL5iQ7xzo8hjwom8tRA2idG3oAGDUC21n54okVGMz6C1yvVTRW Q10AIJ7f7Ijal5wPIWdQczdG1sDPa5GiqyI4Frro5XDNqK9sdfI0bcFjUNm+LNqRrynq aASQ== X-Gm-Message-State: ALoCoQnR4ItUwhPl6aKoWieeXZfb1qtNGrWn6E0jzntTW9YnF1ZkhwgJa2crpTQlAh/uxw2udmie6nA6zIJJb8RQ7DA9R8lSbw== X-Received: by 10.180.85.42 with SMTP id e10mr17927626wiz.17.1430734810067; Mon, 04 May 2015 03:20:10 -0700 (PDT) Received: from mail.la.pnsol.com (eu1sys200aog107.obsmtp.com. [207.126.144.123]) by mx.google.com with SMTPS id s3sm270806wiz.1.2015.05.04.03.20.08 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 04 May 2015 03:20:10 -0700 (PDT) X-Relaying-Domain: pnsol.com Received: from mail.la.pnsol.com ([89.145.213.110]) (using TLSv1) by eu1sys200aob107.postini.com ([207.126.147.11]) with SMTP ID DSNKVUdH2N8g7uXrp9yH7Q4h/zaXVs6IIdpU@postini.com; Mon, 04 May 2015 10:20:08 UTC Received: from ba6-office.pnsol.com ([172.20.5.199]) by mail.la.pnsol.com with esmtp (Exim 4.76) (envelope-from ) id 1YpDTj-0006oP-6n; Mon, 04 May 2015 11:19:59 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) From: Neil Davies In-Reply-To: <0F8CB21C-792F-4F95-BC49-BED3DF0A2100@unimore.it> Date: Mon, 4 May 2015 11:21:13 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <72DB0260-F0DF-426F-A3F3-ECF5D8AF228F@pnsol.com> <766042D4-0C90-4C77-9033-07B8E436C35B@pnsol.com> <2F4DCB53-1E46-4829-B2F8-F8131664D1FF@pnsol.com> <0F8CB21C-792F-4F95-BC49-BED3DF0A2100@unimore.it> To: Paolo Valente X-Mailer: Apple Mail (2.1878.6) Cc: Jonathan Morton , bloat Subject: Re: [Bloat] Detecting bufferbloat from outside a node X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 May 2015 10:20:48 -0000 On 4 May 2015, at 11:10, Paolo Valente wrote: > I have tried to fully digest this information (thanks), but there is = still some piece that I am missing. To highlight it, I would like to try = with an oversimplified example. I hope this will make it easier to point = out flaws in my understanding. >=20 > Suppose that one wants/needs to discover whether outbound and/or = inbound packets experience high, internal queueing delays, in a given = node A, because some buffers are bloated (inside the node). For any = packet leaving or entering the node, we have that, regardless of whether = the packet exits from the node after experiencing a high internal = output-queueing delay, or whether the packet will experience a high = internal input-queueing delay after being received by node, the per-hop = or end-to-end delays experienced by the packet outside the node are = exactly the same. If this statement is true, then, since no information = of any sort is available about queueing delays inside the node, and = since the delays measurable from outside the node are invariant with = respect to the internal queueing delays, how can we deduce internal = delays from external ones? Paolo,=20 as you surmise - without appropriate observation points you can=E2=80=99t = *definitively* isolate where the =E2=80=9Cdelay=E2=80=9D (really =E2=88=86= Q) is accruing. If you can construct time-traces at more than one = observation point you can isolate it to between observation points. If you have a model of how it is supposed to behave (i.e. have knowledge = about the intermediate network elements along with some idea of their = configuration) you can start (by observing the pattern of change in the = delay/loss) to make inferences that certain elements are being driven in = a certain way - but beware, the model of inference is key - use of = =E2=80=9Cbandwidth=E2=80=9D or even just =E2=80=9Cdelay=E2=80=9D = doesn=E2=80=99t work - it does appear to work if you use the =E2=88=86Q|G,= S,V as the basis set Neil >=20 > Thanks, > Paolo >=20 > Il giorno 28/apr/2015, alle ore 12:23, Neil Davies = ha scritto: >=20 >>=20 >> On 28 Apr 2015, at 10:58, Sebastian Moeller wrote: >>=20 >>> Hi Neil, >>>=20 >>>=20 >>> On Apr 28, 2015, at 09:17 , Neil Davies = wrote: >>>=20 >>>> Jonathan >>>>=20 >>>> The timestamps don't change very quickly - dozens (or more) of = packets can have the same timestamp, so it doesn't give you the = appropriate discrimination power. Timed observations at key points gives = you all you need (actually, appropriately gathered they give you all you = can possibly know - by observation) >>>=20 >>> But this has two issues: >>> 1) =E2=80=9Ctimed observations=E2=80=9D: relatively easy if all = nodes are under your control otherwise hard. I know about the CERN = paper, but they had all nodes under their control, symmetric bandwidth = and shipload of samples, so over the wild internet =E2=80=9Ctimed = observations=E2=80=9D are still hard (and harder as the temporal = precision requirement goes up) >>=20 >> =E2=88=86Q (with its improper CDF semantics and G,S and V basis set) = has composition and de-composisition properties - this means that you = don=E2=80=99t need to be able to observe everywhere - even in Lucian=E2=80= =99s case his observation points were limited (certain systems) - the = rest of the analysis is derived using the properies of the =E2=88=86Q = calculus. >>=20 >> Lucian also demonstrated how the standard timing observations (which = include issues of clock drift and distributed accuracy) can be resolved = in a practical situation - he reproduced - starting from libpcap = captures on machines - results that CERN guys build specialist h/w with = better than 20ns timing only 5 years before. >>=20 >> The good thing about Lucian=E2=80=99s thesis is that it is in the = public domain - but we use the same approach over wide (i.e world) = networks and get same properties (unfortunately that is done in a = commercial context). This all arises because we can perform the = appropriate measurement error analysis, and hence use standard = statistical techniques. >>=20 >>>=20 >>> 2) =E2=80=9Ckey points=E2=80=9D: once you know the key points you = already must have a decent understanding on the effective topology of = the network, which again over the wider internet is much harder than if = one has all nodes under control. >>=20 >> Not really - the key points (as a start) are the end ones - and those = you have (reasonable) access to - and even if you don=E2=80=99t have = access to the *actual* end points - you can easily spin up a measurement = point that is very close (in =E2=88=86Q terms) to the ones you are = interested in - AWS and Google Compute are your friends here. >>=20 >>>=20 >>>=20 >>> I am not sure how Paolo=E2=80=99s =E2=80=9Cno-touching=E2=80=9D = problem fits into the requirements for your deltaQ (meta-)math ;) >>=20 >> I see =E2=80=9Cno touching=E2=80=9D as =E2=80=9Cno modification=E2=80=9D= - you can=E2=80=99t deduce information in the absence of data - what = you need to understand is the minimum data requirements to achieve the = measurement outcome - =E2=88=86Q calculus gives you that handle. >>=20 >>>=20 >>> Best Regards >>> Sebastian >>>=20 >>>>=20 >>>> Neil >>>>=20 >>>> On 28 Apr 2015, at 00:11, Jonathan Morton = wrote: >>>>=20 >>>>> On 27 Apr 2015 23:31, "Neil Davies" wrote: >>>>>>=20 >>>>>> Hi Jonathan >>>>>>=20 >>>>>> On 27 Apr 2015, at 16:25, Jonathan Morton = wrote: >>>>>>=20 >>>>>>> One thing that might help you here is the TCP Timestamps option. = The timestamps thus produced are opaque, but you can observe them and = measure the time intervals between their production and echo. You should = be able to infer something from that, with care. >>>>>>>=20 >>>>>>> To determine the difference between loaded and unloaded states, = you may need to observe for an extended period of time. Eventually = you'll observe some sort of bulk flow, even if it's just a software = update cycle. It's not quite so certain that you'll observe an idle = state, but it is sufficient to observe an instance of the link not being = completely saturated, which is likely to occur at least occasionally. >>>>>>>=20 >>>>>>> - Jonathan Morton >>>>>>=20 >>>>>> We looked at using TCP timestamps early on in our work. The = problem is that they don't really help extract the fine-grained = information needed. The timestamps can move in very large steps, and the = accuracy (and precision) can vary widely from implementation to = implementation. >>>>>=20 >>>>> Well, that's why you have to treat them as opaque, just like I = said. Ignore whatever meaning the end host producing them might embed in = them, and simply watch which ones get echoed back and when. You only = have to rely on the resolution of your own clocks. >>>>>=20 >>>>>> The timestamps are there to try and get a gross (if my memory = serves me right ~100ms) approximation to the RTT - not good enough for = reasoning about TCP based interactive/"real time" apps >>>>>=20 >>>>> On the contrary, these timestamps can indicate much better = precision than that; in particular they indicate an upper bound on the = instantaneous RTT which can be quite tight under favourable = circumstances. On a LAN, you could reliably determine that the RTT was = below 1ms this way. >>>>>=20 >>>>> Now, what it doesn't give you is a strict lower bound. But you can = often look at what's going on in that TCP stream and determine that = favourable circumstances exist, such that the upper bound RTT estimate = is probably reasonably tight. Or you could observe that the stream is = mostly idle, and thus probably influenced by delayed acks and Nagle's = algorithm, and discount that measurement accordingly. >>>>>=20 >>>>> - Jonathan Morton >>>>>=20 >>>>=20 >>>> _______________________________________________ >>>> Bloat mailing list >>>> Bloat@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/bloat >>>=20 >>=20 >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >=20 >=20 > -- > Paolo Valente =20 > Algogroup > Dipartimento di Fisica, Informatica e Matematica =09 > Via Campi, 213/B > 41125 Modena - Italy =20 > homepage: http://algogroup.unimore.it/people/paolo/ >=20