From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6C6423CB39 for ; Thu, 7 Dec 2023 15:06:00 -0500 (EST) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-a1f37fd4b53so109107566b.1 for ; Thu, 07 Dec 2023 12:06:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701979559; x=1702584359; darn=lists.bufferbloat.net; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=SfSfnGSy78r+C0mlaQEwytJ3awRY2dzqdXSx0Cs/0lI=; b=CDkrvj2XHHp3TcYIY41NbVfonTbx81PL8DpUtRVSdYL7JN+WLVb/C4RFDVoV+WjS31 x1oQi6cFOKJfVyUUqaCyzPXnuoGYvxUsSZl3q4GflByB7tHfqsuoPehIgBpqaPsFQyvM jxlTHe6wZWpMV4woQLA9jNdCh7G9m/6IunFtgI+WLezBMDiQ6rNA1XjOtAYSPgnRL4ce raUvts7unUO0JMzpiv6ukYLp6d1TK3AqyuWa75yS0A4H6co7P9LTQfAFsdcrAWWjC09Z IyN8j5uxRLi8R0mCKcsbzWhqi1F6shlkOhvBDbk8CehX9srobOXo2opuOT9Pyom36xLL CnJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701979559; x=1702584359; h=content-transfer-encoding:cc:to:subject:message-id:date:from :references:in-reply-to:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SfSfnGSy78r+C0mlaQEwytJ3awRY2dzqdXSx0Cs/0lI=; b=crYYLF1jbxz7qJ0UdOc/w7MhYPtYP9qXIsxdqTFYc/Gs5ggzG0yb5SBG//bodiZFTF 8OVl0IGpPE4tAF/pyBISAt29oyNlGEfc5MXLHEkR6mR+Sj9wpe2fj4T4VSPMCEPPAgWw m1h97591009RSBScirEieH5Js8HNU8h7KoURP+3tuKyuZoKBKZ4b4xAYM21ju2c4WiR1 4aK4OmQATzFZuk6CWDNfrj7QsxoeVwvGzuF4+nIt0ZoWRaQVSsDyDI5xeScFXfRTfj6q apLV1i17uqWHqS/39x3Rm3Esals06Ds7Om0UE3nmzL8rXvj4gzAbX5gbT/0wTst/2OAR EZ8w== X-Gm-Message-State: AOJu0Yy52fdMF4TS1OwuT+m5Xk6jtzIjkWM3Qv2dwKRa7dyHksHtanxG DNsBeO0RGgGKEjxCRDWLp5IH0C4o7IiH63FNZoTf5+49 X-Google-Smtp-Source: AGHT+IFDAfIrBOJK9gk3gym8ARCFT3hUd3XzYIUTHUAk6RyNFWa8iwC6n+z9CU+Rq9RCaO00Bmmnf2PxPbajxIoDZuc= X-Received: by 2002:a17:906:2b0f:b0:a1d:2d2b:2b37 with SMTP id a15-20020a1709062b0f00b00a1d2d2b2b37mr1734033ejg.10.1701979558283; Thu, 07 Dec 2023 12:05:58 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a05:6f02:c916:b0:60:2fe7:f04d with HTTP; Thu, 7 Dec 2023 12:05:57 -0800 (PST) In-Reply-To: <2dc77c3c-3dc5-47a2-b15a-b643f7ff02bf@caida.org> References: <6CA05409-11BD-4733-9CB1-14400736F050@pch.net> <2dc77c3c-3dc5-47a2-b15a-b643f7ff02bf@caida.org> From: Sauli Kiviranta Date: Thu, 7 Dec 2023 22:05:57 +0200 Message-ID: To: Ricky Mok Cc: starlink@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] [NNagain] CFP march 1 - network measurement conference X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Dec 2023 20:06:00 -0000 Thank you Jack, Bill and Ricky for your comments! (And everyone after!) Jack: "> amount (bytes, datagrams) presumed lost and re-transmitted by the sender= " I would consider those lost packets and just recovered through time complexity e.g. retransmission with TCP and that retransmission may require another retransmission etc. Then those re-transmitted data are counted towards overhead. Equivalent for UDP would be redundancy as in paying lost data recovery with Space Complexity. If the payload can not be recovered then that would be considered waste of the whole payload as packet loss addition. "> amount (bytes, datagrams) discarded at the receiver because they were already received" This is simply from my perspective just overhead. Any extra cost in Space Complexity over the original payload is inefficiency and should be just counted as such. "> amount (bytes, datagrams) discarded at the receiver because they arrived too late to be useful" This is more tricky, because we are taking stance on use-case. Better would be just to characterize in terms of time distribution as percentiles at some specific useful intervals e.g. RTT multiples or 100ms, 500ms, 1000ms etc to give rough estimate how much time needs to be paid to recover if feedback loop is involved. "> With such data, it would be possible to measure things like "useful throughput", i.e., the data successfully delivered from source to destination which was actually useful for the associated user's application." Here we have few different components at play. 1. Use-case required bandwidth, use-case consumed bandwidth. 2. Link saturation as non-use-case specific how much of the available path are we able to utilize effectively assuming that use case is equal or larger than this for example for file transfer as we do not have real time budget from the use-case perspective. Bill: "> All of that can probably be derived from sufficiently finely-grained TCP data. i.e. if you had a PCAP of a TCP flow that constituted the measurement, you=E2=80=99d be able to derive all of the above." What I would say that cannot be derived is the behavior of transport considering these combinations. What was very interesting, and what I have also experienced was here: https://www.youtube.com/watch?v=3DXHls8PvCVws&t=3D319s Specially the second talk about measurements at fine grained payload resolution. You see a very specific pattern on TCP when the payload size increases you will introduce extra RTT. Thus I think what is done for example in iperf is not exactly good method as there is just 1 payload size (?) and this result shows that the payload size itself will determine your results. So we need more dimensions in our tests. "> Bandwidth: The data transfer capacity available on the test path. Presumably the goal of a TCP transaction measurement would be to enable this calculation." This in the light of the previous video e.g. "we know X is available, how was it utilized?" "> Transfer Efficiency: The ratio of useful payload data to the overhead da= ta. This is a how-its-used rather than a property-of-the-network. If there are network-inherent overheads, they=E2=80=99re likely to be not directly visible to endpoints, only inferable, and might require external knowledge of the network. So, I=E2=80=99d put this out-of-scope." I think differently, this is extremely important factor, and it is getting more and more important. How much waste there is to useful data. Every single extra bit is waste. Is it redundant data for recovery on non-feedbackloop utilizing transports or just ACK for retransmission the goal is same, we need to construct the payload, either by paying in time budget (wait for rentransmission) or space budget (add enough redundancy to make sure we can recreate regardless of lost data). "> RTT is measurable. If Latency is RTT minus processing delay on the remote end, I=E2=80=99m not sure it=E2=80=99s really measurable, per se, wi= thout the remote end being able to accurately clock itself, or an independent vantage point adjacent to the remote end. This is the old one-way-delay measurement problem in different guise, I think. Anyway, I think RTT is easy and necessary, and I think latency is difficult and probably an anchor not worth attaching to anything we want to see done in the near term. Latency jitter likewise." Yes, this is difficult without time sync. That is why it has to be laboratory conditions or glass to glass across relay. Very very tricky to make sure the timings are correct to no garbage measurements! For truthfully understanding performance from gradient of use-cases we need to be able to distinguish the baseline network fluctuation (e.g. Starlink) and what delays on top of that are caused by our transport.Thus I would really like to keep these 2 separate, as they will allow us to distinguish did we spend extra 300ms on few retransmission loops or did we get extra 5ms latency just as a error from the gradient in the baseline. "> This seems like it can be derived from a PCAP, but doesn=E2=80=99t reall= y constitute an independent measurement." Agree, if we condition the network for lab test, it would be good to be able to derive the conditioned loss to make sure we know what we are doing. "> Energy Efficiency: The amount of energy consumed to achieve the test res= ult. Not measurable." We should try! See attached picture about QUIC, just from the previous email, these things matter when we have constrained devices in IoT or space, or drones etc. We can get artificially good performance if we just burn enormous amount of energy. We should try to make sure we measure total energy spent on the overall transmission. Even if difficult, we should absolutely worry about the cost not only in bandwidth, time but also energy as consequence of computational complexity of some approaches at our disposal even for error correction. "> Did I overlook something? Out-of-order delivery is the fourth classical quality criterion. There are folks who argue that it doesn=E2=80=99t matter anymore, and other= s who (more compellingly, to my mind) argue that it=E2=80=99s at least as relevant as ever." Good point, is this concern for receive buffer size bloating if we need to make jitter buffer and pay in time complexity to recover misordered data? My perspective is mostly on application layer so forgive if I missed your point. "Thus, for an actual measurement suite: - A TCP transaction =E2=80=A6from which we can observe: - Loss - RTT (which I=E2=80=99ll just call =E2=80=9CLatency=E2=80=9D because that= =E2=80=99s what people have called it in the past) - out-of-order delivery - Jitter in the above three, if the transaction continues long enough =E2=80=A6and we can calculate: - Goodput" I see, yes, my comments would have been doing all that TCP/LTP/UDP/QUIC... whatnot to have finally good overview where we stand as humanity on our ability to transfer bits for arbitrary use-case with quadrant on data intensity X axis and event rate on Y axis followed by environmental factors as latency and packetloss. "In addition to these, I think it=E2=80=99s necessary to also associate a traceroute (and, if available and reliable, a reverse-path traceroute) in order that it be clear what was measured, and a timestamp, and a digital signature over the whole thing, so we can know who=E2=80=99s attest= ing to the measurement." Yes, specially if we do tests in the wilderness, fully agreed! Overall all tests should be so accurately documented that they can be reproduced within error margins. If not, its not really science or even engineering, just artisans with rules of thumb! Best regards, Sauli On 12/7/23, Ricky Mok via Starlink wrote: > How about applications? youtube and netflix? > > (TPC of this conference this year) > > Ricky > > On 12/6/23 18:22, Bill Woodcock via Starlink wrote: >> >>> On Dec 6, 2023, at 22:46, Sauli Kiviranta via Nnagain >>> wrote: >>> What would be a comprehensive measurement? Should cover all/most releva= nt >>> areas? >> It=E2=80=99s easy to specify a suite of measurements which is too heavy = to be >> easily implemented or supported on the network. Also, as you point out, >> many things can be derived from raw data, so don=E2=80=99t necessarily r= equire >> additional specific measurements. >> >>> Payload Size: The size of data being transmitted. >>> Event Rate: The frequency at which payloads are transmitted. >>> Bitrate: The combination of rate and size transferred in a given test. >>> Throughput: The data transfer capability achieved on the test path. >> All of that can probably be derived from sufficiently finely-grained TCP >> data. i.e. if you had a PCAP of a TCP flow that constituted the >> measurement, you=E2=80=99d be able to derive all of the above. >> >>> Bandwidth: The data transfer capacity available on the test path. >> Presumably the goal of a TCP transaction measurement would be to enable >> this calculation. >> >>> Transfer Efficiency: The ratio of useful payload data to the overhead >>> data. >> This is a how-its-used rather than a property-of-the-network. If there >> are network-inherent overheads, they=E2=80=99re likely to be not directl= y visible >> to endpoints, only inferable, and might require external knowledge of th= e >> network. So, I=E2=80=99d put this out-of-scope. >> >>> Round-Trip Time (RTT): The ping delay time to the target server and bac= k. >>> RTT Jitter: The variation in the delay of round-trip time. >>> Latency: The transmission delay time to the target server and back. >>> Latency Jitter: The variation in delay of latency. >> RTT is measurable. If Latency is RTT minus processing delay on the remo= te >> end, I=E2=80=99m not sure it=E2=80=99s really measurable, per se, withou= t the remote end >> being able to accurately clock itself, or an independent vantage point >> adjacent to the remote end. This is the old one-way-delay measurement >> problem in different guise, I think. Anyway, I think RTT is easy and >> necessary, and I think latency is difficult and probably an anchor not >> worth attaching to anything we want to see done in the near term. Laten= cy >> jitter likewise. >> >>> Bit Error Rate: The corrupted bits as a percentage of the total >>> transmitted data. >> This seems like it can be derived from a PCAP, but doesn=E2=80=99t reall= y >> constitute an independent measurement. >> >>> Packet Loss: The percentage of packets lost that needed to be recovered= . >> Yep. >> >>> Energy Efficiency: The amount of energy consumed to achieve the test >>> result. >> Not measurable. >> >>> Did I overlook something? >> Out-of-order delivery is the fourth classical quality criterion. There >> are folks who argue that it doesn=E2=80=99t matter anymore, and others w= ho (more >> compellingly, to my mind) argue that it=E2=80=99s at least as relevant a= s ever. >> >> Thus, for an actual measurement suite: >> >> - A TCP transaction >> >> =E2=80=A6from which we can observe: >> >> - Loss >> - RTT (which I=E2=80=99ll just call =E2=80=9CLatency=E2=80=9D because = that=E2=80=99s what people have >> called it in the past) >> - out-of-order delivery >> - Jitter in the above three, if the transaction continues long enough >> >> =E2=80=A6and we can calculate: >> >> - Goodput >> >> In addition to these, I think it=E2=80=99s necessary to also associate a >> traceroute (and, if available and reliable, a reverse-path traceroute) i= n >> order that it be clear what was measured, and a timestamp, and a digital >> signature over the whole thing, so we can know who=E2=80=99s attesting t= o the >> measurement. >> >> -Bill >> >> >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink >