From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 491423B2A4 for ; Sun, 17 Mar 2019 07:45:27 -0400 (EDT) Received: from [192.168.42.220] ([77.182.103.198]) by mail.gmx.com (mrgmx103 [212.227.17.168]) with ESMTPSA (Nemesis) id 0Mbfyr-1hMKek08PN-00J1LL; Sun, 17 Mar 2019 12:45:15 +0100 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) From: Sebastian Moeller In-Reply-To: <94B04C6B-5997-4971-9698-57BEA3AE5C0E@tzi.org> Date: Sun, 17 Mar 2019 12:45:13 +0100 Cc: Greg White , Ingemar Johansson S , "bloat@lists.bufferbloat.net" Content-Transfer-Encoding: quoted-printable Message-Id: <166A7220-875F-4FA0-A8EE-17F11037EC76@gmx.de> References: <94B04C6B-5997-4971-9698-57BEA3AE5C0E@tzi.org> To: Carsten Bormann X-Mailer: Apple Mail (2.3445.9.1) X-Provags-ID: V03:K1:ExBDv4M3h3BeG82zYbTjBxle5EzTIppXuFjYrDcOfDO8ECQNjRc r69DMn3dazic/ZoFRQaeOEoBqGWG1jPu7ToSMrbVDbzGWSRVbTYHQZw2ARCI7zanYXwJ1jZ qemgTbid3fe4UZCFNFMmpvOIy+0dg9uoW09GU4J/J5/AIXDNhW179FFDlW/8FmRmRIrT9cV TziWJfUKmpLzLnrEfHQLA== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:snLElWzG/yQ=:RTdveK5pLUUxtaVOGibW/s 15mj2N2zIPJsUnJ5b2DDd1atluFhIQkAPQlRYmljrITMSkx5AxSBgvoOb53liYPiTsnf72LCw UhQZCE+7Jd1ZkuNJeN5eW1PvC6UzV3tddzcspdGtlk5NWxsHdl2LJkoQxJfyT5SuFJWcAo1gG KoSDijcSIMnaFZW/Anhgz2pvGpoo5q7BNiL4v1sL6Qr+AJzjo6GPuAKbZdwi8Pto7EveUrwPr AinTXewmmoMRd879X4vLFaO/YQnsY4485hYQ6D9GxVIVtTXp0U9NbV0vYCXKUX+XFcM8hiObD hjxbB/t8M1uCE4iHXw3jjN9ZXKiIfSG0n2q0GcqMAjp/khBEjO8cOsvX2ZMY56Zwd6yytPD+X 5mkfZ4Q9ZSXfmeklWr+Br98W0aaS6SsYeIjT3JnUZydZ3OhYQ1EuOQV5AtsPKUAG46zsEYkW6 jebDkphR/0zH1FQNDpLoLKL9+p+FDvVNNf8vmjxVE1k9wkmnRwcNttMmdyypSNIHV0b5qkqn2 xskw+gxk1Zc0FbrqHq8TEQqVBmIlI+7cJaGS/TN+q9IkqRIOn2b8ekFrZff6POOhafvML0vWj rjltoSaiGkEbSX+gpGK8G9PRIaF9wfRLN6YJ0RPHRMSsD5tyMSpdQbVkmkc1aVj41UzLbGtBD eVQENJAX+c9pgztW8OVy/sV/Is0VCXv4lH1UR1W/GH9/zvd2D+AtHZ0bzuxsNtMTJzrKMr23M gsDi02BRITX1pDcaP0aWumMzpWyi/h7gWQ1W9JN+o1CGnASjfioL8yTkmJL17tcaULoYn9qTC 6wugxWsf+GuIMwuQ4oGZ0WojLmVIgK7O600mGQ9BwL9Myp9awgoLCKUg++cg8aYUGw4vCr9bP kmSrDFdC2tUprWegAVCHrfCcx/co3RQcOOuEtv/HVOXTxxzYA+Wsr8Pz8Z0IzOGSgD42quOJY q6l9UJ9q+KQ== Subject: Re: [Bloat] Packet reordering and RACK (was The "Some Congestion Experienced" ECN codepoint) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Mar 2019 11:45:27 -0000 Hi Carsten, > On Mar 17, 2019, at 11:23, Carsten Bormann wrote: >=20 > On Mar 14, 2019, at 22:43, Sebastian Moeller wrote: >>=20 >> if a specific link technology is prone to introduce reordering due to = retransmit it might as well try to clean up after itself >=20 > The end-to-end argument applies: Ultimately, there needs to be = resequencing at the end anyway, so any reordering in the network would = be a performance optimization. It turns out that keeping packets lying = around in some buffer somewhere in the network just to do resequencing = before they exit an L2 domain (or a tunnel) is a pessimization, not an = optimization. I do not buy the end to end argument here, because in the = extreme why do ARQ on individual links anyway, we can just leave it to = the end-points to do the ARQ and TCP does anyway. The point is = transport-ARQ allows to use link technologies that otherwise would not = be acceptable at all. So doing ARQ on the individual links already = indicates that somethings are more efficient to not only do e2e. I just = happen to think that re-ordering falls into the same category, at least = for users stuck behind a slow link as is typical at the edge of the = internet. To put numbers to my example, assume I am on a 1/1 Mbps link and I get = TCP data at 1 Mbps rate and MTU1500 packets (I am going to keep the = numbers approximate) and I get a burst of say 10 packets containing say = 10 individual messages for my application telling the position of say an = object in 3d space each packet is going to "hog" the link for: 1000 ms/s * (1500 * 8 = b/packet ) / (1000 * 1000 b/s) =3D 12 ms So I get access to messages/new positions every 12 ms and I can display = this smoothly Now if the first packet gets r-odered to be last, I either drop that = packet and accept a 12 ms gap or if that is not an option I get to wait = 9*12 =3D 108ms before positions can be updated, that IMHO shows why = re-ordering is terrible even if TCP would be more tolerant. Especially = in the context of L4S something like this seems to be totally = unacceptable if ultra-low latency is supposed to be anything more than = marketing.=20 >=20 > For three decades now, we have acted as if there is no cost for = in-order delivery from L2 =E2=80=94 not because that is true, but = because deployed transport protocol implementations were built and = tested with simple links that don=E2=80=99t reorder. =20 Well, that is similar to the argument for performing non-aligned = loads fast in hardware, yes this comes with a considerable cost in = complexity and it is harder to make this go fast than just allowing = aligned loads and fixing up unaligned loads by trapping to software, but = from a user perspective the fast hardware beats the fickle only make = aligned loads go fast approach any old day. > Techniques for ECMP (equal-cost multi-path) have been developed that = appease that illusion, but they actually also are pessimizations at = least in some cases. Sure, but if I understand correctly, this is partly due to the = fact that transport people opted not to do the re-sorting on a = flow-by-flow basis; that would solve the blocking issue from the = transport perspective, sure the affected flow would still suffer from = some increased delay, but as I tried to show above that might be still = smaller than the delay incurred by doing the re-sorting after the = bottleneck link. What is wrong with my analysis? >=20 > The question at hand is whether we can make the move back to = end-to-end resequencing techniques that work well, But we can not, we can make TCP more robust, but what I predict = if RACK allows for 100ms delay transports will take this as the new the = new goal and will keep pushing against that limit; and all in the name = of bandwidth over latency. > at least within some limits that we still have to find. > That probably requires some evolution at the end-to-end transport = implementation layer. We are in a better position to make that happen = than we have been for a long time. Probably true, but also not very attractive from an end-user = perspective.... unless this will allow transport innovations that will = allow massively more bandwidth at a smallish latency cost. Best Regards Sebastian >=20 > Gr=C3=BC=C3=9Fe, Carsten >=20