From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout21.telekom.de (mailout21.telekom.de [194.25.225.215]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E13DE3B2A4 for ; Wed, 2 Nov 2022 05:41:16 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telekom.de; i=@telekom.de; q=dns/txt; s=dtag1; t=1667382077; x=1698918077; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=lcn3ke6Ij0AQ070Pwi88RoWWGKD4++JTDMzNz5Sx8n0=; b=G76hAe12oNPJOnXD+P3js/CwyfZvgtSaPI5NAuyr5EvLl0W11tF1nx7A 3tyVnUn8J9+reMfLCJ5gl1KXf3Z/T33A1xwDgeI+POy5icqzZ+quYVnqP 56HDMbNwtIyC1dbNQSHgbk/GdnTBq+05whm1Qs1ypSDNCxr0h1NPeHyjC Q9loLWVM+FKVkMaABAXa48vpVlwAz3niyILone98ADSgJYHhlt0VWS3TI yCqaM3SvrBIwm9KWBHpxk2LvNv7Fzqn1MfvPKvLl7AI0YKcPSlGSANrUG 2iAPYEeDCz1GVv5ClLZDkqlVe85GURkbi4B8gTM6CpOF/nPgH3yK7rQG4 A==; Received: from qde9xy.de.t-internal.com ([10.171.254.32]) by MAILOUT21.dmznet.de.t-internal.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 02 Nov 2022 10:41:15 +0100 IronPort-SDR: LU2wOf1iq5Rwlb602LMTJydCOZiyy1vsuDmAi8PEZRNGrNpmWOMN6fnqDMxF41UcKqNDEmafNX wQzD5MVTK3kjBROgHdg8q+kW9BP96kPmM= X-IronPort-AV: E=Sophos;i="5.95,232,1661810400"; d="scan'208";a="599394743" X-MGA-submission: =?us-ascii?q?MDFE3oYElU83vALCnhXoiF0QGn9DaqA3OiksPh?= =?us-ascii?q?FVwgg2g9nWiqYlc7KSl0K6Ch5rMlOxp0qiI58mpffpccrUtGm5s5et2D?= =?us-ascii?q?4JyJKb3v3ZRgLx8iQoYzWKnzNefa/gUkToS3o6l8ZpTwOnXXO5HpO5/I?= =?us-ascii?q?QbhDss1CCPjNwqe5HACMEyow=3D=3D?= Received: from he105717.emea1.cds.t-internal.com ([10.169.118.53]) by QDE9Y1.de.t-internal.com with ESMTP/TLS/ECDHE-RSA-AES128-SHA256; 02 Nov 2022 10:41:15 +0100 Received: from HE105716.EMEA1.cds.t-internal.com (10.169.118.52) by HE105717.emea1.cds.t-internal.com (10.169.118.53) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Wed, 2 Nov 2022 10:41:14 +0100 Received: from HE104160.emea1.cds.t-internal.com (10.171.40.36) by HE105716.EMEA1.cds.t-internal.com (10.169.118.52) with Microsoft SMTP Server (TLS) id 15.0.1497.42 via Frontend Transport; Wed, 2 Nov 2022 10:41:14 +0100 Received: from DEU01-FR2-obe.outbound.protection.outlook.com (104.47.11.175) by O365mail03.telekom.de (172.30.0.232) with Microsoft SMTP Server (TLS) id 15.0.1497.36; Wed, 2 Nov 2022 10:41:14 +0100 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c/OlX+95n85kfQoOP4E41emnFPnr6iszxhP30TfLivfRw/mWoVRvgt4wnZg+BeqzKDjuzLO1gwnJBz3fcnbCXqvPwJoUiHUWR2m1oCovk5PP1k3bKpuj9z9/UFxBORT4+6RCfj/D5K2V0b4BUTMot3BQduDu3FbbOLhxabtYXlVPqI1lJZEh/7OY6yrJRqwgSxsp7mCJpacBOzGJLn1OPkvamgCqkf0Bgo5/4fJFwXM89KfVvEhzClcndG2Gx2stX6jBNfsMtgA9qVn5vrw+DhZlgdI/vdBBCYwgeKTLlwOhdCFY1bkRWh0o4Zijrs/k2LCoLgShDgQNZESxSGKDrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lcn3ke6Ij0AQ070Pwi88RoWWGKD4++JTDMzNz5Sx8n0=; b=LQhNdYxWR5SoXeSvN37/NhWhK8JLugZ1fQ5Be2kBC7n3PTMcVPvod2TUeVXwmdsTkBpGW5ALDlCQrgC73MQnfa/efK++GFc6rYzJxNbMnOsF9VLE1QmWlv18kYP+CSvsNhnHd53CQPucu9zoez/cfZOVwHwRooIWZnVfq985vDE5dQfH2qwTNgw3WnW9wAOx4TcG2p00VdERfAsX5Dw8zy/3Lj/0nW97a5WzRQ5r8qOLjNJm3V3KD3E8Yj2+EKEkZ0KOydteRyN4db9u+Bd5vje9gilqfrf5iIyCfmwuzeC0axiNosabXc0TIg5nXv59/rgw9ns3TPzmPKjIioLJnA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=telekom.de; dmarc=pass action=none header.from=telekom.de; dkim=pass header.d=telekom.de; arc=none Received: from FR2P281MB1527.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:8b::11) by BE1P281MB2950.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:4f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5769.21; Wed, 2 Nov 2022 09:41:13 +0000 Received: from FR2P281MB1527.DEUP281.PROD.OUTLOOK.COM ([fe80::54f3:8989:c087:dabf]) by FR2P281MB1527.DEUP281.PROD.OUTLOOK.COM ([fe80::54f3:8989:c087:dabf%3]) with mapi id 15.20.5791.022; Wed, 2 Nov 2022 09:41:13 +0000 From: To: , CC: , Thread-Topic: [ippm] [Rpm] lightweight active sensing of bandwidth and buffering Thread-Index: AQHY7pSSe1jemA1+oU+yIbz2XUvgw64rWCug Date: Wed, 2 Nov 2022 09:41:13 +0000 Message-ID: References: <0a8cc31c7077918bf84fddf9db50db02@rjmcmahon.com> <344f2a33b6bcae4ad4390dcb96f92589@rjmcmahon.com> <261B90F5-FD4E-46D5-BEFE-6BF12D249A28@gmx.de> In-Reply-To: <261B90F5-FD4E-46D5-BEFE-6BF12D249A28@gmx.de> Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=telekom.de; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: FR2P281MB1527:EE_|BE1P281MB2950:EE_ x-ms-office365-filtering-correlation-id: 5b0ed156-0dcd-4e5f-cff8-08dabcb6619f x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: bA1eYP2WA78JqceyaCG+AqChwSlXVW4R/OLgUv0fMNS7uh+QMU8ig1YfS0r+o7j938ewQToxP/h4Iq2YbVN0pHZ9NjVfHnoGepOjMCVTSfr8BlnfNuuIwj7LLJNg7+G1Ui10MBaJ9XKsLN9rVs8X1ONJPIvfHi37/aW3fvEUovu4VZe59fA5DzDGrSdIZIs3Jxy5SKYYYxnoMGthJsc0hHF2zdtdQd67Kiavfeq2VzawF4n+eg3IabhE5Uzs6OyOxGQvI4uvud8NbKBfO8OXFSDEHrJr/CGwn//I3Iby6eScOtIF4oCfK2vGiqtLHPNm4BweweVKnqjigxA41vdt+vvw7CGlh6Ynopehl0JsSMOid+k2pZ8BGJVbvYIvsGr2aNhdSvNbG6Mss8rrYJpZ8akURYwVc0ScsuT7cLFNOyS9sY51TyxZqOaLnMNy3XGdMTRFHyYI++nx5EHKxmh9QhiGp+N9vkQi5Y0c21Pfs2ajMjlyKyXkxV44LsWMyuCi0VKo/pbpau1fnKVEfAvzehv5ERH8xyrMNBDJYCVm1iOO+n1w+ZRpbPbEW2V/W0vUUHIi3/LnsLJDdMauM+/i7TLSeAu1VzYQLX70k0GtwUutCURrjLLdJR7le/mMmS+zl+KfUchr/KwYFpgnXAQVO2Qwp1pWoGUN0aziWpRSrZvBnkSvXP3aDvuERWh122ufJ9p59EJlZ1cGctfBSGuvhcXkUag4AUIeakBuWYoAMeKZuIcKoWg6KP7R8fAk91b6Am94AdXaXNUE2Ucu0e71DvgV/IMDJIxbiXSzeEqYJhY= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:FR2P281MB1527.DEUP281.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(13230022)(4636009)(366004)(396003)(39860400002)(136003)(376002)(346002)(451199015)(38070700005)(33656002)(86362001)(2906002)(5660300002)(38100700002)(82960400001)(122000001)(83380400001)(66446008)(4326008)(52536014)(66946007)(76116006)(8676002)(66476007)(66556008)(7696005)(64756008)(6506007)(53546011)(186003)(478600001)(8936002)(71200400001)(316002)(9686003)(26005)(54906003)(41300700001)(966005)(110136005)(55016003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?d2SVQs/LRz6zolzD3RyYq6I3eChlqTnHgHQ+5BW8QfE2+s3ETMCO6VAOjV?= =?iso-8859-1?Q?NIJ+/WTP3E5/HUqMTJvWcADOy0qm86qKwsPDqEH5fFcbAikZ9/TrySFsLm?= =?iso-8859-1?Q?qt6BlpzzNQlYWwf7vf/2GlwQFa2RYG3Z0pU2c+dXMBV4Ez0M2utCuHeHTM?= =?iso-8859-1?Q?XBE2FFddkXJUCaRzKUL73uLIm+G9IyhAop5zY3oWopyTBU0jmTF2BfWZuW?= =?iso-8859-1?Q?XjNRztsZtKCqfPGVeKOaugK/JFX6PFw6fSve7ixwL7pEtalBI0UcfPoklL?= =?iso-8859-1?Q?+hxYwKL0aqi3NDuQKydNLfGXbmbPHa7SOO+cfAokzAgUa/xhxnYV0NEZZd?= =?iso-8859-1?Q?3gXNGkA5T/vz+8oMYg2VaEcYuhTSjP1IvjLV8TDnPkBxpbiebhY42khYya?= =?iso-8859-1?Q?KglYt2zjJpX8tJMj6Dvxo1AL8Kfs9mA57dPHQFrnVZYGwk+t/vAfMcvr+q?= =?iso-8859-1?Q?J8rP5UWh2AWvrbXaieZ5DtYHTJnN6iJwR9XL7KiRlBbq5gyMqGawb2IoZB?= =?iso-8859-1?Q?DEr2j379JBvGWG+Q1MiTh+0wQ3cT1p4R24F75nQfCxALGYej74yhsd7CwB?= =?iso-8859-1?Q?BgFIXKZuE99KK5v19RmMIQDQv8774Hv4wuwgq2OBNps1W4V1RlOBGE7/7t?= =?iso-8859-1?Q?/GA9LEX/ZhcwW8TSIJ23YBAPYdMgbi2sOMNsEkyOqw4dyYQRAZdo2mTgVp?= =?iso-8859-1?Q?TMUxyKrW94oaWdyPtyhkRxYQLUljFpVHKC64PSTqYdgRIfVgkEquJ0Hcni?= =?iso-8859-1?Q?TicmYHAJ30FAzlbiyy7g+GsySoHIW7c0hUUmY1gB0V+BbPYUkgy223ulpc?= =?iso-8859-1?Q?edS4DRbdw+1mL0NppYX/LWQGm743U+2wdwNksjaTQABfwWMZ8XAK5E0xT2?= =?iso-8859-1?Q?NATmzZqa2J9yrpFMzPVjkg3V1G1eGup51VVnhvnSWutKerLG9g6LkUEGYp?= =?iso-8859-1?Q?+WE0xNtvX/y0Y/8ugS9Ar4ewieliMK01UXxeJMyNv6YBNcA1wYli7Q0Leq?= =?iso-8859-1?Q?5NmodhUULhSkfHoTh9y+PXK/FOSqV1io456ottqFh/1EReir0pifYDDymO?= =?iso-8859-1?Q?/GgoU3y6D8FwcvOC11jgNCGbkTn0Kmty4IktJuy2cVYGQsBPc5nZdtlgFE?= =?iso-8859-1?Q?5hiWG9MkfbErA0SRUpQXT1uH4X0KzF0qJwFFr81SneIFiQ0DkxRyDRBCpE?= =?iso-8859-1?Q?bYeCaNEPzbKECd+y/smuKkMUhSMJv1gIUwHs71RY2qd+5W1PkR30VGUQoD?= =?iso-8859-1?Q?y/ZTNrOh6b1TwhFZ8CY1aKXC6TJ4/i5IoSB8mIFJFiCaLvh+sx3yufJlDj?= =?iso-8859-1?Q?srweHPhHRpSVG/qqN0WJQD8Hzh06HlhOrR2NNWHZbVeZF05Hfo9s5XtP91?= =?iso-8859-1?Q?qoP5J+yfQcnnVMkCwo/zi+oVZB+hBYYgXlvu50Rg5OkAyoSnSk9fcxJ8a4?= =?iso-8859-1?Q?tK+LwWeDeKMdtuX/ibXOp3By2kr7AfjTESL2p1HsCTJyr+ZhV6xzVQT9Pc?= =?iso-8859-1?Q?iA39bYxvpkEZ36kkGzbEU+e4RtuaphNvHgnDMD1J9Zf5B8rup08LVVwGIT?= =?iso-8859-1?Q?H/B4TG9EfzftVbb5Cs+ESuwPpnRrMUZq1QoSltQ2psuj5gED2ZKiTfUwOo?= =?iso-8859-1?Q?wa21v0UNriVGnc9lvFMYRTnsR+lQGreNGZ?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: FR2P281MB1527.DEUP281.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 5b0ed156-0dcd-4e5f-cff8-08dabcb6619f X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Nov 2022 09:41:13.4123 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bde4dffc-4b60-4cf6-8b04-a5eeb25f5c4f X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 9YPQxprLl/wDtZ/De+UDHM9OLIfYp9D2eZoGjrYMhV9gabboj+bi+JPv+ambRV1U7rGJ4CFxQm8VzKAFdfhPanFKCR7YAltzcgMN3HF3fS0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BE1P281MB2950 X-OriginatorOrg: telekom.de Subject: Re: [Rpm] [ippm] lightweight active sensing of bandwidth and buffering X-BeenThere: rpm@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: revolutions per minute - a new metric for measuring responsiveness List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Nov 2022 09:41:17 -0000 Bob, Sebastian, not being active on your topic, just to add what I observed on congestion: - starts with an increase of jitter, but measured minimum delays still rema= in constant. Technically, a queue builds up some of the time, but it isn't = present permanently. - buffer fill reaches a "steady state", called bufferbloat on access I thin= k; technically, OWD increases also for the minimum delays, jitter now decre= ases (what you've described that as "the delay magnitude" decreases or "min= imum CDF shift" respectively, if I'm correct). I'd expect packet loss to oc= cur, once the buffer fill is on steady state, but loss might be randomly di= stributed and could be of a low percentage. - a sudden rather long load burst may cause a jump-start to "steady-state"= buffer fill. The above holds for a slow but steady load increase (where th= e measurement frequency determines the timescale qualifying "slow"). - in the end, max-min delay or delay distribution/jitter likely isn't an ea= sy to handle single metric to identify congestion. Regards, Ruediger > On Nov 2, 2022, at 00:39, rjmcmahon via Rpm w= rote: >=20 > Bufferbloat shifts the minimum of the latency or OWD CDF. [SM] Thank you for spelling this out explicitly, I only worked on a vage i= mplicit assumption along those lines. However what I want to avoid is using= delay magnitude itself as classifier between high and low load condition a= s that seems statistically uncouth to then show that the delay differs betw= een the two classes;).=20 Yet, your comment convinced me that my current load threshold (at least fo= r the high load condition) probably is too small, exactly because the "base= " of the high-load CDFs coincides with the base of the low-load CDFs implyi= ng that the high-load class contains too many samples with decent delay (wh= ich after all is one of the goals of the whole autorate endeavor). > A suggestion is to disable x-axis auto-scaling and start from zero. [SM] Will reconsider. I started with start at zero, end then switched to a= n x-range that starts with the delay corresponding to 0.01% for the reflect= or/condition with the lowest such value and stops at 97.5% for the reflecto= r/condition with the highest delay value. My rationale is that the base del= ay/path delay of each reflector is not all that informative* (and it can st= ill be learned from reading the x-axis), the long tail > 50% however is whe= re I expect most differences so I want to emphasize this and finally I want= ed to avoid that the actual "curvy" part gets compressed so much that all l= ines more or less coincide. As I said, I will reconsider this *) We also maintain individual baselines per reflector, so I could just plo= t the differences from baseline, but that would essentially equalize all re= flectors, and I think having a plot that easily shows reflectors with outly= ing base delay can be informative when selecting reflector candidates. Howe= ver once we actually switch to OWDs baseline correction might be required a= nyways, as due to colck differences ICMP type 13/14 data can have massive o= ffsets that are mostly indicative of un synched clocks**. **) This is whyI would prefer to use NTP servers as reflectors with NTP req= uests, my expectation is all of these should be reasonably synced by defaul= t so that offsets should be in the sane range.... >=20 > Bob >> For about 2 years now the cake w-adaptive bandwidth project has been=20 >> exploring techniques to lightweightedly sense bandwidth and=20 >> buffering problems. One of my favorites was their discovery that ICMP=20 >> type 13 got them working OWD from millions of ipv4 devices! >> They've also explored leveraging ntp and multiple other methods, and=20 >> have scripts available that do a good job of compensating for 5g and=20 >> starlink's misbehaviors. >> They've also pioneered a whole bunch of new graphing techniques,=20 >> which I do wish were used more than single number summaries=20 >> especially in analyzing the behaviors of new metrics like rpm,=20 >> samknows, ookla, and >> RFC9097 - to see what is being missed. >> There are thousands of posts about this research topic, a new post on=20 >> OWD just went by here. >> https://forum.openwrt.org/t/cake-w-adaptive-bandwidth/135379/793 >> and of course, I love flent's enormous graphing toolset for=20 >> simulating and analyzing complex network behaviors. > _______________________________________________ > Rpm mailing list > Rpm@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/rpm _______________________________________________ ippm mailing list ippm@ietf.org https://www.ietf.org/mailman/listinfo/ippm