From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670121.outbound.protection.outlook.com [40.107.67.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 75D253B2A4 for ; Thu, 4 Jun 2020 08:25:25 -0400 (EDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hrkszTt3dXa1fsuWJKoYCzZ3nrvoampFsxHE+Q6eYdlTY749GcuSY9m4zRFfuiblNHoA+1SBC6cQzD+5u9p0GkCkS7gtxpXszG052lb55lw4ruqSizATBfyFJGPQf8T0alI20s/WhwhuzZflYFv548rQviuewaQOlHQOaS3zPs68mz/dX7rXXgMr7SyxHj5aqO+icorldVwA/Z31fwmf00vRrWHYQ7U1Lftl2llfJwxSOtLbdv3sUfJad5BcdVzTO4vx+QBtIIwQ+bKKxr+6LVIy6Y/E9VQoXfbCLHxaLduOSFPAxBjAUIihMzn5dhr5ztSvd7isjvv2iqcNa3wXoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WI5B+Jq/p+isc2rMTnb/JWhjj3LQHlBaTZHiQb6XA/I=; b=P026LjQaerBcCxlr1+xwibVsSqMTq5C/doILX5+BA50WgRuAdBGOQmpF2WlNC5rSBIPv3RRDgoBVwz6gp6kt8mRA8DSMcmdcHCYfCiAJTTzV6otYYdbkt9B47Uk194vbf22I+4FFPCVpEdkVJeRVDn1Yrxjd1B0jS7usc5zPbHmeHeGhhoaIY/G8tCiVoFZxjgHDpA3205rxBzpJWYwozmM0Q/P6nDpE2aLZEf34jzTsDUa0PAf5PFoUlRw4DXfaK3M5wI6eo4/TUJzASNvRxYI5WmArvoXaIaaEIieixLPYbp7x3CViNUH3wlr776PdaR5oPTdXOcPRJZK7BPA2jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=indexexchange.com; dmarc=pass action=none header.from=indexexchange.com; dkim=pass header.d=indexexchange.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=indexexchange.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WI5B+Jq/p+isc2rMTnb/JWhjj3LQHlBaTZHiQb6XA/I=; b=ixAUTq9IzvjCeexob8w1tLOU9uH4bTvQ+LtTdGb5Nxrn4pQHG3HkWKnZclr22kajx5nN765C1HQ4j1zP0+y+nk63Yx4jccsH3Wyjm1w3SuWSLXpNe5IszjCt+47zlNjPccYPfNuUd5ToDKjtbQNpbcKPeXzIZjEiqHpmaVY4bkw= Authentication-Results: indexexchange.com; dkim=none (message not signed) header.d=none;indexexchange.com; dmarc=none action=none header.from=indexexchange.com; Received: from YT1PR01MB2492.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:e::27) by YT1PR01MB3931.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:12::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3066.18; Thu, 4 Jun 2020 12:25:24 +0000 Received: from YT1PR01MB2492.CANPRD01.PROD.OUTLOOK.COM ([fe80::fc35:ff09:c066:9011]) by YT1PR01MB2492.CANPRD01.PROD.OUTLOOK.COM ([fe80::fc35:ff09:c066:9011%5]) with mapi id 15.20.3066.018; Thu, 4 Jun 2020 12:25:24 +0000 Reply-To: dave.collier-brown@indexexchange.com To: =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , Jonathan Morton Cc: bloat References: <87k10njdzj.fsf@toke.dk> From: Dave Collier-Brown Organization: Index Exchange Message-ID: Date: Thu, 4 Jun 2020 08:25:22 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 In-Reply-To: <87k10njdzj.fsf@toke.dk> Content-Type: multipart/alternative; boundary="------------40EE4B8372C7027D3233DC2D" Content-Language: en-US X-ClientProxiedBy: CH2PR19CA0025.namprd19.prod.outlook.com (2603:10b6:610:4d::35) To YT1PR01MB2492.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:e::27) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [IPv6:2607:fea8:5620:487::e183] (2607:fea8:5620:487::e183) by CH2PR19CA0025.namprd19.prod.outlook.com (2603:10b6:610:4d::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3066.18 via Frontend Transport; Thu, 4 Jun 2020 12:25:23 +0000 X-Originating-IP: [2607:fea8:5620:487::e183] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4df7dc7c-2b12-49f7-de08-08d808825adb X-MS-TrafficTypeDiagnostic: YT1PR01MB3931: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-Forefront-PRVS: 04244E0DC5 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kQbQKZMfqRP+jYm7FSuvKX2BXZboH3JFzOLhuSsmUQzSfgWW0taKj6fudcYCsAm9wSEtCE21gzGddGzyPkZgrUAXhbHSINFgJhxo2k3upHAZuP24wIDbt7KrsN1e/CmxsT9Y1W27/q0ecBt52HVBjcbbVlgHRAzoUd4QeYeUoCwmGgzJNp+gdJkifndxz0ZeOsbMcsXf6ncxIU1BJpcVUeSsvnunZowNeos6LVMJn0zFZ4K3PUSzVQfDXtYfUxOn7vgybO1xb7hT46X7A1h+NXOV8a8ZmAyChHPHao731iyMVOJVDhqlZ7VIPWHxwM1HCsvZXiRGPnppY0AQok9TGOFdwr3IBaMbSQYq7w1Sr/cOrABryctnw/OyUnF/82rRq93fxhdnI9njdjoAd9qefpei/NIhJ4q9y6iqkbS4xFBQIgzI1PC+02BzLRiOygqx09EyGFRgA52LSd78aNGea0dKe3VTX8F/jHL/MaTmpmI= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YT1PR01MB2492.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(136003)(396003)(376002)(346002)(366004)(39860400002)(6486002)(31696002)(53546011)(110136005)(33964004)(36916002)(2616005)(316002)(31686004)(5660300002)(52116002)(8676002)(86362001)(8936002)(2906002)(3450700001)(66476007)(478600001)(186003)(66946007)(166002)(66574014)(36756003)(83380400001)(66556008)(4326008)(16526019)(43043002)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: LK3AxgJeZWoBhY6qZ5R6p2oYuQLjmSEDA7fsjZeYEHjGBzMNvB+R80MobHjqzEfE6eotFaLi0PqW6NjOrfqZx6+MUgxr+JKQGt4uiLHuv3NG+QCTBCuSuSZAobg2yOPdxW/kybxCtOPTQ8hdID0zQndgh9w3Tun5/p5iLARcswtVsKZ4hZCDzGImL6VSkDXh2KvRPzst5/SsQg3NHcdL+Y1GXRdKmgFS65gPf1fGUw62meBA9eTBJCcPDgydQuuxLNuNWQ2JNvM7k/SqrNb0b8iYPpyZwjvAaFdWfc6SF9r9FLaGOlZ5qy/+WqPY0Q2L8QdtjyxutZwB9zssw0MzxpGhOdBdauGLVzfTh22x8gMDuUBSk6lzP9X70ISqMF+oVnUcS6V5VZgdJMpgAn+p4+JZF4LpttokvCVw0MRgo65JCXPQ+wTrP/w1Xjxd2AKl/VioqTfAe1uhTtwe9pdJp3rlUVQAA6uz+431sjh77QJFWDGsyw8k2EzihhK9+JNZMf8YencKn3NMK056JlpvHg== X-OriginatorOrg: indexexchange.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4df7dc7c-2b12-49f7-de08-08d808825adb X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2020 12:25:24.0641 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: b07c0690-22b8-4366-8d8d-7b845d088e18 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YDnhDT3YnVauciBEQtKrggX1msw/k4S4wvI7t7hRiFJ4P1H/m+3cYzDSGvcOI5+bUrFoDRYBDoRg3HIgrum9WJb0o7McE/MSLDA9tgHo/NDY2hqzAep8xTZJDWrJyCx2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT1PR01MB3931 Subject: Re: [Bloat] What's a good non-intrusive way to look at bloat (and perhaps things like gout (:-)) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2020 12:25:25 -0000 --------------40EE4B8372C7027D3233DC2D Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable Thanks, that sounds like a good long-term approach. I know our CTO loves eB= PF! As and introduction I'll start prothletising pping (;-)) --dave On 2020-06-04 6:56 a.m., Toke H=C3=B8iland-J=C3=B8rgensen wrote: Jonathan Morton write= s: On 4 Jun, 2020, at 1:21 am, Dave Collier-Brown wrote: We've good tools to measure network performance under stress, by the simple expedient of stressing it, but is there a good approach I could recommend to my company to monitor a bunch of reasonably modern links, without the measurement significantly affecting their state? I don't mind increasing bandwidth usage, but I'm downright grumpy about adding to the service time: I have a transaction that times out for gross slowness if it takes much more that an tenth of a second, and it involves a scatter-gather interaction with at least 10 customers in that time. I'm topically interested in bloat, but really we should understand "everything" about our links. If they can get the bloats like cattle, they can probably get the gout, like King Henry the Eighth (;-)) My platform is Centos 8, and I have lots of Smarter Colleagues to help. My first advice would be to browse pollere.net for tools - like pping (passive ping), which monitors the latency of flows in transit. That should give you some interesting information without adding any load at all. There is also connmon (https://github.com/pollere/connmon). Ah, good idea, totally forgot about Kathy's tools! :) I figure one could probably implement something like connmon in eBPF (as an XDP or TC hook program) and have it run as an always-on monitor with fairly low overhead. Dave, if you have development resources to throw at this, I'll be happy to help with pointers on how to get the eBPF bits working. I believe CentOS 8.2+ should have the needed kernel support... Of course, you could also just use the connmon utility as-is if you have CPU cycles to spare for the extra overhead (it looks like it's using libpcap to capture the packets and process them in userspace). -Toke -- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest dave.collier-brown@indexexchange.com | -- Mark Twain CONFIDENTIALITY NOTICE AND DISCLAIMER : This telecommunication, including a= ny and all attachments, contains confidential information intended only for= the person(s) to whom it is addressed. Any dissemination, distribution, co= pying or disclosure is strictly prohibited and is not a waiver of confident= iality. If you have received this telecommunication in error, please notify= the sender immediately by return electronic mail and delete the message fr= om your inbox and deleted items folders. This telecommunication does not co= nstitute an express or implied agreement to conduct transactions by electro= nic means, nor does it constitute a contract offer, a contract amendment or= an acceptance of a contract offer. Contract terms contained in this teleco= mmunication are subject to legal review and the completion of formal docume= ntation and are not binding until same is confirmed in writing and has been= signed by an authorized signatory. --------------40EE4B8372C7027D3233DC2D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Thanks, that sounds like a good long-term approach. I = know our CTO loves eBPF! As and introduction I'll start prothletising pping= (;-))

--dave

On 2020-06-04 6:56 a.m., Toke H=C3=B8iland-J= =C3=B8rgensen wrote:
Jonathan Morton <chromatix99@gmail.=
com> writes:


On 4 Jun, 2020, at 1:21 am, Dave Col=
lier-Brown <dave.collier-brown@indexexchange.com> wr=
ote:

We've good tools to measure network performance under stress, by the
simple expedient of stressing it, but is there a good approach I
could recommend to my company to monitor a bunch of reasonably modern
links, without the measurement significantly affecting their state?

I don't mind increasing bandwidth usage, but I'm downright grumpy
about adding to the service time: I have a transaction that times out
for gross slowness if it takes much more that an tenth of a second,
and it involves a scatter-gather interaction with at least 10
customers in that time.

I'm topically interested in bloat, but really we should understand
"everything" about our links. If they can get the bloats like cat=
tle,
they can probably get the gout, like King Henry the Eighth (;-))

My platform is Centos 8, and I have lots of Smarter Colleagues to
help.

My first advice would be to browse pollere.net for tools - like pping
(passive ping), which monitors the latency of flows in transit. That
should give you some interesting information without adding any load
at all. There is also connmon (https://github.com/pollere/connmon).

Ah, good idea, totally forgot about Kathy's tools! :)

I figure one could probably implement something like connmon in eBPF (as
an XDP or TC hook program) and have it run as an always-on monitor with
fairly low overhead. Dave, if you have development resources to throw at
this, I'll be happy to help with pointers on how to get the eBPF bits
working. I believe CentOS 8.2+ should have the needed kernel support...

Of course, you could also just use the connmon utility as-is if you have
CPU cycles to spare for the extra overhead (it looks like it's using
libpcap to capture the packets and process them in userspace).

-Toke

--=20
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-brown@indexexchange.com |              -- =
Mark Twain

CONFIDENTIALITY NOTICE AND D= ISCLAIMER : T= his telecommunication, including any and all attachments, contains confiden= tial information intended only for the person(s) to whom it is addressed. Any dissemination, distribution, copying or discl= osure is strictly prohibited and is not a waiver of confidentiality. If you= have received this telecommunication in error, please notify the sender im= mediately by return electronic mail and delete the message from your inbox and deleted items folders. This tel= ecommunication does not constitute an express or implied agreement to condu= ct transactions by electronic means, nor does it constitute a contract offe= r, a contract amendment or an acceptance of a contract offer. Contract terms contained in this telecommunication ar= e subject to legal review and the completion of formal documentation and ar= e not binding until same is confirmed in writing and has been signed by an = authorized signatory.

--------------40EE4B8372C7027D3233DC2D--