From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-DM3-obe.outbound.protection.outlook.com (mail-dm3nam03on0103.outbound.protection.outlook.com [104.47.41.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D7C293B29E for ; Sun, 19 Nov 2017 15:24:58 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cornellprod.onmicrosoft.com; s=selector1-cornell-edu; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=oUlcVbuCbfwX/yqCc/0Orh8RT3g6rKIoCE6X2HtsiYM=; b=AUMjhLTXiCNedr1z/lK4I1n1ottn1W69guThB7/qdy6JdcGsHZwKAEXW4jVJhNwyJDEQ8KC0wGVuofrgC4+zykxpQWrMZHWVC20BAwH8/P3TYW/SxDFqCCoEQ192cK2XbQEKtA+UM9nCXstujT/ZqmVfHdkunIHQqaWFxah1WB4= Received: from BN6PR04MB1187.namprd04.prod.outlook.com (10.173.199.12) by BN6PR04MB1188.namprd04.prod.outlook.com (10.173.199.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.239.5; Sun, 19 Nov 2017 20:24:57 +0000 Received: from BN6PR04MB1187.namprd04.prod.outlook.com ([10.173.199.12]) by BN6PR04MB1187.namprd04.prod.outlook.com ([10.173.199.12]) with mapi id 15.20.0239.009; Sun, 19 Nov 2017 20:24:57 +0000 From: Ken Birman To: 'Dave Taht' , Matthias Tafelmeier CC: Bob Briscoe , "bloat@lists.bufferbloat.net" Thread-Topic: [Bloat] DETNET Thread-Index: AQHTXKmP09aNAYbFf0S3Kj7WJlYhI6MV2m+DgARy6QCAAcMIWYAAHb4Q Message-ID: References: <4d54f24f-ce83-34a0-41f3-9f728420d548@gmx.net> <87shdr0vt6.fsf@nemesis.taht.net> <79f4d92c-74f4-8cd0-9d38-e51a668cb9b6@gmx.net> <796aa11e-9e35-cf34-e456-6ae98d1875d6@bobbriscoe.net> <87fu9f72za.fsf@nemesis.taht.net> <87a7zirue9.fsf@nemesis.taht.net> In-Reply-To: <87a7zirue9.fsf@nemesis.taht.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=kpb3@cornell.edu; x-originating-ip: [69.201.32.239] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BN6PR04MB1188; 6:R5VuUGYsyXRXEDqpJ4CMNvX8NZ3POdowTICVHgdHtIwIT69XkV5sC7UELXxdUYRwSn/dsYw8a2DXdevn1toIPZLhSWc0ZxXNMN/ztg//ZJVisuxW/jxpynYSO+9d/kIpuIQuDPJhfRULZ97OX0ouBkdOx/xUwnYGsOzPttSBQPW5Pkh2bm+NOVZJqwnU/AzI2Np4kIFll2p/LT5V8D7qezgBj6Cb/GL9547ngBwTGNR+Q8T5RurEtn3HBqcWXv0/GTU1PUkkFiiMtxudMcP0pZ1bt5rskpx/KTUtKHs+zi7zAT8JycoLefRWdERE5WgXyVu5+qiQ+CTcubLCswTNQ2iSGc+Uo8hYsQdbuADsqNc=; 5:u6ozHDFC4sDHaKJ3SfUwUMu81B7loUvB3yadk4cqaHcGEwCmonhbTDYB7j2n2leeN/G2E9PcXQY/4iXanEgzBwUgDw/SVhhTHVAQa6uHcSuQa8HtZveeuyfcehAuw8ekjUskrahBGbd0eNjdr8zPZGxY86WpDKCzrOJj4ZIm7Zw=; 24:Suymczy3AYg30GLoqYP0h8Isti+a/RLu4IZt3VEq7/VeHKJfKyrdWe5lFqwnU7r016Nrs2tMUtwN2TvzWDVD3e1BNz83d6h7g63qx55Bqec=; 7:wmwc5xHZ3kkqAtIZ0aczqEG32DjYyASDUOMThys7jFIn59FB104L3TSlYHjlJtLARZIrtmh3+xLL6/5g1vj74OJFXRhctlvpmBNG2lT5/oROKzYctSpfI4PEDBGcyPMVJ9SJwJt/n5y/6p6Dyrcq3LJbbU4guRwv92xridyg/StYyj/Wg9/a+z/RuGMZdrLhZjMulmTXuptlTY+wFUEuBJiNYWYa5VU9FaJexMMdnBIm++C7aIBSOnNeZQ1tRwwB x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: acf407a9-9843-4064-aa45-08d52f8b9a26 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4627115)(8989060)(201703031133081)(201702281549075)(8990040)(2017052603199); SRVR:BN6PR04MB1188; x-ms-traffictypediagnostic: BN6PR04MB1188: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(10201501046)(3231022)(6041248)(20161123555025)(20161123562025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123560025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN6PR04MB1188; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN6PR04MB1188; x-forefront-prvs: 0496DF6962 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(346002)(376002)(39860400002)(199003)(189002)(6246003)(2950100002)(97736004)(68736007)(75432002)(2906002)(7736002)(54906003)(9686003)(66066001)(110136005)(316002)(786003)(3280700002)(105586002)(106356001)(86362001)(93886005)(189998001)(88552002)(14454004)(229853002)(305945005)(33656002)(6506006)(102836003)(6116002)(3846002)(6436002)(25786009)(2900100001)(54356999)(5660300001)(50986999)(76176999)(45080400002)(3660700001)(8936002)(478600001)(99286004)(81156014)(81166006)(77096006)(101416001)(4326008)(53936002)(55016002)(8676002)(74316002)(7696004)(88722002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN6PR04MB1188; H:BN6PR04MB1187.namprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: cornell.edu does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: cornell.edu X-MS-Exchange-CrossTenant-Network-Message-Id: acf407a9-9843-4064-aa45-08d52f8b9a26 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Nov 2017 20:24:57.5196 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5d7e4366-1b9b-45cf-8e79-b14b27df46e1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR04MB1188 X-Mailman-Approved-At: Mon, 11 Dec 2017 11:18:07 -0500 Subject: Re: [Bloat] DETNET X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Date: Sun, 19 Nov 2017 20:24:59 -0000 X-Original-Date: Sun, 19 Nov 2017 20:24:57 +0000 X-List-Received-Date: Sun, 19 Nov 2017 20:24:59 -0000 The Microsoft SIGCOMM paper on deploying RDMA ran into issues with PPF pack= ets (used by the RDMA technology to prevent buffer overruns): on Ethernet (= RoCE) they employ an IP packet associated with Enterprise VLAN functionalit= y, and Microsoft disables that feature on their Azure routers. They work a= round this by using a different feature of the router: modern datacenter ro= uters support DiffServ, but few data centers use it. So they enabled Diffs= erv and split TCP/IP off as one traffic class, and treat RDMA as a second t= raffic class. They use DCQCN, which has its own RDMA flow control manageme= nt, but as a fallback allow the hardware to still generate PPF packets, usi= ng the DiffServ header field to encode the needed PPF "pause sending" infor= mation. But to do this they were forced to change some firmware; no idea i= f this idea of their's will become widely adopted. Anyhow, with this schem= e, the PPF packets protect against cases where DCQCN isn't fast enough to r= eact. This was all with 100Gbps RDMA. If I understood your prioritized queuing use case better I might be able to= point to something squarely on topic. If this thread is about a specific = scenario, maybe someone could point me to the OP where the scenario was fir= st described?