From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id A87603B29D for ; Fri, 20 Nov 2020 18:25:33 -0500 (EST) Received: by mail-ot1-x342.google.com with SMTP id n11so10352148ota.2 for ; Fri, 20 Nov 2020 15:25:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XEbi2TOf7kQXFkAPTyRGTBIzjIgbB2lYMcgSU8aARSk=; b=ECXJIvK/ojnzquCJu+48wrwXoyNw/u8H1mPCotnwIpghngi3OhgFLid3N22RD5ECEC B090VLhQxh2CX3UPg5FRDfhkgJvfs4qIRoiZIjydgYMeplY9P9QBTEvs8CsPBeWjRL9r EbBW0A3vz+2gFz3DIQKUWKm/90TQjx0MTvxKk+TwwF5EcELj6s5vXwpDtMP61SW41Fe/ m3VIliAooSanrQ2PHu+hayylFx9JCtbeAbZsbIE1E8OjdMG6YtfH7rkbmBC9GROihTNm 5b+eH93vtkVK3GjtgLDk5/UIbdIWeyaVgY0B5mGNyJ/Ez6AxB+GOOwvUOKyTR/H+xsVW awPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XEbi2TOf7kQXFkAPTyRGTBIzjIgbB2lYMcgSU8aARSk=; b=jkUGNnAlm6OMb230OiaqFSkG0uFakF7H2RMtcphcndm2kaigu5fR+Ns3F06FdrdJUy 1B/e/xtD0tIg/j6ptMfIsCOcJVMUFXskdS0Cg0klIHnBJYGqe80wdS2hE5M4FsviVxT7 DfZwwhjEOGM+da4ZLuhqqlCbLyQmdqHcTkN+o97Az/wLeStqZzTXhCCWwrf35okSS1TL f/0ANaZivn3zaIFAK4tUZt+ws33DVbHYp7SkwNUw5C8KqPHrqo5eOGcL09ky+0n+P/NU 5IccRpEEgm4QJFzifBPvYEwatPmatCeWApEc7M8psobvOtUxT+d06m8R2lsMBi9858Qs VDnw== X-Gm-Message-State: AOAM533E6zsEbzi2o3I9T3Ff0XjGrP3rYEPOyvLexuaPOGYzC4h6Pepw FZLEcKVbVup6dTHj5NWQqCfz0WaRPAx/Nh23Pro= X-Google-Smtp-Source: ABdhPJy+Yg5JBSMSYoDNGypdRwL9Pt1MR7CVxxpBvF17nrdc0iMm6yr7b/5wx+PmMyXoDCvc9rNLxoXhgpjjvrwhWnM= X-Received: by 2002:a9d:4c18:: with SMTP id l24mr14937554otf.291.1605914732749; Fri, 20 Nov 2020 15:25:32 -0800 (PST) MIME-Version: 1.0 References: <32080.1597787724@localhost> <3A782CD0-01F1-40FA-9DFD-B969BD11A566@gmail.com> <87blhq3pdb.fsf@toke.dk> <875z7ujgxe.fsf@toke.dk> In-Reply-To: From: Daniel Sterling Date: Fri, 20 Nov 2020 18:25:21 -0500 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Jonathan Morton , bloat , Michael Richardson Content-Type: text/plain; charset="UTF-8" Subject: [Bloat] finally got the proper cake experience (was: Re: openwrt e1000e (was: Re: cake + ipv6)) X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 23:25:33 -0000 On Tue, Nov 17, 2020 at 7:00 PM Daniel Sterling wrote: > Wanted to give an update on this. All my issues (odd latency, slow > throughput, etc) went away when I switched from the in-kernel e1000e > driver to Intel's NAPI driver. OK y'all, after switching to this driver, I configured my openwrt snapshot setup as follows: ** e1000e loaded with InterruptThrottleRate=0: rmmod r8169 ; modprobe r8169 rmmod e1000e ; insmod e1000e InterruptThrottleRate=0 ** All ethtool -K offloading turned off: rx tx sg tso gso gro all off ** flow control / pause frames turned off: autoneg off rx off tx off ** ring buffers set to smallest setting: rx 64 tx 64 ** byte queue limits set to 3000 echo 3000 > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/net/eth1/queues/tx-0/byte_queue_limits/limit_max echo 3000 > /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/net/eth0/queues/tx-0/byte_queue_limits/limit_max cake configured with besteffort and no bandwidth limit: /usr/sbin/tc qdisc add dev $WAN handle 1: root cake besteffort internet nat egress ack-filter dual-srchost ethernet /usr/sbin/tc qdisc add dev $LAN handle 1: root cake besteffort internet ingress dual-dsthost ethernet AND ALL MY LATENCY IS GONE. I've finally experienced the Zen of low latency and high throughput of the Promised Cake. This is amazing. Previously, I was able to get low latency by setting the cake "bandwidth" setting to about 40-50mbps out of the LAN interface. Higher than that and I'd see latency. But now, I can run as fast as the wifi / wired connections allow with no latency anywhere on the network. This is truly wild. Why did no one tell me I didn't need that bandwidth setting?? I finally pushed and pushed and messed with things until I got it working this way. If this is the intended final configuration, it needs to be proselytized! Truly wire-line speed and no latency. This is just, there are no words. The dream is real Enormous thank you to everyone who worked on cake! Thanks, Dan