From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DC26E3B29D for ; Wed, 23 Sep 2020 21:07:31 -0400 (EDT) Received: by mail-oi1-x232.google.com with SMTP id 26so1918035ois.5 for ; Wed, 23 Sep 2020 18:07:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qVuZnwaZRc34GVVmQNIDgizKArwr/nSmKl267sKnsyU=; b=IoMxCJ3mz4K1AezbzMMEKWtnAPk6ODbgoIwHKZjoz88a1kjbNVL5ENwpsdRCvk+z5Y VW1TBx0rYsZp/U4uXjOw/x5jdB2CdNcCZkQPtyO2aBU27Tzgo0U1JJy0ThLCwP0YLP7W SFQmZkUQ1AxqLNUgdWNLrwlEo+RhjVxjvQ9z0QnFoOnWYfd0dkDXc2AGw2MJ0YCBTHxM BtxsDT6+epfEcfRLu6tzKdiV9YmGzJ7yhGZ9SZN8tQ63e3ebBQQgdYLz5rl/fc94kbFo MLOBM4n42G27fYQvq/hZSY0IiFBMI4nv51EXXVevy0Y6qeGTeDOn5xn13gzCPmopioje POlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qVuZnwaZRc34GVVmQNIDgizKArwr/nSmKl267sKnsyU=; b=NuzIEGmwWLYNS0aQ+hvdKS9mXvQK0xXSYSMdztWl7MI3rK6SJUVuhedTRLd+Ep42x0 5hNFQ4WaZgyhsz9+WPswa4xSGrMjVxr5g9zrcb+kp1GGhGw4W59TP7pYQBHlxkBwgEsV zAgZSNX6wC/qOEoM6cFDMyEA9+OfL5uztuMSAkCBY0/JxxWoOGICX+3RuRfyyjXYcUzj wbmsxhyqyj6fPRCe7johEgwY/Ay3u9mVoX6z1nJV8S79eORqHy5CmfM7sCnpXgGs9O9r Yzua0jPzYMCnL/jEOBstrp9yxMBhQxCFg2qwoa1IYGa2803G1pj9sKn96wK7UZM8litW XxpA== X-Gm-Message-State: AOAM530KBa3yI7f2UzImVMiFrAgPe/AfnLBIViHkPNy8++U6PpNNuV0e BessVyOsHdxgcxoDaQuhlnBsNu8svZmzF8NtPus= X-Google-Smtp-Source: ABdhPJzDEqYG20gnLlxJTWltWTYi4f3N6OT44/F8LaXAXl0hdY3niyK5IsxwDokbWcB8fhUZbBT/WtwTaxYD1i7Qs00= X-Received: by 2002:aca:60d5:: with SMTP id u204mr1177351oib.8.1600909650857; Wed, 23 Sep 2020 18:07:30 -0700 (PDT) MIME-Version: 1.0 References: <32080.1597787724@localhost> <3A782CD0-01F1-40FA-9DFD-B969BD11A566@gmail.com> In-Reply-To: <3A782CD0-01F1-40FA-9DFD-B969BD11A566@gmail.com> From: Daniel Sterling Date: Wed, 23 Sep 2020 21:07:20 -0400 Message-ID: To: Jonathan Morton Cc: Michael Richardson , bloat Content-Type: text/plain; charset="UTF-8" Subject: Re: [Bloat] cake + ipv6 X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Sep 2020 01:07:32 -0000 On Wed, Sep 23, 2020 at 2:13 PM Jonathan Morton wrote: > It fits my mental model, yes, though obviously the ideal would be to recognise that the xbox is a singular machine. Are you seeing a larger disparity than that? If so, is it even larger than four connections would justify without host-fairness? Thanks Jonathan, Unfortunately I fear I've been running an improperly configured cake all this time-- leave it to me to take a piece of cake and mess it up :) I should say, even with it being possibly misconfigured, it has still worked amazingly well for me! I have no end of gratitude for y'alls work -- So taking your advice, with my cake properly configured, the answer is no -- it looks like host-fairness is working correctly -- with a notable exception -- read on for details... I was using these probably wrong settings: WAN: besteffort bandwidth $UPBANDWIDTH internet nat egress ethernet LAN: besteffort bandwidth $BANDWIDTH internet nat ingress ethernet So, I was using triple-isolate instead of the correct src/dest isolation options per NIC, and I was using "nat" on the LAN side, which may have been further breaking host isolation as per the openwrt wiki: "don't add the nat option on LAN interfaces (this option should only be used in the WAN interface) or Per-Host Isolation stops working" - https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details I changed my configuration to: $WAN handle 1: root cake besteffort bandwidth $UPBANDWIDTH internet nat egress dual-srchost ethernet $LAN handle 1: root cake besteffort bandwidth $BANDWIDTH internet ingress dual-dsthost ethernet So: root@OpenWrt:~# tc -s qdisc | grep cake qdisc cake 1: dev eth1 root refcnt 2 bandwidth 20Mbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 38 mpu 84 qdisc cake 1: dev eth0 root refcnt 2 bandwidth 40Mbit besteffort dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 38 mpu 84 WAN: * nat option enabled * egress option (default) enabled * dual-srchost LAN: * nat option disabled (default) * ingress option enabled * dual-dsthost This worked well, but I ran into a bizarre issue: My xbox FPS gaming UDP stream would start at a low latency (even while other systems were doing bulk downloads), but after some time, the latency went from a steady 30ms (+/- <10ms for jitter) to over 100ms ! 100ms is of course notably the default cake rtt parameter. Resetting cake (tc del and then tc add again) fixed this; the same stream's latency went back down to about 30ms. This gaming stream uses well under 512kbps down and less than half that up, normally. This is worrisome! My use-case may be an edge case, as I have AT&T gigabit fiber but I limit it to 40mbit with cake, since empirically that's the highest throughput I've found that I can use while also maintaining the low jitter / latency over wifi. So certainly the WAN NIC is getting a lot of traffic that cake on the LAN NIC is throwing away. That is, there is basically always a large number of bytes in LAN cake's backlog. For now I've solved this by using the "prio" tc qdisc such that xbox traffic bypasses cake, and all other traffic goes thru cake. Is this a known issue at all? I was quite surprised to see this behaviour. Let me know if I can help debug this. It likely only happens when multiple other hosts are doing bulk downloads / video streaming. Thanks, Dan