From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vn0-x232.google.com (mail-vn0-x232.google.com [IPv6:2607:f8b0:400c:c0f::232]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 5242321F286 for ; Tue, 14 Apr 2015 18:35:40 -0700 (PDT) Received: by vnbf1 with SMTP id f1so10411785vnb.5 for ; Tue, 14 Apr 2015 18:35:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:content-transfer-encoding; bh=yybemTsmO4yt/lnB8mr72tcdvz4BqLjI0FmHdQbl7co=; b=EmMi7VD2ilTI6fSCJmEu4MVjGrU1gb2GCKI+5kptVwpg+74ieeYOr6BelXNYCPF1YK YM98IY2jpWTNNFLyIq6iT9sCpR2531JIXqPlcWSCAAZXZR7yBSKrjUx/JgDGBC/iJO7F wwxMi6SLWvW41vUbffdCVjzjICPDxbNeBQgEtfIcdP1Ixgcc8Ki0/k9nJs6Mn6weHr1N M9Z8dq+58/6QJks1AS+EbwdJzu3b5P6/cXjTnbhe5jy/QH1gyIE7gi0K9gf3QU/FqGLk HQJg9DXt25/XWgWjFuvJwygBgpWKDYNRdCWLG2BkNj/Op721J+N1nF4XKDe1VtJBgAjN 6JcQ== X-Received: by 10.60.44.241 with SMTP id h17mr18763209oem.71.1429061739863; Tue, 14 Apr 2015 18:35:39 -0700 (PDT) MIME-Version: 1.0 Sender: white.phoenix@gmail.com Received: by 10.202.188.8 with HTTP; Tue, 14 Apr 2015 18:35:09 -0700 (PDT) In-Reply-To: References: <558D3A0C-75A0-4707-95DF-790F29F825AE@gmx.de> From: leetminiwheat Date: Tue, 14 Apr 2015 21:35:09 -0400 X-Google-Sender-Auth: ScDbfj2U0PMTFSFXo4HyW9T2oJM Message-ID: To: Sebastian Moeller Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: cerowrt-devel Subject: Re: [Cerowrt-devel] squash/ignore DSCP and mangle table questions X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Apr 2015 01:36:09 -0000 On Mon, Apr 13, 2015 at 5:39 PM, Sebastian Moeller wrote: > This looks reasonable. > > > > > ~ # ifconfig > > ge00 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5D > > collisions:0 txqueuelen:1000 > > > > gw00 Link encap:Ethernet HWaddr 22:4E:7F:91:9F:5C > > collisions:0 txqueuelen:1000 > > > > > > ifb4ge00 Link encap:Ethernet HWaddr C2:00:73:A7:1E:30 > > collisions:0 txqueuelen:32 > > > > > > ifb4gw00 Link encap:Ethernet HWaddr 9A:BA:17:27:79:A5 > > collisions:0 txqueuelen:32 > > > > lo Link encap:Local Loopback > > collisions:0 txqueuelen:0 > > > > se00 Link encap:Ethernet HWaddr 22:4E:7F:91:9F:5C > > collisions:0 txqueuelen:1000 > > > > sw00 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5C > > collisions:0 txqueuelen:1000 > > > > sw10 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5E > > collisions:0 txqueuelen:1000 > > I had a look at my cerowrt router and I also see txqueuelen at 10= 00, but IIRC that does not matter much anymore, since the wndr37/800 suppor= t BQL: > cat /sys/class/net/ge00/queues/tx-0/byte_queue_limits/limit_max > 3000 > So even with txqueuelen =3D 1000 the tx queue will only hold 3000 bytes. = For fib and friends it does not really matter as far as I can tell. Interesting, I had been reading about BQL but didn't fully understand it until now. This clears up a lot of confusion, thank you. I assume tweaking ring parameters from default RX:128 and TX:32 doesn't matter anymore thenr? > > and after rebooting and running my [ethtool/txqueuelen]commands (still = testing different values. should the IFBs txqueuelen be left alone? I left = at default): > > > > ~ # ifconfig > > ge00 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5D > > collisions:0 txqueuelen:8 > > > > gw00 Link encap:Ethernet HWaddr 22:4E:7F:91:9F:5C > > collisions:0 txqueuelen:4 > > > > > > ifb4ge00 Link encap:Ethernet HWaddr C2:00:73:A7:1E:30 > > collisions:0 txqueuelen:32 > > > > > > ifb4gw00 Link encap:Ethernet HWaddr 9A:BA:17:27:79:A5 > > collisions:0 txqueuelen:32 > > > > lo Link encap:Local Loopback > > collisions:0 txqueuelen:0 > > > > se00 Link encap:Ethernet HWaddr 22:4E:7F:91:9F:5C > > collisions:0 txqueuelen: 8 > > > > sw00 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5C > > collisions:0 txqueuelen: 4 > > > > sw10 Link encap:Ethernet HWaddr 20:4E:7F:91:9F:5E > > collisions:0 txqueuelen: 4 > > > > > > If you have time and netperf-wrapper it would be good to convince yoursel= f and us again, that txqueuelen really does not matter for BQL=E2=80=99d in= terfaces by running RRUL tests with and without your modifications=E2=80=A6= . Will do if I can get a friend to set up a netperf server in a node at his datacenter he works at, though I believe he's using Debian Jessy without any kind of QoS, but his datacenter routers might screw with the packets. Maybe I can hook a spare box up to the WAN port temporarily to test with. ###########################################################################= # Off topic a bit, if I might pick some of your brains - does anyone have experience with Intel gigE NICs and any advice on how they might be tweaked for lowest latency possible when connected to a debloated router? Most of the websites on this topic are inexperienced, outdated, or have no evidence to back their assumptions. Throughput is not a concern, as the machine in question is for gaming only, on Windows 8.1 (though in linux it has similarly named settings), connected via gigE to CeroWRT/WNDR3800 gateway which is directly connected via gigE to fiber ONT providing 32/25 (Cero's SQM capped to 28.8/22.5). ((bittorrent traffic still has latency issues though even with really low bandwidth caps, measured from a linux box, perhaps ISP related)) Intel Network Interface is forced into MSI mode (message signaled interrupt= s) My adapter settings settings below, those with an asterisk have been modifi= ed *Adaptive Inter-Frame Spacing: Disabled *Flow Control: Disabled (Options: Disabled, Rx & Tx Enabled, Rx Enabled, Tx Enabled) ## Default is Rx Enabled. I really don't see a benefit to pause frames when we have QoS algorithms. Receive Buffers: 256 (value: 80-2048, this is the equivalent of Ring parameters in Linux) Transmit Buffers: 512 (value: 80-2048, this is the equivalent of Ring parameters in Linux) ## Buffer thoughts: maybe a lower ratio such as 128 and 256 or 64 and 128? Documentation says: ## each descriptor comes with a 2048 byte buffer. ## Why more TX than RX? Our architecture favors the nondeterministic RX side for priority so there is more turnover of descriptors ## than the TX side. Plus the O/S can sometimes not return TX assets back to the driver in a timely fashion. IPv4 Checksum Offload: Rx & Tx Enabled Protocol ARP Offload: Enabled Protocol NS Offload: Enabled TCP Checksum Offload (IPv4): Rx & Tx Enabled (Options: Disabled, Rx Enabled, Tx Enabled) TCP Checksum Offload (IPv6): Rx & Tx Enabled (Options: Disabled, Rx Enabled, Tx Enabled) UDP Checksum Offload (IPv4): Rx & Tx Enabled (Options: Disabled, Rx Enabled, Tx Enabled) UDP Checksum Offload (IPv6): Rx & Tx Enabled (Options: Disabled, Rx Enabled, Tx Enabled) Large Send Offload V2 (IPv4): Enabled (this is called TSO, TCP Segmentation Offload, in linux) Large Send Offload V2 (IPv6): Enabled (this is called TSO, TCP Segmentation Offload, in linux) ## Offload thoughts: Should any offloads be disabled if the host machine is fast enough? do these cause delays? *Jumbo Packet: Disabled ## Probably don't want anything potentially latency sensitive being clumped up into a huge packet, especially when the destination is WAN which the router would have to segment anyways *Interrupt Moderation: Disabled (This is probably similar to large receive offload (LRO), description says it reduces interrupts.) *Interrupt Moderation Rate: Off (Options: Adaptive, Extreme, High, Low, Medium, Minimal, Off) ## Presumably Interrupt Moderation might cause latency by aggregating packets into a buffer? Receive Side Scaling: Enabled (Documentation says this makes processing scale to multiple CPU cores) Maximum Number of RSS Queues: 2 Queue (Options: 1 Queue, 2 Queue) ## Scaling up to 2 cpu cores? unsure if this would cause latency Gigabit Master Slave Mode: Auto Detect (Options: Auto Detect, Force Master Mode, Force Slave Mode) Enable PME: Disabled Energy Efficient Ethernet: Disabled System Idle Power Saver: Disabled