[Cerowrt-devel] squash/ignore DSCP and mangle table questions

Tue Apr 14 21:35:09 EDT 2015

On Mon, Apr 13, 2015 at 5:39 PM, Sebastian Moeller <moeller0 at gmx.de> wrote:
>         This looks reasonable.
>
> >
> >  ~ # ifconfig
> > ge00        Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5D
> >                collisions:0 txqueuelen:1000
> >
> > gw00       Link encap:Ethernet  HWaddr 22:4E:7F:91:9F:5C
> >                collisions:0 txqueuelen:1000
> >
> >
> > ifb4ge00   Link encap:Ethernet  HWaddr C2:00:73:A7:1E:30
> >                collisions:0 txqueuelen:32
> >
> >
> > ifb4gw00  Link encap:Ethernet  HWaddr 9A:BA:17:27:79:A5
> >                collisions:0 txqueuelen:32
> >
> > lo            Link encap:Local Loopback
> >               collisions:0 txqueuelen:0
> >
> > se00       Link encap:Ethernet  HWaddr 22:4E:7F:91:9F:5C
> >               collisions:0 txqueuelen:1000
> >
> > sw00      Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5C
> >              collisions:0 txqueuelen:1000
> >
> > sw10      Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5E
> >              collisions:0 txqueuelen:1000
>
>         I had a look at my cerowrt router and I also see txqueuelen at 1000, but IIRC that does not matter much anymore, since the wndr37/800 support BQL:
> cat /sys/class/net/ge00/queues/tx-0/byte_queue_limits/limit_max
> 3000
> So even with txqueuelen = 1000 the tx queue will only hold 3000 bytes. For fib and friends it does not really matter as far as I can tell.

Interesting, I had been reading about BQL but didn't fully understand
it until now. This clears up a lot of confusion, thank you.

I assume tweaking ring parameters from default RX:128 and TX:32
doesn't matter anymore thenr?

> > and after rebooting and running my [ethtool/txqueuelen]commands (still testing different values. should the IFBs txqueuelen be left alone? I left at default):
> >
> >  ~ # ifconfig
> > ge00        Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5D
> >                collisions:0 txqueuelen:8
> >
> > gw00       Link encap:Ethernet  HWaddr 22:4E:7F:91:9F:5C
> >                collisions:0 txqueuelen:4
> >
> >
> > ifb4ge00   Link encap:Ethernet  HWaddr C2:00:73:A7:1E:30
> >                collisions:0 txqueuelen:32
> >
> >
> > ifb4gw00  Link encap:Ethernet  HWaddr 9A:BA:17:27:79:A5
> >                collisions:0 txqueuelen:32
> >
> > lo            Link encap:Local Loopback
> >               collisions:0 txqueuelen:0
> >
> > se00       Link encap:Ethernet  HWaddr 22:4E:7F:91:9F:5C
> >               collisions:0 txqueuelen: 8
> >
> > sw00      Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5C
> >              collisions:0 txqueuelen: 4
> >
> > sw10      Link encap:Ethernet  HWaddr 20:4E:7F:91:9F:5E
> >              collisions:0 txqueuelen: 4
> >
> >
>
> If you have time and netperf-wrapper it would be good to convince yourself and us again, that txqueuelen really does not matter for BQL’d interfaces by running RRUL tests with and without your modifications….

Will do if I can get a friend to set up a netperf server in a node at
his datacenter he works at, though I believe he's using Debian Jessy
without any kind of QoS, but his datacenter routers might screw with
the packets. Maybe I can hook a spare box up to the WAN port
temporarily to test with.

############################################################################

Off topic a bit, if I might pick some of your brains - does anyone
have experience with Intel gigE NICs and any advice on how they might
be tweaked for lowest latency possible when connected to a debloated
router? Most of the websites on this topic are inexperienced,
outdated, or have no evidence to back their assumptions.

Throughput is not a concern, as the machine in question is for gaming
only, on Windows 8.1 (though in linux it has similarly named
settings), connected via gigE to CeroWRT/WNDR3800 gateway which is
directly connected via gigE to fiber ONT providing 32/25 (Cero's SQM
capped to 28.8/22.5). ((bittorrent traffic still has latency issues
though even with really low bandwidth caps, measured from a linux box,
perhaps ISP related))

Intel Network Interface is forced into MSI mode (message signaled interrupts)

My adapter settings settings below, those with an asterisk have been modified

*Adaptive Inter-Frame Spacing: Disabled
*Flow Control: Disabled (Options: Disabled, Rx & Tx Enabled, Rx
Enabled, Tx Enabled)
## Default is Rx Enabled. I really don't see a benefit to pause frames
when we have QoS algorithms.
Receive Buffers: 256 (value: 80-2048, this is the equivalent of Ring
parameters in Linux)
Transmit Buffers: 512 (value: 80-2048, this is the equivalent of Ring
parameters in Linux)
## Buffer thoughts: maybe a lower ratio such as 128 and 256 or 64 and
128? Documentation says:
##   each descriptor comes with a 2048 byte buffer.
##   Why more TX than RX?  Our architecture favors the
nondeterministic RX side for priority so there is more turnover of
descriptors
##   than the TX side.  Plus the O/S can sometimes not return TX
assets back to the driver in a timely fashion.
IPv4 Checksum Offload: Rx & Tx Enabled
Protocol ARP Offload: Enabled
Protocol NS Offload: Enabled
TCP Checksum Offload (IPv4): Rx & Tx Enabled (Options: Disabled, Rx
Enabled, Tx Enabled)
TCP Checksum Offload (IPv6): Rx & Tx Enabled (Options: Disabled, Rx
Enabled, Tx Enabled)
UDP Checksum Offload (IPv4): Rx & Tx Enabled (Options: Disabled, Rx
Enabled, Tx Enabled)
UDP Checksum Offload (IPv6): Rx & Tx Enabled (Options: Disabled, Rx
Enabled, Tx Enabled)
Large Send Offload V2 (IPv4): Enabled (this is called TSO, TCP
Segmentation Offload, in linux)
Large Send Offload V2 (IPv6): Enabled (this is called TSO, TCP
Segmentation Offload, in linux)
## Offload thoughts: Should any offloads be disabled if the host
machine is fast enough? do these cause delays?
*Jumbo Packet: Disabled
## Probably don't want anything potentially latency sensitive being
clumped up into a huge packet,
     especially when the destination is WAN which the router would
have to segment anyways
*Interrupt Moderation: Disabled (This is probably similar to large
receive offload (LRO), description says it reduces interrupts.)
*Interrupt Moderation Rate: Off (Options: Adaptive, Extreme, High,
Low, Medium, Minimal, Off)
## Presumably Interrupt Moderation might cause latency by aggregating
packets into a buffer?
Receive Side Scaling: Enabled (Documentation says this makes
processing scale to multiple CPU cores)
Maximum Number of RSS Queues: 2 Queue (Options: 1 Queue, 2 Queue)
## Scaling up to 2 cpu cores? unsure if this would cause latency
Gigabit Master Slave Mode: Auto Detect (Options: Auto Detect, Force
Master Mode, Force Slave Mode)
Enable PME: Disabled
Energy Efficient Ethernet: Disabled
System Idle Power Saver: Disabled