Re: [Cerowrt-devel] Correct syntax for cake commands and atm issues.

[Sebastian Moeller <moeller0@gmx.de>]

[-- Attachment #1: Type: text/plain, Size: 3300 bytes --]

Hi Fred,

On Jul 10, 2015, at 21:14 , Fred Stratton <fredstratton@imap.cc> wrote:

> 
> 
> On 10/07/15 19:46, Sebastian Moeller wrote:
>> Hi Fred,
>> 
>> your results seem to indicate that cake is not active at all, as the latency under load is abysmal (a quick check is to look at the median in relation to the min and the 90% number, in your examples all of these are terrible). Could you please post the result of the following commands on your router:
>> 1) cat /etc/config/sqm
> config queue 'eth1'
>    option qdisc 'fq_codel'
>    option script 'simple.qos'
>    option qdisc_advanced '0'
>    option linklayer ‘none'

	For an ADSL link I could swear link layer ‘none’ is not optimal ;)

>    option enabled '0'
>    option interface 'eth1'
>    option download '0'
>    option upload ‘0'

	Not that it matters as you do not use the sqm-scripts machinery.

> 
> 
>> 2) tc -d qdisc
> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev eth1 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 0: dev wlan1 root
> qdisc fq_codel 0: dev wlan1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 0: dev wlan0 root
> qdisc fq_codel 0: dev wlan0 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan0 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan0 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 0: dev wlan0 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc cake 8002: dev pppoe-wan root refcnt 2 bandwidth 850Kbit besteffort flows raw

	You should enable the atm link layer accounting on both ingress (ifb4pppoe-wan) and egress (pppoe-wan).

> qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
> qdisc cake 8001: dev ifb4pppoe-wan root refcnt 2 bandwidth 11500Kbit besteffort flows atm overhead 40
> 
>> 3) tc -d class show dev pppoe-wan
> class cake 8002:2fa parent 8002:
> 
> 
>> 4) tc -d class show dev ifb4pppoe-wqn
> 
> class cake 8001:6e parent 8001:
>> 5) /etc/init.d/sqm stop
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: run.sh stop
> SQM: /usr/lib/sqm/run.sh SQM for interface eth1 is not enabled, skipping over...
>> 6) /etc/init.d/sqm start
> /etc/init.d/sqm start
> SQM: Trying to start/stop SQM on all interfaces.
> SQM: /usr/lib/sqm/run.sh SQM for interface eth1 is not enabled, skipping over…

	Alas, not activated so this does not give any diagnostic output… Newer sqm-scripts should work with cake and luci-app-sqm also works well to set up cake. Please find the most recent files for sqm-scripts attached. The content of the sqm folder should replace the content of /usr/lib/sqm on your router.

[-- Attachment #2: sqm.zip --]
[-- Type: application/zip, Size: 21639 bytes --]

[-- Attachment #3: Type: text/plain, Size: 75 bytes --]

 The content of sqm.lua should replace /usr/lib/lua/luci/model/cbi/sqm.lua 

[-- Attachment #4: sqm.lua --]
[-- Type: application/octet-stream, Size: 9571 bytes --]

--[[
LuCI - Lua Configuration Interface

Copyright 2014 Steven Barth <steven@midlink.org>
Copyright 2014 Dave Taht <dave.taht@bufferbloat.net>

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

$Id$
]]--

local wa = require "luci.tools.webadmin"
local fs = require "nixio.fs"
local net = require "luci.model.network".init()
local sys = require "luci.sys"
--local ifaces = net:get_interfaces()
local ifaces = sys.net:devices()
local path = "/usr/lib/sqm"

m = Map("sqm", translate("Smart Queue Management"),
	translate("With <abbr title=\"Smart Queue Management\">SQM</abbr> you " ..
		"can enable traffic shaping, better mixing (Fair Queueing)," ..
		" active queue length management (AQM) " ..
		" and prioritisation on one " ..
		"network interface."))

s = m:section(TypedSection, "queue", translate("Queues"))
s:tab("tab_basic", translate("Basic Settings"))
s:tab("tab_qdisc", translate("Queue Discipline"))
s:tab("tab_linklayer", translate("Link Layer Adaptation"))
s.addremove = true -- set to true to allow adding SQM instances in the GUI
s.anonymous = true

-- BASIC
e = s:taboption("tab_basic", Flag, "enabled", translate("Enable this SQM instance."))
e.rmempty = false

-- sm: following jow's advise, be helpful to the user and enable
--     sqm's init script if even a single sm instance/interface
--     is enabled; this is unexpected in that the init script gets
--     enabled as soon as at least one sqm instance is enabled
--     and that state is saved, so it does not require "Save & Apply"
--     to effect the init scripts.
--     the implementation was inpired/lifted from 
--     https://github.com/openwrt/luci/blob/master/applications/luci-app-minidlna/luasrc/model/cbi/minidlna.lua
function e.write(self, section, value)
    if value == "1" then
	luci.sys.init.enable("sqm")
	m.message = translate("The SQM GUI has just enabled the sqm initscript on your behalf. Remember to disable the sqm initscript manually under System Startup menu in case this change was not wished for.")
--	luci.sys.call("/etc/init.d/sqm start >/dev/null")
--    else
--	luci.sys.call("/etc/init.d/sqm stop >/dev/null")
--	luci.sys.init.disable("sqm")
    end
    return Flag.write(self, section, value)
end
-- TODO: inform the user what we just did...

n = s:taboption("tab_basic", ListValue, "interface", translate("Interface name"))
-- sm lifted from luci-app-wol, the original implementation failed to show pppoe-ge00 type interface names
for _, iface in ipairs(ifaces) do
--     if iface:is_up() then
--	n:value(iface:name())
--     end
	if iface ~= "lo" then 
		n:value(iface) 
	end
end
n.rmempty = false


dl = s:taboption("tab_basic", Value, "download", translate("Download speed (kbit/s) (ingress) set to 0 to selectively disable ingress shaping:"))
dl.datatype = "and(uinteger,min(0))"
dl.rmempty = false

ul = s:taboption("tab_basic", Value, "upload", translate("Upload speed (kbit/s) (egress) set to 0 to selectively disable egress shaping:"))
ul.datatype = "and(uinteger,min(0))"
ul.rmempty = false

-- QDISC

c = s:taboption("tab_qdisc", ListValue, "qdisc", translate("Queueing discipline"))
c:value("fq_codel", "fq_codel ("..translate("default")..")")
c:value("efq_codel")
c:value("nfq_codel")
c:value("sfq")
c:value("codel")
c:value("ns2_codel")
c:value("pie")
c:value("sfq")
c:value("cake")
c.default = "fq_codel"
c.rmempty = false

local qos_desc = ""
sc = s:taboption("tab_qdisc", ListValue, "script", translate("Queue setup script"))
for file in fs.dir(path) do
  if string.find(file, ".qos$") then
    sc:value(file)
  end
  if string.find(file, ".qos.help$") then
    fh = io.open(path .. "/" .. file, "r")
    qos_desc = qos_desc .. "<p><b>" .. file:gsub(".help$", "") .. ":</b><br />" .. fh:read("*a") .. "</p>"
  end
end
sc.default = "simple.qos"
sc.rmempty = false
sc.description = qos_desc

ad = s:taboption("tab_qdisc", Flag, "qdisc_advanced", translate("Show and Use Advanced Configuration. Advanced options will only be used as long as this box is checked."))
ad.default = false
ad.rmempty = true

squash_dscp  = s:taboption("tab_qdisc", ListValue, "squash_dscp", translate("Squash DSCP on inbound packets (ingress):"))
squash_dscp:value("1", "SQUASH")
squash_dscp:value("0", "DO NOT SQUASH")
squash_dscp.default = "1"
squash_dscp.rmempty = true
squash_dscp:depends("qdisc_advanced", "1")

squash_ingress = s:taboption("tab_qdisc", ListValue, "squash_ingress", translate("Ignore DSCP on ingress:"))
squash_ingress:value("1", "Ignore")
squash_ingress:value("0", "Allow")
squash_ingress.default = "1"
squash_ingress.rmempty = true
squash_ingress:depends("qdisc_advanced", "1")

iecn = s:taboption("tab_qdisc", ListValue, "ingress_ecn", translate("Explicit congestion notification (ECN) status on inbound packets (ingress):"))
iecn:value("ECN", "ECN ("..translate("default")..")")
iecn:value("NOECN")
iecn.default = "ECN"
iecn.rmempty = true
iecn:depends("qdisc_advanced", "1")

eecn = s:taboption("tab_qdisc", ListValue, "egress_ecn", translate("Explicit congestion notification (ECN) status on outbound packets (egress)."))
eecn:value("NOECN", "NOECN ("..translate("default")..")")
eecn:value("ECN")
eecn.default = "NOECN"
eecn.rmempty = true
eecn:depends("qdisc_advanced", "1")

ad2 = s:taboption("tab_qdisc", Flag, "qdisc_really_really_advanced", translate("Show and Use Dangerous Configuration. Dangerous options will only be used as long as this box is checked."))
ad2.default = false
ad2.rmempty = true
ad2:depends("qdisc_advanced", "1")

ilim = s:taboption("tab_qdisc", Value, "ilimit", translate("Hard limit on ingress queues; leave empty for default."))
-- ilim.default = 1000
ilim.isnumber = true
ilim.datatype = "and(uinteger,min(0))"
ilim.rmempty = true
ilim:depends("qdisc_really_really_advanced", "1")

elim = s:taboption("tab_qdisc", Value, "elimit", translate("Hard limit on egress queues; leave empty for default."))
-- elim.default = 1000
elim.datatype = "and(uinteger,min(0))"
elim.rmempty = true
elim:depends("qdisc_really_really_advanced", "1")


itarg = s:taboption("tab_qdisc", Value, "itarget", translate("Latency target for ingress, e.g 5ms [units: s, ms, or  us]; leave empty for automatic selection, put in the word default for the qdisc's default."))
itarg.datatype = "string"
itarg.rmempty = true
itarg:depends("qdisc_really_really_advanced", "1")

etarg = s:taboption("tab_qdisc", Value, "etarget", translate("Latency target for egress, e.g. 5ms [units: s, ms, or  us]; leave empty for automatic selection, put in the word default for the qdisc's default."))
etarg.datatype = "string"
etarg.rmempty = true
etarg:depends("qdisc_really_really_advanced", "1")



iqdisc_opts = s:taboption("tab_qdisc", Value, "iqdisc_opts", translate("Advanced option string to pass to the ingress queueing disciplines; no error checking, use very carefully."))
iqdisc_opts.rmempty = true
iqdisc_opts:depends("qdisc_really_really_advanced", "1")

eqdisc_opts = s:taboption("tab_qdisc", Value, "eqdisc_opts", translate("Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully."))
eqdisc_opts.rmempty = true
eqdisc_opts:depends("qdisc_really_really_advanced", "1")

-- LINKLAYER
ll = s:taboption("tab_linklayer", ListValue, "linklayer", translate("Which link layer to account for:"))
ll:value("none", "none ("..translate("default")..")")
ll:value("ethernet", "Ethernet with overhead: select for e.g. VDSL2.")
ll:value("atm", "ATM: select for e.g. ADSL1, ADSL2, ADSL2+.")
-- ll:value("adsl")	-- reduce the options
ll.default = "none"

po = s:taboption("tab_linklayer", Value, "overhead", translate("Per Packet Overhead (byte):"))
po.datatype = "and(integer,min(-1500))"
po.default = 0
po.isnumber = true
po.rmempty = true
po:depends("linklayer", "ethernet")
-- po:depends("linklayer", "adsl")
po:depends("linklayer", "atm")


adll = s:taboption("tab_linklayer", Flag, "linklayer_advanced", translate("Show Advanced Linklayer Options, (only needed if MTU > 1500). Advanced options will only be used as long as this box is checked."))
adll.rmempty = true
adll:depends("linklayer", "ethernet")
-- adll:depends("linklayer", "adsl")
adll:depends("linklayer", "atm")

smtu = s:taboption("tab_linklayer", Value, "tcMTU", translate("Maximal Size for size and rate calculations, tcMTU (byte); needs to be >= interface MTU + overhead:"))
smtu.datatype = "and(uinteger,min(0))"
smtu.default = 2047
smtu.isnumber = true
smtu.rmempty = true
smtu:depends("linklayer_advanced", "1")

stsize = s:taboption("tab_linklayer", Value, "tcTSIZE", translate("Number of entries in size/rate tables, TSIZE; for ATM choose TSIZE = (tcMTU + 1) / 16:"))
stsize.datatype = "and(uinteger,min(0))"
stsize.default = 128
stsize.isnumber = true
stsize.rmempty = true
stsize:depends("linklayer_advanced", "1")

smpu = s:taboption("tab_linklayer", Value, "tcMPU", translate("Minimal packet size, MPU (byte); needs to be > 0 for ethernet size tables:"))
smpu.datatype = "and(uinteger,min(0))"
smpu.default = 0
smpu.isnumber = true
smpu.rmempty = true
smpu:depends("linklayer_advanced", "1")

lla = s:taboption("tab_linklayer", ListValue, "linklayer_adaptation_mechanism", translate("Which linklayer adaptation mechanism to use; for testing only"))
lla:value("cake")
lla:value("htb_private")
lla:value("tc_stab", "tc_stab ("..translate("default")..")")
lla.default = "tc_stab"
lla.rmempty = true
lla:depends("linklayer_advanced", "1")

-- PRORITIES?

return m

[-- Attachment #5: Type: text/plain, Size: 5696 bytes --]

. These should allow you to set up cake from inside the sqm gui (but it is only lightly tested).


> 
>> 
>> hopefully these give some insight what might have happened.
>> 
>> And finally I would love to learn the output of:
>> sh betterspeedtest.sh -4 -H netperf-eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4 ; sh netperfrunner.sh -4 -H netperf-eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4
> 
> betterspeedtest.sh not installed

	Too bad, this would be nice as it measures downlink and uplink sequentially instead of simultaneously so it can help figure out if only one direction is improperly shaped. Could I convince you to install Rich’s betterspeedtest.sh script as well, it should be in the same repository as netperfrunner.sh?

> 
> sh betterspeedtest.sh -4 -H netperf-eu.bufferbloat.ne
> t -t 150 -p netperf-eu.bufferbloat.net -n 4 ; sh netperfrunner.sh -4 -H netperf-
> eu.bufferbloat.net -t 150 -p netperf-eu.bufferbloat.net -n 4
> sh: can't open 'betterspeedtest.sh'
> 2015-07-10 20:10:55 Testing netperf-eu.bufferbloat.net (ipv4) with 4 streams down and up while pinging netperf-eu.bufferbloat.net. Takes about 150 seconds.
> Download:  6.8 Mbps
>   Upload:  0.59 Mbps
>  Latency: (in msec, 152 pings, 0.00% packet loss)
>      Min: 73.911
>    10pct: 232.211
>   Median: 308.556
>      Avg: 305.686
>    90pct: 376.183
>      Max: 412.553

	This just shows that latency still is bounded badly...

> 
>> 
>> 
>> Many Thanks & Best Regards
>> 	Sebastian
>> 
>> On Jul 10, 2015, at 20:25 , Fred Stratton <fredstratton@imap.cc> wrote:
>> 
>>> By your command
>>> Rebooted to rerun qdisc script, rather than changing qdiscs from the command-line, so suboptimal process as end-point changed.
>>> 
>>> script configuring qdiscs and overhead 40 on
>>> 
>>> sh netperfrunner.sh -H netperf-eu.bufferbloat.net -p 2.96.48.1
>>> 2015-07-10 18:22:08 Testing netperf-eu.bufferbloat.net (ipv4) with 4 streams down and up while pinging 2.96.48.1. Takes about 60 seconds.
>>> Download:  6.73 Mbps
>>>   Upload:  0.58 Mbps
>>>  Latency: (in msec, 62 pings, 0.00% packet loss)
>>>      Min: 24.094
>>>    10pct: 172.654
>>>   Median: 260.563
>>>      Avg: 253.580
>>>    90pct: 330.003
>>>      Max: 411.145
>>> 
>>> script configuring qdiscs on flows raw
>>> 
>>> sh netperfrunner.sh -H netperf-eu.bufferbloat.net -p
>>> 78.145.32.1
>>> 2015-07-10 18:49:21 Testing netperf-eu.bufferbloat.net (ipv4) with 4 streams down and up while pinging 78.145.32.1. Takes about 60 seconds.
>>> Download:  6.75 Mbps
>>>   Upload:  0.59 Mbps
>>>  Latency: (in msec, 59 pings, 0.00% packet loss)
>>>      Min: 23.605
>>>    10pct: 169.789
>>>   Median: 282.155
>>>      Avg: 267.099
>>>    90pct: 333.283
>>>      Max: 376.509
>>> 
>>> script configuring qdiscs and overhead 36 on
>>> 
>>> sh netperfrunner.sh -H netperf-eu.bufferbloat.net -p
>>> 80.44.96.1
>>> 2015-07-10 19:20:18 Testing netperf-eu.bufferbloat.net (ipv4) with 4 streams down and up while pinging 80.44.96.1. Takes about 60 seconds.
>>> Download:  6.56 Mbps
>>>   Upload:  0.59 Mbps
>>>  Latency: (in msec, 62 pings, 0.00% packet loss)
>>>      Min: 22.975
>>>    10pct: 195.473
>>>   Median: 281.756
>>>      Avg: 271.609
>>>    90pct: 342.130
>>>      Max: 398.573
>>> 
>>> 
>>> On 10/07/15 16:19, Alan Jenkins wrote:
>>>> I'm glad to hear there's a working version (even if it's not in the current build :).
>>>> 
>>>> Do you have measurable improvements with overhead configured (v.s. unconfigured)?
>>>> 
>>>> I've used netperfrunner from CeroWrtScripts, e.g.
>>>> 
>>>> sh netperfrunner.sh -H netperf-eu.bufferbloat.net -p $ISP_ROUTER
>>>> 
>>>> I believe accounting for overhead helps on this two-way test, because a) it saturates the uplink b) about half that bandwidth is tiny ack packets (depending on bandwidth asymmetry).  And small packets have proportionally high overhead.
>>>> 
>>>> (But it seems to only make a small difference for me, which always surprises Seb).
>>>> 
>>>> Alan
>>>> 
>>>> On 10/07/15 15:52, Fred Stratton wrote:
>>>>> You are absolutely correct.
>>>>> 
>>>>> I tried both a numeric overhead value, and alternatively 'pppoe-vcmux'
>>>>> and 'ether-fcs' in the build I crafted based on r46006, which is lupin
>>>>> undeclared version 2. Everything works as stated.
>>>>> 
>>>>> On lupin undeclared version 4, the current release based on r46117, the
>>>>> values were not recognised.
>>>>> 
>>>>> Thank you.
>>>>> 
>>>>> I had cake running on a Lantiq ADSL gateway running the same r46006
>>>>> build. Unfortunately this was bricked by attempts to get homenet
>>>>> working, so I have nothing to report about gateway usage at present.
>>>>> 
>>>>> 
>>>>> 
>>>>> On 10/07/15 13:57, Jonathan Morton wrote:
>>>>>> You're already using correct syntax - I've written it to be quite
>>>>>> lenient and use sensible defaults for missing information. There are
>>>>>> several sets of keywords and parameters which are mutually orthogonal,
>>>>>> and don't depend on each other, so "besteffort" has nothing to do with
>>>>>> "overhead" or "atm".
>>>>>> 
>>>>>> What's probably happening is that you're using a slightly old version
>>>>>> of the cake kernel module which lacks the overhead parameter entirely,
>>>>>> but a more up to date tc which does support it. We've seen this
>>>>>> combination crop up ourselves recently.
>>>>>> 
>>>>>> - Jonathan Morton
>>>>>> 
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>