[Cerowrt-devel] routers you can throw off the back of a truck

Valent Turkovic valent at otvorenamreza.org
Sun Jan 17 19:45:11 EST 2016

Hi everybody,

our mission is to provide internet in humanitarian and crisis
situations to as many people as possible, and to make bandwidth fairly
distributed between users and to have low bufferbloat.

Issue with working out in crisis stations is that you don't know what
kind of uplink you will get, and bandwidth you get is much often
variable than constant. I have tested 3G/4G uplinks and download can
be in and range from 100Mbps to 5 Mbps, especially with mobile nodes
(we sometimes put router, battery and 3G/4G modem in backpack and go
where it is needed).

Is there anyway to configure any queuing technique in Linux kernel
that so it distributes bandwidth equally between users and to keep lag
(bufferbloat) as low as possible, but without needing to define
absolute values for download and upload. For bandwidth distribution I
know that Mikrotik has really awesome technique called PCQ [1] and
AFAIK there is no technique as PCQ in Linux kernel, the closes I have
found is SFQ, if you know any other similar queueing techniques in
Linux kernel please share.

I have been testing different queue setup scripts for fq_codel in
OpenWrt, to see how they work with properly setup bandwidth limits and
also with completely wrong bandwidth limits. Test have been done with
3G/4G USB modems and on 100/10 Mbps fibre optics connection with
TP-LINK WDR3600 router running latest Chaos Calmer OpenWrt.

Also I have also seen that pfSense mentions that HFSC [2] that is
"very effective for VoIP on links that degrade quickly, such as 3G/4G,
but it can be complex to configure and tweak for proper operation." so
I tested HFSC queuing technique with fq_codel with  and results show
that it also doesn't work if bandwith is not configured correctly

So far all fq_codel + queue setup scripts on OpenWrt I have tested
give low bufferbloat results only when banddwidth is configured to
around 90% of available bandwidth, once I setup it to 200% of
available bandwidth I get really bad bufferbloat results, so any
current options just don't work if bandwidth changes.

Please don't assume I know anything, if you have any idea what I could
test or how to get better results in real world conditions please

Here are my current results it they are interesting to anybody.

Fiber connection 100/10 Mbps
no qos: 150/110 ms - 94/10.4 Mbps - http://www.dslreports.com/speedtest/2635531
no qos: 235/132 ms - 94.2/10.4 Mbps -
no qos: 160/130 ms - 96.8/10.5 Mbps -

Fiber connection: SQM 95000/9500 on eth0.2
fq_codel + nxt_routed_hfsc.qos: 203/42 ms - 95.2/7.2 Mbps -
fq_codel + layer_cake.qos: 193/135 ms - 93.1/7.7 Mbps -
fq_codel + layer_cake.qos: 201/116 ms - 93.6/8.2 Mbps -

Fiber connection: 100/10 Mbps - SQM 90/9 Mbps (90000/9000) on eth0.2
fq_codel + simple: 63/58 ms - 83/4 Mbps -
fq_codel + simple: 87/55 ms - 84.2/4.6 Mbps -
fq_codel + nxt_routed_hfsc: 130/46 ms - 94.6/6.9 Mbps -
fq_codel + nxt_routed_hfsc: 160/38 ms - 93/9.1 Mbps -
fq_codel + layer_cake: 140/120 ms - 91.3/6.6 Mbps -
fq_codel + layer_cake: 140/190 ms - 92.5/10.2 Mbps -
fq_codel + simplest: 41/35 ms - 81.4/3.7 Mbps -
fq_codel + simplest: 57/56 ms - 84.5/8.8 Mbps -
fq_codel + piece_of_cake: 175/119 ms - 93/7.5 Mbps -
fq_codel + piece_of_cake: 215/146 ms - 89.6/11.7 Mbps -
fq_codel + simple_pppoe: 61/48 ms - 84.1/8.5 Mbps -

Fiber connection: 100/10 Mbps - SQM 200/20 Mbps (200000/20000) on eth0.2
fq_codel + simple: 102/132 ms - 94.8/10.3 Mbps -
fq_codel + nxt_routed_hfsc: 150/133 ms - 94.1/10.5 Mbps -
fq_codel + simplest: 178/148 ms - 92.7/10.5 Mbps -
fq_codel + piece_of_cake: 192/122 ms - 95/10.7 Mbps -


[1] http://wiki.mikrotik.com/wiki/Manual:Queues_-_PCQ
[2] https://doc.pfsense.org/index.php/Traffic_Shaping_Guide

On Sun, Jan 17, 2016 at 9:31 PM, Outback Dingo <outbackdingo at gmail.com> wrote:
> On Sun, Jan 17, 2016 at 7:46 PM, Dave Taht <dave.taht at gmail.com> wrote:
>> http://www.meshpoint.me/ design and purpose seems very cool.
>> Given all the nifty new autoconfig stuff in homenet like hnetd and
>> source specific babel - perhaps we are closer than ever before to
>> actually being able to throw this kind off stuff off the back of a
>> truck and have it "just work".
>> If only we had bufferbloat-free lte uplink hardware (has any
>> materialized, perhaps in a mini-pci-e form factor?)
>> It does now seem semi-possible to hack on enodeBs nowadays:
>> http://bellard.org/lte/
>> Are any LTE providers supporting dhcp-pd or some equivalent for
>> getting routable ipv6 subnets?
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
> There have been "numerous" autoconfiguration setups deployed on OpenWRT by a
> few different people, even myself
> adding in 4G/LTE would be nice, though this has been possible for a while,
> even ive used the BladeRF in a "BTS" environment
> for testing of WhiteSpace wireless VOIP, and Disaster Scenerios, grab an
> Alix APU load OpenWRT, plug a bladeRF into the USB
> load OpenBTS on it and place it out in the wild.... Instant white space
> telco
>> https://www.gofundme.com/savewifi
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel at lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel

More information about the Cerowrt-devel mailing list