[Cerowrt-devel] routers you can throw off the back of a truck

[Cerowrt-devel] routers you can throw off the back of a truck

[Dave Taht]

http://www.meshpoint.me/ design and purpose seems very cool.

Given all the nifty new autoconfig stuff in homenet like hnetd and
source specific babel - perhaps we are closer than ever before to
actually being able to throw this kind off stuff off the back of a
truck and have it "just work".

If only we had bufferbloat-free lte uplink hardware (has any
materialized, perhaps in a mini-pci-e form factor?)

It does now seem semi-possible to hack on enodeBs nowadays:

http://bellard.org/lte/

Are any LTE providers supporting dhcp-pd or some equivalent for
getting routable ipv6 subnets?

--
Dave Täht
Let's go make home routers and wifi faster! With better software!
https://www.gofundme.com/savewifi

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Outback Dingo]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

On Sun, Jan 17, 2016 at 7:46 PM, Dave Taht <dave.taht@gmail.com> wrote:

> http://www.meshpoint.me/ design and purpose seems very cool.
>
> Given all the nifty new autoconfig stuff in homenet like hnetd and
> source specific babel - perhaps we are closer than ever before to
> actually being able to throw this kind off stuff off the back of a
> truck and have it "just work".
>
> If only we had bufferbloat-free lte uplink hardware (has any
> materialized, perhaps in a mini-pci-e form factor?)
>
> It does now seem semi-possible to hack on enodeBs nowadays:
>
> http://bellard.org/lte/
>
> Are any LTE providers supporting dhcp-pd or some equivalent for
> getting routable ipv6 subnets?
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
>

There have been "numerous" autoconfiguration setups deployed on OpenWRT by
a few different people, even myself
adding in 4G/LTE would be nice, though this has been possible for a while,
even ive used the BladeRF in a "BTS" environment
for testing of WhiteSpace wireless VOIP, and Disaster Scenerios, grab an
Alix APU load OpenWRT, plug a bladeRF into the USB
load OpenBTS on it and place it out in the wild.... Instant white space
telco



> https://www.gofundme.com/savewifi
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2416 bytes --]

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

Hi everybody,

our mission is to provide internet in humanitarian and crisis
situations to as many people as possible, and to make bandwidth fairly
distributed between users and to have low bufferbloat.

Issue with working out in crisis stations is that you don't know what
kind of uplink you will get, and bandwidth you get is much often
variable than constant. I have tested 3G/4G uplinks and download can
be in and range from 100Mbps to 5 Mbps, especially with mobile nodes
(we sometimes put router, battery and 3G/4G modem in backpack and go
where it is needed).

Is there anyway to configure any queuing technique in Linux kernel
that so it distributes bandwidth equally between users and to keep lag
(bufferbloat) as low as possible, but without needing to define
absolute values for download and upload. For bandwidth distribution I
know that Mikrotik has really awesome technique called PCQ [1] and
AFAIK there is no technique as PCQ in Linux kernel, the closes I have
found is SFQ, if you know any other similar queueing techniques in
Linux kernel please share.

I have been testing different queue setup scripts for fq_codel in
OpenWrt, to see how they work with properly setup bandwidth limits and
also with completely wrong bandwidth limits. Test have been done with
3G/4G USB modems and on 100/10 Mbps fibre optics connection with
TP-LINK WDR3600 router running latest Chaos Calmer OpenWrt.

Also I have also seen that pfSense mentions that HFSC [2] that is
"very effective for VoIP on links that degrade quickly, such as 3G/4G,
but it can be complex to configure and tweak for proper operation." so
I tested HFSC queuing technique with fq_codel with  and results show
that it also doesn't work if bandwith is not configured correctly

So far all fq_codel + queue setup scripts on OpenWrt I have tested
give low bufferbloat results only when banddwidth is configured to
around 90% of available bandwidth, once I setup it to 200% of
available bandwidth I get really bad bufferbloat results, so any
current options just don't work if bandwidth changes.

Please don't assume I know anything, if you have any idea what I could
test or how to get better results in real world conditions please
share...

Here are my current results it they are interesting to anybody.

Fiber connection 100/10 Mbps
no qos: 150/110 ms - 94/10.4 Mbps - http://www.dslreports.com/speedtest/2635531
no qos: 235/132 ms - 94.2/10.4 Mbps -
http://www.dslreports.com/speedtest/2635586
no qos: 160/130 ms - 96.8/10.5 Mbps -
http://www.dslreports.com/speedtest/2635646

Fiber connection: SQM 95000/9500 on eth0.2
fq_codel + nxt_routed_hfsc.qos: 203/42 ms - 95.2/7.2 Mbps -
http://www.dslreports.com/speedtest/2629376
fq_codel + layer_cake.qos: 193/135 ms - 93.1/7.7 Mbps -
http://www.dslreports.com/speedtest/2629461
fq_codel + layer_cake.qos: 201/116 ms - 93.6/8.2 Mbps -
http://www.dslreports.com/speedtest/2629502

Fiber connection: 100/10 Mbps - SQM 90/9 Mbps (90000/9000) on eth0.2
fq_codel + simple: 63/58 ms - 83/4 Mbps -
http://www.dslreports.com/speedtest/2629696
fq_codel + simple: 87/55 ms - 84.2/4.6 Mbps -
http://www.dslreports.com/speedtest/2629939
fq_codel + nxt_routed_hfsc: 130/46 ms - 94.6/6.9 Mbps -
http://www.dslreports.com/speedtest/2630019
fq_codel + nxt_routed_hfsc: 160/38 ms - 93/9.1 Mbps -
http://www.dslreports.com/speedtest/2630082
fq_codel + layer_cake: 140/120 ms - 91.3/6.6 Mbps -
http://www.dslreports.com/speedtest/2629597
fq_codel + layer_cake: 140/190 ms - 92.5/10.2 Mbps -
http://www.dslreports.com/speedtest/2630122
fq_codel + simplest: 41/35 ms - 81.4/3.7 Mbps -
http://www.dslreports.com/speedtest/2629988
fq_codel + simplest: 57/56 ms - 84.5/8.8 Mbps -
http://www.dslreports.com/speedtest/2630228
fq_codel + piece_of_cake: 175/119 ms - 93/7.5 Mbps -
http://www.dslreports.com/speedtest/2629749
fq_codel + piece_of_cake: 215/146 ms - 89.6/11.7 Mbps -
http://www.dslreports.com/speedtest/2630294
fq_codel + simple_pppoe: 61/48 ms - 84.1/8.5 Mbps -
http://www.dslreports.com/speedtest/2630153

Fiber connection: 100/10 Mbps - SQM 200/20 Mbps (200000/20000) on eth0.2
fq_codel + simple: 102/132 ms - 94.8/10.3 Mbps -
http://www.dslreports.com/speedtest/2630347
fq_codel + nxt_routed_hfsc: 150/133 ms - 94.1/10.5 Mbps -
http://www.dslreports.com/speedtest/2630386
fq_codel + simplest: 178/148 ms - 92.7/10.5 Mbps -
http://www.dslreports.com/speedtest/2630488
fq_codel + piece_of_cake: 192/122 ms - 95/10.7 Mbps -
http://www.dslreports.com/speedtest/2635507


Cheers,
Valent.


[1] http://wiki.mikrotik.com/wiki/Manual:Queues_-_PCQ
[2] https://doc.pfsense.org/index.php/Traffic_Shaping_Guide

On Sun, Jan 17, 2016 at 9:31 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
>
>
> On Sun, Jan 17, 2016 at 7:46 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> http://www.meshpoint.me/ design and purpose seems very cool.
>>
>> Given all the nifty new autoconfig stuff in homenet like hnetd and
>> source specific babel - perhaps we are closer than ever before to
>> actually being able to throw this kind off stuff off the back of a
>> truck and have it "just work".
>>
>> If only we had bufferbloat-free lte uplink hardware (has any
>> materialized, perhaps in a mini-pci-e form factor?)
>>
>> It does now seem semi-possible to hack on enodeBs nowadays:
>>
>> http://bellard.org/lte/
>>
>> Are any LTE providers supporting dhcp-pd or some equivalent for
>> getting routable ipv6 subnets?
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>
>
> There have been "numerous" autoconfiguration setups deployed on OpenWRT by a
> few different people, even myself
> adding in 4G/LTE would be nice, though this has been possible for a while,
> even ive used the BladeRF in a "BTS" environment
> for testing of WhiteSpace wireless VOIP, and Disaster Scenerios, grab an
> Alix APU load OpenWRT, plug a bladeRF into the USB
> load OpenBTS on it and place it out in the wild.... Instant white space
> telco
>
>
>>
>> https://www.gofundme.com/savewifi
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Jonathan Morton]

> On 18 Jan, 2016, at 02:45, Valent Turkovic <valent@otvorenamreza.org> wrote:
> 
> Is there anyway to configure any queuing technique in Linux kernel
> that so it distributes bandwidth equally between users and to keep lag
> (bufferbloat) as low as possible, but without needing to define
> absolute values for download and upload.

Recent versions of Cake have an “autorate_ingress” flag, which can track capacity when deployed on the downstream end of the link.  I use it myself to assist with 3G, where downlink quality and contention vary frequently.

I haven’t yet found a robust way to automatically sense link capacity from the upstream side.  You’ll therefore need to set a conservative static value for the uplink capacity.

 - Jonathan Morton

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

Hi Jonathan,

On Mon, Jan 18, 2016 at 9:30 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 18 Jan, 2016, at 02:45, Valent Turkovic <valent@otvorenamreza.org> wrote:
>>
>> Is there anyway to configure any queuing technique in Linux kernel
>> that so it distributes bandwidth equally between users and to keep lag
>> (bufferbloat) as low as possible, but without needing to define
>> absolute values for download and upload.
>
> Recent versions of Cake have an “autorate_ingress” flag, which can track capacity when deployed on the downstream end of the link.  I use it myself to assist with 3G, where downlink quality and contention vary frequently.

Just found your presentation [1] and this could be what I'm looking
for... and I'm very interested to test it further. My initial tests
[2] of piece_of_cake and layered_cake didn't show it in good light, it
had quite high latency when compared with other sqm scripts in
OpenWrt. Any ideas why? I have 100/10 Mbps fiber connection and during
the test I put 90/9 Mbps limit in SQM and got those results.

Can you please share your sqm qos script, or just how you invoke tc
manually and I'll test it on my routers and see what happens then:)

> I haven’t yet found a robust way to automatically sense link capacity from the upstream side.  You’ll therefore need to set a conservative static value for the uplink capacity.
>
>  - Jonathan Morton

From your presentation I see that if we had a daemon working in
background and somehow measured tcp latency (how?) and then we could
use it to raise/lower bandwidth limits on cake until we get best
possible results. Ideally I would like to use a queueing mechanism
that auto-configures everything.

@everybody any ideas how to tweak current "simple.qos" and
"simplest.qos" scripts in OpenWrt for 3G and fiber optics? On fiber
optic connection idle latency is around 30ms and on 3G connection is
around 60ms, do I need to change 5ms default in fq_codel to these
values? How?

Cheers,
Valent.


[1] http://www.bufferbloat.net/attachments/224/cake-battlemesh-v8.pdf
[2] http://pastebin.com/raw/BcizDmVX

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Alan Jenkins]

On 18/01/2016, Valent Turkovic <valent@otvorenamreza.org> wrote:

> @everybody any ideas how to tweak current "simple.qos" and
> "simplest.qos" scripts in OpenWrt for 3G and fiber optics? On fiber
> optic connection idle latency is around 30ms and on 3G connection is
> around 60ms, do I need to change 5ms default in fq_codel to these
> values? How?

Target should stay at 5ms, unless bandwidth is low ( < 2Mbit).
qos-scripts will tweak it according to the bandwidth you set.

Interval needs to match the expected round-trip-time, as a maximum
across most endpoints.  The 100ms default interval works well in many
cases... a base of 60ms might argue to increase it slightly for 3G, if
you expect some trans-continental destinations.[1]

Source: an update from the original Codel author.  Might not sound
directly related to your question, but I guess 3G falls under their
definition of a "bursty MAC".

"Many experimenters believe the value of target needs to be increased
independently of interval in this case. This note is to help explain
why this is not so."  http://pollere.net/Pdfdocs/noteburstymacs.pdf

The problem is that you need either a fixed rate you can use, or an
underlying connection that isn't itself horribly over-buffered.
(Ethernet with BQL, or dialup modems in the era of wondershaper...)
3G doesn't guarantee either :(.  I assume it also has the wireless
problem, of retry-induced latency on poor signal.  (ala Taht's
"infinite retry" bug).

If Johnathans new "autorate_ingress" solves or mitigates it for
ingress, that would be miraculous.  I had assumed it was most
applicable to point-to-point links which change the bitrate they use
over time due to changing signal quality, e.g. point-to-point wireless
v.s. weather.  (IIRC he had this clever idea of measuring the maximum
bitrate of two packets received back-to-back).

Btw make sure you've measured the idle latency during the busiest
period for the 3G network (for which it is still usable).  People
observe high variations during the day.

Regards
Alan

[1] E.g. from Europe you might expect some US endpoints, but not worry
about performance degradation to Australian-only servers.  You can try
pinging the world from your browser with
http://www.dslreports.com/tools/pingtest :).

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Jonathan Morton]

> On 18 Jan, 2016, at 11:43, Valent Turkovic <valent@otvorenamreza.org> wrote:
> 
> Can you please share your sqm qos script, or just how you invoke tc
> manually and I'll test it on my routers and see what happens then:)

The autorate_ingress option is just a flag.  Specify it after the bandwidth parameter to give it a sane starting point, say 1Mbit.  I think some of the more recent GUIs have a field for “advanced” or “experimental” options like this.  Once it sees some traffic, it should settle down reasonably quickly to the real link capacity, minus a small margin to establish itself as the bottleneck.

Eg: tc qdisc replace dev ifb0 root cake bandwidth 1Mbit autorate_ingress

As a reminder, autorate_ingress only works *downstream* of the bottleneck link.  Use it on the external interface’s *ingress* if possible.

> From your presentation I see that if we had a daemon working in
> background and somehow measured tcp latency (how?) and then we could
> use it to raise/lower bandwidth limits on cake until we get best
> possible results. Ideally I would like to use a queueing mechanism
> that auto-configures everything.

Right.  The autorate_ingress feature works entirely in kernelspace, and effectively takes care of the downstream half of the equation. The upstream half turns out to be a much harder problem, because we can only measure the uplink capacity when it is saturated, and typical consumer traffic doesn’t do that very often.  If we did have a saturating bulk upstream TCP flow, then we could examine its RTT profile in userspace, under the assumption that the downlink was taken care of.

One reasonable approach might be to use a userspace tool to periodically scrape the downlink speed out of autorate_ingress, and set the uplink speed to some fixed fraction of that (using tc qdisc change, for least disruption).  It might even make sense for 3G to inherently have such a ratio.  If it does, does anyone know what it is?

> @everybody any ideas how to tweak current "simple.qos" and
> "simplest.qos" scripts in OpenWrt for 3G and fiber optics? On fiber
> optic connection idle latency is around 30ms and on 3G connection is
> around 60ms, do I need to change 5ms default in fq_codel to these
> values? How?

These are essentially internet-scale latencies, especially if you’re just pinging the gateway immediately beyond the link, so the defaults will work fine.  The most recent versions of tc-adv include a set of intuitive keywords to specify commonly-encountered RTT ranges; the one for “internet” is 100ms, which corresponds to the Codel default parameters.

The 5ms figure is the target *queuing* latency, which should be considerably less than the estimated RTT; you really don’t want to be consistently adding 60ms of queuing on top of your 60ms inherent 3G latency.

 - Jonathan Morton

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

On Mon, Jan 18, 2016 at 12:29 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>
>> On 18 Jan, 2016, at 11:43, Valent Turkovic <valent@otvorenamreza.org> wrote:
>>
>> Can you please share your sqm qos script, or just how you invoke tc
>> manually and I'll test it on my routers and see what happens then:)
>
> The autorate_ingress option is just a flag.  Specify it after the bandwidth parameter to give it a sane starting point, say 1Mbit.  I think some of the more recent GUIs have a field for “advanced” or “experimental” options like this.  Once it sees some traffic, it should settle down reasonably quickly to the real link capacity, minus a small margin to establish itself as the bottleneck.
>
> Eg: tc qdisc replace dev ifb0 root cake bandwidth 1Mbit autorate_ingress

# tc qdisc replace dev eth0.2 root cake bandwidth 1Mbit autorate_ingress
Unknown qdisc "cake", hence option "bandwidth" is unparsable

So this is the reason I saw "bad" results when using cake... cake
qdisc isn't even available in latest Chaos Chalmer... but Luci shows
it as an option, really strange.
Cake script [1] is located in /usr/lib/sqm/piece_of_cake.qos but there
is no cake kernel module as far as I can see:

# opkg list | grep sched
kmod-sched - 3.18.20-1 - Extra kernel schedulers modules for IP traffic
kmod-sched-connmark - 3.18.20-1 - Traffic shaper conntrack mark support
kmod-sched-core - 3.18.20-1 - Core kernel scheduler support for IP traffic
kmod-sched-esfq - 3.18.20-1 - Traffic shaper ESFQ support

Again trough accidental discovery it looks like ESFQ [2] would also be
an nice addition to codel. How about efq_codel insead of fq_codel ?
Has anybody tried using ESFQ with codel?

But back to OpenWrt... are there Cake packages for OpenWrt available anywhere?

> As a reminder, autorate_ingress only works *downstream* of the bottleneck link.  Use it on the external interface’s *ingress* if possible.
>

I'll try this as soon as I get cake working on OpenWrt...


>> From your presentation I see that if we had a daemon working in
>> background and somehow measured tcp latency (how?) and then we could
>> use it to raise/lower bandwidth limits on cake until we get best
>> possible results. Ideally I would like to use a queueing mechanism
>> that auto-configures everything.
>
> Right.  The autorate_ingress feature works entirely in kernelspace, and effectively takes care of the downstream half of the equation. The upstream half turns out to be a much harder problem, because we can only measure the uplink capacity when it is saturated, and typical consumer traffic doesn’t do that very often.  If we did have a saturating bulk upstream TCP flow, then we could examine its RTT profile in userspace, under the assumption that the downlink was taken care of.
>
> One reasonable approach might be to use a userspace tool to periodically scrape the downlink speed out of autorate_ingress, and set the uplink speed to some fixed fraction of that (using tc qdisc change, for least disruption).  It might even make sense for 3G to inherently have such a ratio.  If it does, does anyone know what it is?
>
>> @everybody any ideas how to tweak current "simple.qos" and
>> "simplest.qos" scripts in OpenWrt for 3G and fiber optics? On fiber
>> optic connection idle latency is around 30ms and on 3G connection is
>> around 60ms, do I need to change 5ms default in fq_codel to these
>> values? How?
>
> These are essentially internet-scale latencies, especially if you’re just pinging the gateway immediately beyond the link, so the defaults will work fine.  The most recent versions of tc-adv include a set of intuitive keywords to specify commonly-encountered RTT ranges; the one for “internet” is 100ms, which corresponds to the Codel default parameters.
>
> The 5ms figure is the target *queuing* latency, which should be considerably less than the estimated RTT; you really don’t want to be consistently adding 60ms of queuing on top of your 60ms inherent 3G latency.

Thanks!

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

Forgot one link :)

[2] http://fatooh.org/esfq-2.6/

On Mon, Jan 18, 2016 at 3:07 PM, Valent Turkovic
<valent@otvorenamreza.org> wrote:
> On Mon, Jan 18, 2016 at 12:29 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>
>>> On 18 Jan, 2016, at 11:43, Valent Turkovic <valent@otvorenamreza.org> wrote:
>>>
>>> Can you please share your sqm qos script, or just how you invoke tc
>>> manually and I'll test it on my routers and see what happens then:)
>>
>> The autorate_ingress option is just a flag.  Specify it after the bandwidth parameter to give it a sane starting point, say 1Mbit.  I think some of the more recent GUIs have a field for “advanced” or “experimental” options like this.  Once it sees some traffic, it should settle down reasonably quickly to the real link capacity, minus a small margin to establish itself as the bottleneck.
>>
>> Eg: tc qdisc replace dev ifb0 root cake bandwidth 1Mbit autorate_ingress
>
> # tc qdisc replace dev eth0.2 root cake bandwidth 1Mbit autorate_ingress
> Unknown qdisc "cake", hence option "bandwidth" is unparsable
>
> So this is the reason I saw "bad" results when using cake... cake
> qdisc isn't even available in latest Chaos Chalmer... but Luci shows
> it as an option, really strange.
> Cake script [1] is located in /usr/lib/sqm/piece_of_cake.qos but there
> is no cake kernel module as far as I can see:
>
> # opkg list | grep sched
> kmod-sched - 3.18.20-1 - Extra kernel schedulers modules for IP traffic
> kmod-sched-connmark - 3.18.20-1 - Traffic shaper conntrack mark support
> kmod-sched-core - 3.18.20-1 - Core kernel scheduler support for IP traffic
> kmod-sched-esfq - 3.18.20-1 - Traffic shaper ESFQ support
>
> Again trough accidental discovery it looks like ESFQ [2] would also be
> an nice addition to codel. How about efq_codel insead of fq_codel ?
> Has anybody tried using ESFQ with codel?
>
> But back to OpenWrt... are there Cake packages for OpenWrt available anywhere?
>
>> As a reminder, autorate_ingress only works *downstream* of the bottleneck link.  Use it on the external interface’s *ingress* if possible.
>>
>
> I'll try this as soon as I get cake working on OpenWrt...
>
>
>>> From your presentation I see that if we had a daemon working in
>>> background and somehow measured tcp latency (how?) and then we could
>>> use it to raise/lower bandwidth limits on cake until we get best
>>> possible results. Ideally I would like to use a queueing mechanism
>>> that auto-configures everything.
>>
>> Right.  The autorate_ingress feature works entirely in kernelspace, and effectively takes care of the downstream half of the equation. The upstream half turns out to be a much harder problem, because we can only measure the uplink capacity when it is saturated, and typical consumer traffic doesn’t do that very often.  If we did have a saturating bulk upstream TCP flow, then we could examine its RTT profile in userspace, under the assumption that the downlink was taken care of.
>>
>> One reasonable approach might be to use a userspace tool to periodically scrape the downlink speed out of autorate_ingress, and set the uplink speed to some fixed fraction of that (using tc qdisc change, for least disruption).  It might even make sense for 3G to inherently have such a ratio.  If it does, does anyone know what it is?
>>
>>> @everybody any ideas how to tweak current "simple.qos" and
>>> "simplest.qos" scripts in OpenWrt for 3G and fiber optics? On fiber
>>> optic connection idle latency is around 30ms and on 3G connection is
>>> around 60ms, do I need to change 5ms default in fq_codel to these
>>> values? How?
>>
>> These are essentially internet-scale latencies, especially if you’re just pinging the gateway immediately beyond the link, so the defaults will work fine.  The most recent versions of tc-adv include a set of intuitive keywords to specify commonly-encountered RTT ranges; the one for “internet” is 100ms, which corresponds to the Codel default parameters.
>>
>> The 5ms figure is the target *queuing* latency, which should be considerably less than the estimated RTT; you really don’t want to be consistently adding 60ms of queuing on top of your 60ms inherent 3G latency.
>
> Thanks!

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

Here OpenWrt listsall available qdiscs:

ls -1 /tmp/run/sqm/available_qdiscs/
codel
fq_codel
pie
sfq

And found the reason why Cake isn't included in OpenWrt:
https://github.com/openwrt/packages/issues/1824

Just my luck :(

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]


Jonathan Morton <chromatix99@gmail.com> wrote:
> I haven’t yet found a robust way to automatically sense link capacity from
> the upstream side.  You’ll therefore need to set a conservative static
> value for the uplink capacity.

As the maintainer of a PPPoE concentrator, and operator of some networks,
I've been considering whether one can estimate the bandwidth using round
trip PPP IPCP keep alives.   Clearly, if both ends participate in time
stamping then it is much better, but I've been wondering if we can do some
incremental deployment on one side or the other.

Sadly, I mostly just think about this while cycling; I haven't written any
code yet.




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Dave Taht]

On Sun, Jan 17, 2016 at 4:45 PM, Valent Turkovic
<valent@otvorenamreza.org> wrote:
> Hi everybody,
>
> our mission is to provide internet in humanitarian and crisis
> situations to as many people as possible, and to make bandwidth fairly
> distributed between users and to have low bufferbloat.
>
> Issue with working out in crisis stations is that you don't know what
> kind of uplink you will get, and bandwidth you get is much often
> variable than constant. I have tested 3G/4G uplinks and download can
> be in and range from 100Mbps to 5 Mbps, especially with mobile nodes
> (we sometimes put router, battery and 3G/4G modem in backpack and go
> where it is needed).

Not being sure of the state of the LTE/3g/4g market (I haven't paid
much attention in several years) I posted the question to sierra
wireless's site
(registration required) here
https://forum.sierrawireless.com/viewtopic.php?f=121&t=9444

A far better way to try and find the right LTE hardware to use, IMHO,
would be for all those concerned to go to each manufacturer of
wireless gear's forums and raise the question, rather than being
isolated here. huawei, etc. (pick one, post the question, tell the
lists here the link to the thread, and we'll see what happened. wash,
rinse, repeat)

I have mildly more hope for pci-e derived lte interfaces being fixable
than usb ones, but know very little about how the hardware is
currently designed.

I would like very much to know of one bufferbloat-free lte interface
or of one that could be fixed as part of make-wifi-fast.
>
> Is there anyway to configure any queuing technique in Linux kernel
> that so it distributes bandwidth equally between users and to keep lag
> (bufferbloat) as low as possible, but without needing to define
> absolute values for download and upload. For bandwidth distribution I
> know that Mikrotik has really awesome technique called PCQ [1] and
> AFAIK there is no technique as PCQ in Linux kernel, the closes I have
> found is SFQ, if you know any other similar queueing techniques in
> Linux kernel please share.
>
> I have been testing different queue setup scripts for fq_codel in
> OpenWrt, to see how they work with properly setup bandwidth limits and
> also with completely wrong bandwidth limits. Test have been done with
> 3G/4G USB modems and on 100/10 Mbps fibre optics connection with
> TP-LINK WDR3600 router running latest Chaos Calmer OpenWrt.
>
> Also I have also seen that pfSense mentions that HFSC [2] that is
> "very effective for VoIP on links that degrade quickly, such as 3G/4G,
> but it can be complex to configure and tweak for proper operation." so
> I tested HFSC queuing technique with fq_codel with  and results show
> that it also doesn't work if bandwith is not configured correctly
>
> So far all fq_codel + queue setup scripts on OpenWrt I have tested
> give low bufferbloat results only when banddwidth is configured to
> around 90% of available bandwidth, once I setup it to 200% of
> available bandwidth I get really bad bufferbloat results, so any
> current options just don't work if bandwidth changes.
>
> Please don't assume I know anything, if you have any idea what I could
> test or how to get better results in real world conditions please
> share...
>
> Here are my current results it they are interesting to anybody.
>
> Fiber connection 100/10 Mbps
> no qos: 150/110 ms - 94/10.4 Mbps - http://www.dslreports.com/speedtest/2635531
> no qos: 235/132 ms - 94.2/10.4 Mbps -
> http://www.dslreports.com/speedtest/2635586
> no qos: 160/130 ms - 96.8/10.5 Mbps -
> http://www.dslreports.com/speedtest/2635646
>
> Fiber connection: SQM 95000/9500 on eth0.2
> fq_codel + nxt_routed_hfsc.qos: 203/42 ms - 95.2/7.2 Mbps -
> http://www.dslreports.com/speedtest/2629376
> fq_codel + layer_cake.qos: 193/135 ms - 93.1/7.7 Mbps -
> http://www.dslreports.com/speedtest/2629461
> fq_codel + layer_cake.qos: 201/116 ms - 93.6/8.2 Mbps -
> http://www.dslreports.com/speedtest/2629502
>
> Fiber connection: 100/10 Mbps - SQM 90/9 Mbps (90000/9000) on eth0.2
> fq_codel + simple: 63/58 ms - 83/4 Mbps -
> http://www.dslreports.com/speedtest/2629696
> fq_codel + simple: 87/55 ms - 84.2/4.6 Mbps -
> http://www.dslreports.com/speedtest/2629939
> fq_codel + nxt_routed_hfsc: 130/46 ms - 94.6/6.9 Mbps -
> http://www.dslreports.com/speedtest/2630019
> fq_codel + nxt_routed_hfsc: 160/38 ms - 93/9.1 Mbps -
> http://www.dslreports.com/speedtest/2630082
> fq_codel + layer_cake: 140/120 ms - 91.3/6.6 Mbps -
> http://www.dslreports.com/speedtest/2629597
> fq_codel + layer_cake: 140/190 ms - 92.5/10.2 Mbps -
> http://www.dslreports.com/speedtest/2630122
> fq_codel + simplest: 41/35 ms - 81.4/3.7 Mbps -
> http://www.dslreports.com/speedtest/2629988
> fq_codel + simplest: 57/56 ms - 84.5/8.8 Mbps -
> http://www.dslreports.com/speedtest/2630228
> fq_codel + piece_of_cake: 175/119 ms - 93/7.5 Mbps -
> http://www.dslreports.com/speedtest/2629749
> fq_codel + piece_of_cake: 215/146 ms - 89.6/11.7 Mbps -
> http://www.dslreports.com/speedtest/2630294
> fq_codel + simple_pppoe: 61/48 ms - 84.1/8.5 Mbps -
> http://www.dslreports.com/speedtest/2630153
>
> Fiber connection: 100/10 Mbps - SQM 200/20 Mbps (200000/20000) on eth0.2
> fq_codel + simple: 102/132 ms - 94.8/10.3 Mbps -
> http://www.dslreports.com/speedtest/2630347
> fq_codel + nxt_routed_hfsc: 150/133 ms - 94.1/10.5 Mbps -
> http://www.dslreports.com/speedtest/2630386
> fq_codel + simplest: 178/148 ms - 92.7/10.5 Mbps -
> http://www.dslreports.com/speedtest/2630488
> fq_codel + piece_of_cake: 192/122 ms - 95/10.7 Mbps -
> http://www.dslreports.com/speedtest/2635507
>
>
> Cheers,
> Valent.
>
>
> [1] http://wiki.mikrotik.com/wiki/Manual:Queues_-_PCQ
> [2] https://doc.pfsense.org/index.php/Traffic_Shaping_Guide
>
> On Sun, Jan 17, 2016 at 9:31 PM, Outback Dingo <outbackdingo@gmail.com> wrote:
>>
>>
>> On Sun, Jan 17, 2016 at 7:46 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>> http://www.meshpoint.me/ design and purpose seems very cool.
>>>
>>> Given all the nifty new autoconfig stuff in homenet like hnetd and
>>> source specific babel - perhaps we are closer than ever before to
>>> actually being able to throw this kind off stuff off the back of a
>>> truck and have it "just work".
>>>
>>> If only we had bufferbloat-free lte uplink hardware (has any
>>> materialized, perhaps in a mini-pci-e form factor?)
>>>
>>> It does now seem semi-possible to hack on enodeBs nowadays:
>>>
>>> http://bellard.org/lte/
>>>
>>> Are any LTE providers supporting dhcp-pd or some equivalent for
>>> getting routable ipv6 subnets?
>>>
>>> --
>>> Dave Täht
>>> Let's go make home routers and wifi faster! With better software!
>>
>>
>> There have been "numerous" autoconfiguration setups deployed on OpenWRT by a
>> few different people, even myself
>> adding in 4G/LTE would be nice, though this has been possible for a while,
>> even ive used the BladeRF in a "BTS" environment
>> for testing of WhiteSpace wireless VOIP, and Disaster Scenerios, grab an
>> Alix APU load OpenWRT, plug a bladeRF into the USB
>> load OpenBTS on it and place it out in the wild.... Instant white space
>> telco
>>
>>
>>>
>>> https://www.gofundme.com/savewifi
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Jonathan Morton]

> On 18 Jan, 2016, at 16:07, Valent Turkovic <valent@otvorenamreza.org> wrote:
> 
> Again trough accidental discovery it looks like ESFQ [2] would also be
> an nice addition to codel. How about efq_codel insead of fq_codel ?
> Has anybody tried using ESFQ with codel?

Hmmm…

> The most useful modification, for me, is the ability to change "hash type" so that ESFQ allocates bandwidth fairly per source IP rather than per connection.

You might want to read the recent thread on the Cake list about “triple flow isolation”.  That does everything ESFQ does - and more.

 - Jonathan Morton

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Jonathan Morton]

> On 18 Jan, 2016, at 18:14, Michael Richardson <mcr@sandelman.ca> wrote:
> 
> Jonathan Morton <chromatix99@gmail.com> wrote:
>> I haven’t yet found a robust way to automatically sense link capacity from
>> the upstream side.  You’ll therefore need to set a conservative static
>> value for the uplink capacity.
> 
> As the maintainer of a PPPoE concentrator, and operator of some networks,
> I've been considering whether one can estimate the bandwidth using round
> trip PPP IPCP keep alives.   Clearly, if both ends participate in time
> stamping then it is much better, but I've been wondering if we can do some
> incremental deployment on one side or the other.
> 
> Sadly, I mostly just think about this while cycling; I haven't written any
> code yet.

In most PPPoE deployments I know about, there is also a modem from which the actual, precise link rates can usually be queried.  Where that’s not the case, IPCP (or is it LPCP?) probes would be a reasonable workaround, but it must still be understood that the signal it provides is only valid under saturating traffic, which complicates implementation.

 - Jonathan Morton

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Michael Richardson]

Jonathan Morton <chromatix99@gmail.com> wrote:
    >> Jonathan Morton <chromatix99@gmail.com> wrote:
    >>> I haven’t yet found a robust way to automatically sense link capacity from
    >>> the upstream side.  You’ll therefore need to set a conservative static
    >>> value for the uplink capacity.
    >>
    >> As the maintainer of a PPPoE concentrator, and operator of some networks,
    >> I've been considering whether one can estimate the bandwidth using round
    >> trip PPP IPCP keep alives.   Clearly, if both ends participate in time
    >> stamping then it is much better, but I've been wondering if we can do some
    >> incremental deployment on one side or the other.
    >>
    >> Sadly, I mostly just think about this while cycling; I haven't written any
    >> code yet.

    > In most PPPoE deployments I know about, there is also a modem from
    > which the actual, precise link rates can usually be queried.  Where
    > that’s not the case, IPCP (or is it LPCP?) probes would be a reasonable
    > workaround, but it must still be understood that the signal it provides
    > is only valid under saturating traffic, which complicates
    > implementation.

Yes, you are rtight, I want to do LPCP echo requests.

The modem might know what speed it has with the tower/DSLAM, but won't know how
congested the backhaul link is.   There are some third party/white label 3G
arrangements in Canada that use PPP/L2TP back to the third-party provider,
but most route the IPv4 (only) packets via IPsec or MPLS.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Valent Turkovic]

Jonathan do you know why cake doesn't compile for mips platform?
How do you test Cake? Do you use OpenWrt or some other platform?
Can anyone help to get Cake compiled on OpenWrt?

On Mon, Jan 18, 2016 at 3:16 PM, Valent Turkovic
<valent@otvorenamreza.org> wrote:
> Here OpenWrt listsall available qdiscs:
>
> ls -1 /tmp/run/sqm/available_qdiscs/
> codel
> fq_codel
> pie
> sfq
>
> And found the reason why Cake isn't included in OpenWrt:
> https://github.com/openwrt/packages/issues/1824
>
> Just my luck :(

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[Jonathan Morton]

> On 22 Jan, 2016, at 01:40, Valent Turkovic <valent@otvorenamreza.org> wrote:
> 
> Jonathan do you know why cake doesn't compile for mips platform?
> How do you test Cake? Do you use OpenWrt or some other platform?
> Can anyone help to get Cake compiled on OpenWrt?

The bug report is several months old.  Cake has been developed and cleaned up significantly since then.  It’s a shame the log didn’t include the lines immediately *preceding* “some warnings being treated as errors”.

I myself test by compiling against a recent net-next kernel on x86 and ppc32, and an Ubuntu kernel (which is significantly older) on amd64.  I do have a mips router and some armv6/7 units as well, but I haven’t actively been using them for testing - it shouldn’t be any harder to get it running *just* because it’s mips, though.

 - Jonathan Morton

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[moeller0]

Hi Jonathan,

> On Jan 19, 2016, at 00:06 , Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> 
>> On 18 Jan, 2016, at 18:14, Michael Richardson <mcr@sandelman.ca> wrote:
>> 
>> Jonathan Morton <chromatix99@gmail.com> wrote:
>>> I haven’t yet found a robust way to automatically sense link capacity from
>>> the upstream side.  You’ll therefore need to set a conservative static
>>> value for the uplink capacity.
>> 
>> As the maintainer of a PPPoE concentrator, and operator of some networks,
>> I've been considering whether one can estimate the bandwidth using round
>> trip PPP IPCP keep alives.   Clearly, if both ends participate in time
>> stamping then it is much better, but I've been wondering if we can do some
>> incremental deployment on one side or the other.
>> 
>> Sadly, I mostly just think about this while cycling; I haven't written any
>> code yet.
> 
> In most PPPoE deployments I know about, there is also a modem from which the actual, precise link rates can usually be queried.  Where that’s not the case, IPCP (or is it LPCP?) probes would be a reasonable workaround, but it must still be understood that the signal it provides is only valid under saturating traffic, which complicates implementation.

	Erm, if one send probes of differing sizes, one can calculate the total effective bandwidth even without saturating the link (which unfortunately is an inseparable compound of ingress and egress, but will be heavily dominated by the lower, so typically the egress. And it should be relatively simple to flood the link while sending probes, in other words Valent’s router should be able to create the saturatng load on demand ;)

Best Regards
	Sebastian

> 
> - Jonathan Morton
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Make-wifi-fast] routers you can throw off the back of a truck

[moeller0]

Hi Valent,

> On Jan 22, 2016, at 07:37 , Jonathan Morton <chromatix99@gmail.com> wrote:
> 
> 
>> On 22 Jan, 2016, at 01:40, Valent Turkovic <valent@otvorenamreza.org> wrote:
>> 
>> Jonathan do you know why cake doesn't compile for mips platform?
>> How do you test Cake? Do you use OpenWrt or some other platform?
>> Can anyone help to get Cake compiled on OpenWrt?
> 
> The bug report is several months old.  Cake has been developed and cleaned up significantly since then.  It’s a shame the log didn’t include the lines immediately *preceding* “some warnings being treated as errors”.
> 
> I myself test by compiling against a recent net-next kernel on x86 and ppc32, and an Ubuntu kernel (which is significantly older) on amd64.  I do have a mips router and some armv6/7 units as well, but I haven’t actively been using them for testing - it shouldn’t be any harder to get it running *just* because it’s mips, though.

	I believe Kevin Darbyshire-Bryant created a nice how-to for including cake into a recent openwrt firmware (see last section of  http://www.bufferbloat.net/projects/codel/wiki/Cake ) which I believe works well. So well actually that I believe both Arokh’s ( https://forum.openwrt.org/viewtopic.php?id=50914 ) as well Hnyman’s ( https://forum.openwrt.org/viewtopic.php?id=28392 ) community builds now include cake (not sure whether they contain the most recent cake though) so it could be as si ole as just flashing one of their pre-compiled firmwares and get into testing almost immediately.

Best Tegards
	Sebastian


> 
> - Jonathan Morton
> 
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

Re: [Cerowrt-devel] routers you can throw off the back of a truck

[moeller0]

Hi Richard,

> On Jan 19, 2016, at 02:06 , Michael Richardson <mcr@sandelman.ca> wrote:
> 
> 
> Jonathan Morton <chromatix99@gmail.com> wrote:
>>> Jonathan Morton <chromatix99@gmail.com> wrote:
>>>> I haven’t yet found a robust way to automatically sense link capacity from
>>>> the upstream side.  You’ll therefore need to set a conservative static
>>>> value for the uplink capacity.
>>> 
>>> As the maintainer of a PPPoE concentrator, and operator of some networks,
>>> I've been considering whether one can estimate the bandwidth using round
>>> trip PPP IPCP keep alives.   Clearly, if both ends participate in time
>>> stamping then it is much better, but I've been wondering if we can do some
>>> incremental deployment on one side or the other.
>>> 
>>> Sadly, I mostly just think about this while cycling; I haven't written any
>>> code yet.
> 
>> In most PPPoE deployments I know about, there is also a modem from
>> which the actual, precise link rates can usually be queried.  Where
>> that’s not the case, IPCP (or is it LPCP?) probes would be a reasonable
>> workaround, but it must still be understood that the signal it provides
>> is only valid under saturating traffic, which complicates
>> implementation.
> 
> Yes, you are rtight, I want to do LPCP echo requests.

	It is a pity, that these do not carry a (high-resolution) time stamp per default. It would be sweet if at least the bidirectional RTT of each of the periodic keep-alive probes would be accessible, say somewhere under /sys… Then userspace would have an easy method to get information about the most relevant set of links.

> 
> The modem might know what speed it has with the tower/DSLAM, but won't know how
> congested the backhaul link is.  

	Then there is the issue of lack of standardized link speed reporting, which probably gets worse if we start looking at different link technologies (xDSL, LTE, dial-up, ...)

> There are some third party/white label 3G
> arrangements in Canada that use PPP/L2TP back to the third-party provider,
> but most route the IPv4 (only) packets via IPsec or MPLS.

	Then again the further away the bottleneck actually is the less effective our artificial bottleneck is going to be. (It is for example not guaranteed that pur traffic actually gets any bandwidth guarantee so reducing the bandwidth might simply ceed more bandwidth to other users without improving the latency under load, as on those links we do not compete with ourselves only, but I digress)

Best Regards
	Sebastian

> 
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Make-wifi-fast] routers you can throw off the back of a truck

[Valent Turkovic]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

On Fri, Jan 22, 2016 at 8:11 AM, moeller0 <moeller0@gmx.de> wrote:

> Hi Valent,
>
> > On Jan 22, 2016, at 07:37 , Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> >
> >> On 22 Jan, 2016, at 01:40, Valent Turkovic <valent@otvorenamreza.org>
> wrote:
> >>
> >> Jonathan do you know why cake doesn't compile for mips platform?
> >> How do you test Cake? Do you use OpenWrt or some other platform?
> >> Can anyone help to get Cake compiled on OpenWrt?
> >
> > The bug report is several months old.  Cake has been developed and
> cleaned up significantly since then.  It’s a shame the log didn’t include
> the lines immediately *preceding* “some warnings being treated as errors”.
> >
> > I myself test by compiling against a recent net-next kernel on x86 and
> ppc32, and an Ubuntu kernel (which is significantly older) on amd64.  I do
> have a mips router and some armv6/7 units as well, but I haven’t actively
> been using them for testing - it shouldn’t be any harder to get it running
> *just* because it’s mips, though.
>
>         I believe Kevin Darbyshire-Bryant created a nice how-to for
> including cake into a recent openwrt firmware (see last section of
> http://www.bufferbloat.net/projects/codel/wiki/Cake ) which I believe
> works well. So well actually that I believe both Arokh’s (
> https://forum.openwrt.org/viewtopic.php?id=50914 ) as well Hnyman’s (
> https://forum.openwrt.org/viewtopic.php?id=28392 ) community builds now
> include cake (not sure whether they contain the most recent cake though) so
> it could be as si ole as just flashing one of their pre-compiled firmwares
> and get into testing almost immediately.
>
>
Thanks Sebastian,
I have been doing battle on other fronts (hardware for MeshPoint) and now
finally got back to dealing with cake... I'll try instructions, they look
pretty straight forward.

What is the reason why cake in still not included by default in latest
trunk? It would make testing much easier for wider number of people...

Cheers,
Valent.

[-- Attachment #2: Type: text/html, Size: 2904 bytes --]