[NNagain] upgrading old routers to modern, secure FOSS

[NNagain] upgrading old routers to modern, secure FOSS

[Dave Taht]

I loved that this guy and his ISP burned a couple weeks learning how
to build openwrt, built something exactly to the need, *had it work
the first time* and are in progress to update in place 200+ routers to
better router software, that just works, with videoconferencing, IPv6
support, and OTA functionality. No need for a truck roll, and while
the available bandwidth deep in these mountains in Mexico is meager,
it is now enough for most purposes.

https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/

I have no idea how many of this model routers were sold or are still
deployed (?), but the modest up front cost of this sort of development
dwarves that of deployment. Ongoing maintenance is a problem, but at
least they are in a position now to rapidly respond to CVEs and other
problems when they happen, having "seized control of the methods of
computation" again.

OpenWrt is known to run on 1700 different models, already, (with easy
ports to obscure ones like this box) - going back over a decade in
some cases.

Another favorite story of mine was the ISP in New Zealand that
deployed LibreQos and had all their support calls (from gamers and
videoconferencers) cease overnight. The support tech, formerly drowned
in angst from the users, set to work automating an reflashing 600 old
agw routers they had "retired" on the shelf, and then distributing
them to customers as extenders because the wifi finally worked right
with the fq_codel stuff now in that release.

I feel like I am tooting my own horn here a bit too much, but solving
the right problems like MTTR, MTBF, bufferbloat, and taking back
control of your software infrastructure while being able to customize
it for purpose, and turning what otherwise would be ewaste into
something that will last a decade more, is my inner "green", my inner
stewart brand.

Compare that to so many others being marketed to, to death, that buy
the latest (and often inferior) thing, every few months, perpetually
fooled by promises that do not pay off in the field, and often, really
lousy MTBF. Good embedded software takes many years to develop, say,
oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
many eyeballs to fully debug and get to lots of 9s of reliability.

Back when I was even more radical about good, open, embedded, software
than now, I used to say: "Friends don't let friends run factory
firmware.". I do wish somehow the long term maintence costs of
hardware with a decade plus service lifetime would be adaquately
covered. Insurance? by law? a formal setaside from the purchase price?
Otherwise we run the risk of turning the world's internet into a giant
toxic waste dump that will require Superfund levels of cleanup, one
day, and ever more contributions to trillions of dollars of fraud, and
persistent actors having first broken down the front door, perpetually
on the inside, wreaking more havoc. Somehow preventing that mess, up
front, seems cheaper.

Take this string of vulns:
https://www.google.com/search?q=cisco+router+vulnerability

(try that search string with *any* manufacturer - juniper, netgear, tplink,

There is a new vuln going around about some very old software in a
cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
worked on this while at montavista in the early 00s!)  - abandonware,
toxic waste...

Anyway, in Mexico at least, 200+ routers are going to be a lot better,
through the actions of all that contribute to linux, openwrt, and one
smart and caring engineer.

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [NNagain] upgrading old routers to modern, secure FOSS

[le berger des photons]

[-- Attachment #1: Type: text/plain, Size: 4164 bytes --]

you've convinced me to go see libre qos.  thanks.

On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
nnagain@lists.bufferbloat.net> wrote:

> I loved that this guy and his ISP burned a couple weeks learning how
> to build openwrt, built something exactly to the need, *had it work
> the first time* and are in progress to update in place 200+ routers to
> better router software, that just works, with videoconferencing, IPv6
> support, and OTA functionality. No need for a truck roll, and while
> the available bandwidth deep in these mountains in Mexico is meager,
> it is now enough for most purposes.
>
>
> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>
> I have no idea how many of this model routers were sold or are still
> deployed (?), but the modest up front cost of this sort of development
> dwarves that of deployment. Ongoing maintenance is a problem, but at
> least they are in a position now to rapidly respond to CVEs and other
> problems when they happen, having "seized control of the methods of
> computation" again.
>
> OpenWrt is known to run on 1700 different models, already, (with easy
> ports to obscure ones like this box) - going back over a decade in
> some cases.
>
> Another favorite story of mine was the ISP in New Zealand that
> deployed LibreQos and had all their support calls (from gamers and
> videoconferencers) cease overnight. The support tech, formerly drowned
> in angst from the users, set to work automating an reflashing 600 old
> agw routers they had "retired" on the shelf, and then distributing
> them to customers as extenders because the wifi finally worked right
> with the fq_codel stuff now in that release.
>
> I feel like I am tooting my own horn here a bit too much, but solving
> the right problems like MTTR, MTBF, bufferbloat, and taking back
> control of your software infrastructure while being able to customize
> it for purpose, and turning what otherwise would be ewaste into
> something that will last a decade more, is my inner "green", my inner
> stewart brand.
>
> Compare that to so many others being marketed to, to death, that buy
> the latest (and often inferior) thing, every few months, perpetually
> fooled by promises that do not pay off in the field, and often, really
> lousy MTBF. Good embedded software takes many years to develop, say,
> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
> many eyeballs to fully debug and get to lots of 9s of reliability.
>
> Back when I was even more radical about good, open, embedded, software
> than now, I used to say: "Friends don't let friends run factory
> firmware.". I do wish somehow the long term maintence costs of
> hardware with a decade plus service lifetime would be adaquately
> covered. Insurance? by law? a formal setaside from the purchase price?
> Otherwise we run the risk of turning the world's internet into a giant
> toxic waste dump that will require Superfund levels of cleanup, one
> day, and ever more contributions to trillions of dollars of fraud, and
> persistent actors having first broken down the front door, perpetually
> on the inside, wreaking more havoc. Somehow preventing that mess, up
> front, seems cheaper.
>
> Take this string of vulns:
> https://www.google.com/search?q=cisco+router+vulnerability
>
> (try that search string with *any* manufacturer - juniper, netgear, tplink,
>
> There is a new vuln going around about some very old software in a
> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
> worked on this while at montavista in the early 00s!)  - abandonware,
> toxic waste...
>
> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
> through the actions of all that contribute to linux, openwrt, and one
> smart and caring engineer.
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
>

[-- Attachment #2: Type: text/html, Size: 5170 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 4882 bytes --]

Such a great thing to read! Another LibreQoS man aboard.

Btw, Ignacio might be here, but cc'ing him anyway.

All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <
nnagain@lists.bufferbloat.net> wrote:

> you've convinced me to go see libre qos.  thanks.
>
> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
>
>> I loved that this guy and his ISP burned a couple weeks learning how
>> to build openwrt, built something exactly to the need, *had it work
>> the first time* and are in progress to update in place 200+ routers to
>> better router software, that just works, with videoconferencing, IPv6
>> support, and OTA functionality. No need for a truck roll, and while
>> the available bandwidth deep in these mountains in Mexico is meager,
>> it is now enough for most purposes.
>>
>>
>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>>
>> I have no idea how many of this model routers were sold or are still
>> deployed (?), but the modest up front cost of this sort of development
>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>> least they are in a position now to rapidly respond to CVEs and other
>> problems when they happen, having "seized control of the methods of
>> computation" again.
>>
>> OpenWrt is known to run on 1700 different models, already, (with easy
>> ports to obscure ones like this box) - going back over a decade in
>> some cases.
>>
>> Another favorite story of mine was the ISP in New Zealand that
>> deployed LibreQos and had all their support calls (from gamers and
>> videoconferencers) cease overnight. The support tech, formerly drowned
>> in angst from the users, set to work automating an reflashing 600 old
>> agw routers they had "retired" on the shelf, and then distributing
>> them to customers as extenders because the wifi finally worked right
>> with the fq_codel stuff now in that release.
>>
>> I feel like I am tooting my own horn here a bit too much, but solving
>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>> control of your software infrastructure while being able to customize
>> it for purpose, and turning what otherwise would be ewaste into
>> something that will last a decade more, is my inner "green", my inner
>> stewart brand.
>>
>> Compare that to so many others being marketed to, to death, that buy
>> the latest (and often inferior) thing, every few months, perpetually
>> fooled by promises that do not pay off in the field, and often, really
>> lousy MTBF. Good embedded software takes many years to develop, say,
>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
>> many eyeballs to fully debug and get to lots of 9s of reliability.
>>
>> Back when I was even more radical about good, open, embedded, software
>> than now, I used to say: "Friends don't let friends run factory
>> firmware.". I do wish somehow the long term maintence costs of
>> hardware with a decade plus service lifetime would be adaquately
>> covered. Insurance? by law? a formal setaside from the purchase price?
>> Otherwise we run the risk of turning the world's internet into a giant
>> toxic waste dump that will require Superfund levels of cleanup, one
>> day, and ever more contributions to trillions of dollars of fraud, and
>> persistent actors having first broken down the front door, perpetually
>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>> front, seems cheaper.
>>
>> Take this string of vulns:
>> https://www.google.com/search?q=cisco+router+vulnerability
>>
>> (try that search string with *any* manufacturer - juniper, netgear,
>> tplink,
>>
>> There is a new vuln going around about some very old software in a
>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>> worked on this while at montavista in the early 00s!)  - abandonware,
>> toxic waste...
>>
>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
>> through the actions of all that contribute to linux, openwrt, and one
>> smart and caring engineer.
>>
>> --
>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>>
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
>

[-- Attachment #2: Type: text/html, Size: 7470 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Dave Taht]

On Mon, Oct 23, 2023 at 10:04 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> I loved that this guy and his ISP burned a couple weeks learning how
> to build openwrt, built something exactly to the need, *had it work
> the first time* and are in progress to update in place 200+ routers to
> better router software, that just works, with videoconferencing, IPv6
> support, and OTA functionality. No need for a truck roll, and while
> the available bandwidth deep in these mountains in Mexico is meager,
> it is now enough for most purposes.
>
> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/

In looking over that blog entry again today I know I overfocus on the
"bufferbloat" result, and the fact that he could indeed run a
speedtest while maintaining a good videoconfernce, which I really wish
more folk tested for. However it fails multiple checkboxes in the test
results, which others might be more inclined to look at.

4k video streaming: Failed. However this network is MORE than capable
of 1024p streaming. 4k is difficult to discern except on large,
expensive televisions. It was not all that long ago that 1024p was
considered good enough, and IMHO, still is.

Videoconferencing: Failed. Well, the test is wrong, probably having
too low a bar for the upload as a cutoff. Videoconferencing needs oh,
500kb/sec to work decently, and only facetime tends to try for 4k.
Having comprehensible voice, with a few video artifacts is ok,
incomprehensible voice, is not.

Low Latency gaming: Failed. The waveform test conflates two things
that it shouldn't - the effects of bufferbloat (none, in this case),
and the physical distance to the most local server, which was 70ms,
where the cutoff is 50ms in this test.

I wish that the city-dwellers of BEAD so in love with fiber would
insert 70ms of rural delay into all their testing. If someone would go
to all these enormous conferences about BEAD, and do that, the need
for cdns and uIXPs would become dramatically apparent in what they are
building out.

https://blog.cloudflare.com/tag/latency/

>
> I have no idea how many of this model routers were sold or are still
> deployed (?), but the modest up front cost of this sort of development
> dwarves that of deployment. Ongoing maintenance is a problem, but at
> least they are in a position now to rapidly respond to CVEs and other
> problems when they happen, having "seized control of the methods of
> computation" again.
>
> OpenWrt is known to run on 1700 different models, already, (with easy
> ports to obscure ones like this box) - going back over a decade in
> some cases.
>
> Another favorite story of mine was the ISP in New Zealand that
> deployed LibreQos and had all their support calls (from gamers and
> videoconferencers) cease overnight. The support tech, formerly drowned
> in angst from the users, set to work automating an reflashing 600 old
> agw routers they had "retired" on the shelf, and then distributing
> them to customers as extenders because the wifi finally worked right
> with the fq_codel stuff now in that release.
>
> I feel like I am tooting my own horn here a bit too much, but solving
> the right problems like MTTR, MTBF, bufferbloat, and taking back
> control of your software infrastructure while being able to customize
> it for purpose, and turning what otherwise would be ewaste into
> something that will last a decade more, is my inner "green", my inner
> stewart brand.
>
> Compare that to so many others being marketed to, to death, that buy
> the latest (and often inferior) thing, every few months, perpetually
> fooled by promises that do not pay off in the field, and often, really
> lousy MTBF. Good embedded software takes many years to develop, say,
> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
> many eyeballs to fully debug and get to lots of 9s of reliability.
>
> Back when I was even more radical about good, open, embedded, software
> than now, I used to say: "Friends don't let friends run factory
> firmware.". I do wish somehow the long term maintence costs of
> hardware with a decade plus service lifetime would be adaquately
> covered. Insurance? by law? a formal setaside from the purchase price?
> Otherwise we run the risk of turning the world's internet into a giant
> toxic waste dump that will require Superfund levels of cleanup, one
> day, and ever more contributions to trillions of dollars of fraud, and
> persistent actors having first broken down the front door, perpetually
> on the inside, wreaking more havoc. Somehow preventing that mess, up
> front, seems cheaper.
>
> Take this string of vulns:
> https://www.google.com/search?q=cisco+router+vulnerability
>
> (try that search string with *any* manufacturer - juniper, netgear, tplink,
>
> There is a new vuln going around about some very old software in a
> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
> worked on this while at montavista in the early 00s!)  - abandonware,
> toxic waste...
>
> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
> through the actions of all that contribute to linux, openwrt, and one
> smart and caring engineer.
>
> --
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Dave Taht]

On Mon, Oct 23, 2023 at 10:47 AM Frantisek Borsik via Nnagain
<nnagain@lists.bufferbloat.net> wrote:
>
> Such a great thing to read! Another LibreQoS man aboard.

Well, here is me trying to counter the now-too-popular idea that you
can shape everything at the middlebox at the ISP. It is a good start
to do that! The packet shaping *and* observability REALLY helps! But
in the end everything always works better if you have competent fq +
rfc7567 support at every possible bottleneck link, especially on
variable rate wireless transports, and especially in the face of
potentially difficult traffic like bittorrent, nat awareness and per
host/per flow fq have to happen at the customer site....

Correct backpressure at the right point, further modified by smart
algorithms is 20x less cpu intensive than shaping. The thing that bugs
me most is I see people trying to shape wifi to a fixed rate, which
works if you only test one device at "the right distance",  and never
otherwise. A lot of wifi manufactures like eero and google wifi got
this, but so far my testing of wifi6 was so miserable I decided to
stop working on it. I REALLY hope someone at the wifi conference this
week can show test results with multiple stations at multiple
distances doing the right things, as we did at gfiber here, back in
*2014*, see page 13:

http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf

Because wifi6 and 7 are otherwise a huge step backwards.

> Btw, Ignacio might be here, but cc'ing him anyway.

i loved so much that *starting from scratch* he was able to assemble a
good solution out of OpenWrt. It restores my faith in the engineering
world, to know many folk younger than me still have basic C and
makefile skills. Thx, man!

I have been doing embedded linux software since 1998, and am often a
bit grouchy about "the kids today" being unable to compile a kernel,
write a bit of xml for the dts stuff, and boom, have a totally custom
machine that does exactly what is needed. I see many wifi
manufacturers completely unable to build or modify or modify the
ancient crap they get from the chip manufacturer, including the chip
manufacturer!

Training AIs or engaging in social media are far more popular and
highly paid skills nowadays.


>
> All the best,
>
> Frank
>
> Frantisek (Frank) Borsik
>
>
>
> https://www.linkedin.com/in/frantisekborsik
>
> Signal, Telegram, WhatsApp: +421919416714
>
> iMessage, mobile: +420775230885
>
> Skype: casioa5302ca
>
> frantisek.borsik@gmail.com
>
>
>
> On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <nnagain@lists.bufferbloat.net> wrote:
>>
>> you've convinced me to go see libre qos.  thanks.
>>
>> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <nnagain@lists.bufferbloat.net> wrote:
>>>
>>> I loved that this guy and his ISP burned a couple weeks learning how
>>> to build openwrt, built something exactly to the need, *had it work
>>> the first time* and are in progress to update in place 200+ routers to
>>> better router software, that just works, with videoconferencing, IPv6
>>> support, and OTA functionality. No need for a truck roll, and while
>>> the available bandwidth deep in these mountains in Mexico is meager,
>>> it is now enough for most purposes.
>>>
>>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>>>
>>> I have no idea how many of this model routers were sold or are still
>>> deployed (?), but the modest up front cost of this sort of development
>>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>>> least they are in a position now to rapidly respond to CVEs and other
>>> problems when they happen, having "seized control of the methods of
>>> computation" again.
>>>
>>> OpenWrt is known to run on 1700 different models, already, (with easy
>>> ports to obscure ones like this box) - going back over a decade in
>>> some cases.
>>>
>>> Another favorite story of mine was the ISP in New Zealand that
>>> deployed LibreQos and had all their support calls (from gamers and
>>> videoconferencers) cease overnight. The support tech, formerly drowned
>>> in angst from the users, set to work automating an reflashing 600 old
>>> agw routers they had "retired" on the shelf, and then distributing
>>> them to customers as extenders because the wifi finally worked right
>>> with the fq_codel stuff now in that release.
>>>
>>> I feel like I am tooting my own horn here a bit too much, but solving
>>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>>> control of your software infrastructure while being able to customize
>>> it for purpose, and turning what otherwise would be ewaste into
>>> something that will last a decade more, is my inner "green", my inner
>>> stewart brand.
>>>
>>> Compare that to so many others being marketed to, to death, that buy
>>> the latest (and often inferior) thing, every few months, perpetually
>>> fooled by promises that do not pay off in the field, and often, really
>>> lousy MTBF. Good embedded software takes many years to develop, say,
>>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
>>> many eyeballs to fully debug and get to lots of 9s of reliability.
>>>
>>> Back when I was even more radical about good, open, embedded, software
>>> than now, I used to say: "Friends don't let friends run factory
>>> firmware.". I do wish somehow the long term maintence costs of
>>> hardware with a decade plus service lifetime would be adaquately
>>> covered. Insurance? by law? a formal setaside from the purchase price?
>>> Otherwise we run the risk of turning the world's internet into a giant
>>> toxic waste dump that will require Superfund levels of cleanup, one
>>> day, and ever more contributions to trillions of dollars of fraud, and
>>> persistent actors having first broken down the front door, perpetually
>>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>>> front, seems cheaper.
>>>
>>> Take this string of vulns:
>>> https://www.google.com/search?q=cisco+router+vulnerability
>>>
>>> (try that search string with *any* manufacturer - juniper, netgear, tplink,
>>>
>>> There is a new vuln going around about some very old software in a
>>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>>> worked on this while at montavista in the early 00s!)  - abandonware,
>>> toxic waste...
>>>
>>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
>>> through the actions of all that contribute to linux, openwrt, and one
>>> smart and caring engineer.
>>>
>>> --
>>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>> _______________________________________________
>>> Nnagain mailing list
>>> Nnagain@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/nnagain
>>
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [NNagain] upgrading old routers to modern, secure FOSS

[David Lang]

On Mon, 23 Oct 2023, Dave Taht via Nnagain wrote:

> 4k video streaming: Failed. However this network is MORE than capable
> of 1024p streaming. 4k is difficult to discern except on large,
> expensive televisions.

if you watch for sales, you can get fairly cheal 4k equipment. I have 27" 4k 
monitors and a 42" 4k tv and spent ~$200 each for them. That's not as cheap as 
1080p equipment is now, but it's not bad and you can easily spend more for 1080p 
equipment if you favor name brands

David Lang

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Frantisek Borsik]

[-- Attachment #1: Type: text/plain, Size: 8921 bytes --]

If you can help me to define how exactly it should be done, I will ask WLPC
<https://www.thewlpc.com/conferences/prague-czech-republic-2023> folks and
we will see if there is someone willing to do it.

On the other hand, this is why I was telling you about wireless LAN guys
and their admiration for Juniper, Ruckus and other big buffer guys...

 I REALLY hope someone at the wifi conference this
> week can show test results with multiple stations at multiple
> distances doing the right things, as we did at gfiber here, back in
> *2014*, see page 13:
>
> http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
> Because wifi6 and 7 are otherwise a huge step backwards.



All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Mon, Oct 23, 2023 at 8:11 PM Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Oct 23, 2023 at 10:47 AM Frantisek Borsik via Nnagain
> <nnagain@lists.bufferbloat.net> wrote:
> >
> > Such a great thing to read! Another LibreQoS man aboard.
>
> Well, here is me trying to counter the now-too-popular idea that you
> can shape everything at the middlebox at the ISP. It is a good start
> to do that! The packet shaping *and* observability REALLY helps! But
> in the end everything always works better if you have competent fq +
> rfc7567 support at every possible bottleneck link, especially on
> variable rate wireless transports, and especially in the face of
> potentially difficult traffic like bittorrent, nat awareness and per
> host/per flow fq have to happen at the customer site....
>
> Correct backpressure at the right point, further modified by smart
> algorithms is 20x less cpu intensive than shaping. The thing that bugs
> me most is I see people trying to shape wifi to a fixed rate, which
> works if you only test one device at "the right distance",  and never
> otherwise. A lot of wifi manufactures like eero and google wifi got
> this, but so far my testing of wifi6 was so miserable I decided to
> stop working on it. I REALLY hope someone at the wifi conference this
> week can show test results with multiple stations at multiple
> distances doing the right things, as we did at gfiber here, back in
> *2014*, see page 13:
>
>
> http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
>
> Because wifi6 and 7 are otherwise a huge step backwards.
>
> > Btw, Ignacio might be here, but cc'ing him anyway.
>
> i loved so much that *starting from scratch* he was able to assemble a
> good solution out of OpenWrt. It restores my faith in the engineering
> world, to know many folk younger than me still have basic C and
> makefile skills. Thx, man!
>
> I have been doing embedded linux software since 1998, and am often a
> bit grouchy about "the kids today" being unable to compile a kernel,
> write a bit of xml for the dts stuff, and boom, have a totally custom
> machine that does exactly what is needed. I see many wifi
> manufacturers completely unable to build or modify or modify the
> ancient crap they get from the chip manufacturer, including the chip
> manufacturer!
>
> Training AIs or engaging in social media are far more popular and
> highly paid skills nowadays.
>
>
> >
> > All the best,
> >
> > Frank
> >
> > Frantisek (Frank) Borsik
> >
> >
> >
> > https://www.linkedin.com/in/frantisekborsik
> >
> > Signal, Telegram, WhatsApp: +421919416714
> >
> > iMessage, mobile: +420775230885
> >
> > Skype: casioa5302ca
> >
> > frantisek.borsik@gmail.com
> >
> >
> >
> > On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
> >>
> >> you've convinced me to go see libre qos.  thanks.
> >>
> >> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
> >>>
> >>> I loved that this guy and his ISP burned a couple weeks learning how
> >>> to build openwrt, built something exactly to the need, *had it work
> >>> the first time* and are in progress to update in place 200+ routers to
> >>> better router software, that just works, with videoconferencing, IPv6
> >>> support, and OTA functionality. No need for a truck roll, and while
> >>> the available bandwidth deep in these mountains in Mexico is meager,
> >>> it is now enough for most purposes.
> >>>
> >>>
> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
> >>>
> >>> I have no idea how many of this model routers were sold or are still
> >>> deployed (?), but the modest up front cost of this sort of development
> >>> dwarves that of deployment. Ongoing maintenance is a problem, but at
> >>> least they are in a position now to rapidly respond to CVEs and other
> >>> problems when they happen, having "seized control of the methods of
> >>> computation" again.
> >>>
> >>> OpenWrt is known to run on 1700 different models, already, (with easy
> >>> ports to obscure ones like this box) - going back over a decade in
> >>> some cases.
> >>>
> >>> Another favorite story of mine was the ISP in New Zealand that
> >>> deployed LibreQos and had all their support calls (from gamers and
> >>> videoconferencers) cease overnight. The support tech, formerly drowned
> >>> in angst from the users, set to work automating an reflashing 600 old
> >>> agw routers they had "retired" on the shelf, and then distributing
> >>> them to customers as extenders because the wifi finally worked right
> >>> with the fq_codel stuff now in that release.
> >>>
> >>> I feel like I am tooting my own horn here a bit too much, but solving
> >>> the right problems like MTTR, MTBF, bufferbloat, and taking back
> >>> control of your software infrastructure while being able to customize
> >>> it for purpose, and turning what otherwise would be ewaste into
> >>> something that will last a decade more, is my inner "green", my inner
> >>> stewart brand.
> >>>
> >>> Compare that to so many others being marketed to, to death, that buy
> >>> the latest (and often inferior) thing, every few months, perpetually
> >>> fooled by promises that do not pay off in the field, and often, really
> >>> lousy MTBF. Good embedded software takes many years to develop, say,
> >>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
> >>> many eyeballs to fully debug and get to lots of 9s of reliability.
> >>>
> >>> Back when I was even more radical about good, open, embedded, software
> >>> than now, I used to say: "Friends don't let friends run factory
> >>> firmware.". I do wish somehow the long term maintence costs of
> >>> hardware with a decade plus service lifetime would be adaquately
> >>> covered. Insurance? by law? a formal setaside from the purchase price?
> >>> Otherwise we run the risk of turning the world's internet into a giant
> >>> toxic waste dump that will require Superfund levels of cleanup, one
> >>> day, and ever more contributions to trillions of dollars of fraud, and
> >>> persistent actors having first broken down the front door, perpetually
> >>> on the inside, wreaking more havoc. Somehow preventing that mess, up
> >>> front, seems cheaper.
> >>>
> >>> Take this string of vulns:
> >>> https://www.google.com/search?q=cisco+router+vulnerability
> >>>
> >>> (try that search string with *any* manufacturer - juniper, netgear,
> tplink,
> >>>
> >>> There is a new vuln going around about some very old software in a
> >>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
> >>> worked on this while at montavista in the early 00s!)  - abandonware,
> >>> toxic waste...
> >>>
> >>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
> >>> through the actions of all that contribute to linux, openwrt, and one
> >>> smart and caring engineer.
> >>>
> >>> --
> >>> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> >>> Dave Täht CSO, LibreQos
> >>> _______________________________________________
> >>> Nnagain mailing list
> >>> Nnagain@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/nnagain
> >>
> >> _______________________________________________
> >> Nnagain mailing list
> >> Nnagain@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/nnagain
> >
> > _______________________________________________
> > Nnagain mailing list
> > Nnagain@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/nnagain
>
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
>

[-- Attachment #2: Type: text/html, Size: 13522 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Sebastian Moeller]

Hi Dave,


> On Oct 23, 2023, at 19:58, Dave Taht via Nnagain <nnagain@lists.bufferbloat.net> wrote:
> 
> On Mon, Oct 23, 2023 at 10:04 AM Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> I loved that this guy and his ISP burned a couple weeks learning how
>> to build openwrt, built something exactly to the need, *had it work
>> the first time* and are in progress to update in place 200+ routers to
>> better router software, that just works, with videoconferencing, IPv6
>> support, and OTA functionality. No need for a truck roll, and while
>> the available bandwidth deep in these mountains in Mexico is meager,
>> it is now enough for most purposes.
>> 
>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
> 
> In looking over that blog entry again today I know I overfocus on the
> "bufferbloat" result, and the fact that he could indeed run a
> speedtest while maintaining a good videoconfernce, which I really wish
> more folk tested for. However it fails multiple checkboxes in the test
> results, which others might be more inclined to look at.
> 
> 4k video streaming: Failed. However this network is MORE than capable
> of 1024p streaming. 4k is difficult to discern except on large,
> expensive televisions. It was not all that long ago that 1024p was
> considered good enough, and IMHO, still is.

	[SM] With my aging eyes I agree "full HD" aka 1920 by 1080 still looks plenty fine to me, even on our biggest screen (43").However the older I get the less picky I get, even SD resolution will not keep me from watching things if the content is compelling ;)
	-> 4K streaming is reported as failure due to insufficient download capacity.


> Videoconferencing: Failed. Well, the test is wrong, probably having
> too low a bar for the upload as a cutoff. Videoconferencing needs oh,
> 500kb/sec to work decently, and only facetime tends to try for 4k.
> Having comprehensible voice, with a few video artifacts is ok,
> incomprehensible voice, is not.

	[SM] Videoconferencing reported as failure due to insufficient upload capacity, I am sure though that 10.6/3.46 Mbps will be enough for decent video conferencing for a single seat.

> 
> Low Latency gaming: Failed. The waveform test conflates two things
> that it shouldn't - the effects of bufferbloat (none, in this case),
> and the physical distance to the most local server, which was 70ms,
> where the cutoff is 50ms in this test.

	[SM] the cutoff is reported as "95th Percentile Latency < 40 ms" which is indeed harsh. 



Here is the expanded list of the grading rules:
We use the following criteria to determine if a particular service will work on your Internet connection. Of course, these criteria are far from perfect, but we think they’re a good general guideline.
	• Web Browsing:
		• Download speed > 2 Mbps
		• Upload speed > 100 Kbps
		• Latency < 500 ms
	• Audio Calls:
		• Download speed > 100 Kbps
		• Upload speed > 100 Kbps
		• 95th Percentile Latency < 400 ms
	• 4K Video Streaming:
		• Download speed > 25 Mbps
	• Video Conferencing:
		• Download speed > 10 Mbps
		• Upload speed > 5 Mbps
		• 95th Percentile Latency < 400 ms
	• Low Latency Gaming:
		• Download speed > 10 Mbps
		• Upload speed > 3 Mbps
		• 95th Percentile Latency < 40 ms



> I wish that the city-dwellers of BEAD so in love with fiber would
> insert 70ms of rural delay into all their testing.

	[SM] In fiber 70ms RTT is good for 70 *100 = 7000 Km, that is a lot of latency, sure there are other delays other than propagation delay, but I wish we could wire up more rural ares with better topologies that avoid 7000 Km detours... here however the issue might well be more cloudflare sparsity in MX, they only mention Maxico City and Queretaro... Maxico is quite large, but even then 70ms indicates clear potential.

BUT I also think that we should be able to build an internet infrastructure that can cope decently with such delays!


> If someone would go
> to all these enormous conferences about BEAD, and do that, the need
> for cdns and uIXPs would become dramatically apparent in what they are
> building out.
> 
> https://blog.cloudflare.com/tag/latency/
> 
>> 
>> I have no idea how many of this model routers were sold or are still
>> deployed (?), but the modest up front cost of this sort of development
>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>> least they are in a position now to rapidly respond to CVEs and other
>> problems when they happen, having "seized control of the methods of
>> computation" again.
>> 
>> OpenWrt is known to run on 1700 different models, already, (with easy
>> ports to obscure ones like this box) - going back over a decade in
>> some cases.
>> 
>> Another favorite story of mine was the ISP in New Zealand that
>> deployed LibreQos and had all their support calls (from gamers and
>> videoconferencers) cease overnight. The support tech, formerly drowned
>> in angst from the users, set to work automating an reflashing 600 old
>> agw routers they had "retired" on the shelf, and then distributing
>> them to customers as extenders because the wifi finally worked right
>> with the fq_codel stuff now in that release.
>> 
>> I feel like I am tooting my own horn here a bit too much, but solving
>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>> control of your software infrastructure while being able to customize
>> it for purpose, and turning what otherwise would be ewaste into
>> something that will last a decade more, is my inner "green", my inner
>> stewart brand.
>> 
>> Compare that to so many others being marketed to, to death, that buy
>> the latest (and often inferior) thing, every few months, perpetually
>> fooled by promises that do not pay off in the field, and often, really
>> lousy MTBF. Good embedded software takes many years to develop, say,
>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
>> many eyeballs to fully debug and get to lots of 9s of reliability.
>> 
>> Back when I was even more radical about good, open, embedded, software
>> than now, I used to say: "Friends don't let friends run factory
>> firmware.". I do wish somehow the long term maintence costs of
>> hardware with a decade plus service lifetime would be adaquately
>> covered. Insurance? by law? a formal setaside from the purchase price?
>> Otherwise we run the risk of turning the world's internet into a giant
>> toxic waste dump that will require Superfund levels of cleanup, one
>> day, and ever more contributions to trillions of dollars of fraud, and
>> persistent actors having first broken down the front door, perpetually
>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>> front, seems cheaper.
>> 
>> Take this string of vulns:
>> https://www.google.com/search?q=cisco+router+vulnerability
>> 
>> (try that search string with *any* manufacturer - juniper, netgear, tplink,
>> 
>> There is a new vuln going around about some very old software in a
>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>> worked on this while at montavista in the early 00s!)  - abandonware,
>> toxic waste...
>> 
>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
>> through the actions of all that contribute to linux, openwrt, and one
>> smart and caring engineer.
>> 
>> --
>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Jack Haverty]

[-- Attachment #1: Type: text/plain, Size: 2350 bytes --]

On 10/23/23 10:58, Dave Taht via Nnagain wrote:
> I wish that the city-dwellers of BEAD so in love with fiber would 
> insert 70ms of rural delay into all their testing.
FYI, in case someone wants to pursue such real-world testing....

When we were testing TCP/IP software about 40 years ago there was a 
similar problem of how to do tests in a lab which realistically 
simulated real-world conditions.   We created a software tool called 
"Flakeway" which enable traffic flows to be delayed, duplicated, 
re-ordered, deleted or mangled.   That enabled realistic testing even 
when the machines being tested were all in a lab connected to the same LAN.

That software is long gone, but might be easily rewritten today.  It was 
literally a weekend hack.   Here's how it worked.

The basic design took advantage of a "feature" of the IP protocols. When 
an IP datagram is to be sent to another computer on the same Ethernet, 
the IP address isn't big enough to encode the Ethernet address.  So the 
ARP mechanism is used to get the appropriate mapping between an IP 
address and the required Ethernet address for the destination host.  The 
sender issues an ARP request that says "Where is IP address x.x.x.x"?  
The computer which is configured as that IP address responds with "It's 
me, and my Ethernet address is xx:xx:xx:xx:xx:xx"

When the Flakeway, running on some other computer on the same LAN, saw 
such an ARP exchange for a traffic flow it was supposed to manipulate, 
it would immediately send it's own ARP response, saying "No, it's me, 
and my Ethernet address is..."

We discovered that most computers simply believed the latest ARP 
information it received.   So it was easy for the Flakeway to insert 
itself into any IP traffic flow and do its work, without any changes to 
software in any other computer.  It was handy not only for testing but 
also for diagnosing all sorts of problems, simply capturing the traffic 
flows for later analysis (similar to wireshark).

That was all done in the IPV4 world, 40+ years ago, so I'm not sure how 
it might relate to today's Internet.   We reported this "feature" to 
IETF and some IEEE 802.x committee as a likely vulnerability, but I'm 
not sure if anything changed.

But something similar might be possible in today's world to improve 
real-world testing?

Jack Haverty

[-- Attachment #2: Type: text/html, Size: 2859 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

On Mon, 23 Oct 2023, Jack Haverty via Nnagain wrote:

> We discovered that most computers simply believed the latest ARP information 
> it received.   So it was easy for the Flakeway to insert itself into any IP 
> traffic flow and do its work, without any changes to software in any other 
> computer.  It was handy not only for testing but also for diagnosing all 
> sorts of problems, simply capturing the traffic flows for later analysis 
> (similar to wireshark).
>
> That was all done in the IPV4 world, 40+ years ago, so I'm not sure how it 
> might relate to today's Internet.   We reported this "feature" to IETF and 
> some IEEE 802.x committee as a likely vulnerability, but I'm not sure if 
> anything changed.

This is commonly used today for failover/load balancing

David Lang

[-- Attachment #2: Type: text/plain, Size: 146 bytes --]

_______________________________________________
Nnagain mailing list
Nnagain@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/nnagain

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Karl Auerbach]

[-- Attachment #1: Type: text/plain, Size: 2470 bytes --]


On 10/23/23 11:53 AM, Jack Haverty via Nnagain wrote:
> On 10/23/23 10:58, Dave Taht via Nnagain wrote:
>> I wish that the city-dwellers of BEAD so in love with fiber would 
>> insert 70ms of rural delay into all their testing.
> FYI, in case someone wants to pursue such real-world testing....
>
> When we were testing TCP/IP software about 40 years ago there was a 
> similar problem of how to do tests in a lab which realistically 
> simulated real-world conditions.   We created a software tool called 
> "Flakeway" which enable traffic flows to be delayed, duplicated, 
> re-ordered, deleted or mangled.   That enabled realistic testing even 
> when the machines being tested were all in a lab connected to the same 
> LAN.

When we were doing TCP "bakeoffs" at the FTP Software facility we 
dreamed of having such a device.

When Steve Casner and I were doing entertainment grade audio/video back 
in the late 1990s we discovered that we were in great need of something 
like Postel's Flakeway.  (Receiving RTP code and codecs, especially when 
dealing with multiple lip-synched streams, can be very sensitive to 
inter-packet timing and packet reception order - it was very hard for us 
to reliably test all the code paths.)

So a few years later  I implemented Jon's Flakeway idea, but at layer 2 
rather than 3.  (It was far from a weekend hack.)  I've now gone through 
multiple generations of the idea and sell it as a (reasonably 
successful) testing product.  I'll attach a screen shot so that one can 
get an idea of what it does. (Hopefully the mail handler for this list 
doesn't get upset with the attachment.)

(We've also got versions that do some protocol tracking and rewrite 
packets in "interesting" ways on the fly.  We've had some less-than-fun 
[for the customer] experiences such as when a phone vendor wanted us to 
exercise their IPv6 code but only had their firmware based IPv4 ready 
[and 200, 000+ units already in customer hands].  Within a couple of 
minutes we had found issues with their TCP stack - it seems that far too 
much IP and TCP code was written in C and used the default signed 
integer data type rather than unsigned and thus has troubles when the 
high order bit in a packet field changes.  Perhaps the must vulnerable 
protocol on the net is SIP - I sometimes believe that it should have as 
its icon a target with an over-large bullseye with a bunch of arrows in 
that bullseye.)

         --karl--


[-- Attachment #2: kmax-c-graph-page.png --]
[-- Type: image/png, Size: 1438493 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[rjmcmahon]

I get the nostalgia for old tools and old routers but the industry is 
providing better tools for testing TCP and much better hw & network 
stacks. One such tool is iperf 2, and it's actively maintained and 
released as open source. https://sourceforge.net/projects/iperf2/

TCP is not static either. There is lot of development including new 
socket options, CCAs, etc that need testing. Also, single flows are no 
longer enough. Each end device likely has dozens, if not more, TCP state 
machines active at any one moment in time and are multicore so testing 
really needs to be multithreaded and multiflow.

The analytics side needs to support statistical & multivariate analysis. 
Libraries like python scikit learn need to be considered in the design. 
That's why the flows code based upon python3 is also released as open 
source with examples on how to do things like kolmogorov-smirnov CDF 
distances.

I see a shortage in network expertise, including staying current with 
current transistor designs and silicon, vs a tools issue. Networking 
keeps progressing and staying current requires continuous efforts.

Home networks today are embarrassing to me. Our industry is woefully 
behind here.

Not trying to be rude, just calling it as I see it.

Bob
> On 10/23/23 11:53 AM, Jack Haverty via Nnagain wrote:
>> On 10/23/23 10:58, Dave Taht via Nnagain wrote:
>>> I wish that the city-dwellers of BEAD so in love with fiber would 
>>> insert 70ms of rural delay into all their testing.
>> FYI, in case someone wants to pursue such real-world testing....
>> 
>> When we were testing TCP/IP software about 40 years ago there was a 
>> similar problem of how to do tests in a lab which realistically 
>> simulated real-world conditions.   We created a software tool called 
>> "Flakeway" which enable traffic flows to be delayed, duplicated, 
>> re-ordered, deleted or mangled.   That enabled realistic testing even 
>> when the machines being tested were all in a lab connected to the same 
>> LAN.
> 
> When we were doing TCP "bakeoffs" at the FTP Software facility we
> dreamed of having such a device.
> 
> When Steve Casner and I were doing entertainment grade audio/video
> back in the late 1990s we discovered that we were in great need of
> something like Postel's Flakeway.  (Receiving RTP code and codecs,
> especially when dealing with multiple lip-synched streams, can be very
> sensitive to inter-packet timing and packet reception order - it was
> very hard for us to reliably test all the code paths.)
> 
> So a few years later  I implemented Jon's Flakeway idea, but at layer
> 2 rather than 3.  (It was far from a weekend hack.)  I've now gone
> through multiple generations of the idea and sell it as a (reasonably
> successful) testing product.  I'll attach a screen shot so that one
> can get an idea of what it does. (Hopefully the mail handler for this
> list doesn't get upset with the attachment.)
> 
> (We've also got versions that do some protocol tracking and rewrite
> packets in "interesting" ways on the fly.  We've had some
> less-than-fun [for the customer] experiences such as when a phone
> vendor wanted us to exercise their IPv6 code but only had their
> firmware based IPv4 ready [and 200, 000+ units already in customer
> hands].  Within a couple of minutes we had found issues with their TCP
> stack - it seems that far too much IP and TCP code was written in C
> and used the default signed integer data type rather than unsigned and
> thus has troubles when the high order bit in a packet field changes. 
> Perhaps the must vulnerable protocol on the net is SIP - I sometimes
> believe that it should have as its icon a target with an over-large
> bullseye with a bunch of arrows in that bullseye.)
> 
>         --karl--
> 
> 
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Karl Auerbach]

On 10/23/23 2:54 PM, rjmcmahon wrote:
> Home networks today are embarrassing to me. Our industry is woefully 
> behind here.
>
I would be more expansive.

(Bringing this back to network neutrality - my argument, not clearly 
suggested below, is that "neutrality" is more than bandwidth or 
connectivity but ought also ought to include other aspects including 
robust and repairable service in the face of reasonably foreseeable 
events.  By-the-way, when I was involved in the early days of the net, I 
worked for groups [such as the US Joint Chiefs] who thought that routers 
being vaporized by nuclear explosions were "reasonable foreseeable".)

The lawyer half of me lives in fear of the harm that can come from bad 
code in network devices.  I've seen the growth of strict product 
liability laws in the consumer space (sometimes resulting in those silly 
"do not eat" labels on silica gel packets, but also resulting in 
important steps, like pressure-release closures on cleaning products 
that contain dry sodium hydroxide, or dual braking systems in automobiles.)

And the railroad nut in me remembers that Murphy's law is as strong as 
ever.  (Just ask "Why are highway and railroad traffic control signals 
red and green [actually a quite bluish green]?" [Hint, they originally 
were red and white, and sometimes the red colored lens would fall out.])

When I was working with the DARPA Robotics Challenge my job was to 
introduce network problems - the kinds of things that can happen in real 
life when a robot operates in a disaster zone.  I could introduce a 
simple change - like increasing the level of lost Ethernet frames when a 
robot went through a door into a (simulated) concrete reactor building - 
and the robot would simply stop or fall over.

I've seen videos of animal surgery performed by remote control over a 
long distance (50km) network link where the doctors presumed that the 
net was endlessly flawless.  (I have this mental image of a robotic 
scalpel overshooting its cut due to a non-idempotent command contained 
in a packet that was replicated on the net.)

And I've seen users of satellites fail to remember that every now and 
then, from the point of view of a ground station, a satellite may 
transit across the face of the sun (a highly predictable event) and be 
temporarily blinded and unable to receive data.

Many of our implementations today are hanging on only because modern 
machines have gobs upon gobs of memory and nobody notices if a couple of 
gigabytes leak or are uselessly allocated for a few minutes.

(For instance, one way to stop a Linux stack is to send it patterns of 
tiny IPv4 fragments that overlap or have gaps so that reassembly is not 
possible (or difficult) and buffers just sit there waiting for a rather 
long timeout before being reclaimed.)

It seems that everybody and her brothers think they can write code.  And 
they do.  And in our open source world the code they write is often 
protocol code.  Often it is badly written protocol code containing 
monumental flaws, such as use of "integer" types in C (when "unsigned 
uint16" or similar is needed), failure to recognize that number spaces 
wrap, assumptions that "everything is in ASCII" or that character 
sequences do not contain null bytes. (Last time I looked some major 
libraries went down in flames when string data in packets happened to 
contain nulls - the code was using ancient Unix/C string routines.)  I 
once sent several SIP phones into the weeds when I sent length fields 
(in text form) with leading zero characters (e.g. 050 rather than 50) - 
some code treated that as octal!)

It would certainly be nice if we had a body of network implementation 
design/implementation rules - similar in concept to engineering design 
rules used in bridges, aircraft, electrical networks, etc - for use when 
writing code.  Any one who wanted to do something outside of those rules 
could do so, but would be strongly "encouraged" to seek the advice and 
oversight of others.

Once the Interop show net was brought to a stop (by infinitely looping 
packets) when two brands of routers had different notions how to expand 
IPv4 multicast addresses into MAC addresses.  (I can't remember the 
details, but when every light in the NOC turned red everybody in the 
Interop NOC turned to look at me, guessing [incorrectly in this 
instance] that I was the cause.])

It would be nice if we built our network devices so that they each had a 
little introspective daemon that frequently asked "am I healthy, am I 
still connected, are packets still moving through me?"  (For consumer 
devices an answer of "no" could trigger a full device reboot or reset.)

For larger devices, such as routers, we could have some machinery, 
internal or external, that did a bit of modelling and informed the 
routing machinery of anticipated queue lengths and similar metrics.  
Then the router could monitor itself to check if it was wobbling outside 
of those anticipated ranges and take appropriate action to signal the 
issue.  (I was once quite surprised to learn on at least one large type 
of router that it was difficult-to-impossible to obtain queue length 
data because so much function had been pushed into hardware that had few 
test or measurement points.)

My grandfather and father were radio and TV repair guys.  I learned from 
an early age the value of good tools and of looking outside the basic 
operation of a device for symptoms. (You could often hear a failing 
capacitor or inductor; or you could smell a slowly burning resistor.)  
Our modern networks and code usually lack that kind of observational 
(and active testing) plane.

I can see a big net neutrality differentiator between providers being 
"time to detect" and "time to repair".

         --karl--

Re: [NNagain] upgrading old routers to modern, secure FOSS

[David Lang]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

On Mon, 23 Oct 2023, Karl Auerbach via Nnagain wrote:

> It would be nice if we built our network devices so that they each had a 
> little introspective daemon that frequently asked "am I healthy, am I 
> still connected, are packets still moving through me?"  (For consumer 
> devices an answer of "no" could trigger a full device reboot or reset.)

I agree with a lot of what you say, but I want to throw in a word of caution 
here. I have seen systems go from 'slow but functioning' to 'completely down and 
requires a complete datacenter shutdown to recover' because of automated 
response systems that decided to restart something when it didn't respond fast 
enough, triggering a cascade of failures that prevented any service from being 
able to start into a healthy state.

I've also implemented monitoring on APs to restart them if they don't have a 
path to the Internet, resulting in continual reboots when there is a transitory 
issue (now changed to only check their next hop and only shut down wifi to avoid 
becoming a black hole for that SSID

to err is human, to really mess things up requires a computer, and automation 
removes the oversight from the computer allowing it to do more damage faster.

David Lang

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Karl Auerbach]

I couldn't agree with you more - we need to take care that any control 
or management systems we create are not turn-off-the-Internet switches 
in disguise.

My even larger fear is that as we increasingly cross-link our various 
forms of infrastructure that protective measures will put us into 
(hopefully transient) neo-stone age with a long, difficult recovery.

I have had a long interest that comes from comparing the relative robust 
response of living organisms to the rather brittle responses of our 
technologies.

Living things have an option that is not usually available to our 
technologies - death of the individual.

The one lesson I've been able to draw out so far is that living things 
often have layers of responsive mechanisms that arise because 
evolutionary processes typically do not erase old machinery, but, 
rather, add new responses.  If the new response proves inadequate then 
the old mechanisms are still there and might offer a useful solution to 
whatever condition has happened.

The corollary that I derived from that is that we ought to be designing 
our network systems with layers of response machinery, often working 
somewhat at cross purposes, and with the goal being survival rather than 
optimal use of resources.

How to do this in practice remains somewhat elusive, at least to me.

     --karl--

On 10/23/23 4:39 PM, David Lang wrote:
> On Mon, 23 Oct 2023, Karl Auerbach via Nnagain wrote:
>
>> It would be nice if we built our network devices so that they each 
>> had a little introspective daemon that frequently asked "am I 
>> healthy, am I still connected, are packets still moving through me?"  
>> (For consumer devices an answer of "no" could trigger a full device 
>> reboot or reset.)
>
> I agree with a lot of what you say, but I want to throw in a word of 
> caution here. I have seen systems go from 'slow but functioning' to 
> 'completely down and requires a complete datacenter shutdown to 
> recover' because of automated response systems that decided to restart 
> something when it didn't respond fast enough, triggering a cascade of 
> failures that prevented any service from being able to start into a 
> healthy state.
>
> I've also implemented monitoring on APs to restart them if they don't 
> have a path to the Internet, resulting in continual reboots when there 
> is a transitory issue (now changed to only check their next hop and 
> only shut down wifi to avoid becoming a black hole for that SSID
>
> to err is human, to really mess things up requires a computer, and 
> automation removes the oversight from the computer allowing it to do 
> more damage faster.
>
> David Lang

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Dave Taht]

On Mon, Oct 23, 2023 at 10:44 AM le berger des photons via Nnagain
<nnagain@lists.bufferbloat.net> wrote:
>
> you've convinced me to go see libre qos.  thanks.

Thank you, but that was not my intent. I was actually trying to course
correct the growing QoE industry and their ISP customers to be
measuring and deploying the right things at the routers themselves,
and keep up to date with events. I had been patiently trying to find
folk with clue on this otherwise excellent WISP talk thread:
https://www.facebook.com/groups/wisptalk/permalink/2234396776891325/

(it is a really great group of network operators, btw) over the week..

And it was really great, as by the end of it we had established that
fq_codel was a key part of everyone´s (ubnt, bequant/cambium, preseem,
LibreQos´s "secret sauce") and I had a chance to communicate that we
had fixed a fairly large bug^H^H^H misfeature in codel in 2018 in CAKE
and did not know who to tell about it. I have a long list of vendors
that have listed fq_codel or CAKE as part of their products now, that
I reached out to some effect over the past few years, getting mikrotik
to backport some stuff in particular to their 5.7 kernel release from
5.15.

Sometimes tho I get back the blithe dismissal and I have to play my
theme song to recover.

https://www.youtube.com/watch?v=qGzUTrnqEDA

It bothers me to know that for the next 15 years, that bug will be
still shipping in billions of "new" products, leveraging old kernels,
that people will use. Poor Van Jacobson had identified many problems
with his 90s RED idea, took 16 years to find a fix, and is still
waiting for even one big vendor to make it available in silicon. And
so it goes. I am getting better at just accepting things as they are
and trying just to fix what I can.

Another song... https://www.youtube.com/watch?v=yMj-icLHiw4

>
> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <nnagain@lists.bufferbloat.net> wrote:
>>
>> I loved that this guy and his ISP burned a couple weeks learning how
>> to build openwrt, built something exactly to the need, *had it work
>> the first time* and are in progress to update in place 200+ routers to
>> better router software, that just works, with videoconferencing, IPv6
>> support, and OTA functionality. No need for a truck roll, and while
>> the available bandwidth deep in these mountains in Mexico is meager,
>> it is now enough for most purposes.
>>
>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>>
>> I have no idea how many of this model routers were sold or are still
>> deployed (?), but the modest up front cost of this sort of development
>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>> least they are in a position now to rapidly respond to CVEs and other
>> problems when they happen, having "seized control of the methods of
>> computation" again.
>>
>> OpenWrt is known to run on 1700 different models, already, (with easy
>> ports to obscure ones like this box) - going back over a decade in
>> some cases.
>>
>> Another favorite story of mine was the ISP in New Zealand that
>> deployed LibreQos and had all their support calls (from gamers and
>> videoconferencers) cease overnight. The support tech, formerly drowned
>> in angst from the users, set to work automating an reflashing 600 old
>> agw routers they had "retired" on the shelf, and then distributing
>> them to customers as extenders because the wifi finally worked right
>> with the fq_codel stuff now in that release.
>>
>> I feel like I am tooting my own horn here a bit too much, but solving
>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>> control of your software infrastructure while being able to customize
>> it for purpose, and turning what otherwise would be ewaste into
>> something that will last a decade more, is my inner "green", my inner
>> stewart brand.
>>
>> Compare that to so many others being marketed to, to death, that buy
>> the latest (and often inferior) thing, every few months, perpetually
>> fooled by promises that do not pay off in the field, and often, really
>> lousy MTBF. Good embedded software takes many years to develop, say,
>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
>> many eyeballs to fully debug and get to lots of 9s of reliability.
>>
>> Back when I was even more radical about good, open, embedded, software
>> than now, I used to say: "Friends don't let friends run factory
>> firmware.". I do wish somehow the long term maintence costs of
>> hardware with a decade plus service lifetime would be adaquately
>> covered. Insurance? by law? a formal setaside from the purchase price?
>> Otherwise we run the risk of turning the world's internet into a giant
>> toxic waste dump that will require Superfund levels of cleanup, one
>> day, and ever more contributions to trillions of dollars of fraud, and
>> persistent actors having first broken down the front door, perpetually
>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>> front, seems cheaper.
>>
>> Take this string of vulns:
>> https://www.google.com/search?q=cisco+router+vulnerability
>>
>> (try that search string with *any* manufacturer - juniper, netgear, tplink,
>>
>> There is a new vuln going around about some very old software in a
>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>> worked on this while at montavista in the early 00s!)  - abandonware,
>> toxic waste...
>>
>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
>> through the actions of all that contribute to linux, openwrt, and one
>> smart and caring engineer.
>>
>> --
>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Robert McMahon]

[-- Attachment #1: Type: text/plain, Size: 7638 bytes --]

Thanks, this is very interesting. I wrote code to DMA packets in an early Cisco switch and the hardware ASIC that did the movement across the fabric would provide a simple status of success or not. Unfortunately, the ASIC would at times indicate success and never move the packet across the fabric and then went to a state of using the wrong egress for all subsequent packets. It wasn't possible to change the ASIC as that was locked down years earlier. Luckily, we had the ability to query the ASIC to get more information on what it actually did so the code could see when it needed a fix. We did the FDIR, lost a bunch of packets, and assumed TCP would handle it. Of course, TCP designers assumed the loss was due to congestion so those state machines were incorrect but would ultimately recover.

I started my career working on a NASA FDDI network. SW had gotten so complex that all the states could not be inspected by humans like done on the Shuttle, nor even tested by conmputers. The strategy became commercial off the shelf (COTS) because, through "market magic", it was assumed fully tested.

I think the same naivety is now applied to open source code. There is no magic here either. Testing is way beyond simple scenarios repeated over and over again as the only test that matters.

Networks and distributed systems have bugs. I think a current Linux kernel is 30M lines of code and 1100 config options. Good luck in testing that.

This is beyond complex and not easy. FDIR has to be designed in from the get go.

Bob

On Oct 23, 2023, 4:22 PM, at 4:22 PM, Karl Auerbach <karl@cavebear.com> wrote:
>On 10/23/23 2:54 PM, rjmcmahon wrote:
>> Home networks today are embarrassing to me. Our industry is woefully
>> behind here.
>>
>I would be more expansive.
>
>(Bringing this back to network neutrality - my argument, not clearly
>suggested below, is that "neutrality" is more than bandwidth or
>connectivity but ought also ought to include other aspects including
>robust and repairable service in the face of reasonably foreseeable
>events.  By-the-way, when I was involved in the early days of the net,
>I
>worked for groups [such as the US Joint Chiefs] who thought that
>routers
>being vaporized by nuclear explosions were "reasonable foreseeable".)
>
>The lawyer half of me lives in fear of the harm that can come from bad
>code in network devices.  I've seen the growth of strict product 
>liability laws in the consumer space (sometimes resulting in those
>silly
>"do not eat" labels on silica gel packets, but also resulting in
>important steps, like pressure-release closures on cleaning products
>that contain dry sodium hydroxide, or dual braking systems in
>automobiles.)
>
>And the railroad nut in me remembers that Murphy's law is as strong as
>ever.  (Just ask "Why are highway and railroad traffic control signals
>red and green [actually a quite bluish green]?" [Hint, they originally
>were red and white, and sometimes the red colored lens would fall
>out.])
>
>When I was working with the DARPA Robotics Challenge my job was to
>introduce network problems - the kinds of things that can happen in
>real
>life when a robot operates in a disaster zone.  I could introduce a
>simple change - like increasing the level of lost Ethernet frames when
>a
>robot went through a door into a (simulated) concrete reactor building
>- 
>and the robot would simply stop or fall over.
>
>I've seen videos of animal surgery performed by remote control over a
>long distance (50km) network link where the doctors presumed that the
>net was endlessly flawless.  (I have this mental image of a robotic
>scalpel overshooting its cut due to a non-idempotent command contained
>in a packet that was replicated on the net.)
>
>And I've seen users of satellites fail to remember that every now and
>then, from the point of view of a ground station, a satellite may
>transit across the face of the sun (a highly predictable event) and be
>temporarily blinded and unable to receive data.
>
>Many of our implementations today are hanging on only because modern
>machines have gobs upon gobs of memory and nobody notices if a couple
>of
>gigabytes leak or are uselessly allocated for a few minutes.
>
>(For instance, one way to stop a Linux stack is to send it patterns of
>tiny IPv4 fragments that overlap or have gaps so that reassembly is not
>
>possible (or difficult) and buffers just sit there waiting for a rather
>
>long timeout before being reclaimed.)
>
>It seems that everybody and her brothers think they can write code. 
>And
>they do.  And in our open source world the code they write is often
>protocol code.  Often it is badly written protocol code containing
>monumental flaws, such as use of "integer" types in C (when "unsigned
>uint16" or similar is needed), failure to recognize that number spaces
>wrap, assumptions that "everything is in ASCII" or that character
>sequences do not contain null bytes. (Last time I looked some major
>libraries went down in flames when string data in packets happened to
>contain nulls - the code was using ancient Unix/C string routines.)  I
>once sent several SIP phones into the weeds when I sent length fields
>(in text form) with leading zero characters (e.g. 050 rather than 50) -
>
>some code treated that as octal!)
>
>It would certainly be nice if we had a body of network implementation
>design/implementation rules - similar in concept to engineering design
>rules used in bridges, aircraft, electrical networks, etc - for use
>when
>writing code.  Any one who wanted to do something outside of those
>rules
>could do so, but would be strongly "encouraged" to seek the advice and
>oversight of others.
>
>Once the Interop show net was brought to a stop (by infinitely looping
>packets) when two brands of routers had different notions how to expand
>
>IPv4 multicast addresses into MAC addresses.  (I can't remember the
>details, but when every light in the NOC turned red everybody in the
>Interop NOC turned to look at me, guessing [incorrectly in this
>instance] that I was the cause.])
>
>It would be nice if we built our network devices so that they each had
>a
>little introspective daemon that frequently asked "am I healthy, am I
>still connected, are packets still moving through me?"  (For consumer
>devices an answer of "no" could trigger a full device reboot or reset.)
>
>For larger devices, such as routers, we could have some machinery,
>internal or external, that did a bit of modelling and informed the
>routing machinery of anticipated queue lengths and similar metrics. 
>Then the router could monitor itself to check if it was wobbling
>outside
>of those anticipated ranges and take appropriate action to signal the
>issue.  (I was once quite surprised to learn on at least one large type
>
>of router that it was difficult-to-impossible to obtain queue length
>data because so much function had been pushed into hardware that had
>few
>test or measurement points.)
>
>My grandfather and father were radio and TV repair guys.  I learned
>from
>an early age the value of good tools and of looking outside the basic
>operation of a device for symptoms. (You could often hear a failing
>capacitor or inductor; or you could smell a slowly burning resistor.) 
>Our modern networks and code usually lack that kind of observational
>(and active testing) plane.
>
>I can see a big net neutrality differentiator between providers being
>"time to detect" and "time to repair".
>
>         --karl--

[-- Attachment #2: Type: text/html, Size: 8448 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Ignacio Ocampo]

[-- Attachment #1: Type: text/plain, Size: 9142 bytes --]

Hi there, a quick update since I published the post
<https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/>.
We've now deployed 50+ units (out of the 200+ goal) so far, and it's
working really well!

The fact that I now have full-control and visibility on what is going on
within the system is making a huge difference that ultimately benefits the
end user.

IPv6 works out of the box, shaping the up-link with Cake, being able to
easily configure system parameters (SSID, key, etc) with UCI, and just
seeing a clean `ps` in comparison to the stock firmware with some weird
processes.

As a next step, we're planning to deploy a TR-069 client (or why not write
a simple REST API to consume basic system params and interface with UCI to
set up things).

As David said, we're shaping up-link close to the IBR, and the up-link
close to the customer.

Having full-control on the router is a huge advantage as operator, that
again, benefits the end user.

Thanks!

On Mon, Oct 23, 2023 at 12:11 PM Dave Taht via Nnagain <
nnagain@lists.bufferbloat.net> wrote:

> On Mon, Oct 23, 2023 at 10:47 AM Frantisek Borsik via Nnagain
> <nnagain@lists.bufferbloat.net> wrote:
> >
> > Such a great thing to read! Another LibreQoS man aboard.
>
> Well, here is me trying to counter the now-too-popular idea that you
> can shape everything at the middlebox at the ISP. It is a good start
> to do that! The packet shaping *and* observability REALLY helps! But
> in the end everything always works better if you have competent fq +
> rfc7567 support at every possible bottleneck link, especially on
> variable rate wireless transports, and especially in the face of
> potentially difficult traffic like bittorrent, nat awareness and per
> host/per flow fq have to happen at the customer site....
>
> Correct backpressure at the right point, further modified by smart
> algorithms is 20x less cpu intensive than shaping. The thing that bugs
> me most is I see people trying to shape wifi to a fixed rate, which
> works if you only test one device at "the right distance",  and never
> otherwise. A lot of wifi manufactures like eero and google wifi got
> this, but so far my testing of wifi6 was so miserable I decided to
> stop working on it. I REALLY hope someone at the wifi conference this
> week can show test results with multiple stations at multiple
> distances doing the right things, as we did at gfiber here, back in
> *2014*, see page 13:
>
>
> http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
>
> Because wifi6 and 7 are otherwise a huge step backwards.
>
> > Btw, Ignacio might be here, but cc'ing him anyway.
>
> i loved so much that *starting from scratch* he was able to assemble a
> good solution out of OpenWrt. It restores my faith in the engineering
> world, to know many folk younger than me still have basic C and
> makefile skills. Thx, man!
>
> I have been doing embedded linux software since 1998, and am often a
> bit grouchy about "the kids today" being unable to compile a kernel,
> write a bit of xml for the dts stuff, and boom, have a totally custom
> machine that does exactly what is needed. I see many wifi
> manufacturers completely unable to build or modify or modify the
> ancient crap they get from the chip manufacturer, including the chip
> manufacturer!
>
> Training AIs or engaging in social media are far more popular and
> highly paid skills nowadays.
>
>
> >
> > All the best,
> >
> > Frank
> >
> > Frantisek (Frank) Borsik
> >
> >
> >
> > https://www.linkedin.com/in/frantisekborsik
> >
> > Signal, Telegram, WhatsApp: +421919416714
> >
> > iMessage, mobile: +420775230885
> >
> > Skype: casioa5302ca
> >
> > frantisek.borsik@gmail.com
> >
> >
> >
> > On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
> >>
> >> you've convinced me to go see libre qos.  thanks.
> >>
> >> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
> >>>
> >>> I loved that this guy and his ISP burned a couple weeks learning how
> >>> to build openwrt, built something exactly to the need, *had it work
> >>> the first time* and are in progress to update in place 200+ routers to
> >>> better router software, that just works, with videoconferencing, IPv6
> >>> support, and OTA functionality. No need for a truck roll, and while
> >>> the available bandwidth deep in these mountains in Mexico is meager,
> >>> it is now enough for most purposes.
> >>>
> >>>
> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
> >>>
> >>> I have no idea how many of this model routers were sold or are still
> >>> deployed (?), but the modest up front cost of this sort of development
> >>> dwarves that of deployment. Ongoing maintenance is a problem, but at
> >>> least they are in a position now to rapidly respond to CVEs and other
> >>> problems when they happen, having "seized control of the methods of
> >>> computation" again.
> >>>
> >>> OpenWrt is known to run on 1700 different models, already, (with easy
> >>> ports to obscure ones like this box) - going back over a decade in
> >>> some cases.
> >>>
> >>> Another favorite story of mine was the ISP in New Zealand that
> >>> deployed LibreQos and had all their support calls (from gamers and
> >>> videoconferencers) cease overnight. The support tech, formerly drowned
> >>> in angst from the users, set to work automating an reflashing 600 old
> >>> agw routers they had "retired" on the shelf, and then distributing
> >>> them to customers as extenders because the wifi finally worked right
> >>> with the fq_codel stuff now in that release.
> >>>
> >>> I feel like I am tooting my own horn here a bit too much, but solving
> >>> the right problems like MTTR, MTBF, bufferbloat, and taking back
> >>> control of your software infrastructure while being able to customize
> >>> it for purpose, and turning what otherwise would be ewaste into
> >>> something that will last a decade more, is my inner "green", my inner
> >>> stewart brand.
> >>>
> >>> Compare that to so many others being marketed to, to death, that buy
> >>> the latest (and often inferior) thing, every few months, perpetually
> >>> fooled by promises that do not pay off in the field, and often, really
> >>> lousy MTBF. Good embedded software takes many years to develop, say,
> >>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
> >>> many eyeballs to fully debug and get to lots of 9s of reliability.
> >>>
> >>> Back when I was even more radical about good, open, embedded, software
> >>> than now, I used to say: "Friends don't let friends run factory
> >>> firmware.". I do wish somehow the long term maintence costs of
> >>> hardware with a decade plus service lifetime would be adaquately
> >>> covered. Insurance? by law? a formal setaside from the purchase price?
> >>> Otherwise we run the risk of turning the world's internet into a giant
> >>> toxic waste dump that will require Superfund levels of cleanup, one
> >>> day, and ever more contributions to trillions of dollars of fraud, and
> >>> persistent actors having first broken down the front door, perpetually
> >>> on the inside, wreaking more havoc. Somehow preventing that mess, up
> >>> front, seems cheaper.
> >>>
> >>> Take this string of vulns:
> >>> https://www.google.com/search?q=cisco+router+vulnerability
> >>>
> >>> (try that search string with *any* manufacturer - juniper, netgear,
> tplink,
> >>>
> >>> There is a new vuln going around about some very old software in a
> >>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
> >>> worked on this while at montavista in the early 00s!)  - abandonware,
> >>> toxic waste...
> >>>
> >>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
> >>> through the actions of all that contribute to linux, openwrt, and one
> >>> smart and caring engineer.
> >>>
> >>> --
> >>> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> >>> Dave Täht CSO, LibreQos
> >>> _______________________________________________
> >>> Nnagain mailing list
> >>> Nnagain@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/nnagain
> >>
> >> _______________________________________________
> >> Nnagain mailing list
> >> Nnagain@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/nnagain
> >
> > _______________________________________________
> > Nnagain mailing list
> > Nnagain@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/nnagain
>
>
>
> --
> Oct 30:
> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
> Dave Täht CSO, LibreQos
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
>


-- 
Ignacio Ocampo

[-- Attachment #2: Type: text/html, Size: 12644 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Ignacio Ocampo]

[-- Attachment #1.1: Type: text/plain, Size: 9742 bytes --]

And, as a dogfooding, I'm personally using that router in Seattle! to make
sure the experience I'm providing to my customers in Mexico is
high-quality. I haven't had any problem either (limiting the up-link and
down-link as well).

[image: image.png]

On Mon, Oct 23, 2023 at 11:34 PM Ignacio Ocampo <nafiux@gmail.com> wrote:

> Hi there, a quick update since I published the post
> <https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/>.
> We've now deployed 50+ units (out of the 200+ goal) so far, and it's
> working really well!
>
> The fact that I now have full-control and visibility on what is going on
> within the system is making a huge difference that ultimately benefits the
> end user.
>
> IPv6 works out of the box, shaping the up-link with Cake, being able to
> easily configure system parameters (SSID, key, etc) with UCI, and just
> seeing a clean `ps` in comparison to the stock firmware with some weird
> processes.
>
> As a next step, we're planning to deploy a TR-069 client (or why not write
> a simple REST API to consume basic system params and interface with UCI to
> set up things).
>
> As David said, we're shaping up-link close to the IBR, and the up-link
> close to the customer.
>
> Having full-control on the router is a huge advantage as operator, that
> again, benefits the end user.
>
> Thanks!
>
> On Mon, Oct 23, 2023 at 12:11 PM Dave Taht via Nnagain <
> nnagain@lists.bufferbloat.net> wrote:
>
>> On Mon, Oct 23, 2023 at 10:47 AM Frantisek Borsik via Nnagain
>> <nnagain@lists.bufferbloat.net> wrote:
>> >
>> > Such a great thing to read! Another LibreQoS man aboard.
>>
>> Well, here is me trying to counter the now-too-popular idea that you
>> can shape everything at the middlebox at the ISP. It is a good start
>> to do that! The packet shaping *and* observability REALLY helps! But
>> in the end everything always works better if you have competent fq +
>> rfc7567 support at every possible bottleneck link, especially on
>> variable rate wireless transports, and especially in the face of
>> potentially difficult traffic like bittorrent, nat awareness and per
>> host/per flow fq have to happen at the customer site....
>>
>> Correct backpressure at the right point, further modified by smart
>> algorithms is 20x less cpu intensive than shaping. The thing that bugs
>> me most is I see people trying to shape wifi to a fixed rate, which
>> works if you only test one device at "the right distance",  and never
>> otherwise. A lot of wifi manufactures like eero and google wifi got
>> this, but so far my testing of wifi6 was so miserable I decided to
>> stop working on it. I REALLY hope someone at the wifi conference this
>> week can show test results with multiple stations at multiple
>> distances doing the right things, as we did at gfiber here, back in
>> *2014*, see page 13:
>>
>>
>> http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
>>
>> Because wifi6 and 7 are otherwise a huge step backwards.
>>
>> > Btw, Ignacio might be here, but cc'ing him anyway.
>>
>> i loved so much that *starting from scratch* he was able to assemble a
>> good solution out of OpenWrt. It restores my faith in the engineering
>> world, to know many folk younger than me still have basic C and
>> makefile skills. Thx, man!
>>
>> I have been doing embedded linux software since 1998, and am often a
>> bit grouchy about "the kids today" being unable to compile a kernel,
>> write a bit of xml for the dts stuff, and boom, have a totally custom
>> machine that does exactly what is needed. I see many wifi
>> manufacturers completely unable to build or modify or modify the
>> ancient crap they get from the chip manufacturer, including the chip
>> manufacturer!
>>
>> Training AIs or engaging in social media are far more popular and
>> highly paid skills nowadays.
>>
>>
>> >
>> > All the best,
>> >
>> > Frank
>> >
>> > Frantisek (Frank) Borsik
>> >
>> >
>> >
>> > https://www.linkedin.com/in/frantisekborsik
>> >
>> > Signal, Telegram, WhatsApp: +421919416714
>> >
>> > iMessage, mobile: +420775230885
>> >
>> > Skype: casioa5302ca
>> >
>> > frantisek.borsik@gmail.com
>> >
>> >
>> >
>> > On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <
>> nnagain@lists.bufferbloat.net> wrote:
>> >>
>> >> you've convinced me to go see libre qos.  thanks.
>> >>
>> >> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
>> nnagain@lists.bufferbloat.net> wrote:
>> >>>
>> >>> I loved that this guy and his ISP burned a couple weeks learning how
>> >>> to build openwrt, built something exactly to the need, *had it work
>> >>> the first time* and are in progress to update in place 200+ routers to
>> >>> better router software, that just works, with videoconferencing, IPv6
>> >>> support, and OTA functionality. No need for a truck roll, and while
>> >>> the available bandwidth deep in these mountains in Mexico is meager,
>> >>> it is now enough for most purposes.
>> >>>
>> >>>
>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>> >>>
>> >>> I have no idea how many of this model routers were sold or are still
>> >>> deployed (?), but the modest up front cost of this sort of development
>> >>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>> >>> least they are in a position now to rapidly respond to CVEs and other
>> >>> problems when they happen, having "seized control of the methods of
>> >>> computation" again.
>> >>>
>> >>> OpenWrt is known to run on 1700 different models, already, (with easy
>> >>> ports to obscure ones like this box) - going back over a decade in
>> >>> some cases.
>> >>>
>> >>> Another favorite story of mine was the ISP in New Zealand that
>> >>> deployed LibreQos and had all their support calls (from gamers and
>> >>> videoconferencers) cease overnight. The support tech, formerly drowned
>> >>> in angst from the users, set to work automating an reflashing 600 old
>> >>> agw routers they had "retired" on the shelf, and then distributing
>> >>> them to customers as extenders because the wifi finally worked right
>> >>> with the fq_codel stuff now in that release.
>> >>>
>> >>> I feel like I am tooting my own horn here a bit too much, but solving
>> >>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>> >>> control of your software infrastructure while being able to customize
>> >>> it for purpose, and turning what otherwise would be ewaste into
>> >>> something that will last a decade more, is my inner "green", my inner
>> >>> stewart brand.
>> >>>
>> >>> Compare that to so many others being marketed to, to death, that buy
>> >>> the latest (and often inferior) thing, every few months, perpetually
>> >>> fooled by promises that do not pay off in the field, and often, really
>> >>> lousy MTBF. Good embedded software takes many years to develop, say,
>> >>> oh, 7, while the hardware cycle is closer to 2, nowadays, and requires
>> >>> many eyeballs to fully debug and get to lots of 9s of reliability.
>> >>>
>> >>> Back when I was even more radical about good, open, embedded, software
>> >>> than now, I used to say: "Friends don't let friends run factory
>> >>> firmware.". I do wish somehow the long term maintence costs of
>> >>> hardware with a decade plus service lifetime would be adaquately
>> >>> covered. Insurance? by law? a formal setaside from the purchase price?
>> >>> Otherwise we run the risk of turning the world's internet into a giant
>> >>> toxic waste dump that will require Superfund levels of cleanup, one
>> >>> day, and ever more contributions to trillions of dollars of fraud, and
>> >>> persistent actors having first broken down the front door, perpetually
>> >>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>> >>> front, seems cheaper.
>> >>>
>> >>> Take this string of vulns:
>> >>> https://www.google.com/search?q=cisco+router+vulnerability
>> >>>
>> >>> (try that search string with *any* manufacturer - juniper, netgear,
>> tplink,
>> >>>
>> >>> There is a new vuln going around about some very old software in a
>> >>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>> >>> worked on this while at montavista in the early 00s!)  - abandonware,
>> >>> toxic waste...
>> >>>
>> >>> Anyway, in Mexico at least, 200+ routers are going to be a lot better,
>> >>> through the actions of all that contribute to linux, openwrt, and one
>> >>> smart and caring engineer.
>> >>>
>> >>> --
>> >>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> >>> Dave Täht CSO, LibreQos
>> >>> _______________________________________________
>> >>> Nnagain mailing list
>> >>> Nnagain@lists.bufferbloat.net
>> >>> https://lists.bufferbloat.net/listinfo/nnagain
>> >>
>> >> _______________________________________________
>> >> Nnagain mailing list
>> >> Nnagain@lists.bufferbloat.net
>> >> https://lists.bufferbloat.net/listinfo/nnagain
>> >
>> > _______________________________________________
>> > Nnagain mailing list
>> > Nnagain@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/nnagain
>>
>>
>>
>> --
>> Oct 30:
>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos
>> _______________________________________________
>> Nnagain mailing list
>> Nnagain@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/nnagain
>>
>
>
> --
> Ignacio Ocampo
>


-- 
Ignacio Ocampo

[-- Attachment #1.2: Type: text/html, Size: 13562 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 167329 bytes --]

Re: [NNagain] upgrading old routers to modern, secure FOSS

[Frantisek Borsik]

[-- Attachment #1.1: Type: text/plain, Size: 10708 bytes --]

Thanks for weighing-in, Ignacio. And great job, again!
Looking forward to hear about TR-069 implementation - hope you will write a
blog post documenting it as well.


All the best,

Frank

Frantisek (Frank) Borsik



https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Tue, Oct 24, 2023 at 7:40 AM Ignacio Ocampo via Nnagain <
nnagain@lists.bufferbloat.net> wrote:

> And, as a dogfooding, I'm personally using that router in Seattle! to make
> sure the experience I'm providing to my customers in Mexico is
> high-quality. I haven't had any problem either (limiting the up-link and
> down-link as well).
>
> [image: image.png]
>
> On Mon, Oct 23, 2023 at 11:34 PM Ignacio Ocampo <nafiux@gmail.com> wrote:
>
>> Hi there, a quick update since I published the post
>> <https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/>.
>> We've now deployed 50+ units (out of the 200+ goal) so far, and it's
>> working really well!
>>
>> The fact that I now have full-control and visibility on what is going on
>> within the system is making a huge difference that ultimately benefits the
>> end user.
>>
>> IPv6 works out of the box, shaping the up-link with Cake, being able to
>> easily configure system parameters (SSID, key, etc) with UCI, and just
>> seeing a clean `ps` in comparison to the stock firmware with some weird
>> processes.
>>
>> As a next step, we're planning to deploy a TR-069 client (or why not
>> write a simple REST API to consume basic system params and interface with
>> UCI to set up things).
>>
>> As David said, we're shaping up-link close to the IBR, and the up-link
>> close to the customer.
>>
>> Having full-control on the router is a huge advantage as operator, that
>> again, benefits the end user.
>>
>> Thanks!
>>
>> On Mon, Oct 23, 2023 at 12:11 PM Dave Taht via Nnagain <
>> nnagain@lists.bufferbloat.net> wrote:
>>
>>> On Mon, Oct 23, 2023 at 10:47 AM Frantisek Borsik via Nnagain
>>> <nnagain@lists.bufferbloat.net> wrote:
>>> >
>>> > Such a great thing to read! Another LibreQoS man aboard.
>>>
>>> Well, here is me trying to counter the now-too-popular idea that you
>>> can shape everything at the middlebox at the ISP. It is a good start
>>> to do that! The packet shaping *and* observability REALLY helps! But
>>> in the end everything always works better if you have competent fq +
>>> rfc7567 support at every possible bottleneck link, especially on
>>> variable rate wireless transports, and especially in the face of
>>> potentially difficult traffic like bittorrent, nat awareness and per
>>> host/per flow fq have to happen at the customer site....
>>>
>>> Correct backpressure at the right point, further modified by smart
>>> algorithms is 20x less cpu intensive than shaping. The thing that bugs
>>> me most is I see people trying to shape wifi to a fixed rate, which
>>> works if you only test one device at "the right distance",  and never
>>> otherwise. A lot of wifi manufactures like eero and google wifi got
>>> this, but so far my testing of wifi6 was so miserable I decided to
>>> stop working on it. I REALLY hope someone at the wifi conference this
>>> week can show test results with multiple stations at multiple
>>> distances doing the right things, as we did at gfiber here, back in
>>> *2014*, see page 13:
>>>
>>>
>>> http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf
>>>
>>> Because wifi6 and 7 are otherwise a huge step backwards.
>>>
>>> > Btw, Ignacio might be here, but cc'ing him anyway.
>>>
>>> i loved so much that *starting from scratch* he was able to assemble a
>>> good solution out of OpenWrt. It restores my faith in the engineering
>>> world, to know many folk younger than me still have basic C and
>>> makefile skills. Thx, man!
>>>
>>> I have been doing embedded linux software since 1998, and am often a
>>> bit grouchy about "the kids today" being unable to compile a kernel,
>>> write a bit of xml for the dts stuff, and boom, have a totally custom
>>> machine that does exactly what is needed. I see many wifi
>>> manufacturers completely unable to build or modify or modify the
>>> ancient crap they get from the chip manufacturer, including the chip
>>> manufacturer!
>>>
>>> Training AIs or engaging in social media are far more popular and
>>> highly paid skills nowadays.
>>>
>>>
>>> >
>>> > All the best,
>>> >
>>> > Frank
>>> >
>>> > Frantisek (Frank) Borsik
>>> >
>>> >
>>> >
>>> > https://www.linkedin.com/in/frantisekborsik
>>> >
>>> > Signal, Telegram, WhatsApp: +421919416714
>>> >
>>> > iMessage, mobile: +420775230885
>>> >
>>> > Skype: casioa5302ca
>>> >
>>> > frantisek.borsik@gmail.com
>>> >
>>> >
>>> >
>>> > On Mon, Oct 23, 2023 at 7:44 PM le berger des photons via Nnagain <
>>> nnagain@lists.bufferbloat.net> wrote:
>>> >>
>>> >> you've convinced me to go see libre qos.  thanks.
>>> >>
>>> >> On Mon, Oct 23, 2023 at 7:04 PM Dave Taht via Nnagain <
>>> nnagain@lists.bufferbloat.net> wrote:
>>> >>>
>>> >>> I loved that this guy and his ISP burned a couple weeks learning how
>>> >>> to build openwrt, built something exactly to the need, *had it work
>>> >>> the first time* and are in progress to update in place 200+ routers
>>> to
>>> >>> better router software, that just works, with videoconferencing, IPv6
>>> >>> support, and OTA functionality. No need for a truck roll, and while
>>> >>> the available bandwidth deep in these mountains in Mexico is meager,
>>> >>> it is now enough for most purposes.
>>> >>>
>>> >>>
>>> https://blog.nafiux.com/posts/cnpilot_r190w_openwrt_bufferbloat_fqcodel_cake/
>>> >>>
>>> >>> I have no idea how many of this model routers were sold or are still
>>> >>> deployed (?), but the modest up front cost of this sort of
>>> development
>>> >>> dwarves that of deployment. Ongoing maintenance is a problem, but at
>>> >>> least they are in a position now to rapidly respond to CVEs and other
>>> >>> problems when they happen, having "seized control of the methods of
>>> >>> computation" again.
>>> >>>
>>> >>> OpenWrt is known to run on 1700 different models, already, (with easy
>>> >>> ports to obscure ones like this box) - going back over a decade in
>>> >>> some cases.
>>> >>>
>>> >>> Another favorite story of mine was the ISP in New Zealand that
>>> >>> deployed LibreQos and had all their support calls (from gamers and
>>> >>> videoconferencers) cease overnight. The support tech, formerly
>>> drowned
>>> >>> in angst from the users, set to work automating an reflashing 600 old
>>> >>> agw routers they had "retired" on the shelf, and then distributing
>>> >>> them to customers as extenders because the wifi finally worked right
>>> >>> with the fq_codel stuff now in that release.
>>> >>>
>>> >>> I feel like I am tooting my own horn here a bit too much, but solving
>>> >>> the right problems like MTTR, MTBF, bufferbloat, and taking back
>>> >>> control of your software infrastructure while being able to customize
>>> >>> it for purpose, and turning what otherwise would be ewaste into
>>> >>> something that will last a decade more, is my inner "green", my inner
>>> >>> stewart brand.
>>> >>>
>>> >>> Compare that to so many others being marketed to, to death, that buy
>>> >>> the latest (and often inferior) thing, every few months, perpetually
>>> >>> fooled by promises that do not pay off in the field, and often,
>>> really
>>> >>> lousy MTBF. Good embedded software takes many years to develop, say,
>>> >>> oh, 7, while the hardware cycle is closer to 2, nowadays, and
>>> requires
>>> >>> many eyeballs to fully debug and get to lots of 9s of reliability.
>>> >>>
>>> >>> Back when I was even more radical about good, open, embedded,
>>> software
>>> >>> than now, I used to say: "Friends don't let friends run factory
>>> >>> firmware.". I do wish somehow the long term maintence costs of
>>> >>> hardware with a decade plus service lifetime would be adaquately
>>> >>> covered. Insurance? by law? a formal setaside from the purchase
>>> price?
>>> >>> Otherwise we run the risk of turning the world's internet into a
>>> giant
>>> >>> toxic waste dump that will require Superfund levels of cleanup, one
>>> >>> day, and ever more contributions to trillions of dollars of fraud,
>>> and
>>> >>> persistent actors having first broken down the front door,
>>> perpetually
>>> >>> on the inside, wreaking more havoc. Somehow preventing that mess, up
>>> >>> front, seems cheaper.
>>> >>>
>>> >>> Take this string of vulns:
>>> >>> https://www.google.com/search?q=cisco+router+vulnerability
>>> >>>
>>> >>> (try that search string with *any* manufacturer - juniper, netgear,
>>> tplink,
>>> >>>
>>> >>> There is a new vuln going around about some very old software in a
>>> >>> cisco mx series which is ancient and yet 100k+ are vulnerable -  (I
>>> >>> worked on this while at montavista in the early 00s!)  - abandonware,
>>> >>> toxic waste...
>>> >>>
>>> >>> Anyway, in Mexico at least, 200+ routers are going to be a lot
>>> better,
>>> >>> through the actions of all that contribute to linux, openwrt, and one
>>> >>> smart and caring engineer.
>>> >>>
>>> >>> --
>>> >>> Oct 30:
>>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> >>> Dave Täht CSO, LibreQos
>>> >>> _______________________________________________
>>> >>> Nnagain mailing list
>>> >>> Nnagain@lists.bufferbloat.net
>>> >>> https://lists.bufferbloat.net/listinfo/nnagain
>>> >>
>>> >> _______________________________________________
>>> >> Nnagain mailing list
>>> >> Nnagain@lists.bufferbloat.net
>>> >> https://lists.bufferbloat.net/listinfo/nnagain
>>> >
>>> > _______________________________________________
>>> > Nnagain mailing list
>>> > Nnagain@lists.bufferbloat.net
>>> > https://lists.bufferbloat.net/listinfo/nnagain
>>>
>>>
>>>
>>> --
>>> Oct 30:
>>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>> _______________________________________________
>>> Nnagain mailing list
>>> Nnagain@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/nnagain
>>>
>>
>>
>> --
>> Ignacio Ocampo
>>
>
>
> --
> Ignacio Ocampo
> _______________________________________________
> Nnagain mailing list
> Nnagain@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/nnagain
>

[-- Attachment #1.2: Type: text/html, Size: 15933 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 167329 bytes --]