[Cerowrt-devel] how is everyone's uptime?

[Cerowrt-devel] how is everyone's uptime?

[Dave Taht]

the lab where I was keeping a box up under load had a long enough
power failure last week to keeelll my approximatively 76 days uptime.

Anybody else got any record-holders here?

I am interested in long term memory leakage (run a free),
process usage (runaways like pimd), and growth in disk space usage (df),
so output like

uptime
free
df

periodically would be nice.

-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] how is everyone's uptime?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

On Wed, 10 Dec 2014 14:52:59 -0800, Dave Taht said:
> I am interested in long term memory leakage (run a free),
> process usage (runaways like pimd), and growth in disk space usage (df),
> so output like

root@bogon-gateway-1:~# uptime
 14:31:39 up 33 days,  8:50,  load average: 0.02, 0.05, 0.05
root@bogon-gateway-1:~# free
             total         used         free       shared      buffers
Mem:        126256       100732        25524            0         6156
-/+ buffers:              94576        31680
Swap:            0            0            0
root@bogon-gateway-1:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                    7296       392      6904   5% /
/dev/root                 7680      7680         0 100% /rom
tmpfs                    63128     49364     13764  78% /tmp
/dev/mtdblock5            7296       392      6904   5% /overlay
overlayfs:/overlay        7296       392      6904   5% /
tmpfs                      512         0       512   0% /dev

Biggest issue after 33 days of uptime seems to be /tmp/log/babeld.log:

root@bogon-gateway-1:~# ls -l /tmp/log/babeld.log
-rw-r--r--    1 root     root      49416018 Dec 11 14:34 babeld.log
root@bogon-gateway-1:~# wc /tmp/log/babeld.log
   772310   6176740  49416530 /tmp/log/babeld.log
root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c

Oh kewl.  That seems to have killed it :)

Uptime is now 11 minutes, and the log is much shorter.

root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c
    107 Couldn't determine channel of interface sw00: Invalid argument.
    107 Couldn't determine channel of interface sw10: Invalid argument.

I'm trying to remember why I even want babeld running.  In any case,
the log as it stands now those two messages aren't much help.




[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] how is everyone's uptime?

[Dave Taht]

just send babeld.log to /dev/null via the setting in /etc/config/bab*

I had, on at least one release, left that logging function on. Other
folk, do check.

Do hate running out of memory for no good reason.


On Thu, Dec 11, 2014 at 11:58 AM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Wed, 10 Dec 2014 14:52:59 -0800, Dave Taht said:
>> I am interested in long term memory leakage (run a free),
>> process usage (runaways like pimd), and growth in disk space usage (df),
>> so output like
>
> root@bogon-gateway-1:~# uptime
>  14:31:39 up 33 days,  8:50,  load average: 0.02, 0.05, 0.05
> root@bogon-gateway-1:~# free
>              total         used         free       shared      buffers
> Mem:        126256       100732        25524            0         6156
> -/+ buffers:              94576        31680
> Swap:            0            0            0
> root@bogon-gateway-1:~# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> rootfs                    7296       392      6904   5% /
> /dev/root                 7680      7680         0 100% /rom
> tmpfs                    63128     49364     13764  78% /tmp
> /dev/mtdblock5            7296       392      6904   5% /overlay
> overlayfs:/overlay        7296       392      6904   5% /
> tmpfs                      512         0       512   0% /dev
>
> Biggest issue after 33 days of uptime seems to be /tmp/log/babeld.log:
>
> root@bogon-gateway-1:~# ls -l /tmp/log/babeld.log
> -rw-r--r--    1 root     root      49416018 Dec 11 14:34 babeld.log
> root@bogon-gateway-1:~# wc /tmp/log/babeld.log
>    772310   6176740  49416530 /tmp/log/babeld.log
> root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c
>
> Oh kewl.  That seems to have killed it :)
>
> Uptime is now 11 minutes, and the log is much shorter.

Heh. Sorry!


>
> root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c
>     107 Couldn't determine channel of interface sw00: Invalid argument.
>     107 Couldn't determine channel of interface sw10: Invalid argument.

This is a longstanding bug in either the kernel or babel on correctly
determining the channel on a wireless device in AP mode.

It is not an error, per se, it just means that diversity routing (which
needs to know the channel) doesn't work as well, as the channel
here gets set to "interfering" rather than the correct channel number.


>
> I'm trying to remember why I even want babeld running.  In any case,
> the log as it stands now those two messages aren't much help.
>
>
>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] how is everyone's uptime?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

On Thu, 11 Dec 2014 13:50:44 -0800, Dave Taht said:

> > root@bogon-gateway-1:~# wc /tmp/log/babeld.log
> >    772310   6176740  49416530 /tmp/log/babeld.log
> > root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c
> >
> > Oh kewl.  That seems to have killed it :)
> >
> > Uptime is now 11 minutes, and the log is much shorter.
> 
> Heh. Sorry!

Not your fault.  I don't think the design specs included "have enough
ram/swap to sort a 3/4 million line logfile" :)

More importantly, it rebooted cleanly and without hiccupping any
connections through it.

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] how is everyone's uptime?

[Dave Taht]

well, you would have crashed mysteriously in another 3-4 weeks anyway,
given the growth there. :/

One of the things that bothers me about all the folks trying to use
debian for embedded use is that they don't fix syslogging in general,
and thus a crash is merely a matter of time and flash space.

In embedded, we care a lot about slow memory leaks, and things like
logfiles growing out of control, and openwrt takes care of most of
that for you. But not all of it. You want zero memory growth,
transparent restarts of all daemons from crashes, etc....

I am also increasingly impressed by what the low memory footprint of
openwrt does for you. Even on the rangeley (with 2MB cache) it seems
tons faster and more resistant to temporary glitches to have such a
small working set.

I am thinking a lot of the work that opendaylight, etc, is wasted,
they should just be running a smaller OS....



On Thu, Dec 11, 2014 at 4:18 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Thu, 11 Dec 2014 13:50:44 -0800, Dave Taht said:
>
>> > root@bogon-gateway-1:~# wc /tmp/log/babeld.log
>> >    772310   6176740  49416530 /tmp/log/babeld.log
>> > root@bogon-gateway-1:~# sort /tmp/log/babeld.log | uniq -c
>> >
>> > Oh kewl.  That seems to have killed it :)
>> >
>> > Uptime is now 11 minutes, and the log is much shorter.
>>
>> Heh. Sorry!
>
> Not your fault.  I don't think the design specs included "have enough
> ram/swap to sort a 3/4 million line logfile" :)
>
> More importantly, it rebooted cleanly and without hiccupping any
> connections through it.



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] how is everyone's uptime?

[Aaron Wood]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

Do I win any prizes?  :)

root@cerowrt:~# uptime
 16:27:49 up 74 days,  7:14,  load average: 0.46, 0.17, 0.09

root@cerowrt:~# free             total         used         free
shared      buffers
Mem:        126256        49652        76604            0            0
-/+ buffers:              49652        76604
Swap:            0            0            0

root@cerowrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                   14720     10912      3808  74% /
/dev/root                14720     10912      3808  74% /
tmpfs                    63128      1916     61212   3% /tmp
tmpfs                      512         0       512   0% /dev

I'm not running guest or babel networks, just the 2.4 and 5GHz networks.

However, I did discover a number of core dumps from miniupnpd...

-Aaron

On Wed, Dec 10, 2014 at 2:52 PM, Dave Taht <dave.taht@gmail.com> wrote:

> the lab where I was keeping a box up under load had a long enough
> power failure last week to keeelll my approximatively 76 days uptime.
>
> Anybody else got any record-holders here?
>
> I am interested in long term memory leakage (run a free),
> process usage (runaways like pimd), and growth in disk space usage (df),
> so output like
>
> uptime
> free
> df
>
> periodically would be nice.
>
> --
> Dave Täht
>
> http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2631 bytes --]

Re: [Cerowrt-devel] how is everyone's uptime?

[Dave Taht]

On Thu, Dec 11, 2014 at 4:32 PM, Aaron Wood <woody77@gmail.com> wrote:
> Do I win any prizes?  :)
>
> root@cerowrt:~# uptime
>  16:27:49 up 74 days,  7:14,  load average: 0.46, 0.17, 0.09
>
> root@cerowrt:~# free             total         used         free
> shared      buffers
> Mem:        126256        49652        76604            0            0
> -/+ buffers:              49652        76604
> Swap:            0            0            0
>
> root@cerowrt:~# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> rootfs                   14720     10912      3808  74% /
> /dev/root                14720     10912      3808  74% /
> tmpfs                    63128      1916     61212   3% /tmp
> tmpfs                      512         0       512   0% /dev
>
> I'm not running guest or babel networks, just the 2.4 and 5GHz networks.
>
> However, I did discover a number of core dumps from miniupnpd...

Please file a bug with the core dump. I can't remember how to turn those off
(or on) right now, it is a single sysctl, and I should have turned it
off, if I didn't.

kernel traps?

dmesg weirdnesses?

logmesg?

There are certainly a ton of accumulated cruft we can fix if and when we
go back into a development cycle. I did do a chaos calmer openwrt build
recently with all the new stuff (sqm, cake), and it has been doing well.

But frankly, I would prefer for most of the chaos there to subside and to find
a new, additional platform, to be working on before resuming work,
that can do inbound shaping at up to 300mbit. And
to be more openwrt compatible in whatever we do, whatever that is.

And I remain torn between updating 3.10.50 or moving forward to 3.14
or later and making all you folk reflash.

...

I see this periodically on my 3.10.50 comcast gateway:

Thu Dec 11 16:31:28 2014 daemon.info xinetd[2082]: At least 1
DENY_TIME has expired, global_no_access list updated

Yea! blocking some probes from the outside world!

And this:

Thu Dec 11 16:31:55 2014 daemon.notice netifd: wan6 (1934): Command
failed: Unknown error
Thu Dec 11 16:32:25 2014 daemon.notice netifd: wan6 (1934): Command
failed: Unknown error

I am not happy with having a firewall reload for any reason but have
not summoned the
gumption to figure out how to do it more right.

as for this:

Thu Dec 11 14:58:56 2014 daemon.warn avahi-daemon[2144]: Received
packet from invalid interface.

It think this is also fending off an attacker. That said, mdns is what
the homewrt folk used for hnetd
(which has some big problems still in the daemons, and I have been
having a few exchanges with
the devs on that front.


>
> -Aaron
>
> On Wed, Dec 10, 2014 at 2:52 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> the lab where I was keeping a box up under load had a long enough
>> power failure last week to keeelll my approximatively 76 days uptime.
>>
>> Anybody else got any record-holders here?
>>
>> I am interested in long term memory leakage (run a free),
>> process usage (runaways like pimd), and growth in disk space usage (df),
>> so output like
>>
>> uptime
>> free
>> df
>>
>> periodically would be nice.
>>
>> --
>> Dave Täht
>>
>> http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] how is everyone's uptime?

[Joseph Swick]

On 12/10/2014 05:52 PM, Dave Taht wrote:
> the lab where I was keeping a box up under load had a long enough
> power failure last week to keeelll my approximatively 76 days uptime.
> 
> Anybody else got any record-holders here?
> 
> I am interested in long term memory leakage (run a free),
> process usage (runaways like pimd), and growth in disk space usage (df),
> so output like
> 
> uptime
> free
> df
> 
> periodically would be nice.
> 

(Hopefully this gets through) Here's mine:

root@gw:~# uptime
 01:10:07 up 117 days,  1:43,  load average: 0.26, 0.17, 0.12

root@gw:~# free
             total         used         free       shared      buffers
Mem:        126256        55744        70512            0         6240
-/+ buffers:              49504        76752
Swap:            0            0            0

root@gw:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                    7296       404      6892   6% /
/dev/root                 7680      7680         0 100% /rom
tmpfs                    63128      1272     61856   2% /tmp
/dev/mtdblock5            7296       404      6892   6% /overlay
overlayfs:/overlay        7296       404      6892   6% /
tmpfs

However, while I'm getting an IPv6 address from Comcast, I don't have
IPv6 working inside, and I see this regularly in my logs:

Fri Dec 12 01:23:53 2014 daemon.warn odhcpd[1289]: A default route is
present but there is no public prefix on sw00 thus we don't announce a
default route!
Fri Dec 12 01:23:54 2014 daemon.warn odhcpd[1289]: A default route is
present but there is no public prefix on se00 thus we don't announce a
default route!
Fri Dec 12 01:23:58 2014 daemon.warn odhcpd[1289]: A default route is
present but there is no public prefix on sw00 thus we don't announce a
default route!
Fri Dec 12 01:23:59 2014 daemon.warn odhcpd[1289]: A default route is
present but there is no public prefix on se00 thus we don't announce a
default route!
Fri Dec 12 01:24:02 2014 daemon.warn odhcpd[1289]: A default route is
present but there is no public prefix on sw00 thus we don't announce a
default route!


I had tried to send a message to the list about it a while back, but I
think it may have gone to the bit-bucket due to how I setup my email
aliases (hoping the same fate doesn't happen to this message).  At the
time I thought it may have been PLEBKAC (mine), but I think there's been
a few others who have posted similar error messages.  I just haven't had
a whole lot of time to try to investigate myself, so I've just been
continuing to live with just IPv4.

-- Joseph Swick

Re: [Cerowrt-devel] how is everyone's uptime?

[Jim Gettys]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]

root@router:~# uptime
 17:31:25 up 131 days,  4:03,  load average: 0.23, 0.08, 0.06
root@router:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                    7296       416      6880   6% /
/dev/root                 7680      7680         0 100% /rom
tmpfs                    63128      5992     57136   9% /tmp
/dev/mtdblock5            7296       416      6880   6% /overlay
overlayfs:/overlay        7296       416      6880   6% /
tmpfs                      512         0       512   0% /dev
oot@router:~# free
             total         used         free       shared      buffers
Mem:        126256        66224        60032            0         6572
-/+ buffers:              59652        66604
Swap:            0            0            0




On Wed, Dec 10, 2014 at 5:52 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> the lab where I was keeping a box up under load had a long enough
> power failure last week to keeelll my approximatively 76 days uptime.
>
> Anybody else got any record-holders here?
>
> I am interested in long term memory leakage (run a free),
> process usage (runaways like pimd), and growth in disk space usage (df),
> so output like
>
> uptime
> free
> df
>
> periodically would be nice.
>
> --
> Dave Täht
>
> http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 2973 bytes --]

[Cerowrt-devel] an option for a new platform?

[Erkki Lintunen]

Hello,

while enjoying and reading another thread from the list...

> -------- Forwarded Message --------
> Subject: Re: [Cerowrt-devel] how is everyone's uptime?
> Date: Thu, 11 Dec 2014 16:42:37 -0800
> From: Dave Taht <dave.taht@gmail.com>
[snip]
> But frankly, I would prefer for most of the chaos there to subside and to find
> a new, additional platform, to be working on before resuming work,
> that can do inbound shaping at up to 300mbit. And
> to be more openwrt compatible in whatever we do, whatever that is.

this reminded me that another day I passed a web-page of a platform and
in the hope this has not been on the list yet passing it forward.

<http://www.pcengines.ch/apu.htm>

An interesting tidbit in the platform is the choice of firmware, I
think. Haven't seen any board yet with the similar choice by the
manufacturer. With a quick summing from the vendor part catalog, the
platform is sub 200 EUR (238 USD in current exchange rate) for an about
working assembly of 3x 1GbE, 4G ram, 1G flash, 802.11a/b/g/n radio...

I can't say anything how capable the hw might be for the stated inbound
shaping performance. I have had an ALIX board from their previous
generation for years and its been humming nicely though I haven't pushed
it to its envelope.

Best
Erkki

Re: [Cerowrt-devel] an option for a new platform?

[Dave Taht]

There was a review of that hardware that showed it couldn't push more
than 600Mbit natively (without shaping). I felt that the ethernet
driver could be improved significantly after looking it over, but
didn't care for the 600mbit as a starting point to have to improve
from.

Not ruling it out, though! It met quite a few requirements we have.

On Thu, Dec 11, 2014 at 11:33 PM, Erkki Lintunen <ebirdie@iki.fi> wrote:
>
> Hello,
>
> while enjoying and reading another thread from the list...
>
>> -------- Forwarded Message --------
>> Subject: Re: [Cerowrt-devel] how is everyone's uptime?
>> Date: Thu, 11 Dec 2014 16:42:37 -0800
>> From: Dave Taht <dave.taht@gmail.com>
> [snip]
>> But frankly, I would prefer for most of the chaos there to subside and to find
>> a new, additional platform, to be working on before resuming work,
>> that can do inbound shaping at up to 300mbit. And
>> to be more openwrt compatible in whatever we do, whatever that is.
>
> this reminded me that another day I passed a web-page of a platform and
> in the hope this has not been on the list yet passing it forward.
>
> <http://www.pcengines.ch/apu.htm>
>
> An interesting tidbit in the platform is the choice of firmware, I
> think. Haven't seen any board yet with the similar choice by the
> manufacturer. With a quick summing from the vendor part catalog, the
> platform is sub 200 EUR (238 USD in current exchange rate) for an about
> working assembly of 3x 1GbE, 4G ram, 1G flash, 802.11a/b/g/n radio...
>
> I can't say anything how capable the hw might be for the stated inbound
> shaping performance. I have had an ALIX board from their previous
> generation for years and its been humming nicely though I haven't pushed
> it to its envelope.
>
> Best
> Erkki
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: [Cerowrt-devel] an option for a new platform?

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 3400 bytes --]

Anyone measured what is the actual bottleneck in 300 mb/s shaping?  On an Intel platform you can measure a running piece of code pretty accurately. I ask because it is not obvious a cpu needs to touch much of a frame to do shaping, so it seems more likely that the driver and memory management structures are the bottleneck.

But it is really easy to write very slow code in a machine with limited cache. So maybe that is it.

On a multi core intel arch machine these days it is a surprising fact that a single core can't use up more than about 25 percent of  a socket's memory cycles so to get full i/o speed you need to be running your code on 4 cores or more... this kind of thing can really limit achievable speed of a poorly threaded design.  Architectural optimization needs more than llvm and clean code. You need to think about the whole software pipeline. Debian may not be great out of the box for this reason - it was never designed for routing throughput.

On Dec 12, 2014, Dave Taht <dave.taht@gmail.com> wrote:
>There was a review of that hardware that showed it couldn't push more
>than 600Mbit natively (without shaping). I felt that the ethernet
>driver could be improved significantly after looking it over, but
>didn't care for the 600mbit as a starting point to have to improve
>from.
>
>Not ruling it out, though! It met quite a few requirements we have.
>
>On Thu, Dec 11, 2014 at 11:33 PM, Erkki Lintunen <ebirdie@iki.fi>
>wrote:
>>
>> Hello,
>>
>> while enjoying and reading another thread from the list...
>>
>>> -------- Forwarded Message --------
>>> Subject: Re: [Cerowrt-devel] how is everyone's uptime?
>>> Date: Thu, 11 Dec 2014 16:42:37 -0800
>>> From: Dave Taht <dave.taht@gmail.com>
>> [snip]
>>> But frankly, I would prefer for most of the chaos there to subside
>and to find
>>> a new, additional platform, to be working on before resuming work,
>>> that can do inbound shaping at up to 300mbit. And
>>> to be more openwrt compatible in whatever we do, whatever that is.
>>
>> this reminded me that another day I passed a web-page of a platform
>and
>> in the hope this has not been on the list yet passing it forward.
>>
>> <http://www.pcengines.ch/apu.htm>
>>
>> An interesting tidbit in the platform is the choice of firmware, I
>> think. Haven't seen any board yet with the similar choice by the
>> manufacturer. With a quick summing from the vendor part catalog, the
>> platform is sub 200 EUR (238 USD in current exchange rate) for an
>about
>> working assembly of 3x 1GbE, 4G ram, 1G flash, 802.11a/b/g/n radio...
>>
>> I can't say anything how capable the hw might be for the stated
>inbound
>> shaping performance. I have had an ALIX board from their previous
>> generation for years and its been humming nicely though I haven't
>pushed
>> it to its envelope.
>>
>> Best
>> Erkki
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
>
>--
>Dave Täht
>
>thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
>_______________________________________________
>Cerowrt-devel mailing list
>Cerowrt-devel@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- Sent from my Android device with K-@ Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 4691 bytes --]

Re: [Cerowrt-devel] an option for a new platform?

[Dave Taht]

On Sat, Dec 13, 2014 at 3:02 PM, David P. Reed <dpreed@reed.com> wrote:
> Anyone measured what is the actual bottleneck in 300 mb/s shaping?  On an
> Intel platform you can measure a running piece of code pretty accurately.

Regrettably the MIPs profiler has been broken for as long as I can remember.
I just tried compiling it in again and got it failing on an illegal
op... not unfixable,
but after that you have only short intervals to poke at stuff before running
out of available memory on anything.


>I
> ask because it is not obvious a cpu needs to touch much of a frame to do
> shaping, so it seems more likely that the driver and memory management
> structures are the bottleneck.

I pointed to a specific instruction pattern that was a problem, but I do
think it is a more general problem with caching and context switching.

A recent benchmark of a dual core 1+ghz atom hit 300mbits of shaping
before bottlenecking in softirqd.


So even a low end atom is better
than the 24k mips, and most of the other work points to just plain
running out of 32k cache as the key limitation.

The right place to do the work of inbound and outbound shaping
is in the switch, I think, but that needs a fq_codel enabled switch
in the first place...

... and despite waiting patiently for that to show, no sign of life
from a single silicon vendor yet. I have all the pieces to build
a pretty good fq'd and ecn'd switch now - DRR from netfpga, trimode ethernet
macs from a variety of vendors, timer code from elsewhere, scheduling
from the scenic project...

and someone just built the very simple zynq FPGA interface to
4 ethernet macs with all the needed IP:

http://www.fpgadeveloper.com/2014/11/ethernet-fmc-is-now-available.html

except that I'd rewrite the ethernet device entirely to bolt what is
really needed, in. And I gave away my two zedboards to worthy
developers and have to get another.

>
> But it is really easy to write very slow code in a machine with limited
> cache. So maybe that is it.

Context switches suck on modern arches, far more than I realized
until recently. The software routers are spinning, polling, in userspace.

Sucks on power, works on latency.

>
> On a multi core intel arch machine these days it is a surprising fact that a
> single core can't use up more than about 25 percent of  a socket's memory
> cycles so to get full i/o speed you need to be running your code on 4 cores
> or more... this kind of thing can really limit achievable speed of a poorly
> threaded design.  Architectural optimization needs more than llvm and clean
> code. You need to think about the whole software pipeline. Debian may not be
> great out of the box for this reason - it was never designed for routing
> throughput.

The work on software routing (click, vandervecken, dpdk, opendaylight)
appears to point towards doing it all in userspace as the way to go on
most current architectures.

Regrettably the userspace stacks are not very open, or complete, at present.

>
> On Dec 12, 2014, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> There was a review of that hardware that showed it couldn't push more
>> than 600Mbit natively (without shaping). I felt that the ethernet
>> driver could be improved significantly after looking it over, but
>> didn't care for the 600mbit as a starting point to have to improve
>> from.
>>
>> Not ruling it out, though! It met quite a few requirements we have.
>>
>> On Thu, Dec 11, 2014 at 11:33 PM, Erkki Lintunen <ebirdie@iki.fi> wrote:
>>
>>> Hello,
>>>
>>> while enjoying and reading another thread from the list...
>>>
>>>> -------- Forwarded Message --------
>>>> Subject: Re: [Cerowrt-devel] how is everyone's uptime?
>>>> Date: Thu, 11 Dec 2014 16:42:37 -0800
>>>> From: Dave Taht <dave.taht@gmail.com>
>>>
>>> [snip]
>>>>
>>>> But frankly, I would prefer for most of the chaos there to subside and
>>>> to find
>>>> a new, additional platform, to be working on before resuming work,
>>>> that can do inbound shaping at up to 300mbit. And
>>>> to be more openwrt compatible in whatever we do, whatever that is.
>>>
>>>
>>> this reminded me that another day I passed a web-page of a platform and
>>> in the hope this has not been on the list yet passing it forward.
>>>
>>> <http://www.pcengines.ch/apu.htm>
>>>
>>> An interesting tidbit in the platform is the choice of firmware, I
>>> think. Haven't seen any board yet with the similar choice by the
>>> manufacturer. With a quick summing from the vendor part catalog, the
>>> platform is sub 200 EUR (238 USD in current exchange rate) for an about
>>> working assembly of 3x 1GbE, 4G ram, 1G flash, 802.11a/b/g/n radio...
>>>
>>> I can't say anything how capable the hw might be for the stated inbound
>>> shaping performance. I have had an ALIX board from their previous
>>> generation for years and its been humming nicely though I haven't pushed
>>> it to its envelope.
>>>
>>> Best
>>> Erkki
>>> ________________________________
>>>
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
>>
>>
>
> -- Sent from my Android device with K-@ Mail. Please excuse my brevity.



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks