[Cerowrt-devel] cerowrt-3.10.32-9 released

[Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

Get it at:

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/

I've been running this a few days now with no problems.

+ resync with openwrt head
+ upnp (when enabled) works with a yamaha receiver, torrent, and a few
other things
+ with no ipv6, 0 unaligned instruction traps
+ Latest SQM code
+ Latest dnsmasq with dnssec enabled
+ everything rolled up from the comcast releases

- untested with ipv6 as yet
- haven't tried blue-ray
- My nexus-4 still fails to get an address at 5ghz (but felix's
succeeds) so I'm going to assume
there's something wrong with my nexus-4. A newer nexus-7 works
correctly. There were a ton
of noise rejection patches from openwrt head that made it into this release...
-

It looks like you can increase the dnsmasq cache to 9999 and improve
the hit rate
on the namebench test without impacting memory much. Not that
namebench is representative.
And various test sites for dnssec return green.

In other plus's: a whole bunch of vm boxes were donated by google and
after a bit of fiddling by travis yesterday the build cluster is in
the best shape I've ever seen it.

http://buildbot.openwrt.org:8010/buildslaves

It's my hope that by speeding up build cycle time this will make
openwrt head much more stable,
and thus cerowrt more stable, and speed up the pending barrier breaker
release of openwrt by a lot.

I have kind of taken 2 weeks off from cero and have to look at my
notes for what else is
a barrier to a stable release. As best I recall my last two wishlist
items were procd support
for babeld, and bcp38 support. We have issues still with upnp. hnetd,
and ohybridproxy are entirely untested, and I am fiddling with the
auto target/interval calculation with various methods.

Other stuff?

I am firing up a new ubnt build and giving an edgerouter build a shot.
With that stuff building
again I should be able to start bringing the yurtlab back online for
more extensive automated testing.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Sun, 16 Mar 2014 12:58:28 -0700, Dave Taht said:
> Get it at:
> 
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/

> - untested with ipv6 as yet

Running it on my 3800, IPv6 from my laptop to Google and work and
other places seems to be working just fine in my corner of Comcast land.
My laptop gets a DHCPv6 address, a SLAAC address, and generates itself
a privacy address, and they all are reachable from the outside, and my
Rasberry Pi is happily SLAAC'ing away as well.  As far as I can tell,
my TV and my PS3 are IPv4-only, so that's as much as I can test.

If I catch it misbehaving, or there's something in particular you want
poked, yell...



[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

Valdis:

1) enable upnp and play some games?

2) what is the output of:

cat /sys/kernel/debug/mips/unaligned_instructions

3) It does look like miniupnpd IS correctly rejecting connection
attempts from the outside world, but I think I'd prefer to firewall it
off completely.

# logread

Sat Jan 17 03:30:04 1970 daemon.warn miniupnpd[21982]: HTTP peer
[::ffff:186.215.61.169]:41839 is not from a LAN, closing the
connection
Sat Jan 17 03:30:04 1970 daemon.warn miniupnpd[21982]: HTTP peer
[::ffff:59.90.193.38]:54571 is not from a LAN, closing the connection

And I'd love to improve the defensive sensor facility one day, to log
and forward external cracker attempts to something like a rbl blocklist.

Sat Jan 17 03:30:04 1970 daemon.crit xinetd[1416]: 1416
{process_sensor} Adding 103.26.122.210 to the global_no_access list
for 120 minutes

Still I sleep better knowing these limited protections are working.


On Sun, Mar 16, 2014 at 2:27 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Sun, 16 Mar 2014 12:58:28 -0700, Dave Taht said:
>> Get it at:
>>
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/
>
>> - untested with ipv6 as yet
>
> Running it on my 3800, IPv6 from my laptop to Google and work and
> other places seems to be working just fine in my corner of Comcast land.
> My laptop gets a DHCPv6 address, a SLAAC address, and generates itself
> a privacy address, and they all are reachable from the outside, and my
> Rasberry Pi is happily SLAAC'ing away as well.  As far as I can tell,
> my TV and my PS3 are IPv4-only, so that's as much as I can test.
>
> If I catch it misbehaving, or there's something in particular you want
> poked, yell...
>
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Dave Taht <dave.taht@gmail.com> writes:

> 2) what is the output of:
>
> cat /sys/kernel/debug/mips/unaligned_instructions

Running a build based on 3.10.32-9, I get 0 unaligned instructions after
running a couple of 10sec netperf sessions (even concurrent ones) over
IPv6 (henet tunnel).

On a different box (with an older build) I have several torrent clients
successfully opening ports through miniupnpd, with a config identical to
the one in cerofiles (I note that you can get rid of the 'download' and
'upload' values in /etc/config/upnpd -- they seem to be purely
informational (passed to clients), and likely to be wrong for cerowrt).

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

On Sun, 16 Mar 2014 14:45:45 -0700, Dave Taht said:
> Valdis:
> 
> 1) enable upnp and play some games?

It's enabled, but I don't do the online gaming thing much, so I'll
have to dig around and find something that uses it..

> 2) what is the output of:
> 
> cat /sys/kernel/debug/mips/unaligned_instructions

I've pushed at least a gigabyte of IPv6 through it, and we still
got a big whoppin' "0" there.




[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Sun, Mar 16, 2014 at 5:42 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Sun, 16 Mar 2014 14:45:45 -0700, Dave Taht said:
>> Valdis:
>>
>> 1) enable upnp and play some games?
>
> It's enabled, but I don't do the online gaming thing much, so I'll
> have to dig around and find something that uses it..

torrent does (transmission is what I try, I just also got a receiver that
uses pnp for some reason (and streams audio from various sources
quite well)

It's interesting to run torrent with the SQM system enabled and still
see if things are reasonable. I have generally found that with less
than 12 peers,
web traffic is quite fast, still. Some torrent clients let you set the
diffserv codepoint for flows to CS1 (background) which makes it pretty
invisible even for 50 peers.

>> 2) what is the output of:
>>
>> cat /sys/kernel/debug/mips/unaligned_instructions
>
> I've pushed at least a gigabyte of IPv6 through it, and we still
> got a big whoppin' "0" there.

Assuming that's through wireless and wired, then "ding dong the traps
are dead, the traps are dead"!

>
>
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Michael Richardson]

Dave Taht <dave.taht@gmail.com> wrote:
    > - haven't tried blue-ray

quai?

    > items were procd support
    > for babeld, and bcp38 support. We have issues still with upnp. hnetd,

is bcp38 support something you are expecting to see in the UI scripts or is
there some other place that you'd expect to see it?

while we need to drop packets from ULA src at the *WAN* edge, I am wondering
how we deal with a gentle renumber from the ISP... I'm thinking that we
should just leave all the prefixes that PD gave us in place for some time.
Maybe LRU on 16 prefixes...

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Mon, Mar 17, 2014 at 5:20 AM, Michael Richardson <mcr@sandelman.ca> wrote:
>
> Dave Taht <dave.taht@gmail.com> wrote:
>     > - haven't tried blue-ray
>
> quai?

At  least one blueray player we know of isn't working through the
default  dhcp/dns/upnp setup.

>     > items were procd support
>     > for babeld, and bcp38 support. We have issues still with upnp. hnetd,
>
> is bcp38 support something you are expecting to see in the UI scripts or is
> there some other place that you'd expect to see it?

firewall and UI support would be good

I've modeled  something that basically should  work in my bcp38 repo.

>
> while we need to drop packets from ULA src at the *WAN* edge, I am wondering
> how we deal with a gentle renumber from the ISP... I'm thinking that we
> should just leave all the prefixes that PD gave us in place for some time.
> Maybe LRU on 16 prefixes...

With source specific routing for ipv6 we have no need for bcp38  for ipv6.

It's needed for  ipv4.

That said, surviving an ipv6 renumber is a problem.  Many clients  probably
don't  respect an address assignment  lifetime.

>
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>
>
>
>
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

Dave Taht <dave.taht@gmail.com> writes:

> At least one blueray player we know of isn't working through the
> default dhcp/dns/upnp setup.

Why would a bluray player need upnp? *shudder*

> I've modeled  something that basically should  work in my bcp38 repo.

So, not sure exactly how it's supposed to work; does this hook into the
firewall after NAT'ing has been applied? Otherwise you'd presumably need
to add exceptions for the configured internal network(s)? (I think that
may be what is going on in the bcp script at ln 38, but some sort of
auto-detection of the relevant network(s) would be needed? Or as a
minimum a whitelist configuration option?)

Could double-nat be detected from wan iface hotplug or somesuch?

> That said, surviving an ipv6 renumber is a problem. Many clients
> probably don't respect an address assignment lifetime.

Application-transparent MPTCP from the operating system with automatic
failover? Pretty please? :)

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Toke Høiland-Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

> So, not sure exactly how it's supposed to work; does this hook into the
> firewall after NAT'ing has been applied? Otherwise you'd presumably need
> to add exceptions for the configured internal network(s)? (I think that
> may be what is going on in the bcp script at ln 38, but some sort of
> auto-detection of the relevant network(s) would be needed? Or as a
> minimum a whitelist configuration option?)

Also, is there a reason you're not putting the contents of the ipset
into the firewall configuration file? Then you'd have the GUI sorted
(assuming there's LUCI support for ipset)...

-Toke

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 489 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Mon, Mar 17, 2014 at 7:30 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Dave Taht <dave.taht@gmail.com> writes:
>
>> At least one blueray player we know of isn't working through the
>> default dhcp/dns/upnp setup.
>
> Why would a bluray player need upnp? *shudder*

It's a sony. Where products from that org are concerned, I tend to
suspect they will be reporting back to the mothership.

>> I've modeled  something that basically should  work in my bcp38 repo.
>
> So, not sure exactly how it's supposed to work; does this hook into the
> firewall after NAT'ing has been applied? Otherwise you'd presumably need
> to add exceptions for the configured internal network(s)? (I think that
> may be what is going on in the bcp script at ln 38, but some sort of
> auto-detection of the relevant network(s) would be needed? Or as a
> minimum a whitelist configuration option?)

It would hook into the wan firewall rules regardless of NAT. So there is
no need to specifically exempt internal addresses. The situation we want
to prevent is packets sourced from a NATted address exiting the wan

say your network is 172.30.42.0/24.

Someone starts pinging 172.29.42.1 from inside your network. The default
non-source-specific route will then send those packets out the wan, with
a source address of your default gw and a destination of 172.29.42.1...

where
they will wander the internet until someone drops them, which can be
quite far out. In the case of the dsl box I'm testing today, they do
get dropped at
the first hop. On cable I've seen 3-5 hops.

I didn't claim it all worked yet. The core remaining problem is detecting
a double nat situation via some dhcp hook and adding an exception for that
network and it's default netmask and default gateway.

>
> Could double-nat be detected from wan iface hotplug or somesuch?

I would hope so. But haven't found the hook yet. (and the resulting
table needs to be preserved across dhcp renews and other network
activity, which is in part why it's not setup in the firewall rules in
the testy scripts...)


>> That said, surviving an ipv6 renumber is a problem. Many clients
>> probably don't respect an address assignment lifetime.
>
> Application-transparent MPTCP from the operating system with automatic
> failover? Pretty please? :)

Linux kernel patches for that are available. They are quite invasive and I
don't know when they will make mainline linux.

http://multipath-tcp.org/pmwiki.php?n=Main.Release88

I'd like to see netperf support added to that.

>
> -Toke



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Michael Richardson]

Dave Taht <dave.taht@gmail.com> wrote:
    >> Dave Taht <dave.taht@gmail.com> wrote:
    >> > - haven't tried blue-ray
    >>
    >> quai?

    > At  least one blueray player we know of isn't working through the
    > default  dhcp/dns/upnp setup.

okay, I was imaging you plugging a blue-ray player onto the USB port...
and then... ... ? profit.

    >> > items were procd support
    >> > for babeld, and bcp38 support. We have issues still with upnp. hnetd,
    >>
    >> is bcp38 support something you are expecting to see in the UI scripts or is
    >> there some other place that you'd expect to see it?

    > firewall and UI support would be good

    > I've modeled  something that basically should  work in my bcp38 repo.

    >> while we need to drop packets from ULA src at the *WAN* edge, I am wondering
    >> how we deal with a gentle renumber from the ISP... I'm thinking that we
    >> should just leave all the prefixes that PD gave us in place for some time.
    >> Maybe LRU on 16 prefixes...

    > With source specific routing for ipv6 we have no need for bcp38  for ipv6.

well... if all of our tables are source specific, then I guess I agree.
If we have no default route for a default source, then we have implemented bcp38.

    > It's needed for  ipv4.

That's a different can of worms.
The NAT table won't survive a renumber...

    > That said, surviving an ipv6 renumber is a problem.  Many clients  probably
    > don't  respect an address assignment  lifetime.

I have experimental evidence that it's the opposite: clients give up their
address and on-linkness way too easily.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Mon, Mar 17, 2014 at 7:55 AM, Michael Richardson <mcr@sandelman.ca> wrote:

>     >> while we need to drop packets from ULA src at the *WAN* edge, I am wondering

btw, there is full ipv6 NAT support in this release. Untested.


>     > That said, surviving an ipv6 renumber is a problem.  Many clients  probably
>     > don't  respect an address assignment  lifetime.
>
> I have experimental evidence that it's the opposite: clients give up their
> address and on-linkness way too easily.

I have seen that too, particularly on wireless interfaces that go to sleep.

One thing I liked that dnsmasq did that the current odhcpv6 server
does not is force a RA on a dhcpv4 request. Perhaps hostapd could do
the same
on a reassociation in some sane way.

>
> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Török Edwin]

On 03/16/2014 09:58 PM, Dave Taht wrote:
> Get it at:
> 
> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/
> 
> I've been running this a few days now with no problems.

Can you please add these packages:
 - p910nd
 - luci-app-p910nd
 - wifitoggle

Just upgraded from 3.7.5-2, and it looks good so far.

I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test
if that is actually the correct setting?

I have this setup with my ISP:
cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP

I connect using PPPoE, and AFAIK the ISP is using GPON.
Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down.

Thanks,
--Edwin

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Sebastian Moeller]

Hi Edwin,


On Mar 18, 2014, at 11:00 , Török Edwin <edwin@etorok.net> wrote:

> On 03/16/2014 09:58 PM, Dave Taht wrote:
>> Get it at:
>> 
>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/
>> 
>> I've been running this a few days now with no problems.
> 
> Can you please add these packages:
> - p910nd
> - luci-app-p910nd
> - wifitoggle
> 
> Just upgraded from 3.7.5-2, and it looks good so far.
> 
> I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test
> if that is actually the correct setting?

	If you know that you have per packet overhead (more than the pure ethernet header that is handled with 'none') you should select "ethernet with overhead" and specify the overhead on your line (be sure to add the 14 bytes for the ethernet header as the kernel unhelpfully forgets to take this into account when you use the link layer adjustment method tc_stab)
	For ATM based systems we could use the RTT quantization effects of the ATM cells to deduce the overhead empirically but for links with out quantization that does not work, so I do not know how check which overhead to specify empirically, all you could do is look at the information you have for your link and potentially ask your ISP for more information. Just remember the goal is to supply precise information about the on-wire size of data packets so SQM can calculate the true bandwidth-cost associated with each packet. BTW if anyone in the audience knows how to measure the overhead for ethernet packets, please chime in.
	From your information below I would estimate:
	As far as I know GPON, basically is a ethernet hub solution (with one segment shared between several customers) so there is only typical ethernet overhead, plus potential framing and vlan tags, so if you select "ethernet" as link layer option, you should use the following overhead:
	PPP (2B), PPPoE (6B), ethernet (14B, reguired for tc_stab), potentially VLAN (4B?), potentially ethernet frame check sequence (???B)
	Your ISP should be able to tell you whether he uses VLAN tags on the bottle neck link (it does not matter whether the VLAN tags are actually visible/existent on your end of the GPON modem)
	So somewhere in the 22 to 30bytes range should work. Alas the only way to figure this out for good is to snoop packets on the fiber segment, so realistically you need to ask your ISP, or be happy that 22Bytes is as close to the true overhead as you can get with the information at your hand. And the closer to the actual wire size SQMs supplied bandwidths are the preciser the shaping works.
	That said it looks like each of your packets is like 8bytes larger than the kernel assumes without link layer adjustments or roughly 100*8/64 = 12.5 % for the smallest ethernet packets and 100*8/1500 = 0.5% for the largest, assuming you typically use larger packets than 64 bytes, you should not really notice whether the overhead is set correctly or not. On principle I would recommend to use "ethernet with overhead" but it should not make much of a difference. Especially since you will need to cut the shaper some slack anyways, that is even with link layer adjustments latency will be compromised unless you reduce the bandwidths specified to SQM from the line rates…

Best Regards
	Sebastian 



> 
> I have this setup with my ISP:
> cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP
> 
> I connect using PPPoE, and AFAIK the ISP is using GPON.
> Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down.
> 
> Thanks,
> --Edwin
> 
> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

Regrettably the SQM system on the wndr series of hardware maxes out on
CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g
25/25 works). If you want to apply this code at higher rates, routing
hardware with more "oomph" is needed.

I would be interested in a rrul test of your 50Mbit system. My tests
of verizon at 25/25 showed them well managed on the up, far less well
managed on the down, so in your 50Mbit design you might want to merely
control the down with SQM.

On Tue, Mar 18, 2014 at 8:12 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> Hi Edwin,
>
>
> On Mar 18, 2014, at 11:00 , Török Edwin <edwin@etorok.net> wrote:
>
>> On 03/16/2014 09:58 PM, Dave Taht wrote:
>>> Get it at:
>>>
>>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/
>>>
>>> I've been running this a few days now with no problems.
>>
>> Can you please add these packages:
>> - p910nd
>> - luci-app-p910nd
>> - wifitoggle
>>
>> Just upgraded from 3.7.5-2, and it looks good so far.
>>
>> I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test
>> if that is actually the correct setting?
>
>         If you know that you have per packet overhead (more than the pure ethernet header that is handled with 'none') you should select "ethernet with overhead" and specify the overhead on your line (be sure to add the 14 bytes for the ethernet header as the kernel unhelpfully forgets to take this into account when you use the link layer adjustment method tc_stab)
>         For ATM based systems we could use the RTT quantization effects of the ATM cells to deduce the overhead empirically but for links with out quantization that does not work, so I do not know how check which overhead to specify empirically, all you could do is look at the information you have for your link and potentially ask your ISP for more information. Just remember the goal is to supply precise information about the on-wire size of data packets so SQM can calculate the true bandwidth-cost associated with each packet. BTW if anyone in the audience knows how to measure the overhead for ethernet packets, please chime in.
>         From your information below I would estimate:
>         As far as I know GPON, basically is a ethernet hub solution (with one segment shared between several customers) so there is only typical ethernet overhead, plus potential framing and vlan tags, so if you select "ethernet" as link layer option, you should use the following overhead:
>         PPP (2B), PPPoE (6B), ethernet (14B, reguired for tc_stab), potentially VLAN (4B?), potentially ethernet frame check sequence (???B)
>         Your ISP should be able to tell you whether he uses VLAN tags on the bottle neck link (it does not matter whether the VLAN tags are actually visible/existent on your end of the GPON modem)
>         So somewhere in the 22 to 30bytes range should work. Alas the only way to figure this out for good is to snoop packets on the fiber segment, so realistically you need to ask your ISP, or be happy that 22Bytes is as close to the true overhead as you can get with the information at your hand. And the closer to the actual wire size SQMs supplied bandwidths are the preciser the shaping works.
>         That said it looks like each of your packets is like 8bytes larger than the kernel assumes without link layer adjustments or roughly 100*8/64 = 12.5 % for the smallest ethernet packets and 100*8/1500 = 0.5% for the largest, assuming you typically use larger packets than 64 bytes, you should not really notice whether the overhead is set correctly or not. On principle I would recommend to use "ethernet with overhead" but it should not make much of a difference. Especially since you will need to cut the shaper some slack anyways, that is even with link layer adjustments latency will be compromised unless you reduce the bandwidths specified to SQM from the line rates...
>
> Best Regards
>         Sebastian
>
>
>
>>
>> I have this setup with my ISP:
>> cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP
>>
>> I connect using PPPoE, and AFAIK the ISP is using GPON.
>> Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down.
>>
>> Thanks,
>> --Edwin
>>
>>
>>
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Török Edwin]

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

On 03/18/2014 04:21 PM, Dave Taht wrote:
> Regrettably the SQM system on the wndr series of hardware maxes out on
> CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g
> 25/25 works). If you want to apply this code at higher rates, routing
> hardware with more "oomph" is needed.
> 
> I would be interested in a rrul test of your 50Mbit system. My tests
> of verizon at 25/25 showed them well managed on the up, far less well
> managed on the down, so in your 50Mbit design you might want to merely
> control the down with SQM.

Host: Linux 3.14-rc5, AMD FX(tm)-8350 Eight-Core Processor, cpufreq set to performance, cpb disabled
Router: NETGEAR WNDR3700v2, 3.10.32-9
Target: OpenBSD 5.4, QEMU Virtual CPU version (cpu64-rhel6), 3300.54 MHz

Host <-> Router connected via gigabit ethernet.

I confirmed that my target can do >50Mbit to another server, netperf shows 128*10^6 bit/s / 108 * 10^6 bit/s.

SQM with link-layer none settings:
 * download speed: 46000 kbit/s, upload speed 50000 kbit/s (~95% of measured speed on ISP's speedtest site).
 * fq_codel (default)
 * simple.qos
 * link-layer: none
 * results: <sqm_on_none.png>

SQM off: <sqm_off.png>

SQM with overhead Ethernet overhead 30: <sqm-on-30.png>
SQM with overhead Ethernet overhead 22: <sqm-on-22.png>

Should I also run a rrul46/rrul46compete test?

Best regards,
--Edwin

[-- Attachment #2: sqm_on_22.png --]
[-- Type: image/png, Size: 266647 bytes --]

[-- Attachment #3: sqm_on_30.png --]
[-- Type: image/png, Size: 257373 bytes --]

[-- Attachment #4: sqm_on_none.png --]
[-- Type: image/png, Size: 253387 bytes --]

[-- Attachment #5: sqm_off.png --]
[-- Type: image/png, Size: 242595 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote:
> On 03/18/2014 04:21 PM, Dave Taht wrote:
>> Regrettably the SQM system on the wndr series of hardware maxes out on
>> CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g
>> 25/25 works). If you want to apply this code at higher rates, routing
>> hardware with more "oomph" is needed.
>>
>> I would be interested in a rrul test of your 50Mbit system. My tests
>> of verizon at 25/25 showed them well managed on the up, far less well
>> managed on the down, so in your 50Mbit design you might want to merely
>> control the down with SQM.
>
> Host: Linux 3.14-rc5, AMD FX(tm)-8350 Eight-Core Processor, cpufreq set to performance, cpb disabled
> Router: NETGEAR WNDR3700v2, 3.10.32-9
> Target: OpenBSD 5.4, QEMU Virtual CPU version (cpu64-rhel6), 3300.54 MHz
>
> Host <-> Router connected via gigabit ethernet.
>
> I confirmed that my target can do >50Mbit to another server, netperf shows 128*10^6 bit/s / 108 * 10^6 bit/s.
>
> SQM with link-layer none settings:
>  * download speed: 46000 kbit/s, upload speed 50000 kbit/s (~95% of measured speed on ISP's speedtest site).
>  * fq_codel (default)
>  * simple.qos
>  * link-layer: none
>  * results: <sqm_on_none.png>
>
> SQM off: <sqm_off.png>
>
> SQM with overhead Ethernet overhead 30: <sqm-on-30.png>
> SQM with overhead Ethernet overhead 22: <sqm-on-22.png>
>
> Should I also run a rrul46/rrul46compete test?

If you can test ipv6 too, that would be great.

> Best regards,
> --Edwin

At 8ms of induced extra latency without SQM there doesn't seem to be
much point in running it on your platform. It does look like you are tail
dropping...

A thought would be to leave it
off and try running your link at 100Mbit rather than gige.

# ethtool -s ge00 advertise 0x008

and see if fq_codel alone can break up bursts better.

I note that linux 3.14 TCP is now so highly debloated that it is really hard to
compare the results we get with it with any TCP before it - the TSO offload
fixes, tcp small queues, etc, have really shortened and made more accurate
the control loops.

second note is that the wndr can only forward packets at about 330Mbit
without firewall rules. Add in the firewall rules and you are looking at
sub 120mbit forwarding performance.

I am jealous of your link. :)


-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Török Edwin]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

On 03/18/2014 05:35 PM, Dave Taht wrote:
> On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote:
>> Should I also run a rrul46/rrul46compete test?
> 
> If you can test ipv6 too, that would be great.

Attached IPv6 results.

> 
> At 8ms of induced extra latency without SQM there doesn't seem to be
> much point in running it on your platform. It does look like you are tail
> dropping...
> 
> A thought would be to leave it
> off and try running your link at 100Mbit rather than gige.
> 
> # ethtool -s ge00 advertise 0x008

Well ge00 already runs at 100Mbit, because ISP plugged me into a 100Mbit port on the switch:
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s

They would plug me into a 1Gbit port only if I pay for 1Gbps connectivity, which is in fact only 7 EUR/month more, but its not available in my area yet :)

> 
> and see if fq_codel alone can break up bursts better.
> 
> I note that linux 3.14 TCP is now so highly debloated that it is really hard to
> compare the results we get with it with any TCP before it - the TSO offload
> fixes, tcp small queues, etc, have really shortened and made more accurate
> the control loops.
> 
> second note is that the wndr can only forward packets at about 330Mbit
> without firewall rules. Add in the firewall rules and you are looking at
> sub 120mbit forwarding performance.

Good to know, not much point in purchasing 1Gbit connectivity from my ISP unless I also replace my router.

> 
> I am jealous of your link. :)

Best regards,
--Edwin

[-- Attachment #2: rrul46_compete_sqm_off.png --]
[-- Type: image/png, Size: 262670 bytes --]

[-- Attachment #3: rrul46_compete_sqm_on.png --]
[-- Type: image/png, Size: 276876 bytes --]

[-- Attachment #4: rrul46_sqm_on.png --]
[-- Type: image/png, Size: 211120 bytes --]

[-- Attachment #5: rrul46_sqm_off.png --]
[-- Type: image/png, Size: 225381 bytes --]

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Dave Taht]

On Tue, Mar 18, 2014 at 12:49 PM, Török Edwin <edwin@etorok.net> wrote:
> On 03/18/2014 05:35 PM, Dave Taht wrote:
>> On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote:
>>> Should I also run a rrul46/rrul46compete test?
>>
>> If you can test ipv6 too, that would be great.
>
> Attached IPv6 results.

The behavior I find puzzling is how rapidly you lose your measurement
flows, basically immediately after you hit slow start on the test, packet
loss there occurs and they all die.

My guess is you actually have insufficient buffering on the gpon or onc sides,
which also explains your low latency.

>>
>> At 8ms of induced extra latency without SQM there doesn't seem to be
>> much point in running it on your platform. It does look like you are tail
>> dropping...
>>
>> A thought would be to leave it
>> off and try running your link at 100Mbit rather than gige.
>>
>> # ethtool -s ge00 advertise 0x008
>
> Well ge00 already runs at 100Mbit, because ISP plugged me into a 100Mbit port on the switch:
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Half 1000baseT/Full
>         Advertised pause frame use: No
>         Advertised auto-negotiation: Yes
>         Speed: 100Mb/s

Ah.

> They would plug me into a 1Gbit port only if I pay for 1Gbps connectivity, which is in fact only 7 EUR/month more, but its not available in my area yet :)
>
>>
>> and see if fq_codel alone can break up bursts better.
>>
>> I note that linux 3.14 TCP is now so highly debloated that it is really hard to
>> compare the results we get with it with any TCP before it - the TSO offload
>> fixes, tcp small queues, etc, have really shortened and made more accurate
>> the control loops.
>>
>> second note is that the wndr can only forward packets at about 330Mbit
>> without firewall rules. Add in the firewall rules and you are looking at
>> sub 120mbit forwarding performance.
>
> Good to know, not much point in purchasing 1Gbit connectivity from my ISP unless I also replace my router.

The only things I have found that can push that are x86 boxes mostly.

Toke is evaluating a few new ones.
>
>>
>> I am jealous of your link. :)
>
> Best regards,
> --Edwin



-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

Re: [Cerowrt-devel] cerowrt-3.10.32-9 released

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

On Tue, 18 Mar 2014 11:35:51 -0400, Dave Taht said:

> second note is that the wndr can only forward packets at about 330Mbit
> without firewall rules. Add in the firewall rules and you are looking at
> sub 120mbit forwarding performance.

It's good to know that it can handle any connection I'm likely to get
from Comcast in the reasonable future. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]