* [Cerowrt-devel] cerowrt-3.10.32-9 released @ 2014-03-16 19:58 Dave Taht 2014-03-16 21:27 ` Valdis.Kletnieks ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Dave Taht @ 2014-03-16 19:58 UTC (permalink / raw) To: cerowrt-devel Get it at: http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ I've been running this a few days now with no problems. + resync with openwrt head + upnp (when enabled) works with a yamaha receiver, torrent, and a few other things + with no ipv6, 0 unaligned instruction traps + Latest SQM code + Latest dnsmasq with dnssec enabled + everything rolled up from the comcast releases - untested with ipv6 as yet - haven't tried blue-ray - My nexus-4 still fails to get an address at 5ghz (but felix's succeeds) so I'm going to assume there's something wrong with my nexus-4. A newer nexus-7 works correctly. There were a ton of noise rejection patches from openwrt head that made it into this release... - It looks like you can increase the dnsmasq cache to 9999 and improve the hit rate on the namebench test without impacting memory much. Not that namebench is representative. And various test sites for dnssec return green. In other plus's: a whole bunch of vm boxes were donated by google and after a bit of fiddling by travis yesterday the build cluster is in the best shape I've ever seen it. http://buildbot.openwrt.org:8010/buildslaves It's my hope that by speeding up build cycle time this will make openwrt head much more stable, and thus cerowrt more stable, and speed up the pending barrier breaker release of openwrt by a lot. I have kind of taken 2 weeks off from cero and have to look at my notes for what else is a barrier to a stable release. As best I recall my last two wishlist items were procd support for babeld, and bcp38 support. We have issues still with upnp. hnetd, and ohybridproxy are entirely untested, and I am fiddling with the auto target/interval calculation with various methods. Other stuff? I am firing up a new ubnt build and giving an edgerouter build a shot. With that stuff building again I should be able to start bringing the yurtlab back online for more extensive automated testing. -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 19:58 [Cerowrt-devel] cerowrt-3.10.32-9 released Dave Taht @ 2014-03-16 21:27 ` Valdis.Kletnieks 2014-03-16 21:45 ` Dave Taht 2014-03-17 12:20 ` Michael Richardson 2014-03-18 10:00 ` Török Edwin 2 siblings, 1 reply; 21+ messages in thread From: Valdis.Kletnieks @ 2014-03-16 21:27 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 690 bytes --] On Sun, 16 Mar 2014 12:58:28 -0700, Dave Taht said: > Get it at: > > http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ > - untested with ipv6 as yet Running it on my 3800, IPv6 from my laptop to Google and work and other places seems to be working just fine in my corner of Comcast land. My laptop gets a DHCPv6 address, a SLAAC address, and generates itself a privacy address, and they all are reachable from the outside, and my Rasberry Pi is happily SLAAC'ing away as well. As far as I can tell, my TV and my PS3 are IPv4-only, so that's as much as I can test. If I catch it misbehaving, or there's something in particular you want poked, yell... [-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 21:27 ` Valdis.Kletnieks @ 2014-03-16 21:45 ` Dave Taht 2014-03-17 0:03 ` Toke Høiland-Jørgensen 2014-03-17 0:42 ` Valdis.Kletnieks 0 siblings, 2 replies; 21+ messages in thread From: Dave Taht @ 2014-03-16 21:45 UTC (permalink / raw) To: Valdis Kletnieks; +Cc: cerowrt-devel Valdis: 1) enable upnp and play some games? 2) what is the output of: cat /sys/kernel/debug/mips/unaligned_instructions 3) It does look like miniupnpd IS correctly rejecting connection attempts from the outside world, but I think I'd prefer to firewall it off completely. # logread Sat Jan 17 03:30:04 1970 daemon.warn miniupnpd[21982]: HTTP peer [::ffff:186.215.61.169]:41839 is not from a LAN, closing the connection Sat Jan 17 03:30:04 1970 daemon.warn miniupnpd[21982]: HTTP peer [::ffff:59.90.193.38]:54571 is not from a LAN, closing the connection And I'd love to improve the defensive sensor facility one day, to log and forward external cracker attempts to something like a rbl blocklist. Sat Jan 17 03:30:04 1970 daemon.crit xinetd[1416]: 1416 {process_sensor} Adding 103.26.122.210 to the global_no_access list for 120 minutes Still I sleep better knowing these limited protections are working. On Sun, Mar 16, 2014 at 2:27 PM, <Valdis.Kletnieks@vt.edu> wrote: > On Sun, 16 Mar 2014 12:58:28 -0700, Dave Taht said: >> Get it at: >> >> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ > >> - untested with ipv6 as yet > > Running it on my 3800, IPv6 from my laptop to Google and work and > other places seems to be working just fine in my corner of Comcast land. > My laptop gets a DHCPv6 address, a SLAAC address, and generates itself > a privacy address, and they all are reachable from the outside, and my > Rasberry Pi is happily SLAAC'ing away as well. As far as I can tell, > my TV and my PS3 are IPv4-only, so that's as much as I can test. > > If I catch it misbehaving, or there's something in particular you want > poked, yell... > > -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 21:45 ` Dave Taht @ 2014-03-17 0:03 ` Toke Høiland-Jørgensen 2014-03-17 0:42 ` Valdis.Kletnieks 1 sibling, 0 replies; 21+ messages in thread From: Toke Høiland-Jørgensen @ 2014-03-17 0:03 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 649 bytes --] Dave Taht <dave.taht@gmail.com> writes: > 2) what is the output of: > > cat /sys/kernel/debug/mips/unaligned_instructions Running a build based on 3.10.32-9, I get 0 unaligned instructions after running a couple of 10sec netperf sessions (even concurrent ones) over IPv6 (henet tunnel). On a different box (with an older build) I have several torrent clients successfully opening ports through miniupnpd, with a config identical to the one in cerofiles (I note that you can get rid of the 'download' and 'upload' values in /etc/config/upnpd -- they seem to be purely informational (passed to clients), and likely to be wrong for cerowrt). -Toke [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 21:45 ` Dave Taht 2014-03-17 0:03 ` Toke Høiland-Jørgensen @ 2014-03-17 0:42 ` Valdis.Kletnieks 2014-03-17 0:54 ` Dave Taht 1 sibling, 1 reply; 21+ messages in thread From: Valdis.Kletnieks @ 2014-03-17 0:42 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 424 bytes --] On Sun, 16 Mar 2014 14:45:45 -0700, Dave Taht said: > Valdis: > > 1) enable upnp and play some games? It's enabled, but I don't do the online gaming thing much, so I'll have to dig around and find something that uses it.. > 2) what is the output of: > > cat /sys/kernel/debug/mips/unaligned_instructions I've pushed at least a gigabyte of IPv6 through it, and we still got a big whoppin' "0" there. [-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 0:42 ` Valdis.Kletnieks @ 2014-03-17 0:54 ` Dave Taht 0 siblings, 0 replies; 21+ messages in thread From: Dave Taht @ 2014-03-17 0:54 UTC (permalink / raw) To: Valdis Kletnieks; +Cc: cerowrt-devel On Sun, Mar 16, 2014 at 5:42 PM, <Valdis.Kletnieks@vt.edu> wrote: > On Sun, 16 Mar 2014 14:45:45 -0700, Dave Taht said: >> Valdis: >> >> 1) enable upnp and play some games? > > It's enabled, but I don't do the online gaming thing much, so I'll > have to dig around and find something that uses it.. torrent does (transmission is what I try, I just also got a receiver that uses pnp for some reason (and streams audio from various sources quite well) It's interesting to run torrent with the SQM system enabled and still see if things are reasonable. I have generally found that with less than 12 peers, web traffic is quite fast, still. Some torrent clients let you set the diffserv codepoint for flows to CS1 (background) which makes it pretty invisible even for 50 peers. >> 2) what is the output of: >> >> cat /sys/kernel/debug/mips/unaligned_instructions > > I've pushed at least a gigabyte of IPv6 through it, and we still > got a big whoppin' "0" there. Assuming that's through wireless and wired, then "ding dong the traps are dead, the traps are dead"! > > > -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 19:58 [Cerowrt-devel] cerowrt-3.10.32-9 released Dave Taht 2014-03-16 21:27 ` Valdis.Kletnieks @ 2014-03-17 12:20 ` Michael Richardson 2014-03-17 13:40 ` Dave Taht 2014-03-18 10:00 ` Török Edwin 2 siblings, 1 reply; 21+ messages in thread From: Michael Richardson @ 2014-03-17 12:20 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel Dave Taht <dave.taht@gmail.com> wrote: > - haven't tried blue-ray quai? > items were procd support > for babeld, and bcp38 support. We have issues still with upnp. hnetd, is bcp38 support something you are expecting to see in the UI scripts or is there some other place that you'd expect to see it? while we need to drop packets from ULA src at the *WAN* edge, I am wondering how we deal with a gentle renumber from the ISP... I'm thinking that we should just leave all the prefixes that PD gave us in place for some time. Maybe LRU on 16 prefixes... -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | network architect [ ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 12:20 ` Michael Richardson @ 2014-03-17 13:40 ` Dave Taht 2014-03-17 14:30 ` Toke Høiland-Jørgensen 2014-03-17 14:55 ` Michael Richardson 0 siblings, 2 replies; 21+ messages in thread From: Dave Taht @ 2014-03-17 13:40 UTC (permalink / raw) To: Michael Richardson; +Cc: cerowrt-devel On Mon, Mar 17, 2014 at 5:20 AM, Michael Richardson <mcr@sandelman.ca> wrote: > > Dave Taht <dave.taht@gmail.com> wrote: > > - haven't tried blue-ray > > quai? At least one blueray player we know of isn't working through the default dhcp/dns/upnp setup. > > items were procd support > > for babeld, and bcp38 support. We have issues still with upnp. hnetd, > > is bcp38 support something you are expecting to see in the UI scripts or is > there some other place that you'd expect to see it? firewall and UI support would be good I've modeled something that basically should work in my bcp38 repo. > > while we need to drop packets from ULA src at the *WAN* edge, I am wondering > how we deal with a gentle renumber from the ISP... I'm thinking that we > should just leave all the prefixes that PD gave us in place for some time. > Maybe LRU on 16 prefixes... With source specific routing for ipv6 we have no need for bcp38 for ipv6. It's needed for ipv4. That said, surviving an ipv6 renumber is a problem. Many clients probably don't respect an address assignment lifetime. > > -- > ] Never tell me the odds! | ipv6 mesh networks [ > ] Michael Richardson, Sandelman Software Works | network architect [ > ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [ > > > > > -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 13:40 ` Dave Taht @ 2014-03-17 14:30 ` Toke Høiland-Jørgensen 2014-03-17 14:39 ` Toke Høiland-Jørgensen 2014-03-17 14:55 ` Dave Taht 2014-03-17 14:55 ` Michael Richardson 1 sibling, 2 replies; 21+ messages in thread From: Toke Høiland-Jørgensen @ 2014-03-17 14:30 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 955 bytes --] Dave Taht <dave.taht@gmail.com> writes: > At least one blueray player we know of isn't working through the > default dhcp/dns/upnp setup. Why would a bluray player need upnp? *shudder* > I've modeled something that basically should work in my bcp38 repo. So, not sure exactly how it's supposed to work; does this hook into the firewall after NAT'ing has been applied? Otherwise you'd presumably need to add exceptions for the configured internal network(s)? (I think that may be what is going on in the bcp script at ln 38, but some sort of auto-detection of the relevant network(s) would be needed? Or as a minimum a whitelist configuration option?) Could double-nat be detected from wan iface hotplug or somesuch? > That said, surviving an ipv6 renumber is a problem. Many clients > probably don't respect an address assignment lifetime. Application-transparent MPTCP from the operating system with automatic failover? Pretty please? :) -Toke [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 14:30 ` Toke Høiland-Jørgensen @ 2014-03-17 14:39 ` Toke Høiland-Jørgensen 2014-03-17 14:55 ` Dave Taht 1 sibling, 0 replies; 21+ messages in thread From: Toke Høiland-Jørgensen @ 2014-03-17 14:39 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 662 bytes --] Toke Høiland-Jørgensen <toke@toke.dk> writes: > So, not sure exactly how it's supposed to work; does this hook into the > firewall after NAT'ing has been applied? Otherwise you'd presumably need > to add exceptions for the configured internal network(s)? (I think that > may be what is going on in the bcp script at ln 38, but some sort of > auto-detection of the relevant network(s) would be needed? Or as a > minimum a whitelist configuration option?) Also, is there a reason you're not putting the contents of the ipset into the firewall configuration file? Then you'd have the GUI sorted (assuming there's LUCI support for ipset)... -Toke [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 489 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 14:30 ` Toke Høiland-Jørgensen 2014-03-17 14:39 ` Toke Høiland-Jørgensen @ 2014-03-17 14:55 ` Dave Taht 1 sibling, 0 replies; 21+ messages in thread From: Dave Taht @ 2014-03-17 14:55 UTC (permalink / raw) To: Toke Høiland-Jørgensen; +Cc: cerowrt-devel On Mon, Mar 17, 2014 at 7:30 AM, Toke Høiland-Jørgensen <toke@toke.dk> wrote: > Dave Taht <dave.taht@gmail.com> writes: > >> At least one blueray player we know of isn't working through the >> default dhcp/dns/upnp setup. > > Why would a bluray player need upnp? *shudder* It's a sony. Where products from that org are concerned, I tend to suspect they will be reporting back to the mothership. >> I've modeled something that basically should work in my bcp38 repo. > > So, not sure exactly how it's supposed to work; does this hook into the > firewall after NAT'ing has been applied? Otherwise you'd presumably need > to add exceptions for the configured internal network(s)? (I think that > may be what is going on in the bcp script at ln 38, but some sort of > auto-detection of the relevant network(s) would be needed? Or as a > minimum a whitelist configuration option?) It would hook into the wan firewall rules regardless of NAT. So there is no need to specifically exempt internal addresses. The situation we want to prevent is packets sourced from a NATted address exiting the wan say your network is 172.30.42.0/24. Someone starts pinging 172.29.42.1 from inside your network. The default non-source-specific route will then send those packets out the wan, with a source address of your default gw and a destination of 172.29.42.1... where they will wander the internet until someone drops them, which can be quite far out. In the case of the dsl box I'm testing today, they do get dropped at the first hop. On cable I've seen 3-5 hops. I didn't claim it all worked yet. The core remaining problem is detecting a double nat situation via some dhcp hook and adding an exception for that network and it's default netmask and default gateway. > > Could double-nat be detected from wan iface hotplug or somesuch? I would hope so. But haven't found the hook yet. (and the resulting table needs to be preserved across dhcp renews and other network activity, which is in part why it's not setup in the firewall rules in the testy scripts...) >> That said, surviving an ipv6 renumber is a problem. Many clients >> probably don't respect an address assignment lifetime. > > Application-transparent MPTCP from the operating system with automatic > failover? Pretty please? :) Linux kernel patches for that are available. They are quite invasive and I don't know when they will make mainline linux. http://multipath-tcp.org/pmwiki.php?n=Main.Release88 I'd like to see netperf support added to that. > > -Toke -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 13:40 ` Dave Taht 2014-03-17 14:30 ` Toke Høiland-Jørgensen @ 2014-03-17 14:55 ` Michael Richardson 2014-03-17 15:18 ` Dave Taht 1 sibling, 1 reply; 21+ messages in thread From: Michael Richardson @ 2014-03-17 14:55 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel Dave Taht <dave.taht@gmail.com> wrote: >> Dave Taht <dave.taht@gmail.com> wrote: >> > - haven't tried blue-ray >> >> quai? > At least one blueray player we know of isn't working through the > default dhcp/dns/upnp setup. okay, I was imaging you plugging a blue-ray player onto the USB port... and then... ... ? profit. >> > items were procd support >> > for babeld, and bcp38 support. We have issues still with upnp. hnetd, >> >> is bcp38 support something you are expecting to see in the UI scripts or is >> there some other place that you'd expect to see it? > firewall and UI support would be good > I've modeled something that basically should work in my bcp38 repo. >> while we need to drop packets from ULA src at the *WAN* edge, I am wondering >> how we deal with a gentle renumber from the ISP... I'm thinking that we >> should just leave all the prefixes that PD gave us in place for some time. >> Maybe LRU on 16 prefixes... > With source specific routing for ipv6 we have no need for bcp38 for ipv6. well... if all of our tables are source specific, then I guess I agree. If we have no default route for a default source, then we have implemented bcp38. > It's needed for ipv4. That's a different can of worms. The NAT table won't survive a renumber... > That said, surviving an ipv6 renumber is a problem. Many clients probably > don't respect an address assignment lifetime. I have experimental evidence that it's the opposite: clients give up their address and on-linkness way too easily. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | network architect [ ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [ ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-17 14:55 ` Michael Richardson @ 2014-03-17 15:18 ` Dave Taht 0 siblings, 0 replies; 21+ messages in thread From: Dave Taht @ 2014-03-17 15:18 UTC (permalink / raw) To: Michael Richardson; +Cc: cerowrt-devel On Mon, Mar 17, 2014 at 7:55 AM, Michael Richardson <mcr@sandelman.ca> wrote: > >> while we need to drop packets from ULA src at the *WAN* edge, I am wondering btw, there is full ipv6 NAT support in this release. Untested. > > That said, surviving an ipv6 renumber is a problem. Many clients probably > > don't respect an address assignment lifetime. > > I have experimental evidence that it's the opposite: clients give up their > address and on-linkness way too easily. I have seen that too, particularly on wireless interfaces that go to sleep. One thing I liked that dnsmasq did that the current odhcpv6 server does not is force a RA on a dhcpv4 request. Perhaps hostapd could do the same on a reassociation in some sane way. > > -- > ] Never tell me the odds! | ipv6 mesh networks [ > ] Michael Richardson, Sandelman Software Works | network architect [ > ] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [ > -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-16 19:58 [Cerowrt-devel] cerowrt-3.10.32-9 released Dave Taht 2014-03-16 21:27 ` Valdis.Kletnieks 2014-03-17 12:20 ` Michael Richardson @ 2014-03-18 10:00 ` Török Edwin 2014-03-18 12:12 ` Sebastian Moeller 2 siblings, 1 reply; 21+ messages in thread From: Török Edwin @ 2014-03-18 10:00 UTC (permalink / raw) To: cerowrt-devel On 03/16/2014 09:58 PM, Dave Taht wrote: > Get it at: > > http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ > > I've been running this a few days now with no problems. Can you please add these packages: - p910nd - luci-app-p910nd - wifitoggle Just upgraded from 3.7.5-2, and it looks good so far. I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test if that is actually the correct setting? I have this setup with my ISP: cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP I connect using PPPoE, and AFAIK the ISP is using GPON. Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down. Thanks, --Edwin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 10:00 ` Török Edwin @ 2014-03-18 12:12 ` Sebastian Moeller 2014-03-18 14:21 ` Dave Taht 0 siblings, 1 reply; 21+ messages in thread From: Sebastian Moeller @ 2014-03-18 12:12 UTC (permalink / raw) To: Török Edwin; +Cc: cerowrt-devel Hi Edwin, On Mar 18, 2014, at 11:00 , Török Edwin <edwin@etorok.net> wrote: > On 03/16/2014 09:58 PM, Dave Taht wrote: >> Get it at: >> >> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ >> >> I've been running this a few days now with no problems. > > Can you please add these packages: > - p910nd > - luci-app-p910nd > - wifitoggle > > Just upgraded from 3.7.5-2, and it looks good so far. > > I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test > if that is actually the correct setting? If you know that you have per packet overhead (more than the pure ethernet header that is handled with 'none') you should select "ethernet with overhead" and specify the overhead on your line (be sure to add the 14 bytes for the ethernet header as the kernel unhelpfully forgets to take this into account when you use the link layer adjustment method tc_stab) For ATM based systems we could use the RTT quantization effects of the ATM cells to deduce the overhead empirically but for links with out quantization that does not work, so I do not know how check which overhead to specify empirically, all you could do is look at the information you have for your link and potentially ask your ISP for more information. Just remember the goal is to supply precise information about the on-wire size of data packets so SQM can calculate the true bandwidth-cost associated with each packet. BTW if anyone in the audience knows how to measure the overhead for ethernet packets, please chime in. From your information below I would estimate: As far as I know GPON, basically is a ethernet hub solution (with one segment shared between several customers) so there is only typical ethernet overhead, plus potential framing and vlan tags, so if you select "ethernet" as link layer option, you should use the following overhead: PPP (2B), PPPoE (6B), ethernet (14B, reguired for tc_stab), potentially VLAN (4B?), potentially ethernet frame check sequence (???B) Your ISP should be able to tell you whether he uses VLAN tags on the bottle neck link (it does not matter whether the VLAN tags are actually visible/existent on your end of the GPON modem) So somewhere in the 22 to 30bytes range should work. Alas the only way to figure this out for good is to snoop packets on the fiber segment, so realistically you need to ask your ISP, or be happy that 22Bytes is as close to the true overhead as you can get with the information at your hand. And the closer to the actual wire size SQMs supplied bandwidths are the preciser the shaping works. That said it looks like each of your packets is like 8bytes larger than the kernel assumes without link layer adjustments or roughly 100*8/64 = 12.5 % for the smallest ethernet packets and 100*8/1500 = 0.5% for the largest, assuming you typically use larger packets than 64 bytes, you should not really notice whether the overhead is set correctly or not. On principle I would recommend to use "ethernet with overhead" but it should not make much of a difference. Especially since you will need to cut the shaper some slack anyways, that is even with link layer adjustments latency will be compromised unless you reduce the bandwidths specified to SQM from the line rates… Best Regards Sebastian > > I have this setup with my ISP: > cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP > > I connect using PPPoE, and AFAIK the ISP is using GPON. > Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down. > > Thanks, > --Edwin > > > > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 12:12 ` Sebastian Moeller @ 2014-03-18 14:21 ` Dave Taht 2014-03-18 15:22 ` Török Edwin 0 siblings, 1 reply; 21+ messages in thread From: Dave Taht @ 2014-03-18 14:21 UTC (permalink / raw) To: Sebastian Moeller; +Cc: Török Edwin, cerowrt-devel Regrettably the SQM system on the wndr series of hardware maxes out on CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g 25/25 works). If you want to apply this code at higher rates, routing hardware with more "oomph" is needed. I would be interested in a rrul test of your 50Mbit system. My tests of verizon at 25/25 showed them well managed on the up, far less well managed on the down, so in your 50Mbit design you might want to merely control the down with SQM. On Tue, Mar 18, 2014 at 8:12 AM, Sebastian Moeller <moeller0@gmx.de> wrote: > Hi Edwin, > > > On Mar 18, 2014, at 11:00 , Török Edwin <edwin@etorok.net> wrote: > >> On 03/16/2014 09:58 PM, Dave Taht wrote: >>> Get it at: >>> >>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.32-9/ >>> >>> I've been running this a few days now with no problems. >> >> Can you please add these packages: >> - p910nd >> - luci-app-p910nd >> - wifitoggle >> >> Just upgraded from 3.7.5-2, and it looks good so far. >> >> I'm not sure about the SQM Link Layer Adaptation, the wiki says that I should leave it as 'none' for Fiber, but how can I test >> if that is actually the correct setting? > > If you know that you have per packet overhead (more than the pure ethernet header that is handled with 'none') you should select "ethernet with overhead" and specify the overhead on your line (be sure to add the 14 bytes for the ethernet header as the kernel unhelpfully forgets to take this into account when you use the link layer adjustment method tc_stab) > For ATM based systems we could use the RTT quantization effects of the ATM cells to deduce the overhead empirically but for links with out quantization that does not work, so I do not know how check which overhead to specify empirically, all you could do is look at the information you have for your link and potentially ask your ISP for more information. Just remember the goal is to supply precise information about the on-wire size of data packets so SQM can calculate the true bandwidth-cost associated with each packet. BTW if anyone in the audience knows how to measure the overhead for ethernet packets, please chime in. > From your information below I would estimate: > As far as I know GPON, basically is a ethernet hub solution (with one segment shared between several customers) so there is only typical ethernet overhead, plus potential framing and vlan tags, so if you select "ethernet" as link layer option, you should use the following overhead: > PPP (2B), PPPoE (6B), ethernet (14B, reguired for tc_stab), potentially VLAN (4B?), potentially ethernet frame check sequence (???B) > Your ISP should be able to tell you whether he uses VLAN tags on the bottle neck link (it does not matter whether the VLAN tags are actually visible/existent on your end of the GPON modem) > So somewhere in the 22 to 30bytes range should work. Alas the only way to figure this out for good is to snoop packets on the fiber segment, so realistically you need to ask your ISP, or be happy that 22Bytes is as close to the true overhead as you can get with the information at your hand. And the closer to the actual wire size SQMs supplied bandwidths are the preciser the shaping works. > That said it looks like each of your packets is like 8bytes larger than the kernel assumes without link layer adjustments or roughly 100*8/64 = 12.5 % for the smallest ethernet packets and 100*8/1500 = 0.5% for the largest, assuming you typically use larger packets than 64 bytes, you should not really notice whether the overhead is set correctly or not. On principle I would recommend to use "ethernet with overhead" but it should not make much of a difference. Especially since you will need to cut the shaper some slack anyways, that is even with link layer adjustments latency will be compromised unless you reduce the bandwidths specified to SQM from the line rates... > > Best Regards > Sebastian > > > >> >> I have this setup with my ISP: >> cerowrt router <---(Ethernet) ----> (ISP on premise switch for multiple apartments) <----> (ISP device) <--- (fiber optics) ---> ISP >> >> I connect using PPPoE, and AFAIK the ISP is using GPON. >> Currently I have ~50 Mbps up/down speed, but I could upgrade to 1000 Mbps up/down. >> >> Thanks, >> --Edwin >> >> >> >> _______________________________________________ >> Cerowrt-devel mailing list >> Cerowrt-devel@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cerowrt-devel > > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 14:21 ` Dave Taht @ 2014-03-18 15:22 ` Török Edwin 2014-03-18 15:35 ` Dave Taht 0 siblings, 1 reply; 21+ messages in thread From: Török Edwin @ 2014-03-18 15:22 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 1357 bytes --] On 03/18/2014 04:21 PM, Dave Taht wrote: > Regrettably the SQM system on the wndr series of hardware maxes out on > CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g > 25/25 works). If you want to apply this code at higher rates, routing > hardware with more "oomph" is needed. > > I would be interested in a rrul test of your 50Mbit system. My tests > of verizon at 25/25 showed them well managed on the up, far less well > managed on the down, so in your 50Mbit design you might want to merely > control the down with SQM. Host: Linux 3.14-rc5, AMD FX(tm)-8350 Eight-Core Processor, cpufreq set to performance, cpb disabled Router: NETGEAR WNDR3700v2, 3.10.32-9 Target: OpenBSD 5.4, QEMU Virtual CPU version (cpu64-rhel6), 3300.54 MHz Host <-> Router connected via gigabit ethernet. I confirmed that my target can do >50Mbit to another server, netperf shows 128*10^6 bit/s / 108 * 10^6 bit/s. SQM with link-layer none settings: * download speed: 46000 kbit/s, upload speed 50000 kbit/s (~95% of measured speed on ISP's speedtest site). * fq_codel (default) * simple.qos * link-layer: none * results: <sqm_on_none.png> SQM off: <sqm_off.png> SQM with overhead Ethernet overhead 30: <sqm-on-30.png> SQM with overhead Ethernet overhead 22: <sqm-on-22.png> Should I also run a rrul46/rrul46compete test? Best regards, --Edwin [-- Attachment #2: sqm_on_22.png --] [-- Type: image/png, Size: 266647 bytes --] [-- Attachment #3: sqm_on_30.png --] [-- Type: image/png, Size: 257373 bytes --] [-- Attachment #4: sqm_on_none.png --] [-- Type: image/png, Size: 253387 bytes --] [-- Attachment #5: sqm_off.png --] [-- Type: image/png, Size: 242595 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 15:22 ` Török Edwin @ 2014-03-18 15:35 ` Dave Taht 2014-03-18 16:49 ` Török Edwin 2014-03-18 17:06 ` Valdis.Kletnieks 0 siblings, 2 replies; 21+ messages in thread From: Dave Taht @ 2014-03-18 15:35 UTC (permalink / raw) To: Török Edwin; +Cc: cerowrt-devel On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote: > On 03/18/2014 04:21 PM, Dave Taht wrote: >> Regrettably the SQM system on the wndr series of hardware maxes out on >> CPU at about 50Mbit down, 10Mbit up, or any combination thereof (e.g >> 25/25 works). If you want to apply this code at higher rates, routing >> hardware with more "oomph" is needed. >> >> I would be interested in a rrul test of your 50Mbit system. My tests >> of verizon at 25/25 showed them well managed on the up, far less well >> managed on the down, so in your 50Mbit design you might want to merely >> control the down with SQM. > > Host: Linux 3.14-rc5, AMD FX(tm)-8350 Eight-Core Processor, cpufreq set to performance, cpb disabled > Router: NETGEAR WNDR3700v2, 3.10.32-9 > Target: OpenBSD 5.4, QEMU Virtual CPU version (cpu64-rhel6), 3300.54 MHz > > Host <-> Router connected via gigabit ethernet. > > I confirmed that my target can do >50Mbit to another server, netperf shows 128*10^6 bit/s / 108 * 10^6 bit/s. > > SQM with link-layer none settings: > * download speed: 46000 kbit/s, upload speed 50000 kbit/s (~95% of measured speed on ISP's speedtest site). > * fq_codel (default) > * simple.qos > * link-layer: none > * results: <sqm_on_none.png> > > SQM off: <sqm_off.png> > > SQM with overhead Ethernet overhead 30: <sqm-on-30.png> > SQM with overhead Ethernet overhead 22: <sqm-on-22.png> > > Should I also run a rrul46/rrul46compete test? If you can test ipv6 too, that would be great. > Best regards, > --Edwin At 8ms of induced extra latency without SQM there doesn't seem to be much point in running it on your platform. It does look like you are tail dropping... A thought would be to leave it off and try running your link at 100Mbit rather than gige. # ethtool -s ge00 advertise 0x008 and see if fq_codel alone can break up bursts better. I note that linux 3.14 TCP is now so highly debloated that it is really hard to compare the results we get with it with any TCP before it - the TSO offload fixes, tcp small queues, etc, have really shortened and made more accurate the control loops. second note is that the wndr can only forward packets at about 330Mbit without firewall rules. Add in the firewall rules and you are looking at sub 120mbit forwarding performance. I am jealous of your link. :) -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 15:35 ` Dave Taht @ 2014-03-18 16:49 ` Török Edwin 2014-03-18 17:03 ` Dave Taht 2014-03-18 17:06 ` Valdis.Kletnieks 1 sibling, 1 reply; 21+ messages in thread From: Török Edwin @ 2014-03-18 16:49 UTC (permalink / raw) To: Dave Taht; +Cc: cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 1755 bytes --] On 03/18/2014 05:35 PM, Dave Taht wrote: > On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote: >> Should I also run a rrul46/rrul46compete test? > > If you can test ipv6 too, that would be great. Attached IPv6 results. > > At 8ms of induced extra latency without SQM there doesn't seem to be > much point in running it on your platform. It does look like you are tail > dropping... > > A thought would be to leave it > off and try running your link at 100Mbit rather than gige. > > # ethtool -s ge00 advertise 0x008 Well ge00 already runs at 100Mbit, because ISP plugged me into a 100Mbit port on the switch: Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 100Mb/s They would plug me into a 1Gbit port only if I pay for 1Gbps connectivity, which is in fact only 7 EUR/month more, but its not available in my area yet :) > > and see if fq_codel alone can break up bursts better. > > I note that linux 3.14 TCP is now so highly debloated that it is really hard to > compare the results we get with it with any TCP before it - the TSO offload > fixes, tcp small queues, etc, have really shortened and made more accurate > the control loops. > > second note is that the wndr can only forward packets at about 330Mbit > without firewall rules. Add in the firewall rules and you are looking at > sub 120mbit forwarding performance. Good to know, not much point in purchasing 1Gbit connectivity from my ISP unless I also replace my router. > > I am jealous of your link. :) Best regards, --Edwin [-- Attachment #2: rrul46_compete_sqm_off.png --] [-- Type: image/png, Size: 262670 bytes --] [-- Attachment #3: rrul46_compete_sqm_on.png --] [-- Type: image/png, Size: 276876 bytes --] [-- Attachment #4: rrul46_sqm_on.png --] [-- Type: image/png, Size: 211120 bytes --] [-- Attachment #5: rrul46_sqm_off.png --] [-- Type: image/png, Size: 225381 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 16:49 ` Török Edwin @ 2014-03-18 17:03 ` Dave Taht 0 siblings, 0 replies; 21+ messages in thread From: Dave Taht @ 2014-03-18 17:03 UTC (permalink / raw) To: Török Edwin; +Cc: cerowrt-devel On Tue, Mar 18, 2014 at 12:49 PM, Török Edwin <edwin@etorok.net> wrote: > On 03/18/2014 05:35 PM, Dave Taht wrote: >> On Tue, Mar 18, 2014 at 11:22 AM, Török Edwin <edwin@etorok.net> wrote: >>> Should I also run a rrul46/rrul46compete test? >> >> If you can test ipv6 too, that would be great. > > Attached IPv6 results. The behavior I find puzzling is how rapidly you lose your measurement flows, basically immediately after you hit slow start on the test, packet loss there occurs and they all die. My guess is you actually have insufficient buffering on the gpon or onc sides, which also explains your low latency. >> >> At 8ms of induced extra latency without SQM there doesn't seem to be >> much point in running it on your platform. It does look like you are tail >> dropping... >> >> A thought would be to leave it >> off and try running your link at 100Mbit rather than gige. >> >> # ethtool -s ge00 advertise 0x008 > > Well ge00 already runs at 100Mbit, because ISP plugged me into a 100Mbit port on the switch: > Advertised link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Half 1000baseT/Full > Advertised pause frame use: No > Advertised auto-negotiation: Yes > Speed: 100Mb/s Ah. > They would plug me into a 1Gbit port only if I pay for 1Gbps connectivity, which is in fact only 7 EUR/month more, but its not available in my area yet :) > >> >> and see if fq_codel alone can break up bursts better. >> >> I note that linux 3.14 TCP is now so highly debloated that it is really hard to >> compare the results we get with it with any TCP before it - the TSO offload >> fixes, tcp small queues, etc, have really shortened and made more accurate >> the control loops. >> >> second note is that the wndr can only forward packets at about 330Mbit >> without firewall rules. Add in the firewall rules and you are looking at >> sub 120mbit forwarding performance. > > Good to know, not much point in purchasing 1Gbit connectivity from my ISP unless I also replace my router. The only things I have found that can push that are x86 boxes mostly. Toke is evaluating a few new ones. > >> >> I am jealous of your link. :) > > Best regards, > --Edwin -- Dave Täht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Cerowrt-devel] cerowrt-3.10.32-9 released 2014-03-18 15:35 ` Dave Taht 2014-03-18 16:49 ` Török Edwin @ 2014-03-18 17:06 ` Valdis.Kletnieks 1 sibling, 0 replies; 21+ messages in thread From: Valdis.Kletnieks @ 2014-03-18 17:06 UTC (permalink / raw) To: Dave Taht; +Cc: T?r?k Edwin, cerowrt-devel [-- Attachment #1: Type: text/plain, Size: 360 bytes --] On Tue, 18 Mar 2014 11:35:51 -0400, Dave Taht said: > second note is that the wndr can only forward packets at about 330Mbit > without firewall rules. Add in the firewall rules and you are looking at > sub 120mbit forwarding performance. It's good to know that it can handle any connection I'm likely to get from Comcast in the reasonable future. :) [-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --] ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2014-03-18 17:08 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-03-16 19:58 [Cerowrt-devel] cerowrt-3.10.32-9 released Dave Taht 2014-03-16 21:27 ` Valdis.Kletnieks 2014-03-16 21:45 ` Dave Taht 2014-03-17 0:03 ` Toke Høiland-Jørgensen 2014-03-17 0:42 ` Valdis.Kletnieks 2014-03-17 0:54 ` Dave Taht 2014-03-17 12:20 ` Michael Richardson 2014-03-17 13:40 ` Dave Taht 2014-03-17 14:30 ` Toke Høiland-Jørgensen 2014-03-17 14:39 ` Toke Høiland-Jørgensen 2014-03-17 14:55 ` Dave Taht 2014-03-17 14:55 ` Michael Richardson 2014-03-17 15:18 ` Dave Taht 2014-03-18 10:00 ` Török Edwin 2014-03-18 12:12 ` Sebastian Moeller 2014-03-18 14:21 ` Dave Taht 2014-03-18 15:22 ` Török Edwin 2014-03-18 15:35 ` Dave Taht 2014-03-18 16:49 ` Török Edwin 2014-03-18 17:03 ` Dave Taht 2014-03-18 17:06 ` Valdis.Kletnieks
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox