Development issues regarding the cerowrt test router project
 help / color / mirror / Atom feed
* [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
@ 2013-11-12  5:47 Richard E. Brown
  2013-11-12  9:11 ` Sebastian Moeller
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Richard E. Brown @ 2013-11-12  5:47 UTC (permalink / raw)
  To: cerowrt-devel

I used the sysupgrade process to upgrade my primary router from 3.7.5-2 firmware to 3.10.18-1.

- I initially goofed, and installed the wrong build firmware (I installed the WNDR3800 image on a WNDR3700v2 router.) The symptoms were that the router worked, but not very well. Speedtest was gave miserable speeds; netalyzr didn’t work at all. (It said there were serious problems: see http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-26536-45539c09-7334-456b-b81a ) I was able to download the proper system upgrade firmware, but it took forever. Don’t do it :-)

- After installing the proper image (for WNDR3700v2), PPPoE didn’t immediately come up on my 7000/768kbps ADSL from Fairpoint. I had to go to the Edit page for the ge00 interface, and click Apply (without making any changes to the saved settings). This caused the link to come right up.

- The henet 6in4 tunnel did not work. The router received the expected global IPv6 address, and handed an IPv6 global address to my notebook, but neither the router nor the notebook were able to ping ipv6.google.com. I removed that interface from the configs using the GUI.

- Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 

- The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 

- This may be related to the netalyzr test - after netalyzr completed a run that complained that nothing worked (see above), these errors stopped for a while.

- However, using NetalyzrCLI.jar, I got the following results where most everything worked: http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-24217-0cc69e65-e649-4be6-b2c5

- The PPPoE running on ge00 link seemed to bounce every 10-15 minutes, and I often had to bring it up manually.

- Reverting to 3.7.5-2.


[  992.386718] ------------[ cut here ]------------
[  992.390625] WARNING: at net/sched/sch_hfsc.c:1428 hfsc_dequeue+0x258/0x49c [sch_hfsc]()
[  992.398437] Modules linked in: ifb ath9k iptable_nat ath9k_common pppoe nf_nat_ipv4 nf_conntrack_ipv4 mac80211 cfg80211 ath9k_hw xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_state xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY ts_kmp ts_fsm ts_bm pptp pppox ppp_async nf_nat_irc nf_nat_ftp nf_defrag_ipv4 nf_conntrack_irc nf_conntrack_ftp libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ah ipt_REJECT ipt_MASQUERADE ipt_ECN ip_tables crc_ccitt compat ath sch_teql sch_tbf sch_sfq sch_red sch_qfq sch_prio sch_pie sch_ns2_codel sch_nfq_codel sch_netem sch_htb sch_gred sch_efq_codel sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police act_ipt act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress xt_set ip_set_list_set ip_set_hash_netport ip_set_hash_netiface ip_set_hash_net ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_NPT ip6t_MASQUERADE ip6table_nat nf_nat_ipv6 nf_nat ip6t_REJECT ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables nf_conntrack_ipv6 nf_conntrack nf_defrag_ipv6 pppoatm ppp_generic slhc ip_gre gre sit ipcomp xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet esp4 ah4 ip6_tunnel tunnel6 tunnel4 ip_tunnel tun tcp_ledbat af_key xfrm_user xfrm_ipcomp xfrm_algo vfat fat autofs4 br2684 atm nls_utf8 nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_iso8859_1 nls_cp437 ipv6 chainiv eseqiv crypto_wq sha1_generic krng rng md5 hmac des_generic deflate zlib_inflate zlib_deflate cbc authenc aead arc4 crypto_blkcipher usb_storage input_polldev leds_gpio ohci_hcd ledtrig_timer ledtrig_default_on ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 crc16 jbd2 mbcache button_hotplug input_core usbcore nls_base usb_common crc32c crypto_hash [last unloaded: ifb]
[  992.593750] CPU: 0 PID: 1 Comm: procd Tainted: G        W    3.10.18 #1
[  992.601562] Stack : 00000006 00000000 00000000 00000000 00000000 00000000 803a2abe 0000003b
[  992.601562] 	  838281a8 802e101c 80382a30 8033173b 00000001 82de5788 803a0000 803a0000
[  992.601562] 	  00400100 800796c8 00000003 80077080 00000000 00000000 802e28e4 83831d84
[  992.601562] 	  00831d84 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  992.601562] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 83831d10
[  992.601562] 	  ...
[  992.636718] Call Trace:
[  992.636718] [<8006e4e4>] show_stack+0x48/0x70
[  992.644531] [<80077204>] warn_slowpath_common+0x78/0xa8
[  992.648437] [<8007724c>] warn_slowpath_null+0x18/0x24
[  992.652343] [<82e41744>] hfsc_dequeue+0x258/0x49c [sch_hfsc]
[  992.660156] [<80226a48>] __qdisc_run+0xdc/0x18c
[  992.664062] [<8020bb9c>] net_tx_action+0xdc/0x104
[  992.667968] [<8007e0fc>] __do_softirq+0xc8/0x1b4
[  992.671875] [<8007e298>] do_softirq+0x48/0x68
[  992.675781] [<8007e4d4>] irq_exit+0x54/0x70
[  992.679687] [<8006082c>] ret_from_irq+0x0/0x4
[  992.687500] 
[  992.687500] ---[ end trace 8987bf849bc7e685 ]---


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12  5:47 [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report Richard E. Brown
@ 2013-11-12  9:11 ` Sebastian Moeller
       [not found]   ` <51BF9432-6FC2-4A14-B147-13F1E779CA93@gmail.com>
  2013-11-13  4:21   ` Richard E. Brown
  2013-11-12  9:40 ` Fred Stratton
  2013-11-15  1:56 ` Richard E. Brown
  2 siblings, 2 replies; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-12  9:11 UTC (permalink / raw)
  To: Richard E. Brown; +Cc: cerowrt-devel

Hi Richard,


On Nov 12, 2013, at 06:47 , Richard E. Brown <richb.hanover@gmail.com> wrote:

> I used the sysupgrade process to upgrade my primary router from 3.7.5-2 firmware to 3.10.18-1.
> 
> - I initially goofed, and installed the wrong build firmware (I installed the WNDR3800 image on a WNDR3700v2 router.) The symptoms were that the router worked, but not very well. Speedtest was gave miserable speeds; netalyzr didn’t work at all. (It said there were serious problems: see http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-26536-45539c09-7334-456b-b81a ) I was able to download the proper system upgrade firmware, but it took forever. Don’t do it :-)
> 
> - After installing the proper image (for WNDR3700v2), PPPoE didn’t immediately come up on my 7000/768kbps ADSL from Fairpoint. I had to go to the Edit page for the ge00 interface, and click Apply (without making any changes to the saved settings). This caused the link to come right up.
> 
> - The henet 6in4 tunnel did not work. The router received the expected global IPv6 address, and handed an IPv6 global address to my notebook, but neither the router nor the notebook were able to ping ipv6.google.com. I removed that interface from the configs using the GUI.
> 
> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 

	Just curious, did you specify overhead and encapsulation?

> 
> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 

	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
to check what is up with the AQM system...

Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?

> 
> - This may be related to the netalyzr test - after netalyzr completed a run that complained that nothing worked (see above), these errors stopped for a while.
> 
> - However, using NetalyzrCLI.jar, I got the following results where most everything worked: http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-24217-0cc69e65-e649-4be6-b2c5
> 
> - The PPPoE running on ge00 link seemed to bounce every 10-15 minutes, and I often had to bring it up manually.
> 
> - Reverting to 3.7.5-2.
> 
> 
> [  992.386718] ------------[ cut here ]------------
> [  992.390625] WARNING: at net/sched/sch_hfsc.c:1428 hfsc_dequeue+0x258/0x49c [sch_hfsc]()
> [  992.398437] Modules linked in: ifb ath9k iptable_nat ath9k_common pppoe nf_nat_ipv4 nf_conntrack_ipv4 mac80211 cfg80211 ath9k_hw xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_state xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY ts_kmp ts_fsm ts_bm pptp pppox ppp_async nf_nat_irc nf_nat_ftp nf_defrag_ipv4 nf_conntrack_irc nf_conntrack_ftp libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ah ipt_REJECT ipt_MASQUERADE ipt_ECN ip_tables crc_ccitt compat ath sch_teql sch_tbf sch_sfq sch_red sch_qfq sch_prio sch_pie sch_ns2_codel sch_nfq_codel sch_netem sch_htb sch_gred sch_efq_codel sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police act_ipt act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress xt_set ip_set_list_set ip_set_hash_netport ip_set_hash_netiface ip_set_hash_net ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_NPT ip6t_MASQUERADE ip6table_nat nf_nat_ipv6 nf_nat ip6t_REJECT ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables nf_conntrack_ipv6 nf_conntrack nf_defrag_ipv6 pppoatm ppp_generic slhc ip_gre gre sit ipcomp xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet esp4 ah4 ip6_tunnel tunnel6 tunnel4 ip_tunnel tun tcp_ledbat af_key xfrm_user xfrm_ipcomp xfrm_algo vfat fat autofs4 br2684 atm nls_utf8 nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_iso8859_1 nls_cp437 ipv6 chainiv eseqiv crypto_wq sha1_generic krng rng md5 hmac des_generic deflate zlib_inflate zlib_deflate cbc authenc aead arc4 crypto_blkcipher usb_storage input_polldev leds_gpio ohci_hcd ledtrig_timer ledtrig_default_on ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 crc16 jbd2 mbcache button_hotplug input_core usbcore nls_base usb_common crc32c crypto_hash [last unloaded: ifb]
> [  992.593750] CPU: 0 PID: 1 Comm: procd Tainted: G        W    3.10.18 #1
> [  992.601562] Stack : 00000006 00000000 00000000 00000000 00000000 00000000 803a2abe 0000003b
> [  992.601562] 	  838281a8 802e101c 80382a30 8033173b 00000001 82de5788 803a0000 803a0000
> [  992.601562] 	  00400100 800796c8 00000003 80077080 00000000 00000000 802e28e4 83831d84
> [  992.601562] 	  00831d84 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [  992.601562] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 83831d10
> [  992.601562] 	  ...
> [  992.636718] Call Trace:
> [  992.636718] [<8006e4e4>] show_stack+0x48/0x70
> [  992.644531] [<80077204>] warn_slowpath_common+0x78/0xa8
> [  992.648437] [<8007724c>] warn_slowpath_null+0x18/0x24
> [  992.652343] [<82e41744>] hfsc_dequeue+0x258/0x49c [sch_hfsc]
> [  992.660156] [<80226a48>] __qdisc_run+0xdc/0x18c
> [  992.664062] [<8020bb9c>] net_tx_action+0xdc/0x104
> [  992.667968] [<8007e0fc>] __do_softirq+0xc8/0x1b4
> [  992.671875] [<8007e298>] do_softirq+0x48/0x68
> [  992.675781] [<8007e4d4>] irq_exit+0x54/0x70
> [  992.679687] [<8006082c>] ret_from_irq+0x0/0x4
> [  992.687500] 
> [  992.687500] ---[ end trace 8987bf849bc7e685 ]---
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12  5:47 [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report Richard E. Brown
  2013-11-12  9:11 ` Sebastian Moeller
@ 2013-11-12  9:40 ` Fred Stratton
  2013-11-12 17:24   ` Richard E. Brown
  2013-11-15  1:56 ` Richard E. Brown
  2 siblings, 1 reply; 14+ messages in thread
From: Fred Stratton @ 2013-11-12  9:40 UTC (permalink / raw)
  To: Richard E. Brown, cerowrt-devel

After your helpful writeup as an ADSL user, I am expanding on the 
previous remarks I made. SM is making his contribution as I write.

For several years, I have been with Telefonica O2 as an ISP. This 
network gave subscribers a dynamic ipv4 address which did not change 
unless the gateway WAN MAC changed. Sky bought the network to get the 
400 000 subscribers and migrated everyone to their network, closing the 
O2 setup. I left, moving to TalkTalk, a network which gives out a 
different ipv4 address every time it retrains, two or three times a day 
at present, using their proprietary dynamic line management unchanged.

I threw away the supplied Trendchip box, and use a 1483-bridged TP-Link 
TD-W8970 to connect to the phone line.

Here, PPPoE always comes up using the 3.10.18-1 build. Henet is more 
problematic. I lose ipv6 connectivity every five or so retrains, and 
have to reboot the router.

The latest builds allow concurrent downloading or the use of torrents 
and video streaming 90 per cent of the time. tc-stab is apparently 
functional.

Changing ISP has been a very illuminating exercise from the point of 
view of 'bufferbloat'. O2 lost a lot of packets through their site 
screening mechanism. TalkTalk subcontracts its deep packet inspection to 
Huawei in China. This minimises packet loss.

Have set AQM to circa 70 percent of download sync and 95 per cent of 
upload sync, based on empirical incremental changes.


On 12/11/13 05:47, Richard E. Brown wrote:
> I used the sysupgrade process to upgrade my primary router from 3.7.5-2 firmware to 3.10.18-1.
>
> - I initially goofed, and installed the wrong build firmware (I installed the WNDR3800 image on a WNDR3700v2 router.) The symptoms were that the router worked, but not very well. Speedtest was gave miserable speeds; netalyzr didn’t work at all. (It said there were serious problems: see http://n3.netalyzr.icsi.berkeley.edu/summary/id=36ea240d-26536-45539c09-7334-456b-b81a ) I was able to download the proper system upgrade firmware, but it took forever. Don’t do it :-)
>
> - After installing the proper image (for WNDR3700v2), PPPoE didn’t immediately come up on my 7000/768kbps ADSL from Fairpoint. I had to go to the Edit page for the ge00 interface, and click Apply (without making any changes to the saved settings). This caused the link to come right up.
>
> - The henet 6in4 tunnel did not work. The router received the expected global IPv6 address, and handed an IPv6 global address to my notebook, but neither the router nor the notebook were able to ping ipv6.google.com. I removed that interface from the configs using the GUI.
>
> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds
>
> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis.
>
> - This may be related to the netalyzr test - after netalyzr completed a run that complained that nothing worked (see above), these errors stopped for a while.
>
> - However, using NetalyzrCLI.jar, I got the following results where most everything worked: http://n2.netalyzr.icsi.berkeley.edu/summary/id=43ca208a-24217-0cc69e65-e649-4be6-b2c5
>
> - The PPPoE running on ge00 link seemed to bounce every 10-15 minutes, and I often had to bring it up manually.
>
> - Reverting to 3.7.5-2.
>
>
> [  992.386718] ------------[ cut here ]------------
> [  992.390625] WARNING: at net/sched/sch_hfsc.c:1428 hfsc_dequeue+0x258/0x49c [sch_hfsc]()
> [  992.398437] Modules linked in: ifb ath9k iptable_nat ath9k_common pppoe nf_nat_ipv4 nf_conntrack_ipv4 mac80211 cfg80211 ath9k_hw xt_time xt_tcpudp xt_tcpmss xt_string xt_statistic xt_state xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY ts_kmp ts_fsm ts_bm pptp pppox ppp_async nf_nat_irc nf_nat_ftp nf_defrag_ipv4 nf_conntrack_irc nf_conntrack_ftp libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ah ipt_REJECT ipt_MASQUERADE ipt_ECN ip_tables crc_ccitt compat ath sch_teql sch_tbf sch_sfq sch_red sch_qfq sch_prio sch_pie sch_ns2_codel sch_nfq_codel sch_netem sch_htb sch_gred sch_efq_codel sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police act_ipt act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress xt_set ip_set_list_set ip_set_hash_netport ip_set_hash_netiface ip_set_hash_net ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_NPT ip6t_MASQUERADE ip6table_nat nf_nat_ipv6 nf_nat ip6t_REJECT ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables nf_conntrack_ipv6 nf_conntrack nf_defrag_ipv6 pppoatm ppp_generic slhc ip_gre gre sit ipcomp xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet esp4 ah4 ip6_tunnel tunnel6 tunnel4 ip_tunnel tun tcp_ledbat af_key xfrm_user xfrm_ipcomp xfrm_algo vfat fat autofs4 br2684 atm nls_utf8 nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_iso8859_1 nls_cp437 ipv6 chainiv eseqiv crypto_wq sha1_generic krng rng md5 hmac des_generic deflate zlib_inflate zlib_deflate cbc authenc aead arc4 crypto_blkcipher usb_storage input_polldev leds_gpio ohci_hcd ledtrig_timer ledtrig_default_on ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 crc16 jbd2 mbcache button_hotplug input_core usbcore nls_base usb_common crc32c crypto_hash [last unloaded: ifb]
> [  992.593750] CPU: 0 PID: 1 Comm: procd Tainted: G        W    3.10.18 #1
> [  992.601562] Stack : 00000006 00000000 00000000 00000000 00000000 00000000 803a2abe 0000003b
> [  992.601562] 	  838281a8 802e101c 80382a30 8033173b 00000001 82de5788 803a0000 803a0000
> [  992.601562] 	  00400100 800796c8 00000003 80077080 00000000 00000000 802e28e4 83831d84
> [  992.601562] 	  00831d84 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [  992.601562] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 83831d10
> [  992.601562] 	  ...
> [  992.636718] Call Trace:
> [  992.636718] [<8006e4e4>] show_stack+0x48/0x70
> [  992.644531] [<80077204>] warn_slowpath_common+0x78/0xa8
> [  992.648437] [<8007724c>] warn_slowpath_null+0x18/0x24
> [  992.652343] [<82e41744>] hfsc_dequeue+0x258/0x49c [sch_hfsc]
> [  992.660156] [<80226a48>] __qdisc_run+0xdc/0x18c
> [  992.664062] [<8020bb9c>] net_tx_action+0xdc/0x104
> [  992.667968] [<8007e0fc>] __do_softirq+0xc8/0x1b4
> [  992.671875] [<8007e298>] do_softirq+0x48/0x68
> [  992.675781] [<8007e4d4>] irq_exit+0x54/0x70
> [  992.679687] [<8006082c>] ret_from_irq+0x0/0x4
> [  992.687500]
> [  992.687500] ---[ end trace 8987bf849bc7e685 ]---
>
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12  9:40 ` Fred Stratton
@ 2013-11-12 17:24   ` Richard E. Brown
  0 siblings, 0 replies; 14+ messages in thread
From: Richard E. Brown @ 2013-11-12 17:24 UTC (permalink / raw)
  To: Fred Stratton, cerowrt-devel

On Nov 12, 2013, at 4:40 AM, Fred Stratton <fredstratton@imap.cc> wrote:

> After your helpful writeup as an ADSL user, I am expanding on the previous remarks I made. SM is making his contribution as I write.
> 
> For several years, I have been with Telefonica O2 as an ISP. This network gave subscribers a dynamic ipv4 address which did not change unless the gateway WAN MAC changed. Sky bought the network to get the 400 000 subscribers and migrated everyone to their network, closing the O2 setup. I left, moving to TalkTalk, a network which gives out a different ipv4 address every time it retrains, two or three times a day at present, using their proprietary dynamic line management unchanged.
> 
> I threw away the supplied Trendchip box, and use a 1483-bridged TP-Link TD-W8970 to connect to the phone line.
> 
> Here, PPPoE always comes up using the 3.10.18-1 build. Henet is more problematic. I lose ipv6 connectivity every five or so retrains, and have to reboot the router.
> 
> The latest builds allow concurrent downloading or the use of torrents and video streaming 90 per cent of the time. tc-stab is apparently functional.
> 
> Changing ISP has been a very illuminating exercise from the point of view of 'bufferbloat'. O2 lost a lot of packets through their site screening mechanism. TalkTalk subcontracts its deep packet inspection to Huawei in China. This minimises packet loss.
> 
> Have set AQM to circa 70 percent of download sync and 95 per cent of upload sync, based on empirical incremental changes.

What other information could I collect if I find that PPPoE isn’t coming up reliably?

re: HEnet tunnels not coming up. There’s an easier way to reconnect the tunnel by going to a HTTPS link. It’s described in the IPv6 Tunnel page on the wiki: http://www.bufferbloat.net/projects/cerowrt/wiki/IPv6_Tunnel (Check out step 3, manually re-establishing the tunnel)

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
       [not found]   ` <51BF9432-6FC2-4A14-B147-13F1E779CA93@gmail.com>
@ 2013-11-12 17:26     ` Richard E. Brown
  2013-11-12 21:17       ` Sebastian Moeller
  0 siblings, 1 reply; 14+ messages in thread
From: Richard E. Brown @ 2013-11-12 17:26 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cerowrt-devel

> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>>> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 
>> 
>> 	Just curious, did you specify overhead and encapsulation?

No, I simply used the defaults on that page.

>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>> 
>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>> to check what is up with the AQM system...
>> 
>> Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?

Yes, I was using QoS in 3.7.5-2, and I kept the old configuration files (so I didn’t have to re-enter credentials for my DSL link, etc.) I guess it seems likely that I may have been running *both* (!)

Would a better upgrade path be to start with 3.7.5-2, then disable the QoS, then flash with 3.10.18-1? (My intuition tells me that this would remove the QoS settings from the loop…)

It’s pretty easy to re-flash with 3.10.18-1 and run the tc commands. If disabling QoS makes sense, then after I do the ’tc’ experiment, I’ll re-flash with 3.7.5-2, turn off QoS, then reflash to 3.10.18-1. But it’ll have to wait ’til dark so no one else is using it. :-)

Rich



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12 17:26     ` Richard E. Brown
@ 2013-11-12 21:17       ` Sebastian Moeller
  2013-11-12 23:06         ` Sebastian Moeller
  0 siblings, 1 reply; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-12 21:17 UTC (permalink / raw)
  To: Richard E. Brown; +Cc: cerowrt-devel

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

Hi Richard,


On Nov 12, 2013, at 18:26 , Richard E. Brown <richb.hanover@gmail.com> wrote:

>> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>>> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 
>>> 
>>> 	Just curious, did you specify overhead and encapsulation?
> 
> No, I simply used the defaults on that page.

	Ah, you might want to try setting the link layer adaptation mechanism to tc_stab, the link layer to adls or atm and the overhead to 40 (or look at http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ to figure out the fixed per packet overhead of your link). This should allow you to specify a larger percentage of your link rate as shaped rate… But see the attached screenshot of my AQM tab



> 
>>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>>> 
>>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>>> to check what is up with the AQM system...
>>> 
>>> Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?
> 
> Yes, I was using QoS in 3.7.5-2, and I kept the old configuration files (so I didn’t have to re-enter credentials for my DSL link, etc.) I guess it seems likely that I may have been running *both* (!)

	Probably just QOS, as I think both QOS and AQM start out by tearing down all discs… but the hfsc error should not affect AQM since it uses HTB.

> 
> Would a better upgrade path be to start with 3.7.5-2, then disable the QoS, then flash with 3.10.18-1? (My intuition tells me that this would remove the QoS settings from the loop…)

	Mhhh, I guess that should work.
> 
> It’s pretty easy to re-flash with 3.10.18-1 and run the tc commands. If disabling QoS makes sense, then after I do the ’tc’ experiment, I’ll re-flash with 3.7.5-2, turn off QoS, then reflash to 3.10.18-1.

	 This should not fix your PPPoE issues though… so maybe it is not the best time to switch…

> But it’ll have to wait ’til dark so no one else is using it. :-)

	Hihi, same issue on my side :) No "playing" with the network while my wife is using it 

Best
	Sebastian


> 
> Rich
> 
> 


[-- Attachment #2.1: Type: text/html, Size: 4486 bytes --]

[-- Attachment #2.2: PastedGraphic-1.tiff --]
[-- Type: image/tiff, Size: 167998 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12 21:17       ` Sebastian Moeller
@ 2013-11-12 23:06         ` Sebastian Moeller
  2013-11-12 23:11           ` Sebastian Moeller
  0 siblings, 1 reply; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-12 23:06 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cerowrt-devel

Hi Richard, hi Dave, hi list,

so I could not resist the lure of 3.10.18-1 and upgraded my 3.10.11-2.; which turned out to be slightly more involved than I had expected.

1) SYSUPGRADE
root@nacktmulle:/# sysupgrade -d 60 -n /home/persistent/cerowrts/3.10.18-1/openwrt-ar71xx-generic-wndr3700v2-squashfs-sysupgrade.bin
 
killall: watchdog: no process killed
Sending TERM to remaining processes ... netifd dynamic_dns_upd sleep minissdpd lighttpd crond lighttpd pimd snmpd xinetd dbus-daemon dnsmasq zebra babeld watchquagga smbd nmbd avahi-daemon ahcpd rngd ntpd ubusd askfirst 
Sending KILL to remaining processes ... ubusd askfirst 
Switching to ramdisk...
mount:  /proc is not a block device
umount: /tmp/root: not mounted
Failed to switch over to ramfs. Please reboot.

Rebooting still returned me back to 3.10.11-2

2)  MTD
root@nacktmulle:/tmp# mtd -r write /tmp/firmware.img 
Usage: mtd [<options> ...] <command> [<arguments> ...] <device>[:<device>...]

The device is in the format of mtdX (eg: mtd4) or its label.
mtd recognizes these commands:
        unlock                  unlock the device
        refresh                 refresh mtd partition
        erase                   erase all data on device
        write <imagefile>|-     write <imagefile> (use - for stdin) to device
        jffs2write <file>       append <file> to the jffs2 partition on the device
        fixtrx                  fix the checksum in a trx header on first boot
Following options are available:
        -q                      quiet mode (once: no [w] on writing,
                                           twice: no status messages)
        -n                      write without first erasing the blocks
        -r                      reboot after successful command
        -f                      force write without trx checks
        -e <device>             erase <device> before executing the command
        -d <name>               directory for jffs2write, defaults to "tmp"
        -j <name>               integrate <file> into jffs2 data when writing an image
        -p                      write beginning at partition offset
        -o offset               offset of the image header in the partition(for fixtrx)
        -F <part>[:<size>[:<entrypoint>]][,<part>...]
                                alter the fis partition table to create new partitions replacing
                                the partitions provided as argument to the write command
                                (only valid together with the write command)

Example: To write linux.trx to mtd4 labeled as linux and reboot afterwards
         mtd -r write linux.trx linux

Still no upgrade performed, but at least it is clearer why, my command was incomplete… BUT I seem to recall that it was exactly this command that actually allowed me to install 3.10.11-2 in the first place, weird.

3) LUCI (http://gw.home.lan:81/cgi-bin/luci/;stok=19113c7f25269daca52ed92ef4d4b802/admin/system/flashops/)
I disabled the Keep Settings checkbox, uploaded the image (after making sure /tmp had enough space) followed the "flash image…" link e voila, 3.10.18-1 up and running in no time

I have no idea what the GUI actually does differently from calling sysupgrade on the command line.


So the upshot is Juergen Botz is right and the GUI seems to work, at least if one does not keep the old configuration. (And for that problem I followed caves advice and just saved /overlay before upgrading, so I could see the old configuration files and compare.)

Since I am using cerowrt as secondary router I have no input on the PPPoE issues….

best
	Sebastian




On Nov 12, 2013, at 22:17 , Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi Richard,
> 
> 
> On Nov 12, 2013, at 18:26 , Richard E. Brown <richb.hanover@gmail.com> wrote:
> 
>>> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>> 
>>>>> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 
>>>> 
>>>> 	Just curious, did you specify overhead and encapsulation?
>> 
>> No, I simply used the defaults on that page.
> 
> 	Ah, you might want to try setting the link layer adaptation mechanism to tc_stab, the link layer to adls or atm and the overhead to 40 (or look at http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ to figure out the fixed per packet overhead of your link). This should allow you to specify a larger percentage of your link rate as shaped rate… But see the attached screenshot of my AQM tab
> 
> 
> <PastedGraphic-1.tiff>
> 
>> 
>>>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>>>> 
>>>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>>>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>>>> to check what is up with the AQM system...
>>>> 
>>>> Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?
>> 
>> Yes, I was using QoS in 3.7.5-2, and I kept the old configuration files (so I didn’t have to re-enter credentials for my DSL link, etc.) I guess it seems likely that I may have been running *both* (!)
> 
> 	Probably just QOS, as I think both QOS and AQM start out by tearing down all discs… but the hfsc error should not affect AQM since it uses HTB.
> 
>> 
>> Would a better upgrade path be to start with 3.7.5-2, then disable the QoS, then flash with 3.10.18-1? (My intuition tells me that this would remove the QoS settings from the loop…)
> 
> 	Mhhh, I guess that should work.
>> 
>> It’s pretty easy to re-flash with 3.10.18-1 and run the tc commands. If disabling QoS makes sense, then after I do the ’tc’ experiment, I’ll re-flash with 3.7.5-2, turn off QoS, then reflash to 3.10.18-1.
> 
> 	 This should not fix your PPPoE issues though… so maybe it is not the best time to switch…
> 
>> But it’ll have to wait ’til dark so no one else is using it. :-)
> 
> 	Hihi, same issue on my side :) No "playing" with the network while my wife is using it 
> 
> Best
> 	Sebastian
> 
> 
>> 
>> Rich
>> 
>> 
> 
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel

-- 
Sandra, Okko, Joris, & Sebastian Moeller
Telefon: +49 7071 96 49 783, +49 7071 96 49 784, +49 7071 96 49 785
GSM: +49-1577-190 31 41
GSM: +49-1517-00 70 355

Moltkestrasse 6
72072 Tuebingen
Deutschland


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12 23:06         ` Sebastian Moeller
@ 2013-11-12 23:11           ` Sebastian Moeller
  2013-11-15 12:35             ` Sebastian Moeller
  0 siblings, 1 reply; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-12 23:11 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cerowrt-devel

Hi All,

it turns out that not being able/willing to read can make you do busy work. It seems I forgot to add  "firmware" as device to my mtd invocation… I guess I would never have tried the GUI if I had gotten the mtd command right the first time :)


best
	Sebastian


On Nov 13, 2013, at 00:06 , Sebastian Moeller <sebastian.moeller@gmail.com> wrote:

> Hi Richard, hi Dave, hi list,
> 
> so I could not resist the lure of 3.10.18-1 and upgraded my 3.10.11-2.; which turned out to be slightly more involved than I had expected.
> 
> 1) SYSUPGRADE
> root@nacktmulle:/# sysupgrade -d 60 -n /home/persistent/cerowrts/3.10.18-1/openwrt-ar71xx-generic-wndr3700v2-squashfs-sysupgrade.bin
> 
> killall: watchdog: no process killed
> Sending TERM to remaining processes ... netifd dynamic_dns_upd sleep minissdpd lighttpd crond lighttpd pimd snmpd xinetd dbus-daemon dnsmasq zebra babeld watchquagga smbd nmbd avahi-daemon ahcpd rngd ntpd ubusd askfirst 
> Sending KILL to remaining processes ... ubusd askfirst 
> Switching to ramdisk...
> mount:  /proc is not a block device
> umount: /tmp/root: not mounted
> Failed to switch over to ramfs. Please reboot.
> 
> Rebooting still returned me back to 3.10.11-2
> 
> 2)  MTD
> root@nacktmulle:/tmp# mtd -r write /tmp/firmware.img 
> Usage: mtd [<options> ...] <command> [<arguments> ...] <device>[:<device>...]
> 
> The device is in the format of mtdX (eg: mtd4) or its label.
> mtd recognizes these commands:
>        unlock                  unlock the device
>        refresh                 refresh mtd partition
>        erase                   erase all data on device
>        write <imagefile>|-     write <imagefile> (use - for stdin) to device
>        jffs2write <file>       append <file> to the jffs2 partition on the device
>        fixtrx                  fix the checksum in a trx header on first boot
> Following options are available:
>        -q                      quiet mode (once: no [w] on writing,
>                                           twice: no status messages)
>        -n                      write without first erasing the blocks
>        -r                      reboot after successful command
>        -f                      force write without trx checks
>        -e <device>             erase <device> before executing the command
>        -d <name>               directory for jffs2write, defaults to "tmp"
>        -j <name>               integrate <file> into jffs2 data when writing an image
>        -p                      write beginning at partition offset
>        -o offset               offset of the image header in the partition(for fixtrx)
>        -F <part>[:<size>[:<entrypoint>]][,<part>...]
>                                alter the fis partition table to create new partitions replacing
>                                the partitions provided as argument to the write command
>                                (only valid together with the write command)
> 
> Example: To write linux.trx to mtd4 labeled as linux and reboot afterwards
>         mtd -r write linux.trx linux
> 
> Still no upgrade performed, but at least it is clearer why, my command was incomplete… BUT I seem to recall that it was exactly this command that actually allowed me to install 3.10.11-2 in the first place, weird.
> 
> 3) LUCI (http://gw.home.lan:81/cgi-bin/luci/;stok=19113c7f25269daca52ed92ef4d4b802/admin/system/flashops/)
> I disabled the Keep Settings checkbox, uploaded the image (after making sure /tmp had enough space) followed the "flash image…" link e voila, 3.10.18-1 up and running in no time
> 
> I have no idea what the GUI actually does differently from calling sysupgrade on the command line.
> 
> 
> So the upshot is Juergen Botz is right and the GUI seems to work, at least if one does not keep the old configuration. (And for that problem I followed caves advice and just saved /overlay before upgrading, so I could see the old configuration files and compare.)
> 
> Since I am using cerowrt as secondary router I have no input on the PPPoE issues….
> 
> best
> 	Sebastian
> 
> 
> 
> 
> On Nov 12, 2013, at 22:17 , Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> Hi Richard,
>> 
>> 
>> On Nov 12, 2013, at 18:26 , Richard E. Brown <richb.hanover@gmail.com> wrote:
>> 
>>>> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>> 
>>>>>> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 
>>>>> 
>>>>> 	Just curious, did you specify overhead and encapsulation?
>>> 
>>> No, I simply used the defaults on that page.
>> 
>> 	Ah, you might want to try setting the link layer adaptation mechanism to tc_stab, the link layer to adls or atm and the overhead to 40 (or look at http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ to figure out the fixed per packet overhead of your link). This should allow you to specify a larger percentage of your link rate as shaped rate… But see the attached screenshot of my AQM tab
>> 
>> 
>> <PastedGraphic-1.tiff>
>> 
>>> 
>>>>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>>>>> 
>>>>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>>>>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>>>>> to check what is up with the AQM system...
>>>>> 
>>>>> Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?
>>> 
>>> Yes, I was using QoS in 3.7.5-2, and I kept the old configuration files (so I didn’t have to re-enter credentials for my DSL link, etc.) I guess it seems likely that I may have been running *both* (!)
>> 
>> 	Probably just QOS, as I think both QOS and AQM start out by tearing down all discs… but the hfsc error should not affect AQM since it uses HTB.
>> 
>>> 
>>> Would a better upgrade path be to start with 3.7.5-2, then disable the QoS, then flash with 3.10.18-1? (My intuition tells me that this would remove the QoS settings from the loop…)
>> 
>> 	Mhhh, I guess that should work.
>>> 
>>> It’s pretty easy to re-flash with 3.10.18-1 and run the tc commands. If disabling QoS makes sense, then after I do the ’tc’ experiment, I’ll re-flash with 3.7.5-2, turn off QoS, then reflash to 3.10.18-1.
>> 
>> 	 This should not fix your PPPoE issues though… so maybe it is not the best time to switch…
>> 
>>> But it’ll have to wait ’til dark so no one else is using it. :-)
>> 
>> 	Hihi, same issue on my side :) No "playing" with the network while my wife is using it 
>> 
>> Best
>> 	Sebastian
>> 
>> 
>>> 
>>> Rich
>>> 
>>> 
>> 
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> -- 
> Sandra, Okko, Joris, & Sebastian Moeller
> Telefon: +49 7071 96 49 783, +49 7071 96 49 784, +49 7071 96 49 785
> GSM: +49-1577-190 31 41
> GSM: +49-1517-00 70 355
> 
> Moltkestrasse 6
> 72072 Tuebingen
> Deutschland
> 


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12  9:11 ` Sebastian Moeller
       [not found]   ` <51BF9432-6FC2-4A14-B147-13F1E779CA93@gmail.com>
@ 2013-11-13  4:21   ` Richard E. Brown
  2013-11-13 13:56     ` Sebastian Moeller
  1 sibling, 1 reply; 14+ messages in thread
From: Richard E. Brown @ 2013-11-13  4:21 UTC (permalink / raw)
  To: cerowrt-devel


On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
> 
> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
> to check what is up with the AQM system...

Here’s the output as requested from my router that started out as 3.7.5-2 with QoS enabled, and then was flashed via sysupgrade through the web GUI to 3.10.18-1, preserving the settings. 

Rich
 -----------------------------------------------------

root@cerowrt:~# tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
qdisc fq_codel a: dev se00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
qdisc fq_codel 110: dev ge00 parent 1:10 limit 600p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
qdisc mq 1: dev sw00 root
qdisc fq_codel 10: dev sw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev sw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev sw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev sw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw01 root
qdisc fq_codel 10: dev gw01 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw01 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw01 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw01 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw00 root
qdisc fq_codel 10: dev gw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc hfsc 1: dev ifb0 root refcnt 2 default 30
qdisc fq_codel 100: dev ifb0 parent 1:10 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 200: dev ifb0 parent 1:20 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 300: dev ifb0 parent 1:30 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 400: dev ifb0 parent 1:40 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc mq 1: dev sw10 root
qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw11 root
qdisc fq_codel 10: dev gw11 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw11 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw11 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw11 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw10 root
qdisc fq_codel 10: dev gw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
class hfsc 1: root
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 0 level 2

class hfsc 1:1 parent 1: sc m1 0bit d 0us m2 6000Kbit ul m1 0bit d 0us m2 6000Kbit
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 2 work 260 bytes level 1

class hfsc 1:10 parent 1:1 leaf 100: rt m1 1360Kbit d 325us m2 600000bit ls m1 1360Kbit d 325us m2 3333Kbit ul m1 0bit d 0us m2 6000Kbit
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 0 level 0

class hfsc 1:20 parent 1:1 leaf 200: rt m1 3126Kbit d 325us m2 3000Kbit ls m1 3126Kbit d 325us m2 1666Kbit ul m1 0bit d 0us m2 6000Kbit
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 0 level 0

class hfsc 1:30 parent 1:1 leaf 300: ls m1 0bit d 100.0ms m2 833000bit ul m1 0bit d 0us m2 6000Kbit
 Sent 260 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 2 work 260 bytes level 0

class hfsc 1:40 parent 1:1 leaf 400: ls m1 0bit d 200.0ms m2 166000bit ul m1 0bit d 0us m2 6000Kbit
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 period 0 level 0

class fq_codel 300:22f parent 300:
 (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  deficit 40 count 0 lastcount 0 ldelay 15us
class htb 1:10 parent 1:1 leaf 110: prio 0 rate 650000bit ceil 650000bit burst 1599b cburst 1599b
 Sent 54458 bytes 263 pkt (dropped 0, overlimits 0 requeues 0)
 rate 1984bit 1pps backlog 0b 0p requeues 0
 lended: 263 borrowed: 0 giants: 0
 tokens: 244610 ctokens: 244610

class htb 1:1 root rate 650000bit ceil 650000bit burst 1599b cburst 1599b
 Sent 54458 bytes 263 pkt (dropped 0, overlimits 0 requeues 0)
 rate 1984bit 1pps backlog 0b 0p requeues 0
 lended: 0 borrowed: 0 giants: 0
 tokens: 244610 ctokens: 244610

class fq_codel 110:2e7 parent 110:
 (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  deficit -55 count 0 lastcount 0 ldelay 12us
root@cerowrt:~#

I then reflashed 3.7.5-2, and disabled the QoS setting from the web GUI, and then flashed via sysupgrade through the web GUI to 3.10.18-1, again preserving the settings. I got these results:

root@cerowrt:~# tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
qdisc fq_codel a: dev se00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
qdisc fq_codel 110: dev ge00 parent 1:10 limit 600p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
qdisc htb 1: dev ifb0 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
qdisc fq_codel 110: dev ifb0 parent 1:10 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc mq 1: dev sw00 root
qdisc fq_codel 10: dev sw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev sw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev sw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev sw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw01 root
qdisc fq_codel 10: dev gw01 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw01 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw01 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw01 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc fq_codel a: dev pppoe-ge00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
qdisc mq 1: dev gw00 root
qdisc fq_codel 10: dev gw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev sw10 root
qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw11 root
qdisc fq_codel 10: dev gw11 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw11 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw11 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw11 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
qdisc mq 1: dev gw10 root
qdisc fq_codel 10: dev gw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev gw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev gw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev gw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
class htb 1:10 parent 1:1 leaf 110: prio 0 rate 6000Kbit ceil 6000Kbit burst 1599b cburst 1599b
 Sent 69125382 bytes 210311 pkt (dropped 0, overlimits 0 requeues 0)
 rate 2254Kbit 991pps backlog 0b 0p requeues 0
 lended: 210311 borrowed: 0 giants: 0
 tokens: 4767 ctokens: 4767

class htb 1:1 root rate 6000Kbit ceil 6000Kbit burst 1599b cburst 1599b
 Sent 69125382 bytes 210311 pkt (dropped 0, overlimits 0 requeues 0)
 rate 2254Kbit 991pps backlog 0b 0p requeues 0
 lended: 0 borrowed: 0 giants: 0
 tokens: 4767 ctokens: 4767

class fq_codel 110:292 parent 110:
 (dropped 26, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  deficit -709 count 1 lastcount 1 ldelay 10us
class htb 1:10 parent 1:1 leaf 110: prio 0 rate 650000bit ceil 650000bit burst 1599b cburst 1599b
 Sent 14582488 bytes 192897 pkt (dropped 5, overlimits 0 requeues 0)
 rate 541856bit 909pps backlog 0b 0p requeues 0
 lended: 192897 borrowed: 0 giants: 0
 tokens: 276028 ctokens: 276028

class htb 1:1 root rate 650000bit ceil 650000bit burst 1599b cburst 1599b
 Sent 14582488 bytes 192897 pkt (dropped 0, overlimits 0 requeues 0)
 rate 541856bit 909pps backlog 0b 0p requeues 0
 lended: 0 borrowed: 0 giants: 0
 tokens: 276028 ctokens: 276028

class fq_codel 110:e4 parent 110:
 (dropped 8911, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  deficit 26 count 82 lastcount 49 ldelay 10us
root@cerowrt:~#

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-13  4:21   ` Richard E. Brown
@ 2013-11-13 13:56     ` Sebastian Moeller
  2013-11-13 15:53       ` Richard E. Brown
  0 siblings, 1 reply; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-13 13:56 UTC (permalink / raw)
  To: Richard E. Brown; +Cc: cerowrt-devel

Hi Richard,

thanks for doing this.

On Nov 13, 2013, at 05:21 , "Richard E. Brown" <richb.hanover@gmail.com> wrote:

> 
> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>> 
>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>> to check what is up with the AQM system...
> 
> Here’s the output as requested from my router that started out as 3.7.5-2 with QoS enabled, and then was flashed via sysupgrade through the web GUI to 3.10.18-1, preserving the settings. 
> 
> Rich
> -----------------------------------------------------
> 
> root@cerowrt:~# tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
> qdisc fq_codel a: dev se00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
> qdisc fq_codel 110: dev ge00 parent 1:10 limit 600p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
> qdisc mq 1: dev sw00 root
> qdisc fq_codel 10: dev sw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev sw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev sw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev sw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw01 root
> qdisc fq_codel 10: dev gw01 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw01 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw01 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw01 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw00 root
> qdisc fq_codel 10: dev gw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc hfsc 1: dev ifb0 root refcnt 2 default 30

	And here we have it a "residual" hfsc queueing discipline that somehow  was left over from QOS, this caused the hfsc watchdog errors.


> qdisc fq_codel 100: dev ifb0 parent 1:10 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 200: dev ifb0 parent 1:20 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 300: dev ifb0 parent 1:30 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 400: dev ifb0 parent 1:40 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 1: dev sw10 root
> qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw11 root
> qdisc fq_codel 10: dev gw11 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw11 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw11 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw11 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw10 root
> qdisc fq_codel 10: dev gw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms

	The number of fq_codels on all interfaces are cerowrt's work, as is the htb on ge00

> class hfsc 1: root
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 0 level 2
> 
> class hfsc 1:1 parent 1: sc m1 0bit d 0us m2 6000Kbit ul m1 0bit d 0us m2 6000Kbit
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 2 work 260 bytes level 1
> 
> class hfsc 1:10 parent 1:1 leaf 100: rt m1 1360Kbit d 325us m2 600000bit ls m1 1360Kbit d 325us m2 3333Kbit ul m1 0bit d 0us m2 6000Kbit
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 0 level 0
> 
> class hfsc 1:20 parent 1:1 leaf 200: rt m1 3126Kbit d 325us m2 3000Kbit ls m1 3126Kbit d 325us m2 1666Kbit ul m1 0bit d 0us m2 6000Kbit
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 0 level 0
> 
> class hfsc 1:30 parent 1:1 leaf 300: ls m1 0bit d 100.0ms m2 833000bit ul m1 0bit d 0us m2 6000Kbit
> Sent 260 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 2 work 260 bytes level 0
> 
> class hfsc 1:40 parent 1:1 leaf 400: ls m1 0bit d 200.0ms m2 166000bit ul m1 0bit d 0us m2 6000Kbit
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> period 0 level 0
> 
> class fq_codel 300:22f parent 300:
> (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>  deficit 40 count 0 lastcount 0 ldelay 15us
> class htb 1:10 parent 1:1 leaf 110: prio 0 rate 650000bit ceil 650000bit burst 1599b cburst 1599b
> Sent 54458 bytes 263 pkt (dropped 0, overlimits 0 requeues 0)
> rate 1984bit 1pps backlog 0b 0p requeues 0
> lended: 263 borrowed: 0 giants: 0
> tokens: 244610 ctokens: 244610
> 
> class htb 1:1 root rate 650000bit ceil 650000bit burst 1599b cburst 1599b
> Sent 54458 bytes 263 pkt (dropped 0, overlimits 0 requeues 0)
> rate 1984bit 1pps backlog 0b 0p requeues 0
> lended: 0 borrowed: 0 giants: 0
> tokens: 244610 ctokens: 244610
> 
> class fq_codel 110:2e7 parent 110:
> (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>  deficit -55 count 0 lastcount 0 ldelay 12us
> root@cerowrt:~#

	So as we expected the preserved QOS setting did not work well with cerowrt's defaults.

> 
> I then reflashed 3.7.5-2, and disabled the QoS setting from the web GUI, and then flashed via sysupgrade through the web GUI to 3.10.18-1, again preserving the settings. I got these results:
> 
> root@cerowrt:~# tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
> qdisc fq_codel a: dev se00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
> qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17
> qdisc fq_codel 110: dev ge00 parent 1:10 limit 600p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
> qdisc htb 1: dev ifb0 root refcnt 2 r2q 10 default 10 direct_packets_stat 0 ver 3.17

	And now we also have htb on ifb0, so this is a consistent cerowrt state. I now begin to understand why Dave recommends against using the "keep configuration" setting for sys upgrade :) there is just too much in flux for it to work for all sub packages. That said it should be possible to restrict the set of kept settings to those settings that should not differ between cerowrt and openWRT (I think the PPPoE setting should qualify). But I have not looked into that yet.

> qdisc fq_codel 110: dev ifb0 parent 1:10 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc mq 1: dev sw00 root
> qdisc fq_codel 10: dev sw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev sw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev sw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev sw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw01 root
> qdisc fq_codel 10: dev gw01 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw01 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw01 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw01 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc fq_codel a: dev pppoe-ge00 root refcnt 2 limit 1000p flows 1024 quantum 1000 target 5.0ms interval 100.0ms ecn
> qdisc mq 1: dev gw00 root
> qdisc fq_codel 10: dev gw00 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw00 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw00 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw00 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev sw10 root
> qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw11 root
> qdisc fq_codel 10: dev gw11 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw11 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw11 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw11 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> qdisc mq 1: dev gw10 root
> qdisc fq_codel 10: dev gw10 parent 1:1 limit 800p flows 1024 quantum 500 target 10.0ms interval 100.0ms
> qdisc fq_codel 20: dev gw10 parent 1:2 limit 800p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 30: dev gw10 parent 1:3 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
> qdisc fq_codel 40: dev gw10 parent 1:4 limit 1000p flows 1024 quantum 300 target 5.0ms interval 100.0ms
> class htb 1:10 parent 1:1 leaf 110: prio 0 rate 6000Kbit ceil 6000Kbit burst 1599b cburst 1599b
> Sent 69125382 bytes 210311 pkt (dropped 0, overlimits 0 requeues 0)
> rate 2254Kbit 991pps backlog 0b 0p requeues 0
> lended: 210311 borrowed: 0 giants: 0
> tokens: 4767 ctokens: 4767
> 
> class htb 1:1 root rate 6000Kbit ceil 6000Kbit burst 1599b cburst 1599b
> Sent 69125382 bytes 210311 pkt (dropped 0, overlimits 0 requeues 0)
> rate 2254Kbit 991pps backlog 0b 0p requeues 0
> lended: 0 borrowed: 0 giants: 0
> tokens: 4767 ctokens: 4767
> 
> class fq_codel 110:292 parent 110:
> (dropped 26, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>  deficit -709 count 1 lastcount 1 ldelay 10us
> class htb 1:10 parent 1:1 leaf 110: prio 0 rate 650000bit ceil 650000bit burst 1599b cburst 1599b
> Sent 14582488 bytes 192897 pkt (dropped 5, overlimits 0 requeues 0)
> rate 541856bit 909pps backlog 0b 0p requeues 0
> lended: 192897 borrowed: 0 giants: 0
> tokens: 276028 ctokens: 276028
> 
> class htb 1:1 root rate 650000bit ceil 650000bit burst 1599b cburst 1599b
> Sent 14582488 bytes 192897 pkt (dropped 0, overlimits 0 requeues 0)
> rate 541856bit 909pps backlog 0b 0p requeues 0
> lended: 0 borrowed: 0 giants: 0
> tokens: 276028 ctokens: 276028
> 
> class fq_codel 110:e4 parent 110:
> (dropped 8911, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>  deficit 26 count 82 lastcount 49 ldelay 10us
> root@cerowrt:~#
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-13 13:56     ` Sebastian Moeller
@ 2013-11-13 15:53       ` Richard E. Brown
  0 siblings, 0 replies; 14+ messages in thread
From: Richard E. Brown @ 2013-11-13 15:53 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cerowrt-devel

Hi Sebastian,

Thanks for looking at this. So it seems that keeping the configuration when using sysupgrade caused the problem.

> I now begin to understand why Dave recommends against using the "keep configuration" setting for sys upgrade :) there is just too much in flux for it to work for all sub packages.
> That said it should be possible to restrict the set of kept settings to those settings that should not differ between cerowrt and openWRT (I think the PPPoE setting should qualify). But I have not looked into that yet.

Another (and better, and simpler) alternative would be for me to add the PPPoE credential-setting lines to the existing script on the Files tab of the wiki. That way, I could use sysupgrade back to the factory defaults, then use the script to get consistent settings on my router.

I’ll post when I’ve updated it. Best,

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12  5:47 [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report Richard E. Brown
  2013-11-12  9:11 ` Sebastian Moeller
  2013-11-12  9:40 ` Fred Stratton
@ 2013-11-15  1:56 ` Richard E. Brown
  2013-11-15  2:27   ` David Personette
  2 siblings, 1 reply; 14+ messages in thread
From: Richard E. Brown @ 2013-11-15  1:56 UTC (permalink / raw)
  To: cerowrt-devel

Update on my field report:

I reflashed from 3.7.5-2 to 3.10.18-1 using sysupgrade via the web GUI. I did *not* keep settings.

After letting the router restart and setting up assorted parameters, it seems completely fine. It feels snappier than 3.7.5-2, but I have no objective measurements of this. I did watch ping times during a heavy-ish download, and they hardly changed.

PPPoE comes up as desired on boot (in my earlier experiment with the 3.7.5-2 settings kept around, it didn’t)

mDNS works as desired across interfaces/subnets of this router. 

I’m feeling pretty cocky, so I installed 3.10.18-1 as my primary router. But as my father used to remind me, "Pride cometh before a fall.” So I’m keeping the 3.7.5-2 binary handy.

Good work, everyone!

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-15  1:56 ` Richard E. Brown
@ 2013-11-15  2:27   ` David Personette
  0 siblings, 0 replies; 14+ messages in thread
From: David Personette @ 2013-11-15  2:27 UTC (permalink / raw)
  To: cerowrt-devel

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

Also, while I haven't tested this. After reboots I got an IP a good bit
faster than previous builds (IE without the long timeout in fixdaemons),
it's possible that the dnsmasq (at least) startup issue is fixed in
3.10.18...

-- 
David P.


On Thu, Nov 14, 2013 at 8:56 PM, Richard E. Brown
<richb.hanover@gmail.com>wrote:

> Update on my field report:
>
> I reflashed from 3.7.5-2 to 3.10.18-1 using sysupgrade via the web GUI. I
> did *not* keep settings.
>
> After letting the router restart and setting up assorted parameters, it
> seems completely fine. It feels snappier than 3.7.5-2, but I have no
> objective measurements of this. I did watch ping times during a heavy-ish
> download, and they hardly changed.
>
> PPPoE comes up as desired on boot (in my earlier experiment with the
> 3.7.5-2 settings kept around, it didn’t)
>
> mDNS works as desired across interfaces/subnets of this router.
>
> I’m feeling pretty cocky, so I installed 3.10.18-1 as my primary router.
> But as my father used to remind me, "Pride cometh before a fall.” So I’m
> keeping the 3.7.5-2 binary handy.
>
> Good work, everyone!
>
> Rich
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>

[-- Attachment #2: Type: text/html, Size: 1915 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report
  2013-11-12 23:11           ` Sebastian Moeller
@ 2013-11-15 12:35             ` Sebastian Moeller
  0 siblings, 0 replies; 14+ messages in thread
From: Sebastian Moeller @ 2013-11-15 12:35 UTC (permalink / raw)
  To: Sebastian Moeller; +Cc: cerowrt-devel

Hi All,

small update, I have a hunch that comment 3 of https://dev.openwrt.org/ticket/13958
might be relevant for us:
> additional 2cents: turns out i had mount-utils installed, whose /usr/bin/mount had different output than the busybox one. this broke the function rootfs_type() in /lib/upgrade/common.sh, causing the sysupgrade script to switch something unswitchable...


indeed we have a binary in /usr/bin/mount and:
export PATH=/usr/bin:/usr/sbin:/bin:/sbin
in /etc/profile

So, it might be enough to replace the following in /lib/upgrade/commons.sh

rootfs_type() {
        mount | awk '($3 ~ /^\/$/) && ($5 !~ /rootfs/) { print $5 }'
}

with
rootfs_type() {
        busy box mount | awk '($3 ~ /^\/$/) && ($5 !~ /rootfs/) { print $5 }'
}

to get sys upgrade to work again. 

But I have not tested that (and most likely will not be able to do so before next week); yet I wanted to document this potential fix for the greater public...


Best Regards
	Sebastian


On Nov 13, 2013, at 00:11 , Sebastian Moeller <moeller0@gmx.de> wrote:

> Hi All,
> 
> it turns out that not being able/willing to read can make you do busy work. It seems I forgot to add  "firmware" as device to my mtd invocation… I guess I would never have tried the GUI if I had gotten the mtd command right the first time :)
> 
> 
> best
> 	Sebastian
> 
> 
> On Nov 13, 2013, at 00:06 , Sebastian Moeller <sebastian.moeller@gmail.com> wrote:
> 
>> Hi Richard, hi Dave, hi list,
>> 
>> so I could not resist the lure of 3.10.18-1 and upgraded my 3.10.11-2.; which turned out to be slightly more involved than I had expected.
>> 
>> 1) SYSUPGRADE
>> root@nacktmulle:/# sysupgrade -d 60 -n /home/persistent/cerowrts/3.10.18-1/openwrt-ar71xx-generic-wndr3700v2-squashfs-sysupgrade.bin
>> 
>> killall: watchdog: no process killed
>> Sending TERM to remaining processes ... netifd dynamic_dns_upd sleep minissdpd lighttpd crond lighttpd pimd snmpd xinetd dbus-daemon dnsmasq zebra babeld watchquagga smbd nmbd avahi-daemon ahcpd rngd ntpd ubusd askfirst 
>> Sending KILL to remaining processes ... ubusd askfirst 
>> Switching to ramdisk...
>> mount:  /proc is not a block device
>> umount: /tmp/root: not mounted
>> Failed to switch over to ramfs. Please reboot.
>> 
>> Rebooting still returned me back to 3.10.11-2
>> 
>> 2)  MTD
>> root@nacktmulle:/tmp# mtd -r write /tmp/firmware.img 
>> Usage: mtd [<options> ...] <command> [<arguments> ...] <device>[:<device>...]
>> 
>> The device is in the format of mtdX (eg: mtd4) or its label.
>> mtd recognizes these commands:
>>       unlock                  unlock the device
>>       refresh                 refresh mtd partition
>>       erase                   erase all data on device
>>       write <imagefile>|-     write <imagefile> (use - for stdin) to device
>>       jffs2write <file>       append <file> to the jffs2 partition on the device
>>       fixtrx                  fix the checksum in a trx header on first boot
>> Following options are available:
>>       -q                      quiet mode (once: no [w] on writing,
>>                                          twice: no status messages)
>>       -n                      write without first erasing the blocks
>>       -r                      reboot after successful command
>>       -f                      force write without trx checks
>>       -e <device>             erase <device> before executing the command
>>       -d <name>               directory for jffs2write, defaults to "tmp"
>>       -j <name>               integrate <file> into jffs2 data when writing an image
>>       -p                      write beginning at partition offset
>>       -o offset               offset of the image header in the partition(for fixtrx)
>>       -F <part>[:<size>[:<entrypoint>]][,<part>...]
>>                               alter the fis partition table to create new partitions replacing
>>                               the partitions provided as argument to the write command
>>                               (only valid together with the write command)
>> 
>> Example: To write linux.trx to mtd4 labeled as linux and reboot afterwards
>>        mtd -r write linux.trx linux
>> 
>> Still no upgrade performed, but at least it is clearer why, my command was incomplete… BUT I seem to recall that it was exactly this command that actually allowed me to install 3.10.11-2 in the first place, weird.
>> 
>> 3) LUCI (http://gw.home.lan:81/cgi-bin/luci/;stok=19113c7f25269daca52ed92ef4d4b802/admin/system/flashops/)
>> I disabled the Keep Settings checkbox, uploaded the image (after making sure /tmp had enough space) followed the "flash image…" link e voila, 3.10.18-1 up and running in no time
>> 
>> I have no idea what the GUI actually does differently from calling sysupgrade on the command line.
>> 
>> 
>> So the upshot is Juergen Botz is right and the GUI seems to work, at least if one does not keep the old configuration. (And for that problem I followed caves advice and just saved /overlay before upgrading, so I could see the old configuration files and compare.)
>> 
>> Since I am using cerowrt as secondary router I have no input on the PPPoE issues….
>> 
>> best
>> 	Sebastian
>> 
>> 
>> 
>> 
>> On Nov 12, 2013, at 22:17 , Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>> Hi Richard,
>>> 
>>> 
>>> On Nov 12, 2013, at 18:26 , Richard E. Brown <richb.hanover@gmail.com> wrote:
>>> 
>>>>> On Nov 12, 2013, at 4:11 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>>>>> 
>>>>>>> - Had to enable and set AQM parameters, since they’re saved differently from the QoS settings in the 3.7.5-2 firmware. Set parameters to ~ 90% of link speeds 
>>>>>> 
>>>>>> 	Just curious, did you specify overhead and encapsulation?
>>>> 
>>>> No, I simply used the defaults on that page.
>>> 
>>> 	Ah, you might want to try setting the link layer adaptation mechanism to tc_stab, the link layer to adls or atm and the overhead to 40 (or look at http://ace-host.stuart.id.au/russell/files/tc/tc-atm/ to figure out the fixed per packet overhead of your link). This should allow you to specify a larger percentage of your link rate as shaped rate… But see the attached screenshot of my AQM tab
>>> 
>>> 
>>> <PastedGraphic-1.tiff>
>>> 
>>>> 
>>>>>>> - The kernel.log shows lots of the stack traces below: 2-5 per second on a long-term basis. 
>>>>>> 
>>>>>> 	These look quite weird, the error is a slow patch warning from hfsc_schedule_watchdog . But, hfsc is the queuing discipline used by stock OpenWrt, cerowrt , so far, has only used HTB (last I checked was cerowrt 3.10.11-3). So my guess is that you were running the default QOS system instead (or worse in addition) to cerowrt's. It would be great to see the output of:
>>>>>> tc -d qdisc ; tc -s class show dev ifb0 ; tc -s class show dev ge00
>>>>>> to check what is up with the AQM system...
>>>>>> 
>>>>>> Did you by any change use the QOS tab in 3.7.5-2 instead of running AQM or simple_qos.sh from rc.local/ifup? If so did you direct sys upgrade to keep the old configuration files?
>>>> 
>>>> Yes, I was using QoS in 3.7.5-2, and I kept the old configuration files (so I didn’t have to re-enter credentials for my DSL link, etc.) I guess it seems likely that I may have been running *both* (!)
>>> 
>>> 	Probably just QOS, as I think both QOS and AQM start out by tearing down all discs… but the hfsc error should not affect AQM since it uses HTB.
>>> 
>>>> 
>>>> Would a better upgrade path be to start with 3.7.5-2, then disable the QoS, then flash with 3.10.18-1? (My intuition tells me that this would remove the QoS settings from the loop…)
>>> 
>>> 	Mhhh, I guess that should work.
>>>> 
>>>> It’s pretty easy to re-flash with 3.10.18-1 and run the tc commands. If disabling QoS makes sense, then after I do the ’tc’ experiment, I’ll re-flash with 3.7.5-2, turn off QoS, then reflash to 3.10.18-1.
>>> 
>>> 	 This should not fix your PPPoE issues though… so maybe it is not the best time to switch…
>>> 
>>>> But it’ll have to wait ’til dark so no one else is using it. :-)
>>> 
>>> 	Hihi, same issue on my side :) No "playing" with the network while my wife is using it 
>>> 
>>> Best
>>> 	Sebastian
>>> 
>>> 
>>>> 
>>>> Rich
>>>> 
>>>> 
>>> 
>>> _______________________________________________
>>> Cerowrt-devel mailing list
>>> Cerowrt-devel@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> 
>> -- 
>> Sandra, Okko, Joris, & Sebastian Moeller
>> Telefon: +49 7071 96 49 783, +49 7071 96 49 784, +49 7071 96 49 785
>> GSM: +49-1577-190 31 41
>> GSM: +49-1517-00 70 355
>> 
>> Moltkestrasse 6
>> 72072 Tuebingen
>> Deutschland
>> 
> 


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2013-11-15 12:36 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-12  5:47 [Cerowrt-devel] CeroWrt 3.10.18-1 Field Report Richard E. Brown
2013-11-12  9:11 ` Sebastian Moeller
     [not found]   ` <51BF9432-6FC2-4A14-B147-13F1E779CA93@gmail.com>
2013-11-12 17:26     ` Richard E. Brown
2013-11-12 21:17       ` Sebastian Moeller
2013-11-12 23:06         ` Sebastian Moeller
2013-11-12 23:11           ` Sebastian Moeller
2013-11-15 12:35             ` Sebastian Moeller
2013-11-13  4:21   ` Richard E. Brown
2013-11-13 13:56     ` Sebastian Moeller
2013-11-13 15:53       ` Richard E. Brown
2013-11-12  9:40 ` Fred Stratton
2013-11-12 17:24   ` Richard E. Brown
2013-11-15  1:56 ` Richard E. Brown
2013-11-15  2:27   ` David Personette

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox