From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-x242.google.com (mail-it0-x242.google.com [IPv6:2607:f8b0:4001:c0b::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 414DB3B2A2 for ; Fri, 11 Nov 2016 12:39:10 -0500 (EST) Received: by mail-it0-x242.google.com with SMTP id n68so11989713itn.3 for ; Fri, 11 Nov 2016 09:39:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=YzFmaJer1RyC/t6xV7FuTvRJnjAm69M9i0idUNOvT5I=; b=hdcB7lxpkXxPOQ/WXBcDoDf5OQiY85bZv61zE+kPRB1+t/2QTnFCHb6x7jOdOcwlbX CiP40KzE4YGKkliRH49ZOex5sEchEnjreYoFirSW7KRhc6WKp6dzUIyCrPOelQj045k3 jKzGob0Muv/NH7tg8fj5FZfgFhA7jcDsu0vE3eC1ohMA3I1QeDr9IImBDyYViQWv0VKD 0wa3XT0YtU8wb4soa5lWD+G8LqDExcMSmGlX97l3ci73OeL1MsFhYXuVm1IXZUTZncRl lE7ogqbbT/m6CRdXq/As95d+oM20scsltH9lVKEKaiYHtuFlJ5DD0AM4nJhw8Ni3/zQV dopg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=YzFmaJer1RyC/t6xV7FuTvRJnjAm69M9i0idUNOvT5I=; b=hhqPbErOPKaf6tpDmC4BWqmXgppXGj+XwqVanp0UZCaJrf8OlXASaTBomtFIiHkTa4 3Fn1y9jIh6chxAZbr7lfcJoU4suvKbO4SkoWP6lRZxQEviYA4COkWwn6oULx5GbtDhMv Iqc6k/2LFyxb/R+7VNodh1SuAnuWFcLLSYSJi8FCMboOhFtOj1oNZSK8ps8VrEZm9tSf MSltazgl/oqyT55xLZu9qGm5Y3u6Ro7q3n0SuZ6BrxbJRtpipBzb0Vaq0rjc8xPSjWPB BjBPMT8KYDG45S7ik706KXn6PPkaCFQeyVAVmFoe5+IYfV0EdrXHTiPqVL8UlOu+GDqI BmCQ== X-Gm-Message-State: ABUngveWdbr5cWbPemEo+Y99EML13X2/ZNwBmm5bw/X1XWGfZjux/H3lr2KmuHKhHxfdwu/ju2KCK/dp+OL4Sw== X-Received: by 10.36.237.3 with SMTP id r3mr9422196ith.76.1478885949346; Fri, 11 Nov 2016 09:39:09 -0800 (PST) MIME-Version: 1.0 Received: by 10.36.22.80 with HTTP; Fri, 11 Nov 2016 09:38:28 -0800 (PST) In-Reply-To: References: <3e5942d2-6d6b-0e01-8aa6-98c3535c26ef@gmail.com> <46c0133b-b6f8-fe1f-4d2e-0cf6088e024d@gmail.com> From: Outback Dingo Date: Sat, 12 Nov 2016 01:38:28 +0800 Message-ID: To: Noah Causin Cc: cake@lists.bufferbloat.net Content-Type: text/plain; charset=UTF-8 Subject: Re: [Cake] Cake with Deep Packet Inspection X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2016 17:39:10 -0000 On Fri, Nov 11, 2016 at 10:56 PM, Outback Dingo wrote: > On Fri, Nov 11, 2016 at 10:22 PM, Noah Causin wrote: >> What I would try is deleting your build source folder and start from >> scratch. >> >> Then edit source/package/kernel/kmod-sched-cake/Makefile >> >> change the PKG_SOURCE_VERSION to 70169dba14daa6ef7907af9c1e922ef9f797993a >> >> That's the latest stable version of cake. >> >> https://github.com/dtaht/sch_cake/commit/70169dba14daa6ef7907af9c1e922ef9f797993a >> >> > > Appreciate the attempt to help, but it did this even prior to adding > cake previously when just using loading sqm > ill give it a go but i believe the issue is deeper then just cake. like I stated made the changes and still seeing issues [ 149.490000] INFO: rcu_sched detected stalls on CPUs/tasks: [ 149.490000] 3-...: (0 ticks this GP) idle=b7c/0/0 softirq=3733/3733 fqs=0 [ 149.500000] (detected by 1, t=6003 jiffies, g=1039, c=1038, q=10655) [ 149.500000] Task dump for CPU 3: [ 149.510000] swapper/3 R running 0 0 1 0x00100000 [ 149.510000] Stack : 00000000 00005ac7 8e049880 7ffdecec 00000000 80418a90 8045a2a4 80410000 [ 149.510000] 8041275c 00000001 00000001 80412680 80412724 80410000 00000000 800132e8 [ 149.510000] 00000000 00000001 8fc64000 8fc65ec0 80410000 8005c870 1100fc03 00000003 [ 149.510000] 00000000 80410000 8045a2a4 8005c868 80410000 8001ade0 1100fc03 00000000 [ 149.510000] 00000004 804124a0 000000a0 8001ade8 fefffbdb aefffbfb fff77fe7 dfffffef [ 149.510000] ... [ 149.550000] Call Trace: [ 149.550000] [<8000bba8>] __schedule+0x574/0x758 [ 149.560000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 [ 149.560000] [ 149.560000] rcu_sched kthread starved for 6009 jiffies! g1039 c1038 f0x0 s3 ->state=0x1 [ 1326.670000] INFO: rcu_sched detected stalls on CPUs/tasks: [ 1326.670000] 2-...: (0 ticks this GP) idle=998/0/0 softirq=56566/56566 fqs=0 [ 1326.680000] (detected by 1, t=6003 jiffies, g=14603, c=14602, q=9455) [ 1326.680000] Task dump for CPU 2: [ 1326.690000] swapper/2 R running 0 0 1 0x00100000 [ 1326.690000] Stack : 00000000 00000001 0000000a 00000000 00000000 00000001 8045a2a4 80410000 [ 1326.690000] 8041275c 00000001 00000000 804125e0 80412724 80410000 00000000 800132e8 [ 1326.690000] 1100fc03 00000002 8fc62000 8fc63ec0 80410000 8005c870 1100fc03 00000002 [ 1326.690000] 00000000 80410000 8045a2a4 8005c868 80410000 8001ade0 1100fc03 00000000 [ 1326.690000] 00000004 804124a0 000000a0 8001ade8 fbffbfdf 7fff7b3b 7ffaeff8 67fbffff [ 1326.690000] ... [ 1326.730000] Call Trace: [ 1326.730000] [<8000bba8>] __schedule+0x574/0x758 [ 1326.740000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 [ 1326.740000] [ 1326.740000] rcu_sched kthread starved for 6009 jiffies! g14603 c14602 f0x0 s3 ->state=0x1 [ 1392.200000] INFO: rcu_sched detected stalls on CPUs/tasks: [ 1392.200000] 2-...: (0 ticks this GP) idle=c80/0/0 softirq=56566/56566 fqs=0 [ 1392.210000] (detected by 3, t=6003 jiffies, g=14669, c=14668, q=11973) [ 1392.210000] Task dump for CPU 2: [ 1392.220000] swapper/2 R running 0 0 1 0x00100000 [ 1392.220000] Stack : 00000000 4e88ab25 00000148 ffffffff 0000011d 773206f0 8045a2a4 80410000 [ 1392.220000] 8041275c 00000001 00000000 804125e0 80412724 80410000 000010d9 800132e8 [ 1392.220000] 1100fc03 00000002 8fc62000 8fc63ec0 80410000 8005c870 1100fc03 00000002 [ 1392.220000] 00000000 80410000 8045a2a4 8005c868 80410000 8001ade0 1100fc03 00000000 [ 1392.220000] 00000004 804124a0 000000a0 8001ade8 fbffbfdf 7fff7b3b 7ffaeff8 67fbffff [ 1392.220000] ... [ 1392.260000] Call Trace: [ 1392.260000] [<8000bba8>] __schedule+0x574/0x758 [ 1392.270000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 [ 1392.270000] [ 1392.270000] rcu_sched kthread starved for 6009 jiffies! g14669 c14668 f0x0 s3 ->state=0x1 > > >> >> On 11/11/2016 9:08 AM, Outback Dingo wrote: >>> >>> On Fri, Nov 11, 2016 at 9:04 PM, Noah Causin >>> wrote: >>>> >>>> Do you build your firmware from the git repository? >>>> >>>> >>> Yes it is built from a recent trunk, as of yesterday >>> >>> >>>> On 11/10/2016 11:28 PM, Outback Dingo wrote: >>>>> >>>>> On Fri, Nov 11, 2016 at 10:52 AM, Outback Dingo >>>>> wrote: >>>>>> >>>>>> On Fri, Nov 11, 2016 at 10:42 AM, Noah Causin >>>>>> wrote: >>>>>>> >>>>>>> I took an extra router I had and bridged two ports on the router's >>>>>>> switch, >>>>>>> so they just pass traffic. (eth0.2 and eth0.3) >>>>>>> >>>>>>> Clients >-[Main Router] --[Extra Router]--[Cable-Modem] >>>>>>> >>>>>>> The extra router is passive. It acts like it's part of the Ethernet >>>>>>> cable >>>>>>> between the main router and cable modem. It does not interfere. >>>>>>> >>>>>>> The extra router needs these packages: >>>>>>> >>>>>>> kmod-ebtables, kmod-ebtables-ipv4, kmod-crypto-pcompress and the two >>>>>>> packages compiled from the Makefile I showed below (iptables-mod-ndpi >>>>>>> and >>>>>>> iptables-mod-ndpi) >>>>>>> >>>>>>> I edited /etc/sysctl.conf and change the last two lines to this: >>>>>>> >>>>>>> net.bridge.bridge-nf-call-ip6tables=1 >>>>>>> net.bridge.bridge-nf-call-iptables=1 >>>>>>> >>>>>>> This enables bridge firewalling, so the traffic between the two ports >>>>>>> can be >>>>>>> marked. >>>>>>> >>>>>>> I then added firewall rules to LuCIs custom firewall rules tab. >>>>>>> >>>>>>> Example Rules, modify classes as desired: >>>>>>> >>>>>>> iptables -t mangle -A FORWARD -m ndpi --steam -m mac ! --mac-source >>>>>>> Your >>>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs0 >>>>>>> iptables -t mangle -A FORWARD -m ndpi --youtube -m mac ! --mac-source >>>>>>> Your >>>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs2 >>>>>>> >>>>>>> iptables -t mangle -A FORWARD -m ndpi --netflix -m mac ! --mac-source >>>>>>> Your >>>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs3 >>>>>>> iptables -t mangle -A FORWARD -m ndpi --skype -m mac ! --mac-source >>>>>>> Your >>>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs4 >>>>>>> >>>>>>> Have your main router use some form of DiffServ for both upload and >>>>>>> download >>>>>>> on its WAN interface. For upload traffic, you just need the two >>>>>>> packages >>>>>>> from the makefile to be installed on the main router and create >>>>>>> firewall >>>>>>> rules like this: >>>>>>> >>>>>>> iptables -t mangle -A FORWARD -o eth2 -m ndpi --netflix -j DSCP >>>>>>> --set-dscp-class cs3 >>>>>>> iptables -t mangle -A FORWARD -o eth2 -m ndpi --skype -j DSCP >>>>>>> --set-dscp-class cs4 >>>>>>> >>>>>>> If you need help building a custom firmware image, just let me know. >>>>>>> >>>>>> Ok got it, its in-line.... right now im fighting the GFW of China >>>>>> using shadowsocks and chinadns so im pretty customized already, >>>>>> i was having issues with sqm and kernel segfaults on 4.4.30 ill build >>>>>> a cake and DPI image and see how that goes, maybe Ill try this >>>>>> after i deem sqm is stable with cake on this build. It a ZBT-WG3526 >>>>>> router... so its ralink based >>>>> >>>>> like i stated on LEDE major issues when sqm is enabled.... even with >>>>> cake... is nobody else seeing these issues? >>>>> >>>>> [ 569.020000] 1-...: (6 GPs behind) idle=236/0/0 softirq=50607/50613 >>>>> fqs=1 >>>>> [ 569.020000] (detected by 0, t=6002 jiffies, g=5799, c=5798, q=573) >>>>> [ 569.020000] Task dump for CPU 1: >>>>> [ 569.020000] swapper/1 R running 0 0 1 0x00100000 >>>>> [ 569.020000] Stack : 00000000 00000001 00000015 00000000 00000000 >>>>> 00000001 8045a2a4 80410000 >>>>> [ 569.020000] 8041275c 00000001 00000001 80412540 80412724 >>>>> 80410000 00000000 800132e8 >>>>> [ 569.020000] 1100fc03 00000001 8fc60000 8fc61ec0 80410000 >>>>> 8005c870 1100fc03 00000001 >>>>> [ 569.020000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>>> 8001ade0 1100fc03 00000000 >>>>> [ 569.020000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>>> aefffbff fff77fe7 dfffffef >>>>> [ 569.020000] ... >>>>> [ 569.020000] Call Trace: >>>>> [ 569.020000] [<8000bba8>] __schedule+0x574/0x758 >>>>> [ 569.020000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>>> [ 569.020000] >>>>> [ 569.020000] rcu_sched kthread starved for 6001 jiffies! g5799 c5798 >>>>> f0x0 s3 ->state=0x1 >>>>> [ 629.130000] INFO: rcu_sched detected stalls on CPUs/tasks: >>>>> [ 629.130000] 1-...: (8 GPs behind) idle=756/0/0 softirq=50607/50613 >>>>> fqs=1 >>>>> [ 629.130000] (detected by 0, t=6002 jiffies, g=5801, c=5800, q=653) >>>>> [ 629.130000] Task dump for CPU 1: >>>>> [ 629.130000] swapper/1 R running 0 0 1 0x00100000 >>>>> [ 629.130000] Stack : 00000000 bfd47c67 00000094 ffffffff 00000081 >>>>> 00000011 8045a2a4 80410000 >>>>> [ 629.130000] 8041275c 00000001 00000001 80412540 80412724 >>>>> 80410000 000010d9 800132e8 >>>>> [ 629.130000] 1100fc03 00000001 8fc60000 8fc61ec0 80410000 >>>>> 8005c870 1100fc03 00000001 >>>>> [ 629.130000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>>> 8001ade0 1100fc03 00000000 >>>>> [ 629.130000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>>> aefffbff fff77fe7 dfffffef >>>>> [ 629.130000] ... >>>>> [ 629.130000] Call Trace: >>>>> [ 629.130000] [<8000bba8>] __schedule+0x574/0x758 >>>>> [ 629.130000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>>> [ 629.130000] >>>>> [ 629.130000] rcu_sched kthread starved for 6001 jiffies! g5801 c5800 >>>>> f0x0 s3 ->state=0x1 >>>>> [ 692.940000] INFO: rcu_sched detected stalls on CPUs/tasks: >>>>> [ 692.940000] 1-...: (0 ticks this GP) idle=4c2/0/0 >>>>> softirq=50613/50613 fqs=0 >>>>> [ 692.940000] 2-...: (25 GPs behind) idle=55a/0/0 softirq=28459/28460 >>>>> fqs=0 >>>>> [ 692.940000] 3-...: (25 GPs behind) idle=5c2/0/0 softirq=30711/30982 >>>>> fqs=0 >>>>> [ 692.940000] (detected by 0, t=6002 jiffies, g=5824, c=5823, q=3190) >>>>> [ 692.940000] Task dump for CPU 1: >>>>> [ 692.940000] swapper/1 R running 0 0 1 0x00100000 >>>>> [ 692.940000] Stack : 00000000 3646fb94 000000af ffffffff 0000008f >>>>> d2f1a9fc 8045a2a4 80410000 >>>>> [ 692.940000] 8041275c 00000001 00000001 80412540 80412724 >>>>> 80410000 000010d9 800132e8 >>>>> [ 692.940000] 00000000 00000001 8fc60000 8fc61ec0 80410000 >>>>> 8005c870 1100fc03 00000001 >>>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>>> 8001ade0 1100fc03 00000000 >>>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>>> aefffbff fff77fe7 dfffffef >>>>> [ 692.940000] ... >>>>> [ 692.940000] Call Trace: >>>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>>> [ 692.940000] >>>>> [ 692.940000] Task dump for CPU 2: >>>>> [ 692.940000] swapper/2 R running 0 0 1 0x00100000 >>>>> [ 692.940000] Stack : 00000000 00000001 00000010 00000000 00000000 >>>>> 00010001 8045a2a4 80410000 >>>>> [ 692.940000] 8041275c 00000001 00000000 804125e0 80412724 >>>>> 80410000 00000000 800132e8 >>>>> [ 692.940000] 1100fc03 00000002 8fc62000 8fc63ec0 80410000 >>>>> 8005c870 1100fc03 00000002 >>>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>>> 8001ade0 1100fc03 00000000 >>>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fbffbfdf >>>>> 7fff7b3b 7ffaeff8 67fbffff >>>>> [ 692.940000] ... >>>>> [ 692.940000] Call Trace: >>>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>>> [ 692.940000] >>>>> [ 692.940000] Task dump for CPU 3: >>>>> [ 692.940000] swapper/3 R running 0 0 1 0x00100000 >>>>> [ 692.940000] Stack : 00000000 a69c5765 000000a3 ffffffff 0000008f >>>>> 00000020 8045a2a4 80410000 >>>>> [ 692.940000] 8041275c 00000001 00000001 80412680 80412724 >>>>> 80410000 000010d9 800132e8 >>>>> [ 692.940000] 1100fc03 00000003 8fc64000 8fc65ec0 80410000 >>>>> 8005c870 1100fc03 00000003 >>>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>>> 8001ade0 1100fc03 00000000 >>>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fefffbdb >>>>> aefffbfb fff77fe7 dfffffef >>>>> [ 692.940000] ... >>>>> [ 692.940000] Call Trace: >>>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>>> [ 692.940000] >>>>> [ 692.940000] rcu_sched kthread starved for 6002 jiffies! g5824 c5823 >>>>> f0x0 s3 ->state=0x1 >>>>> root@lede:~# packet_write_wait: Connection to 192.168.20.1 port 22: >>>>> Broken >>>>> pipe >>>>> >>>>> >>>>>>> On 11/10/2016 9:06 PM, Outback Dingo wrote: >>>>>>> >>>>>>> On Fri, Nov 11, 2016 at 9:55 AM, Noah Causin >>>>>>> wrote: >>>>>>> >>>>>>> I finally got my project working. >>>>>>> >>>>>>> I integrated the NDPI Deep Packet Inspection engine into my LEDE >>>>>>> build, >>>>>>> so I >>>>>>> could prioritize applications using Cake. >>>>>>> >>>>>>> http://www.ntop.org/products/deep-packet-inspection/ndpi/ >>>>>>> >>>>>>> NDPI integrates into IPTables, which allows me to DSCP mark packets. >>>>>>> Cake >>>>>>> reads the DiffServ markings and puts the traffic into appropriate >>>>>>> classes. >>>>>>> >>>>>>> I found a Makefile which successfully compiles: >>>>>>> >>>>>>> >>>>>>> https://github.com/981213/lede_src/blob/0d344bc2958838dcbc547a8f0a3d8842e6f6d2f8/package/my_package/ndpi-netfilter/Makefile >>>>>>> >>>>>>> The system works very well. Steam traffic is deprioritized to allow >>>>>>> applications like YouTube, Netflix, and Skype to receive higher >>>>>>> amounts >>>>>>> of >>>>>>> the available bandwidth. >>>>>>> >>>>>>> What I do for ingress is bridge two ports on an extra router, enable >>>>>>> bridge >>>>>>> firewalling, and create IPTables rules to mark downstream packets. >>>>>>> The >>>>>>> router I use is a D-Link DGL-5500, which is comparable to an Archer >>>>>>> C7. >>>>>>> >>>>>>> IPv6 support is not available in this netfilter module, but the IPv4 >>>>>>> support >>>>>>> is great. >>>>>>> >>>>>>> Noah Causin >>>>>>> >>>>>>> very nice.... however, can you describe a bit better how you >>>>>>> configured for ingress ? a second router? configuration file ? your >>>>>>> high level description seems a bit confusing to me >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Cake mailing list >>>>>>> Cake@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/cake >>>>>>> >>>>>>> >>