From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-x243.google.com (mail-it0-x243.google.com [IPv6:2607:f8b0:4001:c0b::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EF9E03B2A2 for ; Fri, 11 Nov 2016 09:57:32 -0500 (EST) Received: by mail-it0-x243.google.com with SMTP id n68so11064697itn.3 for ; Fri, 11 Nov 2016 06:57:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=K4gDqhpiUf8aT6760dhPQvxgtCLh4JAofywoAYH7yZU=; b=Ey5WdVN2qs0ZTwZ+dR/r015c9VcwQ3YOwap6bmzUeT35wBKaONUpT9jYwo+kf/PoAC XwE0uahtGcjKLFjU02U/RgjXFiSIwecwHgGXo6EUVGQ5EIXgQbPQBaImXZb25ldrHCRA r2W0WumEB12yMCrB+9c8LelxQpdbO8hEBhqMh7Jua8oqNVYzqKAUxOT7Yxf+jsRCEs8f v9P8m5EMFP4QM3EqJAa8G47IRDPAQ4uVMuP3A6H5VXmaT52JHqymbgpGe7+iwDEqFiAe v7ClpeKXvD+oIWwZZQRWlEC7MipC6ZcpJAB4CIWZXxMK+Ye59EMNhi3kGaL0oFksKEsd zKvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=K4gDqhpiUf8aT6760dhPQvxgtCLh4JAofywoAYH7yZU=; b=de1RQV7gdz2+J3FOqhxzfnSMDbFlLf8T2065N638Yfnap0q8R1hjDy5wtZAbwFjlfd iMdeq+Pdegqq/YUfiotEZRO8kuokd4t8z+bcjGYN4SNe6HGE8uLI08CiEmWn15TI9/xq YbgPTZPKUyXvwIrVjRuWrKbBpqiF9w2SVDx/elxOPgEiAO52QX6vId7wgJN/2Wg2bjpM WMhQoLpNeyuRsiu+lcQTFhNsTtLHfBEkoAquRl+009U+x6ERkPahBEz/IHkoGkZ4uQLx 9WRbN69/hCY4b6HC7QCsXRgTZQSjeXo97+YeV62ahcibk63cYaq3/NeJYH02Mn8LXW5J opKg== X-Gm-Message-State: ABUngveQUrLu1VWI/0y5S3jhUhsAWFYhxS3rTvayVanma2tuUbnneQoqcDA4RwFSTJscLV3Cxa+82qEiHUAhMw== X-Received: by 10.36.90.19 with SMTP id v19mr21541895ita.91.1478876252239; Fri, 11 Nov 2016 06:57:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.36.22.80 with HTTP; Fri, 11 Nov 2016 06:56:51 -0800 (PST) In-Reply-To: References: <3e5942d2-6d6b-0e01-8aa6-98c3535c26ef@gmail.com> <46c0133b-b6f8-fe1f-4d2e-0cf6088e024d@gmail.com> From: Outback Dingo Date: Fri, 11 Nov 2016 22:56:51 +0800 Message-ID: To: Noah Causin Cc: cake@lists.bufferbloat.net Content-Type: text/plain; charset=UTF-8 Subject: Re: [Cake] Cake with Deep Packet Inspection X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Nov 2016 14:57:33 -0000 On Fri, Nov 11, 2016 at 10:22 PM, Noah Causin wrote: > What I would try is deleting your build source folder and start from > scratch. > > Then edit source/package/kernel/kmod-sched-cake/Makefile > > change the PKG_SOURCE_VERSION to 70169dba14daa6ef7907af9c1e922ef9f797993a > > That's the latest stable version of cake. > > https://github.com/dtaht/sch_cake/commit/70169dba14daa6ef7907af9c1e922ef9f797993a > > Appreciate the attempt to help, but it did this even prior to adding cake previously when just using loading sqm ill give it a go but i believe the issue is deeper then just cake. > > On 11/11/2016 9:08 AM, Outback Dingo wrote: >> >> On Fri, Nov 11, 2016 at 9:04 PM, Noah Causin >> wrote: >>> >>> Do you build your firmware from the git repository? >>> >>> >> Yes it is built from a recent trunk, as of yesterday >> >> >>> On 11/10/2016 11:28 PM, Outback Dingo wrote: >>>> >>>> On Fri, Nov 11, 2016 at 10:52 AM, Outback Dingo >>>> wrote: >>>>> >>>>> On Fri, Nov 11, 2016 at 10:42 AM, Noah Causin >>>>> wrote: >>>>>> >>>>>> I took an extra router I had and bridged two ports on the router's >>>>>> switch, >>>>>> so they just pass traffic. (eth0.2 and eth0.3) >>>>>> >>>>>> Clients >-[Main Router] --[Extra Router]--[Cable-Modem] >>>>>> >>>>>> The extra router is passive. It acts like it's part of the Ethernet >>>>>> cable >>>>>> between the main router and cable modem. It does not interfere. >>>>>> >>>>>> The extra router needs these packages: >>>>>> >>>>>> kmod-ebtables, kmod-ebtables-ipv4, kmod-crypto-pcompress and the two >>>>>> packages compiled from the Makefile I showed below (iptables-mod-ndpi >>>>>> and >>>>>> iptables-mod-ndpi) >>>>>> >>>>>> I edited /etc/sysctl.conf and change the last two lines to this: >>>>>> >>>>>> net.bridge.bridge-nf-call-ip6tables=1 >>>>>> net.bridge.bridge-nf-call-iptables=1 >>>>>> >>>>>> This enables bridge firewalling, so the traffic between the two ports >>>>>> can be >>>>>> marked. >>>>>> >>>>>> I then added firewall rules to LuCIs custom firewall rules tab. >>>>>> >>>>>> Example Rules, modify classes as desired: >>>>>> >>>>>> iptables -t mangle -A FORWARD -m ndpi --steam -m mac ! --mac-source >>>>>> Your >>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs0 >>>>>> iptables -t mangle -A FORWARD -m ndpi --youtube -m mac ! --mac-source >>>>>> Your >>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs2 >>>>>> >>>>>> iptables -t mangle -A FORWARD -m ndpi --netflix -m mac ! --mac-source >>>>>> Your >>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs3 >>>>>> iptables -t mangle -A FORWARD -m ndpi --skype -m mac ! --mac-source >>>>>> Your >>>>>> Main Router's Mac Address -j DSCP --set-dscp-class cs4 >>>>>> >>>>>> Have your main router use some form of DiffServ for both upload and >>>>>> download >>>>>> on its WAN interface. For upload traffic, you just need the two >>>>>> packages >>>>>> from the makefile to be installed on the main router and create >>>>>> firewall >>>>>> rules like this: >>>>>> >>>>>> iptables -t mangle -A FORWARD -o eth2 -m ndpi --netflix -j DSCP >>>>>> --set-dscp-class cs3 >>>>>> iptables -t mangle -A FORWARD -o eth2 -m ndpi --skype -j DSCP >>>>>> --set-dscp-class cs4 >>>>>> >>>>>> If you need help building a custom firmware image, just let me know. >>>>>> >>>>> Ok got it, its in-line.... right now im fighting the GFW of China >>>>> using shadowsocks and chinadns so im pretty customized already, >>>>> i was having issues with sqm and kernel segfaults on 4.4.30 ill build >>>>> a cake and DPI image and see how that goes, maybe Ill try this >>>>> after i deem sqm is stable with cake on this build. It a ZBT-WG3526 >>>>> router... so its ralink based >>>> >>>> like i stated on LEDE major issues when sqm is enabled.... even with >>>> cake... is nobody else seeing these issues? >>>> >>>> [ 569.020000] 1-...: (6 GPs behind) idle=236/0/0 softirq=50607/50613 >>>> fqs=1 >>>> [ 569.020000] (detected by 0, t=6002 jiffies, g=5799, c=5798, q=573) >>>> [ 569.020000] Task dump for CPU 1: >>>> [ 569.020000] swapper/1 R running 0 0 1 0x00100000 >>>> [ 569.020000] Stack : 00000000 00000001 00000015 00000000 00000000 >>>> 00000001 8045a2a4 80410000 >>>> [ 569.020000] 8041275c 00000001 00000001 80412540 80412724 >>>> 80410000 00000000 800132e8 >>>> [ 569.020000] 1100fc03 00000001 8fc60000 8fc61ec0 80410000 >>>> 8005c870 1100fc03 00000001 >>>> [ 569.020000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>> 8001ade0 1100fc03 00000000 >>>> [ 569.020000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>> aefffbff fff77fe7 dfffffef >>>> [ 569.020000] ... >>>> [ 569.020000] Call Trace: >>>> [ 569.020000] [<8000bba8>] __schedule+0x574/0x758 >>>> [ 569.020000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>> [ 569.020000] >>>> [ 569.020000] rcu_sched kthread starved for 6001 jiffies! g5799 c5798 >>>> f0x0 s3 ->state=0x1 >>>> [ 629.130000] INFO: rcu_sched detected stalls on CPUs/tasks: >>>> [ 629.130000] 1-...: (8 GPs behind) idle=756/0/0 softirq=50607/50613 >>>> fqs=1 >>>> [ 629.130000] (detected by 0, t=6002 jiffies, g=5801, c=5800, q=653) >>>> [ 629.130000] Task dump for CPU 1: >>>> [ 629.130000] swapper/1 R running 0 0 1 0x00100000 >>>> [ 629.130000] Stack : 00000000 bfd47c67 00000094 ffffffff 00000081 >>>> 00000011 8045a2a4 80410000 >>>> [ 629.130000] 8041275c 00000001 00000001 80412540 80412724 >>>> 80410000 000010d9 800132e8 >>>> [ 629.130000] 1100fc03 00000001 8fc60000 8fc61ec0 80410000 >>>> 8005c870 1100fc03 00000001 >>>> [ 629.130000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>> 8001ade0 1100fc03 00000000 >>>> [ 629.130000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>> aefffbff fff77fe7 dfffffef >>>> [ 629.130000] ... >>>> [ 629.130000] Call Trace: >>>> [ 629.130000] [<8000bba8>] __schedule+0x574/0x758 >>>> [ 629.130000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>> [ 629.130000] >>>> [ 629.130000] rcu_sched kthread starved for 6001 jiffies! g5801 c5800 >>>> f0x0 s3 ->state=0x1 >>>> [ 692.940000] INFO: rcu_sched detected stalls on CPUs/tasks: >>>> [ 692.940000] 1-...: (0 ticks this GP) idle=4c2/0/0 >>>> softirq=50613/50613 fqs=0 >>>> [ 692.940000] 2-...: (25 GPs behind) idle=55a/0/0 softirq=28459/28460 >>>> fqs=0 >>>> [ 692.940000] 3-...: (25 GPs behind) idle=5c2/0/0 softirq=30711/30982 >>>> fqs=0 >>>> [ 692.940000] (detected by 0, t=6002 jiffies, g=5824, c=5823, q=3190) >>>> [ 692.940000] Task dump for CPU 1: >>>> [ 692.940000] swapper/1 R running 0 0 1 0x00100000 >>>> [ 692.940000] Stack : 00000000 3646fb94 000000af ffffffff 0000008f >>>> d2f1a9fc 8045a2a4 80410000 >>>> [ 692.940000] 8041275c 00000001 00000001 80412540 80412724 >>>> 80410000 000010d9 800132e8 >>>> [ 692.940000] 00000000 00000001 8fc60000 8fc61ec0 80410000 >>>> 8005c870 1100fc03 00000001 >>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>> 8001ade0 1100fc03 00000000 >>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fe9ff9db >>>> aefffbff fff77fe7 dfffffef >>>> [ 692.940000] ... >>>> [ 692.940000] Call Trace: >>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>> [ 692.940000] >>>> [ 692.940000] Task dump for CPU 2: >>>> [ 692.940000] swapper/2 R running 0 0 1 0x00100000 >>>> [ 692.940000] Stack : 00000000 00000001 00000010 00000000 00000000 >>>> 00010001 8045a2a4 80410000 >>>> [ 692.940000] 8041275c 00000001 00000000 804125e0 80412724 >>>> 80410000 00000000 800132e8 >>>> [ 692.940000] 1100fc03 00000002 8fc62000 8fc63ec0 80410000 >>>> 8005c870 1100fc03 00000002 >>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>> 8001ade0 1100fc03 00000000 >>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fbffbfdf >>>> 7fff7b3b 7ffaeff8 67fbffff >>>> [ 692.940000] ... >>>> [ 692.940000] Call Trace: >>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>> [ 692.940000] >>>> [ 692.940000] Task dump for CPU 3: >>>> [ 692.940000] swapper/3 R running 0 0 1 0x00100000 >>>> [ 692.940000] Stack : 00000000 a69c5765 000000a3 ffffffff 0000008f >>>> 00000020 8045a2a4 80410000 >>>> [ 692.940000] 8041275c 00000001 00000001 80412680 80412724 >>>> 80410000 000010d9 800132e8 >>>> [ 692.940000] 1100fc03 00000003 8fc64000 8fc65ec0 80410000 >>>> 8005c870 1100fc03 00000003 >>>> [ 692.940000] 00000000 80410000 8045a2a4 8005c868 80410000 >>>> 8001ade0 1100fc03 00000000 >>>> [ 692.940000] 00000004 804124a0 000000a0 8001ade8 fefffbdb >>>> aefffbfb fff77fe7 dfffffef >>>> [ 692.940000] ... >>>> [ 692.940000] Call Trace: >>>> [ 692.940000] [<8000bba8>] __schedule+0x574/0x758 >>>> [ 692.940000] [<800132e8>] r4k_wait_irqoff+0x0/0x20 >>>> [ 692.940000] >>>> [ 692.940000] rcu_sched kthread starved for 6002 jiffies! g5824 c5823 >>>> f0x0 s3 ->state=0x1 >>>> root@lede:~# packet_write_wait: Connection to 192.168.20.1 port 22: >>>> Broken >>>> pipe >>>> >>>> >>>>>> On 11/10/2016 9:06 PM, Outback Dingo wrote: >>>>>> >>>>>> On Fri, Nov 11, 2016 at 9:55 AM, Noah Causin >>>>>> wrote: >>>>>> >>>>>> I finally got my project working. >>>>>> >>>>>> I integrated the NDPI Deep Packet Inspection engine into my LEDE >>>>>> build, >>>>>> so I >>>>>> could prioritize applications using Cake. >>>>>> >>>>>> http://www.ntop.org/products/deep-packet-inspection/ndpi/ >>>>>> >>>>>> NDPI integrates into IPTables, which allows me to DSCP mark packets. >>>>>> Cake >>>>>> reads the DiffServ markings and puts the traffic into appropriate >>>>>> classes. >>>>>> >>>>>> I found a Makefile which successfully compiles: >>>>>> >>>>>> >>>>>> https://github.com/981213/lede_src/blob/0d344bc2958838dcbc547a8f0a3d8842e6f6d2f8/package/my_package/ndpi-netfilter/Makefile >>>>>> >>>>>> The system works very well. Steam traffic is deprioritized to allow >>>>>> applications like YouTube, Netflix, and Skype to receive higher >>>>>> amounts >>>>>> of >>>>>> the available bandwidth. >>>>>> >>>>>> What I do for ingress is bridge two ports on an extra router, enable >>>>>> bridge >>>>>> firewalling, and create IPTables rules to mark downstream packets. >>>>>> The >>>>>> router I use is a D-Link DGL-5500, which is comparable to an Archer >>>>>> C7. >>>>>> >>>>>> IPv6 support is not available in this netfilter module, but the IPv4 >>>>>> support >>>>>> is great. >>>>>> >>>>>> Noah Causin >>>>>> >>>>>> very nice.... however, can you describe a bit better how you >>>>>> configured for ingress ? a second router? configuration file ? your >>>>>> high level description seems a bit confusing to me >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Cake mailing list >>>>>> Cake@lists.bufferbloat.net >>>>>> https://lists.bufferbloat.net/listinfo/cake >>>>>> >>>>>> >