From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 816F13B29E; Thu, 22 Aug 2019 14:56:19 -0400 (EDT) Received: by mail-io1-xd35.google.com with SMTP id j5so14104240ioj.8; Thu, 22 Aug 2019 11:56:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=4eAJJh+sAZrDa6B89tk3wJ0ButHfZ+WodaFAOMkMYiU=; b=LmiV4wAaHUNB2/r3JSFbciC95kFGJxhHk8bEeECdv0tW0pnP8m5ElNn4+BJkBEsdJZ 1qqGU4La4C9Wu3yoQvEX/HqGCAYv6aZdIF6/O/XfkcDlG9K2FdYFZkbpGV5a10do4PW+ yuAcDYdJWz5XkIAOe7x6dzHTexQZR0TCUL7i2FBbz1LWzyJcWMq7DEWlMO4G5+f8kj85 6p9JsLkpyBjXSNSJ4tqq//Uws9dB6t0+oQUG5w1isCwP0E4ohPqjVLYJfg15gldsjhgO KCdfQLyhx4/0tE5KNAyYLHyGb6Q+nPBFzjdWchwijCEGVzbnhLw7YSZna0S1wcekXpUu jwmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=4eAJJh+sAZrDa6B89tk3wJ0ButHfZ+WodaFAOMkMYiU=; b=Ap+uSnCmF+hpo07BQiIhMcmjiWLbTowGhjEFwvAQD85K6oWaf9pDvcjdBJruhJ92H1 79PnG500V0iVseofz+e3rdtuZt+qWGs+dg7xhRnJueJIUlXpqosnG9LP4J+iERFe7Daw VA9mrJh9qOTUoAR/FvCuZEBBu3xa6uN4tiyj901V6exGqZVKxTNRHy1YgW7q7wkdYz14 9H9NvUlqflEvIFZ3NRrrsOQ0v2ZtkwC7yRlVhfW5H1gIxcf7zaSh4J+MDgzAip1G6XD/ zHKwOvKaf8m1yztvRxU3m1Vl5UZuoNMmBhSTtGOZYxrqoV6brhNbmgVYyVKldnYJ2MJk nfTw== X-Gm-Message-State: APjAAAXQDU04ScPo9Dl3tQO01+Np62y+iEAsMBuQyM/TVnkdKzXFVfJO SoFB3dn7jt5YI9vCNpzp6ByPPjXDPVjm70/KmAY= X-Google-Smtp-Source: APXvYqz6+UriqiW8xjisp2t5niXw4QvPIiUx36250WmoRgWNU4mizQ2u2hYya1f3vsGopabujHnJ+vYfScH0hjc8TJI= X-Received: by 2002:a02:c992:: with SMTP id b18mr1035562jap.128.1566500178770; Thu, 22 Aug 2019 11:56:18 -0700 (PDT) MIME-Version: 1.0 References: <87tvabsp99.fsf@toke.dk> <74bccc2b-b805-255f-b6a7-83ade9af6765@newmedia-net.de> <87r25fsn70.fsf@toke.dk> <54438C64-C613-438E-9CB9-6C6D0C5EAFA0@gmail.com> <87sgpvflo4.fsf@taht.net> <87wof6rf7t.fsf@toke.dk> <7656FCDE-C590-4B0C-B191-B9FAC928A762@gmail.com> <5eb4c395-c718-2d28-65a7-9762cf8d5bea@newmedia-net.de> <47AD5102-B66F-44A5-AADE-D167ECB94A61@gmx.de> <1d772664-b6cc-a528-9725-96a431032875@newmedia-net.de> <87v9uqea3x.fsf@taht.net> <87tvaap57q.fsf@toke.dk> <5bbd2b81-9846-3a7a-130c-0f59e04fd2d1@newmedia-net.de> <87ftltdter.fsf@taht.net> <87pnkxnjo4.fsf@toke.dk> In-Reply-To: <87pnkxnjo4.fsf@toke.dk> From: Dave Taht Date: Thu, 22 Aug 2019 11:56:06 -0700 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Cc: Sebastian Gottschall , Dave Taht , Cake List , Battle of the Mesh Mailing List , Make-Wifi-fast Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Make-wifi-fast] Wifi Memory limits in small platforms X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Aug 2019 18:56:19 -0000 On Thu, Aug 22, 2019 at 11:23 AM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > > Sebastian Gottschall writes: > > > Am 22.08.2019 um 19:03 schrieb Dave Taht: > >> Sebastian Gottschall writes: > >> > >>> Am 22.08.2019 um 15:15 schrieb Dave Taht: > >>>> It's very good to know how much folk have been struggling to keep > >>>> things from OOMing on 32MB platforms. I'd like to hope that the > >>>> unified memory management in cake (vs a collection of QoS qdiscs) an= d > >>>> the new fq_codel for wifi stuff (cutting it down to 1 alloc from fou= r) > >>>> help, massively on this issue, but until today I was unaware of how > >>>> much the field may have been patching things out. > >>>> > >>>> The default 32MB memory limits in fq_codel comes from the stressing > >>>> about 10GigE networking from google. 4MB is limit in openwrt, > >>>> which is suitable for ~1Gbit, and is sort of there due to 802.11ac'= s > >>>> maximum (impossible to hit) of a txop that large. > >> I did kind of conflate "qos + fq_codel" vs wifi in this message. It > >> looks like yer staying with me. > >> > >>>> Something as small as 256K is essentially about 128 full size packet= s > >>>> (and often, acks from an ethernet device's rx ring eat 2k). > >>> what i miss in mac80211 is the following option "fq_codel =3D off" > >>> its essential and i will definitly work on a patch to deal with this > >>> way for low memory 802.11n platforms. > >> Well, it would be my hope that turning it off would A) not help that > >> much on memory or cpu and B) show such a dramatic reduction in > >> multi-station performance that you'd immediately turn it on again. > > isnt it better to have a working platform with less performance than a > > crashing platform with no performance? > > i mean i can user older mac80211 versions without that issue on a > > typical nanostation 2/5 which is often used just as CPE device > > So before the queueing patches to mac80211, the maximum packet queue > size for ath9k was 3MB in total, or 2.2MB if only a single AC was used > on the WiFi link (that's 128 packets in the driver + 1000 in the > pfifo_fast qdisc * 2074 bytes for the truesize of a full-size packet). > Whereas now the default is 4MB for a non-vht device. So it's not > actually that big of a difference, and as you've already discovered the > defaults can be changed. > > Would it be helpful to add support for setting the memory limit in > hostapd (to avoid having to patch the kernel default)? hmm. I guess exposing that via netlink, etc is a good idea. Me I just write the sys/kernel/debug/*/*/aqm files. btw: qos_map in my mind, for APs at this point, should default to the best effort queue only. Not sure how to set that in openwrt (I just patched it out of the kernel). 4 queues with 4 ready to go is a lot, and I have some ugly pics from battlemesh when I tested it that I should get around to publishing it. as for that sys file... I'd rather like to expose target and interval, stop disabling ecn dynamically, and have something closer to an ewma for fiddling with the target in the first place..... /me hides > > but with current mac80211 versions (current means last 2-3 years). they > > are just unstable and running out of memory after a while > > the only thing which helped was cutting of the memory limit of fq_codel > > inside mac80211 > > i also have another fancy testunit which is a linksys wrt400 with 32 mb > > ram and 2 ath9k based wifi chipsets. no hope here fonr running stable > > for only 5 minutes even with a single connection under load (my crashin= g > > test is running a hdtv iptv stream converted to unicast using a > > stateless eoip tunnel) > > > >> I try to encourage folk to run the rtt_fair tests in flent when > >> twiddling with wifi. Those really shows how bad things are when you > >> don't have ATF + FQ + Per station aggregation and lots of > >> clients. Single threaded tests are misleading. > > i know but even single threaded tests arent working good on such > > devices. so there is no need to talk about the benefits of atf,fq_codel= etc. > > but there is need to talk about configurable use of it which also allow= s > > to disable it if required. I 110% agree that a system that can stay up for years is much better than one that is fast for 5 minutes! However I'd like a chance, in collaborating with you and your upcoming patches - to try and narrow down crash bugs to various subsystems and be able to get some benchmarks done that I simply couldn't do anymore at the financial conclusion of the make-wifi-fast and cake projects. I think I have a lot of gear that is dd-wrt compatible - apu2, wndr3700s, 3800s.... The reduce truesize patch had helped a lot at the time (2012). There were all kinds of flaky bugs that disappeared. the new drop monitor patchset looks WONDERFUL for seeing more about packet drop behavior in the stack, but it's a 5.3(?) feature only. I note that I run 18.06.1 on my 32MB pico and nanostations on the lupin campus, but I run no gui, few additional applications at all (except babel, snmpd, netperf, and the other core needed daemons). My uptimes are principally governed by power failures. I can't remember the last "crash, crash" I had, and I do track memory leaks (none). That said, I'm painfully aware that I should probably give dd-wrt and openwrt 19.x some testing just to make sure there's no regressions, but have been reluctant to get involved again without more partners in crime, because the scars from deploying 18.x widely are only beginning to heal... and only last week did the needed babel 1.9 upgrade arrive so I can finally redeploy ipv6 universally. I fear my current reliability metrics are so good because I took down ipv6 last year.... Pico: root@pool2:~# free total used free shared buffers Mem: 28480 23796 4684 92 1868 -/+ buffers: 21928 6552 Swap: 0 0 0 root@pool2:~# uptime 11:38:09 up 43 days, 21:37, load average: 0.04, 0.03, 0.04 Same workload over here, on a wndr3800, almost exactly the same config root@couch:~# free total used free shared buffers cached Mem: 60320 22872 37448 68 1960 6120 -/+ buffers/cache: 14792 45528 Swap: 0 0 0 > > Disabling the fq part won't actually gain you much in terms of memory > usage, though, as most of it is packet memory which is already > configurable. > > The one exception to this is the static overhead of 'struct fq_flow', of > which mac80211 currently allocates 4k. That's 300k of memory which is > currently not configurable. But that could be fixed :) > > -Toke -- Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740