From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x232.google.com (mail-oi0-x232.google.com [IPv6:2607:f8b0:4003:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 4C4073B2A4 for ; Tue, 10 Jul 2018 03:02:30 -0400 (EDT) Received: by mail-oi0-x232.google.com with SMTP id 13-v6so40673575ois.1 for ; Tue, 10 Jul 2018 00:02:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=8qziKlu/bjkLs5Yf5tRKBdpPW9fRc6GENGYcjrUhAF0=; b=gnVjLzb5Vx6tBq1nlQzUp65TAYeiQ6ooFj+8G/k6cW8GnFPqy7c6aZYxU4Qp0TtLQP J+jEWVcXCaRA4XhNiMJjPWbdSXhoeSSaesGkpiTxehwDh109gaXQbGlnXbbZ6ZlScgiI j3ouc6UnVKZkoUvn7jFVaj67N/uxoY4n5YAnE7YVqsZ/KoMYwPraOPdl/W6H/a+etiPr 6iddU0ZsTIOGT4Ut9wW0ate6Sa8RfXEXQzV45Z1Y4GXgL2AL8YNFV3nXseAHJ6zhKvOf Ipp4hHWkhRL1zokSewHGbXpCc6/lVpWkj41Duyqrl/b5OsLGCKo2VWmCtbvu/Kcih53R LgrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=8qziKlu/bjkLs5Yf5tRKBdpPW9fRc6GENGYcjrUhAF0=; b=Tbo8AHI/SoVtLxxLVPHuiw5Z+nOz8R04w1Gh6oPO1fNVlcYvufayly9YLg6X3yOzLt i4Ut5+H9SUePmD8GPgB/uzZLZxQtsCtYKmlpBAjHSRezxSfYF1qaOMIldz7Fky/9GuWj E8A9l14QpZlWk7QGz7T8qIb4IXabhnwN7tB7qdyGe0X2Ew8cL9OL2jnfycytCU1/8rUr cEqJYoDg7rz/06qMv25qFQfQE68V1shkWVXqoPvHMKfU3k5Ib8OG2fKngKwOmGZDH8Xk tI/UvOwXi2ZBGVb1IiSOuZ1MvEU/91ShdsWYMUUs4LO/yaV0GT2nqm0b/ML1NbFHOi+L E4NQ== X-Gm-Message-State: APt69E2oTExAJIxeAxfj9UKxsXkuEAQ2eORrkTQE6IjJRCp71jYG9zJZ U+KOGeqxi7F166Xfz+lDvHkNKt1mWufKM6OGrhA= X-Google-Smtp-Source: AAOMgpd/UEOnUktNCcobYk/LnJ5ED03ROQyM5e+/OzF3PtEij8Iq2HomwuOa0WNgorTfTYljTsnhGC1BB+4mC2qmZpk= X-Received: by 2002:aca:cc46:: with SMTP id c67-v6mr28110234oig.197.1531206149569; Tue, 10 Jul 2018 00:02:29 -0700 (PDT) MIME-Version: 1.0 Sender: bkil.hu@gmail.com Received: by 2002:a9d:1d07:0:0:0:0:0 with HTTP; Tue, 10 Jul 2018 00:02:08 -0700 (PDT) In-Reply-To: References: <9E7E043B-2373-46ED-B122-38A287422999@eventide.io> <87d0wu7rbg.fsf@toke.dk> <8A44F1D4-1EB8-4D46-85F9-00C7307FF2D4@heistp.net> From: bkil Date: Tue, 10 Jul 2018 09:02:08 +0200 X-Google-Sender-Auth: veGgLavpfvqHqlfwhQSNAaqS7ww Message-ID: To: Pete Heist Cc: Make-Wifi-fast Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Make-wifi-fast] mesh deployment with ath9k driver changes X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2018 07:02:30 -0000 The CSV can be shaved down a lot by skipping some of the less interesting variable length fields that grow a lot, like the ones related to TIM/BA. I aimed to enumerate any and all fields that we could find any use in, not those fields that are the most interesting. You can experiment with CSVkit or the simple `cut` to see the effect of dropping certain columns. You can also vary snaplen to see the effect on an already made capture by using `editcap` from wireshark-common. It is perfectly legitime to cut short any or all frames shorter than their actual lengths. Tools should handle this, so no need to fill remaining parts with random bytes. Now that you've made a nice pcap anonymizer that is easy to extend, it would be indeed more desirable to share pcap's, especially from a tooling perspective. I've also considered this possibility and snaplength might be workable at a basic level, but you have to go really low. On the card and tcpdump version I tested, the common minimal radiotap size was 30 bytes, hence 58 bytes of snaplength ensured that 0 bytes were left from data frames. You can check this by progressively reducing snaplength and watching data bytes disappear: `tshark -x -V -r dump.pcap` Unfortunately, as radiotap itself is variable length, in case of transmissions with various properties (like HT vs. non-HT), it can get bigger, thus snapping into some interesting fields afterwards. Even with short headers, this snaps in half some mildly interesting frames like beacons. It would be best if we had a smart tool to: * randomize MAC (easiest is hashing with salt or encryption), * only keep radiotap & wlan headers without data, * prune a few sensitive parts from wlan as well (authentication, most beacon IEs, etc.) I've also been following the thread and waiting for anything to pop up because you've been experimenting with some very nice things. I couldn't say for sure, but the IO stat you've shared seems pretty high, especially if it was only a representative value and not a peak. Soft-realtime systems are best left idling - reaching 58% utilization seems like the network could pause pretty often if anything extra comes in. Together with the underflows, I would say that at least part of your problem may not be airtime, but rather CPU time on one or more of your APs/routers when handling peaks, though again both monitoring and airtime investigation could give more insight. CPU time recording is easy (or even forwarding with netcat similar to the pcap examples) https://github.com/bkil/lede-pcap-investigation/blob/master/wrt-cpu-mon.sh Recording key fields inside aqm/xmit could also prove to be useful to correlate with the SmokePing, also a matter of a bit of grep'ping. Although, if this would be the case, shaping should have provided more benefits, but we'll have to think about it. On Tue, Jul 10, 2018 at 2:39 AM, Pete Heist wrote: > >> On Jul 10, 2018, at 1:33 AM, Pete Heist wrote: >> >>> On Jun 30, 2018, at 9:14 PM, bkil wrote: >>> >>> N.b.: It's a pity that networking trace anonymization tools aren't up >>> to the challenge. Simple MAC randomization or hashing with data >>> omission would be just fine for such a use case. >> >> I set out to write a =E2=80=9Csimple=E2=80=9D pcap anonymizer today in G= o and it went smoothly with Ethernet pcaps containing IP data, but if one w= ants to cover radiotap + 802.11 plus all other protocols where MACs can app= ear it's not straightforward. > > So perhaps I can still find a snaplen that covers radiotap + 802.11 but n= ot any of the data, or I can randomize any leftover data beyond the 802.11 = header. I=E2=80=99ll make one more attempt tomorrow. If it works, it might = be easier for analysis than the csv.