Development issues regarding the cerowrt test router project
 help / color / mirror / Atom feed
* [Cerowrt-devel] 2.4GHz WiFi intermittent high latency/low speed issues - possibly ANI, bluetooth, or hwcrypt related?
@ 2015-04-24 15:16 leetminiwheat
       [not found] ` <8a92da2c-a057-4dc1-a600-821798d5e808@reed.com>
  0 siblings, 1 reply; 2+ messages in thread
From: leetminiwheat @ 2015-04-24 15:16 UTC (permalink / raw)
  To: cerowrt-devel

Have not tested 2.4GHz extensively until recently, but I've noticed
some intermittent issues.

Seems kind of random but after about a day or two I start getting
EXTREMELY high pings on wireless devices on 2.4ghz radio especially on
tons of small packets (like synergy - keyboard/mouse over network),
and speedtests drop a ton. Tested on two laptops. kernel debug shows
lots of retransmits, and ANI shows increasing numbers of OFDM and CCK
errors. changing channels doesn't seem to help, and the ath9k module
parameters seem hardcoded without a /sys interface to change them, so
I can't test different options.

I have not messed with any of the txqueuelen options or anything queue
related in /usr/sbin/debloat, they're all at default. Operating in
HT20 mode, WMM enabled/disabled did not matter, nor did ECN on/off
ingress/egress, and SQM on/off didn't matter either.

Any ideas? Could it be bluetooth interference? I noticed bt_coex was
disabled, and there's a few devices around here. or possibly some
really far guest clients causing ANI to act up? There's about 15 other
APs nearby on a scan but nothing higher (lower?) than -79dbI on my
channel according to my phone 10ft from the AP.

A /etc/init.d/network restart usually fixes it but not this time, nor
did reboots. have not tried cold boot, but will try that later though
it's still not a solution if it keeps coming back.

this happens as soon as i move my mouse over to the laptop with synergy:
64 bytes from 172.30.42.1: icmp_seq=101 ttl=64 time=2.25 ms
64 bytes from 172.30.42.1: icmp_seq=102 ttl=64 time=4.95 ms
64 bytes from 172.30.42.1: icmp_seq=103 ttl=64 time=35.9 ms
64 bytes from 172.30.42.1: icmp_seq=104 ttl=64 time=3.16 ms
64 bytes from 172.30.42.1: icmp_seq=105 ttl=64 time=125 ms
64 bytes from 172.30.42.1: icmp_seq=106 ttl=64 time=345 ms
64 bytes from 172.30.42.1: icmp_seq=107 ttl=64 time=3093 ms
64 bytes from 172.30.42.1: icmp_seq=108 ttl=64 time=5696 ms
64 bytes from 172.30.42.1: icmp_seq=109 ttl=64 time=5193 ms
64 bytes from 172.30.42.1: icmp_seq=110 ttl=64 time=4198 ms
64 bytes from 172.30.42.1: icmp_seq=112 ttl=64 time=3360 ms
64 bytes from 172.30.42.1: icmp_seq=114 ttl=64 time=2597 ms
64 bytes from 172.30.42.1: icmp_seq=116 ttl=64 time=2225 ms
64 bytes from 172.30.42.1: icmp_seq=117 ttl=64 time=2107 ms
64 bytes from 172.30.42.1: icmp_seq=118 ttl=64 time=1137 ms
64 bytes from 172.30.42.1: icmp_seq=119 ttl=64 time=485 ms
64 bytes from 172.30.42.1: icmp_seq=120 ttl=64 time=71.3 ms
64 bytes from 172.30.42.1: icmp_seq=121 ttl=64 time=3.12 ms

regular speedtests show about 6mbps... which normally is around 28mbps

# uptime
 11:04:20 up 21 min,  load average: 0.17, 0.21, 0.33

# iw sw00 station dump
Station 00:26:c7:76:ab:80 (on sw00)
        inactive time:  31 ms
        rx bytes:       1249755
        rx packets:     9068
        tx bytes:       2851502
        tx packets:     27751
        tx retries:     18418
        tx failed:      0
        signal:         -50 [-52, -56] dBm
        signal avg:     -50 [-51, -55] dBm
        tx bitrate:     130.0 MBit/s MCS 15
        rx bitrate:     58.5 MBit/s MCS 6
        authorized:     yes
        authenticated:  yes
        preamble:       short
        WMM/WME:        yes
        MFP:            no
        TDLS peer:      no

# iw gw00 station dump
Station bc:52:b7:d6:65:38 (on gw00)
        inactive time:  4699 ms
        rx bytes:       62140
        rx packets:     709
        tx bytes:       283680
        tx packets:     358
        tx retries:     635
        tx failed:      17
        signal:         -88 [-89, -93] dBm
        signal avg:     -87 [-88, -93] dBm
        tx bitrate:     1.0 MBit/s
        rx bitrate:     1.0 MBit/s
        authorized:     yes
        authenticated:  yes
        preamble:       short
        WMM/WME:        yes
        MFP:            no
        TDLS peer:      no
Station e4:92:fb:57:82:b9 (on gw00)
        inactive time:  66 ms
        rx bytes:       657880
        rx packets:     6043
        tx bytes:       6950507
        tx packets:     5663
        tx retries:     5668
        tx failed:      202
        signal:         -85 [-87, -90] dBm
        signal avg:     -85 [-87, -89] dBm
        tx bitrate:     5.5 MBit/s
        rx bitrate:     5.5 MBit/s
        authorized:     yes
        authenticated:  yes
        preamble:       short
        WMM/WME:        yes
        MFP:            no
        TDLS peer:      no
Station 44:a7:cf:d7:76:30 (on gw00)
        inactive time:  2402 ms
        rx bytes:       30852
        rx packets:     349
        tx bytes:       28079
        tx packets:     132
        tx retries:     96
        tx failed:      0
        signal:         -41 [-46, -42] dBm
        signal avg:     -41 [-45, -43] dBm
        tx bitrate:     54.0 MBit/s
        rx bitrate:     36.0 MBit/s
        authorized:     yes
        authenticated:  yes
        preamble:       short
        WMM/WME:        yes
        MFP:            no
        TDLS peer:      no

# cat /sys/kernel/debug/ieee80211/phy0/ath9k/ani
            ANI: ENABLED
      ANI RESET: 9
     OFDM LEVEL: 9
      CCK LEVEL: 7
        SPUR UP: 205
      SPUR DOWN: 205
 OFDM WS-DET ON: 1
OFDM WS-DET OFF: 2
     MRC-CCK ON: 0
    MRC-CCK OFF: 0
    FIR-STEP UP: 201
  FIR-STEP DOWN: 178
 INV LISTENTIME: 0
    OFDM ERRORS: 307392
     CCK ERRORS: 125040

# tc qdisc (tested with/without though)
qdisc fq_codel a: dev se00 root refcnt 2 limit 1514p flows 1024
quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 110: dev ge00 parent 1:11 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev ge00 parent 1:12 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev ge00 parent 1:13 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn
qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
qdisc mq 1: dev sw10 root
qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum
500 target 10.0ms interval 100.0ms
qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum
300 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum
300 target 5.0ms interval 100.0ms
qdisc htb 1: dev sw00 root refcnt 5 r2q 10 default 12
direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 110: dev sw00 parent 1:11 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms
qdisc fq_codel 120: dev sw00 parent 1:12 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms
qdisc fq_codel 130: dev sw00 parent 1:13 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms
qdisc ingress ffff: dev sw00 parent ffff:fff1 ----------------
qdisc htb 1: dev gw00 root refcnt 5 r2q 10 default 12
direct_packets_stat 4 direct_qlen 1000
qdisc fq_codel 110: dev gw00 parent 1:11 limit 1001p flows 1024
quantum 300 target 8.0ms interval 103.0ms
qdisc fq_codel 120: dev gw00 parent 1:12 limit 1001p flows 1024
quantum 300 target 8.0ms interval 103.0ms
qdisc fq_codel 130: dev gw00 parent 1:13 limit 1001p flows 1024
quantum 300 target 8.0ms interval 103.0ms
qdisc ingress ffff: dev gw00 parent ffff:fff1 ----------------
qdisc htb 1: dev ifb4ge00 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 110: dev ifb4ge00 parent 1:11 limit 1001p flows 1024
quantum 500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev ifb4ge00 parent 1:12 limit 1001p flows 1024
quantum 1500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev ifb4ge00 parent 1:13 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn
qdisc htb 1: dev ifb4gw00 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 110: dev ifb4gw00 parent 1:11 limit 1001p flows 1024
quantum 500 target 10.3ms interval 105.3ms ecn
qdisc fq_codel 120: dev ifb4gw00 parent 1:12 limit 1001p flows 1024
quantum 1500 target 10.3ms interval 105.3ms ecn
qdisc fq_codel 130: dev ifb4gw00 parent 1:13 limit 1001p flows 1024
quantum 300 target 10.3ms interval 105.3ms ecn
qdisc htb 1: dev ifb4sw00 root refcnt 2 r2q 10 default 12
direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 110: dev ifb4sw00 parent 1:11 limit 1001p flows 1024
quantum 500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 120: dev ifb4sw00 parent 1:12 limit 1001p flows 1024
quantum 1500 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 130: dev ifb4sw00 parent 1:13 limit 1001p flows 1024
quantum 300 target 5.0ms interval 100.0ms ecn

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [Cerowrt-devel] 2.4GHz WiFi intermittent high latency/low speed issues - possibly ANI, bluetooth, or hwcrypt related?
       [not found] ` <8a92da2c-a057-4dc1-a600-821798d5e808@reed.com>
@ 2015-04-26 14:46   ` leetminiwheat
  0 siblings, 0 replies; 2+ messages in thread
From: leetminiwheat @ 2015-04-26 14:46 UTC (permalink / raw)
  To: David P. Reed; +Cc: cerowrt-devel

Ah, thanks. So it's one of my guest WiFi clients then... wonderful.

Need to think of some way to handle this... <mangle gw00> -j DSCP
--set-dscp 0x08 isn't working, nor is bandwidth limiting. If only all
my devices could use 5GHz I could reserve that for myself. I get <=1ms
pings on 5GHz

On Sun, Apr 26, 2015 at 9:56 AM, David P. Reed <dpreed@reed.com> wrote:
> Classic symptom of bufferbloated link. One station or AP is probably not
> running Fq_codel. If some station starts a file transfer all users of the
> net suffer. Daddy why is the Internet slow? It's because someone is
> uploading and not knowing his packets are being buffered.
>
> On Apr 24, 2015, leetminiwheat <LeetMiniWheat@gmail.com> wrote:
>>
>> Have not tested 2.4GHz extensively until recently, but I've noticed
>> some intermittent issues.
>>
>> Seems kind of random but after about a day or two I start getting
>> EXTREMELY high pings on wireless devices on 2.4ghz radio especially on
>> tons of small packets (like synergy - keyboard/mouse over network),
>> and speedtests drop a ton. Tested on two laptops. kernel debug shows
>> lots of retransmits, and ANI shows increasing numbers of OFDM and CCK
>> errors. changing channels doesn't seem to help, and the ath9k module
>> parameters seem hardcoded without a /sys interface to change them, so
>> I can't test different options.
>>
>> I have not messed with any of the txqueuelen options or anything queue
>> related in /usr/sbin/debloat, they're all at default. Operating in
>> HT20 mode, WMM enabled/disabled did not matter, nor did ECN on/off
>> ingress/egress, and SQM on/off didn't matter either.
>>
>> Any ideas? Could it be bluetooth interference? I noticed bt_coex was
>> disabled, and there's a few devices around here. or possibly some
>> really far guest clients causing ANI to act up? There's about 15 other
>> APs nearby on a scan but nothing higher (lower?) than -79dbI on my
>> channel according to my phone 10ft from the AP.
>>
>> A /etc/init.d/network restart usually fixes it but not this time, nor
>> did reboots. have not tried cold boot, but will try that later though
>> it's still not a solution if it keeps coming back.
>>
>> this happens as soon as i move my mouse over to the laptop with synergy:
>> 64 bytes from 172.30.42.1: icmp_seq=101 ttl=64 time=2.25 ms
>> 64 bytes from 172.30.42.1: icmp_seq=102 ttl=64 time=4.95 ms
>> 64 bytes from 172.30.42.1: icmp_seq=103 ttl=64 time=35.9 ms
>> 64 bytes from 172.30.42.1: icmp_seq=104 ttl=64 time=3.16 ms
>> 64 bytes from 172.30.42.1: icmp_seq=105 ttl=64 time=125 ms
>> 64 bytes from 172.30.42.1: icmp_seq=106 ttl=64 time=345 ms
>> 64 bytes from 172.30.42.1: icmp_seq=107 ttl=64 time=3093 ms
>> 64 bytes from 172.30.42.1: icmp_seq=108 ttl=64 time=5696 ms
>> 64 bytes from 172.30.42.1: icmp_seq=109 ttl=64 time=5193 ms
>> 64 bytes from 172.30.42.1: icmp_seq=110 ttl=64 time=4198 ms
>> 64 bytes from 172.30.42.1: icmp_seq=112 ttl=64 time=3360 ms
>> 64 bytes from 172.30.42.1: icmp_seq=114 ttl=64 time=2597 ms
>> 64 bytes from 172.30.42.1: icmp_seq=116 ttl=64 time=2225 ms
>> 64 bytes from 172.30.42.1: icmp_seq=117 ttl=64 time=2107 ms
>> 64 bytes from 172.30.42.1: icmp_seq=118 ttl=64 time=1137 ms
>> 64 bytes from 172.30.42.1: icmp_seq=119 ttl=64 time=485 ms
>> 64 bytes from 172.30.42.1: icmp_seq=120 ttl=64 time=71.3 ms
>> 64 bytes from 172.30.42.1: icmp_seq=121 ttl=64 time=3.12 ms
>>
>> regular speedtests show about 6mbps... which normally is around 28mbps
>>
>> # uptime
>> 11:04:20 up 21 min,  load average: 0.17, 0.21, 0.33
>>
>> # iw sw00 station dump
>> Station 00:26:c7:76:ab:80 (on sw00)
>> inactive time:  31 ms
>> rx bytes:       1249755
>> rx packets:     9068
>> tx bytes:       2851502
>> tx packets:     27751
>> tx retries:     18418
>> tx failed:      0
>> signal:         -50 [-52, -56] dBm
>> signal avg:     -50 [-51, -55] dBm
>> tx bitrate:     130.0 MBit/s MCS 15
>> rx bitrate:     58.5 MBit/s MCS 6
>> authorized:     yes
>> authenticated:  yes
>> preamble:       short
>> WMM/WME:        yes
>> MFP:            no
>> TDLS peer:      no
>>
>> # iw gw00 station dump
>> Station bc:52:b7:d6:65:38 (on gw00)
>> inactive time:  4699 ms
>> rx bytes:       62140
>> rx packets:     709
>> tx bytes:       283680
>> tx packets:     358
>> tx retries:     635
>> tx failed:      17
>> signal:         -88 [-89, -93] dBm
>> signal avg:     -87 [-88, -93] dBm
>> tx bitrate:     1.0 MBit/s
>> rx bitrate:     1.0 MBit/s
>> authorized:     yes
>> authenticated:  yes
>> preamble:       short
>> WMM/WME:        yes
>> MFP:            no
>> TDLS peer:      no
>> Station e4:92:fb:57:82:b9 (on gw00)
>> inactive time:  66 ms
>> rx bytes:       657880
>> rx packets:     6043
>> tx bytes:       6950507
>> tx packets:     5663
>> tx retries:     5668
>> tx failed:      202
>> signal:         -85 [-87, -90] dBm
>> signal avg:     -85 [-87, -89] dBm
>> tx bitrate:     5.5 MBit/s
>> rx bitrate:     5.5 MBit/s
>> authorized:     yes
>> authenticated:  yes
>> preamble:       short
>> WMM/WME:        yes
>> MFP:            no
>> TDLS peer:      no
>> Station 44:a7:cf:d7:76:30 (on gw00)
>> inactive time:  2402 ms
>> rx bytes:       30852
>> rx packets:     349
>> tx bytes:       28079
>> tx packets:     132
>> tx retries:     96
>> tx failed:      0
>> signal:         -41 [-46, -42] dBm
>> signal avg:     -41 [-45, -43] dBm
>> tx bitrate:     54.0 MBit/s
>> rx bitrate:     36.0 MBit/s
>> authorized:     yes
>> authenticated:  yes
>> preamble:       short
>> WMM/WME:        yes
>> MFP:            no
>> TDLS peer:      no
>>
>> # cat /sys/kernel/debug/ieee80211/phy0/ath9k/ani
>> ANI: ENABLED
>> ANI RESET: 9
>> OFDM LEVEL: 9
>> CCK LEVEL: 7
>> SPUR UP: 205
>> SPUR DOWN: 205
>> OFDM WS-DET ON: 1
>> OFDM WS-DET OFF: 2
>> MRC-CCK ON: 0
>> MRC-CCK OFF: 0
>> FIR-STEP UP: 201
>> FIR-STEP DOWN: 178
>> INV LISTENTIME: 0
>> OFDM ERRORS: 307392
>> CCK ERRORS: 125040
>>
>> # tc qdisc (tested with/without though)
>> qdisc fq_codel a: dev se00 root refcnt 2 limit 1514p flows 1024
>> quantum 1514 target 5.0ms interval 100.0ms ecn
>> qdisc htb 1: dev ge00 root refcnt 2 r2q 10 default 12
>> direct_packets_stat 0 direct_qlen 1000
>> qdisc fq_codel 110: dev ge00 parent 1:11 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 120: dev ge00 parent 1:12 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 130: dev ge00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms ecn
>> qdisc ingress ffff: dev ge00 parent ffff:fff1 ----------------
>> qdisc mq 1: dev sw10 root
>> qdisc fq_codel 10: dev sw10 parent 1:1 limit 800p flows 1024 quantum
>> 500 target 10.0ms interval 100.0ms
>> qdisc fq_codel 20: dev sw10 parent 1:2 limit 800p flows 1024 quantum
>> 300 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 30: dev sw10 parent 1:3 limit 1000p flows 1024 quantum
>> 300 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 40: dev sw10 parent 1:4 limit 1000p flows 1024 quantum
>> 300 target 5.0ms interval 100.0ms
>> qdisc htb 1: dev sw00 root refcnt 5 r2q 10 default 12
>> direct_packets_stat 0 direct_qlen 1000
>> qdisc fq_codel 110: dev sw00 parent 1:11 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms
>> qdisc fq_codel 120: dev sw00 parent 1:12 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms
>> qdisc fq_codel 130: dev sw00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms
>> qdisc ingress ffff: dev sw00 parent ffff:fff1 ----------------
>> qdisc htb 1: dev gw00 root refcnt 5 r2q 10 default 12
>> direct_packets_stat 4 direct_qlen 1000
>> qdisc fq_codel 110: dev gw00 parent 1:11 limit 1001p flows 1024
>> quantum 300 target 8.0ms interval 103.0ms
>> qdisc fq_codel 120: dev gw00 parent 1:12 limit 1001p flows 1024
>> quantum 300 target 8.0ms interval 103.0ms
>> qdisc fq_codel 130: dev gw00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 8.0ms interval 103.0ms
>> qdisc ingress ffff: dev gw00 parent ffff:fff1 ----------------
>> qdisc htb 1: dev ifb4ge00 root refcnt 2 r2q 10 default 12
>> direct_packets_stat 0 direct_qlen 32
>> qdisc fq_codel 110: dev ifb4ge00 parent 1:11 limit 1001p flows 1024
>> quantum 500 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 120: dev ifb4ge00 parent 1:12 limit 1001p flows 1024
>> quantum 1500 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 130: dev ifb4ge00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms ecn
>> qdisc htb 1: dev ifb4gw00 root refcnt 2 r2q 10 default 12
>> direct_packets_stat 0 direct_qlen 32
>> qdisc fq_codel 110: dev ifb4gw00 parent 1:11 limit 1001p flows 1024
>> quantum 500 target 10.3ms interval 105.3ms ecn
>> qdisc fq_codel 120: dev ifb4gw00 parent 1:12 limit 1001p flows 1024
>> quantum 1500 target 10.3ms interval 105.3ms ecn
>> qdisc fq_codel 130: dev ifb4gw00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 10.3ms interval 105.3ms ecn
>> qdisc htb 1: dev ifb4sw00 root refcnt 2 r2q 10 default 12
>> direct_packets_stat 0 direct_qlen 32
>> qdisc fq_codel 110: dev ifb4sw00 parent 1:11 limit 1001p flows 1024
>> quantum 500 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 120: dev ifb4sw00 parent 1:12 limit 1001p flows 1024
>> quantum 1500 target 5.0ms interval 100.0ms ecn
>> qdisc fq_codel 130: dev ifb4sw00 parent 1:13 limit 1001p flows 1024
>> quantum 300 target 5.0ms interval 100.0ms ecn
>>
>> ________________________________
>>
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
> -- Sent with K-@ Mail - the evolution of emailing.

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2015-04-26 14:47 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-24 15:16 [Cerowrt-devel] 2.4GHz WiFi intermittent high latency/low speed issues - possibly ANI, bluetooth, or hwcrypt related? leetminiwheat
     [not found] ` <8a92da2c-a057-4dc1-a600-821798d5e808@reed.com>
2015-04-26 14:46   ` leetminiwheat

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox