From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f43.google.com (mail-la0-f43.google.com [209.85.215.43]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 96C1F21F0AE for ; Fri, 4 Jan 2013 13:36:15 -0800 (PST) Received: by mail-la0-f43.google.com with SMTP id eg20so10518310lab.16 for ; Fri, 04 Jan 2013 13:36:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=5dL1GQUaKmoHWhyYNbahMkjiqeUVgZHBEqP+DJS1VMc=; b=TjNH0kaX3hzDg4tonNCgOGGAY5U2pJOImxG/XIuE47UD52067mX+DODGYJBmRCfO5f e9HDLAK0nvKLXaydOUT4FByaEU2p1qia81li3kSiOuThBo2/AyI/aH74jZ/QzWJhFRrc hT3xmKfdYBhiJXh5fV6wn4cUjfYwsVW1e0xb+B4Z1pK23k7D5uKhY2F9egJa45y/A6K2 ZWXBG+WItyDu9ftlcvm4aOgeGgvmtlGWZdHngMCxht89Ck4gNEzcK0GxJ10JetO6lP/Y jrdXt8VUOtuRelIQ5+KOD/oqL1KpVpG/LvmMd1/WwAp01KqtjUHcJB67nKcVuBV15Y8A k25A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=5dL1GQUaKmoHWhyYNbahMkjiqeUVgZHBEqP+DJS1VMc=; b=DeO52VCuFZFcnxlKW8bJ5Bl+TfqAiQYsZS3ruuUHF1h7qm04zu+cdf1390RYHfdv9Y FHqWHLJPTIkxgIPerFn9ZxmAiwINV0yrWLKSIQoffS8O+8hYxw4r/k1TZ02RroNnB4Wr nCu7gn4F4Aae9Xl4gUWnwVVvdh3VVJzlpKV3s6CM2S8KnqERE9AfjUbuaCTcTsV6W8Da yyF9rmCIN0xEp3JMjn2cK3sW++zeB14TkV+YRxVT252Q1xeVmZfgVJ4w7Ylt83cYA7y8 A4SzBJBzy7MAwZjrT2wl4j8v1uQIIgN2MdUV8WRjMcz8L1FvAS7r97CtJb6h1qOZ4EQY gp5g== MIME-Version: 1.0 Received: by 10.112.26.70 with SMTP id j6mr20426922lbg.55.1357335373188; Fri, 04 Jan 2013 13:36:13 -0800 (PST) Received: by 10.112.49.134 with HTTP; Fri, 4 Jan 2013 13:36:13 -0800 (PST) In-Reply-To: References: Date: Fri, 4 Jan 2013 13:36:13 -0800 Message-ID: From: Jerry Chu To: Dave Taht Content-Type: multipart/alternative; boundary=bcaec5555630c28e7c04d27d4410 X-Gm-Message-State: ALoCoQnNKp5e7JA2YhpUJOgwGPVdTqhSiiaBuU/Kp8HlqaCS+sMs21c9zOmdUhJ/Kl79avuJOO4712KFPs3N28omvpgqcZTjieimrSoAEJY/6tYfQjaZMfkTUi7vF+7Ln+o00qbd0ZHmJugW+NKCkrqMEQEykPgCReuYcpqDt1YQSDMhz4hyLKiDUisGqvVt1b2rC1yGTjpnhvKC3lfvX82XKExVNGZ6sA== X-Mailman-Approved-At: Fri, 04 Jan 2013 13:45:31 -0800 Cc: Eric Dumazet , cerowrt-devel@lists.bufferbloat.net, Yuchung Cheng Subject: Re: [Cerowrt-devel] TFO crashes cerowrt 3.7.1-1 X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2013 21:36:16 -0000 --bcaec5555630c28e7c04d27d4410 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Jan 4, 2013 at 1:21 PM, Dave Taht wrote: > On Fri, Jan 4, 2013 at 12:57 PM, Jerry Chu wrote: > > +ycheng > > > > On Fri, Jan 4, 2013 at 12:43 PM, Maciej Soltysiak > > wrote: > >> > >> Oops, apologies if email was formatted weirdly... > > > > > > The problem you described below is separate from the MIPS router crash > one, > > right? > > I think - but am of course unsure - that we've actually hit a > different (or additional) problem than TFO - merely exposed by testing > fastopen first... > > that said... > > >BTW, we've only tested on x86_64 arch. > > I have multiple arches accumulated for some BQL work, mostly arm, > > They include a raspberry pi, a zedboard, a wndr4300, a nexus 7, and a > couple other boxes in addition to my usual mips based wndr3800s and > nanostation m5s. (which use mildly different mips chips and in > particular the m5 does not share the unaligned rx problem that the > ar71xx has a patch for). The zedboard's been problematic... > > So I will fold in testing tfo to looking hard at the igmp, and > checksum issues seemingly exposed today. Unless someone beats me to it > (I'm tied up til sunday) > > In the interim, at a higher layers, the current release of httpping > has tfo support, and there are patches to polipo, that might benefit > Awesome! > from review and wider testing on x86_64. > > > https://raw.github.com/dtaht/ceropackages-3.3/master/net/polipo/patches/0= 01-server_tfo.patch tfo_qlen is your first line of defense against spoofed TFO attack so you want to pick a value wisely. (50 seems on the safer side.) Jerry > > At least half of the needed support landed in netperf svn recently, too. > > > > > In addition to tcpdump, "netstat -s | grep -i fastopen" may be useful > too. > > > > Thanks, > > > > Jerry (author of TFO server code) > > > >> > >> > >> On Fri, Jan 4, 2013 at 9:42 PM, Maciej Soltysiak > >> wrote: > >>> > >>> I am seeing something strange here, with polipo related to TFO but al= so > >>> DNS. > >>> When I just took 3.7.1-1 and set my windows 7 laptop to use > >>> gw.home.lan:8123 as http proxy it didn't work. What I observed was: > >>> A) after quite a while polipo's response to browser was 504 Host > >>> www.osnews.com lookup failed: Timeout > >>> b) this error in ssh console: Host osnews.com lookup failed: Timeout > >>> (131072) > >>> c) Disabling TFO by adding option useTCPFastOpen 'false' to config > >>> 'polipo' 'general' works around the problem > >>> d) Alternatively, you can keep TFO enabled in polipo but change optio= n > >>> 'dnsUseGethostbyname' from 'reluctantly' to 'true' (!) > >>> This is very weird, because TFO is TCP and the DNS queries fired off = by > >>> polipo are UDP: > >>> root@OpenWrt:/tmp/log# tcpdump -n -v -vv -vvv -x -X -s 1500 -i lo > >>> 20:21:56.160245 IP (tos 0x0, ttl 64, id 50129, offset 0, flags [DF], > >>> proto UDP (17), length 60) > >>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b -> 0xd17f!] > 55396+ > >>> A? www.osnews.com. (32) > >>> 0x0000: 4500 003c c3d1 4000 4011 78dd 7f00 0001 E..<..@.@.x..... > >>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d.. > >>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.osn > >>> 0x0030: 6577 7303 636f 6d00 0001 0001 ews.com..... > >>> 20:21:56.160319 IP (tos 0x0, ttl 64, id 50130, offset 0, flags [DF], > >>> proto UDP (17), length 60) > >>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b -> 0xd164!] > 55396+ > >>> AAAA? www.osnews.com. (32) > >>> 0x0000: 4500 003c c3d2 4000 4011 78dc 7f00 0001 E..<..@.@.x..... > >>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d.. > >>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.osn > >>> 0x0030: 6577 7303 636f 6d00 001c 0001 ews.com..... > >>> 20:21:56.169942 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], prot= o > >>> UDP (17), length 123) > >>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe7a -> 0x5f73!] 553= 96 > >>> q: A? www.osnews.com. 1/2/0 www.osnews.com. [29m3s] A 74.86.31.159 ns= : > >>> osnews.com. [29m3s] NS ns2.swelter.net., osnews.com. [29m3s] NS > >>> ns1.swelter.net. (95) > >>> 0x0000: 4500 007b 0000 4000 4011 3c70 7f00 0001 E..{..@.@. >>> 0x0010: 7f00 0001 0035 b8c8 0067 fe7a d864 8180 .....5...g.z.d.. > >>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.osn > >>> 0x0030: 6577 7303 636f 6d00 0001 0001 c00c 0001 ews.com......... > >>> 0x0040: 0001 0000 06cf 0004 4a56 1f9f c010 0002 ........JV...... > >>> 0x0050: 0001 0000 06cf 0011 036e 7332 0773 7765 .........ns2.swe > >>> 0x0060: 6c74 6572 036e 6574 00c0 1000 0200 0100 lter.net........ > >>> 0x0070: 0006 cf00 0603 6e73 31c0 40 ......ns1.@ > >>> 20:21:56.173901 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], prot= o > >>> UDP (17), length 135) > >>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe86 -> 0x8ecb!] 553= 96 > >>> q: AAAA? www.osnews.com. 1/2/0 www.osnews.com. [54m44s] AAAA > >>> 2607:f0d0:1002:62::3 ns: osnews.com. [29m3s] NS ns1.swelter.net., > >>> osnews.com. [29m3s] NS ns2.swelter.net. (107) > >>> 0x0000: 4500 0087 0000 4000 4011 3c64 7f00 0001 E.....@.@. >>> 0x0010: 7f00 0001 0035 b8c8 0073 fe86 d864 8180 .....5...s...d.. > >>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.osn > >>> 0x0030: 6577 7303 636f 6d00 001c 0001 c00c 001c ews.com......... > >>> 0x0040: 0001 0000 0cd4 0010 2607 f0d0 1002 0062 ........&......b > >>> 0x0050: 0000 0000 0000 0003 c010 0002 0001 0000 ................ > >>> 0x0060: 06cf 0011 036e 7331 0773 7765 6c74 6572 .....ns1.swelter > >>> 0x0070: 036e 6574 00c0 1000 0200 0100 0006 cf00 .net............ > >>> 0x0080: 0603 6e73 32c0 4c ..ns2.L > >>> This is the only DNS traffic I saw during the attempts. The tcpdumps > have > >>> udp bad checksum but when I disabled TFO in polipo, the UDP where > still bad > >>> checksum but they worked. > >>> Really weird. > >>> p.s. UPNP still works for port forwarding negotiation as it did in > >>> 3.6.11-4 > >>> I still couldn't get the UPNP/SSDP broadcasts (udp to 239.255.255.250= ) > to > >>> being forwarded between se00 and sw00/sw10. Last time it worked was > ~3.3.8. > >>> I'm starting not to question why it doesn't work, I'm starting to > wonder why > >>> it did work then ;-) > >>> Regards, > >>> Maciej > >>> On Fri, Jan 4, 2013 at 6:33 PM, Dave Taht wrote= : > >>>> > >>>> On Fri, Jan 4, 2013 at 9:27 AM, Eric Dumazet > >>>> wrote: > >>>> > Sorry, could you give us a copy of the panic stack trace ? > >>>> > >>>> I will get a serial console up on a wndr3800 by sunday. (sorry, just > >>>> landed in california, am in disarray) > >>>> > >>>> The latest dev build of cero for the wndr3800 and wndr3700v2 is at: > >>>> > >>>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.7.1-1/ > >>>> > >>>> -- > >>>> Dave T=E4ht > >>>> > >>>> Fixing bufferbloat with cerowrt: > >>>> http://www.teklibre.com/cerowrt/subscribe.html > >>>> _______________________________________________ > >>>> Cerowrt-devel mailing list > >>>> Cerowrt-devel@lists.bufferbloat.net > >>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel > >>> > >>> > >> > > > > > > -- > Dave T=E4ht > > Fixing bufferbloat with cerowrt: > http://www.teklibre.com/cerowrt/subscribe.html > --bcaec5555630c28e7c04d27d4410 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Fri, Jan 4, 2013 at = 1:21 PM, Dave Taht <dave.taht@gmail.com> wrote:
<= div class=3D"gmail_extra">
On Fri, Jan 4= , 2013 at 12:57 PM, Jerry Chu <hkchu= @google.com> wrote:
> +ycheng
>
> On Fri, Jan 4, 2013 at 12:43 PM, Maciej Soltysiak <maciej@soltysiak.com>
> wrote:
>>
>> Oops, apologies if email was formatted weirdly...
>
>
> The problem you described below is separate from the MIPS router crash= one,
> right?

I think - but am of course unsure - that we've actually hit a
different (or additional) problem than TFO - merely exposed by testing
fastopen first...

that said...

>BTW, we've only tested on x86_64 arch.

I have multiple arches accumulated for some BQL work, mostly arm,

They include a raspberry pi, a zedboard, a wndr4300, a nexus 7, and a
couple other boxes in addition to my usual mips based wndr3800s and
nanostation m5s. (which use mildly different mips chips and in
particular the m5 does not share the unaligned rx problem that the
ar71xx has a patch for). The zedboard's been problematic...

So I will fold in testing tfo to looking hard at the igmp, and
checksum issues seemingly exposed today. Unless someone beats me to it
(I'm tied up til sunday)

In the interim, at a higher layers, the current release of httpping
has tfo support, and there are patches to polipo, that might benefit

Awesome!
=A0
from review and wider testing on x86_64.

https://raw.github.com/dtah= t/ceropackages-3.3/master/net/polipo/patches/001-server_tfo.patch

tfo_qlen is your first line of defense against spoofed = TFO attack so you want to pick a value wisely. (50 seems on the safer side.= )

Jerry



At least half of the needed support landed in netperf svn recently, too.

>
> In addition to tcpdump, "netstat -s | grep -i fastopen" may = be useful too.
>
> Thanks,
>
> Jerry (author of TFO server code)
>
>>
>>
>> On Fri, Jan 4, 2013 at 9:42 PM, Maciej Soltysiak <maciej@soltysiak.com>
>> wrote:
>>>
>>> I am seeing something strange here, with polipo related to TFO= but also
>>> DNS.
>>> When I just took 3.7.1-1 and set my windows 7 laptop to use >>> gw.home.lan:8123 as http proxy it didn't work. What I obse= rved was:
>>> A) after quite a while polipo's response to browser was 50= 4 Host
>>> www.osnews= .com lookup failed: Timeout
>>> b) this error in ssh console: Host osnews.com lookup failed: Timeout
>>> (131072)
>>> c) Disabling TFO by adding option useTCPFastOpen 'false= 9; to config
>>> 'polipo' 'general' works around the problem >>> d) Alternatively, you can keep TFO enabled in polipo but chang= e option
>>> 'dnsUseGethostbyname' from 'reluctantly' to &#= 39;true' (!)
>>> This is very weird, because TFO is TCP and the DNS queries fir= ed off by
>>> polipo are UDP:
>>> root@OpenWrt:/tmp/log# tcpdump -n -v -vv -vvv -x -X -s 1500 -i= lo
>>> 20:21:56.160245 IP (tos 0x0, ttl 64, id 50129, offset 0, flags= [DF],
>>> proto UDP (17), length 60)
>>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b ->= 0xd17f!] 55396+
>>> A? www.osn= ews.com. (32)
>>> 0x0000: 4500 003c c3d1 4000 4011 78dd 7f00 0001 E..<..@.@.x= .....
>>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d= ..
>>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.o= sn
>>> 0x0030: 6577 7303 636f 6d00 0001 0001 ews.com.....
>>> 20:21:56.160319 IP (tos 0x0, ttl 64, id 50130, offset 0, flags= [DF],
>>> proto UDP (17), length 60)
>>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b ->= 0xd164!] 55396+
>>> AAAA? www.= osnews.com. (32)
>>> 0x0000: 4500 003c c3d2 4000 4011 78dc 7f00 0001 E..<..@.@.x= .....
>>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d= ..
>>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.o= sn
>>> 0x0030: 6577 7303 636f 6d00 001c 0001 ews.com.....
>>> 20:21:56.169942 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF= ], proto
>>> UDP (17), length 123)
>>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe7a ->= 0x5f73!] 55396
>>> q: A? www.= osnews.com. 1/2/0 w= ww.osnews.com. [29m3s] A 74.86.31.159 ns:
>>> osnews.com= . [29m3s] NS ns2.swelt= er.net., osnews.com= . [29m3s] NS
>>> ns1.swelt= er.net. (95)
>>> 0x0000: 4500 007b 0000 4000 4011 3c70 7f00 0001 E..{..@.@.<= p....
>>> 0x0010: 7f00 0001 0035 b8c8 0067 fe7a d864 8180 .....5...g.z.d= ..
>>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.o= sn
>>> 0x0030: 6577 7303 636f 6d00 0001 0001 c00c 0001 ews.com.......= ..
>>> 0x0040: 0001 0000 06cf 0004 4a56 1f9f c010 0002 ........JV....= ..
>>> 0x0050: 0001 0000 06cf 0011 036e 7332 0773 7765 .........ns2.s= we
>>> 0x0060: 6c74 6572 036e 6574 00c0 1000 0200 0100 lter.net......= ..
>>> 0x0070: 0006 cf00 0603 6e73 31c0 40 ......ns1.@
>>> 20:21:56.173901 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF= ], proto
>>> UDP (17), length 135)
>>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe86 ->= 0x8ecb!] 55396
>>> q: AAAA? w= ww.osnews.com. 1/2/0 www.osnews.com. [54m44s] AAAA
>>> 2607:f0d0:1002:62::3 ns: osnews.com. [29m3s] NS ns1.swelter.net.,
>>> osnews.com= . [29m3s] NS ns2.swelt= er.net. (107)
>>> 0x0000: 4500 0087 0000 4000 4011 3c64 7f00 0001 E.....@.@.<= d....
>>> 0x0010: 7f00 0001 0035 b8c8 0073 fe86 d864 8180 .....5...s...d= ..
>>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.o= sn
>>> 0x0030: 6577 7303 636f 6d00 001c 0001 c00c 001c ews.com.......= ..
>>> 0x0040: 0001 0000 0cd4 0010 2607 f0d0 1002 0062 ........&.= .....b
>>> 0x0050: 0000 0000 0000 0003 c010 0002 0001 0000 ..............= ..
>>> 0x0060: 06cf 0011 036e 7331 0773 7765 6c74 6572 .....ns1.swelt= er
>>> 0x0070: 036e 6574 00c0 1000 0200 0100 0006 cf00 .net..........= ..
>>> 0x0080: 0603 6e73 32c0 4c ..ns2.L
>>> This is the only DNS traffic I saw during the attempts. The tc= pdumps have
>>> udp bad checksum but when I disabled TFO in polipo, the UDP wh= ere still bad
>>> checksum but they worked.
>>> Really weird.
>>> p.s. UPNP still works for port forwarding negotiation as it di= d in
>>> 3.6.11-4
>>> I still couldn't get the UPNP/SSDP broadcasts (udp to 239.= 255.255.250) to
>>> being forwarded between se00 and sw00/sw10. Last time it worke= d was ~3.3.8.
>>> I'm starting not to question why it doesn't work, I= 9;m starting to wonder why
>>> it did work then ;-)
>>> Regards,
>>> Maciej
>>> On Fri, Jan 4, 2013 at 6:33 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>>>
>>>> On Fri, Jan 4, 2013 at 9:27 AM, Eric Dumazet <edumazet@google.com>
>>>> wrote:
>>>> > Sorry, could you give us a copy of the panic stack tr= ace ?
>>>>
>>>> I will get a serial console up on a wndr3800 by sunday. (s= orry, just
>>>> landed in california, am in disarray)
>>>>
>>>> The latest dev build of cero for the wndr3800 and wndr3700= v2 is at:
>>>>
>>>> http://snapon.lab.bufferbloat.net/~cero2= /cerowrt/wndr/3.7.1-1/
>>>>
>>>> --
>>>> Dave T=E4ht
>>>>
>>>> Fixing bufferbloat with cerowrt:
>>>> http://www.teklibre.com/cerowrt/subscribe.html
>>>> _______________________________________________
>>>> Cerowrt-devel mailing list
>>>> Cer= owrt-devel@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cerowrt-dev= el
>>>
>>>
>>
>



--
Dave T=E4ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscrib= e.html

--bcaec5555630c28e7c04d27d4410--