[Cerowrt-devel] TFO crashes cerowrt 3.7.1-1

Fri Jan 4 16:44:53 EST 2013

On Fri, Jan 4, 2013 at 1:36 PM, Jerry Chu <hkchu at google.com> wrote:
> On Fri, Jan 4, 2013 at 1:21 PM, Dave Taht <dave.taht at gmail.com> wrote:
>>
>> On Fri, Jan 4, 2013 at 12:57 PM, Jerry Chu <hkchu at google.com> wrote:
>> > +ycheng
>> >
>> > On Fri, Jan 4, 2013 at 12:43 PM, Maciej Soltysiak <maciej at soltysiak.com>
>> > wrote:
>> >>
>> >> Oops, apologies if email was formatted weirdly...
>> >
>> >
>> > The problem you described below is separate from the MIPS router crash
>> > one,
>> > right?
>>
>> I think - but am of course unsure - that we've actually hit a
>> different (or additional) problem than TFO - merely exposed by testing
>> fastopen first...
>>
>> that said...
>>
>> >BTW, we've only tested on x86_64 arch.
>>
>> I have multiple arches accumulated for some BQL work, mostly arm,
>>
>> They include a raspberry pi, a zedboard, a wndr4300, a nexus 7, and a
>> couple other boxes in addition to my usual mips based wndr3800s and
>> nanostation m5s. (which use mildly different mips chips and in
>> particular the m5 does not share the unaligned rx problem that the
>> ar71xx has a patch for). The zedboard's been problematic...
>>
>> So I will fold in testing tfo to looking hard at the igmp, and
>> checksum issues seemingly exposed today. Unless someone beats me to it
>> (I'm tied up til sunday)
>>
>> In the interim, at a higher layers, the current release of httpping
>> has tfo support, and there are patches to polipo, that might benefit
>
>
> Awesome!
>
>>
>> from review and wider testing on x86_64.
>>
>>
>> https://raw.github.com/dtaht/ceropackages-3.3/master/net/polipo/patches/001-server_tfo.patch
>
>
> tfo_qlen is your first line of defense against spoofed TFO attack so you
> want to pick a value wisely. (50 seems on the safer side.)

I would like it if TFO client support landed in polipo too, which
looked both tricky to implement, and useful as a means of getting more
TFO support "out there".

Early testing with httping was quite promising over wifi. Eliminating
an RTT there basically doubled throughput for short local aggregates.

http://www.vanheusden.com/httping/ (it's the -F option)

Are there any other test tools available?

>
> Jerry
>
>>
>>
>> At least half of the needed support landed in netperf svn recently, too.
>>
>> >
>> > In addition to tcpdump, "netstat -s | grep -i fastopen" may be useful
>> > too.
>> >
>> > Thanks,
>> >
>> > Jerry (author of TFO server code)
>> >
>> >>
>> >>
>> >> On Fri, Jan 4, 2013 at 9:42 PM, Maciej Soltysiak <maciej at soltysiak.com>
>> >> wrote:
>> >>>
>> >>> I am seeing something strange here, with polipo related to TFO but
>> >>> also
>> >>> DNS.
>> >>> When I just took 3.7.1-1 and set my windows 7 laptop to use
>> >>> gw.home.lan:8123 as http proxy it didn't work. What I observed was:
>> >>> A) after quite a while polipo's response to browser was 504 Host
>> >>> www.osnews.com lookup failed: Timeout
>> >>> b) this error in ssh console: Host osnews.com lookup failed: Timeout
>> >>> (131072)
>> >>> c) Disabling TFO by adding option useTCPFastOpen 'false' to config
>> >>> 'polipo' 'general' works around the problem
>> >>> d) Alternatively, you can keep TFO enabled in polipo but change option
>> >>> 'dnsUseGethostbyname' from 'reluctantly' to 'true' (!)
>> >>> This is very weird, because TFO is TCP and the DNS queries fired off
>> >>> by
>> >>> polipo are UDP:
>> >>> root at OpenWrt:/tmp/log# tcpdump -n -v -vv -vvv -x -X -s 1500 -i lo
>> >>> 20:21:56.160245 IP (tos 0x0, ttl 64, id 50129, offset 0, flags [DF],
>> >>> proto UDP (17), length 60)
>> >>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b -> 0xd17f!]
>> >>> 55396+
>> >>> A? www.osnews.com. (32)
>> >>> 0x0000: 4500 003c c3d1 4000 4011 78dd 7f00 0001 E..<.. at .@.x.....
>> >>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d..
>> >>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.osn
>> >>> 0x0030: 6577 7303 636f 6d00 0001 0001 ews.com.....
>> >>> 20:21:56.160319 IP (tos 0x0, ttl 64, id 50130, offset 0, flags [DF],
>> >>> proto UDP (17), length 60)
>> >>> 127.0.0.1.47304 > 127.0.0.1.53: [bad udp cksum 0xfe3b -> 0xd164!]
>> >>> 55396+
>> >>> AAAA? www.osnews.com. (32)
>> >>> 0x0000: 4500 003c c3d2 4000 4011 78dc 7f00 0001 E..<.. at .@.x.....
>> >>> 0x0010: 7f00 0001 b8c8 0035 0028 fe3b d864 0100 .......5.(.;.d..
>> >>> 0x0020: 0001 0000 0000 0000 0377 7777 066f 736e .........www.osn
>> >>> 0x0030: 6577 7303 636f 6d00 001c 0001 ews.com.....
>> >>> 20:21:56.169942 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
>> >>> UDP (17), length 123)
>> >>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe7a -> 0x5f73!]
>> >>> 55396
>> >>> q: A? www.osnews.com. 1/2/0 www.osnews.com. [29m3s] A 74.86.31.159 ns:
>> >>> osnews.com. [29m3s] NS ns2.swelter.net., osnews.com. [29m3s] NS
>> >>> ns1.swelter.net. (95)
>> >>> 0x0000: 4500 007b 0000 4000 4011 3c70 7f00 0001 E..{.. at .@.<p....
>> >>> 0x0010: 7f00 0001 0035 b8c8 0067 fe7a d864 8180 .....5...g.z.d..
>> >>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.osn
>> >>> 0x0030: 6577 7303 636f 6d00 0001 0001 c00c 0001 ews.com.........
>> >>> 0x0040: 0001 0000 06cf 0004 4a56 1f9f c010 0002 ........JV......
>> >>> 0x0050: 0001 0000 06cf 0011 036e 7332 0773 7765 .........ns2.swe
>> >>> 0x0060: 6c74 6572 036e 6574 00c0 1000 0200 0100 lter.net........
>> >>> 0x0070: 0006 cf00 0603 6e73 31c0 40 ......ns1.@
>> >>> 20:21:56.173901 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
>> >>> UDP (17), length 135)
>> >>> 127.0.0.1.53 > 127.0.0.1.47304: [bad udp cksum 0xfe86 -> 0x8ecb!]
>> >>> 55396
>> >>> q: AAAA? www.osnews.com. 1/2/0 www.osnews.com. [54m44s] AAAA
>> >>> 2607:f0d0:1002:62::3 ns: osnews.com. [29m3s] NS ns1.swelter.net.,
>> >>> osnews.com. [29m3s] NS ns2.swelter.net. (107)
>> >>> 0x0000: 4500 0087 0000 4000 4011 3c64 7f00 0001 E..... at .@.<d....
>> >>> 0x0010: 7f00 0001 0035 b8c8 0073 fe86 d864 8180 .....5...s...d..
>> >>> 0x0020: 0001 0001 0002 0000 0377 7777 066f 736e .........www.osn
>> >>> 0x0030: 6577 7303 636f 6d00 001c 0001 c00c 001c ews.com.........
>> >>> 0x0040: 0001 0000 0cd4 0010 2607 f0d0 1002 0062 ........&......b
>> >>> 0x0050: 0000 0000 0000 0003 c010 0002 0001 0000 ................
>> >>> 0x0060: 06cf 0011 036e 7331 0773 7765 6c74 6572 .....ns1.swelter
>> >>> 0x0070: 036e 6574 00c0 1000 0200 0100 0006 cf00 .net............
>> >>> 0x0080: 0603 6e73 32c0 4c ..ns2.L
>> >>> This is the only DNS traffic I saw during the attempts. The tcpdumps
>> >>> have
>> >>> udp bad checksum but when I disabled TFO in polipo, the UDP where
>> >>> still bad
>> >>> checksum but they worked.
>> >>> Really weird.
>> >>> p.s. UPNP still works for port forwarding negotiation as it did in
>> >>> 3.6.11-4
>> >>> I still couldn't get the UPNP/SSDP broadcasts (udp to 239.255.255.250)
>> >>> to
>> >>> being forwarded between se00 and sw00/sw10. Last time it worked was
>> >>> ~3.3.8.
>> >>> I'm starting not to question why it doesn't work, I'm starting to
>> >>> wonder why
>> >>> it did work then ;-)
>> >>> Regards,
>> >>> Maciej
>> >>> On Fri, Jan 4, 2013 at 6:33 PM, Dave Taht <dave.taht at gmail.com> wrote:
>> >>>>
>> >>>> On Fri, Jan 4, 2013 at 9:27 AM, Eric Dumazet <edumazet at google.com>
>> >>>> wrote:
>> >>>> > Sorry, could you give us a copy of the panic stack trace ?
>> >>>>
>> >>>> I will get a serial console up on a wndr3800 by sunday. (sorry, just
>> >>>> landed in california, am in disarray)
>> >>>>
>> >>>> The latest dev build of cero for the wndr3800 and wndr3700v2 is at:
>> >>>>
>> >>>> http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.7.1-1/
>> >>>>
>> >>>> --
>> >>>> Dave Täht
>> >>>>
>> >>>> Fixing bufferbloat with cerowrt:
>> >>>> http://www.teklibre.com/cerowrt/subscribe.html
>> >>>> _______________________________________________
>> >>>> Cerowrt-devel mailing list
>> >>>> Cerowrt-devel at lists.bufferbloat.net
>> >>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Dave Täht
>>
>> Fixing bufferbloat with cerowrt:
>> http://www.teklibre.com/cerowrt/subscribe.html
>
>

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html