* [Cerowrt-devel] talking at linux plumbers in portugal next week @ 2019-09-03 0:47 Dave Taht 2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson 0 siblings, 1 reply; 12+ messages in thread From: Dave Taht @ 2019-09-03 0:47 UTC (permalink / raw) To: bloat, cerowrt-devel I'm doing a talk on my ipv4 unicast extensions project next week. Toke and jesper and stephen hemminger are there, anyone else? There's also some "bloat" business of course to do. Obviously talk about the state of the bloat, the l4s & sce work, etc. but: I have a bunch of other "plumbing" stuff that's been bugging me lately, and perhaps there are those here that might have more? I should try to find people to talk to about, or knew who I should? 1) Recently (in a limited set of coffee shop tests), I saw 30+% of ipv4 tcp syns being retried. That's scary, I don't know if anyone's tracking that in any way? It seems possible to track it with in-browser statistics, possibly. I'm afraid a syn-limiter is in place with copy-pasted parameters set in the 90s - openwrt's default, last I looked, was 25/sec. 2) I'd really like somehow for atomic route updates to "just work". I think it's working now for ipv4, at least, as I saw a commit go in for it for bird, recently. That plus sorta-kinda-wanting to see pmtu work better for udp is on my mind. 3) Our biggest problem bufferbloat-wise is inbound shaping and I don't know how to make it better aside from resuming work on a smarter policer. 4)? I'm pretty broke as usual (LF is paying for the plane and airbnb (thx LF!), but not food or my time), if anyone could throw a few bucks into my gofundme or patreon, it would be appreciated. -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 0:47 [Cerowrt-devel] talking at linux plumbers in portugal next week Dave Taht @ 2019-09-03 12:23 ` Mikael Abrahamsson 2019-09-03 14:21 ` Dave Taht 0 siblings, 1 reply; 12+ messages in thread From: Mikael Abrahamsson @ 2019-09-03 12:23 UTC (permalink / raw) To: Dave Taht; +Cc: bloat, cerowrt-devel On Mon, 2 Sep 2019, Dave Taht wrote: > with copy-pasted parameters set in the 90s - openwrt's default, last I > looked, was 25/sec. -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN -A syn_flood -m comment --comment "!fw3" -j DROP Well, it's got a burst-size of 50. I agree that this is quite conservative. However, at least in my home we're not seeing drops: # iptables -nvL | grep -A 4 "Chain syn_flood" Chain syn_flood (1 references) pkts bytes target prot opt in out source destination 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ But you might be right that in places with a lot more clients then this might indeed cause problems. -- Mikael Abrahamsson email: swmike@swm.pp.se ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson @ 2019-09-03 14:21 ` Dave Taht 2019-09-03 14:35 ` [Cerowrt-devel] Revising the synflood limit Dave Taht ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Dave Taht @ 2019-09-03 14:21 UTC (permalink / raw) To: Mikael Abrahamsson; +Cc: bloat, cerowrt-devel On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: > > On Mon, 2 Sep 2019, Dave Taht wrote: > > > with copy-pasted parameters set in the 90s - openwrt's default, last I > > looked, was 25/sec. > > -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN > -A syn_flood -m comment --comment "!fw3" -j DROP > > Well, it's got a burst-size of 50. I agree that this is quite > conservative. > > However, at least in my home we're not seeing drops: > > # iptables -nvL | grep -A 4 "Chain syn_flood" > Chain syn_flood (1 references) > pkts bytes target prot opt in out source destination > 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > > But you might be right that in places with a lot more clients then this > might indeed cause problems. Well, *I* long ago had upped those params by 10x and don't see syn drops either on my backbone. But I rather suspect the rest of the world just copy-pasted it. It should scale as a function of bandwidth, I suppose, or get updated as a side effect of setting QoS - or just get bumped up. Start a bug over with openwrt? Take a hard look at other firewall designs? Like I said, though, my big question was is there a browser stat or some other easily accessible stat to see how often syns are rejected? Another context for this was syn negotiation with ecn on and the fallback. Interestingly, I've also seen a pretty big uptick in ecn marking over the last year or so, on one uplink (we do have a lot of guests that run apple gear), it's now at over 10% of of the drop ratio on outbound. This box is - I hope - the last cerowrt box running in the universe - and the only reason it ever goes down is because of a long duration power failure. I've been meaning to replace it for ages... root@lounge:~# uptime 07:14:53 up 55 days, 17:14, load average: 0.16, 0.09, 0.10 outbound: qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn Sent 159378714029 bytes 1038654784 pkt (dropped 426065, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 213282954 ecn_mark 48220 new_flows_len 0 old_flows_len 1 inbound: (where comcast remarks most packets to CS1) qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn Sent 40391986695 bytes 34710741 pkt (dropped 420, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 5687382 ecn_mark 0 new_flows_len 0 old_flows_len 2 qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn Sent 2285974845172 bytes 1748071724 pkt (dropped 61231, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 229072930 ecn_mark 344 new_flows_len 0 old_flows_len 1 > > -- > Mikael Abrahamsson email: swmike@swm.pp.se -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Cerowrt-devel] Revising the synflood limit 2019-09-03 14:21 ` Dave Taht @ 2019-09-03 14:35 ` Dave Taht 2019-09-10 18:08 ` Maciej Sołtysiak 2019-09-03 14:45 ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen 2019-09-03 15:22 ` Sebastian Moeller 2 siblings, 1 reply; 12+ messages in thread From: Dave Taht @ 2019-09-03 14:35 UTC (permalink / raw) To: Mikael Abrahamsson; +Cc: bloat, cerowrt-devel OK, I started a topic over there. It would be good to know how many other firewall tools set a syn limit by default, but that would take way more research. https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] Revising the synflood limit 2019-09-03 14:35 ` [Cerowrt-devel] Revising the synflood limit Dave Taht @ 2019-09-10 18:08 ` Maciej Sołtysiak 2019-09-10 22:35 ` Dave Taht 0 siblings, 1 reply; 12+ messages in thread From: Maciej Sołtysiak @ 2019-09-10 18:08 UTC (permalink / raw) To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat > OK, I started a topic over there. It would be good to know how many > other firewall tools set a syn limit by default, but that would take > way more research. > > https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957 Nice catch! I've 10 to 15 devices on my network. Just in 24h I see I had 10k SYN's rejected! ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] Revising the synflood limit 2019-09-10 18:08 ` Maciej Sołtysiak @ 2019-09-10 22:35 ` Dave Taht 2019-09-23 9:37 ` Maciej Sołtysiak 0 siblings, 1 reply; 12+ messages in thread From: Dave Taht @ 2019-09-10 22:35 UTC (permalink / raw) To: Maciej Sołtysiak; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat I'm not sure if it is a "nice catch" or not yet. It's merely me (now us) tying two anomalies together that might be connected. Can you convert that -j drop to a -j log to see where they come from? For example I was failing to negotiate ecn while at this conference, which also meant a dropped syn. On Tue, Sep 10, 2019 at 7:08 PM Maciej Sołtysiak <maciej@soltysiak.com> wrote: > > > OK, I started a topic over there. It would be good to know how many > > other firewall tools set a syn limit by default, but that would take > > way more research. > > > > https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957 > Nice catch! I've 10 to 15 devices on my network. Just in 24h I see I had 10k SYN's rejected! > -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] Revising the synflood limit 2019-09-10 22:35 ` Dave Taht @ 2019-09-23 9:37 ` Maciej Sołtysiak 0 siblings, 0 replies; 12+ messages in thread From: Maciej Sołtysiak @ 2019-09-23 9:37 UTC (permalink / raw) To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat > I'm not sure if it is a "nice catch" or not yet. It's merely me (now > us) tying two anomalies together that might be connected. Can you > convert that -j drop to a -j log to see where they come from? So, I must say I got rusty with my iptables. When I said I have the same, I had positive packet counters on the limit rule, not the drop rule (or the log rule I added) So, sorry to jump the gun, but I'm not experiencing it. Maciej ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 14:21 ` Dave Taht 2019-09-03 14:35 ` [Cerowrt-devel] Revising the synflood limit Dave Taht @ 2019-09-03 14:45 ` Toke Høiland-Jørgensen 2019-09-03 14:47 ` Dave Taht 2019-09-03 15:22 ` Sebastian Moeller 2 siblings, 1 reply; 12+ messages in thread From: Toke Høiland-Jørgensen @ 2019-09-03 14:45 UTC (permalink / raw) To: Dave Taht, Mikael Abrahamsson; +Cc: cerowrt-devel, bloat Dave Taht <dave.taht@gmail.com> writes: > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: >> >> On Mon, 2 Sep 2019, Dave Taht wrote: >> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I >> > looked, was 25/sec. >> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN >> -A syn_flood -m comment --comment "!fw3" -j DROP >> >> Well, it's got a burst-size of 50. I agree that this is quite >> conservative. >> >> However, at least in my home we're not seeing drops: >> >> # iptables -nvL | grep -A 4 "Chain syn_flood" >> Chain syn_flood (1 references) >> pkts bytes target prot opt in out source destination >> 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ >> >> But you might be right that in places with a lot more clients then this >> might indeed cause problems. > > Well, *I* long ago had upped those params by 10x and don't see syn > drops either on my backbone. But I rather suspect the rest of the > world just copy-pasted it. It should scale as a function of bandwidth, > I suppose, or get updated as a side effect of setting QoS - or just > get bumped up. Start a bug over with openwrt? Take a hard look at > other firewall designs? FWIW: # iptables -nvL syn_flood Chain syn_flood (1 references) pkts bytes target prot opt in out source destination 195K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ # ip6tables -nvL syn_flood Chain syn_flood (1 references) pkts bytes target prot opt in out source destination 396 41508 RETURN tcp * * ::/0 ::/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ 0 0 DROP all * * ::/0 ::/0 /* !fw3 */ rebooted this box today; don't seem to have hit the limit thus far, though... This is on a gigabit link. -Toke ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 14:45 ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen @ 2019-09-03 14:47 ` Dave Taht 2019-09-03 15:10 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 12+ messages in thread From: Dave Taht @ 2019-09-03 14:47 UTC (permalink / raw) To: Toke Høiland-Jørgensen; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote: > > Dave Taht <dave.taht@gmail.com> writes: > > > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: > >> > >> On Mon, 2 Sep 2019, Dave Taht wrote: > >> > >> > with copy-pasted parameters set in the 90s - openwrt's default, last I > >> > looked, was 25/sec. > >> > >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN > >> -A syn_flood -m comment --comment "!fw3" -j DROP > >> > >> Well, it's got a burst-size of 50. I agree that this is quite > >> conservative. > >> > >> However, at least in my home we're not seeing drops: > >> > >> # iptables -nvL | grep -A 4 "Chain syn_flood" > >> Chain syn_flood (1 references) > >> pkts bytes target prot opt in out source destination > >> 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > >> > >> But you might be right that in places with a lot more clients then this > >> might indeed cause problems. > > > > Well, *I* long ago had upped those params by 10x and don't see syn > > drops either on my backbone. But I rather suspect the rest of the > > world just copy-pasted it. It should scale as a function of bandwidth, > > I suppose, or get updated as a side effect of setting QoS - or just > > get bumped up. Start a bug over with openwrt? Take a hard look at > > other firewall designs? > > FWIW: > > # iptables -nvL syn_flood > Chain syn_flood (1 references) > pkts bytes target prot opt in out source destination > 195K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > > # ip6tables -nvL syn_flood > Chain syn_flood (1 references) > pkts bytes target prot opt in out source destination > 396 41508 RETURN tcp * * ::/0 ::/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > 0 0 DROP all * * ::/0 ::/0 /* !fw3 */ > > rebooted this box today; don't seem to have hit the limit thus far, > though... This is on a gigabit link. Hmm. Try to trigger it with --te=upload_streams=200 ? > > -Toke -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 14:47 ` Dave Taht @ 2019-09-03 15:10 ` Toke Høiland-Jørgensen 2019-09-03 15:21 ` Dave Taht 0 siblings, 1 reply; 12+ messages in thread From: Toke Høiland-Jørgensen @ 2019-09-03 15:10 UTC (permalink / raw) To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat Dave Taht <dave.taht@gmail.com> writes: > On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote: >> >> Dave Taht <dave.taht@gmail.com> writes: >> >> > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: >> >> >> >> On Mon, 2 Sep 2019, Dave Taht wrote: >> >> >> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I >> >> > looked, was 25/sec. >> >> >> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN >> >> -A syn_flood -m comment --comment "!fw3" -j DROP >> >> >> >> Well, it's got a burst-size of 50. I agree that this is quite >> >> conservative. >> >> >> >> However, at least in my home we're not seeing drops: >> >> >> >> # iptables -nvL | grep -A 4 "Chain syn_flood" >> >> Chain syn_flood (1 references) >> >> pkts bytes target prot opt in out source destination >> >> 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ >> >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ >> >> >> >> But you might be right that in places with a lot more clients then this >> >> might indeed cause problems. >> > >> > Well, *I* long ago had upped those params by 10x and don't see syn >> > drops either on my backbone. But I rather suspect the rest of the >> > world just copy-pasted it. It should scale as a function of bandwidth, >> > I suppose, or get updated as a side effect of setting QoS - or just >> > get bumped up. Start a bug over with openwrt? Take a hard look at >> > other firewall designs? >> >> FWIW: >> >> # iptables -nvL syn_flood >> Chain syn_flood (1 references) >> pkts bytes target prot opt in out source destination >> 195K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ >> >> # ip6tables -nvL syn_flood >> Chain syn_flood (1 references) >> pkts bytes target prot opt in out source destination >> 396 41508 RETURN tcp * * ::/0 ::/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ >> 0 0 DROP all * * ::/0 ::/0 /* !fw3 */ >> >> rebooted this box today; don't seem to have hit the limit thus far, >> though... This is on a gigabit link. > > Hmm. Try to trigger it with --te=upload_streams=200 ? Sure, that triggers it: # iptables -nvL syn_flood Chain syn_flood (1 references) pkts bytes target prot opt in out source destination 197K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ 275 16480 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ And I get tons of errors from netperf failing to start up. However, the protection is only actually enabled for the INPUT chain; i.e., I had to use the router itself as the netperf target to trigger the rule. So not sure a rule such as this would be the cause of your coffee shop failures? This is with the default openwrt config, BTW: config defaults option syn_flood '1' -Toke ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 15:10 ` Toke Høiland-Jørgensen @ 2019-09-03 15:21 ` Dave Taht 0 siblings, 0 replies; 12+ messages in thread From: Dave Taht @ 2019-09-03 15:21 UTC (permalink / raw) To: Toke Høiland-Jørgensen; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat On Tue, Sep 3, 2019 at 8:10 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote: > > Dave Taht <dave.taht@gmail.com> writes: > > > On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote: > >> > >> Dave Taht <dave.taht@gmail.com> writes: > >> > >> > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: > >> >> > >> >> On Mon, 2 Sep 2019, Dave Taht wrote: > >> >> > >> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I > >> >> > looked, was 25/sec. > >> >> > >> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN > >> >> -A syn_flood -m comment --comment "!fw3" -j DROP > >> >> > >> >> Well, it's got a burst-size of 50. I agree that this is quite > >> >> conservative. > >> >> > >> >> However, at least in my home we're not seeing drops: > >> >> > >> >> # iptables -nvL | grep -A 4 "Chain syn_flood" > >> >> Chain syn_flood (1 references) > >> >> pkts bytes target prot opt in out source destination > >> >> 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > >> >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > >> >> > >> >> But you might be right that in places with a lot more clients then this > >> >> might indeed cause problems. > >> > > >> > Well, *I* long ago had upped those params by 10x and don't see syn > >> > drops either on my backbone. But I rather suspect the rest of the > >> > world just copy-pasted it. It should scale as a function of bandwidth, > >> > I suppose, or get updated as a side effect of setting QoS - or just > >> > get bumped up. Start a bug over with openwrt? Take a hard look at > >> > other firewall designs? > >> > >> FWIW: > >> > >> # iptables -nvL syn_flood > >> Chain syn_flood (1 references) > >> pkts bytes target prot opt in out source destination > >> 195K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > >> > >> # ip6tables -nvL syn_flood > >> Chain syn_flood (1 references) > >> pkts bytes target prot opt in out source destination > >> 396 41508 RETURN tcp * * ::/0 ::/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > >> 0 0 DROP all * * ::/0 ::/0 /* !fw3 */ > >> > >> rebooted this box today; don't seem to have hit the limit thus far, > >> though... This is on a gigabit link. > > > > Hmm. Try to trigger it with --te=upload_streams=200 ? > > Sure, that triggers it: > > # iptables -nvL syn_flood > Chain syn_flood (1 references) > pkts bytes target prot opt in out source destination > 197K 12M RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ > 275 16480 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ > > > And I get tons of errors from netperf failing to start up. > > However, the protection is only actually enabled for the INPUT chain; > i.e., I had to use the router itself as the netperf target to trigger > the rule. So not sure a rule such as this would be the cause of your > coffee shop failures? Good point. I think in the cerowrt case I'd enabled it for all chains and then noticed, and bumped it up. Limiting syn attempts to the router itself makes more sense than doing it on the forward path. The coffee shop tests were fun, but I(we) needed more rigor when doing them. What I'd typically do is go in, get on the wifi, start 6 minutes worth of tests, get in line, get coffee... I would feel a twinge of guilt when I'd start seeing heads pop up from their laptops while running the rrul test, but in two cases I managed to talk to the owner, help 'em get QoS configured right on their router, show'd 'em the difference, and got a free meal out of it... So, it was only discovering that I'd had that 30% rejection figure via the stats on my osx box that I started speculating as to the cause(s), last week. The other big fear is that because OSX is always attempting to negotiate ECN, that that's a cause. A bit more rigor about this and it could turn into a good paper or blog entry, and it's good to get out more for the sake of science, with table service! > > This is with the default openwrt config, BTW: > > > config defaults > option syn_flood '1' > > > -Toke -- Dave Täht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week 2019-09-03 14:21 ` Dave Taht 2019-09-03 14:35 ` [Cerowrt-devel] Revising the synflood limit Dave Taht 2019-09-03 14:45 ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen @ 2019-09-03 15:22 ` Sebastian Moeller 2 siblings, 0 replies; 12+ messages in thread From: Sebastian Moeller @ 2019-09-03 15:22 UTC (permalink / raw) To: Dave Täht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat Not sure this is on-topic, but: https://www.prnewswire.com/news-releases/root-cause-analysis-and-incident-report-on-the-august-ddos-attack-300905405.html https://lists.gt.net/nanog/users/206044 > On Sep 3, 2019, at 16:21, Dave Taht <dave.taht@gmail.com> wrote: > > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote: >> >> On Mon, 2 Sep 2019, Dave Taht wrote: >> >>> with copy-pasted parameters set in the 90s - openwrt's default, last I >>> looked, was 25/sec. >> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN >> -A syn_flood -m comment --comment "!fw3" -j DROP >> >> Well, it's got a burst-size of 50. I agree that this is quite >> conservative. >> >> However, at least in my home we're not seeing drops: >> >> # iptables -nvL | grep -A 4 "Chain syn_flood" >> Chain syn_flood (1 references) >> pkts bytes target prot opt in out source destination >> 2296 113K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */ >> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* !fw3 */ >> >> But you might be right that in places with a lot more clients then this >> might indeed cause problems. > > Well, *I* long ago had upped those params by 10x and don't see syn > drops either on my backbone. But I rather suspect the rest of the > world just copy-pasted it. It should scale as a function of bandwidth, > I suppose, or get updated as a side effect of setting QoS - or just > get bumped up. Start a bug over with openwrt? Take a hard look at > other firewall designs? > > Like I said, though, my big question was is there a browser stat or > some other easily accessible stat to see how > often syns are rejected? Another context for this was syn negotiation > with ecn on and the fallback. > > Interestingly, I've also seen a pretty big uptick in ecn marking over > the last year or so, on one uplink (we do have a lot of guests that > run apple gear), it's now at over 10% of of the drop ratio on > outbound. > > This box is - I hope - the last cerowrt box running in the universe - > and the only reason it ever goes down is because of a long duration > power failure. I've been meaning to replace it for ages... > > root@lounge:~# uptime > 07:14:53 up 55 days, 17:14, load average: 0.16, 0.09, 0.10 > > outbound: > > qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300 > target 5.0ms interval 100.0ms ecn > Sent 159378714029 bytes 1038654784 pkt (dropped 426065, overlimits 0 > requeues 0) > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 213282954 ecn_mark 48220 > new_flows_len 0 old_flows_len 1 > > inbound: (where comcast remarks most packets to CS1) > > qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 1500 > target 5.0ms interval 100.0ms ecn > Sent 40391986695 bytes 34710741 pkt (dropped 420, overlimits 0 requeues 0) > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 5687382 ecn_mark 0 > new_flows_len 0 old_flows_len 2 > qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300 > target 5.0ms interval 100.0ms ecn > Sent 2285974845172 bytes 1748071724 pkt (dropped 61231, overlimits 0 > requeues 0) > backlog 0b 0p requeues 0 > maxpacket 1514 drop_overlimit 0 new_flow_count 229072930 ecn_mark 344 > new_flows_len 0 old_flows_len 1 > > >> >> -- >> Mikael Abrahamsson email: swmike@swm.pp.se > > > > -- > > Dave Täht > CTO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-831-205-9740 > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-09-23 9:38 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-03 0:47 [Cerowrt-devel] talking at linux plumbers in portugal next week Dave Taht 2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson 2019-09-03 14:21 ` Dave Taht 2019-09-03 14:35 ` [Cerowrt-devel] Revising the synflood limit Dave Taht 2019-09-10 18:08 ` Maciej Sołtysiak 2019-09-10 22:35 ` Dave Taht 2019-09-23 9:37 ` Maciej Sołtysiak 2019-09-03 14:45 ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen 2019-09-03 14:47 ` Dave Taht 2019-09-03 15:10 ` Toke Høiland-Jørgensen 2019-09-03 15:21 ` Dave Taht 2019-09-03 15:22 ` Sebastian Moeller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox