Development issues regarding the cerowrt test router project
 help / color / mirror / Atom feed
* [Cerowrt-devel] talking at linux plumbers in portugal next week
@ 2019-09-03  0:47 Dave Taht
  2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson
  0 siblings, 1 reply; 12+ messages in thread
From: Dave Taht @ 2019-09-03  0:47 UTC (permalink / raw)
  To: bloat, cerowrt-devel

I'm doing a talk on my ipv4 unicast extensions project next week. Toke
and jesper and stephen hemminger are there, anyone else?

There's also some "bloat" business of course to do. Obviously talk
about the state of the bloat, the l4s & sce work, etc.

but: I have a bunch of other "plumbing" stuff that's been bugging me
lately, and perhaps there are those here that might have
more? I should try to find people to talk to about, or knew who I should?

1) Recently (in a limited set of coffee shop tests), I saw 30+% of
ipv4 tcp syns being retried. That's scary, I don't know if anyone's
tracking that in any way? It seems possible to track it with
in-browser statistics, possibly. I'm afraid a syn-limiter is in place
with copy-pasted parameters set in the 90s - openwrt's default, last I
looked, was 25/sec.
2) I'd really like somehow for atomic route updates to "just work". I
think it's working now for ipv4, at least, as I saw a commit go
in for it for bird, recently. That plus sorta-kinda-wanting to see
pmtu work better for udp is on my mind.
3) Our biggest problem bufferbloat-wise is inbound shaping and I don't
know how to make it better aside from resuming work on a smarter
policer.
4)?

I'm pretty broke as usual (LF is paying for the plane and airbnb (thx
LF!), but not food or my time), if anyone could throw a few bucks into
my gofundme or patreon, it would be appreciated.

--

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03  0:47 [Cerowrt-devel] talking at linux plumbers in portugal next week Dave Taht
@ 2019-09-03 12:23 ` Mikael Abrahamsson
  2019-09-03 14:21   ` Dave Taht
  0 siblings, 1 reply; 12+ messages in thread
From: Mikael Abrahamsson @ 2019-09-03 12:23 UTC (permalink / raw)
  To: Dave Taht; +Cc: bloat, cerowrt-devel

On Mon, 2 Sep 2019, Dave Taht wrote:

> with copy-pasted parameters set in the 90s - openwrt's default, last I
> looked, was 25/sec.

-A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
-A syn_flood -m comment --comment "!fw3" -j DROP

Well, it's got a burst-size of 50. I agree that this is quite 
conservative.

However, at least in my home we're not seeing drops:

# iptables -nvL | grep -A 4 "Chain syn_flood"
Chain syn_flood (1 references)
  pkts bytes target     prot opt in     out     source               destination
  2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */

But you might be right that in places with a lot more clients then this 
might indeed cause problems.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson
@ 2019-09-03 14:21   ` Dave Taht
  2019-09-03 14:35     ` [Cerowrt-devel] Revising the synflood limit Dave Taht
                       ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Dave Taht @ 2019-09-03 14:21 UTC (permalink / raw)
  To: Mikael Abrahamsson; +Cc: bloat, cerowrt-devel

On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Mon, 2 Sep 2019, Dave Taht wrote:
>
> > with copy-pasted parameters set in the 90s - openwrt's default, last I
> > looked, was 25/sec.
>
> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
> -A syn_flood -m comment --comment "!fw3" -j DROP
>
> Well, it's got a burst-size of 50. I agree that this is quite
> conservative.
>
> However, at least in my home we're not seeing drops:
>
> # iptables -nvL | grep -A 4 "Chain syn_flood"
> Chain syn_flood (1 references)
>   pkts bytes target     prot opt in     out     source               destination
>   2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>      0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>
> But you might be right that in places with a lot more clients then this
> might indeed cause problems.

Well, *I* long ago had upped those params by 10x and don't see syn
drops either on my backbone. But I rather suspect the rest of the
world just copy-pasted it. It should scale as a function of bandwidth,
I suppose, or get updated as a side effect of setting QoS - or just
get bumped up. Start a bug over with openwrt? Take a hard look at
other firewall designs?

Like I said, though, my big question was is there a browser stat or
some other easily accessible stat to see how
often syns are rejected? Another context for this was syn negotiation
with ecn on and the fallback.

Interestingly, I've also seen a pretty big uptick in ecn marking over
the last year or so, on one uplink (we do have a lot of guests that
run apple gear), it's now at over 10% of of the drop ratio on
outbound.

This box is - I hope - the last cerowrt box running in the universe -
and the only reason it ever goes down is because of a long duration
power failure. I've been meaning to replace it for ages...

root@lounge:~# uptime
 07:14:53 up 55 days, 17:14,  load average: 0.16, 0.09, 0.10

outbound:

qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
 Sent 159378714029 bytes 1038654784 pkt (dropped 426065, overlimits 0
requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 213282954 ecn_mark 48220
  new_flows_len 0 old_flows_len 1

inbound: (where comcast remarks most packets to CS1)

qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 1500
target 5.0ms interval 100.0ms ecn
 Sent 40391986695 bytes 34710741 pkt (dropped 420, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 5687382 ecn_mark 0
  new_flows_len 0 old_flows_len 2
qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300
target 5.0ms interval 100.0ms ecn
 Sent 2285974845172 bytes 1748071724 pkt (dropped 61231, overlimits 0
requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1514 drop_overlimit 0 new_flow_count 229072930 ecn_mark 344
  new_flows_len 0 old_flows_len 1


>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

^ permalink raw reply	[flat|nested] 12+ messages in thread

* [Cerowrt-devel] Revising the synflood limit
  2019-09-03 14:21   ` Dave Taht
@ 2019-09-03 14:35     ` Dave Taht
  2019-09-10 18:08       ` Maciej Sołtysiak
  2019-09-03 14:45     ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen
  2019-09-03 15:22     ` Sebastian Moeller
  2 siblings, 1 reply; 12+ messages in thread
From: Dave Taht @ 2019-09-03 14:35 UTC (permalink / raw)
  To: Mikael Abrahamsson; +Cc: bloat, cerowrt-devel

OK, I started a topic over there. It would be good to know how many
other firewall tools set a syn limit by default, but that would take
way more research.

https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 14:21   ` Dave Taht
  2019-09-03 14:35     ` [Cerowrt-devel] Revising the synflood limit Dave Taht
@ 2019-09-03 14:45     ` Toke Høiland-Jørgensen
  2019-09-03 14:47       ` Dave Taht
  2019-09-03 15:22     ` Sebastian Moeller
  2 siblings, 1 reply; 12+ messages in thread
From: Toke Høiland-Jørgensen @ 2019-09-03 14:45 UTC (permalink / raw)
  To: Dave Taht, Mikael Abrahamsson; +Cc: cerowrt-devel, bloat

Dave Taht <dave.taht@gmail.com> writes:

> On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>>
>> On Mon, 2 Sep 2019, Dave Taht wrote:
>>
>> > with copy-pasted parameters set in the 90s - openwrt's default, last I
>> > looked, was 25/sec.
>>
>> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
>> -A syn_flood -m comment --comment "!fw3" -j DROP
>>
>> Well, it's got a burst-size of 50. I agree that this is quite
>> conservative.
>>
>> However, at least in my home we're not seeing drops:
>>
>> # iptables -nvL | grep -A 4 "Chain syn_flood"
>> Chain syn_flood (1 references)
>>   pkts bytes target     prot opt in     out     source               destination
>>   2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>>      0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>>
>> But you might be right that in places with a lot more clients then this
>> might indeed cause problems.
>
> Well, *I* long ago had upped those params by 10x and don't see syn
> drops either on my backbone. But I rather suspect the rest of the
> world just copy-pasted it. It should scale as a function of bandwidth,
> I suppose, or get updated as a side effect of setting QoS - or just
> get bumped up. Start a bug over with openwrt? Take a hard look at
> other firewall designs?

FWIW:

# iptables -nvL syn_flood
Chain syn_flood (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 195K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */

# ip6tables -nvL syn_flood
Chain syn_flood (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  396 41508 RETURN     tcp      *      *       ::/0                 ::/0                 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
    0     0 DROP       all      *      *       ::/0                 ::/0                 /* !fw3 */

rebooted this box today; don't seem to have hit the limit thus far,
though... This is on a gigabit link.

-Toke

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 14:45     ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen
@ 2019-09-03 14:47       ` Dave Taht
  2019-09-03 15:10         ` Toke Høiland-Jørgensen
  0 siblings, 1 reply; 12+ messages in thread
From: Dave Taht @ 2019-09-03 14:47 UTC (permalink / raw)
  To: Toke Høiland-Jørgensen; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Dave Taht <dave.taht@gmail.com> writes:
>
> > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >>
> >> On Mon, 2 Sep 2019, Dave Taht wrote:
> >>
> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I
> >> > looked, was 25/sec.
> >>
> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
> >> -A syn_flood -m comment --comment "!fw3" -j DROP
> >>
> >> Well, it's got a burst-size of 50. I agree that this is quite
> >> conservative.
> >>
> >> However, at least in my home we're not seeing drops:
> >>
> >> # iptables -nvL | grep -A 4 "Chain syn_flood"
> >> Chain syn_flood (1 references)
> >>   pkts bytes target     prot opt in     out     source               destination
> >>   2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
> >>      0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
> >>
> >> But you might be right that in places with a lot more clients then this
> >> might indeed cause problems.
> >
> > Well, *I* long ago had upped those params by 10x and don't see syn
> > drops either on my backbone. But I rather suspect the rest of the
> > world just copy-pasted it. It should scale as a function of bandwidth,
> > I suppose, or get updated as a side effect of setting QoS - or just
> > get bumped up. Start a bug over with openwrt? Take a hard look at
> > other firewall designs?
>
> FWIW:
>
> # iptables -nvL syn_flood
> Chain syn_flood (1 references)
>  pkts bytes target     prot opt in     out     source               destination
>  195K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>
> # ip6tables -nvL syn_flood
> Chain syn_flood (1 references)
>  pkts bytes target     prot opt in     out     source               destination
>   396 41508 RETURN     tcp      *      *       ::/0                 ::/0                 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>     0     0 DROP       all      *      *       ::/0                 ::/0                 /* !fw3 */
>
> rebooted this box today; don't seem to have hit the limit thus far,
> though... This is on a gigabit link.

Hmm. Try to trigger it with --te=upload_streams=200 ?

>
> -Toke



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 14:47       ` Dave Taht
@ 2019-09-03 15:10         ` Toke Høiland-Jørgensen
  2019-09-03 15:21           ` Dave Taht
  0 siblings, 1 reply; 12+ messages in thread
From: Toke Høiland-Jørgensen @ 2019-09-03 15:10 UTC (permalink / raw)
  To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

Dave Taht <dave.taht@gmail.com> writes:

> On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>>
>> Dave Taht <dave.taht@gmail.com> writes:
>>
>> > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> >>
>> >> On Mon, 2 Sep 2019, Dave Taht wrote:
>> >>
>> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I
>> >> > looked, was 25/sec.
>> >>
>> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
>> >> -A syn_flood -m comment --comment "!fw3" -j DROP
>> >>
>> >> Well, it's got a burst-size of 50. I agree that this is quite
>> >> conservative.
>> >>
>> >> However, at least in my home we're not seeing drops:
>> >>
>> >> # iptables -nvL | grep -A 4 "Chain syn_flood"
>> >> Chain syn_flood (1 references)
>> >>   pkts bytes target     prot opt in     out     source               destination
>> >>   2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>> >>      0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>> >>
>> >> But you might be right that in places with a lot more clients then this
>> >> might indeed cause problems.
>> >
>> > Well, *I* long ago had upped those params by 10x and don't see syn
>> > drops either on my backbone. But I rather suspect the rest of the
>> > world just copy-pasted it. It should scale as a function of bandwidth,
>> > I suppose, or get updated as a side effect of setting QoS - or just
>> > get bumped up. Start a bug over with openwrt? Take a hard look at
>> > other firewall designs?
>>
>> FWIW:
>>
>> # iptables -nvL syn_flood
>> Chain syn_flood (1 references)
>>  pkts bytes target     prot opt in     out     source               destination
>>  195K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>>
>> # ip6tables -nvL syn_flood
>> Chain syn_flood (1 references)
>>  pkts bytes target     prot opt in     out     source               destination
>>   396 41508 RETURN     tcp      *      *       ::/0                 ::/0                 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>>     0     0 DROP       all      *      *       ::/0                 ::/0                 /* !fw3 */
>>
>> rebooted this box today; don't seem to have hit the limit thus far,
>> though... This is on a gigabit link.
>
> Hmm. Try to trigger it with --te=upload_streams=200 ?

Sure, that triggers it:

# iptables -nvL syn_flood
Chain syn_flood (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 197K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
  275 16480 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */


And I get tons of errors from netperf failing to start up.

However, the protection is only actually enabled for the INPUT chain;
i.e., I had to use the router itself as the netperf target to trigger
the rule. So not sure a rule such as this would be the cause of your
coffee shop failures?

This is with the default openwrt config, BTW:


config defaults
        option syn_flood '1'


-Toke

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 15:10         ` Toke Høiland-Jørgensen
@ 2019-09-03 15:21           ` Dave Taht
  0 siblings, 0 replies; 12+ messages in thread
From: Dave Taht @ 2019-09-03 15:21 UTC (permalink / raw)
  To: Toke Høiland-Jørgensen; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

On Tue, Sep 3, 2019 at 8:10 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>
> Dave Taht <dave.taht@gmail.com> writes:
>
> > On Tue, Sep 3, 2019 at 7:45 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> >>
> >> Dave Taht <dave.taht@gmail.com> writes:
> >>
> >> > On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >> >>
> >> >> On Mon, 2 Sep 2019, Dave Taht wrote:
> >> >>
> >> >> > with copy-pasted parameters set in the 90s - openwrt's default, last I
> >> >> > looked, was 25/sec.
> >> >>
> >> >> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
> >> >> -A syn_flood -m comment --comment "!fw3" -j DROP
> >> >>
> >> >> Well, it's got a burst-size of 50. I agree that this is quite
> >> >> conservative.
> >> >>
> >> >> However, at least in my home we're not seeing drops:
> >> >>
> >> >> # iptables -nvL | grep -A 4 "Chain syn_flood"
> >> >> Chain syn_flood (1 references)
> >> >>   pkts bytes target     prot opt in     out     source               destination
> >> >>   2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
> >> >>      0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
> >> >>
> >> >> But you might be right that in places with a lot more clients then this
> >> >> might indeed cause problems.
> >> >
> >> > Well, *I* long ago had upped those params by 10x and don't see syn
> >> > drops either on my backbone. But I rather suspect the rest of the
> >> > world just copy-pasted it. It should scale as a function of bandwidth,
> >> > I suppose, or get updated as a side effect of setting QoS - or just
> >> > get bumped up. Start a bug over with openwrt? Take a hard look at
> >> > other firewall designs?
> >>
> >> FWIW:
> >>
> >> # iptables -nvL syn_flood
> >> Chain syn_flood (1 references)
> >>  pkts bytes target     prot opt in     out     source               destination
> >>  195K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
> >>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
> >>
> >> # ip6tables -nvL syn_flood
> >> Chain syn_flood (1 references)
> >>  pkts bytes target     prot opt in     out     source               destination
> >>   396 41508 RETURN     tcp      *      *       ::/0                 ::/0                 tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
> >>     0     0 DROP       all      *      *       ::/0                 ::/0                 /* !fw3 */
> >>
> >> rebooted this box today; don't seem to have hit the limit thus far,
> >> though... This is on a gigabit link.
> >
> > Hmm. Try to trigger it with --te=upload_streams=200 ?
>
> Sure, that triggers it:
>
> # iptables -nvL syn_flood
> Chain syn_flood (1 references)
>  pkts bytes target     prot opt in     out     source               destination
>  197K   12M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>   275 16480 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>
>
> And I get tons of errors from netperf failing to start up.
>
> However, the protection is only actually enabled for the INPUT chain;
> i.e., I had to use the router itself as the netperf target to trigger
> the rule. So not sure a rule such as this would be the cause of your
> coffee shop failures?

Good point. I think in the cerowrt case I'd enabled it for all chains
and then noticed, and bumped it up. Limiting syn attempts to the
router itself makes more sense than doing it on the forward path.

The coffee shop tests were fun, but I(we) needed more rigor when doing
them. What I'd typically do is go in,
get on the wifi, start 6 minutes worth of tests, get in line, get
coffee... I would feel a twinge of guilt when I'd
start seeing heads pop up from their laptops while running the rrul
test, but in two cases I managed to talk to
the owner, help 'em get QoS configured right on their router, show'd
'em the difference, and got a free meal out of it...

So, it was only discovering that I'd had that 30% rejection figure via
the stats on my osx box that I started speculating
as to the cause(s), last week. The other big fear is that because OSX
is always attempting to negotiate ECN, that that's a cause.

A bit more rigor about this and it could turn into a good paper or
blog entry, and it's good to get out more for the
sake of science, with table service!




>
> This is with the default openwrt config, BTW:
>
>
> config defaults
>         option syn_flood '1'
>
>
> -Toke



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week
  2019-09-03 14:21   ` Dave Taht
  2019-09-03 14:35     ` [Cerowrt-devel] Revising the synflood limit Dave Taht
  2019-09-03 14:45     ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen
@ 2019-09-03 15:22     ` Sebastian Moeller
  2 siblings, 0 replies; 12+ messages in thread
From: Sebastian Moeller @ 2019-09-03 15:22 UTC (permalink / raw)
  To: Dave Täht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

Not sure this is on-topic, but:

https://www.prnewswire.com/news-releases/root-cause-analysis-and-incident-report-on-the-august-ddos-attack-300905405.html

https://lists.gt.net/nanog/users/206044


> On Sep 3, 2019, at 16:21, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Tue, Sep 3, 2019 at 5:23 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>> 
>> On Mon, 2 Sep 2019, Dave Taht wrote:
>> 
>>> with copy-pasted parameters set in the 90s - openwrt's default, last I
>>> looked, was 25/sec.
>> 
>> -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
>> -A syn_flood -m comment --comment "!fw3" -j DROP
>> 
>> Well, it's got a burst-size of 50. I agree that this is quite
>> conservative.
>> 
>> However, at least in my home we're not seeing drops:
>> 
>> # iptables -nvL | grep -A 4 "Chain syn_flood"
>> Chain syn_flood (1 references)
>>  pkts bytes target     prot opt in     out     source               destination
>>  2296  113K RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 25/sec burst 50 /* !fw3 */
>>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* !fw3 */
>> 
>> But you might be right that in places with a lot more clients then this
>> might indeed cause problems.
> 
> Well, *I* long ago had upped those params by 10x and don't see syn
> drops either on my backbone. But I rather suspect the rest of the
> world just copy-pasted it. It should scale as a function of bandwidth,
> I suppose, or get updated as a side effect of setting QoS - or just
> get bumped up. Start a bug over with openwrt? Take a hard look at
> other firewall designs?
> 
> Like I said, though, my big question was is there a browser stat or
> some other easily accessible stat to see how
> often syns are rejected? Another context for this was syn negotiation
> with ecn on and the fallback.
> 
> Interestingly, I've also seen a pretty big uptick in ecn marking over
> the last year or so, on one uplink (we do have a lot of guests that
> run apple gear), it's now at over 10% of of the drop ratio on
> outbound.
> 
> This box is - I hope - the last cerowrt box running in the universe -
> and the only reason it ever goes down is because of a long duration
> power failure. I've been meaning to replace it for ages...
> 
> root@lounge:~# uptime
> 07:14:53 up 55 days, 17:14,  load average: 0.16, 0.09, 0.10
> 
> outbound:
> 
> qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 300
> target 5.0ms interval 100.0ms ecn
> Sent 159378714029 bytes 1038654784 pkt (dropped 426065, overlimits 0
> requeues 0)
> backlog 0b 0p requeues 0
>  maxpacket 1514 drop_overlimit 0 new_flow_count 213282954 ecn_mark 48220
>  new_flows_len 0 old_flows_len 1
> 
> inbound: (where comcast remarks most packets to CS1)
> 
> qdisc fq_codel 120: parent 1:12 limit 1001p flows 1024 quantum 1500
> target 5.0ms interval 100.0ms ecn
> Sent 40391986695 bytes 34710741 pkt (dropped 420, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>  maxpacket 1514 drop_overlimit 0 new_flow_count 5687382 ecn_mark 0
>  new_flows_len 0 old_flows_len 2
> qdisc fq_codel 130: parent 1:13 limit 1001p flows 1024 quantum 300
> target 5.0ms interval 100.0ms ecn
> Sent 2285974845172 bytes 1748071724 pkt (dropped 61231, overlimits 0
> requeues 0)
> backlog 0b 0p requeues 0
>  maxpacket 1514 drop_overlimit 0 new_flow_count 229072930 ecn_mark 344
>  new_flows_len 0 old_flows_len 1
> 
> 
>> 
>> --
>> Mikael Abrahamsson    email: swmike@swm.pp.se
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] Revising the synflood limit
  2019-09-03 14:35     ` [Cerowrt-devel] Revising the synflood limit Dave Taht
@ 2019-09-10 18:08       ` Maciej Sołtysiak
  2019-09-10 22:35         ` Dave Taht
  0 siblings, 1 reply; 12+ messages in thread
From: Maciej Sołtysiak @ 2019-09-10 18:08 UTC (permalink / raw)
  To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

> OK, I started a topic over there. It would be good to know how many
> other firewall tools set a syn limit by default, but that would take
> way more research.
>
> https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957
Nice catch! I've 10 to 15 devices on my network. Just in 24h I see I had 10k SYN's rejected!


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] Revising the synflood limit
  2019-09-10 18:08       ` Maciej Sołtysiak
@ 2019-09-10 22:35         ` Dave Taht
  2019-09-23  9:37           ` Maciej Sołtysiak
  0 siblings, 1 reply; 12+ messages in thread
From: Dave Taht @ 2019-09-10 22:35 UTC (permalink / raw)
  To: Maciej Sołtysiak; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

I'm not sure if it is a "nice catch" or not yet. It's merely me (now
us) tying two anomalies together that might be connected. Can you
convert that -j drop to a -j log to see where they come from?

For example I was failing to negotiate ecn while at this conference,
which also meant a dropped syn.

On Tue, Sep 10, 2019 at 7:08 PM Maciej Sołtysiak <maciej@soltysiak.com> wrote:
>
> > OK, I started a topic over there. It would be good to know how many
> > other firewall tools set a syn limit by default, but that would take
> > way more research.
> >
> > https://forum.openwrt.org/t/the-synflood-limit-is-too-low-for-the-modern-internet/43957
> Nice catch! I've 10 to 15 devices on my network. Just in 24h I see I had 10k SYN's rejected!
>


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [Cerowrt-devel] Revising the synflood limit
  2019-09-10 22:35         ` Dave Taht
@ 2019-09-23  9:37           ` Maciej Sołtysiak
  0 siblings, 0 replies; 12+ messages in thread
From: Maciej Sołtysiak @ 2019-09-23  9:37 UTC (permalink / raw)
  To: Dave Taht; +Cc: Mikael Abrahamsson, cerowrt-devel, bloat

> I'm not sure if it is a "nice catch" or not yet. It's merely me (now
> us) tying two anomalies together that might be connected. Can you
> convert that -j drop to a -j log to see where they come from?
So, I must say I got rusty with my iptables.
When I said I have the same, I had positive packet counters on the limit rule, not the drop rule (or the log rule I added)

So, sorry to jump the gun, but I'm not experiencing it.

Maciej


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2019-09-23  9:38 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-03  0:47 [Cerowrt-devel] talking at linux plumbers in portugal next week Dave Taht
2019-09-03 12:23 ` [Cerowrt-devel] [Bloat] " Mikael Abrahamsson
2019-09-03 14:21   ` Dave Taht
2019-09-03 14:35     ` [Cerowrt-devel] Revising the synflood limit Dave Taht
2019-09-10 18:08       ` Maciej Sołtysiak
2019-09-10 22:35         ` Dave Taht
2019-09-23  9:37           ` Maciej Sołtysiak
2019-09-03 14:45     ` [Cerowrt-devel] [Bloat] talking at linux plumbers in portugal next week Toke Høiland-Jørgensen
2019-09-03 14:47       ` Dave Taht
2019-09-03 15:10         ` Toke Høiland-Jørgensen
2019-09-03 15:21           ` Dave Taht
2019-09-03 15:22     ` Sebastian Moeller

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox