[Ecn-sane] osx ecn

[Ecn-sane] osx ecn

[Dave Taht]

netstat -sp tcp has all kinds of useful stats


        7174 client connections attempted to negotiate ECN

                4407 client connections successfully negotiated ECN

                2192 times graceful fallback to Non-ECN connection

                2053 times lost ECN negotiating SYN, followed by retransmission

??
                0 server connection attempted to negotiate ECN

                0 server connection successfully negotiated ECN

                0 time lost ECN negotiating SYN-ACK, followed by retransmission

                90 times received congestion experienced (CE) notification

                0 time CWR was sent in response to ECE

                3298 times sent ECE notification

                25 connections received CE atleast once

                0 connection received ECE atleast once

                830 connections using ECN have seen packet loss but no CE

                12 connections using ECN have seen packet loss and CE

                13 connections using ECN received CE but no packet loss

                9 connections fell back to non-ECN due to SYN-loss

                135 connections fell back to non-ECN due to reordering

                0 connection fell back to non-ECN due to excessive CE-markings



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Mikael Abrahamsson]

On Mon, 26 Aug 2019, Dave Taht wrote:

> netstat -sp tcp has all kinds of useful stats

I am using the settings from 
https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ but I still 
get 0 on attempted ECN connections?

https://mailarchive.ietf.org/arch/msg/iccrg/ghBJK_x9UqD6a5RzqWY-FmpPzBc

I obviously had it working before. I'm on 10.14.6, what are you on?

Btw, I checked what happened if I removed the sysctl settings and then it 
defaults to "2" instead of "1".

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] osx ecn

[Michael Welzl]

run it as root - strange, yes, but then the output is very different.

cheers,
michael


Sent from my iPhone

> On 27 Aug 2019, at 08:47, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> 
>> On Mon, 26 Aug 2019, Dave Taht wrote:
>> 
>> netstat -sp tcp has all kinds of useful stats
> 
> I am using the settings from https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ but I still get 0 on attempted ECN connections?
> 
> https://mailarchive.ietf.org/arch/msg/iccrg/ghBJK_x9UqD6a5RzqWY-FmpPzBc
> 
> I obviously had it working before. I'm on 10.14.6, what are you on?
> 
> Btw, I checked what happened if I removed the sysctl settings and then it defaults to "2" instead of "1".
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> Ecn-sane mailing list
> Ecn-sane@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane

Re: [Ecn-sane] osx ecn

[Sebastian Moeller]

Hi Dave,


> On Aug 27, 2019, at 04:03, Dave Taht <dave.taht@gmail.com> wrote:
> 
>  0 connection fell back to non-ECN due to excessive CE-markings

I wonder whether macosx would be able to gracefully survive a misclassification into a L4S/NQB queue at a dual queue AQM? What is the threshold here?

Best Regards
	Sebastian

Re: [Ecn-sane] osx ecn

[Mikael Abrahamsson]

On Tue, 27 Aug 2019, Michael Welzl wrote:

> run it as root - strange, yes, but then the output is very different.

Thanks, indeed.

$ sudo netstat -sp tcp | egrep 'CE|EC'
 	11097 client connections attempted to negotiate ECN
 		4864 client connections successfully negotiated ECN
 		3800 times graceful fallback to Non-ECN connection
 		467 times lost ECN negotiating SYN, followed by retransmission
 		52 server connections attempted to negotiate ECN
 		52 server connections successfully negotiated ECN
 		0 time lost ECN negotiating SYN-ACK, followed by retransmission
 		427 times received congestion experienced (CE) notification
 		0 time CWR was sent in response to ECE
 		6111 times sent ECE notification
 		15 connections received CE atleast once
 		0 connection received ECE atleast once
 		267 connections using ECN have seen packet loss but no CE
 		1 connection using ECN have seen packet loss and CE
 		14 connections using ECN received CE but no packet loss
 		14 connections fell back to non-ECN due to SYN-loss
 		1 connection fell back to non-ECN due to reordering
 		0 connection fell back to non-ECN due to excessive CE-markings


-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] osx ecn

[Jonathan Morton]

> On 27 Aug, 2019, at 9:58 am, Sebastian Moeller <moeller0@gmx.de> wrote:
> 
>> 0 connection fell back to non-ECN due to excessive CE-markings
> 
> I wonder whether macosx would be able to gracefully survive a misclassification into a L4S/NQB queue at a dual queue AQM? What is the threshold here?

No, I don't think so.  L4S assumes the DCTCP response function which stabilises at 2 CE marks per RTT, which is still a fairly low percentage if the cwnd is high.  I doubt that would be detected as "excessive" in this context.  Also, because the RFC-3168 response to CE is much sharper, the actual marking rate will be lower, as the flow won't stay in the saturating regime where marking occurs.

 - Jonathan Morton

Re: [Ecn-sane] osx ecn

[Sebastian Moeller]

Hi Jonathan,


> On Aug 27, 2019, at 09:05, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 27 Aug, 2019, at 9:58 am, Sebastian Moeller <moeller0@gmx.de> wrote:
>> 
>>> 0 connection fell back to non-ECN due to excessive CE-markings
>> 
>> I wonder whether macosx would be able to gracefully survive a misclassification into a L4S/NQB queue at a dual queue AQM? What is the threshold here?
> 
> No, I don't think so.  L4S assumes the DCTCP response function which stabilises at 2 CE marks per RTT, which is still a fairly low percentage if the cwnd is high.  I doubt that would be detected as "excessive" in this context.  Also, because the RFC-3168 response to CE is much sharper, the actual marking rate will be lower, as the flow won't stay in the saturating regime where marking occurs.
> 
> - Jonathan Morton


thanks for the detailed answer.

Best Regards
	Sebastian

Re: [Ecn-sane] osx ecn

[Dave Taht]

An interesing piece of this is how often the syn attempt is failing.

             2192 times graceful fallback to Non-ECN connection

                2053 times lost ECN negotiating SYN, followed by retransmission

??

Now, there are plenty of reasons to lose syns (most firewalls rate limit these),
besides the ecn flag.

In my case during this boot cycle I've been engaged in a fun
experiment - and one day perhaps a paper - Going to various coffee
shops in the area and running the attached script. The results thus
far are universally dismal,
except for google starbucks which is excellent - but the packet
captures seem to indicate something in addition to
or other than than fq_codel in use (dpi maybe? aruba is the next hop)
AND it did not negotiate ecn when I last tried.

Wifi spread initially through the coffee shop phenomenon, and I've
"lept over the counter" a couple times
to help configure the to-the-isp qos system when I found an owner with
clue, which helps a lot. When I showed 'em the before/after two bought
me lunch (and I can highly recommend https://www.cemitas1.com/ in
davenport for food, music, view & wifi) if that's a motivator for more
folk here "to get out more", please do so. Table service is a nice
thing to have when in deep hack mode and socializing more helps too. :

daves-Air-6:los_gatos_starbucks d$ more testit.sh

#!/bin/sh

T="Los_Gatos_Starbucks"
F="flent -x -H flent-fremont.bufferbloat.net -t $T" # pick a closer
flent server and on linux use --socket-stats
$F --te=download_streams=4 tcp_ndown
$F --te=upload_streams=4 tcp_nup
$F --te=upload_streams=4 tcp_2up_square
$F rrul
$F rrul_be

On Tue, Aug 27, 2019 at 12:02 AM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Tue, 27 Aug 2019, Michael Welzl wrote:
>
> > run it as root - strange, yes, but then the output is very different.
>
> Thanks, indeed.
>
> $ sudo netstat -sp tcp | egrep 'CE|EC'
>         11097 client connections attempted to negotiate ECN
>                 4864 client connections successfully negotiated ECN
>                 3800 times graceful fallback to Non-ECN connection
>                 467 times lost ECN negotiating SYN, followed by retransmission
>                 52 server connections attempted to negotiate ECN
>                 52 server connections successfully negotiated ECN
>                 0 time lost ECN negotiating SYN-ACK, followed by retransmission
>                 427 times received congestion experienced (CE) notification
>                 0 time CWR was sent in response to ECE
>                 6111 times sent ECE notification
>                 15 connections received CE atleast once
>                 0 connection received ECE atleast once
>                 267 connections using ECN have seen packet loss but no CE
>                 1 connection using ECN have seen packet loss and CE
>                 14 connections using ECN received CE but no packet loss
>                 14 connections fell back to non-ECN due to SYN-loss
>                 1 connection fell back to non-ECN due to reordering
>                 0 connection fell back to non-ECN due to excessive CE-markings
>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Dave Taht]

On Mon, Aug 26, 2019 at 11:54 PM Michael Welzl <michawe@ifi.uio.no> wrote:
>
> run it as root - strange, yes, but then the output is very different.
>
> cheers,
> michael
>
>
> Sent from my iPhone

speaking of iphones I have no idea what they do nowadays, any odds you
could do a packet
cap either on it or at a server or via wifi on a middlebox
> > On 27 Aug 2019, at 08:47, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> >
> >> On Mon, 26 Aug 2019, Dave Taht wrote:
> >>
> >> netstat -sp tcp has all kinds of useful stats
> >
> > I am using the settings from https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ but I still get 0 on attempted ECN connections?
> >
> > https://mailarchive.ietf.org/arch/msg/iccrg/ghBJK_x9UqD6a5RzqWY-FmpPzBc
> >
> > I obviously had it working before. I'm on 10.14.6, what are you on?
> >
> > Btw, I checked what happened if I removed the sysctl settings and then it defaults to "2" instead of "1".
> >
> > --
> > Mikael Abrahamsson    email: swmike@swm.pp.se
> > _______________________________________________
> > Ecn-sane mailing list
> > Ecn-sane@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/ecn-sane
>


-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Michael Welzl]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]



> On Aug 27, 2019, at 8:36 PM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Mon, Aug 26, 2019 at 11:54 PM Michael Welzl <michawe@ifi.uio.no <mailto:michawe@ifi.uio.no>> wrote:
>> 
>> run it as root - strange, yes, but then the output is very different.
>> 
>> cheers,
>> michael
>> 
>> 
>> Sent from my iPhone
> 
> speaking of iphones I have no idea what they do nowadays, any odds you
> could do a packet
> cap either on it or at a server or via wifi on a middlebox

I wouldn’t know … but I can guess: some years ago, I did a “jailbreak” on my iPhone, and as a result, I could ssh into it.
So I’m guessing: if you do this with a current one, you can probably still ssh into it and then perhaps run netstat.

Cheers,
Michael


[-- Attachment #2: Type: text/html, Size: 5669 bytes --]

Re: [Ecn-sane] osx ecn

[Ryan Mounce]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On Wed, 28 Aug 2019 at 04:30, Michael Welzl <michawe@ifi.uio.no> wrote:

>
>
> On Aug 27, 2019, at 8:36 PM, Dave Taht <dave.taht@gmail.com> wrote:
>
> On Mon, Aug 26, 2019 at 11:54 PM Michael Welzl <michawe@ifi.uio.no> wrote:
>
>
> run it as root - strange, yes, but then the output is very different.
>
> cheers,
> michael
>
>
> Sent from my iPhone
>
>
> speaking of iphones I have no idea what they do nowadays, any odds you
> could do a packet
> cap either on it or at a server or via wifi on a middlebox
>
>
> I wouldn’t know … but I can guess: some years ago, I did a “jailbreak” on
> my iPhone, and as a result, I could ssh into it.
> So I’m guessing: if you do this with a current one, you can probably still
> ssh into it and then perhaps run netstat.
>
> Cheers,
> Michael
>

You sure can :)

https://gist.github.com/rmounce/74dd5e92e6304688219024349bfaad0e

-Ryan


> --
Regards,
Ryan Mounce

ryan@mounce.com.au
0415 799 929

Sent from mobile

[-- Attachment #2: Type: text/html, Size: 5469 bytes --]

Re: [Ecn-sane] osx ecn

[Dave Taht]

Very cool, thank you for jailbreaking.

I'm a little puzzled over the cwr/ece ratio?

25 times received congestion experienced (CE) notification
37 times CWR was sent in response to ECE
619 times sent ECE notification
1029 connections received CE atleast once

And um, er, while you are here busting your warranty for the sake of
science, I'm dying to know what

 netstat -I each_interface_you_have_especially_lte -qq

shows. They have a really short queue and some form of flow control
and I've never seen anything trigger except though a vm

my laptop:

en0:

     [ sched:  FQ_CODEL  qlength:    0/128 ]

     [ pkts:          0  bytes:          0  dropped pkts:   1087 bytes: 177812 ]

=====================================================

     [ pri: VO (1)      srv_cl: 0x400180        quantum: 600    drr_max: 8 ]

     [ queued pkts: 0   bytes: 0 ]

     [ dequeued pkts: 36767     bytes: 5237639 ]

     [ budget: 0        target qdelay: 10.00 msec       update
interval:100.00 msec ]

     [ flow control: 0  feedback: 0     stalls: 0       failed: 0 ]

     [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]

     [ flows total: 0   new: 0  old: 0 ]

     [ throttle on: 0   off: 0  drop: 0 ]

=====================================================

     [ pri: VI (2)      srv_cl: 0x380100        quantum: 3000   drr_max: 6 ]

     [ queued pkts: 0   bytes: 0 ]

     [ dequeued pkts: 96836     bytes: 36619772 ]

     [ budget: 0        target qdelay: 10.00 msec       update
interval:100.00 msec ]

     [ flow control: 0  feedback: 0     stalls: 0       failed: 0 ]

     [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]

     [ flows total: 0   new: 0  old: 0 ]

     [ throttle on: 0   off: 0  drop: 0 ]

=====================================================

     [ pri: BE (7)      srv_cl: 0x0     quantum: 1500   drr_max: 4 ]

     [ queued pkts: 0   bytes: 0 ]

     [ dequeued pkts: 5212250   bytes: 2820548356 ]

     [ budget: 0        target qdelay: 10.00 msec       update
interval:100.00 msec ]

     [ flow control: 30 feedback: 30    stalls: 0       failed: 0 ]

     [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]

     [ flows total: 0   new: 0  old: 0 ]

     [ throttle on: 0   off: 0  drop: 0 ]

On Wed, Aug 28, 2019 at 4:29 PM Ryan Mounce <ryan@mounce.com.au> wrote:
>
> On Wed, 28 Aug 2019 at 04:30, Michael Welzl <michawe@ifi.uio.no> wrote:
>>
>>
>>
>> On Aug 27, 2019, at 8:36 PM, Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Mon, Aug 26, 2019 at 11:54 PM Michael Welzl <michawe@ifi.uio.no> wrote:
>>
>>
>> run it as root - strange, yes, but then the output is very different.
>>
>> cheers,
>> michael
>>
>>
>> Sent from my iPhone
>>
>>
>> speaking of iphones I have no idea what they do nowadays, any odds you
>> could do a packet
>> cap either on it or at a server or via wifi on a middlebox
>>
>>
>> I wouldn’t know … but I can guess: some years ago, I did a “jailbreak” on my iPhone, and as a result, I could ssh into it.
>> So I’m guessing: if you do this with a current one, you can probably still ssh into it and then perhaps run netstat.
>>
>> Cheers,
>> Michael
>
>
> You sure can :)
>
> https://gist.github.com/rmounce/74dd5e92e6304688219024349bfaad0e
>
> -Ryan
>
>>
> --
> Regards,
> Ryan Mounce
>
> ryan@mounce.com.au
> 0415 799 929
>
> Sent from mobile



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Dave Taht]

fq_codel on osx ethernet (I'd only seen it on wifi before)

https://discussions.apple.com/thread/250589601

On Wed, Aug 28, 2019 at 4:45 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> Very cool, thank you for jailbreaking.
>
> I'm a little puzzled over the cwr/ece ratio?
>
> 25 times received congestion experienced (CE) notification
> 37 times CWR was sent in response to ECE
> 619 times sent ECE notification
> 1029 connections received CE atleast once
>
> And um, er, while you are here busting your warranty for the sake of
> science, I'm dying to know what
>
>  netstat -I each_interface_you_have_especially_lte -qq
>
> shows. They have a really short queue and some form of flow control
> and I've never seen anything trigger except though a vm
>
> my laptop:
>
> en0:
>
>      [ sched:  FQ_CODEL  qlength:    0/128 ]
>
>      [ pkts:          0  bytes:          0  dropped pkts:   1087 bytes: 177812 ]
>
> =====================================================
>
>      [ pri: VO (1)      srv_cl: 0x400180        quantum: 600    drr_max: 8 ]
>
>      [ queued pkts: 0   bytes: 0 ]
>
>      [ dequeued pkts: 36767     bytes: 5237639 ]
>
>      [ budget: 0        target qdelay: 10.00 msec       update
> interval:100.00 msec ]
>
>      [ flow control: 0  feedback: 0     stalls: 0       failed: 0 ]
>
>      [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]
>
>      [ flows total: 0   new: 0  old: 0 ]
>
>      [ throttle on: 0   off: 0  drop: 0 ]
>
> =====================================================
>
>      [ pri: VI (2)      srv_cl: 0x380100        quantum: 3000   drr_max: 6 ]
>
>      [ queued pkts: 0   bytes: 0 ]
>
>      [ dequeued pkts: 96836     bytes: 36619772 ]
>
>      [ budget: 0        target qdelay: 10.00 msec       update
> interval:100.00 msec ]
>
>      [ flow control: 0  feedback: 0     stalls: 0       failed: 0 ]
>
>      [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]
>
>      [ flows total: 0   new: 0  old: 0 ]
>
>      [ throttle on: 0   off: 0  drop: 0 ]
>
> =====================================================
>
>      [ pri: BE (7)      srv_cl: 0x0     quantum: 1500   drr_max: 4 ]
>
>      [ queued pkts: 0   bytes: 0 ]
>
>      [ dequeued pkts: 5212250   bytes: 2820548356 ]
>
>      [ budget: 0        target qdelay: 10.00 msec       update
> interval:100.00 msec ]
>
>      [ flow control: 30 feedback: 30    stalls: 0       failed: 0 ]
>
>      [ drop overflow: 0 early: 0        memfail: 0      duprexmt:0 ]
>
>      [ flows total: 0   new: 0  old: 0 ]
>
>      [ throttle on: 0   off: 0  drop: 0 ]
>
> On Wed, Aug 28, 2019 at 4:29 PM Ryan Mounce <ryan@mounce.com.au> wrote:
> >
> > On Wed, 28 Aug 2019 at 04:30, Michael Welzl <michawe@ifi.uio.no> wrote:
> >>
> >>
> >>
> >> On Aug 27, 2019, at 8:36 PM, Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> On Mon, Aug 26, 2019 at 11:54 PM Michael Welzl <michawe@ifi.uio.no> wrote:
> >>
> >>
> >> run it as root - strange, yes, but then the output is very different.
> >>
> >> cheers,
> >> michael
> >>
> >>
> >> Sent from my iPhone
> >>
> >>
> >> speaking of iphones I have no idea what they do nowadays, any odds you
> >> could do a packet
> >> cap either on it or at a server or via wifi on a middlebox
> >>
> >>
> >> I wouldn’t know … but I can guess: some years ago, I did a “jailbreak” on my iPhone, and as a result, I could ssh into it.
> >> So I’m guessing: if you do this with a current one, you can probably still ssh into it and then perhaps run netstat.
> >>
> >> Cheers,
> >> Michael
> >
> >
> > You sure can :)
> >
> > https://gist.github.com/rmounce/74dd5e92e6304688219024349bfaad0e
> >
> > -Ryan
> >
> >>
> > --
> > Regards,
> > Ryan Mounce
> >
> > ryan@mounce.com.au
> > 0415 799 929
> >
> > Sent from mobile
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Mikael Abrahamsson]

On Wed, 28 Aug 2019, Dave Taht wrote:

> fq_codel on osx ethernet (I'd only seen it on wifi before)
>
> https://discussions.apple.com/thread/250589601

On my iMac running 10.14.6 both en0 (wired ethernet) and en1 (wifi) has 
FQ_CODEL enabled. Interesting!

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] osx ecn

[Dave Taht]

On Wed, Aug 28, 2019 at 8:28 PM Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
> On Wed, 28 Aug 2019, Dave Taht wrote:
>
> > fq_codel on osx ethernet (I'd only seen it on wifi before)
> >
> > https://discussions.apple.com/thread/250589601
>
> On my iMac running 10.14.6 both en0 (wired ethernet) and en1 (wifi) has
> FQ_CODEL enabled. Interesting!

Only microsoft and cablelabs are left for world domination. I should
go bug some folk I know in Microsoft.

Thing is I've not seen it really trigger on wifi at least according to
the stats. I see "flow control".

Could you hit it really hard with flent locally and see what happens?

flent -H myownlanboxalsoatgige --te=upload_streams=200 tcp_nup

and poll the netstat option.

My guess is it just triggers on vms
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Ecn-sane] osx ecn

[Jonathan Morton]

> On 29 Aug, 2019, at 2:45 am, Dave Taht <dave.taht@gmail.com> wrote:
> 
> I'm a little puzzled over the cwr/ece ratio?
> 
> 25 times received congestion experienced (CE) notification

This is at the IP layer, on received data.

> 619 times sent ECE notification

This is at the TCP layer, acking received data.  (ECE is sticky until CWR, so the ratio of the above is related to the average cwnd at congestion onset.)

> 37 times CWR was sent in response to ECE

This is at the TCP layer, on *sent* data, and therefore independent of the other two.

 - Jonathan Morton

Re: [Ecn-sane] osx ecn

[Mikael Abrahamsson]

On Wed, 28 Aug 2019, Dave Taht wrote:

> Could you hit it really hard with flent locally and see what happens?
>
> flent -H myownlanboxalsoatgige --te=upload_streams=200 tcp_nup
>
> and poll the netstat option.
>
> My guess is it just triggers on vms

I ran two concurrent sessions against two machines at home, each with 100 
streams (just to make sure I wasn't bottlenecking anything else apart from 
the iMac port).

TCP upload sum  :       473.50       464.35 Mbits/s         304

TCP upload sum  :       487.96       475.68 Mbits/s         304

My iMac has 30 hours uptime, so there's some other traffic in there as 
well.

en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 
1500 index 7
 	eflags=400009c0<ACCEPT_RTADV,TXSTART,RXPOLL,ARPLL,FASTLN_ON>
 	options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
...
 	nd6 options=201<PERFORMNUD,DAD>
 	media: autoselect (1000baseT <full-duplex>)
 	status: active
 	type: Ethernet
 	link quality: 100 (good)
 	state availability: 0 (true)
 	scheduler: FQ_CODEL
 	link rate: 1.00 Gbps
 	qosmarking enabled: yes mode: none
 	low power mode: disabled


tcp:
 	75183587 packets sent
 		59332651 data packets (3521003642 byte)
 		1387664 data packets (1738258246 bytes) retransmitted
 		0 resend initiated by MTU discovery
 		13380244 ack-only packets (765 delayed)
 		0 URG only packet
 		102 window probe packets
 		1072241 window update packets
 		23163 control packets
 		571945 data packets sent after flow control
 		61 challenge ACKs sent due to unexpected SYN
 		4 challenge ACKs sent due to unexpected RST
 		18314 checksummed in software
 			5196 segments (272795 bytes) over IPv4
 			13118 segments (1494655 bytes) over IPv6
 	45437408 packets received
 		20885349 acks (for 3523699640 byte)
 		2369692 duplicate acks
 		0 ack for unsent data
 		26469614 packets (3766989895 byte) received in-sequence
 		4634 completely duplicate packets (845184 bytes)
 		55 old duplicate packets
 		0 received packet dropped due to low memory
 		1 packet with some dup. data (9 bytes duped)
 		15064 out-of-order packets (21167582 bytes)
 		0 packet (0 byte) of data after window
 		0 window probe
 		300216 window update packets
 		808 packets recovered after loss
 		1584 packets received after close
 		4 bad resets
 		0 discarded for bad checksum
 		17245 checksummed in software
 			4655 segments (319983 bytes) over IPv4
 			12590 segments (1379223 bytes) over IPv6
 		0 discarded for bad header offset field
 		0 discarded because packet too short
 	13206 connection requests
 	83 connection accepts
 	15 bad connection attempts
 	0 listen queue overflow
 	12426 connections established (including accepts)
 	13687 connections closed (including 3494 drops)
 		2003 connections updated cached RTT on close
 		2003 connections updated cached RTT variance on close
 		283 connections updated cached ssthresh on close
 		2335 connections initialized RTT from route cache
 		2335 connections initialized RTT variance from route cache
 		291 connections initialized ssthresh from route cache
 	28 embryonic connections dropped
 	38165161 segments updated rtt (of 8172122 attempts)
 	60615 retransmit timeouts
 		123 connections dropped by rexmit timeout
 		0 connection dropped after retransmitting FIN
 		18 unnecessary packet retransmissionss
 	102 persist timeouts
 		0 connection dropped by persist timeout
 	18871 keepalive timeouts
 		18764 keepalive probes sent
 		20 connections dropped by keepalive
 	11710123 correct ACK header predictions
 	21748077 correct data packet header predictions
 	159802 SACK recovery episodes
 	984157 segment rexmits in SACK recovery episodes
 	1244806263 byte rexmits in SACK recovery episodes
 	3356069 SACK options (SACK blocks) received
 	16314 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 LRO coalesced packet
 		0 time LRO flow table was full
 		0 collision in LRO flow table
 		0 time LRO coalesced 2 packets
 		0 time LRO coalesced 3 or 4 packets
 		0 time LRO coalesced 5 or more packets
 	321561 limited transmits done
 	1550 early retransmits done
 	9623 times cumulative ack advanced along with SACK
 	1069 probe timeouts
 		83 times retransmit timeout triggered after probe
 		0 time probe packets were sent for an interface
 		0 time couldn't send probe packets for an interface
 		13 times fast recovery after tail loss
 		186 times recovered last packet
 		96 SACK based rescue retransmits
 	8180 client connections attempted to negotiate ECN
 		3961 client connections successfully negotiated ECN
 		3489 times graceful fallback to Non-ECN connection
 		417 times lost ECN negotiating SYN, followed by retransmission
 		30 server connections attempted to negotiate ECN
 		30 server connections successfully negotiated ECN
 		0 time lost ECN negotiating SYN-ACK, followed by retransmission
 		23 times received congestion experienced (CE) notification
 		16 times CWR was sent in response to ECE
 		4998 times sent ECE notification
 		14 connections received CE atleast once
 		8 connections received ECE atleast once
 		308 connections using ECN have seen packet loss but no CE
 		2 connections using ECN have seen packet loss and CE
 		20 connections using ECN received CE but no packet loss
 		2 connections fell back to non-ECN due to SYN-loss
 		472 connections fell back to non-ECN due to reordering
 		0 connection fell back to non-ECN due to excessive CE-markings
 		0 connection fell back caused by connection drop due to RST
 		4 connections fell back due to drop after multiple retransmits
 		0 connection fell back due to RST after SYN
 	846 times packet reordering was detected on a connection
 		183836 times transmitted packets were reordered
 		145032 times fast recovery was delayed to handle reordering
 		183836 times retransmission was avoided by delaying recovery
 		0 retransmission not needed
 	22347 retransmissions due to tail loss
 	1146 times DSACK option was sent
 		8877 times DSACK option was received
 		97 times DSACK was disabled on a connection
 		761 times recovered from bad retransmission using DSACK
 		125 times ignored DSACK due to ack loss
 		94 times ignored old DSACK options
 	7 times PMTU Blackhole detection, size reverted
 	5 connections were dropped after long sleep
 	1 connection had stretch ack algorithm disabled
 	0 time a TFO-cookie has been announced
 	0 SYN with data and a valid TFO-cookie have been received
 	0 SYN with TFO-cookie-request received
 	0 time an invalid TFO-cookie has been received
 	0 time we requested a TFO-cookie
 		0 time the peer announced a TFO-cookie
 	0 time we combined SYN with data and a TFO-cookie
 		0 time our SYN with data has been acknowledged
 	0 time a connection-attempt with TFO fell back to regular TCP
 	0 time a TFO-connection blackhole'd
 	0 time a TFO-cookie we sent was wrong
 	0 time did not received a TFO-cookie we asked for
 	0 time TFO got disabled due to heuristicsn
 	0 time TFO got blackholed in the sending direction
 	0 time maximum segment size was changed to default
 	0 time maximum segment size was changed to medium
 	0 time maximum segment size was changed to low
 	842010 timer drifts less or equal to 1 ms
 	10837 timer drifts less or equal to 10 ms
 	2 timer drifts less or equal to 20 ms
 	5 timer drifts less or equal to 50 ms
 	3 timer drifts less or equal to 100 ms
 	6 timer drifts less or equal to 200 ms
 	0 timer drift less or equal to 500 ms
 	0 timer drift less or equal to 1000 ms
 	0 timer drift greater than to 1000 ms

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Ecn-sane] osx ecn

[Ryan Mounce]

On Thu, 29 Aug 2019 at 09:15, Dave Taht <dave.taht@gmail.com> wrote:
>
> Very cool, thank you for jailbreaking.
>
> I'm a little puzzled over the cwr/ece ratio?
>
> 25 times received congestion experienced (CE) notification
> 37 times CWR was sent in response to ECE
> 619 times sent ECE notification
> 1029 connections received CE atleast once
>
> And um, er, while you are here busting your warranty for the sake of
> science, I'm dying to know what
>
>  netstat -I each_interface_you_have_especially_lte -qq
>
> shows. They have a really short queue and some form of flow control
> and I've never seen anything trigger except though a vm

https://gist.github.com/rmounce/dbb05a0bad40c9e37776d15b7d53bdc9

en0 is WiFi
pdp_ip0 is LTE data
pdp_ip1 is VoLTE

This is an iPhone X w/ Qualcomm modem. FWIW I'm on Telstra, which was
(is?) one of the few networks whitelisted by Apple for their cellular
ECN experiment a few years ago.