[Cake] Sce In cake testers wanted

[Cake] Sce In cake testers wanted

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

On Sat, Mar 23, 2019, 8:09 PM Jonathan Morton <chromatix99@gmail.com> wrote:

> > On 23 Mar, 2019, at 4:33 pm, Pete Heist <pete@heistp.net> wrote:
> >
> >> Oddly, I think this is the fastest train I've yet ridden on this trip.
> The speedometer on the internal monitors claims 200kph; the Swedish
> overnight train hit 160kph or so while making up lost time.  They also
> abbreviate the final destination as "Praha hl.n.”
> >
> > That’s clearly hlavní nádraží, from the land of one incredibly
> complicated language to another (Finnish).
>
> Hey, at least Finnish is pronounceable!  I'll grant you the grammar.
>
> Things predictably slowed down a lot once we got past Berlin, but we still
> seem to be on time as far as I can tell.  The railway network must not have
> been so carefully developed in the old DDR, and is evidently congested with
> freight traffic - but that is accounted for in the timetables.
>
> I've just managed to push an initial implementation of SCE marking, using
> a ramp function instead of a step, in Cake. This includes the necessary
> tc-adv patch.  Stats are collected but not yet reported.  Untested…
>
> It defaults to off.  Add "sce" to turn it on.  There are no knobs.  Use
> "no-sce" to turn it off explicitly.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 1725 bytes --]

Re: [Cake] Sce In cake testers wanted

[Kevin Darbyshire-Bryant]

Sent a github PR to make it build - not run it yet.

> On 23 Mar 2019, at 19:15, Dave Taht <dave.taht@gmail.com> wrote:
> 
> 
> On Sat, Mar 23, 2019, 8:09 PM Jonathan Morton <chromatix99@gmail.com> wrote:
> > On 23 Mar, 2019, at 4:33 pm, Pete Heist <pete@heistp.net> wrote:
> > 
> >> Oddly, I think this is the fastest train I've yet ridden on this trip.  The speedometer on the internal monitors claims 200kph; the Swedish overnight train hit 160kph or so while making up lost time.  They also abbreviate the final destination as "Praha hl.n.”
> > 
> > That’s clearly hlavní nádraží, from the land of one incredibly complicated language to another (Finnish).
> 
> Hey, at least Finnish is pronounceable!  I'll grant you the grammar.
> 
> Things predictably slowed down a lot once we got past Berlin, but we still seem to be on time as far as I can tell.  The railway network must not have been so carefully developed in the old DDR, and is evidently congested with freight traffic - but that is accounted for in the timetables.
> 
> I've just managed to push an initial implementation of SCE marking, using a ramp function instead of a step, in Cake. This includes the necessary tc-adv patch.  Stats are collected but not yet reported.  Untested…
> 
> It defaults to off.  Add "sce" to turn it on.  There are no knobs.  Use "no-sce" to turn it off explicitly.
> 
>  - Jonathan Morton
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A

Re: [Cake] Sce In cake testers wanted

[Jonathan Morton]

> On 23 Mar, 2019, at 10:28 pm, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> Sent a github PR to make it build - not run it yet.

LGTM, so I approved it.  I've reached the "bufferboat" and started to settle in, met Rod and Pete…

I brought a Pi Zero W with me as a basic Linux test environment, but I need to find a mini to macro HDMI adapter to make it work with this TV.  Pete says he'll look for one to bring over tomorrow.

So now we probably have *two* working middlebox implementations, using different approaches, but we need a visualisation tool (hi Pete) and some sort of working TCP that uses it (whether using the sqrt rule or the 1/n rule, doesn't really matter - but preferably examples of both).  For the latter, we can temporarily get away with inferring stuff from standard tcptrace.  If there's a working implementation of AccECN for the feedback path, we might as well use that for demo purposes.

I want to be able to demonstrate SCE actually working as designed, in a single-queue implementation, to answer the main argument that some of the L4S folks have latched onto.  But I don't know how soon we'll be able to manage that.

 - Jonathan Morton

Re: [Cake] Sce In cake testers wanted

[Dave Taht]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

There are two openwrt routers on the boat.

On Sun, Mar 24, 2019, 12:53 AM Jonathan Morton <chromatix99@gmail.com>
wrote:

> > On 23 Mar, 2019, at 10:28 pm, Kevin Darbyshire-Bryant <
> kevin@darbyshire-bryant.me.uk> wrote:
> >
> > Sent a github PR to make it build - not run it yet.
>
> LGTM, so I approved it.  I've reached the "bufferboat" and started to
> settle in, met Rod and Pete…
>
> I brought a Pi Zero W with me as a basic Linux test environment, but I
> need to find a mini to macro HDMI adapter to make it work with this TV.
> Pete says he'll look for one to bring over tomorrow.
>
> So now we probably have *two* working middlebox implementations, using
> different approaches, but we need a visualisation tool (hi Pete) and some
> sort of working TCP that uses it (whether using the sqrt rule or the 1/n
> rule, doesn't really matter - but preferably examples of both).  For the
> latter, we can temporarily get away with inferring stuff from standard
> tcptrace.  If there's a working implementation of AccECN for the feedback
> path, we might as well use that for demo purposes.
>
> I want to be able to demonstrate SCE actually working as designed, in a
> single-queue implementation, to answer the main argument that some of the
> L4S folks have latched onto.  But I don't know how soon we'll be able to
> manage that.
>
>  - Jonathan Morton
>
>

[-- Attachment #2: Type: text/html, Size: 1789 bytes --]

Re: [Cake] Sce In cake testers wanted

[Jonathan Morton]

> On 24 Mar, 2019, at 12:54 am, Dave Taht <dave.taht@gmail.com> wrote:
> 
> There are two openwrt routers on the boat.

I mean as a complete system, with a TCP that responds to the signal.

As well as the compatibility cases where a TCP doesn't respond, or one that does respond doesn't get a signal, or a mixture of aware and ignorant flows share the queue.

Can't do any of that without a working TCP.

 - Jonathan Morton

Re: [Cake] Sce In cake testers wanted

[Pete Heist]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…

https://www.alza.cz/akasa-hdmi-mini-hdmi-d2286697.htm?o=5 <https://www.alza.cz/akasa-hdmi-mini-hdmi-d2286697.htm?o=5>

> On Mar 24, 2019, at 12:58 AM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 24 Mar, 2019, at 12:54 am, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> There are two openwrt routers on the boat.
> 
> I mean as a complete system, with a TCP that responds to the signal.
> 
> As well as the compatibility cases where a TCP doesn't respond, or one that does respond doesn't get a signal, or a mixture of aware and ignorant flows share the queue.
> 
> Can't do any of that without a working TCP.
> 
> - Jonathan Morton
> 


[-- Attachment #2: Type: text/html, Size: 1558 bytes --]

Re: [Cake] Sce In cake testers wanted

[Jonathan Morton]

> On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
> 
> I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…

Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.

We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.

 - Jonathan Morton

[Cake] The two SCE tests I have in mind

[Dave Taht]

1) The research into whether bit flipping to the extent that SCE will
do has not been done yet. The study of ECT(0) vs ECT(1) behavior
transiting to CE was a little lightweight.

To test this we going to fire up a ton of nanodes in various data
centers, with low SCE thresholds, and low bandwidths, to flip lots of
bits, and test between the data centers and from as many vantage
points around the net as we can get - do packet captures as well as
flent tests

as a control, set up identical boxes, with SCE disabled, in the same
data centers.

Setup flent, irtt, iperf3.

2) Diffserv bit preservation test

The research going by on the tsvwg mailing seems a bit dated. It is
very straightforward to use irtt to test to see what udp codepoints
survive e2e, and to also leverage this testbed setup. Similarly,
netperf can easily be used to mark tcp. We do not have a good packet
cap tool to verify that the bits are being set right, however I think
irtt can be modified to check for correctness here and produce a
report.

On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>
> > On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
> >
> > I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…
>
> Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.
>
> We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.
>
>  - Jonathan Morton
>
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] The two SCE tests I have in mind

[Luca Muscariello]

[-- Attachment #1: Type: text/plain, Size: 3053 bytes --]

We have done something like that a year ago in our team in Cisco
for a project that you Dave are aware of but that is out of scope for these
lists.

It is important to have vantage points in residential and enterprise
networks.
Cloud to Cloud never goes to transit and the big Clouds have global
footprints.
So you can traverse the planet inside the Cloud WAN (100% bit consistency)
and then traverse a safe peering
point where the adjacency  has been well set (or not).

IPv4/IPv6 of course we'll see a big difference in favour of IPv6.
What Michael has shown is his paper makes a lot of sense to me.
Depending on the use case hit ratio can be very high or very low.




On Sun, Mar 24, 2019 at 9:55 AM Dave Taht <dave.taht@gmail.com> wrote:

> 1) The research into whether bit flipping to the extent that SCE will
> do has not been done yet. The study of ECT(0) vs ECT(1) behavior
> transiting to CE was a little lightweight.
>
> To test this we going to fire up a ton of nanodes in various data
> centers, with low SCE thresholds, and low bandwidths, to flip lots of
> bits, and test between the data centers and from as many vantage
> points around the net as we can get - do packet captures as well as
> flent tests
>
> as a control, set up identical boxes, with SCE disabled, in the same
> data centers.
>
> Setup flent, irtt, iperf3.
>
> 2) Diffserv bit preservation test
>
> The research going by on the tsvwg mailing seems a bit dated. It is
> very straightforward to use irtt to test to see what udp codepoints
> survive e2e, and to also leverage this testbed setup. Similarly,
> netperf can easily be used to mark tcp. We do not have a good packet
> cap tool to verify that the bits are being set right, however I think
> irtt can be modified to check for correctness here and produce a
> report.
>
> On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com>
> wrote:
> >
> > > On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
> > >
> > > I should theoretically arrive at the boat today some time after 3pm,
> having picked up a mini HDMI to HDMI adapter, which we can use with the
> cable that’s there…
> >
> > Awesome.  I'm also setting up a Linux VM on my Mac, which should help
> things along.
> >
> > We're bringing up some actual hardware with the SCE-enabled Cake on it
> now.  Dave wants to investigate various theoretical phenomena the Internet
> might exhibit with a mixture of ECN codepoints; I just want to be sure it
> actually works as intended, before I move on to fiddling with TCP.
> >
> >  - Jonathan Morton
> >
> > _______________________________________________
> > Cake mailing list
> > Cake@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>

[-- Attachment #2: Type: text/html, Size: 4137 bytes --]

Re: [Cake] The two SCE tests I have in mind

[Dave Taht]

On Sun, Mar 24, 2019 at 10:30 AM Luca Muscariello
<luca.muscariello@gmail.com> wrote:
>
> We have done something like that a year ago in our team in Cisco
> for a project that you Dave are aware of but that is out of scope for these lists.
>
> It is important to have vantage points in residential and enterprise networks.

It is hard to get dedicated testing points in those networks. However
providing tools so more volunteers can participate in a larger test
has always proven worthwhile.

> Cloud to Cloud never goes to transit and the big Clouds have global footprints.
> So you can traverse the planet inside the Cloud WAN (100% bit consistency) and then traverse a safe peering
> point where the adjacency  has been well set (or not).

cloud to cloud should be interesting.

Anyway, the first sce enabled version of cake for public testing is
now up at cake-sce-de.teklibre.net. (dns still propigating) It is rate
limited to 40Mbits and thus with 10 flows going, self congests and
thus marks with both SCE and CE. Inititial tests across our cloudy
backbone have been good, no reordering, no drops, while SCE marking
like crazy. The needed code to setup your own server is in my github
sch_cake and tc-adv repos.

We hope to roll that out over linode, google cloud, and aws over the
coming days.

A typical test right now, looks like this:

# server side install if you want to set up your own test server
# after an:

apt-get install flent netperf irtt build-essential flex bison
git clone https://github.com/dtaht/sch_cake
git clone https://github.com/dtaht/tc-adv
git clone https://github.com/dtaht/fq_codel_fast
git clone https://github.com/dtaht/fq_codel_fast
git clone https://github.com/tohojo/sqm-scripts

for i in sch_cake tc-adv fq_codel_fast sqm-scripts
do
cd $i; make; make install; cd ..
done

tc qdisc replace dev eth0 root cake sce bandwidth 40mbit

# Client side test

tcpdump -i eth0 -s 128 -w mylocationinfo.cap
flent -H  cake-sce-de.teklibre.net -l 20 -t my_location_info
--te=download_streams tcp_ndown
killall tcpdump

Go looking at the cap for sack, drops, reorders, and the sce bit being excerted.

(flent's --socket-stats option got broken in a recent tc release which
we need to fix for that and qdisc stats)

if you are running your own server, you can now also see the
sce marking rate via:

tc -s qdisc show dev eth0

qdisc cake 8002: root refcnt 2 bandwidth 40Mbit diffserv3
triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw
overhead 0 sce

 Sent 37542361 bytes 25522 pkt (dropped 0, overlimits 30923 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 124568b of 4Mb
 capacity estimate: 40Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14


                   Bulk  Best Effort        Voice

  thresh       2500Kbit       40Mbit       10Mbit
  target          7.3ms        5.0ms        5.0ms
  interval      102.3ms      100.0ms      100.0ms
  pk_delay          0us        528us        347us
  av_delay          0us        201us         10us
  sp_delay          0us          4us          5us
  backlog            0b           0b           0b
  pkts                0        25209          313
  bytes               0     37500795        41566
  way_inds            0            0            0
  way_miss            0          106            5
  way_cols            0            0            0
  drops               0            0            0
  marks               0           24            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            0            1
  un_flows            0            0            0
  max_len             0         3028         1752
  quantum           300         1220          305
  sce                 0        23105            0


setting up the sqm-scripts to use fq_codel_fast is a matter of setting
the right parameters in a /etc/sqm/eth0.iface.conf file for bandwidth
(40mbit) and qos model (simplest.qos) and

IQDISC_OPTS="ce_threshold 2us" # these are artificially low - 1-2ms
should be closer to "right" on cloudy devices for a real, rather than
test, deployment
EQDISC_OPTS="ce_threshold 2us"


> IPv4/IPv6 of course we'll see a big difference in favour of IPv6.
> What Michael has shown is his paper makes a lot of sense to me.
> Depending on the use case hit ratio can be very high or very low.

I am very behind on reading papers this week.

>
>
>
>
> On Sun, Mar 24, 2019 at 9:55 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> 1) The research into whether bit flipping to the extent that SCE will
>> do has not been done yet. The study of ECT(0) vs ECT(1) behavior
>> transiting to CE was a little lightweight.
>>
>> To test this we going to fire up a ton of nanodes in various data
>> centers, with low SCE thresholds, and low bandwidths, to flip lots of
>> bits, and test between the data centers and from as many vantage
>> points around the net as we can get - do packet captures as well as
>> flent tests
>>
>> as a control, set up identical boxes, with SCE disabled, in the same
>> data centers.
>>
>> Setup flent, irtt, iperf3.
>>
>> 2) Diffserv bit preservation test
>>
>> The research going by on the tsvwg mailing seems a bit dated. It is
>> very straightforward to use irtt to test to see what udp codepoints
>> survive e2e, and to also leverage this testbed setup. Similarly,
>> netperf can easily be used to mark tcp. We do not have a good packet
>> cap tool to verify that the bits are being set right, however I think
>> irtt can be modified to check for correctness here and produce a
>> report.
>>
>> On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>> >
>> > > On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
>> > >
>> > > I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…
>> >
>> > Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.
>> >
>> > We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.
>> >
>> >  - Jonathan Morton
>> >
>> > _______________________________________________
>> > Cake mailing list
>> > Cake@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/cake
>>
>>
>>
>> --
>>
>> Dave Täht
>> CTO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-831-205-9740
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] The two SCE tests I have in mind

[Dave Taht]

I made two goofs in this post.

--te=download_streams=10

and the server and client need to be negiotiating ecn:

https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/

So far as I know the OSX section is out of date, osx always negotiates
ecn now. I do not have data on ios's current configuration of ecn
enablement.

On Sun, Mar 24, 2019 at 3:30 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Sun, Mar 24, 2019 at 10:30 AM Luca Muscariello
> <luca.muscariello@gmail.com> wrote:
> >
> > We have done something like that a year ago in our team in Cisco
> > for a project that you Dave are aware of but that is out of scope for these lists.
> >
> > It is important to have vantage points in residential and enterprise networks.
>
> It is hard to get dedicated testing points in those networks. However
> providing tools so more volunteers can participate in a larger test
> has always proven worthwhile.
>
> > Cloud to Cloud never goes to transit and the big Clouds have global footprints.
> > So you can traverse the planet inside the Cloud WAN (100% bit consistency) and then traverse a safe peering
> > point where the adjacency  has been well set (or not).
>
> cloud to cloud should be interesting.
>
> Anyway, the first sce enabled version of cake for public testing is
> now up at cake-sce-de.teklibre.net. (dns still propigating) It is rate
> limited to 40Mbits and thus with 10 flows going, self congests and
> thus marks with both SCE and CE. Inititial tests across our cloudy
> backbone have been good, no reordering, no drops, while SCE marking
> like crazy. The needed code to setup your own server is in my github
> sch_cake and tc-adv repos.
>
> We hope to roll that out over linode, google cloud, and aws over the
> coming days.
>
> A typical test right now, looks like this:
>
> # server side install if you want to set up your own test server
> # after an:
>
> apt-get install flent netperf irtt build-essential flex bison
> git clone https://github.com/dtaht/sch_cake
> git clone https://github.com/dtaht/tc-adv
> git clone https://github.com/dtaht/fq_codel_fast
> git clone https://github.com/dtaht/fq_codel_fast
> git clone https://github.com/tohojo/sqm-scripts
>
> for i in sch_cake tc-adv fq_codel_fast sqm-scripts
> do
> cd $i; make; make install; cd ..
> done
>
> tc qdisc replace dev eth0 root cake sce bandwidth 40mbit
>
> # Client side test
>
> tcpdump -i eth0 -s 128 -w mylocationinfo.cap
> flent -H  cake-sce-de.teklibre.net -l 20 -t my_location_info
> --te=download_streams tcp_ndown
> killall tcpdump
>
> Go looking at the cap for sack, drops, reorders, and the sce bit being excerted.
>
> (flent's --socket-stats option got broken in a recent tc release which
> we need to fix for that and qdisc stats)
>
> if you are running your own server, you can now also see the
> sce marking rate via:
>
> tc -s qdisc show dev eth0
>
> qdisc cake 8002: root refcnt 2 bandwidth 40Mbit diffserv3
> triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw
> overhead 0 sce
>
>  Sent 37542361 bytes 25522 pkt (dropped 0, overlimits 30923 requeues 0)
>  backlog 0b 0p requeues 0
>  memory used: 124568b of 4Mb
>  capacity estimate: 40Mbit
>  min/max network layer size:           42 /    1514
>  min/max overhead-adjusted size:       42 /    1514
>  average network hdr offset:           14
>
>
>                    Bulk  Best Effort        Voice
>
>   thresh       2500Kbit       40Mbit       10Mbit
>   target          7.3ms        5.0ms        5.0ms
>   interval      102.3ms      100.0ms      100.0ms
>   pk_delay          0us        528us        347us
>   av_delay          0us        201us         10us
>   sp_delay          0us          4us          5us
>   backlog            0b           0b           0b
>   pkts                0        25209          313
>   bytes               0     37500795        41566
>   way_inds            0            0            0
>   way_miss            0          106            5
>   way_cols            0            0            0
>   drops               0            0            0
>   marks               0           24            0
>   ack_drop            0            0            0
>   sp_flows            0            1            0
>   bk_flows            0            0            1
>   un_flows            0            0            0
>   max_len             0         3028         1752
>   quantum           300         1220          305
>   sce                 0        23105            0
>
>
> setting up the sqm-scripts to use fq_codel_fast is a matter of setting
> the right parameters in a /etc/sqm/eth0.iface.conf file for bandwidth
> (40mbit) and qos model (simplest.qos) and
>
> IQDISC_OPTS="ce_threshold 2us" # these are artificially low - 1-2ms
> should be closer to "right" on cloudy devices for a real, rather than
> test, deployment
> EQDISC_OPTS="ce_threshold 2us"
>
>
> > IPv4/IPv6 of course we'll see a big difference in favour of IPv6.
> > What Michael has shown is his paper makes a lot of sense to me.
> > Depending on the use case hit ratio can be very high or very low.
>
> I am very behind on reading papers this week.
>
> >
> >
> >
> >
> > On Sun, Mar 24, 2019 at 9:55 AM Dave Taht <dave.taht@gmail.com> wrote:
> >>
> >> 1) The research into whether bit flipping to the extent that SCE will
> >> do has not been done yet. The study of ECT(0) vs ECT(1) behavior
> >> transiting to CE was a little lightweight.
> >>
> >> To test this we going to fire up a ton of nanodes in various data
> >> centers, with low SCE thresholds, and low bandwidths, to flip lots of
> >> bits, and test between the data centers and from as many vantage
> >> points around the net as we can get - do packet captures as well as
> >> flent tests
> >>
> >> as a control, set up identical boxes, with SCE disabled, in the same
> >> data centers.
> >>
> >> Setup flent, irtt, iperf3.
> >>
> >> 2) Diffserv bit preservation test
> >>
> >> The research going by on the tsvwg mailing seems a bit dated. It is
> >> very straightforward to use irtt to test to see what udp codepoints
> >> survive e2e, and to also leverage this testbed setup. Similarly,
> >> netperf can easily be used to mark tcp. We do not have a good packet
> >> cap tool to verify that the bits are being set right, however I think
> >> irtt can be modified to check for correctness here and produce a
> >> report.
> >>
> >> On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com> wrote:
> >> >
> >> > > On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
> >> > >
> >> > > I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…
> >> >
> >> > Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.
> >> >
> >> > We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.
> >> >
> >> >  - Jonathan Morton
> >> >
> >> > _______________________________________________
> >> > Cake mailing list
> >> > Cake@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/cake
> >>
> >>
> >>
> >> --
> >>
> >> Dave Täht
> >> CTO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-831-205-9740
> >> _______________________________________________
> >> Cake mailing list
> >> Cake@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/cake
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] The two SCE tests I have in mind

[Sebastian Moeller]

Hi All,

while I am late to the party, I could run a client side test tonight. Nothing special, just a residential O2/Telefonica (via L2-bitstream access) VDSL2 50/10 link in Germany. Would a capture of that be of any help? If yes I am happy to test, otherwise I will wait for the dust to settle.

BTW, later this week it would be great to create sharable client and server VMs, just as the LLLLS folks should start distributing, but that certainly is for after the meeting.

P.S.: how likely is that meeting to be on time (I need to figure out how to schedule work around this)?

Best Regards, or better Ahoy to the crew of the buffer boat

	Sebastian



> On Mar 24, 2019, at 11:30, Dave Taht <dave.taht@gmail.com> wrote:
> 
> On Sun, Mar 24, 2019 at 10:30 AM Luca Muscariello
> <luca.muscariello@gmail.com> wrote:
>> 
>> We have done something like that a year ago in our team in Cisco
>> for a project that you Dave are aware of but that is out of scope for these lists.
>> 
>> It is important to have vantage points in residential and enterprise networks.
> 
> It is hard to get dedicated testing points in those networks. However
> providing tools so more volunteers can participate in a larger test
> has always proven worthwhile.
> 
>> Cloud to Cloud never goes to transit and the big Clouds have global footprints.
>> So you can traverse the planet inside the Cloud WAN (100% bit consistency) and then traverse a safe peering
>> point where the adjacency  has been well set (or not).
> 
> cloud to cloud should be interesting.
> 
> Anyway, the first sce enabled version of cake for public testing is
> now up at cake-sce-de.teklibre.net. (dns still propigating) It is rate
> limited to 40Mbits and thus with 10 flows going, self congests and
> thus marks with both SCE and CE. Inititial tests across our cloudy
> backbone have been good, no reordering, no drops, while SCE marking
> like crazy. The needed code to setup your own server is in my github
> sch_cake and tc-adv repos.
> 
> We hope to roll that out over linode, google cloud, and aws over the
> coming days.
> 
> A typical test right now, looks like this:
> 
> # server side install if you want to set up your own test server
> # after an:
> 
> apt-get install flent netperf irtt build-essential flex bison
> git clone https://github.com/dtaht/sch_cake
> git clone https://github.com/dtaht/tc-adv
> git clone https://github.com/dtaht/fq_codel_fast
> git clone https://github.com/dtaht/fq_codel_fast
> git clone https://github.com/tohojo/sqm-scripts
> 
> for i in sch_cake tc-adv fq_codel_fast sqm-scripts
> do
> cd $i; make; make install; cd ..
> done
> 
> tc qdisc replace dev eth0 root cake sce bandwidth 40mbit
> 
> # Client side test
> 
> tcpdump -i eth0 -s 128 -w mylocationinfo.cap
> flent -H  cake-sce-de.teklibre.net -l 20 -t my_location_info
> --te=download_streams tcp_ndown
> killall tcpdump
> 
> Go looking at the cap for sack, drops, reorders, and the sce bit being excerted.
> 
> (flent's --socket-stats option got broken in a recent tc release which
> we need to fix for that and qdisc stats)
> 
> if you are running your own server, you can now also see the
> sce marking rate via:
> 
> tc -s qdisc show dev eth0
> 
> qdisc cake 8002: root refcnt 2 bandwidth 40Mbit diffserv3
> triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw
> overhead 0 sce
> 
> Sent 37542361 bytes 25522 pkt (dropped 0, overlimits 30923 requeues 0)
> backlog 0b 0p requeues 0
> memory used: 124568b of 4Mb
> capacity estimate: 40Mbit
> min/max network layer size:           42 /    1514
> min/max overhead-adjusted size:       42 /    1514
> average network hdr offset:           14
> 
> 
>                   Bulk  Best Effort        Voice
> 
>  thresh       2500Kbit       40Mbit       10Mbit
>  target          7.3ms        5.0ms        5.0ms
>  interval      102.3ms      100.0ms      100.0ms
>  pk_delay          0us        528us        347us
>  av_delay          0us        201us         10us
>  sp_delay          0us          4us          5us
>  backlog            0b           0b           0b
>  pkts                0        25209          313
>  bytes               0     37500795        41566
>  way_inds            0            0            0
>  way_miss            0          106            5
>  way_cols            0            0            0
>  drops               0            0            0
>  marks               0           24            0
>  ack_drop            0            0            0
>  sp_flows            0            1            0
>  bk_flows            0            0            1
>  un_flows            0            0            0
>  max_len             0         3028         1752
>  quantum           300         1220          305
>  sce                 0        23105            0
> 
> 
> setting up the sqm-scripts to use fq_codel_fast is a matter of setting
> the right parameters in a /etc/sqm/eth0.iface.conf file for bandwidth
> (40mbit) and qos model (simplest.qos) and
> 
> IQDISC_OPTS="ce_threshold 2us" # these are artificially low - 1-2ms
> should be closer to "right" on cloudy devices for a real, rather than
> test, deployment
> EQDISC_OPTS="ce_threshold 2us"
> 
> 
>> IPv4/IPv6 of course we'll see a big difference in favour of IPv6.
>> What Michael has shown is his paper makes a lot of sense to me.
>> Depending on the use case hit ratio can be very high or very low.
> 
> I am very behind on reading papers this week.
> 
>> 
>> 
>> 
>> 
>> On Sun, Mar 24, 2019 at 9:55 AM Dave Taht <dave.taht@gmail.com> wrote:
>>> 
>>> 1) The research into whether bit flipping to the extent that SCE will
>>> do has not been done yet. The study of ECT(0) vs ECT(1) behavior
>>> transiting to CE was a little lightweight.
>>> 
>>> To test this we going to fire up a ton of nanodes in various data
>>> centers, with low SCE thresholds, and low bandwidths, to flip lots of
>>> bits, and test between the data centers and from as many vantage
>>> points around the net as we can get - do packet captures as well as
>>> flent tests
>>> 
>>> as a control, set up identical boxes, with SCE disabled, in the same
>>> data centers.
>>> 
>>> Setup flent, irtt, iperf3.
>>> 
>>> 2) Diffserv bit preservation test
>>> 
>>> The research going by on the tsvwg mailing seems a bit dated. It is
>>> very straightforward to use irtt to test to see what udp codepoints
>>> survive e2e, and to also leverage this testbed setup. Similarly,
>>> netperf can easily be used to mark tcp. We do not have a good packet
>>> cap tool to verify that the bits are being set right, however I think
>>> irtt can be modified to check for correctness here and produce a
>>> report.
>>> 
>>> On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>>>> 
>>>>> On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
>>>>> 
>>>>> I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…
>>>> 
>>>> Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.
>>>> 
>>>> We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.
>>>> 
>>>> - Jonathan Morton
>>>> 
>>>> _______________________________________________
>>>> Cake mailing list
>>>> Cake@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/cake
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> _______________________________________________
>>> Cake mailing list
>>> Cake@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] The two SCE tests I have in mind

[Pete Heist]

> On Mar 24, 2019, at 9:55 AM, Dave Taht <dave.taht@gmail.com> wrote:
> 
> 1) The research into whether bit flipping to the extent that SCE will
> do has not been done yet. The study of ECT(0) vs ECT(1) behavior
> transiting to CE was a little lightweight.
> 
> To test this we going to fire up a ton of nanodes in various data
> centers, with low SCE thresholds, and low bandwidths, to flip lots of
> bits, and test between the data centers and from as many vantage
> points around the net as we can get - do packet captures as well as
> flent tests
> 
> as a control, set up identical boxes, with SCE disabled, in the same
> data centers.
> 
> Setup flent, irtt, iperf3.

Sounds good, I’ll offer my home setup with a couple APU2s as a “residential WISP” vantage point. I was unable to access it with OpenVPN from the boat but I’ll sort that out before I go.

> 2) Diffserv bit preservation test
> 
> The research going by on the tsvwg mailing seems a bit dated. It is
> very straightforward to use irtt to test to see what udp codepoints
> survive e2e, and to also leverage this testbed setup. Similarly,
> netperf can easily be used to mark tcp. We do not have a good packet
> cap tool to verify that the bits are being set right, however I think
> irtt can be modified to check for correctness here and produce a
> report.

The irtt server knows what codepoint was negotiated and could easily emit a log message if incoming packets don’t match, but unfortunately the codepoint of incoming packets is not available without using a raw conn (requires root). We’ll want that anyway for UDP lite and I’ll see what’s possible tonight, but for now we can also just use a simple tcpdump filter like this:

tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”

“greater 80” ignores the handshake packets and 0x1 is whatever TOS value we want to make sure the packets contain. We can use different filters for other traffic.

> On Sun, Mar 24, 2019 at 1:45 AM Jonathan Morton <chromatix99@gmail.com> wrote:
>> 
>>> On 24 Mar, 2019, at 8:37 am, Pete Heist <pete@heistp.net> wrote:
>>> 
>>> I should theoretically arrive at the boat today some time after 3pm, having picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s there…
>> 
>> Awesome.  I'm also setting up a Linux VM on my Mac, which should help things along.
>> 
>> We're bringing up some actual hardware with the SCE-enabled Cake on it now.  Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP.
>> 
>> - Jonathan Morton
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740

Re: [Cake] The two SCE tests I have in mind

[Jonathan Morton]

> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
> 
> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”
> 
> “greater 80” ignores the handshake packets and 0x1 is whatever TOS value we want to make sure the packets contain. We can use different filters for other traffic.

Bear in mind that the TOS byte contains ECN as well as DSCP fields, and the latter is left-justified.

 - Jonathan Morton

Re: [Cake] The two SCE tests I have in mind

[Pete Heist]

> On Mar 24, 2019, at 12:08 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
> 
>> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
>> 
>> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”
>> 
>> “greater 80” ignores the handshake packets and 0x1 is whatever TOS value we want to make sure the packets contain. We can use different filters for other traffic.
> 
> Bear in mind that the TOS byte contains ECN as well as DSCP fields, and the latter is left-justified.


Indeed, I was playing with irtt to see if it gladly writes the whole field including the ECN bits (it does). Better common examples we can test for DSCP could be 0x10 or 0xb8.

Re: [Cake] [Ecn-sane] The two SCE tests I have in mind

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]


Jonathan Morton <chromatix99@gmail.com> wrote:
    >> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
    >> 
    >> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”
    >> 
    >> “greater 80” ignores the handshake packets and 0x1 is whatever TOS
    >> value we want to make sure the packets contain. We can use different
    >> filters for other traffic.

    > Bear in mind that the TOS byte contains ECN as well as DSCP fields, and
    > the latter is left-justified.

libpcap should probably learn about DSCN bits to avoid people having to
think so much :-)

Send patches to me/github.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [ 
	

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [Cake] [Ecn-sane] The two SCE tests I have in mind

[Stephen Hemminger]

On Sun, 24 Mar 2019 13:32:03 +0100
Michael Richardson <mcr@sandelman.ca> wrote:

> Jonathan Morton <chromatix99@gmail.com> wrote:
>     >> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
>     >> 
>     >> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”
>     >> 
>     >> “greater 80” ignores the handshake packets and 0x1 is whatever TOS
>     >> value we want to make sure the packets contain. We can use different
>     >> filters for other traffic.  
> 
>     > Bear in mind that the TOS byte contains ECN as well as DSCP fields, and
>     > the latter is left-justified.  
> 
> libpcap should probably learn about DSCN bits to avoid people having to
> think so much :-)
> 
> Send patches to me/github.
> 

Libpcap is ancient history by now. It is like ifconfig, everyone still can't reprogram
their brain; but the tool is on life support.

All development is happening on wireshark/tshark.

Re: [Cake] [Ecn-sane] The two SCE tests I have in mind

[Dave Taht]

On Mon, Apr 1, 2019 at 6:03 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Sun, 24 Mar 2019 13:32:03 +0100
> Michael Richardson <mcr@sandelman.ca> wrote:
>
> > Jonathan Morton <chromatix99@gmail.com> wrote:
> >     >> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
> >     >>
> >     >> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1”
> >     >>
> >     >> “greater 80” ignores the handshake packets and 0x1 is whatever TOS
> >     >> value we want to make sure the packets contain. We can use different
> >     >> filters for other traffic.
> >
> >     > Bear in mind that the TOS byte contains ECN as well as DSCP fields, and
> >     > the latter is left-justified.
> >
> > libpcap should probably learn about DSCN bits to avoid people having to
> > think so much :-)

Well, I still have nothing better than tcptrace + xplot.

> >
> > Send patches to me/github.


> >
>
> Libpcap is ancient history by now. It is like ifconfig, everyone still can't reprogram
> their brain; but the tool is on life support.
>
> All development is happening on wireshark/tshark.
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

Re: [Cake] [Ecn-sane] The two SCE tests I have in mind

[Rodney W. Grimes]

> On Sun, 24 Mar 2019 13:32:03 +0100
> Michael Richardson <mcr@sandelman.ca> wrote:
> 
> > Jonathan Morton <chromatix99@gmail.com> wrote:
> >     >> On 24 Mar, 2019, at 12:05 pm, Pete Heist <pete@heistp.net> wrote:
> >     >> 
> >     >> tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1?
> >     >> 
> >     >> ?greater 80? ignores the handshake packets and 0x1 is whatever TOS
> >     >> value we want to make sure the packets contain. We can use different
> >     >> filters for other traffic.  
> > 
> >     > Bear in mind that the TOS byte contains ECN as well as DSCP fields, and
> >     > the latter is left-justified.  
> > 
> > libpcap should probably learn about DSCN bits to avoid people having to

We need to teach tcpdump and wireshark what the new meaning of the 4th state
ECT bits mean, and that NS now means ESCE.  It already knows what CE and ECE
are, iirc.

> > think so much :-)
> > 
> > Send patches to me/github.
> > 
> 
> Libpcap is ancient history by now. It is like ifconfig, everyone still can't reprogram
> their brain; but the tool is on life support.

That is not correct, wireshark is a GUI built on top of libpcap,
and quiet useless without a working libpcap.

	https://wiki.wireshark.org/libpcap


However, the place(s) that need to learn about the bits so it
can display them correctly is in the wireshark code, and also
in the command line tcpdump code.

> All development is happening on wireshark/tshark.

Rarely, but not unheard of, changes do have to be made to libpcap,
this however is not one of those cases.
Libpcap is a very stable entity today.

-- 
Rod Grimes                                                 rgrimes@freebsd.org

Re: [Cake] Sce In cake testers wanted

[Kevin Darbyshire-Bryant]

and sent a github PR to get tc-adv to report sce enabled/disabled status correctly.



> On 23 Mar 2019, at 21:28, Kevin Darbyshire-Bryant <kevin@darbyshire-bryant.me.uk> wrote:
> 
> Sent a github PR to make it build - not run it yet.
> 
>> On 23 Mar 2019, at 19:15, Dave Taht <dave.taht@gmail.com> wrote:
>> 
>> 
>> On Sat, Mar 23, 2019, 8:09 PM Jonathan Morton <chromatix99@gmail.com> wrote:
>>> On 23 Mar, 2019, at 4:33 pm, Pete Heist <pete@heistp.net> wrote:
>>> 
>>>> Oddly, I think this is the fastest train I've yet ridden on this trip.  The speedometer on the internal monitors claims 200kph; the Swedish overnight train hit 160kph or so while making up lost time.  They also abbreviate the final destination as "Praha hl.n.”
>>> 
>>> That’s clearly hlavní nádraží, from the land of one incredibly complicated language to another (Finnish).
>> 
>> Hey, at least Finnish is pronounceable!  I'll grant you the grammar.
>> 
>> Things predictably slowed down a lot once we got past Berlin, but we still seem to be on time as far as I can tell.  The railway network must not have been so carefully developed in the old DDR, and is evidently congested with freight traffic - but that is accounted for in the timetables.
>> 
>> I've just managed to push an initial implementation of SCE marking, using a ramp function instead of a step, in Cake. This includes the necessary tc-adv patch.  Stats are collected but not yet reported.  Untested…
>> 
>> It defaults to off.  Add "sce" to turn it on.  There are no knobs.  Use "no-sce" to turn it off explicitly.
>> 
>> - Jonathan Morton
>> 
>> _______________________________________________
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> Cheers,
> 
> Kevin D-B
> 
> gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A
> 
> _______________________________________________
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake


Cheers,

Kevin D-B

gpg: 012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A