From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp70.iad3a.emailsrvr.com (smtp70.iad3a.emailsrvr.com [173.203.187.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 8AD0C3BA8E for ; Wed, 30 May 2018 18:57:53 -0400 (EDT) Received: from smtp33.relay.iad3a.emailsrvr.com (localhost [127.0.0.1]) by smtp33.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 5F718564D; Wed, 30 May 2018 18:57:53 -0400 (EDT) X-SMTPDoctor-Processed: csmtpprox beta Received: from smtp33.relay.iad3a.emailsrvr.com (localhost [127.0.0.1]) by smtp33.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 5C0855655; Wed, 30 May 2018 18:57:53 -0400 (EDT) Received: from app42.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp33.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 3D53B564D; Wed, 30 May 2018 18:57:53 -0400 (EDT) X-Sender-Id: dpreed@deepplum.com Received: from app42.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.7.12); Wed, 30 May 2018 18:57:53 -0400 Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app42.wa-webapps.iad3a (Postfix) with ESMTP id 2A4A96003B; Wed, 30 May 2018 18:57:53 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Wed, 30 May 2018 18:57:53 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Wed, 30 May 2018 18:57:53 -0400 (EDT) From: "dpreed@deepplum.com" To: "Dave Taht" Cc: make-wifi-fast@lists.bufferbloat.net MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20180530185753000000_17761" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: Message-ID: <1527721073.171416827@apps.rackspace.com> X-Mailer: webmail/15.0.1-RC Subject: Re: [Make-wifi-fast] =?utf-8?q?emulating_wifi_better_-_coupling_qdisc?= =?utf-8?q?s_in_netem=3F?= X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2018 22:57:53 -0000 ------=_20180530185753000000_17761 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AI would toss netem rather than kludging around what appears to be a fund= amental design choice made in ins conceptualization. Make a "netem2".=0A = =0AFreeBSD has a very nice framework for emulating far more general packet = queuing/routing/... in the kernel, called NetGraph. It's incredibly general= , and could straightforwardly, with high performance, have modules that do = exactly the right emulations of network structures with such blocking, etc.= and even random delays.=0A =0AI know this because in my day job at TidalSc= ale, we heavily use NetGraph to implement new very low level protocols, whi= ch is pretty straightforward, even including complex multi-adapter adaptive= forwarding of our private protocols on 10 and 40 GigE links. Super flexibl= e, entirely in the kernel, running either at real-time priority or not, in = a mix.=0A =0AIn contrast, the Linux TC framework seems very inflexible, as = you've found, in trying to push it to do what it is not designed to do.=0A = =0ASo tossing netem might be far better. I wonder if NetGraph has ever been= ported into some Linux kernel environment...=0A-----Original Message-----= =0AFrom: "Dave Taht" =0ASent: Wednesday, May 30, 2018 = 1:28pm=0ATo: make-wifi-fast@lists.bufferbloat.net=0ASubject: [Make-wifi-fas= t] emulating wifi better - coupling qdiscs in netem?=0A=0A=0A=0AThe match t= o reality of my "wifi slotting" code for netem was so=0Adisappointing that = I was extremely reluctant to push support for it up=0Ato mainline iproute2.= =0A=0AI've now spent months failing to come up with something that=0Acould = emulate in linux the non-duplex behavior and arbitration steps=0Athat wifi = goes through in order to find a new station to transmit to,=0Aor receive fr= om, using netem as a base.=0A=0AGetting that non-duplex behavior right is t= he *single most important=0Athing*, I think, for trying to emulate real wir= eless behaviors in=0Areal time that I can think of (and to thus be able to = run and improve=0Avarious e2e transports against it).=0A=0AA potential tc A= PI seems simple:=0A=0Atc qdisc add dev veth1 root netem coupled # master (A= P)=0Atc qdisc add dev veth2 root netem couple veth1 # client=0Atc qdisc add= dev veth3 root netme couple veth2 # client=0A=0ASomething more complicated= would be to create some sort of=0Aarbitration device and attach that to th= e qdiscs. (which would make=0Ait more possible to write arbitration devices= to emulate lte, gpon,=0Acable, wireless mesh and other non-duplex behavior= s in real time)=0A=0ABut how to convince qdiscs to be arbitrated, only allo= wing one in a=0Aset to transmit at the same time? (and worse, in the long r= un,=0Aallowing MU-MIMO-like behaviors).=0A=0AI'm tempted to *not* put my fa= iled thinking down here in the hope that=0Asomeone says, out there, "oh, th= at's easy, just create this structure=0Awith X API call and use Y function = and you're clear of all the=0Apotential deadlock and RCU issues, and we've = been doing that for=0Ayears, you idiot! Here's the code for how we do it, s= orry we didn't=0Asubmit it earlier."=0A=0AWhat I thought (*and still think*= ) is of creating a superset of the=0Aqdisc_watchdog_schedule_ns() function = is a start at it:=0A=0Atag =3D qdisc_watchdog_create_arb("some identifier")= ;=0Aqdisc_watchdog_schedule_arb(nsec, tag); /* null tag =3D schedule_ns */= =0A=0Awhich doesn't allow that qdisc instance to be run until the arbitrato= r=0Asays it can run (essentially overriding the timeout specified)=0A=0ABut= I actually wouldn't mind something that worked at the veth, or=0Adevice, r= ather than qdisc level...=0A=0Athoughts?=0A=0APS I just spent several days = working on another aspect of the problem,=0Awhich is replaying delay distri= butions (caused by interference and=0Asuch)... and that, sigh, to me, also = belongs in some sort of=0Aarbitration device rather than directly in netem.= Maybe tossing netem=0Aentirely is the answer. I don't know.=0A=0A-- =0A=0A= Dave T=C3=A4ht=0ACEO, TekLibre, LLC=0Ahttp://www.teklibre.com=0ATel: 1-669-= 226-2619=0A_______________________________________________=0AMake-wifi-fast= mailing list=0AMake-wifi-fast@lists.bufferbloat.net=0Ahttps://lists.buffer= bloat.net/listinfo/make-wifi-fast=0A-- =0AReed Online Ltd is a company regi= stered in England and Wales. Company =0ARegistration Number: 6317279.Regist= ered Office: Academy Court, 94 Chancery =0ALane, London WC2A 1DT.=0A=0A ------=_20180530185753000000_17761 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I would toss netem rather = than kludging around what appears to be a fundamental design choice made in= ins conceptualization. Make a "netem2".

=0A

 

= =0A

FreeBSD has a very nice framework for emulating far mor= e general packet queuing/routing/... in the kernel, called NetGraph. It's i= ncredibly general, and could straightforwardly, with high performance, have= modules that do exactly the right emulations of network structures with su= ch blocking, etc. and even random delays.

=0A

 

= =0A

I know this because in my day job at TidalScale, we hea= vily use NetGraph to implement new very low level protocols, which is prett= y straightforward, even including complex multi-adapter adaptive forwarding= of our private protocols on 10 and 40 GigE links. Super flexible, entirely= in the kernel, running either at real-time priority or not, in a mix.

= =0A

 

=0A

In contrast, the Linux TC= framework seems very inflexible, as you've found, in trying to push it to = do what it is not designed to do.

=0A

 

=0A

So tossing netem might be far better. I wonder if NetGraph has = ever been ported into some Linux kernel environment...

=0A

-----Original Message-----
From: "Dave Taht" <dave.taht@gmail.co= m>
Sent: Wednesday, May 30, 2018 1:28pm
To: make-wifi-fast@lis= ts.bufferbloat.net
Subject: [Make-wifi-fast] emulating wifi better - c= oupling qdiscs in netem?

=0A
=0A

The match to reality of my "wifi slotting" code for ne= tem was so
disappointing that I was extremely reluctant to push suppor= t for it up
to mainline iproute2.

I've now spent months fai= ling to come up with something that
could emulate in linux the non-dup= lex behavior and arbitration steps
that wifi goes through in order to = find a new station to transmit to,
or receive from, using netem as a b= ase.

Getting that non-duplex behavior right is the *single most = important
thing*, I think, for trying to emulate real wireless behavio= rs in
real time that I can think of (and to thus be able to run and im= prove
various e2e transports against it).

A potential tc AP= I seems simple:

tc qdisc add dev veth1 root netem coupled # mast= er (AP)
tc qdisc add dev veth2 root netem couple veth1 # client
t= c qdisc add dev veth3 root netme couple veth2 # client

Something= more complicated would be to create some sort of
arbitration device a= nd attach that to the qdiscs. (which would make
it more possible to wr= ite arbitration devices to emulate lte, gpon,
cable, wireless mesh and= other non-duplex behaviors in real time)

But how to convince qd= iscs to be arbitrated, only allowing one in a
set to transmit at the s= ame time? (and worse, in the long run,
allowing MU-MIMO-like behaviors= ).

I'm tempted to *not* put my failed thinking down here in the = hope that
someone says, out there, "oh, that's easy, just create this = structure
with X API call and use Y function and you're clear of all t= he
potential deadlock and RCU issues, and we've been doing that foryears, you idiot! Here's the code for how we do it, sorry we didn't
submit it earlier."

What I thought (*and still think*) is of cr= eating a superset of the
qdisc_watchdog_schedule_ns() function is a st= art at it:

tag =3D qdisc_watchdog_create_arb("some identifier");=
qdisc_watchdog_schedule_arb(nsec, tag); /* null tag =3D schedule_ns *= /

which doesn't allow that qdisc instance to be run until the ar= bitrator
says it can run (essentially overriding the timeout specified= )

But I actually wouldn't mind something that worked at the veth= , or
device, rather than qdisc level...

thoughts?

PS I just spent several days working on another aspect of the problem,which is replaying delay distributions (caused by interference and
= such)... and that, sigh, to me, also belongs in some sort of
arbitrati= on device rather than directly in netem. Maybe tossing netem
entirely = is the answer. I don't know.

--

Dave T=C3=A4ht
= CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619_______________________________________________
Make-wifi-fast mail= ing list
Make-wifi-fast@lists.bufferbloat.net
https://lists.buffe= rbloat.net/listinfo/make-wifi-fast
--
Reed Online Ltd is a compa= ny registered in England and Wales. Company
Registration Number:= 6317279.Registered Office: Academy Court, 94 Chancery
Lane, London W= C2A 1DT.

=0A
 ------=_20180530185753000000_17761--