From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp109.iad3a.emailsrvr.com (smtp109.iad3a.emailsrvr.com [173.203.187.109]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 617833B29E for ; Wed, 22 Apr 2020 13:42:55 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=g001.emailsrvr.com; s=20190322-9u7zjiwi; t=1587577375; bh=i2a/o0P1ZXLH7RqoYyMetxsDiEEY4RnBrb9qpdCQ+TM=; h=Date:Subject:From:To:From; b=ULNXhVulVIGhEmRMGjvhqaxjMDuCO59g4NqqKPrbOBQTcAL7fSg/tfY+lfsa2ENQh KyFTKu44tULWdmrS5kidc8Q/undasf8vczowwd5873e0z9Jbq1VaZ3F416C7/iPpAf 7znkKSaHMyEmBAYyX49fgy/LXkbQM601VP0SIb8k= Received: from app25.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp30.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id E51085475; Wed, 22 Apr 2020 13:42:54 -0400 (EDT) X-Sender-Id: dpreed@deepplum.com Received: from app25.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.7.12); Wed, 22 Apr 2020 13:42:55 -0400 Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app25.wa-webapps.iad3a (Postfix) with ESMTP id CDB9E20042; Wed, 22 Apr 2020 13:42:54 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Wed, 22 Apr 2020 13:42:54 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Wed, 22 Apr 2020 13:42:54 -0400 (EDT) From: "David P. Reed" To: "Luca Muscariello" Cc: "Dave Taht" , "Cake List" , "Maxime Bizon" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20200422134254000000_52480" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <75FEC2D9-BFC8-4FA2-A972-D11A823C5528@gmail.com> <603DFF79-D0C0-41BD-A2FB-E40B95A9CBB0@gmail.com> Message-ID: <1587577374.840114194@apps.rackspace.com> X-Mailer: webmail/17.3.7-RC X-Classification-ID: 26b65163-26ed-45e1-98e3-848d24011b8e-1-1 Subject: Re: [Cake] Advantages to tightly tuning latency X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2020 17:42:55 -0000 ------=_20200422134254000000_52480 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AHaving asymmetric gigabit cable modem service (1 Gb/s down) and very sho= rt latencies (5 ms.) to many servers of interest that can source 1 Gb/s), I= would just comment that I find it very, very useful for "normal" use.=0A = =0APerhaps my point is this: "normal" isn't a narrow gaussian distribution = of performance needs. It's what might be called a time-varying long tailed = distribution.=0A =0AI pay for 1 gb/sec because it is "worth it" to download= from, say, github cloning or a docker container image in under 1 second.= =0A =0ATo think that isn't valuable is to miss the point that the Internet'= s performance isn't about isochronous flows or slow FTPs - it's not about t= hroughput. It's about service delay.=0A =0AAnd congestion control is about = mitigating service delays under load, by eliminating sustained queueing del= ays that build up due to multiplexed use otherwise.=0A =0ATo talk about one= use at a time, and treat an average throughput as the goal metric is to mi= ss the entire point.=0A =0AA home access connection is frequently multiplex= ed over unrelated uses. If you are single, live in your own apartment, ... = you have a very, very warped idea of real usage.=0AOn Wednesday, April 22, = 2020 11:28am, "Luca Muscariello" said:=0A=0A=0A=0A= =0A=0A=0A=0AOn Wed, Apr 22, 2020 at 4:48 PM Dave Taht <[ dave.taht@gmail.co= m ]( mailto:dave.taht@gmail.com )> wrote:On Wed, Apr 22, 2020 at 2:04 AM Lu= ca Muscariello <[ muscariello@ieee.org ]( mailto:muscariello@ieee.org )> wr= ote:=0A >=0A >=0A >=0A > On Wed, Apr 22, 2020 at 12:44 AM Dave Taht <[ dave= .taht@gmail.com ]( mailto:dave.taht@gmail.com )> wrote:=0A >>=0A >> On Tue,= Apr 21, 2020 at 3:33 PM Jonathan Morton <[ chromatix99@gmail.com ]( mailto= :chromatix99@gmail.com )> wrote:=0A >> >=0A >> > > On 22 Apr, 2020, at 1:25= am, Thibaut <[ hacks@slashdirt.org ]( mailto:hacks@slashdirt.org )> wrote:= =0A >> > >=0A >> > > My curiosity is piqued. Can you elaborate on this? Wha= t does [ free.fr ]( http://free.fr ) do?=0A >> >=0A >> > They're a large Fr= ench ISP. They made their own CPE devices, and debloated both them and the= ir network quite a while ago. In that sense, at least, they're a model for= others to follow - but few have.=0A >> >=0A >> > - Jonathan Morton=0A >>= =0A >> they are one of the few ISPs that insisted on getting full source co= de=0A >> to their DSL stack, and retained the chops to be able to modify it= . I=0A >> really admire their revolution v6 product. First introduced in 20= 10,=0A >> it's been continuously updated, did ipv6 at the outset, got fq_co= del=0A >> when it first came out, and they update the kernel regularly. All= =0A >> kinds of great features on it, and ecn is enabled by default for tho= se=0A >> also (things like samba). over 3 million boxes now I hear....=0A >= >=0A >> with <1ms of delay in the dsl driver, they don't need to shape, the= y=0A >> just run at line rate using three tiers of DRR that look a lot like= =0A >> cake. They shared their config with me, and before I lost heart for= =0A >> future internet drafts, I'd stuck it here:=0A >>=0A >> [ https://git= hub.com/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/mi= ddle.mkd ]( https://github.com/dtaht/bufferbloat-rfcs/blob/master/home_gate= way_queue_management/middle.mkd )=0A >>=0A >> Occasionally they share some = data with me. Sometimes I wish I lived in=0A >> paris just so I could have = good internet! (their fiber offering is=0A >> reasonably buffered (not fq_c= odeled) and the wifi... maybe I can get=0A >> them to talk about what they = did)=0A >>=0A >> When [ free.fr ]( http://free.fr ) shipped fq_codel 2 mont= hs after we finalized it, I=0A >> figured the rest of the world was only mo= nths behind. How hard is it=0A >> to add 50 lines of BQL oriented code to a= DSL firmware?=0A >>=0A >=0A > Free has been using SFQ since 2005 (if I rem= ember well).=0A > They announced the wide deployment of SFQ in the [ free.f= r ]( http://free.fr ) newsgroup.=0A > Wi-Fi in the [ free.fr ]( http://free= .fr ) router was not as good though.=0A=0A They're working on it. :)=0A=0A = > In Paris there is a lot of GPON now that is replacing DSL. But there is= =0A > a nation-wide effort funded by local administrations to get fiber=0A = > everywhere. There are small towns in the countryside with fiber.=0A > Pub= lic money has made, and is making that possible.=0A > There is still a litt= le of Euro-DOCSIS, but frankly compared to fiber=0A > it has no chance to s= urvive.=0A=0A I am very, very happy for y'all. Fiber has always been the sa= nest=0A thing. Is there=0A a SPF+ gpon card yet I can plug into a conventio= n open source router yet?=0A=0A >=0A > I currently have 2Gbps/600Mbps acces= s with [ orange.fr ]( http://orange.fr ) and [ free.fr ]( http://free.fr ) = has a subscription=0A > at 10Gbps GPON. I won't tell you the price because = you may feel depressed=0A > compared to other countries where prices are mu= ch higher.=0A=0A I'd emigrate!!!=0A=0A > The challenge becomes to keep up w= ith these link rates in software=0A > as there is a lot of hardware offload= ing.=0A=0AI just meant that these routers tend to use HW offloading =0Aand = kernel qdiscs may be bypassed.=0A =0A At this point, I kind of buy the stan= ford sqrt(bdp) argument. All you=0A really need for gigE+ fiber access to w= ork well=0A for most modern traffic is a fairly short fifo (say, 20ms). Any= form=0A of FQ would help but be hardly noticible. I think=0A there needs t= o be work on the hop between the internet and the subscriber...=0A=0A Web t= raffic is dominated by RTT above 40mbit (presently).=0A streaming video tra= ffic - is no more than 20Mbit, and your occasional=0A big download is a doz= en big streams that would=0A bounce off a short fifo well.=0A gbit access t= o the home is (admittedly glorious, wonderful!) overkill=0A for all present= forms of traffic.=0A=0A I'm pretty sure if I had gig fiber I could come up= with a way to use=0A it up (exiting the cloud entirely comes to mind), but= =0A lacking new applications that demand that much bandwidth...=0A=0A I of = course, would like to see lola ( [ https://lola.conts.it/ ]( https://lola.c= onts.it/ ) ) finally=0A work, and videoconferencing and game stream with hi= gh rates and faster=0A (even raw) encoding also has potential to reduce e2e= latencies=0A enormously at that layer.=0A=0A >=0A > As soon as 802.11ax be= comes the norm, software scheduling will become=0A > a challenge.=0A=0A Do = you mean in fiber or wireless? wireless is really problematic at ANY speed.= =0A=0AI meant that software scheduling becomes a challenge for the same=0Ar= eason as above. Increase in total throughput of the box=0Awill call for har= dware offloading and kernel qdisc may be bypassed.=0AIt is not a challenge = per se, it is a challenge because traffic=0Amay not be managed by the kerne= l.=0A =0A at gfiber, the buffering moved to the wifi, and there are other= =0A problems that really impact achievable bandwidth. When I was last in=0A= paris, I could "hear" 300+ access points from my apt, and could only=0A ge= t 100-200kbit per second out of the wireless n ap I had, unless I=0A cheate= d and stuck my traffic in the VI queue. A friend of mine there,=0A couldn't= even get wifi across the room! Beacons ate into a lot of the=0A available= =0A bandwidth. Since 5ghz (and soon 6ghz - is 6E a thing in france) is=0A s= horter range I'm hoping that's got better, but with=0A 802.11ac and ax peei= ng on half the wifi spectrum by default, I imagine=0A achievable rates in h= igh density locations with many APs will be very=0A low... and very jittery= ... and thus still require good ATF, fq, and=0A aqm technologies.=0A=0A I h= ave high hopes for OFDMA and DU but thus far haven't found an AP=0A doing i= t. I'm not sure what to do about the beaconing problem except=0A offer a fr= ee tradein to all my neighbors still emitting G style=0A frames....=0A=0A A= nd in looking over some preliminary code for the mt76 ax chip, I=0A worry a= bout both bad design of the firmware, and=0A insufficient resources on-chip= to manage well.=0A=0A How is the 5G rollout going in france?=0A=0AGood que= stion. I've just seen a speed test at Gbps on a phone=0Awhich can drain you= r battery in less than 5 minutes. Amazing tech!=0A =0A I recently learned t= hat much of japan is... wait for it... wimax.=0A=0A >=0A > Luca=0A=0A=0A=0A= -- =0A Make Music, Not War=0A=0A Dave T=C3=A4ht=0A CTO, TekLibre, LLC=0A[ = http://www.teklibre.com ]( http://www.teklibre.com )=0A Tel: 1-831-435-0729 ------=_20200422134254000000_52480 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Having asymmetric giga= bit cable modem service (1 Gb/s down) and very short latencies (5 ms.) to m= any servers of interest that can source 1 Gb/s), I would just comment that = I find it very, very useful for "normal" use.

=0A

&n= bsp;

=0A

Perhaps my point is this: "normal" isn't a = narrow gaussian distribution of performance needs. It's what might be calle= d a time-varying long tailed distribution.

=0A

 = ;

=0A

I pay for 1 gb/sec because it is "worth it" to= download from, say, github cloning or a docker container image in under 1 = second.

=0A

 

=0A

To thi= nk that isn't valuable is to miss the point that the Internet's performance= isn't about isochronous flows or slow FTPs - it's not about throughput. It= 's about service delay.

=0A

 

=0A

And congestion control is about mitigating service delays under l= oad, by eliminating sustained queueing delays that build up due to multiple= xed use otherwise.

=0A

 

=0A

To talk about one use at a time, and treat an average throughput as th= e goal metric is to miss the entire point.

=0A

 = ;

=0A

A home access connection is frequently multipl= exed over unrelated uses. If you are single, live in your own apartment, ..= . you have a very, very warped idea of real usage.

=0A

On Wednesday, April 22, 2020 11:28am, "Luca Muscariello" <muscariello= @ieee.org> said:

=0A
=0A<= div dir=3D"ltr">=0A
=0A
=0A
=0A
On Wed, Apr 22, 2020 at 4:48= PM Dave Taht <= dave.taht@gmail.com> wrote:
=0A
On Wed, Apr 22, 2020 at 2:04 AM Luca Muscariello <muscariello@ieee.org= > wrote:
>
>
>
> On Wed, Apr 22, 202= 0 at 12:44 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>>= ; On Tue, Apr 21, 2020 at 3:33 PM Jonathan Morton <chromatix99@gmail.com> wrote:<= br /> >> >
>> > > On 22 Apr, 2020, at 1:25 am, T= hibaut <hacks@s= lashdirt.org> wrote:
>> > >
>> > &g= t; My curiosity is piqued. Can you elaborate on this? What does free.fr do?
= >> >
>> > They're a large French ISP.  They ma= de their own CPE devices, and debloated both them and their network quite a= while ago.  In that sense, at least, they're a model for others to fo= llow - but few have.
>> >
>> >  - Jonath= an Morton
>>
>> they are one of the few ISPs that i= nsisted on getting full source code
>> to their DSL stack, and = retained the chops to be able to modify it. I
>> really admire = their revolution v6 product. First introduced in 2010,
>> it's = been continuously updated, did ipv6 at the outset, got fq_codel
>&= gt; when it first came out, and they update the kernel regularly. All
= >> kinds of great features on it, and ecn is enabled by default for = those
>> also (things like samba). over 3 million boxes now I h= ear....
>>
>> with <1ms of delay in the dsl driv= er, they don't need to shape, they
>> just run at line rate usi= ng three tiers of DRR that look a lot like
>> cake. They shared= their config with me, and before I lost heart for
>> future in= ternet drafts, I'd stuck it here:
>>
>> https://github.co= m/dtaht/bufferbloat-rfcs/blob/master/home_gateway_queue_management/middle.m= kd
>>
>> Occasionally they share some data with= me. Sometimes I wish I lived in
>> paris just so I could have = good internet! (their fiber offering is
>> reasonably buffered = (not fq_codeled) and the wifi... maybe I can get
>> them to tal= k about what they did)
>>
>> When free.fr shipped fq_codel= 2 months after we finalized it, I
>> figured the rest of the w= orld was only months behind. How hard is it
>> to add 50 lines = of BQL oriented code to a DSL firmware?
>>
>
>= ; Free has been using SFQ since 2005 (if I remember well).
> They = announced the wide deployment of SFQ in the free.fr newsgroup.
> Wi-Fi in= the free.f= r router was not as good though.

They're working on it. :)<= br />
> In Paris there is a lot of GPON now that is replacing DSL.= But there is
> a nation-wide effort funded by local administratio= ns to get fiber
> everywhere. There are small towns in the country= side with fiber.
> Public money has made, and is making that possi= ble.
> There is still a little of Euro-DOCSIS, but frankly compare= d to fiber
> it has no chance to survive.

I am very, v= ery happy for y'all. Fiber has always been the sanest
thing. Is there=
a SPF+ gpon card yet I can plug into a convention open source router= yet?

>
> I currently have 2Gbps/600Mbps access wit= h orange.= fr and = free.fr has a subscription
> at 10Gbps GPON. I won't tell you = the price because you may feel depressed
> compared to other count= ries where prices are much higher.

I'd emigrate!!!

= > The challenge becomes to keep up with these link rates in software
> as there is a lot of hardware offloading.
=0A
=0AI just meant t= hat these routers tend to use HW offloading 
=0A
and kernel qdiscs may = be bypassed.
=0A
=0A
 
=0A

At this point, I kind of buy the stanford sqrt= (bdp) argument. All you
really need for gigE+ fiber access to work we= ll
for most modern traffic is a fairly short fifo (say, 20ms). Any fo= rm
of FQ would help but be hardly noticible. I think
there need= s to be work on the hop between the internet and the subscriber...
Web traffic is dominated by RTT above 40mbit (presently).
streami= ng video traffic - is no more than 20Mbit, and your occasional
big do= wnload is a dozen big streams that would
bounce off a short fifo well= .
gbit access to the home is (admittedly glorious, wonderful!) overki= ll
for all present forms of traffic.

I'm pretty sure if I= had gig fiber I could come up with a way to use
it up (exiting the c= loud entirely comes to mind), but
lacking new applications that deman= d that much bandwidth...

I of course, would like to see lola ( = htt= ps://lola.conts.it/ ) finally
work, and videoconferencing and gam= e stream with high rates and faster
(even raw) encoding also has pote= ntial to reduce e2e latencies
enormously at that layer.

&= gt;
> As soon as 802.11ax becomes the norm, software scheduling wi= ll become
> a challenge.

Do you mean in fiber or wirel= ess? wireless is really problematic at ANY speed.
=0A
=0AI meant that s= oftware scheduling becomes a challenge for the same
=0A
reason as above. Increase i= n total throughput of the box
=0A
will call for hardware offloading and kernel qdis= c may be bypassed.
=0A
It is not a challenge per se, it is a challenge because traf= fic
=0A
= may not be managed by the kernel.
=0A
=0A
 
=0A
at gfiber, the buffering = moved to the wifi, and there are other
problems that really impact ac= hievable bandwidth. When I was last in
paris, I could "hear" 300+ acc= ess points from my apt, and could only
get 100-200kbit per second out= of the wireless n ap I had, unless I
cheated and stuck my traffic in= the VI queue. A friend of mine there,
couldn't even get wifi across = the room! Beacons ate into a lot of the
available
bandwidth. Si= nce 5ghz (and soon 6ghz - is 6E a thing in france) is
shorter range I= 'm hoping that's got better, but with
802.11ac and ax peeing on half = the wifi spectrum by default, I imagine
achievable rates in high dens= ity locations with many APs will be very
low... and very jittery... a= nd thus still require good ATF, fq, and
aqm technologies.

= I have high hopes for OFDMA and DU but thus far haven't found an AP
= doing it. I'm not sure what to do about the beaconing problem except
= offer a free tradein to all my neighbors still emitting G style
frame= s....

And in looking over some preliminary code for the mt76 ax= chip, I
worry about both bad design of the firmware, and
insuf= ficient resources on-chip to manage well.

How is the 5G rollout= going in france?=0A
=0A
Good question. I've just seen a speed test at = Gbps on a phone
=0A
which can drain your battery in less than 5 minutes. Amazing te= ch!
=0A
=0A
 
=0A

I recently learned that much of japan is... wait for it= ... wimax.

>
> Luca



--
Make Music, Not War

Dave T=C3=A4ht
CTO, TekLibre, LLC<= br />http://www.teklibre.com
Tel: 1-831-435-0729
=0A=0A=0A ------=_20200422134254000000_52480--