From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp110.iad3a.emailsrvr.com (smtp110.iad3a.emailsrvr.com [173.203.187.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6B60D3CB38 for ; Fri, 27 Mar 2020 15:00:05 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=g001.emailsrvr.com; s=20190322-9u7zjiwi; t=1585335605; bh=+YdGh6uj8178P3tv+Y824/YauSOoUtmbPhRnWYrOZk0=; h=Date:Subject:From:To:From; b=HUjcaDndMhFaKjJji935oqbFOOonFOtgvs7Dy2yHVJAqdygR5iD8qTlyDLPOSQERG Ha9KMKTUcL0i9Jd3Jb1QU135AUsD8RAlQfKs86k09IdqBRB30wYrErGPpW9istZON2 6pDuhi0+h/3OddCeWduJzov0+vtS6U/IDo+Z+RsE= Received: from app51.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp6.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id E57C255F7; Fri, 27 Mar 2020 15:00:04 -0400 (EDT) X-Sender-Id: dpreed@deepplum.com Received: from app51.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.7.12); Fri, 27 Mar 2020 15:00:05 -0400 Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app51.wa-webapps.iad3a (Postfix) with ESMTP id CE357A12D4; Fri, 27 Mar 2020 15:00:04 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Fri, 27 Mar 2020 15:00:04 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Fri, 27 Mar 2020 15:00:04 -0400 (EDT) From: "David P. Reed" To: "Dave Taht" Cc: "bloat" , "Make-Wifi-fast" , "Cake List" , "cerowrt-devel" , "Ken Rice" , "Anthony Minessale II" MIME-Version: 1.0 Content-Type: text/plain;charset=UTF-8 Content-Transfer-Encoding: quoted-printable Importance: Normal X-Priority: 3 (Normal) X-Type: plain In-Reply-To: References: Message-ID: <1585335604.839628636@apps.rackspace.com> X-Mailer: webmail/17.3.3-RC X-Classification-ID: 7c6b1e4c-e1f4-4f6f-b938-07764c892702-1-1 Subject: Re: [Cerowrt-devel] =?utf-8?q?=5BCake=5D_mo_bettah_open_source_multi-?= =?utf-8?q?party_videoconferncing_in_an_age_of_bloated_uplinks=3F?= X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 19:00:05 -0000 Congestion control for real-time video is quite different than for streamin= g. Streaming really is dealt with by a big enough (multi-second) buffering,= and can in principle work great over TCP (if debloated).=0A=0AUDP congesti= on control MUST be end-to-end and done in the application layer, which is u= sually outside the OS kernel. This makes it tricky, because you end up with= latency variation due to eh OS's process scheduler that is on the order of= magnitude of the real-time requirements for air-to-air or light-to-light r= esponse (meaning the physical transition from sound or picture to and from = the transducer).=0A=0AThis creates a godawful mess when trying to do an app= . Whether in WebRTC (peer to peer UDP) or in a Linux userspace app, the sch= eduler has huge variance in delay.=0A=0ANow getting rid of bloat currently = requires TCP to respond to congestion signalling. UDP in the kernel doesn't= do that, and it doesn't tell userspace much either (you can try to detect = packet drops in userspace, but coding that up is quite hard because the sch= dulers get in the way of measurement, and forget about ECN being seen in us= erspace)=0A=0AThis is OS architecture messiness, not a layer 2 or 3 issue.= =0A=0AI've thought about this a lot. Here's my thoughts:=0A=0AI hate puttin= g things in the kernel! It's insecure. But what this says is that for very = historical and stupid reasons (related to the ideas of early timesharing sy= stems like Unix and Multics) folks try to make real-time algorithms look li= ke ordinary "processes" whose notion of controlling temporal behavior is ab= stracted away.=0A=0ASo: =0A1. We really should rethink how timing-sensitive= algorithms are expressed, and it isn't gonna be good to base them on semap= hores and threads that run at random rates. That means a very different OS = conceptual framework. Can this share with, say, the Linux we know and love = - yes, the hardware can be shared. One should be able to dedicate virtual p= rocessors that are not running Linux processes, but instead another computa= tional model (dataflow?).=0AAn example of this (though clunky and unsupport= ed by good tools) is in FreeBSD, it's called *netgraph*. It's a structured = way to write reactive algorithms that are demand or arrival driven. It also= has some security issues, and since it is heavily based on passing mbufs a= round it's really quirky. But I have found it useful for the kind of things= that need to get done in teleconferencing voice and video.=0A=0A2. EBPF is= interesting, because it is more secure, and is again focused on running co= de at kernel level, event-driven. I think it would be a seriously difficul= t lift to get it to the point where one could program the networked media p= rocessing in BPF.=0A=0A3. One of the nice things about KVM (hardware virtua= lization) is that potentially it lets different low level machine models sh= are a common machine. It occurs to me that using VIRTIO network devices and= some kind of VIRTIO media processing devices, that a KVM virtual machine c= ould be hooked up to the packet-level networking drivers in the end device,= isolating the teleconferencing from the rest of the endpoint OS, and creat= ing the right kind of near-bare--metal environment for managing the timing = of network packets and the paths to the screen and audio that would be simp= le and clean and tightly scheduled. KVM could "own" one or more of the phys= ical cores during the teleconference.=0A=0AYou can see, though, that this i= sn't just a "network protocol design" problem. This is only partly a networ= k protocol issue, but one that is coupled with the architecture of the end = systems.=0A=0AI reminisce a little bit thinking back to the 1970's and 80's= when TCP/IP and UDP/IP were being designed. Sadly, it was one of the big p= roblems of communicating between the OS community and the protocol communit= y that the OS community couldn't think outside the "timesharing" system box= , and the protocol community thought of networking like phone calls (sessio= ns). This is where the need for control of timing and buffering got lost. T= he timesharing folks largely thought of networks as for reliable timeless s= equential "streams" of data that had no particular urgency. The network pro= tocol folks were focused on ARQ.=0AOnly a few of us cared about end-to-end = latency bounds (where ends meant keyboard click or audio sample to screen d= isplay change or speaker motion). The packet speech guys did, but most netw= orking guys wanted to toss them under the bus as annoying. And those of us = doing distributed multinode algorithms did, but the remote login and FTP gu= ys were skeptical that would ever matter.=0A=0AIt's the latency, stupid. No= t the reliability, nor the consistency, nor throughput. Unless both the OS = and the path are focused on minimizing latency, a vast set of applications = will suck. Unfortunately, both the OS and network communities are *stuck* i= n a world where latency is uncontrollable, and there are no tools for getti= ng it better.=0A=0A =0A=0AOn Friday, March 27, 2020 1:27pm, "Dave Taht" said:=0A=0A> sort of an outgrowth of this convo:=0A> =0A= > https://lwn.net/SubscriberLink/815751/786d161d06a90f0e/=0A> =0A> I imagin= e worldwide videoconferencing quality could be much better if=0A> we could = convince more folk to=0A> finally install sqm or upgrade to a working docsi= s 3.1 solution, etc.=0A> Maybe some rag somewhere will finally pick up on b= ufferbloat solutions=0A> and run with it? Or we can write some articles? Or= reach out to school=0A> systems? Or?=0A> =0A> I've been fiddling with jits= i, and am about to give freeswitch a try.=0A> Last I looked freeswitch's ot= herwise pretty nifty conference bridge=0A> didn't dynamically adjust at all= due to e2e signalling, but that was=0A> years ago. (?)=0A> =0A> I have to = admit that p2p multiparty videoconferencing seems more=0A> plausible in a d= e-bufferbloated age, but=0A> haven't explored what tools are available. (?)= =0A> =0A> There's also been this somewhat entertaining convo on the ietf mb= one=0A> list: https://mailarchive.ietf.org/arch/msg/mboned/2thFQk_IYn38XCZB= QavhUmOd6tk/=0A> =0A> Around me there has been this huge interest in "strea= ming". The user=0A> agreement for these (see restream.io's) is scary - and = the copyright=0A> police have control... but I am very happy to report that= even a=0A> couple really lousy long distance fq_codel'd ath9k links work *= really*=0A> well (with facebook's implementation), where a non fq_codeled l= ink=0A> (ath10k) failed miserably... and setting up a reflector in nginx al= so=0A> failed miserably.=0A> =0A> Anyone working on the ath10k AQL backport= for openwrt as yet?=0A> =0A> --=0A> Make Music, Not War=0A> =0A> Dave T=C3= =A4ht=0A> CTO, TekLibre, LLC=0A> http://www.teklibre.com=0A> Tel: 1-831-435= -0729=0A> _______________________________________________=0A> Cake mailing = list=0A> Cake@lists.bufferbloat.net=0A> https://lists.bufferbloat.net/listi= nfo/cake=0A> =0A