From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 5B4DC3B2A4; Fri, 27 Mar 2020 16:32:36 -0400 (EDT) Received: by mail-io1-xd2e.google.com with SMTP id q128so11194130iof.9; Fri, 27 Mar 2020 13:32:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=kg8llDEAQiO2WYEv3U1Fhi9tFNuVoPq4yf+9Mjp1csY=; b=Vo0h/yQDmPkfeCUGbKRx4UUHZMYVAS1a3pwtTu/58WnnVLcaEeuYSAFkiokiPM5Oxi W8qKUC6LYPwlViYQkqBDaPgHlOb4Kr2dpC2W+F7OgVnr7dxZ9eVte7/EhwdyqsVpx9+o eBHnuExNyNuAajZzOghbjWR6ANaqzaeqyhbYOca7mT0vltmh42DbW6bbtkp754aTuX6D Wss5ztf0keF6Fv8+22jfbI+APlXC87AC+ZotV/lFphATLLvaji2MuvM6qwch6NmUCW2H crMx3oGHAgWBxSkGLZTcfSv39pa80Zys/M3Z9EUn5RC/PjvdgpqFYNtnP0IWYznskYho YW7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=kg8llDEAQiO2WYEv3U1Fhi9tFNuVoPq4yf+9Mjp1csY=; b=s9B64HwwxCLuFUJ2OGrqAUFAOl+DIS7kiuAiy/VfLurabNARLMzcKWFm5eJKVuFYCb 5++EfTI7JQwzhXxuSpPi5aJ3rIxg2K6mSAzpIVykCNRwwOtYYaKWgqIhXkHe+u/tNX4S x/BD/fcG0RYMmT2+gdk2xUjdM2SztMNa75XUjZLdDrJ3N1v13UpifNcHx9oZJTBhQLO2 Af3rdR+w/Vktgdewme3xxdX8zKQPciy09sL4FpUC+oM9LsxKFv0e0mXrufvzSccayRr1 PkyOWcYsu30jrpHz+2QQS9sjV9iaxPRBDDtaOSTQI5iMCANX7CiFx17ZoGTf6DTnF5wH 6UEw== X-Gm-Message-State: ANhLgQ17u2i4P8b9zTwxdHpZ7rPxrvT0n5XZAQlNFCYEXqCWzXna+GOV PW/rOTr1wFU7BfAXZnxKsO47DyV4x5a3yX3cQnDx5gmlKjA= X-Google-Smtp-Source: ADFU+vtsiYB89DV24+jAIp8yU2Iq6T4UAkfTu9dzrdf1x0ur976V6n9/6Abt9w1w5Qd4uN69K1Cg/d6KZ4qqCXhQjZ4= X-Received: by 2002:a5e:9b01:: with SMTP id j1mr442987iok.27.1585341155521; Fri, 27 Mar 2020 13:32:35 -0700 (PDT) MIME-Version: 1.0 References: <1585335604.839628636@apps.rackspace.com> In-Reply-To: <1585335604.839628636@apps.rackspace.com> From: Dave Taht Date: Fri, 27 Mar 2020 13:32:23 -0700 Message-ID: To: "David P. Reed" Cc: bloat , Make-Wifi-fast , Cake List , cerowrt-devel , Anthony Minessale II , Ken Rice Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Make-wifi-fast] [Cake] mo bettah open source multi-party videoconferncing in an age of bloated uplinks? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Mar 2020 20:32:36 -0000 I don't know to what extent the freeswitch guys would be interested in this thread. I'd like find a good list or forum to talk about the state of the art in videoconferencing ? , the ietf rmcat and webrtc lists are mostly dead. hangouts, jitsi, zoom, etc, seem to be pretty good products nowadays (at least in my fq_codel'd environment), but solid info on how to make them better in the home and for online tele-learning On Fri, Mar 27, 2020 at 12:00 PM David P. Reed wrote: > > Congestion control for real-time video is quite different than for stream= ing. Streaming really is dealt with by a big enough (multi-second) bufferin= g, and can in principle work great over TCP (if debloated). Your encoder still has to adjust to the available bandwidth. The facebook streaming application did this beautifully through my very limited highly shared 5mbit uplink - adjusting quickly to a parallel rrul test in particular by skipping some frames. then lowering the frame rate and quality, but an early attempt of mine to merely reflect rtmp streams did not, neither an attempt with "obs studio". there was about 30 sec of delay in the facebook test - I figure some of this is tuned to visible uplink buffer sizes (still seconds over cell), but also to give the riaa a shot at censoring the audio. (a commercial song crept into - over a mic! - which was detected as infringing on one attempt which automatically muted the audio and keyed a nastygram from fb) I'm going to poke into obs studios underlying code (rtsp anyone?0 at some point, and really - udp with a head dropping aqm is the best thing for transporting video, IMHO. > UDP congestion control MUST be end-to-end and done in the application lay= er, which is usually outside the OS kernel. This makes it tricky, because y= ou end up with latency variation due to eh OS's process scheduler that is o= n the order of magnitude of the real-time requirements for air-to-air or li= ght-to-light response (meaning the physical transition from sound or pictur= e to and from the transducer). We are so far from that point! encoder latencies today are in the 100+ms range. I always liked the opus codec because it can get down to 2.7ms encoding latencies, and a doubled frame rate camera 8ms.... but video encoding rates Im out of date on. (?) One long deferred piece of webrtc/rmcat research I always meant to do was audio and video on separate ports in the stream, and using that 2.7m opus clock and depending on fq at the bottleneck to provide better congestion control information by treating the smaller audio packets as a clock signal. Due to lack of port space and a widespread perception that fq isn't out there, most videoconferencing streams multiplex everything over the same port. With ipv6 in place, well, port space is no longer a problem. > > This creates a godawful mess when trying to do an app. Whether in WebRTC = (peer to peer UDP) or in a Linux userspace app, the scheduler has huge vari= ance in delay. I figure the bounding scheduler latency is still well manageable below a single 60fps frame. > Now getting rid of bloat currently requires TCP to respond to congestion = signalling. UDP in the kernel doesn't do that, and it doesn't tell userspac= e much either (you can try to detect packet drops in userspace, but coding = that up is quite hard because the schdulers get in the way of measurement, = and forget about ECN being seen in userspace) ECN in userspace is easy on udp, except that most api's tend to abstract into a file handle style abstraction and a single return of data, not control information, and the api for getting tos options ugly. APIs that can return data and info (data, packetheader) =3D getudp_someway() probably exist for more modern languages like go, but rarely c or c++. Totally out of date on this, last I looked at the google congestion congtrol code bae was in mozilla... 8 years ago! As for doing udp semi-efficiently in batches... sendmmsg, recvmmsg is a rather underused kernel api. And ugly as sin. With some major limitations. > > This is OS architecture messiness, not a layer 2 or 3 issue. To me the nightmare starts with most cpu context switch latencies being 1000s of clocks nowadays. > > I've thought about this a lot. Here's my thoughts: > > I hate putting things in the kernel! It's insecure. But what this says is= that for very historical and stupid reasons (related to the ideas of early= timesharing systems like Unix and Multics) folks try to make real-time alg= orithms look like ordinary "processes" whose notion of controlling temporal= behavior is abstracted away. On the whole, with the rise of quic - in particular quic, as multiple userspace libs have been emerging - we've got good bases to move forward with more stuff in userspace. > > So: > 1. We really should rethink how timing-sensitive algorithms are expressed= , and it isn't gonna be good to base them on semaphores and threads that ru= n at random rates. That means a very different OS conceptual framework. Can= this share with, say, the Linux we know and love - yes, the hardware can b= e shared. One should be able to dedicate virtual processors that are not ru= nning Linux processes, but instead another computational model (dataflow?). Linux switched to an EDF model for networking in 5.0 > An example of this (though clunky and unsupported by good tools) is in Fr= eeBSD, it's called *netgraph*. It's a structured way to write reactive algo= rithms that are demand or arrival driven. It also has some security issues,= and since it is heavily based on passing mbufs around it's really quirky. = But I have found it useful for the kind of things that need to get done in = teleconferencing voice and video. Neat. > > 2. EBPF is interesting, because it is more secure, and is again focused o= n running code at kernel level, event-driven. I think it would be a seriou= sly difficult lift to get it to the point where one could program the netwo= rked media processing in BPF. But there is huge demand for it, so people are writing way more in it than i ever ever thought possible... or desirable. > > 3. One of the nice things about KVM (hardware virtualization) is that pot= entially it lets different low level machine models share a common machine.= It occurs to me that using VIRTIO network devices and some kind of VIRTIO = media processing devices, that a KVM virtual machine could be hooked up to = the packet-level networking drivers in the end device, isolating the teleco= nferencing from the rest of the endpoint OS, and creating the right kind of= near-bare--metal environment for managing the timing of network packets an= d the paths to the screen and audio that would be simple and clean and tigh= tly scheduled. KVM could "own" one or more of the physical cores during the= teleconference. see also sch_etx and van's teaching nics about time - https://netdevconf.info/0x12/news.html?keynote-recording-is-up there has been a lot of progress on this front in the past few years - having applications say when they want a packet to emerge - but the offload discussion over on the linux list I referenced has seemingly missed this idea entirely. > > You can see, though, that this isn't just a "network protocol design" pro= blem. This is only partly a network protocol issue, but one that is coupled= with the architecture of the end systems. > > I reminisce a little bit thinking back to the 1970's and 80's when TCP/IP= and UDP/IP were being designed. Sadly, it was one of the big problems of c= ommunicating between the OS community and the protocol community that the O= S community couldn't think outside the "timesharing" system box, and the pr= otocol community thought of networking like phone calls (sessions). This is= where the need for control of timing and buffering got lost. The timeshari= ng folks largely thought of networks as for reliable timeless sequential "s= treams" of data that had no particular urgency. The network protocol folks = were focused on ARQ. > Only a few of us cared about end-to-end latency bounds (where ends meant = keyboard click or audio sample to screen display change or speaker motion). I recently got a usb keyboard with truly annoying latencies in it. https://danluu.com/ 's work makes me feel better about people perpetually ignoring this. An Apple II won the benchmark... Last year I got a voice processor, with 70+ms usb latencies for audio - useless for overdubs. same for all the usb based audio mixers made today. thunderbolt based audio gear is hard to find and expensive, I have been scrounging old firewire based stuff. (hey, if anyone has a RME multiface card let me know off list) > The packet speech guys did, but most networking guys wanted to toss them = under the bus as annoying. And those of us doing distributed multinode algo= rithms did, but the remote login and FTP guys were skeptical that would eve= r matter. Yep, we're annoying. But annoyed. And: I really think it would be a less stressed and better communicating world if we got cell phone audio latencies, in partiular, back below 20ms. > It's the latency, stupid. Not the reliability, nor the consistency, nor t= hroughput. Unless both the OS and the path are focused on minimizing latenc= y, a vast set of applications will suck. Unfortunately, both the OS and net= work communities are *stuck* in a world where latency is uncontrollable, an= d there are no tools for getting it better. except ours! :) > > > On Friday, March 27, 2020 1:27pm, "Dave Taht" said: > > > sort of an outgrowth of this convo: > > > > https://lwn.net/SubscriberLink/815751/786d161d06a90f0e/ > > > > I imagine worldwide videoconferencing quality could be much better if > > we could convince more folk to > > finally install sqm or upgrade to a working docsis 3.1 solution, etc. > > Maybe some rag somewhere will finally pick up on bufferbloat solutions > > and run with it? Or we can write some articles? Or reach out to school > > systems? Or? > > > > I've been fiddling with jitsi, and am about to give freeswitch a try. > > Last I looked freeswitch's otherwise pretty nifty conference bridge > > didn't dynamically adjust at all due to e2e signalling, but that was > > years ago. (?) > > > > I have to admit that p2p multiparty videoconferencing seems more > > plausible in a de-bufferbloated age, but > > haven't explored what tools are available. (?) > > > > There's also been this somewhat entertaining convo on the ietf mbone > > list: https://mailarchive.ietf.org/arch/msg/mboned/2thFQk_IYn38XCZBQavh= UmOd6tk/ > > > > Around me there has been this huge interest in "streaming". The user > > agreement for these (see restream.io's) is scary - and the copyright > > police have control... but I am very happy to report that even a > > couple really lousy long distance fq_codel'd ath9k links work *really* > > well (with facebook's implementation), where a non fq_codeled link > > (ath10k) failed miserably... and setting up a reflector in nginx also > > failed miserably. > > > > Anyone working on the ath10k AQL backport for openwrt as yet? > > > > -- > > Make Music, Not War > > > > Dave T=C3=A4ht > > CTO, TekLibre, LLC > > http://www.teklibre.com > > Tel: 1-831-435-0729 > > _______________________________________________ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > --=20 Make Music, Not War Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-435-0729