From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 687343B29D for ; Wed, 28 Sep 2022 14:49:22 -0400 (EDT) Received: by mail-pg1-x52e.google.com with SMTP id 78so12967011pgb.13 for ; Wed, 28 Sep 2022 11:49:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date; bh=lLRHnQeFnsWYIDyV6wxmg5/rxiJ5OoyV015HLhIy014=; b=Lh4FgcAt2IlVBjDH8zXEUA2JULGYpArSPTENxBSlOIBjxXeKQgcmYHQnCgrnOnad3j +Y42yW0sL2+ZLXchsCOKMREw3VdsqN3UQrumy8kqQJ7P2NUTxL/29ehWIzEbznGMomhC 6ysBtkF+CsC4LUJNevDkRt+Q1r0uGtlhJY1ps= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date; bh=lLRHnQeFnsWYIDyV6wxmg5/rxiJ5OoyV015HLhIy014=; b=72dDTeetnl8Y/JC5oQT4bjwtAD8g9DtwTgvE7sHeoXuC5DmOja5X6ImfYhxoDz+LNM MM9Dw7SvyezXTVQRkmV5bGUq1mkx+iL2a7Wm4pr4nSMYVGmupOdxwDEdlglTnH8s577Q FzlKK4UJVmaOY0eYLhmUWYMxTcHsib7jg79SSOOMNPFadxUV6QETF9962BkfODvzRx8Y 2ObDZi+AQDj6CHe5U4sjKMCFlKtic64zDwph6ge7x+cRE8FuaXWhwnJLvdSFlM5XIy5L xzgEjawjwu1n7iIi3e7JhGkMvaWfjjQGwrK4+pOyr08TsLe0VvOg1p/AmU5LOrg65EmB Cp4w== X-Gm-Message-State: ACrzQf0Yy0w6+gqw6XYwMCdHJZq5VhQoVbsUAODyn7oX5EBxL5MEfwXJ 6TTwzzNIWxO1sFeHCJTrm+5Vcg== X-Google-Smtp-Source: AMsMyM7kT0w0Yk9C/ogj8dPOpQqFfZnlNpfLDMi6cueDVAKLb0g1hOdKE4kClaKsz5FzJarKI5eHfg== X-Received: by 2002:a62:2983:0:b0:54e:7cd5:adb3 with SMTP id p125-20020a622983000000b0054e7cd5adb3mr35441043pfp.38.1664390961166; Wed, 28 Sep 2022 11:49:21 -0700 (PDT) Received: from smtpclient.apple (dhcp-72-253-196-65.hawaiiantel.net. [72.253.196.65]) by smtp.gmail.com with ESMTPSA id i6-20020a170902c94600b00177ff4019d9sm4102395pla.274.2022.09.28.11.49.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 28 Sep 2022 11:49:20 -0700 (PDT) From: Eugene Y Chang Message-Id: <0A55475E-9473-4CB7-80D7-D5E92F37E513@ieee.org> Content-Type: multipart/signed; boundary="Apple-Mail=_C06B9470-63A9-453F-8FE6-D15C0D2FA286"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Date: Wed, 28 Sep 2022 08:49:17 -1000 In-Reply-To: <447C034A-2BB7-4462-A6CF-C387832EC022@gmx.de> Cc: Eugene Chang , Bruce Perens , Dave Taht via Starlink To: Sebastian Moeller References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> <9D97FB02-58A5-48B0-9B43-6B7BD2A24099@gmx.de> <3FD7CB2A-8DC8-47FD-82DF-D27B967CBF4F@gmx.de> <45F57AD7-4479-4B06-AD1D-E7CF8D74D748@ieee.org> <447C034A-2BB7-4462-A6CF-C387832EC022@gmx.de> X-Mailer: Apple Mail (2.3696.120.41.1.1) Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2022 18:49:22 -0000 --Apple-Mail=_C06B9470-63A9-453F-8FE6-D15C0D2FA286 Content-Type: multipart/alternative; boundary="Apple-Mail=_9160C723-0C4A-46E1-A1EF-1F2329779BA2" --Apple-Mail=_9160C723-0C4A-46E1-A1EF-1F2329779BA2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 FYI, I believe the old 150ms was at the threshold where a person would = hear the telephony far-end echo. At the (approx) 250ms from the geos, = the far-end echo would interfere with speaking. What is interesting is the speed of sound in air (STP) is about 1ms/foot = (about 35 cm). On a big stage that is about 60 ms stage left to stage = right. The internet transporting sound would be awesome without jitter. Gene ---------------------------------------------- Eugene Chang IEEE Senior Life Member eugene.chang@ieee.org 781-799-0233 (in Honolulu) > On Sep 27, 2022, at 11:54 PM, Sebastian Moeller = wrote: >=20 > Hi Gene, >=20 >> On Sep 28, 2022, at 00:46, Eugene Y Chang = wrote: >>=20 >> Sebastian, >> Good to know. I haven=E2=80=99t had time to follow all the work going = on globally. >> I was only commenting on how the ISP support team behaves. >> How do we bring this better attitude to the US. >=20 > [SM] Good question. In the EU it was actually actually an = official EU regulation by the european council and the parliament = (https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=3DCELEX:32015R21= 20&from=3Dde) that is the basis for the national regulators to act. >=20 >> Sadly the FCC seems stuck accommodating the big ISPs. >=20 > [SM] Similar in Germany the regulator always sees to see itself = both as represntative of the end-user and at the same time as partner of = the ISPs and since ISP have better communication channels to the = regulators they also often seem to accomodate the big ISPs. However if = the relevant law is clear and unambiguous they act according to it. > Not sure though whether "go through the politicians" to improve the = directives of the FCC is a better approach given the state of USian = politics. However it should be conceptually easy to convince politicians = of the issue, it is not that they do not use the internet themselves and = might be open for a demonstration (nicely balanced between the two sides = of the aisle to avoid making this a partisan issue). >=20 >> Of course, we want the FCC, the ISPs, and the networking world to go = beyond speedtest. >=20 > [SM] +1; for all my praise for the local regulators action, they = also have dropped the ball on acceptable latency; they just defined the = minimum internet access people living in Germany are entitled to by law = (something like 10/1.7 Mbps but up to 150ms latency, all measured = against the regulators testing systems in the internet), the rates while = not great seem OK (as an absolute floor) but 150ms latency? What where = they thinking? I bet this comes from the ITU's old characterization of = mouth to ear latencies <=3D 150ms being OK, ignoring that mouth to ear = contains more than pure network delay and that if two of such users = actually try a VoIP call they end up with 300ms delay, which is deeply = in the awkward territory IIRC. >=20 > Regards > Sebastian >=20 >=20 >>=20 >> Gene >> ---------------------------------------------- >> Eugene Chang >> IEEE Senior Life Member >> eugene.chang@ieee.org >> 781-799-0233 (in Honolulu) >>=20 >>=20 >>=20 >>> On Sep 26, 2022, at 9:09 PM, Sebastian Moeller = wrote: >>>=20 >>> Hi Gene, >>>=20 >>>=20 >>>> On Sep 27, 2022, at 05:50, Eugene Y Chang = wrote: >>>>=20 >>>> Of course=E2=80=A6. But the ISP=E2=80=99s maxim is "don=E2=80=99t = believe the customer=E2=80=99s speedtest numbers if it is not with their = host". >>>=20 >>> [SM] In an act of reasonable regulation did the EU give national = regulatory agencies the right to define and set-up national = "speed-tests" (located outside of the access ISPs networks) which ISPs = effectively need to accept. In Germany the local NRA = (Bundes-Netz-Agentur, BNetzA) created a speedtest application against = servers operated on their behalf and will, if a customers demonstrates = the ISP falling short of the contracted rates (according to a somewhat = complicated definition and measurement rules). convince ISPs to follow = the law and either release customers from their contracts immediately or = lower the price in relation to the under-fulfillment. (Unfortunately all = related official web pages are in German only). >>> This put a hold on the practice of gaming speedtests, like = DOCSIS ISPs measuring an in-segment speedtest server which conveniently = hides if a segment's uplink is congested... Not all ISPs gamed their = speedtests, and even the in-segment speedtest can be justified for some = measurements (e.g. when trying to figure out whether congestion is = in-segment or out-of-segment), but the temptation must have been large = to not set-up the most objective spedtest. (We have a saying along the = lines of "making a goat your gardener" which generally is considered a = sub-optimal approach). >>>=20 >>> Regards >>> Sebastian >>>=20 >>>=20 >>>>=20 >>>>=20 >>>> Gene >>>> ---------------------------------------------- >>>> Eugene Chang >>>> IEEE Senior Life Member >>>> eugene.chang@ieee.org >>>> 781-799-0233 (in Honolulu) >>>>=20 >>>>=20 >>>>=20 >>>>> On Sep 26, 2022, at 11:44 AM, Bruce Perens = wrote: >>>>>=20 >>>>> That's a good maxim: Don't believe a speed test that is hosted by = your own ISP. >>>>>=20 >>>>> On Mon, Sep 26, 2022 at 2:36 PM Eugene Y Chang via Starlink = wrote: >>>>> Thank you for the dialog,. >>>>> This discussion with regards to Starlink is interesting as it = confirms my guesses about the gap between Starlinks overly simplified, = over optimistic marketing and the reality as they acquire subscribers. >>>>>=20 >>>>> I am actually interested in a more perverse issue. I am seeing = latency and bufferbloat as a consequence from significant under = provisioning. It doesn=E2=80=99t matter that the ISP is selling a fiber = drop, if (parts) of their network is under provisioned. Two end points = can be less than 5 mile apart and realize 120+ ms latency. Two Labor = Days ago (a holiday) the max latency was 230+ ms. The pattern I see = suggest digital redlining. The older communities appear to have much = more severe under provisioning. >>>>>=20 >>>>> Another observation. Running speedtest appears to go from the edge = of the network by layer 2 to the speedtest host operated by the ISP. = Yup, bypasses the (suspected overloaded) routers. >>>>>=20 >>>>> Anyway, just observing. >>>>>=20 >>>>> Gene >>>>> ---------------------------------------------- >>>>> Eugene Chang >>>>> IEEE Senior Life Member >>>>> eugene.chang@ieee.org >>>>> 781-799-0233 (in Honolulu) >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On Sep 26, 2022, at 11:20 AM, Sebastian Moeller = wrote: >>>>>>=20 >>>>>> Hi Gene, >>>>>>=20 >>>>>>=20 >>>>>>> On Sep 26, 2022, at 23:10, Eugene Y Chang = wrote: >>>>>>>=20 >>>>>>> Comments inline below. >>>>>>>=20 >>>>>>> Gene >>>>>>> ---------------------------------------------- >>>>>>> Eugene Chang >>>>>>> IEEE Senior Life Member >>>>>>> eugene.chang@ieee.org >>>>>>> 781-799-0233 (in Honolulu) >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>>>>>>>=20 >>>>>>>> Hi Eugene, >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = wrote: >>>>>>>>>=20 >>>>>>>>> Ok, we are getting into the details. I agree. >>>>>>>>>=20 >>>>>>>>> Every node in the path has to implement this to be effective. >>>>>>>>=20 >>>>>>>> Amazingly the biggest bang for the buck is gotten by = fixing those nodes that actually contain a network path's bottleneck. = Often these are pretty stable. So yes for fully guaranteed service = quality all nodes would need to participate, but for improving things = noticeably it is sufficient to improve the usual bottlenecks, e.g. for = many internet access links the home gateway is a decent point to = implement better buffer management. (In short the problem are over-sized = and under-managed buffers, and one of the best solution is = better/smarter buffer management). >>>>>>>>=20 >>>>>>>=20 >>>>>>> This is not completely true. >>>>>>=20 >>>>>> [SM] You are likely right, trying to summarize things leads to = partially incorrect generalizations. >>>>>>=20 >>>>>>=20 >>>>>>> Say the bottleneck is at node N. During the period of = congestion, the upstream node N-1 will have to buffer. When node N = recovers, the bufferbloat at N-1 will be blocking until the bufferbloat = drains. Etc. etc. Making node N better will reduce the extent of the = backup at N-1, but N-1 should implement the better code. >>>>>>=20 >>>>>> [SM] It is the node that builds up the queue that profits most = from better queue management.... (again I generalize, the node with the = queue itself probably does not care all that much, but the endpoints = will profit if the queue experiencing node deals with that queue more = gracefully). >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>>=20 >>>>>>>>> In fact, every node in the path has to have the same = prioritization or the scheme becomes ineffective. >>>>>>>>=20 >>>>>>>> Yes and no, one of the clearest winners has been flow = queueing, IMHO not because it is the most optimal capacity sharing = scheme, but because it is the least pessimal scheme, allowing all (or = none) flows forward progress. You can interpret that as a scheme in = which flows below their capacity share are prioritized, but I am not = sure that is the best way to look at these things. >>>>>>>=20 >>>>>>> The hardest part is getting competing ISPs to implement and = coordinate. >>>>>>=20 >>>>>> [SM] Yes, but it turned out even with non-cooperating ISPs there = is a lot end-users can do unilaterally on their side to improve both = ingress and egress congestion. Admittedly especially ingress congestion = would be even better handled with cooperation of the ISP. >>>>>>=20 >>>>>>> Bufferbloat and handoff between ISPs will be hard. The only way = to fix this is to get the unwashed public to care. Then they can say = =E2=80=9Cwe don=E2=80=99t care about the technical issues, just fix = it.=E2=80=9D Until then =E2=80=A6.. >>>>>>=20 >>>>>> [SM] Well we do this one home network at a time (not because = that is efficient or ideal, but simply because it is possible). Maybe, = if you have not done so already try OpenWrt with sqm-scripts (and maybe = cake-autorate in addition) on your home internet access link for say a = week and let us know ih/how your experience changed? >>>>>>=20 >>>>>> Regards >>>>>> Sebastian >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>>=20 >>>>>>>> Regards >>>>>>>> Sebastian >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Gene >>>>>>>>> ---------------------------------------------- >>>>>>>>> Eugene Chang >>>>>>>>> IEEE Senior Life Member >>>>>>>>> eugene.chang@ieee.org >>>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>> On Sep 26, 2022, at 10:48 AM, David Lang = wrote: >>>>>>>>>>=20 >>>>>>>>>> software updates can do far more than just improve recovery. >>>>>>>>>>=20 >>>>>>>>>> In practice, large data transfers are less sensitive to = latency than smaller data transfers (i.e. downloading a CD image vs a = video conference), software can ensure better fairness in preventing a = bulk transfer from hurting the more latency sensitive transfers. >>>>>>>>>>=20 >>>>>>>>>> (the example below is not completely accurate, but I think it = gets the point across) >>>>>>>>>>=20 >>>>>>>>>> When buffers become excessivly large, you have the situation = where a video call is going to generate a small amount of data at a = regular interval, but a bulk data transfer is able to dump a huge amount = of data into the buffer instantly. >>>>>>>>>>=20 >>>>>>>>>> If you just do FIFO, then you get a small chunk of video = call, then several seconds worth of CD transfer, followed by the next = small chunk of the video call. >>>>>>>>>>=20 >>>>>>>>>> But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>>>>>>>=20 >>>>>>>>>> The one thing that Cake needs to work really well is to be = able to know what the data rate available is. With Starlink, this = changes frequently and cake integrated into the starlink dish/router = software would be far better than anything that can be done externally = as the rate changes can be fed directly into the settings (currently = they are only indirectly detected) >>>>>>>>>>=20 >>>>>>>>>> David Lang >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>>>>>>>=20 >>>>>>>>>>> You already know this. Bufferbloat is a symptom and not the = cause. Bufferbloat grows when there are (1) periods of low or no = bandwidth or (2) periods of insufficient bandwidth (aka network = congestion). >>>>>>>>>>>=20 >>>>>>>>>>> If I understand this correctly, just a software update = cannot make bufferbloat go away. It might improve the speed of recovery = (e.g. throw away all time sensitive UDP messages). >>>>>>>>>>>=20 >>>>>>>>>>> Gene >>>>>>>>>>> ---------------------------------------------- >>>>>>>>>>> Eugene Chang >>>>>>>>>>> IEEE Senior Life Member >>>>>>>>>>> eugene.chang@ieee.org >>>>>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> Please help to explain. Here's a draft to start with: >>>>>>>>>>>>=20 >>>>>>>>>>>> Starlink Performance Not Sufficient for Military = Applications, Say Scientists >>>>>>>>>>>>=20 >>>>>>>>>>>> The problem is not availability: Starlink works where = nothing but another satellite network would. It's not bandwidth, = although others have questions about sustaining bandwidth as the = customer base grows. It's latency and jitter. As load increases, = latency, the time it takes for a packet to get through, increases more = than it should. The scientists who have fought bufferbloat, a major = cause of latency on the internet, know why. SpaceX needs to upgrade = their system to use the scientist's Open Source modifications to Linux = to fight bufferbloat, and thus reduce latency. This is mostly just using = a newer version, but there are some tunable parameters. Jitter is a = change in the speed of getting a packet through the network during a = connection, which is inevitable in satellite networks, but will be = improved by making use of the bufferbloat-fighting software, and = probably with the addition of more satellites. >>>>>>>>>>>>=20 >>>>>>>>>>>> We've done all of the work, SpaceX just needs to adopt it = by upgrading their software, said scientist Dave Taht. Jim Gettys, = Taht's collaborator and creator of the X Window System, chimed in: >>>>>>>>>>>> Open Source luminary Bruce Perens said: sometimes = Starlink's latency and jitter make it inadequate to remote-control my = ham radio station. But the military is experimenting with remote-control = of vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>>>>>>>=20 >>>>>>>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>>>>>>>> The key issue is most people don=E2=80=99t understand why = latency matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>>>>>>>=20 >>>>>>>>>>>> First, we have to help people see the symptoms of latency = and how it impacts something they care about. >>>>>>>>>>>> - gamers care but most people may think it is frivolous. >>>>>>>>>>>> - musicians care but that is mostly for a hobby. >>>>>>>>>>>> - business should care because of productivity but they = don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>>>>>>>=20 >>>>>>>>>>>> Second, there needs to be a =E2=80=9COMG, I have been = seeing the action of latency all this time and never knew it! I was = being shafted.=E2=80=9D Once you have this awakening, you can get all = the press you want for free. >>>>>>>>>>>>=20 >>>>>>>>>>>> Most of the time when business apps are developed, =E2=80=9Cw= e=E2=80=9D hide the impact of poor performance (aka latency) or they = hide from the discussion because the developers don=E2=80=99t have a way = to fix the latency. Maybe businesses don=E2=80=99t care because any = employees affected are just considered poor performers. (In bad economic = times, the poor performers are just laid off.) For employees, if they = happen to be at a location with bad latency, they don=E2=80=99t know = that latency is hurting them. Unfair but most people don=E2=80=99t know = the issue is latency. >>>>>>>>>>>>=20 >>>>>>>>>>>> Talking and explaining why latency is bad is not as = effective as showing why latency is bad. Showing has to be with = something that has a person impact. >>>>>>>>>>>>=20 >>>>>>>>>>>> Gene >>>>>>>>>>>> ----------------------------------- >>>>>>>>>>>> Eugene Chang >>>>>>>>>>>> eugene.chang@alum.mit.edu = >>>>>>>>>>>> +1-781-799-0233 (in Honolulu) >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> If you want to get attention, you can get it for free. I = can place articles with various press if there is something interesting = to say. Did this all through the evangelism of Open Source. All we need = to do is write, sign, and publish a statement. What they actually write = is less relevant if they publish a link to our statement. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Right now I am concerned that the Starlink latency and = jitter is going to be a problem even for remote controlling my ham = station. The US Military is interested in doing much more, which they = have demonstrated, but I don't see happening at scale without some = technical work on the network. Being able to say this isn't ready for = the government's application would be an attention-getter. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Thanks >>>>>>>>>>>>>=20 >>>>>>>>>>>>> Bruce >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>>>>>>>> These days, if you want attention, you gotta buy it. A 50k = half page >>>>>>>>>>>>> ad in the wapo or NYT riffing off of It's the latency, = Stupid!", >>>>>>>>>>>>> signed by the kinds of luminaries we got for the fcc wifi = fight, would >>>>>>>>>>>>> go a long way towards shifting the tide. >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht = > wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> The awareness & understanding of latency & impact on QoE = is nearly unknown among reporters. IMO maybe there should be some kind = of background briefings for reporters - maybe like a simple YouTube = video explainer that is short & high level & visual? Otherwise reporters = will just continue to focus on what they know... >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> That's a great idea. I have visions of crashing the = washington >>>>>>>>>>>>>> correspondents dinner, but perhaps >>>>>>>>>>>>>> there is some set of gatherings journalists regularly = attend? >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave = Taht via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>> I still find it remarkable that reporters are still = missing the >>>>>>>>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> -- >>>>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Starlink mailing list >>>>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> -- >>>>>>>>>>>>> Bruce Perens K6BP >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Starlink mailing list >>>>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> -- >>>>>>>>>>>> Bruce Perens K6BP >>>>>>>>>=20 >>>>>>>>> _______________________________________________ >>>>>>>>> Starlink mailing list >>>>>>>>> Starlink@lists.bufferbloat.net >>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>=20 >>>>> _______________________________________________ >>>>> Starlink mailing list >>>>> Starlink@lists.bufferbloat.net >>>>> https://lists.bufferbloat.net/listinfo/starlink >>>>>=20 >>>>>=20 >>>>> -- >>>>> Bruce Perens K6BP >>>>=20 >>>=20 >>=20 >=20 --Apple-Mail=_9160C723-0C4A-46E1-A1EF-1F2329779BA2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 FYI, = I believe the old 150ms was at the threshold where a person would hear = the telephony far-end echo. At the (approx) 250ms from the geos, the = far-end echo would interfere with speaking.

What is interesting is the speed of = sound in air (STP) is about 1ms/foot (about 35 cm). On a big stage that = is about 60 ms stage left to stage right. The internet transporting = sound would be awesome without jitter.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life = Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 27, 2022, at 11:54 PM, Sebastian Moeller <moeller0@gmx.de> = wrote:

Hi Gene,

On Sep 28, 2022, at 00:46, Eugene Y Chang <eugene.chang@ieee.org> wrote:

Sebastian,
Good to know. I haven=E2=80=99t had = time to follow all the work going on globally.
I was only = commenting on how the ISP support team behaves.
How do we = bring this better attitude to the US.

= [SM] Good question. In the EU it was actually actually an = official EU regulation by the european council and the parliament (https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=3DCELE= X:32015R2120&from=3Dde) that is the basis for the national = regulators to act.

Sadly the FCC seems stuck accommodating the big ISPs.

[SM] Similar in Germany the = regulator always sees to see itself both as represntative of the = end-user and at the same time as partner of the ISPs and since ISP have = better communication channels to the regulators they also often seem to = accomodate the big ISPs. However if the relevant law is clear and = unambiguous they act according to it.
Not sure though = whether "go through the politicians" to improve the directives of the = FCC is a better approach given the state of USian politics. However it = should be conceptually easy to convince politicians of the issue, it is = not that they do not use the internet themselves and might be open for a = demonstration (nicely balanced between the two sides of the aisle to = avoid making this a partisan issue).

Of course, we want the = FCC, the ISPs, and the networking world to go beyond speedtest.

[SM] +1; for all my praise for = the local regulators action, they also have dropped the ball on = acceptable latency; they just defined the minimum internet access people = living in Germany are entitled to by law (something like 10/1.7 Mbps but = up to 150ms latency, all measured against the regulators testing systems = in the internet), the rates while not great seem OK (as an absolute = floor) but 150ms latency? What where they thinking? I bet this comes = from the ITU's old characterization of mouth to ear latencies <=3D = 150ms being OK, ignoring that mouth to ear contains more than pure = network delay and that if two of such users actually try a VoIP call = they end up with 300ms delay, which is deeply in the awkward territory = IIRC.

Regards
= Sebastian



Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at 9:09 = PM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Gene,


On Sep 27, 2022, at = 05:50, Eugene Y Chang <eugene.chang@ieee.org> wrote:

Of course=E2=80=A6. But the ISP=E2=80=99s = maxim is "don=E2=80=99t believe the customer=E2=80=99s speedtest numbers = if it is not with their host".

= [SM] In an act of reasonable regulation did the EU give national = regulatory agencies the right to define and set-up national = "speed-tests" (located outside of the access ISPs networks) which ISPs = effectively need to accept. In Germany the local NRA = (Bundes-Netz-Agentur, BNetzA) created a speedtest application against = servers operated on their behalf and will, if a customers demonstrates = the ISP falling short of the contracted rates (according to a somewhat = complicated definition and measurement rules). convince ISPs to follow = the law and either release customers from their contracts immediately or = lower the price in relation to the under-fulfillment. (Unfortunately all = related official web pages are in German only).
This put = a hold on the practice of gaming speedtests, like DOCSIS ISPs measuring = an in-segment speedtest server which conveniently hides if a segment's = uplink is congested... Not all ISPs gamed their speedtests, and even the = in-segment speedtest can be justified for some measurements (e.g. when = trying to figure out whether congestion is in-segment or = out-of-segment), but the temptation must have been large to not set-up = the most objective spedtest. (We have a saying along the lines of = "making a goat your gardener" which generally is considered a = sub-optimal approach).

Regards
= Sebastian




Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:44 AM, Bruce Perens <bruce@perens.com> wrote:

That's a good maxim: Don't believe a speed test that is = hosted by your own ISP.

On Mon, Sep 26, = 2022 at 2:36 PM Eugene Y Chang via Starlink = <starlink@lists.bufferbloat.net> wrote:
Thank you = for the dialog,.
This discussion with regards to Starlink = is interesting as it confirms my guesses about the gap between Starlinks = overly simplified, over optimistic marketing and the reality as they = acquire subscribers.

I am actually = interested in a more perverse issue. I am seeing latency and bufferbloat = as a consequence from significant under provisioning. It doesn=E2=80=99t = matter that the ISP is selling a fiber drop, if (parts) of their network = is under provisioned. Two end points can be less than 5 mile apart and = realize 120+ ms latency. Two Labor Days ago (a holiday) the max latency = was 230+ ms. The pattern I see suggest digital redlining. The older = communities appear to have much more severe under provisioning.

Another observation. Running speedtest appears = to go from the edge of the network by layer 2 to the speedtest host = operated by the ISP. Yup, bypasses the (suspected overloaded) = routers.

Anyway, just observing.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:20 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Gene,


On Sep 26, 2022, at = 23:10, Eugene Y Chang <eugene.chang@ieee.org> wrote:

Comments inline below.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:01 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Eugene,


On Sep 26, 2022, at = 22:54, Eugene Y Chang via Starlink = <starlink@lists.bufferbloat.net> wrote:

Ok, we are getting into the details. I agree.
Every node in the path has to implement this to be = effective.

Amazingly = the biggest bang for the buck is gotten by fixing those nodes that = actually contain a network path's bottleneck. Often these are pretty = stable. So yes for fully guaranteed service quality all nodes would need = to participate, but for improving things noticeably it is sufficient to = improve the usual bottlenecks, e.g. for many internet access links the = home gateway is a decent point to implement better buffer management. = (In short the problem are over-sized and under-managed buffers, and one = of the best solution is better/smarter buffer management).


This is not = completely true.

[SM] You = are likely right, trying to summarize things leads to partially = incorrect generalizations.


Say the bottleneck is at = node N. During the period of congestion, the upstream node N-1 will have = to buffer. When node N recovers, the bufferbloat at N-1 will be blocking = until the bufferbloat drains. Etc. etc.  Making node N better will = reduce the extent of the backup at N-1, but N-1 should implement the = better code.

[SM] It = is the node that builds up the queue that profits most from better queue = management.... (again I generalize, the node with the queue itself = probably does not care all that much, but the endpoints will profit if = the queue experiencing node deals with that queue more gracefully).





In fact, = every node in the path has to have the same prioritization or the scheme = becomes ineffective.

Yes and = no, one of the clearest winners has been flow queueing, IMHO not because = it is the most optimal capacity sharing scheme, but because it is the = least pessimal scheme, allowing all (or none) flows forward progress. = You can interpret that as a scheme in which flows below their capacity = share are prioritized, but I am not sure that is the best way to look at = these things.

The hardest part = is getting competing ISPs to implement and coordinate.

[SM] Yes, but it turned out even = with non-cooperating ISPs there is a lot end-users can do unilaterally = on their side to improve both ingress and egress congestion. Admittedly = especially ingress congestion would be even better handled with = cooperation of the ISP.

Bufferbloat and handoff between ISPs will be = hard. The only way to fix this is to get the unwashed public to care. = Then they can say =E2=80=9Cwe don=E2=80=99t care about the technical = issues, just fix it.=E2=80=9D Until then =E2=80=A6..

[SM] Well we do this one home = network at a time (not because that is efficient or ideal, but simply = because it is possible). Maybe, if you have not done so already try = OpenWrt with sqm-scripts (and maybe cake-autorate in addition) on your = home internet access link for say a week and let us know ih/how your = experience changed?

Regards
= Sebastian






Regards
= Sebastian



Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:48 AM, David Lang <david@lang.hm> wrote:

software updates can do far more than just improve = recovery.

In practice, large data transfers = are less sensitive to latency than smaller data transfers (i.e. = downloading a CD image vs a video conference), software can ensure = better fairness in preventing a bulk transfer from hurting the more = latency sensitive transfers.

(the example = below is not completely accurate, but I think it gets the point = across)

When buffers become excessivly = large, you have the situation where a video call is going to generate a = small amount of data at a regular interval, but a bulk data transfer is = able to dump a huge amount of data into the buffer instantly.

If you just do FIFO, then you get a small = chunk of video call, then several seconds worth of CD transfer, followed = by the next small chunk of the video call.

But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic.

The one thing that Cake needs to work really well is to be = able to know what the data rate available is. With Starlink, this = changes frequently and cake integrated into the starlink dish/router = software would be far better than anything that can be done externally = as the rate changes can be fed directly into the settings (currently = they are only indirectly detected)

David = Lang


On Mon, 26 Sep 2022, = Eugene Y Chang via Starlink wrote:

You already know this. = Bufferbloat is a symptom and not the cause. Bufferbloat grows when there = are (1) periods of low or no bandwidth or (2) periods of insufficient = bandwidth (aka network congestion).

If I = understand this correctly, just a software update cannot make = bufferbloat go away. It might improve the speed of recovery (e.g. throw = away all time sensitive UDP messages).

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:04 AM, Bruce Perens <bruce@perens.com> wrote:

Please help to explain. Here's a draft to start with:

Starlink Performance Not Sufficient for = Military Applications, Say Scientists

The = problem is not availability: Starlink works where nothing but another = satellite network would. It's not bandwidth, although others have = questions about sustaining bandwidth as the customer base grows. It's = latency and jitter. As load increases, latency, the time it takes for a = packet to get through, increases more than it should. The scientists who = have fought bufferbloat, a major cause of latency on the internet, know = why. SpaceX needs to upgrade their system to use the scientist's Open = Source modifications to Linux to fight bufferbloat, and thus reduce = latency. This is mostly just using a newer version, but there are some = tunable parameters. Jitter is a change in the speed of getting a packet = through the network during a connection, which is inevitable in = satellite networks, but will be improved by making use of the = bufferbloat-fighting software, and probably with the addition of more = satellites.

We've done all of the work, = SpaceX just needs to adopt it by upgrading their software, said = scientist Dave Taht. Jim Gettys, Taht's collaborator and creator of the = X Window System, chimed in: <fill in here please>
Open= Source luminary Bruce Perens said: sometimes Starlink's latency and = jitter make it inadequate to remote-control my ham radio station. But = the military is experimenting with remote-control of vehicles on the = battlefield and other applications that can be demonstrated, but won't = happen at scale without adoption of bufferbloat-fighting strategies.

On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = <eugene.chang@alum.mit.edu<mailto:eugene.chang@alum.mit.edu>> = wrote:
The key issue is most people don=E2=80=99t = understand why latency matters. They don=E2=80=99t see it or feel it=E2=80= =99s impact.

First, we have to help people = see the symptoms of latency and how it impacts something they care = about.
- gamers care but most people may think it is = frivolous.
- musicians care but that is mostly for a = hobby.
- business should care because of productivity but = they don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact.

Second, there needs to be a =E2=80=9COMG, I = have been seeing the action of latency all this time and never knew it! = I was being shafted.=E2=80=9D Once you have this awakening, you can get = all the press you want for free.

Most of = the time when business apps are developed, =E2=80=9Cwe=E2=80=9D hide the = impact of poor performance (aka latency) or they hide from the = discussion because the developers don=E2=80=99t have a way to fix the = latency. Maybe businesses don=E2=80=99t care because any employees = affected are just considered poor performers. (In bad economic times, = the poor performers are just laid off.) For employees, if they happen to = be at a location with bad latency, they don=E2=80=99t know that latency = is hurting them. Unfair but most people don=E2=80=99t know the issue is = latency.

Talking and explaining why latency = is bad is not as effective as showing why latency is bad. Showing has to = be with something that has a person impact.

Gene
-----------------------------------
Eugene Chang
eugene.chang@alum.mit.edu = <mailto:eugene.chang@alum.mit.edu>
+1-781-799-0233 = (in Honolulu)





On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:

If you want to get = attention, you can get it for free. I can place articles with various = press if there is something interesting to say. Did this all through the = evangelism of Open Source. All we need to do is write, sign, and publish = a statement. What they actually write is less relevant if they publish a = link to our statement.

Right now I am = concerned that the Starlink latency and jitter is going to be a problem = even for remote controlling my ham station. The US Military is = interested in doing much more, which they have demonstrated, but I don't = see happening at scale without some technical work on the network. Being = able to say this isn't ready for the government's application would be = an attention-getter.

Thanks
Bruce

On Mon, Sep 26, 2022 at = 9:21 AM Dave Taht via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:
These days, if you want attention, you = gotta buy it. A 50k half page
ad in the wapo or NYT = riffing off of It's the latency, Stupid!",
signed by the = kinds of luminaries we got for the fcc wifi fight, would
go = a long way towards shifting the tide.

On = Mon, Sep 26, 2022 at 8:29 AM Dave Taht <dave.taht@gmail.com = <mailto:dave.taht@gmail.com>> wrote:

On Mon, Sep 26, 2022 at 8:20 AM = Livingood, Jason
<Jason_Livingood@comcast.com = <mailto:Jason_Livingood@comcast.com>> wrote:

The = awareness & understanding of latency & impact on QoE is nearly = unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise = reporters will just continue to focus on what they know...

That's a great idea. I have = visions of crashing the washington
correspondents dinner, = but perhaps
there is some set of gatherings journalists = regularly attend?


=EF=BB=BFOn 9/21/22, 14:35, "Starlink on = behalf of Dave Taht via Starlink" = <starlink-bounces@lists.bufferbloat.net = <mailto:starlink-bounces@lists.bufferbloat.net> on behalf of = starlink@lists.bufferbloat.net = <mailto:starlink@lists.bufferbloat.net>> wrote:
I still find it remarkable that reporters are still missing = the
meaning of the huge latencies for starlink, under = load.




--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC


--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>


--
Bruce Perens K6BP
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>



--Bruce Perens K6BP

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink


--
Bruce Perens = K6BP





= --Apple-Mail=_9160C723-0C4A-46E1-A1EF-1F2329779BA2-- --Apple-Mail=_C06B9470-63A9-453F-8FE6-D15C0D2FA286 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEERPTGiBqcibajhTSsv0/8FiYdKmAFAmM0ly4ACgkQv0/8FiYd KmD9lg//XItONLRDETOYjQ71Fkv6UTtayQXDC6D0BmvmYa5qewi6Ila3GIwJHv8W 9ei4vvYIUfYSVI5HVgMf/fkfB/sXXskTdh8N+gAy1RKs2jEfd/+4/UeuXCNkNRjb 5WvAyo8PNq+UQI9Fh3H/uByOfhD67kk48OOK9mPPxBm5i4AzZNovQ/f3jQICEXtM CRXtbYqtui+dTDXkLcBtVck5LVbDr6woHfHMPbSyX8h4EE4Zq4a9JGDCbwgzGfMC gq4AKxA77n+dr/oWevXxlm6xKD+ozO+6mb+jmV8sAeRENLAuB1chsKNtg6iVkdjQ cd/ydDu2t5h39Aarny3P0IVM85XUn1ZRoDHOUxWWhKlvL8I5VMzUsMYPBEur5YA+ XTIllz3KSKRnmNJvheui2KXoeWzAp6Am1iTi7pO3qjFwH7sVBDb0jyLE77eqNbtk gdzdvhzF5zoh2l6F8PNFWYHN1AIm+QGNI4xGhOEcGaRzcsJY+OybaOenJsxJKWzt 4MiDIN8VMihS/oUDN9WblhRECOAtyI7u1OxqfbVEwJ0BhtQzdMkn/BMRMYlIDqmp pW8oehhti+2C4hiCXv6uKfa9+eTB2Kg4VXHb4qP5k5mr/rr+qMxiuxnbMsQ/fzKR teyIkDYfYhUQs+Q3rVci3nnQ3YA/rlQlH6zkDY9jp/8KxdAdJbc= =q58h -----END PGP SIGNATURE----- --Apple-Mail=_C06B9470-63A9-453F-8FE6-D15C0D2FA286--