From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id E59923B2A4 for ; Tue, 27 Sep 2022 18:46:47 -0400 (EDT) Received: by mail-pg1-x52d.google.com with SMTP id bh13so10673393pgb.4 for ; Tue, 27 Sep 2022 15:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date; bh=bEYMWzCoTiy+T26GanM/qO85slDzxLpKUng7MOtq+og=; b=e/6vHiFhst3lQQvawQ5UeHtNk2rn/0JIl0bJ44rWKKoTePcaJeKwiA31Wv8TEIBJRF 0pI6qhu6fiajT5LfqbCdOjarqe3EpAbO3574Ona/28GJSr97bYAm1iAPskWIzk9aD7Zi vRJCIdVnfP0KmWQMJjAzguudOix6Bu2O226aY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date; bh=bEYMWzCoTiy+T26GanM/qO85slDzxLpKUng7MOtq+og=; b=5ioy8sIeBqGizLddhDt1dk8BGzNthx4YJ+wOd0ym5j1o7FDf3iMI461OCe1fg3U/9m ZEcKd1k6vqdffznZdGQt9lg+XkHqQh9TJ6bN+Ge/E/RdgzOW1HK7z8bg877w4jcVpxlF nq4LtrLjz8fhcOpyZYBETxidxKATWHLwY7ZJs7O1xRgOMbnnw1fbq6CrMyO/THNzk/eh +i95xlB7Pqkud6q96IMIWYHUBh+5BktFayFpLBma5VTucuHj+9m7LZ1+kzIFDaVBF+uL vUjV7/Qqu8o5eYAfnffjl6RIgViFsEgMUaSPX1fmp9V4vECEQJl9eZGGBRyqIqS83yQP ibeA== X-Gm-Message-State: ACrzQf1vd5kDuHL/rIp3kRm4JeG/NtM8z1EQ1ER1sH4VtbQPXyoDaNNr jbZmpOQ0ZD+f5YPlPWkeC+UNiUSwLoJaBpSn X-Google-Smtp-Source: AMsMyM5TvXShTIOnA8443T7rYaoxl7wmiyDrs7McpsAm5HMgprOo25jSLmXzu14/YQcI1957oK9NjA== X-Received: by 2002:a05:6a00:190d:b0:550:6db3:b9be with SMTP id y13-20020a056a00190d00b005506db3b9bemr31679299pfi.1.1664318806568; Tue, 27 Sep 2022 15:46:46 -0700 (PDT) Received: from smtpclient.apple (dhcp-72-253-196-65.hawaiiantel.net. [72.253.196.65]) by smtp.gmail.com with ESMTPSA id 5-20020a621605000000b00553b37c7732sm2286191pfw.105.2022.09.27.15.46.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Sep 2022 15:46:45 -0700 (PDT) From: Eugene Y Chang Message-Id: <45F57AD7-4479-4B06-AD1D-E7CF8D74D748@ieee.org> Content-Type: multipart/signed; boundary="Apple-Mail=_DF715FE9-72F8-453B-9D4E-5EA25B91C2B0"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Date: Tue, 27 Sep 2022 12:46:43 -1000 In-Reply-To: <3FD7CB2A-8DC8-47FD-82DF-D27B967CBF4F@gmx.de> Cc: Eugene Chang , Bruce Perens , Dave Taht via Starlink To: Sebastian Moeller References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> <9D97FB02-58A5-48B0-9B43-6B7BD2A24099@gmx.de> <3FD7CB2A-8DC8-47FD-82DF-D27B967CBF4F@gmx.de> X-Mailer: Apple Mail (2.3696.120.41.1.1) Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2022 22:46:49 -0000 --Apple-Mail=_DF715FE9-72F8-453B-9D4E-5EA25B91C2B0 Content-Type: multipart/alternative; boundary="Apple-Mail=_466B73C2-A5C2-451B-9954-5D808CA2B2A6" --Apple-Mail=_466B73C2-A5C2-451B-9954-5D808CA2B2A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Sebastian, Good to know. I haven=E2=80=99t had time to follow all the work going on = globally. I was only commenting on how the ISP support team behaves. How do we bring this better attitude to the US. Sadly the FCC seems stuck accommodating the big ISPs. Of course, we want the FCC, the ISPs, and the networking world to go = beyond speedtest. Gene ---------------------------------------------- Eugene Chang IEEE Senior Life Member eugene.chang@ieee.org 781-799-0233 (in Honolulu) > On Sep 26, 2022, at 9:09 PM, Sebastian Moeller = wrote: >=20 > Hi Gene, >=20 >=20 >> On Sep 27, 2022, at 05:50, Eugene Y Chang = wrote: >>=20 >> Of course=E2=80=A6. But the ISP=E2=80=99s maxim is "don=E2=80=99t = believe the customer=E2=80=99s speedtest numbers if it is not with their = host". >=20 > [SM] In an act of reasonable regulation did the EU give national = regulatory agencies the right to define and set-up national = "speed-tests" (located outside of the access ISPs networks) which ISPs = effectively need to accept. In Germany the local NRA = (Bundes-Netz-Agentur, BNetzA) created a speedtest application against = servers operated on their behalf and will, if a customers demonstrates = the ISP falling short of the contracted rates (according to a somewhat = complicated definition and measurement rules). convince ISPs to follow = the law and either release customers from their contracts immediately or = lower the price in relation to the under-fulfillment. (Unfortunately all = related official web pages are in German only). > This put a hold on the practice of gaming speedtests, like = DOCSIS ISPs measuring an in-segment speedtest server which conveniently = hides if a segment's uplink is congested... Not all ISPs gamed their = speedtests, and even the in-segment speedtest can be justified for some = measurements (e.g. when trying to figure out whether congestion is = in-segment or out-of-segment), but the temptation must have been large = to not set-up the most objective spedtest. (We have a saying along the = lines of "making a goat your gardener" which generally is considered a = sub-optimal approach). >=20 > Regards > Sebastian >=20 >=20 >>=20 >>=20 >> Gene >> ---------------------------------------------- >> Eugene Chang >> IEEE Senior Life Member >> eugene.chang@ieee.org >> 781-799-0233 (in Honolulu) >>=20 >>=20 >>=20 >>> On Sep 26, 2022, at 11:44 AM, Bruce Perens wrote: >>>=20 >>> That's a good maxim: Don't believe a speed test that is hosted by = your own ISP. >>>=20 >>> On Mon, Sep 26, 2022 at 2:36 PM Eugene Y Chang via Starlink = wrote: >>> Thank you for the dialog,. >>> This discussion with regards to Starlink is interesting as it = confirms my guesses about the gap between Starlinks overly simplified, = over optimistic marketing and the reality as they acquire subscribers. >>>=20 >>> I am actually interested in a more perverse issue. I am seeing = latency and bufferbloat as a consequence from significant under = provisioning. It doesn=E2=80=99t matter that the ISP is selling a fiber = drop, if (parts) of their network is under provisioned. Two end points = can be less than 5 mile apart and realize 120+ ms latency. Two Labor = Days ago (a holiday) the max latency was 230+ ms. The pattern I see = suggest digital redlining. The older communities appear to have much = more severe under provisioning. >>>=20 >>> Another observation. Running speedtest appears to go from the edge = of the network by layer 2 to the speedtest host operated by the ISP. = Yup, bypasses the (suspected overloaded) routers. >>>=20 >>> Anyway, just observing. >>>=20 >>> Gene >>> ---------------------------------------------- >>> Eugene Chang >>> IEEE Senior Life Member >>> eugene.chang@ieee.org >>> 781-799-0233 (in Honolulu) >>>=20 >>>=20 >>>=20 >>>> On Sep 26, 2022, at 11:20 AM, Sebastian Moeller = wrote: >>>>=20 >>>> Hi Gene, >>>>=20 >>>>=20 >>>>> On Sep 26, 2022, at 23:10, Eugene Y Chang = wrote: >>>>>=20 >>>>> Comments inline below. >>>>>=20 >>>>> Gene >>>>> ---------------------------------------------- >>>>> Eugene Chang >>>>> IEEE Senior Life Member >>>>> eugene.chang@ieee.org >>>>> 781-799-0233 (in Honolulu) >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>>>>>=20 >>>>>> Hi Eugene, >>>>>>=20 >>>>>>=20 >>>>>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = wrote: >>>>>>>=20 >>>>>>> Ok, we are getting into the details. I agree. >>>>>>>=20 >>>>>>> Every node in the path has to implement this to be effective. >>>>>>=20 >>>>>> Amazingly the biggest bang for the buck is gotten by fixing = those nodes that actually contain a network path's bottleneck. Often = these are pretty stable. So yes for fully guaranteed service quality all = nodes would need to participate, but for improving things noticeably it = is sufficient to improve the usual bottlenecks, e.g. for many internet = access links the home gateway is a decent point to implement better = buffer management. (In short the problem are over-sized and = under-managed buffers, and one of the best solution is better/smarter = buffer management). >>>>>>=20 >>>>>=20 >>>>> This is not completely true. >>>>=20 >>>> [SM] You are likely right, trying to summarize things leads to = partially incorrect generalizations. >>>>=20 >>>>=20 >>>>> Say the bottleneck is at node N. During the period of congestion, = the upstream node N-1 will have to buffer. When node N recovers, the = bufferbloat at N-1 will be blocking until the bufferbloat drains. Etc. = etc. Making node N better will reduce the extent of the backup at N-1, = but N-1 should implement the better code. >>>>=20 >>>> [SM] It is the node that builds up the queue that profits most = from better queue management.... (again I generalize, the node with the = queue itself probably does not care all that much, but the endpoints = will profit if the queue experiencing node deals with that queue more = gracefully). >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>>> In fact, every node in the path has to have the same = prioritization or the scheme becomes ineffective. >>>>>>=20 >>>>>> Yes and no, one of the clearest winners has been flow queueing, = IMHO not because it is the most optimal capacity sharing scheme, but = because it is the least pessimal scheme, allowing all (or none) flows = forward progress. You can interpret that as a scheme in which flows = below their capacity share are prioritized, but I am not sure that is = the best way to look at these things. >>>>>=20 >>>>> The hardest part is getting competing ISPs to implement and = coordinate. >>>>=20 >>>> [SM] Yes, but it turned out even with non-cooperating ISPs there = is a lot end-users can do unilaterally on their side to improve both = ingress and egress congestion. Admittedly especially ingress congestion = would be even better handled with cooperation of the ISP. >>>>=20 >>>>> Bufferbloat and handoff between ISPs will be hard. The only way to = fix this is to get the unwashed public to care. Then they can say =E2=80=9C= we don=E2=80=99t care about the technical issues, just fix it.=E2=80=9D = Until then =E2=80=A6.. >>>>=20 >>>> [SM] Well we do this one home network at a time (not because = that is efficient or ideal, but simply because it is possible). Maybe, = if you have not done so already try OpenWrt with sqm-scripts (and maybe = cake-autorate in addition) on your home internet access link for say a = week and let us know ih/how your experience changed? >>>>=20 >>>> Regards >>>> Sebastian >>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>>=20 >>>>>> Regards >>>>>> Sebastian >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>> Gene >>>>>>> ---------------------------------------------- >>>>>>> Eugene Chang >>>>>>> IEEE Senior Life Member >>>>>>> eugene.chang@ieee.org >>>>>>> 781-799-0233 (in Honolulu) >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>> On Sep 26, 2022, at 10:48 AM, David Lang wrote: >>>>>>>>=20 >>>>>>>> software updates can do far more than just improve recovery. >>>>>>>>=20 >>>>>>>> In practice, large data transfers are less sensitive to latency = than smaller data transfers (i.e. downloading a CD image vs a video = conference), software can ensure better fairness in preventing a bulk = transfer from hurting the more latency sensitive transfers. >>>>>>>>=20 >>>>>>>> (the example below is not completely accurate, but I think it = gets the point across) >>>>>>>>=20 >>>>>>>> When buffers become excessivly large, you have the situation = where a video call is going to generate a small amount of data at a = regular interval, but a bulk data transfer is able to dump a huge amount = of data into the buffer instantly. >>>>>>>>=20 >>>>>>>> If you just do FIFO, then you get a small chunk of video call, = then several seconds worth of CD transfer, followed by the next small = chunk of the video call. >>>>>>>>=20 >>>>>>>> But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>>>>>=20 >>>>>>>> The one thing that Cake needs to work really well is to be able = to know what the data rate available is. With Starlink, this changes = frequently and cake integrated into the starlink dish/router software = would be far better than anything that can be done externally as the = rate changes can be fed directly into the settings (currently they are = only indirectly detected) >>>>>>>>=20 >>>>>>>> David Lang >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>>>>>=20 >>>>>>>>> You already know this. Bufferbloat is a symptom and not the = cause. Bufferbloat grows when there are (1) periods of low or no = bandwidth or (2) periods of insufficient bandwidth (aka network = congestion). >>>>>>>>>=20 >>>>>>>>> If I understand this correctly, just a software update cannot = make bufferbloat go away. It might improve the speed of recovery (e.g. = throw away all time sensitive UDP messages). >>>>>>>>>=20 >>>>>>>>> Gene >>>>>>>>> ---------------------------------------------- >>>>>>>>> Eugene Chang >>>>>>>>> IEEE Senior Life Member >>>>>>>>> eugene.chang@ieee.org >>>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>>>>>=20 >>>>>>>>>> Please help to explain. Here's a draft to start with: >>>>>>>>>>=20 >>>>>>>>>> Starlink Performance Not Sufficient for Military = Applications, Say Scientists >>>>>>>>>>=20 >>>>>>>>>> The problem is not availability: Starlink works where nothing = but another satellite network would. It's not bandwidth, although others = have questions about sustaining bandwidth as the customer base grows. = It's latency and jitter. As load increases, latency, the time it takes = for a packet to get through, increases more than it should. The = scientists who have fought bufferbloat, a major cause of latency on the = internet, know why. SpaceX needs to upgrade their system to use the = scientist's Open Source modifications to Linux to fight bufferbloat, and = thus reduce latency. This is mostly just using a newer version, but = there are some tunable parameters. Jitter is a change in the speed of = getting a packet through the network during a connection, which is = inevitable in satellite networks, but will be improved by making use of = the bufferbloat-fighting software, and probably with the addition of = more satellites. >>>>>>>>>>=20 >>>>>>>>>> We've done all of the work, SpaceX just needs to adopt it by = upgrading their software, said scientist Dave Taht. Jim Gettys, Taht's = collaborator and creator of the X Window System, chimed in: >>>>>>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's = latency and jitter make it inadequate to remote-control my ham radio = station. But the military is experimenting with remote-control of = vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>>>>>=20 >>>>>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>>>>>> The key issue is most people don=E2=80=99t understand why = latency matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>>>>>=20 >>>>>>>>>> First, we have to help people see the symptoms of latency and = how it impacts something they care about. >>>>>>>>>> - gamers care but most people may think it is frivolous. >>>>>>>>>> - musicians care but that is mostly for a hobby. >>>>>>>>>> - business should care because of productivity but they = don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>>>>>=20 >>>>>>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing = the action of latency all this time and never knew it! I was being = shafted.=E2=80=9D Once you have this awakening, you can get all the = press you want for free. >>>>>>>>>>=20 >>>>>>>>>> Most of the time when business apps are developed, =E2=80=9Cwe=E2= =80=9D hide the impact of poor performance (aka latency) or they hide = from the discussion because the developers don=E2=80=99t have a way to = fix the latency. Maybe businesses don=E2=80=99t care because any = employees affected are just considered poor performers. (In bad economic = times, the poor performers are just laid off.) For employees, if they = happen to be at a location with bad latency, they don=E2=80=99t know = that latency is hurting them. Unfair but most people don=E2=80=99t know = the issue is latency. >>>>>>>>>>=20 >>>>>>>>>> Talking and explaining why latency is bad is not as effective = as showing why latency is bad. Showing has to be with something that has = a person impact. >>>>>>>>>>=20 >>>>>>>>>> Gene >>>>>>>>>> ----------------------------------- >>>>>>>>>> Eugene Chang >>>>>>>>>> eugene.chang@alum.mit.edu >>>>>>>>>> +1-781-799-0233 (in Honolulu) >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>>>>>=20 >>>>>>>>>>> If you want to get attention, you can get it for free. I can = place articles with various press if there is something interesting to = say. Did this all through the evangelism of Open Source. All we need to = do is write, sign, and publish a statement. What they actually write is = less relevant if they publish a link to our statement. >>>>>>>>>>>=20 >>>>>>>>>>> Right now I am concerned that the Starlink latency and = jitter is going to be a problem even for remote controlling my ham = station. The US Military is interested in doing much more, which they = have demonstrated, but I don't see happening at scale without some = technical work on the network. Being able to say this isn't ready for = the government's application would be an attention-getter. >>>>>>>>>>>=20 >>>>>>>>>>> Thanks >>>>>>>>>>>=20 >>>>>>>>>>> Bruce >>>>>>>>>>>=20 >>>>>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>>>>>> These days, if you want attention, you gotta buy it. A 50k = half page >>>>>>>>>>> ad in the wapo or NYT riffing off of It's the latency, = Stupid!", >>>>>>>>>>> signed by the kinds of luminaries we got for the fcc wifi = fight, would >>>>>>>>>>> go a long way towards shifting the tide. >>>>>>>>>>>=20 >>>>>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht = > wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>>>>>> > wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> The awareness & understanding of latency & impact on QoE = is nearly unknown among reporters. IMO maybe there should be some kind = of background briefings for reporters - maybe like a simple YouTube = video explainer that is short & high level & visual? Otherwise reporters = will just continue to focus on what they know... >>>>>>>>>>>>=20 >>>>>>>>>>>> That's a great idea. I have visions of crashing the = washington >>>>>>>>>>>> correspondents dinner, but perhaps >>>>>>>>>>>> there is some set of gatherings journalists regularly = attend? >>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave = Taht via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> I still find it remarkable that reporters are still = missing the >>>>>>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> -- >>>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> -- >>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Starlink mailing list >>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> -- >>>>>>>>>>> Bruce Perens K6BP >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Starlink mailing list >>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> -- >>>>>>>>>> Bruce Perens K6BP >>>>>>>=20 >>>>>>> _______________________________________________ >>>>>>> Starlink mailing list >>>>>>> Starlink@lists.bufferbloat.net >>>>>>> https://lists.bufferbloat.net/listinfo/starlink >>>=20 >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/starlink >>>=20 >>>=20 >>> -- >>> Bruce Perens K6BP >>=20 >=20 --Apple-Mail=_466B73C2-A5C2-451B-9954-5D808CA2B2A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Sebastian, 
Good to know. I haven=E2=80=99t= had time to follow all the work going on globally.
I= was only commenting on how the ISP support team behaves.
How do we bring this better attitude to the US.

Sadly the FCC seems = stuck accommodating the big ISPs.
Of course, we = want the FCC, the ISPs, and the networking world to go beyond = speedtest.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life = Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at 9:09 PM, Sebastian Moeller <moeller0@gmx.de> = wrote:

Hi Gene,


On Sep 27, 2022, at = 05:50, Eugene Y Chang <eugene.chang@ieee.org> wrote:

Of course=E2=80=A6. But the ISP=E2=80=99s maxim is "don=E2=80=99= t believe the customer=E2=80=99s speedtest numbers if it is not with = their host".

[SM] In = an act of reasonable regulation did the EU give national regulatory = agencies the right to define and set-up national "speed-tests" (located = outside of the access ISPs networks) which ISPs effectively need to = accept. In Germany the local NRA (Bundes-Netz-Agentur, BNetzA) created a = speedtest application against servers operated on their behalf and will, = if a customers demonstrates the ISP falling short of the contracted = rates (according to a somewhat complicated definition and measurement = rules). convince ISPs to follow the law and either release customers = from their contracts immediately or lower the price in relation to the = under-fulfillment. (Unfortunately all related official web pages are in = German only).
This put a hold on the practice = of gaming speedtests, like DOCSIS ISPs measuring an in-segment speedtest = server which conveniently hides if a segment's uplink is congested... = Not all ISPs gamed their speedtests, and even the in-segment speedtest = can be justified for some measurements (e.g. when trying to figure out = whether congestion is in-segment or out-of-segment), but the temptation = must have been large to not set-up the most objective spedtest. (We have = a saying along the lines of "making a goat your gardener" which = generally is considered a sub-optimal approach).

Regards
Sebastian




Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:44 AM, Bruce Perens <bruce@perens.com> wrote:

That's a good maxim: Don't believe a speed test that is = hosted by your own ISP.

On Mon, Sep 26, = 2022 at 2:36 PM Eugene Y Chang via Starlink = <starlink@lists.bufferbloat.net> wrote:
Thank you = for the dialog,.
This discussion with regards to Starlink = is interesting as it confirms my guesses about the gap between Starlinks = overly simplified, over optimistic marketing and the reality as they = acquire subscribers.

I am actually = interested in a more perverse issue. I am seeing latency and bufferbloat = as a consequence from significant under provisioning. It doesn=E2=80=99t = matter that the ISP is selling a fiber drop, if (parts) of their network = is under provisioned. Two end points can be less than 5 mile apart and = realize 120+ ms latency. Two Labor Days ago (a holiday) the max latency = was 230+ ms. The pattern I see suggest digital redlining. The older = communities appear to have much more severe under provisioning.

Another observation. Running speedtest appears = to go from the edge of the network by layer 2 to the speedtest host = operated by the ISP. Yup, bypasses the (suspected overloaded) = routers.

Anyway, just observing.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:20 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Gene,


On Sep 26, 2022, at = 23:10, Eugene Y Chang <eugene.chang@ieee.org> wrote:

Comments inline below.

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:01 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi Eugene,


On Sep 26, 2022, at = 22:54, Eugene Y Chang via Starlink = <starlink@lists.bufferbloat.net> wrote:

Ok, we are getting into the details. I agree.
Every node in the path has to implement this to be = effective.

Amazingly = the biggest bang for the buck is gotten by fixing those nodes that = actually contain a network path's bottleneck. Often these are pretty = stable. So yes for fully guaranteed service quality all nodes would need = to participate, but for improving things noticeably it is sufficient to = improve the usual bottlenecks, e.g. for many internet access links the = home gateway is a decent point to implement better buffer management. = (In short the problem are over-sized and under-managed buffers, and one = of the best solution is better/smarter buffer management).


This is not = completely true.

[SM] You = are likely right, trying to summarize things leads to partially = incorrect generalizations.


Say the bottleneck is at = node N. During the period of congestion, the upstream node N-1 will have = to buffer. When node N recovers, the bufferbloat at N-1 will be blocking = until the bufferbloat drains. Etc. etc.  Making node N better will = reduce the extent of the backup at N-1, but N-1 should implement the = better code.

[SM] It = is the node that builds up the queue that profits most from better queue = management.... (again I generalize, the node with the queue itself = probably does not care all that much, but the endpoints will profit if = the queue experiencing node deals with that queue more gracefully).





In fact, = every node in the path has to have the same prioritization or the scheme = becomes ineffective.

Yes and = no, one of the clearest winners has been flow queueing, IMHO not because = it is the most optimal capacity sharing scheme, but because it is the = least pessimal scheme, allowing all (or none) flows forward progress. = You can interpret that as a scheme in which flows below their capacity = share are prioritized, but I am not sure that is the best way to look at = these things.

The hardest part = is getting competing ISPs to implement and coordinate.

[SM] Yes, but it turned out even = with non-cooperating ISPs there is a lot end-users can do unilaterally = on their side to improve both ingress and egress congestion. Admittedly = especially ingress congestion would be even better handled with = cooperation of the ISP.

Bufferbloat and handoff between ISPs will be = hard. The only way to fix this is to get the unwashed public to care. = Then they can say =E2=80=9Cwe don=E2=80=99t care about the technical = issues, just fix it.=E2=80=9D Until then =E2=80=A6..

[SM] Well we do this one home = network at a time (not because that is efficient or ideal, but simply = because it is possible). Maybe, if you have not done so already try = OpenWrt with sqm-scripts (and maybe cake-autorate in addition) on your = home internet access link for say a week and let us know ih/how your = experience changed?

Regards
= Sebastian






Regards
= Sebastian



Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:48 AM, David Lang <david@lang.hm> wrote:

software updates can do far more than just improve = recovery.

In practice, large data transfers = are less sensitive to latency than smaller data transfers (i.e. = downloading a CD image vs a video conference), software can ensure = better fairness in preventing a bulk transfer from hurting the more = latency sensitive transfers.

(the example = below is not completely accurate, but I think it gets the point = across)

When buffers become excessivly = large, you have the situation where a video call is going to generate a = small amount of data at a regular interval, but a bulk data transfer is = able to dump a huge amount of data into the buffer instantly.

If you just do FIFO, then you get a small = chunk of video call, then several seconds worth of CD transfer, followed = by the next small chunk of the video call.

But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic.

The one thing that Cake needs to work really well is to be = able to know what the data rate available is. With Starlink, this = changes frequently and cake integrated into the starlink dish/router = software would be far better than anything that can be done externally = as the rate changes can be fed directly into the settings (currently = they are only indirectly detected)

David = Lang


On Mon, 26 Sep 2022, = Eugene Y Chang via Starlink wrote:

You already know this. = Bufferbloat is a symptom and not the cause. Bufferbloat grows when there = are (1) periods of low or no bandwidth or (2) periods of insufficient = bandwidth (aka network congestion).

If I = understand this correctly, just a software update cannot make = bufferbloat go away. It might improve the speed of recovery (e.g. throw = away all time sensitive UDP messages).

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:04 AM, Bruce Perens <bruce@perens.com> wrote:

Please help to explain. Here's a draft to start with:

Starlink Performance Not Sufficient for = Military Applications, Say Scientists

The = problem is not availability: Starlink works where nothing but another = satellite network would. It's not bandwidth, although others have = questions about sustaining bandwidth as the customer base grows. It's = latency and jitter. As load increases, latency, the time it takes for a = packet to get through, increases more than it should. The scientists who = have fought bufferbloat, a major cause of latency on the internet, know = why. SpaceX needs to upgrade their system to use the scientist's Open = Source modifications to Linux to fight bufferbloat, and thus reduce = latency. This is mostly just using a newer version, but there are some = tunable parameters. Jitter is a change in the speed of getting a packet = through the network during a connection, which is inevitable in = satellite networks, but will be improved by making use of the = bufferbloat-fighting software, and probably with the addition of more = satellites.

We've done all of the work, = SpaceX just needs to adopt it by upgrading their software, said = scientist Dave Taht. Jim Gettys, Taht's collaborator and creator of the = X Window System, chimed in: <fill in here please>
Open= Source luminary Bruce Perens said: sometimes Starlink's latency and = jitter make it inadequate to remote-control my ham radio station. But = the military is experimenting with remote-control of vehicles on the = battlefield and other applications that can be demonstrated, but won't = happen at scale without adoption of bufferbloat-fighting strategies.

On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = <eugene.chang@alum.mit.edu<mailto:eugene.chang@alum.mit.edu>> = wrote:
The key issue is most people don=E2=80=99t = understand why latency matters. They don=E2=80=99t see it or feel it=E2=80= =99s impact.

First, we have to help people = see the symptoms of latency and how it impacts something they care = about.
- gamers care but most people may think it is = frivolous.
- musicians care but that is mostly for a = hobby.
- business should care because of productivity but = they don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact.

Second, there needs to be a =E2=80=9COMG, I = have been seeing the action of latency all this time and never knew it! = I was being shafted.=E2=80=9D Once you have this awakening, you can get = all the press you want for free.

Most of = the time when business apps are developed, =E2=80=9Cwe=E2=80=9D hide the = impact of poor performance (aka latency) or they hide from the = discussion because the developers don=E2=80=99t have a way to fix the = latency. Maybe businesses don=E2=80=99t care because any employees = affected are just considered poor performers. (In bad economic times, = the poor performers are just laid off.) For employees, if they happen to = be at a location with bad latency, they don=E2=80=99t know that latency = is hurting them. Unfair but most people don=E2=80=99t know the issue is = latency.

Talking and explaining why latency = is bad is not as effective as showing why latency is bad. Showing has to = be with something that has a person impact.

Gene
-----------------------------------
Eugene Chang
eugene.chang@alum.mit.edu = <mailto:eugene.chang@alum.mit.edu>
+1-781-799-0233 = (in Honolulu)





On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:

If you want to get = attention, you can get it for free. I can place articles with various = press if there is something interesting to say. Did this all through the = evangelism of Open Source. All we need to do is write, sign, and publish = a statement. What they actually write is less relevant if they publish a = link to our statement.

Right now I am = concerned that the Starlink latency and jitter is going to be a problem = even for remote controlling my ham station. The US Military is = interested in doing much more, which they have demonstrated, but I don't = see happening at scale without some technical work on the network. Being = able to say this isn't ready for the government's application would be = an attention-getter.

Thanks

Bruce

On Mon, = Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:
These days, if you want attention, you = gotta buy it. A 50k half page
ad in the wapo or NYT = riffing off of It's the latency, Stupid!",
signed by the = kinds of luminaries we got for the fcc wifi fight, would
go = a long way towards shifting the tide.

On = Mon, Sep 26, 2022 at 8:29 AM Dave Taht <dave.taht@gmail.com = <mailto:dave.taht@gmail.com>> wrote:

On Mon, Sep 26, 2022 at 8:20 AM = Livingood, Jason
<Jason_Livingood@comcast.com = <mailto:Jason_Livingood@comcast.com>> wrote:

The = awareness & understanding of latency & impact on QoE is nearly = unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise = reporters will just continue to focus on what they know...

That's a great idea. I have = visions of crashing the washington
correspondents dinner, = but perhaps
there is some set of gatherings journalists = regularly attend?


=EF=BB=BFOn 9/21/22, 14:35, "Starlink on = behalf of Dave Taht via Starlink" = <starlink-bounces@lists.bufferbloat.net = <mailto:starlink-bounces@lists.bufferbloat.net> on behalf of = starlink@lists.bufferbloat.net = <mailto:starlink@lists.bufferbloat.net>> wrote:
I still find it remarkable that reporters are still missing = the
meaning of the huge latencies for starlink, under = load.




--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC


--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>


--
Bruce Perens K6BP
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>



--Bruce Perens K6BP

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink


--
Bruce Perens = K6BP



= --Apple-Mail=_466B73C2-A5C2-451B-9954-5D808CA2B2A6-- --Apple-Mail=_DF715FE9-72F8-453B-9D4E-5EA25B91C2B0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEERPTGiBqcibajhTSsv0/8FiYdKmAFAmMzfVMACgkQv0/8FiYd KmDe6g//eYBnvma5SiHyIzjEy7ESL/Lj07fM2fLoZDkm/q2BL0sGNtha6DdyTgOT JBRt1eOhfnTyqx2o00L6WtHKAmyzeNY9hJseHx7ZojKjD8KQSkUEaAheGk7QAOV5 NR82wv8YWxgomyaa6dX0qHVuFiXiYGcg9dKavdzTRGSQmwNPiORu2cruHI0xs90/ UFlMFWBRoU9HZ6VU7OQ82ggnZHQ3eGxsP2jmCGOG6HdI+LvOFZ37RVZoHXIm9Jm0 GJKZx0UxzGyqhqU9u0X8fSUmXfAG560GOahopGzQIUmtr/nGCLTBNqIz2jXzkKUM WX1AQNjR+RRJ9ndyXyldMGJr2xO9NKnuHx7TRBrBEBWa9szpwoXwEff/0pIoMgJA ktJnINA3Trcw8rZZPI8ak12FoVm7ksBgKkkeiory+/Og1tlbij5DxTYSapn+cwTP IyxrLh7DgJnhjWziOmNqrcARtQLdTIQK+JwNqkeu8ngNe5WZGmD9aFFtn25ICF4w atC9xpg1PkZsHlayLgY4xbm4cJDkuTp+9LLX9EHiyPn7QNM+qoojx4+iPJjtZKf8 gRLYMjE3sG9Fi3FJ/pveKO6Pn0a86VyMBP2xnuDUpXgNIWddF6B0LthWApQx4OCJ TlLM6Abf6/f817sg8HnRqzwRrCZ9xUD/f7ffgRAWCxVQ47Loudc= =tGhQ -----END PGP SIGNATURE----- --Apple-Mail=_DF715FE9-72F8-453B-9D4E-5EA25B91C2B0--