From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CE0983B2A4 for ; Tue, 27 Sep 2022 20:20:38 -0400 (EDT) Received: by mail-pf1-x42d.google.com with SMTP id d10so9977245pfh.6 for ; Tue, 27 Sep 2022 17:20:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date; bh=Kna0FeytUxZb7zeC1Cikctn6Nj8VDoJjoXbARhNOFsg=; b=MVfqFC6zpZik8kgXRFrHsLGuJ7yrkrrwZeDs7G3qbkPD8uP/SMKgmFuE9IqzxCgpgr QKoGcXxCKTZqdq8tvB9paEm8UL7qfpxSt+iXyxDAwTe2fJkVlf1DP/j03+kIEL20ZxPX lkh0aY88FRB7ZpRtJgOq8b/11w0RHV/gNOjdY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date; bh=Kna0FeytUxZb7zeC1Cikctn6Nj8VDoJjoXbARhNOFsg=; b=sag689o3295YUHOkswPdtGNI1Qm4enoF4zpx0s47VJtI3UBGEec/jVtlzZk3r0/Efn pfjHBCRzsoFkEYPrxL4np1RFjjnq763TNCX52XcryI7CxAa4e3aN0Wb/OBEGeiYKxmk1 HVNwOqN3yZNuyKQkfpeGsMYRZ8N5ZxTv1AjxuP92YKVEKdVHZZIFcyskI6vx6D8XxlqJ BHgJswPBGqPCdpVf9UhR2ypAEaON1dtEFvmfDk/7ygwfbxaUPAjOK6RzpDutTmgn7mUn KZRihGx5m9mp7E44qJ43wBRwFf8/gSpqj2WHYdCirsikLrJfL/BfKiQuuB4Xab6ldsiz jQkw== X-Gm-Message-State: ACrzQf23UVVRvftmkqXPRj3YHTNo2qYwseBjsL0USnb5zhVrwD1FhkJl ZlY7z2Jn9kVRkyv2nYOLdUhAVQ== X-Google-Smtp-Source: AMsMyM4XL273f99Dks+rzucTABNy3b9fGJaEI0utvxqlymYlvD3Hvu9v7ruNM2uYsSPyC+oKrKraxg== X-Received: by 2002:a63:a51e:0:b0:439:857:2758 with SMTP id n30-20020a63a51e000000b0043908572758mr26404867pgf.105.1664324437054; Tue, 27 Sep 2022 17:20:37 -0700 (PDT) Received: from smtpclient.apple (dhcp-72-253-196-65.hawaiiantel.net. [72.253.196.65]) by smtp.gmail.com with ESMTPSA id e11-20020a170902784b00b0017693bb573bsm2124616pln.219.2022.09.27.17.20.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Sep 2022 17:20:35 -0700 (PDT) From: Eugene Y Chang Message-Id: <9E4F2974-A831-4ABA-8467-69607C5F173E@ieee.org> Content-Type: multipart/signed; boundary="Apple-Mail=_1EE90DCD-810D-4766-B0F3-D932A3CC32BB"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Date: Tue, 27 Sep 2022 14:20:33 -1000 In-Reply-To: Cc: Eugene Chang , Sebastian Moeller , Dave Taht via Starlink To: Dave Taht References: <060F7695-D48E-413C-9501-54ECC651ABEB@cable.comcast.com> <07C46DD5-7359-410E-8820-82B319944618@alum.mit.edu> <39E525B8-D356-4F76-82FF-F1F0B3183908@ieee.org> <498p2p23-on1q-op89-p518-1874r3r6rpo@ynat.uz> <8DC6E5EE-2B46-4815-A909-E326507E95B1@ieee.org> <2086C010-B91E-450A-A77F-D0840BC5FCC1@gmx.de> <8E6A87E1-8424-44BE-B582-DF3C8424F584@ieee.org> X-Mailer: Apple Mail (2.3696.120.41.1.1) Subject: Re: [Starlink] It's still the starlink latency... X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Sep 2022 00:20:39 -0000 --Apple-Mail=_1EE90DCD-810D-4766-B0F3-D932A3CC32BB Content-Type: multipart/alternative; boundary="Apple-Mail=_E9A13AF0-380A-45F0-AA55-77D9043DEF70" --Apple-Mail=_E9A13AF0-380A-45F0-AA55-77D9043DEF70 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Yup, all true. I got lots of things to rant about too. Now, Mr. ISP, you signed me up for 1Gbps service. You told me it would = fix my performance problem. You bundled a kit with the Wi-Fi gateway = (router). And you give me bad support if I provide my own router. So why = should you have to know the stuff that Dave is telling me. Yeah, and when I first complained about sub-300Mbps service on my 1Gbps = service, after 3 months of helping the ISP support escalate the problem, = they eventually reported =E2=80=9Coh, we didn=E2=80=99t notice the OLT = was overloaded=E2=80=9D. Eh, don=E2=80=99t they have equipment = monitoring and all that basic stuff. Then they reported it will take 6 = months to get a new OLT shelf added. =E2=80=A6. So lame. So sad. = Deliberately incompetent because it saves them money. Gene ---------------------------------------------- Eugene Chang IEEE Senior Life Member eugene.chang@ieee.org 781-799-0233 (in Honolulu) > On Sep 27, 2022, at 3:55 AM, Dave Taht wrote: >=20 > On Mon, Sep 26, 2022 at 11:36 PM Sebastian Moeller via Starlink > > wrote: >=20 >> Worst, I waited 9 months for them to resolve why I was only getting = 300Mbps on my 1Gbps service (oops, sorry my 700Mbps service). >=20 > Nothing I've seen to date below $400 dollars can actually push a gbit > in both directions at the same time, with > NAT, firewalling, etc. A lot of gear can't even push a gbit in one = direction. >=20 > The long accepted fallacy that a gigabit "router", is just a gigabit > switch, bothers me, but it is just one piece of a long line of > orwellian doublethink about how the net works but is so pervasive we > ended up publishing: >=20 > = https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-rou= ter-read-this-first/90305 = >=20 > I've seen an ostensibly serious proposal that budgeted about $4k/sub > for a symmetric gig fiber rollout that budgeted... > wait for it... $75 dollars for the router, and other proposals that > wanted to punt the firewalling etc to a more centralized server, and > just put a switch at the sub. >=20 >> [SM] I guess mass market internet access is a low margin business, = where expert debugging time is precious. I hope they at least gave you a = rebate for that time. >=20 > Why is it so few take packet captures anymore? (I think I will rant on > that at more length elsewhere) >=20 >>=20 >> Regards >> Sebastian >>=20 >>=20 >>=20 >>>=20 >>>=20 >>> Gene >>> ---------------------------------------------- >>> Eugene Chang >>> IEEE Senior Life Member >>> eugene.chang@ieee.org >>> 781-799-0233 (in Honolulu) >>>=20 >>>=20 >>>=20 >>>> On Sep 26, 2022, at 11:29 AM, Sebastian Moeller = wrote: >>>>=20 >>>> Hi David, >>>>=20 >>>>> On Sep 26, 2022, at 23:22, David Lang wrote: >>>>>=20 >>>>> On Mon, 26 Sep 2022, Eugene Y Chang wrote: >>>>>=20 >>>>>>> On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = wrote: >>>>>>>=20 >>>>>>> Hi Eugene, >>>>>>>=20 >>>>>>>=20 >>>>>>>> On Sep 26, 2022, at 22:54, Eugene Y Chang via Starlink = > = wrote: >>>>>>>>=20 >>>>>>>> Ok, we are getting into the details. I agree. >>>>>>>>=20 >>>>>>>> Every node in the path has to implement this to be effective. >>>>>>>=20 >>>>>>> Amazingly the biggest bang for the buck is gotten by fixing = those nodes that actually contain a network path's bottleneck. Often = these are pretty stable. So yes for fully guaranteed service quality all = nodes would need to participate, but for improving things noticeably it = is sufficient to improve the usual bottlenecks, e.g. for many internet = access links the home gateway is a decent point to implement better = buffer management. (In short the problem are over-sized and = under-managed buffers, and one of the best solution is better/smarter = buffer management). >>>>>>>=20 >>>>>>=20 >>>>>> This is not completely true. Say the bottleneck is at node N. = During the period of congestion, the upstream node N-1 will have to = buffer. When node N recovers, the bufferbloat at N-1 will be blocking = until the bufferbloat drains. Etc. etc. Making node N better will = reduce the extent of the backup at N-1, but N-1 should implement the = better code. >>>>>=20 >>>>> only if node N and node N-1 handle the same traffic with the same = link speeds. In practice this is almost never the case. >>>>=20 >>>> [SM] I note that typically for ingress shaping a = post-true-bottleneck shaper will not work unless we create an artificial = bottleneck by shaping the traffic to below true bottleneck (thereby = creating a new true but artificial bottleneck so the queue develops at a = point where we can control it). >>>> Also if the difference between "true structural" and artificial = bottleneck is small in comparison to the traffic inrush we can see = "traffic back-spill" into the typically oversized and under-managed = upstream buffers, but for reasonably well behaved that happens = relatively rarely. Rarely enough that ingress traffic shaping noticeably = improves latency-under-load in spite of not beeing a guranteed solution. >>>>=20 >>>>=20 >>>>> Until you get to gigabit last-mile links, the last mile is almost = always the bottleneck from both sides, so implementing cake on the home = router makes a huge improvement (and if you can get it on the last-mile = ISP router, even better). Once you get into the Internet fabric, = bottlenecks are fairly rare, they do happen, but ISPs carefully watch = for those and add additional paths and/or increase bandwith to avoid = them. >>>>=20 >>>> [SM] Well, sometimes such links are congested too for economic = reasons... >>>>=20 >>>> Regards >>>> Sebastian >>>>=20 >>>>=20 >>>>>=20 >>>>> David Lang >>>>>=20 >>>>>>>=20 >>>>>>>> In fact, every node in the path has to have the same = prioritization or the scheme becomes ineffective. >>>>>>>=20 >>>>>>> Yes and no, one of the clearest winners has been flow queueing, = IMHO not because it is the most optimal capacity sharing scheme, but = because it is the least pessimal scheme, allowing all (or none) flows = forward progress. You can interpret that as a scheme in which flows = below their capacity share are prioritized, but I am not sure that is = the best way to look at these things. >>>>>>=20 >>>>>> The hardest part is getting competing ISPs to implement and = coordinate. Bufferbloat and handoff between ISPs will be hard. The only = way to fix this is to get the unwashed public to care. Then they can say = =E2=80=9Cwe don=E2=80=99t care about the technical issues, just fix = it.=E2=80=9D Until then =E2=80=A6.. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>> Regards >>>>>>> Sebastian >>>>>>>=20 >>>>>>>=20 >>>>>>>>=20 >>>>>>>> Gene >>>>>>>> ---------------------------------------------- >>>>>>>> Eugene Chang >>>>>>>> IEEE Senior Life Member >>>>>>>> eugene.chang@ieee.org >>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> On Sep 26, 2022, at 10:48 AM, David Lang = wrote: >>>>>>>>>=20 >>>>>>>>> software updates can do far more than just improve recovery. >>>>>>>>>=20 >>>>>>>>> In practice, large data transfers are less sensitive to = latency than smaller data transfers (i.e. downloading a CD image vs a = video conference), software can ensure better fairness in preventing a = bulk transfer from hurting the more latency sensitive transfers. >>>>>>>>>=20 >>>>>>>>> (the example below is not completely accurate, but I think it = gets the point across) >>>>>>>>>=20 >>>>>>>>> When buffers become excessivly large, you have the situation = where a video call is going to generate a small amount of data at a = regular interval, but a bulk data transfer is able to dump a huge amount = of data into the buffer instantly. >>>>>>>>>=20 >>>>>>>>> If you just do FIFO, then you get a small chunk of video call, = then several seconds worth of CD transfer, followed by the next small = chunk of the video call. >>>>>>>>>=20 >>>>>>>>> But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic. >>>>>>>>>=20 >>>>>>>>> The one thing that Cake needs to work really well is to be = able to know what the data rate available is. With Starlink, this = changes frequently and cake integrated into the starlink dish/router = software would be far better than anything that can be done externally = as the rate changes can be fed directly into the settings (currently = they are only indirectly detected) >>>>>>>>>=20 >>>>>>>>> David Lang >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> On Mon, 26 Sep 2022, Eugene Y Chang via Starlink wrote: >>>>>>>>>=20 >>>>>>>>>> You already know this. Bufferbloat is a symptom and not the = cause. Bufferbloat grows when there are (1) periods of low or no = bandwidth or (2) periods of insufficient bandwidth (aka network = congestion). >>>>>>>>>>=20 >>>>>>>>>> If I understand this correctly, just a software update cannot = make bufferbloat go away. It might improve the speed of recovery (e.g. = throw away all time sensitive UDP messages). >>>>>>>>>>=20 >>>>>>>>>> Gene >>>>>>>>>> ---------------------------------------------- >>>>>>>>>> Eugene Chang >>>>>>>>>> IEEE Senior Life Member >>>>>>>>>> eugene.chang@ieee.org >>>>>>>>>> 781-799-0233 (in Honolulu) >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>> On Sep 26, 2022, at 10:04 AM, Bruce Perens = wrote: >>>>>>>>>>>=20 >>>>>>>>>>> Please help to explain. Here's a draft to start with: >>>>>>>>>>>=20 >>>>>>>>>>> Starlink Performance Not Sufficient for Military = Applications, Say Scientists >>>>>>>>>>>=20 >>>>>>>>>>> The problem is not availability: Starlink works where = nothing but another satellite network would. It's not bandwidth, = although others have questions about sustaining bandwidth as the = customer base grows. It's latency and jitter. As load increases, = latency, the time it takes for a packet to get through, increases more = than it should. The scientists who have fought bufferbloat, a major = cause of latency on the internet, know why. SpaceX needs to upgrade = their system to use the scientist's Open Source modifications to Linux = to fight bufferbloat, and thus reduce latency. This is mostly just using = a newer version, but there are some tunable parameters. Jitter is a = change in the speed of getting a packet through the network during a = connection, which is inevitable in satellite networks, but will be = improved by making use of the bufferbloat-fighting software, and = probably with the addition of more satellites. >>>>>>>>>>>=20 >>>>>>>>>>> We've done all of the work, SpaceX just needs to adopt it by = upgrading their software, said scientist Dave Taht. Jim Gettys, Taht's = collaborator and creator of the X Window System, chimed in: >>>>>>>>>>> Open Source luminary Bruce Perens said: sometimes Starlink's = latency and jitter make it inadequate to remote-control my ham radio = station. But the military is experimenting with remote-control of = vehicles on the battlefield and other applications that can be = demonstrated, but won't happen at scale without adoption of = bufferbloat-fighting strategies. >>>>>>>>>>>=20 >>>>>>>>>>> On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = > wrote: >>>>>>>>>>> The key issue is most people don=E2=80=99t understand why = latency matters. They don=E2=80=99t see it or feel it=E2=80=99s impact. >>>>>>>>>>>=20 >>>>>>>>>>> First, we have to help people see the symptoms of latency = and how it impacts something they care about. >>>>>>>>>>> - gamers care but most people may think it is frivolous. >>>>>>>>>>> - musicians care but that is mostly for a hobby. >>>>>>>>>>> - business should care because of productivity but they = don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact. >>>>>>>>>>>=20 >>>>>>>>>>> Second, there needs to be a =E2=80=9COMG, I have been seeing = the action of latency all this time and never knew it! I was being = shafted.=E2=80=9D Once you have this awakening, you can get all the = press you want for free. >>>>>>>>>>>=20 >>>>>>>>>>> Most of the time when business apps are developed, =E2=80=9Cwe= =E2=80=9D hide the impact of poor performance (aka latency) or they hide = from the discussion because the developers don=E2=80=99t have a way to = fix the latency. Maybe businesses don=E2=80=99t care because any = employees affected are just considered poor performers. (In bad economic = times, the poor performers are just laid off.) For employees, if they = happen to be at a location with bad latency, they don=E2=80=99t know = that latency is hurting them. Unfair but most people don=E2=80=99t know = the issue is latency. >>>>>>>>>>>=20 >>>>>>>>>>> Talking and explaining why latency is bad is not as = effective as showing why latency is bad. Showing has to be with = something that has a person impact. >>>>>>>>>>>=20 >>>>>>>>>>> Gene >>>>>>>>>>> ----------------------------------- >>>>>>>>>>> Eugene Chang >>>>>>>>>>> eugene.chang@alum.mit.edu >>>>>>>>>>> +1-781-799-0233 (in Honolulu) >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>> On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = > = wrote: >>>>>>>>>>>>=20 >>>>>>>>>>>> If you want to get attention, you can get it for free. I = can place articles with various press if there is something interesting = to say. Did this all through the evangelism of Open Source. All we need = to do is write, sign, and publish a statement. What they actually write = is less relevant if they publish a link to our statement. >>>>>>>>>>>>=20 >>>>>>>>>>>> Right now I am concerned that the Starlink latency and = jitter is going to be a problem even for remote controlling my ham = station. The US Military is interested in doing much more, which they = have demonstrated, but I don't see happening at scale without some = technical work on the network. Being able to say this isn't ready for = the government's application would be an attention-getter. >>>>>>>>>>>>=20 >>>>>>>>>>>> Thanks >>>>>>>>>>>>=20 >>>>>>>>>>>> Bruce >>>>>>>>>>>>=20 >>>>>>>>>>>> On Mon, Sep 26, 2022 at 9:21 AM Dave Taht via Starlink = > = wrote: >>>>>>>>>>>> These days, if you want attention, you gotta buy it. A 50k = half page >>>>>>>>>>>> ad in the wapo or NYT riffing off of It's the latency, = Stupid!", >>>>>>>>>>>> signed by the kinds of luminaries we got for the fcc wifi = fight, would >>>>>>>>>>>> go a long way towards shifting the tide. >>>>>>>>>>>>=20 >>>>>>>>>>>> On Mon, Sep 26, 2022 at 8:29 AM Dave Taht = > wrote: >>>>>>>>>>>>>=20 >>>>>>>>>>>>> On Mon, Sep 26, 2022 at 8:20 AM Livingood, Jason >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> The awareness & understanding of latency & impact on QoE = is nearly unknown among reporters. IMO maybe there should be some kind = of background briefings for reporters - maybe like a simple YouTube = video explainer that is short & high level & visual? Otherwise reporters = will just continue to focus on what they know... >>>>>>>>>>>>>=20 >>>>>>>>>>>>> That's a great idea. I have visions of crashing the = washington >>>>>>>>>>>>> correspondents dinner, but perhaps >>>>>>>>>>>>> there is some set of gatherings journalists regularly = attend? >>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> =EF=BB=BFOn 9/21/22, 14:35, "Starlink on behalf of Dave = Taht via Starlink" on behalf of = starlink@lists.bufferbloat.net > = wrote: >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>> I still find it remarkable that reporters are still = missing the >>>>>>>>>>>>>> meaning of the huge latencies for starlink, under load. >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>>=20 >>>>>>>>>>>>> -- >>>>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> -- >>>>>>>>>>>> FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ >>>>>>>>>>>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Starlink mailing list >>>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> -- >>>>>>>>>>>> Bruce Perens K6BP >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Starlink mailing list >>>>>>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> -- >>>>>>>>>>> Bruce Perens K6BP >>>>>>>>=20 >>>>>>>> _______________________________________________ >>>>>>>> Starlink mailing list >>>>>>>> Starlink@lists.bufferbloat.net = >>>>>>>> https://lists.bufferbloat.net/listinfo/starlink = >>>=20 >>=20 >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net = >> https://lists.bufferbloat.net/listinfo/starlink = >=20 >=20 >=20 > -- > FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/ = > Dave T=C3=A4ht CEO, TekLibre, LLC --Apple-Mail=_E9A13AF0-380A-45F0-AA55-77D9043DEF70 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Yup, = all true.
I got lots of things to rant about = too.

Now, Mr. = ISP, you signed me up for 1Gbps service. You told me it would fix my = performance problem. You bundled a kit with the Wi-Fi gateway (router). = And you give me bad support if I provide my own router. So why should = you have to know the stuff that Dave is telling me.

Yeah, and when I first = complained about sub-300Mbps service on my 1Gbps service, after 3 months = of helping the ISP support escalate the problem, they eventually = reported =E2=80=9Coh, we didn=E2=80=99t notice the OLT was = overloaded=E2=80=9D. Eh, don=E2=80=99t they have equipment monitoring = and all that basic stuff.  Then they reported it will take 6 months = to get a new OLT shelf added. =E2=80=A6.  So lame.   So sad. = Deliberately incompetent because it saves them money.


Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life = Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 27, 2022, at 3:55 AM, Dave Taht <dave.taht@gmail.com>= wrote:

On Mon, Sep 26, 2022 at 11:36 PM Sebastian Moeller via = Starlink
<starlink@lists.bufferbloat.net> wrote:

Worst, I waited 9 months for them to = resolve why I was only getting 300Mbps on my 1Gbps service (oops, sorry = my 700Mbps service).

Nothing I've seen to date below $400 dollars can actually = push a gbit
in both = directions at the same time, with
NAT, firewalling, etc. A lot of gear can't even push a gbit = in one direction.

The long accepted fallacy that a gigabit "router", is just a = gigabit
switch, = bothers me, but it is just one piece of a long line of
orwellian = doublethink about how the net works but is so pervasive we
ended up = publishing:

https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and= -need-a-router-read-this-first/90305

I've seen an ostensibly serious proposal that budgeted about = $4k/sub
for a = symmetric gig fiber rollout that budgeted...
wait for = it... $75 dollars for the router, and other proposals that
wanted to = punt the firewalling etc to a more centralized server, and
just put a = switch at the sub.

[SM] I guess mass market internet = access is a low margin business, where expert debugging time is = precious. I hope they at least gave you a rebate for that time.

Why is it so few take packet captures anymore? (I think I = will rant on
that at more = length elsewhere)


Regards
        Sebastian





Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 11:29 AM, Sebastian Moeller <moeller0@gmx.de> wrote:

Hi David,

On Sep 26, 2022, at = 23:22, David Lang <david@lang.hm> wrote:

On Mon, 26 Sep 2022, Eugene Y Chang wrote:

On Sep 26, 2022, at 11:01 AM, Sebastian Moeller = <moeller0@gmx.de> wrote:

Hi = Eugene,


On Sep 26, 2022, at 22:54, Eugene Y Chang via = Starlink <starlink@lists.bufferbloat.net = <mailto:starlink@lists.bufferbloat.net>> wrote:
Ok, we are getting into the details. I agree.

Every node in the path has to implement this = to be effective.

 Amazingly= the biggest bang for the buck is gotten by fixing those nodes that = actually contain a network path's bottleneck. Often these are pretty = stable. So yes for fully guaranteed service quality all nodes would need = to participate, but for improving things noticeably it is sufficient to = improve the usual bottlenecks, e.g. for many internet access links the = home gateway is a decent point to implement better buffer management. = (In short the problem are over-sized and under-managed buffers, and one = of the best solution is better/smarter buffer management).


This is not = completely true. Say the bottleneck is at node N. During the period of = congestion, the upstream node N-1 will have to buffer. When node N = recovers, the bufferbloat at N-1 will be blocking until the bufferbloat = drains. Etc. etc.  Making node N better will reduce the extent of = the backup at N-1, but N-1 should implement the better code.

only if node N and node N-1 = handle the same traffic with the same link speeds. In practice this is = almost never the case.

    [SM] I note that typically for = ingress shaping a post-true-bottleneck shaper will not work unless we = create an artificial bottleneck by shaping the traffic to below true = bottleneck (thereby creating a new true but artificial bottleneck so the = queue develops at a point where we can control it).
    Also if the difference between "true = structural" and artificial bottleneck is small in comparison to the = traffic inrush we can see "traffic back-spill" into the typically = oversized and under-managed upstream buffers, but for reasonably well = behaved that happens relatively rarely. Rarely enough that ingress = traffic shaping noticeably improves latency-under-load in spite of not = beeing a guranteed solution.


Until you get to gigabit = last-mile links, the last mile is almost always the bottleneck from both = sides, so implementing cake on the home router makes a huge improvement = (and if you can get it on the last-mile ISP router, even better). Once = you get into the Internet fabric, bottlenecks are fairly rare, they do = happen, but ISPs carefully watch for those and add additional paths = and/or increase bandwith to avoid them.

    [SM] Well, sometimes such links are = congested too for economic reasons...

Regards
    Sebastian



David Lang


In fact, = every node in the path has to have the same prioritization or the scheme = becomes ineffective.

 Yes = and no, one of the clearest winners has been flow queueing, IMHO not = because it is the most optimal capacity sharing scheme, but because it = is the least pessimal scheme, allowing all (or none) flows forward = progress. You can interpret that as a scheme in which flows below their = capacity share are prioritized, but I am not sure that is the best way = to look at these things.

The = hardest part is getting competing ISPs to implement and coordinate. = Bufferbloat and handoff between ISPs will be hard. The only way to fix = this is to get the unwashed public to care. Then they can say =E2=80=9Cwe = don=E2=80=99t care about the technical issues, just fix it.=E2=80=9D = Until then =E2=80=A6..




Regards
 Sebastian



Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:48 AM, David Lang <david@lang.hm> wrote:

software updates can do far more than just improve = recovery.

In practice, large data transfers = are less sensitive to latency than smaller data transfers (i.e. = downloading a CD image vs a video conference), software can ensure = better fairness in preventing a bulk transfer from hurting the more = latency sensitive transfers.

(the example = below is not completely accurate, but I think it gets the point = across)

When buffers become excessivly = large, you have the situation where a video call is going to generate a = small amount of data at a regular interval, but a bulk data transfer is = able to dump a huge amount of data into the buffer instantly.

If you just do FIFO, then you get a small = chunk of video call, then several seconds worth of CD transfer, followed = by the next small chunk of the video call.

But the software can prevent the one app from hogging so much = of the connection and let the chunk of video call in sooner, avoiding = the impact to the real time traffic. Historically this has required the = admin classify all traffic and configure equipment to implement = different treatment based on the classification (and this requires trust = in the classification process), the bufferbloat team has developed = options (fq_codel and cake) that can ensure fairness between = applications/servers with little or no configuration, and no trust in = other systems to properly classify their traffic.

The one thing that Cake needs to work really well is to be = able to know what the data rate available is. With Starlink, this = changes frequently and cake integrated into the starlink dish/router = software would be far better than anything that can be done externally = as the rate changes can be fed directly into the settings (currently = they are only indirectly detected)

David = Lang


On Mon, 26 Sep 2022, = Eugene Y Chang via Starlink wrote:

You already know this. = Bufferbloat is a symptom and not the cause. Bufferbloat grows when there = are (1) periods of low or no bandwidth or (2) periods of insufficient = bandwidth (aka network congestion).

If I = understand this correctly, just a software update cannot make = bufferbloat go away. It might improve the speed of recovery (e.g. throw = away all time sensitive UDP messages).

Gene
----------------------------------------------
Eugene Chang
IEEE Senior Life Member
eugene.chang@ieee.org
781-799-0233 (in = Honolulu)



On Sep 26, 2022, at = 10:04 AM, Bruce Perens <bruce@perens.com> wrote:

Please help to explain. Here's a draft to start with:

Starlink Performance Not Sufficient for = Military Applications, Say Scientists

The = problem is not availability: Starlink works where nothing but another = satellite network would. It's not bandwidth, although others have = questions about sustaining bandwidth as the customer base grows. It's = latency and jitter. As load increases, latency, the time it takes for a = packet to get through, increases more than it should. The scientists who = have fought bufferbloat, a major cause of latency on the internet, know = why. SpaceX needs to upgrade their system to use the scientist's Open = Source modifications to Linux to fight bufferbloat, and thus reduce = latency. This is mostly just using a newer version, but there are some = tunable parameters. Jitter is a change in the speed of getting a packet = through the network during a connection, which is inevitable in = satellite networks, but will be improved by making use of the = bufferbloat-fighting software, and probably with the addition of more = satellites.

We've done all of the work, = SpaceX just needs to adopt it by upgrading their software, said = scientist Dave Taht. Jim Gettys, Taht's collaborator and creator of the = X Window System, chimed in: <fill in here please>
Open= Source luminary Bruce Perens said: sometimes Starlink's latency and = jitter make it inadequate to remote-control my ham radio station. But = the military is experimenting with remote-control of vehicles on the = battlefield and other applications that can be demonstrated, but won't = happen at scale without adoption of bufferbloat-fighting strategies.

On Mon, Sep 26, 2022 at 12:59 PM Eugene Chang = <eugene.chang@alum.mit.edu<mailto:eugene.chang@alum.mit.edu>> = wrote:
The key issue is most people don=E2=80=99t = understand why latency matters. They don=E2=80=99t see it or feel it=E2=80= =99s impact.

First, we have to help people = see the symptoms of latency and how it impacts something they care = about.
- gamers care but most people may think it is = frivolous.
- musicians care but that is mostly for a = hobby.
- business should care because of productivity but = they don=E2=80=99t know how to =E2=80=9Csee=E2=80=9D the impact.

Second, there needs to be a =E2=80=9COMG, I = have been seeing the action of latency all this time and never knew it! = I was being shafted.=E2=80=9D Once you have this awakening, you can get = all the press you want for free.

Most of = the time when business apps are developed, =E2=80=9Cwe=E2=80=9D hide the = impact of poor performance (aka latency) or they hide from the = discussion because the developers don=E2=80=99t have a way to fix the = latency. Maybe businesses don=E2=80=99t care because any employees = affected are just considered poor performers. (In bad economic times, = the poor performers are just laid off.) For employees, if they happen to = be at a location with bad latency, they don=E2=80=99t know that latency = is hurting them. Unfair but most people don=E2=80=99t know the issue is = latency.

Talking and explaining why latency = is bad is not as effective as showing why latency is bad. Showing has to = be with something that has a person impact.

Gene
-----------------------------------
Eugene Chang
eugene.chang@alum.mit.edu = <mailto:eugene.chang@alum.mit.edu>
+1-781-799-0233 = (in Honolulu)





On Sep 26, 2022, at 6:32 AM, Bruce Perens via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:

If you want to get = attention, you can get it for free. I can place articles with various = press if there is something interesting to say. Did this all through the = evangelism of Open Source. All we need to do is write, sign, and publish = a statement. What they actually write is less relevant if they publish a = link to our statement.

Right now I am = concerned that the Starlink latency and jitter is going to be a problem = even for remote controlling my ham station. The US Military is = interested in doing much more, which they have demonstrated, but I don't = see happening at scale without some technical work on the network. Being = able to say this isn't ready for the government's application would be = an attention-getter.

Thanks
Bruce

On Mon, Sep 26, 2022 at = 9:21 AM Dave Taht via Starlink = <starlink@lists.bufferbloat.net<mailto:starlink@lists.bufferbloat.ne= t>> wrote:
These days, if you want attention, you = gotta buy it. A 50k half page
ad in the wapo or NYT = riffing off of It's the latency, Stupid!",
signed by the = kinds of luminaries we got for the fcc wifi fight, would
go = a long way towards shifting the tide.

On = Mon, Sep 26, 2022 at 8:29 AM Dave Taht <dave.taht@gmail.com = <mailto:dave.taht@gmail.com>> wrote:

On Mon, Sep 26, 2022 at 8:20 AM = Livingood, Jason
<Jason_Livingood@comcast.com = <mailto:Jason_Livingood@comcast.com>> wrote:

The = awareness & understanding of latency & impact on QoE is nearly = unknown among reporters. IMO maybe there should be some kind of = background briefings for reporters - maybe like a simple YouTube video = explainer that is short & high level & visual? Otherwise = reporters will just continue to focus on what they know...

That's a great idea. I have = visions of crashing the washington
correspondents dinner, = but perhaps
there is some set of gatherings journalists = regularly attend?


=EF=BB=BFOn 9/21/22, 14:35, "Starlink on = behalf of Dave Taht via Starlink" = <starlink-bounces@lists.bufferbloat.net = <mailto:starlink-bounces@lists.bufferbloat.net> on behalf of = starlink@lists.bufferbloat.net = <mailto:starlink@lists.bufferbloat.net>> wrote:
I still find it remarkable that reporters are still missing = the
meaning of the huge latencies for starlink, under = load.




--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC


--
FQ World Domination pending: = https://blog.cerowrt.org/post/state_of_fq_codel/<https://blog.cerowrt.o= rg/post/state_of_fq_codel/>
Dave T=C3=A4ht CEO, = TekLibre, LLC
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>


--
Bruce Perens K6BP
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>



--Bruce Perens K6BP

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net = <mailto:Starlink@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/starlink = <https://lists.bufferbloat.net/listinfo/starlink>


--
Sent from = my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink



-- FQ World = Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave T=C3=A4ht = CEO, TekLibre, LLC

= --Apple-Mail=_E9A13AF0-380A-45F0-AA55-77D9043DEF70-- --Apple-Mail=_1EE90DCD-810D-4766-B0F3-D932A3CC32BB Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEERPTGiBqcibajhTSsv0/8FiYdKmAFAmMzk1EACgkQv0/8FiYd KmCt9A/+LGQx3oL6rq96KIML06voUthZ475AHo2e+llEXWrUvhcZNEF4FhQ+mzPg ugBLh2sJjqN2cfkjbYVKjO2s6z+TLVe2hSf7awP/kHqxWmBnRkz3I6l6ZM9ea9tL KEzTUyugMIpCbjKlWqWv1BKaNA/W6fVKm4cLIRZ22pTntyGhT7SS9YFrcArXYlpl EFhwoILg+jQC6FRlOzFl+eRPnJnGXUG91p5n3Vln5Qxgqam8bA/KBd4Pte0KusqP fgLJpDLRy9MBCSriJ8hkiMhvEYumRkGD/9lr947tLGa6t3dstqk1iYncyQvMYfCH dkiZMPLbv0m2lWomrfKMlgACQmkZgBhu/vAfuJ3vzB5eqkVy1+/D+W+pKIvmoaDO FjABARWwUShKfisKT/3dl9zOUi6r+ydjLucFlrP5l5On+CrqaHyEOItUFYO+NWdG jwKZO1DiAsgT4dgw2QW23ZCiiaA9MTK8hcgmRrqYVf8YxUI8MBiAki4RO4dKEUEs F3aHu2QbnSrpymheRPSr03GTwYoBN60Wh7jjwLLy0+bTMitRlZFvQtCNtZQX1uvB a8WI4uRmx574yLRiimxP/c/nZE9rQtOZYvynllRnZlTScERIr5j1kCeRwnKZzYY4 FFaFx7OnxY23tKtkl7EJM7ZdkjocstPL3kjZMmw1ZL8zKy6oXdU= =KMp4 -----END PGP SIGNATURE----- --Apple-Mail=_1EE90DCD-810D-4766-B0F3-D932A3CC32BB--