[Bloat] Apple WWDC Talks on Latency/Bufferbloat

Mon Jun 28 18:54:04 EDT 2021

+Randall

On 06/17/21 - 20:33, Matt Mathis wrote:
> Also consider ippm.  intarea might be a good choice for joint sponsorship,
> but they probably won't want to be the lead.

Indeed, ippm might be a good candidate. Thanks!

> 
> BTW by using two TCP connections you potentially give a free pass to many
> types of networks (e.g. ECMP, SFQ, etc) and certain OS mis features.

Yes, we are aware of that. Which is why we are looking into how to minimize
the effects you mentioned in your previous email so that we can accurately
measure the latency under load on the load-bearing connections.

For the curious ones here. If you run on macOS the networkQuality command-line
with option "-c", you get more verbose output. In particular, you will see
the latency for H2-pings on the load-bearing connections (labeled
lud_self_dl_h2 and lud_self_ul_h2). You can see in the below how the
download load-bearing flows are suffering from too much data in the
TCP-socket buffer and data queued in the server's process behind the
bulk-data transfer (in this case it is Apache Traffic Server - we are
looking into how other server implementations behave).

MacBook-Pro:~ cpaasch$ networkQuality -c

{
  "lud_self_ul_h2" : [
    71.202995300292969,
    89.105010986328125,
    51.216960906982422,
    581.09698486328125,
    155.85601806640625,
    304.031982421875,
    271.76202392578125,
    202.48997497558594,
    139.15800476074219,
    160.45701599121094,
    247.11001586914062,
    626.049072265625,
    399.29306030273438,
    335.45803833007812,
    164.31092834472656
  ],
  "responsiveness" : 1075,
  "ul_throughput" : 28645646,
  "lud_foreign_tcp_handshake_443" : [
    34,
    42,
    34,
    37,
    39,
    36,
    36,
    31
  ],
  "lud_self_dl_h2" : [
    313.34603881835938,
    359.79306030273438,
    699.39007568359375,
    929.51605224609375,
    1653.333984375,
    2466.970947265625,
    2981.800048828125,
    2969.277099609375,
    3595.947021484375,
    3785.244873046875,
    3572.677001953125,
    2802.677978515625
  ],
  "dl_flows" : 20,
  "dl_throughput" : 396551712,
  "ul_flows" : 12,
  "lud_foreign_h2_req_resp" : [
    63,
    75,
    80,
    73,
    70,
    69,
    78,
    96
  ]
}

Christoph

> Thanks,
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
> 
> We must not tolerate intolerance;
>        however our response must be carefully measured:
>             too strong would be hypocritical and risks spiraling out of
> control;
>             too weak risks being mistaken for tacit approval.
> 
> 
> On Thu, Jun 17, 2021 at 6:04 PM Christoph Paasch <cpaasch at apple.com> wrote:
> 
> > Not sure yet - there isn’t a good one that would really fit. Maybe tsvwg
> > or intarea.
> >
> > Suggestions?
> >
> > Cheers,
> > Christoph
> >
> > On Jun 17, 2021, at 5:17 PM, Matt Mathis <mattmathis at google.com> wrote:
> >
> > 
> > Which WG are you targeting?
> >
> > Thanks,
> > --MM--
> > The best way to predict the future is to create it.  - Alan Kay
> >
> > We must not tolerate intolerance;
> >        however our response must be carefully measured:
> >             too strong would be hypocritical and risks spiraling out of
> > control;
> >             too weak risks being mistaken for tacit approval.
> >
> >
> > On Thu, Jun 17, 2021 at 4:43 PM Christoph Paasch <cpaasch at apple.com>
> > wrote:
> >
> >> Hello,
> >>
> >> On 06/17/21 - 11:16, Matt Mathis via Bloat wrote:
> >> > Is there a paper or spec for RPM?
> >>
> >> we try to publish an IETF-draft on the methodology before the upcoming
> >> IETF
> >> in July.
> >>
> >> But, in the mean-time please see inline:
> >>
> >> > There are at least two different ways to define RPM, both of which
> >> might be
> >> > relevant.
> >> >
> >> > At the TCP layer: it can be directly computed from a packet capture.
> >> The
> >> > trick is to time reverse a trace and compute the critical path backwards
> >> > through the trace: what event triggered each segment or ACK, and count
> >> > round trips.  This would be super robust but does not include the
> >> queueing
> >> > required in the kernel socket buffers.  I need to think some more about
> >> > computing TCP RPM from tcp_info or other kernel instrumentation - it
> >> might
> >> > be possible.
> >>
> >> We explicitly opted against measuring purely TCP-level round-trip times.
> >> Because
> >> there are countless transparent TCP-proxies out there that would skew
> >> these
> >> numbers. Our goal with RPM/Responsiveness is to measure how an end-user
> >> would
> >> experience the network. Which means, DNS-resolution, TCP handshake-time,
> >> TLS-handshake, HTTP/2 Request/response. Because, at the end, that's what
> >> actually matters to the users.
> >>
> >> > A different RPM can be done in the application, above TCP, for example
> >> by
> >> > ping-ponging messages.  This would include the delays traversing the
> >> kernel
> >> > socket buffers which have to be at least as large as a full network RTT.
> >> >
> >> > This is perhaps an important point: due to the retransmit and
> >> > reassuebly queues (which are required to implement robust data delivery)
> >> > TCP must be able hold at least a full RTT of data in it's own buffers,
> >> > which means that under some conditions the RTT as seen by the
> >> application
> >> > has be be at least twice the network's RTT, including any bloat in the
> >> > network.
> >>
> >> Currently, we measure RPM on separate connections (not the load-bearing
> >> ones). We are also measuring on the load-bearing connections themselves
> >> through H2 Ping frames. But for the reasons you described we haven't yet
> >> factored it into the RPM-number.
> >>
> >> One way may be to inspect with TCP_INFO whether or not the connections had
> >> retransmissions and then throw away the number. On the other hand, if the
> >> network becomes extremely lossy under working conditions, it does impact
> >> the
> >> user-experience and so it could make sense to take this into account.
> >>
> >>
> >> In the end, we realized how hard it is to accurately measure bufferbloat
> >> within a reasonable time-frame (our goal is to finish the test within ~15
> >> seconds).
> >>
> >> We hope that with the IETF-draft we can get the right people together to
> >> iterate over it and squash out a very accurate measurement that represents
> >> what users would experience.
> >>
> >>
> >> Cheers,
> >> Christoph
> >>
> >>
> >> >
> >> > Thanks,
> >> > --MM--
> >> > The best way to predict the future is to create it.  - Alan Kay
> >> >
> >> > We must not tolerate intolerance;
> >> >        however our response must be carefully measured:
> >> >             too strong would be hypocritical and risks spiraling out of
> >> > control;
> >> >             too weak risks being mistaken for tacit approval.
> >> >
> >> >
> >> > On Sat, Jun 12, 2021 at 9:11 AM Rich Brown <richb.hanover at gmail.com>
> >> wrote:
> >> >
> >> > > > On Jun 12, 2021, at 12:00 PM, bloat-request at lists.bufferbloat.net
> >> wrote:
> >> > > >
> >> > > > Some relevant talks / publicity at WWDC -- the first mentioning
> >> CoDel,
> >> > > > queueing, etc. Featuring Stuart Cheshire. iOS 15 adds a developer
> >> test
> >> > > for
> >> > > > loaded latency, reported in "RPM" or round-trips per minute.
> >> > > >
> >> > > > I ran it on my machine:
> >> > > > nowens at mac1015 ~ % /usr/bin/networkQuality
> >> > > > ==== SUMMARY ====
> >> > > > Upload capacity: 90.867 Mbps
> >> > > > Download capacity: 93.616 Mbps
> >> > > > Upload flows: 16
> >> > > > Download flows: 20
> >> > > > Responsiveness: Medium (840 RPM)
> >> > >
> >> > > Does anyone know how to get the command-line version for current (not
> >> > > upcoming) macOS? Thanks.
> >> > >
> >> > > Rich
> >> > > _______________________________________________
> >> > > Bloat mailing list
> >> > > Bloat at lists.bufferbloat.net
> >> > > https://lists.bufferbloat.net/listinfo/bloat
> >> > >
> >>
> >> > _______________________________________________
> >> > Bloat mailing list
> >> > Bloat at lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/bloat
> >>
> >>