Hi Matt and Jamshid,

On 04.07.20 at 19:29 Matt Mathis via Bloat wrote:

> Key takeaway: pacing is inevitable, because it saves large content
> providers money (more efficient use of the most expensive silicon in
> the data center, the switch buffer memory), however to use pacing we
> walk away from 30 years of experience with TCP self clock, which is
> the foundation of all of our CC research....

Thanks for the interesting read. I have a few comments:

  * IMO, many of the mentioned problems are related to using packet loss
    as congestion signal rather than self-clocking.

  * In principle, one can keep utilization high and queuing delay low
    with a congestion window based and ACK-clock
    driven approach (see TCP LoLa
    https://ieeexplore.ieee.org/document/8109356). However, it currently
    lacks
    heuristics to deal with stretch/aggregated ACKs, but I think one can
    extend this like already done in BBR.

  * Pacing is really useful and I think it is important to keep sending
    in case the ACK-clock is distorted
    by the mentioned problems, but only for a limited time. If one's
    estimate for the correct sending rate
    is too high, the amount of inflight data increases over time, which
    leads to queuing delay and/or loss.
    So having the inflight cap as in BBRv1 is also an important safety
    measure.

  * "The maximum receive rate is probed by sending at 125% of max_BW .
    If the network is already full and flows have reached their fair share,
    the observed max_BW won’t change."
    This assumption isn't true if several flows are present at the
    bottleneck.
    If a flow sends with 1.25*max_BW on the saturated link, *the observed**
    **max_BW will change* (unless all flows are probing at the same
    time) because the probing flow preempts other flows, thereby
    reducing their current share. Together with the applied max-filter
    this is the reason why BBRv1 is constantly overestimating the available
    capacity and thus persistently increasing the amount inflight data
    until the inflight cap is hit. The math is in [32] (section 3) of your
    references. Luckily BBRv2 has much more safeguards built-in.

  * "The lower queue occupancy indicates that it is not generally taking
    capacity away from other transport protocols..."
    I think that this indication is not very robust, e.g., it may hold
    in case
    there isn't significant packet loss observed. Observing an overall
    lower buffer occupancy does not necessarily tell you something about
    the individual flow shares. In BBRv1 you could have starving Cubic
    flows, because they were backing-off due to loss, while BBR kept
    sending.

  * Last but not least, even BBR requires an ACK stream as
    feedback in order to estimate the delivery rate. But it is actually
    not self-clocked and keeps sending "blindly" for a while. This is
    quite useful to deal with the mentioned stretch/aggregated ACKs,
    if done with care.

Regards,
 Roland