[Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Dave Taht]

[-- Attachment #1.1: Type: text/plain, Size: 5773 bytes --]

eric, I thought I'd forward this to one of the bufferbloat.net lists. Feel
free to join. It would also be helpful to duplicate your test scripts here.

I agree with you, bbr is way too slow here. We see it bloat massively in
wifi also. I would like BBR to far more reactive than it is, and also for
it to respond to RFC3168 ECN marks.

You said:

*" *In my opinion, in high bw jitter scenario, window based bw undaption
method is too slow to converge to the actual bottleneck bw,  when actual bw
imcreases, BBR needs at lease 8RTT to catch it, when it decreases, BBR then
needs at most 10 RTT to expire the previous bw maximum, anyway it is too
slow, causing severe buffer-bloating. "

You might find the "flent" tool very useful for getting a more microscopic
view as to what is going on.

A test we would use to get a better picture of what is really going on
would be

flent -H netperf_test_server -x --socket-stats --step-size=.05 -t
the_test_conditions --te=cc_algo=bbr --te=upload_streams=1 tcp_nup

This samples a ton of tcp related stats and lets you plot them with
flent-gui, there are 110 other tests in the suite.

A more complicated invocation can get qdisc stats from the router involved,
or iterate over CCs, or run at longer intervals. In general I have found
running tests for 5 minutes or longer reveal interesting behaviors.

flent -H $S --socket-stats -x --step-size=.05 -t
1-flow-fq_starlink_noecn-${M}-c

c=${CC} --te=cpu_stats_hosts=$R --te=netstat_hosts=$R -4
--te=qdisc_stats_hosts=

$RS --te=qdisc_stats_interfaces=$RV --te=upload_streams=1 --te=cc_algo=$CC
tcp_1

up


pic below

---------- Forwarded message ---------
From: Eric xu <ericbin1224@gmail.com>
Date: Mon, Jul 5, 2021 at 12:27 AM
Subject: Re: [bbr-dev] BBR performance and optimization in cellular
wireless network with high delay jitter
To: BBR Development <bbr-dev@googlegroups.com>


Hi, Neal, thanks for your answer. I still  have two questions to discuss
with you after having read the two links you mentioned above.

1, As mentioned in the second link, BBR's throughput on high-jitter links
(especially in cellular) usually matches CUBIC,  as  demonstrated by the
following figure,
[image: 截图.PNG]
however, what we can infer from the figure is that, the mean
throughput(~14Mbps) is far from the available bw that is greater than
20Mbps as showed in the cumulative distrubution figure. Since CUBIC back
off frequently in the LTE network with high error rate, but BBR does not
acquire much more throughput than CUBIC, the only reason I can imagine is
that BBR is highly affected by high delay jitter although cwnd provision is
applied.

*Is my understanding right? and is there any optimization to further
improve thoughput in high bw and delay jitter scenario? *

*2,  Is there any method to enhance BBR's adaptive ability to high bw and
delay jitter?  *In my opinion, in high bw jitter scenario, window based bw
undaption method is too slow to converge to the actual bottleneck bw,  when
actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
anyway it is too slow, causing severe buffer-bloating.  In high delay
jitter scenario, BBR also will cause bw under utilizaion. *So,  what is
your latest optimizaion to improve the performance in such a high bw and
delay jitter network?*

Thx a lot,
Eric
在2021年7月4日星期日 UTC+8 下午9:59:57<Neal Cardwell> 写道：

> Yes, BBR includes mechanisms to deal with this kind of jitter. Please see
> the March 2019 thread on this topic:
>   https://groups.google.com/g/bbr-dev/c/kBZaq98xCC4
>
> Our experience is that BBR's throughput on high-jitter links (cellular,
> wifi, DOCSIS, datacenter ethernet w/ GRO/LRO) is quite good at this point
> (usually matches CUBIC), and there are others who have documented this as
> well, e.g.:
>   http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
>
> best,
> neal
>
>
> On Sun, Jul 4, 2021 at 8:53 AM Eric xu <ericb...@gmail.com> wrote:
>
>> Hello, dear friends. I wonder there is any optimization to BBR ,  for the
>> reason of BBR low bandwidth utilization in wireless network with high delay
>> jitter.  In my opinion,  High delay jitter will cause BBR to underestimate
>> the BDP, thus limit the send rate even if  pipeline is not full. Right?
>> So,  I wonder, *if there is any optimization to improve bw utilization
>> in wireless network with high delay jitter, even in 5G network? *Thx a
>> lot.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to bbr-dev+u...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com
>> <https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
You received this message because you are subscribed to the Google Groups
"BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to bbr-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com
<https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
.


-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

[-- Attachment #1.2: Type: text/html, Size: 9827 bytes --]

[-- Attachment #2: 截图.PNG --]
[-- Type: image/png, Size: 52872 bytes --]

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Bob McMahon]

[-- Attachment #1.1.1: Type: text/plain, Size: 7256 bytes --]

Curious to the idea of wifi providing the opposite of ECN, i.e. my rate
selection algorithm just went from one stream to two stream so TCP speed up
your feedback loop decisions with a bias to faster?

Bob

On Mon, Jul 5, 2021 at 10:20 AM Dave Taht <dave.taht@gmail.com> wrote:

> eric, I thought I'd forward this to one of the bufferbloat.net lists.
> Feel free to join. It would also be helpful to duplicate your test scripts
> here.
>
> I agree with you, bbr is way too slow here. We see it bloat massively in
> wifi also. I would like BBR to far more reactive than it is, and also for
> it to respond to RFC3168 ECN marks.
>
> You said:
>
> *" *In my opinion, in high bw jitter scenario, window based bw undaption
> method is too slow to converge to the actual bottleneck bw,  when actual bw
> imcreases, BBR needs at lease 8RTT to catch it, when it decreases, BBR then
> needs at most 10 RTT to expire the previous bw maximum, anyway it is too
> slow, causing severe buffer-bloating. "
>
> You might find the "flent" tool very useful for getting a more microscopic
> view as to what is going on.
>
> A test we would use to get a better picture of what is really going on
> would be
>
> flent -H netperf_test_server -x --socket-stats --step-size=.05 -t
> the_test_conditions --te=cc_algo=bbr --te=upload_streams=1 tcp_nup
>
> This samples a ton of tcp related stats and lets you plot them with
> flent-gui, there are 110 other tests in the suite.
>
> A more complicated invocation can get qdisc stats from the router
> involved, or iterate over CCs, or run at longer intervals. In general I
> have found running tests for 5 minutes or longer reveal interesting
> behaviors.
>
> flent -H $S --socket-stats -x --step-size=.05 -t
> 1-flow-fq_starlink_noecn-${M}-c
>
> c=${CC} --te=cpu_stats_hosts=$R --te=netstat_hosts=$R -4
> --te=qdisc_stats_hosts=
>
> $RS --te=qdisc_stats_interfaces=$RV --te=upload_streams=1 --te=cc_algo=$CC
> tcp_1
>
> up
>
>
> pic below
>
> ---------- Forwarded message ---------
> From: Eric xu <ericbin1224@gmail.com>
> Date: Mon, Jul 5, 2021 at 12:27 AM
> Subject: Re: [bbr-dev] BBR performance and optimization in cellular
> wireless network with high delay jitter
> To: BBR Development <bbr-dev@googlegroups.com>
>
>
> Hi, Neal, thanks for your answer. I still  have two questions to discuss
> with you after having read the two links you mentioned above.
>
> 1, As mentioned in the second link, BBR's throughput on high-jitter links
> (especially in cellular) usually matches CUBIC,  as  demonstrated by the
> following figure,
> [image: 截图.PNG]
> however, what we can infer from the figure is that, the mean
> throughput(~14Mbps) is far from the available bw that is greater than
> 20Mbps as showed in the cumulative distrubution figure. Since CUBIC back
> off frequently in the LTE network with high error rate, but BBR does not
> acquire much more throughput than CUBIC, the only reason I can imagine is
> that BBR is highly affected by high delay jitter although cwnd provision is
> applied.
>
> *Is my understanding right? and is there any optimization to further
> improve thoughput in high bw and delay jitter scenario? *
>
> *2,  Is there any method to enhance BBR's adaptive ability to high bw and
> delay jitter?  *In my opinion, in high bw jitter scenario, window based
> bw undaption method is too slow to converge to the actual bottleneck bw,
> when actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
> anyway it is too slow, causing severe buffer-bloating.  In high delay
> jitter scenario, BBR also will cause bw under utilizaion. *So,  what is
> your latest optimizaion to improve the performance in such a high bw and
> delay jitter network?*
>
> Thx a lot,
> Eric
> 在2021年7月4日星期日 UTC+8 下午9:59:57<Neal Cardwell> 写道：
>
>> Yes, BBR includes mechanisms to deal with this kind of jitter. Please see
>> the March 2019 thread on this topic:
>>   https://groups.google.com/g/bbr-dev/c/kBZaq98xCC4
>>
>> Our experience is that BBR's throughput on high-jitter links (cellular,
>> wifi, DOCSIS, datacenter ethernet w/ GRO/LRO) is quite good at this point
>> (usually matches CUBIC), and there are others who have documented this as
>> well, e.g.:
>>   http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
>>
>> best,
>> neal
>>
>>
>> On Sun, Jul 4, 2021 at 8:53 AM Eric xu <ericb...@gmail.com> wrote:
>>
>>> Hello, dear friends. I wonder there is any optimization to BBR ,  for
>>> the reason of BBR low bandwidth utilization in wireless network with high
>>> delay jitter.  In my opinion,  High delay jitter will cause BBR to
>>> underestimate the BDP, thus limit the send rate even if  pipeline is
>>> not full. Right?
>>> So,  I wonder, *if there is any optimization to improve bw utilization
>>> in wireless network with high delay jitter, even in 5G network? *Thx a
>>> lot.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "BBR Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to bbr-dev+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com
>>> <https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com
> <https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> Latest Podcast:
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>
> Dave Täht CTO, TekLibre, LLC
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.1.2: Type: text/html, Size: 11431 bytes --]

[-- Attachment #1.2: 截图.PNG --]
[-- Type: image/png, Size: 52872 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Neal Cardwell]

[-- Attachment #1.1: Type: text/plain, Size: 9419 bytes --]

On Tue, Jul 6, 2021 at 4:34 PM 'Bob McMahon' via BBR Development <
bbr-dev@googlegroups.com> wrote:

> Curious to the idea of wifi providing the opposite of ECN, i.e. my rate
> selection algorithm just went from one stream to two stream so TCP speed up
> your feedback loop decisions with a bias to faster?
>

That could make sense. That sounds a lot lot like the Accel-Brake Control
work:

https://www.usenix.org/conference/nsdi20/presentation/goyal

ABC: A Simple Explicit Congestion Controller for Wireless Networks

Authors:
Prateesh Goyal, MIT CSAIL; Anup Agarwal, CMU; Ravi Netravali, UCLA;
Mohammad Alizadeh and Hari Balakrishnan, MIT CSAIL

Abstract:
We propose Accel-Brake Control (ABC), a simple and deployable explicit
congestion control protocol for network paths with time-varying wireless
links. ABC routers mark each packet with an “accelerate” or “brake”, which
causes senders to slightly increase or decrease their congestion windows.
Routers use this feedback to quickly guide senders towards a desired target
rate. ABC requires no changes to header formats or user devices, but
achieves better performance than XCP. ABC is also incrementally deployable;
it operates correctly when the bottleneck is a non-ABC router, and can
coexist with non-ABC traffic sharing the same bottleneck link. We evaluate
ABC using a Wi-Fi implementation and trace-driven emulation of cellular
links. ABC achieves 30-40% higher throughput than Cubic+Codel for similar
delays, and 2.2× lower delays than BBR on a Wi-Fi path. On cellular network
paths, ABC achieves 50% higher throughput than Cubic+Codel.

neal

Bob
>
> On Mon, Jul 5, 2021 at 10:20 AM Dave Taht <dave.taht@gmail.com> wrote:
>
>> eric, I thought I'd forward this to one of the bufferbloat.net lists.
>> Feel free to join. It would also be helpful to duplicate your test scripts
>> here.
>>
>> I agree with you, bbr is way too slow here. We see it bloat massively in
>> wifi also. I would like BBR to far more reactive than it is, and also for
>> it to respond to RFC3168 ECN marks.
>>
>> You said:
>>
>> *" *In my opinion, in high bw jitter scenario, window based bw undaption
>> method is too slow to converge to the actual bottleneck bw,  when actual bw
>> imcreases, BBR needs at lease 8RTT to catch it, when it decreases, BBR then
>> needs at most 10 RTT to expire the previous bw maximum, anyway it is too
>> slow, causing severe buffer-bloating. "
>>
>> You might find the "flent" tool very useful for getting a more
>> microscopic view as to what is going on.
>>
>> A test we would use to get a better picture of what is really going on
>> would be
>>
>> flent -H netperf_test_server -x --socket-stats --step-size=.05 -t
>> the_test_conditions --te=cc_algo=bbr --te=upload_streams=1 tcp_nup
>>
>> This samples a ton of tcp related stats and lets you plot them with
>> flent-gui, there are 110 other tests in the suite.
>>
>> A more complicated invocation can get qdisc stats from the router
>> involved, or iterate over CCs, or run at longer intervals. In general I
>> have found running tests for 5 minutes or longer reveal interesting
>> behaviors.
>>
>> flent -H $S --socket-stats -x --step-size=.05 -t
>> 1-flow-fq_starlink_noecn-${M}-c
>>
>> c=${CC} --te=cpu_stats_hosts=$R --te=netstat_hosts=$R -4
>> --te=qdisc_stats_hosts=
>>
>> $RS --te=qdisc_stats_interfaces=$RV --te=upload_streams=1
>> --te=cc_algo=$CC tcp_1
>>
>> up
>>
>>
>> pic below
>>
>> ---------- Forwarded message ---------
>> From: Eric xu <ericbin1224@gmail.com>
>> Date: Mon, Jul 5, 2021 at 12:27 AM
>> Subject: Re: [bbr-dev] BBR performance and optimization in cellular
>> wireless network with high delay jitter
>> To: BBR Development <bbr-dev@googlegroups.com>
>>
>>
>> Hi, Neal, thanks for your answer. I still  have two questions to discuss
>> with you after having read the two links you mentioned above.
>>
>> 1, As mentioned in the second link, BBR's throughput on high-jitter links
>> (especially in cellular) usually matches CUBIC,  as  demonstrated by the
>> following figure,
>> [image: 截图.PNG]
>> however, what we can infer from the figure is that, the mean
>> throughput(~14Mbps) is far from the available bw that is greater than
>> 20Mbps as showed in the cumulative distrubution figure. Since CUBIC back
>> off frequently in the LTE network with high error rate, but BBR does not
>> acquire much more throughput than CUBIC, the only reason I can imagine is
>> that BBR is highly affected by high delay jitter although cwnd provision is
>> applied.
>>
>> *Is my understanding right? and is there any optimization to further
>> improve thoughput in high bw and delay jitter scenario? *
>>
>> *2,  Is there any method to enhance BBR's adaptive ability to high bw and
>> delay jitter?  *In my opinion, in high bw jitter scenario, window based
>> bw undaption method is too slow to converge to the actual bottleneck bw,
>> when actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
>> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
>> anyway it is too slow, causing severe buffer-bloating.  In high delay
>> jitter scenario, BBR also will cause bw under utilizaion. *So,  what is
>> your latest optimizaion to improve the performance in such a high bw and
>> delay jitter network?*
>>
>> Thx a lot,
>> Eric
>> 在2021年7月4日星期日 UTC+8 下午9:59:57<Neal Cardwell> 写道：
>>
>>> Yes, BBR includes mechanisms to deal with this kind of jitter. Please
>>> see the March 2019 thread on this topic:
>>>   https://groups.google.com/g/bbr-dev/c/kBZaq98xCC4
>>>
>>> Our experience is that BBR's throughput on high-jitter links (cellular,
>>> wifi, DOCSIS, datacenter ethernet w/ GRO/LRO) is quite good at this point
>>> (usually matches CUBIC), and there are others who have documented this as
>>> well, e.g.:
>>>   http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
>>>
>>> best,
>>> neal
>>>
>>>
>>> On Sun, Jul 4, 2021 at 8:53 AM Eric xu <ericb...@gmail.com> wrote:
>>>
>>>> Hello, dear friends. I wonder there is any optimization to BBR ,  for
>>>> the reason of BBR low bandwidth utilization in wireless network with high
>>>> delay jitter.  In my opinion,  High delay jitter will cause BBR to
>>>> underestimate the BDP, thus limit the send rate even if  pipeline is
>>>> not full. Right?
>>>> So,  I wonder, *if there is any optimization to improve bw utilization
>>>> in wireless network with high delay jitter, even in 5G network? *Thx a
>>>> lot.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "BBR Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to bbr-dev+u...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to bbr-dev+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com
>> <https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>> --
>> Latest Podcast:
>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>>
>> Dave Täht CTO, TekLibre, LLC
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

[-- Attachment #1.2: Type: text/html, Size: 14278 bytes --]

[-- Attachment #2: 截图.PNG --]
[-- Type: image/png, Size: 52872 bytes --]

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Bob McMahon]

[-- Attachment #1.1.1: Type: text/plain, Size: 11069 bytes --]

Thanks for this. Looks interesting. Does ABC (love the acronym) try to
optimize throughput, latency or some combination, e.g. network power
(throughput/delay)?

Things like roams, seen energy over the last n milliseconds, and "I've lost
n TXOP arbitrations over time y" could also have influence on the control
feedback loop to TCP.  Not sure how TCP gets any of these stats.

None of this is end/end though. It's likely just the last/first hop which I
think is probably still useful to TCP designers - not sure.

Bob

On Tue, Jul 6, 2021 at 1:39 PM Neal Cardwell <ncardwell@google.com> wrote:

>
>
> On Tue, Jul 6, 2021 at 4:34 PM 'Bob McMahon' via BBR Development <
> bbr-dev@googlegroups.com> wrote:
>
>> Curious to the idea of wifi providing the opposite of ECN, i.e. my rate
>> selection algorithm just went from one stream to two stream so TCP speed up
>> your feedback loop decisions with a bias to faster?
>>
>
> That could make sense. That sounds a lot lot like the Accel-Brake Control
> work:
>
> https://www.usenix.org/conference/nsdi20/presentation/goyal
>
> ABC: A Simple Explicit Congestion Controller for Wireless Networks
>
> Authors:
> Prateesh Goyal, MIT CSAIL; Anup Agarwal, CMU; Ravi Netravali, UCLA;
> Mohammad Alizadeh and Hari Balakrishnan, MIT CSAIL
>
> Abstract:
> We propose Accel-Brake Control (ABC), a simple and deployable explicit
> congestion control protocol for network paths with time-varying wireless
> links. ABC routers mark each packet with an “accelerate” or “brake”, which
> causes senders to slightly increase or decrease their congestion windows.
> Routers use this feedback to quickly guide senders towards a desired target
> rate. ABC requires no changes to header formats or user devices, but
> achieves better performance than XCP. ABC is also incrementally deployable;
> it operates correctly when the bottleneck is a non-ABC router, and can
> coexist with non-ABC traffic sharing the same bottleneck link. We evaluate
> ABC using a Wi-Fi implementation and trace-driven emulation of cellular
> links. ABC achieves 30-40% higher throughput than Cubic+Codel for similar
> delays, and 2.2× lower delays than BBR on a Wi-Fi path. On cellular network
> paths, ABC achieves 50% higher throughput than Cubic+Codel.
>
> neal
>
> Bob
>>
>> On Mon, Jul 5, 2021 at 10:20 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>>> eric, I thought I'd forward this to one of the bufferbloat.net lists.
>>> Feel free to join. It would also be helpful to duplicate your test scripts
>>> here.
>>>
>>> I agree with you, bbr is way too slow here. We see it bloat massively in
>>> wifi also. I would like BBR to far more reactive than it is, and also for
>>> it to respond to RFC3168 ECN marks.
>>>
>>> You said:
>>>
>>> *" *In my opinion, in high bw jitter scenario, window based bw
>>> undaption method is too slow to converge to the actual bottleneck bw,  when
>>> actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
>>> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
>>> anyway it is too slow, causing severe buffer-bloating. "
>>>
>>> You might find the "flent" tool very useful for getting a more
>>> microscopic view as to what is going on.
>>>
>>> A test we would use to get a better picture of what is really going on
>>> would be
>>>
>>> flent -H netperf_test_server -x --socket-stats --step-size=.05 -t
>>> the_test_conditions --te=cc_algo=bbr --te=upload_streams=1 tcp_nup
>>>
>>> This samples a ton of tcp related stats and lets you plot them with
>>> flent-gui, there are 110 other tests in the suite.
>>>
>>> A more complicated invocation can get qdisc stats from the router
>>> involved, or iterate over CCs, or run at longer intervals. In general I
>>> have found running tests for 5 minutes or longer reveal interesting
>>> behaviors.
>>>
>>> flent -H $S --socket-stats -x --step-size=.05 -t
>>> 1-flow-fq_starlink_noecn-${M}-c
>>>
>>> c=${CC} --te=cpu_stats_hosts=$R --te=netstat_hosts=$R -4
>>> --te=qdisc_stats_hosts=
>>>
>>> $RS --te=qdisc_stats_interfaces=$RV --te=upload_streams=1
>>> --te=cc_algo=$CC tcp_1
>>>
>>> up
>>>
>>>
>>> pic below
>>>
>>> ---------- Forwarded message ---------
>>> From: Eric xu <ericbin1224@gmail.com>
>>> Date: Mon, Jul 5, 2021 at 12:27 AM
>>> Subject: Re: [bbr-dev] BBR performance and optimization in cellular
>>> wireless network with high delay jitter
>>> To: BBR Development <bbr-dev@googlegroups.com>
>>>
>>>
>>> Hi, Neal, thanks for your answer. I still  have two questions to discuss
>>> with you after having read the two links you mentioned above.
>>>
>>> 1, As mentioned in the second link, BBR's throughput on high-jitter
>>> links (especially in cellular) usually matches CUBIC,  as  demonstrated
>>> by the following figure,
>>> [image: 截图.PNG]
>>> however, what we can infer from the figure is that, the mean
>>> throughput(~14Mbps) is far from the available bw that is greater than
>>> 20Mbps as showed in the cumulative distrubution figure. Since CUBIC back
>>> off frequently in the LTE network with high error rate, but BBR does not
>>> acquire much more throughput than CUBIC, the only reason I can imagine is
>>> that BBR is highly affected by high delay jitter although cwnd provision is
>>> applied.
>>>
>>> *Is my understanding right? and is there any optimization to further
>>> improve thoughput in high bw and delay jitter scenario? *
>>>
>>> *2,  Is there any method to enhance BBR's adaptive ability to high bw
>>> and delay jitter?  *In my opinion, in high bw jitter scenario, window
>>> based bw undaption method is too slow to converge to the actual bottleneck
>>> bw,  when actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
>>> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
>>> anyway it is too slow, causing severe buffer-bloating.  In high delay
>>> jitter scenario, BBR also will cause bw under utilizaion. *So,  what is
>>> your latest optimizaion to improve the performance in such a high bw and
>>> delay jitter network?*
>>>
>>> Thx a lot,
>>> Eric
>>> 在2021年7月4日星期日 UTC+8 下午9:59:57<Neal Cardwell> 写道：
>>>
>>>> Yes, BBR includes mechanisms to deal with this kind of jitter. Please
>>>> see the March 2019 thread on this topic:
>>>>   https://groups.google.com/g/bbr-dev/c/kBZaq98xCC4
>>>>
>>>> Our experience is that BBR's throughput on high-jitter links (cellular,
>>>> wifi, DOCSIS, datacenter ethernet w/ GRO/LRO) is quite good at this point
>>>> (usually matches CUBIC), and there are others who have documented this as
>>>> well, e.g.:
>>>>   http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
>>>>
>>>> best,
>>>> neal
>>>>
>>>>
>>>> On Sun, Jul 4, 2021 at 8:53 AM Eric xu <ericb...@gmail.com> wrote:
>>>>
>>>>> Hello, dear friends. I wonder there is any optimization to BBR ,  for
>>>>> the reason of BBR low bandwidth utilization in wireless network with high
>>>>> delay jitter.  In my opinion,  High delay jitter will cause BBR to
>>>>> underestimate the BDP, thus limit the send rate even if  pipeline is
>>>>> not full. Right?
>>>>> So,  I wonder, *if there is any optimization to improve bw
>>>>> utilization in wireless network with high delay jitter, even in 5G network?
>>>>> *Thx a lot.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "BBR Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to bbr-dev+u...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "BBR Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to bbr-dev+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>> --
>>> Latest Podcast:
>>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>>>
>>> Dave Täht CTO, TekLibre, LLC
>>> _______________________________________________
>>> Make-wifi-fast mailing list
>>> Make-wifi-fast@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>>
>>
>> This electronic communication and the information and any files
>> transmitted with it, or attached to it, are confidential and are intended
>> solely for the use of the individual or entity to whom it is addressed and
>> may contain information that is confidential, legally privileged, protected
>> by privacy laws, or otherwise restricted from disclosure to anyone else. If
>> you are not the intended recipient or the person responsible for delivering
>> the e-mail to the intended recipient, you are hereby notified that any use,
>> copying, distributing, dissemination, forwarding, printing, or copying of
>> this e-mail is strictly prohibited. If you received this e-mail in error,
>> please return the e-mail to the sender, delete it from your computer, and
>> destroy any printed copy of it.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to bbr-dev+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.1.2: Type: text/html, Size: 16080 bytes --]

[-- Attachment #1.2: 截图.PNG --]
[-- Type: image/png, Size: 52872 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Jeremy Harris]

On 07/07/2021 00:49, Bob McMahon via Make-wifi-fast wrote:
>  ABC (love the acronym)

Conflict with Approrriate Byte Counting, though.

> Does ABC try to
> optimize throughput, latency or some combination, e.g. network power
> (throughput/delay)?

I read it as trying to optimize throughput of one specific link
(the one next to the detecting location).  They didn't discuss
what happens when there are two such on the path, and most of the
presentation assumed that this link was the limiting one.
-- 
Cheers,
   Jeremy

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 3452 bytes --]

I'm wondering if the whole idea that throughput as the exclusive vector for
optimization is a bit off. Even low latency high throughput designs has
throughput as a driving vector. We now have high occupancy low latency
(HULL) data center switches. We have to oversubscribe everything (and TCP
is more than happy to oblige.) As a group, we all do this because of
exactly why?

It reminds me of "weight" in the space program. Weight drove most every
decision due the energy cost required to get mass out of the influence of
earth's gravity. Structural integrity be damned - just make sure it has
less mass.

An app setting TCP_NODELAY on a socket could signal the end to end path
that the thing to optimize for this flow is latency, i.e. that the app
writing this socket isn't going to be sending lots of data. In this case,
the TCP feedback loop could adjust its vectors as an example. And TCP has
little clue of what the app is going to do, though it can guess at it (and
can guess wrong.)

I think WiFi/wireless (or no cables to act as wave guides) makes the
problem quite challenging for lots of reasons. I also think the silo'ing of
engineering both within companies and between companies adds to these
engineering limitations. What is the OSI layer but a human abstraction? Why
can't TCP be informed about PHY related things, and MACs be aware about TCP
related things?  I here the call of blasphemy, "It must be end knowledge
only!!"

Even the TCP assumption that a dropped packet is due to congestion is
flawed. It might be due to walking past a microwave that is irradiating a
bag of popcorn. Or the device orientation may change just enough that a new
spatial stream is now available, instantly doubling capacity.  Or more
radios can be used as CMOS radios are trending to abundance.

Just thinking out loud a bit. Sorry for the meandering in thoughts,
Bob






On Wed, Jul 7, 2021 at 1:27 AM Jeremy Harris <jgh@wizmail.org> wrote:

> On 07/07/2021 00:49, Bob McMahon via Make-wifi-fast wrote:
> >  ABC (love the acronym)
>
> Conflict with Approrriate Byte Counting, though.
>
> > Does ABC try to
> > optimize throughput, latency or some combination, e.g. network power
> > (throughput/delay)?
>
> I read it as trying to optimize throughput of one specific link
> (the one next to the detecting location).  They didn't discuss
> what happens when there are two such on the path, and most of the
> presentation assumed that this link was the limiting one.
> --
> Cheers,
>    Jeremy
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 4150 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Make-wifi-fast] Fwd: [bbr-dev] BBR performance and optimization in cellular wireless network with high delay jitter

[Neal Cardwell]

[-- Attachment #1.1: Type: text/plain, Size: 11715 bytes --]

On Tue, Jul 6, 2021 at 7:49 PM Bob McMahon <bob.mcmahon@broadcom.com> wrote:

> Thanks for this. Looks interesting. Does ABC (love the acronym) try to
> optimize throughput, latency or some combination, e.g. network power
> (throughput/delay)?
>

ABC tries to optimize both throughput and delay, though the paper does not
mention power specifically.


> Things like roams, seen energy over the last n milliseconds, and "I've
> lost n TXOP arbitrations over time y" could also have influence on the
> control feedback loop to TCP.  Not sure how TCP gets any of these stats.
>
> None of this is end/end though. It's likely just the last/first hop which
> I think is probably still useful to TCP designers - not sure.
>

The paper explains their scheme to convey the accelerate/break signals from
bottlenecks using the ECN bits, and then have the TCP receiver echo them to
the TCP sender using the historic "NS" bit.

neal



> Bob
>
> On Tue, Jul 6, 2021 at 1:39 PM Neal Cardwell <ncardwell@google.com> wrote:
>
>>
>>
>> On Tue, Jul 6, 2021 at 4:34 PM 'Bob McMahon' via BBR Development <
>> bbr-dev@googlegroups.com> wrote:
>>
>>> Curious to the idea of wifi providing the opposite of ECN, i.e. my rate
>>> selection algorithm just went from one stream to two stream so TCP speed up
>>> your feedback loop decisions with a bias to faster?
>>>
>>
>> That could make sense. That sounds a lot lot like the Accel-Brake
>> Control  work:
>>
>> https://www.usenix.org/conference/nsdi20/presentation/goyal
>>
>> ABC: A Simple Explicit Congestion Controller for Wireless Networks
>>
>> Authors:
>> Prateesh Goyal, MIT CSAIL; Anup Agarwal, CMU; Ravi Netravali, UCLA;
>> Mohammad Alizadeh and Hari Balakrishnan, MIT CSAIL
>>
>> Abstract:
>> We propose Accel-Brake Control (ABC), a simple and deployable explicit
>> congestion control protocol for network paths with time-varying wireless
>> links. ABC routers mark each packet with an “accelerate” or “brake”, which
>> causes senders to slightly increase or decrease their congestion windows.
>> Routers use this feedback to quickly guide senders towards a desired target
>> rate. ABC requires no changes to header formats or user devices, but
>> achieves better performance than XCP. ABC is also incrementally deployable;
>> it operates correctly when the bottleneck is a non-ABC router, and can
>> coexist with non-ABC traffic sharing the same bottleneck link. We evaluate
>> ABC using a Wi-Fi implementation and trace-driven emulation of cellular
>> links. ABC achieves 30-40% higher throughput than Cubic+Codel for similar
>> delays, and 2.2× lower delays than BBR on a Wi-Fi path. On cellular network
>> paths, ABC achieves 50% higher throughput than Cubic+Codel.
>>
>> neal
>>
>> Bob
>>>
>>> On Mon, Jul 5, 2021 at 10:20 AM Dave Taht <dave.taht@gmail.com> wrote:
>>>
>>>> eric, I thought I'd forward this to one of the bufferbloat.net lists.
>>>> Feel free to join. It would also be helpful to duplicate your test scripts
>>>> here.
>>>>
>>>> I agree with you, bbr is way too slow here. We see it bloat massively
>>>> in wifi also. I would like BBR to far more reactive than it is, and also
>>>> for it to respond to RFC3168 ECN marks.
>>>>
>>>> You said:
>>>>
>>>> *" *In my opinion, in high bw jitter scenario, window based bw
>>>> undaption method is too slow to converge to the actual bottleneck bw,  when
>>>> actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
>>>> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
>>>> anyway it is too slow, causing severe buffer-bloating. "
>>>>
>>>> You might find the "flent" tool very useful for getting a more
>>>> microscopic view as to what is going on.
>>>>
>>>> A test we would use to get a better picture of what is really going on
>>>> would be
>>>>
>>>> flent -H netperf_test_server -x --socket-stats --step-size=.05 -t
>>>> the_test_conditions --te=cc_algo=bbr --te=upload_streams=1 tcp_nup
>>>>
>>>> This samples a ton of tcp related stats and lets you plot them with
>>>> flent-gui, there are 110 other tests in the suite.
>>>>
>>>> A more complicated invocation can get qdisc stats from the router
>>>> involved, or iterate over CCs, or run at longer intervals. In general I
>>>> have found running tests for 5 minutes or longer reveal interesting
>>>> behaviors.
>>>>
>>>> flent -H $S --socket-stats -x --step-size=.05 -t
>>>> 1-flow-fq_starlink_noecn-${M}-c
>>>>
>>>> c=${CC} --te=cpu_stats_hosts=$R --te=netstat_hosts=$R -4
>>>> --te=qdisc_stats_hosts=
>>>>
>>>> $RS --te=qdisc_stats_interfaces=$RV --te=upload_streams=1
>>>> --te=cc_algo=$CC tcp_1
>>>>
>>>> up
>>>>
>>>>
>>>> pic below
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Eric xu <ericbin1224@gmail.com>
>>>> Date: Mon, Jul 5, 2021 at 12:27 AM
>>>> Subject: Re: [bbr-dev] BBR performance and optimization in cellular
>>>> wireless network with high delay jitter
>>>> To: BBR Development <bbr-dev@googlegroups.com>
>>>>
>>>>
>>>> Hi, Neal, thanks for your answer. I still  have two questions to
>>>> discuss with you after having read the two links you mentioned above.
>>>>
>>>> 1, As mentioned in the second link, BBR's throughput on high-jitter
>>>> links (especially in cellular) usually matches CUBIC,  as  demonstrated
>>>> by the following figure,
>>>> [image: 截图.PNG]
>>>> however, what we can infer from the figure is that, the mean
>>>> throughput(~14Mbps) is far from the available bw that is greater than
>>>> 20Mbps as showed in the cumulative distrubution figure. Since CUBIC back
>>>> off frequently in the LTE network with high error rate, but BBR does not
>>>> acquire much more throughput than CUBIC, the only reason I can imagine is
>>>> that BBR is highly affected by high delay jitter although cwnd provision is
>>>> applied.
>>>>
>>>> *Is my understanding right? and is there any optimization to further
>>>> improve thoughput in high bw and delay jitter scenario? *
>>>>
>>>> *2,  Is there any method to enhance BBR's adaptive ability to high bw
>>>> and delay jitter?  *In my opinion, in high bw jitter scenario, window
>>>> based bw undaption method is too slow to converge to the actual bottleneck
>>>> bw,  when actual bw imcreases, BBR needs at lease 8RTT to catch it, when it
>>>> decreases, BBR then needs at most 10 RTT to expire the previous bw maximum,
>>>> anyway it is too slow, causing severe buffer-bloating.  In high delay
>>>> jitter scenario, BBR also will cause bw under utilizaion. *So,  what
>>>> is your latest optimizaion to improve the performance in such a high bw and
>>>> delay jitter network?*
>>>>
>>>> Thx a lot,
>>>> Eric
>>>> 在2021年7月4日星期日 UTC+8 下午9:59:57<Neal Cardwell> 写道：
>>>>
>>>>> Yes, BBR includes mechanisms to deal with this kind of jitter. Please
>>>>> see the March 2019 thread on this topic:
>>>>>   https://groups.google.com/g/bbr-dev/c/kBZaq98xCC4
>>>>>
>>>>> Our experience is that BBR's throughput on high-jitter links
>>>>> (cellular, wifi, DOCSIS, datacenter ethernet w/ GRO/LRO) is quite good at
>>>>> this point (usually matches CUBIC), and there are others who have
>>>>> documented this as well, e.g.:
>>>>>   http://web.cs.wpi.edu/~claypool/papers/driving-bbr/
>>>>>
>>>>> best,
>>>>> neal
>>>>>
>>>>>
>>>>> On Sun, Jul 4, 2021 at 8:53 AM Eric xu <ericb...@gmail.com> wrote:
>>>>>
>>>>>> Hello, dear friends. I wonder there is any optimization to BBR ,  for
>>>>>> the reason of BBR low bandwidth utilization in wireless network with high
>>>>>> delay jitter.  In my opinion,  High delay jitter will cause BBR to
>>>>>> underestimate the BDP, thus limit the send rate even if  pipeline is
>>>>>> not full. Right?
>>>>>> So,  I wonder, *if there is any optimization to improve bw
>>>>>> utilization in wireless network with high delay jitter, even in 5G network?
>>>>>> *Thx a lot.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "BBR Development" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to bbr-dev+u...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/bbr-dev/bc675976-45f5-4ef6-b337-e693e4d21c6en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "BBR Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to bbr-dev+unsubscribe@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/bbr-dev/303da319-87dd-46c0-928f-cad7f93835f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>>
>>>> --
>>>> Latest Podcast:
>>>> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>>>>
>>>> Dave Täht CTO, TekLibre, LLC
>>>> _______________________________________________
>>>> Make-wifi-fast mailing list
>>>> Make-wifi-fast@lists.bufferbloat.net
>>>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>>>
>>>
>>> This electronic communication and the information and any files
>>> transmitted with it, or attached to it, are confidential and are intended
>>> solely for the use of the individual or entity to whom it is addressed and
>>> may contain information that is confidential, legally privileged, protected
>>> by privacy laws, or otherwise restricted from disclosure to anyone else. If
>>> you are not the intended recipient or the person responsible for delivering
>>> the e-mail to the intended recipient, you are hereby notified that any use,
>>> copying, distributing, dissemination, forwarding, printing, or copying of
>>> this e-mail is strictly prohibited. If you received this e-mail in error,
>>> please return the e-mail to the sender, delete it from your computer, and
>>> destroy any printed copy of it.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "BBR Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to bbr-dev+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvroqyg22Ax6ChOPYicYTjm_GnFapm8iX_5ae5EbvVEBzQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 17217 bytes --]

[-- Attachment #2: 截图.PNG --]
[-- Type: image/png, Size: 52872 bytes --]