I'm seeing something interesting using fq_codel, protectcli vault and opnsense. The background load going active cause the responses per second to increase and the bb times to go from 15 ms to 10 ms. I added four upstream TCP streams (not shown) during the yellow time slots and the bounceback delay became lesser!

rjmcmahon@fedora:~/Code/bloat/iperf2-code$ iperf -c <my-cloud-server> -i 1 -e --bounceback  -t 60 --hide-ips
------------------------------------------------------------
Client connecting to (**hidden**), TCP port 5001 with pid 87070 (1/0 flows/load)
Bounceback test (req/reply size = 100 Byte/ 100 Byte) (server hold req=0 usecs & tcp_quickack)
Bursting request 10 times every 1.00 second(s)
TCP congestion control using cubic
TOS defaults to 0x0 (dscp=0,ecn=0) and nodelay (Nagle off)
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  1] local *.*.*.59%enp4s0 port 56380 connected with *.*.*.123 port 5001 (bb w/quickack req/reply/hold=100/100/0) (icwnd/mss/irtt=14/1448/10624) (ct=10.71 ms) on 2024-12-01 13:49:05.856 (PST)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd(pkts)/RTT(var)   Tx/Rx bytes  RPS(avg)
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=15.609/11.041/19.933/2.256 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=64
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.806/11.047/16.801/1.918 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=68
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=13.946/10.260/16.458/2.416 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=72
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.997/10.714/16.314/1.579 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=67
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.612/11.687/15.938/1.504 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=68
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.919/11.177/15.786/1.339 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=67
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.381/10.485/16.311/2.250 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=70
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.221/10.669/17.170/2.404 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=70
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.417/11.493/16.045/1.400 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=69
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=12.815/11.467/13.680/0.694 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=78
[  1] 10.00-11.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.200/9.071/11.488/0.903 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=98
[  1] 11.00-12.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.236/8.838/11.345/0.807 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=98
[  1] 12.00-13.00 sec  1.95 KBytes  16.0 Kbits/sec    10=9.560/8.512/11.301/0.782 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=105
[  1] 13.00-14.00 sec  1.95 KBytes  16.0 Kbits/sec    10=9.939/9.245/10.944/0.552 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=101
[  1] 14.00-15.00 sec  1.95 KBytes  16.0 Kbits/sec    10=13.722/12.576/15.569/0.855 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=73
[  1] 15.00-16.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.856/10.674/12.693/0.624 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=84
[  1] 16.00-17.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.260/8.872/11.085/0.652 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=97
[  1] 17.00-18.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.044/9.152/11.054/0.755 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=100
[  1] 18.00-19.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.199/8.724/11.377/0.928 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=98
[  1] 19.00-20.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.831/11.103/18.682/1.886 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=67
[  1] 20.00-21.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.657/11.171/15.684/1.574 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=68
[  1] 21.00-22.00 sec  1.95 KBytes  16.0 Kbits/sec    10=15.283/10.757/18.079/2.311 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=65
[  1] 22.00-23.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.960/10.551/16.555/1.991 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=67
[  1] 23.00-24.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.699/10.847/16.979/1.815 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=68
[  1] 24.00-25.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.240/10.476/15.544/1.733 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=70
[  1] 25.00-26.00 sec  1.95 KBytes  16.0 Kbits/sec    10=14.230/10.513/15.785/1.972 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=70
[  1] 26.00-27.00 sec  1.95 KBytes  16.0 Kbits/sec    10=13.530/10.297/15.595/2.433 ms    0   0K/(0)/0(0) us  Tx=1000 Byte Rx=1000 Byte  RPS=74