Re: [Bloat] Win10 Updates vs cake

Re: [Bloat] Win10 Updates vs cake

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]


> On Dec 22, 2017, at 4:12 AM, Mario Hock <mario.hock@kit.edu <mailto:mario.hock@kit.edu>> wrote:
> 
> Can you also track packet loss rates / packet loss probability? And also 
> throughput/progress of the Windows update connections?

Oh, yes... I meant to add in the initial report that during times of Win10 downloads, packet loss for all services went to pieces.

I run Intermapper at home to monitor uptime/packet loss/performance of lots of devices across the net. During those downloads, packet loss frequently jumped to 5-6% for most devices.

Thanks again,

Rich

[-- Attachment #2: Type: text/html, Size: 2465 bytes --]

[Bloat] Win10 Updates vs cake

[Rich Brown]

I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+ connection through Fairpoint. The modem stats page shows its "attainable rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:

Download: 7000 (kbps)
Upload: 925 
Queue Disc: Cake/piece_of_cake.qos
Link Layer: ATM/44 bytes overhead
Advanced Options: default

I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.

Experiments I have tried:

- Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)

- Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.

- The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs.  I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.

- This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.

Any thoughts? What other experiments should I make? Thanks!

Rich

Re: [Bloat] Win10 Updates vs cake

[Ryan Mounce]

I've experienced this recently myself. In my case I have a 100Mbps
link from my ISP and their shaper will queue up to about 120ms worth
of packets (on top of the ~10ms baseline latency). I run cake in
ingress mode at 99.2Mbps, which has normally been enough to keep
everything in check and keep my ISP's queue empty at least in the
steady state.

Boot up a Windows 10 PC that's been unused for a few months, let it
update and bam! 100-130ms RTT, family member's Netflix stream in the
next room stalls completely, and running a quick speedtest on a
different machine (that should get ~50% share of the link under normal
circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
link! I performed a quick pcap while this was happening and determined
that Windows Update had started on the order of 120 parallel HTTP
downloads from 2 different Akamai cache IPs (within my ISP's network
20ms away).

The 20ms jump in latency in your case just indicates that there is a
small buffer in your DSLAM, however it is still being flooded by the
parallel transfers from the CDN.

Windows 10 probably deserves most of the blame for opening so many
parallel connections, however I think there is also some concern here
with Akamai's FastTCP not responding to congestion signals.

Regards,
Ryan Mounce


On 22 December 2017 at 12:31, Rich Brown <richb.hanover@gmail.com> wrote:
> I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+ connection through Fairpoint. The modem stats page shows its "attainable rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:
>
> Download: 7000 (kbps)
> Upload: 925
> Queue Disc: Cake/piece_of_cake.qos
> Link Layer: ATM/44 bytes overhead
> Advanced Options: default
>
> I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
>
> Experiments I have tried:
>
> - Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
>
> - Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
>
> - The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs.  I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
>
> - This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
>
> Any thoughts? What other experiments should I make? Thanks!
>
> Rich
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Win10 Updates vs cake

[Ryan Mounce]

Some further reading on the issue

https://www.mail-archive.com/nanog@nanog.org/msg87442.html
https://forums.whirlpool.net.au/archive/2530363

Regards,
Ryan Mounce


On 22 December 2017 at 13:25, Ryan Mounce <ryan@mounce.com.au> wrote:
> I've experienced this recently myself. In my case I have a 100Mbps
> link from my ISP and their shaper will queue up to about 120ms worth
> of packets (on top of the ~10ms baseline latency). I run cake in
> ingress mode at 99.2Mbps, which has normally been enough to keep
> everything in check and keep my ISP's queue empty at least in the
> steady state.
>
> Boot up a Windows 10 PC that's been unused for a few months, let it
> update and bam! 100-130ms RTT, family member's Netflix stream in the
> next room stalls completely, and running a quick speedtest on a
> different machine (that should get ~50% share of the link under normal
> circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
> link! I performed a quick pcap while this was happening and determined
> that Windows Update had started on the order of 120 parallel HTTP
> downloads from 2 different Akamai cache IPs (within my ISP's network
> 20ms away).
>
> The 20ms jump in latency in your case just indicates that there is a
> small buffer in your DSLAM, however it is still being flooded by the
> parallel transfers from the CDN.
>
> Windows 10 probably deserves most of the blame for opening so many
> parallel connections, however I think there is also some concern here
> with Akamai's FastTCP not responding to congestion signals.
>
> Regards,
> Ryan Mounce
>
>
> On 22 December 2017 at 12:31, Rich Brown <richb.hanover@gmail.com> wrote:
>> I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+ connection through Fairpoint. The modem stats page shows its "attainable rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:
>>
>> Download: 7000 (kbps)
>> Upload: 925
>> Queue Disc: Cake/piece_of_cake.qos
>> Link Layer: ATM/44 bytes overhead
>> Advanced Options: default
>>
>> I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
>>
>> Experiments I have tried:
>>
>> - Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
>>
>> - Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
>>
>> - The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs.  I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
>>
>> - This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
>>
>> Any thoughts? What other experiments should I make? Thanks!
>>
>> Rich
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Win10 Updates vs cake

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 135 bytes --]

Using the latest version of cake (cobalt branch) and - most importantly -
turning on ingress mode should help here.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 174 bytes --]

Re: [Bloat] Win10 Updates vs cake

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Those threads are about 18 months old.  Is it still happening the same way,
or has it become a large number of well-behaved TCP streams?  Cake in
ingress mode can cope with the latter pretty well.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 260 bytes --]

Re: [Bloat] Win10 Updates vs cake

[Ryan Mounce]

On 22 December 2017 at 15:56, Jonathan Morton <chromatix99@gmail.com> wrote:
> Those threads are about 18 months old.  Is it still happening the same way,
> or has it become a large number of well-behaved TCP streams?  Cake in
> ingress mode can cope with the latter pretty well.

At least in my case I am using the latest revision of the cobalt
branch and running cake in ingress mode.

>
> - Jonathan Morton

Re: [Bloat] Win10 Updates vs cake

[Kevin Darbyshire-Bryant]

[-- Attachment #1: Type: text/plain, Size: 1442 bytes --]



> On 22 Dec 2017, at 03:57, Ryan Mounce <ryan@mounce.com.au> wrote:
> 
> Some further reading on the issue
> 
> https://www.mail-archive.com/nanog@nanog.org/msg87442.html
> https://forums.whirlpool.net.au/archive/2530363
> 
> Regards,
> Ryan Mounce
> 

Chaps,

As a result of the following conversation between Dave & myself.

>> Opinions on doing a PR for LEDE so they can enjoy(/crash) the fruits of our latest cobalt.  They’re currently missing ingress mode & ack filtering filtering toys.
> 
> Nah. Let's get it into net-next first this time. I would hope for some
> good comments.


LEDE master until very recently (a week) still had old versions of cake & tc.  Since the upstreaming appears to have stalled on a technical issue, I took the opportunity to update LEDE master.  I had submitted patches for 17.01 yesterday, after having let master ’stew’ a bit in case of problems,  but subsequently pulled them this morning due to new arrivals (and discrepancies) in the cake repo.

As of this email - LEDE master has a full featured cake/cobalt but with a double overhead bug.  LEDE 17.01 has a much older, pre ingress mode, pre ack filter version.

Once the cake repo is sorted out I can redo & resubmit the patches for both master & 1701 - and with a bit of luck we’ll all be in an even place again.

Cheers,

Kevin D-B
+44 7947 355344

012C ACB2 28C6 C53E 9775  9123 B3A2 389B 9DE2 334A


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Bloat] Win10 Updates vs cake

[Rich Brown]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

Kevin,

> Once the cake repo is sorted out I can redo & resubmit the patches for both master & 1701 - and with a bit of luck we’ll all be in an even place again.

Thanks for this update. I'm subscribed to the cake github stream, so I hear about changes there.

If I understand correctly, after those patches are resubmitted, 17.01 will have the newest cake with all bells/whistles. 

Would you post a note here to a) let us know when they're available, and b) how "normal people" (that is, non-developers) can get this newest code? (I'm happy to post instructions to the forum if no one beats me to it.) Thanks.

Rich

[-- Attachment #2: Type: text/html, Size: 1752 bytes --]

Re: [Bloat] Win10 Updates vs cake

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 3860 bytes --]

On Thu, Dec 21, 2017 at 8:55 PM, Ryan Mounce <ryan@mounce.com.au> wrote:

> I've experienced this recently myself. In my case I have a 100Mbps
> link from my ISP and their shaper will queue up to about 120ms worth
> of packets (on top of the ~10ms baseline latency). I run cake in
> ingress mode at 99.2Mbps, which has normally been enough to keep
> everything in check and keep my ISP's queue empty at least in the
> steady state.
>
> Boot up a Windows 10 PC that's been unused for a few months, let it
> update and bam! 100-130ms RTT, family member's Netflix stream in the
> next room stalls completely, and running a quick speedtest on a
> different machine (that should get ~50% share of the link under normal
> circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
> link! I performed a quick pcap while this was happening and determined
> that Windows Update had started on the order of 120 parallel HTTP
> downloads from 2 different Akamai cache IPs (within my ISP's network
> 20ms away).
>
> The 20ms jump in latency in your case just indicates that there is a
> small buffer in your DSLAM, however it is still being flooded by the
> parallel transfers from the CDN.
>
> Windows 10 probably deserves most of the blame for opening so many
> parallel connections, however I think there is also some concern here
> with Akamai's FastTCP not responding to congestion signals.
>

Without packet pacing, this is impossible. All TCP implementations have a
minimum window size of 2 segments. Assuming 1500 bytes per segment, that's
3,000 bytes. Divide by RTT of 20ms for 1.2Mb/s minimum bandwidth. Because
there is no pacing, aka delaying packets, the packets are sent as soon as
the ACK hits. With 120 connections, that's 144Mb/s. It can't respond to
congestion by slowing itself down because it's already as slow as it can
go. Packet pacing is very new and is slowly being standardized. Windows
just needs to reduce the number of connections.


> Regards,
> Ryan Mounce
>
>
> On 22 December 2017 at 12:31, Rich Brown <richb.hanover@gmail.com> wrote:
> > I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+
> connection through Fairpoint. The modem stats page shows its "attainable
> rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:
> >
> > Download: 7000 (kbps)
> > Upload: 925
> > Queue Disc: Cake/piece_of_cake.qos
> > Link Layer: ATM/44 bytes overhead
> > Advanced Options: default
> >
> > I have noticed that Win10 updates cause the network connection to become
> unusable for other services/people, as if I had bufferbloat. But ping times
> remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
> >
> > Experiments I have tried:
> >
> > - Setting download speed to 5000 makes the connection usable for other
> people, although the ping times remain about the same (40-50 msec)
> >
> > - Setting the download speed to 8600 still keeps ping times down, but
> that really harms other people's performance.
> >
> > - The link rates (download and upload) seem to track the SQM setting,
> measured with both YAMon and the built-in real-time graphs.  I get ~6,000
> kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I
> get ~8500 kbps after setting download to 8600.
> >
> > - This doesn't seem to happen when I'm downloading other kinds of files
> (I haven't tried torrenting files...) Downloading non-Win10 update files
> seems to leave the connection in a fairly responsive state.
> >
> > Any thoughts? What other experiments should I make? Thanks!
> >
> > Rich
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 5029 bytes --]

Re: [Bloat] Win10 Updates vs cake

[Ryan Mounce]

On 29 December 2017 at 04:18, Benjamin Cronce <bcronce@gmail.com> wrote:
>
>
> On Thu, Dec 21, 2017 at 8:55 PM, Ryan Mounce <ryan@mounce.com.au> wrote:
>>
>> I've experienced this recently myself. In my case I have a 100Mbps
>> link from my ISP and their shaper will queue up to about 120ms worth
>> of packets (on top of the ~10ms baseline latency). I run cake in
>> ingress mode at 99.2Mbps, which has normally been enough to keep
>> everything in check and keep my ISP's queue empty at least in the
>> steady state.
>>
>> Boot up a Windows 10 PC that's been unused for a few months, let it
>> update and bam! 100-130ms RTT, family member's Netflix stream in the
>> next room stalls completely, and running a quick speedtest on a
>> different machine (that should get ~50% share of the link under normal
>> circumstances with dual-dsthost) yields about 1.2Mbps on a 100Mbps
>> link! I performed a quick pcap while this was happening and determined
>> that Windows Update had started on the order of 120 parallel HTTP
>> downloads from 2 different Akamai cache IPs (within my ISP's network
>> 20ms away).
>>
>> The 20ms jump in latency in your case just indicates that there is a
>> small buffer in your DSLAM, however it is still being flooded by the
>> parallel transfers from the CDN.
>>
>> Windows 10 probably deserves most of the blame for opening so many
>> parallel connections, however I think there is also some concern here
>> with Akamai's FastTCP not responding to congestion signals.
>
>
> Without packet pacing, this is impossible. All TCP implementations have a
> minimum window size of 2 segments. Assuming 1500 bytes per segment, that's
> 3,000 bytes. Divide by RTT of 20ms for 1.2Mb/s minimum bandwidth. Because
> there is no pacing, aka delaying packets, the packets are sent as soon as
> the ACK hits. With 120 connections, that's 144Mb/s. It can't respond to
> congestion by slowing itself down because it's already as slow as it can go.
> Packet pacing is very new and is slowly being standardized. Windows just
> needs to reduce the number of connections.


This can't be the reason, under this extreme load the queue of my
ISP's downstream shaper is permanently full and inducing at least an
additional 100ms of latency. Assuming an RTT to Akamai of 100ms this
gives a minimum bandwidth of 0.24Mbps, or 28.8Mbps for 120
connections. That is less than 1/3 of my link capacity.

Regards,
Ryan Mounce

>
>>
>> Regards,
>> Ryan Mounce
>>
>>
>> On 22 December 2017 at 12:31, Rich Brown <richb.hanover@gmail.com> wrote:
>> > I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+
>> > connection through Fairpoint. The modem stats page shows its "attainable
>> > rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:
>> >
>> > Download: 7000 (kbps)
>> > Upload: 925
>> > Queue Disc: Cake/piece_of_cake.qos
>> > Link Layer: ATM/44 bytes overhead
>> > Advanced Options: default
>> >
>> > I have noticed that Win10 updates cause the network connection to become
>> > unusable for other services/people, as if I had bufferbloat. But ping times
>> > remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
>> >
>> > Experiments I have tried:
>> >
>> > - Setting download speed to 5000 makes the connection usable for other
>> > people, although the ping times remain about the same (40-50 msec)
>> >
>> > - Setting the download speed to 8600 still keeps ping times down, but
>> > that really harms other people's performance.
>> >
>> > - The link rates (download and upload) seem to track the SQM setting,
>> > measured with both YAMon and the built-in real-time graphs.  I get ~6,000
>> > kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I
>> > get ~8500 kbps after setting download to 8600.
>> >
>> > - This doesn't seem to happen when I'm downloading other kinds of files
>> > (I haven't tried torrenting files...) Downloading non-Win10 update files
>> > seems to leave the connection in a fairly responsive state.
>> >
>> > Any thoughts? What other experiments should I make? Thanks!
>> >
>> > Rich
>> > _______________________________________________
>> > Bloat mailing list
>> > Bloat@lists.bufferbloat.net
>> > https://lists.bufferbloat.net/listinfo/bloat
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>

Re: [Bloat] Win10 Updates vs cake

[Mario Hock]

Hi,

except from buffer bloat performance can also be degraded by an 
increased packet loss probability.

Can you also track packet loss rates / packet loss probability? And also 
throughput/progress of the Windows update connections?

If the reason actually is that FastTCP does not respond to congestion in 
a fair manner (as Ryan Mounce suggested) than you should see high packet 
loss (for all connections), high throughput for the Windows update 
connections and very low throughput for regular TCP connections.

Also, it would be interesting to see what happens to fixed bit rate UDP 
streams.

Best, Mario


Am 22.12.2017 um 03:01 schrieb Rich Brown:
> I'm using LEDE 17.01.4 on my Archer C7v2. I have a 7mbps/768kbps ADSL2+ connection through Fairpoint. The modem stats page shows its "attainable rates" (kbps): 13330/1272 and Rates: 8271/1181. My SQM settings are:
> 
> Download: 7000 (kbps)
> Upload: 925
> Queue Disc: Cake/piece_of_cake.qos
> Link Layer: ATM/44 bytes overhead
> Advanced Options: default
> 
> I have noticed that Win10 updates cause the network connection to become unusable for other services/people, as if I had bufferbloat. But ping times remain stable - they jump from ~20-22 msec unloaded to 40-50 msec.
> 
> Experiments I have tried:
> 
> - Setting download speed to 5000 makes the connection usable for other people, although the ping times remain about the same (40-50 msec)
> 
> - Setting the download speed to 8600 still keeps ping times down, but that really harms other people's performance.
> 
> - The link rates (download and upload) seem to track the SQM setting, measured with both YAMon and the built-in real-time graphs.  I get ~6,000 kbps with a 7000 download setting, I got ~3,000kbps at the 5000 setting. I get ~8500 kbps after setting download to 8600.
> 
> - This doesn't seem to happen when I'm downloading other kinds of files (I haven't tried torrenting files...) Downloading non-Win10 update files seems to leave the connection in a fairly responsive state.
> 
> Any thoughts? What other experiments should I make? Thanks!
> 
> Rich
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>