[Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Dave Taht]

I just downloaded and seeded 4 popular torrents overnight using  the
latest version of the transmission-gtk client. I have not paid much
attention to this app or protocol of late (about 2.5 years since last
I did this), I got a little sparked by wanting to test cdg, but did
not get that far.

Some egress stats this morning (fq_codel on the uplink)

bytes 32050522339
packets 3379478
dropped 702799
percent 20.80%
maxpacket 28614

Some notes:

1) The link stayed remarkably usable:

http://snapon.lab.bufferbloat.net/~d/withtorrent/vs64connectedpeers.png

This graph shows what happened when one of the 4 torrents completed.

The percentage of bandwidth the uplink on this test got was a bit
larger than I expected.

Subjectively, web browsing was slower but usable, and my other normal
usages (like ssh and mosh and google music over quic) were seemingly
unaffected. (latency for small flows stayed pretty flat)

2) even with 69 peers going at peak, I generally did not get anywhere
near saturating the 100mbit downlink with torrent alone.

3) Offloads are a pita. Merely counting "packets" here does not show
the real truth of what's going on (max "packet" of 28614 bytes!?), so
linux, benchmarkers, and so on, should also be counting bytes dropped
these days. (cake does peeling of superpackets but I was not testing
that, and it too does not return bytes dropped)

4) *All* the traffic was udp. (uTP) Despite ipv6 being enabled (with
two source specific ipv6 ips), I did not see any ipv6 peers connect.
Bug? Death of torrent over ipv6? Blocking? What?

5) transmission-generated uplink traffic seemed "bursty", but I did
not tear apart the data or code. I will track queue length next time.

6) Although transmission seems to support setting the diffserv bytes,
it did not do so on the udp marked traffic. I think that was a tcp
only option. Also it is incorrect for ipv6 (not using IPV6_TCLASS). I
had figured (before starting the test) that this was going to be a
good test of cake's diffserv support. Sigh. Is there some other client
I could use?

7) transmission ate a metric ton of cpu (30% on a i3) at these speeds.

8) My (cable) link actually is 140mbit down, 11 up. I did not much
care for asymmetric networks when the ratios were 6x1, so 13x1 is way
up there....

Anyway, 20% packet loss of the "right" packets was survivable. I will
subject myself to the same test on other fq or aqms. And, if I can
force myself to, with no aqm or fq. For SCIENCE!

Attention, DMCA lawyers: Please send takedown notices to
bufferbloat-research@/dev/null.org . One of the things truly
astonishing about this is that in 12 hours in one night I downloaded
more stuff than I could ever watch (mp4) or listen to (even in flac
format) in several days of dedicated consumption. And it all just got
rm -rf'd. It occurs to me there is a human upper bound to how much
data one would ever want to consume, and we cracked that limit at
20mbit, with only 4k+ video driving demand any harder. When we started
bufferbloat.net 20mbit downlinks were the best you could easily get.



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And: What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

>
>
> 7) transmission ate a metric ton of cpu (30% on a i3) at these speeds.
>
> 8) My (cable) link actually is 140mbit down, 11 up. I did not much
> care for asymmetric networks when the ratios were 6x1, so 13x1 is way
> up there....
>
> Anyway, 20% packet loss of the "right" packets was survivable. I will
> subject myself to the same test on other fq or aqms. And, if I can
> force myself to, with no aqm or fq. For SCIENCE!
>
> Attention, DMCA lawyers: Please send takedown notices to
> bufferbloat-research@/dev/null.org . One of the things truly
> astonishing about this is that in 12 hours in one night I downloaded
> more stuff than I could ever watch (mp4) or listen to (even in flac
> format) in several days of dedicated consumption. And it all just got
> rm -rf'd. It occurs to me there is a human upper bound to how much
> data one would ever want to consume, and we cracked that limit at
> 20mbit, with only 4k+ video driving demand any harder. When we started
> bufferbloat.net 20mbit downlinks were the best you could easily get.

Linux ISOs are a great way to saturate your download. I have enough
downloaded that while seeding, I can sustain over 10Mb/s, but don't expect
to saturate your upload since they're already heavily seeded, but less so
since I stopped recently.

[-- Attachment #2: Type: text/html, Size: 2386 bytes --]

Re: [Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Dave Taht]

sometimes I pick the wrong week to actually try to benchmark a
protocol in the wild.

https://torrentfreak.com/popular-torrents-being-sabotaged-by-ipv6-peer-flood-150619/

On Fri, Jun 19, 2015 at 9:01 AM, Dave Taht <dave.taht@gmail.com> wrote:
> I just downloaded and seeded 4 popular torrents overnight using  the
> latest version of the transmission-gtk client. I have not paid much
> attention to this app or protocol of late (about 2.5 years since last
> I did this), I got a little sparked by wanting to test cdg, but did
> not get that far.
>
> Some egress stats this morning (fq_codel on the uplink)
>
> bytes 32050522339
> packets 3379478
> dropped 702799
> percent 20.80%
> maxpacket 28614
>
> Some notes:
>
> 1) The link stayed remarkably usable:
>
> http://snapon.lab.bufferbloat.net/~d/withtorrent/vs64connectedpeers.png
>
> This graph shows what happened when one of the 4 torrents completed.
>
> The percentage of bandwidth the uplink on this test got was a bit
> larger than I expected.
>
> Subjectively, web browsing was slower but usable, and my other normal
> usages (like ssh and mosh and google music over quic) were seemingly
> unaffected. (latency for small flows stayed pretty flat)
>
> 2) even with 69 peers going at peak, I generally did not get anywhere
> near saturating the 100mbit downlink with torrent alone.
>
> 3) Offloads are a pita. Merely counting "packets" here does not show
> the real truth of what's going on (max "packet" of 28614 bytes!?), so
> linux, benchmarkers, and so on, should also be counting bytes dropped
> these days. (cake does peeling of superpackets but I was not testing
> that, and it too does not return bytes dropped)
>
> 4) *All* the traffic was udp. (uTP) Despite ipv6 being enabled (with
> two source specific ipv6 ips), I did not see any ipv6 peers connect.
> Bug? Death of torrent over ipv6? Blocking? What?
>
> 5) transmission-generated uplink traffic seemed "bursty", but I did
> not tear apart the data or code. I will track queue length next time.
>
> 6) Although transmission seems to support setting the diffserv bytes,
> it did not do so on the udp marked traffic. I think that was a tcp
> only option. Also it is incorrect for ipv6 (not using IPV6_TCLASS). I
> had figured (before starting the test) that this was going to be a
> good test of cake's diffserv support. Sigh. Is there some other client
> I could use?
>
> 7) transmission ate a metric ton of cpu (30% on a i3) at these speeds.
>
> 8) My (cable) link actually is 140mbit down, 11 up. I did not much
> care for asymmetric networks when the ratios were 6x1, so 13x1 is way
> up there....
>
> Anyway, 20% packet loss of the "right" packets was survivable. I will
> subject myself to the same test on other fq or aqms. And, if I can
> force myself to, with no aqm or fq. For SCIENCE!
>
> Attention, DMCA lawyers: Please send takedown notices to
> bufferbloat-research@/dev/null.org . One of the things truly
> astonishing about this is that in 12 hours in one night I downloaded
> more stuff than I could ever watch (mp4) or listen to (even in flac
> format) in several days of dedicated consumption. And it all just got
> rm -rf'd. It occurs to me there is a human upper bound to how much
> data one would ever want to consume, and we cracked that limit at
> 20mbit, with only 4k+ video driving demand any harder. When we started
> bufferbloat.net 20mbit downlinks were the best you could easily get.
>
>
>
> --
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And: What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Dave Taht]

On Fri, Jun 19, 2015 at 3:14 PM, Benjamin Cronce <bcronce@gmail.com> wrote:

> Linux ISOs are a great way to saturate your download. I have enough
> downloaded that while seeding, I can sustain over 10Mb/s, but don't expect
> to saturate your upload since they're already heavily seeded, but less so
> since I stopped recently.

I went for a representative sample of what torrent gets used for by
ordinary users, randomly picking 4 torrents in the top 10 of music,
tv, movies, and... um, er, ah, pr0n.

...FOR SCIENCE! :)

Going for a linux distro would have yielded a different result. I used
the gutenberg torrent and linux distros primarily in those earlier
tests years ago, and I felt that the data I got then was probably
skewed by that towards linux's behaviors.

The major difference between then and now was no tcp whatsoever, and
no ipv6, where before I had seen a lot of tcp and ipv6. I can redo
those tests at some future point. I think the tcp absence here was
partially upnp not working correctly in my setup last night. I have
been quite concerned that as IW10 moved out there generically that
that would be bad.... and perhaps the lack of ipv6 was due to the
flooding ipv6 issue I posted earlier.

So I'll think about how to go about it properly harder... after I
patch transmission to tos mark packets correctly... (my original
intent was to watch cake do classification) but I would argue to do it
fairly right, consistently, over time, while altering other major
variables like qdisc, would be to continue pulling from the top 10
categories the public is pulling from.

suggestions, anyone?

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

[Bloat] BitTorrent and IPv6 [was: tackling torrent...]

[Juliusz Chroboczek]

[Removed AQM from CC.]

> 4) *All* the traffic was udp. (uTP)

Yes, modern BitTorrent implementations prefer µTP to TCP.  Unfortunately, 
the implementation of µTP in Transmission doesn't perform PMTUD, and gets 
packet sizes wrong most of the time.  (Yes, I spoke about this to Arvid.  
No, he wasn't particularly concerned.)

> Despite ipv6 being enabled (with two source specific ipv6 ips), I did 
> not see any ipv6 peers connect.  Bug? Death of torrent over ipv6? 
> Blocking? What?

Please check that you allow incoming IPv6 to your peer, for both TCP and 
UDP.  Obviously, setting port forwarding for IPv4 is not enough for that — 
you'll need to explicitly set up a hole in your IPv6 firewall.  If using 
OpenWRT:

     config rule
             option target 'ACCEPT'
             option src 'wan'
             option name 'Transmission-v6'
             option family 'ipv6'
             option dest '*'
             option dest_port '12345'

for the right value of 12345.

If you set the logging level in Transmission to Debug and let it run for 
twenty minutes or so, you should see periodic "Starting IPv6 DHT announce 
(... good ...)".  If you keep seeing "firewalled" instead of "good", 
you've got an issue.

Once you're positive you don't have firewalling issues, the following 
might be of interest.

I've done quite a bit of work a few years ago to improve IPv6 support in 
Transmission.  The main problem is how to learn the IPv6 addresses of 
your peers:

1. Many trackers don't support IPv6.

2. IPv6 is supported by PEX (gossip, peer-exchange).  I'm not sure if 
other peers honour IPv6 PEX when connected over IPv4, but Transmission 
certainly does.

3. I've implemented a protocol extension that allows peers connected over 
IPv4 to exchange their IPv6 addresses.  This is only marginally useful, 
but can sometimes allow bootstrapping IPv6 PEX without tracker support.

4. Most importantly, I've implemented IPv6 support for the DHT (BEP-32) [1].  
BEP-32 is now implemented by Transmission, Vuze, Tixati, KTorrent and 
Shareaza.

Points (1) through (3) are interoperable with µTorrent/BitTorrent and 
libtorrent, which constitute the vast majority of deployed BitTorrent 
peers.  Unfortunately, µTorrent and libtorrent do not implement BEP-32, 
and since these implementations constitute the vast majority of deployed 
BitTorrent peers outside China, this seriously limits the effectiveness of 
the IPv6 DHT.  (Yes, I've spoken to the µTorrent developers, and while 
they reviewed BEP-32 favourably, they had no plans at the time to implement 
it.)

[1] http://www.bittorrent.org/beps/bep_0032.html

-- Juliusz

Re: [Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Juliusz Chroboczek]

> https://torrentfreak.com/popular-torrents-being-sabotaged-by-ipv6-peer-flood-150619/

That's really strange — it looks like µTorrent is using up a connection 
slot even before the peer has performed the µTP three-way handshake.

-- Juliusz

Re: [Bloat] tackling torrent on a 10mbit uplink (100mbit down)

[Benjamin Cronce]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

They say "connection" but really mean a "state". Handshake completed or
not, a state must be created. Most torrent clients also self limit the
number of "half opened" connections, as not to kill your $200 firewall with
a flood of new states. I've seen a default of between 10 and 50.

On Sun, Jun 21, 2015 at 10:39 AM, Juliusz Chroboczek <
jch@pps.univ-paris-diderot.fr> wrote:

>
>> https://torrentfreak.com/popular-torrents-being-sabotaged-by-ipv6-peer-flood-150619/
>>
>
> That's really strange — it looks like µTorrent is using up a connection
> slot even before the peer has performed the µTP three-way handshake.
>
> -- Juliusz
>
>

[-- Attachment #2: Type: text/html, Size: 1338 bytes --]

Re: [Bloat] BitTorrent and IPv6 [was: tackling torrent...]

[Dave Taht]

On Sun, Jun 21, 2015 at 8:32 AM, Juliusz Chroboczek
<jch@pps.univ-paris-diderot.fr> wrote:
> [Removed AQM from CC.]
>
>> 4) *All* the traffic was udp. (uTP)
>
>
> Yes, modern BitTorrent implementations prefer µTP to TCP.  Unfortunately,
> the implementation of µTP in Transmission doesn't perform PMTUD, and gets
> packet sizes wrong most of the time.  (Yes, I spoke about this to Arvid.
> No, he wasn't particularly concerned.)

At least in the comcast cable world, path mtu is 1500 bytes.

I have in general, longed to see a ledbat like protocol go back to shrinking
packet sizes to 300-500 bytes under contention or with excessive loss,
keeping the
signal strength "high", while impact low. This was how one version or
another of bittorrent also reduced it's impact on the network, but a decade
ago the smaller packet size put stress on many a network.

>> Despite ipv6 being enabled (with two source specific ipv6 ips), I did not
>> see any ipv6 peers connect.  Bug? Death of torrent over ipv6? Blocking?
>> What?
>
>
> Please check that you allow incoming IPv6 to your peer, for both TCP and
> UDP.  Obviously, setting port forwarding for IPv4 is not enough for that —
> you'll need to explicitly set up a hole in your IPv6 firewall.  If using
> OpenWRT:

The core differentiator here turned out to be the type of torrent. Switching to
downloading ubuntu's various distros showed a HUGE percentage
using ipv6 (and transmission as the server).

It also appeared to show that transmission chose one and only one
of the ipv6 addresses available on this box. (I only got 10MByte/sec
on the download where I could have got as much as 150Mbyte/sec
out of my two (configured) source specific gateways.

Given the huge number of seeders for the ubuntu torrents, there are
very connections (2 tops) extant now that they are downloaded. This was
not the case for the more normal torrents, which ended up with
15 or more peers each after downloaded.

...

With 196 mostly ipv6 peers (I upped the defaults), I got the expected
download throughput for each flow for a fq'd environment with cake,
with a "normal" tcp. e.g. ledbat made no difference, as expected.

This is me capturing the last 30 seconds of an ubuntu download:

http://snapon.lab.bufferbloat.net/~d/withtorrents/last30secondsoftorrentvs1upload.png

As comcast (mis)marks a huge percentage of traffic as CS1, trying
to distinguish between inbound traffic types based on diffserv marking
alone was futile. I am not sure how often 6696 is used...

On the other hand, even beating up the link as hard as this was barely
noticible for normal web, mosh, ssh traffic, and maximum induced
delays for cross traffic stayed quite low. (above)

And on the gripping hand, using rrul, and classifying all outbound
torrent traffic as CS1 led to normal (non-torrent) downloads going
much faster, and
slowing down torrent (making it ack limited) in the download
direction, when it was in it's download phase. And classification on
the upload
direction, managed by cake, knocks it out of the way so it is
unnoticible during ordinary use. I would still argue for capping the
number
of outbound torrent flows relative to loss or delay at some value
lower than 50 and thinking about how to make ledbat less impactful in
a world with 5ms delay...

http://snapon.lab.bufferbloat.net/~d/withtorrents/competing_rrul_be_during_196torrents_classified.png

the periodic 5ms latency spikes are "interesting", but it's noisy
data, a new qdisc, and other factors....

A bit more flent data here:

http://snapon.lab.bufferbloat.net/~d/withtorrents/

I will clean up my patches a bit more and submit somewhere when I get time....

It is still pretty amazing how fast I can get an entire OS at 100Mbit.
The initial discovery and seeding and startup process takes a
significant chunk of the time required.

>
>     config rule
>             option target 'ACCEPT'
>             option src 'wan'
>             option name 'Transmission-v6'
>             option family 'ipv6'
>             option dest '*'
>             option dest_port '12345'
>
> for the right value of 12345.

6969 was the most common inbound port.

>
> If you set the logging level in Transmission to Debug and let it run for
> twenty minutes or so, you should see periodic "Starting IPv6 DHT announce
> (... good ...)".  If you keep seeing "firewalled" instead of "good", you've
> got an issue.
>
> Once you're positive you don't have firewalling issues, the following might
> be of interest.
>
> I've done quite a bit of work a few years ago to improve IPv6 support in
> Transmission.  The main problem is how to learn the IPv6 addresses of your
> peers:

Yep, I see your name across every file I have wanted to modify slightly. :)

> 1. Many trackers don't support IPv6.
>
> 2. IPv6 is supported by PEX (gossip, peer-exchange).  I'm not sure if other
> peers honour IPv6 PEX when connected over IPv4, but Transmission certainly
> does.
>
> 3. I've implemented a protocol extension that allows peers connected over
> IPv4 to exchange their IPv6 addresses.  This is only marginally useful, but
> can sometimes allow bootstrapping IPv6 PEX without tracker support.
>
> 4. Most importantly, I've implemented IPv6 support for the DHT (BEP-32) [1].
> BEP-32 is now implemented by Transmission, Vuze, Tixati, KTorrent and
> Shareaza.

In light of source specific routing, perhaps that BEP needs an addendum.

> Points (1) through (3) are interoperable with µTorrent/BitTorrent and
> libtorrent, which constitute the vast majority of deployed BitTorrent peers.
> Unfortunately, µTorrent and libtorrent do not implement BEP-32, and since
> these implementations constitute the vast majority of deployed BitTorrent
> peers outside China, this seriously limits the effectiveness of the IPv6
> DHT.  (Yes, I've spoken to the µTorrent developers, and while they reviewed
> BEP-32 favourably, they had no plans at the time to implement it.)
>
> [1] http://www.bittorrent.org/beps/bep_0032.html

All good to know. Thank you.

> -- Juliusz



-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] BitTorrent and IPv6 [was: tackling torrent...]

[Juliusz Chroboczek]

> It also appeared to show that transmission chose one and only one 
> of the ipv6 addresses available on this box.

Yes.  My code explicitly binds the UDPv6 socket to a single address.  
Anything else, and the IPv6 DHT gets confused since Kademlia requires 
having a single address for all peers.

We could in principle use the kernel's address selection for outgoing µTP 
connections.  We could even announce multiple addresses over Kademlia, but 
that would require a fair amount of work.

> This is me capturing the last 30 seconds of an ubuntu download:

> http://snapon.lab.bufferbloat.net/~d/withtorrents/last30secondsoftorrentvs1upload.png

Nice.  (Interesting how the "endgame" algorithm causes the download 
throughput to explode.)

> 6969 was the most common inbound port.

It's the default, but aficionados tend to choose a random high port, in 
case 6969 is blocked or throttled.

>> 4. Most importantly, I've implemented IPv6 support for the DHT (BEP-32) [1].  
>> BEP-32 is now implemented by Transmission, Vuze, Tixati, KTorrent and 
>> Shareaza.
> 
> In light of source specific routing, perhaps that BEP needs an addendum.

I'm no longer actively working on BitTorrent, just maintaining libdht in 
my copious free time.  Place aux jeunes.

-- Juliusz

Re: [Bloat] BitTorrent and IPv6 [was: tackling torrent...]

[Juliusz Chroboczek]

>> It also appeared to show that transmission chose one and only one of the 
>> ipv6 addresses available on this box.

> Yes.  My code explicitly binds the UDPv6 socket to a single address.

I've just been informed by a reliable source that Vuze's "mldht" plugin 
does not suffer from this issue -- it runs a different instance of the 
Kademlia algorithm on every address allocated to the host:

    "to the outside world it just looks like separate DHT nodes running
     on each port. you have to look closely to notice the shared internal
     state"

I've also received a copy of the Kademlia routing table of their 
implementation running on a host with 256 distinct IPv4 addresses.

-- Juliusz