[Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Dave Taht]

food for thought.


---------- Forwarded message ----------
From: Jay Ashworth <jra@baylink.com>
Date: Mon, Aug 3, 2015 at 5:19 AM
Subject: Did *bufferbloat* cause the 2010 flashcrash?
To: nanog@nanog.org


This guy seems to think so, and his arguments seem pretty convincing
to me, but I don't understand the financial system as well as I might.

yarchive.net/blog/computers/flash_crash.html

Gettys is namechecked in the piece.

Cheers,
-- jra
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.


-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Livingood, Jason]

There are a lot of possibilities related to buffer bloat and other network
performance issues. For example, at Comcast where I work, I have come to
suspect that buffer bloat was behind the degradation of VoIP (e.g. Vonage)
while customers were using BitTorrent (which led the company down a bad
path to try to fix it but that¹s a long story).
http://www.zdnet.com/article/some-vonage-users-are-frustrated-about-comcast
-connection-quality/

JL




On 8/3/15, 5:54 AM, "Dave Taht" <dave.taht@gmail.com> wrote:

>food for thought.
>
>
>---------- Forwarded message ----------
>From: Jay Ashworth <jra@baylink.com>
>Date: Mon, Aug 3, 2015 at 5:19 AM
>Subject: Did *bufferbloat* cause the 2010 flashcrash?
>To: nanog@nanog.org
>
>
>This guy seems to think so, and his arguments seem pretty convincing
>to me, but I don't understand the financial system as well as I might.
>
>yarchive.net/blog/computers/flash_crash.html
>
>Gettys is namechecked in the piece.
>
>Cheers,
>-- jra
>--
>Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
>
>-- 
>Dave Täht
>worldwide bufferbloat report:
>http://www.dslreports.com/speedtest/results/bufferbloat
>And:
>What will it take to vastly improve wifi for everyone?
>https://plus.google.com/u/0/explore/makewififast
>_______________________________________________
>Bloat mailing list
>Bloat@lists.bufferbloat.net
>https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Steinar H. Gunderson]

On Fri, Aug 07, 2015 at 02:13:20PM +0000, Livingood, Jason wrote:
> while customers were using BitTorrent (which led the company down a bad
> path to try to fix it but that¹s a long story).

But what _is_ a good path to fix the BitTorrent problem[1]? I still haven't
seen a real-life practical solution. LEDBAT is negated by fq_codel, DSCP
markings don't work (they get stripped, and usually were not even set to
begin with), switching to IP-level fairness makes life hard for the torrent
user (not that it seems to even exist in fq_codel; and two-level fq_codel
doesn't seem to exist either). DPI to single out torrent traffic? I think
we've sort of tried that. :-)

[1] For the purposes of the question, the “BitTorrent problem” is when you
and I are on the same network, and your 200+ BitTorrent upload sessions
makes it impossible for me to upload my single cat video to YouTube.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 619 bytes --]

On Fri, 7 Aug 2015, Steinar H. Gunderson wrote:

> [1] For the purposes of the question, the “BitTorrent problem” is when 
> you and I are on the same network, and your 200+ BitTorrent upload 
> sessions makes it impossible for me to upload my single cat video to 
> YouTube.

Isn't this dependent on the upload speed? I would imagine that if it's 
5-10 megabit/s, then using Codel or similar technique that doesn't allow 
the buffer to grow to hundreds of milliseconds, would improve things a 
lot?

For a 500 kilobit/s link, I'm not sure even that would work...?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Toke Høiland-Jørgensen]

Mikael Abrahamsson <swmike@swm.pp.se> writes:

> On Fri, 7 Aug 2015, Steinar H. Gunderson wrote:
>
>> [1] For the purposes of the question, the “BitTorrent problem” is when you and
>> I are on the same network, and your 200+ BitTorrent upload sessions makes it
>> impossible for me to upload my single cat video to YouTube.
>
> Isn't this dependent on the upload speed? I would imagine that if it's 5-10
> megabit/s, then using Codel or similar technique that doesn't allow the buffer
> to grow to hundreds of milliseconds, would improve things a lot?
>
> For a 500 kilobit/s link, I'm not sure even that would work...?

There are two separate issues: 

-Toke

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Toke Høiland-Jørgensen]

Toke Høiland-Jørgensen <toke@toke.dk> writes:

>>> [1] For the purposes of the question, the “BitTorrent problem” is when you and
>>> I are on the same network, and your 200+ BitTorrent upload sessions makes it
>>> impossible for me to upload my single cat video to YouTube.
>>
>> Isn't this dependent on the upload speed? I would imagine that if it's 5-10
>> megabit/s, then using Codel or similar technique that doesn't allow the buffer
>> to grow to hundreds of milliseconds, would improve things a lot?
>>
>> For a 500 kilobit/s link, I'm not sure even that would work...?
>
> There are two separate issues: 

(three if you count me hitting send too soon...)

1. When you turn on bittorrent, everything becomes unbearably slow
   because you have huge amounts of bloat. I.e. web browsing doesn't
   work, your voice calls drop, etc. fq_codel solves this pretty nicely
   (I don't even notice when bittorent is on anymore).

2. Bandwidth sharing between bittorrent and other protocols. Bittorrent
   uses utp to try to be a scavenger that will back off and not use
   bandwidth if other traffic needs it. fq_codel breaks this by
   isolating flows and taking away the signal utp uses to back off.
   Diffserv with a suitable scheduler (cake, for instance) can fix this
   if the bittorrent client is configured correctly; or you could use
   deep(er) packet inspection to try to figure out which traffic is
   bittorent. But we don't have a shrinkwrap solution to this presently,
   AFAIK.

I believe the cat video upload falls into category 2.

-Toke

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Steinar H. Gunderson]

On Fri, Aug 07, 2015 at 04:39:16PM +0200, Toke Høiland-Jørgensen wrote:
> 2. Bandwidth sharing between bittorrent and other protocols. Bittorrent
>    uses utp to try to be a scavenger that will back off and not use
>    bandwidth if other traffic needs it. fq_codel breaks this by
>    isolating flows and taking away the signal utp uses to back off.
>    Diffserv with a suitable scheduler (cake, for instance) can fix this
>    if the bittorrent client is configured correctly;

Well, I already pointed out that DSCP doesn't really work :-) For one, the
torrent client _isn't_ configured correctly. I have yet to actually see any
setting DSCP bits in practice, although I'm sure they exist (e.g. I heard
Dave was sending out patches).

>    or you could use
>    deep(er) packet inspection to try to figure out which traffic is
>    bittorent. But we don't have a shrinkwrap solution to this presently,
>    AFAIK.
> 
> I believe the cat video upload falls into category 2.

Yes, certainly.

In a sense, the only reasonable solution I can see to this problem is to have
two-level fq_codel (fair between senders, fair between flows for each
sender), but that simply doesn't exist. Also you'd need to reverse it for
downloads, and for someone in the middle, who knows what's actually up or
down.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Sebastian Moeller]

Hi Steinar,

On Aug 7, 2015, at 16:23 , Steinar H. Gunderson <sgunderson@bigfoot.com> wrote:

> On Fri, Aug 07, 2015 at 02:13:20PM +0000, Livingood, Jason wrote:
>> while customers were using BitTorrent (which led the company down a bad
>> path to try to fix it but that¹s a long story).
> 
> But what _is_ a good path to fix the BitTorrent problem[1]? I still haven't
> seen a real-life practical solution. LEDBAT is negated by fq_codel, DSCP
> markings don't work (they get stripped, and usually were not even set to
> begin with),

	Except for [1] DSCP marking is a really simple solution as for egress from a home network there is no remapping of code points not under your control, so just teach your bit torrent client to behave (aka mark its outgoing packets as CS1) and use sqm-script’s single.qos, which should be close to what you requested...


> switching to IP-level fairness makes life hard for the torrent
> user (not that it seems to even exist in fq_codel;

	fq_codel does not care as far as I know one can specify different hashing schemes (expect I have not tried it myself so that might not work).

> and two-level fq_codel
> doesn't seem to exist either).

	Build one your self ;) as long as you are willing to dedicate two machines for that it should be “simple”, box one does proper shaping and fq_codel by src-ip and box two does what ever you do on your network, just make sure box one sits close to the real bottleneck…

	Actually, one of the more frequent requests on the openwrt forum seems to be bit torrent users who want per-internal-IP fairness, or in other words want to be able to dedicate a box to torrenting without sacrificing all egress bandwidth to the bit-torrent deity ;) I have not yet attempted to set this up. I hope that cake will make this easy:
tc qdisc add dev eth2 root cake bandwidth 50mbit

Usage: ... cake  [ bandwidth RATE | unlimited ] # unlimited line rate is the default
                 [ besteffort | precedence | diffserv8 | diffserv4 ] # diffserv variants including none
                 [ flowblind | srchost | dsthost | hosts | flows ] # hash algorithm on what fields
                 [ atm ] # atm support (untested)
                 (flowblind) gives pure single queue codel aqm behavior, useful for testing the new codel implementation

My guess is that the srchost hash should help a lot with outgoing torrent traffic.


> DPI to single out torrent traffic? I think
> we've sort of tried that. :-)

	All this does is speeding up bit torrents evolution by selecting against easy to spot “markers” in the data...

> 
> [1] For the purposes of the question, the “BitTorrent problem” is when you
> and I are on the same network, and your 200+ BitTorrent upload sessions
> makes it impossible for me to upload my single cat video to YouTube.

	I though part of the boot torrent challenge was the fact that on ingress it can also wreck havoc?

Best Regards
	Sebastian

> 
> /* Steinar */
> -- 
> Homepage: http://www.sesse.net/
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Steinar H. Gunderson]

On Fri, Aug 07, 2015 at 04:43:19PM +0200, Sebastian Moeller wrote:
>> switching to IP-level fairness makes life hard for the torrent
>> user (not that it seems to even exist in fq_codel;
> 	fq_codel does not care as far as I know one can specify different
> 	hashing schemes (expect I have not tried it myself so that might not
> 	work).

How? I've searched for it, I've checked the documentation, I've looked for it
in the code. What am I missing?

> 	Actually, one of the more frequent requests on the openwrt forum
> 	seems to be bit torrent users who want per-internal-IP fairness, or
> 	in other words want to be able to dedicate a box to torrenting
> 	without sacrificing all egress bandwidth to the bit-torrent deity ;)

Yes, per-internal-IP fairness would go a long way. It's not a perfect
solution by itself, but at least that user only messes up for him/herself.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Toke Høiland-Jørgensen]

"Steinar H. Gunderson" <sgunderson@bigfoot.com> writes:

> On Fri, Aug 07, 2015 at 04:43:19PM +0200, Sebastian Moeller wrote:
>>> switching to IP-level fairness makes life hard for the torrent
>>> user (not that it seems to even exist in fq_codel;
>> 	fq_codel does not care as far as I know one can specify different
>> 	hashing schemes (expect I have not tried it myself so that might not
>> 	work).
>
> How? I've searched for it, I've checked the documentation, I've looked for it
> in the code. What am I missing?

The mechanism is not specific to fq_codel, it's a tc thing. You can
attach arbitrary tc filters which fq_codel will then use as a
classifier. http://lartc.org/howto/lartc.adv-filter.hashing.html has
some (fairly old) information on this, but I've never tried it in
practice with fq_codel so can't give you any more specific pointers.

An alternative, of course, is to try out cake. Jonathan did a
presentation on it at Battlemesh yesterday, which also has instructions
on how to obtain it:
http://battlemesh.org/BattleMeshV8/Agenda?action=AttachFile&do=view&target=cake-battlemesh-v8.pdf

-Toke

Re: [Bloat] Fwd: Did *bufferbloat* cause the 2010 flashcrash?

[Jonathan Morton]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]

Yes, cake has an option to use host isolation instead of flow isolation.
I'm hoping to get both at once working soon.

- Jonathan Morton

[-- Attachment #2: Type: text/html, Size: 185 bytes --]