Re: [Cerowrt-devel] Fwd: You're Invited: DWeb Virtual Meetup 14next Wednesday--Decentralized Storage Ø3DÜBE Ø3DÜBB

[Dave Taht <dave.taht@gmail.com>]

On Sat, May 9, 2020 at 12:34 AM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> On Fri, 08 May 2020 16:40:20 -0700, Dave Taht said:
>
> > while I'm referring to stuff that's actually fun and off-topic... I've long
> > off been watching progress in this area, knowing that
> > "fixing bufferbloat" is a key requirement for technologies like this to
> > succeed.
>
> Amen to that.  Probably around a decade ago, $DAYJOB at the time was approached
> by a vendor who had a very compelling and cost-effective erasure-coded storage
> product that would have allowed us to have the data at 3 sites, and total loss
> of any one would be survivable.  Two sites were about 3 cable miles apart, and
> the third was 95 cable miles and could only run at 10Gig speed back then.
>
> And what sank the deal was uncertainty if we could meet throughput requirements
> under heavy load with one site that distant.  (You'd be *amazed* what happens
> in the first 3 minutes after semester add/drop goes live at midnight, when you
> have 35,000 students. :)
>
> The other great bugaboo - How do you back up a half billion files that total
> 12+ petabytes? (It doesn't help the solution space when the majority of it is
> one research group that (a) will probably eventually data-mine their data for
> $100M+ in research grants and also (b) the terms of some of the grants
> specified a 30 year retention  on the data *and* some wicked nasty PII issues,
> because the data includes identifiable video of people who had not consented to
> be part of the research study...

Well, someone(s) are probably doing that with government money anyway.

> Storage - it turns out you can't just throw 10 terabyte drives at the problem. :)

I still rather like bittorrrent. I've come to understand, over the
years, why it got structured the way it did.
The typical behavior of having 5 open streams and switching between
them every 15 seconds or so
is a direct outgrowth of bufferbloat, compensating for slow start in
an overbuffered universe,  and
maxing out at 100ms observed induced delay, even with the latecomer
problem, works as best
as it can.

the further, more recent adopion of vpns for it add intrinsic delay,
and that also makes the impact
of the protocol on edge networks less.

It's not quite a dead protocol, and the implementations could still be
better of course,

But in no case can I come up with some way to keep file integrity for
10PB with even thousands of
volunteer users providing chunks of storage. Hard problem! That said,
thousands of nodes providing
resiliency for the semester end/drop - far worse than mothers day -
problem - does makes sense.

the whole covid-19 thing - trying to match available hospital
resources against the potential growth
in infection rate - is bufferbloat in the real world - covidbloat -
and I wish more folk understood
that the queue theory problems are the same.

-- 
Make Music, Not War

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-435-0729