Thread by @mikeal

It's been about 8 years since I wrote the original proof-of-concept for PouchDB. github.com/pouchdb/pouchd…

In that time, we haven't seen a significant increase in offline capable applications. I'm going to talk a bit about why that is. [Thread]

In 2010 it was clear that we needed new standards. AppCache was awful, WebSQL was dead (Firefox committed to *not* implement it). From 2010 - 2015 or so all the focus was on driving those standards, first IndexedDB and then Service Workers.

But once these standards were in place and we had some good libraries, like PouchDB on them, things still didn't seem to be progressing. Offline apps were rare, adding offline to an existing application was damn near impossible.

And yet, the case for offline capable apps has continued to grow. Our data needs, bundle sizes, etc are all growing much faster than average mobile connectivity. Developing world is coming online with bandwidth practically unusable without smarter caching.

I wrote and talked about this a little in 2016, but there's a spectrum from "fully online" to "fully offline" and the space in the middle, where connectivity is partial or infrequent has the same problems that p2p has.

Once you look at the whole picture, you realize that you just can't adapt the client/server model of development to work for anything other than "fully connected." There's no transition path.

Now we're seeing the same thing happen with edge computing. Cloud services that are "region aware" just fundamentally don't work when there are hundreds of regions. We can't punt on this anymore, these are effectively decentralized systems and need different primitives.

I still have a lot of love for CouchDB/PouchDB but I'm becoming more and more convinced that it's the wrong approach. It's stuck in a 1990's view of decentralization it inherited from Lotus, where there is a sync point that ends up becoming the authority for everyone else.

The biggest problem with this model is what it does to the developer. It gives you just enough familiarity from the client/server mindset that you can be quickly productive but, down the line, hasn't really taught you decentralization.

I'll say it again, the problem is client/server programming.

The assumptions we carry around from this programming model are deep. The way we think about storage, data, how we mutate data-structures, all of it has to change to tackle these problems.

Look at what is happening with "Edge Computing" right now. They have many of the same problems and who is doing the most interesting work? CDNs.

CDNs are basically distributed content addressable file stores. They've been dealing with these issue behind the scenes forever.

The most impressive thing I've seen to date using Service Workers is from Cloudflare blog.cloudflare.com/introducing-cl…

But... it's a big leap from static files to application data. How do you mutate decentralized data? How do you store/sync just the portion of the data a user needs?

I've been diving deep lately into the IPFS stack. At the lower layers you're working with IPLD (Inter-Planetary Linked Data).

Put simply, it's a container for any data that exposes links to other data.

Every node gets hashes and is content addressable.

This means that you end up with a graph which could be very large. But no matter how large the graph, the root, and every branch, are available via universal content address (CID).

This is the missing primitive I've always needed to make offline work. I'll explain.

Multi-user applications store too much data to keep a full, working copy of the data around for every user.

The solutions for this in Couch/Pouch landscape are to shard the data per user, effectively a database per users, or a "view" of the database per user.

The problem is that, depending on what a user does on a given device, I need different slices of the data.

Sync also becomes very problematic when you're trying to slice up the data this way, it's hard to keep track at a document level of all the different people with different revision history. This is why git uses a merkle tree, which is what IPFS/IPLD give you.

But here's the problem, these data-structures are used very differently than client/server developers are used to.

These are immutable data-structures. Once you change them you get a new hash, and you have to figure out the semantics for sharing that hash around.

No matter how much data you work with in client/server development you get to treat it like a global mutable state. CouchDB/PouchDB allow you to deceive yourself into believing you have the same thing until you get enough conflicts.

This is the biggest challenge of Offline, the biggest challenge for the Decentralize Web, and possibly even Edge Computing.

Teaching developers to work with decentralized data-structures is harder than teaching them a new programming language.

This is a completely different programming model, but it's one we'll need to migrate to because our data needs are growing faster than available mobile bandwidth and that isn't changing.