It’s quite rare for software to offer both a real local file format and also substantial cloud/SaaS behavior (ie other than syncing). Usually it’s one or the other—if the cloud has active behavior, apps are thin clients which don’t expose an accessible on-disk format (eg Notion).
The troubles are well articulated here: inkandswitch.com/local-first.ht…
I recently finished re-architecting @withorbit to expose an on-disk format, thought I’d write a bit about what I learned.
I recently finished re-architecting @withorbit to expose an on-disk format, thought I’d write a bit about what I learned.
@withorbit It’s hard to pull this off, as I&S describe. You need to design a file format which syncs efficiently and reliably, which is already tough. Gets tougher when your SaaS is an active replica with its own public API, and when you layer on indexing/querying, migration, sparsity.
@withorbit There are enterprise solutions which solve these problems (Realm, CouchDB), but one consistent limitation is that they “own” the file format, and often the server component as well. I don’t feel good exposing my on-disk format to local clients if it’s tied to a specific backend.
@withorbit CRDTs are a useful piece of the puzzle, but they don’t solve the problem as fully as people often imagine. The I&S folks are working diligently on many of these. But one problem I don’t know how to solve is the complexity of the resulting opaque file formats.
@withorbit In an ideal world, everything’s just plaintext, so you can modify it with whatever tool you like. No libraries needed. Problems with this include semantic sync, indexing, structured data, etc. SQLite is almost as good, if the schema is well-defined: sqlite.org/appfileformat.…
@withorbit Maybe someday automerge’s serialization format will be as durable and universal as SQLite’s, but in the meantime, it’s certainly not amenable to casual concurrent local access in some user script.
@withorbit Of course, SQLite leaves you to solve syncing. The typical approach is a write-only log of events (CRDT mutations or otherwise) which can be replayed across replicas. Snapshots of entity states are computed from queries across these events.
@withorbit That’s the approach Orbit takes: simple event structures (simpler than CRDTs, sacrificing some of their key guarantees to reduce complexity), with well-defined merge operations, in a SQLite db. Users can write scripts to insert their own events if they like.
@withorbit I’m still not satisfied with this. Reading/writing data yourself requires using Orbit’s libraries or a SQLite library and an understanding of the schema. I’d like to just expose the data as plaintext, but I’m not yet sure how to achieve that practically.
@withorbit One approach is to define Git-style “plumbing” CLI primitives which offer plaintext-like I/O to your data structures. Another is to define a FuseFS layer, @rsnous style.
@withorbit @rsnous Switching gears: another hard thing about implementing this type of data format is that web browsers demand their own implementation. @kirkbyo_ kindly implemented an IDB-based Orbit backend. Excited about absurd-sql here: github.com/jlongster/absu…
@withorbit @rsnous @kirkbyo_ Orbit’s cloud server has its own (third, ugh) backend implementation, which it uses to offer APIs, aggregate analytics (for my research), and services like study reminder notifications.
@withorbit @rsnous @kirkbyo_ I see now why people don’t generally do this. It was a huge amount of work. I’m honestly pretty frustrated with myself for shaving this yak: too much time polishing my telescope, too little time looking at stars.
@withorbit @rsnous @kirkbyo_ As it happens, Orbit’s implementations are quite general (i.e. they have almost no specific knowledge of Orbit’s structures), so perhaps they’ll be of use to others. See store-*, sync, and backend packages here: github.com/andymatuschak/…
• • •
Missing some Tweet in this thread? You can try to
force a refresh
