It’s quite rare for software to offer both a real local file format and also substantial cloud/SaaS behavior (ie other than syncing). Usually it’s one or the other—if the cloud has active behavior, apps are thin clients which don’t expose an accessible on-disk format (eg Notion).
The troubles are well articulated here:…

I recently finished re-architecting @withorbit to expose an on-disk format, thought I’d write a bit about what I learned.
@withorbit It’s hard to pull this off, as I&S describe. You need to design a file format which syncs efficiently and reliably, which is already tough. Gets tougher when your SaaS is an active replica with its own public API, and when you layer on indexing/querying, migration, sparsity.
@withorbit There are enterprise solutions which solve these problems (Realm, CouchDB), but one consistent limitation is that they “own” the file format, and often the server component as well. I don’t feel good exposing my on-disk format to local clients if it’s tied to a specific backend.
@withorbit CRDTs are a useful piece of the puzzle, but they don’t solve the problem as fully as people often imagine. The I&S folks are working diligently on many of these. But one problem I don’t know how to solve is the complexity of the resulting opaque file formats.
@withorbit In an ideal world, everything’s just plaintext, so you can modify it with whatever tool you like. No libraries needed. Problems with this include semantic sync, indexing, structured data, etc. SQLite is almost as good, if the schema is well-defined:…
@withorbit Maybe someday automerge’s serialization format will be as durable and universal as SQLite’s, but in the meantime, it’s certainly not amenable to casual concurrent local access in some user script.
@withorbit Of course, SQLite leaves you to solve syncing. The typical approach is a write-only log of events (CRDT mutations or otherwise) which can be replayed across replicas. Snapshots of entity states are computed from queries across these events.
@withorbit That’s the approach Orbit takes: simple event structures (simpler than CRDTs, sacrificing some of their key guarantees to reduce complexity), with well-defined merge operations, in a SQLite db. Users can write scripts to insert their own events if they like.
@withorbit I’m still not satisfied with this. Reading/writing data yourself requires using Orbit’s libraries or a SQLite library and an understanding of the schema. I’d like to just expose the data as plaintext, but I’m not yet sure how to achieve that practically.
@withorbit One approach is to define Git-style “plumbing” CLI primitives which offer plaintext-like I/O to your data structures. Another is to define a FuseFS layer, @rsnous style.
@withorbit @rsnous Switching gears: another hard thing about implementing this type of data format is that web browsers demand their own implementation. @kirkbyo_ kindly implemented an IDB-based Orbit backend. Excited about absurd-sql here:…
@withorbit @rsnous @kirkbyo_ Orbit’s cloud server has its own (third, ugh) backend implementation, which it uses to offer APIs, aggregate analytics (for my research), and services like study reminder notifications.
@withorbit @rsnous @kirkbyo_ I see now why people don’t generally do this. It was a huge amount of work. I’m honestly pretty frustrated with myself for shaving this yak: too much time polishing my telescope, too little time looking at stars.
@withorbit @rsnous @kirkbyo_ As it happens, Orbit’s implementations are quite general (i.e. they have almost no specific knowledge of Orbit’s structures), so perhaps they’ll be of use to others. See store-*, sync, and backend packages here:…

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Andy Matuschak

Andy Matuschak Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @andy_matuschak

10 Sep
A bit of fun: this GIF is a "legendary" @robin_____sloan Amulet (8 8's).

base64: R0lGODdhBgAJAPEAAEeEqvij+U6wk386YSwAAAAABgAJAAACD4RvERciC9pjMDooq0KoAAA7

hash: B36F0888888889A57D77E14E9B8D4B95E11C6B6C0B48FB0D7F5E634B2D129623…
@robin_____sloan I was unable to reveal the amulet via the smart contract, alas, since the text is specified via `string` rather than `bytes`, and UTF-8 encoding rules muck up the string. But/so:

Title: "A picture worth a thousand words"
Carbon offset URL:…
@robin_____sloan (Hope you don't mind the subversion, Robin… can a GIF be poetry?)
Read 5 tweets
8 Sep
One reason we don't have more interesting, quality structured text editors: it's *really* hard to implement table-stakes editing operations well, particularly on web.

In this video, I attempt to arrow up/down and shift+up/down to select inter-line in 8 outliners. Very yikes.
Roam was the only web outliner which got arrow up/down navigation mostly right, though with some unexpected glitches at EOLs.

None of the web outliners support interline selection. OO doesn't either. Bear does great but ofc isn't really structured. And org-mode wins the day.
This sounds so nit-picky and trivial, but I think the difficulty of getting basic editing ops done well in simple outline UX illustrates just how painful it is to make a structured text editor nice enough to live in.

There'd be a lot of value in finding a good abstraction here.
Read 4 tweets
7 Sep
@metaLulie Not a solopreneur but similar situation. Trying to work non-coercively, I’ve found it useful to understand my monkey-brain proclivities. Monkey has inertia, finds it hard to get started, easy to continue. Monkey wants easy gratification in the moment, meaningfulness in hindsight.
@metaLulie Very concretely: I find routine incredibly helpful. I’m not rigid about it—I’m happy to change it upon reflection—but everything is easier with well-considered defaults. It’s simply *true* that if I start my morning on Twitter, I will p>0.7 get no deep work done that morning.
@metaLulie I’ll regret it: in hindsight, Twitter was fun for the first 10m, then I got unwittingly sucked in, and it wasn’t really that fun. So I start my day with the internet off. Sounds like coercion, but doesn’t feel that way in the moment. It feels like “oh, right… thanks, past me.”
Read 8 tweets
3 Sep
Funny how much sheer pleasurability and inspiration matter for habits. Ever since I got a new piano, my old practice time goals (which I often struggled to meet) feel comically low. Now somewhat effortlessly 3 months ahead of target… gotta ratchet up the goal!
Making the habit more pleasurable is an oft-suggested strategy. I think it'd be pretty enabling to assemble a wiki-style database of per-habit pleasure/inspiration-increasing strategies!
Something about this feels wrong. Like: why should I come up with ways to make myself to want to play the piano more? Shouldn't I just naturally want it?

Yet: deliberate practice is unpleasurable! (Ericsson et al '93) *Playing* is, but need much of the former to get the latter.
Read 4 tweets
2 Sep
"In intellectual work, it’s much easier to get twice as fast than twice as good."…

Interesting to consider when this is an effective strategy. In many design tasks, simply being able to generate (& discard) more alts will produce better results. But…
… this is probably only true when the problem can be solved with something like pre-existing approaches. Inventing wholly new representations needs 2x-good > 2x-fast.

In most software tasks (but often not the interesting ones), 2x-fast is usually higher-leverage than 2x-good.
Reminds me also of experiences learning piano. For years, was trying to move as fast as possible onto harder repertoire. Then had a great teacher who made me slow way down, play stuff from years ago, focus on musicality. 2x good was much more rewarding than 2x fast.
Read 5 tweets
17 Aug
A consistently valuable interface design prompt for me: how would I do [X activity] with paper/pen/physical tools? The answer is often so much more fluid and improvisational than typical UI idioms and "data structures" tend to produce!
If I "highlight" something on a computer, it needs to represent that with a literal start offset and length. But annotating a physical document is often intentionally vague and gestural—attach a sticky to the general vicinity of an area, shade a line down the margin, etc.
I can improvise purpose-specific solutions with a physical highlighter: draw a box around something; make special marks in a personal code; write big-font marginalia. Most interfaces permit only pre-set verbs and a small surrounding space of creative repurposing.
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!