Thread by @mykola

The one we never shipped, where we built a distributed graph database with a built-in compute fabric using event sourcing and immutable delta CRDTs.

The value is in what it taught me, from writing a binary format to wrangling Hadoop to how products can fail.

https://twitter.com/berkun/status/1409918939786665985

What if you wrote a database that didn't store state? What if you only tracked changes, via append-only logs of (hypernode) events carrying relational data between different nodes in your hypergraph?

What if state could be derived at any time from those changes? So you could pre-populate a downstream SQL database by parsing the changes and turning them into SQL writes. You could group events by target for quicker queries.

If your database is a giant relational pool of delta events, every response to every query from your database is a subset of that pool. This leads to interesting architectural capabilities - Datomic was just coming out when we were building this, doing some similar stuff.

We had so many cool features. We could dynamically generate JSON representations of complex graphs in response to incredibly complex graph queries.

But my favorite part of this whole thing was that when you pointed to a node, it received a notification. Which means...

We had nodes that were functions. You could invoke a function by creating a new "instance" of it and then connecting, via a graph, other nodes to its argument input edges. Then you could bind other functions etc to its output edges.

And the function itself returns delta events.

So we did this by reserving the last 4 bits in the GUID for a "node type" key. 1 was a regular node, 2 was a hypernode, 3 was a "negate" node, etc. 4 was an "input" node, meaning "treat this as function input"

Which is how we implemented queries. They were delta events too.

Realizing this made us realize that wait, we can think of our whole database as a function. Deltas are its inputs, and its default implementation is persistence and indexing. But by overriding the default behavior we generalized what it meant to "write to the database".

friends I _hand-wrote_ a binary protocol for this, because we didn't want to use protobuf as it was too new and we wanted a bit more control over a few things.

I hand-wrote and hand-optimized a binary protocol for a graph database. I did that. That was so fun and hard!

Because I wrote the binary protocol, I was also able to write a custom compression engine for it. We got it to the point where only the bits necessary for each level of abstraction were written. All events are in a tx, part of its guid is the tx guid. So factor that out on write.

I'll never forget the day my old coworker had a bug, and I asked him to open his console and then eyeballed a few lines of hex data to understand where the bug was. I don't think anyone has ever looked at me more like I'm a wizard.

the core insight for this whole thing, btw, is in the event delta schema, which I'm not going to publish here because I didn't come up with it. I inherited it from boss, who spent a decade trying to build this system. I'm technically still under NDA, I think, tho company is gone.

What I will ONE DAY DO, however, is publish my own version of this system using my own schema, which is heavily informed by the one we used but which makes a few different assumptions in pursuit of a few different goals.

Maybe the best way to think of this is: picture Redux, but without custom reducers because the actions themselves work with a general-case reducer to build a single giant easily-queriable store from which you can trivially query and cache stateful objects. cc @dan_abramov

(using Hadoop at the time we were horizontally scalable beyond the needs of a Fortune Ten company that hired us to do a proof of concept where we consumed all of their conflicting upstream databases, resolved the conflicts and served an always-fresh SQL interface for latest data)

The thing is, this was _ten years ago_ and nobody I tried to talk to about it understood it. "Why don't you just use {SQL/Mongo}?" was a common refrain.

This is a new _kind_ of database, which is actually a compute fabric, with a full stream computing engine built into it.

(And did I mention that this was a _fully relational NoSQL_ according to Codd's Relational Algebra?! People didn't even understand what that meant. Hell WE didn't fully understand what that meant, but my boss saw it all in a crystal clear vision I came to grok.)

BTW when I say we had nodes that were functions, I mean you could define a thing and say "language: javascript, behavior: function(input_deltas) { ... return output_deltas }" and it would find a worker instance with javascript to run your string of code against the input.

You want to update your function's behavior? No sweat, just point to a new behavior. Everything is immutable so you lose nothing, but you could e.g. re-run yesterday's query as if you had used a different function, or whatever.

(Oh, right -- we didn't resolve conflicts of any sort. If you said Foo == 42 and I say Foo == 21, anyone querying the system for the value of Foo gets "Myk says it's 21, so-and-so says it's 42, which value would you like to use? Or we can average them, take the latest, etc")

All writes had full provenance, so you know what user generated the delta, you know what system they're on, etc. That's all in the metadata of the delta.

You can share deltas, btw, between systems. Two instances that have never met can handshake and exchange schemas and nodes.

What I didn't realize until almost a decade after I left this company was that my real passion in this work had been metaphysical in nature.

This was a model of reality that worked for me, and I could see reality as a hypergraph of deltas over time -- each of us as a local cache

Things possible with our database:

"Run this query as if it were last week"

"Run this query as if you were Myk. Run again as Dan. Return the intersecting deltas after applying this schema to them."

"Send the results of this query to my friend Sarah's database."

Because the deltas are so fine-grained, this effectively gives you _cell-level_ access control on a derived tabular view of your data.

Or it lets you serialize all of your deltas directly into a JSON store for your front end.

Or cache important personal data on a private device