I'm not involved with @nats_io anymore, but let me tell you why I've written so much about it, why I used it, why I worked on it full-time for a while, and why I think it's a good addition to @CloudNativeFdn.
@nats_io @CloudNativeFdn The way we build systems has fundamentally changed. Containers, serverless, IoT—computation is increasingly distributed and ephemeral, yet our systems mindset is lagging.
Life is mostly asynchronous, and so are distributed software systems. This introduces a host of new failure modes and design implications.
Striving for distributed systems that provide "single-system image" is fundamentally flawed (christophermeiklejohn.com/lasp/erlang/20…). Yes, you say things like Spanner and CockroachDB suggest otherwise, but it's more nuanced than that.
The natural tendency is to build distributed systems as if they aren't distributed at all because it's easier to reason about strong consistency, reliable messaging, and—generally speaking—predictability.
This asks a lot of the guarantees provided by our infrastructure and requires us to encode our business rules and constraints into the language our infrastructure understands. This causes a few problems...
First, we have to know how to translate our application semantics into these low-level operations while avoiding any impedance mismatch.
In the context of messaging, guaranteed delivery doesn't really mean anything to our application which cares about what’s done with the messages (TCP doesn't care about your application semantics).
It's important not to conflate the transport protocol with the business-transaction protocol. See the end-to-end argument (web.mit.edu/Saltzer/www/pu…).
Second, we preclude ourselves from using a lot of generalized solutions and, in some cases, end up having to engineer specialized ones ourselves. It's not clear how well this scales in practice, but I can speak from experience that building good distributed systems is HARD.
Third, we pay a performance penalty that could otherwise be avoided (vldb.org/pvldb/vol8/p18…). When we think low level, we pay the upfront cost of entry.
Lastly, and probably most important, we put ourselves at the mercy of our infrastructure and hope it makes good on its promises—it often doesn't.
On systems that do claim some form of guarantee, it's best to look at what level that guarantee really runs out.
This suggests we need to rethink the way we design systems by shifting the focus from system properties and guarantees to business rules and application invariants.
Push invariants up into the business layer where we specify correctness in terms of application semantics rather than low-level infrastructure primitives.
By providing only basic guarantees, use cases that don't need stronger guarantees do not pay the cost of their implementation. It's always possible to add stronger guarantees on top, but it's not always possible to remove them from below.
E.g. replayable instead of guaranteed, idempotent instead of exactly-once, commutative instead of ordered.
This isn't FUD and doesn't mean we can't have good abstractions and should just give up (see the work @cmeik is doing with Lasp and distributed programming), but if you need business-level guarantees, build them into the business layer.
End-to-end system semantics matter much more than the semantics of a single building block. E.g. TCP doesn't prevent data loss. If you can't tolerate data loss, you need to account for it end to end. We can't outsource our systems thinking to our Legos.
This is why I like NATS. It's no-frills. It flips the way we think about systems on its head. It embraces the "negative space" and asynchrony and forces us to think more carefully about how we design our systems, and that usually leads to more reliable software in the long run.
Tweetstorm over.
