Thread by @arbedout

Problems in distributed systems that are suddenly much harder when you're building an exchange: a non-exhaustive list. (Thread).

Logging is *hard*. Much harder than you'd think, because most of the time logging is actually easy.

An exchange that only logs every ask and bid and every order filled can couple their logging infrastructure with their COGS - you get paid for things that you log, so...

...log everything, and invest in scaling up safe in the knowledge that your costs should rise in line with revenue.

Logging all quotes kills this, b/c anyone stuffing the order book with quotes is now filling your logging pipeline with data that isn't tied to revenue.

(Also you kind of have to log everything, always, in real time to your data lake, with as-soon-as-possible ability to query it as structured data, for debugging purposes.)

Concurrency is *hard*. Much harder than you'd think, because most of the time concurrency is actually easy.

In a traditional web app environment, you can mask parts of your infra that suck at concurrency. "Just put a cache in front of it" totally works for content....

...and serving HTTP 503s in event of occasional deadlock is fine when the only thing you're losing is page views from anon visitors.

Traders not only won't forgive you for serving 503s, they'll construct an arbitrage strategy around it if they're sharp and you're predictable.

(And I feel like this goes without saying, but for sake of completeness: you can't cache most of the order book for obvious reasons, and the rest of it because your customers definitely won't let you get away with it.)

An aside: concurrency in trading environments might be one problem that's so hard it *might actually be worth* the hassles of switching to a new language. Or at least, Erlang and Golang make concurrency way easier to grok than other languages.

Debugging is *hard*. Probably as hard as you'd think. It's one thing to debug the order book - hopefully you sorted out your logging earlier! - but how to deal with debugging "performance" issues with non-deterministic data?

Imagine a market maker consistently getting orders filled with a lower bid floor than they'd expect. Is that their problem? Your problem? Where's the demarcation line? Can you trace all trades end to end and at least identify what set of quotes they're experiencing problems with?

(Even harder: you really can't share much debugging data with the customers here, they'll absolutely use it to build a trading strategy.)

Interested to hear from others if there's any more to be added to this list. As a bonus for reading this far, here's a few links to the LMAX Disruptor architecture and design decisions folks there made:

martinfowler.com/articles/lmax.…
baeldung.com/lmax-disruptor…
stackoverflow.com/questions/6559…