Problems in distributed systems that are suddenly much harder when you're building an exchange: a non-exhaustive list. (Thread).
Logging is *hard*. Much harder than you'd think, because most of the time logging is actually easy.

An exchange that only logs every ask and bid and every order filled can couple their logging infrastructure with their COGS - you get paid for things that you log, so...
...log everything, and invest in scaling up safe in the knowledge that your costs should rise in line with revenue.

Logging all quotes kills this, b/c anyone stuffing the order book with quotes is now filling your logging pipeline with data that isn't tied to revenue.
(Also you kind of have to log everything, always, in real time to your data lake, with as-soon-as-possible ability to query it as structured data, for debugging purposes.)
Concurrency is *hard*. Much harder than you'd think, because most of the time concurrency is actually easy.

In a traditional web app environment, you can mask parts of your infra that suck at concurrency. "Just put a cache in front of it" totally works for content....
...and serving HTTP 503s in event of occasional deadlock is fine when the only thing you're losing is page views from anon visitors.

Traders not only won't forgive you for serving 503s, they'll construct an arbitrage strategy around it if they're sharp and you're predictable.
(And I feel like this goes without saying, but for sake of completeness: you can't cache most of the order book for obvious reasons, and the rest of it because your customers definitely won't let you get away with it.)
An aside: concurrency in trading environments might be one problem that's so hard it *might actually be worth* the hassles of switching to a new language. Or at least, Erlang and Golang make concurrency way easier to grok than other languages.
Debugging is *hard*. Probably as hard as you'd think. It's one thing to debug the order book - hopefully you sorted out your logging earlier! - but how to deal with debugging "performance" issues with non-deterministic data?
Imagine a market maker consistently getting orders filled with a lower bid floor than they'd expect. Is that their problem? Your problem? Where's the demarcation line? Can you trace all trades end to end and at least identify what set of quotes they're experiencing problems with?
(Even harder: you really can't share much debugging data with the customers here, they'll absolutely use it to build a trading strategy.)
Interested to hear from others if there's any more to be added to this list. As a bonus for reading this far, here's a few links to the LMAX Disruptor architecture and design decisions folks there made:

martinfowler.com/articles/lmax.…
baeldung.com/lmax-disruptor…
stackoverflow.com/questions/6559…

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with شتر دیدی؟ ندیدی

شتر دیدی؟ ندیدی Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @arbedout

Sep 22, 2021
Decentralizing the Internet in a tweet:

Embed details for a fidelity bond and associated list of TLDs in nameserver's SOA record; client resolvers can compare age and amount locked in fidelity bond to decide which NS is authoritative for a given TLD

That's it, that's the tweet
(Whether offering alernatives to the root nameserver is enough to count as 'decentralizing the Internet' is an open question, but it's at least a step in the right direction IMHO.)
Every attempt (that I know of) to compete with current ICANN / IANA root servers - Namecoin, Handshake, OneName, OpenNIC, and so on - ends up running into the land rush problem.

Namely: how do you stop squatters from ruining your new namespace?

Read 7 tweets
Mar 23, 2021
I need to put all of my networking protocol rants into one place for b̶o̶r̶i̶n̶g̶ ̶m̶u̶t̶u̶a̶l̶s̶ ̶t̶o̶ ̶d̶e̶a̶t̶h̶ ̶i̶n̶ ̶t̶h̶e̶ ̶D̶M̶s̶ easy access. Brace yourself:

The CLNP fiasco at the 1992 IAB meeting:

How layering allowed the ARPANet to grow:
Speaking of layering, here's some Gopher vs. HTTP:

Read 9 tweets
Sep 21, 2020
Still digesting the #FinCENFiles dump, but going over it so far has helped crystallize something for me:

I don't think the dollar is going to be replaced anytime soon, but I don't have the same kind of confidence in the dollar *system*.
I don't quite grok how dollars can continue to exist without the dollar system.

But I also don't get why Tether trades pari passu to the dollar (even after Bitfinex admitted it was less than 100% reserve backed!)
I don't get why mafiosos continued to trade in Italian Lira well after it was demonetized. bloomberg.com/news/articles/…
Read 7 tweets
Aug 3, 2019
Tell me what you do at $DAYJOB writing software, and I'll tell you what food service industry job you'd be comfortable working at.

Thread!
You work at a nameless Fortune 500 company, churning out updates to the internal homegrown CRM software. You desperately want to leave but you have student loans to pay and this was the first temp-to-hire gig you could find.

You're working at McDonalds, and you effing hate it.
You work at a seed funded startup, desperately churning out code to grab more and more total addressable market share. You declined the Fortune 500 gig cuz you're cooler than that.

You'd be at a "hip" fast-casual spot like QDoba. You think it's better than McDonalds. It's not.
Read 19 tweets
Jul 31, 2019
It's days like this I wonder - is it time for the ground up beetles, or the wheat poultices?

(Thread on how George Washington died. Mostly.)
On the evening of Dec 12, 1799, Washington came down with a nasty cough. It progressively worsened over the next 36 hours, until he woke up in the middle of the night on the 14th in severe respiratory distress.
The laypeople around him - the estate overseer and his personal secretary - first made him an elixir of molasses, vinegar and butter to treat his sore, swollen throat.

He damn near choked him death on it.

pbs.org/newshour/healt…
Read 11 tweets
Jun 6, 2019
Was it the State that vouched for the money, or was it money that vouched for the State? On the Lydians vs. the Abbasids:
"...the sekka, the right of the ruler to have his name inscribed on coins, and the ḵoṭba, his right to have his name mentioned in the sermon at the congregational worship service on Friday, were the two formal prerogatives of sovereignty....."
Tired: when the money vouches for your head of state
Wired: when the money vouches for your prophet

Constantinople vs. Caliph Abd'Al-Malik in the coin wars:
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(