Thread by @theBuoyantMan

To truly learn how to deal with data at scale,

Get a regular 4-8gb ram laptop/desktop with a good fibernet connection.

Your task is to

- Fetch live tick data from the broker
- Store all the ticks you're able to receive
- Design the storage for efficiency & ease of use

- Also design a separate data store of the same tickdata optimising for speed
- Collect tick data of one instrument and do this.
- Find a way to collect live tick data of multiple instruments.
- Use the stored tick data for backtesting
- Optimise the backtesting process & speed

This also involves optimising the data store. If you're able to accomplish this at scale with your laptop, with say 400-500gb of data over a period of time maybe, you'll acquire the relevant skills along the way to appear for a HFT firm's interview.

When you have to build a backtest engine for faster turnaround time of testing strategies (close to or better than how amibroker does it) and also for most efficient use of the same data in your live trading, you'll pick up the relevant system design, database design skills.

You'll also pick up the relevant low-latency coding skills that is required for you to code strategies that work in high-frequency timeframes. Even otherwise, you'd essentially want to use your laptop or cloud device to fetch exactly what you need in real time from large data.

This will eventually help you understand how to drive system design for various purposes, with a separation of concern, while keeping the system highly scalable. This will take a year or two of focused work, but certainly doable in 5 years.

I may not be exactly accurate about how the process works, as I am in the process of doing this. But, along the way, you'll be able to refine these things.

So,

1. Pick up C++ 11 and STL
2. Head to leetcode and codechef and start solving problems.

3. Learn about computer architecture and system programming along the way.
4. Also very important - learn at least one operating system thoroughly, preferably Linux.