Time for a big systems advice thread!

In distributed systems there's no magic "push everything to prod at once" button. Every service gets pushed independently and nodes within a service get updated incrementally. If you mess up forwards/backwards compatibility you can fail irrecoverably.

So how to avoid this?

1/5: Decouple data and code changes. Never push out a release that changes how data is stored at the same time as the code that uses this new data. If there's a bug and you need to roll back to the old version of your code it won't be able to handle the new data in the new format. Instead push out a release that first changes the data in a way that’s compatible with both the old and new code (e.g., optional fields etc), when that’s stable push out the new code that uses it, then when that’s stable you can change the data to remove backwards compatibility. This is known as a “migration” in the database world and yes it’s annoying, but yes you need to do it. 2/5: Don’t change two services at once. If service A talks to service B, you can’t just add a new API to both of them and push them out. What if someone pushes A but not B? What if there’s a bug in B that needs to be rolled back? Just like with data changes, API changes need to be made in a forwards and backwards compatible way. Engineers forget to do this all the time.

How to be a Principal Engineer/Senior Principal Engineer/Senior Architect/fancy-sounding-title Engineer, a thread:

1. You're evaluated on how much more the company succeeds because you're there, not the lines of code you wrote. If you can unblock someone, do that. If you need to kill a two year project that's not going anywhere, do that. Do what is right, not what makes you look good. 2. Your job is the strategy stuff and the dirty work stuff. All the cool stuff in the middle is for everyone else. You're not too senior to carry a pager or respond to outages, this keeps you in touch with how things are going.