How to be a Principal Engineer/Senior Principal Engineer/Senior Architect/fancy-sounding-title Engineer, a thread:
1. You're evaluated on how much more the company succeeds because you're there, not the lines of code you wrote. If you can unblock someone, do that. If you need to kill a two year project that's not going anywhere, do that. Do what is right, not what makes you look good.
2. Your job is the strategy stuff and the dirty work stuff. All the cool stuff in the middle is for everyone else. You're not too senior to carry a pager or respond to outages, this keeps you in touch with how things are going.
3. If you get too caught up in the day-to-day (the how) to step back and think about whether you're going in the right direction (the why and the what) the team will eventually go adrift and people will lose motivation. You have to proactively own this.
4. Engineers are often bad at strategy because it involves unknowns and engineers like to be confident and exact. Getting good at long term strategy and dealing with unknowns is the #1 skill to develop: medium.com/@jamesacowling…
5. If you spend all day in your little strategy bubble you will probably not have the credibility from the wider engineering team, and if you don't have credibility it's hard to get things done. It helps a lot to have spent serious time at the company building things and continuing to do so.
6. You're on the solutions team, not the complaining team. If you don't like something then go fix it or go speak to someone about getting it fixed. Sometimes you also just need to let things go.
You also need empathy for company leadership. They are probably not a bunch of idiots, they're likely just juggling a different set of priorities. If they really are idiots then quit and go work for a better company.
7. Pick your battles. Not every engineering decision needs to be perfect. Sometimes people need to make small mistakes and learn from them. Sometimes you were wrong and it wasn't a mistake after all. Holding people accountable is way better than micromanaging them.
8. You have to make choices! Choices are decisions where there is no single correct obvious outcome. You have to do it and then you have to own it when it's not perfect. This is usually better than doing nothing.
9. Don't try to hide behind data driven decision making. Sometimes there's just no data to support what has to be an intuitive decision. Just because it's measurable doesn't mean it's the most important factor. Use data to reflect on the decision afterwards and learn from it.
10. Do your best to be made redundant. Mentor folks, try to help people grow, give junior engineers the maximum autonomy and responsibility they can handle. Once someone else can do your job then you can go take on bigger challenges.
11. Don't be an asshole. The tech industry is small and everyone knows the people they trust and who they'd want to work with again. This is the iterative prisoner's dilemma and the optimal strategy (career wise and as a human being) is just to be a good person and try to do the right thing.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
In distributed systems there's no magic "push everything to prod at once" button. Every service gets pushed independently and nodes within a service get updated incrementally. If you mess up forwards/backwards compatibility you can fail irrecoverably.
So how to avoid this?
1/5: Decouple data and code changes. Never push out a release that changes how data is stored at the same time as the code that uses this new data. If there's a bug and you need to roll back to the old version of your code it won't be able to handle the new data in the new format. Instead push out a release that first changes the data in a way that’s compatible with both the old and new code (e.g., optional fields etc), when that’s stable push out the new code that uses it, then when that’s stable you can change the data to remove backwards compatibility. This is known as a “migration” in the database world and yes it’s annoying, but yes you need to do it.
2/5: Don’t change two services at once. If service A talks to service B, you can’t just add a new API to both of them and push them out. What if someone pushes A but not B? What if there’s a bug in B that needs to be rolled back? Just like with data changes, API changes need to be made in a forwards and backwards compatible way. Engineers forget to do this all the time.
3/5: Only allow one version “step” to exist in prod at any time. It’s common to have most of your nodes at version 5 but a few are still at version 4 because they haven’t finished migrating. Never ever allow someone to push to version 6 while version 4 is still running. Otherwise it’s too hard for engineers to reason about which version is “stable” when making multi-step migrations. You need monitoring and alerting for this plus protections agains corner cases, e.g., if a node was offline and then came back online after missing a code push.