Thread by @pragyanatvade

Building reliable software is complex.

Here are 16 lessons I learned the hard way in the last five years as a DevOps engineer 🧵👇

#DevOps #DevOpsCommunity

1. No in-code fallbacks for configs:

If the service can't load the config on startup for any reason, it should just crash. It's easier to diagnose than the result of one borked instance going down an ancient code path due to misconfiguration.

2. Stringent RPC settings:

Zero retries and a timeout like 3X the p99. We are striving for predictability here; sprinkling retries and long timeouts as quick fixes lead to a week-long investigation and headache a year from now.

3. Never give up on local testing:

It keeps dev cycle time much shorter than needing to rely on CI or remote workspaces. Containerise local test environment. It makes it simpler to simulate the production environment in local machines.

4. Avoid state like the plague:

Managing a stateful service is 10X trickier than a stateless one. There are plenty of good-managed DBs and caches; use one of those.

5. Use GIT:

Use it for everything - infrastructure, configuration, code, dashboards, and on-call rotations. Your git repository is your point-in-time recoverable source of truth.

6. Don't waste time on code coverage:

They make excellent charts that have a minimal relationship with how much actual value your change is providing.

7. Prioritize real-world validation:

It's better to push your code to staging and show it does what you wished and doesn't break anything. The next best thing to do is integration tests. The unit test comes at last place.

8. For infrastructure changes, make plans extremely obvious:

Post the terraform plan as a comment on a pull request. There are great tools to make sure the change you think you're making is the change you're making.

9. Use Docker:

It's industry standard for a reason - wrangling dependencies in environments with tools like chef or ansible loses to these nice self-contained artefacts any day.

10. Deploying everything all the time:

Every day that goes by without you deploying increases the chances that it's secretly been broken, and it's tough to track down what went wrong two weeks after the fact.

11. Validate deployments as they go:

Use canary deployments or good readiness probes for validation. Sometimes bad images get built and deployed to prod without anyone noticing till it is too late.

12. Use Kubernetes:

Assuming you have more than one service and more than one instance, you either need service discovery, autoscaling, or deployment versioning. K8S gives infra teams scalability superpowers.

13. Use tools to manage K8S manifest like helm:

You must never use kubectl apply, edit, or delete directly. The resource lifecycle needs to be findable in version control.

14. Avoid operators and CRDs:

K8S has a steep learning curve, and custom operators lead you to the WTF that is going on in the territory. Please keep it simple.

15. Run 3 of everything:

With backups, service and DB instances, two is one, and one is none. Make sure 2 of those three instances can handle the load independently, or else you don't have the fault tolerance you think you do.

16. Structured logs are non-negotiable:

This, plus injected trace ids, gets you 90% of APM functionalities, but much cheaper and less work needed from developers.

That's a wrap!

If you wish to learn more:

1. Follow me @pragyanatvade for more of these
2. RT the tweet below to share this thread with your audience

https://twitter.com/1297136207982284800/status/1585106239943483393