, 18 tweets, 3 min read Read on Twitter
People who think I don’t like Kubernetes: you’re wrong. My team manages dozens of Kubernetes clusters, thousands of pods, and tens of thousands of containers at scale. You need to know where Kubernetes’ strengths and weaknesses are to make it effective.
As a scheduler of containers, Kubernetes does a pretty good job. If you keep it focused on that key task, it can take you miles. As a manager of a large scale distributed infrastructure, it’s not so good.
People I work with are tired of hearing this, but the mantra is: smaller clusters, more of them. Turns out you can manage Kubernetes pretty effectively if you keep it at relatively small scale. Manage the meta-footprint separately.
One of the biggest operational weaknesses of Kubernetes is having a strongly consistent scheduler. When the state machine needs to track network, load balancing, app containers, namespace quotas, etc. it turns out it does a very bad job.
Strip Kubernetes down to its foundational principle: run a set of pods, and you can operationalize it pretty effectively, but even still not at scale. Take the “scale problem” away from Kubernetes and it works pretty well.
For an operator, the task of managing a smaller, distributed footprint of Kubernetes clusters becomes trivial. And they’re predictable when you take away all the “extra” weirdware. Smaller clusters, more of them.
Kubernetes is very weak as an app development utility. And if you’re using it as an app dev framework, you’re in for a long operational tail.
Don’t give app devs direct access to the kubernetes cluster. When you do, it binds the app dev lifecycle to the infrastructure lifecycle — a distinct anti-pattern. Manage the app->cluster interaction through a separate broker.
When apps are built to be agnostic to the fact that they’re running inside of Kubernetes, they do a really great job running there! Don’t bake apps into the infrastructure and your infrastructure and operations will scale.
Custom resource definitions are also a really big weakness in Kubernetes. The second you make Kubernetes the source of truth for anything more than “I should make sure the containers are running”, you’ve set yourself up for failure.
It’s appealing to do these things. Kubernetes has a ton of momentum and the community is very vocal about all the things you can do with it. There’s also not a big contingent who can say “this is an anti-pattern”.
My rule of thumb for everything in infrastructure: keep it as simple as possible. If you need robust complexities, ask yourself why, and if you should rethink your architecture so you don’t need them.
Rule #2: take away the problem of scale from infrastructure, push it back on the apps — things run surprisingly well. Smaller clusters, more of them.
Rule #3: Be able to throw away any part of your infra without affecting the apps that run on it. Keep apps out of the guts of Kubernetes and it’ll work great for you.
One thing I love about Kubernetes is that it has a model for nearly every infrastructure resource you could need, and they are comprehensive. Leverage the model for defining infrastructure resources, but extract the functionality from k8s itself and you’ll be gold.
The Kubernetes API spec should become the canonical way we represent infrastructure. Separate the model from the system ensuring the functionality. I think we’ll see a lot of this in the future.
Smaller clusters, more of them, but how do you manage an app presence across multiple clusters? This is the secret sauce and we do something really fancy to make it work (sarcasm): put the pods on the underlay with macvtap and dhcp the networking.
Turns out networks are great when all they do is switch packets and hand out IP addresses. Get rid of the overlay.
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Dan Woods
Profile picture

Get real-time email alerts when new unrolls (>4 tweets) are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!