Profile picture
Michael Gasch @embano1
, 20 tweets, 5 min read Read on Twitter
I see a lot of people having problems to understand how the Kubernetes platform works at the fundamental level, e.g. resiliency and behaviour. If you start thinking about Kubernetes as a fully event-driven system, there's answers to so many "Why"'s
Think of the API server as an immutable (replicated) log (queue) and stream of events. Events are facts that can be causally related ("happened-before") or not related at all (then we say they happened "concurrently"). etcd is an implementation detail.
All controllers, e.g. the scheduler, deployment controller, endpoint controller, Kubelet, etc. can be understood as consumers and/or producers (consumers can be producers as well, and vice versa).
Consumers specify the objects (and optionally namespace) they want to receive events from the API server. This is called a "watch" in Kubernetes. Think of the combination of object+namespace as a dedicated (virtual) event queue the API server handles.
Consumers and producers don't know about each other as they're fully decoupled (by the log) and autonomous. This makes the whole system extremely scalable, robust and extensible (flexible).
Thus, by design, it's a fully asynchronous and eventual consistent platform. Information takes time to propagate from consumer(s) to producer(s). The picture above shows this where the HPA hasn't caught up with the metrics.
There's NO guarantee that the system will converge to the desired state (even if you got an "OK"/ACK from the control plane). Eg: kubectl scale <deploy> --replicas <n> (with cpu requests > cluster_capacity). You'll get a HTTP 200, even though there is no capacity left.
Another example for some unwanted "racy" conditions between controllers when scaling down services:… by @markoluksa
Controllers are stateless. For efficiency and speed received events are placed in an in-memory buffer (typically also modelled as a queue) per controller. What if a controller crashes as it does not persist state?
This is not needed! The event-driven design will replay all (appropriate) events when the controller starts, also known as event-sourcing. This is also very useful as events are "at most once" delivered, i.e. could be lost during transmission.
Kubernetes is level-triggered. If an event gets lost during transmission (e.g. network issue), next time there's a sync you're going to receive the desired state (… by @thockin ).
I described reconciliation in case of the scheduler here:…
Since information can get delivered more than once (e.g. after failure, resync, etc.) and controllers don't talk to each other directly, there's a potential of race conditions when state is to be changed (e.g. a write to same object from different controllers).
This is called optimistic concurrency and needs to be handled in the application layer (i.e. in each controller logic). Idempotency and compare and set (based on monotonically increasing resource versions) are patterns used here.
Event-driven design has many more advantages and can address a lot of problems in distributed systems (backpressure, queuing, retries, scale-out, etc.).
It's not a new idea. Relational databases work the same way (transaction log). The databases is said to be the (stale) cache of the immutable log :)
If you want to learn more about event-driven architecture, make sure to check out this video from @boner
Btw: @the_sttts and @mhausenblas have written a great series of how the API server actually is implemented. Remember: I was describing a thought model for better understanding. Hope it helps to answer some "Why"'s ;)
Closing remark: this is not specific to computer science. Events are the reason for all being. Without events, there would be no change! Think about it, it's an applied and battle-tested design ,)
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Michael Gasch
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!