THREAD: How to gracefully shut down Pods without dropping production traffic in Kubernetes
If you've ever noticed dropped connection after a rolling upgrade, this thread digs into the details.
Let's start: πΈπ©π’π΅ π©π’π±π±π¦π―π΄ πΈπ©π¦π― π’ ππ°π₯ πͺπ΄ π₯π¦ππ¦π΅π¦π₯?
If you've ever noticed dropped connection after a rolling upgrade, this thread digs into the details.
Let's start: πΈπ©π’π΅ π©π’π±π±π¦π―π΄ πΈπ©π¦π― π’ ππ°π₯ πͺπ΄ π₯π¦ππ¦π΅π¦π₯?
1/6
When the Pod is deleted the endpoint controller removes the endpoint from the Services.
The changes are then propagated to all endpoint consumers:
- iptables
- DNS
- ingress
- etc.
When the Pod is deleted the endpoint controller removes the endpoint from the Services.
The changes are then propagated to all endpoint consumers:
- iptables
- DNS
- ingress
- etc.
2/6
While the endpoint controller removes the endpoint, the kubelet is notified of the change.
The kubelet is in charge of
- triggering the preStop hook (if any)
- sending the SIGTERM
- sending the SIGKILL if the process hasn't stopped in time
While the endpoint controller removes the endpoint, the kubelet is notified of the change.
The kubelet is in charge of
- triggering the preStop hook (if any)
- sending the SIGTERM
- sending the SIGKILL if the process hasn't stopped in time
3/6
Since the two flows are triggered independently, there are opportunities for things to go wrong.
What happens if you shut down the application before the endpoints are removed in kube-proxy or the Ingress?
The traffic flows to a delete Pod.
Since the two flows are triggered independently, there are opportunities for things to go wrong.
What happens if you shut down the application before the endpoints are removed in kube-proxy or the Ingress?
The traffic flows to a delete Pod.
4/6
ππ° π©π°πΈ π₯π° πΊπ°πΆ π§πͺπΉ πͺπ΅?
You just need to wait long enough.
ππ°πΈ ππ°π―π¨?
Wait for kube-proxy, ingress, etc. to update their endpoints.
ππ° π©π°πΈ π₯π° πΊπ°πΆ π§πͺπΉ πͺπ΅?
You just need to wait long enough.
ππ°πΈ ππ°π―π¨?
Wait for kube-proxy, ingress, etc. to update their endpoints.
5/6
Is there anything else you should do other before terminating the process?
π¬π²π.
1. Wait!
2. Close all persistent connections (DBs, WebSockets, etc.)
3. Wait for all active connection to drain
Is there anything else you should do other before terminating the process?
π¬π²π.
1. Wait!
2. Close all persistent connections (DBs, WebSockets, etc.)
3. Wait for all active connection to drain
6/6
If you want to explore more, here are a few links:
- freecontent.manning.com/handling-clienβ¦
- kubernetes.io/docs/concepts/β¦
- medium.com/tailwinds-naviβ¦
If you want to explore more, here are a few links:
- freecontent.manning.com/handling-clienβ¦
- kubernetes.io/docs/concepts/β¦
- medium.com/tailwinds-naviβ¦
