Thread by @danielepolencic

THREAD: How to quarantine a Pod in Kubernetes.

This technique helps you with debugging running Pods in production.

The Pod is detached from the Service (no traffic), and you can troubleshoot it live.

Let's get started!

1/6

Imagine you have a Deployment with three replicas.

Each Pod has an 𝚊𝚙𝚙=𝚑𝚎𝚕𝚕𝚘 label.

A Service routes the traffic to your Pods using the selector 𝚊𝚙𝚙=𝚑𝚎𝚕𝚕𝚘

2/6

If you want to isolate a Pod you can overwrite the existing label with: 𝚔𝚞𝚋𝚎𝚌𝚝𝚕 𝚕𝚊𝚋𝚎𝚕 𝚙𝚘𝚍 <𝚙𝚘𝚍-𝚗𝚊𝚖𝚎> 𝚊𝚙𝚙=𝚍𝚎𝚋𝚞𝚐 --𝚘𝚟𝚎𝚛𝚠𝚛𝚒𝚝𝚎

Two things happen next:

3/6

First, the Service stops routing traffic to the Pod because the Service's selector doesn't match the label.

4/6

Then, the ReplicaSet notices that there are only two replicas, but you asked for 3.

The ReplicaSet creates a new Pod.

5/6

At this point, you can debug the Pod live without any traffic interfering with it.

You could use a mix of 𝚔𝚞𝚋𝚎𝚌𝚝𝚕 𝚎𝚡𝚎𝚌, 𝚔𝚞𝚋𝚎𝚌𝚝𝚕 𝚙𝚘𝚛𝚝-𝚏𝚘𝚛𝚠𝚊𝚛𝚍 or 𝚔𝚞𝚋𝚎𝚌𝚝𝚕 𝚍𝚎𝚋𝚞𝚐 to inspect the running process further.

That's it, only 3 easy steps.

6/6

Credit for this technique goes to @pjausovec who originally posted it on Reddit:

reddit.com/r/kubernetes/c…

Do you know any clever technique to debug your Pods in production?

Let me know!