Having multiple tenants sharing a Kubernetes cluster makes sense from a cost perspective, but what's the overhead?

How much should you invest to keep the tenant isolated, and how does it compare to running several clusters?

We ran three experiments and recorded the costs. Before examining the costs, let's look at the scale of the problem.

Most teams partition their cluster by environment.

For example, ten teams might have three environments each (i.e. dev, test and prod).

If you partition the cluster by environment and team, you will have 30 distinct slices.

By default, Kubernetes doesn't recompute and rebalance workloads.

You could have a cluster with fewer overutilized nodes and others with a handful of pods

How can you fix this?

🚨 Spoiler: you can watch Chris talking about this next week:

Continues…👇 bit.ly/k8s-optimize-3
1/

Let's consider a cluster with a single node that can host 2 Pods

You maxed out all available resources so you can scale the cluster to have a second node and spread the load

What if you could choose the best node for your Kubernetes cluster before writing any code?

Imagine being able to estimate:

- Utilization.
- Overcommitment.
- Wasted resources.
- Costs.

And compare the results for multiple setups.

Let me show you how. 1/

First, not all resources in worker nodes can be used to run workloads

You need to account for memory and CPU used by kubelet, kube-proxy, operating system, etc

⎈ 20 Kubernetes threads in 20 weeks ⎈

I shared one (interesting) Kubernetes weekly thread for the past five months.

Here's the complete list: 1/

Isolating Kubernetes pods for debugging

https://twitter.com/danielepolencic/status/1612421361011400705

How does Pod to Pod communication work in Kubernetes?

How does the traffic reach the pod?

Let's dive into how low-level networking works in Kubernetes. 1/

When you deploy a Pod, the following things happen:

➀ The pod gets its own network namespace
➁ An IP address is assigned
➂ Any containers in the pod share the same networking namespace and can see each other on localhost

If you've ever interacted with a Kubernetes cluster in any way, chances are it was powered by etcd under the hood.

But even though etcd is at the heart of how Kubernetes works, it's rare to interact with it directly daily.

Let's explore how it works! 1/

Architecturally speaking, the Kubernetes API server is a CRUD application that stores manifests and serves data

Hence, it needs a database to store its persisted data, which is where etcd fits into the picture

In Kubernetes, you can use labels to assign key-value pairs to any resources.

Labels are ubiquitous and necessary to everyday operations such as creating services.

However, how should you name and use those labels? 1/

Let's start with the basics

Any resource in Kubernetes can have labels

Some labels are vital (e.g. service's selector, operators, etc.), and others are useful to tag resources (e.g. labelling a deployment)

Does Kubernetes rebalance your Pods?

If there's a node that has more space, does Kubernetes recompute and balance the workloads?

Let's have a look 👉 1/

You have a cluster with a single node that can host 2 Pods

If the node crashes, you will experience downtime

You could have a second node with one Pod each to prevent this

In Kubernetes, what should I use as memory requests and limits?

And what happens when you don't set them?

Let's dive into it. 1/

In Kubernetes, you have two ways to specify how much CPU a pod can use:

➀ "Requests" are usually used to determine the average consumption
➁ "Limits" set the max number of resources allowed

When running an EKS cluster, you might face two issues:

- Running out of IP addresses assigned to pods.
- Low pod count per node (due to ENI limits).

Let me show you how you can fix those. 1/

Before we start, here is some background on how intra-node networking works in Kubernetes

When a node is created, the kubelet delegates:

➀ Creating the container to the Container Runtime
➁ Attaching the container to the network to the CNI
➂ Mounting volumes to the CSI

Kubernetes exposes a robust API that lets you control every aspect of the cluster.

Most of the time, it's hidden behind kubectl, but no one stops you from using it directly.

Let's dive into it. 1/

First, let's recap how the API works

When you type a command, kubectl:

- Validates the request client-side
- Generate YAML on the file (e.g. `kubectl run`)
- Construct the runtime object

In Kubernetes, what should I use as CPU requests and limits?

Popular answers include:

- Always use limits!
- NEVER use limits, only requests!
- I don't use either; is it OK?

Let's dive into it 1/

In Kubernetes, you have two ways to specify how much CPU a pod can use:

➀ "Requests" are usually used to determine the average consumption
➁ "Limits" set the max number of resources allowed

Running new apps in Kubernetes is straightforward

But what happens when you have legacy apps that:

- Log to file instead of stdout?
- Has no support for Prometheus?
- Has no support for HTTPS?

First, let's cover the basics 1/

In Kubernetes, a pod is a collection of containers

Containers are isolated using two Linux primitives: control groups and namespaces

➀ Control groups are for limiting resources (max 256MB of memory)
➁ Namespaces are for isolation (you can only see this folder)

How do you gracefully shut down Pods without dropping traffic?

If you've ever noticed dropped connection after a rolling upgrade, this thread digs into the details

Let's start: what happens when a Pod is deleted? 1/

When you type `kubectl delete pod`, the pod is deleted, and the endpoint controller removes its IP address and port (endpoint) from the Services and etcd

You can observe this with `kubectl describe service`

How does the scheduler work in Kubernetes?

The scheduler is in charge of deciding where your pods are deployed in the cluster.

It might sound like an easy job, but it's rather complicated!

Let's dive into it. 1/

Let's start with the basic

When you submit a deployment with kubectl, the API server receives the request, and the resource is stored in etcd

Who creates the pods?

What happens when you create a Pod in Kubernetes?

A surprisingly simple task reveals a complicated workflow that touches several components in the cluster.

Let's dive into it. 1/

Let's start with the obvious: kubectl sends the YAML definition to the API server

In this step, kubectl:

- Discovers the API endpoints
- Negotiates the resource version
- Validates the YAML
- Issues the request

Do you know all the YAML tricks and gotchas?

Are there any YAML tips for Kubernetes?

Read along! 1/

Let's cover the basics first

YAML has three basic rules:

➀ Indentation — only 2 or 4 spaces
➁ Maps — key-value pairs
➂ Lists — collections of things

Have you ever tried to ping a Service IP address in Kubernetes?

You might have noticed that it doesn't work

Unless it just works

Confusing, I know

Let me explain 1/

Kubernetes Services exist only in etcd

There's no process listening on the IP address and port of the Service

Try to execute `netstat -ntlp` in a node — there's nothing

How do they work, then?

How does the Ingress controller really work in Kubernetes?

I had to find out for myself, so I built one from scratch in bash 1/

Before diving into the code, here is a quick recap on how the ingress controller works

You can think of it as a router that forwards traffic to the correct pods

THREAD: Isolating a pod for troubleshooting

This technique helps you with debugging running Pods in production

The Pod is detached from the Service (no traffic), and you can troubleshoot it live

Let's get started! 1/

Imagine you have a Deployment with three replicas

Each Pod has an `app=hello` label

A Service routes the traffic to your Pods using the selector `app=hello`

It's the first day in the office of 2023, and it's time to reflect on my Kubernetes journey in 2022!

What did I achieve? 1/

Let's start with a (very) low note

I co-authored only 4 articles on the Learnk8s blog

The good news is that they were absolutely epic:

➀ Kubernetes workload identities
➁ HA Kafka on Kube
➂ RBAC
➃ Hardcore networking

You can find them here: learnk8s.io/archive