Thread by @danielepolencic

THREAD

In Kubernetes ⎈, are there hidden costs on running many cluster nodes?

(spoiler: yes)

Not all CPU and memory in your Kubernetes nodes can be used to run Pods.

CPU and Memory are divided into:

1. Operating system
2. Kubelet, CNI, CRI
3. Pods
4. Eviction threshold

2/6

Let's make a quick example.

Imagine you have a cluster with a single 1GiB / 1vCPU Node.

The following resources are reserved for the kubelet + os:

- 255MiB of Memory
- 60 Mi of CPU

That's 25% of Memory and 6% of CPU that you can't use.

Let's pick GCP's n1-standard-1 instance (1 vCPU, 3.75GB).

Google reserves for the kubelet + os:

- 937MB of Memory
- 60 Mi of CPU

So you can only use ~3GB of Memory for your Pods (~75% of the available).

4/6

EKS has similar (interesting?) limits.

Let's pick an m5.large (2vCPU, 8GiB)

Amazon reserves for the kubelet + os:

- 574MiB of Memory
- 160 Mi of CPU

This time you are lucky. You can use ~93% of the available Memory.

5/6

Cost is not the only metric when choosing an instance type for your cluster, but it is certainly a factor.

If you're interested in exploring more about node sizes, we published an extensive article on our blog:

learnk8s.io/kubernetes-nod…

6/6

Links:

- Allocatable memory and CPU resources on GCP cloud.google.com/kubernetes-eng…
- AWS EKS AMI kubeReserved github.com/awslabs/amazon…
- AKS resource reservations docs.microsoft.com/en-us/azure/ak…
- Official docs kubernetes.io/docs/tasks/adm…