Thread by @jessenoller

Here my quick and dirty Kubernetes issue diagnosis a thread:

1. Random latency talking over network
A: check disk IO on the host, you’re probably exceeding the IO levels on the OS disks. I bet it’s disk

2. My cluster goes down during an upgrade
A: set a pod disruption budget

My containers are running slow or not well balanced and causing weird latency issues.

Check your CPU and memory limits in your yaml. Confirm the request limits are correct for your app and the back pressure from throttling at the container level does crash your app

If you are seeing your workload get hot and THEN crash check check disk IO, dmesg, etc for Linux OOM messages panicing your hosts

I’ve added custom rules to the network...

Double check you’re not blocking or your IT group is not auto banning ports, urls, etc (eg port forwarding being blocked by Dave’s firewall in the IT closet)

Tldr your workload can completely and totally impact Kube operations and stability unless you fully profile your app and it’s containers, especially at the bare Host VM level

Even disk IO overload at the host level lasting a a few milliseconds can cause downstream latency as components recover for minutes/hours depending on the application

Next, check DNS and network throttling. Compare the speed of the worker VM NIC and then run a hot cluster using Kubenet or CNI to calculate your SDN overhead needed (sdn run on hosts in userspace, hello cpu contention)

You also need to factor in the caching at the storage and other layers, exceeding the cache will also cause weird latency and socket failures because computer

IMO this is a blind spot between using kube and running kube in reality. You can’t ignore host level data but the abstraction layer of kube makes it super easy to forget or ignore

The thing that makes this doubly toxic is the workload density packing containers give you so we pack 3 node clusters with hundreds of containers and wonder why it behaves badly

I’m probably wrong anyway

If your app has java components, you may be seeing weird cpu and memory usage and rescheduling failures and so on. Double check your kernel and java versions don’t have cgroup bugs

Part of this is that the market has pegged Kubernetes as a PaaS which is pretty disastrously wrong. A hard part of PaaS and Serverless service is protecting against arbitrary code execute by the customer

Kubernetes isn’t a PaaS. It’s Cluster as a Service, it’s IaaS, not Heroku or app engine or PKS. Theses all blur things but they’re designed for arbitrary code execution protection at the VM or container run time

Maybe it’s@more accurate to say kubernetes isn’t an application PaaS, rather a *datacenter* PaaS- those have very very different meanings and wouldn’t help

If you want to have fun, load a fuse overlay fs on top of your normal disk for all the container components and watch the IO volumes compared to what kube metrics reports

Then run a few 1gb+ containers under load. Container size matters. Also nfs and cifs mounts

Ohhhhhhh! By the way!!! Your Kubernetes sidecars are making this worse. CNI, DNS (Core/Kube) - anything running to support your app is ALSO hyper sensitive to IO issues, Fsync() blocking waits, me means cpu contention.

DNS latency? Probably not the network.
Latency talking to the SPI server... probably not network

Oh yeah, nodes flapping not ready? Docker PLEG? Mysterious panics from external volume mounts (kernel panics), errors in component/system/console logs regarding socket read errors?

Disk pressure/IOPS overload.

Yeah, virus/malware/container runtime scanners running in the host and in the cluster context amplify the load and resource contention. Container heavy / host heavy service meshes as well.

planned for disk space, pvc space, mem and cpu but not sheer IO(ps).

Using lots of fast local storage (add, nvme, etc) helps, if you’re running your own etcd uhhh double check that though.

Even nvme/buses/caches have limits and those amplify at scale *under load* and will *look* like a software/host/managed service failure.

Striping your own raid in LVM and so on are work around but come with a pile of issues to (ceph o_o)

You should treat the os disk as inviolate and move all IO like docker, scanners, monitoring/storage, etc *off* the OS path. Give it as *much* space as you can in terms of IO