Strong agree. Everyone wants actual isolation, but tbh I’m not sure k8s is the best abstraction for the job. It’s so complex and leaves so much room for error. One config gone wrong or one missed patch and it’s game over for your cluster.
You’d need to hire a perfectionist to set up your cluster perfectly and then maintain it for eternity and at that point might as well just build your own orchestrator you trust. And have a team continually pass it down.
It’s a trade-off invest in one of the few people in the world who can actually create a secure cluster or hire folks who can schedule VMs on different nodes. It’s not rocket science, just code.
Kubernetes was not made with security in mind. It was an after thought and as such you will continually be a hamster in a wheel sweating your ass off unless you build your own with hard multi-tenancy in the design.
Chances are if you do hire one of the experts they’ll just wind up building their own orchestrator anyways. I know because I did and it only took a few weeks versus playing catch up with k8s upstream and config management for the rest of my life.
Also chances are you don’t actually need hard multi-tenancy (you only need that if you are running apps for unknowns... like a lamda) and in that case do whatever you want, just don’t open your cluster to the public internet.
