Kubernetes has made huge improvements in the ability to run stateful workloads including databases and message queues, but I still prefer not to run them on Kubernetes.
Kubernetes can only meet stateful workloads half way and I lack the expertise to manage a production configuration of Kafka, RabbitMQ, or Postgres on static infrastructure, let alone a Kubernetes cluster.
Kubernetes makes it easier to deploy stateful services not manage them. Stateful services must meet Kubernetes half way and manage their own cluster membership, failover, and replication. CockroachDB and Consul are two great examples, but far from perfect.
Even when stateful services do the right things managing state is still hard. Mixing stateful and stateless applications on the same cluster elevates the complexity of the entire cluster. Cluster security and upgrades become much harder.
The solution I've been using: isolate stateful services to a dedicated set of machines or leverage a managed service. If I really need to run stateful services on a shared Kubernetes cluster, I isolate them to a dedicated node pool and disable dynamic scheduling.
Kubernetes supports stateful workloads; I don't.
