Thread by @adamhjk

Huh. One the one hand: it’s automatic deployment and management of a complex database across multiple clusters. One the other hand, holy guacamole that’s a lot of management tech. I really like the post - details, details.

https://twitter.com/alexwilliams/status/1435604248331493385

I don’t think it’s possible to do inherently complicated management tasks “simply” - too many edge cases stack up on you.

I’m reminded of what happened in the Chef community over time - we all headed to patterns like Helm, then patterns like Operators, and eventually unwound it all and said it’s better to ignore the edge cases you don’t care about.

I don’t really have a solution, or even a criticism. More an observation framed like a question - does the existence of a programable reconciliation loop that exposes more scheduling concerns alter the dynamics of how widely applicable generic automation is at large scale?

I think the answer is going to be no, in the end. For software like Cassandra, you’re getting pretty niche: do you want to manage your own cluster, scheduler, deployment, and automation? How about when you find a bug in the automation?

This is versus buying a managed service, or building bespoke automation that handles your most critical tasks.

We’ve focused most of the last decade plus on trying to build frameworks for hands off automation, but we haven’t focused on making operating what we have directly any easier.

Think of all the operations this Cassandra automation needs to do. How do we call them? Monitor progress? Can we do them bespoke? Can we layer them?

Infra Automation today removes direct control in order to provide benefits of repeatability and scale by having you codify your behaviors in code. But as St. Burgess has tried to tell us for so long, perhaps we aren’t modeling the problem well?

That we would give up direct control to gain automation doesn’t really make sense. We wind up with giant hammers, like an operator - encapsulate all this complicated behavior, hope it’s bug free, and run it. If it doesn’t work right, well…

Start from scratch, or fix the bug if you can. But you probably cannot, because you lack the domain expertise both in Cassandra *and* in operators *and* in this specific complicated code base.

So you’re back to bespoke automation. And you can’t leverage any of the work the operator already does that might be bug free, because it’s all bundled up in a giant ball of behavior in the operator.

The way we compose the automation forces this choice. It’s essentially a hard requirement of the interfaces we choose to expose. Move all the logic into ever more complex systems we cannot operate or decompose, and hope it’s bug free and fits your use case.

Or write your own bespoke automation for your use case, and at least be able to understand its behavior, even if it doesn’t cover every edge case. But want to reuse an interface for a single operation? Not a thing, chicken wing.

I think it’s a composition problem with how our frameworks function. We don’t model the things we want to *do* in a reusable way. Instead we bundle them all up, expose all the knobs and hope.

So it isn’t really that we can’t have generic automation at scale - it’s that the interface to it won’t be “the god operator for cassandra”. It will be “here are the things cassandra can do operationally”, and then we compose *those* into reusable complicated behaviors.

We need to be able to interact with the system safely at every level. That’s impossible with every current automation approach, essentially. It’s always all or nothing.

It’s this way by design. Not just in k8s - chef, puppet, ansible, Pulumi, terraform, whatever - We all thought it would work. I’m increasingly convinced that it won’t. Giant balls of complicated behavior you can’t decompose fail us as generic answers to our problems.