Thread by @tpierrain

Interesting techniques applied to wrong usages may hurt. A thread 🔽

Seems obvious in lots of domain but not in our Software development field (note: I didn’t talk about software Engineering on purpose)

Me: Why do we have such an increase of infrastructure costs recently?

SRE: It seems that we have a new “OTF env” fad since last month… As a result, more than 70% of our global infrastructure costs are related to non-production environments...

Me: Whaaat? How does this happen? What is “OTF env” exactly?

SRE: OTF envs means “On-The-Fly environments”. It is a mechanism we have recently built in order to setup our entire stack (monolith, new apis, db, message brokers etc) on a K8 dev cluster in a couple of minutes.

It looks like some devs are using it for every new feature. They create their own OTF env for every new Merge Request.

Me: I understand the usage of OTF envs for stress tests or in order to quickly setup an iso-prod env (to reproduce and troubleshoot an issue). But…

…mounting a brand new set of pods and infra for every one of us doesn’t look like a scalable approach in term of cost and in term of Continuous Delivery practices.

I’ll talk to them.

Me: Hi, could you please tell me why you are using OTF environments? There is a bunch of them alive and it costs us a lot of money.

Dev: OTF env? Yeah, it’s a new way for us to work with our Product Manager in a stable environment for every new feature. This is so cool!

Me: What is the life time of such OTF env?

Dev: until we finished our feature and its QA, and until we’ve merged it into main.

Me: what you just describe me can last days, didn’t we say that we will going towards Continuous Delivery? That instead of releasing every 2 weeks like now, we will start working on releasing multiple times a day?

Dev: Precisely. The team that have asked this OTF envs capability are saying that they are doing Continuous Delivery.

Me: What?!? Working on long lived branches like this have nothing to deal with Continuous Delivery IMO.

Dev: they told us they were doing it, and that they were releasing their new features whenever they wanted (after their final merge).

Me: ok, Got it. It works because they are a small team working on a green field project (i.e. not legacy) where things are properly modularized.

I.e. they do not have merging conflict. But I still does not understand why they are not doing trunk-based dev in such conditions. It would have cost us less in infrastructure.

Dev: actually they do have merging issues some time to time.

Me: I’m not really surprised…

Dev: those cases arrived when someone just released in production their version. The next ones can’t do that so they are forced to merge. And I heard some cases were they were having more than 3 different releases to make in a row…

…In that case the last one may have suffered from doing multiple merges in a row. But it should be rare

Me: reason why it would have been better to identify earlier such conflicts and to reduce the complexity of the merge IMO. But anyway…

…I understand they journey looking at the size of the team etc. But what about you? You are working on the legacy which is not completely modularized. Your merge sessions should be awful if you are working in isolation for such amount of time…

Dev: Yes but it is like before. But now we have OTF envs to work quietly around a feature with our Product Manager.

Me : but what was the problem working with collective environments such as DEV & UAT ones?

Dev: They aren’t stable enough. I don’t want my work to be jeopardized by others.

Me: fair enough but why aren’t they stable? Aren’t you using feature toogles so that you can hide uncompleted features?

Dev: we are supposed to but we are still working like the old days, with draft and broken things pushed all along and a consolidation period made before the code freeze, just before the release.

Me: don’t you think that using workarounds such as OTF env for your own stability/isolation will prevent us from collectively managing our real issues (such as the low quality of what is pushed between 2 releases)?

Dev: Dunno, but OTF are really cool!

Me like…