Mandatory reading - these two blog posts from the folks at Turbine Labs (RIP) about the perils of conflating "deploy" and "release". I also referenced these ideas in my post on testing in production (medium.com/@copyconstruct…)
blog.turbinelabs.io/deploy-not-equ… and blog.turbinelabs.io/deploy-not-equ…
blog.turbinelabs.io/deploy-not-equ… and blog.turbinelabs.io/deploy-not-equ…
These ideas aren't really anything new - feature flagging, gradual traffic shifting, canaries are all things some people have been doing for over a decade now. The lack of standardized tooling to codify these practices into normal release workflows is what has deterred adoption.
If nothing else, the rise of open source infrastructure projects like Kubernetes, Envoy, istio and Spinnaker help standardize some of these by offering a set of well-defined APIs to build workflows around (much preferable to infra teams rolling their own custom configs and hacks)
Istio, for example, supports something called "mirroring traffic" (tapping in Envoy's ability to shadow traffic), which provides a standardized way to test a change against live production traffic out of band. Having a standardized way to do this is such a relief.
Having robust testing, release and deployment pipelines can have a huge impact on the reliability of a service. But not many companies see it as their core differentiator (perhaps rightfully so?) to invest a ton of effort into it. This is a void that can be filled by OSS tools.
Or SaaS products - but the challenge here for SaaS is it's impossible to build the "One Workflow To Rule Them All" and have every company buy into it. It is much easier with solid OSS primitives atop which additional features and workflows can be built.
TLDR - there's no need for every company to be building these tools from scratch (and the reason they have to do so is also why thorough adoption of these practices remains somewhat poor). Easy-to-plug-in and easy to use OSS tooling can greatly help with adoption.
"Testing in production" used to be a meme. Now Netflix is hiring people to work on just that (jobs.netflix.com/jobs/867867). I hope they OSS some of the tools they build.
Cloud providers can - and I suspect, soon will - be doing a better job of providing some of these primitives too.
Cloud providers can - and I suspect, soon will - be doing a better job of providing some of these primitives too.
