,
4 tweets,
1 min read
Read on Twitter
I've had many failures.
Once had an app in production that was using two EC2 instances as it's backend. I was "tidying up" and deleted them as I thought I was in a different region. I had a backup and didn't lose anything...
... except ...
Once had an app in production that was using two EC2 instances as it's backend. I was "tidying up" and deleted them as I thought I was in a different region. I had a backup and didn't lose anything...
... except ...
... I'd missed a single hard coded URL in the app. This URL referenced one of the instances, and so the app stopped working completely.
I realised almost immediately what happened and the mistake I'd made, as did every other user of the app...
The app?...
I realised almost immediately what happened and the mistake I'd made, as did every other user of the app...
The app?...
... used by a very large brand to manage and communicate with their sales workforce across the country.
So I had to create a new version of the app and roll it out without the bug... Took about 6 hours to resolve it.
My fault.
Completely avoidable.
Everybody makes mistakes.
So I had to create a new version of the app and roll it out without the bug... Took about 6 hours to resolve it.
My fault.
Completely avoidable.
Everybody makes mistakes.
I thought I'd don't everything right, and I nearly had. If I'd not had the hard coded URL, then there may have been a blip of downtime on the app but that's all.
Taught me the importance of separating environments, CD, and robust release processes.
Taught me the importance of separating environments, CD, and robust release processes.
