Thread by @hbhanoo

with 89 days left in my #yeartolive, I thought I'd share some old #amazon stories. The first is about how the CTO told me I had caused "The Worst Bug in Amazon's History" (which I hope is no longer the case), and still got to keep my job.

Straight out of school I got to join this amazing team which was working on converting a monolithic c binary which ran the whole website (compiler flags for different locales) into a Service Oriented Architecture (though that word didn't exist at the time).

In 2003 I started working on a project to help build some internationalization into the web app container. We had to account for old browsers that didn't behave well with different character encoding systems (utf8 still wasn't widespread)

As part of that, I had to modify the code that handled query parameters. At the time, I'm embarrassed to say, our code didn't have too many unit tests. As it so happens, I forgot to null out one of my member variables in the constructor initializer (C++).

The effect of this was that *sometimes* (but not always, and not deterministically), the code would silently drop any input that came after a '#'.
The unfortunate thing is that apartment numbers often start with a '#'.

Because it was non-deterministic, we had no way to know which addresses had been affected by this. The more unfortunate thing is that shipping providers penalized us everytime we had a incompletely addressed package.

Because it was non-deterministic, we had no way to know which customers to ask to update their address until we got charged.

One day I was in my office coding away when the CTO Al (prior to @Werner - who I ended up working for later) walked into our shared office and asks "so who caused the form parameter bug?" I raised my hand right away, not knowing anything about the impact at the time.

He said something like (my memory is hazy): "Congratulations. That is the worst bug in amazon's history. You just cost the company somewhere between $10M - $15M." He wasn't angry. He just said it matter-of-factly, and then walked out. I was shell-shocked.

@guidoism @clazier @rsnodgrass may have been in the room at the time. Then our tech lead Paul turns to me and says something like "Sometimes you just have to eat your dogshit without the ketchup!"

(...which still doesn't make sense to me. Though it definitely felt like I'd just eaten dogshit). I don't remember anything else about that day.

The amazing thing about amazon's culture, though, is that this didn't have a negative impact on my career, and there was no finger pointing (after that).

The culture was forward looking and all about fixing problems, and making sure they never happened again; never about looking backwards and trying to place blame.