Let me tell you about the time that I (indirectly) logged out about 40M Android users from @Twitter with a single character bug. They couldn't log back on for several hours, and the follow on impact on logged in users was ... not great.
In 2013 I wrote out the HTTP Date header using strftime() in Twitter's new edge proxy. Instead of using %Y for the year, I used %G. I can't really say why, probably because it comes first in the man page.
%G outputs the "week year." What the fuck is a week year you ask? Good question. Look it up. I had no idea what it was either until... later.
This code ran in production for a year and a half without any issue, until a few days before NYE '15, at which point the "week year" switched to '15, while the calendar year was still '14. Oops.
I don't recall the full details, but this exposed an issue with how Android was doing auth token validation, which was relying on the HTTP Date header. When the date jumped forward by a year, the token become "too old," was deleted, and the user was force logged out. Oops.
Meanwhile, I'm in the middle of nowhere in Idaho without any real internet access. People are franticly debugging and I start getting random text messages. This eventually escalates to SMS debugging as I'm being sent copy/pasted code via text.
Eventually, someone figures out the bug, it gets fixed and deployed after several hours, and now Android users can start logging back in. However, of course, many of them don't. Oops.
This event kicked off a shitshow of absolutely epic proportions that ran for at least another month, the details of which I won't go into. I will say that how the incident was handled ultimately led to me and quite a few other people leaving Twitter. Blameless postmortems anyone?
Being as far removed from this incident as I am now (and knowing what I know now), I actually find it pretty funny, but I assure you it was not funny at the time.
I often say to the juniors on my team: if you don't break something in a way that someone notices every (long) once in a while you are not doing big enough things. I try to live what I preach. 😉
In summary: do big things, 🔥 brightly, and when you fuck up, own it. And... never for%Get the oopses. 😉
