Profile picture
Matt Klein @mattklein123
, 11 tweets, 2 min read Read on Twitter
Let me tell you about the time that I (indirectly) logged out about 40M Android users from @Twitter with a single character bug. They couldn't log back on for several hours, and the follow on impact on logged in users was ... not great.
In 2013 I wrote out the HTTP Date header using strftime() in Twitter's new edge proxy. Instead of using %Y for the year, I used %G. I can't really say why, probably because it comes first in the man page.
%G outputs the "week year." What the fuck is a week year you ask? Good question. Look it up. I had no idea what it was either until... later.
This code ran in production for a year and a half without any issue, until a few days before NYE '15, at which point the "week year" switched to '15, while the calendar year was still '14. Oops.
I don't recall the full details, but this exposed an issue with how Android was doing auth token validation, which was relying on the HTTP Date header. When the date jumped forward by a year, the token become "too old," was deleted, and the user was force logged out. Oops.
Meanwhile, I'm in the middle of nowhere in Idaho without any real internet access. People are franticly debugging and I start getting random text messages. This eventually escalates to SMS debugging as I'm being sent copy/pasted code via text.
Eventually, someone figures out the bug, it gets fixed and deployed after several hours, and now Android users can start logging back in. However, of course, many of them don't. Oops.
This event kicked off a shitshow of absolutely epic proportions that ran for at least another month, the details of which I won't go into. I will say that how the incident was handled ultimately led to me and quite a few other people leaving Twitter. Blameless postmortems anyone?
Being as far removed from this incident as I am now (and knowing what I know now), I actually find it pretty funny, but I assure you it was not funny at the time.
I often say to the juniors on my team: if you don't break something in a way that someone notices every (long) once in a while you are not doing big enough things. I try to live what I preach. 😉
In summary: do big things, 🔥 brightly, and when you fuck up, own it. And... never for%Get the oopses. 😉
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Matt Klein
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($3.00/month or $30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!