Thread by @evanderkoogh

I know I should know better, but I am again blown away by how incredible @honeycombio is..

We had an incredible weird bug where sometimes a header wouldn't be set for what seemed like a particular customer.
And it would crash the server process for only one particular employee.

And also for me, but not my co-founder.

After a quick @honeycombio query on all the requests from my IP with their sitename gave us the request IDs to search Cloudwatch and found the bug. Turns out if a cookie wasn't set a regex would explode..

But... that should be impossible!

Because if the cookie isn't set on the request, the server would set that cookie on the response itself.

Surprise surprise that wasn't the case.

So it would only blow up for people who had not visited that site before. Turns out my co-founder had done that, I hadn't.

So I found the cause of the crashes, but we still had another bug: Sometimes no cookie header was set.

Another query, this time for all requests where response content-type was html, but the set-cookie header didn't contain linc_guid broken down by site.

And the one site showed up, but also a few others. All of them using NextJS. Flipping to the code the only real code path split was an if statement on the content type to check if it was html. So I had a hunch.

Next query was all requests where set-cookie did *not* contain linc_guid, where content-type contained html, broken down by content-type. Result was a bunch of request where content-type was 'text/html; charset=utf-8'.

A query for all requests where set-cookie header *did* contain linc_guid, where content-type contained html, broken down by content-type, the result was lot of requests where content-type was 'text/html'.

That is when I noticed the typo in the if-statement.

Total time hunting down, fixing and deploying those fixes was <15 minutes. It probably took me more time to write up this Twitter thread :)

I am certain it would be impossible to run @linc_bot with such a small team without @honeycombio.