Thread by @samsworldofno

One of my favourite stories about product / engineering / scaling / really knowing your customer:

📈 I worked on an e-commerce project which had been very successful and started handling a lot of volume, straight away.

This was before virtualisation and elastic scaling, so the servers we had were the servers we had. Keeping the site running in busy times was hard.

An odd behaviour started: the site would crash on the hour, every hour. To the minute.

🤯 We couldn't figure out why.

There was no rush of traffic to the site, but the servers got crazy busy and started falling over. We were baffled. What's more, it wasn't consistent - sometimes it was in the evening, sometimes in the day.

At the same time, we had an issue with the back office part of the site. One page, the order list, showed all pending orders, and because the site had been very successful, this page had become heavy - a page weight of ~5MB.

This was 15 years ago, and 5MB, refreshed constantly by 100+ call centre agents, was killing the client's internet connection.

💡 We came up with a pragmatic solution: the call centre agents would rely on the browser-cached copy of the order list.

They'd load the list, open individual orders in new tabs, and then close the tab and revert to the already-loaded page to work on the next order. They were advised to only reload the main screen once an hour or so.

I happened to be working in client's office one evening, when the site was crashing on the hour, and the mystery was solved.

🕗 As the clock struck 8pm, the call centre director walked out of his office and called out to the whole floor: "Now... everybody refresh their order lists!"

Because the main site shared servers and DB with the back office software... it promptly crashed. Whoops. 🤦‍♂️

And why did it only crash sometimes? It depended which call centre manager was on shift - only some of them had started asking for the hourly refresh to be done on the hour. We would have NEVER figured that out.