Thread by @adamely

On the eve of Black Friday and with Cyber Monday approaching (is it already kind of here?) I’ll tell you a story about bot ops at a kind of big global retailer and how we ensured people got PS5s. A thread… 1/I have no idea how many

Let’s level set. Yes, some bots get the thing you want but NO not all or most go to the bots. There is tech and human ops processes to fight this but first let’s talk about the bot operators.

There are two types of “bots”. The first is simple automation that individuals run in their browsers. These really aren’t the real threat to the supply and consumers that you have to worry about.

These are just fast refresh and data entry. You can do the same with pre-populating your payment info and refreshing fast. They just give a slight edge over people not using them but really low volume success.

The bigger ones to watch are the commercial bots being sole or operated by large groups. They have a dedicated developer mapping site workflows and APIs (especially mobile app APIs that are often unprotected - hint api security is a real thing to think about

These groups work over and over leading up to drops to map out the anti-bot protections sites implement. They watch for site workflow, anti-fraud, anti-bot changes and react to those

So as a retailer you do counter ops. You look at how traffic comes in, note the patterns, note the outliers, identify the bots going direct to check out, cart, etc…

But that’s not enough. The bots will walk the same path as real user to outsmart this. So you also look at session tokens. See multiple IPs/devices using the same session token? Flag it. But still not enough

Bot operators map out how long a token is good for until the device is ever retested as a bot (e.g. how long until the device might get a captcha check again) so before a drop they’ll get a clean session with enough time to buy during the drop before that token gets expired

But still not enough. These operators continuously test the controls and look for unprotected paths or weak settings. So counter ops becomes important. Release a tiny drop (~20 units) and watch what happens, use that intel to prep for the real drop.

And…seconds before the drop goes live flip on all the new protections, invalidate sessions, suspend accounts flagged as bots, close down guest check out, take action so fast that the bot operators can’t respond to your new changes fast enough effectively crippling their op

But…is that enough? Mostly but to really know the order review process is then enacted. Look for accounts, email addresses, addresses, payment info, devices that successfully got more than 1 unit, these are suspect cancel orders for the ones that are bots

Take the units, add to inventory and start all of that over again because so will the bot operators. Of course to the average consumer, none of this is seen. They think they lost the race to a bot when in really there are 100’s of thousands of ppl trying to buy < 1,000 units

That’s really just the simple version of what these teams go through and doesn’t touch on account take over, card fraud, and all the other fun things. Teams put a lot of effort into API security, anti-bot, device trust, anti-fraud, and order review.

Things like waiting queues, which consumers don’t always love, really help since they give a longer period of time to test for bots and remove them from the queue

With all that, appreciate the teams helping you get that new thing and protecting your account while fighting an active, live adversary in real-time. Now go eat pie.

Oh…and I was reminded about ensuring we knew what the counter party was doing by subscribing to their mailing lists, reading their blogs, and watching their discussions on social! This JUST landed in my throw away inbox.

Re-read this thread. So much more to add and sooo many typos. No more watching tv and typing w/o editing.