CAPTCHAs were originally conceived as a stopgap; an inconvenience imposed on legitimate users until we find better solutions to automated bots. But now that they're being used to train commercial ML systems, we're likely stuck with them for a lot longer.
The original idea was that CAPTCHAs force the adversary to do AI-hard research that would be useful on its own. But now they're giving the adversary the option to do something much more insidious: research how to automatically fool life-critical ML into being falsely trained.
Are there safeguards against this? Maybe. Maybe not. But all the incentives are completely different from what the original CAPTCHA idea envisioned, with potentially much higher stakes.
Anyway, leave it to the Internet to find creative ways to make bad ideas worse.
