,
11 tweets,
3 min read
Read on Twitter
This is such an extreme outlier I'm ready to believe that a nation state actor may have an expert AI team* reverse engineering and gaming social graphs and recommendation algorithms at scale silently.
* Asymmetric black hat research (see below) and/or StuxNet level competent
1/N
* Asymmetric black hat research (see below) and/or StuxNet level competent
1/N
It's quite possible for machine learning to have exploits as fundamentally severe and retrospectively obvious as the NSA's 13+ year head start in differential cryptography. White hat research is a terrible proxy for black hat research - especially for AI.
Our social media companies have issues:
- Most social media companies aren't yet competent at blocking even obvious issues let alone adversarial
- There's little to no transparency for good actors (researchers, journalists, ...)
- Profit motives above all
- Most social media companies aren't yet competent at blocking even obvious issues let alone adversarial
- There's little to no transparency for good actors (researchers, journalists, ...)
- Profit motives above all
Even with minimal access good actors have found many fundamental flaws in these ecosystems. Flaws that _should_ have been painfully obvious. These good actors generally have nearly no resources. How competent might a black hat researcher be in their place?
White hat researchers - and even grey hat researchers - are fundamentally not the same attack model as black hat researchers. How many of these companies have machine learning red teams? How many are given permission to scrape from a million botnet'ed IPs?
en.wikipedia.org/wiki/Red_team#…
en.wikipedia.org/wiki/Red_team#…
How many red teams spawn a million bot accounts? How many manipulate real people and coordinate with public figures? How many are allowed to slander or continuously attack individuals in a given clique?
---
The answer is likely zero for both practical and ethical reasons.
---
The answer is likely zero for both practical and ethical reasons.
Black hat researchers have none of these limitations. Low hanging but ethically dubious fruit - from data breaches to manipulating real people - are all ripe for them to pluck. They can play with variables white/grey hat researchers could only dream about in ethics reviews.
In machine learning simple techniques can be scarily effective when you have the right ingredients. How effective could we be if we dropped our ethical foundation? We'll never answer that ourselves but our black hat research cousins may be publishing arXiv papers in plain sight.
To reiterate a point missed by many:
"(a) Asymmetric black hat research _and/or_ (b) StuxNet level competent"
Even without any (b) level competence, script kiddies (especially financed + motivated by a nation state) could well have found an easily exploited loophole by (a).
"(a) Asymmetric black hat research _and/or_ (b) StuxNet level competent"
Even without any (b) level competence, script kiddies (especially financed + motivated by a nation state) could well have found an easily exploited loophole by (a).
That's what I mean by asymmetric black hat research - vulnerabilities that are easy to discover by unscrupulous actors and difficult to detect by good actors (either public researchers/journalists or internal teams due to bureaucracy/lack of profit motive/missing expertise/...).
Continued discussion as to why the defenses and processes that were developed in counteracting SEO are likely inadequate for what we're up against. They're related but entirely different beasts.
