,
9 tweets,
2 min read
Read on Twitter
#GPT2 is published and suddenly every ML researcher is an expert on infosec. so much interest in dual-use issues _literally_ overnight is awesome! let's not confuse knowing the limitations of language modeling with knowing how threat actors operate (1/n)
I'm seeing a lot of anecdotal and hypothetical arguments from people with zero background in this kind of work (on both sides of the weights' non-release debate). for a field that treasures empirical study, this is perplexing.
when the media, marketeers, and scientists from other fields mischaracterize our work, we get upset. it's dismissive and damaging to everything we work for. it causes stakeholders and decision makers to make misinformed judgments. why would we risk doing similar to infosec?
researchers _cannot_ be dismissive of the notion that releasing potentially dangerous models could be unwise. the long-term AI safety crowd is likely watching this unfold with bated breath and considerable disappointment
the infosec community has largely operated on the belief that potentially malicious software should be released openly so that defenses and solutions can be developed in the open. the prospect of advanced AI governance _needs_ to challenge this belief. if not now, when?
it doesn't matter that GPT-2 *probably* isn't that dangerous or that OAI's motivations are *probably* half-spoken. AI governance will be critical and it's wildly underfunded by current labs dominated by industrial concerns. conversations need to happen, and we owe this one to OAI
as capabilities progress, we may need to probe and evaluate models and agents we can't access for regulatory, competitive, or security reasons. trust complicates this effort. why not develop more nuanced ways of continuing open research efforts without exposing ourselves to risk?
we can learn a great deal from the infosec community and its bug bounty programs, but AI governance may require a new trust model and new tech to support it. let's not dismiss that, let's explore it (\end)
PS, a more eloquent message from @jachiam0, including some things I wish I'd considered 👏👏👏
