Thread by @WriteArthur

Meta has released a huge new AI language model called OPT-175B and made it available to a broad array of researchers. It also released a technical report with some truly extraordinary findings about just how dangerous this machine can be. 🧵

#AI #OPT175B

Here's the report. Everyone should read it.
arxiv.org/pdf/2205.01068…

Bottom line is this: across tests, they found that "OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes.”

Comparing it to GPT-3, another language model released last year, the team found that OPT-175B ‘has a higher toxicity rate” and it “appears to exhibit more stereotypical biases in almost all categories except for religion.”

They found that OPT-175B can make harmful content “even when provided with a relatively innocuous prompt." Meaning that it might do some nasty things regardless of whether or not you tell it to.

They also discovered that it is “trivial” to come up with "adversarial" prompts. i.e. it’s easy to trick the system into creating toxic stuff. OpenAI made a similar discovery when testing DALL-E. No matter how many guardrails you set, there's always a way.

https://twitter.com/WriteArthur/status/1512434954629746700

Because the model is large and complex, the authors also admit that they seem to only be able to sort of guess why it's so noxious. When explaining the reasons for its toxic behavior, they use phrases like, “we suspect" and "this is likely due to."

That being said, they do have a strong hunch that its toxicity could have a lot to do with the fact that "a primary source" for the system is a giant database of "unmoderated" text from...wait for it...Reddit.

By their own admission, this dataset from Reddit "has a higher incidence rate for stereotypes and discriminatory text" than similar datasets drawn from sources such as Wikipedia.

So this hunch seems to be…a good one. Imagine if instead of teaching your child to read and write using "Where the Wild Things Are" and "Goodnight Moon," you just used millions of pages of things people say on Reddit.

They also hint at a vexing catch-22: in order to be able to detect and filter toxic outputs, the system needs to be highly familiar with said toxic language. But this can also increase its open-ended capacity to be toxic....

...which reminds me of this AI for inventing non-toxic drugs that turned out to be insanely good at, well, inventing toxic drugs.

https://twitter.com/writearthur/status/1503393942016086025

So, that's the report. To be clear, the team says that OPT-175B "is premature for commercial deployment" but their findings point to just how much more work needs to be done before that happens.

A big part of why Meta released the model is so that a broader community can help address these issues. Looks like it's going to take a whole bunch of smart people to figure this out.

TBS, there may also be questions as to whether the researchers have set sufficient groundwork for that to happen. Not to mention whether OPT-175B will create real harms, even at this experimental stage.

For example, though some mitigation measures do exist to prevent harms arising from these systems, the authors admit that they have not applied such measures to OPT-175B. Why? Because their "primary goal" here was to replicate GPT-3.

That's significant. It appears that they chose not to attempt to reduce the system's propensity to be harmful because they fear that doing so would also reduce its performance relative to a competitor's AI.

They also refrained from applying what they admit is necessary scrutiny to the AI's training dataset because their "current practice is to feed the model with as much data as possible and minimal selection."

Meanwhile, their model card for the system's training dataset seems to be a bit thin on detail.

For example, they state that they are not aware of any "tasks for which the dataset should not be used."

They also decided not to comment on whether there is "anything a future user could do to mitigate...undesirable harms" arising from certain uses of the dataset, such as uses that "could result in unfair treatment of individuals or groups."

In the model card, they also seem to contradict themselves. In one place, they admit that the dataset used to train the system does "relate to people" but on the following page they claim that it does not.

...This contradiction is non-trivial. By saying “no” the second time, they were able to skip all the questions about whether the people in the dataset gave consent to be included.

In sum, the company's transparency with this has been welcomed, and with good reason. But that transparency has revealed some challenging and uncomfortable questions.
OK, that's the thread.