Thread by @emilymbender

A journalist asked me to comment on the release of GPT-4 a few days ago. I generally don't like commenting on what I haven't seen, but here is what I said:

#DataDocumentation #AIhype

>>

"One thing that is top of mind for me ahead of the release of GPT-4 is OpenAI's abysmal track record in providing documentation of their models and the datasets they are trained on.

>>

Since at least 2017 there have been multiple proposals for how to do this documentation, each accompanied by arguments for its importance.

>>

Some that would be appropriate to the GPT models include Data Statements for Natural Language Processing (Bender & Friedman 2018, Bender et al 2021), Datasheets for Datasets (Gebru et al 2018, 2021) and Model Cards for Model Reporting (Mitchell et al 2019).

>>

Without clear and thorough documentation of what is in the dataset and the properties of the trained model, we are not positioned to understand its biases and other possible negative effects, to work on how to mitigate them, or fit between model and use case.

>>

That is, not beyond the obvious first pass questions of: Is this a use case where synthetic text is even appropriate? Very few use cases are."

>>

It was an easy prediction to make, given @OpenAI's track record for sure. Still, I could wish that it wasn't so thoroughly validated.

https://twitter.com/emilymbender/status/1635697381244272640