Thread by @BlancheMinerva

ITT: an OAI employee admits that the text-davinci API models are not from their papers.

Until @OpenAI actually documents the connection between the models in their papers and the models released via APIs, #NLProc researchers need to stop using them to do research.

https://twitter.com/janleike/status/1584618242756132864

@OpenAI This is not a minor point either. Apparently the text-davinci-002 API “is an instruct model. It doesn't uses a similar but slightly different [sic] training technique but it's not derived from davinci. Hence it's not a fair comparison.”

@OpenAI Note that the text-davinciplus-002 model that he mentions isn’t publicly available AFAIK. So external researchers trying to study the InstructGPT models not only are running the wrong models, but they can’t study the correct ones.

https://twitter.com/janleike/status/1542224580483330049?s=20&t=2yY4wBKko_eSo4VdGKVLZA

@OpenAI This is a disaster for scientific research… every external paper that evaluates the impact of instruction-tuning on model performance using the OpenAI API needs to be reviewed and very likely retracted.

@OpenAI This is far from the first time that @OpenAI has made secretive moves that undermine scientific research. The codex models have been changed over time for example, without any announcement or documentation: And even the original text APIs are problematic, as before @AiEleuther

@OpenAI @AiEleuther did a systematic analysis, many papers were written that made false assumptions about the size of the OpenAI GPT-3 models (they’re every other size starting from davinci = 175B and counting down)

blog.eleuther.ai/gpt3-model-siz…

@OpenAI @AiEleuther For example, @DanHendrycks’s widely cited “Measuring Massive Multitask Language Understanding” paper is wrong about the size of the models it evaluates

arxiv.org/abs/2009.03300

@OpenAI @AiEleuther @DanHendrycks Since Dan’s paper only looks at the OpenAI API, there was really no way for him to tell something was wrong. But here’s scaling plots for MMMLU categories comparing the OpenAI API models to publicly available ones. If MMMLU was right about model sizes

@OpenAI @AiEleuther @DanHendrycks the second smallest OpenAI model (blue) would the size of the largest FairSeq model (green)

Plot is from the GPT-NeoX paper: aclanthology.org/2022.bigscienc…