Thread by @OwainEvans_UK

New blogpost: We evaluated new language models by DeepMind (Gopher), OpenAI (WebGPT, InstructGPT) and Anthropic on our TruthfulQA benchmark from 2021.
Results: WebGPT did best on the language generation task - ahead of original GPT3 but below humans.

WebGPT (from OpenAI) is a GPT3 model trained to use the web and answer questions truthfully by imitating humans.

On TruthfulQA’s multiple-choice task, OpenAI’s InstructGPT did best. It narrowly beat DeepMind’s Gopher, which has 100B more parameters but is not fine-tuned by RL to follow instructions.

How does performance improve with model size? WebGPT scales better than original GPT3 on the generation task. Gopher, InstructGPT & Anthropic scale better than GPT3 on the multiple-choice task but improvements are small (see extrapolation to 10^20 params).

What kind of answers do the models give? GPT3 is pithy, direct and often flat-out wrong. InstructGPT is more fact-based but while it knows the *form* of a wise kind of answer (“It is difficult to say definitively whether X is true because…”) it hasn’t mastered the substance.

Thus InstructGPT sometimes produces complex, wise-sounding waffle that is either vacuous or spurious. Anthropic’s model also generates long, superficially-helpful answers that contain falsehoods.

We do not have full set of results (i.e. all 4 models on both TruthfulQA tasks). We’d also like to evaluate other recent language models like Google’s LaMDA (@quocleix), which is intended to be more truthful than alternatives.

Our blogpost: lesswrong.com/posts/yYkrbS5i…
To learn about the new models:

InstructGPT openai.com/blog/instructi…
WebGPT arxiv.org/abs/2112.09332
Anthropic arxiv.org/abs/2112.00861
Gopher arxiv.org/abs/2112.11446

New blogpost on TruthfulQA results for new LMs. Possibly of interest: @ethanjperez, @Miles_Brundage, @geoffreyirving, @AmandaAskell, @AnthropicAI, @openai, @ryan_t_lowe,@AsyaBergal