The course of history now critically depends on whether GPT-4 and GPT-5 will show diminishing performance returns.
600 billion parameters: arxiv.org/abs/2006.16668
"We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art."
"We demonstrate that such a giant model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to achieve far superior quality for translation from 100 languages to English compared to the prior art."
You might have missed this or didn't care: gwern.net/newsletter/202…
This is, by a large margin, the most important thing you can read anywhere. It's equivalent to an article discussing empirical evidence for a 1% probability of a huge asteroid hitting earth before 2030.
This is, by a large margin, the most important thing you can read anywhere. It's equivalent to an article discussing empirical evidence for a 1% probability of a huge asteroid hitting earth before 2030.
GPT-2: *writes poems*
Skeptics: Meh
GPT-3: *writes code for a simple but functioning app*
Skeptics: Gimmick.
GPT-4: *proves simple but novel math theorems*
Skeptics: Interesting but not useful.
GPT-5: *creates GPT-6*
Skeptics: Wait! What?
GPT-6: *FOOM*
Skeptics: *dead*
Skeptics: Meh
GPT-3: *writes code for a simple but functioning app*
Skeptics: Gimmick.
GPT-4: *proves simple but novel math theorems*
Skeptics: Interesting but not useful.
GPT-5: *creates GPT-6*
Skeptics: Wait! What?
GPT-6: *FOOM*
Skeptics: *dead*
"I think we right now have the ability to build an orders-of-magnitude more powerful system than we already have, and I think GPT-3 is the trigger for 100x-larger projects at Google and Facebook and the like, with timelines measured in months." lesswrong.com/posts/N6vZEnCn…
"We might need zero more cognitive breakthroughs, just more refinement / efficiency / computing power to create an AI capable of making an Industrial Revolution sized impact." lesswrong.com/posts/3nDR23ks…
GPT-N might achieve human-level intelligence simply by scaling the same architecture further. But people are also working on improving the architecture, with quite a lot of success reddit.com/r/MachineLearn…
More about GPT-3's astonishing capabilities. Be sure to read the whole comment: lesswrong.com/posts/W3DbNmuM… 
How 2020 might be remembered:
GPT-3 ≫ Recession ⩾ COVID-19
GPT-3 ≫ Recession ⩾ COVID-19
