9/ Neural nets can consume GBs of memory. GPUs only have MBs of on-chip memory. So GPUs store neural nets on external memory soldered next to it on the PCB.
The problem is external memory is 10-100x slower & more power hungry vs. on-chip memory. They are also very expensive.
10/ Large models like Google’s Neural Machine Translation don’t even fit in one GPU’s external memory. Often they have to be split up across dozens of GPUs/servers. This increases latency by another 10-100x.
Ideally the whole model fits on a single chip—that's Cerebras' WSE.
11/ Cerebras’ Wafer Scale Engine (WSE) is *one chip* holding 400,000 cores and 18GB of memory. Neural network training happens on one piece of silicon rather than spread across dozens of boards, servers, interconnects. If it works, one chip can replace a rack of GPU servers.
12/ Healthy skepticism is warranted. The industry has never seen anything like this before. It might not live up to these lofty goals for all kinds of reasons. Cerebras says they have customers in trials now and official benchmarks in November so we’ll see.
13/ Lastly, while it’s easy to get excited about what this will do for existing AI algorithms, whether it's 10x or 100x speedup on GPT-2 or BERT, the real excitement is what it will enable.
14/ As @ylecun has pointed out, hardware informs software. The kind of neural nets we have today is a function of the GPUs we have. If wafer sized chips become the norm, we could invent entire new classes of algorithms. Exciting days ahead! /fin
@ylecun Ps. Credit goes to @anandtech for the slide photos from Hot Chips.
anandtech.com/show/14758/hot…
@ylecun @anandtech Specs comparison: Cerebras Wafer Scale Engine vs. Nvidia Volta GPU vs. Nvidia DGX-1 server
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.