Thread by @TheSequenceAI

Deep dive into "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" by Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He & @jeffra45

It proposes an optimizer to build huge language pre-trained models.

Thread👇🏼 🔎
thesequence.substack.com/p/-edge22-mach…

Zero Redundancy Optimizer (ZeRO) is an optimization module that maximizes both memory and scaling efficiency.

2/

It tries to address the limitations of data parallelism and model parallelism while achieving the merits of both

thesequence.substack.com/p/-edge22-mach…

3/

ZeRO uses an approach called ZeRO-powered data parallelism, removing the memory redundancies across data-parallel processes

4/

+Link: arxiv.org/abs/1910.02054

+Code: github.com/microsoft/Deep…

#ML #AI #ArtificialIntelligence #DeepLearning #DataScience #article #research
5/

DeepSpeed is a new open-source framework focused on optimizing the training of massively large deep learning models.

Includes the first implementation of ZeRO as well as other optimization methods.
6/

TheSequence Edge covers:

+An ML concept you should learn
+A review of an impactful research paper
+New ML framework or platform and how you can use it
thesequence.substack.com/subscribe
7/