Discover and read the best of Twitter Threads about #microsoftBuild
Most recents (1)
Watching @karpathy presentation from today and taking twitter notes, come along for the ride:
If you're like only the practical tips, skip to #32
@karpathy starts with stages:
1 - Pre-training - months x thousands of GPUs
2, 3, 4 - Finetuning stages that take hours or days
1/
If you're like only the practical tips, skip to #32
@karpathy starts with stages:
1 - Pre-training - months x thousands of GPUs
2, 3, 4 - Finetuning stages that take hours or days
1/
Before pre-training happens, there are 2 preparation steps.
Data collection - Get tons of data from different sources (here Andrej LLaMa mixture)
Tokenization - a lossless translations between pieces of words and integers.
2/
Data collection - Get tons of data from different sources (here Andrej LLaMa mixture)
Tokenization - a lossless translations between pieces of words and integers.
2/
"You shouldn't judge the power of the model just by the number of parameters it contains"
LLaMa has trained on 1-1.4 Trillion tokens vs 300B tokens in GPT-3.
3/
LLaMa has trained on 1-1.4 Trillion tokens vs 300B tokens in GPT-3.
3/
