Read on Twitter
Last week, we discussed techniques to speed up the training speed of large language models🔥💨
How about saving memory during inference? 🧠💾 Check out int8 & int4 quantization, which is supported in Lit-LLama 👉github.com/Lightning-AI/l…
🧵1/3
#LLMs #ML #DeepLearning
How about saving memory during inference? 🧠💾 Check out int8 & int4 quantization, which is supported in Lit-LLama 👉github.com/Lightning-AI/l…
🧵1/3
#LLMs #ML #DeepLearning
How does int8 quantization work? 🤔
It's a 2-part procedure with
1) using 8bit quantization
2) 16-bit matmuls for outlier feature dimensions
Check out the LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale paper for details arxiv.org/abs/2208.07339
🧵2/3
It's a 2-part procedure with
1) using 8bit quantization
2) 16-bit matmuls for outlier feature dimensions
Check out the LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale paper for details arxiv.org/abs/2208.07339
🧵2/3
And how about int4? 🤔
It's a one-shot weight quantization method based on approximate second-order information⚙️📉
For more details, see GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: 📚🔍 arxiv.org/abs/2210.17323
🧵3/3
It's a one-shot weight quantization method based on approximate second-order information⚙️📉
For more details, see GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: 📚🔍 arxiv.org/abs/2210.17323
🧵3/3
• • •
Missing some Tweet in this thread? You can try to
force a refresh
