Read on Twitter
Ladies and Gentlemen. GPT-4 👇 
*Ported from HF's code
Firing LLaMA-7B 8xMLP MoE now, wish me luck!
OK we are going somewhere! it definitely learns so the router net is probably fine.
Now I just need to understand if we want to freeze the backbone on all machines when training the experts or we can somehow sync them and we are ready to train wizardcoder-hermes-airoboros experts
Now I just need to understand if we want to freeze the backbone on all machines when training the experts or we can somehow sync them and we are ready to train wizardcoder-hermes-airoboros experts
• • •
Missing some Tweet in this thread? You can try to
force a refresh
