We're excited to open-source Mistral 🚀 - a codebase for accessible large-scale LM training, built as part of Stanford's CRFM (crfm.stanford.edu).

We're releasing 10 GPT-2 Small & Medium models with different seeds & 600+ checkpoints per run!

github.com/stanford-crfm/… [1/4]
At 10:20 PDT, @laurel_orr1 and I will be talking at the Workshop for #FoundationModels (crfm.stanford.edu/workshop.html) about Mistral, as well as our journey towards transparent and accessible training.

We hope to see you there - bring your questions! [2/4]
This was developed by a brilliant team - Jason Bolton, @Tianyi_Zh, @krandiash, @Avanika15, @RishiBommasani, @deepakn94, Tony Lee.

We're also grateful to @tatsu_hashimoto, @jurafsky, @chrmanning, @ChrisGPotts, @HazyResearch, @percyliang!

@StanfordHAI @StanfordAILab [3/4]
We're building this codebase for the community; we really want your engagement on how to "open up" training so that it's more accessible.

We're holding office hours twice a month to hear from all of you - we'd love to see you (github.com/stanford-crfm/…)! [4/4]

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Siddharth Karamcheti

Siddharth Karamcheti Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @siddkaramcheti

26 Aug
In addition to the codebase, @laurel_orr1 and I wrote up a blog post (with the rest of the Propulsion team!) describing a bit more about Mistral and our journey in more detail.

Check it out here, and we'd love to hear your thoughts: crfm.stanford.edu/blog.html [1/5]
I really hope that our voices came through; we tried to keep it light, while also hitting on the hurdles we encountered along the way!

Not everything made it into the blog, so we also recorded a light & lively 25-min podcast: soundcloud.com/propulsion-mix… [2/5]
Big thanks to everyone who helped us build Mistral -- from @Thom_Wolf & @StasBekman who helped us navigate @huggingface Transformers, to @carey_phelps for providing support with @weights_biases.

Also huge shoutout to @BlancheMinerva from #EleutherAI for providing feedback! [3/5]
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(