Thread by @percyliang

📣 CRFM announces PubMedGPT, a new 2.7B language model that achieves a new SOTA on the US medical licensing exam. The recipe is simple: a standard Transformer trained from scratch on PubMed (from The Pile) using @mosaicml on the MosaicML Cloud, then fine-tuned for the QA task.

Details: We took Hugging Face’s Transformer implementation, added FlashAttention, built our own tokenizer, and trained over 300B tokens (110 GB text) on 128 A100 GPUs for ~6.25 days. We did full fine-tuning on downstream tasks (e.g., MedQA-USMLE) for evaluation.

PubMedGPT is also capable of generation, but like most LMs, it will fabricate content (so don’t trust it!). This is a pressing area for LM research, and we hope that the release of this model can help researchers evaluate and improve the reliability of generation.

We hope that PubMedGPT can serve as a foundation model for biomedical researchers; can it be adapted fruitfully for tasks such as medical text simplification, information retrieval, and knowledge completion? There's a lot more to do!

There are many large, interesting datasets across different sectors - e.g., medicine, law, finance. Rather than relying on a single 100B+ parameter foundation model, we think there’s a lot of value that can be captured by <10B parameter models trained on domain-specific datasets.

Blog: crfm.stanford.edu/2022/12/15/pub…
GitHub: github.com/stanford-crfm/…
Model: huggingface.co/stanford-crfm/…

Thanks to Elliot Bolton, @dlwh, @michiyasunaga, @tonyh_lee, @chrmanning, and
at @StanfordHAI’s Center for Research on Foundation Models (CRFM), and @abhi_venigalla, @jefrankle, @mcarbin at @mosaicml on the MosaicML Cloud.