Thread by @LucileSaulnier

🎉 So proud to announce that our "Distributed Deep Learning in Open Collaborations" paper has been accepted at #NeurIPS2021! 🎊

Blog post: huggingface.co/blog/collabora…
Arxiv paper: arxiv.org/abs/2106.10207
Teaser: via @YouTube

#researchpaper #MachineLearning

🔐🌐This work proposes DeDLOC: a method to collaboratively train large neural networks with diverse and scattered computing resources

Among the experiments conducted, we trained SahajBERT a Bengali language model with 40 volunteers that is competitive with SOTA models 😱

Open Sourced model checkpoints

✨Pretrained : huggingface.co/neuropark/saha…
🧩Classification: huggingface.co/neuropark/saha…
🖌️Named Entity Recognition: huggingface.co/neuropark/saha…

And this project is itself the result of a fruitful collaboration with @YandexAI, @HSE_eng, @mipt_eng, @UofT, @VectorInst and @NeuroPark.

➡️

https://twitter.com/m_ryabinin/status/1406988626374762503?s=20

A huge thank you to all my co-authors: @michael_diskin, Alexey Bukhtiyarov, @m_ryabinin, @qlhoest, Anton Sinitsin, Dmitry Popov, Dmitry Pyrkin, Maxim Kashirin, sasha_borzunov, @avillanovamoral, @deniskamazur, Ilia Kobelev, @YJernite, @Thom_Wolf, and Gennady Pekhimenko

I'm sure there are many great things to come in this field of research! 🔭