๐จModel Alert๐จ
๐๏ธโโ๏ธ State-of-the-art sentence & paragraph embedding models
๐ปState-of-the-art semantic search models
๐ขState-of-the-art on MS MARCO for dense retrieval
๐1.2B training pairs corpus
๐ฉโ๐215M Q&A-training pairs
๐Everything Available: SBERT.net
๐งต
๐๏ธโโ๏ธ State-of-the-art sentence & paragraph embedding models
๐ปState-of-the-art semantic search models
๐ขState-of-the-art on MS MARCO for dense retrieval
๐1.2B training pairs corpus
๐ฉโ๐215M Q&A-training pairs
๐Everything Available: SBERT.net
๐งต
๐จ All-Purpose Sentence & Paragraph Embeddings Models
As part of the JAX community week from @huggingface we collected a corpus of 1.2 billion training pairs => Great embeddings for sentences & paragraphs
Models: sbert.net/docs/pretraineโฆ
Training Data: huggingface.co/datasets/senteโฆ
As part of the JAX community week from @huggingface we collected a corpus of 1.2 billion training pairs => Great embeddings for sentences & paragraphs
Models: sbert.net/docs/pretraineโฆ
Training Data: huggingface.co/datasets/senteโฆ
๐จSemantic Search Models
Performing well on out-of-domain data is challenging for neural retrieval models. By training on 215 million (question, answer)-pairs, we get models that generalize well across domains.
Models: sbert.net/docs/pretraineโฆ
Data: huggingface.co/sentence-transโฆ
Performing well on out-of-domain data is challenging for neural retrieval models. By training on 215 million (question, answer)-pairs, we get models that generalize well across domains.
Models: sbert.net/docs/pretraineโฆ
Data: huggingface.co/sentence-transโฆ
๐จState-of-the-art MS MARCO Models
We mined 160 million hard negatives for the MS MARCO dataset and scored them with a CrossEncoder.
We then trained Bi-Encoders on these using MarginMSE Loss (arxiv.org/abs/2010.02666)
Code: sbert.net/examples/trainโฆ
Data: huggingface.co/datasets/senteโฆ
We mined 160 million hard negatives for the MS MARCO dataset and scored them with a CrossEncoder.
We then trained Bi-Encoders on these using MarginMSE Loss (arxiv.org/abs/2010.02666)
Code: sbert.net/examples/trainโฆ
Data: huggingface.co/datasets/senteโฆ
๐1.2 Billion Training Pairs
Training on large datasets is essential to generalize well across domains and tasks.
Previous models were trained on rather small datasets of a few 100k train pairs and had issues on specialized topics.
We collected 1.2B training pairs from ...
Training on large datasets is essential to generalize well across domains and tasks.
Previous models were trained on rather small datasets of a few 100k train pairs and had issues on specialized topics.
We collected 1.2B training pairs from ...
... different sources: Reddit, Scientific Publications, WikiAnswers, StackExchange, Yahoo Answers, Quora, CodeSearch...
They are shared in a unified format here: huggingface.co/datasets/senteโฆ
They are shared in a unified format here: huggingface.co/datasets/senteโฆ
๐ซIn July, @PatrickPlaten & @psuraj28 organized the JAX community event.
A large group of people joined to train the best embedding models in existence:
discuss.huggingface.co/t/train-the-beโฆ
We collected 1.2B training pairs and trained the models for about a week on a TPU-v3-8
A large group of people joined to train the best embedding models in existence:
discuss.huggingface.co/t/train-the-beโฆ
We collected 1.2B training pairs and trained the models for about a week on a TPU-v3-8
๐บHow did we train these models?
As part of the event I gave two talks how to train these models:
Code: github.com/nreimers/se-pyโฆ
Video: Detailed explanation about the training code
Video: Theory on training embedding models:
As part of the event I gave two talks how to train these models:
Code: github.com/nreimers/se-pyโฆ
Video: Detailed explanation about the training code
Video: Theory on training embedding models:
๐บ๐ณ What about multilingual models?
Improved multilingual models are in progress.
Multilingual Knowledge Distillation (arxiv.org/abs/2004.09813) requires good parallel data.
- Parallel data for sentences: huggingface.co/datasets/senteโฆ
- Parallel data for paragraphs: ๐ง In progress
Improved multilingual models are in progress.
Multilingual Knowledge Distillation (arxiv.org/abs/2004.09813) requires good parallel data.
- Parallel data for sentences: huggingface.co/datasets/senteโฆ
- Parallel data for paragraphs: ๐ง In progress
โข โข โข
Missing some Tweet in this thread? You can try to
force a refresh
ใ
