𝐄𝐧𝐝𝟐𝐄𝐧𝐝 𝐕𝐢𝐬𝐢𝐨𝐧-𝐑𝐀𝐆 𝐰𝐢𝐭𝐡 𝐂𝐨𝐡𝐞𝐫𝐞

Our data is multi-modal 🖼️, but most RAG pipelines are still text-only.

This causes massive problems with complex visual information.

With Cmd-A-Vision from @cohere you now get a sota vision model for Vision-RAG Traditional Text-RAG tries to converts images to markdown. However, it looses a lot of the rich information represented in the image 😡

𝐒𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐒𝐞𝐚𝐫𝐜𝐡 𝐨𝐧 𝟏𝟎𝟎𝐌 𝐝𝐨𝐜𝐬 - 𝐖𝐢𝐭𝐡 𝟏𝟎𝟎𝐌𝐁 𝐨𝐟 𝐌𝐞𝐦𝐨𝐫𝐲

GPU-poor and Memory-poor, and not having 500GB of memory to embed & index 100M docs?

Still want to participate at TREC-RAG 2024?

Introducing 𝐃𝐢𝐬𝐤𝐕𝐞𝐜𝐭𝐨𝐫𝐈𝐧𝐝𝐞𝐱 𝐕𝐞𝐜𝐭𝐨𝐫 𝐒𝐞𝐚𝐫𝐜𝐡 𝐟𝐨𝐫 𝐌𝐞𝐦𝐨𝐫𝐲-𝐏𝐨𝐨𝐫
100M embeddings with 1024 dimensions in float32 requires 381GB. Adding an HNSW vector index, you quickly need 500 GB of memory.

How do you make it available to the Memory-Poor?

🇺🇳𝟐𝟓𝟎𝐌 𝐖𝐢𝐤𝐢𝐩𝐞𝐝𝐢𝐚 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠𝐬 𝐢𝐧 𝟑𝟎𝟎+ 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞𝐬 🇺🇳

What could you build if your RAG has access to Wikipedia in all 300+ languages?

Available for anyone to use, using our state-of-the-art multilingual embedding model:
huggingface.co/datasets/Coher…
@pinecone showed that RAG makes LLMs better. The more data LLMs can retrieve from, the better (higher faithfulness = more factually correct).

But access to large retrieval datasets so far is challenging😡

We are here to change this 🚀

pinecone.io/blog/rag-study/

🔎Semantic Search Hackathon🔍

Today I will kick-off a 7-day virtual hackathon focused on semantic search.

A special focus will be our multilingual embedding model, which makes multilingual search 10x easier while giving you way better results.

Details:
lablab.ai/event/semantic… Most search functions are rather useless😰

Search on Wikipedia for "the capital of the United States" and the capital punishment article is ranked first. The article for Washington D.C. is not even among the top-20 results 👎

Semantic search can make your search so much better

🇺🇳Semantic Search finally works across languages! 🇺🇳

Semantic Search gives great search results, but worked so far just for English😰

Glad to share our new cohere multilingual embedding model for 100+ languages. And the results are amazing 📈

Details:
txt.cohere.ai/multilingual/ The usage is extremely simple:
- Sign-up for a free dev API key: dashboard.cohere.ai/welcome/regist…
- Install the SDK: pip install cohere
- Call co.embed with the new model

🧑‍🏫How to adapt text embedding models to a domain?

😟Text embedding models perform poorly on unseen domains
❓How to encode words you have never seen?

🎉Adaptive pre-training and Generative Pseudo Labeling can help

A 🧵 with methods & results 😟Text embeddings models perform often poorly on unseen domains.

The issue is that they don't know what certain words mean and how to represent them in the vector space.

If you have never seen the word BERT, how to know that it is connected to deep learning & NLP?

🎉Sentence-Transformers v2.2.0 released:

💬 T5 added for computing embeddings
💾 Sentence-T5 and GTR models in PyTorch on @huggingface model hub
🔒 Loading private models from @huggingface hub

github.com/UKPLab/sentenc… Sentence-T5 (arxiv.org/abs/2108.08877) and GTR (arxiv.org/abs/2112.07899) are two recent dense embedding models by @GoogleAI trained on 2B community Q&A pairs. I converted these models to PyTorch:
huggingface.co/sentence-trans…
huggingface.co/sentence-trans…

GPT-3 Embeddings by @OpenAI was announced this week.

📈 I was excited and tested them on 20 datasets
😢 Sadly they are worse than open models that are 1000 x smaller
💰 Running @OpenAI models can be a 1 million times more expensive

tinyurl.com/gpt3-emb I tested the text similarity models on 14 datasets from different domains (emails, papers, online communities) on various tasks (clustering, retrieval, paraphrase mining).

The 175B model is actually worse than a tiny MiniLM 22M parameter model that can run in your browser.

🚨Model Alert🚨
🏋️‍♂️ State-of-the-art sentence & paragraph embedding models
🍻State-of-the-art semantic search models
🔢State-of-the-art on MS MARCO for dense retrieval
📂1.2B training pairs corpus
👩‍🎓215M Q&A-training pairs
🌐Everything Available: SBERT.net
🧵 🚨 All-Purpose Sentence & Paragraph Embeddings Models
As part of the JAX community week from @huggingface we collected a corpus of 1.2 billion training pairs => Great embeddings for sentences & paragraphs

Models: sbert.net/docs/pretraine…
Training Data: huggingface.co/datasets/sente…

📺How to train state-of-the-art sentence embeddings? 📺

Just uploaded my 3-part video series on the theory how to train state-of-the-art sentence embedding models:
📺 Part 1 - Applications & Definition
- Why do we need dense representation?
- Definition of dense representation
- What does "semantically similar" mean?
- Applications: Clustering, Search, Zero- & Few-Shot-Classification...

New Project: EasyNMT (github.com/UKPLab/EasyNMT)

Easy-to-use, state-of-the-art Neural Machine Translation using @huggingface and @fairseq.

- Translation for 150+ languages
- Sentence & document translation
- Automatic Language Detection

Colab example: colab.research.google.com/drive/1X47vgSi… Currently 4 state-of-the-art models are supported:
- OPUS-MT models from @HelsinkiNLP (individual models for 150+languages)
- mBART50 many-to-many translation for 50 langs from @facebookai
- m2m_100 many-to-many translation for 100 langs from @facebookai (418M and 1.2B version)