Read on Twitter
Finetuning the embedding model can allow for more meaningful embedding representations, leading to better retrieval performance.
@llama_index has abstraction for finetuning sentence transformers embedding models that makes this process quite seamless.
Let's see how it works 👇
@llama_index has abstraction for finetuning sentence transformers embedding models that makes this process quite seamless.
Let's see how it works 👇
@llama_index Finetuning means updating the model weights themselves over a set of data corpus to make the model work better for specific use-cases.
E.g. for embedding ArXiv papers, we want the embeddings to align semantically with the concepts and not filler words like “This paper is…”.
E.g. for embedding ArXiv papers, we want the embeddings to align semantically with the concepts and not filler words like “This paper is…”.
@llama_index .@llama_index has guides on how to finetune embeddings in different ways:
- finetune the embedding model itself (only sentence transformers)
- finetune an adapter over any black-box embedding model (stay tuned for this one 🔥)
- finetune the embedding model itself (only sentence transformers)
- finetune an adapter over any black-box embedding model (stay tuned for this one 🔥)
@llama_index 3 Steps for finetuning embeddings:
1. Prepare the data via generate_qa_embeddings_pairs()
2. finetune model via SentenceTransformersFinetuneEngine
3. Evaluate the model
1. Prepare the data via generate_qa_embeddings_pairs()
2. finetune model via SentenceTransformersFinetuneEngine
3. Evaluate the model
@llama_index Here we use two pdfs: lyft.pdf and uber.pdf.
we'll use lyft.pdf to create the training set and evaluate the finetuned model using evaluation set created from uber.pdf.
we create nodes from both training and evaluation pdfs.
we'll use lyft.pdf to create the training set and evaluate the finetuned model using evaluation set created from uber.pdf.
we create nodes from both training and evaluation pdfs.
@llama_index Next we use generate_qa_embedding_pairs() to create the training and evaluation datasets from the nodes.
For each chunk, synthetic queries are created for that chunk via LLMs.
Each pair of (generated question, text chunk used as context) becomes a datapoint in the datasets.
For each chunk, synthetic queries are created for that chunk via LLMs.
Each pair of (generated question, text chunk used as context) becomes a datapoint in the datasets.
@llama_index Next we use the SentenceTransformersFinetuneEngine to create the finetuning engine.
We pass it the necessary parameters like the train and evaluation dataset, model to finetune, path to output the finetuned model etc.
We pass it the necessary parameters like the train and evaluation dataset, model to finetune, path to output the finetuned model etc.
@llama_index Finally, we finetune it using the engine created before and evaluate the new model comparing it with the base model and OpenAIEmbeddings
@llama_index In the hit rate metric, new model performs significantly well from the base model it was finetuned on and even performs almost as good as openai embedding model.
More on finetuning and the evaluation method in the official docs:
gpt-index.readthedocs.io/en/latest/exam…
More on finetuning and the evaluation method in the official docs:
gpt-index.readthedocs.io/en/latest/exam…
@llama_index Thanks for reading.
I write about AI, LLMs, RAG etc. and try to make complex topics as easy as possible.
Stay tuned for more ! 🔥 #AI #RAG
I write about AI, LLMs, RAG etc. and try to make complex topics as easy as possible.
Stay tuned for more ! 🔥 #AI #RAG
https://twitter.com/1355239433432403968/status/1711468959306785125
• • •
Missing some Tweet in this thread? You can try to
force a refresh
