Thread by @clusteredbytes

Previously we've seen how to improve retrieval by funetuning an embedding model.

@llama_index also supports finetuning an adapter on top of existing models, which lets us improve retrieval without updating our existing embeddings. 🚀

Let's see how it works 👇🧵

@llama_index For adapters, we pull apart every single layer of the transformer and add randomly initialized new weights.

Then, instead of finetuning all the weights, we freeze the weights of the pre-trained model, only finetune the newly added weights.

We apply similar technique here 👇

@llama_index Here we "freeze" the document embeddings, and then we train a transformation on the query embedding instead.

Thus we're not limited to only Sentence Transformer models.

We can apply this on top of any existing model without re-embedding existing data.

@llama_index The linear adapter:

The query embedding is updated using this linear transformation of the adapter:

updated_q = W*q + b

We train the linear adapter on the training corpus to find the best value for the weight and bias, W and b.

@llama_index 3 steps for finetuning adapters:

1. generate set of synthetic query-context pairs from training and evaluation dataset.
2. Fine-tuning our linear adapter on top of an existing model (e.g. ada)
3. Get the updated model using the base model and the adapter.

@llama_index we use the generate_qa_embedding_pairs function from LlamaIndex to generate both training and evaluation datasets.

@llama_index Now, we create the finetune engine with all the parameters.

@llama_index Next we perform finetuning using engine.finetune().

Finetuning adapter is not resource hungry and can be done on a macbook, no beefy GPU required.

@llama_index Then, we get the model with finetuned adapter using engine.get_finetuned_model().

We can also get the model with the finetuned adapter from the base model and the model path of the trained adapter:

After getting the model, we use it as usual.

@llama_index Instead of simple linear adapter, LlamaIndex also supports Advanced transformation using Deeper Neural Networks e.g. TwoLayerNN or even our own custom model by subclassing the BaseAdapter class.

Stay tuned for guides on these.

@llama_index As document embeddings are unchanged, we can choose to arbitrarily re-train this adapter in the future on top of changing data distributions.

Though performance increase is not as good as finetuning the entire model, but still slightly better than the pre-trained model.

@llama_index Full guide with benchmarks in the official documentation: docs.llamaindex.ai/en/stable/exam…

@llama_index Thanks for reading.

I write about AI, LLMs, RAG etc. and try to make complex topics as easy as possible.

Stay tuned for more ! 🔥 #AI #RAG

https://twitter.com/1355239433432403968/status/1718010908151095715