Thread by @Neeva

Considering using a LLM for production, but it's too slow?

Well fear not! Today you can with Double Pseudo Labeling!

Here's why the @Neeva team did, and we've never looked back!

🧵

Before we get started, let's briefly go over #NeevaAI and what it is...

✅ Our AI generates answers
✅ Our Search makes sure answers are timely & factual

All in all, @Neeva combines search + AI to generate a single, cited for each of your searches.

#NeevaAI works by combining search & phases of generative LLMs to generate AI answers

Phase 1️⃣: Run per-doc summarization and question-answering models on the top results

Phase 2️⃣: Run a cross-document attributed summarizer to synthesize a single answer for the query

Example⬇️

Now let's get into it...

Multi billion parameters LLMs are very effective at making summaries of documents and web pages across domains & languages.

While their quality is high, LLMs have an insatiable appetite for pods of A100s and they can be slow to generate summaries.

Although using foundational APIs can move this burden behind elsewhere, they still can be too slow or expensive for many applications.

Waiting a few seconds to summarize an email may be ok, but users can’t wait minutes to summarize all the relevant documents for a search query.

The bar on latency is much higher.

At @Neeva our models need to run in a few hundred milliseconds but need to behave like the models that take 10s of seconds.

We've already covered how we reduce latency at @Neeva, like in this thread from our co-founder and CEO @RamaswmySridhar 👇

https://twitter.com/RamaswmySridhar/status/1621870491945533440

As well as here, where we shared how @Neeva achieved ~10x reduction in latency of a fine-tuned encoder-decoder model. 👀

https://twitter.com/Neeva/status/1622640441064579076

And also here, where we explain how @Neeva dropped the tail latency of #NeevaAI by over 3x! 😲

https://twitter.com/Neeva/status/1624526258208927746

Most recently, we covered how @Neeva used asymmetric layer pruning of encoder-decoder models to drop the latency of #NeevaAI by another 1.5-2x. 🙌

https://twitter.com/Neeva/status/1633856484609085443

Today, we're covering how pseudo labeling can be leveraged to train and optimize models.

Traditional pseudo labeling works by using a really large language model (that might be unshippable in production – e.g. Bloom) to do data augmentation & generate labels.

The approach here is simple. 🧠

1️⃣ Spin up your local member of the LLM three comma club
2️⃣ Generate labels for a few thousand examples
3️⃣ Use these pseudo labels train a smaller two comma model, that you can serve in production

This works pretty well but here at @Neeva we aren’t easily satisfied. 😏

To push further, we explored the use of multiple stages of pseudo labeling to improve performance.

Instead of shrinking down from a really large LLM directly to a servable LLM, we first shrink down to an intermediate size LLM.

Then, we generate a much larger set of labels (think millions) from the intermediate size LLM and use them to train our final servable model.

Next, we apply our previously discussed asymmetric pruning.

This final compressed model allows us to generate great summaries in under 400 ms! 🎊

Progressively shrinking models while increasing the amount of labeled data in each stage leaves us with...

🥁🥁🥁

A model which behaves like a model with an order of magnitude fewer parameters!

Enjoyed this thread?

⭐ ⭐ ⭐ ⭐ ⭐

👆 Be sure to hit that follow button to keep up-to-date with @Neeva's latest LLM learnings from the wild.