Got a pipeline with **multiple prompts**, like a DSPy program? What's the right way to jointly optimize these prompts?
Introducing MIPRO, a Multi-prompt Instruction Proposal Optimizer.
We integrated MIPRO into DSPy. It can deliver +11% gains over existing DSPy optimizers!
🧵👇
Introducing MIPRO, a Multi-prompt Instruction Proposal Optimizer.
We integrated MIPRO into DSPy. It can deliver +11% gains over existing DSPy optimizers!
🧵👇
MIPRO takes your Language Program (in DSPy), some training inputs (possibly without labels), and a metric to maximize (for accuracy, you’ll just need labels for final output).
Then it'll propose & validate instructions + examples for all prompts in your system!
Then it'll propose & validate instructions + examples for all prompts in your system!
What makes MIPRO special?
MIPRO is itself a multi-stage Language Program!
It implements an automated data scientist. First, it looks at your data to find interesting trends that can improve instruction quality! We’ve seen this find some pretty intuitive patterns!
MIPRO is itself a multi-stage Language Program!
It implements an automated data scientist. First, it looks at your data to find interesting trends that can improve instruction quality! We’ve seen this find some pretty intuitive patterns!
Then, MIPRO runs your program on several inputs to collect examples of each module.
It'll keep the “good” traces (inputs/outputs) of each module, where the final output passes the metric’s validation. E.g., the answers are correct or the output fits the required criteria.
It'll keep the “good” traces (inputs/outputs) of each module, where the final output passes the metric’s validation. E.g., the answers are correct or the output fits the required criteria.
After that, MIPRO leverages the data summary & the few-shot examples it created to propose many plausible instructions — for each module in your DSPy program!
Then, MIPRO uses the proposed instructions & examples (for each module!) as candidates for joint optimization.
Then, MIPRO uses the proposed instructions & examples (for each module!) as candidates for joint optimization.
We treat the instructions/examples of each module as hyperparameters of the program and use a Bayesian Optimizer (TPE) to efficiently search this space — while pruning bad combinations quickly.
TPE helps our optimizer select better combinations over time.
TPE helps our optimizer select better combinations over time.
Note: If you’d been using DSPy's BayesianSignatureOptimizer before this announcement, then you’ve been using early versions of MIPRO!
It’s been public for the past 2.5 months — and so many people have already used it successfully! E.g. @CShorten30 here:
It’s been public for the past 2.5 months — and so many people have already used it successfully! E.g. @CShorten30 here:
We built this easy- and free-to-use notebook showing MIPRO on a retrieval-augmented DSPy program.
Everything in there is cached already, so you don’t even need an API key to try it:
colab.research.google.com/github/stanfor…
Everything in there is cached already, so you don’t even need an API key to try it:
colab.research.google.com/github/stanfor…
It's been amazing collaborating with @michaelryan207 @lateinteraction on this work! And big thank you to @StanfordAILab and @stanfordnlp for making all of this possible 🙌
• • •
Missing some Tweet in this thread? You can try to
force a refresh
