Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Lance Martin Profile picture
@RLanceMartin
May 30, 2023 • 7 tweets • 7 min read • Read on X
Scrolly
There's a lot of questions abt smaller, open source LLMs vs larger, closed models for tasks like question answering. So, we added @MosaicML MPT-7B & @lmsysorg Vicuna-13b to @LangChainAI auto-evaluator. You test them on your own Q+A use-case ... autoevaluator.langchain.com/playground Image
... great pod w/ @jefrankle @swyx @abhi_venigalla on MPT-7B, so used auto-evaluator to benchmark it on a test set of 5 Q+A pairs from the GPT3 paper. Results are close to larger models and it's very fast (kudos to @MosaicML inference team!) ...
Image
... @sfgunslinger also deployed Vicuna-13b on @replicatehq and it achieves performance parity w/ the larger models on this test set. Prompt eng may further improve this (v helpful discussion w/ the folks at @replicatehq / @JoeEHoover); we are looking into improving latency ... Image
... @AnthropicAI Claude-100k (discussed in more depth below) is very impressive for a "retriever-free" approach (pass the full 75 pg GPT3 pdf into the model's context window!), but costs higher latency ...
Image
... @swyx @transitive_bs and @simonw make the interesting point that smaller (open source) models as "reasoning engines" coupled to retrievers (to fetch relevant data) may be sufficient for many apps (Q+A, agents, etc). It's def worth testing them.
... all auto-evaluator code is open source here:
github.com/langchain-ai/a…
... and on app usage: in "Playground" simply upload any PDF doc. The app will use an LLM to auto-generate a Q+A eval set, run on a user-selected chain (model, retriever, etc) built w/ @LangChainAI, use a GPT4 to grade, and store each expt for you. Image

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Lance Martin

Lance Martin Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @RLanceMartin

Lance Martin Profile picture
May 11
self-verification (Outcomes) + self-learning (Dreaming) are two of the most interesting new features we shared at Code With Claude last week.

a few notes + video links to the talks ... Image
1/ Outcomes in Claude Managed Agents is just a "Ralph loop" to verify output vs a user provided rubric. it uses a grader sub-agent for the verification. some interesting points on the benefits of an isolated verifier here:
anthropic.com/engineering/ha…Image
2/ @jess__yan + i showed a toy example of Outcomes: i had an Managed Agent make a generative UI w/ metrics (charts, graphs) rendered as svg. i used an Outcomes loop to improve the render timing - Claude figured out various tricks (prompting, etc) to speed it up.
Read 7 tweets
Lance Martin Profile picture
Apr 10
i co-wrote the Anthropic engineering blog on Claude Managed Agents, and wanted to share some thoughts on agent harnesses + infrastructure for long-horizon tasks ... 🧵
anthropic.com/engineering/ma…Image
1/ Claude can perform tasks over increasingly long time-horizons. newer models (Mythos) will push this further. with Claude Managed Agents, we needed to design infrastructure that can support long-horizon work. Image
2/ we also needed a way to update the agent harness: each harness encodes assumptions about what Claude can't do. those assumptions go stale as Claude gets more capable, and can even hobble the model. Image
Read 11 tweets
Lance Martin Profile picture
Mar 22, 2024
Gave this short talk on RAG vs long context LLMs at a few meetups recently. Tries to pull together threads from a few recent projects + papers I really like.

Just put on YT, a few highlights w papers below ...
1/ Can long context LLMs retrieve & reason over multiple facts as a RAG system does? @GregKamradt and I dug into this w/ multi-needle-in-a-haystack on GPT4. Retrieval is not guaranteed: worse for more needles, worse at doc start, worse w/ reasoning.
Image
2/ Nice paper (@adamlerer & @alex_peys) suggests this may be due to recency bias from training: recent tokens are typically most informative for predicting the next one. Not good for context augmented generation.
arxiv.org/pdf/2310.01427…
Image
Read 11 tweets
Lance Martin Profile picture
Aug 25, 2023
Check out these new guides for 13 popular LLM use-cases. Part of a major community effort to improve the @LangChainAI docs + add CoLabs prototyping.

1/13: Open source LLMs
How to use many open source LLMs on your device
python.langchain.com/docs/guides/lo…
Image
2/13: Agents
How to quickly test various types of agents
python.langchain.com/docs/use_cases…
Image
3/13: RAG (retrieval augmented generation)
How to do RAG at multiple levels of abstraction
python.langchain.com/docs/use_cases…
Image
Read 14 tweets
Lance Martin Profile picture
Aug 23, 2023
GPT-3.5 and LLaMA2 fine-tuning guides 🪄

Considering LLM fine-tuning? Here's two new CoLab guides for fine-tuning GPT-3.5 & LLaMA2 on your data using LangSmith for dataset management and eval. We also share our lessons learned in a blog post here:

blog.langchain.dev/using-langsmit…
Image
... 1/ When to fine-tune? Fine-tuning is not advised for teaching an LLM new knowledge (see references from @OpenAI and others in our blog post). It's best for tasks (e.g., extraction) focused on "form, not facts":
anyscale.com/blog/fine-tuni…
... 2/ With this in mind, we fine-tuned LLaMA-7b-chat & GPT-3.5-turbo for knowledge graph triple extraction (see details in blog post and CoLab). Notebooks here:
LLaMA CoLab:
GPT-3.5-turbo CoLab:

colab.research.google.com/drive/1tpywvzw…
colab.research.google.com/drive/1YCyDHPS…
Read 9 tweets
Lance Martin Profile picture
Aug 12, 2023
Code understanding 🖥️🧠

LLMs excel at code analysis / completion (e.g., Co-Pilot, Code Interpreter, etc). Part 6 of our initiative to improve @LangChainAI docs covers code analysis, building on contributions of @cristobal_dev + others:
https://t.co/2DsxdjbYeypython.langchain.com/docs/use_cases…
Image
1/ Copilot and related tools (e.g., @codeiumdev) have dramatically accelerated dev productivity and shown that LLMs excel at code understanding / completion
2/ But, RAG for QA/chat on codebases is challenging b/c text splitters may break up elements (e.g., fxns, classes) and fail to preserve context about which element each code chunk comes from.
Read 6 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

Send Email!

:(