For code language models, every token is a new chance to break a program. What if LLMs wrote code like people, decomposing programs into solvable parts? They can solve competition-level coding problems by writing natural language programs in Parsel🐍, beating prior SoTA by >75%! Plot showing competition-level pass rate of Parsel using Cod
Parsel 🐍: A Unified Natural Language Framework for Algorithmic Reasoning
Work done w/ @qhwang3 @GabrielPoesia @noahdgoodman @nickhaber
Website [🕸️]: zelikman.me/parselpaper/
Paper [📜]: zelikman.me/parselpaper/pa…
Code [💻]: github.com/ezelikman/pars… Flow chart visualizing Parsel - first, the language model de
In the paper where OpenAI introduced Codex, they showed that code language models fail to generate programs that chain together many simple tasks, while humans can - Parsel solves this by separating out decomposition and implementation Line chart with performance vs number of chained components.
Plus, excitingly, when LLMs write Parsel to generate step-by-step robotic plans from high-level tasks, the plans are consistently more accurate than a zero-shot planner baseline - more than 2/3 of the time! We've also shown Parsel can prove theorems, but highlight key challenges Parsel to VirtualHome example and Parsel to Lean (theorem pr
Our initial goal was to let people write code in natural language, but we found LLMs are also good Parsel coders! We just asked GPT-3 to "think step by step to come up with a clever algorithm" (see arxiv.org/abs/2205.11916), then asked to translate into Parsel given a few examples
To understand the quality of the generated Parsel programs, @GabrielPoesia (an experienced competitive coder) solved a bunch of competition-level APPS problems with Parsel. He solved 5/10 problems in 6 hours, with 3 where GPT-3 failed, suggesting there's still a long way to go! Scaling curve showing how APPS performance improves with the
This new version of the paper goes into more detail on how Parsel addresses the limitations of code language models and better quantifies the ability of LLMs to generate Parsel programs. We think there's still a ton more to be done - we look forward to hearing your thoughts!

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Eric Zelikman

Eric Zelikman Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @ericzelikman

Dec 8, 2022
ChatGPT can write stories and then tell DALLE-2 prompts to illustrate them. I asked it to write a children's story about "a robot that wanted to be a human." Here's the story it came up with: (0/11)
Once upon a time, in a land far, far away, there was a robot named Robby who lived in a world full of machines. Robby was different from the other robots, though. He didn't want to spend his days following orders and carrying out tasks like the other robots did.
(1/11) The image shows a robot standing among a group of other robo
Instead, Robby dreamed of being a human. He longed to feel the sun on his face, to breathe fresh air, and to experience all the wonderful things that humans did. But most of all, Robby wanted to be able to make his own choices and live his own life. (2/11) The image shows a robot walking towards the viewer, with a d
Read 17 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(