Thread by @DrJimFan

What if we set GPT-4 free in Minecraft? ⛏️

I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves itself by writing, refining, committing, and retrieving *code* from a skill library.

GPT-4 unlocks… twitter.com/i/web/status/1…

Generally capable, autonomous agents are the next frontier of AI. They continuously explore, plan, and develop new skills in open-ended worlds, driven by survival & curiosity.

Minecraft is by far the best testbed with endless possibilities for agents:

https://twitter.com/DrJimFan/status/1595459499732926464?s=20

Voyager has 3 key components:

1) An iterative prompting mechanism that incorporates game feedback, execution errors, and self-verification to refine programs;
2) A skill library of code to store & retrieve complex behaviors;
3) An automatic curriculum to maximize exploration.

First, Voyager attempts to write a program to achieve a particular goal, using a popular Javascript Minecraft API (Mineflayer). The program is likely incorrect at the first try. The game environment feedback and javascript execution error (if any) help GPT-4 refine the program.

Second, Voyager incrementally builds a skill library by storing the successful programs in a vector DB. Each program can be retrieved by the embedding of its docstring. Complex skills are synthesized by composing simpler skills, which compounds Voyager’s capabilities over time.

Third, an automatic curriculum proposes suitable exploration tasks based on the agent’s current skill level & world state, e.g. learn to harvest sand & cactus before iron if it finds itself in a desert rather than a forest.

Think of it as an in-context form of *novelty search*.

Putting these all together, here’s the full data flow design that drives lifelong learning in a vast 3D voxel world without any human intervention.

Let’s look at some experiments!

We evaluate Voyager systematically against other LLM-based agent techniques, such as ReAct, Reflexion, and the popular AutoGPT in Minecraft.

Voyager discovers 63 unique items within 160 prompting iterations, 3.3x more than the next best approach.

The novelty-seeking automatic curriculum naturally compels Voyager to travel extensively. Without being explicitly instructed to do so, Voyager traverses 2.3x longer distances and visits more terrains than the baselines, which are “lazier” and often stuck in local areas.

How good is the “trained model”, i.e. skill library after lifelong learning?

We clear the agent’s inventory/armors, spawn a new world, and test with unseen tasks. Voyager solves them significantly faster. Our skill library even boosts AutoGPT, since code is easily transferrable.

Voyager is currently text-only, but can be augmented by visual perception in the future. We do a preliminary study where humans act like an image captioning model and provide feedback to Voyager.

It is able to construct complex 3D structures, such as a Nether Portal and a house.

Let agents emerge in Minecraft! All open-source: voyager.minedojo.org

This work is co-authored by my team at NVIDIA: @guanzhi_wang (our awesome intern), @yuqi_xie5, @YunfanJiang, @AjayMandlekar, @ChaoweiX, @yukez, @DrJimFan (myself, co-advisor), @AnimaAnandkumar (co-advisor).