Thread by @poolio

Happy to announce DreamFusion, our new method for Text-to-3D!

dreamfusion3d.github.io

We optimize a NeRF from scratch using a pretrained text-to-image diffusion model. No 3D data needed!

Joint work w/ the incredible team of @BenMildenhall @ajayj_ @jon_barron

#dreamfusion

DreamFusion generates 3D models from diverse text prompts. Check out our gallery of hundreds of 3D models:
dreamfusion3d.github.io/gallery.html

We build on Dream Fields, replacing CLIP with a new loss computed from the Imagen text-to-image diffusion model (imagen.research.google) :

https://twitter.com/ajayj_/status/1499167579495010304

The 3D model we generate is an improved NeRF that produces a 3D volume with density, color, and surface normals:

DreamFusion represents appearance as a material color, which can be combined with normals for rendering under different lighting conditions:

We can even take several 3D models generated by DreamFusion and compose them into new scenes:

Check out the paper for more details, including a distillation-based loss function that could enable many new applications of pretrained diffusion models: arxiv.org/abs/2209.14988

This was an incredibly fun team effort w/ NeRF wizards @BenMildenhall & @jon_barron, and NeRF + diffusion expert @ajayj_ (graduating this year!).

We're excited to incorporate our methods with open source models and enable a new future for 3D generation! 🚀
#dreamfusion