Very glad I can finally talk about our newly-minted #CVPR2022 paper. We extended mip-NeRF to handle unbounded "360" scenes, and it got us ~photorealistic renderings and beautiful depth maps. Explainer video: and paper: arxiv.org/abs/2111.12077
We "contract" Euclidean space into a bounded domain, which gets hard because we need to warp the mip-NeRF Gaussians that model 3D volumes of space. The trick for making this work is linearizing the contraction (thanks JAX!) and using the same math as an extended Kalman filter.
Big scenes need big models! We use a tiny MLP (queried repeatedly) to model coarse scales, and a huge MLP (queried sparingly) to model the finest scale, and distill the huge "NeRF MLP" into that tiny "proposal MLP". The trick: histograms from both MLPs *must* bound each other.
The inherent ambiguity of these big unbounded scenes means that "floaters" are everywhere. We fix this with a simple regularizer that minimizes distortion (à la k-means) which encourages histograms along rays to resemble delta spikes. This works really well, try it out!
Congrats to my excellent teammates @BenMildenhall, @dorverbin, @_pratul_, and @PeterHedman3. Give them a follow if you want to see some cool new NeRF work from us in the coming weeks.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
