How can we learn NeRF from a SINGLE portrait image? Check out @gaochen315's recent work leverages new (meta-learning) and old (3D morphable model) tricks to make it work! This allows us to synthesize new views and manipulate FOV.
Project: portrait-nerf.github.io
Project: portrait-nerf.github.io
Work led by the amazing Chen Gao (@gaochen315) and in collaboration with friends from Google (Yichang Shih, Wei-Sheng Lai, and Chia-Kai Liang).
Paper: arxiv.org/abs/2012.05903
Paper: arxiv.org/abs/2012.05903
So, how does it work?
Training a NeRF from a single image from scratch won't work because it cannot recover the correct shape. The rendering results look fine at the original viewpoint but produce large errors at novel views.
Training a NeRF from a single image from scratch won't work because it cannot recover the correct shape. The rendering results look fine at the original viewpoint but produce large errors at novel views.
How about transferring learning? Pretraining a NeRF on a collection of a multi-view face dataset.
Well, this works okay, but still performs poorly on *unseen* subjects due to diverse appearance and shape variations.
Well, this works okay, but still performs poorly on *unseen* subjects due to diverse appearance and shape variations.
Our idea: Pretraining our neural implicit model so that it can QUICKLY ADAPT to an unseen subject using meta-learning.
(This is similar to another excellent project on learning initialization for implicit representations: matthewtancik.com/learnit)
(This is similar to another excellent project on learning initialization for implicit representations: matthewtancik.com/learnit)
In our application, however, the shapes of the subjects vary. We compensate for the shape variability using 3D morphable models and learn the implicit function in the *3D canonical coordinate*.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
