Thread by @dchaplot

Presenting SEAL: Self-supervised Embodied Active Learning! #NeurIPS2021

SEAL is a self-supervised framework to close the action-perception loop. It improves perception & action models by just moving in the physical world w/o any human supervision.

devendrachaplot.github.io/projects/seal

1/N

SEAL consists of two phases, Action, where we learn an active exploration policy, and Perception, where we train the Perception Model on data gathered using the exploration policy and labels obtained using spatio-temporal label propagation.

2/N

Learning Action: We define an intrinsic motivation reward called Gainful Curiosity to train the active exploration policy to learn the behavior of maximizing exploration of objects with high confidence.

3/N

Learning Perception: We present a method called 3DLabelProp to obtain self-supervised labels from the 3D Semantic map. The set of observations and self-supervised labels are used to improve the object detection and instance segm performance from 34.8/32.5 to 41.2/37.3 AP50.

4/N

Improved Perception model in-turn leads to better Object-Goal Navigation: 54% -> 62% success rate by just moving around in training environments, without having access to any additional human annotations or map information.

5/N

PDF: arxiv.org/abs/2112.01001

Web: devendrachaplot.github.io/projects/seal

Presentation:

Slides: devendrachaplot.github.io/talks/neurips2…

w. @mihdalal, @_saurabhg, Jitendra Malik, @rsalakhu

6/N