Thread by @du_yilun

Check out neural descriptor fields!
yilundu.github.io/ndf/

We present a self-supervised method to obtain SE(3) equivariant descriptors of 3D shapes. These descriptors enable us generalize pick and place demonstrations to arbitrary novel SE(3) poses and objects instances

(1/6)

This is joint work with amazing collaborators @anthonysimeono_ (Project Colead), @taiyasaki, Josh Tenenbaum, Alberto Rodriguez, @pulkitology (Equal Advising) @vincesitzmann (Equal Advising)

(2/6)

We learn SE(3) dense descriptors of shapes, by utilizing the intermediate network activations of an occupancy network. We further obtain SO(3) equivariance of dense features through vector neurons and translation equivariance through pointcloud centering.

(3/6)

These neural descriptor fields (NDFs) enable us to infer semantic correspondence between separate objects. Below, we illustrate energy heatmaps of descriptor distances between separate shapes.

(4/6)

By defining a set of query points, we may further utilize NDF values at query points to define neural pose descriptor fields, which assign separate sets of descriptors to poses of rigid bodies. Energy optimization over descriptors sets transfer poses between shapes.

(5/6)

In this work, we use these neural pose descriptors to enable generalizable manipulation. However, these ideas are generally applicable for all neural fields, and we are excited for future applications of neural descriptors to other domains!

(6/6)