Thread by @atroyn

there have been quite a few new datasets for autonomous vehicle (AV) perception released lately, including semantic seg for LIDAR point clouds, labeled maps, high fidelity pose etc. etc.

they are also the most popular (KITTI though it’s super old especially) to report results

here are some

level5.lyft.com/dataset/
apolloscape.auto
nuscenes.org

it’s fine that they exist but ultimately they won’t help to significantly improve perception and efforts in AV in particular, or machine perception / computer vision research in general

the focus on AV makes sense because of the nature of competition in this market; there is no business yet

- the competition between VC backed/ad company backed companies is entirely for engineering and research talent

these datasets are mainly an attempt to penetrate academia, and a category of researcher industry can’t reach because their goal is to become a professor (still very possible in this field, plenty of room at the top in robotics in general, computer vision in particular)

however as results are increasingly reported against these datasets, research focuses on a narrow subset of the set of problems to be solved.

new and more interesting datasets need to be developed in order to generate real advances

here are some proposals:
- multi vehicle/scene capture, where multiple moving cameras are capturing the same scene (and each other)

- high fidelity indoor dynamic scenes with calibrated lighting (Oxford multimotion is a start but not enough)

- defined human object / human-human interactions in dynamic scenes

- deformable nonrigid objects and surfaces with semantic segmentation and 3D data

these proposed datasets present a much broader and general class of perception problem. in particular it is my belief that most ML approaches will not replicate under the proposed conditions

this is the only way forward