Thread by @SharonYixuanLi

How can we make neural networks learn both the knowns and unknowns? Check out our #ICLR2022 paper “VOS: Learning What You Don’t Know by Virtual Outlier Synthesis”, a general learning framework that suits both object detection and classification tasks. 1/n

arxiv.org/abs/2202.01197

(2/) Joint work with @xuefeng_du @MuCai7. Deep networks often struggle to reliably handle the unknowns. In self-driving, an object detection model trained to recognize known objects (e.g., cars, stop signs) can produce a high-confidence prediction for an unseen object of a moose.

(3/) The problem arises due to the lack of knowledge of unknowns during training time. Neural networks are typically optimized only on the in-distribution data. The resulting decision boundary, despite being useful on ID tasks, can be ill-fated for OOD detection. See Figure 1(b).

(4/) Ideally, a model should learn a compact decision boundary between ID and OOD data, like Fig 1(c). However, this is non-trivial due to the lack of supervision of unknowns. This motivates our paper: Can we synthesize virtual outliers for effective model regularization?

(5/) In this paper, we propose a novel unknown-aware learning framework dubbed VOS (Virtual Outlier Synthesis), which optimizes the dual objectives of both ID task and OOD detection performance.

(6/) VOS consists of three components tackling challenges of outlier synthesis and model regularization with synthesized outliers. Key to our method, we show that sampling in the feature space is more tractable than synthesizing images in the high-dimensional pixel space.

(7/) VOS offers several compelling advantages: (1) VOS is a general learning framework that is suitable for both object detection and classification tasks. (2) VOS enables adaptive outlier synthesis, which can be flexibly used without manual data collection or cleaning.

(8/) We evaluate our method on common OOD detection benchmarks, along with a more challenging yet underexplored task in the context of object detection. As part of our study, we also curated OOD test datasets that allow future research to evaluate object-level OOD detection.

(9/) More broadly, our work builds on the insights from the energy-based OOD learning framework and improves the regularization loss. We were also inspired by the early work on using GAN-based synthesis by @kimin_le2

(10/) Happy to get feedback if you have more detailed comments. The code and data is publicly available at: github.com/deeplearning-w…