Thread by @XDWang101

🎉CutLER is accepted to #CVPR2023!

tl;dr: CutLER is an unsupervised object detector surpassing prev SOTA by 2.7x on 11 datasets across various domains, e.g. natural images, painting, sketch.

Codes/demos are released!
Developed during my internship at FAIR-Meta AI @MetaAI

1/n

https://twitter.com/_akhaliq/status/1618794235704090624

Code: github.com/facebookresear…

HuggingFace demos (MaskCut): huggingface.co/spaces/faceboo…
HuggingFace demos (CutLER): huggingface.co/spaces/faceboo…

Colab demos: colab.research.google.com/drive/1NgEyFHv…

2/n

Demos results on 11 benchmarks spanning a variety of domains, including video frames, paintings, clip arts, complex scenes, etc.

3/n

Cut-and-Learn has two main stages: 1) generating pseudo-masks with our MaskCut without human annotations and 2) learning unsupervised detectors from pseudo-masks of unlabeled data.

4/n

Stage 1: The proposed MaskCut method can be used to provide segmentation masks for multiple instances of each image, leveraging a self-supervised pretrained DINO model.

5/n

MaskCut can produce high-quality segmentation masks for multiple objects!

Colab demo @GoogleColab: colab.research.google.com/drive/1X05lKL_…
HuggingFace demo @huggingface : huggingface.co/spaces/faceboo…
Replicate demo: replicate.com/cjwbw/cutler

6/n

Stage 2: After producing pseudo-masks with MaskCut, we then learn a detector using our loss-dropping strategy that is robust to objects missed by MaskCut. We further improve the model using multiple rounds of self-training.

7/n

Additional CutLER demos showcasing its ability to detect and segment small objects!

Try it out:
Colab demos @GoogleColab: colab.research.google.com/drive/1NgEyFHv…
HuggingFace demos @huggingface: huggingface.co/spaces/faceboo…

8/n

CutLER can also serve as a strong pretrained model for fully-supervised and semi-supervised object detection, surpassing MoCo-v2 by 7.3% APbox and 6.6% APmask on COCO when training with 5% labels.

9/n

Many thanks to my collaborators 🤗 Ishan @imisra_, Rohit @_rohitgirdhar_ and Stella, and the incredible internship experience at FAIR-Meta AI @MetaAI! Can't wait to be back next summer! 🎉🎉🎉

n/n