Thread by @sarameghanbeery

Check out our paper today at #CVPR2020 🎉🎉🎉
cvpr20.com/event/context-…

Context R-CNN: Long Term Temporal Context for Per Camera Object Detection

Come ask questions at the live Q&As!

First session: June 18, 3-5 PM PDT
Second session: June 19, 3-5 AM PDT

Our @CVPR spotlight vid:

In static cameras (like #cameratraps), relevant context for identifying objects can be spread out across long time horizons. For example, these two images come from the same camera and are uncannily similar, but they were taken a month apart! Turns out animals are pretty habitual

We propose a simple and flexible method for aggregating context from up to a month of data, using attention! We first build an (unsupervised) “memory bank” for each location. We add context for each object by finding features in the memory bank that help us identify that object.

The best news is, it really helps! When testing this method on the #SnapshotSerengeti Dataset (lila.science/datasets/snaps…), we saw a 47.6% relative improvement in mAP from the baseline Faster R-CNN model 🎉🎉🎉

Check out the results in our paper: openaccess.thecvf.com/content_CVPR_2…

The biggest improvements are on challenging cases, ones that even a human might need some context to identify correctly. Static cameras collect lots of difficult-to-ID images because they can't rely on a human photographer to make sure the objects of interest are easy to see & ID

Huge thanks to my co-authors @jonathanhuang11, @GuanhangW, @vivekrathod, and Ronny Votel! And to the @WildInsights AI team, the Visual Dynamics team at @GoogleAI, and Pietro's lab at @Caltech for letting me bounce ideas off of you all 🥰

We also find that increasing the time horizon that we use when providing context increases performance! When we use longer time horizons we are better able to take advantage of how habitual animals are. That warthog that uses the same game trail every day? Got em. 🐗

If you want to see how it works, we've set up a demo in @GoogleColab with our Snapshot Serengeti trained models: colab.research.google.com/github/tensorf…

We provided a few test images, but feel free to try it on other sets of camera trap data from Snapshot Serengeti or elsewhere in East Africa!

The model was trained on Season 1-6, with the train split locations recommended by lila.science/datasets/snaps…. Any images without ground truth bounding box labels got given 'weakly supervised' bounding box labels using the @Microsoft_Green #MegaDetector prior to training.

The code is open-sourced in the @TensorFlow Object Detection API, release notes here: github.com/tensorflow/mod…