Introducing RepNet, a model that counts repetitions in videos of *any* action
w @yusufaytar, @JonathanTompson, @psermanet and Andrew Zisserman
Paper: openaccess.thecvf.com/content_CVPR_2…
Project: sites.google.com/repnet
Video:
#CVPR2020 #computervision #deeplearning
w @yusufaytar, @JonathanTompson, @psermanet and Andrew Zisserman
Paper: openaccess.thecvf.com/content_CVPR_2…
Project: sites.google.com/repnet
Video:
#CVPR2020 #computervision #deeplearning
Here's an overview of RepNet's architecture. 
An integral part of RepNet is the temporal self-similarity matrix (TSM) which not only makes it easy to count repetitions but also drives generalization to actions and domains not seen during training.
What’s interesting is that we can train RepNet with synthetic data. We generate videos with repetitions by randomly sampling a clip of length N from a video and playing it over and over. This allows us to scale training to a large dataset without human annotation effort.
On the left, we show an example of a synthetic repeating video. These videos are too simple for the model. To make the task more difficult, we add camera motion augmentation to the video (shown on the right).
We train RepNet to predict the period from the video.
We train RepNet to predict the period from the video.
After being trained on synthetic videos, RepNet can count repetitions in real videos across many domains and actions.
RepNet can also detect changes in speed of the action.
Here we show an example of RepNet measuring the number of oscillations in a biological process as seen under a microscope. Video from nature.com/articles/s4200…
We will be presenting this as a part of our paper “Counting Out Time: Class Agnostic Video Repetition Counting in the Wild” at #CVPR2020. Please come by our virtual poster (cvpr20.com/event/counting…) tomorrow 9AM -11 AM PDT or 9PM -11PM PDT for live discussion.
