Thread by @cinemarob1

Deep Learning is a blessing! I used to work night shifts in the film biz 🎥

I spent countless hours finding the exact moment when the clapper 🎬 slapped shut, in 10K+ video clips 😓

It took me ~64 hours to automate clapper detection using DL 👨‍💻

Here's the deep dive 🧵:

I will walk you through each step of my project:
=====
1/ Collect data
2/ Set up LabelStudio
3/ Draw bounding boxes
4/ Set up DVC
5/ Spin up EC2
6/ Install IceVision
7/ Run notebooks
8/ Connect WandB
9/ Try more models
10/ Infinite training loop
11/ Deploy

1/ Collect Image Data:

There's a Chrome extension called YouTube Screenshot.

To start the project, I spent 2 hours collecting 50+ images of slates 🎬 from a variety of instructional videos.

rli.to/Yb61z

2/ Set up LabelStudio:

@LabelStudioHQ makes a lovely tool for image data annotation called LabelStudio. It runs locally on my M1 MacBook via @Docker

Here's a thread about setting up LS on any *nix based machine, including mac:

https://twitter.com/cinemarob1/status/1572915171428900864

3/ Draw Bounding Boxes:

Any object detection project requires training data. I crafted my own by manually drawing bounding boxes in LabelStudio.

Labeling the first 50 images only took about an hour.

4/ Set up DVC:

With new image data and bbox annotations in hand, it was time to move into the cloud.

That's where @DVCorg shines. I used DVC to sync my data to S3—plus version control my dataset.

More on that here:

https://twitter.com/cinemarob1/status/1570797450587213825

5/ Spin up an EC2 Instance:

I needed GPU power to train computer vision models.

Over several days, I played with @HashiCorp Terraform to make provisioning EC2 spot instances effortless.

Now I could rev up when needed, and shut down to save costs:

https://twitter.com/cinemarob1/status/1576966326937067521

6/ Install IceVision:

There's no faster way to start experimenting with object detection than to load @ai_fast_track's "IceVision" library. It has abstractions on top of:

- @fastdotai
- @PyTorch
- MMDetection

Installation in minutes.

rli.to/qrJOd

7/ Tweak & Run Training Notebooks:

#IceVision demo notebooks are easily tweaked to fit your needs.

They leverage nice tools like Albumentations to help your models generalize.

Beyond object detection, they have:
- Instance segmentation
- Semantic segmentation
- Keypoints

8/ Connect WandB:

Experiment tracking is key as you try various backbones and hyper-parameters (e.g. image size).

@weights_biases makes it easy to track model runs—and plugs right into IceVision.

This is another step that took minutes, not hours.

9/ Try Various Models:

For my purposes, I tried:
- Faster-RCNN
- YOLOv5
- EfficientDet
- VFNet (lite)

Thanks to WandB, it became clear that Faster-RCNN was giving me superior results.

Each model run took ~8 minutes on an ec2 g4dn.xlarge spot instance.

10/ Infinite training loop:

Time to expand beyond the initial 50 images.

I used my first model to auto-label new images.

Noting where it didn't do well, I fixed errant labels by hand.

The model was re-trained with the new labeled data rolled in.

Repeat.

11/ Deploy:

@huggingface makes it easy to demo computer vision projects on their "spaces".

Here's a working demo of my film slate detector. Upload your own images:
rli.to/kvCU9

12/ Deploy (pt 2):

I ran my detector over some infamous behind-the-scenes footage from Tarantino's "Inglorious Basterds". I hope this sparks joy:

rli.to/Mh2yU

TL;DR:

* Built a 🖼 dataset w/ @LabelStudioHQ
* @DVCorg to track/version data
* Trained an object detector w/ @awscloud GPU spots
* Used #icevision CV framework
* Tracked experiments w/ @weights_biases
* Built a semi-supervised training loop ♺
* Deployed on @huggingface spaces

That's a wrap! 🎞

If you enjoyed this thread:

I share posts about Deep Learning, Computer Vision, and Tech Ethics.

1. Give me a follow @cinemarob1 for more updates.
2. RT the tweet below to share this thread with your audience

https://twitter.com/cinemarob1/status/1587476860551692289?t=hIrXRCkNe7Wwk9sTVEYxjw&s=19