Martin P. Profile picture
Nov 3, 2021 21 tweets 14 min read Read on X
1/21
We’re proud to present a new Deep Learning approach for Content-Aware Frame Interpolation (CAFI). Ever wanted to increase the frame rate of your images using Deep Learning? CAFI might be the right tool for you.
It’s CAFI time!
github.com/mpriessner/CAFI
biorxiv.org/content/10.110…
2/21
Delighted to share this work, a collaboration with @dgaboriau, @SheridanArlo, Tchern Lenn & Jonathan Chubb (not on Twitter), @manorlaboratory, @Vilar_lab and @LaineBioImaging
biorxiv.org/content/10.110…
3/21
CAFI provides Deep Learning-based temporal super-resolution for fast bioimaging. It increases the frame rate of any microscope modality by interpolating an image in between two consecutive images via “intelligent” interpolation, providing 2x increase in temporal resolution. Image
4/21
Our CAFI implementations are based on two neural networks (ZoomingSlowMo (ZS) & DAIN) developed for the video frame interpolation task in computer vision research. Credit goes to @bao_wenbo, @mukosame for their amazing work!
github.com/Mukosame/Zoomi…
github.com/baowenbo/DAIN Image
5/21
Surprisingly, models trained on naturally moving objects (such as cars) already outperforms classical interpolations, like bicubic (BIC), bilinear (BIL) or frame duplication (NONE). Fine-tuning (FT) on representative microscopy images further improves their performance. Image
6/21
CAFI was successfully demonstrated on 6 datasets from 3 different microscopy modalities: point-scanning confocal, spinning-disk confocal and confocal brightfield.
Example 1: GFP labelled dictyostelium - Spinning disk confocal microscope. Data courtesy: T. Lenn and J. Chubb.
7/21
CAFI also performed well on fluorescently labelled lysosomes of SH-SY5Y cells.
Example 2: Lysosomes in SHSY5Y cells labelled with the lysosomal copper(I) stain FLCS1 - Point-scanning confocal microscope.
8/21
Transmitted light time-course of live SH-SY5Y obtained using a confocal brightfield microscope.
Example 3: SH-SY5Y cells - Confocal brightfield microscope.
9/21
Like any other deep learning approaches, CAFI sometimes leads to artefacts. Validation for a particular application is essential. Examples of artifacts highlighted with white arrows. GT: ground truth, BIL: bilinear interpolation. Image
10/21
But wait for it - CAFI can do even better!
By iteratively applying it on the same image sequence it is possible to improve the frequency even further (2x CAFI + 2x CAFI = 4x #iCAFI)! ImageImage
11/21
See the comparison of 2x CAFI and 4x iCAFI on the Lysosomal dynamics dataset.
CAFI & iCAFI Comparison 1: Lysosomes - Confocal microscope.
12/21
See the comparison of 2x CAFI and 4x iCAFI on the SH-SY5Y cell movement dataset.
CAFI & iCAFI Comparison 2: SH-SY5Y- Confocal brightfield microscope.
13/21
We quantitatively assessed CAFI performance on common tracking metrics from the ISBI particles tracking challenge, using #TrackMate, on simulated particles and real lysosomes. Interestingly, Brownian motion cannot be predicted by CAFI, as expected from a random process.
14/21
You may ask yourself: “What’s the difference between interpolating in temporal dimension and in axial dimension?” The answer is basically none.
We also demonstrated that CAFI can interpolate volumes as well as time-courses, improving the axial plane density 2-fold #zCAFI. Image
15/21
zCAFI was successfully used for axial interpolation of electron microscopy images of rat hippocampal slices (data courtesy of @manorlaboratory). We show here again that CAFI performs better than classical interpolations such as bilinear or bicubic.
16/21
How about interpolation in t- and z-dimensions for your 4D image dataset?
CAFI can also do it: sequential interpolation in both t- and z-dimension of 3D+t datasets #CAFI (green) + #zCAFI (magenta) = #tzCAFI. Image
17/21
Sequential application of CAFI on first temporal then axial dimension (tzCAFI) increases image density of a 4D (3D+t) spinning disk confocal dataset of fibronectin-labelled A2780 cells. (Data obtained from @RiinaKaukonen and @johannaivaska, nature.com/articles/nprot…)
18/21
Want to try yourself on your own images?
We made CAFI available as #ZeroCostDL4Mic notebooks for both re-training and for inference.
ZoomingSlowMo4Mic: myminiurl.net/kUCxU
DAIN4Mic: myminiurl.net/2DPIo
Data available on Zenodo @ZENODO_ORG: myminiurl.net/HpuRo
19/21
Last but not least, we also made some tutorial videos for the user to kick-start your CAFI experience.
Tutorial Video DAIN4Mic:
Tutorial Video ZoomingSlowMo4Mic: Image
20/21
Looking forward to your feedback! Let us know what you think!
It’s CAFI time! #CAFI Image
21/21
Thank you to all the funding bodies who supported this work! @EPSRC, @WaittFdn, @NIHClinicalCntr, @nxwm_network, @ChanZuckerberg, @The_MRC, @wellcometrust, and @BBSRC

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Martin P.

Martin P. Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(