1/21
We’re proud to present a new Deep Learning approach for Content-Aware Frame Interpolation (CAFI). Ever wanted to increase the frame rate of your images using Deep Learning? CAFI might be the right tool for you.
It’s CAFI time!
github.com/mpriessner/CAFI
biorxiv.org/content/10.110…
We’re proud to present a new Deep Learning approach for Content-Aware Frame Interpolation (CAFI). Ever wanted to increase the frame rate of your images using Deep Learning? CAFI might be the right tool for you.
It’s CAFI time!
github.com/mpriessner/CAFI
biorxiv.org/content/10.110…
2/21
Delighted to share this work, a collaboration with @dgaboriau, @SheridanArlo, Tchern Lenn & Jonathan Chubb (not on Twitter), @manorlaboratory, @Vilar_lab and @LaineBioImaging
biorxiv.org/content/10.110…
Delighted to share this work, a collaboration with @dgaboriau, @SheridanArlo, Tchern Lenn & Jonathan Chubb (not on Twitter), @manorlaboratory, @Vilar_lab and @LaineBioImaging
biorxiv.org/content/10.110…
3/21
CAFI provides Deep Learning-based temporal super-resolution for fast bioimaging. It increases the frame rate of any microscope modality by interpolating an image in between two consecutive images via “intelligent” interpolation, providing 2x increase in temporal resolution.
CAFI provides Deep Learning-based temporal super-resolution for fast bioimaging. It increases the frame rate of any microscope modality by interpolating an image in between two consecutive images via “intelligent” interpolation, providing 2x increase in temporal resolution.
4/21
Our CAFI implementations are based on two neural networks (ZoomingSlowMo (ZS) & DAIN) developed for the video frame interpolation task in computer vision research. Credit goes to @bao_wenbo, @mukosame for their amazing work!
github.com/Mukosame/Zoomi…
github.com/baowenbo/DAIN
Our CAFI implementations are based on two neural networks (ZoomingSlowMo (ZS) & DAIN) developed for the video frame interpolation task in computer vision research. Credit goes to @bao_wenbo, @mukosame for their amazing work!
github.com/Mukosame/Zoomi…
github.com/baowenbo/DAIN
5/21
Surprisingly, models trained on naturally moving objects (such as cars) already outperforms classical interpolations, like bicubic (BIC), bilinear (BIL) or frame duplication (NONE). Fine-tuning (FT) on representative microscopy images further improves their performance.
Surprisingly, models trained on naturally moving objects (such as cars) already outperforms classical interpolations, like bicubic (BIC), bilinear (BIL) or frame duplication (NONE). Fine-tuning (FT) on representative microscopy images further improves their performance.
6/21
CAFI was successfully demonstrated on 6 datasets from 3 different microscopy modalities: point-scanning confocal, spinning-disk confocal and confocal brightfield.
Example 1: GFP labelled dictyostelium - Spinning disk confocal microscope. Data courtesy: T. Lenn and J. Chubb.
CAFI was successfully demonstrated on 6 datasets from 3 different microscopy modalities: point-scanning confocal, spinning-disk confocal and confocal brightfield.
Example 1: GFP labelled dictyostelium - Spinning disk confocal microscope. Data courtesy: T. Lenn and J. Chubb.
7/21
CAFI also performed well on fluorescently labelled lysosomes of SH-SY5Y cells.
Example 2: Lysosomes in SHSY5Y cells labelled with the lysosomal copper(I) stain FLCS1 - Point-scanning confocal microscope.
CAFI also performed well on fluorescently labelled lysosomes of SH-SY5Y cells.
Example 2: Lysosomes in SHSY5Y cells labelled with the lysosomal copper(I) stain FLCS1 - Point-scanning confocal microscope.
8/21
Transmitted light time-course of live SH-SY5Y obtained using a confocal brightfield microscope.
Example 3: SH-SY5Y cells - Confocal brightfield microscope.
Transmitted light time-course of live SH-SY5Y obtained using a confocal brightfield microscope.
Example 3: SH-SY5Y cells - Confocal brightfield microscope.
9/21
Like any other deep learning approaches, CAFI sometimes leads to artefacts. Validation for a particular application is essential. Examples of artifacts highlighted with white arrows. GT: ground truth, BIL: bilinear interpolation.
Like any other deep learning approaches, CAFI sometimes leads to artefacts. Validation for a particular application is essential. Examples of artifacts highlighted with white arrows. GT: ground truth, BIL: bilinear interpolation.
10/21
But wait for it - CAFI can do even better!
By iteratively applying it on the same image sequence it is possible to improve the frequency even further (2x CAFI + 2x CAFI = 4x #iCAFI)!
But wait for it - CAFI can do even better!
By iteratively applying it on the same image sequence it is possible to improve the frequency even further (2x CAFI + 2x CAFI = 4x #iCAFI)!
11/21
See the comparison of 2x CAFI and 4x iCAFI on the Lysosomal dynamics dataset.
CAFI & iCAFI Comparison 1: Lysosomes - Confocal microscope.
See the comparison of 2x CAFI and 4x iCAFI on the Lysosomal dynamics dataset.
CAFI & iCAFI Comparison 1: Lysosomes - Confocal microscope.
12/21
See the comparison of 2x CAFI and 4x iCAFI on the SH-SY5Y cell movement dataset.
CAFI & iCAFI Comparison 2: SH-SY5Y- Confocal brightfield microscope.
See the comparison of 2x CAFI and 4x iCAFI on the SH-SY5Y cell movement dataset.
CAFI & iCAFI Comparison 2: SH-SY5Y- Confocal brightfield microscope.
13/21
We quantitatively assessed CAFI performance on common tracking metrics from the ISBI particles tracking challenge, using #TrackMate, on simulated particles and real lysosomes. Interestingly, Brownian motion cannot be predicted by CAFI, as expected from a random process.
We quantitatively assessed CAFI performance on common tracking metrics from the ISBI particles tracking challenge, using #TrackMate, on simulated particles and real lysosomes. Interestingly, Brownian motion cannot be predicted by CAFI, as expected from a random process.
14/21
You may ask yourself: “What’s the difference between interpolating in temporal dimension and in axial dimension?” The answer is basically none.
We also demonstrated that CAFI can interpolate volumes as well as time-courses, improving the axial plane density 2-fold #zCAFI.
You may ask yourself: “What’s the difference between interpolating in temporal dimension and in axial dimension?” The answer is basically none.
We also demonstrated that CAFI can interpolate volumes as well as time-courses, improving the axial plane density 2-fold #zCAFI.
15/21
zCAFI was successfully used for axial interpolation of electron microscopy images of rat hippocampal slices (data courtesy of @manorlaboratory). We show here again that CAFI performs better than classical interpolations such as bilinear or bicubic.
zCAFI was successfully used for axial interpolation of electron microscopy images of rat hippocampal slices (data courtesy of @manorlaboratory). We show here again that CAFI performs better than classical interpolations such as bilinear or bicubic.
16/21
How about interpolation in t- and z-dimensions for your 4D image dataset?
CAFI can also do it: sequential interpolation in both t- and z-dimension of 3D+t datasets #CAFI (green) + #zCAFI (magenta) = #tzCAFI.
How about interpolation in t- and z-dimensions for your 4D image dataset?
CAFI can also do it: sequential interpolation in both t- and z-dimension of 3D+t datasets #CAFI (green) + #zCAFI (magenta) = #tzCAFI.
17/21
Sequential application of CAFI on first temporal then axial dimension (tzCAFI) increases image density of a 4D (3D+t) spinning disk confocal dataset of fibronectin-labelled A2780 cells. (Data obtained from @RiinaKaukonen and @johannaivaska, nature.com/articles/nprot…)
Sequential application of CAFI on first temporal then axial dimension (tzCAFI) increases image density of a 4D (3D+t) spinning disk confocal dataset of fibronectin-labelled A2780 cells. (Data obtained from @RiinaKaukonen and @johannaivaska, nature.com/articles/nprot…)
18/21
Want to try yourself on your own images?
We made CAFI available as #ZeroCostDL4Mic notebooks for both re-training and for inference.
ZoomingSlowMo4Mic: myminiurl.net/kUCxU
DAIN4Mic: myminiurl.net/2DPIo
Data available on Zenodo @ZENODO_ORG: myminiurl.net/HpuRo
Want to try yourself on your own images?
We made CAFI available as #ZeroCostDL4Mic notebooks for both re-training and for inference.
ZoomingSlowMo4Mic: myminiurl.net/kUCxU
DAIN4Mic: myminiurl.net/2DPIo
Data available on Zenodo @ZENODO_ORG: myminiurl.net/HpuRo
19/21
Last but not least, we also made some tutorial videos for the user to kick-start your CAFI experience.
Tutorial Video DAIN4Mic:
Tutorial Video ZoomingSlowMo4Mic:
Last but not least, we also made some tutorial videos for the user to kick-start your CAFI experience.
Tutorial Video DAIN4Mic:
Tutorial Video ZoomingSlowMo4Mic:
21/21
Thank you to all the funding bodies who supported this work! @EPSRC, @WaittFdn, @NIHClinicalCntr, @nxwm_network, @ChanZuckerberg, @The_MRC, @wellcometrust, and @BBSRC
Thank you to all the funding bodies who supported this work! @EPSRC, @WaittFdn, @NIHClinicalCntr, @nxwm_network, @ChanZuckerberg, @The_MRC, @wellcometrust, and @BBSRC
• • •
Missing some Tweet in this thread? You can try to
force a refresh
