Thread by @jytinevez

A 2nd post about the new #TrackMate version:
Using TrackMate to segment 3D objects using a slice-by-slice approach.

1/n

Now that we have nice Deep-Learning based and Machine-Learning based segmentation algorithms that work especially well in 2D, we can use them to segment 3D objects with TrackMate.

2/n

The idea is to tell TrackMate "this is not a 3D image but a 2D+T image that you will track".
For instance you could use StarDist on the individual 2D slice to segment sections of the object, then merging the multiple 2D contours in a single 3D object.

3/n

For the merge across Z-slices we simply use TrackMate trackers, then swap T with Z. So we formulate merging as tracking across Z.

4/n

It works in practice. This is what @guijacquemet and @JwPylvanainen did on a drosophila embryo:

5/n

And on arabidopsis:

6/n

This looks like a TrackMate hack, but I would humbly argue that it's going to be an important and useful analysis pipeline.
Here is my logic, to be discussed:

7/n

For instance Deep-Learning algos are really effective and useful in Life-Sciences.
There are several frameworks for DL that can accept 3D images.
But in our humble and limited experience 3D models are less generalizable than 2D ones.

8/n

And it seems that 2D models are widely prevalent compared to 3D models. It's kind of expected. The annotation of images is much harder and longer in 3D compared to 2D. So it is common for a 3D model to 'see' a lower amount of objects during the training phase than in 2D.

9/n

The slice-by-slice approach allows for benefiting from the 2D model in 3D case. It fits well with many imaging modalities we use in Life-Science microscopy (confocal & friends).
There are already many framework that use this approach for 3D (for instance #cellpose)

10/n

Using TrackMate for this allows specifying the Z-merging step as a tracking problem, via a GUI. So it is accessible to anyone, without programming knowledge.

11/n

It turned to be so useful for our projects that we made a new tracker especially useful for this approach in the new TrackMate:
The overlap tracker. Simple, best explained with a figure:

12/n

(Works also very well for 2D+T or 3D+T tracking problems where the objects do not move too much between frames.)

Here is a step-by-step tutorial @JwPylvanainen wrote that shows how to use this approach for segmentation with StarDist:

imagej.net/plugins/trackm…

13/n