Difference between revisions of "TrackMate Accuracy"

Line 6: Line 6:
 
== The ISBI 2012 single particle challenge. ==
 
== The ISBI 2012 single particle challenge. ==
  
In 2011-2012, an ISBI Grand Challenge was organized for the [http://bioimageanalysis.org/track/ Single-Particle Tracking algorithms]. Though TrackMate does not offer a completely new algorithm, product of an original Research work, we took the chance and participated in the challenge. The results and the methodology to compute the accuracy of a tracking algorithms were published<ref name="ISBIpaper"> [http://www.nature.com/nmeth/journal/v11/n3/full/nmeth.2808.html Chenouard ''et al.'', Objective comparison of particle tracking methods. '''Nature Methods, 2014 ''']</ref> thereafter.
+
In 2011-2012, an ISBI Grand Challenge was organized for the [http://bioimageanalysis.org/track/ Single-Particle Tracking algorithms]. Though TrackMate does not offer a completely new algorithm, product of an original Research work, we took the chance and participated in the challenge. The results and the methodology to compute the accuracy of a tracking algorithms were published<ref name="ISBIpaper"> [http://www.nature.com/nmeth/journal/v11/n3/full/nmeth.2808.html Chenouard ''et al.'', "Objective comparison of particle tracking methods", '''Nature Methods, 2014 ''']</ref> thereafter.
  
Unsurprisingly, we did not score amongst the best. At the time, TrackMate was in version 1.1, and ship a stripped down version of the better performing Jaqaman ''et al.'' LAP framework<ref name="Jaqaman">[http://www.nature.com/nmeth/journal/v5/n8/full/nmeth.1237.html Jaqaman et al., "Robust single-particle tracking in live-cell time-lapse sequences", '''Nature Methods, 2008''']</ref>. See the [[TrackMate algorithms#LAP_trackers|LAP trackers section]] for algorithm details. Plus, TrackMate was was young at the time, and some bugs did not help.  
+
Unsurprisingly, we did not score amongst the best. At the time, TrackMate was in version 1.1, and ship a stripped down version of the better performing Jaqaman ''et al.'' LAP framework<ref name="Jaqaman">[http://www.nature.com/nmeth/journal/v5/n8/full/nmeth.1237.html Jaqaman ''et al.'', "Robust single-particle tracking in live-cell time-lapse sequences", '''Nature Methods, 2008''']</ref>. See the [[TrackMate algorithms#LAP_trackers|LAP trackers section]] for algorithm details. Plus, TrackMate was was young at the time, and some bugs did not help.  
  
  
Line 19: Line 19:
 
* The linear motion tracker based on Kalman filter.
 
* The linear motion tracker based on Kalman filter.
 
* The plain Nearest neighbor tracker for reference.
 
* The plain Nearest neighbor tracker for reference.
 +
 +
 +
=== Scenarios. ===
  
 
It's best to directly read the paper<ref name="ISBIpaper"/> to know what is behind these measures, but here is a brief survey of how they are done. The ISBI dataset covers four scenarios:  
 
It's best to directly read the paper<ref name="ISBIpaper"/> to know what is behind these measures, but here is a brief survey of how they are done. The ISBI dataset covers four scenarios:  
* MICROTUBULE: Particles have a slightly elongated shape to mimic MT tip staining. They undergo a roughly constant velocity motion.
 
* RECEPTOR: Particles are spherical and undergo a [http://en.wikipedia.org/wiki/Tethered_particle_motion tethered motion].
 
  
 +
{| class="wikitable"
 +
|-
 +
! Scenario name
 +
! Particle shape
 +
! Motion type
 +
|-
 +
| MICROTUBULE
 +
| Slightly elongated shape to mimic MT tip staining.
 +
| Roughly constant velocity motion.
 +
|-
 +
| RECEPTOR
 +
| rowspan=3 | Spherical.
 +
| [http://en.wikipedia.org/wiki/Tethered_particle_motion Tethered motion]: switch between Brownian and directed motion with random orientation for the later.
 +
|-
 +
| VESICLE
 +
| [http://en.wikipedia.org/wiki/Brownian_motion Brownian motion].
 +
|-
 +
| VIRUS
 +
| Switch between Brownian and directed motion with fixed orientation for the later.
 +
|}
 +
 +
 +
For each scenario, images covers several particle density:
 +
* low: 60-100 / frame
 +
* mid: 400-500 / frame
 +
* high: 700-1000 / frame
 +
to check how a tracking algorithm behaves when particles get very dense.
 +
 +
Also, particles SNR spans several values: 1, 2, 4, 7 (plus 3 for the RECEPTOR scenario). As said on the [http://bioimageanalysis.org/track/ challenge page]: "SNR=4 is a critical level at which methods may start to break down".
 +
 +
 +
=== Accuracy measurements. ===
 +
 +
For each scenario and condition, the method returns numerous values that characterizes the accuracy of a tracking algorithm. They are detailed on [http://bioimageanalysis.org/track/PerformanceMeasures.pdf this technical paper]. We plot below only three of them:
 +
 +
* The <b>Jaccard similarity between tracks</b>, that quantifies how well the tracks returned by the algorithm match the ground truth. This value assesses the accuracy of the [[TrackMate_algorithms#Spot_trackers|spot tracker]] you pick in TrackMate. It ranges from 0 (terrible) to 1 (found tracks = ground truth).
 +
 +
* The <b>Jaccard similarity between detections</b>, that quantifies how well the particle detected by the detection algorithm match the ground truth. It depends strongly on the [[TrackMate_algorithms#Spot_detectors|spot detector]] you pick in TrackMate, and ranges from 0 to 1 like the above quantity.
 +
 +
* The <b>RMSE of detection positions</b> that quantifies how precise is the location of the detected particles.
 +
 +
 +
=== Parameter used. ===
 +
 +
Unless otherwise specified below, I always used the LoG detector as a spot detector, with an estimated particle diameter of 4, and used sub-pixel accuracy. For SNR below 4, this detector was completely confused and the detection results are dominated by noise. I did not make anything special to improve its sensitivity below this limit.
 +
 +
When the histogram of detection quality returned by the detector as not bimodal, I pick a quality threshold that yielded approximately the expected number of particles in the sequence.
 +
 +
The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.
 +
 +
 +
{| class="wikitable"
 +
|-
 +
! Spot tracker
 +
! Parameter
 +
! Value
 +
|-
 +
| rowspan =3 | [[TrackMate_algorithms#Linear_motion_tracker.|Linear motion tracker]]
 +
| Initial search radius
 +
| 10
 +
|-
 +
| Search radius
 +
| 7
 +
|-
 +
| Max frame gap
 +
| 3
 +
|-
 +
| rowspan = 3 | [[TrackMate_algorithms#LAP_trackers LAP|Brownian motion]]
 +
| Max linking distance
 +
| 7
 +
|-
 +
| Max gap-closing distance
 +
| 10
 +
|-
 +
| Max frame gap
 +
| 3
 +
|-
 +
| Nearest neighbor
 +
| Max search distance
 +
| 10
 +
|}
  
 +
Finally, for SNR<4, I filtered out tracks that had less than 4 detections.
  
  

Revision as of 09:39, 16 January 2015

The problem with tracking algorithms is that they always give an answer.

This answer can be completely irrelevant, even non-physical, and there is no built-in flags that would indicate something wrong. The best way to avoid basing your downstream analysis on faulty tracking results is to know in what situation the tracker works the best, and what are its limitations. This is the aim of this page for the trackers and detectors shipped with TrackMate.


The ISBI 2012 single particle challenge.

In 2011-2012, an ISBI Grand Challenge was organized for the Single-Particle Tracking algorithms. Though TrackMate does not offer a completely new algorithm, product of an original Research work, we took the chance and participated in the challenge. The results and the methodology to compute the accuracy of a tracking algorithms were published[1] thereafter.

Unsurprisingly, we did not score amongst the best. At the time, TrackMate was in version 1.1, and ship a stripped down version of the better performing Jaqaman et al. LAP framework[2]. See the LAP trackers section for algorithm details. Plus, TrackMate was was young at the time, and some bugs did not help.


TrackMate v2.7.x series accuracy against the ISBI dataset.

From v2.7.x, TrackMate ships a new tracker that can deal specifically with linear motion. We though it was the right time to re-run the accuracy assessment with the ISBI challenge data. The people behind Icy offered the website to host the challenge data, and it is still available today[3] for download.

The figures below shows the comparison of accuracy for the 3 classes of tracking algorithms available in TrackMate:

  • The LAP framework derived from Jaqaman et al.[2].
  • The linear motion tracker based on Kalman filter.
  • The plain Nearest neighbor tracker for reference.


Scenarios.

It's best to directly read the paper[1] to know what is behind these measures, but here is a brief survey of how they are done. The ISBI dataset covers four scenarios:

Scenario name Particle shape Motion type
MICROTUBULE Slightly elongated shape to mimic MT tip staining. Roughly constant velocity motion.
RECEPTOR Spherical. Tethered motion: switch between Brownian and directed motion with random orientation for the later.
VESICLE Brownian motion.
VIRUS Switch between Brownian and directed motion with fixed orientation for the later.


For each scenario, images covers several particle density:

  • low: 60-100 / frame
  • mid: 400-500 / frame
  • high: 700-1000 / frame

to check how a tracking algorithm behaves when particles get very dense.

Also, particles SNR spans several values: 1, 2, 4, 7 (plus 3 for the RECEPTOR scenario). As said on the challenge page: "SNR=4 is a critical level at which methods may start to break down".


Accuracy measurements.

For each scenario and condition, the method returns numerous values that characterizes the accuracy of a tracking algorithm. They are detailed on this technical paper. We plot below only three of them:

  • The Jaccard similarity between tracks, that quantifies how well the tracks returned by the algorithm match the ground truth. This value assesses the accuracy of the spot tracker you pick in TrackMate. It ranges from 0 (terrible) to 1 (found tracks = ground truth).
  • The Jaccard similarity between detections, that quantifies how well the particle detected by the detection algorithm match the ground truth. It depends strongly on the spot detector you pick in TrackMate, and ranges from 0 to 1 like the above quantity.
  • The RMSE of detection positions that quantifies how precise is the location of the detected particles.


Parameter used.

Unless otherwise specified below, I always used the LoG detector as a spot detector, with an estimated particle diameter of 4, and used sub-pixel accuracy. For SNR below 4, this detector was completely confused and the detection results are dominated by noise. I did not make anything special to improve its sensitivity below this limit.

When the histogram of detection quality returned by the detector as not bimodal, I pick a quality threshold that yielded approximately the expected number of particles in the sequence.

The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.


Spot tracker Parameter Value
Linear motion tracker Initial search radius 10
Search radius 7
Max frame gap 3
Brownian motion Max linking distance 7
Max gap-closing distance 10
Max frame gap 3
Nearest neighbor Max search distance 10

Finally, for SNR<4, I filtered out tracks that had less than 4 detections.


References

  1. 1.0 1.1 Chenouard et al., "Objective comparison of particle tracking methods", Nature Methods, 2014
  2. 2.0 2.1 Jaqaman et al., "Robust single-particle tracking in live-cell time-lapse sequences", Nature Methods, 2008
  3. January 2015