Difference between revisions of "TrackMate Accuracy"

Line 74: Line 74:
  
 
The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.
 
The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.
 
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 110: Line 109:
  
  
== References ==
+
=== Results. ===
 +
 
 +
 
 +
==== Microtubule scenario. ====
 +
 
 +
 
 +
==== Receptor scenario. ====
 +
 
 +
[[File:TrackMate RECEPTOR LAP Brownian motion, Linear motion tracker & Nearest neighbor.png|800px]]
 +
 
 +
 
 +
==== Vesicle scenario. ====
 +
 
 +
 
 +
==== Virus scenario. ====
 +
 
 +
 
 +
 
 +
 
 +
 
 +
== References. ==
  
 
<references/>
 
<references/>

Revision as of 09:43, 16 January 2015

The problem with tracking algorithms is that they always give an answer.

This answer can be completely irrelevant, even non-physical, and there is no built-in flags that would indicate something wrong. The best way to avoid basing your downstream analysis on faulty tracking results is to know in what situation the tracker works the best, and what are its limitations. This is the aim of this page for the trackers and detectors shipped with TrackMate.


The ISBI 2012 single particle challenge.

In 2011-2012, an ISBI Grand Challenge was organized for the Single-Particle Tracking algorithms. Though TrackMate does not offer a completely new algorithm, product of an original Research work, we took the chance and participated in the challenge. The results and the methodology to compute the accuracy of a tracking algorithms were published[1] thereafter.

Unsurprisingly, we did not score amongst the best. At the time, TrackMate was in version 1.1, and ship a stripped down version of the better performing Jaqaman et al. LAP framework[2]. See the LAP trackers section for algorithm details. Plus, TrackMate was was young at the time, and some bugs did not help.


TrackMate v2.7.x series accuracy against the ISBI dataset.

From v2.7.x, TrackMate ships a new tracker that can deal specifically with linear motion. We though it was the right time to re-run the accuracy assessment with the ISBI challenge data. The people behind Icy offered the website to host the challenge data, and it is still available today[3] for download.

The figures below shows the comparison of accuracy for the 3 classes of tracking algorithms available in TrackMate:

  • The LAP framework derived from Jaqaman et al.[2].
  • The linear motion tracker based on Kalman filter.
  • The plain Nearest neighbor tracker for reference.


Scenarios.

It's best to directly read the paper[1] to know what is behind these measures, but here is a brief survey of how they are done. The ISBI dataset covers four scenarios:

Scenario name Particle shape Motion type
MICROTUBULE Slightly elongated shape to mimic MT tip staining. Roughly constant velocity motion.
RECEPTOR Spherical. Tethered motion: switch between Brownian and directed motion with random orientation for the later.
VESICLE Brownian motion.
VIRUS Switch between Brownian and directed motion with fixed orientation for the later.


For each scenario, images covers several particle density:

  • low: 60-100 / frame
  • mid: 400-500 / frame
  • high: 700-1000 / frame

to check how a tracking algorithm behaves when particles get very dense.

Also, particles SNR spans several values: 1, 2, 4, 7 (plus 3 for the RECEPTOR scenario). As said on the challenge page: "SNR=4 is a critical level at which methods may start to break down".


Accuracy measurements.

For each scenario and condition, the method returns numerous values that characterizes the accuracy of a tracking algorithm. They are detailed on this technical paper. We plot below only three of them:

  • The Jaccard similarity between tracks, that quantifies how well the tracks returned by the algorithm match the ground truth. This value assesses the accuracy of the spot tracker you pick in TrackMate. It ranges from 0 (terrible) to 1 (found tracks = ground truth).
  • The Jaccard similarity between detections, that quantifies how well the particle detected by the detection algorithm match the ground truth. It depends strongly on the spot detector you pick in TrackMate, and ranges from 0 to 1 like the above quantity.
  • The RMSE of detection positions that quantifies how precise is the location of the detected particles.


Parameter used.

Unless otherwise specified below, I always used the LoG detector as a spot detector, with an estimated particle diameter of 4, and used sub-pixel accuracy. For SNR below 4, this detector was completely confused and the detection results are dominated by noise. I did not make anything special to improve its sensitivity below this limit.

When the histogram of detection quality returned by the detector as not bimodal, I pick a quality threshold that yielded approximately the expected number of particles in the sequence.

The three spot trackers were configured as indicated in the table below. It's not very sensible to keep always the same parameter across different scenario, but with what you can tune in TrackMate, there is little room for fine-tuning.

Spot tracker Parameter Value
Linear motion tracker Initial search radius 10
Search radius 7
Max frame gap 3
Brownian motion Max linking distance 7
Max gap-closing distance 10
Max frame gap 3
Nearest neighbor Max search distance 10

Finally, for SNR<4, I filtered out tracks that had less than 4 detections.


Results.

Microtubule scenario.

Receptor scenario.

TrackMate RECEPTOR LAP Brownian motion, Linear motion tracker & Nearest neighbor.png


Vesicle scenario.

Virus scenario.

References.

  1. 1.0 1.1 Chenouard et al., "Objective comparison of particle tracking methods", Nature Methods, 2014
  2. 2.0 2.1 Jaqaman et al., "Robust single-particle tracking in live-cell time-lapse sequences", Nature Methods, 2008
  3. January 2015