Fijiyama

Revision as of 08:58, 12 February 2020 by Fijiyama (talk | contribs) (Transformations)
Revision as of 08:58, 12 February 2020 by Fijiyama (talk | contribs) (Transformations)
Fijiyama (Fiji)
Author Romain Fernandez & Cédric Moisy
Maintainer Romain Fernandez
Source on GitHub
Initial release February 18th, 2020
Development status unstable, active
Category Registration


(This software is currently in beta version. Check the top of the page to know the date of the first stable version release. If you cannot wait to see it, follow the installation instructions to download it, and use it at your own risks.)

Fijiyama (Yet Another Matching and Alignment tool for Fiji) is a tool for registration and alignment of 3D stacks/images collected on the same object but using different imaging modalities (MRI, X-rays, Microscopy, Photography, ...) and/or during time-lapse monitoring (successive observations of the same sample at different times).

Contents

Multi-modal and/or Multi-time Registration

CONTEXT: 3D images acquired with various imaging devices or during monitoring experiments often come with different orientations and positions, making it difficult to analyze. Fixing this misalignment in 2D can be difficult but it is even harder in 3D. In addition, as each modality can highlights different structures, images from multiple devices can be very dissimilar. In this situation, cross-structure and -phenomenon analysis from multimodal 3D-images or time-series can be compromised.

AIM: Fijiyama aims to address these issues by helping users to register, align and combine 3D images or time-series in a unique reference geometry. Therefore, stacks of 3D images a registered in a coherent multi-modal and/or multi-time volume that can be explored using other tools.

VERSATILITY: Fijiyama is not dedicated to a specific scale, a specific biological tissue, nor a specific imaging device. It has been developed for multi-modalities, including the most common imaging approaches in biology and medicine: microscopy, MRI, X-rays, photography... and therefore works in several conditions.


Reg present 1.png

Demo

An example of Fijiyama application: registration of two stacks of MRI images, acquired on a vine cutting at a 35-days interval (see "training tutorial" for details).

Plugin features

  • Two 3D images registration (training / settings mode)
  • N-times / M-modalities 3D series registration (main feature)
  • Composition of successive ITK transformations into a single transformation
  • Transformation of a 3D image (or a 4D / 5D hyperimage) with an ITK transformation

In order to address the complex task of multi-modal registration, computation is based on Block-matching, a generic purpose registration algorithm, developed in the field of medical imaging. Another common algorithm (ITK iconic registration) is available in the plugin as an option. To learn more about the algorithmic bases of the plugin, see section "The maths behind".

Installation

Add the plugin and its dependencies to your current Fiji distribution using the Fiji update manager (Help > Update > Manage update sites...). First, activate both ImageJ-ITK and IJPB-Plugins repositories (by checking the associated checkboxes in the list). Then, add the Fijiyama repository to the list (by clicking on Add update site and filling the fields : name = "Fijiyama", site = https://sites.imagej.net/Fijiyama). Then check the associated checkbox. Restart Fiji. The corresponding Fijiyama entry should be visible in the menu Plugins > Registration.

Registration manager

>>>TODO:Mettre a jour sos dans appli et dans figure et refaire le snapshot <<<

Fijiyama reg manager.png

Start Fijiyama from the "Plugins" menu of ImageJ (Plugins > Registration). Click on "Two 3D images registration" and follow the instructions (choose the 'reference' and moving' images, and select the 'output' directory). Once the registration manager opens, explore the manager and read the "tooltips" pop-ups, the contextual help, until you feel comfortable.

The registration manager window shows four parts (top to bottom):

  • Log window : a guide to remember last operations, and to understand what the plugin expects from you
  • Settings menus and buttons : useful to set the parameters of the future action
  • Action buttons : the first line (Start, Abort, and Undo) drives the lifecycle of the current action, defined by the menus. Save can be used to save the progress made in the registration, as a "checkpoint". Export computes the resulting aligned 3D images, according with the succession of transformations performed. Help can be used at any time to open a contextual help message.
  • Actions list : stores the different steps during the registration process.

Preparing your data

In order to register your data, Fijiyama needs properly formatted dataset:

  • Two 3D stacks/images of the same object. Use your own data or the provided example datasets here : File:Test dataset 01 vine crops.zip and follow the steps watching the video above.
  • An empty "Output" directory, that will be used to store the Fijiyama configuration file, the intermediate results (transformations) and the final results (transformations and resulting 3D images). The configuration file (*.fjm) keep a full track of your work. When saving the current registration experiment, Fijiyama stores the successive registration steps and the associated computed files. When restarting later from this point, you can review the whole process, and eventually "undo" one or more steps.

Before starting registration, check that your images are well calibrated:

  • Intensities: verify there is a proper contrast between your object of interest and the background when opened in ImageJ. If needed, use Image > Adjust > Brightness / Contrast, and set the min and max values, then close the Brightness / Contrast window.
  • Voxel sizes: using ImageJ, check the image properties (Image > Properties...) and correct the 4 central values, if needed. Unit of length should have another value than "pixel" (it can be "mm", "cm", "µm" for example). Check coherence of pixel width / height / depth according to your unit of length. If needed, open the "info" menu (Image > Show info...), to get dicom parameters, or any other parameters. To visualize if the voxel depth is coherent with the pixel size, along X and Y axis, you can render your object as a 3D volume using the 3D viewer (plugins > 3D viewer).

Once signal intensity and voxel sizes have been checked, save your modified images.

Tutorial 1 : Two 3D images registration

In order to achieve an efficient registration of two 3D images, we recommend the following procedure:

  • Step 1 : Prepare your data or download the example set number 1 : File:Test dataset 01 vine crops.zip
  • Step 2 : Select 'Manual registration' in the first menu and run it. Superimpose roughly the two volumes manually (centers should be roughly aligned, with angle < 15 degrees). Press on 'Position ok' (green button) when ready. This first step will help Fijiyama finding the correct orientation.
  • Step 3 : Select 'Automatic registration' with default parameters (Block matching). If you want to monitor the registration during the run, select the 'Display automatic registration' in the “Manual registration viewer” menu. Before starting this action, notice the estimated time required to complete the action (calculated from your computer settings).
  • Step 3-bis (optional, depending on the data) : Select 'Automatic registration' with default parameters (Block matching), and select "Vector field" in the "Transformation to estimate" menu.
  • Step 4 : Select 'Align both images with XYZ' with default parameters. Turn the scene to improve its orientation, if needed for a more convenient representation of your data (production of figures or analyses). In the example dataset, we choose to align the red cylinder with the white lines (Z axis), and align the wounding point (a hole in the surface) with the axis X or Y.
  • Step 5 : Save the final transformation using the 'Save current state' button.
  • Step 6 : Export the registered 3D images in the “output/Exported_data” directory previously selected, using the 'Export results' button.

Tutorial 2 : Multimodal time-series registration

Once comfortable with the "two images registration" module, try a “series registration” following these steps :

  • Step 1 : Prepare your data or download the example set number 2 : link TODO >>>>blabla<<<<.

Data names should respect a generic form, for example : img_t{Time}_mod{ModalityName}.tif , where {Time} represents the successive values for each time point, and {ModalityName} represents the name of each modality. In the example dataset, this is already done, since the image names are : img_t1_modRX.tif , img_t1_modMRI.tif , img_t2_modMRI.tif , img_t3_modRX.tif , img_t3_modMRI.tif

  • Step 2 : Use the "Two images" training module to identify the best pipeline of actions for registering your images, from a modality to another, or from an observation time to another.
  • Step 3 : Start the module 'Series registration', and follow the instructions. The manager will ask the pipeline for inter-time registration, then for inter-modalities registration. Once done, it will compose these steps into a full pipeline and process. You can repeatedly press on 'start this action', and execute all the steps.
  • Step 4 : Once all actions needing human intervention are performed, use the 'Chain run' button to ask Fijiyama to execute successively all the steps automatically.
  • Step 5 : After an optional final manual alignment, and results exportation using the 'Export results' button. The results are saved in the output/Exported_data directory.

Complementary features

Compose transformations

Chaining linear and/or dense transformations is a basis of the plugin for series processing. This module is a helper module to use this function standalone. Guessing you already computed :

  • the transform T12 that aligned Image1.tif with Image2.tif
  • the transform T23 that aligned Image2.tif with Image3.tif

You may want to align Image1 with Image3 in order to compare the three 3D images altogether. However, applying successive transformations will result in a strong blurring effect. In order to avoid this blur, you may want to combine the two transformation steps into a single one.

This module asks the successive transformations to be composed (in the example T12 then T23), and produce the resulting combined transform.

Apply a transformation to a 3d/4d/5d Image

This module is a helper function to apply an ITK transformation to a 3D image or to a 4D/5D hyperimage. In case of hyperimage, the transformation is interpreted as a 3D transformation, and is applied to each channel / frame of the hyperimage before gathering them back into the shape of an hyperimage.

Versatility Tests

Fijiyama plugin has been tested on several data (including vegetal and human tissues), at different scales (from micro to macro imaging), and on different imaging modalities (microscopy, MRI, X-rays, and photographies).

See an example in the video below : Human abdomen CT-scan and MRI registration (Courtesy of Samuel Merigeaud, Tridilogy SARL)

Accuracy Test

The registration algorithms used in Fijiyama have been studied in the literature, including accuracy tests (see the corresponding papers in the section "references" below). Therefore, we conducted a complementary study to evaluate results quality within each of our datasets. To evaluate the overall final alignment, we computed the “mean mismatch after registration”, measured as the average distance between corresponding points in the registered volume. In our first dataset, where the tissues undergo large deformations during the 35-days gap between the two images acquisition, we measured a 'maximal mismatch' (over all measured points) of 2.1 pixels, and a 'mean resulting mismatch' lower than one pixel (mean=0.76 pix/27µm, std=0.45 pix/16µm).

This measurement process is integrated in the plugin as an extra functionality, to help users evaluating their results (see video below)

-Video : “Accuracy Test : Quantifying mismatch”

Possible issues and solutions proposed

  • Dealing with large datasets : biological images are sometimes very large. In the field of time-lapse microscopy, BigStitcher plugin handles such data, using BigDataViewer, as its registration algorithm, perfect for monomodal data of lightsheet microscopy, is well fitted to it.

Fijiyama is based on the Blockmatching registration algorithm, robust for inter-modality. This algorithm is memory consuming and computation intensive, so user might consider down sampling the data before starting Fijiyama. When starting Fijiyama, it will automatically detect your computer settings and might propose to subsample your dataset to fasten the registration process. This will not impact the quality of the final results since the transformations will finally be performed in the original calibrated space and resolution.

  • Translation in 3D under Windows : under Windows, translation mode in the 3D viewer using SHIFT+drag can be hazardous. To address this, we associated surnumerous controls to the numerical keypad. Using “4” and “6” keys, you can translate the selected volume along the X axis. Using “8” and “2” along the Y axis. Using “5” and “0”, along the Z axis. Rotations can be handled with “7” and “9” (X axis), “1” and “3” (Y axis), and characters “p” and “o” for the Z axis.
  • 3D view disappears : depending on the sequence of translations / rotations applied to the scene, the viewer can "freeze". Turn the objects to a 90 degrees’ angle, and the viewer come back
  • Other issues ? please tell us ! Try to "repeat" your bug from the beginning, identify the context producing the bug, then send an email to corresponding author, including :

- a compressed copy of your output dir - the copy of the content of the ImageJ log window (probably a hundred lines of sentences in red telling where Exceptions were encountered) - Any useful commentary - if possible, the content of the black log window - if possible, a link to the dataset used

The maths behind

Registration of two images is achieved estimating a geometrical transformation that can be applied to an image (the "moving one"), to superimpose it with another image (the "reference" one).

Transformations

Fijiyama handles the following transformations families :

  • Rigid transformations (Translation + Rotation)
  • Similarities (Rigid + isotropic homothetic factor)
  • Dense vector field (one vector per voxel)

For more details about transformations, check the supplementary data of the official publication (see section "Cite this work" below)

Optimizers

Common categories of registration optimizers, associated with different registration strategies :

  • ITK iconic algorithm optimizes iteratively a transformation. The optimization process is guided by the superposition improvement measured using a global similarity measure between the two 3D images. Choosing this measure, we assume a relationship between intensities of reference and moving images, and that this relationship is valid and identical in any point of the image. That weak assumption can lead to robustness issues when dealing with multi-modal registration.
  • The Block-Matching algorithm (reference) is a hybrid method, using a similarity measure to establish correspondences between images, then using these correspondences to compute a global transformation for the whole image. Assuming that the relationship between intensities of reference and moving images can be valid in a local neighborhood, blockmatching uses it to compare subparts of the 3D images to identify correspondences. This algorithm is the default one in Fijiyama.
  • Fijiyama does not have a "geometric" registration algorithm yet. Principle of such algorithms is to extract features from each image, and to establish correspondences between extracted features. Such approach can lead to faster but less robust algorithms as it is difficult to define a feature extraction method that is modality-independent, object-independent, and scale-independent. But in the case of monomodal registration, such algorithms can provide great success (reference : BigStitcher).

Citing this work

  • R. Fernandez and C. Moisy, “Fijiyama : a versatile registration tool for 3D multimodal time-lapse monitoring of biological tissues in Fiji” (under review)

References

  • Grocott, Timothy, Paul Thomas, et Andrea E. Münsterberg. 2016. « Atlas Toolkit: Fast Registration of 3D Morphological Datasets in the Absence of Landmarks ». Scientific Reports 6 (1): 1-7. https://doi.org/10.1038/srep20732.
  • Hörl, David, Fabio Rojas Rusak, Friedrich Preusser, Paul Tillberg, Nadine Randel, Raghav K. Chhetri, Albert Cardona, et al. 2019. « BigStitcher: Reconstructing High-Resolution Image Datasets of Cleared and Expanded Samples ». Nature Methods 16 (9): 870-74. https://doi.org/10.1038/s41592-019-0501-0.
  • Ourselin, S., A. Roche, S. Prima, et N. Ayache. 2000. « Block Matching: A General Framework to Improve Robustness of Rigid Registration of Medical Images ». In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2000, édité par Scott L. Delp, Anthony M. DiGoia, et Branislav Jaramaz, 1935:557-66. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-40899-4_57.
  • Schindelin, Johannes, Ignacio Arganda-Carreras, Erwin Frise, Verena Kaynig, Mark Longair, Tobias Pietzsch, Stephan Preibisch, et al. 2012. « Fiji: An Open-Source Platform for Biological-Image Analysis ». Nature Methods 9 (7): 676-82. https://doi.org/10.1038/nmeth.2019.
  • Yoo, Terry S., Michael J. Ackerman, William E. Lorensen, Will Schroeder, Vikram Chalana, Stephen Aylward, Dimitris Metaxas, et Ross Whitaker. 2002. « Engineering and Algorithm Design for an Image Processing Api: A Technical Report on ITK--the Insight Toolkit ». Studies in Health Technology and Informatics 85: 586-92.

Software dependencies and acknowledgements

  • ITK/SimpleItk : We especially thanks the SimpleItk and ImageJ-ITK projects to bring ITK functionalities wrapped to the Java world of ImageJ. The class used in Fijiyama to define 3D transformations (ItkTransform) is based on the SimpleItk class "Transform", and extends some of its functionalities. SimpleItk/ITK also provides the ITK iconic registration algorithm.
  • ImageScience : Rigid transform and similarity estimation from correspondence points are based on ImageScience and VIBLib

License

This program is an open-source free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation (http://www.gnu.org/licenses/gpl.txt).

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.