Fijiyama

Revision as of 05:22, 28 February 2020 by Fijiyama (talk | contribs) (Version tracking)
Fijiyama (Fiji)
Author Romain Fernandez & Cédric Moisy
Maintainer Romain Fernandez
Source on GitHub
Initial release February 10th, 2020
Development status stable, active
Category Registration


Current Fijiyama version : Felicity ficus (Release date : February 21th, 2020).

The plugin Fijiyama (Yet Another Matching and Alignment tool for Fiji) is a generic purpose tool for registration and alignment of 3D image series from various imaging modalities (MRI, X-rays, Microscopy, Photography, ...).

Multi-modal time-series Registration

Context: 3D images acquired with various imaging devices or during monitoring experiments often come with different orientations and positions, making it difficult to analyze. Fixing this misalignment in 2D is difficult and become intractable in 3D. In addition, as each modality highlights different structures, images from multiple devices can be very dissimilar. In this situation, structures and phenomenon analysis can be compromised.

Goal: Fijiyama aims to address these issues by helping users to register, align and combine multi-modal 3D images or time-series in a unique reference geometry. After registration, stacks are combined in a coherent multi-modal and/or multi-time volume that can be explored using the ImageJ viewer.

Versatility: Fijiyama is not scale-specific, tissue-specific, nor modality-specific. It has been tested on the most common imaging approaches in biology and medicine: MRI, X-rays, photography...


Reg present 1.png

Demo

An example of Fijiyama application: registration of two stacks of MRI images, acquired on a vine cutting at a 35-days interval (see "training tutorial" for details).

Plugin features

  • Two images registration (training / settings mode)
  • N-times / M-modalities 3D series registration
  • Composition of successive ITK transformations into a single transformation
  • Transformation of a 3D image (or a 4D / 5D hyperimage) with an ITK transformation

In order to address the complex task of multi-modal registration, computation is based on the Block Matching algorithm (Ourselin et al., 2000), a generic purpose registration algorithm, developed in the field of medical imaging. Another common algorithm (Iconic registration) is available in the plugin as an option. To learn more about the algorithmic bases of the plugin, see section "The maths behind".

Installation

Install Fiji : follow this link https://fiji.sc/ to download and install Fiji. Run it a first in order to let him manage the first updates, then restart it.
Install Fijiyama : In Fiji, run the Update manager (Help > Update). Click on "OK" to close the popup, then click on the button Manage update sites... to open the update list. In this list :

  • Activate both ImageJ-ITK and IJPB-Plugins repositories (by checking the associated checkboxes in the list).
  • Add the Fijiyama repository (by clicking on the button Add update site and filling the fields : name = "Fijiyama", site = https://sites.imagej.net/Fijiyama), then check the associated checkbox.


Restart Fiji. A Fijiyama entry should be visible in the menu Plugins > Registration. If it's not the case, go back to the update manager, and check that the three repositories are checked (ImageJ-ITK, IJPB-Plugins repositories and Fijiyama).

Preparing your data

In order to register your data, Fijiyama needs properly formatted dataset:

  • For two images pairwise registration : Two 3D stacks/images of the same object. Use your own data or the provided example datasets here: case study 1, two-images registration
  • For series registration : N 3D stacks/images of the same object. Use your own data or the provided example datasets here: case study 4, times series registration.
  • An empty "Output" directory, that will be used to store the Fijiyama configuration file, the intermediate results (transformations) and the final results (transformations and resulting 3D images). The configuration file (*.fjm) keep a full track of your work. When saving the current registration experiment, Fijiyama stores the successive registration steps and the associated computed files. When restarting later from this point, you can review the whole process, and "undo" one or more steps.


Before starting registration, check your images calibration:

  • Intensities: verify there is a proper contrast between your object of interest and the background when opened in ImageJ. If needed, use Image > Adjust > Brightness / Contrast, and set the min and max values, then close the Brightness / Contrast window.
  • Voxel sizes: using ImageJ, check the image properties (Image > Properties...) and correct the 4 central values, if needed. Unit of length should have another value than "pixel" (it can be "mm", "cm", "µm" for example). Check coherence of pixel width / height / depth according to your unit of length. If needed, open the "info" menu (Image > Show info...), to get dicom parameters, or any other parameters. To visualize if the voxel depth is coherent with the pixel size, along X and Y axis, you can render your object as a 3D volume using the 3D viewer (plugins > 3D viewer).

Once signal intensity and voxel sizes have been checked, save your modified images. Then run Fijiyama, and select the two images registration mode or the series registration mode.

The registration manager window

Fijiyama reg manager.png

Once a registration scenario starts, you can interact with the registration manager window through (see image) :

  • The log window : a guide to remember last operations, and to understand what the plugin expects from you
  • The settings menus and buttons : useful to set the parameters of the future action
  • The action buttons : the first line (Start, Abort, and Undo) drives the lifecycle of the current action, defined by the menus and the current settings. Save can be used to save the progress made in the registration, as a "checkpoint". Export computes the resulting aligned 3D images, according with the succession of transformations performed. Help can be used at any time to open a contextual help message.
  • The actions list : stores the successive steps accomplished during the registration process.

Tutorial 1 : Two images registration

In order to achieve an efficient registration of two 3D images, we recommend the following procedure:

  • Step 1 : Prepare your data or download the example set: case study 1, two-images registration
  • Step 2 : Select 'Manual registration' in the first menu and run it. Superimpose roughly the two volumes manually (centers should be roughly aligned, with angle < 15 degrees). Press on 'Position ok' (green button) when ready. This first step will help Fijiyama finding the correct orientation.
  • Step 3 : Select 'Automatic registration' with default parameters (Block matching). If you want to monitor the registration during the run, select the 'Display automatic registration' in the “Manual registration viewer” menu. Before starting this action, notice the estimated time required to complete the action (calculated from your computer settings).
  • Step 3-bis (optional, depending on the data) : Select 'Automatic registration' with default parameters (Block matching), and select "Vector field" in the "Transformation to estimate" menu.
  • Step 4 : Select 'Align both images with XYZ' with default parameters. Turn the scene to improve its orientation, if needed for a more convenient representation of your data (production of figures or analyses). In the example dataset, we choose to align the red cylinder with the white lines (Z axis), and align the wounding point (a hole in the surface) with the axis X or Y.
  • Step 5 : Save the final transformation using the 'Save current state' button.
  • Step 6 : Export the registered 3D images in the “output/Exported_data” directory previously selected, using the 'Export results' button.

Tutorial 2 : Multimodal time-series registration

Once comfortable with the "two images registration" module, try a “series registration” following these steps. If you feel lost, you can rely on the tutorial here :

Data names should respect a generic form, for example : img_t{Time}_mod{ModalityName}.tif , where {Time} represents the successive values for each time point, and {ModalityName} represents the name of each modality. In the example dataset, this is already done, since the image names are : MRI_D0.tif , MRI_D1.tif , MRI_D2.tif , MRI_D3.tif, which corresponds to a monomodal series with the generic expression MRI_D{Time}.tif .

  • Step 2 : Use the "Two images" training module to identify the best pipeline of actions for registering your images, from a modality to another, or from an observation time to another.
  • Step 3 : Start the module 'Series registration', and follow the instructions. The manager will ask the pipeline for inter-time registration, then for inter-modalities registration. Once done, it will compose these steps into a full pipeline and process. You can repeatedly press on 'start this action', and execute all the steps.
  • Step 4 : Once all actions needing human intervention are performed, use the 'Chain run' button to ask Fijiyama to execute successively all the steps automatically.
  • Step 5 : Run the last actions. The manual alignment step will set the reference geometry in the image axis, and the save and export steps will stores the actions and the results. The results are saved in the output/Exported_data directory.

Complementary features

Compose transformations

Chaining linear and/or dense transformations is one of the plugin core features for series processing. This function can also be used as a helper.
Guessing you already computed :

  • the transform T12 that aligned Image1.tif with Image2.tif
  • the transform T23 that aligned Image2.tif with Image3.tif

You may want to align Image1 with Image3 in order to compare the three 3D images altogether. However, applying successive transformations will result in a strong blurring effect. In order to avoid this blur, you may want to combine the two transformation steps into a single one.

Use the compose transformations function the successive transformations to be composed (in the example T12 then T23), and produce the resulting combined transform.

Apply a transformation to a 3d/4d/5d Image

This module is a helper function to apply an ITK transformation to a 3D image or to a 4D/5D hyperimage. In case of hyperimage, the transformation is interpreted as a 3D transformation, and is applied to each channel / frame of the hyperimage before the transformed 3d images are gathered back in an hyperstack.

Versatility Tests

Fijiyama plugin has been tested on several data (including vegetal and human tissues), at different scales (from micro to macro imaging), and on different imaging modalities (microscopy, MRI, X-rays, and photographies).

See an example in the video below : Human abdomen CT-scan and MRI registration (Courtesy of Samuel Merigeaud, Tridilogy SARL)

Accuracy Test

Fijiyama can be used to quantify the resulting mismatch. This measurement process is integrated in the plugin as an extra functionality, to help users evaluating their results (see video below)

-Video : “Accuracy Test : Quantifying mismatch”

Version tracking

Fijiyama is a living project, with frequent updates. Thanks to the Fiji updater, your version will be update with the latest one each time the fiji update starts.

Major update include new features released, or major refactoring, while minor update (change in the release time) should not modify deeply the behaviour of the plugin, to keep results reproducible. These two informations can be seen in the launching interface of Fijiyama.

Major update are symbolized with a change in the plugin version name. Minor update can be seen as the release time is modified.


Elastic Eucalyptus First release: 2020/02/10 Last release: 2020/02/20 Integration of the multimodal series registration feature. Redefinition of default settings.
Felicity Ficus First release: 2020/02/20 Last release: 2020/02/28 Integration of the time-lapse multimodal series feature
(Future version) Gorgeous grapevine First release: 2020/03/.. Last release: 2020/03/.. ...

Possible issues and solutions proposed

  • Dealing with large datasets : biological images are sometimes very large. In the field of time-lapse microscopy, BigStitcher plugin handles such data, using BigDataViewer, as its registration algorithm, perfect for monomodal data of lightsheet microscopy, is well fitted to it.

Fijiyama is based on the Blockmatching registration algorithm, robust for inter-modality. This algorithm is memory consuming and computation intensive. When starting Fijiyama, the plugin analyze your computer settings and might propose to subsample your dataset to fasten the registration process. This will not impact the appearance of the final results since the exportation procedure is performed using the initial images, at the chosen resolution.

  • Translation in 3D under Windows : under Windows, translation mode in the 3D viewer using SHIFT+drag can be hazardous. To address this, we associated surnumerous controls to the numerical keypad. Using “4” and “6” keys, you can translate the selected volume along the X axis. Using “8” and “2” along the Y axis. Using “5” and “0”, along the Z axis. Rotations can be handled with “7” and “9” (X axis), “1” and “3” (Y axis), and characters “p” and “o” for the Z axis.
  • 3D view disappears : depending on the sequence of translations / rotations applied to the scene, the viewer can "freeze". Turn the objects to a 90 degrees’ angle, and the viewer come back
  • Other issues ? please tell us ! Try to "repeat" your bug from the beginning, identify the context producing the bug, then send an email to corresponding author, including :

- a compressed copy of your output dir - the copy of the content of the ImageJ log window (probably a hundred lines of sentences in red telling where Exceptions were encountered) - Any useful commentary - if possible, the content of the black log window - if possible, a link to the dataset used

The maths behind

Registration of two images is achieved estimating a geometrical transformation that can be applied to an image (the "moving one"), to superimpose it with another image (the "reference" one).

Transformations

Fijiyama handles the following transformations families :

  • Rigid transformations (translation + rotation)
  • Similarities (rigid + isotropic homothetic factor)
  • Dense vector field (A generic representation of a non-linear transformation)

For more details about transformations, check the supplementary data of the official publication (see section "Cite this work" below)

Optimizers

Common categories of registration optimizers, associated with different registration strategies :

  • ITK (Yoo et al. 2002) iconic algorithm optimizes iteratively a transformation. The optimization process is guided by the superposition improvement measured using a global similarity measure between the two 3D images. Choosing this measure, we assume a relationship between intensities of reference and moving images, and that this relationship is valid and identical in any point of the image. That weak assumption can lead to robustness issues when dealing with multi-modal registration.
  • The Block-Matching algorithm (Ourselin et al. 2000) is a hybrid method, using a similarity measure to establish correspondences between subparts of images, then using these correspondences to compute a global transformation for the whole image. Assuming that the relationship between intensities of reference and moving images can be valid in a local neighborhood, blockmatching uses it to compare subparts of the 3D images to identify correspondences. This algorithm is the default one in Fijiyama.
  • Fijiyama does not have a "geometric" registration algorithm yet. Principle of such algorithms is to extract features from each image, and to establish correspondences between extracted features. Such approach can lead to faster but less robust algorithms as it is difficult to define a feature extraction method that is modality-independent, object-independent, and scale-independent. But for monomodal registration, such algorithms can provide great success, like BigStitcher with microscopy images

Citing this work

  • R. Fernandez and C. Moisy, “Fijiyama: a registration tool for 3D multimodal time-lapse imaging” (under review)

References

  • Hörl, David, Fabio Rojas Rusak, Friedrich Preusser, Paul Tillberg, Nadine Randel, Raghav K. Chhetri, Albert Cardona, et al. 2019. « BigStitcher: Reconstructing High-Resolution Image Datasets of Cleared and Expanded Samples ». Nature Methods 16 (9): 870-74. https://doi.org/10.1038/s41592-019-0501-0.
  • Ourselin, S., A. Roche, S. Prima, et N. Ayache. 2000. « Block Matching: A General Framework to Improve Robustness of Rigid Registration of Medical Images ». In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2000, édité par Scott L. Delp, Anthony M. DiGoia, et Branislav Jaramaz, 1935:557-66. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-40899-4_57.
  • Schindelin, Johannes, Ignacio Arganda-Carreras, Erwin Frise, Verena Kaynig, Mark Longair, Tobias Pietzsch, Stephan Preibisch, et al. 2012. « Fiji: An Open-Source Platform for Biological-Image Analysis ». Nature Methods 9 (7): 676-82. https://doi.org/10.1038/nmeth.2019.
  • Yoo, Terry S., Michael J. Ackerman, William E. Lorensen, Will Schroeder, Vikram Chalana, Stephen Aylward, Dimitris Metaxas, et Ross Whitaker. 2002. « Engineering and Algorithm Design for an Image Processing Api: A Technical Report on ITK--the Insight Toolkit ». Studies in Health Technology and Informatics 85: 586-92.

Software dependencies and acknowledgements

  • ITK/SimpleItk : We especially thanks the SimpleItk and ImageJ-ITK projects to bring ITK functionalities wrapped to the Java world of ImageJ. The class used in Fijiyama to define 3D transformations (ItkTransform) is based on the ITK/SimpleITK class "Transform", and extends some of its functionalities. ITK/SimpleITK also provides the ITK iconic registration algorithm.
  • ImageScience : Rigid transform and similarity estimation from correspondence points are based on ImageScience and VIBLib

License

This program is an open-source free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation (http://www.gnu.org/licenses/gpl.txt).

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.