Scijava-parallel

Revision as of 02:15, 7 October 2019 by Velissarious (talk | contribs)
Scijava-parallel (Fiji)
Author Petr Bainar, Jan Kožusznik
Maintainer Jan Kožusznik
Source on github
Initial release October 2019
Latest version October 2019
Category Category:Development


Scijava-parallel is a framework that provides an access point for using computational resources such as remote computers or HPC clusters. It hides details about configuration and initialization of such resources.

Types of users

Scijava-parallel is designed to be used by three types of users and we call them: developers, Fiji users and parallelization paradigm providers.

A Developer is a user who uses its defined API for accessing configured computational resources. Remote computational resources are accessible through components that implement an interface derived from the interface ParallelParadigm. These components are called parallelization paradigm implementations and specialized interfaces are called parallelization paradigms.

A Fiji user provides configurations for parallelization paradigm implementations. The configurations are called profiles. Users also select one configuration that is used when some code requests a parallelization paradigm.

A parallelization paradigm provider offers new parallelization paradigm implementations or new types of parallelization paradigm.

Developers

Developers obtain parallelization paradigm implementation that is in selected profile by method getParadigmOfType. Desired parallelization paradigm is specified when the method is used and when parallelization paradigm implementation does not implement the type null value is returned. ParallelService.png

Base type ParallelParadimg contains three methods:

  • init - initializes resources required for and used by parallelization paradigm;
  • close - disposes initialized resource;
  • getStatus - returns status indicating whether the paradigm is initialized or disposed.

First parallelization paradigm is type RPCParadigm that has methods:

  List<Map<String, Object>> runAll(String commandName, List<Map<String, Object>> parameters)
  List<CompletableFuture<Map<String, Object>>> runAllAsync(String commandName, List<Map<String, Object>> parameters);

The methods cause parallel execution of specified command for parameters. A size of a parameters defines how many executions are performed. Parallelization and distribution of requests is solved by the parallelization paradigm implementation used. Three implementations of parallelization paradigm exist:

  • LocalMultithreadedParadigm in the module scijava-parallel that executes commands in the same instance of running Fiji and uses multiple threads for parallelization.
  • FSTRPCParadigm in the module scijava-parallel-fst that executes commands in different instances of running Fiji. Fiji can run in a different computer or an HPC cluster. Communication is realized through TCP/IP and data are serialized by library fst.
  • ImageJServerParadigm in the module scijava-parallel-imagej-server that also executes commands in different instances of running Fiji. The paradigm uses ability of imagej-server to execute commands through its provided REST API.

Fiji users

Scijava-parallel offers plugin that allows management of paradigms and profiles. It is accessible from menu Plugins > Scijava parallel > Paradigm Profiles Manager. The plugin shows the following form:

ParadigmProfilesManager.png

Profiles can be:

  • created - button Create at the bottom;
  • edited - button Edit in the central part;
  • deleted - button Delete in the central part; and
  • selected - button Select in the central part.

Profile creation and editing

During the creation of a profile a user specifies the implementation of an associated paradigm - FSTRPCParadigm is selected in the screenshot - paradigm manager. Paradigm managers are supplied together with parallelization paradigm implementations by parallelization paradigm providers. Paradigm managers manage resources used by a parallelization paradigm implementation. There are three paradigm managers available that differ in the location where they start a server handling requests:

  • Inprocess FSTPRC - in an actual running instance of Fiji;
  • Local FSTRPC - on local machine as an another process;
  • FSTRPC on HPC - on remote HPC cluster through ssh.

This is done in order to allow the user to provide basic settings during a profile creation. The settings required change according to the selected paradigm manager. Local FSTPRC paradigm manager requires the user to provide the path to a Fiji installation in a local filesystem while FSTRPC on HPC requires the user to provide information that will allow connection to a remote HPC cluster - host name,credentials - and the location of a Fiji installation on HPC cluster.

Profile selection

This action marks a paradigm profile as selected. An instance of the paradigm associated with a selected profile is returned as the result of calling the method getParadigmOfType in case that the parallelization paradigm implementation provides (implements an interface) a requested parallelization paradigm type. In the present implementation of the parallel service, only one profile can be marked as selected but we see it as an implementation detail that could be changed in the future.

Parallelization paradigm management

The manager permits the user to perform basic management of the parallelization paradigm implementation in a selected profile. A user can initialize and close a profile by the check box Paradigm Active. The paradigm manager FSTRPC on HPC offers a possibility to leave a Fiji, that is started in a job on a remote HPC, running while is an associated parallelization paradigm implementation closed. This property is specified by a user during

Parallelization paradigm providers

A Parallelization paradigm implementation requires only to implement one or more parallelization paradigms and to annotate this class as plugin:

@Plugin(type = ParallelizationParadigm.class)
public class MPIParadigmImpl implements MPIParadigm

The parallelization paradigm implementation very often also requires some specific initialization. Settings for the initialization are expected in an instance of ParallelizationParadigmProfile or its subclass. Scijava-parallel offers a framework for a profile creation, its storing in persistent storage and paradigm initialization with contained settings. The framework provides the user interface described earlier.

The framework provides interface ParadigmManager and its implementation is used by the framework for a profile creation (method createProfile) and its editing (method editProfile). Parallelization paradigm implementation is initialized by methodprepareParadigm. Parallelization paradigm provider implements the interface ParadigmManager and annotates it as @Plugin with a type ParadigmManager.

Paradigm-manager.png

A class ParadigmManagerUsingRunner supports very often scenario how ParallelizationParadigm is initialized: external software is launched - is is always Fiji in case of existing implementation - and parallelization paradigm is initialized with information how to access the software (host name, port number). It works with profile type ParadigmProfileUsingRunner that contains object of a type RunnerSettings or its subtype (e.g. LocalImagejRunnerSettings). ParadigmManagerUsingRunner starts external software through an interface ServerRunner. ParadigmManagerUsingRunner edits settings with an object that implements RunnerSettingsEditor. Implementing class should be annotated with the annotation @Plugin with the type RunnerSettingsEditor. It enables to scijava-parallel find an editor for a given type of settings. There are registered editors for existing implementations of RunnerSettings.

A developer makes non-abstract child of the class ParadigmManagerUsingRunner. The child need to implement only two abstract methods: getTypeOfRunner and initParadigm. It is possible to create a new implementation of ServerRunner or reuse one from existing: LocalImagejRunner, HPCImageJRunner.

Paradigm-manager-using-runner.jpg

Installation

Scijava-parallel can be installed directly into Fiji from its update site https://sites.imagej.net/Scijava-parallel/. We also provide modified version of Labkit that has support for scijava-parallel. This version of Labkit has own update site https://sites.imagej.net/Imglib2-labkit-cluster/. It only demonstrates scijava-parallel and it is not meant as a production software.