Difference between revisions of "2017-12-04 - Fiji + KNIP hackathon"

(Adding conclusion of parallelization technical discussions)
(Hackathon Progress: Add hackathon achievements for Curtis Rueden)
Line 97: Line 97:
 
* demoed the new client to the community
 
* demoed the new client to the community
 
* encouraged members of the community to use annotations specifying whether a SciJava plugin can run headless or not
 
* encouraged members of the community to use annotations specifying whether a SciJava plugin can run headless or not
 +
 +
=== SciJava - (Curtis Rueden) ===
 +
 +
* Added a minimum Java build version feature to pom-scijava-base ({{GitHub | org=scijava | repo=pom-scijava-base | commit=54bf6664462db3cb538a39986005ac91bded13d5 | label=scijava/pom-scijava-base@54bf6664}}).
 +
* Implemented <code>#@script</code> directive ({{GitHub | org=scijava | repo=scijava-common | commit=d9dce68bf18b4db0fb812e4685f44fb58d6a988e | label=scijava/scijava-common@d9dce68b}}, {{GitHub | org=scijava | repo=scijava-common | issue=294 | label=scijava/scijava-common#294}}).
 +
* In response to a question from {{Person|milkyklim}}, cleaned up the <code>PrefService</code> and fixed related parameter persistence bugs ({{GitHub | org=scijava | repo=scijava-common | commit=195878b7277fff3f350e1e5c457ae395e53bf34e | label=scijava/scijava-common@195878b7}}).
 +
* With {{Person|haesleinhuepf}} and {{Person|frauzufall}}, developed a <code>SearchService</code> API for extensible text-based searches ({{GitHub | org=scijava | repo=scijava-search | label=scijava/scijava-search}}). Initially supports searching SciJava modules (e.g. commands and scripts) and ImageJ web resources. Also offers a code snippet executer, as suggested by {{Person|kephale}}.
 +
* In response to a question from {{Person|rimadoma}}, fixed a bug with <code>DynamicCommand</code> validater method callbacks ({{GitHub | org=scijava | repo=scijava-common | commit=13e4ec325d951cb89b1cba89bcab8889ca25e65f | label=scijava/scijava-common@13e4ec32}}).
 +
* In response to a question from {{Person|rimadoma}}, added a short-term workaround for the fact that you cannot execute <code>DynamicCommand</code> via <code>CommandService#run</code> ({{GitHub | org=scijava | repo=scijava-common | commit=753dd703144371dc715bceb2fd77901e97788002 | label=scijava/scijava-common@753dd703}}).
 +
 +
=== ImageJ Ops - (Curtis Rueden) ===
 +
 +
* Merged {{Person|gselzer}}'s improvements to the transform namespace for intervals ({{GitHub | org=imagej | repo=imagej-ops | pr=515 | label=imagej/imagej-ops#515}}).
 +
* Merged {{Person|xulman}}'s Gabor and bigauss filters ({{GitHub | org=imagej | repo=imagej-ops | pr=485 | label=imagej/imagej-ops#485}}).
 +
* Merged {{Person|gselzer}}'s Frangi vesselness filter ({{GitHub | org=imagej | repo=imagej-ops | pr=525 | label=imagej/imagej-ops#525}}).
 +
* Merged {{Person|bnorthan}}'s bug-fix to the IFFT op ({{GitHub | org=imagej | repo=imagej-ops | pr=529 | label=imagej/imagej-ops#529}}).
 +
* Merged {{Person|bnorthan}}'s diffraction-based kernel, useful for generating PSFs for deconvolution ({{GitHub | org=imagej | repo=imagej-ops | pr=530 | label=imagej/imagej-ops#530}}).
 +
* Merged {{Person|kephale}}'s morphological thinning ops, ported from KNIME ({{GitHub | org=imagej | repo=imagej-ops | pr=317 | label=imagej/imagej-ops#317}}).
 +
* Merged {{Person|EikeHeinz}}'s Sobel and Hessian filters ({{GitHub | org=imagej | repo=imagej-ops | pr=349 | label=imagej/imagej-ops#349}}).
 +
* With {{Person|gab1one}}, worked on fusion ops for combining RAIs that overlap ({{GitHub | org=imagej | repo=imagej-ops | pr=230 | label=imagej/imagej-ops#230}}). PR was closed without merge, but a demo of RAI fusion was pushed to my sandbox ({{GitHub | org=ctrueden | repo=sandbox | commit=b551811277bf5898d2bf838abb92959bc3385de6 | label=ctrueden/sandbox@b5518112}}).
 +
* In response to a question from {{Person|milkyklim}}, pushed an example of using the slice op to iterate an op over planes ({{GitHub | org=ctrueden | repo=sandbox | commit=753dd703144371dc715bceb2fd77901e97788002 | label=ctrueden/sandbox@753dd703}}).
 +
 +
=== ImageJ Legacy - (Curtis Rueden) ===
 +
 +
* In response to a question from {{Person|milkyklim}}, fixed bug with parameter visibility ({{GitHub | org=imagej | repo=imagej-legacy | commit=e89ce40e38956488a41e806dc0cb51cc12611341 | label=imagej/imagej-legacy@e89ce40e}}).
 +
 +
=== ImageJ Launcher - (Curtis Rueden) ===
 +
 +
* Merged {{Person|stelfrich}}'s CI configuration for AppVeyor and Travis CI ({{GitHub | org=imagej | repo=imagej-launcher | pr=49 | label=imagej/imagej-launcher#49}}).
 +
* Made minimal changes required for the the Launcher to work with Java 9 ({{GitHub | org=imagej | repo=imagej-launcher | commit=e501d695f571c82148e35afdbc154835801fbf49 | label=imagej/imagej-launcher@e501d695}}).
 +
 +
=== Bio-Formats - (Curtis Rueden) ===
 +
 +
* In response to a question from {{Person|tischi}} and {{Person|hoerldavid}}, filed a PR to enable the high-level Bio-Formats API to accept file patterns directly ({{GitHub | org=openmicroscopy | repo=bioformats | pr=3019 | label=openmicroscopy/bioformats#3019}}).
 +
* Fixed new and existing Java-8 installations of Fiji to work properly with Bio-Formats again ([http://forum.imagej.net/t/fiji-and-bioformats-conflicting-jars/5736/9?u=ctrueden 1]). This entailed creating a script that instructs users of the old Java 6 version of Fiji to please enable the Fiji-Legacy update site, and instructs users of the new Java 8 version of Fiji to please enable the Java-8 update site.
  
 
[[Category:News]]
 
[[Category:News]]

Revision as of 11:34, 18 December 2017

From Monday, December 4, 2017 through Friday, December 15, 2017, the Max Planck Institute of Molecular Cell Biology and Genetics hosts ~50 developers at the Center for Systems Biology in Dresden, Germany for a hackathon to develop ImageJ2 and Fiji core infrastructure and plugins.


https://gitter.im/fiji/hackathon_dd_2017


Voluntary hackathon calendar

https://tinyurl.com/ybjcq9qw

Hackathon google doc

https://docs.google.com/document/d/1h4uCt4PAEdeGQQwwVZC73o_Anq6-AI2KZiSzxY8kWng/edit

Hackathon on Twitter (#hackdd17)

https://twitter.com/hashtag/hackdd17?vertical=default&src=hash

Technical Discussions

N5, not HDF5

For more info on N5, check out the github repository here.

  • feels like HDF5, but stores chunks(blocks) in separate files in the file system.
  • is a Java library, but Constantin Pape already wrote a C++/python version of it: z5 (also matches "zarr" library), https://github.com/constantinpape/z5
  • attributes are stored in an additional JSON file
  • Discussion: should we define standard now as to how data should be stored in there to prevent an emergence of a zoo of different flavors as there is for HDF5?
    • how to do time series where each timestep / angle could have different image size
    • if we want a general "N5 viewer" for images, we'd have to add calibration data
    • put this information around the N5 dataset, because it behaves more like a dataset within an H5 file.
    • Perhaps make it versioned? Because a duck is not always a duck...
  • why another file format?
    • parallel writes (awesome for clusters with shared filesystem)
    • there is a special type for label blocks
    • blocks can have a halo
    • the block grid does not need to be filled dense, some blocks could be missing
    • couldn't this just be another flavor of HDF5?
  • are parallel writes to the same block prevented by some kind of locks?
  • the HDF5 team should be included in the discussions to learn from their mistakes - there is lots of information on parallel writing of HDF5 files out there
  • Try to write a N5 dataset into a FUSE filesystem/file??? Could this be a work-around for the many-small-files issue?

BigDataViewer

  • There is a fork of BigDataViewer for JavaFx. [1]
  • BigDataViewer will be splitted into a UI independent part, and the Swing UI. This will make it possible to merge the JavaFx and Swing Version of BigDataViewer .

We discussed opportunities to improvement for the Bdv design:

  • Slim the core of bdv, make it less smart and more predictable.
  • Bdv handle for different UI toolkits,
  • Use property pattern for settings like active source ...
  • Actively add/remove views
  • Update views (force cache-dropping) - for BDV add/remove views
  • Access to color settings, overlap rendering, …
  • Allow Grouping of views (externally is ok)
  • Consider Accumulator / Composite that works with type other than ARGBType
  • Make it easy to disable or replace Dialogs
  • Make it easy to remove an overlay
  • BdvVistools show function for converters/volatile

Matrix and Vector libraries

ImageJ/Fiji parallelization

  • initial focus on embarassingly parallel data
  • specific parallel (cluster) implementation will be abstracted
  • from the system architecture point of view, the design can be inspired by the search bar feature implementation
  • started use case definition

Hackathon Progress

ilastik - (Dominik Kutra, Carsten Haubold)

  • developed an ImageJ2 plugin that allows to stream raw data and predictions from the work in progress ilastik processing backend: https://github.com/ilastik/org.ilastik.bdvsource
  • discussed with Philipp Hanslovsky about the benefits of using the N5 format for communication
    • the label block format together with the fact that datasets can be stored sparse could be a great foundation for label storage / communication
    • played around with Constantin Pape's C++/Python N5 reader as available format on the ilastik server
    • noticed that the only thing needed to serve a N5 dataset on the web is to serve the directory contents. Philipp wrote a N5 REST reader for BigCAT to talk to the ilastik server that way
  • worked on some caching problems in the ilastik server
  • discussed possible interfaces between Matthias Arzt's LabKit and ilastik-backend for interactive segmentation in ImageJ/BigDataViewer with an ilastik classifier backend

SciView - (Kyle Harrington, Ulrik Günther, Curtis Rueden)

  • set up automatic deployment of Fiji update site that works on all operating sustems
  • imglib2 images as textures on 3D objects
  • imagej2-style logging
  • public availability of SciView plugin

ImageJ server - (Curtis Rueden, Petr Bainar)

  • discussed its design with respect to the ImageJ/Fiji architecture as well as its future use cases
  • identified the most imminent issues, raised them on GitHub and brainstormed possible solutions
  • worked on a new JavaScript client with enhanced UI, making use of the Angular framework
  • demoed the new client to the community
  • encouraged members of the community to use annotations specifying whether a SciJava plugin can run headless or not

SciJava - (Curtis Rueden)

ImageJ Ops - (Curtis Rueden)

ImageJ Legacy - (Curtis Rueden)

ImageJ Launcher - (Curtis Rueden)

Bio-Formats - (Curtis Rueden)

  • In response to a question from Christian Tischer and David Hörl, filed a PR to enable the high-level Bio-Formats API to accept file patterns directly (openmicroscopy/bioformats#3019).
  • Fixed new and existing Java-8 installations of Fiji to work properly with Bio-Formats again (1). This entailed creating a script that instructs users of the old Java 6 version of Fiji to please enable the Fiji-Legacy update site, and instructs users of the new Java 8 version of Fiji to please enable the Java-8 update site.