[ImageJ-devel] [fiji-devel] ImgLib2 + KNIME hackathon report

[Mark Hiner]

Hi all,

SCIFIOCellImg with Memory Service
> * Development branch: https://github.com/scifio/scifio/tree/cache-imps
> * Refactored SCIFIOCells to cache via phantom reference instead of
> finalization (done)
> * Created MemoryService to manage a new RefHandler class of plugins and
> provide configurable memory safeguards (done)
> * Taking a test case of re-writing each pixel in a dataset, improved cache
> performance from 59.6s write, 57.1s read to 35.2s write, 1.6s read (done)
> * Two tests still failing unpredictably due to garbage collection reliance
> (I think). As soon as these are fixed, the update SCIFIOCell/MemoryService
> functionality will be released in SCIFIO 0.7.4.
> * Integration to KNIME started but not finished (easy task, thanks Mark!)
>
>
>
 Just wanted to let people know that this functionality is now available in
the SCIFIO-0.7.4 release <https://github.com/scifio/scifio/releases>.

 I ended up focusing the changes to the PhantomReference refactoring and
performance improvements. If anyone's interested why, I wrote up an
explanation on the original ticket<https://github.com/scifio/scifio/issues/69>
.

 To elaborate on the write/read test I was using as a metric, Christian
Dietz provided a ~90MB dataset and made a simple program that iterates over
each pixel, writing it to a random value, then iterates over the dataset
again reading each pixel's value.

 With the 0.7.3 cell cache, the writing operation actually took over 100s
when limiting the JVM to 50MB ram. Reading the data back took at least 50s.

 By adjusting when certain operations run, moving a lot of cleanup to
separate threads, and avoiding unnecessary disk checks, as of 0.7.4 the
same operation now takes ~16s to write and ~2s to read on the same machine.
And of course the code is significantly improved, as we're no longer
relying on overridden finalize methods or System.gc calls.

 Anyway, I'm sure there will still be more improvements to make but I think
the cell cache was significantly enhanced. Thanks to Christian for
providing the motivation + PhantomReference info!

- Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20131211/f9fc0198/attachment.html>