[ImageJ-devel] [ome-devel] strategy and 6D data

Thu Dec 6 14:39:10 CST 2012

Hi Curtis, Ian et al.

Thanks for raising this issue again.  We've certainly discussed it several times, on these lists and in our regular weekly meetings.  Fundamentally, Curtis is correct, we definitely need to address the issue of N-dimensional data thoroughly and completely.  This was the basis for the various studies we did of this problem in 2011/2012, the final summaries of which are:

http://www.openmicroscopy.org/site/support/file-formats/working-with-ome-xml/6d-7d-and-8d-storage

http://trac.openmicroscopy.org/ome/wiki/WorkPlan/NDimensionalData
(deprecated page)

Curtis stated he didn't like the Modulo solution, and frankly, from a modelling point of view, we're not fond of that either.   We fully appreciate the limitations of the approach.  As detailed above, however, the approach does allow us to move forward, begin to explore these different data types, and start to work with real applications in a broad number of domains, and allow us to develop knowledge and experience with new data types and their usage.  That provides the foundation for a significant re-design of essentially all of OME's models and software.  Given the number of people who depend on these resources, we need to initiate large breaking changes when we are sure of what we are doing, and have good example datasets from as wide variety domains as possible.  For this reason, we previously decided to provide Ian and other with this type of multi-dimensional data with support for a Modulo-based approach, help get their analysis integrated with OME's tool, and then get, for example, global analysis on FLIM data running in production.

This does put off the inevitable, and we fully accept the additional burden this creates.  However, we are committed to making significant changes to OME's APIs through a series of defined, known steps, where people working with OME-TIFF, Bio-Formats and OMERO are provided explicit, well-documented, working specifications and examples, are allowed to transition to a new framework.

Frankly, the other issue is that there are a large number of critical things to achieve in OME.  For a number of years, many institutions have insisted that we provide support for direct access to data from the filesystem, improved support for analysis, export, etc.  To do each of these correctly takes significant work and concentration by the whole team to deliver functionality across the whole stack.  That means we have to make strategic decisions, and choose our priorities.  We don't have infinite resources, but what we do build and release has to be of high quality and usability.

Finally, discussions of N-dim will certainly continue to appear on the weekly team meeting agenda (https://www.openmicroscopy.org/site/community/minutes/conference-calls/2012).  Those meetings can also be used for more detailed, focussed conversations.

Thanks for the thoughtful comments;  look forward to more.

Cheers,

Jason

From: ome-devel-bounces at lists.openmicroscopy.org.uk [mailto:ome-devel-bounces at lists.openmicroscopy.org.uk] On Behalf Of Curtis Rueden
Sent: Wednesday, December 05, 2012 3:16 PM
To: Munro, Ian
Cc: ome-devel Development; ImageJ Developers
Subject: Re: [ome-devel] strategy and 6D data

Hi Ian,

> Clearly you have a better feel for the downsides of the ModuloAlong?
> solution than we do so can I ask how you intend to store your data in
> OMERO?

When I wrote about my concerns with the Modulo approach last year, my goal was to spark a technical discussion that could lead to a truly N-dimensional solution in the OME schema. But that did not happen-the rest of the team did not share my concerns-so the Modulo approach is AFAIK the only way to store lifetime data within OMERO in the present day.

So it seems to me your best bet is to use Modulo, since that is what the OMERO system provides, and hope that it stays well-supported in the long term. The other option-proposing and implementing a truly N-dimensional data model-would be something that only the core OMERO developers realistically have the capability to do, given the complexity of the OMERO software stack.

As for how LOCI will store spectral lifetime data in OMERO: we aren't planning on doing so until the OME data model fully supports it. In the meantime, it will be enough for us for Bio-Formats and ImageJ2 to support it, since they are already N-dimensional.

Regards,
Curtis

On Fri, Nov 30, 2012 at 11:44 AM, Munro, Ian <i.munro at imperial.ac.uk<mailto:i.munro at imperial.ac.uk>> wrote:
Hi Curtis

Thans for the feedback. It's much appreciated.

Actually I oversimplified as our code is a Matlab GUI wrapping cross-platform c++ fitting code so bringing this together with slim may be relatively easy as the 2 GUIs seem to have a lot of common features.

In the shorter term I think we need to use the same strategy to store data in OMERO.

Clearly you have a better feel for the downsides of the ModuloAlong? solution than we do so can I ask how you intend to store your data in OMERO?

Regards

Ian

On 28 Nov 2012, at 20:10, Curtis Rueden wrote:

Hi Ian,

Thanks for this update; very informative. It looks like you have done a lot to bring FLIM processing into OMERO, which is great.

I want to draw your attention to some very similar work we are doing within ImageJ:
    http://loci.wisc.edu/software/slim-plugin

The SLIM Plugin is an ImageJ plugin (written in Java, of course), which uses the SLIM-curve library (written in cross-platform C) to perform the curve fitting (of TCSPC data).

> The Imperial/Photonics Group's main FLIM analysis software is
> internally named "GlobalProcessing". It is written in MATLAB and
> provides state of the art fitting of time domain data for both
> laser-scanning time-correlated single photon (TCSPC) and wide-field
> time-gated FLIM images.

Perhaps GlobalProcessing and SLIM-curve could join forces? It is much easier to access C code from Java than MATLAB code, but there are clearly features in GlobalProcessing missing from SLIM-curve (e.g., wide-field), and vice versa.

> 1) We intend to standardise on encoding FLIM data using ModuloAlongT.

Personally, I dislike ModuloAlongT, especially as a long-term solution. There are certain dimension orders that are simply impossible using it (e.g., XYbCZT, where "b" is lifetime bins, and T is actual time points). I would much favor us developing a true N-dimensional OME data model, and using that, moving forward.

> 2) Where should the functionality currently in IC-importer fit into
> the OMERO eco-system?

I agree that this functionality would work well as a Bio-Formats reader.

Regards,
Curtis

On Wed, Nov 28, 2012 at 5:11 AM, Munro, Ian <i.munro at imperial.ac.uk<mailto:i.munro at imperial.ac.uk>> wrote:
Hello all

Please find attached a document describing where the work of the Imperial satellite currently stands.
We've reached a point where our group is making decisions, regarding the handling of data with a 6th dimension,
that may well have future  implications for  the OMERO clients or for other groups working with 5+ dimensions.

As  a result we're looking for any feedback/improvements that anyone can offer on our current approach
In particular the 3 areas  highlighted on the second page of the document.

Thanks in advance for your time.

Ian

Ian

_______________________________________________
ome-devel mailing list
ome-devel at lists.openmicroscopy.org.uk<mailto:ome-devel at lists.openmicroscopy.org.uk>
http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-devel

The University of Dundee is a registered Scottish Charity, No: SC015096
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20121206/4187fed8/attachment.html>