[ImageJ-devel] EuclideanSpace and imagej.display.Display
Lee Kamentsky
leek at broadinstitute.org
Thu Jan 19 14:47:36 CST 2012
Pretty impressive progress - I'm hoping that it's easier to make coding
choices with a more comprehensive model of the N-D space. Generally,
thanks for following through. The only specific point I have is that you
are probably right about the difficulties posed by the nature of the
spectral dimension. I can't think of a clean way to handle both kinds of
spectral "positions" in a unified way, so perhaps indexes and reference
to the channel labels is the best that can be done.
Thanks,
--Lee
On 1/19/2012 3:33 PM, Curtis Rueden wrote:
> Hi Lee,
>
> I'm running into a number of problems with overlays that are
> caused by the fact that the EuclideanSpace is now defined on the
> Dataset and not on the Display used to composite datasets and
> overlays. There are more than a few issues:
>
>
> Over the past few weeks, and during the Dresden hackathon, I finally
> found time to pursue a lot of these issues, and have made changes to
> the codebase to improve the data/display hierarchy.
>
> * The ImageDisplay interface now implements CalibratedInterval,
> which extends not only EuclideanSpace but also Interval. Further,
> unlike Data objects, ImageDisplay now also implements
> PositionableByAxis (which extends Localizable and Positionable).
> So an ImageDisplay always has a current position in the
> N-dimensional structure, and can report what that is.
> * Data objects (Datasets and Overlays) now implement
> CalibratedInterval, but not PositionableByAxis, since it makes no
> sense to ask a Data object what its current position is.
> * However, the DataView object, which wraps a Data and provides
> visualization-specific metadata about how that Data is currently
> being visualized, *does* implement PositionableByAxis.
> * An ImageDisplay is still a collection of DataViews as before, but
> uses the CombinedInterval class to combine the N-dimensional
> spaces of its constituent views into a single aggregate space.
>
> Hopefully these changes are along the lines of what we discussed when
> I visited Broad all those months ago.
>
> * Trac issue ImageJ 558 - the user takes a Z-stack, multichannel
> image and thresholds it to get a BinaryMaskOverlay and
> associated ROI. The ROI is properly defined in a 4-dimensional
> space. However, when displayed, a single X/Y plane should be
> sampled and displayed, but there is no mechanism to retrieve
> the plane being displayed or the color display mode. The code
> iterates through the entire stack and that's what causes it to
> update slowly.
>
>
> I haven't had time to examine this issue in much detail, but the fix
> you implemented long ago seems fine for the time being.
>
> * Datasets have axes which capture the metadata of what a
> particular dimension means. Overlays don't have that.
>
>
> As mentioned above, things have now been unified so that all Data
> objects (including both Datasets and Overlays) are N-dimensional with
> axes, by extending the CalibratedInterval interface, which in turn
> extends Interval and CalibratedSpace (which extends EuclideanSpace).
>
> * Channels are really categorical, not ordinal - there's no
> reason the red channel is zero, the green is one and the blue
> is two and, with a channel-stack, the DAPI channel in one
> image might be the PI channel in a second. You can properly
> relate one channel to the other through metadata, but overlays
> don't have that.
>
>
> I think channels are sometimes categorical, and sometimes ordinal. For
> example, a hyperspectral dataset that provides 32 channels ranging
> from 400 nm through 550 nm sampled every 5 nm would be a continuous
> domain, similar to other dimensions. But if you have two
> channels---one transmitted channel captured using a grayscale camera,
> and another channel detected from fluorescence caused by laser
> scanning---those are definitely categorical.
>
> Still, ultimately, they become ordinal in that you must provide an
> index for each channel. That doesn't mean you should assume continuity
> though, of course.
>
> * You might want to composite datasets - take two images and
> place them adjacently. What happens if they have different
> axes? What happens if they have different channel layouts?
>
>
> In general, some math needs to happen. The new CombinedInterval
> <http://fiji.sc/cgi-bin/gitweb.cgi?p=imagej2/.git;a=blob;f=core/data/src/main/java/imagej/data/CombinedInterval.java;h=24aec4a1305a28aaead11709733ef96e677a1672;hb=refs/heads/svn/trunk>
> class handles the union of multiple CalibratedInterval objects, fairly
> naively at the moment, but with the potential to improve the logic as
> you suggest in the future. We are also planning to tap into Unidata's
> Ucar Units <https://github.com/ctrueden/ucar-units> project to assist
> with this.
>
> So, what I'm thinking is the following:
>
> * A display represents the EuclideanSpace that's used to
> composite overlays and datasets. Think of the display as a
> big N-dimensional container and the overlays and datasets
> float inside that container.
> o The display has a set of axes that are the union of all of
> the axes that appear in the view data objects.
> o There are two behaviors for a view object that does not
> have an axis in the space. Perhaps the user selects the
> behavior?:
> + The value in space outside of a single plane defined
> by a coordinate location along the axis is nan or null.
> + The value in space along the missing axis is the same
> at all locations along that axis.
>
>
> Yes, this is as we discussed at Broad a few months back. It is really
> working now! :-)
>
> Currently, if a constituent DataView lacks an axis from the aggregate
> space, the value for that axis can be specified explicitly, and
> defaults to 0 otherwise. So, for example, all Overlays currently have
> the X and Y axes, and no other axes. Embedding a rectangle overlay in
> a 4D dataset (with, say, XYZT) causes that rectangle to be visible at
> Z=0, T=0, and no other Z and T positions, unless the
> DataView.setPosition(long, AxisType) method is used to override a
> different position for that dimension.
>
> In the future, we could enable the user to specify an alternative
> behavior as you suggest, such that the overlay becomes visible at
> *all* positions of that axis. But for now it is always a constant value.
>
> * A display window holds the information about what plane is
> being displayed and how the channels are composited.
> o The views are asked to display according to the plane
> information. A plane is an interval where the min=max for
> all dimensions but X and Y. The views could construct
> appropriate projectors based on the interval. The display
> window should also control the channel compositing
> information and the views should reference the display
> window's compositing information instead of maintaining
> their own copies.
>
>
> Right, as things stand now, the ImageDisplay tracks its current
> position (i.e., it implements PositionableByAxis) and the DisplayPanel
> provides the GUI that actually presents the information onscreen.
> However, one vital remaining task is to finish decoupling these two
> classes. Currently there is still some unfortunate coupling between
> ImageDisplay, ImageCanvas, DisplayPanel and DisplayWindow, which needs
> to change. The ImageDisplay and DataViews should be completely
> UI-agnostic, with the active user interface components subscribing to
> display events and updating themselves accordingly. This will help
> eliminate some problematic sections of code, particularly
> SwingDatasetView and SwingOverlayView, which are discouraged to use in
> ImageJ plugins due to their Swing-specific nature.
>
> * Perhaps there is a SpatialDataObject or perhaps the DataObject
> is always spatial. In any case, the axis information and
> channel metadata should be pushed up the inheritance hierarchy
> so that Overlay has a first-class representation of that and
> operations that generate overlays from datasets copy the
> information to the overlays they create.
>
>
> We went the route of "data objects are always spatial." And yeah, a
> lot of functionality was pushed up into AbstractData, AbstractDataView
> and AbstractImageDisplay. Hopefully this will reduce the amount of
> dataset- and overlay-specific code throughout the system.
>
> * There is a spectral dimension, but the channel axis doesn't
> capture that. Channels are categorical and should have names
> so they can be matched in the same way that axes are matched.
>
>
> It is a good point that we should add some support for categorical
> dimensions in general. It is not enough to have calibrations; it
> should be possible to label each axis position individually. This is
> useful for a variety of reasons. We will need to revisit this idea
> later, when time permits.
>
> It's some work to refactor these things, but as the code base
> grows and becomes more entrenched it will become increasingly
> difficult to refactor, so we should do it sooner rather than
> later. I don't think this is much coding at this point and if you
> look at the code, there are places where this organization
> clarifies some things.
>
>
> The coding work has been nontrivial, but certainly doable. There is
> more left to do as well, but I think we are in reasonable shape.
>
> Thanks very much for your comments! I really appreciate how much
> thought and effort you have put into the ImageJ design over the past year.
>
> Regards,
> Curtis
>
>
> On Fri, Jun 3, 2011 at 1:44 PM, Lee Kamentsky <leek at broadinstitute.org
> <mailto:leek at broadinstitute.org>> wrote:
>
> I'm running into a number of problems with overlays that are
> caused by the fact that the EuclideanSpace is now defined on the
> Dataset and not on the Display used to composite datasets and
> overlays. There are more than a few issues:
>
> * Trac issue ImageJ 558 - the user takes a Z-stack, multichannel
> image and thresholds it to get a BinaryMaskOverlay and
> associated ROI. The ROI is properly defined in a 4-dimensional
> space. However, when displayed, a single X/Y plane should be
> sampled and displayed, but there is no mechanism to retrieve
> the plane being displayed or the color display mode. The code
> iterates through the entire stack and that's what causes it to
> update slowly.
>
> * Datasets have axes which capture the metadata of what a
> particular dimension means. Overlays don't have that.
> * Channels are really categorical, not ordinal - there's no
> reason the red channel is zero, the green is one and the blue
> is two and, with a channel-stack, the DAPI channel in one
> image might be the PI channel in a second. You can properly
> relate one channel to the other through metadata, but overlays
> don't have that.
> * You might want to composite datasets - take two images and
> place them adjacently. What happens if they have different
> axes? What happens if they have different channel layouts?
>
> So, what I'm thinking is the following:
>
> * A display represents the EuclideanSpace that's used to
> composite overlays and datasets. Think of the display as a
> big N-dimensional container and the overlays and datasets
> float inside that container.
> o The display has a set of axes that are the union of all of
> the axes that appear in the view data objects.
> o There are two behaviors for a view object that does not
> have an axis in the space. Perhaps the user selects the
> behavior?:
> + The value in space outside of a single plane defined
> by a coordinate location along the axis is nan or null.
> + The value in space along the missing axis is the same
> at all locations along that axis.
> * A display window holds the information about what plane is
> being displayed and how the channels are composited.
> o The views are asked to display according to the plane
> information. A plane is an interval where the min=max for
> all dimensions but X and Y. The views could construct
> appropriate projectors based on the interval. The display
> window should also control the channel compositing
> information and the views should reference the display
> window's compositing information instead of maintaining
> their own copies.
> * Perhaps there is a SpatialDataObject or perhaps the DataObject
> is always spatial. In any case, the axis information and
> channel metadata should be pushed up the inheritance hierarchy
> so that Overlay has a first-class representation of that and
> operations that generate overlays from datasets copy the
> information to the overlays they create.
> * There is a spectral dimension, but the channel axis doesn't
> capture that. Channels are categorical and should have names
> so they can be matched in the same way that axes are matched.
>
> It's some work to refactor these things, but as the code base
> grows and becomes more entrenched it will become increasingly
> difficult to refactor, so we should do it sooner rather than
> later. I don't think this is much coding at this point and if you
> look at the code, there are places where this organization
> clarifies some things.
>
> --Lee
>
>
>
>
> _______________________________________________
> ImageJ-devel mailing list
> ImageJ-devel at imagej.net
> http://imagej.net/mailman/listinfo/imagej-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20120119/f5566a30/attachment.html>
More information about the ImageJ-devel
mailing list