[ImageJ-devel] Regarding Target Java 1.5 for imglib2 core

Sat Mar 31 22:29:19 CDT 2012

Hi Tobias & everyone,

This has all been happening a bit too fast for me and I'm kind of lost
> right now about what exactly is going on here.  So could someone please
> explain what happened?
>

Yes, I am sorry I could not send a mail explaining the situation earlier,
but we were fully occupied just fixing all the build problems that had
accumulated over the past few days.

Your confusion is understandable, because there were several unrelated
issues that required solving.

*== 1) TrakEM2 precompiled was out of date ==
*
Probably the least significant problem was that building Fiji without
having modules/TrakEM2 initialized caused compile errors, due to the
precompiled TrakEM2 jar not matching the rest of the codebase. Fixing this
required rebuilding TrakEM2 from source and committing the new jar (
https://github.com/fiji/fiji/commit/34ae3188). Doing so was complicated
only by the fact that the Fiji build system needed a little TLC first (see
#2 next).

*== 2) Bugs in the Fiji build system and ImageJ launcher ==
*
Johannes has been working very hard on the ImageJ launcher, in preparation
for inclusion with ImageJ2. Initially, much of that work was done in
fiji.git on ImageJ.c. The code has been steadily improving, but as with all
such changes, new bugs can be introduced or brought to light as a result:

A) Dscho & I tracked down a bug with the launcher on OS X, noticed by
Steffi: https://github.com/fiji/fiji/commit/a7d90f38

B) Dscho fixed another bug specific to OS X:
https://github.com/fiji/fiji/commit/25e58077

C) Steffi fixed a problem with Fiji build on Windows:
https://github.com/fiji/fiji/commit/a85acab1

Once those problems were fixed, we were able to finish solving the tougher
issues, below.

*== 3) The OpenJDK6 Javac bug ==
*
The OpenJDK6 problem has nothing to do with Java5. There are several usages
of generics in ImgLib that Fiji's OpenJDK6 compiler failed to handle
properly. To resolve it, Johannes upgraded Fiji's version of OpenJDK6 to
the latest available (https://github.com/fiji/fiji/commit/402dce11).
However, this latest version introduced some new problems when compiling
SPIM registration, so Johannes partially reverted it (
https://github.com/fiji/fiji/commit/82bd2950). But still a few bugs
remained when compiling ImgLib. To work around them, I introduced the ugly
"HACK" changes (https://github.com/imagej/imglib/commit/5748a1be).

I know these HACKs are unpleasant, but the fact is that we are targeting
three different compilers: Oracle's Java6, OpenJDK6, and Eclipse's JDT
compiler. If ImgLib fails to compile with any of them, then our current
workflows have problems.

In the future, as OpenJDK7 becomes the gold standard for compilation, and
Java7 is more easily available for all major platforms, the dichotomy
between compilation with Fiji build (which uses OpenJDK6) and Maven on the
command line (which uses your installed Java, often Oracle's Java6,
depending on your platform) should disappear. We will likely need to remain
cautious indefinitely with respect to Eclipse's JDT compiler, though.

> Is there anything I have to be careful about now, so that I don't break
> anything?

As always, just because something works in Eclipse does not mean it will
work with Oracle's Java6, or with OpenJDK6. In a perfect world, we would
all test our changes with all three compilers (on all three OSes too!), but
personally I think asking everyone to always do that is unrealistic. That's
what Jenkins is for. Just don't be surprised if sometimes the build breaks
even though Eclipse had no problems. And if you see that one of your
changes broke the build, it is extremely helpful if you can find a
fix/workaround, rather than waiting for Dscho or me to fix it. J-Y asked in
another thread ("Fiji does not compile anymore under Mac and Windows")
about the best way to test compilation in Fiji build, etc.—on Monday I will
reply there with more details, and create a new page on the Fiji wiki about
it.

As a side note, it can happen the other way too: yesterday Johannes merged
the ImageJ launcher to ImageJ2, and it is all working fine from the command
line, but not from Eclipse. ;-) The reality is that it takes work to keep
everything running smoothly across the different paradigms.

*== 4) The Java5 compatibility snafu ==
*
Recently, I asked whether we should migrate the imglib2-io project into
SCIFIO (a.k.a. Bio-Formats), and people responded favorably. So Mark Hiner
& I did it on a topic branch, and when Melissa Linkert visited last week,
we merged the changes (https://github.com/openmicroscopy/bioformats/pull/39,
). However, we forgot one important detail: SCIFIO still uses a Java5
compiler to build. (Actually, the entire OME software stack does.) So the
OME Jenkins failed to build SCIFIO with the new ImgLib I/O classes, and we
scrambled to find a solution. We created a temporary Java5 version of
imglib2.jar that we could use, and fixed a couple of other oversights (
https://github.com/openmicroscopy/bioformats/pull/45).

That fixed the build problems, but Dscho & I wanted to find a more
permanent solution to the build compatibility problem between SCIFIO and
ImgLib2. I had an idea for a compromise—target Java 1.5 for the core
imglib2 project—which I implemented on Thursday (
https://github.com/imagej/imglib/commit/c66f26e6). I did not know that
Eclipse would misbehave when importing the project until afterwards—but
that problem is now fixed (see below).

As an aside, I would like to point out that compiling ImgLib to target
Java5 is nothing new. Fiji build has done this from the beginning, and not
just for imglib2 core, but for *all* of ImgLib (
https://github.com/fiji/fiji/commit/702c921b). You'll notice the line
"javaVersion=1.5" which tells Fiji build to compile the code to class
version 49.0 (Java 1.5). So those of you building ImgLib via Fiji build are
already using class version 49.0.

Are there plans to figure out the performance implications of using the old
> class format?

As Dscho says, we have no plans to test it ourselves. However, it would be
very easy to run the ImgLib2 benchmarks (
https://github.com/imagej/imglib/blob/master/imglib2/ij/src/test/scripts/benchmark.sh)
with a Java5 jar vs. a Java6 one.

I have absolutely no idea if performance is impacted by using a
Java5-compatible jar. Personally, I doubt it. However, Johannes mentioned
offhand that it might be a possibility, and I agreed, so I mentioned it in
the commit message. But it is pure speculation. ImgLib has proven to be
extremely sensitive to small changes (because the JIT is sensitive), so we
thought it worth mentioning. But I did not mean to suggest that such
performance problems were necessarily likely—merely conceivable.

Can I (locally) just remove the maven-compiler-plugin section from the pom
> and have everything working as before?
>

The question is now moot, because in a commit earlier today, I disabled the
Java5 targeting by default (https://github.com/imagej/imglib/commit/b67f246f).
It must now be enabled explicitly using a profile. Things should now work
in Eclipse again, exactly as before. Jenkins now turns on the Java5
targeting when it builds, so that the imglib2.jar deployed to the Maven
repository (which SCIFIO builds against) will remain Java5 compatible. But
those of us using Eclipse will not suffer any inconvenience.

Regarding making local changes to a POM: in general I would discourage it.
If you find yourself needing to do so, we should discuss what problem you
are trying to solve, and what an easier and more effective solution might
be.

*== 5) Continuing changes to ImgLib2 I/O classes ==
*
We have been testing ImageJ2's I/O plugins, which use the ImgLib2 ImgOpener
and ImgSaver classes. The ImgSaver in particular is very new, and still
under active development. Unfortunately, the turnaround time to merge
changes into SCIFIO, and make them available from ImageJ's Maven
repository, is days to weeks. So while there is another pull request
pending to update the ImgLib I/O code in SCIFIO (
https://github.com/openmicroscopy/bioformats/pull/49), for the impending
ImageJ2 beta we needed a solution more rapidly.

So what we did is to commit the latest version back to the imglib2-io
project of imglib.git as well. Now, the code lives in both projects, with
two different package names (net.imglib2.io, and ome.scifio.img). At the
moment, the code is in sync. The long term plan is still to remove the
imglib2-io project in favor of it being part of SCIFIO, but it will have to
wait until we can achieve the speed of integrated multi-project development
we are accustomed to with Fiji. The good news is that there is now a
working ImgSaver available for use (feedback welcome!).

Hopefully this clarifies the recent changes to Fiji and ImgLib. Please let
us know if anything else is unclear.

Regards,
Curtis

On Sat, Mar 31, 2012 at 5:10 AM, Tobias Pietzsch <pietzsch at mpi-cbg.de>wrote:

> Hi guys,
>
> This has all been happening a bit too fast for me and I'm kind of lost
> right now about what exactly is going on here.  So could someone please
> explain what happened?
>
> There have been ugly "HACK"s introduced like the following:
>
> -final T buffer = Util.getTypeFromInterval( fftImage ).createVariable();
> +// HACK: Explicit assignment is needed for OpenJDK javac.
> +final T fftImageType = Util.getTypeFromInterval( fftImage );
> +final T buffer = fftImageType.createVariable();
>
> It seems, that this has been limited (fortunately!!!) to
> imglib2-algorithms-gpl.  But why? Surely, there are similar constructs in
> other parts of imglib. Why are they not affected?
>
> Are there plans to figure out the performance implications of using the
> old class format?  Is there anything I have to be careful about now, so
> that I don't break anything?  Can I (locally) just remove the
> maven-compiler-plugin section from the pom and have everything working as
> before?
>
> And honestly, I don't like that this was done without any advance warning
> or discussion. This seems to be a big thing with potential
> performance implications.
>
> best regards,
> Tobias
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20120331/38b5070b/attachment-0001.html>