[ImageJ-devel] [fiji-devel] Re: Regarding Target Java 1.5 for imglib2 core

Mon Apr 2 04:56:39 CDT 2012

Hi Curtis,

I second Tobias in thanking very much for this comprehensive explanation
(and the preceding confusion).  To clarify our situation:

We had definitely noticed a massive amount of Jenkins build failure
messages but since this occurred in the context of Johannes' announced
changes of the system to better catch ImageJ1 regressions, we took this
for the usual construction site noise.

We always test our changes by building Fiji using Fiji's Build.sh and
were not aware that Jenkins uses even another compiler.  That way, I
noticed Fiji not building on my desktop computer where the repository is
a messy construction site as well.  So, in the assumption that this may
be my fault having something not checked out in the proper way, I tested
it on my laptop in a more cleaned up repository, and believe it or not,
everything built fine---until the very end.  Conclusion: no need to
move, they will get their stuff done and I'll fix my desktop later.
Both computers are Ubuntu, 10.4 and 11.10, no $JAVA_HOME set, imglib and
TrakEM2 submodules checked out.

Best regards and again thanks for the explanation,
Stephan

On Mon, 2012-04-02 at 11:27 +0200, Tobias Pietzsch wrote: 
> Hi Curtis,
> 
> Thanks a lot for the explanations. That was really helpful.
> 
> I thought it was all somehow caused by targeting java 1.5.
> I did not know that Fiji used the 49.0 class version anyway.
> So I'm much more comfortable with that now. I will still run
> the benchmark to compare the class versions, just to know for
> sure.
> 
> best regards,
> Tobias
> 
> On 04/01/2012 05:29 AM, Curtis Rueden wrote:
> > Hi Tobias & everyone,
> >
> >     This has all been happening a bit too fast for me and I'm kind of lost
> >     right now about what exactly is going on here.  So could someone please
> >     explain what happened?
> >
> >
> > Yes, I am sorry I could not send a mail explaining the situation
> > earlier, but we were fully occupied just fixing all the build problems
> > that had accumulated over the past few days.
> >
> > Your confusion is understandable, because there were several unrelated
> > issues that required solving.
> >
> >
> > *== 1) TrakEM2 precompiled was out of date ==
> > *
> > Probably the least significant problem was that building Fiji without
> > having modules/TrakEM2 initialized caused compile errors, due to the
> > precompiled TrakEM2 jar not matching the rest of the codebase. Fixing
> > this required rebuilding TrakEM2 from source and committing the new jar
> > (https://github.com/fiji/fiji/commit/34ae3188). Doing so was complicated
> > only by the fact that the Fiji build system needed a little TLC first
> > (see #2 next).
> >
> >
> > *== 2) Bugs in the Fiji build system and ImageJ launcher ==
> > *
> > Johannes has been working very hard on the ImageJ launcher, in
> > preparation for inclusion with ImageJ2. Initially, much of that work was
> > done in fiji.git on ImageJ.c. The code has been steadily improving, but
> > as with all such changes, new bugs can be introduced or brought to light
> > as a result:
> >
> > A) Dscho & I tracked down a bug with the launcher on OS X, noticed by
> > Steffi: https://github.com/fiji/fiji/commit/a7d90f38
> >
> > B) Dscho fixed another bug specific to OS X:
> > https://github.com/fiji/fiji/commit/25e58077
> >
> > C) Steffi fixed a problem with Fiji build on Windows:
> > https://github.com/fiji/fiji/commit/a85acab1
> >
> > Once those problems were fixed, we were able to finish solving the
> > tougher issues, below.
> >
> >
> > *== 3) The OpenJDK6 Javac bug ==
> > *
> > The OpenJDK6 problem has nothing to do with Java5. There are several
> > usages of generics in ImgLib that Fiji's OpenJDK6 compiler failed to
> > handle properly. To resolve it, Johannes upgraded Fiji's version of
> > OpenJDK6 to the latest available
> > (https://github.com/fiji/fiji/commit/402dce11). However, this latest
> > version introduced some new problems when compiling SPIM registration,
> > so Johannes partially reverted it
> > (https://github.com/fiji/fiji/commit/82bd2950). But still a few bugs
> > remained when compiling ImgLib. To work around them, I introduced the
> > ugly "HACK" changes (https://github.com/imagej/imglib/commit/5748a1be).
> >
> > I know these HACKs are unpleasant, but the fact is that we are targeting
> > three different compilers: Oracle's Java6, OpenJDK6, and Eclipse's JDT
> > compiler. If ImgLib fails to compile with any of them, then our current
> > workflows have problems.
> >
> > In the future, as OpenJDK7 becomes the gold standard for compilation,
> > and Java7 is more easily available for all major platforms, the
> > dichotomy between compilation with Fiji build (which uses OpenJDK6) and
> > Maven on the command line (which uses your installed Java, often
> > Oracle's Java6, depending on your platform) should disappear. We will
> > likely need to remain cautious indefinitely with respect to Eclipse's
> > JDT compiler, though.
> >
> >     Is there anything I have to be careful about now, so that I don't
> >     break anything?
> >
> >
> > As always, just because something works in Eclipse does not mean it will
> > work with Oracle's Java6, or with OpenJDK6. In a perfect world, we would
> > all test our changes with all three compilers (on all three OSes too!),
> > but personally I think asking everyone to always do that is unrealistic.
> > That's what Jenkins is for. Just don't be surprised if sometimes the
> > build breaks even though Eclipse had no problems. And if you see that
> > one of your changes broke the build, it is extremely helpful if you can
> > find a fix/workaround, rather than waiting for Dscho or me to fix it.
> > J-Y asked in another thread ("Fiji does not compile anymore under Mac
> > and Windows") about the best way to test compilation in Fiji build,
> > etc.—on Monday I will reply there with more details, and create a new
> > page on the Fiji wiki about it.
> >
> > As a side note, it can happen the other way too: yesterday Johannes
> > merged the ImageJ launcher to ImageJ2, and it is all working fine from
> > the command line, but not from Eclipse. ;-) The reality is that it takes
> > work to keep everything running smoothly across the different paradigms.
> >
> >
> > *== 4) The Java5 compatibility snafu ==
> > *
> > Recently, I asked whether we should migrate the imglib2-io project into
> > SCIFIO (a.k.a. Bio-Formats), and people responded favorably. So Mark
> > Hiner & I did it on a topic branch, and when Melissa Linkert visited
> > last week, we merged the changes
> > (https://github.com/openmicroscopy/bioformats/pull/39, ). However, we
> > forgot one important detail: SCIFIO still uses a Java5 compiler to
> > build. (Actually, the entire OME software stack does.) So the OME
> > Jenkins failed to build SCIFIO with the new ImgLib I/O classes, and we
> > scrambled to find a solution. We created a temporary Java5 version of
> > imglib2.jar that we could use, and fixed a couple of other oversights
> > (https://github.com/openmicroscopy/bioformats/pull/45).
> >
> > That fixed the build problems, but Dscho & I wanted to find a more
> > permanent solution to the build compatibility problem between SCIFIO and
> > ImgLib2. I had an idea for a compromise—target Java 1.5 for the core
> > imglib2 project—which I implemented on Thursday
> > (https://github.com/imagej/imglib/commit/c66f26e6). I did not know that
> > Eclipse would misbehave when importing the project until afterwards—but
> > that problem is now fixed (see below).
> >
> > As an aside, I would like to point out that compiling ImgLib to target
> > Java5 is nothing new. Fiji build has done this from the beginning, and
> > not just for imglib2 core, but for *all* of ImgLib
> > (https://github.com/fiji/fiji/commit/702c921b). You'll notice the line
> > "javaVersion=1.5" which tells Fiji build to compile the code to class
> > version 49.0 (Java 1.5). So those of you building ImgLib via Fiji build
> > are already using class version 49.0.
> >
> >
> >     Are there plans to figure out the performance implications of using
> >     the old class format?
> >
> >
> > As Dscho says, we have no plans to test it ourselves. However, it would
> > be very easy to run the ImgLib2 benchmarks
> > (https://github.com/imagej/imglib/blob/master/imglib2/ij/src/test/scripts/benchmark.sh)
> > with a Java5 jar vs. a Java6 one.
> >
> > I have absolutely no idea if performance is impacted by using a
> > Java5-compatible jar. Personally, I doubt it. However, Johannes
> > mentioned offhand that it might be a possibility, and I agreed, so I
> > mentioned it in the commit message. But it is pure speculation. ImgLib
> > has proven to be extremely sensitive to small changes (because the JIT
> > is sensitive), so we thought it worth mentioning. But I did not mean to
> > suggest that such performance problems were necessarily likely—merely
> > conceivable.
> >
> >
> >     Can I (locally) just remove the maven-compiler-plugin section from
> >     the pom and have everything working as before?
> >
> >
> > The question is now moot, because in a commit earlier today, I disabled
> > the Java5 targeting by default
> > (https://github.com/imagej/imglib/commit/b67f246f). It must now be
> > enabled explicitly using a profile. Things should now work in Eclipse
> > again, exactly as before. Jenkins now turns on the Java5 targeting when
> > it builds, so that the imglib2.jar deployed to the Maven repository
> > (which SCIFIO builds against) will remain Java5 compatible. But those of
> > us using Eclipse will not suffer any inconvenience.
> >
> > Regarding making local changes to a POM: in general I would discourage
> > it. If you find yourself needing to do so, we should discuss what
> > problem you are trying to solve, and what an easier and more effective
> > solution might be.
> >
> >
> > *== 5) Continuing changes to ImgLib2 I/O classes ==
> > *
> > We have been testing ImageJ2's I/O plugins, which use the ImgLib2
> > ImgOpener and ImgSaver classes. The ImgSaver in particular is very new,
> > and still under active development. Unfortunately, the turnaround time
> > to merge changes into SCIFIO, and make them available from ImageJ's
> > Maven repository, is days to weeks. So while there is another pull
> > request pending to update the ImgLib I/O code in SCIFIO
> > (https://github.com/openmicroscopy/bioformats/pull/49), for the
> > impending ImageJ2 beta we needed a solution more rapidly.
> >
> > So what we did is to commit the latest version back to the imglib2-io
> > project of imglib.git as well. Now, the code lives in both projects,
> > with two different package names (net.imglib2.io
> > <http://net.imglib2.io>, and ome.scifio.img). At the moment, the code is
> > in sync. The long term plan is still to remove the imglib2-io project in
> > favor of it being part of SCIFIO, but it will have to wait until we can
> > achieve the speed of integrated multi-project development we are
> > accustomed to with Fiji. The good news is that there is now a working
> > ImgSaver available for use (feedback welcome!).
> >
> >
> > Hopefully this clarifies the recent changes to Fiji and ImgLib. Please
> > let us know if anything else is unclear.
> >
> > Regards,
> > Curtis
> >
> >
> > On Sat, Mar 31, 2012 at 5:10 AM, Tobias Pietzsch <pietzsch at mpi-cbg.de
> > <mailto:pietzsch at mpi-cbg.de>> wrote:
> >
> >     Hi guys,
> >
> >     This has all been happening a bit too fast for me and I'm kind of lost
> >     right now about what exactly is going on here.  So could someone please
> >     explain what happened?
> >
> >     There have been ugly "HACK"s introduced like the following:
> >
> >     -final T buffer = Util.getTypeFromInterval( fftImage ).createVariable();
> >     +// HACK: Explicit assignment is needed for OpenJDK javac.
> >     +final T fftImageType = Util.getTypeFromInterval( fftImage );
> >     +final T buffer = fftImageType.createVariable();
> >
> >     It seems, that this has been limited (fortunately!!!) to
> >     imglib2-algorithms-gpl.  But why? Surely, there are similar
> >     constructs in
> >     other parts of imglib. Why are they not affected?
> >
> >     Are there plans to figure out the performance implications of using the
> >     old class format?  Is there anything I have to be careful about now, so
> >     that I don't break anything?  Can I (locally) just remove the
> >     maven-compiler-plugin section from the pom and have everything
> >     working as before?
> >
> >     And honestly, I don't like that this was done without any advance
> >     warning or discussion. This seems to be a big thing with potential
> >     performance implications.
> >
> >     best regards,
> >     Tobias
> >
> >
> 