[ImageJ-devel] [LOCI ImageJ] IJ2 Plugin Service

Tue May 6 11:17:26 CDT 2014

Hi Jay,

> The iteration method it uses requires directories, which is fine and
> dandy when running from within Eclipse but my program's JarFile
> doesn't look like a directory so the exact same mechanism doesn't
> work.

EclipseHelper is intended to work when running from Eclipse, and never
otherwise. That is why it does not support generating annotations from
classes within JAR files.

> However, it looks like a method analogous to this approach but for
> JarFiles might work., right?

I don't understand why you would ever need to do that... the point of
EclipseHelper is to generate the annotations (META-INF/json/*) as close to
compile time as possible -- not later at runtime, which would be the only
time your code would be stored within a JAR file.

> 0) Does my take on the situation seem right?

While the technical aspects of what you are saying are correct to my
knowledge (NB: I did not investigate the usage of getURLs()), I do not
think it makes sense to invoke an EclipseHelper-like program during the
actual execution of your program. By doing that, you would lose all the
performance advantages of generating the annotations in advance.

For ImageJ2 we use Maven from the command line to bundle our JARs and that
generates the annotations as expected. An alternate approach, if you really
really want Eclipse to do it for you instead, you might be to do it as part
of Maven's generate-sources step; see the ImageJ OPS project for an example
of that [1]. You'd just have to change the groovy script to invoke the
annotation processor manually. But I haven't seriously explored doing
things that way. And JEX would need to be structured as a Maven project for
that...

> 1) If I edit the programs jar file during run time by adding the
> META-INF/json/<package>.Plugin jar entry, will that majorly hose
> things?

Yep, I would be surprised if that did not cause problems. I strongly advise
not doing that.

> In other words, if I can get the META-INF/json/<package>.Plugin
> written to my program's jar analogous to the other referenced jars,
> then everything will be OK?

Yes, the goal is definitely to somehow generate your JAR file such that it
contains the META-INF/json/* metadata from the get-go, without needing to
pull any tricks the first time JEX is launched.

Another possibility is to add a .factorypath and
.settings/org.eclipse.jdt.apt.core.prefs file, similar to:

.settings/org.eclipse.jdt.apt.core.prefs:
eclipse.preferences.version=1
org.eclipse.jdt.apt.aptEnabled=true
org.eclipse.jdt.apt.genSrcDir=target/classes
org.eclipse.jdt.apt.reconcileEnabled=false

.factorypath:
<factorypath>
  <factorypathentry kind="VARJAR"
id="M2_REPO/org/scijava/scijava-common/2.18.1/scijava-common-2.18.1.jar"
enabled="true" runInBatchMode="true"/>
</factorypath>

But you'd want to ensure it matches your version of scijava-common, and
also point it at wherever you have that JAR stored rather than into the
Maven repository (which you presumably don't have since you aren't using
Maven).

Eclipse unfortunately makes invoking annotation processors quite
complicated... [2]

Regards,
Curtis

[1] https://github.com/imagej/imagej-ops/blob/imagej-ops-0.3.0/pom.xml#L169
[2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=280542

On Wed, Apr 30, 2014 at 7:49 PM, Jay Warrick <warrick at wisc.edu> wrote:

> Hi All,
>
> Thanks for all these replies folks. Digging around yesterday afternoon and
> evening essentially painted a picture much like what Johannes indicated in
> his email. The getResource() method works just fine and works roughly the
> same whether you do A) jars-in-a-jar or B) jars (reference libraries) in a
> folder next to a jar (my program), which are two of the 3 possible modes of
> exporting runnable jars. The third option, C), actually extracts all the
> class files and reorganizes them into a single flat jar (i.e., not
> jar-in-jar but just a single uber-jar of classes in folders), which I
> haven't used in the past for other reasons as well.
>
> Instead, it seems (I say seems because I only dabble in ClassLoaders)
> their is no way to compile a list of my program's class files for indexing
> annotation using just the method getResource() or getResources() without
> knowing the name of at least a single resource in my program's jar. The
> getResources() method just gives a list of the referenced jars and not the
> main program's jar itself.  However, if I want sci-java to work agnostic of
> who's using it, I can't assume within the sci-java code that I ever know of
> a specific resource to look for. Even if getResource() does give a path to
> my program's jar, the jar does not have the META-INF/json/<package>.Plugin
> file that contains the results of indexing. The key to the EclipseHelper is
> that it actually uses the getURLs() method of the URLClassLoader instead of
> the getResources(). When running from eclipse, this method gives the path
> to my referenced jars and the programs bin folder. When running from jar,
> the getURLs() method returns both the referenced libraries and my programs
> jar file (yay). However, since the jar file is not a directory, the
> EclipseHelper method does not index it and write the
> META-INF/json/<package>.Plugin file. The iteration method it uses requires
> directories, which is fine and dandy when running from within Eclipse but
> my program's JarFile doesn't look like a directory so the exact same
> mechanism doesn't work. However, it looks like a method analogous to this
> approach but for JarFiles might work., right? I think, to work, it would
> have to iteratively index the contents of the JarFile and add a
> META-INF/json/<package>.Plugin file as a new Jar entry (as far as I can see
> anyway, I could definitely be wrong). I am trying to hack something
> together to show whether this would work at all but if any of you think
> this approach would benefit others, I'd welcome suggestions and would try
> to clean it up as much as I can for a pull request. It seems like the
> EclipseHelper is used when the class loader is an instance of
> URLClassLoader (plus some other tests...) which happens to occur both when
> running from eclipse and when exported as a runnable jar. So, on the
> surface at least, it looks like the EclipseHelper class could be used for
> both scenarios but am happy to change tacks if you think otherwise.
>
> However, I have a couple specific questions, that if answered would help
> me to know if this might work at all and move forward.
>
> *0) Does my take on the situation seem right?*
>
> *1) If I edit the programs jar file during run time by adding the
> META-INF/json/<package>.Plugin jar entry, will that majorly hose things?*
>
> *2) The Index class still uses the getResources() method to find things
> and it will still only link to the referenced jar files. In this case we
> can't assume that the ClassLoader is a URLClassLoader to be able to use the
> getURLs method. I don't recall noticing when or how the indexing mechanism
> looks for these META-INF/json/<package>.Plugin files to compile all the
> annotation information. I assume it is via the getResource() method because
> I believe by knowing what to look for (i.e., a
> META-INF/json/<package>.Plugin file), we can just ask for the resource
> directly? In other words, if I can get the META-INF/json/<package>.Plugin
> written to my program's jar analogous to the other referenced jars, then
> everything will be OK?*
>
> Thanks thanks thanks,
>
> Jay
>
> On Apr 30, 2014, at 12:51 PM, Johannes Schindelin <
> Johannes.Schindelin at gmx.de> wrote:
>
> Hi,
>
> On Wed, 30 Apr 2014, Curtis Rueden wrote:
>
> when I export the application as a runnable jar, ij.plugin()service
> returns all the Command.class plugins but 0 plugins of type
> JEXPlugin.class
>
>
> This is an issue we have discussed before: Eclipse creates uberjars using a
> "jar-in-jar" approach, and SciJava Common's plugin mechanism does not read
> the metadata out of a jar-in-jar.
>
>
> Please note that the jar-in-jar poses no problem, unless the ClassLoader
> used to access them is broken: it needs to support the getResources() call
> properly and find the resource files contained in the nested .jar files.
>
> However, in the reported case I believe it is not triggered by the uber
> jar or jar-in-jar scenario.
>
> Background: The internal technique behind the plugins uses annotation
> processors run at compile time. They basically look at each file that has
> a @Plugin annotation and write out index files that get included into the
> .jar files.
>
> Except that Eclipse -- violating the Java specification -- does not run
> annotation processors. At least not by default, and even if you switch it
> on (manually, for each and every project you maintain, one by one), it
> *still* only runs them on full builds (i.e. after Project>Clean).
>
> So it looks to me that in the reported case, the annotation processor is
> never run, and as a consequence, the index file is never written, and
> therefore it cannot be found at runtime.
>
> Of course, Eclipse being such a prevalent platform to develop in, we tried
> to come up with a workaround: whenever the annotation indexes are read, a
> class called "EclipseHelper" tries to detect whether it needs to create
> the index files because Eclipse failed to run the annotation processors.
>
> This works amazingly well because many developers have written unit tests
> and run them before bundling .jar files manually. These unit tests verify
> that plugins work, of course, which is why the EclipseHelper works around
> the problem successfully in most cases.
>
> Also, here is a lesson for everybody choosing to learn from our past
> mistakes and experiences: any possible convenience of uber jars is
> outweighed multiple times over by the disadvantages it incurs to users: it
> makes updating really costly (every time it's time to update, a new
> monster .jar needs to be downloaded), it makes collaboration between
> projects difficult at best, and it certainly asks for version skew.
>
> Ciao,
> Johannes
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20140506/2992a494/attachment.html>