<div dir="ltr">Hi Jay,<div><br></div><div><div>> The requirement to generate such and index was not clear to me from previous</div><div>> conversations and the high-level documentation on the plugin mechanism of</div>
<div>> ImageJ2... maybe I missed something? It was clear how you had to annotate the</div><div>> classes but not the requirement of generating an index or the best way how.</div><div><br></div><div>I want to clarify: Anyone using a javac-based build system has all the annotations generated for them with no effort on their part. That is what a compile-time annotation processor does. This includes:</div>
<div><br></div><div>* Ant builds on the command line</div><div>* Maven builds on the command line</div><div>* classic Makefiles</div><div>* Gradle builds</div><div>* NetBeans builds</div><div>* IntelliJ IDEA builds</div>
<div>
* Manually invoking javac</div><div>* Any Maven-based project in Eclipse (thanks to Dscho's EclipseHelper)</div><div><br></div><div>The _only_ known use case that has this problem is an Eclipse-only development style which does not use Maven _and_ has no unit tests _and_ does not configure a .factorypath _and_ uses Eclipse's "Export JAR" feature to deploy its JARs.</div>
<div><br></div><div>If you want to help improve the situation with this use case, it would be awesome if you could write up this requirement as a page on the ImageJ wiki (<a href="http://wiki.imagej.net/">http://wiki.imagej.net/</a>) together with instructions on ways to configure the project in Eclipse to solve the issue.</div>
<div><br></div><div>> Further, I just imagine other users getting things to work great from</div><div>> Eclipse, then when they export, it all breaks. It doesn't appear there is a</div><div>> mechanism provided in the sci-java common library to streamline index</div>
<div>> generation for more general jar application development.</div></div><div><br></div><div>Anyone following the Maven-based "minimal IJ1 plugin" example [1] will not have the problem, even from Eclipse, thanks to the EclipseHelper -- as well as the fact that they will not need to use Eclipse's "Export JAR" feature but can instead simply run Maven's "package" goal to generate the JAR with annotations (this can be done using "Run As > Maven install" in the Eclipse context menu).</div>
<div><br></div><div>If you have ideas on how to streamline the process further to make things more easy and transparent for Eclipse users not using Maven, then pull requests are very welcome.</div><div><br></div><div>Regards,</div>
<div>Curtis</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 6, 2014 at 12:18 PM, Jay Warrick <span dir="ltr"><<a href="mailto:warrick@wisc.edu" target="_blank">warrick@wisc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Thanks for the reply Curtis.</div><div><br></div>I got it to work but not completely pleased for the reasons you cite. You run the application once, it fixes the jar file but the changes to the jar file on disk are not reflected in the jar file that is loaded already in memory for running the application (at least as indicated by the getResources() method). So, you close, and the second time you run, the getResources() function now sees the index and everything works. This jar is now "fixed" and can then be deployed so others don't have to open once and restart. I didn't get a reply on whether this was even a reasonable approach before I put a lot of work into trying this. I tried it because I really wanted to roll out a few functions to collaborators and I thought there was a chance that changes would be reflected during run time because the same approach was successful when running from Eclipse. It appears that knowledge of jar resources is established more wholesale at initial loading of any classes compared to loading resources from folders as occurs when Eclipse runs. Nothing crashes or anything but the additions to the jar aren't reflected by the getResources() method (which makes sense in hindsight :-).<div>
<br></div><div>However, it seems like the current situation generally forces others to use Maven or some other mechanism of generating the annotation indexes and packaging them into the jar if anyone wants to implement their own sci-java plugins. Do you index the "plugins folder" full of user defined .java, .class, and .jar files separately somewhere in the code that I didn't see? If that is handled, this whole issue is probably moot for ImageJ2 stuff and really relates more to the ease-of-use/flexibility of sci-java common/SciJavaPlugin/PluginService mechanism for other applications. The requirement to generate such and index was not clear to me from previous conversations and the high-level documentation on the plugin mechanism of ImageJ2... maybe I missed something? It was clear how you had to annotate the classes but not the requirement of generating an index or the best way how. Further, I just imagine other users getting things to work great from Eclipse, then when they export, it all breaks. It doesn't appear there is a mechanism provided in the sci-java common library to streamline index generation for more general jar application development. Potentially at least a "semi-manual" java static function to "fix" a jar that doesn't require setting up a Maven build script (or similar) would be useful for those out there that aren't Maven literate yet. I now have the basis for this code if you think it useful. It works after packaging the jar so it doesn't have to be implemented within a build script (i.e., you don't absolutely have to know how to add custom operations to build instructions to Eclipse or Maven). It piggybacks on the EcipseHelper and seems to reproduce the indices produced by the EclipseHelper when running from Eclipse in this instance but I'm not absolutely sure it fits all use cases. You guys would likely be better judges of that. Let me know if you are interested but it's what I'll use for now while I get up to speed on Maven stuff.</div>
<div><br></div><div>Cheers,</div><div><br></div><div>Jay</div><div><div class="h5"><div><br></div><div><div><br></div><div><div><div>On May 6, 2014, at 11:17 AM, Curtis Rueden <<a href="mailto:ctrueden@wisc.edu" target="_blank">ctrueden@wisc.edu</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr">Hi Jay,<div><br></div><div><div>> The iteration method it uses requires directories, which is fine and</div><div>> dandy when running from within Eclipse but my program's JarFile</div>
<div>> doesn't look like a directory so the exact same mechanism doesn't</div>
<div>> work.</div></div><div><br></div><div>EclipseHelper is intended to work when running from Eclipse, and never otherwise. That is why it does not support generating annotations from classes within JAR files.</div>
</div></blockquote><blockquote type="cite"><div dir="ltr">
<div>
<br></div><div><div>> However, it looks like a method analogous to this approach but for</div><div>> JarFiles might work., right?</div></div><div><br></div><div>I don't understand why you would ever need to do that... the point of EclipseHelper is to generate the annotations (META-INF/json/*) as close to compile time as possible -- not later at runtime, which would be the only time your code would be stored within a JAR file.</div>
<div><br></div><div>> 0) Does my take on the situation seem right?</div><div><br></div><div>While the technical aspects of what you are saying are correct to my knowledge (NB: I did not investigate the usage of getURLs()), I do not think it makes sense to invoke an EclipseHelper-like program during the actual execution of your program. By doing that, you would lose all the performance advantages of generating the annotations in advance.</div>
<div><br></div><div>For ImageJ2 we use Maven from the command line to bundle our JARs and that generates the annotations as expected. An alternate approach, if you really really want Eclipse to do it for you instead, you might be to do it as part of Maven's generate-sources step; see the ImageJ OPS project for an example of that [1]. You'd just have to change the groovy script to invoke the annotation processor manually. But I haven't seriously explored doing things that way. And JEX would need to be structured as a Maven project for that...</div>
<div><div><br></div><div>> 1) If I edit the programs jar file during run time by adding the</div><div>> META-INF/json/<package>.Plugin jar entry, will that majorly hose</div><div>> things?</div></div><div>
<br>
</div><div>Yep, I would be surprised if that did not cause problems. I strongly advise not doing that.</div><div><br></div><div><div>> In other words, if I can get the META-INF/json/<package>.Plugin</div><div>> written to my program's jar analogous to the other referenced jars,</div>
<div>> then everything will be OK?</div></div><div><br></div><div>Yes, the goal is definitely to somehow generate your JAR file such that it contains the META-INF/json/* metadata from the get-go, without needing to pull any tricks the first time JEX is launched.</div>
<div><br></div><div>Another possibility is to add a .factorypath and .settings/org.eclipse.jdt.apt.core.prefs file, similar to:</div><div><br></div><div>.settings/org.eclipse.jdt.apt.core.prefs:</div><div><div>eclipse.preferences.version=1</div>
<div>org.eclipse.jdt.apt.aptEnabled=true</div><div>org.eclipse.jdt.apt.genSrcDir=target/classes</div><div>org.eclipse.jdt.apt.reconcileEnabled=false</div></div><div><br></div><div><div>.factorypath:</div><div><factorypath><br>
</div><div><div> <factorypathentry kind="VARJAR" id="M2_REPO/org/scijava/scijava-common/2.18.1/scijava-common-2.18.1.jar" enabled="true" runInBatchMode="true"/></div><div></factorypath></div>
</div><div><br></div></div><div>But you'd want to ensure it matches your version of scijava-common, and also point it at wherever you have that JAR stored rather than into the Maven repository (which you presumably don't have since you aren't using Maven).</div>
<div><br></div><div>Eclipse unfortunately makes invoking annotation processors quite complicated... [2]</div><div><br></div><div>Regards,</div><div>Curtis</div><div><br></div><div>[1] <a href="https://github.com/imagej/imagej-ops/blob/imagej-ops-0.3.0/pom.xml#L169" target="_blank">https://github.com/imagej/imagej-ops/blob/imagej-ops-0.3.0/pom.xml#L169</a></div>
<div>[2] <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=280542" target="_blank">https://bugs.eclipse.org/bugs/show_bug.cgi?id=280542</a></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 30, 2014 at 7:49 PM, Jay Warrick <span dir="ltr"><<a href="mailto:warrick@wisc.edu" target="_blank">warrick@wisc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Hi All,</div><div><br></div>Thanks for all these replies folks. Digging around yesterday afternoon and evening essentially painted a picture much like what Johannes indicated in his email. The getResource() method works just fine and works roughly the same whether you do A) jars-in-a-jar or B) jars (reference libraries) in a folder next to a jar (my program), which are two of the 3 possible modes of exporting runnable jars. The third option, C), actually extracts all the class files and reorganizes them into a single flat jar (i.e., not jar-in-jar but just a single uber-jar of classes in folders), which I haven't used in the past for other reasons as well. <div>
<br></div><div>Instead, it seems (I say seems because I only dabble in ClassLoaders) their is no way to compile a list of my program's class files for indexing annotation using just the method getResource() or getResources() without knowing the name of at least a single resource in my program's jar. The getResources() method just gives a list of the referenced jars and not the main program's jar itself. However, if I want sci-java to work agnostic of who's using it, I can't assume within the sci-java code that I ever know of a specific resource to look for. Even if getResource() does give a path to my program's jar, the jar does not have the META-INF/json/<package>.Plugin file that contains the results of indexing. The key to the EclipseHelper is that it actually uses the getURLs() method of the URLClassLoader instead of the getResources(). When running from eclipse, this method gives the path to my referenced jars and the programs bin folder. When running from jar, the getURLs() method returns both the referenced libraries and my programs jar file (yay). However, since the jar file is not a directory, the EclipseHelper method does not index it and write the META-INF/json/<package>.Plugin file. The iteration method it uses requires directories, which is fine and dandy when running from within Eclipse but my program's JarFile doesn't look like a directory so the exact same mechanism doesn't work. However, it looks like a method analogous to this approach but for JarFiles might work., right? I think, to work, it would have to iteratively index the contents of the JarFile and add a META-INF/json/<package>.Plugin file as a new Jar entry (as far as I can see anyway, I could definitely be wrong). I am trying to hack something together to show whether this would work at all but if any of you think this approach would benefit others, I'd welcome suggestions and would try to clean it up as much as I can for a pull request. It seems like the EclipseHelper is used when the class loader is an instance of URLClassLoader (plus some other tests...) which happens to occur both when running from eclipse and when exported as a runnable jar. So, on the surface at least, it looks like the EclipseHelper class could be used for both scenarios but am happy to change tacks if you think otherwise.</div>
<div><br></div><div>However, I have a couple specific questions, that if answered would help me to know if this might work at all and move forward.</div><div><br></div><div><b>0) Does my take on the situation seem right?</b></div>
<div><br></div><div><b>1) If I edit the programs jar file during run time by adding the META-INF/json/<package>.Plugin jar entry, will that majorly hose things?</b></div><div><b><br></b></div><div><b>2) The Index class still uses the getResources() method to find things and it will still only link to the referenced jar files. In this case we can't assume that the ClassLoader is a URLClassLoader to be able to use the getURLs method. I don't recall noticing when or how the indexing mechanism looks for these META-INF/json/<package>.Plugin files to compile all the annotation information. I assume it is via the getResource() method because I believe by knowing what to look for (i.e., a META-INF/json/<package>.Plugin file), we can just ask for the resource directly? In other words, if I can get the META-INF/json/<package>.Plugin written to my program's jar analogous to the other referenced jars, then everything will be OK?</b></div>
<div><br></div><div>Thanks thanks thanks,</div><div><br></div><div>Jay</div><div><div><br></div><div><div><div>On Apr 30, 2014, at 12:51 PM, Johannes Schindelin <<a href="mailto:Johannes.Schindelin@gmx.de" target="_blank">Johannes.Schindelin@gmx.de</a>> wrote:</div>
<br><blockquote type="cite">Hi,<br><br>On Wed, 30 Apr 2014, Curtis Rueden wrote:<br><br><blockquote type="cite"><blockquote type="cite">when I export the application as a runnable jar, ij.plugin()service<br>returns all the Command.class plugins but 0 plugins of type<br>
JEXPlugin.class<br></blockquote><br>This is an issue we have discussed before: Eclipse creates uberjars using a<br>"jar-in-jar" approach, and SciJava Common's plugin mechanism does not read<br>the metadata out of a jar-in-jar.<br>
</blockquote><br>Please note that the jar-in-jar poses no problem, unless the ClassLoader<br>used to access them is broken: it needs to support the getResources() call<br>properly and find the resource files contained in the nested .jar files.<br>
<br>However, in the reported case I believe it is not triggered by the uber<br>jar or jar-in-jar scenario.<br><br>Background: The internal technique behind the plugins uses annotation<br>processors run at compile time. They basically look at each file that has<br>
a @Plugin annotation and write out index files that get included into the<br>.jar files.<br><br>Except that Eclipse -- violating the Java specification -- does not run<br>annotation processors. At least not by default, and even if you switch it<br>
on (manually, for each and every project you maintain, one by one), it<br>*still* only runs them on full builds (i.e. after Project>Clean).<br><br>So it looks to me that in the reported case, the annotation processor is<br>
never run, and as a consequence, the index file is never written, and<br>therefore it cannot be found at runtime.<br><br>Of course, Eclipse being such a prevalent platform to develop in, we tried<br>to come up with a workaround: whenever the annotation indexes are read, a<br>
class called "EclipseHelper" tries to detect whether it needs to create<br>the index files because Eclipse failed to run the annotation processors.<br><br>This works amazingly well because many developers have written unit tests<br>
and run them before bundling .jar files manually. These unit tests verify<br>that plugins work, of course, which is why the EclipseHelper works around<br>the problem successfully in most cases.<br><br>Also, here is a lesson for everybody choosing to learn from our past<br>
mistakes and experiences: any possible convenience of uber jars is<br>outweighed multiple times over by the disadvantages it incurs to users: it<br>makes updating really costly (every time it's time to update, a new<br>
monster .jar needs to be downloaded), it makes collaboration between<br>projects difficult at best, and it certainly asks for version skew.<br><br>Ciao,<br>Johannes<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div></div></div></div></div></blockquote></div><br></div>