[ImageJ-devel] ImgLib split?

Thu Apr 10 11:41:05 CDT 2014

Hi Steffi & Tobias,

> I am happy to discuss it with Tobias in detail when I am in Dresden in
> two weeks.

Thanks! I agree that ImgLib2 core has been quite stable lately. Hopefully
there are not too many more API changes needed before a 2.0.0 release.

One thing that has been less stable (and still needs more API changes) is
the imglib2-meta package. It is becoming increasingly clear to me that the
imglib2-meta and ij-data components are redundant and should be merged. We
especially should not have both ImgPlus *and* Dataset. So I want to merge
them and call it something like "imagej-core" (since the current "ij-core"
has nothing to do with images specifically and will be migrating up to the
SciJava layer [1]). Then it would be one less component for the ImgLib2
team to worry about maintaining.

As an added benefit, the current ImgLib2 ROI class hierarchy could also be
migrated into this combined "imagej-core" component, since that one will
*not* be coming out of beta in June. That will give us freedom to improve
it dramatically over the next few months.

If bringing ImgLib2 core out of beta by June is too difficult, then we
could instead do it when we meet in October. That would be a very nice
target for a hackathon!

What do you guys think?

Regards,
Curtis

[1] http://imagej.net/pipermail/imagej-devel/2014-March/001996.html

On Tue, Apr 8, 2014 at 10:54 PM, Stephan Preibisch <
preibischs at janelia.hhmi.org> wrote:

> Hi guys,
>
> I definitely see the need to fix ImgLib2 core for a release, but that
> really did not change much recently I feel -- maybe we finally need to
> define what the core should actually be, we started with that on the
> hackathon in Madison but never really finished.
>
> I support Tobias's position, I have only used ImgLib2 recently, not done
> any development on it. So he has a much better perspective on it than I
> have. I am happy to discuss it with Tobias in detail when I am in Dresden
> in two weeks. We have to do software integration anyways.
>
> Cheers,
> Steffi
>
> On Apr 9, 2014, at 7:24 , Curtis Rueden wrote:
>
> Hi Tobias,
>
> > I must have missed that. Can you send a link?
>
> https://list.nih.gov/cgi-bin/wa.exe?A2=IMAGEJ;8570535e.1404
>
> > I would just ask that Curtis or you explain for a half hour or so
> > these magic release engineering helpers over skype.
>
> Happy to do it, once it is all in place. Early progress is here:
> * https://github.com/scijava/pom-scijava/compare/developer-profiles
> Though in general we are moving toward this:
> * https://github.com/scijava/scijava-maven-plugin
>
> The idea being to have a Maven plugin that eliminates tediousness in
> various ways.
>
> Unfortunately, developing this infrastructure cannot be our priority
> before June 1.
>
> > My fear with splitting subprojects is that this will make it harder to
> > consistently refactor across subprojects, (or clean up behind commits
> > that don't), see this discussion
> > https://github.com/imglib/imglib/pull/23 (last 10 messages or so). How
> > can we pull this off consistently?
>
> That is a fair concern. However, as long as all affected projects are open
> in Eclipse, and coupled with project couplings at the time, the refactoring
> will cover everything. If someone does a refactoring while excluding a
> downstream project, the easiest solution I know to "fix it" later is to
> check out the old version of the code prior to performing the refactoring
> and reperform it with the project linkage in place this time.
>
> > Also I image that we will require quite a bit more of "git logistics"
> > with split projects. For example, assume that I want to make a new
> > topic branch that touches more than one subproject (which easily
> > happens when refactoring).
>
> This is actually a larger concern to me. However, it comes down to how
> mature the project is. When you are in the incubation phase (pre-release),
> then these refactorings are common, and SNAPSHOT couplings with a
> multi-module Maven project are desirable. However, once you become released
> and stable, there is an increased need to maintain backwards compatibility.
> In that case, you really want to *avoid* those sorts of refactorings in the
> future. I mean sure, you can do them, but you must also maintain backwards
> compatibility, meaning that you leave the old API there too (often
> @Deprecated). In that common scenario, you probably actually want to alter
> only one project at a time, then make sure the changes in a "low-level"
> project like imglib2 core do not adversely affect downstream "higher-level"
> projects such as imglib2-realtransform.
>
> > How will Jenkins deal with this decoupled situation: I will merge my
> > topic branches into master in each of the subprojects sequentially.
> > This will produce a lot of failing intermediate builds in Jenkins,
> > right?
>
> No, because each component is pinned to release versions of other
> components. So changing master of one component will have no effect on the
> Jenkins builds of other components. You would need to do a release of
> component A, then update component B's dependency to the new version of A,
> before the Jenkins build would be affected.
>
> It is useful locally to pin a bunch of components together temporarily
> using SNAPSHOTs. But you never want that pin to make it onto the master
> branch, or else the build will be irreproducible. So one of our goals is to
> make it as easy as possible to toggle your local developer configuration
> between release and snapshot couplings, while simultaneously making it as
> difficult as possible to accidentally commit and push code that uses or
> requires the SNAPSHOT bindings to work.
>
> > How about doing decoupled versions without splitting up the git
> > repository? It seems to me that this would be an easy way to avoid the
> > downsides mentioned before.
>
> It would avoid some of them, and cause others...
>
> > One more thing: If you want to bring imglib core out of beta, we
> > should probably do a clean-up. There are things that are in core now,
> > I would not consider ready for release (ROIs come to mind). So either
> > we live with rapidly growing major version numbers due to frequent API
> > breaks (fine with me) or split out the not-quite-ready parts into
> > their own subprojects (also fine with me).
>
> It is not a matter of "if" but "when". We simply cannot leave ImgLib2 in
> beta forever. That said, ImgLib2 does not need to be released before ImageJ
> 2.0.0's June 1 deadline, because that deadline has very limited scope. The
> new N-dimensional data model is not included with that. But we will need to
> bite the bullet eventually, possibly even later this year...
>
> > either we live with rapidly growing major version numbers due to
> > frequent API breaks (fine with me)
>
> To a limited extent I am also fine with that. But I think we should impose
> a limit; e.g., "no more than one major release bump per year" or some such.
> Ideally much less often, if ever. Keep in mind that in a highly extensible
> framework like ImageJ, every breaking change can have enormous disruptive
> consequences. The fact that ImgLib2 is not stable and released with a
> guaranteed-not-to-break-later API has surely stunted its adoption. We can
> discuss and refine more as the projects continue to mature.
>
> Regards,
> Curtis
>
>
> On Tue, Apr 8, 2014 at 1:49 AM, Tobias Pietzsch <pietzsch at mpi-cbg.de>wrote:
>
>> Hi Johannes and Curtis,
>>
>> certainly you saw Curtis' recent mail about our plans for ImageJ2?
>>
>> I must have missed that. Can you send a link?
>>
>> First of all: in principle, I have no problem with that if it is
>> necessary.
>> I would just ask that Curtis or you explain for a half hour or so these
>> magic release engineering helpers over skype. (It would by the way also be
>> nice to know how this currently works. I have no idea, how I would do a
>> "proper beta release" if I wanted to do so... I would appreciate some
>> pointers to documentation or scripts etc.)
>>
>> That being said, here are my concerns and questions:
>>
>> My fear with splitting subprojects is that this will make it harder to
>> consistently refactor across subprojects, (or clean up behind commits that
>> don't), see this discussion https://github.com/imglib/imglib/pull/23 (last
>> 10 messages or so).
>> How can we pull this off consistently?
>>
>> Also I image that we will require quite a bit more of "git logistics"
>> with split projects. For example, assume that I want to make a new topic
>> branch that touches more than one subproject (which easily happens when
>> refactoring). Will I have to make topic branches in all subprojects? Is
>> there a way to relate these other than manually by using the same branch
>> names across projects, etc?
>> How will Jenkins deal with this decoupled situation: I will merge my
>> topic branches into master in each of the subprojects sequentially. This
>> will produce a lot of failing intermediate builds in Jenkins, right? I
>> think this will complicate hunting down errors.
>> Overall, I'm a bit afraid of the additional overhead.
>>
>> How about doing decoupled versions without splitting up the git
>> repository? It seems to me that this would be an easy way to avoid the
>> downsides mentioned before.
>>
>>
>> One more thing: If you want to bring imglib core out of beta, we should
>> probably do a clean-up.
>> There are things that are in core now, I would not consider ready for
>> release (ROIs come to mind).
>> So either we live with rapidly growing major version numbers due to
>> frequent API breaks (fine with me) or split out the not-quite-ready parts
>> into their own subprojects (also fine with me).
>>
>> Stephans, what do you think?
>>
>>
>> best regards,
>> Tobias
>>
>> On Apr 7, 2014, at 11:06 PM, Johannes Schindelin <schindelin at wisc.edu>
>> wrote:
>>
>> Hi Tobias, Stephan & Stephan,
>>
>> certainly you saw Curtis' recent mail about our plans for ImageJ2?
>> Basically, we want to release a version of ImageJ whose user interface
>> looks like ImageJ1, but internally uses all the goodies on which we worked
>> so hard these past years.
>>
>> That includes ImgLib2, of course, so we would need to bring parts of
>> ImgLib2 out of beta. In particular, we found it unwise to always version
>> all of ImgLib2 in unison. Rather, there should be releases of the
>> individual components whenever there should be new releases: bug fixes,
>> API enhancements, API-breaking major new developments.
>>
>> As always, Curtis & I are ready to help with all of that stuff, in
>> particular with helpers making release engineering close to fun. Our
>> central goal in that respect is to make it as easy as possible to switch
>> between A) reproducible builds with release couplings; and B)
>> tightly-coupled builds with snapshot couplings for rapid development
>> across components.
>>
>> The first step would be to break the multi-module ImgLib2 repository apart
>> (much in the way we split out imglib2-tests and friends, except that we
>> split out *all* of the individual projects). We do not see any other way
>> to get only that part of ImgLib2 out of beta that we really need for the
>> ImageJ2 release...
>>
>> Are you okay with that plan?
>>
>> Ciao,
>> Dscho
>>
>> P.S. We are planning to split up imagej.git in very much the same way.
>>
>>
>>
> _______________________________________________
> ImageJ-devel mailing list
> ImageJ-devel at imagej.net
> http://imagej.net/mailman/listinfo/imagej-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20140410/28b945d4/attachment.html>