[ImageJ-devel] SemVer policies for ImageJ2 and Fiji

[Mark Hiner]

Hi Stephan,

>Behavior, however, is critical and complicated to test, having a
>contract that makes guarantees here would indeed be useful

A contract that guarantees this would be amazing. I think we are on the
same page, but just wanted to illustrate how bugs make this an intractable
problem for versioning:

If bug fixes that change behavior would cause MAJOR version bumps, the
introduction of bugs that change behavior must also cause a MAJOR version
bump. Thus you need to know if you have added a bug or not before doing a
release. Failure to identify a bug means the release of versions that do
not follow the versioning scheme, which means we can not actually use the
version to reason about compatibility.

>> 3. Since this is an internal versioning scheme, it may not be easy to
>> compare our versions with external project versions that use SemVer.
>
>Why would we want to do this?

My original thought was dependency convergence - but for that the versions
comparisons are between artifacts with the same versioning system, so all
you need is internal consistency. So this may not actually be important.

>http://sentimentalversioning.org/

This page is great. I either hadn't read it or had forgotten it, so thanks
for sharing!

> https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e

I think the author of this post slightly misunderstands the use of SemVer.
I completely agree with some of the content ('If you expect SemVer to solve
your problems for you, you will be disappointed'). But the author seems to
blame SemVer for failing to account for changes in behavior, when really
the failure was in their use of SemVer.

In the intro of http://semver.org/ is the phrase: "For this system to work,
you first need to declare a public API." If a developer doesn't declare API
to include behavior then they can't expect behavior to respect SemVer. If
they decide API does include behavior and then follow SemVer, then behavior
would be guaranteed between compatible versions - assuming they updated the
version appropriately when changing behavior.

>Instead, we should try to build pom-fiji including all
>downstream modules *overriding* or *upgrading* their dependency versions
>to the highest possible.  Do you have something like this already or do
>you have an idea how that could be done?

We have been throwing around the phrase "Melting Pot" for a while. This is
our dream: to have a Jenkins job that pulls the full SciJava software stack
and tests everything together based - i.e. build Fiji's pom
<https://github.com/fiji/fiji/blob/86622adda86e786dac01e8f7339cba5cb772b50b/pom.xml>
and test. This would check for dependency convergence as you outlined, and
make whatever guarantees of behavior we can automate.

There is a script started to do this
<https://github.com/scijava/scijava-scripts/blob/aab7cfe805638efc25615fdd7c78cedaae3c7197/melting-pot.sh>,
but no tests incorporating it that I know of yet. But it sounds like we're
on the same page, that this is how our software should be tested.

With this in mind, my proposal is:

* Define "API" for our software to be limited to public interfaces, classes
and methods, but not behavior. This is what will be covered by SemVer.
* Develop Melting Pot tests to validate behavior and dependency convergence

Unless there are any objections to this, I will update
http://imagej.net/Architecture#Versioning to explicitly define API as such,
and summarize the concerns that have been raised in this discussion -
including how the Melting Pot will make everything happy.

>we're doomed.

Probably!

>Thanks for all the energy that you're putting into this!

You too!

Best,
Mark

On Fri, Mar 27, 2015 at 10:06 PM, Stephan Saalfeld <
saalfelds at janelia.hhmi.org> wrote:

> Hi Mark,
>
> >[...] but not the traditional definition of public API[1,2].
>
> This is what worries me because for most things that are less trivial
> than addition, `traditional' definitions are not available, i.e.
> `unintended' is not well defined.
>
> > Just to make sure we're on the same page - when you say "changes the
> > output", I assume you mean "changes the behavior but not the API - i.e.
> > return type is unchanged"
>
> In a typed language yes, what about Python?  What I mean is indeed
> change in behavior---correct.  I proposed to treat change in behavior as
> change of API because that would enable to reason about compatibility.
> If we enable PATCH upgrades to change behavior but only require `compile
> time' compatibility (which limits the scope of PATCH to statically typed
> and compiled languages which, I think, was not intended), then all we
> can reason from seeing a PATCH upgrade is that the stuff will compile,
> and somehow run, no guarantees about behavior.  That's not very helpful
> because compiling and running stuff isn't expensive to just try.
> Behavior, however, is critical and complicated to test, having a
> contract that makes guarantees here would indeed be useful.
>
> > In that case, I think it would be cleaner to just eliminate the PATCH
> > number - because every bug fix necessarily changes behavior, right? So
> with
> > this scheme, MAJOR increases = "existing behavior has changed", and MINOR
> > increases = "new behavior is available".
> >
>
> Not necessarily.  Performance improvements, crash-fixes, compatibility
> adjustments to upgraded dependencies (!) would fall into PATCH.  But I
> see that even crashes could be used by dependents as part of the API...
> we're doomed.
>
> > Some problems with creating this versioning scheme:
> > 1. MAJOR versions will increase rapidly. This is aesthetic, but one that
> > people can react very negatively to - and can certainly be confusing if
> > people don't expect MAJOR version bumps to cover bug fixes.
>
> Right---but SemVer is already counterintuitive compared to romantic
> versioning where a MAYOR upgrade is associated with a lot of new
> features, not breakage.
>
> > 2. It's one more thing for external developers to learn. We can't just
> say
> > "we use SemVer".
>
> Very true.
>
> > 3. Since this is an internal versioning scheme, it may not be easy to
> > compare our versions with external project versions that use SemVer.
>
> Why would we want to do this?
>
> > 4. There will be false negatives for MAJOR version compatibility
> > comparisons (instead of what could be considered false positives with
> > SemVer)
>
> I wouldn't call them false but conservative or overly cautious.  It
> would tell you when compatibility is not guaranteed which is useful.
>
> >
> > Unfortunately, #1 above alone makes it very unlikely that we would want
> to
> > adopt this use of version numbers.
>
> I understand and fully agree.  But SemVer with PATCHES of `unintended
> behavior' or other sentimental or romantic versioning schemes (I am
> stealing these terms from
>
> https://gist.github.com/jashkenas/cbd2b088e20279ae2c8e
>
> and
>
> http://sentimentalversioning.org/
>
> which are both exciting reads) aren't useful to reason about
> compatibility.  We should therefore not use them to reason about
> compatibility which you correctly state in the following...
>
> > But what you're trying to do here -
> > ensure compatibility - is fantastic and something that would be great to
> > have.
> >
> > So let's take a step back and look at what guarantees we do and do not
> have
> > right now:
> >
> > + We have reproducible builds (release couplings, which requires -some-
> > versioning scheme to be used)
> > + We have API compatibility (SemVer)
> >
>
> No---there is no strict API compatibility when using PATCH to fix
> `unintended behavior'.  Happened with ImageJ many times over the last
> years: API compile-time compatible that would qualify as PATCH upgrade,
> behavior different, plugins delivering crazy output, discovered only
> much later through bug reports.
>
> > - We do not have strict behavior compatibility
> > - We do not have dependency compatibility
> >
> > Behavior and dependency compatibility are very closely related - if they
> > were covered by a versioning scheme, we could automatically answer the
> > question "is it safe to drop in version X to replace version Y?".
> >
>
> Yes.
>
> > However, I do not think we should conflate these concerns with SemVer.
>
> I could not agree more.
>
> > Instead, two potential options would be:
> >
> > 1) Continue to use SemVer, accept its limitations, be content with
> > reproducibility.
>
> Or any other versioning scheme that people like.  My current impression
> is that SemVer doesn't help much.  I will keep using it, but it isn't
> any better than any other scheme that assigns keys to versions and
> enables to order them.
>
> > 2) Create a separate versioning scheme that covers behavior and
> dependency
> > compatibility. Use it in tandem with SemVer.
> >
>
> You're absolutely right, but---phew---that's probably asking too much of
> people that are already unwilling to adopt to something as simple as
> SemVer.
>
> My current thinking is that we should require contributors to use a
> versioning scheme that can be sorted correctly by
>
> sort -V
>
> Anything else, i.e. compatibility guarantees, aren't possible to express
> with a simple number and require runtime testing (e.g. unit-tests
> promise this int an ideal world).  We should therefore never drop any
> upgrade without significant attempt to test.  We could assume that a
> maven build includes sufficient [unit-]testing to enable deployment if
> it succeeds (this is not true in reality but it is a meaningful
> contract).  I.e. artifacts can be deployed if they were explicitly built
> against upgraded dependencies.  In our heterogeneous environment,
> however, it is unrealistic to expect all contributors to adapt their
> code in time.  Instead, we should try to build pom-fiji including all
> downstream modules *overriding* or *upgrading* their dependency versions
> to the highest possible.  Do you have something like this already or do
> you have an idea how that could be done?
>
> Thanks for all the energy that you're putting into this!
>
> Cheers,
> Stephan
>
>
>
> > If anyone can think of other examples of versioned guarantees that would
> be
> > useful to have, or counter-examples to any claims made here - please
> share!
> >
> > Also, please let me know if any of this is confusing and/or additional
> > examples would be useful.
> >
> > Thanks again for the continued discussion,
> > Mark
> >
> > [1]
> >
> http://en.wikipedia.org/wiki/Application_programming_interface#API_in_object-oriented_languages
> > [2]
> >
> http://stackoverflow.com/questions/2954372/difference-between-spi-and-api
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://imagej.net/pipermail/imagej-devel/attachments/20150330/2b0b493c/attachment.html>