Difference between revisions of "Development Lifecycle"

(Emphasize relationship with Maven SNAPSHOT versions)
(Relationship with Maven SNAPSHOTs: rephrase to de-emphasize "our" code and lie less)
Line 33: Line 33:
 
= Relationship with Maven SNAPSHOTs =
 
= Relationship with Maven SNAPSHOTs =
  
Another way of thinking about the development cycle is through the Maven version number given associated with our code. The idea behind reproducible builds is that when a user gets a particular version of a plugin, they can precisely deduce the state of the code that produced that plugin. Typically, that state is determined by a unique [[Git]] commit. However, it would be impractical and unrealistic to change the Maven version with every single Git commit.
+
Another way of thinking about the development cycle is through the Maven version number given associated with the code. The idea behind reproducible builds is that from a given version of a plugin, the state of the code producing that version can be determined unambiguously. Typically, that state is determined by a unique [[Git]] commit. However, it would be impractical and unrealistic to change the Maven version with every single Git commit.
  
This is why we use [https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN401 SNAPSHOT] versions while "in development" (phases 1 and 2 - "SNAPSHOT coupling"). Using a SNAPSHOT version is conceding reproducibility in favor of convenience. For this reason we do not provide users with SNAPSHOT versions (except potentially for testing) as they can not facilitate reproducible science.
+
This is why [https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN401 SNAPSHOT] versions are used while "in development" (phases 1 and 2 - "SNAPSHOT coupling"). Using a SNAPSHOT version is saying "no guarantees are made as to the reproducibility of this artifact." For this reason, to best facilitate reproducible science, SNAPSHOT versions of code are not provided to users (except potentially for testing).
  
When we want to provide users with an updated version of our code (phases 3 and 4) we change the version to a unique non-SNAPSHOT version for a single Git commit. Then the next commit returns us to SNAPSHOT versioning for development purposes. And thus the cycle repeats.
+
To provide users with an updated version of an artifact (phases 3 and 4) the version is changed to a unique, non-SNAPSHOT, version for a single Git commit. Then the next commit returns to SNAPSHOT versioning for further development. Thus the cycle repeats.
  
 
= Phases in-depth =
 
= Phases in-depth =

Revision as of 08:14, 11 November 2015


Template:Development The SciJava philosophy is to release early, release often. At the same time, we always want to preserve scientific reproducibility. To make this possible we lean on several project management tools. The purpose of this guide is to take you through the process of using these tools with the goal of releasing new versions of your software, and then providing those releases to users.


Phases of development

ImageJ and Fiji are developed according to the SciJava philosophy, thus these applications are used throughout this tutorial to illustrate the development lifecycle.

Whether adding new features, fixing bugs, improving performance, etc... development is the process of making changes, with the goal of exposing these changes to users. To accomplish this, actively developed projects cycle through four general "phases":

What are Maven artifacts?

Artifacts are files, most commonly a JAR encapsulating the compiled classes for a component. Other files that may be produced as artifacts include:
  • The project's POM
  • A jar with the original source files
  • A jar with any generated javadoc
  • A jar with any test files
  1. In development. The source code is modified to add new features, fix bugs, etc... these modifications are expressed as commits by Git, whether on your local filesystem, a topic branch, or a repository fork.
  2. On master. When you have a set of one or more commits that you are happy with (i.e. the feature is complete, or the bug is fixed) they are moved to the master branch of the project's repository on GitHub. This ensures the master branch is always release ready.
  3. Released. When there is a need to make the current master branch public (i.e. it has a critical bug fix or cool new feature that users have requested) Maven is used to cut a release, which is then deployed as an artifact to the ImageJ Maven repository. Developers can now use the new version in their own projects.
  4. Uploaded. Once we've tested our latest release artifact to ensure it doesn't conflict with other components, it can be uploaded to an ImageJ update site - making it available to end users.

The following sections will discuss these phases, and their associated tools and workflows, in more depth.

Relationship with Maven SNAPSHOTs

Another way of thinking about the development cycle is through the Maven version number given associated with the code. The idea behind reproducible builds is that from a given version of a plugin, the state of the code producing that version can be determined unambiguously. Typically, that state is determined by a unique Git commit. However, it would be impractical and unrealistic to change the Maven version with every single Git commit.

This is why SNAPSHOT versions are used while "in development" (phases 1 and 2 - "SNAPSHOT coupling"). Using a SNAPSHOT version is saying "no guarantees are made as to the reproducibility of this artifact." For this reason, to best facilitate reproducible science, SNAPSHOT versions of code are not provided to users (except potentially for testing).

To provide users with an updated version of an artifact (phases 3 and 4) the version is changed to a unique, non-SNAPSHOT, version for a single Git commit. Then the next commit returns to SNAPSHOT versioning for further development. Thus the cycle repeats.

Phases in-depth

Phase 1: In development

Repositories on GitHub are referred to as remotes; when you clone, or check out, a remote you get a local copy of the repository. Development progresses by making changes to your local copy and pushing them back to the remote. GitHub provides tools for controlling user permission levels for each remote repository, therefore how you develop a project depends on whether you are able to push (write) to that project's remote repository or not.

  • Collaborating developer. If you have permission to push directly to the project's remote repository, then you can simply use Git and GitHub to clone the repository and make your changes. For non-trivial changes, you will typically create a topic branch to develop and test the changes. This also provides a forum for discussion and review with your fellow developers.
  • External developer. If you do not have push rights, then you need to go through an additional step called Forking the repository. This will create a remote copy of the repository, to which you have push rights. Your remote fork is referred to as downstream of the original remote repository (which is upstream of your fork). Your development will then take place on your fork, with an additional step later to reconcile with the upstream repository.

Phase 2: On master

Once a feature or fix is complete it can move to the master branch of the repository. How you accomplish this depends on how the changes were developed in Phase 1.

  • Collaborating developer. Minimal changes can be pushed back directly to master on the remote repository. If your work is on a topic branch then you should use a pull request (PR) so that the topic branch can be reviewed before being merged to master.
  • External developer. First push your changes back to a branch of your forked repository (it doesn't necessarily have to be master). Then you can file a pull request (PR) on GitHub to merge your branch back to the official repository. This invites code review from other interested developers. Reviewers might ask for changes to the code to address any issues. After any needed revisions have been made, a project maintainer will accept your changes and then merge to the official master branch.

Phase 3: Released

Note: This step can only be performed by a project maintainer.

Once the master branch of a component has your desired new functionality, the next step is to cut a release version of the component. Normally, the Maven version (in the pom.xml) on master is a SNAPSHOT version, meaning it is unstable and not yet released. However, a stable release artifact can be deployed to the ImageJ Maven repository in one of three methods, described below. As a rule of thumb, SciJava developers currently use:

Method 1: Release-Version Jenkins job

The special Release-Version job of Jenkins (which is visible only to logged in Jenkins users) is triggered to release a particular artifact. There are several advantages to this approach:

  • A tag is created on GitHub to easily reference the release commit
  • The "bump to next release cycle" commit is done automatically
  • The Maven POM references the correct tag rather than HEAD
  • This job is capable of releasing certain artifacts to OSS Sonatype (and hence to Maven Central) as appropriate

Prerequisites:

  1. You will need an ImageJ Jenkins account. If you haven't set this up yet, contact the ImageJ-devel list and ask for an account. A maintainer will send you instructions.

Steps to release:

  1. Visit the Release-Version Jenkins job web page.
  2. Click "Build with Parameters" from the left-hand menu.
  3. Select your component from the dropdown list.
  4. Enter the commit hash corresponding to the most recent commit of the master branch.
  5. Enter the desired version number of the release.
  6. Click the Build button.
  7. Grab a coffee.

Jenkins will take care of all the steps, including pushing the relevant commits and tags, building the code, and deploying the resultant artifact to the ImageJ Maven repository.

Method 2: release-version.sh

The release-version.sh script is what the Release-Version Jenkins job uses under the hood to perform releases. It relies on the Maven Release plugin to do most of the heavy lifting, but also does some extra work (e.g., to ensure releases are deployed to the correct repository).

Prerequisites:

  1. Install the release-version.sh script. The best way to do this is to clone the complete SciJava-scripts repository. That will give you access to other useful scripts and help keep them all up to date.
  2. (optional) If you want to easily use these scripts from any directory, you can add the scijava-scripts folder to your PATH.
  3. You will need an account for maven.imagej.net and the local Maven configuration to deploy to this repository. If you haven't set this up yet, contact the ImageJ-devel list and ask for an account. A maintainer will send you instructions.

Steps to release:

  1. From your project's directory, simply run:
 release-version.sh <NEW_VERSION>



Method 3: Double push to master

Historically, the "double push to master" approach was the recommended way to release—but it has now been replaced by the other two approaches. Nonetheless, there are a few components of Fiji that still use it to do their releases. A push to master happens with the POM version set to a non-SNAPSHOT—e.g., this Trainable_Segmentation commit releasing version 2.2.1. A second push to master is then typically done to "bump to next release cycle" after Jenkins starts building the release—e.g., this Trainable_Segmentation commit bumps to 2.2.2-SNAPSHOT. This second push avoids accidentally having two different commits that purport to be the same non-SNAPSHOT version (since release versions must be unique and immutable).

Prerequisites:

  • Your project needs to be tracked by a Jenkins job that will build and deploy Maven artifacts in response to changes on GitHub. If you aren't sure if such a job exists or not for your project, send a mail to the ImageJ-devel list.
  • Familiarize yourself with the concept of Maven, and in particular the idea of SNAPSHOT and release artifacts. When Jenkins deploys to Maven, it is actually uploading to a separate release or SNAPSHOT repository based on the version defined in the project's pom.xml. SNAPSHOTs can be changed; releases cannot.
  • Familiarize yourself with the Git tagging process. You will need to tag a commit and push the tag to a remote repository for the release process. Although Git tags can be changed or deleted over time, the convention of creating a tag to match each release creates a nice "bookmark" system for developers when they check out the code.

Steps to release:

  1. Make a commit on master that changes the version of your project to a non-SNAPSHOT.
  2. Push your commit to GitHub to trigger the deployment of a Maven release.
  3. Tag the release commit with an appropriate name. Core projects use a standard "component name"-"version" naming scheme, e.g. mysoftware-1.2.3.
  4. Make a second commit incrementing the version to the next SNAPSHOT version.
  5. Push again to GitHub—both the SNAPSHOT-version master branch and the release tag.

The disadvantage to this method is that, since the steps are not automated, there is room for error and inconsistency—not pushing the tag, not making a commit go back to a SNAPSHOT version, etc.



Phase 4: Uploaded

What are ImageJ update sites?

ImageJ update sites are what ImageJ actually queries to download updates. These update sites are versioned, but do not rely on other tools (e.g., Git or Maven) in order to function. Rather, component developers upload new versions of their component(s) using the ImageJ Updater, which makes them available to end users. Typically, update sites are available as web sites via HTTP, with uploads functioning via WebDAV or SSH/SFTP/SCP.

Deploying to the Maven repository creates a stable release artifact of a software component, but for ImageJ-related components, that alone does not put it into the hands of users. To do that, the component must then be uploaded to an ImageJ update site.

ImageJ and Fiji update sites

For core projects, there is an additional layer tying User-facing and Developer-facing components together: the Bill of Materials (BOM). To ensure users and developers see the same functionality, components should only be uploaded to the core update sites when their version is also updated in the corresponding BOM.

When you have a new version of your plugin to release, you should follow the External developer instructions for contributing to corresponding BOM (i.e. pom-fiji in this case). By submitting a pull request that simply modifies the managed version of your component, you will signal to the core Fiji developers that your project is ready for upload.

External update sites

An update site can be hosted anywhere, though the ImageJ web server at http://sites.imagej.net/ offers a personal update site service.

See the distribution page for a discussion of pros and cons of distributing your plugin on a core versus a personal update site.

If you do manage your own update site, you can upload your release yourself.

See the documentation on update sites for further instructions.

Further reading

  • The SciJava versioning guidelines will help you choose appropriate version numbers for your software when performing Maven releases.