Development Lifecycle



Development
Topics
Overview
Philosophy
Architecture
Source code
Project management
Coding style
Debugging
Tools
GitHub
Git
Maven
IDEs
Jenkins
Travis
Dotfiles
Guides
Writing plugins
ImageJ Ops
Contributing to a plugin
Distributing your plugins
Development lifecycle
Building a POM
Developing with Eclipse
Hands-on debugging
Adding new ops
Adding new formats
Using native libraries
Tips for developers
Tips for C++ developers
ImageJ 1.x plugins
Versioning
Logging
Uber-JARs

The SciJava philosophy is to release early, release often. At the same time, we always want to preserve scientific reproducibility. To make this possible we lean on several project management tools. The purpose of this guide is to take you through the process of using these tools with the goal of releasing new versions of your software, and then providing those releases to users.

Phases of development

ImageJ and Fiji are developed according to the SciJava philosophy, thus these applications are used throughout this tutorial to illustrate the development lifecycle.

Whether adding new features, fixing bugs, improving performance, etc... development is the process of making changes, with the goal of exposing these changes to users. To accomplish this, actively developed projects cycle through four general "phases":

What are Maven artifacts?

Artifacts are files, most commonly a JAR encapsulating the compiled classes for a component. Other files that may be produced as artifacts include:
  • The project's POM
  • A jar with the original source files
  • A jar with any generated javadoc
  • A jar with any test files
  1. In development. The source code is modified to add new features, fix bugs, etc... these modifications are expressed as commits by Git, whether on your local filesystem, a topic branch, or a repository fork.
  2. On master. When you have a set of one or more commits that you are happy with (i.e. the feature is complete, or the bug is fixed) they are moved to the master branch of the project's repository on GitHub. This ensures the master branch is always release ready.
  3. Released. When there is a need to make the current master branch public (i.e. it has a critical bug fix or cool new feature that users have requested) Maven is used to cut a release, which is then deployed as an artifact to the ImageJ Maven repository. Developers can now use the new version in their own projects.
  4. Uploaded. Once we've tested our latest release artifact to ensure it doesn't conflict with other components, it can be uploaded to an ImageJ update site - making it available to end users.

The following sections will discuss these phases, and their associated tools and workflows, in more depth.

Relationship with Maven SNAPSHOTs

Another way of thinking about the development cycle is through the Maven version number given associated with the code. The idea behind reproducible builds is that from a given version of a plugin, the state of the code producing that version can be determined unambiguously. Typically, that state is determined by a unique Git commit. However, it would be impractical and unrealistic to change the Maven version with every single Git commit.

This is why SNAPSHOT versions are used while "in development" (phases 1 and 2 - "SNAPSHOT coupling"). Using a SNAPSHOT version is saying "no guarantees are made as to the reproducibility of this artifact." For this reason, to best facilitate reproducible science, SNAPSHOT versions of code are not provided to users (except potentially for testing).

To provide users with an updated version of an artifact (phases 3 and 4) the version is changed to a unique, non-SNAPSHOT, version for a single Git commit. Then the next commit returns to SNAPSHOT versioning for further development. Thus the cycle repeats.

Phases in-depth

When to use a topic branch?

Core SciJava components employ a "release ready master branch" approach:
  • The tip of the master branch is always stable enough to be released, "as good or better" than the state of its last release.
  • Each commit on the master branch should compile with passing tests. This has several advantages—e.g., better bisect-style debugging.

Topic branches are great for isolating potentially disruptive and/or unfinished changes from the master branch, so that it always remains release ready. However, pushing directly to master has a huge time savings over filing a PR and awaiting review for days, weeks or months. Getting changes onto master quickly has many advantages:

  • Fewer conflicts. It avoids conflicts between multiple long-running topic branches.
  • SNAPSHOT builds. Jenkins builds the change into the latest SNAPSHOT build, making it available from the ImageJ Maven repository.
  • Faster support. Supporting the community is less convoluted, with changes released to users more rapidly. Yes, you can link to changes on a topic branch. And yes, you can upload binary builds from that branch. But each extra manual step costs time—better to link directly to the latest SNAPSHOT build. There are even ImageJ update sites which serve the latest builds from master, to make it easier for non-technical users to test changes.
  • Less complex. The more topic branches you have—and in particular, the more integration branches you have—the more complex the system becomes, the more supporting tooling, CI jobs, etc. are needed. And the more developer time is needed to maintain the tooling, sort through topic branches, keep track of open PRs... leaving less time for developing new features and fixing bugs.

Hence, when exactly to use a topic branch is a judgment call, but some good times to use a topic branch are:

  • Breaking. The changes break backwards compatibility.
  • New API. The changes introduce significant new API which will need to remain backwards compatible in the future, and review is desired before committing to that API.
  • Unfinished. The changes are unfinished.
  • Regressing. The changes leave the codebase in a "worse" state somehow.
  • Discussion. To solicit discussion from the community, especially if the changes might be contentious.

Conversely, some situations to push directly to master:

  • Correct. Bug-fixes where the developer is confident the fix is correct.
  • No new API. Small new additions which do not introduce significant future maintenance burden.
  • Unstable. Changes to unstable or experimental components still in their "incubation" period of development (i.e., versioned at 0.x), since there is no promise of backwards compatibility.
  • Unsupported. Changes to "unsupported" components which make no guarantee of backwards compatibility.
Lastly, keep in mind that SciJava favors the release early, release often style of development, to maximize iterations of community feedback. Just because a change makes it to the master branch, does not mean it is set in stone: if a problem is later found, the change can be amended or reverted as quickly as it was added—easy come, easy go.

Phase 1: In development

Repositories on GitHub are referred to as remotes; when you clone, or check out, a remote you get a local copy of the repository. Development progresses by making changes to your local copy and pushing them back to the remote. GitHub provides tools for controlling user permission levels for each remote repository, therefore how you develop a project depends on whether you are able to push (write) to that project's remote repository or not.

  • Collaborating developer. If you have permission to push directly to the project's remote repository, then you can simply use Git and GitHub to clone the repository and make your changes. For non-trivial changes, you will typically create a topic branch to develop and test the changes. This also provides a forum for discussion and review with your fellow developers.
  • External developer. If you do not have push rights, then you need to go through an additional step called Forking the repository. This will create a remote copy of the repository, to which you have push rights. Your remote fork is referred to as downstream of the original remote repository (which is upstream of your fork). Your development will then take place on your fork, with an additional step later to reconcile with the upstream repository.

Phase 2: On master

Once a feature or fix is complete it can move to the master branch of the repository. How you accomplish this depends on how the changes were developed in Phase 1.

  • Collaborating developer. Minimal changes can be pushed back directly to master on the remote repository. If your work is on a topic branch then you should use a pull request (PR) so that the topic branch can be reviewed before being merged to master.
  • External developer. First push your changes back to a branch of your forked repository (it doesn't necessarily have to be master). Then you can file a pull request (PR) on GitHub to merge your branch back to the official repository. This invites code review from other interested developers. Reviewers might ask for changes to the code to address any issues. After any needed revisions have been made, a project maintainer will accept your changes and then merge to the official master branch.

Phase 3: Released

Note: This step can only be performed by a project maintainer.

Once the master branch of a component has your desired new functionality, the next step is to cut a release version of the component. Normally, the Maven version (in the pom.xml) on master is a SNAPSHOT version, meaning it is unstable and not yet released. However, a stable release artifact can be deployed to the ImageJ Maven repository in one of three methods, described below. As a rule of thumb, SciJava developers currently use:

Method 1: Release-Version Jenkins job

The special Release-Version job of Jenkins (which is visible only to logged in Jenkins users) is triggered to release a particular artifact. There are several advantages to this approach:

  • A tag is created on GitHub to easily reference the release commit
  • The "bump to next release cycle" commit is done automatically
  • The Maven POM references the correct tag rather than HEAD
  • This job is capable of releasing certain artifacts to OSS Sonatype (and hence to Maven Central) as appropriate

Prerequisites:

  • You will need an ImageJ Jenkins account. If you haven't set this up yet, contact the ImageJ-devel list and ask for an account. A maintainer will send you instructions.

Steps to release:

  1. Visit the Release-Version Jenkins job web page.
  2. Click "Build with Parameters" from the left-hand menu.
  3. Select your component from the dropdown list.
  4. Enter the commit hash corresponding to the most recent commit of the master branch.
  5. Enter the desired version number of the release.
  6. Click the Build button.
  7. Grab a coffee.

Jenkins will take care of all the steps, including pushing the relevant commits and tags, building the code, and deploying the resultant artifact to the ImageJ Maven repository.

Method 2: release-version.sh

The release-version.sh script is what the Release-Version Jenkins job uses under the hood to perform releases. It relies on the Maven Release plugin to do most of the heavy lifting, but also does some extra work (e.g., to ensure releases are deployed to the correct repository).

Prerequisites:

  • Install the release-version.sh script. The best way to do this is to clone the complete SciJava-scripts repository. That will give you access to other useful scripts and help keep them all up to date.
  • (optional) If you want to easily use these scripts from any directory, you can add the scijava-scripts folder to your PATH.
  • You will need an account for maven.imagej.net and the local Maven configuration to deploy to this repository. If you haven't set this up yet, contact the ImageJ developers on the ImageJ Forum to ask for an account. Then follow this guide to configure your settings.xml and settings-security.xml appropriately on your local machine. In particular, you will need the following block in settings.xml:
    	<servers>
    		<server>
    			<id>imagej.releases</id>
    			<username>{your Nexus username}</username>
    			<password>{your Nexus password hash}</password>
    		</server>
    		<server>
    			<id>imagej.snapshots</id>
    			<username>{your username}</username>
    			<password>{your Nexus password hash}</password>
    		</server>
    	</servers>
    

Steps to release:

  1. From your project's directory, simply run:
   release-version.sh <NEW_VERSION>



Method 3: Double push to master

Historically, the "double push to master" approach was the recommended way to release—but it has now been replaced by the other two approaches. Nonetheless, there are a few components of Fiji that still use it to do their releases. A push to master happens with the POM version set to a non-SNAPSHOT—e.g., this Trainable_Segmentation commit releasing version 2.2.1. A second push to master is then typically done to "bump to next release cycle" after Jenkins starts building the release—e.g., this Trainable_Segmentation commit bumps to 2.2.2-SNAPSHOT. This second push avoids accidentally having two different commits that purport to be the same non-SNAPSHOT version (since release versions must be unique and immutable).

Prerequisites:

  • The repository for your project needs to be linked with a continuous integration tool that will automatically build and deploy Maven artifacts in response to changes on GitHub, e.g. using Travis or Jenkins. If you're not sure if your project has this automation, contact us and we'll help you find out.
  • Familiarize yourself with the concept of Maven, and in particular the idea of SNAPSHOT and release artifacts. When Jenkins deploys to Maven, it is actually uploading to a separate release or SNAPSHOT repository based on the version defined in the project's pom.xml. SNAPSHOTs can be changed; releases cannot.
  • Familiarize yourself with the Git tagging process. You will need to tag a commit and push the tag to a remote repository for the release process. Although Git tags can be changed or deleted over time, the convention of creating a tag to match each release creates a nice "bookmark" system for developers when they check out the code.


Steps to release:

  1. Make a commit on master that changes the version of your project to a non-SNAPSHOT.
  2. Push your commit to GitHub to trigger the deployment of a Maven release.
  3. Tag the release commit with an appropriate name. Core projects use a standard "component name"-"version" naming scheme, e.g. mysoftware-1.2.3.
  4. Make a second commit incrementing the version to the next SNAPSHOT version.
  5. Push again to GitHub—both the SNAPSHOT-version master branch and the release tag.

The disadvantage to this method is that, since the steps are not automated, there is room for error and inconsistency—not pushing the tag, not making a commit go back to a SNAPSHOT version, etc.



Phase 4: Uploaded

What are ImageJ update sites?

ImageJ update sites are what ImageJ actually queries to download updates. These update sites are versioned, but do not rely on other tools (e.g., Git or Maven) in order to function. Rather, component developers upload new versions of their component(s) using the ImageJ Updater, which makes them available to end users. Typically, update sites are available as web sites via HTTP, with uploads functioning via WebDAV or SSH/SFTP/SCP.

Deploying to the Maven repository creates a stable release artifact of a software component, but for ImageJ-related components, that alone does not put it into the hands of users. To do that, the component must then be uploaded to an ImageJ update site.

ImageJ and Fiji update sites

For core projects, there is an additional layer tying User-facing and Developer-facing components together: the Bill of Materials (BOM). To ensure users and developers see the same functionality, components should only be uploaded to the core update sites when their version is also updated in the corresponding BOM.

When you have a new version of your plugin to release, you should follow the External developer instructions for contributing to corresponding BOM (i.e. pom-fiji in this case). By submitting a pull request that simply modifies the managed version of your component, you will signal to the core Fiji developers that your project is ready for upload.

External update sites

An update site can be hosted anywhere, though the ImageJ web server at http://sites.imagej.net/ offers a personal update site service.

See the distribution page for a discussion of pros and cons of distributing your plugin on a core versus a personal update site.

If you do manage your own update site, you can upload your release yourself.

See the documentation on update sites for further instructions.

Further reading

  • The SciJava versioning guidelines will help you choose appropriate version numbers for your software when performing Maven releases.