Bug reporting best practices

Revision as of 06:18, 8 August 2014 by Hinerm (talk | contribs) (Added bug reporting template)


TL;DR Summary

Why put effort into bug reports

For light reading, there are numerous guides and essays on how and why to write excellent bug reports. ImageJ users should be aware that the development community is diverse: from publicly-funded individuals and teams to scientists and user contributors. For a largely open-source community like this, there are several key points to consider when submitting a bug:

  • Teams developing open-source code are typically smaller and lack dedicated testing teams. Developers thus rely on an active and vocal community to provide feedback and guide the development process by identifying the areas in need of attention (complaint-driven development).
  • If you encounter a bug, it is likely interfering with your desired workflow and needs to be resolved quickly. The better the bug report, the faster a developer will be able to reproduce and address the issue. Poorly written bug reports are more likely to sit unanswered - not because developers see the issue as unimportant, rather that the time required to clarify the bug report itself presents a significant barrier when setting priorities within an overflowing schedule.
  • When you find a bug, it is unlikely that you are the only individual affected by it. By reporting a bug in a way that developers can understand, identify and resolve the issue, you are performing a necessary and valuable service to the entire ImageJ community.

Components of a complete bug report

There are three critical components in an ImageJ bug report. If a report is missing any of these components, its usefulness may be limited.

Environment information

ImageJ is a flexible and extensible platform, so the actual "ImageJ environment" can vary from user to user. A common misunderstanding in bug reports is to just report the version of the base ImageJ component (e.g. 1.49e). This is helpful, but says nothing about what plugins, update sites, etc... are in use.

Errors can appear in any component of the software, and in some cases two plugins might work individually but have some negative interaction with each other. Thus a complete view of the ImageJ environment is imperative. Fiji currently ships with a Report a Bug command; this will automatically populate the complete environment information and include it in the bugzilla report; this bug-reporting functionality will be migrated to core ImageJ soon.

Note that if you file a bug manually or are enquiring to a mailing list, you can still copy and paste the contents of the "Useful information about your system" text box from the Report a Bug dialog. It will be greatly appreciated.

Minimal and precise steps to reproduce

When we're in a hurry, it's easy to provide a brief overview a bug without actually describing how to reproduce the error. We are also prone to providing too much information, which can confuse the issue and discourage thorough reading.

The actual text of your bug report should succinctly describe the fewest steps possible to reproduce your issue. For example:

1) Open sample image "blobs"
2) Run auto-threshold command
3) Run subtract background command
At this point, an evil kraken appears and sinks my hard drive.

Additional information is typically unnecessary... if a developer can reproduce the problem, they will do their best to fix it.

Sample data

Developers typically have a cache of sample data for testing their application. That said, we are still striving to reproduce the original environment of the error. Having the original image that caused the error is the best possible way to test.

If your test data is small and public, you can typically attach it to the bug report.

If you are using a Fiji distribution, the easiest and safest way to provide sample data is via the Help>Upload Sample Image command. This sends the image to a website restricted to Fiji wiki users with explicit authentication (reserved for trusted developers). This allows proprietary data to be tested without fear of public distribution.

Note: if you do upload sample data via Fiji, you should mention the name of the dataset in your bug report. This will help developers find your image(s) quickly.

For bugs working with excessively large datasets, you may need to communicate with a developer to determine the best way to test on your data.

While you're waiting...

If you have encountered and reported a bug that is completely blocking your work, you still have options available while waiting for the issue to be resolved.

Disable SCIFIO

In recent months we have begun to provide an alternative to the hard-coded case logic of ImageJ 1.x's image I/O to a new, highly extensible paradigm: ImageJ2. It represents the next steps in scientific image analysis, but due to the vast scope of the overhaul there are inevitably issues remaining. For example, if you run into an issue of the style where a dataset that used to open correctly for you is broken after updating, you can check if you have a Edit>Options>ImageJ2 command and uncheck the option to use SCIFIO. This will allow you to revert to the class image I/O until the new I/O is fixed or improved.

Note: even if disabling SCIFIO fixes the issue for you, please still report the discovered bug. The long-term vision for ImageJ is to migrate completely to the new image I/O paradigm, so if there are problems we need to know about them.

Catastrophic failure

We maintain lifeline ImageJ distributions on the Downloads page. You can use these until any outstanding issues are resolved.