Big Data Support

SNTv5 implemented preliminary support for big data: images too large to fit in CPU/GPU memory (BDV XML/HDF5, N5, and OME-Zarr/OME-NGFF containers) can be opened virtually and browsed or reconstructed without ever loading the full volume. Most of this support is provided through Big Data Viewer and Big Volume Viewer. The following operations are currently supported:

Operation	Status	Comments
Manual and semi-automated tracing	Supported (experimental)1	Multi-dimensional images have not been thoroughly tested
Automated tracing	Supported (experimental)	See Auto-tracing1,2
Tracing along way-points	Supported
Path optimization	Supported
3D Visualization: Interactive re-slicing browser	Supported via BigDataViewer
3D Visualization: Interactive volume viewer	Supported via BigVolumeViewer and sciview	Includes Slab View for restricting rendering to a thin, adjustable region of the volume
Multidimensional operations	Limited support	Tracing handles one active channel/frame at a time for performance reasons. Commands that require exposing all channels/frames to be available at once may not be supported

¹For collaborative tracing, please use HortaCloud: This is SNT’s development team recommended tool for tracing Terabyte-size datasets in a collaborative environment

²See PySNT

Getting Started

Choose Big Data… from the Shortcuts Window. In the prompt you can specify:

• Main volume: The primary image volume. Supported files include: standard files, IMS, BDV XML/HDF5, N5, and OME-Zarr/OME-NGFF containers

• Secondary volume: Optional image volume. Useful for loading a processed version of the main volume, or a secondary channel saved as a separated file

• Reconstruction file(s): Optional. Either a single file (TRACES, SWC, JSON, etc.), or a folder containing multiple reconstruction files. Reconstructions can also be imported later on using the SNT Controls toolbar

• Markers file: Optional. A CSV file containing bookmarked locations. Markers can also be imported later on using the SNT Controls toolbar

• Viewer type: Defines the viewer type. Either:

  • Big Data Viewer (BDV): Interactive reslicing: Interactive re-slicing browser
  • Big Volume Viewer (BVV): 3D rendering: Interactive volume viewer, no tracing
  • Big Volume Viewer (BVV): 3D rendering w/ tracing capabilities: Same as Big Volume Viewer (BVV): 3D rendering, but a new SNT instance is initialized as SNT Stream, turning the BVV viewer into a functional tracing canvas
  • Big Data Viewer (BDV): Interactive reslicing w/ tracing capabilities: A fallback option to the previous choice for when no discrete graphics card is available, or SNT is running on hardware without GPU support

Limitations and Idiosyncrasies

• One channel/frame at a time: for performance reasons, one channel/frame is streamed for tracing at once. Commands that require all channels/frames available simultaneously (e.g., Multispectral Refinement) are not supported
• Fixed search algorithm: the A* search algorithm is used for interactive tracing; the alternate NBA*/Fast Marching options available for in-core images may not be offered in Stream mode
• Image statistics: Access to cost-function tuning and image-statistics options. Compute Real-Time (the default) recalculates local statistics as you trace new regions. Compute Now For Whole Image scans the entire streamed dataset once; because this can be slow depending on dataset size and network/disk speed
• 3D Scene recentering on click: whether/how the view recenters as you place tracing points is configurable from the toolbar’s options menu (Never / Adaptive / Always). Adaptive is the recommended default for streamed data: it skips recentering for clicks that already land close to the focal plane, and briefly gates subsequent clicks after a recenter fires, giving tile streaming time to catch up
• Path fitting and automated tracing: Supported the same way as for in-core images; see Auto-tracing and Path optimization

Slab View

Slab View restricts BVV’s rendering to a thin, adjustable region of the volume, making it easier to inspect internal structure in dense or large datasets without having to rotate away obstructing tissue. Also, it may reduce data fetched from remote sources, depending on BVV’s internal caching behavior. It is a general rendering feature, available in Scene Controls whenever a BVV canvas is open, even outside active SNT Stream tracing sessions.

• Controls: toggle with the Slab View button; a Position slider moves the slab through the volume, and a Thickness spinner controls how much of the volume is included around that position
• Orientation: Slab View must be activated from one of the three canonical planes (XY/XZ/ZY), needed to give the slab an initial axis to work from. Once active, the view can be freely rotated; Slab View only turns itself off if the volume genuinely rotates out of the visible slab
• Slab Paths / Slab Annotations: optional toggles that restrict rendered reconstructions/markers to the same slab window, so paths outside the slab don’t clutter the view. Slab Paths’ on/off state is remembered as a preference across Slab View activations; Slab Annotations resets each time

Supported File Formats

Big volume viewer and Big data viewer are at the core of Big-data support. Standard formats (TIFF, and anything readable via Bio-Formats) are loaded as regular in-memory volumes, with an option to downsample automatically if a volume exceeds the GPU’s 3D texture limits. Imaris (.ims) files are supported via an automatically generated BDV-XML sidecar written next to the original file.

For genuinely large datasets, BDV XML/HDF5, N5, and OME-Zarr/OME-NGFF containers are opened virtually: data is streamed from disk in blocks as needed. N5 and Zarr containers are read directly, with no XML sidecar required.