[ImageJ-devel] [fiji-devel] slow opening of tif files using scifio-based ImgOpener

[Tobias Pietzsch]

Hi Curtis,

Ok, I tried the things you suggested as well as the setGroupFiles( false ) suggested by Lee.
Here are the results:

baseline (unmodified ImgOpener):
loading 50 tif images using ImageJ, 10000 other tif files in same directory
median: 141 ms
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 6953 ms

with setAllowOpenFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 5972 ms

with setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 2331 ms

with both,  setAllowOpenFiles( false ) and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 1070 ms

with TiffReader:
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 5061 ms

with TiffReader and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 10000 other tif files in same directory
median: 568 ms




For comparison, the case where there are no other files in the directory
baseline (unmodified ImgOpener):
loading 50 tif images using ImageJ, 0 other tif files in same directory
median: 140 ms
loading 50 tif images using ImgOpener, 0 other tif files in same directory
median: 1804 ms


with TiffReader and setGroupFiles( false ):
loading 50 tif images using ImgOpener, 0 other tif files in same directory
median: 552 ms

That's still 4x slower than ImageJ1 but already getting a lot closer. Nice!

best regards,
Tobias


On Feb 6, 2013, at 6:35 PM, Curtis Rueden wrote:

> Hi all,
> 
> > Is there a way to turn this off?
> 
> Yes, you can call imageOpener.setAllowOpenFiles(false) which will prevent Bio-Formats from doing any directory listings or opening up any files for the purposes of determining the file format. However, if you set this flag, some file formats will be improperly identified.
> 
> Out of curiosity: how much does the benchmark improve if that flag is set inside ImgOpener's createReader method? Another thing you could test, to determine the performance impact, is to change "new ImageReader" to "new TiffReader", which explicitly tells ImgOpener to assume everything is a TIFF.
> 
> If such changes vastly improve the ImgOpener performance, we could consider adding another ImgOpener signature that lets you explicitly specify the file format.
> 
> > Bio-formats analyzes the folder. Some formats are multi-file.
> 
> Indeed. And some files are even considered a different file format depending on which other files are present in the same or nearby directories.
> 
> Regards,
> Curtis
> 
> P.S. to Melissa: Johannes has some ideas for improving the TIFF reader performance too, which he plans to explore over the next few days. We will keep you posted on any progress.
> 
> 
> On Wed, Feb 6, 2013 at 11:13 AM, Albert Cardona <sapristi at gmail.com> wrote:
> 2013/2/6 Tobias Pietzsch <pietzsch at mpi-cbg.de>
> Hi Melissa,
> 
> That is awesome! Thanks a lot for tackling these problems.
> 
> Out of curiosity, could you briefly explain (if there is an easy explanation) why the 100.000-files-issue happens?
> Is bioformats analyzing the directory the TIFF file lives in? Is there a way to turn this off?
> 
> 
> Yes, Bio-formats analyzes the folder. Some formats are multi-file.
> 
> Albert
> 
> -- 
> http://albert.rierol.net
> http://www.ini.uzh.ch/~acardona/
> 
> -- 
> -- 
> Please avoid top-posting, and please make sure to reply-to-all!
>  
> Mailing list web interface: http://groups.google.com/group/fiji-devel
>  
> --- 
> You received this message because you are subscribed to the Google Groups "Fiji-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fiji-devel+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
>