[ImageJ-devel] [fiji-devel] Imglib2: using threadpools in core algorithms

[Johannes Schindelin]

Hi Tobi,

On Fri, 6 Dec 2013, Tobias Pietzsch wrote:

> On Dec 5, 2013, at 11:56 PM, Curtis Rueden <ctrueden at wisc.edu> wrote:
> 
> > > * Encapsulates internal dependencies such as java.util.concurrent
> > > (e.g., Avian doesn't have that package!).
> > 
> > And Android / J2ME doesn't have java.util.concurrent either. I am
> > interested in porting ImageJ2 to Android (which would necessitate a
> > port of ImgLib2 to Android). The more "weird" Java packages we use,
> > the more involved it will be to refactor those usages out later to
> > make such a thing possible. The "swappability" of SJC services will
> > obviate the need to do it in some cases.
> 
> On a closer look, ThreadService uses java.util.concurrent.Callable and
> import java.util.concurrent.Future.

Yes, they do. At least Callable is so simple that it does not make sense
whatsoever to reimplement it.

> It also extends java.util.concurrent.ThreadFactory. So the Android/AVIAN
> argument is not really valid.

Whoa, slow! The *more* you take from the concurrent package, the *harder*
it gets to support Android/Avian. It's not "take it or leave it". So
Curtis' argument is a very valid one! Just because you use two interfaces
that are easily provided in a support library does not mean that you could
just as easily buy into the complete concurrent package, because that one
is *not* easily provided in a support library!

> To reiterate my previous question why not extend the ExecutorService
> interface instead?
>
> One answer I could give to that myself is that we do not need/want the
> full ExecutorService, e.g., an imglib algorithm should not be allowed to
> shutdown() the ExecutorService.

There you go.

> On the other hand, we do not need/want the full ThreadService, e.g.
> using newThread() from the ThreadFactory is not wanted.

In ImgLib2 core, no you do not need that right now. But scijava-common is
about striking a balance between what ImgLib2 needs and what other
scientific Java projects need.

Personally, I have a strong faith in ImgLib2 being able to cope with the
ThreadService providing the newThread() method and just not use it.

I also have a strong faith that both Lee with his SingleThreadService and
myself with whatever I have to implement to support the ThreadService in
Avian (if ever needed, really) will have a much smaller problem
implementing ThreadService than a full-fledged ExecutorService.

> So could we maybe extract the part of ExecutorService that we are
> interested in (invokeAll, invokeAny, and submit variants) into a new
> interface and make ThreadService extend that?

The idea of ThreadService (it being an interface already) was to *be* that
new interface.

> This would still not allow us to pass a standard ExecutorService into
> imglib algorithms but a ThreadService. I could live with it.

And again, scijava-common already has a default implementation using the
ExecutorService (which we will have to override in the case of Android or
Avian).

> The question then is whether this should live in imglib, in
> scijava-common, or somewhere else. I hesitate making scijava-common a
> dependency of imglib.

Yes, you made that point earlier.

Personally, I am less and less convinced that it is a good idea to go out
of our way to make ImgLib2-core a kitchen sink. It is about
multi-dimensional data processing, after all, not about common
functionality to scientific Java projects.

The name scijava-common was chosen to reflect the desire to put code used
commonly in scientific Java projects into that library rather than
reinvent the wheel in every scientific Java project.

And it is understandable that the same scientists who would not use
Bio-Formats for reading and writing AVIs (because the name suggests that
Bio-Formats is intended for biologists, and they are physicists, after
all) would refuse to link to ImgLib2-core for the useful stuff unrelated
to image processing because they do not want to process images.

Again, that is why scijava-common was started, and we put quite useful
functionality into it that all arose from the Fiji/ImageJ2 projects but is
of wider interest than just image processing, at the same time managing to
keep the library small and sweet.

> I know just having scijava-common as a dependency does not force me to
> use the application container but still… Making imglib a dependency of
> scijava-common is not a good idea either.

Exactly. I think it is a wise idea to trust Curtis on the separation of
concerns because he has spent literally the past four years on that
subject.

In particular, your intuition that imglib2-core should not be a dependency
of scijava-common makes absolute sense from the point of view that modules
need to be separated by virtue of their particular focus: the
scijava-common library's focus being quite obviously *the* base for
scientific Java programming, right?

Ciao,
Dscho