Difference between revisions of "Clojure Scripting"

(details on how to use the interpreter)
(underscore in name)
Line 17: Line 17:
To create scripts, just save them as .clj text files in any folder or subfolder of Fiji's plugins folder, and run "Plugins - Scripting - Refresh Clojure Scripts" to update the menus (it's done automatically at start up as well).
To create scripts, just save them as .clj text files (with an underscore in the name) in any folder or subfolder of Fiji's plugins folder, and run "Plugins - Scripting - Refresh Clojure Scripts" to update the menus (it's done automatically at start up as well).
To edit a script, just edit and save it with your favorite text editor.
To edit a script, just edit and save it with your favorite text editor.

Revision as of 09:29, 8 February 2009

Clojure tutorial for ImageJ

Check out clojure web site and particularly the chapter on Java interoperability.

Clojure is not a scripting language: Clojure compiles directly to JVM bytecode, and thus runs at native speed. Thus one must think of Clojure as a true alternative to Java the language, but much more expressive, flexible and powerful.

Using Clojure inside Fiji

Go to "Plugins - Scripting - Clojure Interpreter". The prompt accepts any clojure code.

See Scripting Help for details on keybindings and how to use the interpreter. Control+) will add all necessary ending parenthesis.

A minimal, complete clojure example:

(import '(ij IJ))
(def gold (IJ/openImage "http://rsb.info.nih.gov/ij/images/AuPbSn40.jpg"))
(.show gold)

To create scripts, just save them as .clj text files (with an underscore in the name) in any folder or subfolder of Fiji's plugins folder, and run "Plugins - Scripting - Refresh Clojure Scripts" to update the menus (it's done automatically at start up as well).

To edit a script, just edit and save it with your favorite text editor.

To execute a script, do any of:

  • Select it from the plugins menus.
  • Type 'l' (L), start typing its name, push the down arrow and then return to execute it.
  • If it was the last executed command, just type shift+r (shortcut to "Process - Repeat Command").

The script is always read directly from the source file, so no updating of menus is needed (unless its file name changes).

Language basics

A ';' defines the start of a comment, just like '//' does in Java.

A function definition declares parameters within [].

Local variables are declared with let, and global variables with def.

Functions are defined with defn, and are visible globally. Hence a function declared within a let statement has access to variables declared in it. This method enables closures.

Calling methods and variables on a java object

There are two ways, the second syntactic sugar of the first. Below, imp is a pointer to an ImagePlus:

; java-ish way:
(. imp (getProcessor))

; shorter java-ish way:
(. imp getProcessor)

; lisp-ish way:
(.getProcessor imp)

To call a method on an object returned by a method call, there is a simplified way:

; double way:
(. (. imp (getProcessor)) (getPixels))

; simplified double way:
(.. imp (getProcessor) (getPixels))

; super simplified (less parenthesis than java!)
(.. imp getProcessor getPixels)

; or lisp-ish way:
(.getPixels (.getProcessor imp))

Any number of concatenated method calls, both for static methods or for instances:

; Concatenated call to static and instance methods
(.. WindowManager getCurrentImage getProcessor getPixels)

To call a variable or 'field', just do it like a method call but without the parentheses:

(. imp changes)

or more lisp-ish:

(.changes imp)

To enhance readability, use import when appropriate. Imports remain visible throughout the current namespace:

(import '(java.awt Color Rectangle)
        '(ij.plugin.filter PlugInFilter))

(new Rectangle 0 0 500 500)
; It's the same as:
(Rectangle. 0 0 500 500)

; A static field, call like a namespace:

Choose whatever matches your mental schemes best.

Calling static fields and methods: namespace syntax

To call a static field or method, use namespace syntax:

(println "does all: " ij.plugin.filter.PlugInFilter/DOES_ALL)

(ij.IJ/log "Some logged text")

Above, notice how a class name is used instead of a pointer to call static fields and methods. Static fields and methods are just variables and functions that exist within the namespace of the class in which they are declared. Hence Clojure's namespace syntax makes way more sense than java code, that doesn't do such distinction and allows for loads of confusion (java allows invoking static methods and fields using a pointer to an instance of the class in which such static methods and fields are declared).

Defining variables: obtaining the current image

As a local variable imp declared within a let statement:

(let [imp (ij.WindowManager/getCurrentImage)]
    ; print its name
    (println (.getTitle imp)))

; Variable imp NOT visible from outside let statement:
(println (.getTitle imp))
---> ERROR

As a general variable visible from the entire namespace:

(def *imp* (ij.WindowManager/getCurrentImage))

(println (.getTitle *imp*))

By convention, in lisp global variables are written with asterisks in the name.

A let statement lets you declare any number of paired variable name / values, even referring to each other in sequence:

(let [imp (ij.WindowManager/getCurrentImage)
      ip (.getProcessor imp)
      pix (.getPixels ip)
      pix2 (.getPixels (.getProcessor imp))]
    ; do some processing ...
    (println (str "number of pixels: " (count pix))))

Any number of let statements may be nested together:

(let [imp (ij.WindowManager/getCurrentImage)]
    ; do whatever processing here
    (let [ip (.getProcessor imp]
        ; print first pixel
	(println (str (.getPixel ip 0 0))))))

Creating objects: invoking constructors

A constructor is invoked by adding a dot '.' to the name of a class, followed by the arguments. Below, we create an ImageProcessor and then an ImagePlus with it, and finally we print the ImagePlus, which invokes toString() on it (like in java):

(let [ip (ij.process.ByteProcessor. 400 400)
      imp (ij.ImagePlus. "my image" ip)]
  (println imp))

An alternative syntax is to use the java-like new keyword, but it's unnecessarily verbose:

(let [ip (new ij.process.ByteProcessor 400 400)
      imp (new ij.ImagePlus "my image" ip)]
  (println imp))

Defining a closure

In the following a function is declared within the scope of the local variable rand, which contains an instance of java.util.Random. All calls to the function rand-double will use the same random number generator instance with seed 69997.

The dotimes loop will then print 10 different pseudo-random numbers. If the rand was a new Random with seed 69997 every time, all 10 numbers would be exactly the same.

You can think of a function inside a closure as a static function using a static variable (in Java), but it's more than that, since the function will be able to access parameters on the global namespace and also in any other local namespace in which the let is declared. For example, another let, or even another defn!

(let [rand (java.util.Random. 69997)]
    (defn rand-double []
        (.nextDouble rand)))

(dotimes [i 10]
    (println (rand-double)))

Above, note the dot '.' after Random, which indicates we are calling the constructor (with a single parameter 69997, the pseudorandom generator seed to be used). Alternatively, one may use the java-like syntax: (new java.util.Random 69997) -- note the absence of a dot now.

Looping an array of pixels

For example, to find the min and max values:

; Obtain the pixels array from the current image
(let [imp (ij.WindowManager/getCurrentImage)
      pixels (.. imp (getProcessor) (getPixels))
      min (apply min pixels)
      max (apply max pixels)]
    (println (str "min: " min ", max: " max)))

The above code does not explicitly loop the pixels: it simply applies a function to an array.

To loop pixels one by one, use any of the following:

(let [imp (ij.WindowManager/getCurrentImage)
      pixels (.. imp (getProcessor) (getPixels))]

      ; First loop with "dotimes"
      (dotimes [i (count pixels)]
          (println (aget pixels i)))

      ; Second loop: with "loop -- recur"
      (loop [i 0
             len (count pixels)]
         (if (< i len)
               (println (str i ": " (aget pixels i)))
               (recur (inc i)

Above, notice that the loop -- recur construct is essentially a let declaration, with a second call (recur) to reset the variables to something else. In this case, the next index in the array. Note how the len is simply given the same value over and over, just to avoid calling (count pixels) at each iteration.

Of course, there are lispier ways to loop an array of pixels. For example, to obtain the average of all pixels, we can use function reduce, which takes the first two elements of a list, applies a function to them, and then applies the function to the result and the next element, etc:

(let [fp (.getProcessor (ij.IJ/getImage))
      pix (.getPixels fp)]
   (if (instance? ij.process.FloatProcessor fp)
      (println "Average pixel intensity" (/ (reduce + pix) (count pix)))
      (println "Not a 32-bit image")))

Above, notice that one could have used also apply to apply + to all element of an array, with the same result:

      (println "Average pixel intensity" (/ (apply + pix) (count pix)))

To sum all pixels in an 8-bit image, one needs first to bit-and all bytes to 255, so they become integers and can be summed. But of course we should not bit-and the sum! To solve this, reduce accepts a first value (in this case, zero):

(let [bp (.getProcessor (ij.IJ/getImage))
      pix (.getPixels bp)]
  (if (instance? ij.process.ByteProcessor bp)
    (println "Average intensity: " (float (/ (reduce (fn [x1 x2] (+ x1 (bit-and x2 255))) 0 pix) (count pix))))
    (println "Not an 8-bit image")))

It could even be done using a local variable, but it's ugly and undesirable (why create it when it's not really needed)? Notice we need to create the local variable "sum" because variables declared by let are immutable.

(let [bp (.getProcessor (ij.IJ/getImage))
      pix (.getPixels bp)]
  (if (instance? ij.process.ByteProcessor bp)
    (with-local-vars (sum 0)
      (doseq [pixel pix]
        (var-set sum (+ (var-get sum) (bit-and pixel 255))))
      (println (float (/ (var-get sum) (count pix)))))
    (println "Not an 8-bit image")))

Executing commands from the menus

Any ImageJ menu command may be run on the active image:

(ij.IJ/doCommand "Add Noise")

Be aware that the above starts a new Thread and forks. For reliable control, try the run method, which will wait until the plugin finishes execution.

(ij.IJ/run "Add Noise")

For even more reliable control, run the command directly on a specified image, instead of a possibly changing current image:

(let [imp (ij.IJ/getImage)]
  (ij.IJ/run imp "Subtract..." "value=25"))

To find out which arguments can any command accept, open the Plugins - Macros - Macro Recorder and run the command manually.

Creating and using Clojure scripts as ImageJ plugins

Simply create a text file with the script inside, and place it in the plugins menu or any subfolder. Then call Plugins - Scripting - Refresh Clojure Scripts to make it appear on the menus.

If the macros/StartupMacros.txt includes a call to the Refresh Clojure Scripts inside the AutoRun macro, then all your Clojure scripts will appear automatically on startup.

To modify an script which exists already as a menu item, simply edit its file and run it by selecting it from the menus. No compiling necessary, and no need to call Refresh Clojure Scripts either (ther latter only for new scripts or at start up.)

Very important: all scripts and commands from the interpreter will run within the same thread, and within the same clojure context.

Using java beans for quick and convenient access to an object's fields

Essentially it's all about using get methods in a trivial way. For example:

(let [imp (ij.WindowManager/getCurrentImage)
      b (bean imp)]
   (println (:title b)
            (:width b)  
            (:height b)))

Eventually Clojure may add support for set methods as well.


Fixing overexposed images: setting a pixel value to a desirable one for all overexposed pixels

The problem: Leginon or the Gatan TEM camera acquired an overexposed image, and set all pixels beyond range to zero.

The solution: iterate all pixels; if the pixel is zero then set it to a desirable value, such as the maximum value of the main curve in the histogram (push 'auto' on the Brightness and Contrast dialog to see it.)

In the example below, the fix function is called with the current image and the value 32500 as a floating point number. Notice also the type definition (which is optional) of the float pixel array, to enhance execution speed:

 ; Assumes a FloatProcessor image
 (defn fix [imp max]
 (let [#^floats pix (.getPixels (.getProcessor imp))]
   (loop [i (int 0)]
     (if (< i (alength pix))
         (if (= 0 (aget pix i)) (aset pix i (float max)))
         (recur (inc i)))))))
 (let [imp (ij.IJ/getImage)]
   (fix imp (float 32500))
   (.updateAndDraw imp))

Creating a script for ImageJ

Simply write the clojure script in a text file, and follow these conventions:

1. Add an underscore "_" to its file name, and the extension ".clj": fix_leginon_images.clj
2. Save it under fiji/plugins/ folder, or a subfolder.

When done, just run the "PlugIns/Scripting/Refresh Clojure Scripts" plugin.

Once saved and in the menus, you need not call refresh scripts ever again for that script. Just edit and save it's text file, and run it again from the menus. Next time Fiji opens, the script will automatically appear in the menus.

See Scripting Help for more details, including how to use the built-in dynamic interpreter.

Example Clojure plugins included in Fiji

Open the plugins/Examples/ folder in Fiji installation directory. You'll find three Clojure examples:

  • Multithreaded_Image_Processing.clj: illustrate, with macros (via defmacro), how to automatically multithread the processing of an image using arbitrary subdivision of the image, such as one line per thread, for as many threads as cores the CPU has.
  • blend_two_images.clj: illustrates how to open two images from an URL, and blend the gray image into each channel of the color image.
  • celsius_to_fahrenheit.clj: illustrates the usage of a Swing GUI, and how to instantiate anonymous classes from an interface (via proxy Clojure function). This example is taken from the Clojure website.
  • random_noise_example.clj: illustrates how to declare a function inside a closure (for private access to, in this case, a unique instance of a random number generator), and then fill all pixels of a ByteProcessor image with a random byte value.
  • Command_Launcher_Clojure.clj: illustrates how to create a GUI with a KeyListener, so that the typed text changes color from red to black when it matches the name of an ImageJ command. This example is also under the Scripting comparisons, along equivalent versions written in Java, Jython, Javascript and JRuby.
  • Dynamic ROI Profiler: illustrates how to add a MouseMotionListener and a WindowListener to an ImageWindow of an open image. Reads out the ROI (Region Of Interest), and if it's a line, polyline or rectangle, plots the profile of pixel intensity along the line. As the mouse moves or edits the ROI on the image, the profile is updated.


Defining the output stream

The default output stream is at variable *out*. The Clojure Interpreter binds it to the PrintWriter that prints to the interpreter text area, by executing any typed in code within the following binding:

(binding [*out* (Clojure.Clojure_Interpreter/getStdOut)]
   ; any typed input here


  • To list all existing namespaces:
>>> (all-ns)
(#<Namespace: xml> #<Namespace: zip> #<Namespace: clojure> #<Namespace: set> #<Namespace: user>)
  • To list all functions and variables of a specific namespace, first get the namespace object by name:
(ns-map (find-ns 'xml))

Note above the quoted string "xml" to avoid evaluating it to a (non-existing) value.

  • To list all functions and variables of all namespaces:
(map ns-map (all-ns))

A nicer way to print all public functions and variables from all namespaces, sorted alphabetically:

(doseq [name (all-ns)]
  (doseq [[k v] (sort (ns-publics name))]
    (println k v)))

Note above we use destructuring: the [k v] take the values of the key and the value of each entry in the ns-publics table. Actually, since we first sort the table, the k and v take the first and second values of each array pair in the sorted list of array pairs returned on applying sort to the ns-publics-generated table.

Forget/Remove all variables from a namespace

To forget all variables from the user namespace, do:

(map #(ns-unmap 'user %)
     (keys (ns-interns 'user)))

The above maps the function ns-unmap to each variable name declared in the user namespace (using # to create a lambda function), which is the same as the prompt namespace. To get the names of the variables, we use ns-interns to retrieve the map of variable names versus the variable contents, and extract the keys from it into a list.

Thanks to AWizzArd from #clojure at irc.freenode.net for the tip.

JVM arguments

  • To get the arguments passed to the JVM, see contents of variable *command-line-args*
(println *command-line-args*)


  • To list all methods of an object:
(defn print-java-methods [obj]
  (doseq [method (seq (.getMethods (if (= (class obj) java.lang.Class)
                                    (identity obj)
                                    (class obj))))]
    (println method)))
 ; Inspect an object named imp, perhaps an image
 (print-java-methods imp)

 public synchronized boolean ij.ImagePlus.lock()
 public void ij.ImagePlus.setProperty(java.lang.String,java.lang.Object)
 public java.lang.Object ij.ImagePlus.getProperty(java.lang.String)
  • To list constructors, just use .getConstructors instead of .getMethods.

(Thanks to Craig McDaniel for posting the above function to Clojure's mailing list.)

Lambda functions


  • To declare functions on the fly, lambda style, with regex for arguments:

For example, declare a function that takes 2 arguments, and returns the value of the first argument divided by 10, and multiplied by the second argument:

(let [doer #(* (/ %1 10) %2)]
  (doer 3 2))

Of course there's no need to name the function, the above is just for illustration.

Mapping a function to all elements in a list

  • To declare a nameless function, and apply it to each element of a list:

In this case, increment by one each value of a list from 0 to 9:

(let [numbers (range 10)
      add-one (fn [x] (+ x 1))]
  (map add-one numbers))

There is no need to declare the names, the above is just for illustration. Above, we could have defined the function as #(+ %1 1):

(map #(+ %1 1) (range 10))

... or of course use the internal function inc which does exactly that:

(map inc (range 4))

Beware that the map function above applies the given function to each element of a list, and returns a new list with the results.

Built-in documentation

Use the function doc to query any other function. For example, the list generator function range:

(doc range)

([end] [start end] [start end step])
  Returns a lazy seq of nums from start (inclusive) to end
  (exclusive), by step, where start defaults to 0 and step to 1.

Above, notice the function has three groups of possible arguments, denoted in brackets.

When not knowing what to search for, you may try find-doc instead, which takes a string or regular expression as argument:

 user=> (find-doc "ns-")
   Returns a map of the aliases for the namespace.
   Returns a map of the import mappings for the namespace.
 ... etc.

Defining documentation for your own functions

So where does the documentation come from? Every definition of a function or macro or multimethod may take a description string before the arguments:

(defn area
  "Computes the area of a rectangle."
  (* (.width r) (.height r)))

... which the doc function prints, formatted:

 user=> (doc area)                           
   Computes the area of a rectangle.

A fibonacci sequence: lazy and infinite sequences

A beautiful example of using lazy sequences and applying functions to one or more sequences at a time.

Below, the sequence fibs is defined in such a way that it contains all possible fibonacci numbers. Since such sequence is infinite, we declared it as lazy sequence, that creates new elements only when they are asked for.

The lazy-cat clojure function creates such lazy sequence by concatenation of two sequences: the first sequence is 0, 1 (which takes the role of feeder or initialization sequence), and the second sequence is the result of a map operation over two subsets of the fibs sequence itself: the full and the full minus the first element (hence the rest operation to obtain the list of all elements without the first).

A map operation applies a function to each element of a sequence, or, when two or more sequences are provided, to the corresponding elements: those at index 0 in all sequences, those at index 1 in all sequences, etc.

To generate the fibonacci sequence of numbers, a sum + operation is mapped to the numbers contained in the self sequence fibs and to the corresponding elements of the self sequence fibs minus the first element, i.e. shifted by one.

In short, the lazy sequence fibs is an abstract way of representing a potentially infinite sequence, with an implementation containing a full abstract definition of all fibonacci numbers.

Then we just take the first 10 elements of such lazy sequence, which are created on the fly.

(def fibs (lazy-cat [0 1]
                    (map + fibs (rest fibs))))

(take 10 fibs)

Which outputs:

(0 1 1 2 3 5 8 13 21 34)

Creating shallow and deep sequences from java arrays

Many clojure functions take sequences, not native java arrays, as arguments. A java native array can be wrapped by a shallow sequence like the following:

>>> (def pixels (into-array (range 10)))
>>> pixels
>>> (def seq-pix (seq pixels))
>>> seq-pix
(0 1 2 3 4 5 6 7 8 9)

Now if we modify the native array, the sequence will reflect that change too when read:

>>> (aset pixels 3 99)
>>> seq-pix
(0 1 2 99 4 5 6 7 8 9)

The array was not duplicated. The only new object created was the shallow sequence:

>>> (class seq-pix)

To create a true deep duplicate of the array, one can do:

>>> (def pixels2 (vec pixels))
>>> (class pixels2)
>>> pixels2
[0 1 2 99 4 5 6 7 8 9]
>>> (def seq-pix2 (seq pixels2))
>>> (class seq-pix2)

Or, in short:

(def seq-pix2 (seq (vec pixels)))

So now any changes to the original pixels array will not affect the new sequence:

>>> (aset pixels 3 101)
>>> seq-pix2
(0 1 2 99 4 5 6 7 8 9)

Thanks to Chouser and wwmorgan for examples on #clojure at irc.freenode.net

Generating java classes in .class files from clojure code

Using ahead of time (AOT) compilation with gen-class, any clojure code can be compiled to a java class. Such class can then be used from java code, or from any scripting language like jython, jruby, javascript, and any other.

One way to do so is to place a gen-class declaration in a namespace block.

Be aware: the namespace must match the folder structure where the .clj file is and the file name of the .clj file. For example, to generate a class named fj.tests.process.FloatProcessorPlus, you need a clojure file under fj/tests/process/FloatProcessorPlus.clj .

To compile the clojure code to .class files, you need:

  1. A classes/ folder in the current directory where clojure.lang.Repl is run. This folder will receive the generated .class files.
  2. Add to your classpath the top-level folder, in the example the 'fj' folder, and also the folder containing the .clj file itself.
  3. Add to your classpath the classes/ folder as well.
  4. In a clojure.lang.Repl, use the compile function.

For example:

$ mkdir classes
$ java -cp .:../../ij.jar:../../jars/clojure.jar:./classes/:./fj/tests/process/ clojure.lang.Repl
user=> (compile 'fj.tests.process.FloatProcessorPlus)

The following clojure example contains a namespace declaration that includes some imports and also the gen-class. In the gen-class block, we define which class our code extends (in this case, ij.process.FloatProcessor), and which methods are to be created (with specific argument signatures and return object signature).

Later, the compiler will assign a clojure function to each declared method, using the prefix string plus the method name to match a function.

For example, with prefix "fp-" and method "fillValue", the compiler will look for the clojure function "fp-fillValue".

Finally, a main method is not directly declared, but exists if a function named prefix + main ("fp-main" in the example) exists. We can use the main method to run the new class as an application.

The example clojure code:

; Albert Cardona 20081203
; Save this file as fj/tests/process/FloatProcessorPlus.clj

; and compile it from a Repl or clojure script like:
; (compile 'fj.tests.process.FloatProcessorPlus)
; Be sure to set the classpath to point to the folder containing the above file, for example:
; $ cd fiji/plugins/
; $ mkdir -p tests/fj/tests/process
; $ cd tests/fj/tests/process/
; $ vim FloatProcessorPlus.clj
; ...
; $ cd ../../../
; $ mkdir classes
; $ java -cp ../../ij.jar:../../jars/clojure.jar:./classes/:.:./fj/tests/process/ clojure.lang.Repl
; user=> (compile 'fj.tests.process.FloatProcessorPlus)
; fj.tests.process.FloatProcessorPlus
; user=>
; The compilation will place the proper .class files under the proper directory
; structure in the ./classes/ folder.
; Then run like any other java class with a static public void main method:
; $ java -cp .:../../ij.jar:../../jars/clojure.jar:./classes fj.tests.process.FloatProcessorPlus

(ns fj.tests.process.FloatProcessorPlus
   (:import (ij ImagePlus)
            (ij.process FloatProcessor)
            (java.util Random))
    ; Could also use :implements
    :extends ij.process.FloatProcessor
    ; Specify methods to expose as public,
    ; with specific parameter types and return type:
    :methods [[fillMin [] void]
              [fillMax [] void]
              [fillValue [float] void]
              [randomize [] void]]
    ; Define a function prefix for the exposed methods: for example,
    ; the fillMin public method is implemented by function fp-fillMin.
    :prefix "fp-"))

(defn fp-fillValue [this value]
  "Set each pixel in the image to the given value."
  (.setPixels this
              (into-array Float/TYPE (replicate
                            (* (.getWidth this) (.getHeight this))
                            (float value)))))

(defn fp-fillMin [this]
  (.fillValue this Float/MIN_VALUE))

(defn fp-fillMax [this]
  (.fillValue this Float/MAX_VALUE))
; Declaring a function to be used as a java method, within a closure:
(let [generator (Random. (System/currentTimeMillis))]
  (defn fp-randomize [this]  
    (.setPixels this (into-array Float/TYPE
                              (fn [x] (.nextFloat generator))
                              (range (* (.getWidth this) (.getHeight this))))))))

; This function is seen as the static public void main function of a java class:
; (add a parameter, like [args], if you would like to access the command-line args)
(defn fp-main []
  "Test the generated class" 
  (let [imp (ImagePlus. "Test" (fj.tests.process.FloatProcessorPlus. 100 100))
        ip (.getProcessor imp)] ; Testing access on "ImageProcessor" type
    (.show imp)
    ; Test some methods of our extended FloatProcessor class:
    (.randomize ip)
    (.findMinAndMax ip)
    (.updateAndDraw imp)))

References, concurrency, transactions and synchronization

Clojure supports thread concurrency without explicit locks. Compared to java code, this is a gigantic step forward: locks, and particularly multiple locks, are very hard to get right and very, very hard to debug properly (but see debugging multithreaded java programs).

The most basic building blocks are references, which are created with the ref function, and modified within transaction blocks (defined by dosync) using commute or alter functions (and others).

To read out the value of a reference, call deref or just @ on it:

; Create a new reference named 'id' initializated to value zero:
(def id (ref 0))
-> 0

; Read out the value of the reference:
-> 0

; Increase the id by one, using built-in function "inc":
(dosync (alter id inc))
-> 1

; Set the value to 20 (ignoring the current value, given in cv:
(dosync (alter id (fn [cv] 20)))
-> 20

References are not type specific: any object may be assigned to the same reference. Whether that makes any sense is up to you.

The commute and alter functions replace the value of the reference with that of the return value of a function given as argument. The function given as argument to commute and alter is given, in turn, the value of the reference (i.e. the dereferenced reference) and any other further arguments. The difference between commute and alter is that commute returns the dereferenced reference after the transaction is done, which may be different already (because of concurrent modifications) than the value that was set to the reference within the transaction; whereas alter returns the value that it had while the transaction was being done (i.e. the value returned by the function, the same that gets set as the value of the reference).

In the following example, a unique id counter is incremented continuously by 1, and all ids are collected, unordered, into a vector. Both the next available id and the vector of all accessed ids are stored in references.

Keep in mind the vector of ids assigned to the reference named 'ls' is always immutable: what we assign to the reference 'ls' below is a new vector, resulting from adding a new id to the old vector of ids. This immutability enables other threads to read the vector without locks. For performance, keep in mind that vectors, like many other clojure data structures, have structural sharing, so the new vector is not a duplication--even if it behaves like one.

The assignment is done in a transaction, so no matter how many concurrent threads try to do so, the resulting vector will have all the ids.

; Albert Cardona 2008-12-18
; Example clojure program using references and concurrent threads
; that alter the value of the references.
; 10 threads running concurrently
; each thread runs 100000 iterations
; in each iteration the thread increments a counter 'id'
;   and adds it to a list 'ls' of ids.
; At the end, we print the current value of 'id'
; and the length of the list 'ls' of ids.
; No locks!

(ns fj.test.concurrent
  (:import (java.util.concurrent Executors TimeUnit)))

(let [ls (ref []) ; A reference to a vector storing a list if ids.
      id (ref 0)  ; The next unique id available.
      n_threads 10
      n_iterations 100000
      exec (Executors/newFixedThreadPool n_threads)]
  (println "Running" n_threads "threads x" n_iterations "iterations/thread...")
  (dotimes [i n_threads]
    (.submit exec (fn []
                    (dotimes [t n_iterations]
                      ; Obtain the next unique id:
                      ; (Note we use "alter" and not "commute", because alter
                      ; returns the result of the applied function, whereas
                      ; commute would return the dereferenced ref, which could
                      ; have already changed.  Thanks to AWizzards for spotting
                      ; this.)
                      (let [next-id (dosync (alter id inc))]
                        ; Create a new vector made of
                        ;  all previous ids and next-id,
                        ;  and set it as the current list of ids:
                        (dosync (commute ls conj next-id)))))
  (.shutdown exec)
  (.awaitTermination exec 10 TimeUnit/MINUTES)
  (println "... done!")
  ; If there was any clash in setting the reference to the list of ids,
  ; the count would be less than 1000000:
  (println "Number of listed ids:" (count @ls))
  ; If any id was used twice, the next available id would be less than 1000000:
  (println "Next available id:" @id)
  ; Check that there aren't any repeated ids:
  (println "Number of repeated ids:"
           (- (count @ls)
              (count (set @ls))))) ; Make a hash set (with unique entries) from the list of ids

Using try/catch/finally and throwing Exceptions

  (println "Going to throw ...")
  (throw (Exception. "Testing error catching"))
  (println "Should not print, an Exception is thrown before")
  (catch Exception e
    (println "Oops ... an error ocurred.")
    (.printStackTrace e))
    (println "Cleaning up!")))

Of course you can throw any kind of exception you want. For example, in checking function arguments:

(import '(java.awt Rectangle))

(defn area [#^Rectangle r]
  (if (not (instance? Rectangle r))
    (throw (IllegalArgumentException. "Can only compute the area of a Rectangle.")))
  (* (.width r) (.height r)))

Above, despite the type declaration, one can pass any value to the area function and it will still work, but of course our class check will cut execution:

user=> (area 10)                          
java.lang.IllegalArgumentException: Can only compute the area of a Rectangle. (NO_SOURCE_FILE:0)

user=> (area (Rectangle. 0 0 10 10))

Executing a command in a shell and capturing its output

First we define the macro exec:

(import '(java.io BufferedReader InputStreamReader))

(defmacro exec
  "Execute a command on the shell, passing to the given function
   the lazy sequence of lines read as output, and the rest of arguments."
  [cmd pred & args]
  `(with-open [br# (BufferedReader. (InputStreamReader. (.getInputStream (.exec (Runtime/getRuntime) ~cmd))))]
    (~pred (line-seq br#) ~@args)))

Some explanations on the above macro syntax (see also clojure's macro syntax page):

  • The backquote ` quotes the next expression, as defined by: `( <any code here> ). Which means the code block is not evaluated. But, unlike simple quote ', the backquote enables evaluation of expressions within the block when tagged with a ~ (a tilde).
  • The ~ (tilde) evaluates the immediate expression. Can only be used in the context of a backquoted code block.
  • The ~@ means evaluate and expand, which has the efect of placing the elements of a list as if they where declared in the code, without the list enclosure. So: `(~@(str "this" "that")) results in: "thisthat". In the example above, we expand the & args, which is a list containing all arguments given to the exec macro beyond the first and second (which are bound to cmd and pred, respectively). In this way, we lay down the proper function call of the pred, which is expected to be a function name (a predicate); the reason we use ~ on it is to evaluate pred so that it renders the pointer to the function itself. That pred function, by design, must accept a lazy sequence of text lines and any number of arguments afterwards.
  • The # tagged at the end of a name expands to (gensym name), which results in creating a uniquely named symbol, to avoid name collisions.
  • Any code present outside the backquote (none, in the case above) will be executed at macro read time, not at code execution time (aka run time)! So any precomputations are possible before laying down the code that will be executed at run time.

Then we give the macro a command to execute and a function to process its stdout output.

; List all files in the home directory:
(exec "ls /home/albert/"
      #(doseq [line %1] (println line)))

A second example, printing the file size of each listed file in the home directory:

; Print the size of each file in the home directory:
(import '(java.io File))

(let [dir "/home/albert/"]
  (exec (str "ls " dir)
        #(doseq [line %1]
          (println (.length (File. (str dir line)))))))

A third example, telling the music player XMMS2 to jump to a specific track in its playlist:

(let [track-number 125]
  (exec (str "xmms2 jump " track-number)
        (fn [lines] lines)))