Linux command line tutorial

Connecting to the cluster

We will use SSH (secure shell) to connect to the cluster. On Mac OSX we need to simply open a terminal window (under Applications->Utilities), on PC we may need to download an ssh client such as Putty.

In the terminal type:

ssh madmax.mpi-cbg.de

and enter your password

tomancak@madmax.mpi-cbg.de's password: 

Now you are on the cluster

Last login: Sun May 19 09:01:24 2013 from 10.1.7.16
"Platform HPC" 3.1 (build 7211) Management Node
Welcome to the MPI-CBG cluster masternode.
[tomancak@madmax ~]$

Moving around

pwd - print working directory

[tomancak@madmax ~]$ pwd
/home/tomancak

cd - change directory

[tomancak@madmax ~]$ cd /projects
[tomancak@madmax projects]$ pwd
/projects

~ - tilda means home directory

[tomancak@madmax projetcs]$ cd ~
[tomancak@madmax ~]$ pwd
/home/tomancak

/ - slash demarcates directories and by itself means the root directory

[tomancak@madmax ~]$ cd /
[tomancak@madmax /]$ pwd
/

. - current directory, if we want to for example execute a script in the current directory we do it like this:

./executable_script

This tells the interpreter to look for the file executable_script in the current directory and not somewhere else.

.. - one directory up

[tomancak@madmax projects]$ cd ..
[tomancak@madmax /]

pressing tab - auto-completion, we don't need to type long names of directories, start typing the name and press tab and the linux system will either auto-complete or show you the options you have

[tomancak@madmax ~]$ cd /projects/toman + press tab
[tomancak@madmax ~]$ cd /projects/tomancak_lightsheet
[tomancak@madmax tomancak_lightsheet]$

Manipulating files

ls

more

tail

nano

mv

cp

scp

rm

mkdir

rmdir

gunzip

tar

Permissions

chmod

su

sudo

Pipes and redirects

|

>

wc

Misc

which

top

watch

grep

sort

find

man

Using it on the cluster

Here we put all of that goodness to work to make our life on a cluster easier.

Once we submit the jobs we want to monitor the progress. For example bjobs -r gives us an output summarising our running jobs. If there are many its not that useful. How about counting them?

bjobs -r | wc -l
160

More useful, we have 160 jobs running. Still how about the pending ones? We can do that:

bjobs -r | grep PEND | wc -l
40

Ok 40. Now, how many jobs actually finished successfully? Lets say they create some .tif files a few directories down

ls ../../output/*.tif | wc -l
200

Good, looks like 200 are finished. Its tempting to keep pressing enter on those commands but its also tiresome. Using

watch -n5 'ls ../../output/*.tif | wc -l' 

outputs continuously

Every 5.0s: ls ../../output/*.tif | wc -l                                                                                             
Mon May 20 21:16:37 2013
180

i.e. we get the number of files automatically updated every 5 seconds (without the -n paramater every 2s by default).