Page history Edit this page How do I edit this website?

Linux command line tutorial

Connecting to the cluster

We will use SSH (secure shell) to connect to the cluster. On macOS we need to simply open a terminal window (under ApplicationsUtilities), on PC we may need to download an ssh client such as Putty.

In the terminal type:

ssh madmax.mpi-cbg.de

and enter your password:

tomancak@madmax.mpi-cbg.de's password:   

Now you are on the cluster!

Last login: Sun May 19 09:01:24 2013 from 10.1.7.16  
"Platform HPC" 3.1 (build 7211) Management Node  
Welcome to the MPI-CBG cluster masternode.  
[tomancak@madmax ~]$

Moving around

pwd - print working directory

[tomancak@madmax ~]$ pwd  
/home/tomancak

cd - change directory

[tomancak@madmax ~]$ cd /projects  
[tomancak@madmax projects]$ pwd  
/projects

~ - tilda means home directory

[tomancak@madmax projects]$ cd ~  
[tomancak@madmax ~]$ pwd  
/home/tomancak

/ - slash demarcates directories and by itself means the root directory

[tomancak@madmax ~]$ cd /  
[tomancak@madmax /]$ pwd  
/

. - current directory, if we want to, for example, execute a script in the current directory we do it like this:

./executable_script  

This tells the interpreter to look for the file executable_script in the current directory and not somewhere else.

.. - one directory up

[tomancak@madmax projects]$ cd ..  
[tomancak@madmax /]

pressing tab - auto-completion, we don’t need to type long names of directories, start typing the name and press tab and the linux system will either auto-complete or show you the options you have.

[tomancak@madmax ~]$ cd /projects/toman + press tab  
[tomancak@madmax ~]$ cd /projects/tomancak_lightsheet  
[tomancak@madmax tomancak_lightsheet]$

Manipulating files

ls

more

tail

nano

mv

cp

scp

rm

mkdir

rmdir

gunzip

tar

Permissions

chmod

su

sudo

Pipes and redirects

|

>

wc

Misc

which

top

watch

grep

sort

find

man

Using it on the cluster

Here we put all of that goodness to work to make our life on a cluster easier.

Once we submit the jobs, we want to monitor the progress. For example bjobs -r gives us an output summarising our running jobs. If there are many, it’s not that useful. How about counting them?

bjobs -r | wc -l  
160

More usefully, we have 160 jobs running. Still how about the pending ones? We can do that:

bjobs -r | grep PEND | wc -l  
40

Ok 40. Now, how many jobs actually finished successfully? Let’s say they create some .tif files in ../../ouput:

ls ../../output/*.tif | wc -l  
200

Good, looks like 200 are finished. It’s tempting to keep pressing enter on those commands but it’s also tiresome. Using:

watch -n5 'ls ../../output/*.tif | wc -l' 

outputs continuously:

Every 5.0s: ls ../../output/*.tif | wc -l                                                                                               
Mon May 20 21:16:37 2013  
180

i.e. we get the number of files automatically updated every 5 seconds (without the -n paramater, every 2s by default).