Linux command line tutorial
Contents
Connecting to the cluster
We will use SSH (secure shell) to connect to the cluster. On Mac OSX we need to simply open a terminal window (under Applications->Utilities), on PC we may need to download an ssh client such as Putty.
In the terminal type:
ssh madmax.mpi-cbg.de
and enter your password
tomancak@madmax.mpi-cbg.de's password:
Now you are on the cluster
Last login: Sun May 19 09:01:24 2013 from 10.1.7.16 "Platform HPC" 3.1 (build 7211) Management Node Welcome to the MPI-CBG cluster masternode. [tomancak@madmax ~]$
Moving around
pwd - print working directory
[tomancak@madmax ~]$ pwd /home/tomancak
cd - change directory
[tomancak@madmax ~]$ cd /projects [tomancak@madmax projects]$ pwd /projects
~ - tilda means home directory
[tomancak@madmax projetcs]$ cd ~ [tomancak@madmax ~]$ pwd /home/tomancak
/ - slash demarcates directories and by itself means the root directory
[tomancak@madmax ~]$ cd / [tomancak@madmax /]$ pwd /
. - current directory, if we want to for example execute a script in the current directory we do it like this:
./executable_script
This tells the interpreter to look for the file executable_script in the current directory and not somewhere else.
.. - one directory up
[tomancak@madmax projects]$ cd .. [tomancak@madmax /]
pressing tab - auto-completion, we don't need to type long names of directories, start typing the name and press tab and the linux system will either auto-complete or show you the options you have
[tomancak@madmax ~]$ cd /projects/toman + press tab [tomancak@madmax ~]$ cd /projects/tomancak_lightsheet [tomancak@madmax tomancak_lightsheet]$
Manipulating files
ls
more
tail
nano
mv
cp
scp
rm
mkdir
rmdir
gunzip
tar
Permissions
chmod
su
sudo
Pipes and redirects
|
>
wc
Misc
which
top
watch
grep
sort
find
man
Using it on the cluster
Here we put all of that goodness to work to make our life on a cluster easier.
Once we submit the jobs we want to monitor the progress. For example bjobs -r gives us an output summarising our running jobs. If there are many its not that useful. How about counting them?
bjobs -r | wc -l 160
More useful, we have 160 jobs running. Still how about the pending ones? We can do that:
bjobs -r | grep PEND | wc -l 40
Ok 40. Now, how many jobs actually finished successfully? Lets say they create some .tif files a few directories down
ls ../../output/*.tif | wc -l 200
Good, looks like 200 are finished. Its tempting to keep pressing enter on those commands but its also tiresome. Using
watch -n5 'ls ../../output/*.tif | wc -l'
outputs continuously
Every 5.0s: ls ../../output/*.tif | wc -l Mon May 20 21:16:37 2013 180
i.e. we get the number of files automatically updated every 5 seconds (without the -n paramater every 2s by default).
