This is a Gramble, which of course is short for a GROMACS Bramble, or, in other words, a Raspberry Pi 2 model B cluster running GROMACS. Given the ARM processor in a Raspberry Pi 2 does not allow SIMD instructions like the more complex (and expensive) Intel chips, why would I want to do such a thing? Well, I wanted to learn how to setup a simple compute cluster.
And this is what I did. Unless stated otherwise you need to do this on both machines (or how ever many you are using).
1. Install Ubuntu 14.04 LTS Server onto the microSD cards
Each Raspberry Pi 2 runs off a microSD card; on a computer with a microSD slot (I used an iMac) download the Ubuntu 14.04 LTS image and copy it onto the microSD card as described here. Then you simply push it back into the slot on the Raspberry Pi and power it up. Note that Ubuntu does not run on the model A and, at the time of writing, only ran on the model B. If your microSD card is bigger than 2GB you might want to resize the partition.
First, let’s update the installed software and also install the ssh server so we can remotely connect.
sudo apt-get update -y sudo apt-get upgrade -y sudo apt-get install openssh-server
3. Setup the network
Now my setup was a bit strange. I was using an old Apple Time Capsule I had; both Raspberry Pis were connected to this via ethernet cables and the Time Capsule itself was in “Extend Wireless Network” mode since our main wireless router is somewhere else in the house. Ideally, I’d want a dual-homed headnode with one public IP address and then a private network for communication within the cluster. Instead each of the two Raspberry Pis have their own IP that is dynamically assigned by my router, but this will do for now.
sudo nano /etc/hosts
so it reads
127.0.0.1 localhost 192.168.0.12 rasp0 192.168.0.18 rasp1
Also edit the hostname
sudo nano /etc/hostname
so it matches
4. Add an MPI user
We will need a special user that can log in without passwords to all the nodes that SLURM will use later on. As I understand it, giving it a uid less than 1000 stops the user appearing in any login GUI.
sudo adduser mpiuser --uid 999
5. Install NFS
We will share a folder on the headnode with all the compute nodes using the NFS protocol. This means we’ll only need to install applications on the headnode and they will be accessible from any compute node. Also this is where the GROMACS output files will be written.
on the headnode (rasp0)
sudo apt-get install nfs-kernel-server
on the compute node (rasp1)
sudo apt-get install nfs-common
on the headnode (rasp0) add the following to
/home/mpiuser *(rw,sync,no_subtree_check) /apps *(rw,sync,no_subtree_check)
This will export the folders
/home/mpiuser on rasp0 to all the compute nodes (in this case just rasp1). You need to make sure all folders shared by NFS exist on both machines. So on both machines
sudo mkdir /apps
You don’t need to
mkdir /home/mpiuser as creating this user will have automatically created a home directory for it. Now on the headnode
sudo service nfs-kernel-server start
On the compute node (rasp1),
sudo ufw allow from 192.168.1.0/24 sudo mount rasp0:/home/mpiuser /home/mpiuser sudo mount rasp0:/apps /apps
The first line opens a port in the firewall. although I’m definitely sure I needed to do this. The last two manually mount the NFS share from the headnode (rasp0). To set it up so this happens automatically
sudo nano /etc/fstab
rasp0:/home/mpiuser /home/mpiuser nfs rasp0:/apps /apps nfs
then we can force a remount via
sudo mount -a
6. Create an SSH key pair to allow passwordless login
Because we now have /apps and /home/mpiuser shared with all nodes of the cluster (ok, just rasp1, but you know what I mean) we can simply on the headnode create an ssh keypair as mpiuser and it will be shared with all the compute nodes. So on rasp0
su mpiuser ssh-keygen -t rsa cd .ssh/ cat id_rsa.pub >> authorized_keys
I didn’t use a passphrase during key generation. I expect this is a bad thing and I did read you could use a key chain, but as this is a toy cluster I’m going to stick my fingers in my ears and pretend I didn’t read that. If you haven’t created an ssh keypair before, it is fairly simple – it creates a public and a private key. This is described in more detail here. The key things are that the private key (.ssh/id_rsa) should only be readable by the mpiuser and no-one else. In Linux-land, this means it should have permissions of
400 – this is how it will be created. Secondly, any remote machine will allow a passwordless login if the public key for that user is in
.ssh/authorized_keys; this explains the last line above.
Let’s test it. Since we are already the mpiuser and we are on rasp0
Should automatically log you into rasp1. If you try the same thing as the default ubuntu user it will prompt you for your password as that user doesn’t have an ssh keypair setup.
7. Compile GROMACS
As we are thinking about NFS, let’s compile GROMACS in
/apps so the gmx binary can be run from any of the compute node(s). We need a few things before we begin.
sudo apt-get install build-essential cmake
The first package contains the compilers you’ll need to, well, compile GROMACS and cmake is the build tool GROMACS uses. So as the mpiuser,
cd /apps mkdir src cd src/ wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.2.tar.gz cd gromacs-5.1.2/ mkdir build-gcc48 cd build-gcc48 cmake .. -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=‘/apps/gromacs/5.1.2’ -DBUILD_SHARED_LIBS=off make -j 4 sudo make install
Note the unusual
-DCBUILD_SHARED_LIBS flag in the GROMACS cmake command; this is to get around an error when compiling GROMACS on the Raspberry Pi. You shouldn’t normally need this flag. Now the make command will take at least ten minutes, so put the kettle on.
Because of dynamic linking, you’ll also need to install the compilers on the compute nodes via
sudo apt-get install build-essential
I suspect using environment modules might avoid this issue; I’m going to play with these and if I can get it to work will write another post.
Once you’ve done this, then on any machine you should be able to run GROMACS via
source /apps/gromacs/5.1.2/bin/GMXRC gmx mdrun
8. Install a cluster management and job scheduling system (SLURM)
Despite the fact the machines I’ve used in the past have tended to use PBS or SGE (and so my fingers can type qstat really, really fast), I chose to use SLURM as
- it is available as an Ubuntu package
- our University high performance computing centre have recently started using it and they recommended it
- it is actively developed and I can’t work out, or at least remember for more than five minutes, what is going on with SGE
- it has documentation! and tutorials!
- it is open source (GPL2)
- I liked the name
To install on rasp0
sudo apt-get install slurm-llnl
This also installs MUNGE as a pre-requisite (see the next section). Now SLURM appeared to want to use
/usr/bin/mail and complained when it couldn’t find it so I also installed
sudo apt-get install mailutils
which drops you into a setup screen and I chose the “local” option.
Also on rasp1
sudo apt-get install slurm-llnl
9. Get MUNGE working
MUNGE creates and validates credentials and SLURM uses it. On the headnode
This creates a key
/etc/munge/munge.key. Now copy this key to
/etc/munge/ on all nodes (you may need to fiddle with permissions etc to use the NFS share). There appears to be a bug with Ubuntu and MUNGE, but the workaround is to do the following on all nodes
sudo nano /etc/default/munge
and add the line
now start the service
sudo service munge start
Check it is running
ps -e | grep munge
10. Get SLURM working
This was the bit I wasn’t looking forward to as job schedulers have, frankly, scared me. But it turns out this was one of the easiest steps. If we are the mpiuser on rasp0. SLURM comes with a very simple configuration file that we can edit.
cp /usr/share/doc/slurm-llnl/examples/slurm.conf.simple.gz . gunzip slurm.conf.simple.gz nano slurm.conf.simple
All I did was change the lines so they read
ControlMachine=rasp0 .. NodeName=rasp[0-1] Procs=4 State=UNKNOWN PartitionName=test Nodes=rasp[0-1] Default=YES MaxTime=INIFINITE State=UP
You’ll notice I’ve identified rasp0 as both the ControlMachine (i.e. headnode) and also a Node (compute node) belonging to the test Partition. On a regular cluster you probably don’t want the headnode also being a compute node, but I only had two Raspberry Pis so I thought why not? This also shows the syntax for referring to multiple nodes. If you want a more complex configuration an online configuration tool is provided. There is also an even more complicated online configurator. A note of caution: these may not work with the version of SLURM installed by apt-get (2.6.5) since the current version is 15.08. That doesn’t mean 2.6.5 is old; they’ve changed the numbering system recently.
Finally copy the file to the right place on all nodes
sudo cp slurm.conf.simple /etc/slurm-llnl/slurm.conf
and (on the headnode, rasp0)
sudo service slurm start
on the compute node
sudo slurmd -c
srun -N1 hostname
11. Submit a GROMACS job to the queue
I’m going to assume you have prepared a TPR file called md.tpr and have copied it into /home/ubuntu (and we are now the default user, ubuntu).
Let’s do some simple benchmarking – remember a Raspberry Pi has 4 cores. So first, let’s create a series of TPR files
cp md.tpr md-1.tpr cp md.tpr md-2.tpr cp md.tpr md-4.tpr
Now let’s create some SLURM job submission files. This is the one for running on two cores – you’ll need to change the
--job-name SBATCH flags and the
-ntmpi GROMACS flags depending on the number of cores.
sudo nano md-2.slurm.sh
and copy in
#!/bin/bash #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=2 #SBATCH --time=00:15:00 #SBATCH --job-name=md-2 source /apps/gromacs/5.1.2/bin/GMXRC srun gmx mdrun -deffnm md-2 -ntmpi 2 -ntomp 1 -maxh 0.1 -resethway -noconfout
This will run for 6 minutes, resetting the GROMACS timers after 3 minutes. It won’t write out a final GRO file as this can affect the timings. Hopefully you’ll find that, whilst useful and fun machines, Raspberry Pis are really slow at running GROMACS! To submit the jobs
To check the queue we can issue
and to cancel we can use