Setting up a GROMACS cluster

3. Compile GROMACS

We could simply compile GROMACS in the usual way, but install it into /opt/software. There are several problems with this, the main being that it will be dynamically linked to various compiler libraries on the headnode and so the binary won’t work on the compute nodes (unless we also install the same compiler there too). Instead we shall use environment modules, which has already been installed on all machines. There are also some good walk-throughs here and here. Roughly speaking, we shall the pre-requisites for GROMACS, including gcc, into /opt/software and then setup ‘module files’ for each so that we can dynamically alter what is in our $PATH. This makes maintaining multiple versions of code (or the same version compiled using different compilers) much more straightforward and is the approach commonly taken by large computing clusters but it can be useful on a single workstation.

The environment modules package has been installed in /usr/share/modules

First we need to alter where module looks for modulefiles (the $MODULEPATH).

$ sudo vim /usr/share/modules/init/.modulespath

Comment out all the lines and add this to the end

/opt/software/modules

Now we can activate modules by loading this shell script

$ source /etc/profile.d/modules.sh

The structure I have adopted is to download and compile code in

/opt/software/src

and then install it in

/opt/software/apps

with modulesfiles located in

/opt/software/modules

Compiling gcc

The first module to compile is a compiler. We’ll need subversion and also flex. This is all still on the headnode

$ sudo apt-get install subversion flex

First, let’s check what versions are

$ svn ls svn://gcc.gnu.org/svn/gcc/tags | grep gcc | grep release

The latest stable version is 5.3.0 so let’s install that one.

/opt/software/src/gcc/$ sudo svn co svn://gcc.gnu.org/svn/gcc/tags/gcc_5_3_0_release/
/opt/software/src/gcc/$ cd gcc_5_3_0_release/

Download the prerequisites and make a build directory

/opt/software/src/gcc/gcc_5_3_0_release/$ sudo ./contrib/download_prerequisites
/opt/software/src/gcc/gcc_5_3_0_release/$ sudo mkdir build
/opt/software/src/gcc/gcc_5_3_0_release/$ cd build
/opt/software/src/gcc/gcc_5_3_0_release/build$ sudo ../configure --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/opt/software/apps/gcc/5.3.0 --disable-multilib
/opt/software/src/gcc/gcc_5_3_0_release/build/$ make -j 16
/opt/software/src/gcc/gcc_5_3_0_release/build/$ sudo make install

The make command will take up to an hour. Change the number after the -j flag to match the number of cores on your machine. If this works then we should now have two versions of gcc installed; 4.8.3 from build-essential (/usr/bin/gcc) that is in our $PATH by default and now 5.3.0

$ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.1) 4.8.4

$ /opt/software/apps/gcc/5.3.0/bin/gcc --version
gcc (GCC) 5.3.0

Now we need to make a module file

$ cd /opt/software/modules
/opt/software/modules/$ sudo mkdir gcc
/opt/software/modules/$ cd gcc
/opt/software/modules/gcc/$ sudo touch 5.3.0
/opt/software/modules/gcc/$ sudo vim 5.3.0

and mine looks something like this

#%Module1.0#####################################################################
##
## Application module file
##
proc ModulesHelp { } {
global version
global app
puts stderr "\tAdds `app-$version' to your PATH environment variable and necessary libraries"
}

set app gcc
set version 5.3.0

module-whatis "loads the necessary `$app-$version' library paths"

set BASEPATH /opt/software/apps/$app/$version/

setenv $app $BASEPATH
setenv GCCDIR $BASEPATH

prepend-path PATH $BASEPATH/bin/
prepend-path LD_LIBRARY_PATH $BASEPATH/lib:$BASEPATH/lib64
prepend-path MANPATH $BASEPATH/man/

prepend-path --delim " " CPPFLAGS " -I$BASEPATH/include"
prepend-path --delim " " CFLAGS " -I$BASEPATH/include"
prepend-path --delim " " LDFLAGS " -L$BASEPATH/lib/ -L$BASEPATH/lib64/"

setenv cc gcc
setenv cxx gcc
setenv f77 gfortran
setenv f90 gfortran
setenv f95 gfortran
setenv fc gfortran

setenv CC gcc
setenv CXX gfortran
setenv F77 gfortran
setenv F90 gfortran
setenv F95 gfortran
setenv FC gfortran

Now if we ask what modules are available, we should find there is one

$ module avail

---------------------------- /opt/software/modules -----------------------------
gcc/5.3.0

And no modules are loaded

$ module list
No Modulefiles Currently Loaded.

At the moment the system gcc installed when we ran sudo apt-get install build-essential should be in our $PATH

$ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.1) 4.8.4

If we load the gcc module (no need to specify which version as there is only one at present)

$ module load gcc
$ module list
Currently Loaded Modulefiles:
1) gcc/5.3.0
$ gcc --version
gcc (GCC) 5.3.0

Then you can see the gcc compiler we have just installed is now the active one!

Compiling Open-MPI

Next let’s compile open-mpi. The latest stable version is 1.10.2

$ cd /opt/software/src
/opt/software/src/$ sudo mkdir openmpi
/opt/software/src/$ cd openmpi
/opt/software/src/openmpi/$ wget https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.2.tar.gz
/opt/software/src/openmpi/$ tar zxvf openmpi-1.10.2.tar.gz
/opt/software/src/openmpi/$ cd openmpi-1.10.2
/opt/software/src/openmpi/$ sudo mkdir build-gcc53
/opt/software/src/openmpi/$ cd build-gcc53
/opt/software/src/openmpi/build-gcc53$ sudo ../configure --prefix=/opt/software/apps/openmpi/1.10.2/ --enable-shared=yes --enable-static=yes --with-slurm
/opt/software/src/openmpi/build-gcc53$ sudo make -j 16
/opt/software/src/openmpi/build-gcc53$ sudo make install

Then create a modules file

$ cd /opt/software/modules
/opt/software/modules/$ sudo mkdir openmpi
/opt/software/modules/$ cd openmpi
/opt/software/modules/openmpi/$ sudo vim 1.10.2

Mine looks like

#%Module1.0#####################################################################
##
## openmpi modulefile
##
##
proc ModulesHelp { } {
global version

puts stderr "\tAdds 64-bit OpenMPI compiled with GCC to your environment."
puts stderr "\tDirectory: $root"
}

module-whatis "adds OpenMPI to your environment variables"

set app openmpi
set version 1.10.2
set root /opt/software/apps/openmpi/1.10.2

prepend-path PATH $root/bin
prepend-path LD_RUN_PATH $root/lib
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path MANPATH $root/share/man

append-path -d " " CPPFLAGS -I$root/include
append-path -d " " LDFLAGS -L$root/lib

setenv MPI_HOME $root
setenv MPIRUN $root/bin/mpirun

Now we have two modules

$ module avail
---------------------------- /opt/software/modules -----------------------------
gcc/5.3.0 openmpi/1.10.2

### Compiling GROMACS

Now, at last, we can tackle GROMACS. I could have also put FFTW into modules, but GROMACS is quite happy downloading it and compiling FFTW itself so I haven’t bothered. Also if we were using GPUs, I should have installed CUDA into my modules, but it looks like the GPU cards I can fit in the limited space inside an Xserve are not powerful enough, so I’m going to ignore that too for the present. First, let’s download and unpack GROMACS

$ cd /opt/software/src
/opt/software/src/$ sudo mkdir gromacs
/opt/software/src/$ cd gromacs
/opt/software/src/gromacs/$ wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.2.tar.gz
/opt/software/src/gromacs/$ tar zxvf gromacs-5.1.2.tar.gz
/opt/software/src/gromacs/$ cd gromacs-5.1.2
/opt/software/src/gromacs/gromacs-5.1.2/$ sudo vim install.sh

In this shell script (install.sh) I’m going to record all the options I gave to GROMACS. This is useful since they are a bit more complex and I’m likely to compile GROMACS fairly frequently. This is one I have adapted from my old lab and first compiles “normal” GROMACS and then the MPI version of mdrun.

#! /bin/bash

GROMACS_SRC='/opt/software/src/gromacs/gromacs-5.1.2'
INSTALL_DIR='/opt/software/apps/gromacs/5.1.2'

source /etc/profile.d/modules.sh
module purge
module load openmpi/1.10.2
module load gcc/5.3.0

mkdir $GROMACS_SRC/build-gcc53
cd $GROMACS_SRC/build-gcc53

echo $PWD

cmake $GROMACS_SRC \
-DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
-DGMX_MPI=OFF \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DGMX_BUILD_MDRUN_ONLY=OFF \
-DGMX_BUILD_OWN_FFTW=ON
-DGMX_X11=OFF \
-DGMX_GPU=OFF \
-DGMX_DEFAULT_SUFFIX=OFF \
-DGMX_CPU_ACCELERATION=AVX \
-DBUILD_SHARED_LIBS=OFF \
-DGMX_EXTERNAL_BLAS=OFF \
-DGMX_EXTERNAL_LAPACK=OFF

make -j 16

make install

cd ..

mkdir $GROMACS_SRC/build-gcc53-openmpi
cd $GROMACS_SRC/build-gcc53-openmpi

cmake $GROMACS_SRC \
-DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
-DGMX_MPI=ON \
-DCMAKE_C_COMPILER=mpicc \
-DCMAKE_CXX_COMPILER=mpicxx \
-DGMX_BUILD_MDRUN_ONLY=ON \
-DGMX_BUILD_OWN_FFTW=ON \
-DGMX_X11=OFF \
-DGMX_GPU=OFF \
-DGMX_DEFAULT_SUFFIX=ON \
-DBUILD_SHARED_LIBS=OFF \
-DGMX_EXTERNAL_BLAS=OFF \
-DGMX_EXTERNAL_LAPACK=OFF

make -j 16

make install

Then we can run this as root

/opt/software/src/gromacs/gromacs-5.1.2/$ sudo ./install.sh

and wait. Then create a module file

$ cd /opt/software/modulefiles
/opt/software/modulefiles/$ sudo mkdir gromacs
/opt/software/modulefiles/$ cd gromacs
/opt/software/modulefiles/gromacs/$ sudo vim 5.1.2

which looks like

#%Module1.0#####################################################################
##
## gromacs Modulefile
##
##
proc ModulesHelp { } {
global version

puts stderr "\tAdds 64-bit GROMACS compiled with GCC to your environment."
puts stderr "\tDirectory: $root"
}

module-whatis "adds GROMACS to your environment variables"

if {! [ is-loaded openmpi/1.10.2 ] } {
module load openmpi/1.10.2
}
prereq openmpi/1.10.2

if {! [ is-loaded gcc/5.3.0 ] } {
module load gcc/5.3.0
}
prereq gcc/5.3.0

set version 5.1.2
set app gromacs
set root /opt/software/apps/gromacs/${version}

prepend-path PATH $root/bin
prepend-path LD_RUN_PATH $root/lib
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path MANPATH $root/share/man

setenv GMXDIR $root
setenv GMXBIN $root/bin
setenv GMXLDLIB $root/lib
setenv GMXMAN $root/share/man
setenv GMXDATA $root/share
setenv GMXFONT 10x20

So now

$ module avail
---------------------------- /opt/software/modules -----------------------------
gcc/5.3.0 gromacs/5.1.2 openmpi/1.10.2

Let’s try it on a compute node

$ ssh node01
$ module add gromacs
$ gmx -h
:-) GROMACS - gmx, VERSION 5.1.2 (-:

GROMACS is written by:
Emile Apol Rossen Apostolov Herman J.C. Berendsen Par Bjelkmar

Hurray it works!