As I’ve described here, here and here, I ran a Hackathon instead of the more traditional project in the third week of the Bioinformatics module course for around 30 students from the University of Oxford Interdisciplinary Biosciences programme in November 2017.
One of the students, Ellen Pasternack, wrote a guest blog post for Microsoft which you can read here in which she explains how they came to investigate the genetics of bird migration.
In this post, I’ll spell out some of the problems we encountered using Microsoft Azure to run a 3-week course for about 30 postgraduates in a typical “computer lab”. As you’ll see, a group of cloud-naive highly intelligent postgraduates are capable of breaking nearly anything and, perhaps, might constitute the perfect resilience test for your cloud software, if it survives that is…
As noted in the Azure instructions, we chose the simplest approach which was for all teachers and students to share the same Azure Subscription, using different Resource Groups for different parts of the course. As this is about 30-40 people in total, you quickly end up with a LOT of different VMs, Public IP addresses etc that, if they aren’t well named, gets very confusing and no-one wants to clean things up as they are afraid they’ll delete someone else’s Virtual Machine.
But even so you will find the odd, err, pedantic student. We asked them to call their VM “phylogenetics_YOURNAME” so guess what? Yep. Someone made one called “phylogenetic_YOURNAME”. The only solution I can see to this is ruthless deletion of anything that violates your naming scheme in BOFH style.
Because we didn’t and, as we hadn’t followed (a), quickly discovered that, by default, you can’t have any more than 60 Public IP addresses per subscription. (Later we also found there is a limit of 100 Security Groups per subscription). This led to some slightly panicked deleting during which an over-zealous student deleted the Image that one of the teachers had lovingly created for their course….
Fortunately, the Image that was deleted was for the previous day’s course (not the next day’s). One of the teachers found that you can put delete locks on resources which is a very, very good idea for anything that you value as students can and will delete everything to get their VM working. More on this soon.
Not a new problem. In one of the practicals, because they were accessing the remote VM graphically via RDP (rather than simply via the terminal) it was necessary to have port 3389 open, as well as the default port 22. The practical clearly explained that you should attach the pre-defined security group that had this rule, but about half the class ignored this and created their own security group which, of course, had no port 3389 open, so they couldn’t connect graphically and then complained.
Which brings us back to here, again. You need to picture the scene; about 15 students can only access their VM via the terminal as they haven’t followed the instructions and so are a bit grumpy and the other half are happily working away using an RDP clone on Ubuntu. Then one of the students, for reasons unknown, edits the shared Security Group and deletes the port 3389 rule. Instantly, half the class (who were feeling smug) are booted off their remote VM. Instant panic and confusion. Try figuring that one out on the hoof. We couldn’t – it was only later looking through the logs that we could see students trying to edit the Shared Security Group.
So, after all of this, I was pleasantly surprised that the feedback was positive and all the students could see how using cloud could help them in their research.
This is the second year that I have organised a three-week course on Bioinformatics from PhD students in their first year of the Oxford Interdisciplinary Biosciences programme. As last year, I run the third week as a Hackathon and in preparation the students had to choose a scientific paper in the first week, form teams and then present back on the Monday of the Hackathon (third) week. Unlike last year, there was very little change in teams as the Hackathon started. Also unlike last year, we had the full capability of Microsoft Azure for them to play with (which should make e.g. running meaningful molecular dynamics simulations much easier).
At the end of the course, I asked the students to fill in some online feedback. I won’t go through of it here, but will instead focus on how they found using Microsoft Azure throughout the three weeks and what they thought of the hackathon. Let’s tackle Azure first.
These are the average scores from the relevant questions:
On the whole, they were positive about using Azure, with most averages hovering around “Agree”. This does, of course, hide a range of opinions, so whilst, on average, they agreed with the statement “Azure made the computer practicals easier”, that hides an almost exactly even split between the number that disagreed to some extent (11) and those that agreed to some extent (10), the only difference being that more people Strongly agreed than Strongly disagreed, resulting in the final weighted score. We did encounter some serious difficulties, mainly as a result of our inexperience as teachers in using Azure, and so I am both not surprised in this split and also encouraged. For the other scores, I’ll give an indication of the number of people who disagreed to some extent (i.e. Disagree + Strongly disagree) and the number who agreed to some extent (i.e. Agree + Strongly agree).
There was more support for “creating and accessing my own Virtual Machine was straightforward” 15 agreeing and 5 not. This reflected, I hope, the effort I had put into creating an easy and comprehensive walkthrough about how to do this. One should also consider Azure in the context of the alternative (as that is never perfect either), which here is remotely deploying the right software to a homogenous set of Dell PCs running Ubuntu 16.04 LTS Desktop. On the first day we had a whole bunch of issues with logins etc which is reflected in the average score above, which breaks down into 8 agreeing and 10 disagreeing. The next question was a bit pessimistic: “In the future I would just stick to just using the DTC workstations for the computer practicals”, and 9 disagreed and only 6 agreed.
The questions looking to the future brought out more optimism. Eight students said that “I intend using the Bioinformatics Azure subscription after the course” and only 5 disagreed with 11 abstentions. This was no pure sentiment either as at least three of the hackathon groups continued using Azure after the course and burnt through the remaining resources that had been allocated to that subscription sometime over the Christmas vacation. Finally, all 24 students agreed (or strongly agreed) with the statement “I can see how cloud computing could be useful in my research” which I interpret as they could see past the niggles and appreciate the potential of cloud computing.
You can read about some of the problems that we bumped into in a separate post.
I reported how last year the students much preferred a hackathon to a more traditional “project” and not much has changed. I asked the same questions:
Of the 24 students who answered my feedback questionnaire, 23 of them enjoyed the hackathon and 22 preferred it to doing a project by myself. Two found it hard to work in a team, whilst a bit over half the class sat on the fence when it came to GitHub leaving 9 to agree with the statement “GitHub is useful” and 1 to disagree. There was a full range of answers to “Prizes motivated me”, but I would always have some form of geeky-but-not-too-expensive prize at the end of a Hackathon whatever the answer to this question. Lastly, and most pleasingly, all 24 students agreed or strongly agreed with the statement “I learnt some skills during the hackathon that will be useful during my DPhil”.
So I think we’ll keep doing a hackathon in the third week! Here are photos of some of the teams just after they’ve delivered their presentation on the last Friday – at least one of them had been up most of the night coding.
Last year I took over coordinating a three-week Bioinformatics module for the Interdisciplinary Bioscience Doctoral Training Centre in Oxford. Much of the course is taken up with computational practicals which we usually run on the low-spec desktop PCs that they have in the DTC. This year though we have been fortunate to receive a grant from Microsoft Azure to try using their cloud instead which I believe has three main advantages.
These instructions are therefore aimed at students and teachers who are using Azure to create and use Virtual Machines in a computational practical, but they might be useful if you are trying to use Azure for the first time.
Importantly, these instructions follow the procedure we chose to use, which is for all students and teachers to belong to the same subscription. This means everyone can automatically see any Images that have been created, but as I will explain in a subsequent post, it can bring some (unexpected) problems as well. There are probably more sophisticated ways of using Azure but this is how we did it (and by the time you read this, Microsoft may well have improved things yet further).
If you don’t have a Microsoft account, go to https://signup.live.com and create one. It can be easier if you use a personal email address (like Gmail, Hotmail etc) rather than an institutional or business email address and Microsoft may try and link that to e.g. your University Office365 account if you have one.
After entering your email address and password Microsoft will send you a confirmation email with a code you will need to enter on the next page. Then you’ll need to prove you are a human by transcribing a Captcha. (This sometimes took me several attempts to get right)
If you already have a Microsoft account, please check by logging in to
https://account.microsoft.com/account
If you can’t remember the password, please reset it.
Whether you have just setup an account, or logged into your existing Microsoft account, you need to be able to see your Microsoft account screen as shown below.
If you are a student, at this point you need to let your course organiser know your Microsoft identity email address so they can add to you the course Azure subscription (probably using the Educator portal). This will give you access to some funds to launch and run Virtual Machines. If you are running Azure yourself, you’ll need to create your own pay-as-you-go subscription which has your credit card details attached – you can do this in the Azure Portal. All the following instructions assume you you are a student getting access through your course organiser.
If you did send your email address to your course organiser there is one more important step. You will get an email like this from invites@microsoft.com and you must click Get Started which will then invite you to login and take you through to the Azure Portal.
If the above steps didn’t take you to the Portal, navigate there and login. First we need to check you’ve been added to the subscription for your course.
Your email address will be shown at the top right of the screen. If you look directly beneath it you will see either a long alphanumeric string or an email address. What is correct will depend on exactly where your funds to use Azure have come from; since we were awarded some time by Microsoft, we saw the email address of the administrative owner of the grant.
Now let’s launch (“spin-up”) a small Linux virtual machine and log in remotely. Log into the Azure portal with your Microsoft identify and on the Dashboard click Create Resources (highlighted with a red box).
On the next window, choose Ubuntu Server 16.04 LTS, which at the time of writing was the most recent Long Term Support version of the Ubuntu Linux distribution.
On the next page you will need to enter some options. These are
Since we are all sharing the same subscription, and hence all our resources will appear in the same list, make sure you can identify what is yours by, for example, appending your initials to the name. Don’t call anything “test” or “VirtualMachine”! I’m going to call mine LinuxTestPWF as PWF is my initials so I can instantly see it is mine.
Once you’ve filled out the options, click OK.
Now we get to choose the size of the Virtual Machine. By size we mean the number and speed of CPU cores it has, how much physical memory. There are a large number of options, and each location only has a subset. Understandably, our free access doesn’t give us access to some of the largest and most expensive options, for example, there are Virtual Machines with 4x NVIDIA K80 GPU cards which would allow you to run molecular dynamics blazingly fast.
By default Azure shows you three small, cheap, general-purpose instance types. (If you want to see the full list available to you at your chosen location, click View all). Let’s choose the first one which is a D2S_V3 instance as that should be good enough. Click it and then choose Select
Now we are presented with some additional options. Leave all of them set to the defaults, except Auto-shutdown. This is a nifty feature that, when activated, will automatically shutdown your instance at the time specified (7pm by default), so if you forget to stop your Virtual Machine you will only be charged up to 7pm that day.
Finally, the Azure Portal summarises all the choices we’ve made and asks us if we are happy to proceed.
You don’t have to check the box – just click Create and wait.
You’ll start to see some messages about your Virtual Machine being deployed. Azure is now finding an empty slot in the UK South datacenter and copying in an Ubuntu 16.04 image, adding the user you specified, booting it up and connecting it to the network etc. This can take a few minutes.
When it tells you it is done, click the All resources blade on the left. Notice that there is more than one resource here; Azure has not only created a Virtual Machine called LinuxTestPWF, but has also created a Virtual Network and a Network Interface and a Public IP address. Here you can see one of the key points of cloud; everything has been abstracted. In other words, my Virtual Machine is independent of the IP address which is independent of the Storage. Click on your Virtual Machine (if the list is very long, you might want to Filter by name, or change the type so only Virtual Machines are displayed).
This pane is very useful; it shows us all sorts of specifics, like whether the VM is running, the Location it is in, its Size, and, crucially, its IP address (in blue). Copy the Public IP address and, in a Terminal on your workstation or laptop, ssh into the remote Virtual Machine, putting your user name and the IP address of your Virtual Machine.
You should then arrive at a prompt on your Azure Virtual Machine! Well done!
At this point you can install additional software and configure your Virtual Machine until it is how you want it. To copy some files (here, hello.txt) from your workstation to the VM, open another Terminal and use scp (or rsync if you are familiar with that)
Now comes the neat bit. Let’s say we want to go home, but we are not finished using our Virtual Machine and we don’t want to pay for having our VM “up” overnight. Well, if we set an Auto-shutdown time we could let it run until then. Or we could Stop the VM, which would free up the slot in the datacenter, release the IP address and hence all we would be paying for would the disk space necessary to store the virtual machine image. Then tomorrow morning we could Start the VM back up again and it would be exactly as we left it! One key difference is the IP address as Azure will have assigned our VM a different one. Try it! Just be warned both stopping and starting a VM can take a few minutes each. Click Stop, wait until the Status is listed as Stopped (deallocated), then click Start.
This highlights an important difference between Stop and Delete. The latter will deallocate the Virtual Machine AND delete the Virtual Machine image so there is no going back. Since storage is generally a lot cheaper than using a computing slot in the Datacenter, it is ok to keep some stopped Virtual Machines, but please tidy up, by deleting resources, as you will rapidly get confused which one is right one. Since we all share the same subscription, the list of resources will get long very quickly…
This section is more relevant for instructors, but I’ve included it since it shows you how you can setup a Virtual Machine with input files, instructions and programs and then capture it as an Image which other people belonging to the same subscription can use to launch Virtual Machines, allowing you to rapidly allow other people to try out your code, or in our case, run a Bioinformatics exercise. Don’t go through it step-by-step unless you want to really want to try it out. This might be useful for the Hackathon though..
There are some very good instructions by Microsoft on how to create an image of a Linux Virtual Machine so I will just summarise the main steps here. On your local computer you would also need to have installed the Azure Command Line Interface (CLI) but you may find it already installed.
As a simple (but daft) test we will save the Virtual Machine that was created in the previous section which is identical to a clean Ubuntu 16.04 server with one key difference; there is a file called hello.txt at the root of the filesystem.
First, we must deprovision the VM. Typically, you also remove the last installed user, so that when people specify their own user when they setup their own Virtual Machine based on this image you only get one user account. This means that you can’t leave anything important in the $HOME directory of the user account. If you want to leave the user account simply omit the +user flag in the following. In a terminal that is logged into your Virtual Machine in the state that you wish to save and create an Image type.
sudo waagent -deprovision+user
Once you answer yes there is no going back. Azure will delete the user account and start the process of turning this Virtual Machine into an Image. Quit the Virtual Machine by typing exit.
All the following commands are in your local Terminal and use the Azure CLI. First we need to login to Azure on the command line.
Copy the URL into a browser, enter the code and then login into Azure, closing the browser window when prompted. After about 20 seconds some text will appear in the Terminal and are now logged in! First we deallocate the Virtual Machine, changing the resource group and name where appropriate (this step can take a minute or two)
az vm deallocate --resource-group 0_Azure_Test --name LinuxTestPWF
Next we generalize it
az vm generalize --resource-group 0_Azure_Test --name LinuxTestPWF
Finally, we create the Image. Make sure you choose a suitable name!
az image create --resource-group 0_Azure_Test --source LinuxTestPWF --name UbuntuTestImage
To summarise we have created an Image called UbuntuTestImage in the 0_Azure_Test Resource. Anyone belonging to the DTP_Bioinformatics subscription can then see this image.
This is very similar to the steps we went through to create an Ubuntu 16.04 Virtual Machine. Except this time, click on the name of the Image that is stored in the DTP_Bioinformatics subscription and then click Create VM.
Now you should see this screen, exactly the same as before! Simply pick up the instructions from this point in the first section and soon you’ll be able to remotely login using ssh.
It looks innocuous sitting on the desk, an Oxford Nanopore MinION, but it can produce a huge data of data from a single sequencing run. Since the nanowire works by inferring which base is in the pore by how much it reduces the flow of ions (and hence current) through the pore, the raw data is commonly called “squiggles”. Converting each of these squiggles to a sequence of nucleotides is “base-calling”. Ideally, you want to do this in real-time, i.e. as the squiggles are being produced by the MinION. Interestingly, this is becoming a challenge for the molecular biologists and bioinformaticians in our group since the flow of data is now high enough that a high-spec laptop struggles to cope. It may becoming obvious that this is not my field of expertise – and you’d be right – but I do know something about speeding up computational jobs through the use of GPUs and/or computing clusters. There appear to be two main use-cases we have for base-calling the squiggles. I’m only going to talk about nanonetcall,
Nick Sanderson is leading the charge here in our group (that is his hand in the photo above). He has built a workstation with a GPU and SSD disc and was playing around with the ability of nanonetcall to use multiple threads. This is our base case, which has a single process with one OpenMP thread, so by definition has a speedup of 1.0x. The folder sample-data/ contains a large number of squiggle files (fast5 files).
OMP_NUM_THREADS=1 nanonetcall sample_data/ --jobs 1 > /dev/null
Let’s first try using 2 OpenMP threads and a single process.
OMP_NUM_THREADS=2 nanonetcall sample_data/ --jobs 1 > /dev/null
This makes no difference whatsoever. In common with running GROMACS simulations, OpenMP doesn’t help much, in my hands at least. Let’s rule out using additional OpenMP threads and simply increase
We can simply try increasing the number of jobs to run in parallel:
OMP_NUM_THREADS=1 nanonetcall sample_data/ --jobs 2 > /dev/null
OMP_NUM_THREADS=1 nanonetcall sample_data/ --jobs 4 > /dev/null
OMP_NUM_THREADS=1 nanonetcall sample_data/ --jobs 8 > /dev/null
OMP_NUM_THREADS=1 nanonetcall sample_data/ --jobs 12 > /dev/null
These lead to speedups of 1.8x, 3.0x, 3.2x and 3.2x. These were all run on my early-2013 MacPro which has a single 6-core 3.5 GHz Intel Xeon CPU, so can run up to 12 processes with hyperthreading. I don’t know exactly how nanonetcall is parallelised, but at least a good chunk of it is Python, it is no surprise that it doesn’t scale that well since Python will always struggle due to the limitations inherent in being an interpreted language (which in Python’s case means the GIL). Now parts of nanonetcall are cleverly written using OpenCL so it can make use of a GPU if one is available. My MacPro has two AMD FirePro D300 cards. Good graphics cards, but I would have chosen better GPUs if I could. Even so using a single GPU gives a speedup of 3.3x.
nanonetcall sample_data/ --platforms Apple:0:1 Apple:1:1 --exc_opencl > /dev/null
I suggested we try one of my favourite apparently-little-known unix tools, GNU Parallel. This is simple but truly awesome command that you can install via a package manager like apt-get on Ubuntu and MacPorts on a Mac. The simplest way to use it is in a pipe like this;
find sample-data/ -name '*.fast5' | parallel -j 12 nanonetcall {} > /dev/null
This needs explaining. The find command will produce a long list of all the fast5 files. Parallel then consumes these, and will first launch 12 nanonetcall jobs, each running on a single core. As soon as one of these finishes, parallel will launch another nanonetcall job to process the next fast5 in the list. In this way parallel will ensure that there are 12 nanonetcall jobs running at all times and we rapidly work out way through the squiggles. This results in a speed up of 4.8x, so not linear, but certainly better than trying to use the ‘internal’ parallelisation of nanonetcall.
But we can do better because we can use parallel to overload the GPU too.
find sample-data/ -name '*fast5' | parallel nanonetcall {} --platforms Apple:0:1 --exc_opencl > /dev/null
This yields our fastest speed-up of 5.7x. But perhaps the GPU is getting too overloaded, so let’s try sharing the loads
find sample-data-1/ -name '*.fast5' | parallel -j 6 nanonetcall {} --platforms Apple:1:1 --exc_opencl > /dev/null &
find sample-data-2/ -name '*.fast5' | parallel -j 6 nanonetcall {} --platforms Apple:0:1 --exc_opencl > /dev/null
where I’ve split the data equally into two folders. Sure enough, this now gives a speedup of 9.9x. Now, remember there are only 12 virtual cores, so if we try running more processes, we should start to see a performance penalty, but let’s try!
find sample-data-1/ -name '*.fast5' | parallel -j 12 nanonetcall {} --platforms Apple:1:1 --exc_opencl > /dev/null &
find sample-data-2/ -name '*.fast5' | parallel -j 12 nanonetcall {} --platforms Apple:0:1 --exc_opencl > /dev/null
Unexpectedly, this ekes out a bit more speed at 10.9x! So by ignoring the inherently poor scalability built-in to nanonetcall and using GNU Parallel in harness with two GPUs, we have managed to speedup the base-calling by a factor of nearly eleven. I expect Nick to manage even higher speedups using more powerful GPUs.
I sit in the same office as two of our bioinformaticians and even with a good setup for live-basecalling, it sounds like there are still occasions when they need to baseball a large dataset (~50GB) of squiggle files. This might be because part of the live base calling process failed, or even the MinION writing files to a different folder due to some software update, or perhaps you simply want to compare several different pieces of basecalling software or even just compare across versions. You want to load the data onto some “computational resource”, press go and have it chew its way through the squiggle files as quickly and reliably as possible. There are clearly many ways to do this; here I am going to look at a simple linux computing cluster with a head node and a large number of slave nodes. Jobs are distributed across the slave nodes using a batch scheduler and queuing system (I use SLURM).
Hang Phan, another bioinformatician, had a large dataset of squiggles that needed 2D base calling and she wanted to try nanonetcall. To demonstrate the idea, I simply installed nanonetcall on my venerable but perfectly serviceable cluster of Apple Xserves. Then it is just a matter of rsync`ing the data over, writing a simple bit of Python to (a) group together sets of fast5 files (I chose groups of 50) and then (b) create a SLURM job submission file for each group and finally (c) submit the job to the queue.
The advantages of this well-established “bare metal” approach are that
– it is truly scalable: if you want to speed up the process, you add more nodes
– it is reliable; my Ubuntu/SLURM/NFS cluster wasn’t switched off for over half a year (and this isn’t unusual)
– you can walk away and let the scheduler handle what runs when
As you can see from my other posts, I am a fan of cloud and distributed computing, but in this case a good old-fashioned computing cluster (preferably one with GPUs!) fits the bill nicely.
I’m planning to launch a citizen science project, bashthebug.net, in 2017 which has two distinct ways anyone can help combat antibiotic resistance. I’ve revamped and relaunched what will ultimately become the public-facing project website – please have a look.
The first strand is closer to the light of day and will help the international Tuberculosis consortium, CRyPTIC. This global group of researchers, of which I am a part, will be collecting over 100,000 samples from patients with TB. Each sample will be tested to see which antibiotics are effective as well as having the genome of its M.tuberculosis bacterium sequenced. In practice, because each sample is measured at least three different times, that means looking at 300,000 96-well plates. Step forward Zooniverse! This type of large-scale image classification is exactly the sort of thing Zooniverse Citizen Science projects excel at. I hope to launch this project in early 2017.
The second citizen science project is more complex and I have recently applied for funding. As described in my Research, I am developing methods that can predict whether novel or rarely-observed mutations cause resistance to an antibiotic (or not). These require a lot of computer resource and the idea is to build a volunteer computing project, like [climate prediction.net](http://climate prediction.net), using the BOINC framework, so that volunteers can download a program onto their laptop or desktop. When they’re not using their computer, the program will retrieve part of a problem and run the simulations on their machine before returning the results over the internet. These type of project is more complicated and requires more infrastructure to be setup, but with some luck, I’d hope to have a soft launch late in 2017.
DOCKER is cool. But what is it? From the DOCKER webpage
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.
I like to think of it as somewhere in between virtualenv and a virtual machine. Although the DOCKER website is focussed on commercial software development, and so talks about building and shipping applications, DOCKER could be of huge use to myself as a computational scientist. For example, rather than make a series of input files for my simulations available, along with a list of which software versions I used, I could instead simply make a DOCKER image available that contains all the compiled software I used along with all the input files. Then anyone should, in principle, be able to reproduce my research.
Make no mistake: reproducibility is, rightly, a coming trend. But surely all scientific results are reproduced?. Turns out if the experiment or simulation was difficult to do the answer is not so much. And when concerted efforts have been made to reproduce results reported in high impact journals, the answer is often, well, disconcerting at the very least. In a now famous study, Begley & Ellis from a pharmaceutical company, Amgen, reported that their in-house scientists were unable to reproduce 47 out of 53 landmark experimental studies in haematology and oncology. They were looking at novel, exciting findings which are more likely to be challenging to reproduce (although the pressure to over-sell is also stronger). I have no reason to think computational studies are much better. The past few years there have been a flurry of papers, comments and best practices. One can even now make a DOCKER image available via GitHub with a DOI so it can be cited independently of an article.
As I’d like to do this in the future, I’ve started to play with DOCKER and GROMACS. Since my workstation is a Mac, the DOCKER host has to run within a lightweight Linux virtual machine. First I installed DOCKER. Then I opened a DOCKER Quick Terminal and checked everything was working by downloading the hello-world image and running it
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4276590986f6: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:4f32210e234b4ad5cac92efacc0a3d602b02476c754f13d517e1ada048e5a8ba
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.
Let’s get try something more real, like an Ubuntu 16.04 Server image.
$ docker run -it ubuntu bash
This drops me inside the Ubuntu image. Let’s compile GROMACS!
root@4b511a41dbf0:/# apt-get update -y
root@4b511a41dbf0:/# apt-get upgrade -y
root@4b511a41dbf0:/# apt-get install build-essential cmake wget openssh-server -y
root@4b511a41dbf0:/# wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.2.tar.gz
root@4b511a41dbf0:/# tar zxvf gromacs-5.1.2.tar.gz
root@4b511a41dbf0:/# cd gromacs-5.1.2
root@4b511a41dbf0:/# mkdir build
root@4b511a41dbf0:/# cd build
root@4b511a41dbf0:/# cmake .. -DGMX_BUILD_OWN_FFTW=ON
root@4b511a41dbf0:/# make
root@4b511a41dbf0:/# make install
root@4b511a41dbf0:/# cd
Now let’s copy over a TPR file to see how fast GROMACS is within a DOCKER container
root@4b511a41dbf0:/# scp fowler@somewhere.else:benchmark.tpr .
root@4b511a41dbf0:/# source /usr/local/gromacs/bin/GMXRC
root@4b511a41dbf0:/# gmx mdrun -s benchmark -resethway -noconfout -maxh 0.1
Note that this is a single CPU DOCKER image. I was worried that since the DOCKER host was running inside a Linux VM it would be slow compared to running natively in Mac OS X so I ran three repeats of each and DOCKER was only 1.7% slower…
To save this DOCKER image locally, quit the session
$ docker commit -m "Installed GROMACS 5.1.2 for benchmarking" -a "Philip W Fowler" c5f1cf30c96b philipwfowler/gromacs-5.1.2
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
philipwfowler/gromacs-5.1.2 latest 73e44c120bfa 6 seconds ago 809 MB
ubuntu latest c5f1cf30c96b 2 weeks ago 120.8 MB
hello-world latest 94df4f0ce8a4 3 weeks ago 967 B
Done. More soon on multiple cores, can-we-use-the-GPU? and using DOCKER on Amazon Web Services.
If you want to compile GROMACS to run on a GPU Amazon Web Services EC2 instance, please first read these instructions on how to compile GROMACS on an AMI without CUDA. These instructions then explain how to install the CUDA toolkit and compile GROMACS against it.
The first few steps are loosely based on these instructions, except rather than download the NVIDIA driver, we shall download the CUDA toolkit since this includes an NVIDIA driver. First we need to make sure the kernel is updated
sudo yum install kernel-devel-`uname -r`
sudo reboot
Safest to do a quick reboot here. Assuming you are in your HOME directory, move into your packages folder.
cd packages/
And download the CUDA toolkit (version 7.5 at present)
wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run
sudo /bin/bash cuda_7.5.18_linux.run
It will ask you to accept the license and then asks you a series of questions. I answer Yes to everything except installing the CUDA samples. Now add the following to the end of your ~/.bash_profile using a text editor
export PATH; PATH="/usr/local/cuda-7.5/bin:$PATH"
export LD_LIBRARY_PATH; LD_LIBRARY_PATH="/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH"
Now we can build GROMACS against the CUDA toolkit. I’m assuming you’ve already downloaded a version of GROMACS and probably installed a non-CUDA version of GROMACS (so you’ll already have one build directory). Let’s make another build directory. You can call it what you want, but some kind of consistent naming can be helpful. The -j 4 flag assumes you have four cores to compile on – this will depend on the EC2 instance you have deployed. Obviously the more cores, the faster, but GROMACS only takes minutes, not hours.
mkdir build-gcc48-cuda75
cd build-gcc48-cuda75
cmake .. -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs/5.0.7-cuda/ -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda
make -j 4
sudo make install
To load all the GROMACS tools into your $PATH, run this command and you are done!
source /usr/local/gromacs/5.0.7-cuda/bin/GMXRC
If you run this mdrun binary on a GPU instance it should automatically detect the GPU and run on it, assuming your MDP file options support this. If it does you will see this zip by in the log file as GROMACS starts up
1 GPU detected:
#0: NVIDIA GRID K520, compute cap.: 3.0, ECC: no, stat: compatible
1 GPU auto-selected for this run.
Mapping of GPU to the 1 PP rank in this node: #0
Will do PME sum in reciprocal space for electrostatic interactions.
Depending on the size and forcefield you are using you should get a speedup of at least a factor two, and realistically three, using a GPU in combination with the CPUs. For example, see these benchmarks.
These are some quick instructions on how to build a more recent version of GCC than is provided by the devel-tools package on the Cent OS based Amazon Linux AMI. (currently GCC 4.8.3) You may, for example, wish to use a more recent version to compile GROMACS – that is my interest. If so, then these instructions assume you have done all the steps up to, but not including, compiling GROMACS in this post. Compiling GCC needs several GB of disk space so if you use the default 8GB for an EC2 AMI it will run out of disk space; increasing this to 12 GB is sufficient.
First let’s find out what versions of GCC are available.
[ec2-user@ip-172-30-0-42 ~]$ svn ls svn://gcc.gnu.org/svn/gcc/tags | grep gcc | grep release
...
gcc_4_8_3_release/
gcc_4_8_4_release/
gcc_4_8_5_release/
gcc_4_9_0_release/
gcc_4_9_1_release/
gcc_4_9_2_release/
gcc_4_9_3_release/
gcc_5_1_0_release/
gcc_5_2_0_release/
gcc_5_3_0_release/
As you can see when I wrote this 5.3.0 was the most recent stable version, so let’s try that one. I’m going to compile everything inside a folder called packages/ so let’s create that then use subversion to check out version 5.3.0 (this is going to download a lot of files so will take a minute or two)
[ec2-user@ip-172-30-0-42 ~]$ mkdir ~/packages
[ec2-user@ip-172-30-0-42 ~]$ cd ~/packages
[ec2-user@ip-172-30-0-42 packages]$ svn co svn://gcc.gnu.org/svn/gcc/tags/gcc_5_3_0_release/
A gcc_5_3_0_release/config-ml.in
A gcc_5_3_0_release/libitm
...
A gcc_5_3_0_release/fixincludes/fixopts.c
A gcc_5_3_0_release/install-sh
A gcc_5_3_0_release/ylwrap
U gcc_5_3_0_release
Checked out revision 232268.
[ec2-user@ip-172-30-0-42 packages]$ cd gcc_5_3_0_release/
GCC needs some prerequisites which are installed by this script.
[ec2-user@ip-172-30-0-42 gcc_5_3_0_release]$ ./contrib/download_prerequisites
--2016-01-12 13:24:23-- ftp://gcc.gnu.org/pub/gcc/infrastructure/mpfr-2.4.2.tar.bz2
=> ‘mpfr-2.4.2.tar.bz2’
Resolving gcc.gnu.org (gcc.gnu.org)... 209.132.180.131
...
isl-0.14.tar.bz2 100%[=====================>] 1.33M 693KB/s in 2.0s
2016-01-12 13:24:39 (693 KB/s) - ‘isl-0.14.tar.bz2’ saved [1399896]
Go up a level, make a build directory and move there.
[ec2-user@ip-172-30-0-42 gcc_5_3_0_release]$ cd ..
[ec2-user@ip-172-30-0-42 packages]$ mkdir gcc_5_3_0_release_build/
[ec2-user@ip-172-30-0-42 packages]$ cd gcc_5_3_0_release_build/
Now we are in a position to compile GCC 5.3.0. This took about 50 min using all eight cores of a c3.2xlarge instance, so this is a good moment to go and have lunch. Note that since the instance I am compiling on has 8 virtual CPUs, I can use the -j 8 flag to tell make to use up to 8 threads during compilation which will speed things up. If you are using a micro instance, just omit the
-j 8 (but good luck as that would take a long time).
[ec2-user@ip-172-30-0-42 gcc_5_3_0_release_build]$ ../gcc_5_3_0_release/configure && make -j 8 && sudo make install && echo "success" && date
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether ln works... yes
...
Hopefully you now have a newer version of GCC to compile binaries with. With any luck it might even give you a performance boost.
So we have created an Amazon Machine Image (AMI) with GROMACS installed. In this post I will examine the sort of single core performance you can expect and much this is likely to cost compared to other compute options you might have.
To test the different types of instances you can deploy our GROMACS image on, we need a benchmark system to test. For this I’ve chosen a peptide MFS transporter in a simple POPC lipid bilayer solvated by water. This is very similar to the simulations found in this paper. Or to put it another way: 78,000 atoms in a cube, most of which are water, some belong to lipids and the rest, protein. It is fully atomistic and is described using the CHARMM27 forcefield.
I tried to use a range of compute resources to provide a good comparison for AWS. First, and most obviously, I used my workstation on my desk, which is a late-2013 MacPro which has 12 Intel Xeon cores. In our department we also have a small compute cluster, each node of which has 16 cores. Some of these nodes also have a K20 GPU. Then I also have access to a much larger computing cluster run by the University. Unfortunately, since the division I am in has decided not to contribute to its running, I have to pay for any significant usage.
Rather than test all the different types of instances available on EC2, I tested an example from each of the current (m4) and older generation (m3) of non-burstable general purpose instances. I also tested an example from the latest generation of compute instances (c4) and finally the smaller instance from the GPU instances (g2).
The performance, in nanoseconds per day for a single compute core, is shown on the left (bigger is better).
One worry about AWS EC2 is that for a highly-optimised compute code, like GROMACS, performance might suffer due to the layers of virtualisation, but, as you can see, even the current generation of general purpose instances is as fast as my MacPro workstation. The fastest machine, perhaps unsurprisingly, is the new University compute cluster. On AWS, the compute c2 class is faster than the current general purpose m4 class, which in turn is faster than the older generation general purpose m3 class. Finally, as you might expect, using a GPU boosts performance by slightly more than 3x.
I’m going to do a “real” comparison. So if I buy a compute cluster and keep it in the department I only have to pay the purchase cost but none of the running costs. So I’m assuming the workstation is £2,500 and a single 16-core node is £4,000 and both of these have a five year lifetime. Alternatively I can use the university’s high performance computing clusters at 2p per core hour. This obviously is unfair on the university facility as this does include operational costs, like electricity, staff etc, and you can see that reflected in the difference in costs.
So AWS EC2 more or less expensive? This hinges on whether you use it in the standard “on demand” manner or instead get access through bidding via the market. The later is significantly cheaper but you only have access whilst your bid price is above the current “spot price” and so you can’t guarantee access and your simulations have to be able to cope with restarts. Since the spot price varies with time, I took the average of two prices at different times on Wed 13 Jan 2016.
As you can see AWS is more expensive per core hour if you use it “on demand”, but is cheaper than the university facility if you are willing to surf the market. Really, though we should be considering the cost efficiency i.e. the cost per nanosecond as this also takes into account the performance.
When we do this an interesting picture emerges: using AWS EC2 via bidding on the market is cheaper than using the university facility and can be as cheap as buying your own hardware even if you don’t have to pay the running costs. Furthermore, as you’d expect, using a GPU decreases cost and so should be a no-brainer for GROMACS.
Of course, this assumes lots of people don’t start using the EC2 market, thereby driving the spot price up…