Guide to compiling Chaste dependencies from source on ARCUS-B

This guide supplants InstallGuides/Arc as far as installing stuff and setting up the environment. And since these instructions are specific to arcus-b, jump straight to InstallGuides/Arc (running code on arcus-b) after following this page.

Some available modules on ARCUS-B are old, or may have been compiled with an older Intel compiler. This may cause compatibility issues when using newer versions of Chaste (post introduction of C++11).

This guide runs through compiling new versions of each Chaste dependency from source, and relies on the following modules:

• python/2.7 for PyCml dependencies
• gcc/5.4.0 as a fallback for dependencies such as boost which prefer it to intel
• intel-compilers/2017 intel-mkl/2017 intel-mpi/2017 for compiling most things, such as PETSc
• cmake/3.8.0
• git/1.9.0

First, in case you have already loaded modules through ~/.bash_profile, run module list to find out which modules are currently loaded. Although it is probably unnecessary, just to make sure that you are in the same situation as these instructions, unload all of these modules by running module unload <loaded_module_1> <loaded_module_2> .... Then run the following to load the required modules

module load python/2.7
module unload intel-compilers/2013 intel-mkl/2013
module load gcc/5.4.0
module load intel-compilers/2017 intel-mkl/2017 intel-mpi/2017
module load cmake/3.8.0
module load git/1.9.0

Make a directory to install dependencies. There should be more than enough space in your $HOME, and these files are less likely to be periodically deleted.

export SOFTWARE=$HOME/software
mkdir $SOFTWARE

PETSc (with HDF5 and PARMETIS)

This PETSc install will use Intel's MPI and MKL.

cd $SOFTWARE

wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.6.4.tar.gz
tar -zxf petsc-lite-3.6.4.tar.gz
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.1/src/hdf5-1.10.1.tar.bz2

MPI_ROOT_DIR=/system/software/linux-x86_64/compilers/intel/intelPS-2017/impi/2017.2.191/intel64/
MKL_ROOT_DIR=/system/software/linux-x86_64/compilers/intel/intelPS-2017/compilers_and_libraries_2017/linux/mkl/lib/intel64/

cd $SOFTWARE/petsc-3.6.4/
export PETSC_DIR=`pwd`

export PETSC_ARCH=linux-intel-opt-mkl
./configure --with-batch --with-make-np=10 --with-fc=0 --with-x=false --with-ssl=false --known-mpi-shared-libraries=1 --with-mpi-dir=$MPI_ROOT_DIR --download-hdf5=../hdf5-1.10.1.tar.bz2 --download-parmetis=1 --download-metis=1 --with-shared-libraries --with-debugging=0 --with-blas-lapack-dir=$MKL_ROOT_DIR

When this has finished, you will be prompted to submit a job as part of the installation process: this is necessary due to the --with-batch specifier, as the installer cannot test the MPI configuration on the head node.

Still inside $SOFTWARE/petsc-3.6.4/, create the following petsc_job.sh (taking care to change <your_email_address>:

#!/bin/bash

# Use the devel partition
#SBATCH --partition=devel

# set the number of nodes
#SBATCH --nodes=1

# set max wallclock time
#SBATCH --time=00:01:00

# set name of job
#SBATCH --job-name=petsc_config

# mail alert at start, end and abortion of execution
#SBATCH --mail-type=ALL

# send mail to this address
#SBATCH --mail-user=<your_email_address>

# run the application
export PETSC_DIR=$SOFTWARE/petsc-3.6.4
export PETSC_ARCH=linux-intel-opt-mkl
$SOFTWARE/petsc-3.6.4/conftest-linux-intel-opt-mkl

and run it with sbatch petsc_job.sh. Because it is set to run on the devel partition, it should start almost instantly, and will only take a few seconds.

Then, finish the installation with

cd $SOFTWARE/petsc-3.6.4
python reconfigure-linux-intel-opt-mkl.py
make all

Boost

cd $SOFTWARE
wget http://downloads.sourceforge.net/project/boost/boost/1.58.0/boost_1_58_0.tar.bz2
tar -jxf boost_1_58_0.tar.bz2
cd $SOFTWARE/boost_1_58_0/
./bootstrap.sh --prefix=$SOFTWARE/boost_1_58 && ./b2 -j 10 install

VTK

cd $SOFTWARE
wget https://github.com/Kitware/VTK/archive/v7.1.1.tar.gz
tar -zxf v7.1.1.tar.gz
mkdir $SOFTWARE/vtk-build-7.1 && cd $SOFTWARE/vtk-build-7.1
cmake -DCMAKE_INSTALL_PREFIX=$SOFTWARE/vtk-7.1 -DCMAKE_INSTALL_RPATH=$SOFTWARE/vtk-7.1/lib -DCMAKE_BUILD_TYPE=Release ../VTK-7.1.1 && make -j10 && make install

XSD

XSD just needs extracting from the archive:

cd $SOFTWARE
wget http://www.codesynthesis.com/download/xsd/4.0/linux-gnu/x86_64/xsd-4.0.0-x86_64-linux-gnu.tar.bz2
tar -jxf xsd-4.0.0-x86_64-linux-gnu.tar.bz2

Xerces

cd $SOFTWARE
wget http://archive.apache.org/dist/xerces/c/3/sources/xerces-c-3.1.4.tar.bz2
tar -jxf xerces-c-3.1.4.tar.bz2 && cd $SOFTWARE/xerces-c-3.1.4
./configure --enable-netaccessor-socket --prefix=$SOFTWARE/xercesc_3_1_4 && make && make install

SUNDIALS (CVODE)

Following the rest of these instructions, downloading and compiling Sundials/CVODE results in the error undefined reference to pthread_create when running cmake. You should therefore link to the versions already on arcus-b when it comes up later.

PyCML

Amara and friends seem to already be available with the Python 2.7 module. Double check by running:

cd $SOFTWARE
pip install --user "python-dateutil==1.5"
pip install --user "Amara==1.2.0.2"
pip install --user "rdflib==2.4.2"
pip install --user lxml

Tidy up

If the above has all completed successfully, you can run the following to free up space:

rm $SOFTWARE/boost_1_58_0.tar.bz2
rm -rf $SOFTWARE/boost_1_58_0/

rm $SOFTWARE/v7.1.1.tar.gz
rm -rf $SOFTWARE/vtk-build-7.1
rm -rf $SOFTWARE/VTK-7.1.1

rm $SOFTWARE/xsd-4.0.0-x86_64-linux-gnu.tar.bz2

rm $SOFTWARE/xerces-c-3.1.4.tar.bz2
rm -rf $SOFTWARE/xerces-c-3.1.4

rm $SOFTWARE/hdf5-1.10.1.tar.bz2
rm $SOFTWARE/petsc-lite-3.6.4.tar.gz

Setting the Environment

Adding the following to your ~/.bash_profile (and remove anything Chaste-related from your ~/.bashrc) will allow CMake to pick up all the pacakges:

unset LD_LIBRARY_PATH

# Keep a software directory in $HOME
export SOFTWARE=$HOME/software
alias cdsoftware='cd $SOFTWARE'

# ----- Chaste config ----- #

# Module commands

module load cmake/3.8.0
module load python/2.7
module load git/1.9.0

module unload gcc/4.8.2
module load gcc/5.4.0
module unload intel-compilers/2013 intel-mkl/2013
module load intel-compilers/2017 intel-mkl/2017 intel-mpi/2017

# PETSc
export PETSC_DIR=$SOFTWARE/petsc-3.6.4
export PETSC_ARCH=linux-intel-opt-mkl

# Boost
export BOOST_ROOT=$SOFTWARE/boost_1_58

# XSD
export XSD_ROOT=$SOFTWARE/xsd-4.0.0-x86_64-linux-gnu

# Xerces
export XERCESC_INCLUDE_DIR=$SOFTWARE/xercesc_3_1_4/include
export XERCESC_LIBRARY_DIR=$SOFTWARE/xercesc_3_1_4/lib

# CVODE
export SUNDIALS_ROOT=/system/software/linux-x86_64/lib/cvode/2.7.0

# Define CMake variables needed by Chaste
Chaste_OPTS="-DCMAKE_BUILD_TYPE=RELEASE -DChaste_UPDATE_PROVENANCE=OFF -DChaste_ERROR_ON_WARNING=OFF"
BOOST_OPTS="-DBoost_NO_BOOST_CMAKE=BOOL:ON -DBOOST_ROOT=$BOOST_ROOT"
VTK_OPTS="-DVTK_DIR=$SOFTWARE/vtk-7.1/lib/cmake/vtk-7.1"

# Combine the CMake lines defined above
export Chaste_REL_OPTS="$Chaste_OPTS $BOOST_OPTS $VTK_OPTS"

# ----- End Chaste config ----- #

# Useful aliases
alias cdchaste='cd /path/to/build/dir'
alias cmchaste='cmake $Chaste_REL_OPTS /path/to/chaste/src'
alias myjobs='squeue -o "%.7i %.30j %.2t %.19S %.10M %.6D %R" -u <username>'
alias gitpull='git -C /path/to/your/chaste/project pull'

Building Chaste

Make sure you have started a new session since adding the above to your ~/.bash_profile, i.e. literally log out and log back in.

cd /path/to/build/dir  # or just "cdchaste"
# make sure the build directory exists and that you are in it, otherwise the next line will ruin your life
rm -rf *  # just to make sure it's a clean build, don't do this every time
cmake $Chaste_REL_OPTS /path/to/chaste/src  # or just "cmchaste"
make -j11 chaste_heart