Introduction

Here you will find a step by step guide to downloading, configuring, and running the Einstein Toolkit. You may use this tutorial on a workstation or laptop, or on a supported cluster. Configuring the Einstein Toolkit on an unsupported cluster is beyond the scope of this tutorial. If you find something that does not work, please feel free to edit the wiki or mail users@einsteintoolkit.org.

Prerequisites

When using the Einstein Toolkit on a laptop or workstation you will need a number of packages installed in order to download, compile and use the Einstein Toolkit components.

Before you build, type "which mpirun". If nothing is found, please load a version of mpi into your path before continuing. The Einstein Toolkit can build many of the dependencies it needs for itself. However, this can take time, and make a long build process even longer. If you have administrative priviliges on your machine, you would do well to install the following packages. Note that on Centos, you will have to do "yum install -y epel-release" for some of the important packages below to install.

Package	Debian /Ubuntu	fedora	centos	opensuse
jpeg	apt-get install -y libjpeg-turbo?-dev	dnf install -y libjpeg-turbo-devel	yum install -y libjpeg-turbo-devel
curl	apt-get install -y curl	dnf install -y curl	yum install -y curl	zypper install -y curl
gcc	apt-get install -y gcc	dnf install -y gcc	yum install -y gcc	zypper install -y gcc
git	apt-get install -y git	dnf install -y git	yum install -y git	zypper install -y git
lapack	apt-get install -y liblapack-dev	dnf install -y lapack-devel	yum install -y lapack-devel	zypper install -y lapack-devel
make	apt-get install -y make	dnf install -y make	yum install -y make	zypper install -y make
subversion	apt-get install -y subversion	dnf install -y subversion	yum install -y subversion	zypper install -y subversion
g++	apt-get install -y g++	dnf install -y gcc-c++	yum install -y gcc-c++	zypper install -y gcc-c++
which		dnf install -y which	yum install -y which	zypper install -y which
papi	apt-get install -y libpapi-dev	dnf install -y papi-devel	yum install -y papi-devel	zypper install -y papi-devel
python	apt-get install -y python	dnf install -y python	yum install -y python	zypper install -y python
hwloc	apt-get install -y libhwloc-dev	dnf install -y hwloc-devel	yum install -y hwloc-devel	zypper install -y hwloc-devel
mpi	libopenmpi-dev libhdf5-openmpi-dev	openmpi-devel hdf5-openmpi-devel	openmpi-devel hdf5-openmpi-devel	zypper install -y openmpi-devel
ssl	apt-get install -y libssl-dev	dnf install -y openssl-devel	yum install -y openssl-devel	zypper install -y libopenssl-devel
libtool		dnf install -y libtool-ltdl-devel	yum install -y libtool-ltdl-devel
numa	apt-get install -y numactl	dnf install -y numactl-devel	yum install -y numactl-devel	zypper install -y libnuma-devel
gfortran	apt-get install -y gfortran	dnf install -y gcc-gfortran	yum install -y gcc-gfortran	zypper install -y gcc-fortran
xargs		dnf install -y findutils	yum install -y findutils
hdf5	apt-get install -y libhdf5-dev	dnf install -y hdf5-devel	yum install -y hdf5-devel	zypper install -y hdf5-devel
fftw	apt-get install -y libfftw3-dev	dnf install -y fftw-devel	yum install -y fftw-devel	zypper install -y libfftw3-3
patch	apt-get install -y patch	dnf install -y patch	yum install -y patch	zypper install -y patch
gsl	apt-get install -y libgsl-dev	dnf install -y gsl-devel	yum install -y gsl-devel	zypper install -y gsl-devel
pkg-config	apt-get install -y pkg-config	dnf install -y pkgconfig	yum install -y pkgconfig	zypper install -y pkg-config

Download

A script called GetComponents is used to fetch the components of the Einstein Toolkit. GetComponents serves as convenient wrapper around lower level tools like git and svn to download the codes that make up the Einstein toolkit from their individual repositories. You may download and make it executable as follows:

%cd ~/

!curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2017_06/GetComponents
!chmod a+x GetComponents

GetComponents accepts a thorn list as an argument. To check out the needed components:

!./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2017_06/einsteintoolkit.th

%cd ~/Cactus

Configure and build

The recommended way to compile the Einstein Toolkit is to use the Simulation Factory ("SimFactory").

Configuring SimFactory for your machine

The ET depends on various libraries, and needs to interact with machine-specific queueing systems and MPI implementations. As such, it needs to be configured for a given machine. For this, it uses SimFactory. Generally, configuring SimFactory means providing an optionlist, for specifying library locations and build options, a submit script for using the batch queueing system, and a runscript, for specifying how Cactus should be run, e.g. which mpirun command to use.

!./simfactory/bin/sim setup-silent

After this step is complete you will find your machine's default setup under ./simfactory/mdb/machines/<hostname >.ini You can edit some of these settings freely, such as "description", "basedir" etc. Some entry edits could result in simulation start-up warnings and/or errors such as "ppn" (processor-per-node meaning number of cores on your machine), "num-threads" (number of threads per core) so such edits must be done with some care.

Building the Einstein Toolkit

Assuming that SimFactory has been successfully set up on your machine, you should be able to build the Einstein Toolkit with the command below. The option "--mdbkey make 'make -j2'" sets the make command that will be used by the script. The number (in the example below it is 2) is the number of processors used when building. Even in parallel, this step may take a while, as it compiles all the thorns specified in manifest/einsteintoolkit.th.

!./simfactory/bin/sim build --mdbkey make 'make -j5' --thornlist ../einsteintoolkit.th

Running

You can now run the Einstein Toolkit with a simple test parameter file.

!./simfactory/bin/sim create-submit helloworld \
    --parfile arrangements/CactusExamples/HelloWorld/par/HelloWorld.par --walltime 0:5:0

The above command will submit the simulation to the queue naming it "helloworld" and ask for a 5 minutes long job time, if you are running on a cluster, or run it immediately in the background if you are on a personal laptop or workstation. You can check the status of the simulation with the command below. You can run this command repeatedly until the job shows

[ACTIVE (FINISHED)...

as it's state. Prior to that, it may show up as QUEUED or RUNNING.

!simfactory/bin/sim list-simulations helloworld

Once it finished you can look at the output with the command below.

!simfactory/bin/sim show-output helloworld

If you see

INFO (HelloWorld): Hello World!

anywhere in the above output, then congratulations, you have successfully downloaded, compiled and run the Einstein Toolkit! You may now want to try some of the other tutorials to explore some interesting physics examples.

What follows is the much more computationally intensive example of simulating a static TOV star. We will first use two sed commands to strip this down from a production quality run to something more manageable. Note that we use 5 procs for this simulation, which may be more than your machine supports. If that is the case, please adjust procs and ppn-used accordingly.

# modify parameter file for smaller memory footprint using sed by changing CoordBase::dx from 8 to 12
!sed '/CoordBase::d[xyz]/s/8/12/' < par/static_tov.par > par/static_tov_small.par
# Shorten the run time to 200
!sed '/cctk_final_time/s/1000/250/' <  par/static_tov_small.par >  par/static_tov_small_short.par

# start simulation, watch log output
!./simfactory/bin/sim create-run static_tov \
  --parfile=par/static_tov_small_short.par --procs=5 --num-threads=1 --ppn-used=5  --walltime=8:0:0

# This cell enables inline plotting in the notebook
%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

Plotting the Output

Numpy has a routine called genfromtxt() which is an extremely efficient reader of textual arrays of floating point numbers. This is well-suited to Cactus .asc files.

lin_data = np.genfromtxt("/home/jovyan/simulations/static_tov/output-0000/static_tov_small_short/hydrobase-rho.maximum.asc")

This is all you need to do to plot the data once you've loaded it. Note, this uses Python array notation to grab columns 1 and 2 of the data file.

plt.plot(lin_data[:,1],lin_data[:,2])