<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi. Sorry for the late response.</p>
<div class="moz-cite-prefix">On 10/2/2025 2:10 PM, Omar Elías
Velasco Castillo wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hello, </div>
<div><br>
</div>
<div>This is a followup for retaking my questions about my
failing to submit and run simulations in a remote cluster. </div>
<div><br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can
you show us what the error message(s) are?</blockquote>
<div dir="ltr"><br>
</div>
<div>Yes, the tail of my err file reads:</div>
<div><br>
</div>
<div><font face="monospace">+ set -<br>
ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from
LD_PRELOAD cannot be preloaded: ignored.<br>
ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from
LD_PRELOAD cannot be preloaded: ignored.</font></div>
<div><font face="monospace">--------------------------------------------------------------------------<br>
Primary job terminated normally, but 1 process returned<br>
a non-zero exit code. Per user-direction, the job has been
aborted.<br>
--------------------------------------------------------------------------<br>
ERROR: ld.so: object '/lib64/libpapi.so.5.2.0.0' from
LD_PRELOAD cannot be preloaded: ignored.<br>
/home/ia/ovelasco/simulations/tov_ET_decisiva/SIMFACTORY/exe/cactus_sim:
error while loading shared libraries: libpapi.so.5.2.0.0:
cannot open shared object file: No such file or directory</font></div>
</div>
</blockquote>
<p><font face="monospace">Is it possible this library is installed
on the head node and not the compute nodes? That could be the
problem.</font></p>
<p><font face="monospace">Do you need papi? Maybe build without it.
Why is there a preload?</font></p>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><font face="monospace">=>> PBS: job killed: walltime
864033 exceeded limit 864000<br>
mpirun: abort is already in progress...hit ctrl-c again to
forcibly terminate</font></div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace"><br>
</font></div>
<div dir="ltr"><br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Maybe.
I'm not 100% sure what you are doing. Can you be clearer about
how you are running the ET?</blockquote>
<div><br>
</div>
<div>Sure, as I mentioned in a previous email, my intention is
to run and submit the ET in a queue of a remote machine using
Simfactory using either PBS or SLURM, as I work in different
remote machines from time to time. The key point is that, I
notice that while the sim and cactus_sim build is being done,
it seems that some lines printed on the shell indicate that
when a module is not found in the machine, the ET compiler
"builds a bundle" for those modules that weren't found in the
compilation from some thorns (examples below showing creation
of bundles for GSL and HDF5):</div>
</div>
</blockquote>
That is correct.
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>
<div><font face="monospace"><br
class="gmail-Apple-interchange-newline">
********************************************************************************<br>
Running configuration script for thorn GSL:<br>
GSL selected, but GSL_DIR not set. Checking pkg-config ...<br>
GSL not found. Checking standard paths ...<br>
GSL not found.<br>
Using bundled GSL...<br>
Finished running configuration script for thorn GSL.</font></div>
<div><font face="monospace">********************************************************************************<br>
Running configuration script for thorn HDF5:<br>
Additional requested language support: Fortran<br>
HDF5 selected, but HDF5_DIR not set. Checking pkg-config
...<br>
HDF5 not found. Checking standard paths ...<br>
HDF5 not found.<br>
Using bundled HDF5...<br>
Finished running configuration script for thorn HDF5.</font></div>
<div><br>
</div>
</div>
<div>First, these messages about GSL and HDF5 not being found
surprise me because both modules are located in the /usr
directory and I pointed to them in the configurations file.</div>
</div>
</blockquote>
It may be that HDF5 isn't built with the correct options. You need
hdf5+hl+fortran+cxx+mpi (i.e. you need fortran, cxx, mpi, and hl
enabled).
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>Second. If this build was completed succesfully in the home
or login shell of that machine, then I assume I can run and
submit simulations with the configuration and runscript files
of my machine in PBS or SLURM queues, am I right? If yes then,
why I still haven't been capable of doing so? My simulations
die within some seconds after being started.</div>
</div>
</blockquote>
That should be the case, but every cluster is a special animal.
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div>I include here the optionlist and runscript that I use for
a machine that uses PBS as an example.</div>
<div><br>
</div>
<div>Optionlist:</div>
<div><br>
</div>
<div><font face="monospace">VERSION = 2025 #For Einstein Toolkit
2022_11<br>
<br>
CPP = cpp<br>
#CC = gcc<br>
#CXX = g++<br>
CC = mpicc<br>
CXX = mpic++<br>
</font></div>
</div>
</blockquote>
<font face="monospace">You should use gcc and g++ here.</font>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><font face="monospace"><br>
FPP = cpp<br>
#F90 = gfortran<br>
F90 = mpif90<br>
</font></div>
</div>
</blockquote>
<font face="monospace">Again, gfortran, not mpif90.</font>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><font face="monospace"><br>
CPPFLAGS = -I/usr/include<br>
CPPFLAGS += -DCCTK_VECTOR_DISABLE_TESTS<br>
CPPFLAGS = -D__USE_ISOC99<br>
PPFLAGS = -D_GLIBCXX_USE_C99_MATH<br>
<br>
LDFLAGS = -L/usr/lib64 -lssl -lcrypto -rdynamic<br>
<br>
<br>
FPPFLAGS = -traditional<br>
<br>
CFLAGS = -g -std=gnu99<br>
CXXFLAGS = -g -std=gnu++11 -fpermissive<br>
F90FLAGS = -g -fcray-pointer -ffixed-line-length-none <br>
<br>
DEBUG = no<br>
CPP_DEBUG_FLAGS =<br>
C_DEBUG_FLAGS =<br>
CXX_DEBUG_FLAGS =<br>
<br>
OPTIMISE = yes<br>
CPP_OPTIMISE_FLAGS =<br>
C_OPTIMISE_FLAGS = -O2<br>
CXX_OPTIMISE_FLAGS = -O2<br>
F90_OPTIMISE_FLAGS = -O2<br>
<br>
PROFILE = no<br>
CPP_PROFILE_FLAGS =<br>
C_PROFILE_FLAGS = -pg<br>
CXX_PROFILE_FLAGS = -pg<br>
F90_PROFILE_FLAGS = -pg<br>
<br>
WARN = yes<br>
CPP_WARN_FLAGS = -Wall<br>
C_WARN_FLAGS = -Wall<br>
CXX_WARN_FLAGS = -Wall<br>
F90_WARN_FLAGS = -Wall<br>
<br>
OPENMP = yes<br>
CPP_OPENMP_FLAGS = -fopenmp<br>
FPP_OPENMP_FLAGS = -D_OPENMP<br>
C_OPENMP_FLAGS = -fopenmp<br>
CXX_OPENMP_FLAGS = -fopenmp<br>
F90_OPENMP_FLAGS = -fopenmp<br>
<br>
VECTORISE = no<br>
VECTORISE_ALIGNED_ARRAYS = no<br>
VECTORISE_INLINE = yes<br>
<br>
PTHREADS_DIR = NO_BUILD<br>
<br>
# Para hallar todas estas opciones, hacemos: ldconfig -p |
grep nombredelpaquete<br>
MPI_DIR = /software/TEST/local<br>
# <br>
<br>
LAPACK_DIR = /usr<br>
#/usr/lib64/liblapack.so.3<br>
# lapack-3.4.2-8.el7.x86_64<br>
<br>
BLAS_DIR = /usr<br>
#/usr/lib64/libblas.so.3<br>
# blas-3.4.2-8.el7.x86_64<br>
<br>
HDF5_DIR = /usr<br>
#/usr/lib64/libhdf5.so.8<br>
# hdf5-1.8.12-13.el7.x86_64<br>
<br>
HWLOC_DIR = /usr<br>
#/usr/lib64/libhwloc.so.5<br>
# hwloc-1.11.8-4.el7.x86_64<br>
<br>
JPEG_DIR = /usr<br>
#/usr/lib64/libjpeg.so.62<br>
# <br>
<br>
YAML_DIR = /usr<br>
#/usr/lib64/libyaml-0.so.2<br>
# /usr/lib64/libyaml-0.so.2.0.<br>
<br>
ZLIB_DIR = /usr<br>
#/usr/lib64/imlib2/loaders/zlib.so<br>
# zlib-1.2.7-21.el7_9.x86_64<br>
<br>
GSL_DIR = /usr<br>
#/usr/lib64/libgsl.so<br>
# gsl-1.15-13.el7.x86_64<br>
<br>
FFTW3_DIR = /usr<br>
#/usr/lib64/libfftw3.so<br>
# fftw-3.3.3-8.el7.x86_64<br>
<br>
PAPI_DIR = /usr<br>
#/usr/lib64/libpapi.so.5.2.0.0<br>
# papi-5.2.0-26.el7.x86_64<br>
<br>
XML2_DIR = /usr<br>
#/usr/lib64/libxml2.so.2<br>
# xml2-0.5-7.el7.x86_64<br>
<br>
NUMA_DIR = /usr<br>
#/usr/lib64/libnuma.so.1<br>
<br>
OPENSSL_DIR = /usr<br>
#/usr/lib64/libssl3.so<br>
# openssl-1.0.2k-26.el7_9.x86_64</font></div>
<div><font face="monospace"><br>
</font></div>
<div><br>
</div>
<div>Runscript:</div>
<div><br>
</div>
<div><font face="monospace">#!/bin/bash<br>
<br>
set -x<br>
set -e<br>
<br>
cd @SIMULATION_DIR@<br>
<br>
# Environment setup<br>
source /opt/rh/devtoolset-8/enable<br>
<br>
module purge<br>
module load lamod/cmake/3.17<br>
module load lamod/fftw/gnu/3.3.8<br>
module load lamod/openmpi/gnu/4.1.0<br>
module load libraries/gsl/2.6_gnu<br>
module load libraries/hdf5/1.10.5_gnu<br>
</font></div>
</div>
</blockquote>
<p><font face="monospace">It's hard for me to tell whether this is a
good environment.</font></p>
<p><font face="monospace">Are you using simfactory to build and run?
It's important to make sure you have the same env for both of
those tasks.</font></p>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><font face="monospace"><br>
echo "Environment diagnostics:"<br>
date<br>
hostname<br>
env<br>
ldd @EXECUTABLE@ | grep -E
"lapack|blas|openssl|stdc++|gfortran"<br>
<br>
# Set runtime parameters<br>
export CACTUS_NUM_PROCS=64<br>
export CACTUS_NUM_THREADS=1<br>
export OMP_NUM_THREADS=1<br>
export GMON_OUT_PREFIX=gmon.out<br>
env | sort > SIMFACTORY/ENVIRONMENT<br>
<br>
echo "Starting simulation at $(date)"<br>
export CACTUS_STARTTIME=$(date +%s)<br>
<br>
</font></div>
</div>
</blockquote>
<font face="monospace">This looks right.</font>
<blockquote type="cite"
cite="mid:CAM3SPH2C6KftAWwUsajJxzd310LLdpMjk5H0c2WpTSy5RKA24g@mail.gmail.com">
<div dir="ltr">
<div><font face="monospace">mpirun -np $CACTUS_NUM_PROCS -x
LD_LIBRARY_PATH @EXECUTABLE@ -L 3 @PARFILE@<br>
<br>
echo "Simulation finished at $(date)"<br>
touch segment.done</font></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">El jue, 18 sept 2025 a las
14:04, Steven Brandt (<<a
href="mailto:sbrandt@cct.lsu.edu" moz-do-not-send="true"
class="moz-txt-link-freetext">sbrandt@cct.lsu.edu</a>>)
escribió:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 9/17/2025 12:11 PM, Omar Elías Velasco Castillo
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Dear Einstein Toolkit team,<br>
<br>
I hope this message finds you well. I am a
beginner with the Einstein Toolkit. On
personal workstations I have been able to
compile and run tutorial simulations at low
resolution, but I am facing problems on two
different clusters. I would like to ask two
questions:<br>
<br>
1. <b>Are there ET versions prior to 2022_05
(e.g. 2019–2020 releases) that can still be
downloaded and compiled successfully?</b>
When I try to fetch them from the website
using ./GetComponents, the process fails
(CactusSourceJar.git is not created and some
components do not download). Since some of the
nodes I use have older GCC versions (8 or 10)
and limited modules, a stable older release
might be more practical.<br>
<br>
2. During compilation, I notice that thorns
(such as GSL and HDF5, for example) fall back
to using the bundled versions because system
modules are not found. The build completes
successfully, but jobs fail immediately after
submission to PBS or SLURM queues.<br>
</div>
</div>
</div>
</div>
</blockquote>
Can you show us what the error message(s) are?
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
<b>What is the role of the bundled versions in
this case</b>?<b> If the build uses bundled
GSL/HDF5, do I still need to load
corresponding, compatible modules in the
runscript?</b><br>
<br>
Could this mismatch explain why jobs die right
after submission?<br>
</div>
</div>
</div>
</div>
</blockquote>
<p>Maybe. I'm not 100% sure what you are doing. Can you
be clearer about how you are running the ET?</p>
<p>--Steve</p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
Any advice on handling these issues would be
very helpful. Thank you very much for your
time and support.</div>
<div><br>
</div>
<div>Greetings,</div>
<div><br>
</div>
<div>O.V.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Users mailing list
<a href="mailto:Users@einsteintoolkit.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">Users@einsteintoolkit.org</a>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://lists.einsteintoolkit.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</body>
</html>