[Users] Running with SLURM

[Warren, Jessica Sawyer]

Hello,

I am trying to get ET setup on the Quartz cluster at Indiana University (https://kb.iu.edu/d/qrtz).  Simfactory compiles, however, I've run into a problem when a simulation begins and keep getting a segmentation fault (one per MPI process).  The cluster uses SLURM, which is new to me, and RHEL 8.  I've attached the relevant portion of an example output file (with stderr and stdout merged), the corresponding batch file that produced it when submitted, and below are the key settings from the machine file and the one line from the generic mpi runscript that I have edited.  Any guidance would be much appreciated.

INI settings:
max-num-smt     = 1   # no hyperthreading on system
num-smt         = 1
ppn             = 128 # physical cores
mpn             = 2   # NUMA domains ("sockets")
max-num-threads = 128 # do not oversubscribe PUs
num-threads     = 4   # using 4 threads per process; make 2 may be better
nodes           = 1  # use more for actual runs

Edited runscript line:
srun --nodes=@NODES@ --ntasks-per-node=@NODE_PROCS@ --cpus-per-task=@NUM_THREADS@ @EXECUTABLE@ -L 3 @PARFILE@

Thank you,
Jessica


Dr. Jessica S. Warren
Physics Lecturer
Indiana University Northwest
warrenjs at iun.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20220808/4ca15723/attachment-0001.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: batch.txt
Url: http://lists.einsteintoolkit.org/pipermail/users/attachments/20220808/4ca15723/attachment-0002.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: output.txt
Url: http://lists.einsteintoolkit.org/pipermail/users/attachments/20220808/4ca15723/attachment-0003.txt