Hello Konrad,

To my understanding, it seems like the .cfg file has something wrong ( http://lists.einsteintoolkit.org/pipermail/users/2018-February/006103.html - i.e. the num-smt of your cluster should be 2, that should mean hyper threading - NOTE: I could be wrong with that). Moreover, you should use --procs=24 when submitting (from my experience this flag should be interpreted as the number of mpi task, that should correspond to the physical cores).

I am only providing my experience, but for sure, some more experienced users/developers could drive you to the most correct answer.

Regards, Federico Cipolletta.

Il sab 3 lug 2021, 19:00 users-request@einsteintoolkit.org ha scritto:

Send Users mailing list submissions to users@einsteintoolkit.org

To subscribe or unsubscribe via the World Wide Web, visit http://lists.einsteintoolkit.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-request@einsteintoolkit.org

You can reach the person managing the list at users-owner@einsteintoolkit.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..."

Today's Topics:

1. Submitting a simulation via simfactory on a cluster (Konrad Topolski)

---

Message: 1 Date: Sat, 3 Jul 2021 17:59:24 +0200 From: Konrad Topolski k.topolski2@student.uw.edu.pl Subject: [Users] Submitting a simulation via simfactory on a cluster To: users@einsteintoolkit.org Message-ID: < CAEx5SMHM6GszcaqE3_L0Rvx5+U29D6STMYH2d0PW777acyqrRA@mail.gmail.com> Content-Type: text/plain; charset="utf-8"

Hi

I am trying to get things right setting up simfactory on my cluster. I have created the okeanos.ini and okeanos.cfg files, as well as built the Cactus executable that's called cactus_okeanos. First I want to debug the machine setup, then I hope to take care of the submission script.

To submit a job, I use: simfactory/bin/sim create-submit GW150914_4 --parfile par/GW150914/GW150914.rpar --walltime 24:00:00 --configuration=okeanos --machine=okeanos

The error I am getting is the following:

Warning: Too many threads per process specified: specified num-threads=48 (ppn-used is 24) Warning: Number of used cores per node, number of SMT threads, and number of threads per process are inconsistent: ppn-used=24, num-smt=1, num-threads=48_threads (ppn-used*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of threads per process are inconsistent: procs=1, num-threads=48 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=1, ppn-used=24 (procs must be an integer multiple of ppn-used)

The relevant part of my machine .ini file looks like this: cpu = Intel Xeon E5-2690 v3 cpufreq = 2.6 ppn = 24 # or more max-num-threads = 48 # or more num-threads = 48 # or more nodes = 1084 memory = 131072 #any node has 128GB ram min-ppn = 24 maxwalltime = 48:00:00

I know that each node in my cluster consists of 12 sockets for cpus, each cpu has two cores and each core supports multithreading (=2), so it's 12x2x2 = 48. What am I getting wrong?

Best regards Konrad Topolski