Thank you Erik - it seems that when I keep the num-threads variable in machine.ini file, it overrides any argument given to simfactory executable. Removing it appears to fix the problem.
If I don't provide --configuration=okeanos and --machine=okeanos though, it defaults to any local (login) node I used to ./simfactory/bin/sim setup simfactory. So I suppose in each invocation I need to specify it?
After commenting out num-threads in okeanos.ini, here's what works and what doesn't:
./simfactory/bin/sim create-submittopoltopolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4 Warning: Unknown machine name nid00069 Error: Unknown local machine nid00069. Please use 'sim setup' to create a local machine entry from the generic template. Aborting Simfactory.
topolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4 --machine=okeanos Warning: Current Working directory does not match Cactus sourcetree, changing to /home/topolski/Cactus Warning: simulation "GW150914_MPI" does not exist or is not readable Parameter file: /lustre/tetyda/home/topolski/Cactus/GW150914_MPI.rpar Error: Executable /home/topolski/Cactus/exe/cactus_sim for configuration sim does not exist or is not readable Aborting Simfactory.
topolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim create-submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4 --machine=okeanos --configuration=okeanos Warning: Current Working directory does not match Cactus sourcetree, changing to /home/topolski/Cactus Parameter file: /lustre/tetyda/home/topolski/Cactus/GW150914_MPI.rpar Skeleton Created Job directory: "/home/topolski/simulations/GW150914_MPI" Executable: "/home/topolski/Cactus/exe/cactus_okeanos" Option list: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/cfg/OptionList" Submit script: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/run/SubmitScript" Run script: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/run/RunScript" Parameter file: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/par/GW150914_MPI.rpar" Assigned restart id: 0 Executing submit command: sbatch /home/topolski/simulations/GW150914_MPI/output-0000/SIMFACTORY/SubmitScript Submit finished, job id is 739787
And the last attempt yields the desired result.
So I suppose this is the correct way?
śr., 14 lip 2021 o 01:25 Erik Schnetter schnetter@cct.lsu.edu napisał(a):
Konrad
Changing the number of MPI processes and OpenMP threads with Simfactory when restarting works the same way as setting them in the first place. For example, your first run might be submitted with
./simfactory/bin/sim submit poisson --parfile=poisson.par --procs=120 --num-threads=4
You can restart this simulations with
./simfactory/bin/sim submit poisson
which will re-use the original settings. You can also restart with
./simfactory/bin/sim submit poisson --procs=160 --num-threads=8
to change these settings.
If this does not work, then your machine might be configured wrong. For example, you say that you specify the number of MPI processes by setting "--num-threads", which sounds suspicious.
The default-generated machine configuration only works for workstations or laptops. If you run this script on an HPC system, it will generate a nonsense configuration, and might even hide a "real" configuration if one is present.
-erik
On Tue, Jul 13, 2021 at 5:23 PM Konrad Topolski k.topolski2@student.uw.edu.pl wrote:
Hi,
I am currently trying to find what the optimal number of MPI processes
is for my purposes.
I have managed to change the number of MPI processes when restarting a
simulation from a checkpoint - but using the bare executable, not simfactory.
Now, I would like to learn how to do it in simfactory.
I have learned that to successfully steer the number of threads per 1
MPI process (which, combined with a total number of threads requested, yields the total number of MPI processes), I change the num-thread variable in the machine.ini file.
This is probably (certainly?) suboptimal, so if there's a proper way,
I'd like to learn it.
I submit/recover simulations via . /simfactory/bin/sim submit <sim_name> --parfile <parfile_name>
--recover --procs NUM_PROCS --machine=okeanos --configuration=okeanos
If I don't use the --machine option specifying my cluster, it will
default to some config with max nodes = 1 (generic?). Which is why I steer MPI processes via num-thread.
Trying to recover a simulation via simfactory with a new machine file
(with num-thread changed) yields an error message:
INFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 4 processes WARNING level 0 from host nid00392 process 0 in thorn Carpet, file
/lustre/tetyda/home/topolski/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148:
-> The environment variable CACTUS_NUM_PROCS is set to 96, but there
are 4 MPI processes. This may indicate a severe problem with the MPI startup mechanism.
What can I do to recover a simulation via simfactory and use a different
number of MPI processes?
While I'm at it, can I also change parameters such as the number of
refinement levels or make new guesses for AHFinderDirect, in case the previously-used parameters did not provide high enough resolution for a successful find?
Best regards Konrad Topolski
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/