<div dir="ltr"><div><br></div><div>Thank you Erik - it seems that when I keep the num-threads variable in machine.ini file, it overrides any argument given to simfactory executable.</div><div>Removing it appears to fix the problem.<br></div><div><br></div><div>If I don't provide --configuration=okeanos and --machine=okeanos though, it defaults to any local (login) node I used to ./simfactory/bin/sim setup simfactory. </div><div>So I suppose in each invocation I need to specify it? </div><div><br></div><div>After commenting out num-threads in okeanos.ini, here's what works and what doesn't:</div><div><br></div><div>./simfactory/bin/sim create-submittopoltopolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4<br>Warning: Unknown machine name nid00069<br>Error: Unknown local machine nid00069. Please use 'sim setup' to create a local machine entry from the generic template.<br>Aborting Simfactory.</div><div><br></div><div>topolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4 --machine=okeanos<br>Warning: Current Working directory does not match Cactus sourcetree, changing to /home/topolski/Cactus<br>Warning: simulation "GW150914_MPI" does not exist or is not readable<br>Parameter file: /lustre/tetyda/home/topolski/Cactus/GW150914_MPI.rpar<br>Error: Executable /home/topolski/Cactus/exe/cactus_sim for configuration sim does not exist or is not readable<br>Aborting Simfactory.</div><div><br></div><div>topolski@okeanos-login1:~/Cactus> ./simfactory/bin/sim create-submit GW150914_MPI --parfile GW150914_MPI.rpar --define N 28 --walltime=1:00:00 --procs 192 --num-threads=4 --machine=okeanos --configuration=okeanos<br>Warning: Current Working directory does not match Cactus sourcetree, changing to /home/topolski/Cactus<br>Parameter file: /lustre/tetyda/home/topolski/Cactus/GW150914_MPI.rpar<br>Skeleton Created<br>Job directory: "/home/topolski/simulations/GW150914_MPI"<br>Executable: "/home/topolski/Cactus/exe/cactus_okeanos"<br>Option list: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/cfg/OptionList"<br>Submit script: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/run/SubmitScript"<br>Run script: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/run/RunScript"<br>Parameter file: "/home/topolski/simulations/GW150914_MPI/SIMFACTORY/par/GW150914_MPI.rpar"<br>Assigned restart id: 0 <br>Executing submit command: sbatch /home/topolski/simulations/GW150914_MPI/output-0000/SIMFACTORY/SubmitScript<br>Submit finished, job id is 739787<br></div><div><br></div><div>And the last attempt yields the desired result. </div><div><br></div><div>So I suppose this is the correct way? <br></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">śr., 14 lip 2021 o 01:25 Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu">schnetter@cct.lsu.edu</a>> napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Konrad<br>
<br>
Changing the number of MPI processes and OpenMP threads with<br>
Simfactory when restarting works the same way as setting them in the<br>
first place. For example, your first run might be submitted with<br>
<br>
./simfactory/bin/sim submit poisson --parfile=poisson.par --procs=120<br>
--num-threads=4<br>
<br>
You can restart this simulations with<br>
<br>
./simfactory/bin/sim submit poisson<br>
<br>
which will re-use the original settings. You can also restart with<br>
<br>
./simfactory/bin/sim submit poisson --procs=160 --num-threads=8<br>
<br>
to change these settings.<br>
<br>
If this does not work, then your machine might be configured wrong.<br>
For example, you say that you specify the number of MPI processes by<br>
setting "--num-threads", which sounds suspicious.<br>
<br>
The default-generated machine configuration only works for<br>
workstations or laptops. If you run this script on an HPC system, it<br>
will generate a nonsense configuration, and might even hide a "real"<br>
configuration if one is present.<br>
<br>
-erik<br>
<br>
<br>
On Tue, Jul 13, 2021 at 5:23 PM Konrad Topolski<br>
<<a href="mailto:k.topolski2@student.uw.edu.pl" target="_blank">k.topolski2@student.uw.edu.pl</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I am currently trying to find what the optimal number of MPI processes is for my purposes.<br>
> I have managed to change the number of MPI processes when restarting a simulation from a checkpoint - but using the bare executable, not simfactory.<br>
><br>
> Now, I would like to learn how to do it in simfactory.<br>
><br>
> I have learned that to successfully steer the number of threads per 1 MPI process (which, combined with a total number of threads requested, yields the total number of MPI processes), I change the num-thread variable in the machine.ini file.<br>
> This is probably (certainly?) suboptimal, so if there's a proper way, I'd like to learn it.<br>
><br>
> I submit/recover simulations via .<br>
> /simfactory/bin/sim submit <sim_name> --parfile <parfile_name> --recover --procs NUM_PROCS --machine=okeanos --configuration=okeanos<br>
><br>
> If I don't use the --machine option specifying my cluster, it will default to some config with max nodes = 1 (generic?). Which is why I steer MPI processes via num-thread.<br>
><br>
> Trying to recover a simulation via simfactory with a new machine file (with num-thread changed) yields an error message:<br>
><br>
> INFO (Carpet): MPI is enabled<br>
> INFO (Carpet): Carpet is running on 4 processes<br>
> WARNING level 0 from host nid00392 process 0<br>
> in thorn Carpet, file /lustre/tetyda/home/topolski/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:148:<br>
> -> The environment variable CACTUS_NUM_PROCS is set to 96, but there are 4 MPI processes. This may indicate a severe problem with the MPI startup mechanism.<br>
><br>
> What can I do to recover a simulation via simfactory and use a different number of MPI processes?<br>
><br>
> While I'm at it, can I also change parameters such as the number of refinement levels or make new guesses for AHFinderDirect, in case the previously-used parameters did not provide high enough resolution for a successful find?<br>
><br>
> Best regards<br>
> Konrad Topolski<br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
> <a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
<br>
<br>
<br>
-- <br>
Erik Schnetter <<a href="mailto:schnetter@cct.lsu.edu" target="_blank">schnetter@cct.lsu.edu</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
</blockquote></div>