<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">Sorry, I just have a quick follow-up on this subject, to clarify.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> Am I misinterpreting when </span><span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I read</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"> "</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">num-threads is
larger than the value for cores or procs"?</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">The cluster I am using (ThornyFlat) gives me the formula: </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">procs* num-stm
= np*num-threads</span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Disregarding simultaneous multithreading (</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">num-stm
=</span><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> 1</span></span></span><span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">)
and considering </span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">np>1, </span><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">gives </span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">procs>num-threads</span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;background-color:rgb(255, 255, 255);display:inline !important">
<div style="margin:0px;font-size:12pt"><span style="margin:0px;font-family:Calibri, Helvetica, sans-serif">What I heard at the last ETK meeting was that </span><span style="margin:0px;font-size:14.6667px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif"><span style="margin:0px;font-size:12pt;font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255)"><i>procs
is nr. processes, and it will be divided by the num threads.</i></span></span></div>
<div style="margin:0px;font-size:12pt"><span style="margin:0px;font-family:Calibri, Helvetica, sans-serif">I thought that<span> </span></span><span style="margin:0px"><span style="margin:0px;font-size:14.6667px;font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;background-color:rgb(255, 255, 255);display:inline !important"><span style="margin:0px"><span style="margin:0px;background-color:rgb(255, 255, 255);display:inline !important"><span style="margin:0px"><span style="margin:0px;background-color:rgb(255, 255, 255);display:inline !important"><span style="margin:0px;font-size:12pt;font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255);display:inline !important">if
we divide procs by num-threads, we get </span><span style="margin:0px;font-size:16px;font-family:Calibri, Arial, Helvetica, sans-serif;background-color:rgb(255, 255, 255);display:inline !important"><span style="margin:0px;font-size:12pt;font-family:Calibri, Helvetica, sans-serif;background-color:rgb(255, 255, 255);display:inline !important">np.</span></span></span></span></span></span></span></span></div>
</span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">Indeed,
testing it, I get: </span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255); display: inline !important;">--procs=24
--num-threads=1 <=> np=24</span><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> </span></span></span></span></span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:"Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif;font-size:14.6667px;background-color:rgb(255, 255, 255);display:inline !important"><span><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">--procs</span><span style="background-color:rgb(255, 255, 255);display:inline !important"><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> </span><span><span style="background-color: rgb(255, 255, 255); display: inline !important; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">=24
--num-threads=2 <=> np=12</span><span style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"> </span></span></span></span></span></span></span></span></div>
<div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div>
<div dir="ltr">_______________________</div>
<div dir="ltr"><i><font size="2">Maria C. Babiuc Hamilton, Ph.D.</font></i></div>
<div dir="ltr"><i><font size="2">Professor, Department of Physics </font></i></div>
<div dir="ltr"><i style="font-size:12pt"><font size="2"><i style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><font size="2"></font></i>College of Science, </font></i><i style="font-size:12pt"><font size="2">Marshall University,</font></i></div>
<div dir="ltr"><i><font size="2">1 John Marshall Drive, </font></i><i style="font-size:12pt"><font size="2">Huntington, WV, 25755</font></i></div>
<div dir="ltr"><i><font size="2"><i style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><font size="2"><i style="font-size:12pt"><font size="2">Room S 257, <span id="ms-rterangepaste-start"></span><i style="font-size:16px"><font size="2">Phone:
(304)696-2754</font></i></font></i></font></i><br>
</font></i></div>
<div dir="ltr"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> users-bounces@einsteintoolkit.org <users-bounces@einsteintoolkit.org> on behalf of Roland Haas <rhaas@illinois.edu><br>
<b>Sent:</b> Monday, January 18, 2021 5:17 PM<br>
<b>To:</b> KARIMA SHAHZAD <02141911015@student.qau.edu.pk><br>
<b>Cc:</b> users@einsteintoolkit.org <users@einsteintoolkit.org><br>
<b>Subject:</b> Re: [Users] TAT/ Slab Error</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Hello Karima,<br>
<br>
> Here is the error attached I encountered while evolving black holes<br>
> binary. The work-around I guess is to increase the number of<br>
> processes (if I am not wrong), but I am not quite sure how to do<br>
> that. I need some help to get around this issue.<br>
Assuming you use simulation factory you need to pass options to make<br>
sure it creates multiple MPI processes. The auto-detection code tends<br>
to choose a number of threads equal to the the number of cores on your<br>
laptop and and a single MPI rank.<br>
<br>
Eg if you have a 4 core laptop then you have to use:<br>
<br>
./simfactory/bin/sim submit testrun01 --cores 4 --num-threads 2<br>
--parfile ...<br>
<br>
which uses a total of 4 cores and starts 2 threads per MPI rank so that<br>
you end up with 4 / 2 = 2 MPI ranks each of which uses 2 threads.<br>
<br>
Similarly on clusters, the basic idea is to set --num-threads so that<br>
there are more than 1 MPI rank started (ie --num-threads is larger than<br>
the value for --cores or --procs [which are synonyms for each other]).<br>
<br>
If not using simulation factory you have to manually use mpirun and<br>
OMP_NUM_THREADS. Eg:<br>
<br>
export OMP_NUM_THREADS=2<br>
<br>
mpirun -np 2 /home/karima/simulations/bbhHr/SIMFACTORY/exe/cactus_sim<br>
-L 3 /home/karima/simulations/bbhHr/output-0000/BBHHigherRes.par<br>
<br>
which starts 2 MPI ranks each will use 2 OpenMP threads.<br>
<br>
Yours,<br>
Roland<br>
<br>
-- <br>
My email is as private as my paper mail. I therefore support encrypting<br>
and signing email messages. Get my PGP key from <a href="https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpgp.mit.edu%2F&amp;data=04%7C01%7Cbabiuc%40marshall.edu%7C6164afef3f5144f4e6ad08d8bbfee273%7C239ab2783bba4c78b41d8508a541e025%7C0%7C0%7C637466050685932327%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=bIBVt8EYRo%2FekfxirOK1RXSZyr5K8aAReTD76%2FpaLHk%3D&amp;reserved=0">
https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpgp.mit.edu%2F&amp;data=04%7C01%7Cbabiuc%40marshall.edu%7C6164afef3f5144f4e6ad08d8bbfee273%7C239ab2783bba4c78b41d8508a541e025%7C0%7C0%7C637466050685932327%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=bIBVt8EYRo%2FekfxirOK1RXSZyr5K8aAReTD76%2FpaLHk%3D&amp;reserved=0</a>
.<br>
</div>
</span></font></div>
</div>
</body>
</html>