<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On 4 Feb 2017, at 20:13, Gwyneth Allwright <<a href="mailto:allgwy001@myuct.ac.za">allgwy001@myuct.ac.za</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Hi All,<br><br>I'm trying to get the Einstein Toolkit installed on an HPC cluster running SLES. The trouble is that Cactus tries to use all the available processors even when I specify a smaller number (by setting ppn in my PBS script).<br><br>As a test, I tried running the compilation with a parameter file that required about 5 GB of RAM. In my PBS script, I set nodes=1 and ppn=3, and then ran using openmpi-1.10.1:<br><br>mpirun -hostfile $PBS_NODEFILE <ET exe> <parameter file><br><br>This resulted in the simulation running on all 24 available processors, even though I'd only requested 3. Since PBS and MPI are integrated, I was told that using -np with mpirun wouldn't help. <br><br>Does anyone know how to address this issue?<br></div></blockquote><div><br></div><div>Hi Gwyneth,</div><div><br></div><div>A few questions:</div><div><br></div><div>Can you check which nodes are appearing in $PBS_NODEFILE?</div><div><br></div><div>When you say it's running on all 24 available processors, what do you mean? Do you mean that "top" shows 24 processes, or that the CPU is at 100%? Could it be that you do in fact have three processes, but due to OpenMP, each process actually has 12 threads?</div><div><br></div><div>Can you do</div><div><br></div><div>ps -u $USER f</div><div><br></div><div>on the node while Cactus is running? This will show you a tree view of the processes that are running.</div><div><br></div><div>Is the parameter file using PUGH or Carpet? The standard output from the first process should show you the number of processes that Cactus thinks it is running on.</div><div><br></div><div>When you compiled the ET, did you specify the location of the MPI library with MPI_DIR? If Cactus failed to find the library automatically, it will have automatically built its own version of OpenMPI and linked with that version. If you then use an mpirun from a different installation of MPI, things will not work correctly.</div><div><br></div><div>Maybe you can increase the verbosity of mpirun, to check how many processes it is trying to start.</div><div><br></div></div><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>-- </div><div>Ian Hinder</div><div><a href="http://members.aei.mpg.de/ianhin">http://members.aei.mpg.de/ianhin</a></div></div></div></div></div>
</div>
<br></body></html>