<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 18 Sep 2019, at 04:14, 黎旭翔 <<a href="mailto:jukcoeng@pku.edu.cn" class="">jukcoeng@pku.edu.cn</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><span class="" style="">Dear friends,</span>
<div class="" style=""><br class="">
</div>
<div class="" style="">We are trying to use those examples listed on the website to test if it works on our HPC.</div>
<div class="" style=""><br class="">
</div>
<div class="" style=""><a href="https://einsteintoolkit.org/gallery.html" class="">https://einsteintoolkit.org/gallery.html</a></div>
<div class="" style=""><br class="">
</div>
<div class="" style=""> It goes well with a simple tov equation, which takes 2-5 min. But when we use it to simulate GW150914 BH merger or solve tov equation with high precision and long time, it seems that it will stop at an iteration point without any further
output, even an error. What really confuses us is that for different tests it stops at different points. Could you please help us find out what goes wrong with the simulation? I will attach the log.txt and parameter files to this mail. Thanks for your time!</div>
<div class="" style=""><br class="">
</div>
<div class="" style="">Here we used a partition with 144 nodes. Sometimes with a specific —procs and —num-threads number in the shell file, the simulation finished successfully. In other time it came across the problem above. In the two neutron star output
files, the job stoped at two different iteration points, and was cancelled due to time out or by hand.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Hi,</div>
<div><br class="">
</div>
<div>Can you tell us the simfactory command line that you used to submit the simulation?</div>
<div><br class="">
</div>
<div>From the log file, it looks like it might be wrong, or the machine might not be set up correctly in simfactory.</div>
<div><br class="">
</div>
<div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numprocs = 8</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::nodeprocs = 8</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numthreads = 18</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::hostname =
<a href="http://b01.hpc.pku.edu.cn" class="">b01.hpc.pku.edu.cn</a></div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::ppn = 144</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::ppnused = 144</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::procsrequested = 144</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::pbsSimulationName= GW150914-0000</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::cpufreq = </div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::user = 1801110076</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::memory = 0</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::nodes = 1</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::procs = 144</div>
<div>[LOG:2019-06-01 16:07:23] restart.userRun(simulationName)::numsmt = 1</div>
<div class=""><br class="">
</div>
</div>
<div>In particular, ppn = 144 looks wrong.</div>
<div><br class="">
</div>
<div>Erik, can you confirm?</div>
<div><br class="">
</div>
<div>If it's trying to run on too few nodes, it will run out of memory, as Steve suggested.</div>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
-- <br class="">
Ian<b class=""><span class="Apple-converted-space"> </span></b>Hinder<br class="">
Research Software Engineer<br class="">
University of Manchester, UK</div>
</div>
</div>
<br class="">
</body>
</html>