<html>#2008: NaNs when running static tov on >40 cores
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td>Gwyneth Allwright</td></tr>
<tr><td style='text-align:right'> Status:</td><td>resolved</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td></td></tr>
<tr><td style='text-align:right'> Version:</td><td>ET_2016_05</td></tr>
<tr><td style='text-align:right'> Type:</td><td>bug</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>minor</td></tr>
<tr><td style='text-align:right'>Component:</td><td>Carpet</td></tr>
</table>
<p>Changes (by Roland Haas):</p>
<p><table>
<tr><td>status:</td><td>resolved (was open)</td></tr>
</table></p>
<p>I've been trying to run the static tov example parameter file on an HPC cluster using >40 cores, but this results in NaNs in the data. I can't remember whether the issue first appears at 40 or 41 cores (and won't be able to check this for the next few days), but using 41+ cores definitely gives me NaNs. I remember testing with 39 cores and several other lower values (down to 4), but these runs all seemed fine. </p>
<p>So far, I've been able to run larger simulations (e.g. BBHs) on more than 40 cores (same cluster) without any apparent issues.</p>
<p>I'll attach the static tov parameter file I used (I think I changed one or two outdated parameters), as well as the PBS script and error/output files from a static tov run on 44 cores.</p>
<p>The static tov runs were intended for speed test purposes. The results of the speed tests I've run so far (on fewer than 40 cores) seem quite strange to me, so I'm going to attach a text file with walltimes and CPU times for these runs, too. Any comments would be appreciated!</p>
<p><strong>Keyword:</strong></p>
<p>Comment (by Roland Haas):</p>
<p>I have verified (on LSU’s melete05 machine which has 80 cores) that indeed git hash 49d31796 "ML_BSSN: SYNC after computing initial Gamma, dtalpha and dtbeta vars" of <a data-is-external-link="true" href="https://bitbucket.org/einsteintoolkit/mclachlan" rel="nofollow">mclachlan</a> fixes this issue.</p>
<p>This was fixed as part of #2182 on 2018-08-22.</p>
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2008/nans-when-running-static-tov-on-40-cores'>https://bitbucket.org/einsteintoolkit/tickets/issues/2008/nans-when-running-static-tov-on-40-cores</a></p>
</html>