<html>#2289: CT_MultiLevel tests abort when run using more than one thread via simfactory
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td>Roland Haas</td></tr>
<tr><td style='text-align:right'> Status:</td><td>new</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td>ET_2019_10</td></tr>
<tr><td style='text-align:right'> Version:</td><td></td></tr>
<tr><td style='text-align:right'> Type:</td><td>bug</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>minor</td></tr>
<tr><td style='text-align:right'>Component:</td><td>EinsteinToolkit thorn</td></tr>
</table>
<p>Comment (by Roland Haas):</p>
<p>True. I reverted the commit and re-generated all data on my workstation using gcc9, -O2 and using 2 MPI ranks and 1 thread per rank. </p>
<p>This dataset then fails to pass on some of the clusters (eg stampede2, comet) but passes on others (and obviously on my workstation and presumably on the Jenkins system since it is a Ubuntu VM).</p>
<p>Running with very high thread count I do see differences (on my workstation) for the reduced quantities, which is can kind of understand since an <code>omp reduce</code> will produce different answers depending on the number of threads used (and there is a such a one in CT_MultiLevel).</p>
<p>I have checked and on Stampede2 the number of iterations output to stdout is the same as on my workstation so it is not “simply” an issue of roundoff level differences in the error norms triggering one more iteration.</p>
<p>One will have to see if the failing clusters are all using aggressive Intel compiler optimizations or if it is eg something that has to do with vectorization.</p>
<p>Unfortunately even with the recent changes the <code>CT_MultiLevel</code> tests still have a tendency to hang (just never finish) on the OSX Macports testing system, which has me very confused.</p>
<p>Have you run the testsuites on a Mac (using MacPorts ideally) recently?</p>
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2289/ct_multilevel-tests-abort-when-run-using'>https://bitbucket.org/einsteintoolkit/tickets/issues/2289/ct_multilevel-tests-abort-when-run-using</a></p>
</html>