<html>#2289: CT_MultiLevel tests abort when run using more than one thread via simfactory
<table style='border-spacing: 1ex 0pt; '>
<tr><td style='text-align:right'> Reporter:</td><td>Roland Haas</td></tr>
<tr><td style='text-align:right'> Status:</td><td>new</td></tr>
<tr><td style='text-align:right'>Milestone:</td><td>ET_2019_10</td></tr>
<tr><td style='text-align:right'> Version:</td><td></td></tr>
<tr><td style='text-align:right'> Type:</td><td>bug</td></tr>
<tr><td style='text-align:right'> Priority:</td><td>minor</td></tr>
<tr><td style='text-align:right'>Component:</td><td>EinsteinToolkit thorn</td></tr>
</table>
<p>Comment (by Roland Haas):</p>
<p><span class="ap-mention" data-atlassian-id="557058:99374cb3-e897-42a6-b5c8-a191a22a46af">@Eloisa Bentivegna</span> LoopControl’s <code>LC_LOOP</code> macros themselves do not enable any parallel threading. They need to be surround by a <code>#pragma omp parallel</code> section (see eg their use in Llama). Without they are just single threaded. Having had a look at CT_MultiLevel’s code it seems that there is no multi-threading going on. And indeed if I run the poisson test with 4 threads I only see 1 core (per MPI rank) in use even when removing the <code>num_threads</code> setting.</p>
<p>That is to say: CT_MultiLevel always (with the exception of a reduction and the loops in CT_Analytic and CT_Dust), even before introducing the <code>num_threads</code>setting, used only a single thread.</p>
<p>There should have never been a chance for a race condition (since there was only one runner) and any claim that I have made about there being one in the Gauss-Seidel iteration was false, sorry.</p>
<p>I certainly leaves me worried why your data produced with gcc (4.8.2 admittedly) and -O2 would differ what is produced on the tutorial server (also gcc) or my workstation. The affected test seems to be the boostedpuncture test (which now of course produces bit-identical results for me whether run with <code>OMP_NUM_THREADS=1</code> or <code>OMP_NUM_THREADS=64</code> on my workstation (gcc9, -O2, 24 cores so I am oversubscribing it).</p>
<p>My understanding of what <code>Carpet::num_threads</code> can be used for would be if one somehow cannot pass the <code>OMP_NUM_THREADS</code> variable to the executable (though I do not know of a MPI implementation that would not give me some way of passing ENV variables) in which case one would set <code>CACTUS_NUM_THREADS</code> and the <code>num_threads</code> parameter to the same value, which will not cause Carpet to abort the run.</p>
<p>For the upcoming release I would try reverting <a data-is-external-link="true" href="https://bitbucket.org/eloisa/ctthorns/commits/f81354537b5adb
<p>--<br/>
Ticket URL: <a href='https://bitbucket.org/einsteintoolkit/tickets/issues/2289/ct_multilevel-tests-abort-when-run-using'>https://bitbucket.org/einsteintoolkit/tickets/issues/2289/ct_multilevel-tests-abort-when-run-using</a></p>
</html>