<div dir="ltr">Hi Sam,<div><br></div><div>I can confirm that if I set OMP_NUM_TRHEADS=1 the test passes.</div><div><br></div><div>Best,</div><div>Gabriele</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 2:35 PM Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu">scupp1@my.apsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-6741243620787451910">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Gabriele,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I've been experimenting with this, and I've reproduced this issue on a different machine. In my case, it runs with 1 OMP thread but has the segfault with 2. Can you run that test parfile (repos/GRHayLET/GRHayLHDX/test/Balsara0.par) without OMP? If this is
the source, it will help narrow down the problem. I can run the test with multiple threads on my local machine though, so I'm not sure why OMP only causes a segfault sometimes.<br>
<div id="m_-6741243620787451910Signature">
<div>
<div id="m_-6741243620787451910divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> Samuel Cupp
<div id="m_-6741243620787451910divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Postdoctoral Researcher<br>
</span></div>
<div id="m_-6741243620787451910divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Department of Physics</span></div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif"><span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont">
<span>University of Idaho</span><span id="m_-6741243620787451910ms-rterangepaste-end"></span></span></div>
</span>
<div id="m_-6741243620787451910divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
</div>
</div>
</div>
</div>
</div>
<div id="m_-6741243620787451910appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6741243620787451910divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Gabriele Bozzola <<a href="mailto:bozzola.gabriele@gmail.com" target="_blank">bozzola.gabriele@gmail.com</a>><br>
<b>Sent:</b> Sunday, October 29, 2023 6:16 PM<br>
<b>To:</b> Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>><br>
<b>Cc:</b> Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>>; <a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a> <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] meeting minutes for 2023-10-19</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>I verified that the testsuite passes if I remove the GRHayL thorns.</div>
<div>Other than that, ET seems to work fine.</div>
<div><br>
</div>
<div>Attached is the cfg file I used to compile ET on Anvil.</div>
<div><br>
</div>
<div>Best,</div>
<div>Gabriele</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Oct 26, 2023 at 12:45 PM Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Gabriele,<br>
</div>
<div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We discussed this in the call this morning, and there's a few things we can try. First, it would be helpful if you created a ticket so we can track progress on the issue. This is especially true since the testsuite shouldn't hang if a test fails. The expected
behavior would be for it to continue with testing, but for some reason it didn't. Also, it might help to remove GRHayLHDX from the thornlist, recompile, and see if that is the only test failing. Knowing if this is specific to the thorn or a broader issue would
help diagnose the source of the problem.<br>
</div>
<div id="m_-6741243620787451910x_m_8295053389503033264Signature">
<div>
<div id="m_-6741243620787451910x_m_8295053389503033264divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> Samuel Cupp
<div id="m_-6741243620787451910x_m_8295053389503033264divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Postdoctoral Researcher<br>
</span></div>
<div id="m_-6741243620787451910x_m_8295053389503033264divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Department of Physics</span></div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif"><span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont">
<span>University of Idaho</span><span id="m_-6741243620787451910x_m_8295053389503033264ms-rterangepaste-end"></span></span></div>
</span>
<div id="m_-6741243620787451910x_m_8295053389503033264divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
</div>
</div>
</div>
</div>
</div>
<div id="m_-6741243620787451910x_m_8295053389503033264appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6741243620787451910x_m_8295053389503033264divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Gabriele Bozzola <<a href="mailto:bozzola.gabriele@gmail.com" target="_blank">bozzola.gabriele@gmail.com</a>><br>
<b>Sent:</b> Tuesday, October 24, 2023 1:27 PM<br>
<b>To:</b> Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>><br>
<b>Cc:</b> Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>>;
<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a> <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] meeting minutes for 2023-10-19</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>This is just CPU. The .out for the entire testsuite was attached to my first email.</div>
<div><br>
</div>
<div>Best,</div>
<div>Gabriele</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Oct 24, 2023 at 12:14 PM Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus?<br>
</div>
<div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313Signature">
<div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> Samuel Cupp
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Postdoctoral Researcher<br>
</span></div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Department of Physics</span></div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif"><span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont">
<span>University of Idaho</span><span id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313ms-rterangepaste-end"></span></span></div>
</span>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
</div>
</div>
</div>
</div>
</div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Gabriele Bozzola <<a href="mailto:bozzola.gabriele@gmail.com" target="_blank">bozzola.gabriele@gmail.com</a>><br>
<b>Sent:</b> Tuesday, October 24, 2023 10:07 AM<br>
<b>To:</b> Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>><br>
<b>Cc:</b> Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>>;
<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a> <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] meeting minutes for 2023-10-19</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi Sam,
<div><br>
</div>
<div>I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.</div>
<div><br>
</div>
<div>I am attaching the .out for the test.</div>
<div><br>
</div>
<div>Best,</div>
<div>Gabriele</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. <<a href="mailto:scupp1@my.apsu.edu" target="_blank">scupp1@my.apsu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Gabriele,<br>
</div>
<div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD
tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?<br>
</div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232Signature">
<div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<br>
</div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> Samuel Cupp
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Postdoctoral Researcher<br>
</span></div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif">
<span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont"> Department of Physics</span></div>
<div dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif"><span style="font-family:Calibri,Helvetica,sans-serif,serif,EmojiFont">
<span>University of Idaho</span><span id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232ms-rterangepaste-end"></span></span></div>
</span>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
</div>
</div>
</div>
</div>
</div>
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-6741243620787451910x_m_8295053389503033264x_m_3247450434824616313x_m_5681150511870418232divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Users <<a href="mailto:users-bounces@einsteintoolkit.org" target="_blank">users-bounces@einsteintoolkit.org</a>> on behalf of Gabriele Bozzola <<a href="mailto:bozzola.gabriele@gmail.com" target="_blank">bozzola.gabriele@gmail.com</a>><br>
<b>Sent:</b> Thursday, October 19, 2023 5:46 PM<br>
<b>To:</b> Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>><br>
<b>Cc:</b> <a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a> <<a href="mailto:users@einsteintoolkit.org" target="_blank">users@einsteintoolkit.org</a>><br>
<b>Subject:</b> Re: [Users] meeting minutes for 2023-10-19</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>Is there a tested configuration for anvil?</div>
<div><br>
</div>
<div>I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the </div>
<div>MPI runs, and the test suite does not complete even with a walltime of 6 hours.</div>
<div><br>
</div>
<div>I am attaching the output.</div>
<div><br>
</div>
<div>Best,</div>
<div>Gabriele</div>
</div>
<br>
<div>
<div dir="ltr">On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt <<a href="mailto:sbrandt@cct.lsu.edu" target="_blank">sbrandt@cct.lsu.edu</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Present: Steve, Peter, Sam, Zach, Leo<br>
<br>
Release<br>
Gallery Examples<br>
- TOV is Peter<br>
- BBH is Steve<br>
- Roland might have students for the other three?<br>
Tests<br>
- Need to be run<br>
- Clusters:<br>
Stampede2 isn't listed, and Delta and Anvil are added.<br>
- Release name: Lise Meitner <a href="https://en.wikipedia.org/wiki/Lise_Meitner" rel="noreferrer" target="_blank">
https://en.wikipedia.org/wiki/Lise_Meitner</a><br>
- Autogenerated codes need to be regenerated, Leo will do it<br>
<br>
Mailing list moderation<br>
- Remind Roland to send Peter the password<br>
<br>
Unanswered question on the mailing list<br>
- Need answer for Enzo. Roland was going to answer?<br>
- Ticket 2749: Leo says he fixed it.<br>
- Ticket 2647: One of Zach's students is looking at that in Grail (sp)<br>
- Ticket 2609: Steve thinks he approved PR for Roland<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@einsteintoolkit.org" target="_blank">Users@einsteintoolkit.org</a><br>
<a href="http://lists.einsteintoolkit.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/users</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div></blockquote></div>