Hi Sam,

I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.

I am attaching the .out for the test.

Best, Gabriele

On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.

I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Users users-bounces@einsteintoolkit.org on behalf of Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Thursday, October 19, 2023 5:46 PM *To:* Steven R. Brandt sbrandt@cct.lsu.edu *Cc:* users@einsteintoolkit.org users@einsteintoolkit.org *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hello,

Is there a tested configuration for anvil?

I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours.

I am attaching the output.

Best, Gabriele

On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:

Present: Steve, Peter, Sam, Zach, Leo

Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation - Remind Roland to send Peter the password

Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Hi Sam,

This is just CPU. The .out for the entire testsuite was attached to my first email.

Best, Gabriele

On Tue, Oct 24, 2023 at 12:14 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Tuesday, October 24, 2023 10:07 AM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.

I am attaching the .out for the test.

Best, Gabriele

On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.

I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Users users-bounces@einsteintoolkit.org on behalf of Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Thursday, October 19, 2023 5:46 PM *To:* Steven R. Brandt sbrandt@cct.lsu.edu *Cc:* users@einsteintoolkit.org users@einsteintoolkit.org *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hello,

Is there a tested configuration for anvil?

I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours.

I am attaching the output.

Best, Gabriele

On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:

Present: Steve, Peter, Sam, Zach, Leo

Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation - Remind Roland to send Peter the password

Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Hi Sam,

I verified that the testsuite passes if I remove the GRHayL thorns. Other than that, ET seems to work fine.

Attached is the cfg file I used to compile ET on Anvil.

Best, Gabriele

On Thu, Oct 26, 2023 at 12:45 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, We discussed this in the call this morning, and there's a few things we can try. First, it would be helpful if you created a ticket so we can track progress on the issue. This is especially true since the testsuite shouldn't hang if a test fails. The expected behavior would be for it to continue with testing, but for some reason it didn't. Also, it might help to remove GRHayLHDX from the thornlist, recompile, and see if that is the only test failing. Knowing if this is specific to the thorn or a broader issue would help diagnose the source of the problem.

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Tuesday, October 24, 2023 1:27 PM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

This is just CPU. The .out for the entire testsuite was attached to my first email.

Best, Gabriele

On Tue, Oct 24, 2023 at 12:14 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Tuesday, October 24, 2023 10:07 AM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.

I am attaching the .out for the test.

Best, Gabriele

On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.

I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Users users-bounces@einsteintoolkit.org on behalf of Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Thursday, October 19, 2023 5:46 PM *To:* Steven R. Brandt sbrandt@cct.lsu.edu *Cc:* users@einsteintoolkit.org users@einsteintoolkit.org *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hello,

Is there a tested configuration for anvil?

I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours.

I am attaching the output.

Best, Gabriele

On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:

Present: Steve, Peter, Sam, Zach, Leo

Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation - Remind Roland to send Peter the password

Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Hi Sam,

I can confirm that if I set OMP_NUM_TRHEADS=1 the test passes.

Best, Gabriele

On Thu, Nov 2, 2023 at 2:35 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, I've been experimenting with this, and I've reproduced this issue on a different machine. In my case, it runs with 1 OMP thread but has the segfault with 2. Can you run that test parfile (repos/GRHayLET/GRHayLHDX/test/Balsara0.par) without OMP? If this is the source, it will help narrow down the problem. I can run the test with multiple threads on my local machine though, so I'm not sure why OMP only causes a segfault sometimes.

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Sunday, October 29, 2023 6:16 PM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I verified that the testsuite passes if I remove the GRHayL thorns. Other than that, ET seems to work fine.

Attached is the cfg file I used to compile ET on Anvil.

Best, Gabriele

On Thu, Oct 26, 2023 at 12:45 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, We discussed this in the call this morning, and there's a few things we can try. First, it would be helpful if you created a ticket so we can track progress on the issue. This is especially true since the testsuite shouldn't hang if a test fails. The expected behavior would be for it to continue with testing, but for some reason it didn't. Also, it might help to remove GRHayLHDX from the thornlist, recompile, and see if that is the only test failing. Knowing if this is specific to the thorn or a broader issue would help diagnose the source of the problem.

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Tuesday, October 24, 2023 1:27 PM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

This is just CPU. The .out for the entire testsuite was attached to my first email.

Best, Gabriele

On Tue, Oct 24, 2023 at 12:14 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Tuesday, October 24, 2023 10:07 AM *To:* Cupp, Samuel D. scupp1@my.apsu.edu *Cc:* Steven R. Brandt sbrandt@cct.lsu.edu; users@einsteintoolkit.org < users@einsteintoolkit.org> *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.

I am attaching the .out for the test.

Best, Gabriele

On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. scupp1@my.apsu.edu wrote:

Hi Gabriele, It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.

I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

---

*From:* Users users-bounces@einsteintoolkit.org on behalf of Gabriele Bozzola bozzola.gabriele@gmail.com *Sent:* Thursday, October 19, 2023 5:46 PM *To:* Steven R. Brandt sbrandt@cct.lsu.edu *Cc:* users@einsteintoolkit.org users@einsteintoolkit.org *Subject:* Re: [Users] meeting minutes for 2023-10-19

Hello,

Is there a tested configuration for anvil?

I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the MPI runs, and the test suite does not complete even with a walltime of 6 hours.

I am attaching the output.

Best, Gabriele

On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt sbrandt@cct.lsu.edu wrote:

Present: Steve, Peter, Sam, Zach, Leo

Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation - Remind Roland to send Peter the password

Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland

---

Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

Hi Steve, These are the minutes for an older meeting, so they hadn't switched yet. You are currently assigned the BNS gallery example.

Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho ________________________________ From: Users users-bounces@einsteintoolkit.org on behalf of Steven Brandt sbrandt@cct.lsu.edu Sent: Monday, November 6, 2023 10:36 AM To: users@einsteintoolkit.org users@einsteintoolkit.org Subject: Re: [Users] meeting minutes for 2023-10-19

I think the gallery examples differs from previous weeks? I think I was trading BBH with some other test?

--Steve

On 10/19/2023 9:35 AM, Steven R. Brandt wrote:

Present: Steve, Peter, Sam, Zach, Leo

Release Gallery Examples - TOV is Peter - BBH is Steve - Roland might have students for the other three? Tests - Need to be run - Clusters: Stampede2 isn't listed, and Delta and Anvil are added. - Release name: Lise Meitner https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikiped...https://en.wikipedia.org/wiki/Lise_Meitner - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation - Remind Roland to send Peter the password

Unanswered question on the mailing list - Need answer for Enzo. Roland was going to answer? - Ticket 2749: Leo says he fixed it. - Ticket 2647: One of Zach's students is looking at that in Grail (sp) - Ticket 2609: Steve thinks he approved PR for Roland

---

Users mailing list Users@einsteintoolkit.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.einst...http://lists.einsteintoolkit.org/mailman/listinfo/users

_______________________________________________ Users mailing list Users@einsteintoolkit.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.einst...http://lists.einsteintoolkit.org/mailman/listinfo/users