[Users] meeting minutes for 2023-10-19

Thu Nov 2 16:35:53 CDT 2023

Hi Gabriele,
   I've been experimenting with this, and I've reproduced this issue on a different machine. In my case, it runs with 1 OMP thread but has the segfault with 2. Can you run that test parfile (repos/GRHayLET/GRHayLHDX/test/Balsara0.par) without OMP? If this is the source, it will help narrow down the problem. I can run the test with multiple threads on my local machine though, so I'm not sure why OMP only causes a segfault sometimes.

   Samuel Cupp
   Postdoctoral Researcher
   Department of Physics
   University of Idaho
________________________________
From: Gabriele Bozzola <bozzola.gabriele at gmail.com>
Sent: Sunday, October 29, 2023 6:16 PM
To: Cupp, Samuel D. <scupp1 at my.apsu.edu>
Cc: Steven R. Brandt <sbrandt at cct.lsu.edu>; users at einsteintoolkit.org <users at einsteintoolkit.org>
Subject: Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I verified that the testsuite passes if I remove the GRHayL thorns.
Other than that, ET seems to work fine.

Attached is the cfg file I used to compile ET on Anvil.

Best,
Gabriele

On Thu, Oct 26, 2023 at 12:45 PM Cupp, Samuel D. <scupp1 at my.apsu.edu<mailto:scupp1 at my.apsu.edu>> wrote:
Hi Gabriele,
   We discussed this in the call this morning, and there's a few things we can try. First, it would be helpful if you created a ticket so we can track progress on the issue. This is especially true since the testsuite shouldn't hang if a test fails. The expected behavior would be for it to continue with testing, but for some reason it didn't. Also, it might help to remove GRHayLHDX from the thornlist, recompile, and see if that is the only test failing. Knowing if this is specific to the thorn or a broader issue would help diagnose the source of the problem.

   Samuel Cupp
   Postdoctoral Researcher
   Department of Physics
   University of Idaho
________________________________
From: Gabriele Bozzola <bozzola.gabriele at gmail.com<mailto:bozzola.gabriele at gmail.com>>
Sent: Tuesday, October 24, 2023 1:27 PM
To: Cupp, Samuel D. <scupp1 at my.apsu.edu<mailto:scupp1 at my.apsu.edu>>
Cc: Steven R. Brandt <sbrandt at cct.lsu.edu<mailto:sbrandt at cct.lsu.edu>>; users at einsteintoolkit.org<mailto:users at einsteintoolkit.org> <users at einsteintoolkit.org<mailto:users at einsteintoolkit.org>>
Subject: Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

This is just CPU. The .out for the entire testsuite was attached to my first email.

Best,
Gabriele

On Tue, Oct 24, 2023 at 12:14 PM Cupp, Samuel D. <scupp1 at my.apsu.edu<mailto:scupp1 at my.apsu.edu>> wrote:
Do you know if any other CarpetX tests fail? Also, is this using gpus or cpus?

   Samuel Cupp
   Postdoctoral Researcher
   Department of Physics
   University of Idaho
________________________________
From: Gabriele Bozzola <bozzola.gabriele at gmail.com<mailto:bozzola.gabriele at gmail.com>>
Sent: Tuesday, October 24, 2023 10:07 AM
To: Cupp, Samuel D. <scupp1 at my.apsu.edu<mailto:scupp1 at my.apsu.edu>>
Cc: Steven R. Brandt <sbrandt at cct.lsu.edu<mailto:sbrandt at cct.lsu.edu>>; users at einsteintoolkit.org<mailto:users at einsteintoolkit.org> <users at einsteintoolkit.org<mailto:users at einsteintoolkit.org>>
Subject: Re: [Users] meeting minutes for 2023-10-19

Hi Sam,

I pulled the latest version. Tests are not failing anymore, but looking at the .out, it seems that the Balsara0 test fails and stalls execution of the entire testsuite.

I am attaching the .out for the test.

Best,
Gabriele

On Fri, Oct 20, 2023 at 2:32 PM Cupp, Samuel D. <scupp1 at my.apsu.edu<mailto:scupp1 at my.apsu.edu>> wrote:
Hi Gabriele,
   It looks like some of the failures are in GRHayLHD. I've been having trouble getting consistent output with nproc>1. The data doesn't change, it just gets rearranged in the datafile. I made changes to the tests this week, so could you rerun the GRHayLHD tests and tell me if they fail? If they do, I'll change the tests so that they only run for nproc=1.

I don't know why it would take that long to run, however. Nothing in the ET CI suggests it should take that long, best I can tell. Do you have an idea of which test is taking so long?

   Samuel Cupp
   Postdoctoral Researcher
   Department of Physics
   University of Idaho
________________________________
From: Users <users-bounces at einsteintoolkit.org<mailto:users-bounces at einsteintoolkit.org>> on behalf of Gabriele Bozzola <bozzola.gabriele at gmail.com<mailto:bozzola.gabriele at gmail.com>>
Sent: Thursday, October 19, 2023 5:46 PM
To: Steven R. Brandt <sbrandt at cct.lsu.edu<mailto:sbrandt at cct.lsu.edu>>
Cc: users at einsteintoolkit.org<mailto:users at einsteintoolkit.org> <users at einsteintoolkit.org<mailto:users at einsteintoolkit.org>>
Subject: Re: [Users] meeting minutes for 2023-10-19

Hello,

Is there a tested configuration for anvil?

I compiled it one last weekend and ran the tests with the master branch. I found a few failures in the
MPI runs, and the test suite does not complete even with a walltime of 6 hours.

I am attaching the output.

Best,
Gabriele

On Thu, Oct 19, 2023 at 7:36 AM Steven R. Brandt <sbrandt at cct.lsu.edu<mailto:sbrandt at cct.lsu.edu>> wrote:
Present: Steve, Peter, Sam, Zach, Leo

Release
     Gallery Examples
     - TOV is Peter
     - BBH is Steve
     - Roland might have students for the other three?
     Tests
     - Need to be run
     - Clusters:
         Stampede2 isn't listed, and Delta and Anvil are added.
     - Release name: Lise Meitner https://en.wikipedia.org/wiki/Lise_Meitner
     - Autogenerated codes need to be regenerated, Leo will do it

Mailing list moderation
     - Remind Roland to send Peter the password

Unanswered question on the mailing list
     - Need answer for Enzo. Roland was going to answer?
     - Ticket 2749: Leo says he fixed it.
     - Ticket 2647: One of Zach's students is looking at that in Grail (sp)
     - Ticket 2609: Steve thinks he approved PR for Roland

_______________________________________________
Users mailing list
Users at einsteintoolkit.org<mailto:Users at einsteintoolkit.org>
http://lists.einsteintoolkit.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20231102/a3ce8805/attachment-0001.htm>