Users December 2022

users@lists.einsteintoolkit.org

12 participants
16 discussions

About the next Einstein Toolkit Summer School
by Mário Raia Neto 05 Jan '23

05 Jan '23

Dear Einstein Toolkit Team: First of all, I hope to find you safe, as well as all your beloved ones. I wish you a happy new year. Now, I'm a undergraduate physics student from Brazil and I work with general relativity. I wish to learn more about numerical relativity and due to that I would like to know if will occur a next edition of the "Einstein Toolkit Summer School"; it will be a next summer school for the next academic year? Best regards, Mário Raia Neto.

2 1

Re: [Users] Issue with Multiple Node Simulation on cluster
by Spandan Sarma 19306 27 Dec '22

27 Dec '22

Dear Erik and Steven, Thank you so much for the suggestions. We changed the runscript to add -x OMP_NUMTHREADS to the command line and it worked in solving the issue with the total number of threads being 144. Now it sets to 32 (equal to the number of procs). Also, the iterations have increased to 132105 for 32 procs (24 hr walltime) compared to just 240 before. Although this is a huge increase, we expected it to be a bit more. For a shorter walltime (30 mins) we received iterations - 2840, 2140, 1216 for procs - 32, 16, 8. Are there any more changes that we can do to improve on this? The new runscript and the output file (as a drive link) are attached below (no changes were made to the machine file, option list and the submit script from before). p32_omp.out <https://drive.google.com/file/d/1TpM03UQ6vfxJS2ArxzZuI3MBD6JzrTnP/view?usp=…> On Thu, Dec 15, 2022 at 2:49 PM Spandan Sarma 19306 <spandan19(a)iiserb.ac.in> wrote: > Dear Erik and Steven, > > Thank you so much for the suggestions. We changed the runscript to add -x > OMP_NUMTHREADS to the command line and it worked in solving the issue with > the total number of threads being 144. Now it sets to 32 (equal to the > number of procs). > > Also, the iterations have increased to 132105 for 32 procs (24 hr > walltime) compared to just 240 before. Although this is a huge increase, we > expected it to be a bit more. For a shorter walltime (30 mins) we received > iterations - 2840, 2140, 1216 for procs - 32, 16, 8. Are there any more > changes that we can do to improve on this? > > The new runscript and the output file for 32 procs are attached below (no > changes were made to the machine file, option list and the submit script > from before). > > On Fri, Dec 9, 2022 at 8:13 PM Steven R. Brandt <sbrandt(a)cct.lsu.edu> > wrote: > >> It's not too late to do a check, though, to see if all other nodes have >> the same OMP_NUM_THREADS value. Maybe that's the warning? It sounds like >> it should be an error. >> >> --Steve >> >> On 12/8/2022 5:23 PM, Erik Schnetter wrote: >> > Steve >> > >> > Code that runs as part of the Cactus executable is running too late >> > for this. At that time, OpenMP has already been initialized. >> > >> > There is the environment variable "CACTUS_NUM_THREADS" which is >> > checked at run time, but only if it is set (for backward >> > compatibility). Most people do not bother setting it, leaving this >> > error undetected. There is a warning output, but these are generally >> > ignored. >> > >> > -erik >> > >> > On Thu, Dec 8, 2022 at 3:48 PM Steven R. Brandt <sbrandt(a)cct.lsu.edu> >> wrote: >> >> We could probably add some startup code in which MPI broadcasts the >> >> OMP_NUM_THREADS setting to all the other processes and either checks >> the >> >> value of the environment variable or calls omp_set_num_threads() or >> some >> >> such. >> >> >> >> --Steve >> >> >> >> On 12/8/2022 9:03 AM, Erik Schnetter wrote: >> >>> Spandan >> >>> >> >>> The problem is likely that MPI does not automatically forward your >> >>> OpenMP setting to the other nodes. You are setting the environment >> >>> variable OMP_NUM_THREADS in the run script, and it is likely necessary >> >>> to forward this environment variable to the other processes as well. >> >>> Your MPI documentation will tell you how to do this. This is likely an >> >>> additional option you need to pass when calling "mpirun". >> >>> >> >>> -erik >> >>> >> >>> On Thu, Dec 8, 2022 at 2:50 AM Spandan Sarma 19306 >> >>> <spandan19(a)iiserb.ac.in> wrote: >> >>>> Hello, >> >>>> >> >>>> >> >>>> This mail is in continuation to the ticket, “Issue with compiling ET >> on cluster”, by Shamim. >> >>>> >> >>>> >> >>>> So after Roland’s suggestion, we found that using the –prefix >> <openmpi-directory> command along with hostfile worked successfully in >> simulating a multiple node simulation in our HPC. >> >>>> >> >>>> >> >>>> Now we find that the BNSM gallery simulation evolves for only 240 >> iterations on 2 nodes (16+16 procs, 24 hr walltime), which is very slow >> with respect to, simulation on 1 node (16 procs, 24 hr walltime) evolved >> for 120988 iterations. The parallelization process goes well within 1 node, >> we received iterations - 120988, 67756, 40008 for procs - 16, 8, 4 (24 hr >> walltime) respectively. We are unable to understand what is causing this >> issue when openmpi is given 2 nodes (16+16 procs). >> >>>> >> >>>> >> >>>> In the output files we found the following, which may be an >> indication towards the issue: >> >>>> >> >>>> IINFO (Carpet): MPI is enabled >> >>>> >> >>>> INFO (Carpet): Carpet is running on 32 processes >> >>>> >> >>>> INFO (Carpet): This is process 0 >> >>>> >> >>>> INFO (Carpet): OpenMP is enabled >> >>>> >> >>>> INFO (Carpet): This process contains 1 threads, this is thread 0 >> >>>> >> >>>> INFO (Carpet): There are 144 threads in total >> >>>> >> >>>> INFO (Carpet): There are 4.5 threads per process >> >>>> >> >>>> INFO (Carpet): This process runs on host n129, pid=20823 >> >>>> >> >>>> INFO (Carpet): This process runs on 1 core: 0 >> >>>> >> >>>> INFO (Carpet): Thread 0 runs on 1 core: 0 >> >>>> >> >>>> INFO (Carpet): This simulation is running in 3 dimensions >> >>>> >> >>>> INFO (Carpet): Boundary specification for map 0: >> >>>> >> >>>> nboundaryzones: [[3,3,3],[3,3,3]] >> >>>> >> >>>> is_internal : [[0,0,0],[0,0,0]] >> >>>> >> >>>> is_staggered : [[0,0,0],[0,0,0]] >> >>>> >> >>>> shiftout : [[1,0,1],[0,0,0]] >> >>>> >> >>>> WARNING level 1 from host n131 process 21 >> >>>> >> >>>> in thorn Carpet, file >> /home2/mallick/ET9/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:426: >> >>>> >> >>>> -> The number of threads for this process is larger its number >> of cores. This may indicate a performance problem. >> >>>> >> >>>> >> >>>> This is something that we couldn’t understand as we asked for only >> 32 procs, with num-threads set to 1. The command that we used to submit our >> job was: >> >>>> >> >>>> ./simfactory/bin/sim create-submit p32_mpin_npn --procs=32 >> --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 >> --parfile=par/nsnstohmns1.par --walltime=24:10:00 >> >>>> >> >>>> >> >>>> I have attached the out file, runscript, submitscript, optionlist, >> machine file for reference. Thanks in advance for help. >> >>>> >> >>>> >> >>>> Sincerely, >> >>>> >> >>>> -- >> >>>> Spandan Sarma >> >>>> BS-MS' 19 >> >>>> Department of Physics (4th Year), >> >>>> IISER Bhopal >> >>>> _______________________________________________ >> >>>> Users mailing list >> >>>> Users(a)einsteintoolkit.org >> >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users >> >>> >> >> _______________________________________________ >> >> Users mailing list >> >> Users(a)einsteintoolkit.org >> >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > >> > >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > > > -- > Spandan Sarma > BS-MS' 19 > Department of Physics (4th Year), > IISER Bhopal > -- Spandan Sarma BS-MS' 19 Department of Physics (4th Year), IISER Bhopal

2 2

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 19 Dec '22

19 Dec '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

2 1

Is the old CactusEinstein arrangement available?
by Bernard Kelly 19 Dec '22

19 Dec '22

Hi. I wanted to take a look at the old thorn PsiKadelia, which was part of the CactusEinstein arrangement. How would I go about obtaining that thorn (or the arrangement as a whole)? I tried adding a few lines to the einsteintoolkit.th for the ET_2022_05 release, as follows: # CactusEinstein thorns !TARGET = $ARR !TYPE = git !URL = https://bitbucket.org/cactuscode/cactuseinstein.git !REPO_BRANCH = $ET_RELEASE !REPO_PATH= $2 !CHECKOUT = CactusEinstein/PsiKadelia ... but this fails, even if I remove the REPO_BRANCH line. Is CactusEinstein accessible at all? If so, how? thanks, Bernard -- ------------------------------------------------------------------ Bernard Kelly -- CRESST Assistant Research Scientist, NASA/GSFC Gravitational Astrophysics Laboratory --- Code 663 Phone: +1 (301) 286-7243 *** Fax: +1 (301) 286-2226 Web: http://science.gsfc.nasa.gov/sed/bio/bernard.j.kelly ORCID: orcid.org/0000-0002-3326-4454 ------------------------------------------------------------------

3 3

Re: [Users] Issue with Multiple Node Simulation on cluster
by Erik Schnetter 19 Dec '22

19 Dec '22

Spandan It is quite possible that different build options, different MPI options, or different parameter settings would improve the performance of your calculation. Performance optimisation is a difficult topic, and it's impossible to say anything in general. A good starting point would be to run your simulation on a different system, to run a different parameter file with a known setup on your system, and then to compare. -erik On Thu, Dec 15, 2022 at 4:19 AM Spandan Sarma 19306 <spandan19(a)iiserb.ac.in> wrote: > > Dear Erik and Steven, > > Thank you so much for the suggestions. We changed the runscript to add -x OMP_NUMTHREADS to the command line and it worked in solving the issue with the total number of threads being 144. Now it sets to 32 (equal to the number of procs). > > Also, the iterations have increased to 132105 for 32 procs (24 hr walltime) compared to just 240 before. Although this is a huge increase, we expected it to be a bit more. For a shorter walltime (30 mins) we received iterations - 2840, 2140, 1216 for procs - 32, 16, 8. Are there any more changes that we can do to improve on this? > > The new runscript and the output file for 32 procs are attached below (no changes were made to the machine file, option list and the submit script from before). > > On Fri, Dec 9, 2022 at 8:13 PM Steven R. Brandt <sbrandt(a)cct.lsu.edu> wrote: >> >> It's not too late to do a check, though, to see if all other nodes have >> the same OMP_NUM_THREADS value. Maybe that's the warning? It sounds like >> it should be an error. >> >> --Steve >> >> On 12/8/2022 5:23 PM, Erik Schnetter wrote: >> > Steve >> > >> > Code that runs as part of the Cactus executable is running too late >> > for this. At that time, OpenMP has already been initialized. >> > >> > There is the environment variable "CACTUS_NUM_THREADS" which is >> > checked at run time, but only if it is set (for backward >> > compatibility). Most people do not bother setting it, leaving this >> > error undetected. There is a warning output, but these are generally >> > ignored. >> > >> > -erik >> > >> > On Thu, Dec 8, 2022 at 3:48 PM Steven R. Brandt <sbrandt(a)cct.lsu.edu> wrote: >> >> We could probably add some startup code in which MPI broadcasts the >> >> OMP_NUM_THREADS setting to all the other processes and either checks the >> >> value of the environment variable or calls omp_set_num_threads() or some >> >> such. >> >> >> >> --Steve >> >> >> >> On 12/8/2022 9:03 AM, Erik Schnetter wrote: >> >>> Spandan >> >>> >> >>> The problem is likely that MPI does not automatically forward your >> >>> OpenMP setting to the other nodes. You are setting the environment >> >>> variable OMP_NUM_THREADS in the run script, and it is likely necessary >> >>> to forward this environment variable to the other processes as well. >> >>> Your MPI documentation will tell you how to do this. This is likely an >> >>> additional option you need to pass when calling "mpirun". >> >>> >> >>> -erik >> >>> >> >>> On Thu, Dec 8, 2022 at 2:50 AM Spandan Sarma 19306 >> >>> <spandan19(a)iiserb.ac.in> wrote: >> >>>> Hello, >> >>>> >> >>>> >> >>>> This mail is in continuation to the ticket, “Issue with compiling ET on cluster”, by Shamim. >> >>>> >> >>>> >> >>>> So after Roland’s suggestion, we found that using the –prefix <openmpi-directory> command along with hostfile worked successfully in simulating a multiple node simulation in our HPC. >> >>>> >> >>>> >> >>>> Now we find that the BNSM gallery simulation evolves for only 240 iterations on 2 nodes (16+16 procs, 24 hr walltime), which is very slow with respect to, simulation on 1 node (16 procs, 24 hr walltime) evolved for 120988 iterations. The parallelization process goes well within 1 node, we received iterations - 120988, 67756, 40008 for procs - 16, 8, 4 (24 hr walltime) respectively. We are unable to understand what is causing this issue when openmpi is given 2 nodes (16+16 procs). >> >>>> >> >>>> >> >>>> In the output files we found the following, which may be an indication towards the issue: >> >>>> >> >>>> IINFO (Carpet): MPI is enabled >> >>>> >> >>>> INFO (Carpet): Carpet is running on 32 processes >> >>>> >> >>>> INFO (Carpet): This is process 0 >> >>>> >> >>>> INFO (Carpet): OpenMP is enabled >> >>>> >> >>>> INFO (Carpet): This process contains 1 threads, this is thread 0 >> >>>> >> >>>> INFO (Carpet): There are 144 threads in total >> >>>> >> >>>> INFO (Carpet): There are 4.5 threads per process >> >>>> >> >>>> INFO (Carpet): This process runs on host n129, pid=20823 >> >>>> >> >>>> INFO (Carpet): This process runs on 1 core: 0 >> >>>> >> >>>> INFO (Carpet): Thread 0 runs on 1 core: 0 >> >>>> >> >>>> INFO (Carpet): This simulation is running in 3 dimensions >> >>>> >> >>>> INFO (Carpet): Boundary specification for map 0: >> >>>> >> >>>> nboundaryzones: [[3,3,3],[3,3,3]] >> >>>> >> >>>> is_internal : [[0,0,0],[0,0,0]] >> >>>> >> >>>> is_staggered : [[0,0,0],[0,0,0]] >> >>>> >> >>>> shiftout : [[1,0,1],[0,0,0]] >> >>>> >> >>>> WARNING level 1 from host n131 process 21 >> >>>> >> >>>> in thorn Carpet, file /home2/mallick/ET9/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:426: >> >>>> >> >>>> -> The number of threads for this process is larger its number of cores. This may indicate a performance problem. >> >>>> >> >>>> >> >>>> This is something that we couldn’t understand as we asked for only 32 procs, with num-threads set to 1. The command that we used to submit our job was: >> >>>> >> >>>> ./simfactory/bin/sim create-submit p32_mpin_npn --procs=32 --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par --walltime=24:10:00 >> >>>> >> >>>> >> >>>> I have attached the out file, runscript, submitscript, optionlist, machine file for reference. Thanks in advance for help. >> >>>> >> >>>> >> >>>> Sincerely, >> >>>> >> >>>> -- >> >>>> Spandan Sarma >> >>>> BS-MS' 19 >> >>>> Department of Physics (4th Year), >> >>>> IISER Bhopal >> >>>> _______________________________________________ >> >>>> Users mailing list >> >>>> Users(a)einsteintoolkit.org >> >>>> http://lists.einsteintoolkit.org/mailman/listinfo/users >> >>> >> >> _______________________________________________ >> >> Users mailing list >> >> Users(a)einsteintoolkit.org >> >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > >> > >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > -- > Spandan Sarma > BS-MS' 19 > Department of Physics (4th Year), > IISER Bhopal -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

2 1

Meeting minutes for 2022-12-15
by Leonardo Rosa Werneck 19 Dec '22

19 Dec '22

Hi all, Here are the minutes for today’s meeting. Cheers, Leo ------------ Chair : Sam Cupp Notes : Leo Werneck Present: Leo Werneck, Peter Diener, Sam Cupp, Steve Brandt - Any updates on github & Ian Hindler's account? * Something was compromised with his account, but github canceled the token and looks like nothing bad happened. Ian is working on it. - ET mailing list archives not using good SSL certificates * Steve will contact Roland, since this was not discussed last week but was in the agenda. - Safety feature to avoid HDF5 files from being corrupted * Leo requests a feature that would allow the user to e.g., generate one output file per restart. With kuibit, there was interested in switching from the ASCII data files to the HDF5 in our research group. However, in a recent simulation it turned out that a node failure caused a crash as one of the HDF5 was being written to and we lost all data for an important gridfunction. If one HDF5 file was written per restart (or another safety feature was in place), then this would have not been an issue, as only one of the chunks of data would have been corrupted. Leo will open a ticket about this. - Unanswered questions on the mailing list * Spandan Sarma had an issue where it seemed the code didn't scale as expected, but Peter pointed out that the scaling is more or less what's expected. - Open tickets * All website related tickets are likely more interesting to Roland, so we won't be discussing them today. Next meeting : possibly Jan 5, 2023 Next chair : pending Next minute taker: pending ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo(a)uidaho.edu <mailto:leonardo@uidaho.edu> https://leowerneck.github.io <https://leowerneck.github.io/>

4 4

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 14 Dec '22

14 Dec '22

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Meeting minutes for 2022-12-08
by Steven R. Brandt 14 Dec '22

14 Dec '22

Present: Steve, Roland, Keith, Yosef, Sam, Zach, Gabriele, Leo Next ET release chair search: Leo, deputy chair: Sam Call for new contribution proposals: Fishbone Moncrief Gallery Example developed at the ICERM Brown workshop CarpetX prerelease These are the things we said we wanted: https://docs.einsteintoolkit.org/et-docs/CarpetX_Transition Have we finalized how CarpetX works? E.g. boundary conditions Maybe more people need to be involved in discussing the BC's? Need to schedule a BC call to make sure we have all our requirements? Will Hyperslab-based BC's work? Reach out to AMReX developers? Some classes might need to be renamed, maybe the repo, some thorns should be rearranged Gabriele Bozzala Does not test that the code is converging Want the ability to run convergence tests for Thorn Developers Bring up on the CarpetX call? Emails about Ian Hinder's account being compromised. Not sure whether this is a concern. Zach has said that there is confusion over name of the toolkit. And it's true. Can we make it clearer that ET is not Cactus. (Einstein Toolkit != Cactus) Don't want people to feel non-cactus codes are being washed out. The debate should continue. Ticket 963 Peter ran the gallery example and constraint damping. It made a big difference in the number of orbits. Reduced orbits from 5 to 2. Not sure what's wrong. Restarting with constraints output. Will discuss with Zach. Chair: Sam Minute Taker: Roland

2 1

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 12 Dec '22

12 Dec '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Issue with Multiple Node Simulation on cluster
by Spandan Sarma 19306 09 Dec '22

09 Dec '22

Hello, This mail is in continuation to the ticket, “Issue with compiling ET on cluster”, by Shamim. So after Roland’s suggestion, we found that using the –prefix <openmpi-directory> command along with hostfile worked successfully in simulating a multiple node simulation in our HPC. Now we find that the BNSM gallery simulation evolves for only 240 iterations on 2 nodes (16+16 procs, 24 hr walltime), which is very slow with respect to, simulation on 1 node (16 procs, 24 hr walltime) evolved for 120988 iterations. The parallelization process goes well within 1 node, we received iterations - 120988, 67756, 40008 for procs - 16, 8, 4 (24 hr walltime) respectively. We are unable to understand what is causing this issue when openmpi is given 2 nodes (16+16 procs). In the output files we found the following, which may be an indication towards the issue: IINFO (Carpet): MPI is enabled INFO (Carpet): Carpet is running on 32 processes INFO (Carpet): This is process 0 INFO (Carpet): OpenMP is enabled INFO (Carpet): This process contains 1 threads, this is thread 0 INFO (Carpet): There are 144 threads in total INFO (Carpet): There are 4.5 threads per process INFO (Carpet): This process runs on host n129, pid=20823 INFO (Carpet): This process runs on 1 core: 0 INFO (Carpet): Thread 0 runs on 1 core: 0 INFO (Carpet): This simulation is running in 3 dimensions INFO (Carpet): Boundary specification for map 0: nboundaryzones: [[3,3,3],[3,3,3]] is_internal : [[0,0,0],[0,0,0]] is_staggered : [[0,0,0],[0,0,0]] shiftout : [[1,0,1],[0,0,0]] WARNING level 1 from host n131 process 21 in thorn Carpet, file /home2/mallick/ET9/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:426: -> The number of threads for this process is larger its number of cores. This may indicate a performance problem. This is something that we couldn’t understand as we asked for only 32 procs, with num-threads set to 1. The command that we used to submit our job was: ./simfactory/bin/sim create-submit p32_mpin_npn --procs=32 --ppn=16 --num-threads=1 --ppn-used=16 --num-smt=1 --parfile=par/nsnstohmns1.par --walltime=24:10:00 I have attached the out file, runscript, submitscript, optionlist, machine file for reference. Thanks in advance for help. Sincerely, -- Spandan Sarma BS-MS' 19 Department of Physics (4th Year), IISER Bhopal

3 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users December 2022