Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

Re: [Users] [External] Re: Running with SLURM
by Warren, Jessica Sawyer 18 Aug '22

18 Aug '22

Hi Roland, The admins reinstalled openmpi and it now runs the hello script correctly. However, the Toolkit would still produce seg faults after srun. Switching to mvapich seems to have largely done the trick though, as the TOV job is now able to start executing. As long as there is only 1 MPI process (with however many threads), the TOV job runs to completion correctly. However, anytime there are multiple MPI processes, it crashes at the first time iteration: INFO (TOVSolver): Done interpolation. --------------------------------------------------------------------------- Iteration Time | ADMBASE::alp | HYDROBASE::rho | minimum maximum | minimum maximum --------------------------------------------------------------------------- 0 0.000 | 0.6698612 0.9966374 | 1.000000e-10 0.0012800 Rank 1 with PID 3964893 received signal 11 Writing backtrace to static_tov/backtrace.1.txt srun: error: c40: task 1: Segmentation fault (core dumped) The backtrace is attached, as well as the last portion of the output, and it looks like the issue is tied to Carpet. Are there some settings in the parameter file that need adjusting or setting to fix this? Or perhaps specific settings for the number of ranks and threads? Thank you, Jessica Dr. Jessica S. Warren Physics Lecturer Indiana University Northwest warrenjs(a)iun.edu ________________________________ From: Roland Haas Sent: Thursday, August 11, 2022 8:32 AM To: Warren, Jessica Sawyer Cc: users(a)einsteintoolkit.org Subject: Re: [Users] [External] Re: Running with SLURM Hello Jessica, If you get the same error from hello-world and from Cactus then it would seem that there is still something off with the MPI stack. The -lmpi_cxx option instructs the linker to link in C++ bindings for MPI though for just the hello world example, it being C code, this is not required and -lmpi alone is sufficient. I would see two options that would let you get running somewhat quickly: 1. report your issues with OpenMPI and hello-world (including link to the source code on the web, and the exact command line to compile) to the admins and ask them for help 1.5 instead of using gcc to compile for OpenMPI do use the MPI official compiler wrapper mpicc which would just be: mpicc -o hello hello.c that is you do not have to pass and library or inlcude options. If this fails, I would definitely talk to the admins. 2. compile hello-world using mvapich. For this the easiest way is to make sure to load the mvapich module and then use the same compiler wrapper invication to compile: mpicc -o hello hello.c If 2 works then you can also compile the Einstein Toolkit with mvapich. You have to make sure to load the correct module before compiling the toolkit and then ExternalLibraries/MPI should figure out (from the mpicc wrapper) how to compile the toolkit. Yours, Roland > Hi Roland, > > Thank you so much. The compute nodes are able to be used for > compilation, and the directories match what is listed in > make.MPI.defn. When doing the 'hello' example you linked to, it was > unable to compile due to a linker error (/usr/bin/ld: cannot find > -lmpi_cxx). I re-ran it in verbose mode and found the directory it > was searching did exist and did have lmpi but not lmpi_cxx. The > admins said they had had some issues installing openmpi (couldn't > recall exactly what), and recommended mpavich (since that does have > lmpicxx installed and is their preferred implementation). However, > they reinstalled openmpi in an effort to get that to work and it did > allow the 'hello' script to compile, but when executed it produced: > > -------------------------------------------------------------------------- > No OpenFabrics connection schemes reported that they were able to be > used on a specific port. As such, the openib BTL (OpenFabrics > support) will be disabled for this port. > > Local host: h1 > Local device: mlx5_0 > Local port: 1 > CPCs attempted: rdmacm, udcm > -------------------------------------------------------------------------- > Hello world from processor h1.quartz.uits.iu.edu, rank 0 out of 1 > processors > > Similarly, doing the TOV job via sbatch, after the srun command it > gave the same OpenFabrics message (for each MPI rank) and then the > same segmentation faults as before. I've contacted the admins about > this and am waiting to hear back. Do you have any recommendations - > perhaps it would be easier to try switching over to mvapich? If so, > could you point me to some resources on how to reconfigure? > > Thank you, > Jessica > > Dr. Jessica S. Warren > Physics Lecturer > Indiana University Northwest > warrenjs(a)iun.edu > ________________________________ > From: Roland Haas <rhaas(a)illinois.edu> > Sent: Tuesday, August 9, 2022 9:48 AM > To: Warren, Jessica Sawyer <warrenjs(a)iun.edu> > Cc: users(a)einsteintoolkit.org <users(a)einsteintoolkit.org> > Subject: [External] Re: [Users] Running with SLURM > > Hello Jessica, > > You may also find something useful in the setting up a new machine > seminar presentation: > > https://urldefense.com/v3/__https://www.einsteintoolkit.org/seminars/2022_0… > > Yours, > Roland > > -- > My email is as private as my paper mail. I therefore support > encrypting and signing email messages. Get my PGP key from > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9JAgxc4juluJwklwT… > . -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 2

A Carpet error occurred in an unequal mass bns simulation
by 白济民 18 Aug '22

18 Aug '22

Hi, An error reported by the Thorn Carpet occurred when I run a bns merger simulation in the unequal mass case when applying a certain grid structure, which reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ exec: Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:262: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<T, D>::operator&(const CarpetLib::bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ However, I don't think the grid structure I applied in my simulation is peculiar: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CoordBase::xmin = -400 CoordBase::xmax = 400 CoordBase::ymin = -400 CoordBase::ymax = 400 CoordBase::zmin = 0 CoordBase::zmax = 400 CoordBase::spacing = "numcells" CoordBase::ncells_x = 40 CoordBase::ncells_y = 40 CoordBase::ncells_z = 20 ... CarpetRegrid2::num_centres = 3 CarpetRegrid2::regrid_every = 128 CarpetRegrid2::snap_to_coarse = "yes" CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::freeze_unaligned_parent_levels = "yes" # ----------- Region 1 -------------------- CarpetRegrid2::active_1 = "yes" CarpetRegrid2::num_levels_1 = 7 CarpetRegrid2::position_x_1 = -17.4316 CarpetRegrid2::position_y_1 = 0.0 CarpetRegrid2::radius_1[1] = 240.0 CarpetRegrid2::radius_1[2] = 120.0 CarpetRegrid2::radius_1[3] = 60.0 CarpetRegrid2::radius_1[4] = 30.0 CarpetRegrid2::radius_1[5] = 20.0 CarpetRegrid2::radius_1[6] = 15.0 # ----------- Region 2 -------------------- CarpetRegrid2::active_2 = "yes" CarpetRegrid2::num_levels_2 = 7 CarpetRegrid2::position_x_2 = 23.1912 CarpetRegrid2::position_y_2 = -0.0 CarpetRegrid2::radius_2[1] = 240.0 CarpetRegrid2::radius_2[2] = 120.0 CarpetRegrid2::radius_2[3] = 60.0 CarpetRegrid2::radius_2[4] = 30.0 CarpetRegrid2::radius_2[5] = 18.0 CarpetRegrid2::radius_2[6] = 14.0 # ----------- Region 3 -------------------- CarpetRegrid2::active_3 = "yes" CarpetRegrid2::num_levels_3 = 3 CarpetRegrid2::position_x_3 = 0 CarpetRegrid2::radius_3[1] = 240.0 CarpetRegrid2::radius_3[2] = 120.0 CarpetRegrid2::radius_3[3] = 66.0 CarpetRegrid2::radius_3[4] = 33.0 CarpetRegrid2::radius_x_3[5] = 18.0 CarpetRegrid2::radius_y_3[5] = 18.0 CarpetRegrid2::radius_z_3[5] = 15.0 CarpetRegrid2::radius_3[6] = 12.0 CarpetRegrid2::radius_3[7] = 8.0 # ----------------------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Though the resulotion is relatively low for test. I am wondering what the error indicates and I'm looking forward to addressing it with your help. Thank you in advance, Jimmy

2 1

Boundary conditions on a sphere
by Raimon Luna 18 Aug '22

18 Aug '22

Dear all, I would like to know if there is any way ("hacky" or not) to implement boundary conditions on some variables on a sphere within the computational domain. In other words, if there is a way to impose boundary conditions on some subset of the fields at some finite radial coordinate, smaller than the size of the computational domain, while letting the rest of the fields propagate freely until the boundary of such computational domain. I assume this would be easier with a multipatch system, but solutions for a Cartesian grid would also be ok. Thank you very much for your attention! Raimon

2 2

meeting minutes for 2022-08-18
by Roland Haas 18 Aug '22

18 Aug '22

Present: Roland, Peter, Zach, Steve, Atul. Yosef, Beyhan, Cheng-Hsin New contributions: * Zach, Cheng-Hsin: NRPyElliptic, can compile with ET * Hayley, Zach: FLRW started review, interacting, Python dependency removed * Alireza, Beyhan: time constraints, will likely skip this review and release * Sam, Peter: SelfForce1D checked out and compiling Moving back to 9:00am * accepted, no objections gallery examples: * BBH - Steve is working on it (simulation finished) functionality to retire: * Tmunu inheritance from ADMBase, StaticConformal * REQUIRES THHORN is deprecated, Roland is replacing its use by REQUIRES * also add CoordGauge Messages on list: * http://lists.einsteintoolkit.org/pipermail/users/2022-August/008654.html is an internal Carpet issue, Roland thinks that it means that Carpet tries to copy between refinement levels with different resolution, wold need stack backtrace to see exactly where it is a called from * http://lists.einsteintoolkit.org/pipermail/users/2022-August/008649.html is an MPI failure on a new cluster. Roland was in contact with her, error message indicates a failure in MPI in reduction call. Strange thing is that error happens relatively late, since there should have been reductions before. Peter says he has seen delayed SEGFAULTs if one does not ask or enough memory in the submit script (#SBATCH --mem XXX).Beyhan notes that the output indicates that 1GB is required for the simulations Open tickets: * Zach worked on FishboneMoncrief ID thorn (https://bitbucket.org/einsteintoolkit/tickets/issues/2503) with significant updates in sympy / NRPy+. Will include improved integration with ADMBase's methods. * Zach found that the proposed change can lead to slight differences wrt to centering (https://bitbucket.org/einsteintoolkit/tickets/issues/181) * Improve McLachlan accuracy (https://bitbucket.org/einsteintoolkit/tickets/issues/963), Zach added a comment, found big improvement, Peter had looked in during hackathon at UIdaho, but could not find obvious change ET conventions * Zach brought up old code in ADMBase and non-intuitive required usage * Yosef and Steve brought up suggested code style. Cactus has one in MaintGuide.pdf chapter A2 "Coding Style" (not online but in Cactus/docs/) Zach suggested adding discussions on CarpetX transition to the agenda * full transition and retirement for Carpet may be years in the future * many thorns currently require re-implementation, adapting * Steve mentions that for a while both stacks may need to be maintained * Yosef suggests that if these are many small changes then these would be good hackathon projects * Sam Cupp is working on porting IllinoisGRMHD to CarpetX * Roland mentioned weekly CarpetX call Wed 9am Central Time (join http://lists.einsteintoolkit.org/mailman/listinfo/carpetx-developers to receive Zoom invite ) * add as recurring item agenda ET workshop in Dublin: * no call during workshop (2 weeks from now) * Steve, Peter, Roland in person * Steve will present on NRPy+, will interact with Zach Chair next week: Zach Minutes next week: Roland (or Leo) Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 17 Aug '22

17 Aug '22

Hello, Please consider joining the weekly Einstein Toolkit phone call at 8:30 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 16 Aug '22

16 Aug '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

2 1

Thorn SphericalSlice
by Alex Vañó Viñuales 15 Aug '22

15 Aug '22

Dear all, I am trying to set up the multipatch domain with Carpet and I am following what is done in the LlamaWaveHyperboloidal thorn ( http://lists.einsteintoolkit.org/pipermail/users/2013-July/003115.html ). Its parameter file activates a thorn called SphericalSlice (with the capabilities of SphericalSurface and more useful features), which I have only been able to find listed in https://docs.einsteintoolkit.org/et-docs/Thorns_we_know_of . The svn repo given does not seem to work for me. Is the SphericalSlice thorn still supported and accessible? Thank you very much for your attention! Have a nice weekend. Alex -- Alex Vano-Vinuales Instituto Superior Técnico Departamento de Física/CENTRA Gabinete 2-8.8 Av. Rovisco Pais 1 1049-001 Lisboa PORTUGAL

2 3

Meeting minutes for 2002-08-11
by Steven R. Brandt 11 Aug '22

11 Aug '22

Present: Steve (minutes), Roland, Bruno, Peter, Beyhan (chair), Yosef, Zach, Leo Done with the 3rd Spec Benchmark: CPU8 We have some licensing issues that Roland discovered. Canuda progress: The review has started and sent some comments to the authors so Roland is optimistic. FLRW Solver progress: Zach isn't a cosmologist, but he is reviewing based on coding. Selfforce-1D: Peter has started, so it's progressing NRPyEllipticET: Leo says the review will start early next week, tests were done by hand. Some cleaning up of the code has been in progress. A script for the empirical formula for computing the damping was added because the referee asked for it. Roland Haas is trying to make sure the reviews are done in a timely way and the results of the published paper can be reproduced. Steve suggested adding coverage testing for new thorns. ID Reader for Elliptica. Beyhan has compiled it, but that's all so far. This may not get reviewed by the end of the month and so we might move it to the next release. Roland is not sure if the library of data files are ready. Not in a publication. Roland would like the reader in the toolkit, but would prefer not to have something that's rushed. BBH Gallery example is running. New thorns should probably add a gallery example. This would also create more visibility. Zach thinks the self-force code should have a gallery example. Functionality to remove ADMBase and StaticConformal inheritance from TmunuBase. Zach wants StaticConformal to go altogether. We could deprecate it. Could replace it with a stub of a thorn that just errors out. Steve suggests we add compile/runtime warnings during the deprecation. Spec code has such a mechanism. We could do something similar. Yosef wants to not build deprecated unless the user chooses an option. Maybe disable it in the thorn list. So announce removal, then disabled, then removed. Open tickets: Does number 2626 have an update? Roland is not sure the changes actually make things go faster. Ticket 2598? Mostly about clusters at West Virginia. Started by Maria. The maintainer is Anush. Roland will check on the pull request. Ticket 2625? Gabrielle Bozolla made a pull request. Yosef tried to use GetComponents to checkout thorns. Multiple branches cannot doesn't work. Needs --no-shallow. Yosef askes if we want to move the meeting back to 9am? We will put the question on the mailing list. Steve will try and update the intro to cactus presentation to use the wave equtaion instead of the TOV star. Volunteers for the next call: Roland will take minutes. Yosef will be chair.

1 0

Running with SLURM
by Warren, Jessica Sawyer 11 Aug '22

11 Aug '22

Hello, I am trying to get ET setup on the Quartz cluster at Indiana University (https://kb.iu.edu/d/qrtz). Simfactory compiles, however, I've run into a problem when a simulation begins and keep getting a segmentation fault (one per MPI process). The cluster uses SLURM, which is new to me, and RHEL 8. I've attached the relevant portion of an example output file (with stderr and stdout merged), the corresponding batch file that produced it when submitted, and below are the key settings from the machine file and the one line from the generic mpi runscript that I have edited. Any guidance would be much appreciated. INI settings: max-num-smt = 1 # no hyperthreading on system num-smt = 1 ppn = 128 # physical cores mpn = 2 # NUMA domains ("sockets") max-num-threads = 128 # do not oversubscribe PUs num-threads = 4 # using 4 threads per process; make 2 may be better nodes = 1 # use more for actual runs Edited runscript line: srun --nodes=@NODES@ --ntasks-per-node=@NODE_PROCS@ --cpus-per-task=@NUM_THREADS@ @EXECUTABLE@ -L 3 @PARFILE@ Thank you, Jessica Dr. Jessica S. Warren Physics Lecturer Indiana University Northwest warrenjs(a)iun.edu

2 4

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 10 Aug '22

10 Aug '22

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users