Users September 2022

users@lists.einsteintoolkit.org

15 participants
19 discussions

Meeting Minutes 2022-9-15
by Cupp, Samuel D. 15 Sep '22

15 Sep '22

Present: Sam, Steve, Keith, Gabriele, Peter, Leo, Zach, Bruno New ET modules: * no news on Canuda review * Haley has addressed reviewers' comments for FLRW solver * Sam has addressed reviewers' comments for selfforce-1d; only remaining task is to regenerate the FORD documentation * Leo has addressed reviewers' comments for NRPyEllipticET; notebook which generates the code is not yet ready, but this is currently being worked on * Baikal update: Zach hasn't heard back from Helvi. Since the new Baikal is waiting for inclusion, the old Baikal's notebook isn't being maintained in favor of the new Baikal Other questions: * For the particle tracer Leo is working on, he needs to access the cctk_timefac for the finest level, but cctk_timefac only gives the value for the current time level. He does this by accessing the Carpet parameter time_refinement_factors using CCTK_ParameterGet() but wanted to know if there was a better/more 'proper' way of accessing the parameters. The function is necessary because the parameter is private and cannot be accessed via the interface. Steve said that this is how he would do it, but Leo should send the question to the mailing list to see if Roland or Erik have other suggestions. * We would like to shut down the svn servers which aren't being used anymore and have been maintained for preservation of old thornlists for a long time. By shutting them down, we can see if anyone complains about it and also track down anywhere else on the website that might reference them. * There is no established policy on deprecating or renaming thorns/parameters. Zach proposed that we should set a clear policy for the timescale on which these things can be done, and the procedure for doing so. * Someone asked about whether there will be recordings of the European ET meeting, but Peter said there were not. However, they do intend to upload the slides at some point. * Next European meeting was set and will be in Aveiro, Portugal. Questions on the mailing list: No one answered 'Running with SLURM' from last week; Steve will respond to it. http://lists.einsteintoolkit.org/pipermail/users/2022-August/008667.html Open tickets: Gabriele brought up ticket issue #2629 https://bitbucket.org/einsteintoolkit/tickets/issues/2629/mol_pseudoevoluti… about when one should schedule in CCTK_ANALYSIS vs MoL_PseudoEvolution. The particular concern is how this will interact with parameters such as the calculate_constraints_every in LeanBSSNMoL, as some choices of output_every type parameters will be invalid depending on the number of active refinement levels. However, since these calculations are expensive, it is important to be able to control the calculation to only occur when it will be output. Zach said that one solution is to just leave it up to the user to decide if it is sane to update these quantities every n timesteps. This is similar to the issue Leo has been facing with the particle tracer, as it must compute and output only so many iterations. However, Cactus counts iterations based on the smallest possible refinement level, even if it isn't active. This leads to the need to determine the dtfac for the finest possible level, which is the topic of the question raised earlier. He then errors out if the parameters will produce incorrect output. Since several thorns need this feature, discussion arose on whether this should be provided as an aliased function by Carpet to easily provide access to the needed information to validate the given values of these types of output_every parameters. Carpet could set some out_every_safe that provides the minimal safest out_every. The chair for next week (9/22) is Leo, and the minutes taker is Bruno. Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

2 3

Issue with black hole binary evolution employing IGM
by Federico Cattorini 15 Sep '22

15 Sep '22

Hello everyone, I am experiencing some issue in a number of GRMHD simulations of black hole binaries employing IllinoisGRMHD. As an example, I will write about an unequal-mass BHB configuration (with q = 2) that I'm running. After approximately ten orbits, the run stops with no error codes or any other message that could help me identify the issue. The last lines of the standard output are INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160260e+03 ***** INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393 VL= 179 rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02, ErrDenom: 4.838e+13 | 4.51 iters/gridpt INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160269e+03 ***** Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST I tried restarting my simulation from the latest checkpoint, but the same sudden stop occurred at the same timestep. At first, I thought about some problem with IGM. The last INFO is printed by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints in it to identify the spot where the error occurs. Unfortunately, I drew a blank, since the stop seems to occur just after the end of IllinoisGRMHD_driver_evaluate_MHD_rhs: INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to time: 3.160251e+03 ***** INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM ***** INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs ***** Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST I tried to restart the simulation and run it on pure MPI. It ran for a few more iterations, then stopped as well: INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to time: 3.156831e+03 ***** INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM ***** INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs ***** Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST The simulation setup is as follows: Allocated: Nodes: 10 Cores per node: 48 SLURM setting SLURM_NNODES : 10 SLURM_NPROCS : 20 SLURM_NTASKS : 20 SLURM_CPUS_ON_NODE : 48 SLURM_CPUS_PER_TASK : 24 SLURM_TASKS_PER_NODE: 2(x10) Running: MPI processes: 20 OpenMP threads per process: 24 MPI processes per node: 2.0 OpenMP threads per core: 1.0 OpenMP threads per node: 48 while the pure-MPI setup is Allocated: Nodes: 10 Cores per node: 48 SLURM setting SLURM_NNODES : 10 SLURM_NPROCS : 480 SLURM_NTASKS : 480 SLURM_CPUS_ON_NODE : 48 SLURM_CPUS_PER_TASK : 1 SLURM_TASKS_PER_NODE: 48(x10) Running: MPI processes: 480 OpenMP threads per process: 1 MPI processes per node: 48.0 OpenMP threads per core: 1.0 OpenMP threads per node: 48 I am using The Lorentz version of ET. I've had this issue for two binary BH simulations, both unequal-mass with q = 2. My colleague Giacomo Fedrigo experienced the same problem running an equal-mass simulation. I attach the q = 2 (s_UUmis_Q2) parameter file and the ET config-info file. Also, I attach the st. error and output of my q = 2 run and of Giacomo's run (b1_UUmis_a12b_pol3_r56_gauss_9). The st. outputs were cut for readability reasons. Can someone please help me with this? Thanks in advance, Federico

2 1

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 14 Sep '22

14 Sep '22

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 12 Sep '22

12 Sep '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Problems with simfactory compilation with ThornList-ET_2018_02.
by Michal Pirog 12 Sep '22

12 Sep '22

Hi there. Let me ask you here about a little help... I downloaded "ThornList-ET_2018_02" from: https://bitbucket.org/zach_etienne/wvuthorns_diagnostics/src/master/ then I run "./GetComponents ThornList-ET_2018_02" and I've got an output as you can see in "output.dat". It created a Cactus directory for me, so I went there and tried to compile it by: ./simfactory/bin/sim setup-silent ./simfactory/bin/sim build -j12 --thornlist ../ThornList-ET_2018_02 --optionlist ../dns4.cfg My dns4.cfg option list is probably irrelevant, but I'm sending it too. The compilation failed, as you can see in x4x.out. Any idea? Any simple and straightforward solution? PS: Let me also mention that when I try to get components again, I have an info related to these particular components which are mentioned in the compilation errors. Look at the output2.dat. Michal Pirog

2 1

Meeting minutes for 2022-09-08
by Leonardo Rosa Werneck 08 Sep '22

08 Sep '22

Hi all, Here are the minutes from today’s meeting. Chair: Peter Minutes: Leo Present: Peter Diener, Leo Werneck, Sam Cupp, Zach Etienne, Keith Dow, Steve Brandt, Beyhan, Gabriele Bozzola, Roland Haas, Yosef Zlochower * Updates on new thorns proposed for inclusion and revisions: - NRPyEllipticET: got nice feedback from reviewers. Leo will be working on updates over the next couple of weeks. - SelfForce1D: no progress during the past week. - Baikal: Updates are mostly going to be applied to the code generation scripts. - FLRW Solver: thorn has been thoroughly reviewed for best programming practices and is looking good. - Canuda Solver: No update since no reviewer or champion present. * Zach announces: CarpetX-compatible versions of Baikal and IllinoisGRMHD will be developed. "BaikalX" may already exist, will look into it. * Steve asks: can NRPyElliptic handle the surfaces of NSs when generating ID? Zach says: BNS initial data will be the subject of a future project. There is no doubt that NRPyElliptic can solve the elliptic problem, but the techniques requires for handling the NS surface are yet not completely clear. * Zach suggests: some of ADMBase's functionalities (e.g., setting ID) should be moved elsewhere, as currently the thorn does (too) much more than what its name suggests. Cheers, Leo ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo(a)uidaho.edu <mailto:leonardo@uidaho.edu> https://leowerneck.github.io <https://leowerneck.github.io/>

1 0

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 07 Sep '22

07 Sep '22

1 0

Setting up ETK on Discoverer
by Jay Vijay Kalinani 06 Sep '22

06 Sep '22

Hi all, I am trying to perform simulations on the Discoverer cluster ( https://docs.discoverer.bg/index.html) using the latest Einstein Toolkit (ETK) release (ET_2022_05) and the Spritz GRMHD code. To compile ETK on Discoverer, I am attaching the simfactory configuration files which I had newly prepared. I am also attaching the list of modules which were loaded. To submit the simulation, for instance, I use the following simfactory command: sim submit BNS_IF_fluxCT_dx018_q10_RPA_RotGas_E8e49 --parfile=./par/BNS_IF_fluxCT_dx018_q10_RPA_RotGas_E8e49.par --config=newspritzgnu --machine=discoverer --procs=256 --num-threads=1 --ppn-used=128 --walltime=24:00:00 Unfortunately, my simulations crash after running for some time. My guess is that I might not be correctly setting the configuration options or flags during compilation, which might affect my simulation during runtime, but I am not completely certain. I am attaching the output file, the error file, the parfile as well as the generated backtrace for the simulation which used 256 procs. I also looked at the hexadecimal addresses in the backtrace with addr2line, but unfortunately all of them return "??:0" I also noticed that when changing the number of processors, the simulation crashes at different times. But if I keep the number of processors as the same, the simulation always crashes at the same point. For instance, simulation with 256 processors ran for about 2 hours on the cluster, and crashed after completing about 4600 iterations. One submitted with 1280 processors ran for about 12 hours and crashed after completing about 32000 iterations. Simulation with 1792 processors instead crashed soon after the start of the simulation (within few minutes), even before reaching iteration 0. For all cases, I always set number of threads as 1. If you have any suggestions or insights on why the simulations crash and in case I have any incorrect settings in the configuration files, kindly let me know. I would greatly appreciate your help. If you need any further information from my side, please let me know too. Thank you very much. Kind regards, Jay Kalinani

2 1

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 05 Sep '22

05 Sep '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users September 2022