Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 26 Sep '22

26 Sep '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Computing Apparent Horizons
by Mario Imbrogno 24 Sep '22

24 Sep '22

Dear Einstein-Toolkit community, my name is Mario Imbrogno and I am a Ph.D. student at the University of Calabria (UniCal, Cosenza, Italy) under the supervision of prof. Sergio Servidio. Our research group had developed a spectral code to solve the Einstein equations via a pseudo-spectral technique [Meringolo C, Servidio S, and Veltri P "A spectral method algorithm for numerical simulations of gravitational fields." Classical and Quantum Gravity 38.7 (2021): 075027], and I am studying different cases of merging two and more black holes, in vacuum conditions. Now I have to compute the apparent horizons (AHs) just for our initial conditions, and I have read that the Einstein Toolkit implements the AHFinderDirect, a code that can detect the AHs. My questions are: is there a way to compute such AHs by, just uploading our 3D cartesian BSSN fields and using the AHFinderDirect code? Who is(are) the user(s) that mainly use(s) such code? Who can I ask? Where can I find a detailed guide in which I can learn how to upload my data using the AHFinderDirect code? If someone will help us, there is no problem for us to include as a co-author of our paper. Best regards, Mario Imbrogno

2 3

Issue with black hole binary evolution employing IGM
by Federico Cattorini 23 Sep '22

23 Sep '22

Hello Erik, Thanks for your reply. I followed your suggestions and found two strings that may be indicative of what's going on. In the standard outputs 'CCTK_Proc14.out' and 'CCTK_Proc15.out' the last lines read INFO (IllinoisGRMHD): Font fix failed! INFO (IllinoisGRMHD): i,j,k = 67 63 16, stats.failure_checker = 0 x,y,z = 3.392857e+00 8.892857e+00 2.392857e+00 , index=111739 st_i = -1.002115e+08 2.298583e+08 -1.221746e+08, rhostar = 1.573103e+02, Bi = -1.064528e+03 1.120144e+03 2.972675e+03, gij = 6.643816e+00 5.521615e-01 4.380688e-01 7.244355e+00 -1.685406e-03 6.830374e+00, Psi6 = 1.803534e+01 I assume this means that there are issues in the con2prim of IGM. That INFO is printed by harm_primitives_lowlevel.C: // Use the new Font fix subroutine > int font_fix_applied=0; > if(check!=0) { > font_fix_applied=1; > CCTK_REAL u_xl=1e100, u_yl=1e100, u_zl=1e100; // Set to insane > values to ensure they are overwritten. > if (gamma_equals2==1) { > check = > font_fix_gamma_equals2(u_xl,u_yl,u_zl,CONSERVS,PRIMS,METRIC_PHYS,METRIC_LAP_PSI4,eos); > } else { > check = > font_fix_general_gamma(u_xl,u_yl,u_zl,CONSERVS,PRIMS,METRIC_PHYS,METRIC_LAP_PSI4,eos); > } > //Translate to HARM primitive now: > prim[UTCON1] = METRIC_PHYS[GUPXX]*u_xl + METRIC_PHYS[GUPXY]*u_yl + > METRIC_PHYS[GUPXZ]*u_zl; > prim[UTCON2] = METRIC_PHYS[GUPXY]*u_xl + METRIC_PHYS[GUPYY]*u_yl + > METRIC_PHYS[GUPYZ]*u_zl; > prim[UTCON3] = METRIC_PHYS[GUPXZ]*u_xl + METRIC_PHYS[GUPYZ]*u_yl + > METRIC_PHYS[GUPZZ]*u_zl; > if (check==1) { > CCTK_VInfo(CCTK_THORNSTRING,"Font fix failed!"); > CCTK_VInfo(CCTK_THORNSTRING,"i,j,k = %d %d %d, > stats.failure_checker = %d x,y,z = %e %e %e , index=%d st_i = %e %e %e, > rhostar = %e, Bi = %e %e %e, gij = %e %e %e %e %e %e, Psi6 = > %e",i,j,k,stats.failure_checker,X[index],Y[index],Z[index],index,mhd_st_x_orig,mhd_st_y_orig,mhd_st_z_orig,rho_star_orig,PRIMS[BX_CENTER],PRIMS[BY_CENTER],PRIMS[BZ_CENTER],METRIC_PHYS[GXX],METRIC_PHYS[GXY],METRIC_PHYS[GXZ],METRIC_PHYS[GYY],METRIC_PHYS[GYZ],METRIC_PHYS[GZZ],METRIC_LAP_PSI4[PSI6]); > exit(1); // Let's exit instead of printing potentially GBs of log > files. Uncomment if you really want to deal with a mess. > } > } > stats.failure_checker+=font_fix_applied*10000; > stats.font_fixed=font_fix_applied; > Can I do anything that may help pinpoint the cause of this error? Thanks in advance, Federico Il giorno gio 15 set 2022 alle ore 15:30 Erik Schnetter <schnetter(a)gmail.com> ha scritto: > Federico > > Thanks for including the output, that is helpful. > > There are parameters "Carpet::verbose" and "Carpet::veryverbose". You > can set them to "yes" and recover from a checkpoint. This gives more > information about what the code is doing, and thus where it crashes. > > The output you attached is only from the first MPI process. Other > processes' output might contain a clue. You can add the command line > option "-roe" to Cactus when you run the simulation. This will collect > output from all processes. > > -erik > > On Thu, Sep 15, 2022 at 9:20 AM Federico Cattorini > <f.cattorini(a)campus.unimib.it> wrote: > > > > Hello everyone, > > > > I am experiencing some issue in a number of GRMHD simulations of black > hole binaries employing IllinoisGRMHD. > > > > As an example, I will write about an unequal-mass BHB configuration > (with q = 2) that I'm running. > > > > After approximately ten orbits, the run stops with no error codes or any > other message that could help me identify the issue. The last lines of the > standard output are > > > > INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: > 3.160260e+03 ***** > > INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393 VL= > 179 rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02, ErrDenom: > 4.838e+13 | 4.51 iters/gridpt > > INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: > 3.160269e+03 ***** > > Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST > > > > > > > > I tried restarting my simulation from the latest checkpoint, but the > same sudden stop occurred at the same timestep. > > > > At first, I thought about some problem with IGM. The last INFO is > printed by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints in > it to identify the spot where the error occurs. > > Unfortunately, I drew a blank, since the stop seems to occur just after > the end of IllinoisGRMHD_driver_evaluate_MHD_rhs: > > > > INFO (IllinoisGRMHD): ***** line 52: entering > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to > time: 3.160251e+03 ***** > > INFO (IllinoisGRMHD): ***** line 100: > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > INFO (IllinoisGRMHD): ***** line 204: just before > reconstruct_set_of_prims_PPM ***** > > INFO (IllinoisGRMHD): ***** DEBUG END of > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST > > > > > > I tried to restart the simulation and run it on pure MPI. It ran for a > few more iterations, then stopped as well: > > > > INFO (IllinoisGRMHD): ***** line 52: entering > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to > time: 3.156831e+03 ***** > > INFO (IllinoisGRMHD): ***** line 100: > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > INFO (IllinoisGRMHD): ***** line 204: just before > reconstruct_set_of_prims_PPM ***** > > INFO (IllinoisGRMHD): ***** DEBUG END of > IllinoisGRMHD_driver_evaluate_MHD_rhs ***** > > Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST > > > > > > The simulation setup is as follows: > > > > Allocated: > > Nodes: 10 > > Cores per node: 48 > > SLURM setting > > SLURM_NNODES : 10 > > SLURM_NPROCS : 20 > > SLURM_NTASKS : 20 > > SLURM_CPUS_ON_NODE : 48 > > SLURM_CPUS_PER_TASK : 24 > > SLURM_TASKS_PER_NODE: 2(x10) > > Running: > > MPI processes: 20 > > OpenMP threads per process: 24 > > MPI processes per node: 2.0 > > OpenMP threads per core: 1.0 > > OpenMP threads per node: 48 > > > > > > while the pure-MPI setup is > > > > Allocated: > > Nodes: 10 > > Cores per node: 48 > > SLURM setting > > SLURM_NNODES : 10 > > SLURM_NPROCS : 480 > > SLURM_NTASKS : 480 > > SLURM_CPUS_ON_NODE : 48 > > SLURM_CPUS_PER_TASK : 1 > > SLURM_TASKS_PER_NODE: 48(x10) > > Running: > > MPI processes: 480 > > OpenMP threads per process: 1 > > MPI processes per node: 48.0 > > OpenMP threads per core: 1.0 > > OpenMP threads per node: 48 > > > > > > I am using The Lorentz version of ET. > > > > I've had this issue for two binary BH simulations, both unequal-mass > with q = 2. My colleague Giacomo Fedrigo experienced the same problem > running an equal-mass simulation. > > > > I attach the q = 2 (s_UUmis_Q2) parameter file and the ET config-info > file. Also, I attach the st. error and output of my q = 2 run and of > Giacomo's run (b1_UUmis_a12b_pol3_r56_gauss_9). The st. outputs were cut > for readability reasons. > > > > Can someone please help me with this? > > > > Thanks in advance, > > > > Federico > > _______________________________________________ > > Users mailing list > > Users(a)einsteintoolkit.org > > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > -- > Erik Schnetter <schnetter(a)gmail.com> > http://www.perimeterinstitute.ca/personal/eschnetter/ >

4 4

Meeting Minutes 2022-09-22
by Bruno Giacomazzo 22 Sep '22

22 Sep '22

Present: Leo Werneck (chair), Bruno Giacomazzo (minute taker), Steven Brandt, Beyhan, Peter Diener, Gabriele Bozzola, Keith, Roland Haas, Samuel Cupp, Zachariah Etienne *1) vote on modules suggested for inclusion for the next release* *1a) Canuda scalar and complex fields* -- proposed by Miguel Zilhao Zach asks if it was reviewed since he didn't hear anything recently. Roland said that the review finished. No one opposed the inclusion. *1b) FLRW solver* -- proposed by Hayley Macpherson No one opposed the inclusion. *1c) SelfForce-1D* -- proposed by Samuel Cupp Peter mentioned that there are new functionalities that have been added. Roland asked if there are new examples, but Peter said no. Steve asked for examples with movies. No one opposed the inclusion. *1d) NRPyEllipticET* -- proposed by Leo Werneck, Thiago Assumpcao, Zach Etienne Roland mentioned that there are no test cases included with it. No one opposed the inclusion with the condition that at least a test case is added. Leo mentioned that some documentation of WVUThorns does not show correctly online and Roland said that html latex is stricter than pdflatex when checking latex syntax. *2) ET release* Roland said that is coming up in November, all thorns compile, and all tests pass. Test results are available on http://einsteintoolkit.org/testsuite_results/index.php Steve reminded people that the subversion servers are going to be switched off. There seems to be only one still used and maybe it is time to move to another versioning system any repository still on svn. Gabriele said that kuibit will soon require python 3. Zach mentioned that NRPy will continue to support python 2 since python 2 is still used on many HPC systems. Steve suggested checking which is the default version of python available in the clusters we use. *4) unanswered question on mailing list* Regarding configuration of workstation (Indrani Banerjee): no volunteer found to answer this question yet. Not clear whether it would be possible to compute waveforms on a workstation. *5) open tickets sorted by update time* The following tickets were discussed: #2644: NaNs in ADMBASE metric components with RNSID -> no volunteer found #2642: Toolkit Fails to Compile on Sawtooth (Idaho National Lab) -> it seems that C++ 14 standard is needed and this should solve the issue. #2643: Adding read/write declarations to Meudon ID thorns -> it is a pull request (please have a look) #2641: ML_BSSN macro usage bypasses personalized read/write macros *6) tickets ready for review* #2630: WaveMoL example computes energy from unsynchronized grid variables #181: AEILocalInterp should not off-centre the interpolation stencil by default #2364: avoid opening and closing HDF5 files multiple times during output #963: Improve McLachlan accuracy No comment on any of these tickets. *7) Other comments* Zach said that the update to Baikal will be a regular update (as suggested by Roland) and he will open a pull request. Next Chair: Steve Next Minute Tacker: Roland

1 0

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 21 Sep '22

21 Sep '22

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 19 Sep '22

19 Sep '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Meeting Minutes 2022-9-15
by Cupp, Samuel D. 15 Sep '22

15 Sep '22

Present: Sam, Steve, Keith, Gabriele, Peter, Leo, Zach, Bruno New ET modules: * no news on Canuda review * Haley has addressed reviewers' comments for FLRW solver * Sam has addressed reviewers' comments for selfforce-1d; only remaining task is to regenerate the FORD documentation * Leo has addressed reviewers' comments for NRPyEllipticET; notebook which generates the code is not yet ready, but this is currently being worked on * Baikal update: Zach hasn't heard back from Helvi. Since the new Baikal is waiting for inclusion, the old Baikal's notebook isn't being maintained in favor of the new Baikal Other questions: * For the particle tracer Leo is working on, he needs to access the cctk_timefac for the finest level, but cctk_timefac only gives the value for the current time level. He does this by accessing the Carpet parameter time_refinement_factors using CCTK_ParameterGet() but wanted to know if there was a better/more 'proper' way of accessing the parameters. The function is necessary because the parameter is private and cannot be accessed via the interface. Steve said that this is how he would do it, but Leo should send the question to the mailing list to see if Roland or Erik have other suggestions. * We would like to shut down the svn servers which aren't being used anymore and have been maintained for preservation of old thornlists for a long time. By shutting them down, we can see if anyone complains about it and also track down anywhere else on the website that might reference them. * There is no established policy on deprecating or renaming thorns/parameters. Zach proposed that we should set a clear policy for the timescale on which these things can be done, and the procedure for doing so. * Someone asked about whether there will be recordings of the European ET meeting, but Peter said there were not. However, they do intend to upload the slides at some point. * Next European meeting was set and will be in Aveiro, Portugal. Questions on the mailing list: No one answered 'Running with SLURM' from last week; Steve will respond to it. http://lists.einsteintoolkit.org/pipermail/users/2022-August/008667.html Open tickets: Gabriele brought up ticket issue #2629 https://bitbucket.org/einsteintoolkit/tickets/issues/2629/mol_pseudoevoluti… about when one should schedule in CCTK_ANALYSIS vs MoL_PseudoEvolution. The particular concern is how this will interact with parameters such as the calculate_constraints_every in LeanBSSNMoL, as some choices of output_every type parameters will be invalid depending on the number of active refinement levels. However, since these calculations are expensive, it is important to be able to control the calculation to only occur when it will be output. Zach said that one solution is to just leave it up to the user to decide if it is sane to update these quantities every n timesteps. This is similar to the issue Leo has been facing with the particle tracer, as it must compute and output only so many iterations. However, Cactus counts iterations based on the smallest possible refinement level, even if it isn't active. This leads to the need to determine the dtfac for the finest possible level, which is the topic of the question raised earlier. He then errors out if the parameters will produce incorrect output. Since several thorns need this feature, discussion arose on whether this should be provided as an aliased function by Carpet to easily provide access to the needed information to validate the given values of these types of output_every parameters. Carpet could set some out_every_safe that provides the minimal safest out_every. The chair for next week (9/22) is Leo, and the minutes taker is Bruno. Samuel Cupp Postdoctoral Researcher Department of Physics University of Idaho

2 3

Issue with black hole binary evolution employing IGM
by Federico Cattorini 15 Sep '22

15 Sep '22

Hello everyone, I am experiencing some issue in a number of GRMHD simulations of black hole binaries employing IllinoisGRMHD. As an example, I will write about an unequal-mass BHB configuration (with q = 2) that I'm running. After approximately ten orbits, the run stops with no error codes or any other message that could help me identify the issue. The last lines of the standard output are INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160260e+03 ***** INFO (IllinoisGRMHD): C2P: Lev: 9 NumPts= 569160 | Fixes: Font= 393 VL= 179 rho*= 2 | Failures: 0 InHoriz= 0 / 0 | Error: 7.124e-02, ErrDenom: 4.838e+13 | 4.51 iters/gridpt INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 9, Integrating to time: 3.160269e+03 ***** Simfactory Done at date: gio 04 ago 2022 11:43:01 CEST I tried restarting my simulation from the latest checkpoint, but the same sudden stop occurred at the same timestep. At first, I thought about some problem with IGM. The last INFO is printed by IllinoisGRMHD_driver_evaluate_MHD_rhs.C, so I put some prints in it to identify the spot where the error occurs. Unfortunately, I drew a blank, since the stop seems to occur just after the end of IllinoisGRMHD_driver_evaluate_MHD_rhs: INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** Iter. # 353949, Lev: 10, Integrating to time: 3.160251e+03 ***** INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM ***** INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs ***** Simfactory Done at date: gio 04 ago 2022 19:44:55 CEST I tried to restart the simulation and run it on pure MPI. It ran for a few more iterations, then stopped as well: INFO (IllinoisGRMHD): ***** line 52: entering IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** Iter. # 353565, Lev: 10, Integrating to time: 3.156831e+03 ***** INFO (IllinoisGRMHD): ***** line 100: IllinoisGRMHD_driver_evaluate_MHD_rhs ***** INFO (IllinoisGRMHD): ***** line 204: just before reconstruct_set_of_prims_PPM ***** INFO (IllinoisGRMHD): ***** DEBUG END of IllinoisGRMHD_driver_evaluate_MHD_rhs ***** Simfactory Done at date: ven 05 ago 2022 19:00:13 CEST The simulation setup is as follows: Allocated: Nodes: 10 Cores per node: 48 SLURM setting SLURM_NNODES : 10 SLURM_NPROCS : 20 SLURM_NTASKS : 20 SLURM_CPUS_ON_NODE : 48 SLURM_CPUS_PER_TASK : 24 SLURM_TASKS_PER_NODE: 2(x10) Running: MPI processes: 20 OpenMP threads per process: 24 MPI processes per node: 2.0 OpenMP threads per core: 1.0 OpenMP threads per node: 48 while the pure-MPI setup is Allocated: Nodes: 10 Cores per node: 48 SLURM setting SLURM_NNODES : 10 SLURM_NPROCS : 480 SLURM_NTASKS : 480 SLURM_CPUS_ON_NODE : 48 SLURM_CPUS_PER_TASK : 1 SLURM_TASKS_PER_NODE: 48(x10) Running: MPI processes: 480 OpenMP threads per process: 1 MPI processes per node: 48.0 OpenMP threads per core: 1.0 OpenMP threads per node: 48 I am using The Lorentz version of ET. I've had this issue for two binary BH simulations, both unequal-mass with q = 2. My colleague Giacomo Fedrigo experienced the same problem running an equal-mass simulation. I attach the q = 2 (s_UUmis_Q2) parameter file and the ET config-info file. Also, I attach the st. error and output of my q = 2 run and of Giacomo's run (b1_UUmis_a12b_pol3_r56_gauss_9). The st. outputs were cut for readability reasons. Can someone please help me with this? Thanks in advance, Federico

2 1

Einstein Toolkit Meeting Reminder
by reminders＠einsteintoolkit.org 14 Sep '22

14 Sep '22

1 0

Agenda for Thursday's Meeting
by reminders＠einsteintoolkit.org 12 Sep '22

12 Sep '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users