Users April 2022

users@lists.einsteintoolkit.org

16 participants
25 discussions

Problem regriding using trigger thorn
by Dhruv Desai 10 May '22

10 May '22

Hello, I am running a simulation with Daniel Siegel where we use a base level plus 6 refinement levels, like the following: CarpetRegrid2::radius_1[1] = 336.0 # dx/64 CarpetRegrid2::radius_1[2] = 168.0 # dx/64 CarpetRegrid2::radius_1[3] = 84.0 # dx/64 CarpetRegrid2::radius_1[4] = 42.0 # dx/64 CarpetRegrid2::radius_1[5] = 21.0 # dx/64 CarpetRegrid2::radius_1[6] = 10.5 # dx/64 We ran simulations of a neutrino-driven wind from a Neutron star, out to ~150 ms, and now we want to turn on a high magnetic field. We have tried B-fields of varying strengths, but if we try a B-field strength that is too high, we find errors generated on the innermost refinement level boundary. We believe if we enlarge the innermost box size ie. CarpetRegrid2::radius_1[6] = 10.5 --> ~19-20, these errors would disappear. But we are having trouble doing this. In our attempt, we activate the trigger thorn with the following settings: ActiveThorns = "Trigger" Trigger::Trigger_Debug = 1 Trigger::Trigger_Once[0]=1 Trigger::Trigger_Number =1 Trigger::Trigger_Checked_Variable[0]="GRHydro::GRHydro_rho_min" Trigger::Trigger_Reduction [0]="" Trigger::Trigger_Relation [0]="<" Trigger::Trigger_Checked_Value [0]= 1.1 Trigger::Trigger_Reaction [0]="steerparam" Trigger::Trigger_Steered_Parameter_Value [0] = "19.32" Trigger::Trigger_Steered_Parameter_Name [0] = "radius_1[6]" Trigger::Trigger_Steered_Parameter_Thorn [0] = "CarpetRegrid2" (We use a trivially satisfied trigger condition so that the parameter value is reset immediately upon restart.) Since the CarpetRegrid2 variable "radius_1[6]" is not steerable, we had to modify this to make it such. After turning debug on, we see the following output: INFO (Trigger): Testing triggers INFO (Trigger): last_checked: -1 INFO (Trigger): trigger nr. 0 fulfilled for GRHydro_rho_min (0.000000<1.100000) INFO (Trigger): Steering parameter INFO (Trigger): Parameter steered Which supposedly successfully resets the value correctly. We believe that even though this may be the case, the actual grid is not changing. We plot the first 2 boundary levels and show them in the attached file. There must be interpolation of the second innermost grid onto the innermost grid, and we believe this is not actually handled mid-simulation and thus doesn't not result in the change we expect. Is there any simple way of changing the inner grid easily? Any help is much appreciated! [image: image(1).png] Thanks, Dhruv Desai

3 6

Re: [Users] Segmentation Fault with CarpetReduce
by Bruno Giacomazzo 03 May '22

03 May '22

DG, it could also be that there is a bug in the code, but other compilers are not affected by it. My strategy is usually to make sure that I can reproduce the problem on the cluster when running just with one core. In this way it is easier to run within a debugger and spot the problem. Best regards, Bruno Il giorno ven 29 apr 2022 alle ore 08:12 Garrison, David <Garrison(a)uhcl.edu> ha scritto: > It seems to be an error on a cluster I use but not on my Mac. It may be > an issue with how the cluster is configured. > > -DG > > On Apr 28, 2022, at 9:48 AM, Bruno Giacomazzo <bruno.giacomazzo(a)unimib.it> > wrote: > > DG, > thanks for the parameter file. > > During today's call we also saw that you mentioned that you encountered a > segmentation fault when using one ghost point. Do you have a backtrace? Did > you try running the code in a debugger to check which variable is causing > it? > > Cheers, > Bruno > > Il giorno gio 21 apr 2022 alle ore 17:32 Garrison, David < > Garrison(a)uhcl.edu> ha scritto: > >> I don’t use standard Einstein Toolkit thorns. I also don’t use >> refinement levels in Carpet. Please see attached for an example. The >> error occurs when I change the ghost zone size to 1. >> >> -DG >> >> -- >> David Garrison, Ph.D. >> Associate Dean for the College of Science and Engineering, >> Professor and Founding Chair of Physics >> University of Houston-Clear Lake >> Bayou 3611/STEM 2252 >> Houston, TX 77058 >> >> BAYOU: 281-283-3717 >> STEM : 281-283-3796 >> https://www.uhcl.edu/science-engineering/faculty/garrison-david >> http://www.uhcl.edu/physics >> >> "If we knew what it was we were doing, it would not be called research, >> would it?" — Albert Einstein. >> >> >> >> On Apr 21, 2022, at 9:24 AM, Bruno Giacomazzo <bruno.giacomazzo(a)unimib.it> >> wrote: >> >> DG, >> if you could share the parameter file that would be useful in order to >> try reproducing the error with standard Einstein Toolkit thorns. I do not >> see any problem with the lines of code you posted in the email. >> >> Do you get the same problem also when using Carpet without refinement >> levels? >> >> Best regards, >> Bruno >> >> Il giorno gio 31 mar 2022 alle ore 19:17 Garrison, David < >> Garrison(a)uhcl.edu> ha scritto: >> >> Hello, >> >> It is a custom code so I am attaching the part I used to do the >> reductions instead. >> >> handle = CCTK_ReductionHandle("average"); >> ierr = CCTK_Reduce(cctkGH, -1, handle, 1, CCTK_VARIABLE_REAL, >> &scale_factor, 1, CCTK_VarIndex("MHD_Analysis::aa_out_avg")); >> ierr = CCTK_Reduce(cctkGH, -1, handle, 1, CCTK_VARIABLE_REAL, &utime, 1, >> CCTK_VarIndex("MHD_Analysis::univ_age")); >> >> >> -DG >> >> >> On Mar 31, 2022, at 9:21 AM, Bruno Giacomazzo <bruno.giacomazzo(a)unimib.it> >> wrote: >> >> DG, >> could you share the parameter file you are using? So that we may have >> a better look at the problem. >> >> Thanks, >> Bruno >> >> >> Il giorno mer 30 mar 2022 alle ore 20:52 Garrison, David < >> Garrison(a)uhcl.edu> ha scritto: >> >> Hello, >> >> Whenever I set the number of ghost zones to 1, I get segmentation faults >> from my reduction operations. This wasn’t an issue with Pugh or when I use >> 2 or more ghost zones with Carpet. Has anyone else had this problem? >> >> -DG >> >> -- >> David Garrison, Ph.D. >> Interim Associate Dean for the College of Science and Engineering, >> Professor and Founding Chair of Physics >> University of Houston-Clear Lake >> Bayou 3611/STEM 2252 >> Houston, TX 77058 >> >> BAYOU: 281-283-3717 >> STEM : 281-283-3796 >> https://www.uhcl.edu/science-engineering/faculty/garrison-david >> http://www.uhcl.edu/physics >> >> "If we knew what it was we were doing, it would not be called research, >> would it?" — Albert Einstein. >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> <https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listin…> >> >> >> >> -- >> >> Prof. Bruno Giacomazzo >> Department of Physics >> University of Milano-Bicocca >> Piazza della Scienza 3 >> 20126 Milano >> Italy >> >> email: bruno.giacomazzo(a)unimib.it >> phone: (+39) 02 6448 2321 >> web: http://www.brunogiacomazzo.org >> <https://urldefense.com/v3/__http://www.brunogiacomazzo.org/__;!!BCR0FSePrR4…> >> >> ---------------------------------------------------------------------- >> There are only 10 types of people in the world: >> Those who understand binary, and those who don't >> ---------------------------------------------------------------------- >> >> >> -- >> David Garrison, Ph.D. >> Interim Associate Dean for the College of Science and Engineering, >> Professor and Former Chair of Physics >> University of Houston-Clear Lake >> Bayou 3611 >> Houston, TX 77058 >> >> Tel: 281-283-3796 >> Fax: 281-283-3709 >> http://sce.uhcl.edu/garrison >> http://www.uhcl.edu/physics >> >> "If we knew what it was we were doing, it would not be called research, >> would it?" ‹ Albert Einstein. >> >> >> >> >> >> >> -- >> >> Prof. Bruno Giacomazzo >> Department of Physics >> University of Milano-Bicocca >> Piazza della Scienza 3 >> 20126 Milano >> Italy >> >> email: bruno.giacomazzo(a)unimib.it >> phone: (+39) 02 6448 2321 >> web: http://www.brunogiacomazzo.org >> <https://urldefense.com/v3/__http://www.brunogiacomazzo.org/__;!!BCR0FSePrR4…> >> >> ---------------------------------------------------------------------- >> There are only 10 types of people in the world: >> Those who understand binary, and those who don't >> ---------------------------------------------------------------------- >> >> >> > > -- > > Prof. Bruno Giacomazzo > Department of Physics > University of Milano-Bicocca > Piazza della Scienza 3 > 20126 Milano > Italy > > email: bruno.giacomazzo(a)unimib.it > phone: (+39) 02 6448 2321 > web: http://www.brunogiacomazzo.org > <https://urldefense.com/v3/__http://www.brunogiacomazzo.org/__;!!BCR0FSePrR4…> > > ---------------------------------------------------------------------- > There are only 10 types of people in the world: > Those who understand binary, and those who don't > ---------------------------------------------------------------------- > > > -- Prof. Bruno Giacomazzo Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy email: bruno.giacomazzo(a)unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

2 2

CarpetRegrid2
by Beyhan Karakaş 30 Apr '22

30 Apr '22

Hello Wayne, I suppose the error resulted from not having enough refinement levels on region 3. If you get this error while Triggering the 7th refinement level of region 3, then you need to have CarpetRegrid2::radius_3[8] for Trigger thorn (copied from BNS gallery example parfile > Trigger::Trigger_Steered_Scalar ][1] = > "CarpetRegrid2::num_levels[2]" # == num_levels_3 > Trigger::Trigger_Steered_Scalar_Value [1] = "8" > ) and if you are already doing this then you did not set 7th refinement level CarpetRegrid2::radius_3[7] If you would get this error at the beginning of the simulation, you would need to make sure that you have enough refinement levels at the beginning > CarpetRegrid2::num_levels_3 = 8 (If you set to 8 you can only have 7 (CarpetRegrid2::radius_2[7]) > > If this is wrong. someone could correct it. Best regards, Beyhan. > > Message: 1 > Date: Fri, 29 Apr 2022 04:43:58 +0000 > From: "Barkhouse, Wayne" <wayne.barkhouse(a)und.edu> > Subject: [Users] CarpetRegrid2 > To: "users(a)einsteintoolkit.org" <users(a)einsteintoolkit.org> > Message-ID: > < > BY3PR08MB70754D5C51A30EA94CBFE8E6EBFC9(a)BY3PR08MB7075.namprd08.prod.outlook.com > > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > When running a BNS simulation, we occasionally run into the following > error message: > > ERROR from host node1 process 0 > in thorn CarpetRegrid2, file > /home/wbark/Software/EinsteinToolkit/Cactus/configs/sim/build/CarpetRegrid2/regrid.cc:80: > -> The radius of refinement level 7 of region 3 is [-1,-1,-1], which is > not non-negative > > Is there a general fix for this that we are missing? > > Thank you!! > > cheers, > Wayne > > *************************************** > Dr. Wayne Barkhouse > Professor > Department of Physics and Astrophysics > University of North Dakota > Grand Forks, ND 58202-7129 > > Email: wayne.barkhouse(a)und.edu > Phone: 701-777-3520 > FaX: 701-777-3523 > *************************************** >

2 1

CarpetRegrid2
by Barkhouse, Wayne 29 Apr '22

29 Apr '22

Hi, When running a BNS simulation, we occasionally run into the following error message: ERROR from host node1 process 0 in thorn CarpetRegrid2, file /home/wbark/Software/EinsteinToolkit/Cactus/configs/sim/build/CarpetRegrid2/regrid.cc:80: -> The radius of refinement level 7 of region 3 is [-1,-1,-1], which is not non-negative Is there a general fix for this that we are missing? Thank you!! cheers, Wayne *************************************** Dr. Wayne Barkhouse Professor Department of Physics and Astrophysics University of North Dakota Grand Forks, ND 58202-7129 Email: wayne.barkhouse(a)und.edu Phone: 701-777-3520 FaX: 701-777-3523 ***************************************

2 1

reduction operation in CarpetIOScalar
by 白济民 28 Apr '22

28 Apr '22

Hi, I'm concerning about how outscalar_reductions (e.g. one of its default, sum) in the thorn CarpetIOScalar counting values on carpet grid with multiple refinement levels. The output section in the gallery parfile nsnstohmns.par has one line reading IOScalar::outScalar_vars="...(e.g. HydroBase::rho)", so does it count values on a specified refinement level and is the counting different among different outscalar_vars? I also wonder what does "icount iaverage inorm" mean in outscalar_reductions' default. Thanks in advance for your help! Yours sincerely, Jimmy

2 2

Meeting minutes 2022-04-28
by Bruno Giacomazzo 28 Apr '22

28 Apr '22

Present: Steven Brandt, Peter Diener, Roland Haas, Miguel. G. Linares, Beyhan, Bruno Giacomazzo, Leo R. Werneck, Erik Schnetter, Yosef E. Zlochower, Zach Etienne. Chair: Peter D. Minutes: Bruno G. 1) Next ET release Yosef: no particular update. In a few days we need to see the gallery example complete and test the docker images. We are on schedule anyway. Steve: I will take care of the docker images. And will take care of finding a volunteer for the BBH example. Pieter: running testsuites on a machine in Stockholm on which I have access. Steve: current version of testsuites will run also the testsuites requiring one core (and not only the ones requiring two cores) if one selects two cores. Roland: no change in the testsuite status. Some changes to SuperMUC. And some problems on Expanse. Bruno: for the next release I will provide simfactory files for Marconi A3 and Galileo 100. 2) ET summer school at Idaho Zach: sent a final reminder for the registration deadline. Those that did not ask for financial support should pay the conference fee by the end of this week. We will have a final response for the requests of financial supports by the middle of next week. Next week we will have a meeting to discuss the program and talks. Got also a request to discuss about a standard data format to be used to share results (e.g., GW data). More than 100 people registered, about 30 will be in person. 3) SPEC benchmark contribution Steve: no update Erik: I or Steve could give a tutorial on how to use SPEC Steve: We will have this tutorial in these meeting slots and announce it in the mailing list. Peter: It may be on May 5th. Steve: I cannot on 5th Erik: Let's move to the following week (May 12) Peter: OK, let's plan for May 12 4) CarpetX videos/demos Steve: It would be nice to have some CarpetX videos/demos for a booth we are organizing for a HPC conference. Demos running on a single PC would be nice, so that it can run in local without requiring internet connection. 5) Unanswered questions on mailing list - email about "reduction operation in CarpetIOScalar" Erik will answer to this. - email about "Segmentation Fault with CarpetReduce" Bruno will ask about backtrace file or running it in a debugger to check where the segmentation fault is coming from. 6) Open tickets #963 Peter will have a look at this #2602 Peter should be done with the review within a week. It could go into the November release. #1568 Zach is reporting that Formaline seems to have problems on some systems. A bug reported specific to it should be submitted (Bernard Kelly reported some problems). #2256 Peter points out that this ticket is from 2019 (it is about moving the wiki to bitbucket). Roland created the ticket and it seems to be assigned to Steve, but he was not involved in this. Roland thinks we should do something about that. #2206 Also this ticket is quite old (set LORENE 2 as default). Bruno points out that we may want to include the Kadath thorns inside the Einstein Toolkit (https://bitbucket.org/fukaws/). 7) Open Issues #2364 Roland is going to take care of this #332 Roland points out that this is very old 8) Other nothing 9) Next calls May 5 Chair: Beyhan Minutes: Leo R. Werneck May 12 Chair: Erik Minutes: ??? May 19 Chair: ??? Minutes: Miguel G. Linares

1 0

Segmentation Fault with CarpetReduce
by Garrison, David 28 Apr '22

28 Apr '22

Hello, Whenever I set the number of ghost zones to 1, I get segmentation faults from my reduction operations. This wasn’t an issue with Pugh or when I use 2 or more ghost zones with Carpet. Has anyone else had this problem? -DG -- David Garrison, Ph.D. Interim Associate Dean for the College of Science and Engineering, Professor and Founding Chair of Physics University of Houston-Clear Lake Bayou 3611/STEM 2252 Houston, TX 77058 BAYOU: 281-283-3717 STEM : 281-283-3796 https://www.uhcl.edu/science-engineering/faculty/garrison-david http://www.uhcl.edu/physics "If we knew what it was we were doing, it would not be called research, would it?" — Albert Einstein.

2 5

Einstein Toolkit Meeting Reminder
by users＠einsteintoolkit.org 27 Apr '22

27 Apr '22

Hello, Please consider joining the weekly Einstein Toolkit phone call at 9:00 am US central time on Thursdays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --The Maintainers

1 0

Final Reminder, Registration Deadline May 1 for 2022 NA ET School
by Leonardo Rosa Werneck 27 Apr '22

27 Apr '22

Dear colleagues, This is the final reminder that the deadline to register for the 2022 North American Einstein Toolkit School is this Sunday, May 1, 2022. The school—held at the University of Idaho June 13-17, 2022—will provide an opportunity for researchers and students to gather and learn about the Einstein Toolkit. To register for the event, please visit: https://einsteintoolkit.github.io/et2022uidaho/register.html <https://einsteintoolkit.github.io/et2022uidaho/register.html> This will be a fully hybrid meeting with both in-person and remote participation welcome. Please select the in-person option at registration in case you are undecided about whether you will be able to attend in person. A nominal registration fee will be required for participants who attend in person. A limited amount of travel support is available for participants who wish to attend in person, including waiver of registration fee. Please indicate during registration if you would like to be considered for this support. This year, a special field trip will be offered for in-person participants to nearby LIGO Hanford, whose direct observation of the gravitational waves from a coalescing pair of black holes led to the 2017 Nobel Prize in Physics. During the visit, LIGO Hanford will be gearing up for the start of O4 (Observing Run 4), due to start at the end of this year. On the same day, virtual participants will have the opportunity to attend an Einstein Toolkit hackathon. For more information, please visit our website: https://einsteintoolkit.github.io/et2022uidaho/ <https://einsteintoolkit.github.io/et2022uidaho/> If you have any questions, please contact the organizers at workshop(a)einsteintoolkit.org <mailto:workshop@einsteintoolkit.org>. Sincerely, The Scientific Organizing Committee 2022 North American Einstein Toolkit Summer School Gabrielle Allen Steven Brandt Zachariah Etienne Roland Haas Christiana Pantelidou Leonardo Werneck Helvi Witek ------ Leonardo Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo(a)uidaho.edu <mailto:leonardo@uidaho.edu> https://leowerneck.github.io <https://leowerneck.github.io/>

1 0

Agenda for Thursday's Meeting
by users＠einsteintoolkit.org 25 Apr '22

25 Apr '22

Please update the Wiki with agenda items for Thursday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --The Maintainers

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users April 2022