Users October 2018

users@lists.einsteintoolkit.org

18 participants
31 discussions

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 31 Oct '19

31 Oct '19

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --Steve

8 94

Agenda for Monday's Meeting
by Steven R. Brandt 30 Oct '19

30 Oct '19

Please update the Wiki with agenda items for Monday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --Steve

3 46

New Einstein Toolkit Release (Chien-Shiung Wu)
by Steven R. Brandt 30 Oct '19

30 Oct '19

=== Release Announcement === We are pleased to announce the sixteenth release (code name "Tesla") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are Five new thorns have been added: * GiRaFFE * GiRaFFEfood * ShiftedKerrSchild * GiRaFFE_to_HydroBase * ID_converter_GiRaFFE In addition, bug fixes accumulated since the previous release in Feb 2018 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics code GRHydro. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 200 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: == Larger changes since last release == * Support has been added for GCC8. * Built-in OpenMPI tarball has been upgraded to 1.10.7 Note, we still recommend using the system MPI that is installed natively on your system. * And many bug fixes. Notably, a significant correctness problem for NewRad was addressed which caused many simulations with more than two ranks to fail https://trac.einsteintoolkit.org/ticket/2182 == New thorns or tools == * GiRaFFE (~1,600 lines of code): Solves the GRFFE equations in the formalism of Paschalidis and Shapiro [2] using the same reconstruction (PPM) and staggered vector potential methods as IllinoisGRMHD. * GiRaFFEfood (~1,300 lines of code): Code validation initial data and basic diagnostics. Contains Five 1D tests in flat spacetime (fast wave, Alfvén wave, degenerate Alfvén wave, "three-wave", FFE breakdown), One 3D test in flat spacetime (aligned rotator), and Three 3D tests in curved spacetimes ("Exact Wald" solution, "Magnetospheric Wald" solution, and split monopole). * ShiftedKerrSchild (~230 lines of code): Sets up all 3+1 quantities for a Kerr-Schild spacetime with an arbitrary radial shift, to mitigate effects of extreme spacetime curvature near r_{KS}=0. See Appendix of code release paper for full discussion. * GiRaFFE_to_HydroBase and ID_converter_GiRaFFE (~230 lines of code): Acts as a translation layer to convert 3-velocities in HydroBase's Valencia formalism to/from the 3-velocity found in the induction equation (vi = ui/u0; native to GiRaFFE). == Upcoming changes for the next releases == == New thorns planned for the next relases == * Seed_Magnetic_Fields-modified Extended Seed_Magnetic_Fields thorn for binary neutron stars. Supercedes Seed_Magnetic_Fields thorn. * Meudon_Bin_NS-modified Modifications to Meudon BNS initial data thorn to disable the overwriting of initial lapse/shift, which acts to significantly reduce coordinate eccentricity. Supercedes Meudon_Bin_NS thorn. * VolumeIntegrals_GRMHD Nice GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. * VolumeIntegrals_vacuum Nice GRMHD volume integration thorn, currently depends on ML_BSSN. There is a bit of code duplication and duplicated functionality between VI_GRMHD and VI_vacuum, to ensure that VI_vacuum can be used without enabling a GRMHD code. There is probably a better way of doing this, but I haven't had the time to think deeply about this. * particle_tracerET Solves the ODE \partial_t xi = vi for typically thousands of tracer particles, using an RK4 integration atop the current timestepping. E.g., one RK4 substep in the particle integration might occur every 16 RK4 substeps in the GRMHD evolution. These tracer particle positions are quite useful for visualizing magnetic field lines in a consistent way from frame-to-frame in a movie (recall that in the GRMHD approximation, the magnetic field lines stay attached the fluid elements they thread Flux Freezing?). Note that the velocity must be consistent with the velocity appearing in the GRMHD induction equation. This thorn reads in the HydroBase vel[] vector gridfunction, which assumes the Valencia formalism, and converts it into the induction equation velocity. * smallbPoynET Computes bi, b2, and three spatial components of Poynting flux. It also computes (-1-u0), which is useful for tracking unbound matter. * LeanBSSNMoL and NPScalars: LeanBSSNMoL evolves Einstein's Equations with the BSSN formulation. NPScalars computes the Newman-Penrose scalars. For the conformal factor it uses the "W" version, i.e. W=\gamma^{-1/6}, and it employs the usual "1+log" and "Gamma-driver" gauge conditions. Also available, in the "new_gauge" branch, is the shift condition of Figueras et al for their evolutions of higher dimensional BHs, see: https://arxiv.org/abs/1702.01755 https://arxiv.org/abs/1512.04532 * Proca arrangement: This repository (https://bitbucket.org/canuda/proca/src) provides the tools to evolve the Einstein-Proca system as first described in https://arxiv.org/abs/1505.00797 == How to upgrade from Tesla (ET_2018_02) == To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. == Machine notes == Supported (tested) machines include: * Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (Homebrew and MacPorts) installations * Bluewaters * Comet * Cori * Draco * Edison * Golub * Hydra * Marconi * Minerva * Queenbee 2 * Stampede 2 * SuperMIC * Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * A new configuration for KNL nodes is being worked on, but not yet included in the release (but compilation works and tests mostly pass). All repositories participating in this release carry a branch ET_2018_02 marking this release. These release branches will be updated if severe errors are found. The "Chien-Shiung Wu" Release Team on behalf of the Einstein Toolkit Consortium (2018-09-17) * Steven R. Brandt * Samuel D. Cupp * Peter Diener * Roland Haas * Roberto De Pietri Sep, 2018

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #1326: running loopcontrol on strange number of threads fails
by Erik Schnetter 11 Apr '19

11 Apr '19

Do you have a backtrace? -erik On Sunday, March 22, 2015, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #1326: running loopcontrol on strange number of threads fails > ------------------------------------+--------------------------------------- > Reporter: rhaas | Owner: > Type: defect | Status: new > Priority: minor | Milestone: > Component: EinsteinToolkit thorn | Version: > Resolution: | Keywords: LoopControl > ------------------------------------+--------------------------------------- > > Comment (by rhaas): > > This still happens even with current (Sun Mar 22 18:46:21 CET 2015) trunk, > though failure looks a bit different now: > {{{ > actus_sim: > /data/rhaas/postdoc/gr/ET_trunk/configs/sim/build/LoopControl/loopcontrol.cc:312: > T {anonymous}::divexact(T, T) [with T = int]: Assertion `i % j == 0' > failed > }}} > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1326#comment:1> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Compiling ET Using PGI
by Christopher Ramirez 29 Nov '18

29 Nov '18

Hello, I've been getting the following errors when trying to compile using PGI compilers and am wondering if were any additional steps needed to compile successfully. COMPILING arrangements/CactusBase/Boundary/src/ScalarBoundary.c PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 41) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 41) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 42) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 42) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 45) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 45) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 71) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 71) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 29) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 29) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 30) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 30) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 60) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 60) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 61) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 61) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 67) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 67) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 68) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 68) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Misc.h: 43) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Misc.h: 43) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 39) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 39) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 40) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 40) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 76) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 76) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 77) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 77) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 216) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 216) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 228) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 228) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 301) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 301) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 321) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 321) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 381) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 381) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 393) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 393) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 467) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 467) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 487) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 487) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 40) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 40) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 42) PGC-F-0008-Error limit exceeded (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 42) PGC/x86-64 Linux 17.10-0: compilation aborted make[3]: *** [ScalarBoundary.c.o] Error 2 make[2]: *** [make.checked] Error 2

4 8

Hydro_RNSID Inherits from GRHydro
by Zach Etienne 05 Nov '18

05 Nov '18

Hello, Hydro_RNSID inherits from GRHydro, which goes against the usual policy that initial data thorns depend only on HydroBase / ADMBase. Is there a plan to fix this? Is a fix even difficult? -Zach * * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University *https://math.wvu.edu/~zetienne/ <https://math.wvu.edu/~zetienne/>* https://blackholesathome.net

3 2

Problem with reproducing the NsNsToHMNS
by hassan.khalvati＠physics.sharif.edu 29 Oct '18

29 Oct '18

Dear all, I need some guidance on reproducing the NsNsToHMNS which is provided in the gallery section of ET's website. but there are a bunch of warnings during the simulation. I would be grateful if anyone could possibly tell me what is the minimum limit on core numbers for the NsNs and also check the warnings below. I tried with 4 procs at first , but now I am trying on 28 procs. but there are same warnings. I don't know if I have to take them serious or not ! Because even with these warnings, the simulation is still running... Gratefully, Hassan *****Here are all warnings***** : Warning: thorn CoordBase already scheduled for activation WARNING level 3 from host compute-0-46.local process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn IOUtil, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/IOUtil/CheckpointRecovery.cc:730: -> No HDF5 checkpoint files with basefilename 'checkpoint.chkpt' found in recovery directory 'NsNs' Formaline::Formaline_RegisterWarnings: [meta] Register to receive warnings and info messages from the flesh WARNING level 3 from host compute-0-46.local process 0 while executing schedule bin CCTK_STARTUP, routine TerminationTrigger::TerminationTrigger_StartSignalHandler in thorn TerminationTrigger, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/TerminationTrigger/signal.c:64: -> Could not register parameter change monitor for 'signal_numbers' INFO (Vectors): Using vector size 2 for architecture SSE2 (64-bit precision) INFO (Formaline): Writing tarballs with the Cactus sources into the directory "NsNs/cactus-source" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 INFO (MoL): Using Runge-Kutta 4 as the time integrator. INFO (SymBase): Symmetry on lower x-face: rotating_symmetry_180 INFO (SymBase): Symmetry on lower z-face: reflection_symmetry INFO (MoL): The maximum number of evolved variables is 1250. 30 are registered. INFO (MoL): The maximum number of slow evolved variables is 1250. 0 are registered. INFO (MoL): The maximum number of constrained variables is 1250. 37 are registered. INFO (MoL): The maximum number of SandR variables is 1250. 0 are registered. INFO (MoL): The maximum number of evolved array variables is 1250. 0 are registered. INFO (MoL): The maximum number of constrained array variables is 1250. 0 are registered. INFO (MoL): The maximum number of SandR array variables is 1250. 0 are registered. INFO (MoL): The maximum size of any array variables is 0. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). INFO (NSTracker): Found star at (15.1875,0,0) INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- WARNING level 2 from host compute-0-46.local process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_calculate.F90:130: -> There were not enough past time levels available for accurate calculations INFO (TimerReport): Timer Report at iteration 0 time 0 INFO (TimerReport): Top timers at iteration 0 time 0 ----------------------------------- Hassan Khalvati Master Student Department of Physics Sharif University of Technology Tehran, Azadi Hassan.khalvati(a)physics.sharif.edu

3 5

meeting minutes for 2018-10-29
by Roland Haas 29 Oct '18

29 Oct '18

Present: Bill, Peter, Sam, Roland, Zach BNS gallery example: * no further progresss, should have time two weeks from now ET infrasstructure: * will have presentation for Helvi, Bill, Zach on how the pieces of ET infrastructure fit together * Zach is planning to have a longer term project of more automated code generation from NRPy+ code http://blackholesathome.net C++11 support: * Zach states that intel 15 exists on tianhe, currently intel 14 in use The US will not yet have changed times next week. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Question about GW150914.rpar simulation
by Benjamin Chardi Marco 24 Oct '18

24 Oct '18

Dear friends, We are trying to use the EinsteinToolKit GW150914.rpar binary balckhole merge simulation as use case to test that our container orchestration product OpenShift can be used for HPC. Our test environment only has 30 CPUs so we need to execute that simulation in a reasonable time. Please can you tell us how to modify GW150914.rpar in order to get a less precise simulation executed in a 30 CPUs cluster in a reasonable time (~ few days). Now we can run the simulation GW150914.rpar using OpenMPI + EinsteinToolKit, but it takes so long to be executed (~ weeks). We believe that GW150914.rpar EinsteinToolKit is a great use case to test OpenShift for HPC, and of course we will reference to EinsteinToolKit is our final report as a use case for Openshift in HPC mode. Many thanks in advance for your help, Benja -- Benjamín Chardí Marco Senior Red Hat Consultant RHCE #100-107-341 bchardim(a)redhat.com Mobile: 0034 654 344 878

3 4

Re: [Users] Question about GW150914.rpar simulation
by Roland Haas 24 Oct '18

24 Oct '18

Hello Benja, you will still see both black holes. Half this size is really "reduce the radius of the sphere by half" and not showing only one half of the domain. Eg the original rpar file may have had a domain that goes out to a radius of 2000M while the new one goes only to 1000M. The black holes are never further apart than about 20M or so though so they are always included. Yours, Roland > Hi Ronal, > > Many thanks for your help, we are going to try the suggested rpar. > Just one question, if we use "domain half its current size" as you > suggested, when we will plot phi.*.xy.h5 result files we will see the two > "black holes" as figure " > https://docs.einsteintoolkit.org/et-docs/File:vt-5.png" > or just one ? > > Cheers, > Benja > > > On Tue, Oct 23, 2018 at 7:47 PM Roland Haas <rhaas(a)illinois.edu> wrote: > > > Hello Benja, > > > > attached please find a modified rpar file where I made two changes: > > > > * changed the boundary condition to be of Robin type instead of > > Dirichlet type, which reduces reflections on the boundary (the line > > NewRad::z_is_radial = "yes") > > * made the domain half its current size which reduces memory footprint > > and runtime but will induce some reflections off the boundary, this > > makes the simulation smaller so that is uses less memory > > * then ran with very low resolution (N=24 instead of N=28) this makes > > the simulation runs faster > > > > I gave it a test run on my workstation (12 cores, 96GB of RAM) and it > > runs at ~4.1 M/hour. Since the full simulation is > > about 1000 M this will finish in 10 days. > > > > If this is too slow (which is may well be) then you can try and reduce > > the finite difference order to 6 from 8 by changing the lines (they > > are not consecutive in the file): > > > > Driver::ghost_size = 5 > > Coordinates::patch_boundary_size = 5 > > Coordinates::additional_overlap_size = 3 > > Coordinates::outer_boundary_size = 5 > > ML_BSSN::fdOrder = 8 > > SummationByParts::order = 8 > > Interpolate::interpolator_order = 5 > > WeylScal4::fdOrder = 8 > > to: > > > > Driver::ghost_size = 4 > > Coordinates::patch_boundary_size = 4 > > Coordinates::additional_overlap_size = 3 > > Coordinates::outer_boundary_size = 4 > > ML_BSSN::fdOrder = 6 > > SummationByParts::order = 6 > > Interpolate::interpolator_order = 3 > > WeylScal4::fdOrder = 6 > > > > which gives me a run speed of ~6.9M/hr (so 7 days runtime). > > > > This is the command line to start the simulation: > > > > simfactory/bin/sim create-submit GW150914_24 --define N 24 \ > > --parfile ~/runs/devel/GW150914.rpar --procs 12 --walltime 24:00:00 > > > > Yours, > > Roland > > > > > Dear friends, > > > > > > We are trying to use the EinsteinToolKit GW150914.rpar binary > > > balckhole merge simulation as use case to test that our container > > > orchestration product OpenShift can be used for HPC. > > > Our test environment only has 30 CPUs so we need to execute that > > > simulation in a reasonable time. > > > > > > Please can you tell us how to modify GW150914.rpar in order to get a > > > less precise simulation executed in a 30 CPUs cluster in a reasonable > > > time (~ few days). > > > Now we can run the simulation GW150914.rpar using OpenMPI + > > > EinsteinToolKit, but it takes so long to be executed (~ weeks). > > > > > > We believe that GW150914.rpar EinsteinToolKit is a great use case to > > > test OpenShift for HPC, and of course we will reference to > > > EinsteinToolKit is our final report as a use case for Openshift in > > > HPC mode. > > > > > > Many thanks in advance for your help, > > > Benja > > > > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2018