Users September 2018

users@lists.einsteintoolkit.org

18 participants
28 discussions

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 31 Oct '19

31 Oct '19

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --Steve

8 94

Agenda for Monday's Meeting
by Steven R. Brandt 30 Oct '19

30 Oct '19

Please update the Wiki with agenda items for Monday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --Steve

3 46

New Einstein Toolkit Release (Chien-Shiung Wu)
by Steven R. Brandt 30 Oct '19

30 Oct '19

=== Release Announcement === We are pleased to announce the sixteenth release (code name "Tesla") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are Five new thorns have been added: * GiRaFFE * GiRaFFEfood * ShiftedKerrSchild * GiRaFFE_to_HydroBase * ID_converter_GiRaFFE In addition, bug fixes accumulated since the previous release in Feb 2018 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics code GRHydro. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 200 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: == Larger changes since last release == * Support has been added for GCC8. * Built-in OpenMPI tarball has been upgraded to 1.10.7 Note, we still recommend using the system MPI that is installed natively on your system. * And many bug fixes. Notably, a significant correctness problem for NewRad was addressed which caused many simulations with more than two ranks to fail https://trac.einsteintoolkit.org/ticket/2182 == New thorns or tools == * GiRaFFE (~1,600 lines of code): Solves the GRFFE equations in the formalism of Paschalidis and Shapiro [2] using the same reconstruction (PPM) and staggered vector potential methods as IllinoisGRMHD. * GiRaFFEfood (~1,300 lines of code): Code validation initial data and basic diagnostics. Contains Five 1D tests in flat spacetime (fast wave, Alfvén wave, degenerate Alfvén wave, "three-wave", FFE breakdown), One 3D test in flat spacetime (aligned rotator), and Three 3D tests in curved spacetimes ("Exact Wald" solution, "Magnetospheric Wald" solution, and split monopole). * ShiftedKerrSchild (~230 lines of code): Sets up all 3+1 quantities for a Kerr-Schild spacetime with an arbitrary radial shift, to mitigate effects of extreme spacetime curvature near r_{KS}=0. See Appendix of code release paper for full discussion. * GiRaFFE_to_HydroBase and ID_converter_GiRaFFE (~230 lines of code): Acts as a translation layer to convert 3-velocities in HydroBase's Valencia formalism to/from the 3-velocity found in the induction equation (vi = ui/u0; native to GiRaFFE). == Upcoming changes for the next releases == == New thorns planned for the next relases == * Seed_Magnetic_Fields-modified Extended Seed_Magnetic_Fields thorn for binary neutron stars. Supercedes Seed_Magnetic_Fields thorn. * Meudon_Bin_NS-modified Modifications to Meudon BNS initial data thorn to disable the overwriting of initial lapse/shift, which acts to significantly reduce coordinate eccentricity. Supercedes Meudon_Bin_NS thorn. * VolumeIntegrals_GRMHD Nice GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. * VolumeIntegrals_vacuum Nice GRMHD volume integration thorn, currently depends on ML_BSSN. There is a bit of code duplication and duplicated functionality between VI_GRMHD and VI_vacuum, to ensure that VI_vacuum can be used without enabling a GRMHD code. There is probably a better way of doing this, but I haven't had the time to think deeply about this. * particle_tracerET Solves the ODE \partial_t xi = vi for typically thousands of tracer particles, using an RK4 integration atop the current timestepping. E.g., one RK4 substep in the particle integration might occur every 16 RK4 substeps in the GRMHD evolution. These tracer particle positions are quite useful for visualizing magnetic field lines in a consistent way from frame-to-frame in a movie (recall that in the GRMHD approximation, the magnetic field lines stay attached the fluid elements they thread Flux Freezing?). Note that the velocity must be consistent with the velocity appearing in the GRMHD induction equation. This thorn reads in the HydroBase vel[] vector gridfunction, which assumes the Valencia formalism, and converts it into the induction equation velocity. * smallbPoynET Computes bi, b2, and three spatial components of Poynting flux. It also computes (-1-u0), which is useful for tracking unbound matter. * LeanBSSNMoL and NPScalars: LeanBSSNMoL evolves Einstein's Equations with the BSSN formulation. NPScalars computes the Newman-Penrose scalars. For the conformal factor it uses the "W" version, i.e. W=\gamma^{-1/6}, and it employs the usual "1+log" and "Gamma-driver" gauge conditions. Also available, in the "new_gauge" branch, is the shift condition of Figueras et al for their evolutions of higher dimensional BHs, see: https://arxiv.org/abs/1702.01755 https://arxiv.org/abs/1512.04532 * Proca arrangement: This repository (https://bitbucket.org/canuda/proca/src) provides the tools to evolve the Einstein-Proca system as first described in https://arxiv.org/abs/1505.00797 == How to upgrade from Tesla (ET_2018_02) == To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. == Machine notes == Supported (tested) machines include: * Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (Homebrew and MacPorts) installations * Bluewaters * Comet * Cori * Draco * Edison * Golub * Hydra * Marconi * Minerva * Queenbee 2 * Stampede 2 * SuperMIC * Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * A new configuration for KNL nodes is being worked on, but not yet included in the release (but compilation works and tests mostly pass). All repositories participating in this release carry a branch ET_2018_02 marking this release. These release branches will be updated if severe errors are found. The "Chien-Shiung Wu" Release Team on behalf of the Einstein Toolkit Consortium (2018-09-17) * Steven R. Brandt * Samuel D. Cupp * Peter Diener * Roland Haas * Roberto De Pietri Sep, 2018

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #1326: running loopcontrol on strange number of threads fails
by Erik Schnetter 11 Apr '19

11 Apr '19

Do you have a backtrace? -erik On Sunday, March 22, 2015, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #1326: running loopcontrol on strange number of threads fails > ------------------------------------+--------------------------------------- > Reporter: rhaas | Owner: > Type: defect | Status: new > Priority: minor | Milestone: > Component: EinsteinToolkit thorn | Version: > Resolution: | Keywords: LoopControl > ------------------------------------+--------------------------------------- > > Comment (by rhaas): > > This still happens even with current (Sun Mar 22 18:46:21 CET 2015) trunk, > though failure looks a bit different now: > {{{ > actus_sim: > /data/rhaas/postdoc/gr/ET_trunk/configs/sim/build/LoopControl/loopcontrol.cc:312: > T {anonymous}::divexact(T, T) [with T = int]: Assertion `i % j == 0' > failed > }}} > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1326#comment:1> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Compiling ET Using PGI
by Christopher Ramirez 29 Nov '18

29 Nov '18

Hello, I've been getting the following errors when trying to compile using PGI compilers and am wondering if were any additional steps needed to compile successfully. COMPILING arrangements/CactusBase/Boundary/src/ScalarBoundary.c PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 41) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 41) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 42) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 42) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 45) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 45) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cGH.h: 71) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cGH.h: 71) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 29) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 29) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 30) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 30) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 60) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 60) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 61) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 61) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 67) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 67) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Coord.h: 68) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Coord.h: 68) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Misc.h: 43) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Misc.h: 43) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 39) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 39) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 40) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 40) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 76) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 76) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/cctk_Interp.h: 77) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/cctk_Interp.h: 77) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 216) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 216) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 228) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 228) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 301) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 301) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 321) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 321) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 381) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 381) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 393) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 393) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/src/include/util_Table.h: 467) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 467) PGC-S-0040-Illegal use of symbol, CCTK_COMPLEX16 (/home/ramirez/Cactus/src/include/util_Table.h: 487) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/src/include/util_Table.h: 487) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 40) PGC-W-0156-Type not specified, 'int' assumed (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 40) PGC-S-0040-Illegal use of symbol, CCTK_REAL8 (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 42) PGC-F-0008-Error limit exceeded (/home/ramirez/Cactus/arrangements/CactusBase/Boundary/src/Boundary.h: 42) PGC/x86-64 Linux 17.10-0: compilation aborted make[3]: *** [ScalarBoundary.c.o] Error 2 make[2]: *** [make.checked] Error 2

4 8

mass estimation of neutron star
by 林家暉 07 Oct '18

07 Oct '18

Dear whom it may concern, I simulated the binary neutron star merger as in gallery code(https://einsteintoolkit.org/gallery/bns/index.html). I tried to find out the mass of the merged neutron star. For test, I calculated the initial mass of each neutron star from the file rho.xy.h5 which is the rest mass density. What I computed is about 0.9 solar mass , however it supposed to be much larger because the baryonic mass of the initial data of each neutron star is about 1.45 solar mass. I also calculated magnetic energy which corresponds to about 1e-6 solar mass which is not a possible reason of the discrepancy. I also considered velocity which may increase mass due to the relativity but the maximum of the initial velocity is only about 0.1c which also does not raise the mass too much. What is the possible reason that the total rest mass is so much smaller than the baryonic mass ? Thank you. Best regards, Chia-Hui

2 12

Issue running the default qc0-mclachlan.par
by Gomard-Henshaw, Chad 06 Oct '18

06 Oct '18

Hello, When running the default qc0 simulation, I get an error (see attached). This was run using the following command in the windows linux subshell: ./simfactory/bin/sim create-run qc05 \ --parfile=par/qc0-mclachlan.par The simulation runs for about an hour before aborting; I get partial output files but only with two data points. Can you please advise on how to address this issue? Thanks! -- Chad Henshaw Georgia Institute of Technology - Physics

3 8

Compile Errors
by Christopher Ramirez 02 Oct '18

02 Oct '18

Hello, Using PGI 18.4 and intel mpi provided by PSXE 2018. I'm using the newest version of ET at the moment, and when compiling I'm reaching the current errors. I'm not too familiar with the errors I'm getting for piranha, nor am I understanding whats calling the intlc library. I've linked directly to the containing folder inside of my config file and am thinking it's not the library I assumed it was (provided by intel). Could someone please review and let me know if you're familiar with any of these? Thank you. pa: file ReParse.cc.o, line 1311: unknown variable reference: &2&542 ipa: file ReParse.cc.o, line 1312: unknown variable reference: &2&543 ipa: file ReParse.cc.o, line 2850: unknown variable reference: &2&660 ipa: file ReParse.cc.o, line 2853: unknown variable reference: &2&661 ipa: file ReParse.cc.o, line 3187: unknown variable reference: &2&683 ipa: file ReParse.cc.o, line 3188: unknown variable reference: &2&684 ipa: file ReParse.cc.o, line 3218: unknown variable reference: &2&688 ipa: file ReParse.cc.o, line 3219: unknown variable reference: &2&689 ipa: file ReParse.cc.o, line 3613: unknown variable reference: &2&729 ipa: file ReParse.cc.o, line 3614: unknown variable reference: &2&730 ipa: file AutoGrammar.cc.o, line 1093: unknown variable reference: &2&293 ipa: file AutoGrammar.cc.o, line 1094: unknown variable reference: &2&294 ipa: file AutoGrammar.cc.o, line 1155: unknown variable reference: &2&300 ipa: file AutoGrammar.cc.o, line 1156: unknown variable reference: &2&301 ipa: file AutoGrammar.cc.o, line 1602: unknown variable reference: &2&334 ipa: file AutoGrammar.cc.o, line 1605: unknown variable reference: &2&335 ipa: file AutoGrammar.cc.o, line 1939: unknown variable reference: &2&357 ipa: file AutoGrammar.cc.o, line 1940: unknown variable reference: &2&358 ipa: file AutoGrammar.cc.o, line 1970: unknown variable reference: &2&362 ipa: file AutoGrammar.cc.o, line 1971: unknown variable reference: &2&363 ipa: file AutoGrammar.cc.o, line 2216: unknown variable reference: &2&388 ipa: file AutoGrammar.cc.o, line 2217: unknown variable reference: &2&389 ipa: file Lookup.cc.o, line 1810: unknown variable reference: &2&448 ipa: file Lookup.cc.o, line 1811: unknown variable reference: &2&449 ipa: file Lookup.cc.o, line 1872: unknown variable reference: &2&454 ipa: file Lookup.cc.o, line 1873: unknown variable reference: &2&455 ipa: file Lookup.cc.o, line 2371: unknown variable reference: &2&497 ipa: file Lookup.cc.o, line 2374: unknown variable reference: &2&498 ipa: file Lookup.cc.o, line 2708: unknown variable reference: &2&520 ipa: file Lookup.cc.o, line 2709: unknown variable reference: &2&521 ipa: file Lookup.cc.o, line 2739: unknown variable reference: &2&525 ipa: file Lookup.cc.o, line 2740: unknown variable reference: &2&526 ipa: file Lookup.cc.o, line 3081: unknown variable reference: &2&563 ipa: file Lookup.cc.o, line 3082: unknown variable reference: &2&564 The following function(s) are called, but no IPA information is available: assert, get_mpi_comm Linking without IPA /usr/bin/ld: cannot find -lintlc child process exit status 1: /usr/bin/ld make[1]: *** [/home/ramirez/Cactus/exe/cactus_drewz] Error 2

3 4

Error: job id is negative
by Ferguson, Deborah L 01 Oct '18

01 Oct '18

Periodically when continuing a run, I get the error Error: job id is negative Aborting Simfactory It is fixed by deleting the symlink to the previous active directory and then resubmitting. Although that fix works, when I want to submit a job multiple times at once (so that when one output is finished running, the next one starts automatically), this error prevents me from doing that. Anyone have any ideas on what might be the underlying cause of this issue and how to actually fix it? Thanks! Deborah ----------------------------------------- Deborah Ferguson Graduate Student Georgia Institute of Technology Center for Relativistic Astrophysics Boggs 1-66

3 3

Fwd: [new user][help] Problem during install: libcblas
by Arthur Reis 27 Sep '18

27 Sep '18

Hello, I'm a new user and have been trying to install the Einstein Toolkit following the instructions at https://nbviewer.jupyter.org/github/nds-org/jupyter-et/blob/master/CactusTu… . (I'm using arch-linux x64) I was able to follow the procedures and succesfully submitted the "helloworld" simulation job. Listing the simulations I have: > [ianncunha@oppenheimer Cactus]$ ./simfactory/bin/sim list-sim > Warning: job status is U > helloworld [ACTIVE (FINISHED), restart 0000, job id 6582] > But trying to get the output I get the following error: > /home/ianncunha/simulations/helloworld/SIMFACTORY/exe/cactus_sim: error > while loading shared libraries: libcblas.so.3: cannot open shared object > file: No such file or directory > > After I installed atlas-lapack, I was unable to import numpy due to the same error. I moved a copy of libcblas.so.3 by hand to /usr/lib64 and got numpy working again, but I'm still getting the error in simfactory. Could anyone help me giving some insight on how to solve this problem? Thanks, Arthur

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users September 2018