Users October 2019

users@lists.einsteintoolkit.org

19 participants
28 discussions

Missing file for configuring build ubuntu.cfg
by David N Bradley 04 Nov '19

04 Nov '19

Hello All: I am in process of installing the ET system, and can not find the file ubuntu.cfg needed for configuring for Mint. ./simfactory/bin/sim setup-silent --optionlist=ubuntu.cfg --runscript debian.sh recently reformated and installed mint 19.1 and lost my old setups. David

4 3

meeting minutes for 2019-10-31
by Haas, Roland 31 Oct '19

31 Oct '19

Present: Bill, Chris, Helvi, Maria, Roland, Steve, Yosef, Zach New meeting software: * will use zoom, hosted by LSU (https://lsu.zoom.us/j/631356683) * volunteers for note takers - Bill - Helvi - Roland - Steve - Zach * volunteers for leads - Helvi - Roland - Steve - Peter - Zach * other tasks - approve tutorial servers - approve ET membership requests - spam check in the mailing list filters - tutorial maintenance post release cleanup: * Roland to upload BNS gallery example, close ticket * Helvi to upload missing galery examples * Helvi to update main ET page with release announcement * Steve to check that TODO list for release preparation is complete remove content from cactuscode.org: * move content to sub-page of einsteintoolkit.org to remove extra server to maintain * construct list of page to keep before move * keep URL and redirect to the new page next ET release * call to propose code to be included - BaikalETK inclusion, Helvi agreed to review - possibly NRPy+, given that there are NRPy+ generated thorns LSU meeting for ET * aim for first week of August * has some participant support money left, will use to bring in students * will have ET tutorial Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 31 Oct '19

31 Oct '19

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. For details on how to connect and what agenda items are to be discussed, use the link below. https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call --Steve

8 94

Agenda for Monday's Meeting
by Steven R. Brandt 30 Oct '19

30 Oct '19

Please update the Wiki with agenda items for Monday's meeting. Thanks! https://docs.einsteintoolkit.org/et-docs/meeting_agenda --Steve

3 46

New Einstein Toolkit Release (Chien-Shiung Wu)
by Steven R. Brandt 30 Oct '19

30 Oct '19

=== Release Announcement === We are pleased to announce the sixteenth release (code name "Tesla") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are Five new thorns have been added: * GiRaFFE * GiRaFFEfood * ShiftedKerrSchild * GiRaFFE_to_HydroBase * ID_converter_GiRaFFE In addition, bug fixes accumulated since the previous release in Feb 2018 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics code GRHydro. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 200 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: == Larger changes since last release == * Support has been added for GCC8. * Built-in OpenMPI tarball has been upgraded to 1.10.7 Note, we still recommend using the system MPI that is installed natively on your system. * And many bug fixes. Notably, a significant correctness problem for NewRad was addressed which caused many simulations with more than two ranks to fail https://trac.einsteintoolkit.org/ticket/2182 == New thorns or tools == * GiRaFFE (~1,600 lines of code): Solves the GRFFE equations in the formalism of Paschalidis and Shapiro [2] using the same reconstruction (PPM) and staggered vector potential methods as IllinoisGRMHD. * GiRaFFEfood (~1,300 lines of code): Code validation initial data and basic diagnostics. Contains Five 1D tests in flat spacetime (fast wave, Alfvén wave, degenerate Alfvén wave, "three-wave", FFE breakdown), One 3D test in flat spacetime (aligned rotator), and Three 3D tests in curved spacetimes ("Exact Wald" solution, "Magnetospheric Wald" solution, and split monopole). * ShiftedKerrSchild (~230 lines of code): Sets up all 3+1 quantities for a Kerr-Schild spacetime with an arbitrary radial shift, to mitigate effects of extreme spacetime curvature near r_{KS}=0. See Appendix of code release paper for full discussion. * GiRaFFE_to_HydroBase and ID_converter_GiRaFFE (~230 lines of code): Acts as a translation layer to convert 3-velocities in HydroBase's Valencia formalism to/from the 3-velocity found in the induction equation (vi = ui/u0; native to GiRaFFE). == Upcoming changes for the next releases == == New thorns planned for the next relases == * Seed_Magnetic_Fields-modified Extended Seed_Magnetic_Fields thorn for binary neutron stars. Supercedes Seed_Magnetic_Fields thorn. * Meudon_Bin_NS-modified Modifications to Meudon BNS initial data thorn to disable the overwriting of initial lapse/shift, which acts to significantly reduce coordinate eccentricity. Supercedes Meudon_Bin_NS thorn. * VolumeIntegrals_GRMHD Nice GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. * VolumeIntegrals_vacuum Nice GRMHD volume integration thorn, currently depends on ML_BSSN. There is a bit of code duplication and duplicated functionality between VI_GRMHD and VI_vacuum, to ensure that VI_vacuum can be used without enabling a GRMHD code. There is probably a better way of doing this, but I haven't had the time to think deeply about this. * particle_tracerET Solves the ODE \partial_t xi = vi for typically thousands of tracer particles, using an RK4 integration atop the current timestepping. E.g., one RK4 substep in the particle integration might occur every 16 RK4 substeps in the GRMHD evolution. These tracer particle positions are quite useful for visualizing magnetic field lines in a consistent way from frame-to-frame in a movie (recall that in the GRMHD approximation, the magnetic field lines stay attached the fluid elements they thread Flux Freezing?). Note that the velocity must be consistent with the velocity appearing in the GRMHD induction equation. This thorn reads in the HydroBase vel[] vector gridfunction, which assumes the Valencia formalism, and converts it into the induction equation velocity. * smallbPoynET Computes bi, b2, and three spatial components of Poynting flux. It also computes (-1-u0), which is useful for tracking unbound matter. * LeanBSSNMoL and NPScalars: LeanBSSNMoL evolves Einstein's Equations with the BSSN formulation. NPScalars computes the Newman-Penrose scalars. For the conformal factor it uses the "W" version, i.e. W=\gamma^{-1/6}, and it employs the usual "1+log" and "Gamma-driver" gauge conditions. Also available, in the "new_gauge" branch, is the shift condition of Figueras et al for their evolutions of higher dimensional BHs, see: https://arxiv.org/abs/1702.01755 https://arxiv.org/abs/1512.04532 * Proca arrangement: This repository (https://bitbucket.org/canuda/proca/src) provides the tools to evolve the Einstein-Proca system as first described in https://arxiv.org/abs/1505.00797 == How to upgrade from Tesla (ET_2018_02) == To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. == Machine notes == Supported (tested) machines include: * Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (Homebrew and MacPorts) installations * Bluewaters * Comet * Cori * Draco * Edison * Golub * Hydra * Marconi * Minerva * Queenbee 2 * Stampede 2 * SuperMIC * Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * A new configuration for KNL nodes is being worked on, but not yet included in the release (but compilation works and tests mostly pass). All repositories participating in this release carry a branch ET_2018_02 marking this release. These release branches will be updated if severe errors are found. The "Chien-Shiung Wu" Release Team on behalf of the Einstein Toolkit Consortium (2018-09-17) * Steven R. Brandt * Samuel D. Cupp * Peter Diener * Roland Haas * Roberto De Pietri Sep, 2018

1 1

Loop over the grid points of the base grid
by Erik Rodrígo Jiménez Vázquez 25 Oct '19

25 Oct '19

Dear all I want to subtract an array at t-dt and t times only at the base grid (refinement level 0), following http://lists.einsteintoolkit.org/pipermail/users/2013-May/003004.html, I implemented the suggestion in my code as BEGIN_GLOBAL_MODE(cctkGH) { ENTER_LEVEL_MODE(cctkGH, 0) { for(int k=0;k<cctk_lsh[2];k++) { for(int j=0;j<cctk_lsh[1];j++) { for(int i=0;i<cctk_lsh[0];i++) { int index= CCTK_GFINDEX3D(cctkGH,i,j,k); lres += abs(phi[index]-phi_p[index]); } } } } LEAVE_LEVEL_MODE; } END_GLOBAL_MODE; However I got a segmentation fault. I think is because of the array index, if I comment the line lres += abs(phi[index]-phi_p[index]); the code works fine. My questions are: 1) How can I perform this operation only at the base grid?, 2) How can I get the correct index at the base level? .. any help is very welcome. Thanks. Erik

3 2

Add a lapse condition
by Erik Rodrígo Jiménez Vázquez 25 Oct '19

25 Oct '19

Dear all Is there a way to add different gauge conditions for the lapse? I'm trying to add a gauge shock avoid lapse for the Bona-Masso family of slicing conditions dot alpha = -alpha^2 f(alpha) K with f(alpha)=1+constant/alpha^2 I don't know if I should add a thorn that complements the McLachlan thorn, or what is the correct procedure to implement the gauge condition? Furthermore, this kind of gauge condition allows to the lapse become negative which I know is a serious drawback when the lapse collapses to zero, however I want to do some numerical experiments with this kind of lapse. Any help is very welcome. Thanks Erik

3 4

Meeting Minutes for 2019-10-24
by Hamilton, Maria 24 Oct '19

24 Oct '19

Present: Bill Gabella, Yu Liu, Steve Brandt, Peter Diener, Maria Babiuc-Hamilton, Helvi Vitek, Roland Haas ET release * release day issue: scheduled for Oct 27th which is a Sunday, rescheduled for Monday Oct. 28th. * things Steve needs to finish: DOI, release announcement, branches * please read and check release announcement at: <https://bitbucket.org/einsteintoolkit/www.git> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbitbucket.org%2feinstei… * gallery examples: Helvi finished BBH, Poisson done, waiting for data * BNS Roland and Maria are running on clusters, just waiting for the data to be extra sure * testsuite failing on "golub" private machine, to be removed for the upcoming release * Roland mentioned that testing VM on his osx macports some MPI using tests hang * Steve reported that Centos8 was release, nothing works, issue to be delayed * He raised the question of choosing to support either macports or homebrew on MacOS, issue to be delayed * Peter closed tickets on SelfForce, and all other users, is past time * Recap: to be done with release on OCT. 28th. * Yu Liu past meeting question: does ET_2019_10 work on MacOSX with Catalina OS and homebrew? -- No, to be looked into after the release. regards, Maria _______________________ Maria C. Babiuc Hamilton Professor of Physics, Ph.D. S257, College of Science, Marshall University, 1 John Marshall Drive, Huntington, WV, 25755 Phone: (304)696-2754

1 0

Question about "New star location not set (18.4258, 0.763088, 0) because too far from old (15.0469, 1.82812, 0)."
by ZhiChao Zhao 24 Oct '19

24 Oct '19

Dear all, I am using ET to simulate binary neutron star merge. I use NStracker to tracker the stars. However, I found a lot of logs that likes : “New star location not set (18.4258,0.763088,0) because too far from old (15.0469,1.82812,0).” I think maybe it shows there is some error. Could you list some possible unreasonable uses that that can cause this error? If parfile and initial_data are necessary, my parfile and the initial_data are here: https://github.com/yanyuechuixue/parfiles Thank you !

2 1

Re: [Users] The simulation stops suddenly, problem with restarting from checkpoint, and with number of processors
by Haas, Roland 23 Oct '19

23 Oct '19

Hello Hassan, using create-run is likely not the best way to run these simulations. Create-run is really intended for interactive use and will output stdout and stderr to screen and never to a file (which is why there are no *.out and *.err files). The command to use to run production simulations is "create-submit" (or just "submit") which will start the simulation in the background. I cannot really make a good guess as to why the simulation ran faster / slower using the options given as there are usually too many things interacting. Looking at the RunScript files nothing seems to be obviously wrong, though OpenMPI's core binding could affect things. Please see https://www.open-mpi.org/faq/?category=tuning#using-paffinity-v1.6 for a description of the options available. Usually you will want something like --bind-to-socket or --bind-to-none (with the exact name of the option depending a bit on the OpenMPI version used). I am also not sure (without seeing at least the out files and possibly the ini files for the machine) how many threads / MPI ranks were used without the --num-threads option. If it ended up using just threads and a single MPI ranks then a speed difference of 2 or so is not unlikely. Are you running this on a Mac or a Linux box? Yours, Roland > Dear Roland, > sorry for the delay. > I could resume the simulation, doing based on your suggestion to add a line > for jobid, but =999985 nor =999999. > The *.err and *.out are still missing. Please find the files attached. > > Here is the script I used to run the simulation > > ./simfactory/bin/sim create-run the-last-one \ > --parfile=par/bbh-2res-1mass-10sep-final.par --ppn-used=56 > --num-threads=14 --num-smt=2 > > > There is another problem besides this. In the script line above, I used > ppn-used=56 and procs=56 before and the "htop" was showing that all 56 > threads are being used 100%. but when I changed to what is written in the > above line ( --ppn-used=56 --num-threads=14 --num-smt=2 ), the "htop" was > showing that only a portion of cores are being used and non was performing > on 100%, however, the simulation was *running faster more than fourfold. * > I am still confused about specifying the number of cores. > > Bests, > Hassan > > On Thu, 3 Oct 2019 at 19:13, Haas, Roland <rhaas(a)illinois.edu> wrote: > > > Hello Hassan, Erik, > > > > does this issue still persist? Since there are no *.out and no *.err > > file, would you mind providing a list of the files that are present? > > > > Ie the output of somehing like: > > > > ls -lR /home/cosmo/simulations/the-last-one/output-0001/ > > > > as well as the simfactory log file > > (home/cosmo/simulations/the-last-one/log.txt), please? > > > > Note that the files should always be saved. There names are given in > > the SubmitScript (exactly how depends on the queuing system used, it > > may be good if you could include the file SubmitScript and RunScript > > as well). > > > > Yours, > > Roland > > > > > Dear Erik, > > > Thank you for your reply, but there are not *.out or *.err files in the > > > output directory or anywhere else. Was there an option that I have had to > > > activate that to save these files? > > > > > > Hassan > > > > > > On Wed, 11 Sep 2019 at 21:54, Erik Schnetter <schnetter(a)cct.lsu.edu> > > wrote: > > > > > > > Hassan > > > > > > > > The last lines of the simulation output might not include the error > > > > message. There should be two files in the output directory, one ending > > > > in *.out, the other ending in *.err. The latter might have an actual > > > > error message. > > > > > > > > To see whether all cores are used, you can look at the startup output > > > > of Carpet. This would be near the beginning of the *.out file above > > > > (within the first 1000 lines or so). To get more detailed output, you > > > > can activate the thorn "SystemTopology" in your parameter file. This > > > > will provide more details regarding cores and threads in your output. > > > > > > > > -erik > > > > > > > > > > > > On Wed, Sep 11, 2019 at 12:19 PM Hassan Khalvati < > > hassan.kh92(a)gmail.com> > > > > wrote: > > > > > > > > > > Dear All, > > > > > I had a simulation running for nearly 5 days and it stops today with > > no > > > > reason, no errors, and no termination. > > > > > the first thing I need help with is that I can not find the cause > > that > > > > the simulation has been stopped. The last lines during the simulation > > have > > > > been attached as a text file. > > > > > > > > > > > > > > > The second problem is that I can not restart from the checkpoint. > > there > > > > is an error : > > > > > > > > > > ./simfactory/bin/sim submit the-last-one > > > > --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 > > > > > Error: job id is negative > > > > > Aborting Simfactory. > > > > > > > > > > > > > > > I looked up in email archives, and I did what Roland has suggested, > > to > > > > add a line for jobid, (jobid = 999999) in the properties.ini file, but > > I > > > > am still getting errors > > > > > > > > > > ./simfactory/bin/sim submit the-last-one > > > > --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 > > > > > Warning: job status is U > > > > > Warning: job status is U > > > > > Assigned restart id: 1 > > > > > Warning: Too many used cores per node specified: specified > > ppn-used=56 > > > > (ppn is 28) > > > > > Executing submit command: exec nohup > > > > > > /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < > > > > /dev/null > /dev/null 2> /dev/null & echo $! > > > > > Submit finished, job id is 8907 > > > > > > > > > > > > > > > > > > > > I changed the lines in the properties.ini file for procs, and again > > > > getting error > > > > > > > > > > > > > > > ./simfactory/bin/sim submit the-last-one > > > > --parfile=par/bbh-2res-1mass-10sep-final.par > > > > > Assigned restart id: 1 > > > > > Executing submit command: exec nohup > > > > > > /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < > > > > /dev/null > /dev/null 2> /dev/null & echo $! > > > > > Submit finished, job id is 10517 > > > > > > > > > > And finally, I am confused about the option for the "ppn, procs, > > and > > > > ..." numbers in the Simfactory. I have attached my CPU information. It > > is a > > > > double 14 core Xeon(R) CPU E5-2680, with 2 threads per core. my > > submission > > > > command was: > > > > > ./simfactory/bin/sim create-run the-last-one > > > > --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 > > --ppn-used=56 > > > > > but in the properties.ini file, it is mentioned that: > > > > > numprocs = 4 > > > > > nodeprocs = 4 > > > > > numthreads = 14 > > > > > I have also attached the properties.ini file. Is it using only 4 > > cores? > > > > I looked up in the Simfactory docs, and also ET's wiki. I can not get a > > > > clear picture of how the option of the number of processors works. > > However, > > > > with the same command line, I have mentioned above, --procs=56 > > > > --ppn-used=56, the simulation was performing well, I want to know if > > it is > > > > using total number of processors on my system or not. I would be > > grateful > > > > if anyone could help me with each of these issues. > > > > > > > > > > Attachments are: > > > > > parameter file, > > > > > properties.ini, > > > > > simulation-last-lines, > > > > > CPU info, > > > > > and the log.txt file. > > > > > > > > > > > > > > > > > > > > Sincerely, > > > > > Hassan > > > > > > > > > > > > > > > -- > > > > > > > > > > Hassan Khalvati > > > > > Sharif University of Technology, Tehran > > > > > Hassan.Khalvati(a)physics.sharif.edu > > > > > Hassan.kh92(a)gmail.com > > > > > > > > > > _______________________________________________ > > > > > Users mailing list > > > > > Users(a)einsteintoolkit.org > > > > > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > > > > > > > > > > > > > -- > > > > Erik Schnetter <schnetter(a)cct.lsu.edu> > > > > http://www.perimeterinstitute.ca/personal/eschnetter/ > > > > > > > > > > > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2019