Users February 2018

users@lists.einsteintoolkit.org

16 participants
25 discussions

meeting minutes for 2018-02-19
by Roland Haas 19 Feb '18

19 Feb '18

Present: Roland, Bhavesh, Erik, Jim, Peter, Roberto, Samuel, Steve, Yosef, Frank, Qian, Ian, con2prim workshop at PI: * Erik had hosted a workshop to learn about current state of the art of con2prim methods and eos tables for hydro codes * Erik gave a quick report on what was learned at the workshop * good design should have a "cleaning" stage that removes non-phyisical points in a well defined manner so that the actual root finding can be robust * people interested in this: Philipp Moesta, David Radice, Wolfgang Kastaun big tickets items for the next release: * want to get them in now to have them well tested for next ET release * Roland, Ian, Steve have leftover "cleanup" tickets from the release * need to update trac landing page to point to current release new thorns: * GiRAFFE: make sure there is outside interest and push by authors * Lean-Public, Proca: make sure there is outside interest and push by authors * suggest third-party-thorns registration capability, add this to discussion items for next week BNS gallery example: * Roberto suggests the difference could be due to F90 vs C++ code, can check which code was originally used to compute the gallery example * Roland to re-run with code from time of gallery example creation to check on differences between clusters ET workshop at GT: * date (June 18th - June 20th) are fixed now * forming a scientific organization committee right now * working on registration procedure * will send url of website to users mailing list once known OpenMP scalability on Stampede SKX: * Jim is testing threading changes in OpenMP usage during prolongation by Erik * see this email for results: http://lists.einsteintoolkit.org/pipermail/users/2018-February/006093.html * finds ~25% benefit when using newer code * should update OpenMP settings in Simfactory to have more MPI ranks per node (right now uses 2 which is among the worst possible settings) Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Synchronization in the ADMBase schedule.ccl
by Cupp, Samuel 16 Feb '18

16 Feb '18

Hey all, I am working on the new synchronization in the presync branch, and there's a scheduling comment I don't understand. On line 234 (master branch) of the ADMBase schedule.ccl, the following comment is given before scheduling a sync for ADM grid functions: > For Mesh Refinement it is possible (eg Cowling approximation) that ADMBase variables will need synchronization even when not evolved (eg when a new refined grid appears) According to this, a sync should be triggered in some cases at the mesh refinement stage. Since I don't understand the reasons for this, I can't reproduce this behavior in the new method. Why does the mesh refinement require a sync only sometimes? Is there a specific scheduled subroutine which creates this behavior, or is it inherent in the AMR process itself? Samuel Cupp Graduate Student Department of Physics & Astronomy Louisiana State University

2 1

2D configuration with Carpet
by Miguel Zilhão 14 Feb '18

14 Feb '18

hi all, i have a 2+1 simulation that i would like to run with the ET, and i would like to use Carpet for the mesh refinement in the spatial (x and y) directions. is this possible? is there an example parameter file that i could have a look at? many thanks, Miguel

4 10

best practices for groups
by Eric West 14 Feb '18

14 Feb '18

Hi All, Do research groups distribute Cactus/EinsteinToolkit files amongst different users on the same machine? I understand that the Cactus philosophy is that it is meant to be distributed. But it seems redundant for every member of a group, using the same machine, to have their own copies of everything. I am curious how other groups handle this. What parts of the toolkit are shared amongst multiple users, if any? What parts are not shared? For context, our group is very small (currently myself and a couple of grad students). The disk quota for the group on our HPC machine is 150GB. Right now that is not prohibitively small. I'm more concerned about sprawling config files, parfiles, etc. Although if there are things that can be done to reduce the overall disk usage, that would be an added bonus. So I am wondering what practices other groups employ. Thanks, Eric -- Eric J West Assistant Professor Department of Physics and Astronomy University of Minnesota Duluth

2 1

meeting minutes for 2018-02-12
by Roland Haas 12 Feb '18

12 Feb '18

Present: Steve, Peter, Roland, Ian, Erik, Miguel, ET release: * add announcement prominently on ET website * Steve asked to have ET tutorial notebook updated on NDS machine * announce that we are now asking for larger contributions ** create tickets for them, tag them with next release milestone to it ** each ticket will have a champion who is responsible for its progress and is invested in it ** discuss progress on items at tbe beginning of each month ** stalled items will be closed and dropped * action item: go through items that were dropped before the release ET US meeting: * dates: Monday June 18th to 20th at GT * program: TBA Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

Re: [Users] questions about compilation
by ian.hinder＠aei.mpg.de 10 Feb '18

10 Feb '18

> On 1 Feb 2018, at 04:47, 林家暉 <r06222015(a)ntu.edu.tw> wrote: > > Dear Roland and whom it may concern, > I have done the simulation of the example code of GW150914. > However when I tried to visualize the result by VisIt ( https://docs.einsteintoolkit.org/et-docs/GW150914_VisIt_Tutorial#Installing… ), I can not find the .vtk files which represent the shape and properties of relevant 2D topological spheres in the output data. Only .h5 files are in my output file. > What is the possible reason for it ? If you look at the parameter file, it has the lines # out3d_every = rl0_every * 2 out3d_every = 0 You need to swap the comment, so that it reads out3d_every = rl0_every * 2 # out3d_every = 0 i.e. you need to have a nonzero out3d_every parameter. The default in the parameter file is NOT to output this 3D data, as it totals 3 TB for the whole simulation, which is more than most people want to generate! Note that this parameter is used for both the 3D gridfunction output, as well as the horizon surface data (in VTK files). If you just want the horizon files, then you can change QuasiLocalMeasures::output_vtk_every = $out3d_every to QuasiLocalMeasures::output_vtk_every = $rl0_every * 2 without changing out3d_every. I'm afraid you will have to rerun the simulation, as the VTK files were not generated when you ran it the first time. > By the way , I have commended some module load in the edison.ini file, listed below: > # module load cray-petsc/3.7.6.0 > # module load curl/7.28.1 > # module load hwloc/1.7.2 > # module load numactl/2.0.10 > I wonder whether this module commended resulted in the disappearance of .vtk files. No, this should be unrelated. -- Ian Hinder http://members.aei.mpg.de/ianhin

2 8

New Einstein Toolkit Release (Tesla)
by Steven R. Brandt 09 Feb '18

09 Feb '18

We are pleased to announce the sixteenth release (code name "Tesla") of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are: * A new thorn, Hydro_RNSID which models a rotating neutron star. * Tutorials have been updated and the install process for new users has been simplified. In addition, bug fixes accumulated since the previous release in June 2017 have been included. The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics code GRHydro. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics. The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in. Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers. For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org. The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 200 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization. The changes between this and the previous release include: === Larger changes since last release === * The support for generic machines is more robust, and the ET should compile, run, and pass the test suites out of the box on new Linux machines. * A Jupyter-based Tutorial (https://einsteintoolkit.org/documentation/new-user-tutorial) is now available. * The AVX512 instruction set used on the Intel "Knight's Landing" platform is now supported. * PITTNullCode now has test outputs * EOS_Omni polytrope supports hybrid equations of date with up to 10 pieces === New thorns or tools === * The Hydro_RNSID thorn which provides initial data for a rotating neutron star. === Upcoming changes for the next releases === * New thorns: * GiRaFFE, which models plasma flows in a dynamic spacetime * Changes to WVUThorns_Diagnostics * Seed_Magnetic_Fields-modified: Extended Seed_Magnetic_Fields thorn for binary neutron stars. Supercedes Seed_Magnetic_Fields thorn. * Meudon_Bin_NS-modified: Modifications to Meudon BNS initial data thorn to disable the overwriting of initial lapse/shift, which acts to significantly reduce coordinate eccentricity. Supercedes Meudon_Bin_NS thorn. * VolumeIntegrals_GRMHD-new: Performs volume integrals on arbitrary "Swiss-cheese"-like topologies, and even interoperates with Carpet to track NS centers of mass. * VolumeIntegrals_vacuum-new: Ensures that VI_vacuum can be used without enabling a GRMHD code. * particle_tracerET-new: Solves the ODE D_t xi = vi for typically thousands of tracer particles, using an RK4 integration atop the current timestepping. * smallbPoynET-new: Computes b^i, b^2, and three spatial components of Poynting flux. It also computes (-1-u0), which is useful for tracking unbound matter === How to upgrade from Hack (ET_2017_06) === To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version. See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions. === Machine notes === Supported (tested) machines include: - Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (Homebrew and MacPorts) installations - Bluewaters - Comet - Cori - Draco - Edison - Golub - Hydra - Marconi - Minerva - Queenbee 2 - Stampede 2 - Supermike II - Wheeler * TACC machines: defs.local.ini needs to have sourcebasedir = $WORK and basedir = $SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine. * A new configuration for KNL nodes is being worked on, but not yet included in the release (but compilation works and tests mostly pass). All repositories participating in this release carry a branch ET_2018_02 marking this release. These release branches will be updated if severe errors are found. The "Tesla" Release Team on behalf of the Einstein Toolkit Consortium (2018-02-15) Steven R. Brandt Peter Diener Roland Haas Ian Hinder Feb, 2018

1 0

ETK Workshop Dates June 18 - June 20
by Ferguson, Deborah L 08 Feb '18

08 Feb '18

Dear all, The Einstein Toolkit Workshop will be held at Georgia Tech this year from June 18 to June 20. Thank you to everyone who participated in the the poll and helped select the dates. We will send information about registering for the workshop within the next couple of weeks. Best, Deborah ----------------------------------------- Deborah Ferguson Graduate Student Georgia Institute of Technology Center for Relativistic Astrophysics Boggs 1-66

1 0

setting up an unsupported cluster
by Eric West 08 Feb '18

08 Feb '18

Hi All, I am trying setup ET on a new cluster (hosted by the Minnesota Supercomputing Institute at U of Minnesota, Twin Cities). I have been using the mdb file bluewaters.ini as a template, replacing bluewaters specs with my specs where necessary. I've attached my mdb file for reference. I am able to build simfactory using the --machine=mesabi flag. I am able to run the testsuite with no failed tests (although several "unrunnable" tests, which I am assuming is ok?). I can submit jobs, and they seem to run to completion just fine. However, every time I log in, I am on a different login node, which triggers an "unknown machine name" error, unless I have previously built simfactory on that particular node. I have tried to mimick bluewaters' aliaspattern line in hopes that it would do the trick. But I must be doing something wrong. What do I need to include in my mdb file to force the system to recognize that all of the login nodes are on the same machine? For a bit more background: MSI uses a two step login process. First you ssh into a login machine. Then you ssh from there into one of the clusters. The machine I eventually reach is named mesabi, and the login hosts are named ln000[1-6]. Any help is greatly appreciated. Thanks, Eric -- Eric J West Assistant Professor Department of Physics and Astronomy University of Minnesota Duluth

5 5

Release Meeting, Wednesday, 2/5/2018, 9AM Central Time
by Steven R. Brandt 05 Feb '18

05 Feb '18

The next (and possibly final) release meeting for Tesla will be Wednesday, 2/5/2018, 9AM Central Time. Please call in to https://hangouts.google.com/hangouts/_/stevenrbrandt.com/etcall if you are interested. --Steve

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users February 2018