Users June 2016

users@lists.einsteintoolkit.org

18 participants
30 discussions

Thoughts on external libraries
by Erik Schnetter 10 Jun '16

10 Jun '16

The Einstein Toolkit is a complex piece of software. Almost by definition, and certainly by design, the Cactus framework chooses to build (and thus depend) upon other software, rather than re-inventing the wheel. Thus we have to deal with linking against external software, such as MPI and HDF5, and a long rat's tail of others (BLAS, FFTW, GSL, hwloc, LAPACK, OpenSSL, PAPI, PETSc, zlib). It is an unfortunate reality of current HPC programming that using libraries remains a very difficult task. Our current way of handling these in Cactus is two-fold: (1) there is a flexible mechanism to find, check for, or manually specify the location of external libraries (2) external libraries can be bundled in a Cactus thorn, and be built automatically (3) Cactus can choose automatically between these two I claim that the current mechanism does not work well, for at least the following reasons: - if Cactus builds an external library, it does so too often (once per configuration or executable, not once per system or compiler version) - the current convenience mechanism it overly complex, in the sense that people don't understand any more under what conditions an external library is built; Cactus does things differently from what people expect, and errors go unnoticed - bundling source code for external libraries makes for large thorns, even if it is not used - the current implementation is complex and difficult to maintain, for a variety of reasons that have been extensively discussed - there are several other (minor) problems that are nonetheless difficult to solve Furthermore, people have different preferences regarding building or using external libraries, ranging from making it simple for new users, to reducing download time/disk space, reducing build time, to obtaining the best possible performance. These conflicting goals (all valid!) make it difficult to work towards a better solution. To solve this problem, I make the following radical suggestion: >>> We stop building external libraries as part of building a Cactus executable. <<< It was a good and promising idea, but it failed to deliver; it is causing more problems than it solves. In the new system, there will be two kinds of external libraries: - like MPI or HDF5: an external library must be present that Cactus uses - like LORENE: we always build our own (more on this below) Clearly, we still must make it easy to install a missing external library. Building PETSc is not easy, and then pointing Cactus to a self-built PETSc is not easy either. There are various options to build an external library outside Cactus: (1) apt-get install or similar (for local machines) (2) module/softenv (on many HPC systems) (3) the ET maintainers build it, and make it publicly available, on a set of "important" machines (4) manual build (configure; make; make install) -- often not difficult, if brief (ten lines) of instructions are given (5) we provide thorns and parameter files that automate this, as in "exe/cactus_sim build-hdf5.par"; these would need to be run explicitly once per user on every system (6) using a professional package manager that doesn't require root access, such as Spack <https://github.com/LLNL/spack> Regarding thorns like LORENE: We should switch from distributing the LORENE source code as tarball to keeping the unpacked source code in the thorn, with a set of patches applied. In this way, LORENE can be built as regular Cactus thorn, together with other Cactus source code, and will be linked in a natural way -- no configuration magic necessary at all. All we need is a makefile that lists the LORENE source files. -erik -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

3 3

Resolving two BH
by rahul kashyap 08 Jun '16

08 Jun '16

Hello Everyone, I'm working on binary black hole mergers with large mass ratio (20:1 or more). By using higher refinement level, I have been able to resolve the smaller BH, but it doesn't remain resolved even for a single time step. So far, I have tried changing the CarpetRegrid2::regrid_every and AHFinderDirect::find_every to as small as 1. Could someone please help me where I'm going wrong? I'm sure there are other problems too with my parameter file. I have attached the par file if anyone would be kind enough to give me further suggestions for this problem. Thanks, Rahul

5 7

How to install and run the ET....
by Steven R. Brandt 07 Jun '16

07 Jun '16

step 1: Install "docker" via yum, apt-get, whatever your OS uses. step 2: Make sure you're in the docker group (i.e. run "usermod -aG docker <user>" as root). Once this is done, open a new shell. step 3: Start the docker server. On Fedora, issue "service start docker" as root. On SUSE, it's "systemctl docker start". step 4: As yourself, run "docker pull stevenrbrandt/et" This will consume about 4GB of disk. step 5: As yourself, run "docker run -i -t stevenrbrandt/et" At this point you will find yourself logged in to the docker vm and inside the Cactus directory with a successfully compiled ET. You can run the testsuite, or whatever else you want to do. When you log out, by default changes to the VM are discarded. Cheers, Steve

2 1

Updated test results
by Ian Hinder 07 Jun '16

07 Jun '16

Hi, Following the ET call, I fixed a problem in the pipeline which processes the test results. The updated results are available at the usual place (https://build.barrywardell.net/view/EinsteinToolkitMulti/job/EinsteinToolki…). -- Ian Hinder http://members.aei.mpg.de/ianhin

1 1

Re: [Users] [ET Trac] [Einstein Toolkit] #1908: Cactus: don't print schedule tree by default
by Erik Schnetter 06 Jun '16

06 Jun '16

My suggestion: AFTER. The release, disable the schedule output to stdout by default. Instead, output it to a file by default. That file could be a log file capturing all level 3 warning output, so that we don't end up with too many files. -erik On Monday, June 6, 2016, Einstein Toolkit <trac-noreply(a)einsteintoolkit.org> wrote: > #1908: Cactus: don't print schedule tree by default > --------------------------+------------------------------------------------- > Reporter: knarf | Owner: > Type: enhancement | Status: new > Priority: optional | Milestone: > Component: Cactus | Version: development version > Resolution: | Keywords: > --------------------------+------------------------------------------------- > > Comment (by knarf): > > I agree that some 'phrases' might be nice. I also agree that the > scheduling tree isn't the only information you hardly need when running > simulations, or building executables. We have to do something to cut > output like this down to a minimum that is useful for most people, along > with the possibility to easily enable more output when needed. I've chosen > the schedule tree because of it's sheer length in terms of lines, and for > the not-very-invasive nature of the change (just the default value of an > already existing parameter). > > What I would like to avoid is that this ticket becomes a collection of > wishes for very different things, a kind of über-ticket for cutting down > on the output while building/running. We could open a ticket like that, > and link this ticket there, if this is felt to be helpful. I would like to > concentrate here on the default for printing the scheduling tree: should > this be 'yes', or 'no'? > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1908#comment:2> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 06 Jun '16

06 Jun '16

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with the URL: https://hangouts.google.com/call/5xt7voshalqmdcztr4zhfeox2ia Frank Löffler

2 1

meeting minutes for 2016-05-16
by Roland Haas 02 Jun '16

02 Jun '16

Present: Erik, Frank, Barry, Ian, Matt, Yosef, Zach, Steve, Eloisa LIGO September event simulation * Barry, Eloisa, and Ian have set up a parameter file for the GW event that LIGO detected in September * the simulation is short since the event was high masss * there is a webpage and tutorial visualization using VisIt available * provide full 3d output which is ~1TB and lightweight data which is about 500GB * an extensive Mathemtatica Notebook using SimulationTools [http://simulationtools.org/] is provided showing how to do all sorts of analysis for the simulation * restructured gallery page to have little headings about each example Feedback: * add pointer to LIGO publications eg the PRL article * point out scaling of results * get DOI or simimlar for this, plan to have "proper" publication as well, want to also publicize it in talks and to communities that do not yet produce full GR simulations ordinarily eg the physics today magazine, conceivably also include in LIGO data releases * try provide data online suggest eg NCSA's data services * suggest to make plots and movies available for download ET release: * hwloc https://trac.einsteintoolkit.org/ticket/1717 * HDF5 https://trac.einsteintoolkit.org/ticket/1851 suggest to apply https://gist.github.com/eschnett/baa8b20fd64eb524c42f https://trac.einsteintoolkit.org/raw-attachment/ticket/1851/0002-detect.sh-… and https://trac.einsteintoolkit.org/raw-attachment/ticket/1851/0001-detect.sh-… Roland to review * https://trac.einsteintoolkit.org/ticket/1853 accept revert on master for release and fix diagonal output, move into feature branch, on feature branch will merge the diagonal part of the old code and the non-diagonal code of the new code * piraha error messages https://trac.einsteintoolkit.org/ticket/1881 accept current solution * AHFinderDirect https://trac.einsteintoolkit.org/ticket/1880 Roland will commit new data, many of the now failing tests fail due to extra lines of output that was not found before * create wiki page with outstanding issues and progress documenting why each item is fixed/not fixed * collect suggestions for new release name Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

3 7

Postdoctoral position in theoretical astrophysics
by Agnieszka Janiuk 02 Jun '16

02 Jun '16

Dear users, There will be a postdoctoral position starting this fall in my group in Warsaw. Job advertisement is now on AAS site: https://jobregister.aas.org/job_view?JobID=54165 please feel free to distribute this information and contact me further if interested. best regards, Agnieszka -- dr hab. Agnieszka Janiuk (Associate Professor) Center for Theoretical Physics Polish Academy of Sciences Al. Lotnikow 32/46, 02-668 Warsaw, Poland Tel: (+48) 22 1162006 Tel: (+48) 22 8470920 Fax: (+48) 22 8431369 email: agnes(a)cft.edu.pl www: http://www.cft.edu.pl/p_view/p_view.php?login=agnes

1 0

Test failures on OS X
by Ian Hinder 01 Jun '16

01 Jun '16

Hi, I have run the ET tests on OS X. There are 5 failures (https://build.barrywardell.net/view/EinsteinToolkitMulti/job/EinsteinToolki…): IOHDF5.test_recover/1procs SphericalHarmonicReconGen.SpEC-h5-test/1procs CT_MultiLevel.boostedpuncture/2procs IOHDF5.test_recover/2procs SphericalHarmonicReconGen.SpEC-h5-test/2procs and also when running the test EinsteinInitialData/Exact/test/Schwarzschild_EF.par on one process on OS X, the test hung after running. The Cactus process was a zombie, and mpirun was waiting for it to finish. I killed the mpirun process, and everything went smoothly after that. Several of the test failures seem to be related to HDF5 (see the output pages from the link above). For example, I get these warnings: HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0: #000: H5Dio.c line 170 in H5Dread(): can't read data major: Dataset minor: Read failed #001: H5Dio.c line 418 in H5D__read(): unable to set up type info major: Dataset minor: Unable to initialize object #002: H5Dio.c line 953 in H5D__typeinfo_init(): not a datatype major: Invalid arguments to routine minor: Inappropriate type WARNING[L1,P0] (IOHDF5Util): HDF5 call 'H5Dread (dataset, rec_info->hdf5type, H5S_ALL, H5S_ALL, H5P_DEFAULT, data)' returned error code -1 and the diffs say phi.xl: substantial differences significant differences on 165 (out of 165) lines maximum absolute difference in column 2 is 1.4849088247045 maximum relative difference in column 2 is 3.18869248642747 for example, in IOHDF5.test_recover/1procs. This might be because MacPorts is now using HDF5 1.10.0, and I don't know how much this has been tested with Cactus. It introduces a format change in HDF5, but the new version is supposed to be able to read files written by old versions, so that shouldn't be an issue here. Erik, have you run the tests on any system with HDF5 1.10.0? I'm also a bit concerned that Cactus didn't abort after those HDF5 errors. -- Ian Hinder http://members.aei.mpg.de/ianhin

4 6

status release
by Frank Loeffler 31 May '16

31 May '16

Hi, I would like to give you a little update about the release. All testsuites pass now (on Jenkins). For the most part, this was 'fixing the testsuites', rather than fixing actual code. Some tickets still need to be addressed, and/or need more input: #1897 [1]: external library build issues #1853 [2]: diagonal Carpet output. The troubling patch will be moved to a feature branch and be reverted from both release and master until fixed. #1850 [4]: Stampede performance problem (reviewed ok, but not resolved) Would-be-nice tickets that potentially could make it into the release include: #1881 [3]: improve piraha error messages (needs review, and a decision about inclusion in the release, which I'll leave to the reviewer). #1894 [5]: TwoPuncture potential segfault fix (reviewed ok, but not pushed) I propose we'll use today (and tomorrow for people in the EU) to fix all of those. I'll create release branches after that, hopefully tomorrow morning (US time). I'll then send around a request to start testing as heavily as you can/dare. If you do start working on one of those tickets, leave a note here: https://trac.einsteintoolkit.org/ticket/1895 to avoid duplicated work. What is the current status of the re-worked gallery? Frank Loeffler [1] https://trac.einsteintoolkit.org/ticket/1897 [2] https://trac.einsteintoolkit.org/ticket/1853 [3] https://trac.einsteintoolkit.org/ticket/1881 [4] https://trac.einsteintoolkit.org/ticket/1850 [5] https://trac.einsteintoolkit.org/ticket/1894

4 8

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users June 2016