Users June 2014

users@lists.einsteintoolkit.org

8 participants
21 discussions

Einstein Toolkit Meeting Reminder
by Frank Loeffler 14 May '15

14 May '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. My agenda contains: - Fall workshop - PLEASE let us know if you come [1]! - Hotel booking - Release planning - release date: Mon Oct 24 2011 - freeze date: Mon Sep 26 - the day of the call! - branch name: ET_2011_10 - List of release-concerning bugs [2] - ET paper: proposed "freeze" by Oct 3rd (Mon after this call) As always: feel free to add to this list. Frank Loeffler [1] https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Fall_2011#Fall_Einstei… [2] https://trac.einsteintoolkit.org/query?status=!closed&milestone=ET_2011_11&…

10 31

PARAMCHECK and non fatal but inefficient parameter choices
by Roland Haas 03 Oct '14

03 Oct '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, the PARAMCHECK routines normally abort if inconsistent parameters are given. I have a situation where the parameter choice is valid and the result of the simulation should be identical with or without the parameter set (assuming not bugs...), however setting the parameter is likely going to make the simulation slower. Im this particular case the parameter is CarpetLib::use_higher_order_restriction when used for a run with Carpet::refinement_centering = "vertex". In that case restriction is just a copy operation and independent of use_higher_order_restriction, for cell-centered runs this is not true anymore and this is the situation that use_higher_order_restriction was implemented for. However there is code in Carpet/Evolve.cc that calls POSTREGRID multiple times if use_higher_order_restriction is set (it has to in order to get symmetry, outer boundaries and also ghost zones right). For vertex centering nothing should happen (since there is a second call to POSTRESTRICT after all restriction is done). Options are: * do nothing since these parameters are perfectly valid if slow * warn with level CCTK_WARN_COMPLAIN ("the user should know about this but the problem is not terribly surprising") * warn with level CCTK_WARN_ALERT ("the results of this run will be wrong, and this will surprise the user, but we can still continue the run") * abort since the user most likely did not want to set the parameter Note that current practise in most code seems to be to use CCTK_WARN_COMPLAIN (Level 1) for warnings about situations that do not invalidate the result and CCTK_WARN_ABORT (Level 0) for anything that invalidates results, ie. we do not adhere to the meaning of the levels documented in src/include/cctk_WarnLevel.h . Yours, Roland - -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlNqrvQACgkQTiFSTN7SboVuBQCg0IhvBi92zRKRVKoe216Q6wND NEYAn25pqT33i90JbhXmBJ90jKNAP4wd =+2Fk -----END PGP SIGNATURE-----

3 4

Stampede
by Ian Hinder 07 Aug '14

07 Aug '14

Hi all, Has anyone run into problems recently with Cactus jobs on Stampede? I've had jobs die when checkpointing, and also mysteriously hanging for no apparent reason. These might be separate problems. The checkpointing issue occurred when I submitted several jobs and they all started checkpointing at the same time after 3 hours. The hang happened after a few hours of evolution, with GDB reporting > MPIDI_CH3I_MRAILI_Get_next_vbuf (vc_ptr=0x7fff00d9a8d8, vbuf_ptr=0x13) > at src/mpid/ch3/channels/mrail/src/gen2/ibv_channel_manager.c:296 > 296 for (; i < mv2_MPIDI_CH3I_RDMA_Process.polling_group_size; > ++i) Unfortunately I didn't ask for a backtrace. I'm using mvapich2. I've been in touch with support and they said the dying while checkpointing coincided with the filesystems being hit hard by my jobs, which makes sense, but they didn't see any problems in their logs, and they have no idea about the mysterious hang. I repeated the hanging job and it ran fine. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

5 13

Notes from the Einstein Toolkit Call 2014-06-30
by Erik Schnetter 30 Jun '14

30 Jun '14

Einstein Toolkit Call 2014-06-40 Participants: Erik Schnetter, Frank Löffler, Ian Hinder, Roland Haas, Steve Brandt, Yosef Zlochower Notes: Tickets: - hwloc (please review patch) - GetComponents (please review patch) - parameter parsing (please review patch) Illinois GRMHD: - need to integrate better with EinsteinBase Transition to git: - Frank created mappings from user names to emails, sent to Ian - will check out flesh into "repos" subdirectory - may want to convert CactusBase first, and the flesh later Time level handling and checkpointing: - still possible to create a checkpoint that cannot be recovered - problems is the time hierarchy (th class): grid variables can have more time levels than th - note: there is a parameter max_timelevels that can be set to prevent this - one solution: check during storage allocation whether max_timelevels is set correctly - Roland to create ticket describing discussed solution - MoL should maybe be changed to require fewer time levels; Ian to create ticket for this Automated tests: - will need to update from Ubuntu 12.04 to 14.04 -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu/.

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 30 Jun '14

30 Jun '14

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. Some tickets within the last week that are still open: - 1635: hwloc requiring a library version but not checking for it - 1396: GetComponents and displaying promts - 1518: parameter and expression parsing in Cactus Other projects include - Make Illinois code work with standard - Carpet - ADMBase - HydroBase - git transition As always, please add to this list if you like. Frank Loeffler

2 1

Carpet::max_timelevels not steerable
by Roland Haas 27 Jun '14

27 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, I am trying to recover a checkpoint that was written with the attached parfile and get errors - --8<-- cactus_bns_all: /mnt/data/rhaas/postdoc/gr/Zelmani/arrangements/Carpet/CarpetLib/src/th.hh:79: CCTK_REAL8 th::get_time(int, int, int) const: Assertion `tl>=0 and tl<timelevels' failed. Rank 0 with PID 31684 received signal 6 Writing backtrace to tov-mol/backtrace.0.txt - --8<-- and indeed the checkpoint contains - --8<-- ML_ADMCONSTRAINTS::H\ it=0\ tl=2\ rl=0\ c=0 Dataset {21, 21, 36} - --8<-- Setting carpet::max_timelevels when recovering however does not have the desired effect since the parameter is not steerable. There was a lengthy discussion as to how best fix this issue of not being able to recover checkpoints when the parameter file was "funny" in https://trac.einsteintoolkit.org/ticket/626 (yes, that one :-) ). My question now is: why is carpet::max_timelevels not steerable? It seems to me as if the most common use will be exactly to make a checkpoint recoverable. Was this just an oversight? Ideally no such parameter should be needed. Either CarpetIOHDF5 should ignore (or: fail with a meaningful error when it encounters them) timelevels in checkpoint files that the current simulation cannot handle or Carpet should be able to cope with the maximum possible number of timelevels (determined to be the maximum over all groups of CCTK_MaxTimeLevels()), I think. Yours, Roland PS The attached parfile is a proxy for a real world parfile where we attempt to do something similar (but the real world one runs on many cores only), so just disabling ML_ADMCONSTRAINTS is not an option. - -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlOtsEwACgkQTiFSTN7SboWd2ACfUluOYaUdMQ329HG2GVV+RzTe cgcAn0Xdk+o2LXXk9DFO5EjcVCVpBI5e =VT73 -----END PGP SIGNATURE-----

1 0

ET_2014_05 fails to build on Linux Mint 17
by Dumsani Ndzinisa 26 Jun '14

26 Jun '14

Hi everyone, I have a fresh checkout of the latest stable release of the Einstein Toolkit (ET_2014_05) which I'm trying to build on my laptop. The laptop is running on Linux Mint 17, and I have adapted configuration options from the bundled "ubuntu.cfg" file. On my machine, I have gcc version 4.8.2, g++ version 4.8.2, and gfortran 4.8.2 as well. However, when building the toolkit (using the thornlist "einsteintoolkit.th) the build always fails no matter what I try. For instance, on my very first attempt, I didn't use any of my locally installed libraries but opted for the ones bundled with the toolkit. There was one error message in that case pointing to the PAPI library having failed to get configured. I then decided to comment out this thorn from the thornlist (in the .th file). From there on, still building with the bundled external libraries, I have been persistently getting an error message (in the linking stage) to the effect a certain library (lnuma) could not be found. Below is the error message that gets returned onto the screen: /usr/bin/ld: cannot find -lnuma collect2: error: ld returned 1 exit status /home/dumsani/Cactus/lib/make/make.configuration:147: recipe for target '/home/dumsani/Cactus/exe/cactus_test-ET6' failed make[1]: *** [/home/dumsani/Cactus/exe/cactus_test-ET6] Error 1 Makefile:254: recipe for target 'test-ET6' failed make: *** [test-ET6] Error 2 I would appreciate if anyone with some idea came to my rescue here. Cactus and the ET are still very new to me. I've been stuck on this for almost a week now. And I am of the idea that it shouldn't be taking me that long. Together with my advisor, we have made a checkout of Cactus (Llama thornlist) earlier and he did oriente me to the process until we had a successful build (we got the executable) for the llama thornlist. So, now the idea is for me to get some more practice with Cactus/ET by checking out the full toolkit, configuring, building and then running some example simulations provided in the toolkit. Thank you in advance for your help.

3 7

Editing .cfg file for a machine to use locally installed external libraries (Simfactory, optionlist)
by Dumsani Ndzinisa 26 Jun '14

26 Jun '14

Hi everyone, I have been able to successfully build an Einstein toolkit simulation using the latest release of the toolkit (ET_2014_05). I am particularly following the 2013 notes by Miguel Zilhao and Frank Loeffler, "An Introduction to the Einstein Toolkit" (arXiv:1305:5299v2 [gr-qc] ). In my build I have used the bundled Cactus libraries for almost all required external libraries. I appreciate this as an entry level step for a beginner like myself. But, as advised in the notes as well, making use of locally installed copies of these libraries may speed up the overall building process somehow - something which I will definitely need moving forward. However, my attempt at this just proved to be completely incorrect. The linker vehemently complained about not being able to find blas, lapack, mpi, etc.. I thought this could only mean one thing - that I did not enter the directory paths (for the libraries and the header files to be "included") and library names correctly. For instance, in my .cfg file, this is what I had entered for the BLAS library BLAS_DIR = NO_BUILD BLAS_LIB_DIRS = /usr/lib/libblas BLAS_INC_DIRS = /usr/include BLAS_LIBS = libblas3 libblas-dev It turned out that the linker could not proceed well with this. I can attach my entire cfg file for anyone who might wish to view it in order to see where I'm going wrong and provide suggestions. Under the fields such as BLAS_LIB_DIRS and BLAS_INC_DIRS, I have been using results returned by the commalnd " locate <package/library name>". Of course, this command returns a plethora of directory/file locations where there are occurrences of <package/library name>. I would then choose those starting with /usr/... Then for fields like BLAS_LIBS, I used output returned by the command "dpkg --get-selections | grep <package/library name> ". However, sometimes this command returns quite a number of things and I would get a bit confused about what to "choose" and leave out as the the name of the library(-ies). This, for example, has been the case with my MPI since I have installed both OPENMPI and MPICH. How exactly should this information be entered in the .cfg file such that the build uses the locally installed external libraries? Thank you in advance for your help.

1 0

Re: [Users] ET_2014_05 fails to build on Linux Mint 17
by Dumsani Ndzinisa 26 Jun '14

26 Jun '14

Hi Ian, Ah I'm sorry I didn't realize that my replying wasn't directed to the group. I guess this all has to do with me being very on almost everything here! I will append my previous response here. Many thanks to you!!! Commenting out hwloc from the thornlist has done the trick. My ET has compiled successfully now. Being the newbie that I am, please do expect me to get back again and bother you will plenty of elementary questions. Thanks again. My previous response which was communicated off the group...here....... > Hi Ian, > > Yes, I do have a local installation of the hwloc library. And I think > you've really hit the > nail on the head in your diagnosis. When I went through the make file, > and also looked > at the error message my understanding was that Cactus is trying to > bring in numa > when linking, and it looks for it in the wrong place. Running the command > dpkg --get-selections | grep hwloc, this is the output I get: > hwloc-nox install > libhwloc-dev:amd64 install > libhwloc5:amd64 install > > I'll implement the suggestions you gave to me, try to build again and > get back here > with some feedback. On 26/06/2014 12:53, Ian Hinder wrote: > > On 26 Jun 2014, at 12:51, Dumsani Ndzinisa <g14n8326(a)campus.ru.ac.za > <mailto:g14n8326@campus.ru.ac.za>> wrote: > >> Hi Ian, >> >> Yes, I do have a local installation of the hwloc library. And I think >> you've really hit the >> nail on the head in your diagnosis. When I went through the make >> file, and also looked >> at the error message my understanding was that Cactus is trying to >> bring in numa >> when linking, and it looks for it in the wrong place. Running the command >> dpkg --get-selections | grep hwloc, this is the output I get: >> hwloc-nox install >> libhwloc-dev:amd64 install >> libhwloc5:amd64 install >> >> I'll implement the suggestions you gave to me, try to build again and >> get back here >> with some feedback. > > Hi Dumsani, > > Glad to be of help! By the way, you replied just to me, not to the > mailing list. > >> >> >> On 26/06/2014 12:22, Ian Hinder wrote: >>> On 26 Jun 2014, at 11:56, Dumsani Ndzinisa <g14n8326(a)campus.ru.ac.za >>> <mailto:g14n8326@campus.ru.ac.za>> wrote: >>> >>>> Hi everyone, >>>> >>>> I have a fresh checkout of the latest stable release of the >>>> Einstein Toolkit >>>> (ET_2014_05) which I'm trying to build on my laptop. The laptop is >>>> running on Linux Mint 17, and I have adapted configuration options from >>>> the bundled "ubuntu.cfg" file. On my machine, I have gcc version 4.8.2, >>>> g++ version 4.8.2, and gfortran 4.8.2 as well. >>>> >>>> However, when building the toolkit (using the thornlist >>>> "einsteintoolkit.th <http://einsteintoolkit.th>) >>>> the build always fails no matter what I try. For instance, on my >>>> very first >>>> attempt, I didn't use any of my locally installed libraries but opted >>>> for the >>>> ones bundled with the toolkit. There was one error message in that case >>>> pointing to the PAPI library having failed to get configured. I >>>> then decided >>>> to comment out this thorn from the thornlist (in the .th file). >>>> From there >>>> on, still building with the bundled external libraries, I have been >>>> persistently >>>> getting an error message (in the linking stage) to the effect a certain >>>> library (lnuma) could not be found. Below is the error message that >>>> gets >>>> returned onto the screen: >>>> >>>> /usr/bin/ld: cannot find -lnuma >>>> collect2: error: ld returned 1 exit status >>>> /home/dumsani/Cactus/lib/make/make.configuration:147: recipe for target >>>> '/home/dumsani/Cactus/exe/cactus_test-ET6' failed >>>> make[1]: *** [/home/dumsani/Cactus/exe/cactus_test-ET6] Error 1 >>>> Makefile:254: recipe for target 'test-ET6' failed >>>> make: *** [test-ET6] Error 2 >>>> >>>> I would appreciate if anyone with some idea came to my rescue here. >>>> Cactus and the ET are still very new to me. I've been stuck on this for >>>> almost a week now. And I am of the idea that it shouldn't be taking me >>>> that long. Together with my advisor, we have made a checkout of Cactus >>>> (Llama thornlist) earlier and he did oriente me to the process until we >>>> had a successful build (we got the executable) for the llama >>>> thornlist. So, >>>> now the idea is for me to get some more practice with Cactus/ET by >>>> checking out the full toolkit, configuring, building and then >>>> running some >>>> example simulations provided in the toolkit. >>> Hi, >>> >>> Can you try commenting out the thorn "hwloc" from your thornlist? >>> Grepping through the source for "numa", the only use of this >>> library seems to be hwloc. hwloc is a thorn which assists in >>> binding software threads to physical computational cores and >>> processes to processor sockets for efficiency, but it is not >>> necessary for the toolkit to run. >>> >>> There might be something wrong with the hwloc configure script >>> (Cactus/arrangements/ExternalLibraries/hwloc/configure.sh). The >>> relevant lines appear to be >>> >>>> # Add libnuma manually, if necessary >>>> if grep -q '[-]lnuma' ${HWLOC_LIB_DIR}/libhwloc.la 2>/dev/null; then >>>> if ! echo '' ${HWLOC_LIBS} '' | grep -q ' numa '; then >>>> HWLOC_LIBS="${HWLOC_LIBS} numa" >>>> fi >>>> fi >>> This seems to be attempting to add the numa library to the link line >>> if it is found in libhwloc.la. The error message indicates that >>> numa has been added to the link line, but the library is not >>> available on the link path. >>> >>> The most recent change to hwloc, in April (i.e. before the release), is: >>> >>> "Correct detecting whether -lnuma is necessary" >>> (http://git.barrywardell.net/?p=arrangements/ExternalLibraries/hwloc.git;a=c…) >>> authoreschnett <eschnett@152c557c-6d84-4bc7-9b1a-dfca721279c7> >>> Sat, 19 Apr 2014 23:07:32 +0200 (21:07 +0000) >>> committereschnett <eschnett@152c557c-6d84-4bc7-9b1a-dfca721279c7> >>> Sat, 19 Apr 2014 23:07:32 +0200 (21:07 +0000) >>> >>> Probably Erik has some insight into this. >>> >>> We have an automated build and test process running under Ubuntu >>> 12.04, and this is working fine. It also builds hwloc from source, >>> rather than relying on the Ubuntu version. Do you happen to have >>> hwloc installed? Maybe it is conflicting with the self-built version? >>> >>> dpkg -l hwloc >>> >>> We should probably upgrade the build and test system to Ubuntu >>> 14.04, as that is now the latest stable release. Maybe something in >>> Ubuntu has changed with the hwloc library. >>> >> > > -- > Ian Hinder > http://numrel.aei.mpg.de/people/hinder >

1 1

Einstein Toolkit Meeting Reminder
by Frank Loeffler 23 Jun '14

23 Jun '14

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. Some tickets within the last two weeks that are still open: - 1630: ML_BSSN harmonic shift is ill-posed - 1631: dgfe branch of McLachlan generats NaNs - 1633: backport MoL revision 225 (to release) - 1632: problem with compilation Other projects include - Make Illinois code work with standard Carpet - git transition: Ian & Frank chatted, but no conversion was done so far. As always, please add to this list if you like. Frank Loeffler

1 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users June 2014