Users August 2015

users@lists.einsteintoolkit.org

18 participants
30 discussions

Benchmarking results for McLachlan rewrite
by Erik Schnetter 17 Aug '15

17 Aug '15

I ran the Simfactory benchmark for ML_BSSN on both the current version and the "rewrite" branch to see whether this branch is ready for production use. I ran this benchmark on a single node of Shelob at LSU. In both cases, using 2 OpenMP threads and 8 MPI processes per node was fastest, so I am reporting these results below. Since I was interested in the performance of McLachlan, this is a unigrid vacuum benchmark using fourth order differencing. One noteworthy difference is that dissipation as implemented in the "rewrite" branch is finally approximately as fast as thorn Dissipation, and I have thus used this option for the "rewrite" branch. Here are the high-level results: current: 3.03136e-06 sec per grid point rewrite: 2.85734e-06 sec per grid point That is, the rewrite branch is about 5% faster. More detailed timing results for CCTK_EVOL are as follows: Current: ---------------------------------------------------------------------------------------- Time Time Imblnc Timer gettimeof getrusage percent secs percent secs secs ---------------------------------------------------------------------------------------- 97.8% 373.1 0.0% |_CallEvol 373.1 738.5 97.8% 373.1 0.0% | |_CCTK_EVOL 373.1 738.4 97.8% 373.0 0.0% | | |_CallFunction 373 738.3 10.3% 39.3 36.9% | | | |_syncs 62.27 122.6 10.3% 39.3 36.9% | | | | |_Sync 62.25 122.6 2.1% 8.0 18.6% | | | | | |_comm_state[3].state_fill_sen 9.839 19.49 6.1% 23.2 46.3% | | | | | |_comm_state[6].state_do_some_ 43.18 84.73 1.5% 5.5 1.8% | | | | | |_comm_state[7].state_empty_re 5.61 10.84 87.3% 333.2 6.4% | | | |_thorns 310.3 614.5 21.8% 83.0 3.2% | | | | |_ML_BSSN_Advect 84.45 168.8 8.6% 32.9 51.3% | | | | |_ML_BSSN_NewRad 0.0354 0.05899 14.1% 53.6 1.9% | | | | |_ML_BSSN_RHS1 54.68 109.1 16.0% 61.1 2.0% | | | | |_ML_BSSN_RHS2 62.35 124.5 5.1% 19.4 9.0% | | | | |_ML_BSSN_convertToADMBase 18.63 37.17 4.2% 16.1 3.1% | | | | |_ML_BSSN_convertToADMBaseDtLaps 16.11 31.98 2.4% 9.1 20.0% | | | | |_ML_BSSN_enforce 11.34 22.65 8.3% 31.7 15.3% | | | | |_MoL_Add 33.12 62.48 1.0% 3.8 48.9% | | | | |_ReflectionSymmetry_Apply 7.461 14.93 5.2% 19.8 3.1% | | | | |_dissipation_add 20.47 40.87 Rewrite: -------------------------------------------------------------------------------------------------- Time Time Imblnc Timer gettimeof getrusage percent secs percent secs secs -------------------------------------------------------------------------------------------------- 99.0% 356.1 0.0% |_CallEvol 356.1 703.8 98.9% 356.0 0.0% | |_CCTK_EVOL 356.1 703.8 98.9% 356.0 0.0% | | |_CallFunction 356 703.7 14.0% 50.4 37.3% | | | |_syncs 80.43 158.2 14.0% 50.4 37.3% | | | | |_Sync 80.42 158.2 2.1% 7.7 7.7% | | | | | |_comm_state[3].state_fill_send_buffers. 8.341 16.51 9.7% 35.0 45.2% | | | | | |_comm_state[6].state_do_some_work.step 63.77 125.2 1.5% 5.6 2.5% | | | | | |_comm_state[7].state_empty_recv_buffers 5.629 10.87 84.8% 305.1 9.9% | | | |_thorns 275.2 544.4 5.5% 19.8 7.1% | | | | |_ML_BSSN_ADMBaseEverywhere 19.58 39.15 4.4% 16.0 5.3% | | | | |_ML_BSSN_ADMBaseInterior 15.91 31.63 2.3% 8.3 16.6% | | | | |_ML_BSSN_EnforceEverywhere 10.01 20.02 10.8% 39.0 5.0% | | | | |_ML_BSSN_EvolutionInteriorSplitBy1 38.75 77.28 21.2% 76.4 4.6% | | | | |_ML_BSSN_EvolutionInteriorSplitBy2 75.82 151.5 23.3% 83.9 5.8% | | | | |_ML_BSSN_EvolutionInteriorSplitBy3 83.78 166.8 9.7% 35.0 48.9% | | | | |_ML_BSSN_NewRad 0.1266 0.156 6.0% 21.5 20.7% | | | | |_MoL_Add 23.12 43.16 0.9% 3.3 49.4% | | | | |_ReflectionSymmetry_Apply 6.508 12.97 As can be seen, the non-McLachlan numbers are comparable, although not identical. This is to be expected. The main RHS evaluation is split over several routines in each case; these are (RHS1, RHS2, Advect, dissipation_add) for the current version and (EvolutionInteriorSplitBy[1,2,3]) for the rewrite branch. The way in which the RHS evaluation is actually split is of course different for both cases. With these numbers in hand, I think we are ready to switch to the rewrite branch. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 21

Post-processing frameworks
by Ian Hinder 13 Aug '15

13 Aug '15

Hi, There was a discussion today during the Einstein Toolkit workshop (http://agenda.albanova.se/conferenceDisplay.py?confId=4936) about frameworks for analysing and post-processing simulation data from the Einstein Toolkit. We would like to collect together information about any such frameworks so that people know what is available. Ideally, these should be publicly available. I have created a wiki page (https://docs.einsteintoolkit.org/et-docs/Analysis_and_post-processing) and added very brief descriptions of SimulationTools and PostCactus. If you have such a framework or other tools, it would be great if you could add some information to this wiki page about them. Thanks! -- Ian Hinder http://members.aei.mpg.de/ianhin

2 1

std::out_of_range error while checkpointing
by Miguel Zilhão 12 Aug '15

12 Aug '15

hi all, i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6 when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine. i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same. i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380 so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899 this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash. i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same. has anyone seen such an error before? anything else i could provide to help diagnose this? thanks, Miguel

2 6

problem with par file not finding thorns
by David Bradley 11 Aug '15

11 Aug '15

Greetings I tried to run full-040.par simulation from the examples directory and it failed to find several of the thorns. Error: Thorn ADMConstraints not found Error: Thorn BSSN_MoL not found Error: Thorn LegoExcision not found Activation failed - 3 errors in activation sequence I am not sure how to proceed

2 1

McLachlan help
by James Healy 11 Aug '15

11 Aug '15

Hello, I have been running some tests on Stampede comparing the run speed of McLachlan to RIT's evolution thorn LazEv. I started with the qc0-mclachlan.par parameter file included with the Einstein Toolkit, added a few refinement levels, increased the resolution and changed McLachlan to be 8th order (and increased the number of ghost zones to 5). I also increased the initial separation so the finest grids aren't already overlapping. To compare with LazEv, I removed the McLachlan and Dissipation thorns and replaced them with LazEv. Everything else in the parameter file is exactly the same. I tried using both the McLachlan master and rewrite branches. The grid setup is 10 levels of refinement, dx=4M on the coarsest with outer boundary at 400M, M/128 on the finest with r=0.6M, CFL is 0.25. Both use 8th order spatial differencing with ghost_size=5 and 5th order dissipation. Below is a summary of the results as reported at iteration 256 from Carpet::physical_time_per_hour: McLachlan - rewrite branch: 3.0596110 M/hr McLachlan - master branch: 3.8033607 M/hr LazEv - 4.1941544 M/hr I am using the stampede-impi.cfg configuration file in simfactory. "module list" returns: 1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC 2) Linux 4) intel/13.0.2.146 6) cluster 8) impi/4.1.0.030 Attached is my parameter file. I pasted the McLachlan parameters below. Are there any optimizations that I can use for McLachlan? Are the parameters I am using for it what would be used for production runs? ML_BSSN::harmonicN = 1 # 1+log ML_BSSN::harmonicF = 2.0 # 1+log ML_BSSN::ShiftGammaCoeff = 0.75 ML_BSSN::BetaDriver = 1.0 ML_BSSN::LapseAdvectionCoeff = 1.0 ML_BSSN::ShiftAdvectionCoeff = 1.0 ML_BSSN::MinimumLapse = 1.0e-8 ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas" ML_BSSN::my_rhs_boundary_condition = "NewRad" Boundary::radpower = 2 ML_BSSN::ML_log_confac_bound = "none" ML_BSSN::ML_metric_bound = "none" ML_BSSN::ML_Gamma_bound = "none" ML_BSSN::ML_trace_curv_bound = "none" ML_BSSN::ML_curv_bound = "none" ML_BSSN::ML_lapse_bound = "none" ML_BSSN::ML_dtlapse_bound = "none" ML_BSSN::ML_shift_bound = "none" ML_BSSN::ML_dtshift_bound = "none" ML_BSSN::fdOrder = 8 ActiveThorns = "Dissipation" Dissipation::order = 5 Dissipation::vars = " ML_BSSN::ML_metric ML_BSSN::ML_trace_curv ML_BSSN::ML_curv ML_BSSN::ML_Gamma ML_BSSN::ML_lapse ML_BSSN::ML_shift ML_BSSN::ML_dtlapse ML_BSSN::ML_dtshift " ActiveThorns = "ML_ADMConstraints" Thanks, Jim Healy

4 7

No Einstein Toolkit Meeting
by Roland Haas 10 Aug '15

10 Aug '15

Hi, There will not be an Einstein Toolkit meeting coming Monday. The next meeting will be in about one week on Monday, August 17th, at the usual time. Roland Haas

1 0

debugging kranc .m
by Comer Duncan 05 Aug '15

05 Aug '15

I am working on a kranc file and am getting output errors when I do make. I would like to know if there are any options to turn on more verbose output so I can better see where the problem is arising and hopefully fix them. Currently the make runs as follows: ComerMacProRetina:m comerduncan$ make BrillAnalytic.out \rm -rf BrillAnalytic.out ./runmath.sh BrillAnalytic.m Using Kranc installation at ../../../repos/Kranc/Bin/.. Mathematica 10.0 for Mac OS X x86 (64-bit) Copyright 1988-2014 Wolfram Research, Inc. $RecursionLimit::reclim: Recursion depth of 1024 exceeded. $RecursionLimit::reclim: Recursion depth of 1024 exceeded. $RecursionLimit::reclim: Recursion depth of 1024 exceeded. General::stop: Further output of $RecursionLimit::reclim will be suppressed during this calculation. Creating thorn CT_AnalyticBrillWave Processing tensorial arguments Computing reflection symmetries Creating (component-based) Kranc thorn Processing arguments to CreateKrancThorn Verifying arguments Error: Invalid arguments: lookup[_Rule,_Symbol] in CreateKrancThorn/ParameterDatabase make: *** [BrillAnalytic.out] Error 1 The recursion depth problem is worrisome as is the complaint about invalid arguments. Any ideas would be appreciated. Comer

2 1

postprocessing data in Cactus
by Roland Haas 04 Aug '15

04 Aug '15

Hello all, for the upcoming ET workshop I wanted to grab a 5 minutes slot and drum up support for more advanced postprocessing features in Cactus and to compare what Cactus can do (which is very little I am afraid) to what other codes can do. As far as I know, multiple persons have already written their own specialized tools to do some offline postprocessing but there seems to be no concerted effort to produce a more general tool. So as a starter I wanted to give a short list of existing postprocessing tools that are used (mostly I am going to emphasis postprocessing using 3d and 2d datasets) and wanted to ask if people were to send me a short (a couple of lines) description of posprocessing tools that they use/have written and their capabilities (pictures are also welcome). I'd try and use those as starting points for a discussion at the meeting. We have an informal list of visualization tools (https://docs.einsteintoolkit.org/et-docs/Visualization_of_simulation_results) however I am looking for actual analysis facilities eg. tools that would allow one to compute things like ADM energy or unbound matter fraction based on saved simulation data (so simulationtools rather than VisIt). Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

2 1

meeting minutes for 2015-08-03
by Roland Haas 03 Aug '15

03 Aug '15

Present: Josh, Yosef, Steve, Peter, Matt, Erik, Frank, Roland No call next week due to ET meeting in Stockholm. * Apply clang-format-3.5 to CactusBase and EinsteinBase. * PITTNullCode is mostly solved, patch needs to be applied, test tolerances can then be tightened * IllinoisGRMHD: Josh is using it, Frank is trying it * script to process backtraces, ticket 1798: fine to include in either flesh or Carpet (depending on whether Carpet actively writes the backtrace or not) * rewrite branch of McLachlan: still one open issue in comments, Erik to open a new ticket with the outstanding issue * general agreement that having MoL allocate storage for ScratchSpace itself is fine, want to change client thorns to no longer set these parameters Yours, Roland

1 0

Re: [Users] Einstein Toolkit Meeting Reminder
by Frank Loeffler 01 Aug '15

01 Aug '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 and the conference id is 118682#. Frank Loeffler

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users August 2015