Users July 2015

users@lists.einsteintoolkit.org

11 participants
17 discussions

Re: [Users] [ET Trac] [Einstein Toolkit] #1326: running loopcontrol on strange number of threads fails
by Erik Schnetter 11 Apr '19

11 Apr '19

Do you have a backtrace? -erik On Sunday, March 22, 2015, Einstein Toolkit < trac-noreply(a)einsteintoolkit.org> wrote: > #1326: running loopcontrol on strange number of threads fails > ------------------------------------+--------------------------------------- > Reporter: rhaas | Owner: > Type: defect | Status: new > Priority: minor | Milestone: > Component: EinsteinToolkit thorn | Version: > Resolution: | Keywords: LoopControl > ------------------------------------+--------------------------------------- > > Comment (by rhaas): > > This still happens even with current (Sun Mar 22 18:46:21 CET 2015) trunk, > though failure looks a bit different now: > {{{ > actus_sim: > /data/rhaas/postdoc/gr/ET_trunk/configs/sim/build/LoopControl/loopcontrol.cc:312: > T {anonymous}::divexact(T, T) [with T = int]: Assertion `i % j == 0' > failed > }}} > > -- > Ticket URL: <https://trac.einsteintoolkit.org/ticket/1326#comment:1> > Einstein Toolkit <http://einsteintoolkit.org> > The Einstein Toolkit > _______________________________________________ > Trac mailing list > Trac(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/trac > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 2

Benchmarking results for McLachlan rewrite
by Erik Schnetter 17 Aug '15

17 Aug '15

I ran the Simfactory benchmark for ML_BSSN on both the current version and the "rewrite" branch to see whether this branch is ready for production use. I ran this benchmark on a single node of Shelob at LSU. In both cases, using 2 OpenMP threads and 8 MPI processes per node was fastest, so I am reporting these results below. Since I was interested in the performance of McLachlan, this is a unigrid vacuum benchmark using fourth order differencing. One noteworthy difference is that dissipation as implemented in the "rewrite" branch is finally approximately as fast as thorn Dissipation, and I have thus used this option for the "rewrite" branch. Here are the high-level results: current: 3.03136e-06 sec per grid point rewrite: 2.85734e-06 sec per grid point That is, the rewrite branch is about 5% faster. More detailed timing results for CCTK_EVOL are as follows: Current: ---------------------------------------------------------------------------------------- Time Time Imblnc Timer gettimeof getrusage percent secs percent secs secs ---------------------------------------------------------------------------------------- 97.8% 373.1 0.0% |_CallEvol 373.1 738.5 97.8% 373.1 0.0% | |_CCTK_EVOL 373.1 738.4 97.8% 373.0 0.0% | | |_CallFunction 373 738.3 10.3% 39.3 36.9% | | | |_syncs 62.27 122.6 10.3% 39.3 36.9% | | | | |_Sync 62.25 122.6 2.1% 8.0 18.6% | | | | | |_comm_state[3].state_fill_sen 9.839 19.49 6.1% 23.2 46.3% | | | | | |_comm_state[6].state_do_some_ 43.18 84.73 1.5% 5.5 1.8% | | | | | |_comm_state[7].state_empty_re 5.61 10.84 87.3% 333.2 6.4% | | | |_thorns 310.3 614.5 21.8% 83.0 3.2% | | | | |_ML_BSSN_Advect 84.45 168.8 8.6% 32.9 51.3% | | | | |_ML_BSSN_NewRad 0.0354 0.05899 14.1% 53.6 1.9% | | | | |_ML_BSSN_RHS1 54.68 109.1 16.0% 61.1 2.0% | | | | |_ML_BSSN_RHS2 62.35 124.5 5.1% 19.4 9.0% | | | | |_ML_BSSN_convertToADMBase 18.63 37.17 4.2% 16.1 3.1% | | | | |_ML_BSSN_convertToADMBaseDtLaps 16.11 31.98 2.4% 9.1 20.0% | | | | |_ML_BSSN_enforce 11.34 22.65 8.3% 31.7 15.3% | | | | |_MoL_Add 33.12 62.48 1.0% 3.8 48.9% | | | | |_ReflectionSymmetry_Apply 7.461 14.93 5.2% 19.8 3.1% | | | | |_dissipation_add 20.47 40.87 Rewrite: -------------------------------------------------------------------------------------------------- Time Time Imblnc Timer gettimeof getrusage percent secs percent secs secs -------------------------------------------------------------------------------------------------- 99.0% 356.1 0.0% |_CallEvol 356.1 703.8 98.9% 356.0 0.0% | |_CCTK_EVOL 356.1 703.8 98.9% 356.0 0.0% | | |_CallFunction 356 703.7 14.0% 50.4 37.3% | | | |_syncs 80.43 158.2 14.0% 50.4 37.3% | | | | |_Sync 80.42 158.2 2.1% 7.7 7.7% | | | | | |_comm_state[3].state_fill_send_buffers. 8.341 16.51 9.7% 35.0 45.2% | | | | | |_comm_state[6].state_do_some_work.step 63.77 125.2 1.5% 5.6 2.5% | | | | | |_comm_state[7].state_empty_recv_buffers 5.629 10.87 84.8% 305.1 9.9% | | | |_thorns 275.2 544.4 5.5% 19.8 7.1% | | | | |_ML_BSSN_ADMBaseEverywhere 19.58 39.15 4.4% 16.0 5.3% | | | | |_ML_BSSN_ADMBaseInterior 15.91 31.63 2.3% 8.3 16.6% | | | | |_ML_BSSN_EnforceEverywhere 10.01 20.02 10.8% 39.0 5.0% | | | | |_ML_BSSN_EvolutionInteriorSplitBy1 38.75 77.28 21.2% 76.4 4.6% | | | | |_ML_BSSN_EvolutionInteriorSplitBy2 75.82 151.5 23.3% 83.9 5.8% | | | | |_ML_BSSN_EvolutionInteriorSplitBy3 83.78 166.8 9.7% 35.0 48.9% | | | | |_ML_BSSN_NewRad 0.1266 0.156 6.0% 21.5 20.7% | | | | |_MoL_Add 23.12 43.16 0.9% 3.3 49.4% | | | | |_ReflectionSymmetry_Apply 6.508 12.97 As can be seen, the non-McLachlan numbers are comparable, although not identical. This is to be expected. The main RHS evaluation is split over several routines in each case; these are (RHS1, RHS2, Advect, dissipation_add) for the current version and (EvolutionInteriorSplitBy[1,2,3]) for the rewrite branch. The way in which the RHS evaluation is actually split is of course different for both cases. With these numbers in hand, I think we are ready to switch to the rewrite branch. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

3 21

McLachlan help
by James Healy 11 Aug '15

11 Aug '15

Hello, I have been running some tests on Stampede comparing the run speed of McLachlan to RIT's evolution thorn LazEv. I started with the qc0-mclachlan.par parameter file included with the Einstein Toolkit, added a few refinement levels, increased the resolution and changed McLachlan to be 8th order (and increased the number of ghost zones to 5). I also increased the initial separation so the finest grids aren't already overlapping. To compare with LazEv, I removed the McLachlan and Dissipation thorns and replaced them with LazEv. Everything else in the parameter file is exactly the same. I tried using both the McLachlan master and rewrite branches. The grid setup is 10 levels of refinement, dx=4M on the coarsest with outer boundary at 400M, M/128 on the finest with r=0.6M, CFL is 0.25. Both use 8th order spatial differencing with ghost_size=5 and 5th order dissipation. Below is a summary of the results as reported at iteration 256 from Carpet::physical_time_per_hour: McLachlan - rewrite branch: 3.0596110 M/hr McLachlan - master branch: 3.8033607 M/hr LazEv - 4.1941544 M/hr I am using the stampede-impi.cfg configuration file in simfactory. "module list" returns: 1) TACC-paths 3) cluster-paths 5) xalt/0.4.6 7) TACC 2) Linux 4) intel/13.0.2.146 6) cluster 8) impi/4.1.0.030 Attached is my parameter file. I pasted the McLachlan parameters below. Are there any optimizations that I can use for McLachlan? Are the parameters I am using for it what would be used for production runs? ML_BSSN::harmonicN = 1 # 1+log ML_BSSN::harmonicF = 2.0 # 1+log ML_BSSN::ShiftGammaCoeff = 0.75 ML_BSSN::BetaDriver = 1.0 ML_BSSN::LapseAdvectionCoeff = 1.0 ML_BSSN::ShiftAdvectionCoeff = 1.0 ML_BSSN::MinimumLapse = 1.0e-8 ML_BSSN::my_initial_boundary_condition = "extrapolate-gammas" ML_BSSN::my_rhs_boundary_condition = "NewRad" Boundary::radpower = 2 ML_BSSN::ML_log_confac_bound = "none" ML_BSSN::ML_metric_bound = "none" ML_BSSN::ML_Gamma_bound = "none" ML_BSSN::ML_trace_curv_bound = "none" ML_BSSN::ML_curv_bound = "none" ML_BSSN::ML_lapse_bound = "none" ML_BSSN::ML_dtlapse_bound = "none" ML_BSSN::ML_shift_bound = "none" ML_BSSN::ML_dtshift_bound = "none" ML_BSSN::fdOrder = 8 ActiveThorns = "Dissipation" Dissipation::order = 5 Dissipation::vars = " ML_BSSN::ML_metric ML_BSSN::ML_trace_curv ML_BSSN::ML_curv ML_BSSN::ML_Gamma ML_BSSN::ML_lapse ML_BSSN::ML_shift ML_BSSN::ML_dtlapse ML_BSSN::ML_dtshift " ActiveThorns = "ML_ADMConstraints" Thanks, Jim Healy

4 7

Meeting minutes for 2015-07-27
by Roland Haas 27 Jul '15

27 Jul '15

Hello all, Present were: Josh, Matt, Roland, Peter, Yosef, Steve, Christine, Erik Merge rewrite of McLachlan: * Ian not present. Skipped. The RIT group tried McLachlan starting from ET parfile. Find that McLachlan runs significantly slower than LazEv. Will move discussion to the mailing list. Number of timelevels: * Roland raises the question of whether CCTK_MaxTimeLevels should be changed to return what CCTK_MaxActiveTimeLevels such that CCTK_MaxTimeLevels is an upper bound on CCTK_ActiveTimelevels at this point in time * consensus: rename CCTK_MaxTimeLevels to CCTK_MaxDeclaredTimeLevels * consensus: require that interface.ccl declared at least 2 timelevels to allow dynamic increase * no clear agreement yet about whether to have the driver copy data forward in time * currently Carpet internally copies data during time level cycling * suggestion to add restriction that grid functions with multiple timelevels must exist globally * worry about situation where a third party thorn enables more timelevels for variables of a thorn * want to provide as many safety features as we can rather than allow all possible uses of the code * static_assert: accept with suggestion to add CCTK_ATTRIBUTE_UNUSED Yours, Roland

1 0

Einstein Toolkit Meeting Reminder
by Steven R. Brandt 26 Jul '15

26 Jul '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 and the conference id is 118682#. Steven R. Brandt

1 1

McLachlan-rewrite has landed
by Erik Schnetter 24 Jul '15

24 Jul '15

I just pushed all the changes related to the McLachlan-rewrite merge. Apart from the McLachlan Kranc scripts and the respective generated code, there are a few thorns that needed to have test results updated. I see zero test failures with these changes on Bethe, Blue Waters, Carver, Datura, Edison, Hopper, Mike, Nvidia (a Perimeter workstation), Philip, Queen Bee, Shelob, and Stampede. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

2 11

Kranc and xAct
by Comer Duncan 22 Jul '15

22 Jul '15

Do Kranc and xAct play well together? Just curious since sometimes I do things in xAct and then may want to do the same in Kranc and it would be nice if I could just say import xAct and the work done in it to a developing Kranc script. Of course, there are several pieces of xAct, so maybe my question is not formulated precisely enough, but what do you think? Thanks. Comer

3 2

Meeting minutes for 2015-07-20
by Steven R. Brandt 22 Jul '15

22 Jul '15

Hello all, Present were Steve, Peter, Erik, Zach, Joseph, Matt * no objections to merging the rewrite branch of McLachlan - the last objection was Ian, who worried about performance. Performance is as good as before. The performance of McL dissipation is much faster. - there was some discussion about the completeness of tests, whether it was to roundoff and what sort of comparisons were done. * no objections to reformatting some util thorns with clang-format, excepting thorn Accelerator. * no objections to ticket #1786, which proposed switching to c99 inline semnatics. The ticket was moved to review_ok Tickets #1794 Backport of mpi detect.pl changes #816 include SphericalHarmonicReconASCII from PITTNULLCode #1780 Correct name of Weinberg pseudotenser are all in review_ok. Someone needs to apply them. Cheers, Steve

3 3

best way to maintain CCTK_COMPLEX precision with Fortran?
by Bernard Kelly 20 Jul '15

20 Jul '15

Hi all. The Cactus documentation shows a set of C routines CCTK_CmplxXXXXXX that presumably preserve precision between CCTK_REAL and CCTK_COMPLEX types. Is there a "right" way to handle this in Cactus with Fortran (that isn't "recode in C")? Context: I have some Fortran code in a Cactus thorn that works with real & complex fields. I've noticed that when the numbers are very large in magnitude, taking a simple real(z) drops precision --- a finite but large real part ends up as +/- Inf instead. I suspect it's dropping to single precision at some stage. Switching to "dreal" doesn't help, and avoiding the use of CCTK_COMPLEX altogether --- while possible --- leads to longer, less readable code. Thanks, Bernard

2 3

'trunk' in branch thorn
by Frank Loeffler 15 Jul '15

15 Jul '15

Hi, I just removed some extra 'trunk' directories that made it into (really 'into' some of the branches. This most likely happened using a script to create these branches, and I am sorry it wasn't caught earlier. Everything worked, of course, because these extra files didn't disturb anything. I now removed these, which should only have the effect of making a checkout of a release a bit smaller. Only four thorns had this problem: pciutils, PETSc, OpenBLAS, and OpenCL Frank

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users July 2015