Users January 2014

users@lists.einsteintoolkit.org

17 participants
22 discussions

Einstein Toolkit Meeting Reminder
by Frank Loeffler 14 May '15

14 May '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. My agenda contains: - Fall workshop - PLEASE let us know if you come [1]! - Hotel booking - Release planning - release date: Mon Oct 24 2011 - freeze date: Mon Sep 26 - the day of the call! - branch name: ET_2011_10 - List of release-concerning bugs [2] - ET paper: proposed "freeze" by Oct 3rd (Mon after this call) As always: feel free to add to this list. Frank Loeffler [1] https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Fall_2011#Fall_Einstei… [2] https://trac.einsteintoolkit.org/query?status=!closed&milestone=ET_2011_11&…

10 31

Re: [Users] CarpetReduce: Formerly working par file now yields 'Simulation domain volume and reduction weight sum differ'
by Scott Hawley 24 Mar '14

24 Mar '14

Hi Steve, Making that change didn't affect the error. You're off the hook. ;-) I'm happy to hear other suggestions. I've never seen this error before. Thanks, Scott From: "Steven R. Brandt" <sbrandt(a)cct.lsu.edu<mailto:sbrandt@cct.lsu.edu>> Date: Monday, January 27, 2014 10:32 AM To: Scott Hawley <scott.hawley(a)belmont.edu<mailto:scott.hawley@belmont.edu>> Subject: Re: [Users] CarpetReduce: Formerly working par file now yields 'Simulation domain volume and reduction weight sum differ' Let's just make sure it's not my fault somehow (I doubt that it is). Go to src/util/ParseFile.c and change line 134 to look like this: int piraha_active = 0; See if that makes any difference. If it doesn't, change the 0 back to 1. If it does, please let me know. Cheers, Steve On 01/23/2014 10:57 AM, Scott Hawley wrote: Hi. I never received any responses to this. Hoping it was just because that was a busy time of year. Trying again. Thanks, Scott From: Scott Hawley <scott.hawley(a)belmont.edu<mailto:scott.hawley@belmont.edu>> Date: Friday, December 20, 2013 1:19 PM To: "users(a)einsteintoolkit.org<mailto:users@einsteintoolkit.org>" <users(a)einsteintoolkit.org<mailto:users@einsteintoolkit.org>> Subject: [Users] CarpetReduce: Formerly working par file now yields 'Simulation domain volume and reduction weight sum differ' Hi, I'm pretty sure I haven't changed a par file which "used to work." Now I'm getting the 'Simulation domain volume and reduction weight sum differ' error. I've read a few previous posts about this... http://lists.einsteintoolkit.org/pipermail/users/2011-September/001457.html http://lists.einsteintoolkit.org/pipermail/users/2012-September/002540.html http://lists.einsteintoolkit.org/pipermail/trac/2011-October/002465.html ...but haven't understood how they apply to my case. I'm not using CarpetMask, and haven't changed anything about the boundary points or grid structure. Can anyone offer a suggestion? Thanks. Output log and par file follow. -Scott % /opt/local/bin/openmpirun -np 1 ./exe/cactus_bruiser -roe --logdir=log /Users/shawley/runs/cactus/bruiser/bruiser_test.par IN DIRECTORY: /Users/shawley/Cactus -------------------------------------------------------------------------------- 10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org<http://www.cactuscode.org> 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101 -------------------------------------------------------------------------------- Cactus version: 4.1.1 Compile date: Dec 20 2013 (12:34:14) Run date: Dec 20 2013 (13:02:27-0600) Run host: (pid=99492) WARNING level 1 in thorn CarpetReduce processor 0 host Working directory: /Users/shawley/Cactus (line 137 of /Users/shawley/.mojaveconfig/Cactus/bruiser/build/CarpetReduce/mask_test.c): Executable: ./exe/cactus_bruiser -> Simulation domain volume and reduction weight sum differ Parameter file: /Users/shawley/runs/cactus/bruiser/bruiser_test.par -------------------------------------------------------------------------------- % cat bruiser_test.par | grep -v '^#' Cactus::cctk_run_title = "BruiserSetBG, FMR BBH" Cactus::cctk_full_warnings = yes Cactus::cctk_itlast = 0 ActiveThorns = "AEILocalInterp Boundary CartGrid3D CoordBase SymBase Slab Carpet CarpetLib CarpetInterp CarpetReduce CarpetSlab CarpetRegrid IOUtil SphericalSurface NaNChecker ADMBase ADMCoupling ADMMacros CoordGauge SpaceMask StaticConformal BruiserSetBG BruiserConstraints AHFinderDirect ADMAnalysis ADMConstraints LoopControl InitBase CarpetIOBasic CarpetIOScalar CarpetIOASCII TimerReport TmunuBase GenericFD" ActiveThorns = "CarpetIOHDF5 CarpetIOASCII" SphericalSurface::nsurfaces = 3 SphericalSurface::maxntheta = 39 SphericalSurface::maxnphi = 76 SphericalSurface::ntheta [0] = 39 SphericalSurface::nphi [0] = 76 SphericalSurface::nghoststheta[0] = 2 SphericalSurface::nghostsphi [0] = 2 SphericalSurface::ntheta [1] = 39 SphericalSurface::nphi [1] = 76 SphericalSurface::nghoststheta[1] = 2 SphericalSurface::nghostsphi [1] = 2 SphericalSurface::ntheta [2] = 39 SphericalSurface::nphi [2] = 76 SphericalSurface::nghoststheta[2] = 2 SphericalSurface::nghostsphi [2] = 2 IO::out_dir = $parfile Carpet::domain_from_coordbase = yes driver::ghost_size = 4 InitBase::initial_data_setup_method="init_all_levels" Carpet::prolongation_order_time = 0 Carpet::prolongation_order_space = 1 Carpet::convergence_level = 0 ADMMacros::spatial_order = 4 ADMBase::initial_data = "BruiserSetBG, Kerr-Schild" ADMBase::lapse_timelevels = 2 ADMBase::shift_timelevels = 2 ADMConstraints::constraints_timelevels = 2 ADMBase::metric_timelevels = 2 CoordBase::domainsize = minmax CoordBase::spacing = numcells CoordBase::xmin = -36.0 CoordBase::ymin = -36.0 CoordBase::zmin = -36.0 CoordBase::xmax = 36.0 CoordBase::ymax = 36.0 CoordBase::zmax = 36.0 CoordBase::ncells_x = 12 CoordBase::ncells_y = 12 CoordBase::ncells_z = 12 CartGrid3D::type = coordbase CartGrid3D::avoid_origin = no NaNChecker::check_every = 524288 NaNChecker::action_if_found = terminate Carpet::max_refinement_levels = 7 CarpetRegrid::refinement_levels = 7 CarpetRegrid::refined_regions = "manual-coordinate-list" CarpetRegrid::smart_outer_boundaries = yes CarpetRegrid::coordinates = "[ [([-36,-36,-36]:[36,36,36]:[3,3,3])], [([-36,-36,-36]:[36,36,36]:[1.5,1.5,1.5])], [([-19.5,-19.5,-19.5]:[19.5,19.5,19.5]:[0.75,0.75,0.75])], [([-12,-9,-9]:[12,9,9]:[0.375,0.375,0.375])], [([-8.25,-4.125,-4.125]:[-1.875,4.125,4.125]:[0.1875,0.1875,0.1875]),([1.875,-4.125,-4.125]:[8.25,4.125,4.125]:[0.1875,0.1875,0.1875])], [([-6.375,-1.6875,-1.6875]:[-3.75,1.6875,1.6875]:[0.09375,0.09375,0.09375]), ([3.75,-1.6875,-1.6875]:[6.375,1.6875,1.6875]:[0.09375,0.09375,0.09375])] ]" Carpet::verbose = "no" Carpet::veryverbose = "no" BruiserSetBG::bh1_m = 1.0 BruiserSetBG::bh1_a = 0.5 BruiserSetBG::bh1_theta = 0.0 BruiserSetBG::bh1_phi = 1.57 BruiserSetBG::bh1_x = -5.00 BruiserSetBG::bh1_y = 0.0 BruiserSetBG::bh1_z = 0.0 BruiserSetBG::bh1_vx = 1.0e-33 BruiserSetBG::bh1_vy = 0.0 BruiserSetBG::bh1_vz = 0.0 BruiserSetBG::bh2_m = 1.0 BruiserSetBG::bh2_a = 0.5 BruiserSetBG::bh2_theta = 0.0 BruiserSetBG::bh2_phi = 0.0 BruiserSetBG::bh2_x = 5.00 BruiserSetBG::bh2_y = 0.0 BruiserSetBG::bh2_z = 0.0 BruiserSetBG::bh2_vx = 1.0e-33 BruiserSetBG::bh2_vy = 0.0 BruiserSetBG::bh2_vz = 0.0 BruiserSetBG::r_mask = 0.9 BruiserConstraints::skip_solver = "no" BruiserConstraints::use_fma = "no" BruiserConstraints::nlevels = 7 BruiserConstraints::nvcycle = 1 BruiserConstraints::preswp = 3 BruiserConstraints::bruiser_verbosity = 5 BruiserConstraints::sor_factor = 1.4 BruiserConstraints::cgtol = 1.0e-8 SpaceMask::use_mask = yes AHFinderDirect::find_every = 1 AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation" AHFinderDirect::geometry_interpolator_pars = "order=4" AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation" AHFinderDirect::surface_interpolator_pars = "order=4" AHFinderDirect::N_horizons = 2 AHFinderDirect::origin_x [1] = -5.0 AHFinderDirect::initial_guess__coord_sphere__x_center [1] = -5.0 AHFinderDirect::initial_guess__coord_sphere__radius [1] = 2.0 AHFinderDirect::which_surface_to_store_info [1] = 0 AHFinderDirect::origin_x [2] = 5.0 AHFinderDirect::initial_guess__coord_sphere__x_center [2] = 5.0 AHFinderDirect::initial_guess__coord_sphere__radius [2] = 2.0 AHFinderDirect::which_surface_to_store_info [2] = 1 AHFinderDirect::set_mask_for_individual_horizon [2] = no TimerReport::output_all_timers = "yes" IOBasic::outInfo_every = 1 IOBasic::outInfo_reductions = "minimum maximum" IOBasic::outInfo_vars = " ADMBase::alp " IOScalar::outScalar_every = 1 IOScalar::outScalar_vars = " grid::coordinates ADMBase::metric ADMBase::curv " IOASCII::out2D_every = 1 IOASCII::out2D_vars = " BruiserConstraints::phi " IOHDF5::out_criterion = time IOHDF5::out_dt = 1.0 IOHDF5::out_vars = " ADMBase::metric ADMBase::curv BruiserSetBG::bruiser_mask BruiserConstraints::phi " BruiserConstraints::output_method = "IOHDF5" _______________________________________________ Users mailing list Users(a)einsteintoolkit.org<mailto:Users@einsteintoolkit.org>http://lists.einsteintoolkit.org/mailman/listinfo/users

2 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 31 Jan '14

31 Jan '14

1 0

simfactory
by Vassilios Mewes 31 Jan '14

31 Jan '14

Hello all, first time I am trying to use simfactory... I installed the ET trunk version on the MPG hydra cluster in Garching (with Intel 14) and wanted to try simfactory for the first time (on the other clusters I have always compiling and running cactus without simfactory) now, here is what i have done: i more or less copied the supermuc.ini, supermuc-ibmmpi.run and supermuc-ibmmpi.sub to files named hydra* with the respective extensions and then put them to the respective folders, changin username and so on (i dont really know what to use for the aliaspattern, maybe that is hindering me from succeeding) then i want to create a simulation with sim create and the binary i have already compiled, but it fails with the following error: Error: Error: empty/missing run script for configuration ET_TORI_INTEL Aborting Simfactory. even when i supply the file via --runscript, it fails to use it... any ideas what might be going on here? best wishes, Vassili

2 1

Meeting Minutes
by Roland Haas 29 Jan '14

29 Jan '14

Present: Frank, Roland, Matt, Josh Stampede slowdown in Noether release: * Jim reported on the mailing list that setting LoopControl::settle_after_iteration = 0 fixed the issue Include link to archive email in mailman: * do not wait for mailman 3 which might be while before it is released * for einsteintoolkit.org we have root access and can modify mailman * adapt proposed scripts in ticket #719 * further discussion in ticket using intel 14 compiler: * no one has yet successfully compiled the code with it or run a simulation with it Misc: * call to add examples to the gallery http://einsteintoolkit.org/about/gallery/ . Previously Roland and Peter had promised examples, Philipp or Christian R. were volunteered for a collapse (to BH or NS) example * please review tickets: https://trac.einsteintoolkit.org/query?status=review&col=id&col=summary&col… Yours, Roland

3 2

Git: "index pack failure" checking out Noether?
by Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] 27 Jan '14

27 Jan '14

Hi all. I'm only just getting around to checking out the Noether release of ET, and I'm hitting a problem I haven't seen before. All the Git-hosted arrangements are failing on checkout, because "fatal: index-pack failed". Has anyone seen this? A tiny bit of Googling indicates this might be a server-side issue, rather than on my client's end, but since this involves at least two different serves, I suspect the problem is on my end. Can anyone help/advise? The version of Git installed on this machine is "1.8.5.2". Thanks, Bernard P.S. I ran again with --verbose (and without --parallel), but it wasn't much more informative.

2 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 26 Jan '14

26 Jan '14

1 0

ET_2013_11 run performance
by James Healy 24 Jan '14

24 Jan '14

Hello all, On Thursday morning, I pulled a fresh checkout of the newest version of the Einstein Toolkit (ET_2013_11) to use with RIT's LazEv code. I compiled it on stampede using the current stampede.cfg located in simfactory/mdb/optionlists which uses Intel MPI version 4.1.0.030 and the intel compilers version 13.1.1.163 (enabled through a module load). I submitted a short job which I ran previously with ET_2013_05. The results come out the same. However, the run speed as reported in Carpet::physical_time_per_hour is poor. It starts off good, approximately the same as with the previous build, but over time drops to as low as half the speed over 24 hours of evolution. On recovery from checkpoint, the speed is even worse, dropping to below 1/4 of the original run speed. So, I tried using the previous stampede.cfg included in the ET_2013_05 branch of simfactory, the same one I used to compile my ET_2013_05 build. This cfgfile uses the same version of IMPI but different Intel compilers (version 13.0.2.146). The run speed shows the same trends as when using the newer config file. The parameter file I am using is EXACTLY the same, not a single change. The run is fairly lightweight, not requiring many nodes to run (I used 2). It is purely vacuum GR, an R1 configuration (equal-mass, non-spinning BBH system with a small initial separation). Memory usage in all three cases is the same, and is stable throughout the run. Are there new parameters that I should be setting to get similar performance as before? New thorns that I should have active? If not, has anyone else used stampede recently and experienced similar problems? Thanks, Jim Healy Rochester Institute of Technology

7 14

CarpetReduce: Formerly working par file now yields 'Simulation domain volume and reduction weight sum differ'
by Scott Hawley 23 Jan '14

23 Jan '14

Hi, I'm pretty sure I haven't changed a par file which "used to work." Now I'm getting the 'Simulation domain volume and reduction weight sum differ' error. I've read a few previous posts about this... http://lists.einsteintoolkit.org/pipermail/users/2011-September/001457.html http://lists.einsteintoolkit.org/pipermail/users/2012-September/002540.html http://lists.einsteintoolkit.org/pipermail/trac/2011-October/002465.html ...but haven't understood how they apply to my case. I'm not using CarpetMask, and haven't changed anything about the boundary points or grid structure. Can anyone offer a suggestion? Thanks. Output log and par file follow. -Scott % /opt/local/bin/openmpirun -np 1 ./exe/cactus_bruiser -roe --logdir=log /Users/shawley/runs/cactus/bruiser/bruiser_test.par IN DIRECTORY: /Users/shawley/Cactus -------------------------------------------------------------------------------- 10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101 -------------------------------------------------------------------------------- Cactus version: 4.1.1 Compile date: Dec 20 2013 (12:34:14) Run date: Dec 20 2013 (13:02:27-0600) Run host: (pid=99492) WARNING level 1 in thorn CarpetReduce processor 0 host Working directory: /Users/shawley/Cactus (line 137 of /Users/shawley/.mojaveconfig/Cactus/bruiser/build/CarpetReduce/mask_test.c): Executable: ./exe/cactus_bruiser -> Simulation domain volume and reduction weight sum differ Parameter file: /Users/shawley/runs/cactus/bruiser/bruiser_test.par -------------------------------------------------------------------------------- % cat bruiser_test.par | grep -v '^#' Cactus::cctk_run_title = "BruiserSetBG, FMR BBH" Cactus::cctk_full_warnings = yes Cactus::cctk_itlast = 0 ActiveThorns = "AEILocalInterp Boundary CartGrid3D CoordBase SymBase Slab Carpet CarpetLib CarpetInterp CarpetReduce CarpetSlab CarpetRegrid IOUtil SphericalSurface NaNChecker ADMBase ADMCoupling ADMMacros CoordGauge SpaceMask StaticConformal BruiserSetBG BruiserConstraints AHFinderDirect ADMAnalysis ADMConstraints LoopControl InitBase CarpetIOBasic CarpetIOScalar CarpetIOASCII TimerReport TmunuBase GenericFD" ActiveThorns = "CarpetIOHDF5 CarpetIOASCII" SphericalSurface::nsurfaces = 3 SphericalSurface::maxntheta = 39 SphericalSurface::maxnphi = 76 SphericalSurface::ntheta [0] = 39 SphericalSurface::nphi [0] = 76 SphericalSurface::nghoststheta[0] = 2 SphericalSurface::nghostsphi [0] = 2 SphericalSurface::ntheta [1] = 39 SphericalSurface::nphi [1] = 76 SphericalSurface::nghoststheta[1] = 2 SphericalSurface::nghostsphi [1] = 2 SphericalSurface::ntheta [2] = 39 SphericalSurface::nphi [2] = 76 SphericalSurface::nghoststheta[2] = 2 SphericalSurface::nghostsphi [2] = 2 IO::out_dir = $parfile Carpet::domain_from_coordbase = yes driver::ghost_size = 4 InitBase::initial_data_setup_method="init_all_levels" Carpet::prolongation_order_time = 0 Carpet::prolongation_order_space = 1 Carpet::convergence_level = 0 ADMMacros::spatial_order = 4 ADMBase::initial_data = "BruiserSetBG, Kerr-Schild" ADMBase::lapse_timelevels = 2 ADMBase::shift_timelevels = 2 ADMConstraints::constraints_timelevels = 2 ADMBase::metric_timelevels = 2 CoordBase::domainsize = minmax CoordBase::spacing = numcells CoordBase::xmin = -36.0 CoordBase::ymin = -36.0 CoordBase::zmin = -36.0 CoordBase::xmax = 36.0 CoordBase::ymax = 36.0 CoordBase::zmax = 36.0 CoordBase::ncells_x = 12 CoordBase::ncells_y = 12 CoordBase::ncells_z = 12 CartGrid3D::type = coordbase CartGrid3D::avoid_origin = no NaNChecker::check_every = 524288 NaNChecker::action_if_found = terminate Carpet::max_refinement_levels = 7 CarpetRegrid::refinement_levels = 7 CarpetRegrid::refined_regions = "manual-coordinate-list" CarpetRegrid::smart_outer_boundaries = yes CarpetRegrid::coordinates = "[ [([-36,-36,-36]:[36,36,36]:[3,3,3])], [([-36,-36,-36]:[36,36,36]:[1.5,1.5,1.5])], [([-19.5,-19.5,-19.5]:[19.5,19.5,19.5]:[0.75,0.75,0.75])], [([-12,-9,-9]:[12,9,9]:[0.375,0.375,0.375])], [([-8.25,-4.125,-4.125]:[-1.875,4.125,4.125]:[0.1875,0.1875,0.1875]),([1.875,-4.125,-4.125]:[8.25,4.125,4.125]:[0.1875,0.1875,0.1875])], [([-6.375,-1.6875,-1.6875]:[-3.75,1.6875,1.6875]:[0.09375,0.09375,0.09375]), ([3.75,-1.6875,-1.6875]:[6.375,1.6875,1.6875]:[0.09375,0.09375,0.09375])] ]" Carpet::verbose = "no" Carpet::veryverbose = "no" BruiserSetBG::bh1_m = 1.0 BruiserSetBG::bh1_a = 0.5 BruiserSetBG::bh1_theta = 0.0 BruiserSetBG::bh1_phi = 1.57 BruiserSetBG::bh1_x = -5.00 BruiserSetBG::bh1_y = 0.0 BruiserSetBG::bh1_z = 0.0 BruiserSetBG::bh1_vx = 1.0e-33 BruiserSetBG::bh1_vy = 0.0 BruiserSetBG::bh1_vz = 0.0 BruiserSetBG::bh2_m = 1.0 BruiserSetBG::bh2_a = 0.5 BruiserSetBG::bh2_theta = 0.0 BruiserSetBG::bh2_phi = 0.0 BruiserSetBG::bh2_x = 5.00 BruiserSetBG::bh2_y = 0.0 BruiserSetBG::bh2_z = 0.0 BruiserSetBG::bh2_vx = 1.0e-33 BruiserSetBG::bh2_vy = 0.0 BruiserSetBG::bh2_vz = 0.0 BruiserSetBG::r_mask = 0.9 BruiserConstraints::skip_solver = "no" BruiserConstraints::use_fma = "no" BruiserConstraints::nlevels = 7 BruiserConstraints::nvcycle = 1 BruiserConstraints::preswp = 3 BruiserConstraints::bruiser_verbosity = 5 BruiserConstraints::sor_factor = 1.4 BruiserConstraints::cgtol = 1.0e-8 SpaceMask::use_mask = yes AHFinderDirect::find_every = 1 AHFinderDirect::geometry_interpolator_name = "Lagrange polynomial interpolation" AHFinderDirect::geometry_interpolator_pars = "order=4" AHFinderDirect::surface_interpolator_name = "Lagrange polynomial interpolation" AHFinderDirect::surface_interpolator_pars = "order=4" AHFinderDirect::N_horizons = 2 AHFinderDirect::origin_x [1] = -5.0 AHFinderDirect::initial_guess__coord_sphere__x_center [1] = -5.0 AHFinderDirect::initial_guess__coord_sphere__radius [1] = 2.0 AHFinderDirect::which_surface_to_store_info [1] = 0 AHFinderDirect::origin_x [2] = 5.0 AHFinderDirect::initial_guess__coord_sphere__x_center [2] = 5.0 AHFinderDirect::initial_guess__coord_sphere__radius [2] = 2.0 AHFinderDirect::which_surface_to_store_info [2] = 1 AHFinderDirect::set_mask_for_individual_horizon [2] = no TimerReport::output_all_timers = "yes" IOBasic::outInfo_every = 1 IOBasic::outInfo_reductions = "minimum maximum" IOBasic::outInfo_vars = " ADMBase::alp " IOScalar::outScalar_every = 1 IOScalar::outScalar_vars = " grid::coordinates ADMBase::metric ADMBase::curv " IOASCII::out2D_every = 1 IOASCII::out2D_vars = " BruiserConstraints::phi " IOHDF5::out_criterion = time IOHDF5::out_dt = 1.0 IOHDF5::out_vars = " ADMBase::metric ADMBase::curv BruiserSetBG::bruiser_mask BruiserConstraints::phi " BruiserConstraints::output_method = "IOHDF5"

1 1

Meeting Minutes
by Roland Haas 22 Jan '14

22 Jan '14

Present: Frank Loeffler, Roland Haas, Stefan Ruehe, Erik Schnetter, Ian Hinder, Joshua Faber, Matt Kinsey, Yosef Zlochower, James Healy Stampede slowdown in Noether release: * Ian suggests using mvapich2 rather than intel MPI (already in simfactory/trunk) * Roland suggests disabling LoopControl * Erik suggest looking at Cactus' timer output * Jim will test performance with these options * Jim had tested old version with current setup and got good performance * Jim will open a trac ticket to focus discussion Using SPH (or other particle code with Cactus): * some interest in SPH in Cactus, each participant presented what they had done in the past and what they would like to do, Erik gave a very short description of the SPH method * Erik implemented a unigrid SPH type code in Cactus ** particles are tied to the 3d cells to find neighbours ** uses 1d Cactus arrays for particle storage ** not yet parallelized ** would need to add code to exchange particles as particles move * Josh has a working MPI parallelized SPH code ** uses grid to compute gravity forces ** Newtonian version is available for download ** Josh has fixed background GR version ** each process keeps the full particle position list in memory ** neighbour list however is distributed across processes ** parallelized around particles, no attempt to keep particles local to grid * Matt has code that evolves many geodesics ** own classes to handle and store particles ** keeps particles on process that owns grid under particle, otherwise interpolation is too costly ** currently scales to billions of particles ** uses Boost serialization to move particles after each step ** works with AMR grid * Ian has similar code to evolve geodesics ** wants to evolve geodesics, do raytracing ** have geodesic integrator using MoL and mesh refinement ** tested with relatively small number of geodesics (everything on one process) * Stefan has worked on an fixes spacetime SPH code for his diplom thesis ** would like to couple to a GR code ** to be used for stellar disruption ** more discussion via email ** invited to join the ET phone calls (they are all public) * notice that geodesics and SPH evolution likely have different laod characteristics since geodesics do not require gradients between particles. For geodesics the time consuming factor is sending interpolated data over the network. * suggestion to implement a "particle" datatype in Cactus so that the infrastructure knows about it and can offer specialized functions to handle with them * technical needed code to increase grid array size using intel 14 compiler: * Frank and Roland tried on stampede but failed to compile the full configuration, received segfault * Ian succeeded compiling the bare (or less than) full ET * Erik suggest reducing debug information and/or optimization level Elliptic solver in ET: * currently the only one known to work with mesh refinement is Eloisa Bentivegna's elliptic solver. Currently available on bitbucket https://bitbucket.org/eloisa/cosmology/overview * tested for Cosmology (written for bh lattice), and for binary black holes (not particularly in this case) simfactory 3: * Erik, Ian, Barry working on new simfactory implementation * designing interfaces and methods * should be usable by non-cactus tools, offer direct interface from python, eg "import mdb from simfactory" * currently not yet feature complete * still important architectural decisions to make * Erik will consult with Ian and send out url of repository use simfactory (2) to manage external libraries: * want to have simfactory build external libraries so that they can be used by non-Cactus projects as well * ExternalLibraries revert to CactusExternal scheme * need to handle multiple different configurations build at the same time on the * possible presentation next Monday on current status Misc: * Yosef to open ticket about being unable to checkpoint on stampede Yours, Roland

5 6

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users January 2014