Users May 2011

users@lists.einsteintoolkit.org

19 participants
25 discussions

Carpet hg vs git
by Luca Baiotti 27 Jul '11

27 Jul '11

Hello, I am trying to switch to carpet mercurial. I compiled two executables, identical except for the Carpet thorns, one with thorns from the git repository and the other with thorns from the hg repository. The BBH simulations runs with the git version, but (with the same parfile) it fails with the hg version: ... INFO (CarpetReduce): Simulation domain volume: 512000 INFO (CarpetReduce): Reduction weight sum: 511999.999154925 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ Memory statistics from CarpetLib: Current number of objects: 2611 Current allocated memory: 649 MB Maximum number of objects: 2.63e+03 Maximum allocated memory: 652 MB Current administrative memory: 2.79 MB Total allocated used system memory: 665 MB Total allocated free system memory: 17.6 MB gh::allmemory: 708688 dh::allmemory: 1301208 th::allmemory: 26608 ggf::allmemory: 576000 gdata::allmemory: 182224 INFO (IsolatedHorizon): Low Spin < 1e-3 ! Skipping coordinate-transformation! [node154:6197] *** An error occurred in MPI_Allreduce: the reduction operation MPI_SUM is not defined for non-intrinsic datat ypes [node154:6197] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [node154:6197] *** MPI_ERR_OP: invalid reduce operation [node154:6197] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) I am running on damiana, with the same configuration options and qsub scripts for the two executables. When switching from git to hg, should I change, add, or remove some parameter? Here is a list of the carpet parameters (and a few others) that I set in my parfiles: ActiveThorns" Carpet CarpetIOScalar CarpetIOASCII CarpetIOBasic CarpetIOHDF5 CarpetInterp CarpetLib CarpetMask CarpetReduce CarpetRegrid2 Parameters: Carpet::domain_from_coordbase = yes Carpet::max_refinement_levels = 9 CarpetRegrid2::num_centres = 2 CarpetRegrid2::num_levels_1 = 9 CarpetRegrid2::position_x_1 = 4 CarpetRegrid2::radius_1 [1] = 100.0 etc. Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64]" Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 Carpet::regrid_in_level_mode = yes Carpet::regrid_during_initialisation = no CarpetRegrid2::regrid_every = 256 Carpet::init_fill_timelevels = yes InitBase::initial_data_setup_method = "init_all_levels" CarpetTracker::surface [0] = 0 CarpetTracker::surface [1] = 1 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 Carpet::num_integrator_substeps = 4 MoL::MoL_Num_Scratch_Levels = 1 CarpetMask::excluded_surface [0] = 2 CarpetMask::excluded_surface_factor [0] = 1. CarpetMask::excluded_surface [1] = 3 CarpetMask::excluded_surface [2] = 4 Carpet::poison_new_timelevels = yes Carpet::check_for_poison = no CarpetLib::poison_new_memory = yes CarpetLib::poison_value = 200 Carpet::output_timers_every = 768 CarpetLib::print_timestats_every = 768 CarpetLib::print_memstats_every = 768 Cheers, Luca

3 8

"No Einstein Toolkit Meeting" Reminder
by Frank Loeffler 04 Jul '11

04 Jul '11

Hi There will be no Einstein Toolkit meeting today. The next Monday-meeting will happen in two weeks: May 9th at 10am US central time. Frank Loeffler

2 1

problems of building ET on kraken
by Tsatsin Petr 31 May '11

31 May '11

Hello, I'm a new user of the ET. I did an installation of Cactus and EinstainToolkit using the "Tutorial for the new users" on wiki. I have an account on kraken and I modified udb.pm according to the 'ranger' example in a tutorial. Namely I changed: YOUR_LOGIN, YOUR(a)EMAIL.ADDRESS and added set option 'kraken', 'user', 'MY_LOGIN' set option 'kraken', 'allocation', 'MY_ALLOCATION' I tried to compile the test suite on kraken using the command: ./simfactory/sim build --thornlist=manifest/einsteintoolkit.th and I got the following error: 'Simulation Factory: Configuration name(s) not specified -- using default configuration "sim" Uncaught exception from user code: Unknown machine name "krakenpf4.nics.utk.edu" at ./simfactory/sim line 5049. at ./simfactory/sim line 5049 main::get_machine('') called at ./simfactory/sim line 1498 main::command_build() called at ./simfactory/sim line 454' I was wondering if you could help me with this? It's seem like I didn't modify udb.pm in proper way. Thank you. Petr Tsatsin Florida Atlantic University

2 2

CarpetRegrid2 grid structure inconsistencies
by Erik Schnetter 30 May '11

30 May '11

Some time ago, someone (can't find the email right now) reported problems with CarpetRegrid2, where the grid structure may be inconsistent. I have restructured the code, but since this is a fairly invasive change I would like this to be thoroughly tested before committing it. This change may lead to Carpet using a (slightly) different grid structure than before. The differences are not "important"; what potentially changes is the order in which consistency conditions are applied. I expect that, in most cases, there will be no differences. The list of consistency conditions is: - ensure proper nesting (level must be larger than next finer level) - add buffer zones (only once, not really a consistency condition) - combine regions into a single box (if this is efficient) - apply rotating 90/180 symmetry (if requested) - clip at outer boundary (so that boxes that are too large or are outside are cut off) I have uploaded the patch to the Einstein Toolkit TRAC at <https://trac.einsteintoolkit.org/ticket/442>. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 30 May '11

30 May '11

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. Ian Hinder agreed to lead the call this time again (I will be traveling at that time, and cell phone reception is poor here, at best). Points of discussion will be: 1) ET paper: - We should get this out of our ways before the summer - Mostly status on example sections. TOV: nothing happened here since last week, will catch up until next Monday 2) MHD status update (please report in the minutes) 3) REU summer student will work on Hydro Analysis thorn, ala total mass(es), center of mass, total linear/angular momentum, vorticity. Any ideas are welcome, but please keep in mind that this should be simple - it's a undergraduate summer project: about 6 weeks from zero to Cactus developer, implementing some physics and applying it in real simulations. The resulting thorn will be part of the Einstein Toolkit, assuming consensus at that time. Frank Loeffler

2 2

Cactus-built LAPACK, BLAS giving incorrect results
by Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] 28 May '11

28 May '11

Hi all. I recently needed to use a LAPACK routine in a Cactus thorn. Since my local machine's installation of LAPACK & BLAS was old, I decided to force an installation by including the thorns "ExternalLibraries/[BLAS,LAPACK]". This seemed to work, with the resulting .a files ending up in configs/<my config name>/scratch/external/LAPACK, etc. Compiling and running my cactus executable, however, I found the LAPACK gave completely wrong results. For a much simpler test case that (see attached F90 standalone program), consider the 2x2 system A x = b, where A = (3, 2\\2, 6), and b = (5, 8). Correct result for "x" is (1,1). Cactus-forced LAPACK (using routine "dgesv") yields (-2,1). I later installed LAPACK (v. 3.3.0) + BLAS by hand on my machine. Linking against these yields the correct answer. I'm on a 32-bit Linux machine, using the Intel 10 compilers. I'd appreciate any insights on this. Thanks, Bernard

2 6

Re: [Users] [Possible Spam: 10%] Cactus-built LAPACK, BLAS giving incorrect results
by Garrison, David 27 May '11

27 May '11

Hi, I doubt this helps but (-2,1) is a non-normalized eigenvector of the system. Maybe your forced LAPACK was doing the wrong operation. -DG -- David Garrison, Ph.D. Associate Professor and Chair of Physics University of Houston-Clear Lake Bayou 3531 Houston, TX 77058 Tel: 281-283-3796 Fax: 281-283-3709 http://sce.uhcl.edu/garrison For more information about the UHCL Physics Program go to http://www.uhcl.edu/physics or sign up for UHCL newsletters at https://ecommunications.houston.edu/uhcl/join.html "If we knew what it was we were doing, it would not be called research, would it?" - Albert Einstein. -----Original Message----- From: users-bounces(a)einsteintoolkit.org on behalf of Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORECOUNTY] Sent: Fri 5/27/2011 11:09 AM To: users(a)einsteintoolkit.org Subject: [Possible Spam: 10%] [Users] Cactus-built LAPACK, BLAS giving incorrect results Hi all. I recently needed to use a LAPACK routine in a Cactus thorn. Since my local machine's installation of LAPACK & BLAS was old, I decided to force an installation by including the thorns "ExternalLibraries/[BLAS,LAPACK]". This seemed to work, with the resulting .a files ending up in configs/<my config name>/scratch/external/LAPACK, etc. Compiling and running my cactus executable, however, I found the LAPACK gave completely wrong results. For a much simpler test case that (see attached F90 standalone program), consider the 2x2 system A x = b, where A = (3, 2\\2, 6), and b = (5, 8). Correct result for "x" is (1,1). Cactus-forced LAPACK (using routine "dgesv") yields (-2,1). I later installed LAPACK (v. 3.3.0) + BLAS by hand on my machine. Linking against these yields the correct answer. I'm on a 32-bit Linux machine, using the Intel 10 compilers. I'd appreciate any insights on this. Thanks, Bernard

1 0

OpenMP is making it slower?
by Scott Hawley 26 May '11

26 May '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm doing a Cactus run which I've compiled with OpenMp support on my linux machine, and looking at the performance in the routine in which I explicitly added a "parallel do" directive. When I change the environment variable OMP_NUM_THREADS to anything larger than 1, it actually takes LONGER than the 1-thread run. I have *24* processors on my machine. It runs fine on my mac, but not my linux machine. Does anyone have a suggestion? I notice my version of gcc doesn't have --enable-openmp (see below)... but then the worst it should do is not have any effect when OMP_NUM_THREADS is set to anything. Thanks! Scott > mpicc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.5-8' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.4 --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.4.5 (Debian 4.4.5-8) libgomp is version 4.4.5-8 ... but there is no lib64gomp. Odd. I'm using Debian squeeze which supposedly contains lib64gomp. - -- Scott H. Hawley, Ph.D. Asst. Prof. of Physics Chemistry & Physics Dept Office: Hitch 100D Belmont University Tel: +1-615-460-6206 Nashville, TN 37212 USA Fax: +1-615-460-5458 PGP Key at http://sks-keyservers.net -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJN0pn+AAoJENp9UKaXQr6GlwQIAKEYSnYmyiSOEni7yDj36NTq GXZg5wijj6flhsK/r/PKMIFslKa/k1BaONJKG8wY9tgV03QIpv5VuTH2Pw0RF3GF c7E8i5FoxT3p8yvQSS/oTQND32pTiBiIu+S0Yzv8lydNjRkoE2Dt6CJtZVpcEBn2 9993/kb8YlnD6jvLF9ycrqeyppKbWclSwZND3LA7kPEGxZmiLGPWVNBwOFHC5VUs BF+u42mWQQHOGsOiQMGGbgnS/btnlxdYQlvj8rFNpGcOQeWf2FEzlE7biXtMoFbX y0W24EHuoz5ggSq9T7hGcG1rFiLOfdpsJWmiY1I4A/kjHLpRHcXl4H50lt2ifbQ= =LZoK -----END PGP SIGNATURE-----

5 11

Carpet Git vs Mercurial
by Luca Baiotti 25 May '11

25 May '11

Hello, does someone have a list of changes (necessary or recommended) in the parfiles in order to migrate from Carpet-Git to Carpet-Mercurial? (A parfile of mine that works with CarpetGit does not work out of the box with Carpet-Mercurial.) Cheers, Luca

1 0

Meeting minutes
by Roland Haas 25 May '11

25 May '11

Meeting minutes: Present were: Tanja, Roland, Ian, Eloisa, Erik, Bruno, Yosef ET paper: * Bruno still has troubles with RotatingSymmetry180 when using the development version of ET for the BBH example (parfile is in subversion) * will try running without RotatingSymmetry180 * convergence in norms is bad, somewhat better in 1D data for a short time after simulation startup * cosmology example is done for now * other examples had no maintainer on the phone call * example data: ** add parfiles to paper repository ** add data for plots to paper repository ** have single person generate all plots for final version to achieve uniform style Array padding: * Erik suggests introducing array padding into Cactus to avoid cache aliasing issues * re-use existing but (essentially) unused cctk_lssh for loop boundaries of i,j,k * would use cctk_lsh to compute 1D index from i,j,k * would loop i,j,k using cctk_lssh * Cactus doc's already say to use cctk_lssh for this * thorns that are not updated *might* only compute extra unused data for the "padding" points ** thorns using ghostzones/boundary zones as cctk_lsh-nghostzones/boundary_width might cause problems ** thorns computing in the interior might compute extra data and overwrite ghostzone/boundary information * would like to have a mechanism to check for errors during runtime * make padding a runtime parameter in the parameter files * provide example parfile demonstrating speedup Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

5 10

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users May 2011