Users July 2011

users@lists.einsteintoolkit.org

15 participants
21 discussions

Problems with mpich2
by jhar131＠cct.lsu.edu 18 Dec '11

18 Dec '11

When attempting to build a simulation (in fedora), I encounter an undefined reference error pertaining to MPI. I have installed mpich2 and mpich2-devel both automatically with yum -y as suggested in the notation; as well as manually after this error. Any suggestions? Creating cactus_Test in /home/joshua/Cactus/exe from CactusBase/Boundary CactusBase/CartGrid3D CactusBase/CoordBase CactusConnect/HTTPD CactusConnect/HTTPDExtra CactusWave/IDScalarWaveC CactusBase/IOASCII CactusBase/IOBasic CactusIO/IOJpeg CactusBase/IOUtil CactusPUGHIO/IsoSurfacer CactusBase/LocalReduce CactusPUGH/PUGH CactusPUGH/PUGHReduce CactusPUGH/PUGHSlab CactusConnect/Socket CactusBase/SymBase CactusBase/Time CactusWave/WaveBinarySource CactusWave/WaveToyC ExternalLibraries/libjpeg /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trid' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trvalid' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_env2int' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trrealloc' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trspace' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trDebugLevel' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_TrSetMaxMem' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trlevel' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trmalloc' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_env2bool' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_env2range' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trcalloc' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trfree' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_env2str' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trstrdup' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trdump' /usr/lib64/mpich2/lib/libmpich.so: undefined reference to `MPL_trinit' collect2: ld returned 1 exit status

4 6

GetComponents Errors for private thorn
by Scott Hawley 02 Aug '11

02 Aug '11

Hi, I'm trying to add an svn repository for my thorn in my Thornlist file, and it doesn't seem to be working or telling me the nature of the problem: % ./GetComponents --debug --verbose ~/ThornLists/ThornListTexMEx Warning: Nothing will be checked out from svn://svn.belmont.edu/svnrepos/texmex TexMEx Is there something besides "debug" and "verbose" that will tell me WHY it won't be checked out? (I was reading the docs at http://cactuscode.org/documentation/GetComponents.php) Thanks. -Scott PS- The lines in question are: # TexMex !TARGET = $ARR !TYPE = svn !ANONYMOUS_USER = anonymous !ANONYMOUS_PASSWORD = anon !URL = svn://svn.belmont.edu/svnrepos/texmex TexMEx -- Scott H. Hawley, Ph.D. Asst. Prof. of Physics Chemistry & Physics Dept Office: Hitch 100D Belmont University Tel: +1-615-460-6206 Nashville, TN 37212 USA Fax: +1-615-460-5458 PGP Key at http://sks-keyservers.net

6 12

Meeting minutes
by Roland Haas 01 Aug '11

01 Aug '11

Meeting minutes: Present were: Tanja, Roland, Frank, Bruno, Yosef, Peter, Erik, Eloisa, Ian ET paper: MHD: * Christian and Bruno independently looked at PPM and re-split its MHD version from the hydro only version and rearranged code to allow the compiler (ifort) to optimize the routine * Bruno finds about 3% speedup (in PPM) * Erik says that Christian found about 30% speedup * Bruno and Christian will exchange notes * Peter agreed to run some performance tests with hardware counters using a parameter file that Bruno will provide * we still fail the cylindrical blast wave test which many other codes also do not pass (using divergence cleaning) * will try dissipation in the B field * will run white noise test for dissipation to check that divergence cleaning works as expected (see apples with apples paper) CCE: * Yosef committed writeup of sketch of documentation and a sample parameter file based on simfactory's qc0 example * we would like a testsuite, which might however require a large input file for CCE * Yosef is not sure if the PITTNull CCE extractor would work with Multipatch codes since he has no access to any * Christian Reisswigs Worldtube thorn should (there is a Llama version) * Yosef will give a short tutorial at a later date, date TBA testsuites: * see Ian's email http://lists.einsteintoolkit.org/pipermail/users/2011-July/001253.html * test data was regenerated for RotatingSymmetry which was affected by changes in Kranc which affected McLachlan * correct boundary/ghost zones in GRHydro test for second order convergence (2 required for McLachlan due to advective derivatives) * try tapering to find fourth order convergence in constraints paper: * no progress * need to write missing sections (!) * all authors: take half an hour to read sections and make comments * lead author(s) will assign sections to people to work on in the next two weeks, and which sections to keep or drop (depending on their state) * reserve a Monday call for paper talk (with all callers having read the paper) * decide on day for call within the next three days (until Wednesday), Frank will set up a doodle poll Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

3 3

Checkpoint/recovery inconsistencies
by Erik Schnetter 01 Aug '11

01 Aug '11

Christian Ott, Peter Diener, and I tracked down a set of checkpointing/recovery inconsistencies in the Einstein Toolkit. This means that many variables had different values after checkpointing and then recovering them, which should not be the case. We were careful not to use MPI, OpenMP, or any kind of compiler optimisations, so these inconsistencies represent real changes. We found a series of problems, and we have now one possible correction. Unfortunately, this requires changes to several thorns, mostly to schedule.ccl declarations, but also to Carpet. In addition, we find that it is impossible to re-calculate PseudoEvolution variables after recovery -- they have to be checkpointed. In other words, variables such as ADMBase, TmunuBase, or other variables with 3 timelevels need to be checkpointed to be consistent. (Many of the inconsistencies are small, and are caused by differences in the discretisation error. That is, they will vanish in the continuum limit. However, one of the design principles of the Einstein Toolkit is that results should be as much as possible independent of the number of MPI processes, OpenMP threads, and checkpointing/recovery. Disabling checkpointing of such variables could be implemented depending on a parameter, which would also need to ensure that these variables are then recalculated -- with slightly different values -- after recovery. In particular, this would require a new schedule bin in Carpet that is executed after recovery for all timelevels, or alternatively a new schedule option that requests this behaviour from Carpet.) The necessary changes to correct these inconsistencies are in detail: 1. All variables with 3 timelevels need to be checkpointed. This affects ADMBase, TmunuBase, ML_ADMConstraints, ML_ADMQuantities, and the BSSN constraints in ML_BSSN. Since this depends on the number of timelevels, care has to be taken to make the right decision at run time. 2. The schedule group MoL_PseudoEvolution must not be scheduled in post_recover_variables -- since these variables are checkpointed, they do not need to be recalculated. 3. Carpet cannot execute the post_recover_variables bin on the past timelevels, since applying boundary conditions to past timelevels includes prolongation, and the time interpolation necessary for this would have a different discretisation error. 4. Carpet traversed the PostRestrict bin in the incorrect order. It traversed from finest to coarsest (the same order in which restriction has to be applied), but because fine grid boundaries may be interpolate from coarse grid boundaries, this bin must be traversed from coarsest to finest. 5. The scheduling of GRHydro and HydroBase is off in certain places. 6. The scheduling of NaNChecker leaves the NaNmask uninitialised after recovery. 7. The scheduling of various Kranc generated thorns is off in certain places. 8. Certain schedule items requiring the ADMBase variables are executed in the incorrect order, i.e. before the ADMBase variables were available. I attach a diff of a possible set of corrections. However: - This diff does not correct handling the TmunuBase variables -- they are still not checkpointed if they have multiple timelevels. I believe this should be done by thorn TmunuBase itself. - Similarly, handling checkpointing of the ADMBase variables should be moved from ML_BSSN_Helper to ADMBase itself. - I think a new schedule group MoL_PseudoEvolutionBoundaries would make sense, as this would simplify the schedule of thorns which use MoL_PseudoEvolution. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/

4 7

Hyperslab Error
by Garrison, David 28 Jul '11

28 Jul '11

Hello, I am trying to use Generic Hyperslabbing to change an initial array from the normal Cactus MPI distribution to one where all the GF points can exist in one loop on one processor. For example if phi_temp is an N^3 array and phi_in is an (N-2)^3 array, sizex = cctk_gsh[0]; sizey = cctk_gsh[1]; sizez = cctk_gsh[2]; fsizex = cctk_gsh[0]-2; fsizey = cctk_gsh[1]-2; fsizez = cctk_gsh[2]-2; struct xferinfo info1[3]; for (d=0; d<3; ++d) { /* Source array: Cactus layout */ info1[d].src.gsh = cctk_gsh[d]; info1[d].src.lbnd = cctk_lbnd[d]; info1[d].src.lsh = cctk_lsh[d]; info1[d].src.lbbox = cctk_bbox[2*d]; info1[d].src.ubbox = cctk_bbox[2*d+1]; info1[d].src.nghostzones = cctk_nghostzones[d]; /* Destination array: New layout */ info1[d].dst.gsh = cctk_gsh[d]; info1[d].dst.lbnd = 0; info1[d].dst.lsh = cctk_gsh[d]; info1[d].dst.lbbox = cctk_bbox[2*d]; info1[d].dst.ubbox = cctk_bbox[2*d+1]; info1[d].dst.nghostzones = cctk_nghostzones[d]; /* Source slab: whole array */ info1[d].src.off = 0; info1[d].src.len = cctk_gsh[d]; info1[d].src.str = 1; /* Destination slab: whole array */ info1[d].dst.off = 0; info1[d].dst.len = cctk_gsh[d]; info1[d].dst.str = 1; /* No transformation */ info1[d].xpose = d; info1[d].flip = 0; } ierr = Slab_Transfer (cctkGH, cctk_dim, info1, -1, CCTK_VARIABLE_REAL, phi_temp, CCTK_VARIABLE_REAL, phi_temp); assert (! ierr); phi_in = malloc(sizeof(CCTK_REAL)*fsizex*fsizey*fsizez); for(k = 1; k <= fsizez; k++) { for(j = 1; j <= fsizey; j++) { for(i = 1; i <= fsizex; i++) { phi_in[(i-1) + (j-1)*fsizex + (k-1)*fsizex*fsizey] = phi_temp[i + j*sizex + k*sizex*sizey]; } } } Running the code as is causes segmentation fault and some of the output GFs are distorted. If I comment out the Slab_Transfer I don't get a Segmentation fault. If I run the code with a single processor, there is no segmentation fault and no distortion in the GFs. Can you see what I'm doing wrong with the hyperslabbing? There must be a better way to get the transfer to work while avoiding the errors. -DG -- David Garrison, Ph.D. Associate Professor and Chair of Physics University of Houston-Clear Lake Bayou 3531 Houston, TX 77058 Tel: 281-283-3796 Fax: 281-283-3709 http://sce.uhcl.edu/garrison For more information about the UHCL Physics Program go to http://www.uhcl.edu/physics or sign up for UHCL newsletters at https://ecommunications.houston.edu/uhcl/join.html "If we knew what it was we were doing, it would not be called research, would it?" ‹ Albert Einstein.

1 0

Carpet hg vs git
by Luca Baiotti 27 Jul '11

27 Jul '11

Hello, I am trying to switch to carpet mercurial. I compiled two executables, identical except for the Carpet thorns, one with thorns from the git repository and the other with thorns from the hg repository. The BBH simulations runs with the git version, but (with the same parfile) it fails with the hg version: ... INFO (CarpetReduce): Simulation domain volume: 512000 INFO (CarpetReduce): Reduction weight sum: 511999.999154925 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ Memory statistics from CarpetLib: Current number of objects: 2611 Current allocated memory: 649 MB Maximum number of objects: 2.63e+03 Maximum allocated memory: 652 MB Current administrative memory: 2.79 MB Total allocated used system memory: 665 MB Total allocated free system memory: 17.6 MB gh::allmemory: 708688 dh::allmemory: 1301208 th::allmemory: 26608 ggf::allmemory: 576000 gdata::allmemory: 182224 INFO (IsolatedHorizon): Low Spin < 1e-3 ! Skipping coordinate-transformation! [node154:6197] *** An error occurred in MPI_Allreduce: the reduction operation MPI_SUM is not defined for non-intrinsic datat ypes [node154:6197] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [node154:6197] *** MPI_ERR_OP: invalid reduce operation [node154:6197] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) I am running on damiana, with the same configuration options and qsub scripts for the two executables. When switching from git to hg, should I change, add, or remove some parameter? Here is a list of the carpet parameters (and a few others) that I set in my parfiles: ActiveThorns" Carpet CarpetIOScalar CarpetIOASCII CarpetIOBasic CarpetIOHDF5 CarpetInterp CarpetLib CarpetMask CarpetReduce CarpetRegrid2 Parameters: Carpet::domain_from_coordbase = yes Carpet::max_refinement_levels = 9 CarpetRegrid2::num_centres = 2 CarpetRegrid2::num_levels_1 = 9 CarpetRegrid2::position_x_1 = 4 CarpetRegrid2::radius_1 [1] = 100.0 etc. Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64]" Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 Carpet::regrid_in_level_mode = yes Carpet::regrid_during_initialisation = no CarpetRegrid2::regrid_every = 256 Carpet::init_fill_timelevels = yes InitBase::initial_data_setup_method = "init_all_levels" CarpetTracker::surface [0] = 0 CarpetTracker::surface [1] = 1 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 Carpet::num_integrator_substeps = 4 MoL::MoL_Num_Scratch_Levels = 1 CarpetMask::excluded_surface [0] = 2 CarpetMask::excluded_surface_factor [0] = 1. CarpetMask::excluded_surface [1] = 3 CarpetMask::excluded_surface [2] = 4 Carpet::poison_new_timelevels = yes Carpet::check_for_poison = no CarpetLib::poison_new_memory = yes CarpetLib::poison_value = 200 Carpet::output_timers_every = 768 CarpetLib::print_timestats_every = 768 CarpetLib::print_memstats_every = 768 Cheers, Luca

3 8

Test suites
by Ian Hinder 25 Jul '11

25 Jul '11

Hi everyone, All ET tests are now passing with Git Carpet. We have 14 tests failing with Mercurial Carpet in the following thorns: AHFinderDirect, CarpetIOHDF5, CarpetReduce, QuasiLocalMeasures, WeylScal4. http://damiana2.aei.mpg.de/~ianhin/testsuites/einsteintoolkit/ -- Ian Hinder ian.hinder(a)aei.mpg.de

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 25 Jul '11

25 Jul '11

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. I might not be able to call in from the airport (although I will try). In case I cannot "open" the call I ask someone with the necessary knowledge from LSU/ex-LSU to do that. Erik: please lead the call in that case. We need to: - get work on the paper started again, as well as finished - get cce tested / integrated, including test suites, documentation and ideally a short tutorial on how to use the code - get an update of MHD in GRHydro Frank

1 0

CarpetRegrid
by Luca Baiotti 21 Jul '11

21 Jul '11

Hello, this is a general question, not directly related to any specific issue. Is any of you using CarpetRegrid (as opposed to CarpetRegrid2)? Is CarpetRegrid still maintained (and tested) in the new versions of Carpet?) Luca

4 3

OpenMP problems in ET
by Hee Il Kim 20 Jul '11

20 Jul '11

Hi, I recently found OpenMP runs of ET can make different results depending on the number of threads (NT=1 vs. NT neq 1). In some experiments, the difference becomes noticeable only after a long time, but you can see the difference even for the TOV test run with static_tov.par (I compared the time variation of rho_max). With the same parameter setup except for the extended cctk_final time, the difference becomes noticeable around t = 1300. If you take a low order reconstruction such as VanLeerMC2, you can see the difference much earlier. Until t=5000, the differences do not grow drastically and tiny, but non-negligible at all. Things become more serious for more dynamical tests, such as binary neutron star merges. The difference does not disappear even though I remove all the OpenMP parallellization parts of GRHydro. That means something wrong in other than GRHydro. But one more weird thing is here. The result of No-openmp-GRhydro is not equal to the result of Openmp-GRhydro for the same number of threads. This problem can be fixed by removing the openmp parts in GRHydro_Prim2Con.F90 (I checked only non-MHD routines), even though I couldn't find anything wrong in GRHyro_Prim2Con.F90. Now, turning to non-GRHydro, I ran QC0 (qc0-mclachlan.par) much longer than the original final_time. Here also I found the floating point calculation is not repeatable even though the difference is tiny; at t=100 which is the original cctk_final_time of qc0-mclachlan.par 100 0.0595683707179757 (for NT=1) 100 0.0595683707137867 (for NT=4) However, in the private project which uses higher resolution and higher order (8th order) differencing method, I don't see any difference even in the last decimal place at least until t ~ 670. Is it really improved by the higher resolution etc.? I did the TOV tests on two different machines with Intel Xeon cpus and Intel 11.1 compilers. One is Queenbee and the other is a domestic cluster named Tachyon2. Please take a look at the attached optionlist of Tachyon2. In addition to the default option given in the file, also I tested with; -O2 -no-ip -O3 -no-ip -O3 Any hints or comments would be appreciated. Cheers, Hee Il

3 3

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users July 2011