Luca
I have not seen this error before. It seems to me a configuration problem regarding the MPI version that is used for building or when the job is submitted.
When switching to Mercurial Carpet, you may need to change a few parameters. Most of these are technical changes, e.g. parameters that don't exist any more in Mercurial Carpet. None of these parameter changes lead to MPI problems like this.
To debug this further, it would be important to find out which thorn performs this reduction. Using the parameter Carpet::veryverbose may help here.
-erik
On Fri, May 27, 2011 at 8:02 PM, Luca Baiotti baiotti@ile.osaka-u.ac.jp wrote:
Hello,
I am trying to switch to carpet mercurial. I compiled two executables, identical except for the Carpet thorns, one with thorns from the git repository and the other with thorns from the hg repository.
The BBH simulations runs with the git version, but (with the same parfile) it fails with the hg version:
... INFO (CarpetReduce): Simulation domain volume: 512000 INFO (CarpetReduce): Reduction weight sum: 511999.999154925 WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction weight sum differ Memory statistics from CarpetLib: Current number of objects: 2611 Current allocated memory: 649 MB Maximum number of objects: 2.63e+03 Maximum allocated memory: 652 MB Current administrative memory: 2.79 MB Total allocated used system memory: 665 MB Total allocated free system memory: 17.6 MB gh::allmemory: 708688 dh::allmemory: 1301208 th::allmemory: 26608 ggf::allmemory: 576000 gdata::allmemory: 182224 INFO (IsolatedHorizon): Low Spin < 1e-3 ! Skipping coordinate-transformation! [node154:6197] *** An error occurred in MPI_Allreduce: the reduction operation MPI_SUM is not defined for non-intrinsic datat ypes [node154:6197] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [node154:6197] *** MPI_ERR_OP: invalid reduce operation [node154:6197] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
I am running on damiana, with the same configuration options and qsub scripts for the two executables. When switching from git to hg, should I change, add, or remove some parameter?
Here is a list of the carpet parameters (and a few others) that I set in my parfiles:
ActiveThorns" Carpet CarpetIOScalar CarpetIOASCII CarpetIOBasic CarpetIOHDF5 CarpetInterp CarpetLib CarpetMask CarpetReduce CarpetRegrid2
Parameters:
Carpet::domain_from_coordbase = yes Carpet::max_refinement_levels = 9 CarpetRegrid2::num_centres = 2 CarpetRegrid2::num_levels_1 = 9 CarpetRegrid2::position_x_1 = 4 CarpetRegrid2::radius_1 [1] = 100.0
etc.
Carpet::time_refinement_factors = "[1,1,1,2,4,8,16,32,64]"
Carpet::use_buffer_zones = yes Carpet::prolongation_order_space = 5 Carpet::prolongation_order_time = 2 Carpet::regrid_in_level_mode = yes Carpet::regrid_during_initialisation = no CarpetRegrid2::regrid_every = 256 Carpet::init_fill_timelevels = yes InitBase::initial_data_setup_method = "init_all_levels" CarpetTracker::surface [0] = 0 CarpetTracker::surface [1] = 1 MoL::ODE_Method = "rk4" MoL::MoL_Intermediate_Steps = 4 Carpet::num_integrator_substeps = 4 MoL::MoL_Num_Scratch_Levels = 1 CarpetMask::excluded_surface [0] = 2 CarpetMask::excluded_surface_factor [0] = 1. CarpetMask::excluded_surface [1] = 3 CarpetMask::excluded_surface [2] = 4 Carpet::poison_new_timelevels = yes Carpet::check_for_poison = no CarpetLib::poison_new_memory = yes CarpetLib::poison_value = 200 Carpet::output_timers_every = 768 CarpetLib::print_timestats_every = 768 CarpetLib::print_memstats_every = 768
Cheers, Luca _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users