[Users] Carpet hg vs git

[Erik Schnetter]

Luca

I have not seen this error before. It seems to me a configuration
problem regarding the MPI version that is used for building or when
the job is submitted.

When switching to Mercurial Carpet, you may need to change a few
parameters. Most of these are technical changes, e.g. parameters that
don't exist any more in Mercurial Carpet. None of these parameter
changes lead to MPI problems like this.

To debug this further, it would be important to find out which thorn
performs this reduction. Using the parameter Carpet::veryverbose may
help here.

-erik

On Fri, May 27, 2011 at 8:02 PM, Luca Baiotti <baiotti at ile.osaka-u.ac.jp> wrote:
> Hello,
>
> I am trying to switch to carpet mercurial. I compiled two executables,
> identical except for the Carpet thorns, one with thorns from the git
> repository and the other with thorns from the hg repository.
>
> The BBH simulations runs with the git version, but (with the same
> parfile) it fails with the hg version:
>
> ...
> INFO (CarpetReduce): Simulation domain volume: 512000
> INFO (CarpetReduce): Reduction weight sum:     511999.999154925
> WARNING[L1,P0] (CarpetReduce): Simulation domain volume and reduction
> weight sum differ
> Memory statistics from CarpetLib:
>    Current number of objects: 2611
>    Current allocated memory:  649 MB
>    Maximum number of objects: 2.63e+03
>    Maximum allocated memory:  652 MB
>    Current administrative memory: 2.79 MB
>    Total allocated used system memory: 665 MB
>    Total allocated free system memory: 17.6 MB
>    gh::allmemory:    708688
>    dh::allmemory:    1301208
>    th::allmemory:    26608
>    ggf::allmemory:   576000
>    gdata::allmemory: 182224
> INFO (IsolatedHorizon): Low Spin < 1e-3 ! Skipping
> coordinate-transformation!
> [node154:6197] *** An error occurred in MPI_Allreduce: the reduction
> operation MPI_SUM is not defined for non-intrinsic datat
> ypes
> [node154:6197] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
> [node154:6197] *** MPI_ERR_OP: invalid reduce operation
> [node154:6197] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
>
>
> I am running on damiana, with the same configuration options and qsub
> scripts for the two executables. When switching from git to hg, should I
> change, add, or remove some parameter?
>
>
> Here is a list of the carpet parameters (and a few others) that I set in
> my parfiles:
>
> ActiveThorns"
>   Carpet
>   CarpetIOScalar
>   CarpetIOASCII
>   CarpetIOBasic
>   CarpetIOHDF5
>   CarpetInterp
>   CarpetLib
>   CarpetMask
>   CarpetReduce
>   CarpetRegrid2
>
> Parameters:
>
> Carpet::domain_from_coordbase           = yes
> Carpet::max_refinement_levels           = 9
> CarpetRegrid2::num_centres              = 2
> CarpetRegrid2::num_levels_1             = 9
> CarpetRegrid2::position_x_1             = 4
> CarpetRegrid2::radius_1             [1] = 100.0
>
> etc.
>
> Carpet::time_refinement_factors         = "[1,1,1,2,4,8,16,32,64]"
>
> Carpet::use_buffer_zones                = yes
> Carpet::prolongation_order_space        = 5
> Carpet::prolongation_order_time         = 2
> Carpet::regrid_in_level_mode            = yes
> Carpet::regrid_during_initialisation    = no
> CarpetRegrid2::regrid_every             = 256
> Carpet::init_fill_timelevels            = yes
> InitBase::initial_data_setup_method     = "init_all_levels"
> CarpetTracker::surface                      [0] = 0
> CarpetTracker::surface                      [1] = 1
> MoL::ODE_Method                         = "rk4"
> MoL::MoL_Intermediate_Steps             = 4
> Carpet::num_integrator_substeps         = 4
> MoL::MoL_Num_Scratch_Levels             = 1
> CarpetMask::excluded_surface        [0] = 2
> CarpetMask::excluded_surface_factor [0] = 1.
> CarpetMask::excluded_surface        [1] = 3
> CarpetMask::excluded_surface        [2] = 4
> Carpet::poison_new_timelevels           = yes
> Carpet::check_for_poison                = no
> CarpetLib::poison_new_memory            = yes
> CarpetLib::poison_value                 = 200
> Carpet::output_timers_every             = 768
> CarpetLib::print_timestats_every        = 768
> CarpetLib::print_memstats_every         = 768
>
>
> Cheers,
> Luca
> _______________________________________________
> Users mailing list
> Users at einsteintoolkit.org
> http://lists.einsteintoolkit.org/mailman/listinfo/users
>



-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/