[Einstein Toolkit] #495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes"
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" --------------------+------------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Keywords: | --------------------+------------------------------------------------------- In the development (Mercurial) version of Carpet, but not the stable (Git) version, QuasiLocalMeasures fails with the following error message:
... INFO (QuasiLocalMeasures): Landau-Lifshitz angular momentum x: 0.00000 INFO (QuasiLocalMeasures): Landau-Lifshitz angular momentum y: 0.00000 INFO (QuasiLocalMeasures): Landau-Lifshitz angular momentum z: 0.00000 [sl-18:26396] *** An error occurred in MPI_Reduce: the reduction operation MPI_SUM is not defined for non-intrinsic datatypes [sl-18:26396] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0 [sl-18:26396] *** MPI_ERR_OP: invalid reduce operation [sl-18:26396] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
This affects the QuasiLocalMeasures test suite, but was also reported and discussed on the ET mailing list:
http://lists.einsteintoolkit.org/pipermail/users/2011-May/001107.html
Additional debugging was suggested to try to locate the reason for the error.
(Reporting against Carpet even though it might be a problem in QuasiLocalMeasures because it works in the stable Carpet and this will be a possible blocker for promoting the development Carpet to stable).
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
This is most likely not a problem in QLM, since that thorn only uses intrinsic MPI datatypes. I conjecture that this is a problem in CarpetReduce, most likely uncovered by a recent version of an MPI library that is more strict about checking standard conformance.
I am still waiting for mor information. Please send a stack backtrace or similar that helps identify which routine uses which MPI datatype when this error occurs.
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
The only non-intrinsic datatypes that Carpet uses are those defined for complex numbers. Is there a reduction operation called for complex numbers? If so, from where -- is this output, or is this in the thorn itself?
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by bentivegna):
* cc: bentivegna (added)
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by bentivegna):
It is output, not QLM. What is the fix, changing CarpetReduce to handle complex numbers in a standard-compliant form, or making sure QLM uses intrinsic datatypes only?
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
The work-around is the to disable CarpetScalar output for complex quantities.
The solution is to add support for Carpet's complex MPI datatypes to CarpetReduce. MPI does not offer C datatypes for complex numbers. Maybe we can instead use Fortran datatypes -- they should be the same. If so, the code in Carpet which creates new MPI datatypes for complex number should be modified to use the existing Fortran complex number MPI datatypes instead.
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by bentivegna):
A related question: what has changed in CarpetReduce between the git and mercurial versions? The error appears when switching from the former to the latter, with the exact same Open MPI implementation (1.5.4).
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
It is possible that CarpetReduce changed -- I don't recall.
Can we, for the release, state that one should not reduce complex numbers for output?
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by hinder):
I'm attaching a list of all the commits in the Mercurial version which touched Carpet/CarpetReduce. You can generate this list using
{{{ hg log -r 5418b354a3ab:tip Carpet/CarpetReduce }}}
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: review Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------ Changes (by eschnett):
* status: new => review
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: review Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by eschnett):
*ping*
#495: MPI error from Carpet with QuasiLocalMeasures: "MPI_SUM is not defined for non-intrinsic datatypes" ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: closed Priority: major | Milestone: Component: Carpet | Version: Resolution: fixed | Keywords: ---------------------+------------------------------------------------------ Changes (by eschnett):
* status: review => closed * resolution: => fixed
Comment:
Applied.
trac@lists.einsteintoolkit.org
-
Einstein Toolkit