Hi All,
One of my simulations recently aborted with the following error:
cactus_ET: /opt/exp_soft/EinsteinToolkit/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:261: bboxset2::bboxset<T, D> bboxset2::bboxset<T, D>::binary_operator(const F&, const bboxset2::bboxset<T, D>&) const [with F = bboxset2::bboxset<T, D>::operator&(const bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
This was when running on an HPC cluster. On my laptop, it works fine and I don't get this error.
However, commenting out the following section in my parameter file resulted in a successful HPC run:
Activethorns = "CarpetIOHDF5" IOHDF5::one_file_per_group = "yes" IOHDF5::out0D_every = 128 IOHDF5::out0D_vars = " WEYLSCAL4::Psi4r WEYLSCAL4::Psi4i PunctureTracker::pt_loc "
This is the only HDF5 output I'm requesting. Am I doing something silly here?
If it would help, I'd be happy to open a ticket and attach the whole parameter file, backtrace etc.
Gwyneth
On 12 Mar 2017, at 09:08, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi All,
One of my simulations recently aborted with the following error:
cactus_ET: /opt/exp_soft/EinsteinToolkit/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:261: bboxset2::bboxset<T, D> bboxset2::bboxset<T, D>::binary_operator(const F&, const bboxset2::bboxset<T, D>&) const [with F = bboxset2::bboxset<T, D>::operator&(const bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
This was when running on an HPC cluster. On my laptop, it works fine and I don't get this error.
However, commenting out the following section in my parameter file resulted in a successful HPC run:
Activethorns = "CarpetIOHDF5" IOHDF5::one_file_per_group = "yes" IOHDF5::out0D_every = 128 IOHDF5::out0D_vars = " WEYLSCAL4::Psi4r WEYLSCAL4::Psi4i PunctureTracker::pt_loc "
This is the only HDF5 output I'm requesting. Am I doing something silly here?
If it would help, I'd be happy to open a ticket and attach the whole parameter file, backtrace etc.
Hi Gwyneth,
There are two different "bboxset" implementations in Carpet. bboxset1 and bboxset2. Which one is used depends on the optionlist, and I think the availability of certain C++11 features. I suspect that your laptop is using bboxset1, and the cluster is using bboxset2, and the error is only triggering in bboxset2, perhaps because it has a bug, or maybe because of a compiler bug in the cluster's compiler. Can you add
-DCARPET_DISABLE_BBOXSET2
to CPPFLAGS in the cluster's optionlist, and see if the problem goes away?
Another possibility is that you are running on a different number of MPI processes on your laptop and the cluster, and the (possible) bug is only being triggered for one particular domain decomposition. If you are not running on the same number of MPI processes, you could try that, and see if this is responsible. In any case, I think this looks like a bug, so yes, please do open a ticket for this.
Hi Ian, Frank, Erik,
Thanks for the comments!
Ian, I'm using my full cluster core allocation for simulations at the moment, but should be able to get back to you about adjusting CPPFLAGS in a few days' time. The same goes for changing the number of MPI processes (I was indeed using a much higher number of processes when running on the cluster).
Frank, I haven't been able to find a (direct and obvious) indication of which bboxset implementation is used in the output. But I've attached it to the ticket, so you're welcome to have a look. (Erik did say that it was bboxset2.)
Gwyneth
On Sun, Mar 12, 2017 at 4:27 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 12 Mar 2017, at 09:08, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi All,
One of my simulations recently aborted with the following error:
cactus_ET: /opt/exp_soft/EinsteinToolkit/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:261: bboxset2::bboxset<T, D> bboxset2::bboxset<T, D>::binary_operator(const F&, const bboxset2::bboxset<T, D>&) const [with F = bboxset2::bboxset<T, D>::operator&(const bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
This was when running on an HPC cluster. On my laptop, it works fine and I don't get this error.
However, commenting out the following section in my parameter file resulted in a successful HPC run:
Activethorns = "CarpetIOHDF5" IOHDF5::one_file_per_group = "yes" IOHDF5::out0D_every = 128 IOHDF5::out0D_vars = " WEYLSCAL4::Psi4r WEYLSCAL4::Psi4i PunctureTracker::pt_loc "
This is the only HDF5 output I'm requesting. Am I doing something silly here?
If it would help, I'd be happy to open a ticket and attach the whole parameter file, backtrace etc.
Hi Gwyneth,
There are two different "bboxset" implementations in Carpet. bboxset1 and bboxset2. Which one is used depends on the optionlist, and I think the availability of certain C++11 features. I suspect that your laptop is using bboxset1, and the cluster is using bboxset2, and the error is only triggering in bboxset2, perhaps because it has a bug, or maybe because of a compiler bug in the cluster's compiler. Can you add
-DCARPET_DISABLE_BBOXSET2
to CPPFLAGS in the cluster's optionlist, and see if the problem goes away?
Another possibility is that you are running on a different number of MPI processes on your laptop and the cluster, and the (possible) bug is only being triggered for one particular domain decomposition. If you are not running on the same number of MPI processes, you could try that, and see if this is responsible. In any case, I think this looks like a bug, so yes, please do open a ticket for this.
-- Ian Hinder http://members.aei.mpg.de/ianhin
Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111 <+27%2021%20650%209111>. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
On 13 Mar 2017, at 13:14, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Ian, Frank, Erik,
Thanks for the comments!
Ian, I'm using my full cluster core allocation for simulations at the moment, but should be able to get back to you about adjusting CPPFLAGS in a few days' time. The same goes for changing the number of MPI processes (I was indeed using a much higher number of processes when running on the cluster).
Does the problem appear immediately? Can you run on your laptop with the same number of MPI processes as you were using on the cluster when you saw the error?
Frank, I haven't been able to find a (direct and obvious) indication of which bboxset implementation is used in the output. But I've attached it to the ticket, so you're welcome to have a look. (Erik did say that it was bboxset2.)
The error indicates that it is definitely using bboxset2 on the cluster. The question is what is used on your laptop. The logic that decides which implementation to use is in
arrangements/Carpet/CarpetLib/src/defs.hh:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_BASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #endif
Assuming that you are using gcc on your laptop, you could add compile-time output which will tell you which is being used:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_BASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #pragma message "bboxset2 disabled" #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #pragma message "bboxset2 enabled" #pragma message "bboxset2 used" #endif
(the only changes are to add the pragma lines).
On my laptop, this gives
COMPILING arrangements/Carpet/CarpetLib/src/bboxset1.cc In file included from /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/bboxset1.cc:9:0: /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:34:17: note: #pragma message: bboxset2 enabled #pragma message "bboxset2 enabled" ^ /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:35:17: note: #pragma message: bboxset2 used #pragma message "bboxset2 used"
Hi Erik, Ian, Frank:
Thankfully, I managed to get Open MPI working on my laptop and can now run things again!
I tried running the parameter file that produces the error on the HPC. Previously, the error did not appear on my laptop, but now it does (when running on two processes). I don't think there's an issue with a very small domain and a very large number of MPI processes here.
I get:
Assertion failed: (all(stride == other.stride)), function binary_operator, file /Users/gwynethallwright/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh, line 261.
The simulation seems to abort immediately after TwoPunctures finishes setting up the initial data and the actual evolution begins.
Should I try compiling with -DCARPET_DISABLE_BBOXSET2 on my laptop?
Gwyneth
On Mon, Mar 13, 2017 at 2:49 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 13 Mar 2017, at 13:14, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Ian, Frank, Erik,
Thanks for the comments!
Ian, I'm using my full cluster core allocation for simulations at the
moment, but should be able to get back to you about adjusting CPPFLAGS in a few days' time. The same goes for changing the number of MPI processes (I was indeed using a much higher number of processes when running on the cluster).
Does the problem appear immediately? Can you run on your laptop with the same number of MPI processes as you were using on the cluster when you saw the error?
Frank, I haven't been able to find a (direct and obvious) indication of
which bboxset implementation is used in the output. But I've attached it to the ticket, so you're welcome to have a look. (Erik did say that it was bboxset2.)
The error indicates that it is definitely using bboxset2 on the cluster. The question is what is used on your laptop. The logic that decides which implementation to use is in
arrangements/Carpet/CarpetLib/src/defs.hh:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_ BASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #endif
Assuming that you are using gcc on your laptop, you could add compile-time output which will tell you which is being used:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_ BASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #pragma message "bboxset2 disabled" #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #pragma message "bboxset2 enabled" #pragma message "bboxset2 used" #endif
(the only changes are to add the pragma lines).
On my laptop, this gives
COMPILING arrangements/Carpet/CarpetLib/src/bboxset1.cc In file included from /Users/ian/Cactus/EinsteinToolkitGit/ arrangements/Carpet/CarpetLib/src/bboxset1.cc:9:0: /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:34:17: note: #pragma message: bboxset2 enabled #pragma message "bboxset2 enabled" ^ /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:35:17: note: #pragma message: bboxset2 used #pragma message "bboxset2 used"
-- Ian Hinder http://members.aei.mpg.de/ianhin
Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
Gwyneth
Can you produce a stack backtrace? Or can you show more output, in particular while setting Carpet::veryverbose = yes?
I don't think this error is related to the bboxset class, so disabling bboxset2 is unlikely to help.
-erik
On Tue, Mar 21, 2017 at 4:32 PM, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Erik, Ian, Frank:
Thankfully, I managed to get Open MPI working on my laptop and can now run things again!
I tried running the parameter file that produces the error on the HPC. Previously, the error did not appear on my laptop, but now it does (when running on two processes). I don't think there's an issue with a very small domain and a very large number of MPI processes here.
I get:
Assertion failed: (all(stride == other.stride)), function binary_operator, file /Users/gwynethallwright/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh, line 261.
The simulation seems to abort immediately after TwoPunctures finishes setting up the initial data and the actual evolution begins.
Should I try compiling with -DCARPET_DISABLE_BBOXSET2 on my laptop?
Gwyneth
On Mon, Mar 13, 2017 at 2:49 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 13 Mar 2017, at 13:14, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Ian, Frank, Erik,
Thanks for the comments!
Ian, I'm using my full cluster core allocation for simulations at the
moment, but should be able to get back to you about adjusting CPPFLAGS in a few days' time. The same goes for changing the number of MPI processes (I was indeed using a much higher number of processes when running on the cluster).
Does the problem appear immediately? Can you run on your laptop with the same number of MPI processes as you were using on the cluster when you saw the error?
Frank, I haven't been able to find a (direct and obvious) indication of
which bboxset implementation is used in the output. But I've attached it to the ticket, so you're welcome to have a look. (Erik did say that it was bboxset2.)
The error indicates that it is definitely using bboxset2 on the cluster. The question is what is used on your laptop. The logic that decides which implementation to use is in
arrangements/Carpet/CarpetLib/src/defs.hh:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_B ASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #endif
Assuming that you are using gcc on your laptop, you could add compile-time output which will tell you which is being used:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_B ASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #pragma message "bboxset2 disabled" #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #pragma message "bboxset2 enabled" #pragma message "bboxset2 used" #endif
(the only changes are to add the pragma lines).
On my laptop, this gives
COMPILING arrangements/Carpet/CarpetLib/src/bboxset1.cc In file included from /Users/ian/Cactus/EinsteinTool kitGit/arrangements/Carpet/CarpetLib/src/bboxset1.cc:9:0: /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:34:17: note: #pragma message: bboxset2 enabled #pragma message "bboxset2 enabled" ^ /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:35:17: note: #pragma message: bboxset2 used #pragma message "bboxset2 used"
-- Ian Hinder http://members.aei.mpg.de/ianhin
Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
On 21 Mar 2017, at 21:51, Erik Schnetter schnetter@cct.lsu.edu wrote:
Gwyneth
Can you produce a stack backtrace? Or can you show more output, in particular while setting Carpet::veryverbose = yes?
I don't think this error is related to the bboxset class, so disabling bboxset2 is unlikely to help.
I think it's worth a try though!
Hi Ian,
The -DCARPET_DISABLE_BBOXSET2 compile fails with a familiar-looking error:
Undefined symbols for architecture x86_64:
"std::basic_ostream<char, std::char_traits<char> >& output<bbox<int, 0>
(std::basic_ostream<char, std::char_traits<char> >&, std::vector<bbox<int,
0>, std::allocator<bbox<int, 0> > > const&)", referenced from:
bboxset1::bboxset<int, 0>::output(std::basic_ostream<char, std::char_traits<char> >&) const in libthorn_CarpetIOHDF5.a( OutputSlice.cc.o)
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
If you have any suggestions for getting it working, I'd be happy to fiddle away.
Gwyneth
On Tue, Mar 21, 2017 at 11:02 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 21 Mar 2017, at 21:51, Erik Schnetter schnetter@cct.lsu.edu wrote:
Gwyneth
Can you produce a stack backtrace? Or can you show more output, in particular while setting Carpet::veryverbose = yes?
I don't think this error is related to the bboxset class, so disabling bboxset2 is unlikely to help.
I think it's worth a try though!
-- Ian Hinder http://members.aei.mpg.de/ianhin
Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111 <+27%2021%20650%209111>. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
Hi Erik,
I get:
Backtrace from rank 0 pid 94588: Backtrace not available!
in the backtrace file. Is there anything I can try to get it working?
I've attached the output with Carpet::veryverbose = yes to the trac ticket. Let me know if there's anything else you'd like me to send you.
Gwyneth
On Tue, Mar 21, 2017 at 10:51 PM, Erik Schnetter schnetter@cct.lsu.edu wrote:
Gwyneth
Can you produce a stack backtrace? Or can you show more output, in particular while setting Carpet::veryverbose = yes?
I don't think this error is related to the bboxset class, so disabling bboxset2 is unlikely to help.
-erik
On Tue, Mar 21, 2017 at 4:32 PM, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Erik, Ian, Frank:
Thankfully, I managed to get Open MPI working on my laptop and can now run things again!
I tried running the parameter file that produces the error on the HPC. Previously, the error did not appear on my laptop, but now it does (when running on two processes). I don't think there's an issue with a very small domain and a very large number of MPI processes here.
I get:
Assertion failed: (all(stride == other.stride)), function binary_operator, file /Users/gwynethallwright/Cactus /arrangements/Carpet/CarpetLib/src/bboxset2.hh, line 261.
The simulation seems to abort immediately after TwoPunctures finishes setting up the initial data and the actual evolution begins.
Should I try compiling with -DCARPET_DISABLE_BBOXSET2 on my laptop?
Gwyneth
On Mon, Mar 13, 2017 at 2:49 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 13 Mar 2017, at 13:14, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi Ian, Frank, Erik,
Thanks for the comments!
Ian, I'm using my full cluster core allocation for simulations at the
moment, but should be able to get back to you about adjusting CPPFLAGS in a few days' time. The same goes for changing the number of MPI processes (I was indeed using a much higher number of processes when running on the cluster).
Does the problem appear immediately? Can you run on your laptop with the same number of MPI processes as you were using on the cluster when you saw the error?
Frank, I haven't been able to find a (direct and obvious) indication
of which bboxset implementation is used in the output. But I've attached it to the ticket, so you're welcome to have a look. (Erik did say that it was bboxset2.)
The error indicates that it is definitely using bboxset2 on the cluster. The question is what is used on your laptop. The logic that decides which implementation to use is in
arrangements/Carpet/CarpetLib/src/defs.hh:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_B ASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #endif
Assuming that you are using gcc on your laptop, you could add compile-time output which will tell you which is being used:
// Disable bboxset2 if C++11 is not supported #if !defined(HAVE_CCTK_CXX_AUTO_SPECIFIER) || \ !defined(HAVE_CCTK_CXX_LAMBDA) || !defined(HAVE_CCTK_CXX_RANGE_B ASED_FOR) #ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_WARN_DISABLE_BBOXSET2 #endif #undef CARPET_DISABLE_BBOXSET2 #define CARPET_DISABLE_BBOXSET2 #pragma message "bboxset2 disabled" #endif
#ifndef CARPET_DISABLE_BBOXSET2 #define CARPET_ENABLE_BBOXSET2 #define CARPET_USE_BBOXSET2 #pragma message "bboxset2 enabled" #pragma message "bboxset2 used" #endif
(the only changes are to add the pragma lines).
On my laptop, this gives
COMPILING arrangements/Carpet/CarpetLib/src/bboxset1.cc In file included from /Users/ian/Cactus/EinsteinTool kitGit/arrangements/Carpet/CarpetLib/src/bboxset1.cc:9:0: /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:34:17: note: #pragma message: bboxset2 enabled #pragma message "bboxset2 enabled" ^ /Users/ian/Cactus/EinsteinToolkitGit/arrangements/Carpet/CarpetLib/src/defs.hh:35:17: note: #pragma message: bboxset2 used #pragma message "bboxset2 used"
-- Ian Hinder http://members.aei.mpg.de/ianhin
Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/ Disclaimer - University of Cape Town This e-mail is subject to UCT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111 <+27%2021%20650%209111>. If this e-mail is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via csirt@uct.ac.za
Gwyneth
Congratulations, you found another bug in Carpet. There is some internal inconsistency that was detected, forcing the run to abort. Yes, please open a full bug report. In particular a stack backtrace would be helpful.
0D HDF5 output is rarely used. You can use 0D ASCII output instead as a work-around.
(I don't think this has anything to do with bboxset1 vs. bboxset2. You are using bboxset2, which is the modern variant, which is good.)
-erik
On Sun, Mar 12, 2017 at 4:08 AM, Gwyneth Allwright allgwy001@myuct.ac.za wrote:
Hi All,
One of my simulations recently aborted with the following error:
cactus_ET: /opt/exp_soft/EinsteinToolkit/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:261: bboxset2::bboxset<T, D> bboxset2::bboxset<T, D>::binary_operator(const F&, const bboxset2::bboxset<T, D>&) const [with F = bboxset2::bboxset<T, D>::operator&(const bboxset2::bboxset<T, D>&) const [with T = int; int D = 3]::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(stride == other.stride)' failed.
This was when running on an HPC cluster. On my laptop, it works fine and I don't get this error.
However, commenting out the following section in my parameter file resulted in a successful HPC run:
Activethorns = "CarpetIOHDF5" IOHDF5::one_file_per_group = "yes" IOHDF5::out0D_every = 128 IOHDF5::out0D_vars = " WEYLSCAL4::Psi4r WEYLSCAL4::Psi4i PunctureTracker::pt_loc "
This is the only HDF5 output I'm requesting. Am I doing something silly here?
If it would help, I'd be happy to open a ticket and attach the whole parameter file, backtrace etc.
Gwyneth
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
On 12 Mar 2017, at 17:27, Erik Schnetter schnetter@cct.lsu.edu wrote:
Gwyneth
Congratulations, you found another bug in Carpet. There is some internal inconsistency that was detected, forcing the run to abort. Yes, please open a full bug report. In particular a stack backtrace would be helpful.
0D HDF5 output is rarely used. You can use 0D ASCII output instead as a work-around.
(I don't think this has anything to do with bboxset1 vs. bboxset2. You are using bboxset2, which is the modern variant, which is good.)
Hi Erik,
If the bug is in bboxset2, then it might not be present in bboxset1. Having said that, since the number of processes will affect the bboxsets that are used in domain decomposition, and hence what CarpetIOHDF5 sees, I suspect the problem is triggered by the different (larger) number of MPI processes.
Gwyneth, are you using a very large number of MPI processes for a very small domain, like for the static_tov test that you were doing before?
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Gwyneth Allwright -
Ian Hinder