hi all,
i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error
INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6
when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine.
i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same.
i've ran this through gdb, here's the relevant output:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380
so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899
this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash.
i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same.
has anyone seen such an error before? anything else i could provide to help diagnose this?
thanks, Miguel
Miguel
Can you look at https://trac.einsteintoolkit.org/ticket/1800? Can you try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem?
-erik
On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão mzilhao@ffn.ub.es wrote:
hi all,
i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error
INFO (CarpetIOHDF5):
INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5):
terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6
when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine.
i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same.
i've ran this through gdb, here's the relevant output:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry =0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry =true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380
so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899
this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash.
i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same.
has anyone seen such an error before? anything else i could provide to help diagnose this?
thanks, Miguel
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
hi Erik,
Can you look at https://trac.einsteintoolkit.org/ticket/1800? Can you try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem?
thanks, but this does not seem to help... i still get the same error and backtrace in gdb:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x0000000000c70402 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b1b930, fullname=fullname@entry=0x3e52d760 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df4c990, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x0000000000c72e5a in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b1b930, outfile=outfile@entry=167
Miguel
On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão <mzilhao@ffn.ub.es mailto:mzilhao@ffn.ub.es> wrote:
hi all, i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6 when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine. i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same. i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380 so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899 this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash. i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same. has anyone seen such an error before? anything else i could provide to help diagnose this? thanks, Miguel _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
Erik,
some more info. i briefly went through git history and found the offending commit. up to Carpet commit d52042b867eea1e0770b6de87ff3929ab7b9d297 everything works fine for me. from commit 69c73fb12d2ee41f01928122cd178d1bae9f8e13 onward i get the error i mentioned.
indeed, in the current state, if i remove writing of the "active" attribute everything works just fine:
diff --git a/CarpetIOHDF5/src/Output.cc b/CarpetIOHDF5/src/Output.cc index ec9b090..30c017d 100644 --- a/CarpetIOHDF5/src/Output.cc +++ b/CarpetIOHDF5/src/Output.cc @@ -894,10 +894,11 @@ static int AddAttributes (const cGH *const cctkGH, const char *fullname, HDF5_ERROR (H5Awrite (attr, H5T_NATIVE_INT, &ioffset[0])); HDF5_ERROR (H5Aclose (attr));
- ostringstream buf; - buf << (vdd.at(Carpet::map)-> - local_boxes.at(mglevel).at(refinementlevel).at(component).active); - WriteAttribute(dataset, "active", buf.str().c_str()); + // ostringstream buf; + // buf << (vdd.at(Carpet::map)-> + // local_boxes.at(mglevel).at(refinementlevel).at(component).active); + // WriteAttribute(dataset, "active", buf.str().c_str()); + }
if (is_index) {
i can't tell why it breaks once that bit of code is in, though...
thanks, Miguel
On 06/08/15 15:43, Miguel Zilhão wrote:
hi Erik,
Can you look at https://trac.einsteintoolkit.org/ticket/1800? Can you try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem?
thanks, but this does not seem to help... i still get the same error and backtrace in gdb:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x0000000000c70402 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b1b930, fullname=fullname@entry=0x3e52d760 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df4c990, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x0000000000c72e5a in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b1b930, outfile=outfile@entry=167
Miguel
On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão <mzilhao@ffn.ub.es mailto:mzilhao@ffn.ub.es> wrote:
hi all, i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6 when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine. i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same. i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380 so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899 this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash. i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same. has anyone seen such an error before? anything else i could provide to help diagnose this? thanks, Miguel _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Miguel
Thanks for tracking this down. I believe this code looks already different on the current master. Can you just disable it, as you did? The set of active grid points is important for some very new post-processing and visualization tools, but you probably don't need it, and you don't need it in a checkpoint file unless you want to visualize the data stored there.
-erik
On Fri, Aug 7, 2015 at 8:31 AM, Miguel Zilhão mzilhao@ffn.ub.es wrote:
Erik,
some more info. i briefly went through git history and found the offending commit. up to Carpet commit d52042b867eea1e0770b6de87ff3929ab7b9d297 everything works fine for me. from commit 69c73fb12d2ee41f01928122cd178d1bae9f8e13 onward i get the error i mentioned.
indeed, in the current state, if i remove writing of the "active" attribute everything works just fine:
diff --git a/CarpetIOHDF5/src/Output.cc b/CarpetIOHDF5/src/Output.cc index ec9b090..30c017d 100644 --- a/CarpetIOHDF5/src/Output.cc +++ b/CarpetIOHDF5/src/Output.cc @@ -894,10 +894,11 @@ static int AddAttributes (const cGH *const cctkGH, const char *fullname, HDF5_ERROR (H5Awrite (attr, H5T_NATIVE_INT, &ioffset[0])); HDF5_ERROR (H5Aclose (attr));
- ostringstream buf;
- buf << (vdd.at(Carpet::map)->
local_boxes.at(mglevel).at(refinementlevel).at(component).active);
- WriteAttribute(dataset, "active", buf.str().c_str());
- // ostringstream buf;
- // buf << (vdd.at(Carpet::map)->
- // local_boxes.at
(mglevel).at(refinementlevel).at(component).active);
// WriteAttribute(dataset, "active", buf.str().c_str());
}
if (is_index) {
i can't tell why it breaks once that bit of code is in, though...
thanks, Miguel
On 06/08/15 15:43, Miguel Zilhão wrote:
hi Erik,
Can you look at https://trac.einsteintoolkit.org/ticket/1800? Can you
try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem?
thanks, but this does not seem to help... i still get the same error and backtrace in gdb:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x0000000000c70402 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b1b930, fullname=fullname@entry=0x3e52d760 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry =0x3df4c990, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x0000000000c72e5a in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b1b930, outfile=outfile@entry=167
Miguel
On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão <mzilhao@ffn.ub.es <mailto:
mzilhao@ffn.ub.es>> wrote:
hi all, i'm running latest ET Hilbert on openSUSE tumbleweed and i'm havingthe following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error
INFO (CarpetIOHDF5):
INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0,simulation time 0 INFO (CarpetIOHDF5):
terminate called after throwing an instance of'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6
when writing the checkpoint file. this only happens if i run with more than one MPI process; with asingle processor it runs fine.
i'm compiling with gcc-5, but i find the same problem with gcc-4.8.i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same.
i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(charconst*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry =0x3df96370, bbox=..., dataset=83886080, is_index=false) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry =true,indexfile=indexfile@entry=-1) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380
so the relevant bits of code seem to be inCarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899
this seems to be triggered when writing hdf5 output in parallel. ifi remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash.
i have also tried removing all my hdf5 libs and configuring ET withHDF5_DIR=BUILD, but the outcome was the same.
has anyone seen such an error before? anything else i could provideto help diagnose this?
thanks, Miguel _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
hi Erik,
actually, i've just noticed that, for whatever reason, i was using Carpet's master branch... the rest of my ET installation does point to ET_2015_05 (and i thought Carpet did as well). i must have typed 'git checkout master' in the wrong place at some point, and never noticed it.
so the 'fix' i mentioned in my previous email applies to Carpet's current master branch. if i checkout branch ET_2015_05 and compile that (as i thought i was doing), it works fine without me needing to change anything.
thanks, Miguel
On 07/08/15 15:08, Erik Schnetter wrote:
Miguel
Thanks for tracking this down. I believe this code looks already different on the current master. Can you just disable it, as you did? The set of active grid points is important for some very new post-processing and visualization tools, but you probably don't need it, and you don't need it in a checkpoint file unless you want to visualize the data stored there.
-erik
On Fri, Aug 7, 2015 at 8:31 AM, Miguel Zilhão <mzilhao@ffn.ub.es mailto:mzilhao@ffn.ub.es> wrote:
Erik, some more info. i briefly went through git history and found the offending commit. up to Carpet commit d52042b867eea1e0770b6de87ff3929ab7b9d297 everything works fine for me. from commit 69c73fb12d2ee41f01928122cd178d1bae9f8e13 onward i get the error i mentioned. indeed, in the current state, if i remove writing of the "active" attribute everything works just fine: diff --git a/CarpetIOHDF5/src/Output.cc b/CarpetIOHDF5/src/Output.cc index ec9b090..30c017d 100644 --- a/CarpetIOHDF5/src/Output.cc +++ b/CarpetIOHDF5/src/Output.cc @@ -894,10 +894,11 @@ static int AddAttributes (const cGH *const cctkGH, const char *fullname, HDF5_ERROR (H5Awrite (attr, H5T_NATIVE_INT, &ioffset[0])); HDF5_ERROR (H5Aclose (attr)); - ostringstream buf; - buf << (vdd.at <http://vdd.at>(Carpet::map)-> - local_boxes.at <http://local_boxes.at>(mglevel).at(refinementlevel).at(component).active); - WriteAttribute(dataset, "active", buf.str().c_str()); + // ostringstream buf; + // buf << (vdd.at <http://vdd.at>(Carpet::map)-> + // local_boxes.at <http://local_boxes.at>(mglevel).at(refinementlevel).at(component).active); + // WriteAttribute(dataset, "active", buf.str().c_str()); + } if (is_index) { i can't tell why it breaks once that bit of code is in, though... thanks, Miguel On 06/08/15 15:43, Miguel Zilhão wrote: hi Erik, Can you look at <https://trac.einsteintoolkit.org/ticket/1800>? Can you try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem? thanks, but this does not seem to help... i still get the same error and backtrace in gdb: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x0000000000c70402 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b1b930, fullname=fullname@entry=0x3e52d760 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df4c990, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x0000000000c72e5a in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b1b930, outfile=outfile@entry=167 Miguel On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão <mzilhao@ffn.ub.es <mailto:mzilhao@ffn.ub.es> <mailto:mzilhao@ffn.ub.es <mailto:mzilhao@ffn.ub.es>>> wrote: hi all, i'm running latest ET Hilbert on openSUSE tumbleweed and i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- terminate called after throwing an instance of 'std::out_of_range' what(): vector::_M_range_check: __n (which is 1) >= this->size() (which is 1) Rank 1 with PID 5958 received signal 6 when writing the checkpoint file. this only happens if i run with more than one MPI process; with a single processor it runs fine. i'm compiling with gcc-5, but i find the same problem with gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same. i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370, called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 <CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*)>, attribute=attribute@entry=0x1b4a4e8, data=data@entry=0x1b507d0, user_timer=...) at /home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380 so the relevant bits of code seem to be in CarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899 this seems to be triggered when writing hdf5 output in parallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash. i have also tried removing all my hdf5 libs and configuring ET with HDF5_DIR=BUILD, but the outcome was the same. has anyone seen such an error before? anything else i could provide to help diagnose this? thanks, Miguel _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> <mailto:Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>> http://lists.einsteintoolkit.org/mailman/listinfo/users -- Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu> <mailto:schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu>>> http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
Miguel
This was an error in the development version of CarpetIOHDF5 -- at one place, "component" should have been "local_component". This has now been corrected.
-erik
On Fri, Aug 7, 2015 at 11:22 AM, Miguel Zilhão mzilhao@ffn.ub.es wrote:
hi Erik,
actually, i've just noticed that, for whatever reason, i was using Carpet's master branch... the rest of my ET installation does point to ET_2015_05 (and i thought Carpet did as well). i must have typed 'git checkout master' in the wrong place at some point, and never noticed it.
so the 'fix' i mentioned in my previous email applies to Carpet's current master branch. if i checkout branch ET_2015_05 and compile that (as i thought i was doing), it works fine without me needing to change anything.
thanks, Miguel
On 07/08/15 15:08, Erik Schnetter wrote:
Miguel
Thanks for tracking this down. I believe this code looks already different on the current master. Can you just disable it, as you did? The set of active grid points is important for some very new post-processing and visualization tools, but you probably don't need it, and you don't need it in a checkpoint file unless you want to visualize the data stored there.
-erik
On Fri, Aug 7, 2015 at 8:31 AM, Miguel Zilhão <mzilhao@ffn.ub.es mailto: mzilhao@ffn.ub.es> wrote:
Erik, some more info. i briefly went through git history and found theoffending commit. up to Carpet commit d52042b867eea1e0770b6de87ff3929ab7b9d297 everything works fine for me. from commit 69c73fb12d2ee41f01928122cd178d1bae9f8e13 onward i get the error i mentioned.
indeed, in the current state, if i remove writing of the "active"attribute everything works just fine:
diff --git a/CarpetIOHDF5/src/Output.cc b/CarpetIOHDF5/src/Output.cc index ec9b090..30c017d 100644 --- a/CarpetIOHDF5/src/Output.cc +++ b/CarpetIOHDF5/src/Output.cc @@ -894,10 +894,11 @@ static int AddAttributes (const cGH *constcctkGH, const char *fullname, HDF5_ERROR (H5Awrite (attr, H5T_NATIVE_INT, &ioffset[0])); HDF5_ERROR (H5Aclose (attr));
- ostringstream buf; - buf << (vdd.at <http://vdd.at>(Carpet::map)-> - local_boxes.at <http://local_boxes.at(mglevel).at(refinementlevel).at(component).active);
- WriteAttribute(dataset, "active", buf.str().c_str()); + // ostringstream buf; + // buf << (vdd.at <http://vdd.at>(Carpet::map)-> + // local_boxes.at <http://local_boxes.at(mglevel).at(refinementlevel).at(component).active);
+ // WriteAttribute(dataset, "active", buf.str().c_str()); + } if (is_index) { i can't tell why it breaks once that bit of code is in, though... thanks, Miguel On 06/08/15 15:43, Miguel Zilhão wrote: hi Erik, Can you look at <https://trac.einsteintoolkit.org/ticket/1800>?Can you try replacing "reflevel" in line 816 of the file CarpetIOHDF5/src/Output.cc with "refinementlevel", and see whether this avoids the problem?
thanks, but this does not seem to help... i still get the sameerror and backtrace in gdb:
#6 0x00007ffff4f6d4d5 in std::__throw_out_of_range_fmt(charconst*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x0000000000c70402 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry=0x1b1b930, fullname=fullname@entry=0x3e52d760 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df4c990, bbox=..., dataset=83886080, is_index=false) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x0000000000c72e5a in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b1b930, outfile=outfile@entry=167
Miguel On Wed, Aug 5, 2015 at 8:11 PM, Miguel Zilhão <mzilhao@ffn.ub.es mailto:mzilhao@ffn.ub.es <mailto:mzilhao@ffn.ub.es mailto: mzilhao@ffn.ub.es>> wrote:
hi all, i'm running latest ET Hilbert on openSUSE tumbleweedand i'm having the following issue. upon trying to run a simple head-on collision configuration with McLachlan (attached parameter file), i get the error
INFO (CarpetIOHDF5):
INFO (CarpetIOHDF5): Dumping initial checkpoint atiteration 0, simulation time 0 INFO (CarpetIOHDF5):
terminate called after throwing an instance of'std::out_of_range' what(): vector::_M_range_check: __n (which is 1)
= this->size() (which is 1)
Rank 1 with PID 5958 received signal 6 when writing the checkpoint file. this only happens if i run with more than one MPIprocess; with a single processor it runs fine.
i'm compiling with gcc-5, but i find the same problemwith gcc-4.8. i was running this very same configuration just fine a couple of months ago, so it must have been some update i've made in the meantime (either to my OS or to ET). i've also tried with different configurations and the outcome is the same.
i've ran this through gdb, here's the relevant output: #6 0x00007ffff4f6d4d5 instd::__throw_out_of_range_fmt(char const*, ...) () from /usr/lib64/libstdc++.so.6 #7 0x00000000005bb398 in _M_range_check (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:803 #8 at (__n=<optimized out>, this=<optimized out>) at /usr/include/c++/5/bits/stl_vector.h:824 #9 CarpetIOHDF5::AddAttributes (cctkGH=cctkGH@entry =0x1b507d0, fullname=fullname@entry=0x3f2434a0 "ML_BSSN::cA", vdim=3, refinementlevel=refinementlevel@entry=0, request=request@entry=0x3df96370, bbox=..., dataset=83886080, is_index=false) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:899 #10 0x00000000005bdea4 in CarpetIOHDF5::WriteVarChunkedParallel ( cctkGH=cctkGH@entry=0x1b507d0, outfile=outfile@entry=16777216, io_bytes=@0x7fffffffc980: 1110772, request=0x3df96370,
called_from_checkpoint=called_from_checkpoint@entry=true,indexfile=indexfile@entry=-1) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc:706 #11 0x00000000005a233e in CarpetIOHDF5::Checkpoint (cctkGH=0x1b507d0, called_from=0) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/CarpetIOHDF5/src/CarpetIOHDF5.cc:1277 #12 0x000000000041f0d5 in CCTK_CallFunction ( function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), fdata=fdata@entry=0x1b4a4e8, data=data@entry=0x1b507d0) at /home/mzilhao/Trabalho/projectos/ET/Cactus/src/main/ScheduleInterface.c:312 #13 0x0000000000ef6499 in Carpet::CallScheduledFunction ( time_and_mode=time_and_mode@entry=0x1174842 "Meta mode", function=function@entry=0x5a2da0 CarpetIOHDF5::CarpetIOHDF5_InitialDataCheckpoint(cGH*), attribute=attribute@entry=0x1b4a4e8, data=data@entry =0x1b507d0, user_timer=...) at
/home/mzilhao/Trabalho/projectos/ET/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:380
so the relevant bits of code seem to be inCarpetIOHDF5/src/Output.cc:706 and CarpetIOHDF5/src/Output.cc:899
this seems to be triggered when writing hdf5 output inparallel. if i remove checkpointing the run goes fine, and i do get regular 2D hdf5 output. this does not seem to be written in parallel, though, as i get only one file per grid function/group. so it seems to be the parallel output that triggers the crash.
i have also tried removing all my hdf5 libs andconfiguring ET with HDF5_DIR=BUILD, but the outcome was the same.
has anyone seen such an error before? anything else icould provide to help diagnose this?
thanks, Miguel _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> <mailto:Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org>> http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter <schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu> <mailto:schnetter@cct.lsu.edu <mailto:schnetter@cct.lsu.edu
http://www.perimeterinstitute.ca/personal/eschnetter/ _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Miguel Zilhão