Trac May 2011

trac@lists.einsteintoolkit.org

1 participants
122 discussions

[Einstein Toolkit] #163: Assertion `tl>=0 and tl<timelevels' failed error
by Einstein Toolkit 13 Nov '12

13 Nov '12

#163: Assertion `tl>=0 and tl<timelevels' failed error ------------------------------------+--------------------------------------- Reporter: azebrowski@… | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Keywords: | ------------------------------------+--------------------------------------- I'm currently running a Cactus simulation based off the ET mclachlan parameter file. I have some custom thorns which force checkpointing every iteration, and then write new parameter files. The new parameter files are used to run specific functions from the host simulation ("spawning"), but I'm having some problems resuming simulations. Currently, I get this error when resuming: INFO (Carpet): GF: rhs: 818k active, 1440k owned (+76%), 1896k total (+32%), 328 steps/time cactus_sim: /home/azebrowski/Cactus/arrangements/Carpet/CarpetLib/src/th.hh:79: double th::get_time(int, int, int) const: Assertion `tl>=0 and tl<timelevels' failed. [cyder:32759] *** Process received signal *** [cyder:32759] Signal: Aborted (6) I'm guessing there's a parameter I'm not setting properly in my child simulation, could anyone give me a pointer to where I should be looking? I looked for things relating to timelevels in the host/spawned parameter files, but didn't see anything that stood out. My parameter files used and full output are attached to this email, with the disclaimer that I modified the spawned parameter file to run every function instead of skipping some in an attempt to bypass any problems that could be caused by skipping some Carpet function on accident. I've made a bzipped tarball containing the checkpointed data from the simulation. It contains several parameter files. The parameter file of interest here is spawn.par, as it doesn't use any of my custom code but still causes Cactus to abort with an error. I left the other parameter files in on the off chance that I might need to refer to them later. Here is the source parameter file, which creates the spawned simulation: http://www.cct.lsu.edu/~azebrowski/ml-ahfinder-spawn.par Here is the spawned simulation's parameter file: http://www.cct.lsu.edu/~azebrowski/spawn.par Here is the full checkpointed data and another copy of the spawned parameter file: http://www.cct.lsu.edu/~azebrowski/data.tar.bz2 Other information: I ran the simulation using OpenMP with 12 cores to generate the checkpointed data. I've also tried MPI, but that didn't seem to make a difference. Thornlist: http://www.cct.lsu.edu/~azebrowski/ThornList gcc: azebrowski@cyder:~/Cactus$ gcc -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.3-4ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable- shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i486 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) Fortran is gfortran-4.4 I'm using the Mercurial version of Carpet, and the ET development thorns. If you need more information, please let me know. -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/163> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 13

[Einstein Toolkit] #434: Keep track of masked-out volume in CarpetMask
by Einstein Toolkit 08 Nov '12

08 Nov '12

#434: Keep track of masked-out volume in CarpetMask -------------------------+-------------------------------------------------- Reporter: eschnett | Owner: eschnett Type: enhancement | Status: new Priority: minor | Milestone: Component: Carpet | Version: Keywords: | -------------------------+-------------------------------------------------- Keep track of the volume that is masked out by CarpetMask, and take this volume into account when checking in CarpetReduce that the integral over the simulation domain equals the domain volume. -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/434> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 30

[Einstein Toolkit] #361: delta_time when setting up ID with explicit time dependence
by Einstein Toolkit 28 Oct '12

28 Oct '12

#361: delta_time when setting up ID with explicit time dependence ------------------------------------------+--------------------------------- Reporter: eloisa.bentivegna@… | Owner: eschnett Type: defect | Status: new Priority: minor | Milestone: Component: Carpet | Version: ET_2010_11 Keywords: | ------------------------------------------+--------------------------------- In CallInitial, delta_time is used to calculate the time corresponding to the different timelevels (line 399 of Carpet/src/Initialise.cc). At this stage, though, delta_time is always equal to 1, leading to potentially very separated initial-data slices when using init_each_timelevel and an initial-data thorn that uses cctk_time explicitly. Should cctkGH->cctk_delta_time be used here instead? -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/361> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 19

[Einstein Toolkit] #323: GetComponents overwrites thorn list
by Einstein Toolkit 25 Oct '12

25 Oct '12

#323: GetComponents overwrites thorn list ---------------------------+------------------------------------------------ Reporter: eschnett | Owner: eric9 Type: defect | Status: new Priority: critical | Milestone: Component: GetComponents | Version: Keywords: | ---------------------------+------------------------------------------------ I specified a thorn list located in the main Cactus directory. GetComponents overwrote this thorn list with its own thorn list (the one it generates automatically). -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/323> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 18

[Einstein Toolkit] #176: Test parameter files without running them
by Einstein Toolkit 23 Oct '12

23 Oct '12

#176: Test parameter files without running them -------------------------+-------------------------------------------------- Reporter: hinder | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Cactus | Version: Keywords: | -------------------------+-------------------------------------------------- It would be useful to be able to check that the parameters of a run are OK before running it. Since there can be long queue times on supercomputers, it would be good to do this on a local machine which might not have as much memory as is required for the full run. Currently, running the job would cause memory exhaustion on the smaller machine. One solution would be an additional command-line argument to Cactus --exit-after-paramcheck which stops the run cleanly after the PARAMCHECK Cactus bin. This is preferable to a new Cactus parameter, as it would not require the parameter file to be modified. One could imagine this also being potentially used by simfactory automatically when submitting a job (though this would only work if you could run MPI executables on the head node). -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/176> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 12

[Einstein Toolkit] #359: Add MDB entry for Pandora at LSU
by Einstein Toolkit 22 Oct '12

22 Oct '12

#359: Add MDB entry for Pandora at LSU ------------------------+--------------------------------------------------- Reporter: eschnett | Owner: mthomas Type: task | Status: new Priority: major | Milestone: Component: SimFactory | Version: Keywords: | ------------------------+--------------------------------------------------- -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/359> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 3

[Einstein Toolkit] #412: Split Appendices
by Einstein Toolkit 22 Oct '12

22 Oct '12

#412: Split Appendices -------------------------+-------------------------------------------------- Reporter: eschnett | Owner: Type: enhancement | Status: new Priority: minor | Milestone: Component: Cactus | Version: Keywords: | -------------------------+-------------------------------------------------- Currently, the appendices of the users' guide are repeated into the reference manual. This is somewhat confusing, and causes problems because the appendices cannot easily reference other sections (e.g. "see page 15" doesn't make sense since one doesn't know in which document this will be read). I suggest to split the appendices and have some of them in the users' guide while moving others into the reference manual, so that each appendix is included in only one document -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/412> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 1

[Einstein Toolkit] #420: newest revision of simfactory 2.0 submit three instead of one job
by Einstein Toolkit 11 Aug '12

11 Aug '12

#420: newest revision of simfactory 2.0 submit three instead of one job ----------------------------------------------+----------------------------- Reporter: alexander.beck-ratzka@… | Owner: mthomas Type: defect | Status: new Priority: minor | Milestone: Component: SimFactory | Version: ET_2010_11 Keywords: | ----------------------------------------------+----------------------------- I am submitting a simulation using simfactory with the command: [snip] simfactory/bin/sim submit test-whisky-openmp --configuration test-whisky- openmp --parfile=whisky-openmp-test-ali.par --verbose --walltime=48:00:00 --procs=16 --ppn=4 --num-threads=4 --machine=damiana --queue=intel.q [snip] sim then comes with the messages: [snip] Info: Simfactory command: simfactory/bin/../lib/sim.py "submit" "test- whisky-openmp" "--configuration" "test-whisky-openmp" "--parfile=whisky- openmp-test-ali.par" "--verbose" "--walltime=48:00:00" "--procs=16" "-- ppn=4" "--num-threads=4" "--machine=damiana" "--queue=intel.q" Info: Version 1331M The Simulation Factory: Manage Cactus simulations Info: defs: /home/alibeck/programme/Cactus- Luca/Cactus/simfactory/etc/defs.ini Info: defs.local: /home/alibeck/programme/Cactus- Luca/Cactus/simfactory/etc/defs.local.ini Info: Cactus Directory: /home/alibeck/programme/Cactus-Luca/Cactus Info: simenv.COMMAND: submit Info: Executing command: submit Info: Assigned restart_id of: 0002 Info: Found the following restart_ids: [0, 1] Info: Maximum restart id determined to be: 0001 Assigned restart id: 2 Info: Simulation is inactive: submitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0002/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0002/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test- whisky-openmp/output-0002/SIMFACTORY/SubmitScript Submit finished, job id is 259460 Info: Restart 2 is active Info: Assigned restart_id of: 0003 Info: Found the following restart_ids: [0, 1, 2, 2] Info: Maximum restart id determined to be: 0002 Assigned restart id: 3 Info: Simulation is active: presubmitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0003/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0003/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test- whisky-openmp/output-0003/SIMFACTORY/SubmitScript Submit finished, job id is 259461 Info: Restart 2 is active Info: Assigned restart_id of: 0004 Info: Found the following restart_ids: [0, 1, 2, 2, 3] Info: Maximum restart id determined to be: 0003 Assigned restart id: 4 Info: Simulation is active: presubmitting Info: Job allocation information: Info: System: nodes=170 cores/node=4 threads/process=4 Info: Requested: nodes=4 cores=16 cores/node=4 Info: Run: processes=4 threads=16 threads/process=4 Info: Distribution: processes/node=1 threads/node=4 Info: Ratio: threads/core=1.000 cores/thread=1.000 Info: writing to internalDir: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0004/SIMFACTORY Info: saving substituted submitscript contents to: /lustre/AEI/alibeck/simulations/test-whisky- openmp/output-0004/SIMFACTORY/SubmitScript Executing submit command: qsub /lustre/AEI/alibeck/simulations/test- whisky-openmp/output-0004/SIMFACTORY/SubmitScript Submit finished, job id is 259462 [snip] As a result three jobs are queued. [snip] qstat job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- 259460 0.00000 test-whisk alibeck qw 04/29/2011 10:26:55 16 259461 0.00000 test-whisk alibeck hqw 04/29/2011 10:26:56 16 259462 0.00000 test-whisk alibeck hqw 04/29/2011 10:26:56 16 [snip] What is going wrong here? -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/420> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 3

[Einstein Toolkit] #380: AHFinderDirect failure with qc0-mclachlan.par
by Einstein Toolkit 10 Aug '12

10 Aug '12

#380: AHFinderDirect failure with qc0-mclachlan.par --------------------+------------------------------------------------------- Reporter: hinder | Owner: Type: defect | Status: new Priority: major | Milestone: ET_2011_05 Component: Other | Version: Keywords: | --------------------+------------------------------------------------------- Trying to run the par/qc0-mclachlan.par in the ET trunk gives the following error: INFO (AHFinderDirect): proc 0: searching for horizons 1,5/6 WARNING level 0 in thorn CarpetInterp processor 0 host kop193.datura.admin (line 1683 of /home/ianhin/Cactus/etrelease/arrangements/Carpet/CarpetInterp/src/interp.cc): -> Grid function "AHFINDERDIRECT::ahmask" has only 1 active time levels on refinement level 1; this is not en ough for time interpolation See also #373, which might be related. -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/380> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 10

[Einstein Toolkit] #177: EXC_BAD_ACCESS in CarpetIOASCII
by Einstein Toolkit 06 Aug '12

06 Aug '12

#177: EXC_BAD_ACCESS in CarpetIOASCII --------------------+------------------------------------------------------- Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Keywords: | --------------------+------------------------------------------------------- I am trying to run a minimal parameter file for testing purposes (attached). Cactus gets as far as printing the iteration 0 info line and then appears to hang. When I run in a debugger, it reports: Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 13 at address: 0x0000000000000000 0x0000000100b858b7 in CarpetIOASCII::WriteASCII<0> (os=@0x7fff5fbfc5e0, gfdatas={<std::_Vector_base<const gdata *, std::allocator<const gdata *> >> = {_M_impl = {<allocator<const gdata *>> = {<__gnu_cxx::new_allocator<const gdata *>> = {<No data fields>}, <No data fields>}, _M_start = 0x109f0ea10, _M_finish = 0x109f0eab8, _M_end_of_storage = 0x109f0eab8}}, <No data fields>}, gfext=@0x7fff5fbfc960, vi=0, time=0, org=@0x7fff5fbfcafc, dirs=@0x7fff5fbfd150, rl=0, ml=0, m=0, c=0, tl=0, coord_time=0, coord_lower=@0x7fff5fbfc9d8, coord_upper=@0x7fff5fbfc9f0) at ioascii.cc:1378 1378 switch (vartype) { (gdb) bt #0 0x0000000100b858b7 in CarpetIOASCII::WriteASCII<0> (os=@0x7fff5fbfc5e0, gfdatas={<std::_Vector_base<const gdata *, std::allocator<const gdata *> >> = {_M_impl = {<allocator<const gdata *>> = {<__gnu_cxx::new_allocator<const gdata *>> = {<No data fields>}, <No data fields>}, _M_start = 0x109f0ea10, _M_finish = 0x109f0eab8, _M_end_of_storage = 0x109f0eab8}}, <No data fields>}, gfext=@0x7fff5fbfc960, vi=0, time=0, org=@0x7fff5fbfcafc, dirs=@0x7fff5fbfd150, rl=0, ml=0, m=0, c=0, tl=0, coord_time=0, coord_lower=@0x7fff5fbfc9d8, coord_upper=@0x7fff5fbfc9f0) at ioascii.cc:1378 #1 0x0000000100b5b7ee in CarpetIOASCII::IOASCII<0>::OutputDirection (cctkGH=0x109f073f0, vindex=0, alias={_M_dataplus = {<allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x10a06c1c8 "carpet::timing"}}, basefilename={_M_dataplus = {<allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x10a06ba68 "minimal/carpet::timing"}}, dirs=@0x7fff5fbfd150, is_new_file=true, truncate_file=true) at ioascii.cc:610 #2 0x0000000100b554c4 in CarpetIOASCII::IOASCII<0>::OutputVarAs (cctkGH=0x109f073f0, varname=0x10a06c170 "CARPET::physical_time_per_hour", alias=0x10a06be00 "carpet::timing") at ioascii.cc:480 #3 0x0000000100b57370 in CarpetIOASCII::IOASCII<0>::TriggerOutput (cctkGH=0x109f073f0, vindex=0) at ioascii.cc:359 #4 0x0000000100b542b7 in CarpetIOASCII::IOASCII<0>::OutputGH (cctkGH=0x109f073f0) at ioascii.cc:239 #5 0x000000010329a006 in Carpet::OutputGH (cctkGH=0x109f073f0) at OutputGH.cc:55 #6 0x0000000103293b25 in Carpet::OutputGH (where=0x104c70f3c "Initialise::CallAnalysis", cctkGH=0x109f073f0) at Initialise.cc:1382 #7 0x000000010328d432 in Carpet::CallAnalysis (cctkGH=0x109f073f0) at Initialise.cc:530 #8 0x00000001032892d4 in Carpet::Initialise (fc=0x7fff5fbfe2b0) at Initialise.cc:118 #9 0x000000010005806a in main (argc=4, argv=0x7fff5fbfe318) at flesh.cc:80 I do not see how this line "switch(vartype)" where vartype is declared on the stack, could lead to this error. This configuration has been rebuilt with -O0 (Intel compiler) and debugging enabled. This is with the current production (Git) version of Carpet running on Mac OS 10.6.5. The code is compiled as 64 bit. This parameter file has worked in the past, though with 32 bit on Mac OS 10.5. -- Ticket URL: <https://trac.einsteintoolkit.org/ticket/177> Einstein Toolkit <http://einsteintoolkit.org> The Einstein Toolkit

1 6

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac May 2011