Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

Another restart from checkpoint failure
by Jakob Hansen 17 Feb '11

17 Feb '11

Hi, I am having problems with checkpoint restarting for high resolution Carper/McLachlan simulations. For more coarse resolutions I have no problems with checkpoint restarting, nor do I have any problems if the high resolution simulations run for only a few M. But when I do high-resolution simulations until stopped by wall-time and try to restart from checkpoint, I get the following error output : ################################################################# HDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) thread 0: #000: H5Dio.c line 174 in H5Dread(): can't read data major: Dataset minor: Read failed #001: H5Dio.c line 404 in H5D_read(): can't read data major: Dataset minor: Read failed #002: H5Dchunk.c line 1724 in H5D_chunk_read(): unable to read raw data chunk major: Low-level I/O minor: Read failed #003: H5Dchunk.c line 2737 in H5D_chunk_lock(): data pipeline read failed major: Data filters minor: Filter operation failed #004: H5Z.c line 1116 in H5Z_pipeline(): filter returned failure during read major: Data filters minor: Read failed #005: H5Zdeflate.c line 133 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression major: Resource unavailable minor: No space available for allocation WARNING level 1 in thorn CarpetIOHDF5 processor 104 host tachyon3167 (line 1102 of /home01/r632kgw/jakob/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Input.cc): -> HDF5 call 'H5Dread (dataset, datatype, memspace, filespace, xfer, cctkGH->data[patch->vindex][timelevel])' returned error code -1 . . ......etc, etc, ################################################################## Any ideas for possible cause and solution to this? Thanks, Jakob

3 3

Re: [Users] Another restart from checkpoint failure
by Oleg Korobkin 17 Feb '11

17 Feb '11

Hi Jakob, I had the same problem some time ago. Since your HDF5 files are compressed, Cactus tries to unpack them before reading, and this operation somehow is very memory-intensive. Try to repack your checkpoint files before restarting to reduce compression level to zero. You can use the standard h5repack utility for that. Here's a small script to uncompress all checkpoint files and put them to an $OUTDIR directory: ---------------- #!/bin/bash ITER=5555 # which iteration to use OUTDIR=$SCRATCH/tmp # output directory for f in checkpoint.chkpt.it_${ITER}.file_*.h5; do echo processing $f... h5repack -i $f -o $OUTDIR/$f -f NONE done ---------------- This will make your checkpoint files 5-10 times larger, but because they don't need to be unpacked in RAM, less memory is required to read them. Also, try the following: 1. set: CarpetIOHDF5::open_one_input_file_at_a_time = yes IO::abort_on_io_errors = yes 2. use less cores per node. 3. try running on larger number of processors. Cheers, - Oleg Korobkin Frank Loeffler wrote: > Hi, > > On Thu, Feb 17, 2011 at 03:24:04PM +0900, Jakob Hansen wrote: >> #005: H5Zdeflate.c line 133 in H5Z_filter_deflate(): memory allocation >> failed for deflate uncompression >> major: Resource unavailable >> minor: No space available for allocation > >> Any ideas for possible cause and solution to this? > > Looking at these error messages I suggest you first do an h5ls/h5dump to > see if the files are actually ok. If that is so, what I would suspect next > is that reading the files takes more memory than is available, as indicated > by the message above. In this case I suggest to try the workarounds > mentioned in this thread: > > http://lists.einsteintoolkit.org/pipermail/users/2011-February/000852.html > > If this also doesn't help I would suggest to try again using more > available memory, e.g. using more nodes. > > Frank > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Users mailing list > Users(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users

1 0

Re: [Users] compilation problem on x86-64
by Erik Schnetter 15 Feb '11

15 Feb '11

Maria Another solution could be to add the option -m128bit-long-double to all of CFLAGS, CXXFLAGS, F77FLAGS, and F90FLAGS. -erik On Tue, Feb 15, 2011 at 2:07 PM, Erik Schnetter <schnetter(a)cct.lsu.edu> wrote: > Maria > > Do you need real*16 for your work? Otherwise it would be easiest to > just not use this type. For Cactus itself, you can use either of the > options > > REAL16_KIND = 10 > > or > > DISABLE_REAL16 = yes > > to choose what you want to do. Do either of these options help? > > -erik > > On Sat, Feb 12, 2011 at 3:00 PM, Babiuc-Hamilton, Maria > <babiuc(a)marshall.edu> wrote: >> Hi, >> >> I found on the web that x86-64 CPUs do not support require 16-bytes precision, and gfortran 4.5 supports only up to kind=10. I compiled just fine using ifort on another x86-64, but it seems that the intel compilers can do this in software. The new gfortran 4.6 would support a quad precision in software. Was anybody successful in compiling on a x86-64 with GNU compilers? >> >> best, >> Maria >> >> ________________________________________ >> From: users-bounces(a)einsteintoolkit.org [users-bounces(a)einsteintoolkit.org] on behalf of Babiuc-Hamilton, Maria [babiuc(a)marshall.edu] >> Sent: Friday, February 11, 2011 9:30 PM >> To: users(a)einsteintoolkit.org >> Subject: [Users] compilation problem on 10.10 ubuntu >> >> Hello, >> >> I have problems compiling einsteintoolkit on Ubuntu 10.10 with gnu 4.4 or 4.5 compilers. The errors are: >> >> H5fortran_detect.f90:164.26: >> >> REAL(KIND= 16 ) :: a >> 1 >> Error: Kind 16 not supported for type REAL at (1) >> H5fortran_detect.f90:167.21: >> >> a_size = SIZEOF(a) >> 1 >> Error: Symbol 'a' at (1) has no IMPLICIT type >> make[4]: *** [H5fortran_detect.o] Error 1 >> make[3]: *** [all-recursive] Error 1 >> make[2]: *** [all-recursive] Error 1 >> Building HDF5... >> >> CST error 1: >> -> Configuration script for thorn HDF5 returned exit code 1 >> (no error message) >> >> Maria >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> > > > > -- > Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/ > -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.cct.lsu.edu/~eschnett/

1 0

Meeting minutes
by Roland Haas 14 Feb '11

14 Feb '11

Present were: Frank, Tanja, Roland, Peter, Ian, Eloisa, Bruno, Erik, Josh ET paper: * please have a look at your assigned section and either work on it or let Frank know if you do not have time to do so * if you have no section assigned, then please still read through the paper to see if there is anything obviously missing * Please see what you can do to get this done. Now. Visualization scripts: * please add your own methods, even if it is just "I do what XXX does" * Ian has a set of Mathematica routines that he uses and that have support for the data that (his) simulations produce: HDF5, some ASCII files ** currently used internally at AEI ** will be made public eventually, but no fixed timeframe exists for that yet ** if we want to run these remotely on the cluster then we will have to check/ask for Mathematica licenses (Erik will do) ** Ian will give a short EVO presentation on the routines next week * other options are python/plotlib which (hearsay) Christian has been using for a while ** would not require commercial software ** more lightweight than Mathematica ** has to be written first (Bruno will have a look, talk to Christian) * gnuplot/(g)awk/shell ** this is what many of us currently use ** (speaking for GT) these work, but are often very messy since the evolved from hacks * there are two levels of visualization: ** online for a quick overview of what the simulation is doing ** in depth for science/publications ** Erik suggested to have one framework for both * ET workshop: * fall user workshop: nothing discussed since Christian was not here * spring internal workshop at GT: go-ahead from GT, we have to find a convenient date though. ** doodle poll: http://doodle.com/ayigys5kv7zzyxna ** topics will be MHD and the EOS (EOS_Omni) with hopefully introductions to bring everyone up to speed on MHD and the EOS ** if possible will also present the current state of the visualization ** current possible (as of writing this email) dates are Sat Apr 23rd to Wed Apr 27th (incl.) which is the week before the APS April meeting ** if this shrinks to zero we will coarse grain and ask for possible weeks, then for days within possible weeks Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

Meeting Reminder
by Frank Loeffler 14 Feb '11

14 Feb '11

Hi, This is a reminder about our usual Monday Einstein Toolkit phone call at 10 am US central time. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/info/support/ Some points of tomorrows discussion will be: - Progress on the ET paper. This seems to become a common theme. Please see what you can do to get this done. Now. - ET workshops (hands-on in spring, new-users in fall) http://doodle.com/ayigys5kv7zzyxna - "Poll" Visualization of simulation results Please add yourself to https://docs.einsteintoolkit.org/et-docs/Visualization_of_simulation_results Frank Löffler

2 1

restart failure from checkpoint
by Hee Il Kim 14 Feb '11

14 Feb '11

Hi, I used to encounter restart failure with the following messages. This seems to be quite often at the moment (3 from 6 restarts). It doesn't seem to be relevant with NaNs. Is there any way to check my system I/O with Cactus, other than the benchmark I/O given in the homepage of Cactus? Thanks, Hee Il ################# .... INFO (ADMMacros): Spatial finite differencing order: 4 INFO (Time): Timestep set to 0.0703125 (courant_static) INFO (CarpetIOHDF5): reading grid variables on mglevel 0 reflevel 4 WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable AHFINDERDIRECT::ahmask on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::dens on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 1 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::tau on rl 4 and tl 2 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::scon[0] on rl 4 and tl 0 not read completely. Will have to look for it in other files. WARNING[L1,P0] (CarpetIOHDF5): Variable GRHYDRO::scon[0] on rl 4 and tl 1 not read completely. Will have to look for it in other files. ... ...

4 4

compilation problem on x86-64
by Babiuc-Hamilton, Maria 12 Feb '11

12 Feb '11

Hi, I found on the web that x86-64 CPUs do not support require 16-bytes precision, and gfortran 4.5 supports only up to kind=10. I compiled just fine using ifort on another x86-64, but it seems that the intel compilers can do this in software. The new gfortran 4.6 would support a quad precision in software. Was anybody successful in compiling on a x86-64 with GNU compilers? best, Maria ________________________________________ From: users-bounces(a)einsteintoolkit.org [users-bounces(a)einsteintoolkit.org] on behalf of Babiuc-Hamilton, Maria [babiuc(a)marshall.edu] Sent: Friday, February 11, 2011 9:30 PM To: users(a)einsteintoolkit.org Subject: [Users] compilation problem on 10.10 ubuntu Hello, I have problems compiling einsteintoolkit on Ubuntu 10.10 with gnu 4.4 or 4.5 compilers. The errors are: H5fortran_detect.f90:164.26: REAL(KIND= 16 ) :: a 1 Error: Kind 16 not supported for type REAL at (1) H5fortran_detect.f90:167.21: a_size = SIZEOF(a) 1 Error: Symbol 'a' at (1) has no IMPLICIT type make[4]: *** [H5fortran_detect.o] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 Building HDF5... CST error 1: -> Configuration script for thorn HDF5 returned exit code 1 (no error message) Maria _______________________________________________ Users mailing list Users(a)einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users

1 0

compilation problem on 10.10 ubuntu
by Babiuc-Hamilton, Maria 11 Feb '11

11 Feb '11

Hello, I have problems compiling einsteintoolkit on Ubuntu 10.10 with gnu 4.4 or 4.5 compilers. The errors are: H5fortran_detect.f90:164.26: REAL(KIND= 16 ) :: a 1 Error: Kind 16 not supported for type REAL at (1) H5fortran_detect.f90:167.21: a_size = SIZEOF(a) 1 Error: Symbol 'a' at (1) has no IMPLICIT type make[4]: *** [H5fortran_detect.o] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 Building HDF5... CST error 1: -> Configuration script for thorn HDF5 returned exit code 1 (no error message) Maria

1 0

McLachlan constraint time levels
by Bruno Coutinho Mundim 10 Feb '11

10 Feb '11

Hi, I am receiving the messages of the following kind in my simulation with McLachlan: (line 1084 of ./Cactus/arrangements/Carpet/CarpetReduce/src/reduce.cc): -> Grid function "ML_BSSN::H" has only 1 time levels on refinement level 0; this is not enough for time interpolation and it is indeed true that the constraint variables has 1 time level only: CTK_REAL ML_Ham type=GF timelevels=1 tags='tensortypealias="Scalar" tensorweight=0' { H } "ML_Ham" I was wondering if we shouldn't be using 3 time levels instead. Thanks, Bruno.

3 3

new state of trouble tickets / bug reports
by Frank Loeffler 09 Feb '11

09 Feb '11

Hi It is nice to see that the bug tracking system [1] is filling with bug reports. Obviously we don't like that there are bugs, but it is nice that they are reported. However, especially with the review policies of the Cactus and Einstein toolkit it now becomes more and more difficult to have a good overview of reports which are already have patches and are only awaiting review. This is why I now added another state of a ticket: 'review'. It is meant to indicate that the issue at hand has a potential fix attached and is now awaiting review. It is now easy to find those reports using a filter, and corresponding links have been added to the trac home page. I hope that this helps to speed up the review process in the future. Frank

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users