[Users] HDF5 Error, Hilbert

[Geraint Pratten]

Ah, I think you could be right! It seems that I've messed up the submit/run
scripts along the line! Relevant output attached below. I'll try fixing
this now.

Thanks! Much appreciated!
Geraint

INFO (Carpet): MPI is enabled

INFO (Carpet): Carpet is running on 1 processes

WARNING level 1 from host node207.cm.cluster process 0

  while executing schedule bin (none), routine (no thorn)::(no routine)

  in thorn Carpet, file
/mnt/pact/gp234/NR/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:226:

  -> The environment variable CACTUS_NUM_PROCS is set to 6, but there are 1
MPI processes. This may indicate a severe problem with the MPI startup
mechanism.

INFO (Carpet): This is process 0

INFO (Carpet): OpenMP is enabled

INFO (Carpet): This process contains 6 threads, this is thread 0

WARNING level 2 from host node207.cm.cluster process 0

  while executing schedule bin (none), routine (no thorn)::(no routine)

  in thorn Carpet, file
/mnt/pact/gp234/NR/Cactus/arrangements/Carpet/Carpet/src/SetupGH.cc:256:

  -> Although OpenMP is enabled, the environment variable
CACTUS_NUM_THREADS is not set.

INFO (Carpet): There are 6 threads in total

INFO (Carpet): There are 6 threads per process

INFO (Carpet): This process runs on host node207, pid=29183

INFO (Carpet): This process runs on 24 cores: 0-23

INFO (Carpet): Thread 0 runs on 24 cores: 0-23

INFO (Carpet): Thread 1 runs on 24 cores: 0-23

INFO (Carpet): Thread 2 runs on 24 cores: 0-23

INFO (Carpet): Thread 3 runs on 24 cores: 0-23

INFO (Carpet): Thread 4 runs on 24 cores: 0-23

INFO (Carpet): Thread 5 runs on 24 cores: 0-23

On 5 October 2015 at 12:27, Ian Hinder <ian.hinder at aei.mpg.de> wrote:

>
> On 5 Oct 2015, at 12:43, Geraint Pratten <g.pratten at sussex.ac.uk> wrote:
>
> Hi,
>
> I am trying to run a simulation with the same output directory - though
> I've been removing the simulations after each run while trying to fix the
> above issue. I should have write access to the file and there should be
> sufficient storage.
>
>
> Can you check the standard output of the run and see if the number of
> processes and threads are what you expect?  You should see something like
> "Carpet is running on N processes with M threads".  Maybe there is a
> problem in MPI initialisation, and multiple processes are trying to write
> to the same HDF5 file at the same time.
>
>
>
> Geraint
>
> On 5 October 2015 at 11:34, Ian Hinder <ian.hinder at aei.mpg.de> wrote:
>
>>
>> On 1 Oct 2015, at 15:49, Erik Schnetter <schnetter at cct.lsu.edu> wrote:
>>
>> Geraint
>>
>> Some very fundamental HDF5 operations fail. This could be due to a file
>> system problem; maybe you don't have write access to the file, or your disk
>> is full. It could also be that there exists a corrupted HDF5 file, and you
>> are trying to write to it (extend it).
>>
>> HDF5 files can get corrupted if you have opened it for writing, and then
>> the application is interrupted or crashes, or you run out of disk space.
>> Even if you make free space later, a corrupted file remains corrupted.
>>
>>
>> Hi Geraint,
>>
>> Are you trying to run a simulation with output in the same directory as
>> one that was run before?
>>
>>
>> -erik
>>
>>
>> On Thu, Oct 1, 2015 at 9:30 AM, Geraint Pratten <g.pratten at sussex.ac.uk>
>> wrote:
>>
>>> Hi all,
>>>
>>> I recently built the latest release (Hilbert) on the local cluster and
>>> I've been getting a HDF5 error. I've tried building Cactus with both the
>>> local HDF5 package on the cluster as well as letting Cactus build HDF5 from
>>> scratch. Does anyone have any insight into the error I've attached below
>>> (extract of a long chain of errors of the same form)?
>>>
>>> I can dump the output to file or give configuration files if needed.
>>>
>>> Thanks in advance!
>>> Geraint
>>>
>>> ----------
>>>
>>> HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
>>>   #000: H5Ddeprec.c line 193 in H5Dcreate1(): unable to create dataset
>>>     major: Dataset
>>>     minor: Unable to initialize object
>>>   #001: H5Dint.c line 453 in H5D__create_named(): unable to create and
>>> link to dataset
>>>     major: Dataset
>>>     minor: Unable to initialize object
>>>   #002: H5L.c line 1638 in H5L_link_object(): unable to create new link
>>> to object
>>>     major: Links
>>>     minor: Unable to initialize object
>>>   #003: H5L.c line 1882 in H5L_create_real(): can't insert link
>>>     major: Symbol table
>>>     minor: Unable to insert object
>>>   #004: H5Gtraverse.c line 861 in H5G_traverse(): internal path
>>> traversal failed
>>>     major: Symbol table
>>>     minor: Object not found
>>>   #005: H5Gtraverse.c line 641 in H5G_traverse_real(): traversal
>>> operator failed
>>>     major: Symbol table
>>>     minor: Callback failed
>>>   #006: H5L.c line 1674 in H5L_link_cb(): name already exists
>>>     major: Symbol table
>>>     minor: Object already exists
>>> WARNING level 1 from host node207.cm.cluster process 0
>>>   while executing schedule bin (none), routine (no thorn)::(no routine)
>>>   in thorn CarpetIOHDF5, file
>>> /mnt/pact/gp234/NR/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc
>>> <http://output.cc/>:677:
>>>   -> HDF5 call 'dataset = H5Dcreate (outfile, datasetname.str().c_str(),
>>> filedatatype, dataspace, plist)' returned error code -1
>>> WARNING level 1 from host node207.cm.cluster process 0
>>>   while executing schedule bin (none), routine (no thorn)::(no routine)
>>>   in thorn CarpetIOHDF5, file
>>> /mnt/pact/gp234/NR/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc
>>> <http://output.cc/>:689:
>>>   -> HDF5 call 'H5Dwrite (dataset, memdatatype, H5S_ALL, H5S_ALL,
>>> H5P_DEFAULT, data)' returned error code -1
>>> WARNING level 1 from host node207.cm.cluster process 0
>>>   while executing schedule bin (none), routine (no thorn)::(no routine)
>>>   in thorn CarpetIOHDF5, file
>>> /mnt/pact/gp234/NR/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc
>>> <http://output.cc/>:725:
>>>   -> HDF5 call 'attr = H5Acreate (dataset, "level", H5T_NATIVE_INT,
>>> dataspace, H5P_DEFAULT)' returned error code -1
>>> WARNING level 1 from host node207.cm.cluster process 0
>>>   while executing schedule bin (none), routine (no thorn)::(no routine)
>>>   in thorn CarpetIOHDF5, file
>>> /mnt/pact/gp234/NR/Cactus/arrangements/Carpet/CarpetIOHDF5/src/Output.cc
>>> <http://output.cc/>:726:
>>>   -> HDF5 call 'H5Awrite (attr, H5T_NATIVE_INT, &refinementlevel)'
>>> returned error code -1
>>> WARNING level 1 from host node207.cm.cluster process 0
>>>
>>> --
>>> Geraint Pratten
>>> Postdoctoral Research Associate
>>>
>>> Mobile: +44(0) 7581709282
>>> E-mail: G.Pratten at sussex.ac.uk
>>> Skype: geraint.pratten
>>>
>>> School of Mathematical and Physical Sciences
>>> Pevensey 3 Building
>>> University of Sussex
>>> Falmer Campus
>>> Brighton
>>> BN1 9QH
>>> United Kingdom
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at einsteintoolkit.org
>>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>>
>>>
>>
>>
>> --
>> Erik Schnetter <schnetter at cct.lsu.edu>
>> http://www.perimeterinstitute.ca/personal/eschnetter/
>> _______________________________________________
>> Users mailing list
>> Users at einsteintoolkit.org
>> http://lists.einsteintoolkit.org/mailman/listinfo/users
>>
>>
>> --
>> Ian Hinder
>> http://members.aei.mpg.de/ianhin
>>
>>
>
>
> --
> Geraint Pratten
> Postdoctoral Research Associate
>
> Mobile: +44(0) 7581709282
> E-mail: G.Pratten at sussex.ac.uk
> Skype: geraint.pratten
>
> School of Mathematical and Physical Sciences
> Pevensey 3 Building
> University of Sussex
> Falmer Campus
> Brighton
> BN1 9QH
> United Kingdom
>
>
> --
> Ian Hinder
> http://members.aei.mpg.de/ianhin
>
>


-- 
Geraint Pratten
Postdoctoral Research Associate

Mobile: +44(0) 7581709282
E-mail: G.Pratten at sussex.ac.uk
Skype: geraint.pratten

School of Mathematical and Physical Sciences
Pevensey 3 Building
University of Sussex
Falmer Campus
Brighton
BN1 9QH
United Kingdom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20151005/436684f4/attachment-0001.html