#1186: Provide a version of CCTK_WARN which never returns
-------------------------+--------------------------------------------------
Reporter: hinder | Owner:
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: Cactus | Version:
Keywords: |
-------------------------+--------------------------------------------------
CCTK_WARN(0,...) always causes a fatal abort, but the compiler does not
know this. Hence, it incorrectly warns that variables might be used
without being set because it doesn't know that the only path that doesn't
set them terminates the program. I propose introducing a new function
CCTK_WARN_ABORT(...) which calls CCTK_WARN(0,...) and is declared with the
attribute noreturn (http://gcc.gnu.org/onlinedocs/gcc-3.2/gcc/Function-
Attributes.html) so that the compiler knows that it can never return, and
does not make incorrect conclusions when computing variable dependencies
(CCTK_Abort is already defined without an error message).
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1186>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#1255: change default of IOUtil::out_dir from "." to "$parfile"
-------------------------+--------------------------------------------------
Reporter: knarf | Owner:
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: Cactus | Version: development version
Keywords: |
-------------------------+--------------------------------------------------
Currently the default output directory for Cactus output is ".", something
that is hardly every used by anybody. We should strive to have parameter
defaults that work in most circumstances and that are used by most people.
"." is save, but not at all the most wanted choice. "$parfile" is another
sensible choice, should be as save as "." and is used by more (>0) users
than their default. Thus, I propose to test to change the default. Before
getting into this (unknown) amount of work, let me use this ticket to ask
for objections first.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1255>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by hinder):
I think that would already be a good solution. Yes, when I have found an
odd error, I find that h5ls fails, or gives no datasets. This may not be
100% reliable, but it's better than nothing. Yes, you will never get all
the files consistent unless you lock the simulation or have a filesystem
snapshot. I agree that temporal consistency is much less important than
avoiding corrupt hdf5 files; I already have code to handle temporal
inconsistency (two BH_diagnostics output files having different numbers of
iterations, etc).
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:7>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by knarf):
I don't have a corrupted hdf5 file handy, so I have to ask: Is there a
quick way to check if such a file is corrupt, maybe an 'h5ls'? If so, this
would make a selected resync quite easy.
Especially for high-frequency output you might come into trouble with
timestamps if your rsync takes longer than one of the output frequency
steps (and the simulation doesn't stop, which I really don't want to
have). You'll never get all files consistently because the moment you copy
the last the first has already been updated again.
All I care about would be to have all files non-corrupt. Retrying (only
the corrupt files) until this is reached sounds like a possible solution
to me, and would be an incentive to use simfactory over the otherwise also
simple, manual rsync.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:6>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by hinder):
I have been working like this for the last few years, and the occasional
corrupted HDF5 has not been a problem. However, recently I have started
to run simulations with many more files and more frequent output, and I
sometimes get into the situation where every time I sync the simulation,
at least one of the files is corrupt, as they are written very frequently,
and if they are copied before the write finishes, the copy can be corrupt.
A corrupt HDF5 file cannot be used; you cannot read the previous datasets,
you cannot even list the datasets. You just have to sync again and hope
for the best. I also agree that stopping the simulation is to be avoided
if possible. In the far future, we would probably be able to create a
filesystem-level snapshot of the simulation at the end of each iteration.
That would ensure a consistent state which could be easily copied. But
filesystems and OSes are not up to that yet.
Re: writing a time stamp to a separate file: The timestamp on the file
won't be updated until the file is closed.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:5>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by knarf):
I might be interested in such a 'get' command, but personally I am not so
much concerned about consistency. As long as a simulation is running I
accept that files might be inconsistent between each other, and in the
case of hdf5 might also be broken at times. Pausing a simulation and
wasting SUs that way just to get a snapshot of the data 100% reliably
doesn't sound like a good idea to me. Automatically checking that an hdf5
file isn't broken would be nice though. However, making sure the file is
'current' seems unnecessary. I usually don't care to have the absolutely
last timestep if that was just written and would like to avoid another
rsync for that; especially if yet another timestep might be written during
that time.
Of course, as soon as a simulation finished all of this isn't an issue
anymore anyway.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:4>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by eschnett):
Stopping a simulation seems dangerous. I'd prefer something safer. Here is
one idea:
- Cactus writes a timestamp or some other unique identifier for each HDF5
file into a separate file.
- Simfactory checks this file before and after an rsync. If the time
stamps differ, the file has to be copied again.
Another, similar method would be to have Cactus generate checksums for the
HDF5 files. This is expensive, as the whole file would have to be
checksummed, unless there is an HDF5 facility for this. After copying the
file we compare checksums.
Both methods need to be combined with a method to avoid accessing a file
while it is being updated; renaming it to *.tmp is the standard Unix way
to do so.
Yet another way would be to ask Cactus to make copies of all output files.
Such copies are never changed after being created (only deleted if they
are outdated), and such can be safely copied.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by rhaas):
I believe option 2 will break if Cactus starts to write to a file while
simfactory is synching it. The OS will allow the rename and rsync will
happily continue reading from the now changing file, possibly mixing old
and new content. If Cactus finishes adding to the file before rsync
finishes and moves it from tmp to its original name, then simfactory would
not notice this (but rsync most likely would when it does its final check
on the file content).
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:2>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#30: Implement a "get" command
--------------------------+-------------------------------------------------
Reporter: hinder | Owner: mthomas
Type: enhancement | Status: new
Priority: minor | Milestone:
Component: SimFactory | Version:
Resolution: | Keywords:
--------------------------+-------------------------------------------------
Comment (by hinder):
We will need some way to ensure that the retrieved data is in a consistent
state. Truncated ASCII files can be dealt with, though this is not ideal,
but partially-written HDF5 files cannot. This can be a serious problem if
several HDF5 files are being synced, as after each sync, there can be a
high probability that at least one of them is incompletely written. Some
options:
1. SimFactory (on the remote machine) writes a control file (either into
the simulation or somewhere else) which indicates to the simulation that
it should not open any new files for writing, and once it has closed any
currently open file as part of normal operation, it should record this
information in the control file, continue running, and only write new
files once the control file tells it to. This has the disadvantage of
locking the entire simulation for the duration of the transfer of the
active restart. For slow data transfers, this could be a significant
amount of time. This approach has the disadvantage that the different
files will not be in a consistent state; e.g. one output file may have the
current iteration but another may not.
2. Before writing a file, Cactus would move it to a new location
(file.tmp), and only move it back when it was fully written. SimFactory
would not sync tmp files. Any files which had been renamed to *.tmp in
the first pass would then be synced in a separate pass using their
original names, if they exist. Repeat until all files are synced. This
solution also does not maintain a consistent state across multiple files.
This does not require write access to the simulation directory, so could
also be used by collaborators who do not own the simulation.
3. Similar to (1), but only performed at the end of an iteration.
SimFactory would indicate to Cactus to pause the simulation at the end of
the current iteration, when all files are presumably valid on disk.
Cactus would indicate that the simulation had paused in a control file,
and SimFactory would then transfer the data, and unpause the simulation
when it was finished. This would guarantee that the synced data was in a
consistent state. We might want to have some mechanism to ensure that
simulations do not remain paused forever, perhaps by requiring simfactory
to update the control file periodically if it is still syncing.
All of the above apply only to the active restart. I think (3) is the
simplest and most robust. It is also the most expensive in SUs. The
control file location could be customisable, and placed somewhere that all
collaborators have write access.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/30#comment:1>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit
#1257: known_architectures/linux doesn't deal with non-standard gcc version output
--------------------+-------------------------------------------------------
Reporter: knarf | Owner:
Type: defect | Status: new
Priority: minor | Milestone: ET_2013_05
Component: Cactus | Version: development version
Keywords: |
--------------------+-------------------------------------------------------
Currently, known_architectures/linux only looks at the first line of '$F77
--version' to determine the version of a gnu-type fortran compiler.
However, on bluewaters this returns
{{{
/opt/cray/xt-asyncpe/5.16/bin/ftn: INFO: Compiling with
CRAYPE_COMPILE_TARGET=native.
GNU Fortran (GCC) 4.7.2 20120920 (Cray Inc.)
Copyright (C) 2012 Free Software Foundation, Inc.
}}}
The information the script is actually interested in is on the second
line, not the first. Thus, I propose the attached patch (grepping for
"GNU" before choosing the first line). I didn't find a problem with this
patch on other machines, but there might be in case $F77 --version does
not contain GNU in the line with the version.
--
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1257>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit