Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

Fwd: [Hdf-forum] hdf-1.10 and collective metadata success story
by Erik Schnetter 24 May '16

24 May '16

As many of you know, restarting a simulation from checkpoints is often slower than it should be, and we have in the past implemented several performance improvements and work-arounds to address this. However, this is still an unsolved problem. The performance of accessing HDF5 datasets (i.e. opening the datasets and/or reading attributes) was often substantially slower than expected. The performance of reading the data themselves seems to be fine. A few weeks ago, HDF5 1.10 was released. This features (optionally) a new file format that stores metadata (i.e. group directories and attributes) in a more efficient way. I always suspected that this should help with performance, but didn't run benchmarks at scale yet. Today, I see this email from Rob Latham <http://www.mcs.anl.gov/~robl/> on the HDF5 mailing list, which I copy below. Essentially: HDF5 1.10 can provide a significant performance increase, and is worth a try. If you try this at home, remember that HDF5 1.10 can create files that are backward compatible with 1.8. Obviously, if you do this, you will not see the performance benefits of the new file format. Please read the release notes to learn more about this, and about backward compatibility. Note that HDF5 1.10 also ships with a tool "h5format_convert" that converts from the 1.10 to the 1.8 format in case you have tools that can't read the new format. This conversion is very quick. -erik ---------- Forwarded message ---------- From: Rob Latham <robl(a)mcs.anl.gov> Date: Tue, May 24, 2016 at 10:44 AM Subject: [Hdf-forum] hdf-1.10 and collective metadata success story To: hdf-forum(a)lists.hdfgroup.org Since I've spent oh, only the last 8 years beating you up a bit about HDF5 metadata reads at scale, it is only fair that I share with you some good news. Thanks for implementing a scalable metadata read. It's already paying off. There was a user trying to read a large dataset on our Blue Gene. We got him using 1.10, and presto: Here's a quote from the user: Yes, I can now restore data (with hdf5 1.10). I did a large test simulation on cetus (with fewer nodes than I'd usually run, so a lot of data per node), and then followed up by restoring a real 1024-node simulation on mira: this read 6 "field files" of 3.8 GB each and 2 "particle files" of 396 GB and 2 "particle files" of 69 GB in 442 seconds. Previously it was taking on the order of an hour to read in a 4 GB field file. ==rob _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum(a)lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5 -- Erik Schnetter <schnetter(a)gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 23 May '16

23 May '16

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with the URL: https://hangouts.google.com/call/5xt7voshalqmdcztr4zhfeox2ia Frank Löffler

2 1

Cactus only using one "tags" string per group in interface.ccl?
by Bernard Kelly 20 May '16

20 May '16

Hi. I only just found out that these two interface.ccl declarations are *not* equivalent: * tags='checkpoint="no" Prolongation="none"' tags='tensortypealias="Scalar" tensorweight=0 tensorparity=1' * tags='checkpoint="no" Prolongation="none" 'tensortypealias="Scalar" tensorweight=0 tensorparity=1' Both compile, but in the first case, the contents of the first "tags" string is ignored (I realized this only when I was getting crashes from CarpetLib's ggf.cc on attempted prolongation). So given that only one tags value is being used for each group, Is there a reason why the compilation stage SHOULDN'T flag/fail on a repeated tags definition like the first example? Since it didn't, I just assumed it was concatenating the two strings and using all the info. [Yes, I know it's silly to have two assignments where one will do, but as often happens, I inherited some code that did it.] [I'm still using ET_2015_05 for this, in case anything's changed.] Thanks, Bernard

2 1

Merging carpet hdf5 data without creating additional files
by Michael Clark 17 May '16

17 May '16

Hello, I'm interested in ways to manipulate carpet hdf5 data from multiple jobs (e.g. from different output-XXXX directories of a simfactory-managed simulation) for visualization purposes. Currently, if I had a variable I wanted to plot, what I would do is run hdf5_merge on the separate .h5 files to generate a single hdf5 file and then use a visualization tool (like visit) with it. I'm curious if there is a means to do this differently: for instance, if one could load the separate .h5 files in python as some collection of data objects and then combine those objects, no separate and combined .h5 file would need to be produced. Then, if some means existed to create a visualization from that data object (e.g. through yt), the need for the combined .h5 file would be totally obsolete. I suspect this must already be possible in some manner, perhaps using other programs. I'd be curious to hear how, for instance, SimulationTools handles this problem.

7 9

meeting minutes for 2016-05-16
by Roland Haas 16 May '16

16 May '16

Present: Barry, Eloisa, Steven, Zach, Roland, Yosef, Tickets: * 1853: Roland suggests to undo the change to reduce memory which resets the code to the last release * 1880: Roland suggests to update the test data with the new data, will follow up with Erik and Frank * 1889: requires discussion as to what the correct algorithm to compute a normalization constant for relative errors should be ET conference: * best train connection is from Rome, there is also a direct train connection from Munich, from Verona one has to take a bus then the train Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

meeting minutes for 2016-05-09
by Roland Haas 13 May '16

13 May '16

Present: Frank, Josh, Roland, Ian, Philipp, Erik, Peter Please register lunch preferences here http://goo.gl/forms/XAlDgYvxuK so that food can be ordered. Tickets: * failing Dissipation tests: these are caused by changes to AHFinderDirect related to scheduling. * poisson: not clear what causes this. Roland suspects that not all Jenkins slaves are exactly equal and that it only fails on this one * testsuite comparison is script received comments, Frank will look at them * HDF5 zlib etc detection: apply all patches and test, Ian to provide his changes as well Next ET release: * Frank will work toward it but we are waiting or some of the fixes to be tested first Workshop: * please submit your information about lunch and receiption information here: http://goo.gl/forms/XAlDgYvxuK Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

5 8

Einstein Toolkit Meeting Reminder
by Frank Loeffler 06 May '16

06 May '16

1 0

meeting minutes for 2016-05-02
by Roland Haas 02 May '16

02 May '16

Present: Steve, Roland, Eloisa, Matt, Ian, Josh, Erik, Peter, Tickets: * AHFinderDirect: apply proposed patch incl. comments * TimerReport and Cactus require c++11: with current fixes Cactus selects proper compiler options * HDF5 memory fragmentation: happens on stampede, datura due to use of realloc() with an initial allocation which is almost always too small. Ian will extract information out of (private) email thread to obtain an actionable item in the ticket. Suggest to send a bug report to HDF5. Put ticket on hold until we have sufficient information and a test case though. * McLachlan helper thorns: known how to do, ticket is a reminder * Parfile error concerning comments: Steve will take care of it Next ET release: * will have release (see previous minutes) but only maintenance due to no big new features * will discuss details next week when all are back Workshop: * remote access seems desirable but difficult to get set up properly. Will be discussed among organizers and requires some outside interest being voiced. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 29 Apr '16

29 Apr '16

1 0

meeting minutes for 2016-04-25
by Roland Haas 25 Apr '16

25 Apr '16

Present: Ian, Frank, Erik, Roland, Steve, Matt, Josh Tickets: * Steve will take care of https://trac.einsteintoolkit.org/ticket/1872 * Frank suggests to make https://trac.einsteintoolkit.org/ticket/1875 depend on a parameter * Frank added some comments to https://trac.einsteintoolkit.org/ticket/1862 Roland will follow up with David to make sure that all required changes were done Multi-model simulations: * Roland has a branch of PUGH that replicates Carpet's Multi-model framework * suggests to have ExternalLibraries/MPI provide WORLD and UNIVERSE communicators. Erik suggests to provide these via function calls rather than just exposing global variables. Roland will provide a pull request Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users