Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

ET and Llama
by Erik Schnetter 24 Sep '13

24 Sep '13

We discussed the upcoming Einstein Toolkit release and the Llama infrastructure in yesterday's telecon. Unfortunately, my wireless connection was not well enough to support much of a discussion with me. When we (we = main Llama developers) released Llama earlier this year, we did so in the knowledge that the infrastructure supports high-quality production simulations, but lacks documentation and tutorials for newcomers. Our plan was (and is) to attract users who can help us add these tutorials, examples, documentation, and also help clean up the code. As you may imagine, Llama grew organically, and there are also some left-over pieces of unsuccessful sub-projects. Some pieces have also by now migrated into Cactus, and should now either be removed or rewritten and simplified. We intentionally did not submit Llama for inclusion into the Einstein Toolkit yet: Llama should not come "for free" with the Einstein Toolkit, but people should realize and acknowledge that Llama represents an independent project and significant amount of work. We are happy it if people find Llama useful and appreciate any contributions towards making it even more useful. Personally, I expect Llama to become part of the ET, but the code is not ready yet. In summary: - We (we = main Llama deveopers or users) should give a Llama tutorial soon, helping others to get started. - We are looking for a tutorial write-up and examples, in particular discussing some of the unexpected features (memory requirements, typical grid structures for binary systems, time step limitations, etc.) - A code review with the intent to better integrate Llama with the core of Cactus, to remove outdated parts, and consolidate the existing code would be in order. -erik -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu/.

1 0

CarpetRegrid2 with nonexistent radii in some centres
by Ian Hinder 24 Sep '13

24 Sep '13

Hi, Suppose I want to set up a grid structure for a BBH evolution using CarpetRegrid2, but ensure that the outer levels never change. What I usually do is set up two box hierarchies centred on the two BHs, but this doesn't guarantee that the outer levels don't change. Instead, I would like to have three hierarchies, one for the outer levels, and two for the BHs, where the latter only contain boxes on the finest levels. I would implement this by not setting the coarser radii for the BH hierarchies, and leaving them at -1. I was under the impression that this used to be possible, but there is a check in CarpetRegrid2 which prevents this: -> The radius of refinement level 1 of region 2 is [-1,-1,-1], which is non-negative I can't see anything in the code which would make this necessary. Is it safe to disable this check? -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

2 2

Meeting Minutes
by Roland Haas 23 Sep '13

23 Sep '13

Present: Frank, Roland, Zach, Erik, Peter, Ian Zach's UIUC code based MHD code: * code is almost ready, needs some cleaning up * Frank to get into contact with Zach to set up a subversion repository within the ET Gallery (http://einsteintoolkit.org/about/gallery/): * want more examples * NSNS (with and without Llama): Roland * collapse (to BH or to NS): Roland, Philipp or Christian Reisswig * BBH and BH: Peter Important Tickets for next release: * #262 [1] concerns not being able to recover from a checkpoint if the number of timelevels does not match what prolongation_order_time requires also of interest might be #620 [2] which is an enhancement request to avoid having to specify the maximum number of timelevels. We will discuss this on the mailing list in more detail. * #1395 that contains a workaround for svn 1.7 and parallel checkouts [3]. Frank will take care of this. * with more git thorns in the ET ticket #393 [4] should also be adressed * documentation for GRHydro and PITTNull code should be updated GRHydro should include the relevant sections from the recent papers. PITTNull should contain some information in the "Physical system" section of the documentation * there was a suggestion to try and include Llama in next release however this seems to the too short notice since the code still needs cleanup etc and not all of the Llama authors have been contacted * if you have access to the newest Intel compiler, please test #1276 [5] to see if it still has problems with restrict in C code. * for the next workshop, consider "distributed" approach with several institutes broadcasting the lectures and organizing groups for the exercises, consider possibility of participants calling in (though this should be the exception and participants should try and attend as a group) * need to test whichever system might be used (video, slides, audio, questions from autience, chat). Suggestions were either bigbluebutton (most likely version 0.81 will be out by then) or the H323 (polycom) systems at AEI, Caltech, LSU, PI. Yours, Roland [1] https://trac.einsteintoolkit.org/ticket/626 [2] https://trac.einsteintoolkit.org/ticket/620 [3] https://trac.einsteintoolkit.org/ticket/1395 [4] https://trac.einsteintoolkit.org/ticket/392 [5] https://trac.einsteintoolkit.org/ticket/1276

1 0

Einstein Toolkit Meeting Reminder
by Frank Loeffler 21 Sep '13

21 Sep '13

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. As always, we will answer questions and hear comments that come up at the meeting. With the next release being about 2 months away it is time to discuss what we plan to include/develop/fix/extend in/for/until then/it - which is what I would like to talk about on Monday. Frank Löffler

1 0

Recovery fails with out-of-memory
by Ian Hinder 21 Sep '13

21 Sep '13

Hi, Does anyone have a suggestion for how I can make a run recover if HDF5 claims to be out of memory? The original run was on 1056 cores of stampede. Recovery fails both on the same number of cores as well as on 2048. Would a newer version of HDF5 help? Is there some parameter I can set? I am already using CarpetIOHDF5::open_one_input_file_at_a_time = yes. I am using CarpetIOHDF5::compression_level = 9. I wonder if the problem is made worse by having to decompress when reading the file. Logically, I don't see any reason why HDF5 should need more memory to recover than the original run used. > HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0: > #000: H5Dio.c line 174 in H5Dread(): can't read data > major: Dataset > minor: Read failed > #001: H5Dio.c line 449 in H5D__read(): can't read data > major: Dataset > minor: Read failed > #002: H5Dchunk.c line 1735 in H5D__chunk_read(): unable to read raw data chunk > major: Low-level I/O > minor: Read failed > #003: H5Dchunk.c line 2766 in H5D__chunk_lock(): data pipeline read failed > major: Data filters > minor: Filter operation failed > #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure during read > major: Data filters > minor: Read failed > #005: H5Zdeflate.c line 136 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression > major: Resource unavailable > minor: No space available for allocation > WARNING[L1,P254] (CarpetIOHDF5): : > HDF5 call 'H5Dread (dataset, datatype, memspace, filespace, xfer, cctkGH->data[patch->vindex][timelevel])' returned error code -1 -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

2 1

strong scaling tests on Stampede (TACC)
by Bruno Giacomazzo 19 Sep '13

19 Sep '13

Hi, I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs. I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs). First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it? According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede? Let me know if you have any comment or suggestion. Cheers, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

6 10

Carpet with Intel MPI on Stampede
by Ian Hinder 19 Sep '13

19 Sep '13

Hi, I've noticed that I get a number of jobs failing on stampede with the following error: [59:c453-703][../../dapl_poll_rc.c:1360] Intel MPI fatal error: ofa-v2-mlx4_0-1 DTO operation posted for [1:c411-103] completed with error. status=0x8. cookie=0x0 Assertion failed in file ../../dapl_poll_rc.c at line 1362: 0 internal ABORT - process 59 It appears to be intermittent; if I restart from the last checkpoint, the simulation gets a bit further. Switching to MVAPICH appears to improve things (time will tell) but it seems to be significantly slower than Intel MPI. I remember also having problems on SuperMUC with Intel MPI which were solved by switching to IBM MPI there. Has anyone else seen these problems, and if so, do you have a workaround? I'm using the standard simfactory optionlist with 512 cores and num-threads = 8. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

2 2

Gerrit
by Steven R. Brandt 17 Sep '13

17 Sep '13

I heard about Gerrit at the recent PTP meeteing. https://code.google.com/p/gerrit/ Gerrit manages the process of code reviewing in git, and has a lot of customization. You can set things up so that no one can push to the main branch unless their change has been reviewed. You can make rules like, one senior developer and two junior developers must review, etc. By helping to enforce code reviews, Gerrit helps to increase code quality, and helps to train junior developers in understanding the codebase. I think it might be a good alternative to Trac. Cheers, Steve

2 2

Meeting Minutes
by Roland Haas 16 Sep '13

16 Sep '13

Present: Frank, Roland, Matt, Erik, Steve, Peter, Zach open tickets * TwoPunctures d3tensor patch: Roland to test Erik's patch in valgrind, to report back ** hard-coding lower bound to zero would likely not significantly simplify the code * some version of subversion fail in parallel checkout ** Frank has written workaround https://trac.einsteintoolkit.org/ticket/1395 needs review * no objections against moving common GF tags into interface.ccl, Roland to propose a list Kranc changes: Ian had replied to Roland via email and said: --8<-- Here is a summary of what I have done: • Developed some unit and regression tests for Kranc. In addition to standard unit tests which test individual functions (of which there are few), I also generate small thorns and compare the generated output code with the previous version. I include cut-down versions of McLachlan in their many forms (plain, vectorised, OpenCL, CaKernel, DGFE), as well as several examples of specific features. • Major long-overdue cleanup of the internals of Kranc. The external interface has not changed. The only effects should be in the formatting and ordering of the generated code. Some items in CCL files will be reordered and some variable declarations in the code will be made slightly more consistent. • Lots of very old code has been removed or rewritten. A lot of "cruft" accumulates over 10 years! • Aim is to make Kranc significantly more modular; very long functions have been broken up, and each "feature" has been moved to its own source file (e.g. vectorisation, opencl, DGFE etc). • Have taken a conservative and incremental approach to the cleanup, ensuring that the tests pass after each small change. • Still much left to do • Please report any problems you encounter --8<-- ET seminar talk by Zach Etienne: * wait for Zach to finish the code, then schedule a seminar at a convenient date some weeks afterwards to leave enough time for announcements Yours, Roland

1 0

Carpet: using baseextents?
by Scott Hawley 16 Sep '13

16 Sep '13

Goal: I'm trying to determine if a given (fine) grid vertex coincides with a coarser grid point. Frank pointed me to baseextents, and I've come up with the following snippet of code... Just posting this for a sanity check: Am I on the right track? Is there a more robust way one should do this? My code relies on "baseextents" being the "full" extent of a given refinement level, apart from any domain decomposition that occurs when running with multiple MPI processors. (Well, and it relies on the fact that fine grid boundaries must lie along coarse grid points.) BEGIN_MAP_LOOP (cctkGH, CCTK_GF) { BEGIN_LOCAL_COMPONENT_LOOP (cctkGH, CCTK_GF) { DECLARE_CCTK_ARGUMENTS; assert (maps == 1); int const m = 0; // <--- Really? m is never > 0? assert (mglevels == 1); int const rl = reflevel; ivect const ilo = vhh.at(m)->baseextents.at(m).at(rl).lower(); ivect const istr = vhh.at(m)->baseextents.at(m).at(rl).stride(); int ref_ratio = 2; // <-- Do we support refinement ratios > 2? ivect coarse_str = istr * ref_ratio; for (CCTK_INT k = 0 ; k < cctk_lsh[2] ; k=k+1) { for (CCTK_INT j = 0 ; j < cctk_lsh[1] ; j=j+1) { for (CCTK_INT i = 0 ; i < cctk_lsh[0] ; i=i+1) { // does this vertex coincide with a coarser grid vertex? if ( ( 0 == (i - ilo[0]) % coarse_str[0]) && ( 0 == (j - ilo[1]) % coarse_str[1]) && ( 0 == (k - ilo[2]) % coarse_str[2])) { CCTK_VInfo(CCTK_THORNSTRING," We are over a CG vertex!"); } } // end loop over i } // end loop over j } // end loop over k } END_LOCAL_COMPONENT_LOOP; } END_MAP_LOOP; Thank you very m uch, Scott

2 4

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users