Users September 2013

users@lists.einsteintoolkit.org

14 participants
24 discussions

strong scaling tests on Stampede (TACC)
by Bruno Giacomazzo 19 Sep '13

19 Sep '13

Hi, I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs. I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs). First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it? According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede? Let me know if you have any comment or suggestion. Cheers, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

6 10

Carpet with Intel MPI on Stampede
by Ian Hinder 19 Sep '13

19 Sep '13

Hi, I've noticed that I get a number of jobs failing on stampede with the following error: [59:c453-703][../../dapl_poll_rc.c:1360] Intel MPI fatal error: ofa-v2-mlx4_0-1 DTO operation posted for [1:c411-103] completed with error. status=0x8. cookie=0x0 Assertion failed in file ../../dapl_poll_rc.c at line 1362: 0 internal ABORT - process 59 It appears to be intermittent; if I restart from the last checkpoint, the simulation gets a bit further. Switching to MVAPICH appears to improve things (time will tell) but it seems to be significantly slower than Intel MPI. I remember also having problems on SuperMUC with Intel MPI which were solved by switching to IBM MPI there. Has anyone else seen these problems, and if so, do you have a workaround? I'm using the standard simfactory optionlist with 512 cores and num-threads = 8. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

2 2

Gerrit
by Steven R. Brandt 17 Sep '13

17 Sep '13

I heard about Gerrit at the recent PTP meeteing. https://code.google.com/p/gerrit/ Gerrit manages the process of code reviewing in git, and has a lot of customization. You can set things up so that no one can push to the main branch unless their change has been reviewed. You can make rules like, one senior developer and two junior developers must review, etc. By helping to enforce code reviews, Gerrit helps to increase code quality, and helps to train junior developers in understanding the codebase. I think it might be a good alternative to Trac. Cheers, Steve

2 2

Meeting Minutes
by Roland Haas 16 Sep '13

16 Sep '13

Present: Frank, Roland, Matt, Erik, Steve, Peter, Zach open tickets * TwoPunctures d3tensor patch: Roland to test Erik's patch in valgrind, to report back ** hard-coding lower bound to zero would likely not significantly simplify the code * some version of subversion fail in parallel checkout ** Frank has written workaround https://trac.einsteintoolkit.org/ticket/1395 needs review * no objections against moving common GF tags into interface.ccl, Roland to propose a list Kranc changes: Ian had replied to Roland via email and said: --8<-- Here is a summary of what I have done: • Developed some unit and regression tests for Kranc. In addition to standard unit tests which test individual functions (of which there are few), I also generate small thorns and compare the generated output code with the previous version. I include cut-down versions of McLachlan in their many forms (plain, vectorised, OpenCL, CaKernel, DGFE), as well as several examples of specific features. • Major long-overdue cleanup of the internals of Kranc. The external interface has not changed. The only effects should be in the formatting and ordering of the generated code. Some items in CCL files will be reordered and some variable declarations in the code will be made slightly more consistent. • Lots of very old code has been removed or rewritten. A lot of "cruft" accumulates over 10 years! • Aim is to make Kranc significantly more modular; very long functions have been broken up, and each "feature" has been moved to its own source file (e.g. vectorisation, opencl, DGFE etc). • Have taken a conservative and incremental approach to the cleanup, ensuring that the tests pass after each small change. • Still much left to do • Please report any problems you encounter --8<-- ET seminar talk by Zach Etienne: * wait for Zach to finish the code, then schedule a seminar at a convenient date some weeks afterwards to leave enough time for announcements Yours, Roland

1 0

Carpet: using baseextents?
by Scott Hawley 16 Sep '13

16 Sep '13

Goal: I'm trying to determine if a given (fine) grid vertex coincides with a coarser grid point. Frank pointed me to baseextents, and I've come up with the following snippet of code... Just posting this for a sanity check: Am I on the right track? Is there a more robust way one should do this? My code relies on "baseextents" being the "full" extent of a given refinement level, apart from any domain decomposition that occurs when running with multiple MPI processors. (Well, and it relies on the fact that fine grid boundaries must lie along coarse grid points.) BEGIN_MAP_LOOP (cctkGH, CCTK_GF) { BEGIN_LOCAL_COMPONENT_LOOP (cctkGH, CCTK_GF) { DECLARE_CCTK_ARGUMENTS; assert (maps == 1); int const m = 0; // <--- Really? m is never > 0? assert (mglevels == 1); int const rl = reflevel; ivect const ilo = vhh.at(m)->baseextents.at(m).at(rl).lower(); ivect const istr = vhh.at(m)->baseextents.at(m).at(rl).stride(); int ref_ratio = 2; // <-- Do we support refinement ratios > 2? ivect coarse_str = istr * ref_ratio; for (CCTK_INT k = 0 ; k < cctk_lsh[2] ; k=k+1) { for (CCTK_INT j = 0 ; j < cctk_lsh[1] ; j=j+1) { for (CCTK_INT i = 0 ; i < cctk_lsh[0] ; i=i+1) { // does this vertex coincide with a coarser grid vertex? if ( ( 0 == (i - ilo[0]) % coarse_str[0]) && ( 0 == (j - ilo[1]) % coarse_str[1]) && ( 0 == (k - ilo[2]) % coarse_str[2])) { CCTK_VInfo(CCTK_THORNSTRING," We are over a CG vertex!"); } } // end loop over i } // end loop over j } // end loop over k } END_LOCAL_COMPONENT_LOOP; } END_MAP_LOOP; Thank you very m uch, Scott

2 4

Einstein Toolkit Meeting Reminder
by Roland Haas 15 Sep '13

15 Sep '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. Topics: * outstanding patches * moving grid function tags into actual interface.ccl options: ** checkpoint ** tensortypealias ** others * GetComponents -p and svn versions >= 1.7 * Kranc activity As always, we will answer questions and hear comments that come up at the meeting. Roland Haas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlI2RCQACgkQTiFSTN7SboXkmwCfS6nCozvAhraGQOqrX9GHHNY9 QakAnjszSRe/bgVf7BdU7LkwxsKV8W6T =Q42C -----END PGP SIGNATURE-----

1 0

splitting grid ARRAY along given direction only
by Roland Haas 13 Sep '13

13 Sep '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, is there a way to split a grid array (not a grid function) only along a given direction when doing the decomposition into processor components? In my usage case I have a grid array: CCTK_REAL phi TYPE=ARRAY SIZE=nphi,ntheta,nrad DISTRIB=DEFAULT ... and would like to split only in the last (radial) direction so that each process gets full spherical shells (I would have more shells than processes and could also live with empty processes). I looked at the code in SetupGH.cc and it seems as if currently the mapping DISTRIB=DEFAULT => SplitRegions_Automatic automatic is hard-coded and I could not find a way to make SplitRegions_Automatic split only in the z direction (but found a new method to cause "grid structure inconsistent" assert()s). Before I start hacking Carpet and/or Cactus, is there known way on how to achieve this? Yours, Roland - -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iEYEARECAAYFAlIyTzkACgkQTiFSTN7SboVQrwCg1I6UAe/KVZ223+Ud1FwuvnIa xuMAoKmPLHUs6U4ycem6zCGn8rygD0+6 =atMr -----END PGP SIGNATURE-----

3 3

Meeting Minutes
by Roland Haas 12 Sep '13

12 Sep '13

Present: Frank, Roland, Peter, Erik, Matt, Ian, Josh, Maria, Yosef, Philipp git to svn transition: * Ian (and other contributors) created a wiki page outlying ideas for how to organize repositories: https://docs.einsteintoolkit.org/et-docs/Version_control * majority of call participants prefer git over svn, points mentioned are easier management of branches and patches and availability of powerful GUIs * suggest using GUIs even for "advanced" users. For git there is SourceTree for OSX and SmartGit or GitEye (see http://git-scm.com/downloads/guis) Goals to achieve while moving the repositories, some of which might be conflicting, in no particular ordering: * simplify exchanging patches, in particular patches for review where right now we exchange patch files, instead we might want to point to a revision in a branch or private repository * consistency of thorns in toolkit is easier to achieve is strongly dependent thorns are in a single repository * simplify work-cycle for developers, read-only access for end-users is already provided by GetComponents * minimize amount of code downloaded * keep modularity of framework, avoid monolithic repository Opinions: * consistency is given low priority, instead require framework to enforce this (eg via interface.ccl etc). * avoid overly large repository with many unrelated thorns * using branches and cherry picking commits to choose a working set of thorns that mixes cutting edge versions of some thorns and bugfix-only versions of another is hard to do, takes lots of time. * we lack the manpower to keep a always working mainline for all thorns, GRHydro currently follows this paradigm and requires ~0.5 days a week to choose and push patches. The advantage is that Zelmani can do whatever it wants and does not have to worry about "external" users pulling code that contains bugs. This is not in line with the ET philisophy of having trunk branches under development then releases every 6 months. Hosting: * Frank (who will be admin) advertises to host the git repos at CCT, Ian suggests using gitolite as the front-end which uses a plain git repository for admin task (keeps record of admin actions) * desire to be able to automate actions on multiple repositories (assuming we will have multiple/many repositories) * suggestion to consider bitbucket/github as external hoster for speed and/or certificate issues, did not find much support * want to make sure that certificates work and that speed is acceptable * rather spend some money than spend time * bitbucket might be a viable option when using teams and would simplify infrequent tasks, major problem is scriptability * github source code made be available to use even for locally hosted repos if we want a web-based UI, Ian to test this Proposed repository structure: * structure not fully decided on, use wiki page (see above) as location to hash out proper layout Yours, Roland

6 10

reported vs real memory usage + CarpetRegrid2?
by Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] 11 Sep '13

11 Sep '13

Hi. I'm running a vacuum BHB evolution with a larger-than-usual set of inner refinement regions (levels 8, 9, 10 11 have radii of 12M, 8M, 6M, and 4M, respectively) and consequently the memory usage is a bit higher than normal. But I'm finding that it jumps up almost 100% after the first regridding, and stays there. My diagnostic for this is the result of top on each of the nodes (via "qtop.pl", a script on the machine I'm using). Sampled before the first regridding, it shows each core using ~ 1.5% of the node's total memory, while after regridding, it's more like 2.9% (these are Sandy Bridge nodes, with 16 available cores). However, the periodic output message from Carpet reporting the Grid structure etc. shows regions only marginally larger than before, and --- crucially for me --- has a marginally larger "Total required memory" (164 GB -> 167 GB, for instance). So (a) what's using the extra memory, and (b) why isn't Carpet reporting it? How seriously should I be taking that "Total required memory" message? I see this with executables generated from both the last (ET_2012_11) and current (ET_2013_05) stable releases, BTW. I'm attaching the current parameter file and SCROUT from a run. Thanks, Bernard

2 2

Meeting Minutes
by Roland Haas 09 Sep '13

09 Sep '13

Present: Peter, Erik, Frank, Roland, Philipp, Zacharia Etienne, Ian ET projects: * loosing momentum now that MHD paper is (almost) published * next project: staggered vector potential * currently working (with code in Zelmani): cell centered vector potential using algebraic gauge * started to work on reconstruction required for edge based A field * Philipp will make parfiles for cell centered A field available * Doodle poll for edge-A MHD call http://doodle.com/vahwwt5mbd6thmez Zach's MHD code: * Zach has send email too mailing list detailing plans and current status * Ian cleanup up Kranc's code over the last couple days, no changes in results are expected, please contact Ian (possibly via the mailing list so as to avoid duplication) if you experience problems Yours, Roland

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users September 2013