Users February 2013

users@lists.einsteintoolkit.org

20 participants
25 discussions

Einstein Toolkit Meeting Reminder
by Frank Loeffler 14 May '15

14 May '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. My agenda contains: - Fall workshop - PLEASE let us know if you come [1]! - Hotel booking - Release planning - release date: Mon Oct 24 2011 - freeze date: Mon Sep 26 - the day of the call! - branch name: ET_2011_10 - List of release-concerning bugs [2] - ET paper: proposed "freeze" by Oct 3rd (Mon after this call) As always: feel free to add to this list. Frank Loeffler [1] https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Fall_2011#Fall_Einstei… [2] https://trac.einsteintoolkit.org/query?status=!closed&milestone=ET_2011_11&…

10 31

strong scaling tests on Stampede (TACC)
by Bruno Giacomazzo 19 Sep '13

19 Sep '13

Hi, I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs. I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs). First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it? According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede? Let me know if you have any comment or suggestion. Cheers, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

6 10

bug in Dissipation thorn?
by Kentaro Takami 07 Mar '13

07 Mar '13

Dear all, I encountered the problem in which the simulation result is randomly different at each run( of course I used same executable, par file, HPC and so on.). Finally I'm doubting the origin of the problem is "CactusNumerical/Dissipation", because when I don't activate this thorn, the problem can be disappeared. In this thorn, "dissipation.c" calls the fortran subroutine "apply_dissipation.F77" directly without scheduling in CACTUS. I guess the variables between C and fortran become inconsistency in this stage. What do you think? I don't have much knowledge to call fortran subroutine from directly c language. So I need comments from expert. Kentaro TAKAMI

6 23

CarpetIOHDF5::output_index = "yes" by default?
by Frank Loeffler 28 Feb '13

28 Feb '13

Hi, Would somebody object to making CarpetIOHDF5::output_index = "yes" the default? (It is currently "no".) It should not hurt to create these small files. (Or, does it?) Frank

4 5

Build system bug
by Steven R. Brandt 27 Feb '13

27 Feb '13

I use the CACTUS_CONFIGS_DIR feature regularly, and with newer ET checkouts it seems to call problems. Something is issuing a find in the configs dir and not checking the variable. Anybody know where this is? Cheers, Steve COMPILING /home/sbrandt/workspace/funwave2/Cactus/src/main/Parameters.c Updating /home/sbrandt/.mojaveconfig/funwave2/Cfg1/lib/libthorn_Cactus.a Checking status of thorn CactusBindings find: `configs/Cfg1/ThornList': No such file or directory find: `configs/Cfg1/config-data': No such file or directory find: `configs/Cfg1/config-info': No such file or directory

2 1

McLachlan crash
by Vassilios Mewes 27 Feb '13

27 Feb '13

Hello all, as mentioned earlier in the ET phone call, i have attached the par file used in the run... i tried checking for the nans in the output, but the last hdf5 output was too many iterations earlier and they aren't visible in 1D ascii output either as said, the puncture tracker actually aborted the run, not the nan checker... the last lines of the simulation log look like this: INFO (AHFinderDirect): proc 0: searching for horizon 1/1 INFO (AHFinderDirect): AH 1/1: r=0.888981 at (-0.010571,0.042863,0.000000) INFO (AHFinderDirect): AH 1/1: area=37.24715073 m_irreducible=0.8608185171 71872 449.200 | 0.3696424 0.9877759 | 8.7614309 | 1.847988e+05 | 1.489620e-12 0.0001394 | 7.655611e-18 0.0000003 71876 449.225 | 0.3696349 1.0105778 | 8.7608144 | 1.848221e+05 | -0.0000005 0.0001395 | -1.183581e-09 0.0000003 INFO (Refluxing): Refluxing at iteration 71880 on level 3 of 5 INFO (Refluxing): Refluxing on level 3: INFO (Refluxing): Refluxing at iteration 71880 on level 4 of 5 71880 449.250 | 0.3696287 1.9741189 | 8.7609976 | 1.848285e+05 | -0.0000006 0.0001395 | -2.225731e-09 0.0000003 71884 449.275 | 0.3366873 4.239172e+39 | 8.7611688 | 1.848352e+05 | -0.0000003 0.0001395 | -8.130017e-10 0.0000003 INFO (Refluxing): Refluxing at iteration 71888 on level 2 of 5 INFO (Refluxing): Refluxing on level 2: INFO (Refluxing): Refluxing at iteration 71888 on level 3 of 5 INFO (Refluxing): Refluxing on level 3: INFO (Refluxing): Refluxing at iteration 71888 on level 4 of 5 71888 449.300 | 0.0000000 1.130446e+40 | 8.7613044 | 1.848426e+05 | 1.457866e-12 0.0001395 | 7.432056e-18 0.0000003 71892 449.325 | -0.2467648 8.478344e+39 | 8.7609892 | 1.848595e+05 | -0.0000030 0.0001395 | -4.615484e-09 0.0000003 INFO (Refluxing): Refluxing at iteration 71896 on level 3 of 5 INFO (Refluxing): Refluxing on level 3: INFO (Refluxing): Refluxing at iteration 71896 on level 4 of 5 71896 449.350 | 0.0000000 1.599816e+26 | 8.7611584 | 1.848662e+05 | -0.0000037 0.0001395 | -6.024551e-09 0.0000003 71900 449.375 | -1.413057e+39 1.199862e+26 | 8.7613031 | 1.848734e+05 | -0.0000022 0.0001395 | -4.227202e-09 0.0000003 INFO (Refluxing): Refluxing at iteration 71904 on level 1 of 5 INFO (Refluxing): Refluxing on level 1: INFO (Refluxing): Refluxing on level 1 map 0 component 0 direction 0 face 0: [31,31,3]:[31,32,30] INFO (Refluxing): Refluxing on level 1 map 0 component 0 direction 1 face 0: [31,31,3]:[33,31,30] INFO (Refluxing): Refluxing on level 1 map 0 component 0 direction 2 face 0: [31,31,3]:[33,32,3] INFO (Refluxing): Refluxing at iteration 71904 on level 2 of 5 INFO (Refluxing): Refluxing on level 2: INFO (Refluxing): Refluxing at iteration 71904 on level 3 of 5 INFO (Refluxing): Refluxing on level 3: INFO (Refluxing): Refluxing at iteration 71904 on level 4 of 5 71904 449.400 | 0.0000000 0.9877757 | 8.7613184 | 1.848834e+05 | 1.468319e-12 0.0001395 | 7.506929e-18 0.0000003 71908 449.425 | -1.999770e+25 0.9877756 | 8.7605818 | 1.849092e+05 | -0.0000001 0.0001395 | -7.816558e-11 0.0000003 INFO (Refluxing): Refluxing at iteration 71912 on level 3 of 5 INFO (Refluxing): Refluxing on level 3: INFO (Refluxing): Refluxing at iteration 71912 on level 4 of 5 71912 449.450 | -0.1092106 0.9877756 | 8.7604068 | 1.849231e+05 | -0.0000001 0.0001395 | -9.198894e-11 0.0000003 71916 449.475 | -0.1107971 0.9877756 | 8.7603118 | 1.849354e+05 | -0.0000000 0.0001396 | -4.147005e-11 0.0000003 WARNING level 0 in thorn PunctureTracker processor 0 host vives.uv.es (line 224 of /usus/va/vasme/cactus/ET_2012_11/Cactus/configs/ET_TORI_opt/build/PunctureTracker/puncture_tracker.c): -> Shift at puncture #0 is (-nan,-nan,-nan). This likely indicates an error in the simulation. the infos are lapse min/max physical time/hour total time rho min/max press min/max thanks a lot in advance, best wishes, Vassili

3 7

hwloc
by Steven R. Brandt 27 Feb '13

27 Feb '13

I'm obviously doing something wrong. What is it? I have hwloc and hwloc-devel installed. I see no messages about missing headers. COMPILING /home/sbrandt/workspace/funwave2/Cactus/arrangements/CactusBase/IOUtil/src/AdvertisedFiles.c /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc: In member function ‘void {anonymous}::node_topology_info_t::load(hwloc_topology* const&, const {anonymous}::mpi_host_mapping_t&)’: /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:460:59: error: ‘HWLOC_OBJ_CACHE_DATA’ was not declared in this scope /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:460:79: error: ‘hwloc_get_cache_type_depth’ was not declared in this scope /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:469:20: error: ‘const struct hwloc_obj_attr_u::hwloc_cache_attr_s’ has no member named ‘type’ /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:469:28: error: ‘HWLOC_OBJ_CACHE_UNIFIED’ was not declared in this scope /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:470:20: error: ‘const struct hwloc_obj_attr_u::hwloc_cache_attr_s’ has no member named ‘type’ /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:471:20: error: ‘const struct hwloc_obj_attr_u::hwloc_cache_attr_s’ has no member named ‘type’ /home/sbrandt/workspace/funwave2/Cactus/arrangements/ExternalLibraries/hwloc/src/system_topology.cc:471:28: error: ‘HWLOC_OBJ_CACHE_INSTRUCTION’ was not declared in this scope make[3]: *** [system_topology.cc.o] Error 1 Cheers, Steve

2 2

meeting minutes
by Roland Haas 25 Feb '13

25 Feb '13

Present were: Frank, Philipp, Yosef, Peter, Erik, Steve, Josh workshops: * create doodle poll for Monday * Roland put hotel rates for Saga Motor Home and Vagabond Inn into wiki (https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Summer_2013 broken links/OpenDx: * remove broken links from website, replace with generic link to OpenDX reader plugin in svn * add statement that OpenDX is no longer being developped and that the reader is currently orphaned * suggest instead VisIt, DV (ask Bruno to verify instructions), Mathematica (ask Ian if he has pointers to his package) for visualization * Peter will give the reader a try Piranha parameter parser: * currently parses only parameter files, plan to extent to all ccl files * is easier to expand and maintain than current hand-written parsers * currently tested with ET test suite, needs to be tested on whole set of ET (and Cactus since it affects the flesh) machines * should use namespace and adhere to Cactus naming and commenting conventions: either use C++ namespace cctk (and cctki?) and/or prefix all externally visible functions by either cctk or cctki depending on whether they should be user callable or not * remove and replace existing math evaluator/parser by piranha one to have uniform math support * commmit to main flesh branch for easier testing tickets to look at: * radiation boundary condition https://trac.einsteintoolkit.org/ticket/1261 * unless objections are raised Frank will change the default for IOUtil::out_dir from "." to "$parfile" see https://trac.einsteintoolkit.org/ticket/1255 . No objections were raised in the phone call. Yours, Roland

1 0

Piraha in the Flesh
by Steven R. Brandt 25 Feb '13

25 Feb '13

I've hacked a version of Cactus to use Piraha for parameter parsing. Please take a look: https://svn.cactuscode.org/flesh/branches/with_piraha This version is able to run the ET testsuite. This version can understand the variables $parfile, $ENV{name}, and $pi (which is 3.14...). Variables can be standalone, or be substituted from inside strings. It can understand mathematical expressions including grouping, order of operation, and a few functions (sin, cos, tan, exp, sqrt). Other parameters from the parameter file are available for use in mathematical expressions. Some debug printing is in place so you can get an idea of what's going on. Cheers, Steve

5 12

Einstein Toolkit Meeting Reminder
by Roland Haas 24 Feb '13

24 Feb '13

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users February 2013