Users July 2013

users@lists.einsteintoolkit.org

16 participants
22 discussions

Einstein Toolkit Meeting Reminder
by Frank Loeffler 14 May '15

14 May '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. My agenda contains: - Fall workshop - PLEASE let us know if you come [1]! - Hotel booking - Release planning - release date: Mon Oct 24 2011 - freeze date: Mon Sep 26 - the day of the call! - branch name: ET_2011_10 - List of release-concerning bugs [2] - ET paper: proposed "freeze" by Oct 3rd (Mon after this call) As always: feel free to add to this list. Frank Loeffler [1] https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Fall_2011#Fall_Einstei… [2] https://trac.einsteintoolkit.org/query?status=!closed&milestone=ET_2011_11&…

10 31

strong scaling tests on Stampede (TACC)
by Bruno Giacomazzo 19 Sep '13

19 Sep '13

Hi, I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs. I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs). First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it? According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede? Let me know if you have any comment or suggestion. Cheers, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

6 10

database maintenance - 6pm CST - 9pm CST]
by Frank Loeffler 31 Jul '13

31 Jul '13

Hi, There will be a short outage of LSUThorns repositories this evening. See below for details. Frank ----- Forwarded message from Adam Yates <yates(a)cct.lsu.edu> ----- Date: Wed, 31 Jul 2013 15:27:18 -0500 From: Adam Yates <yates(a)cct.lsu.edu> To: allcct(a)cct.lsu.edu Subject: [Allcct] database maintenance - 6pm CST - 9pm CST Hi everyone; We will be performing some database maintenance this evening from 6pm CST to 9pm CST. During this time, some of the following services will be intermittently available: CCT's main website custom database / web applications group websites subversion wikis webmail * any other service not mentioned that accesses db.cct.lsu.edu. If you experience any difficulties after 9pm CST with any of the above services, please let us know. Thanks! Adam -- Adam Yates Manager of Information Technology Center for Computation and Technology (CCT) 231 Johnston Hall Baton Rouge, LA 70803 225.578.8235 _______________________________________________

1 0

Automated Interactive Tutorial for ET
by Dennis Castleberry 29 Jul '13

29 Jul '13

Dear users, In preparation for the user workshop, I had created a Virtualbox virtual machine (VM) with a pre-compiled ET and several other utilities (see ticket #780: https://trac.einsteintoolkit.org/ticket/780) installed to facilitate use of the ET for new users. Unfortunately, another VM had been deployed at the workshop for that same purpose before the existence of my own VM was made known to the organizers; thus mine never saw use. Here are the links to it: https://www.cct.lsu.edu/~dcastl2/etk/etk_2.2.zip<https://www.cct.lsu.edu/~dcastl2/etk/etk_2.2.zip.md5.txt> https://www.cct.lsu.edu/~dcastl2/etk/etk_2.2.zip.md5.txt https://www.cct.lsu.edu/~dcastl2/etk/etk_2.2.vdi.md5.txt<https://www.cct.lsu.edu/~dcastl2/etk/etk_2.2.zip.md5.txt> I would like to propose re-purposing my VM for an automated tutorial. For an (unrelated) automated testing project, I created a system which can automatically detect sub-images within a screen shot and move the mouse cursor to click on the sub-image (see here: http://www.youtube.com/watch?v=0xck7w-UeX8). It can also do automated keyboard entry. I would like to port this system to my ET VM to follow tutorials for setting up ET, running the WaveToy demo, thorn writing, etc. The tutorials would be interactive in the sense that they could be paused to allow for the user to give input. Such a system would provide an interactive ET tutorial which is accessible to everyone. Please let me know if you think this would be a worthwhile endeavor. If so, feel free to make any suggestions toward its development and feature set. Regards, Dennis Castleberry dcastl2(a)tigers.lsu.edu

1 0

problems with Visit 2.6.* and CarpetHDF5
by Bruno Giacomazzo 29 Jul '13

29 Jul '13

Hi, I have some issues using the latest version of Visit. When visualizing data in CarpetHDF5 format for some reasons VisIt does not show properly all the values, i.e., values below a certain threshold are visualized using the same color of the minimum in the colorbar. I have attached two images to make it more clear: rho_visit_263.jpeg shows the density using version 2.6.3, while rho_visit_232.jpeg shows the same quantity but using version 2.3.2. As you can see, 2.6.3 does not show properly the values below ~1e-6 (they all get the same color of 1e-10). I tried both the binary and source versions of visit 2.6.3 and they both show the same problem. I also tried the 2.6.0 binary version and it shows the same problem too. Any idea of what's going on? Thanks, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

2 2

-r & --redirect flags
by Scott Hawley 27 Jul '13

27 Jul '13

Has output redirection changed recently? >From my copy of the User's Guide: " -r[o|e|oe|eo] or--redirect=[o|e|oe|eo] Redirects the standard output (`o') and/or standard error (`e') of each processor to a le. By default, the standard outputs from processors other than processor 0 are discarded " Note that there is no space. But when I use either "-roe" or "--redirect=oe" I get Usage errors, e.g. /opt/local/bin/openmpirun -np 2 ./exe/cactus_bruiser --redirect=oe /Users/shawley/runs/cactus/bruiser/bruiser_test.par IN DIRECTORY: /Users/shawley/Cactus Usage: ./exe/cactus_bruiser [-h] [-O] [-o paramname] [-L n] [-W n] [-E n] [-r[o|e|oe|eo]] [--logdir <dir>] [-b <no|line|full>] [-S] [-T] [-t name] [--parameter-level <level>] [-v] <parameter_file_name> -------------------------------------------------------------------------- openmpirun noticed that the job aborted, but has no info as to the process that caused that situation. -------------------------------------------------------------------------- I must be doing something wrong. Any suggestions? Thanks, Scott

2 1

Einstein Toolkit Meeting Reminder
by Frank Loeffler 21 Jul '13

21 Jul '13

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. In addition to anything that might be brought up at the meeting, the workshop(s) next week need some discussion. On Trac, ticket changes are rare right now, but nevertheless the ET development version finally doesn't show failing tests again. The ticket counter hovers around 400 - which doesn't only include problem reports, but also quite a few wishlist items: of the 400 about 170 are flagged as 'defect'. There will be no phone call on July 29th. Instead, you may find increased number of commits and tickets/resolutions due to a locally increased Einstein Toolkit developer density - in short the ET workshop. Regular calls will resume a week later, on August 5th. Frank Löffler

1 0

simulation problem
by Udayaraj Khanal 18 Jul '13

18 Jul '13

I found there is a directory /scratch/ettest52/simulations/static_tov. There are three directories and a file inside static_tov. They are output-0000, output-0000-active, SIMFACTORY and a file log.txt. Inside output-0000 there is a directory SIMFACTORY. There is not any log files. Inside SIMFACTORY there are three files properties.ini, simulation-1374132863.07 and SubmitScript. The log.txt file in /scratch/ettest52/simulations/static_tov contains following details. [LOG:2013-07-18 02:34:19] self.create(simulationName, parfile)::Creating simulation static_tov [LOG:2013-07-18 02:34:19] self.create(simulationName, parfile)::Simulation directory: /scratch/ettest52/simulations/static_tov [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::Simulation Properties: [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile):: [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::[properties] [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::machine = queenbee [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.18-02.34.19-19349 [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::sourcedir = /home/ettest52/Cactus [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::configuration = sim [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::testsuite = False [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile):: [LOG:2013-07-18 02:34:22] self.create(simulationName, parfile)::Simulation static_tov created [LOG:2013-07-18 02:34:22] self.submit(submitScript)::Restart for simulation static_tov created with restart id 0, long restart id 0000 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::Prepping for submission [LOG:2013-07-18 02:34:22] self.submit(submitScript)::No previous walltime available to be used, using walltime 8:00:00 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::Defined substituion properties for submission [LOG:2013-07-18 02:34:22] self.submit(submitScript)::{'SIMULATION_ID': 'simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.18-02.34.19-19349', 'NODE_PROCS': 1, 'PPN_USED': 8, 'PPN': 8, 'ALLOCATION': 'NO_ALLOCATION', 'WALLTIME_HH': '08', 'CPUFREQ': '2.33', 'USER': 'ettest52', 'RUNDIR': '/scratch/ettest52/simulations/static_tov/output-0000', 'NODES': 4, 'SIMULATION_NAME': 'static_tov', 'WALLTIME': '8:00:00', 'NUM_THREADS': 8, 'EXECUTABLE': '/scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim', 'PROCS_REQUESTED': 32, 'EMAIL': 'urkhanal(a)tucdp.edu.np', 'RESTART_ID': 0, 'CHAINED_JOB_ID': '', 'FROM_RESTART_COMMAND': '', 'NUM_SMT': 1, 'WALLTIME_SECONDS': 28800, 'SIMFACTORY': '/home/ettest52/Cactus/simfactory/bin/sim', 'PROCS': 32, 'SUBMITSCRIPT': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'WALLTIME_HOURS': 8.0, 'WALLTIME_MM': '00', 'PARFILE': '/scratch/ettest52/simulations/static_tov/output-0000/static_tov.par', 'WALLTIME_SS': '00', 'QUEUE': 'checkpt', 'CONFIGURATION': 'sim', 'SOURCEDIR': '/home/ettest52/Cactus', 'HOSTNAME': 'qb3.loni.org', 'NUM_PROCS': 4, 'SCRIPTFILE': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'MEMORY': '8192', 'WALLTIME_MINUTES': 480, 'SHORT_SIMULATION_NAME': 'static_tov-0000'} [LOG:2013-07-18 02:34:22] self.submit(submitScript)::self.Properties: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/properties.ini [LOG:2013-07-18 02:34:22] self.submit(submitScript):: [LOG:2013-07-18 02:34:22] self.submit(submitScript)::[properties] [LOG:2013-07-18 02:34:22] self.submit(submitScript)::machine = queenbee [LOG:2013-07-18 02:34:22] self.submit(submitScript)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.07.18-02.34.19-19349 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::sourcedir = /home/ettest52/Cactus [LOG:2013-07-18 02:34:22] self.submit(submitScript)::configuration = sim [LOG:2013-07-18 02:34:22] self.submit(submitScript)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus [LOG:2013-07-18 02:34:22] self.submit(submitScript)::buildid = build-sim-qb4.loni.org-ettest52-2013.07.18-07.25.24-12102 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::testsuite = False [LOG:2013-07-18 02:34:22] self.submit(submitScript)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim [LOG:2013-07-18 02:34:22] self.submit(submitScript)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList [LOG:2013-07-18 02:34:22] self.submit(submitScript)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript [LOG:2013-07-18 02:34:22] self.submit(submitScript)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript [LOG:2013-07-18 02:34:22] self.submit(submitScript)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par [LOG:2013-07-18 02:34:22] self.submit(submitScript)::chainedjobid = -1 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::ppn = 8 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::procsrequested = 32 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::allocation = NO_ALLOCATION [LOG:2013-07-18 02:34:22] self.submit(submitScript)::user = ettest52 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::numsmt = 1 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::walltime = 8:00:00 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::numprocs = 4 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::nodeprocs = 1 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::numthreads = 8 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::hostname = qb3.loni.org [LOG:2013-07-18 02:34:22] self.submit(submitScript)::ppnused = 8 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::queue = checkpt [LOG:2013-07-18 02:34:22] self.submit(submitScript)::cpufreq = 2.33 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::procs = 32 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::memory = 8192 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::nodes = 4 [LOG:2013-07-18 02:34:22] self.submit(submitScript)::pbsSimulationName= static_tov-0000 [LOG:2013-07-18 02:34:22] self.submit(submitScript):: [LOG:2013-07-18 02:34:22] self.submit(submitScript)::saving substituted submitscript contents to: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript [LOG:2013-07-18 02:34:22] self.submit(submitScript)::Executing submission command: qsub /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript [LOG:2013-07-18 02:34:22] self.makeActive()::Simulation static_tov with restart-id 0 has been made active [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::received raw output: Invalid allocation "NO_ALLOCATION". [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::===================== Allocation information for ettest52 ===================== [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output):: Proj. Name| Alloc| Balance| Deposited| %Used| Days Left| End [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::-------------------------------------------------------------------------------- [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::loni_cactus08|loni_cactus08 on @Dell_Cluster|106210.43| 250000.00| 57.52| 167|2014-01-01 [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output):: [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::Note: Balance and Deposit are measured in CPU-hours [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::qsub: submit filter returned an error code, aborting job submission. [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output):: [LOG:2013-07-18 02:34:23] job_id = self.extractJobId(output)::using submitRegex: (\d+)[.]qb2 [LOG:2013-07-18 02:34:23] self.submit(submitScript)::After searching raw output, it was determined that the job_id is: -1 [LOG:2013-07-18 02:34:23] self.submit(submitScript)::The regex did NOT match anything. No job_id means no control. [LOG:2013-07-18 02:35:29] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 I do not understand what these means. What I have to do to complete simulation? Udayaraj KhanalProfessorCentral Department of PhysicsTribhuvan UniversityKirtipur, Nepal

2 1

dependencies & CST warnings
by Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] 17 Jul '13

17 Jul '13

Hi. I'm compiling an *extremely* simple cactus executable designed to run the testsuite "vectors" in LSUThorns/Vectors. If I were to believe the parameter file vectors.par, I should need exactly the following seven thorns: CartGrid3D CoordBase IOASCII IOUtil PUGH PUGHSlab Vectors However, if I begin to compile a minimal Cactus configuration using only these (in current ET stable release Gauss --- ET_2013_05), I quickly hit the warnings below, implying that I *also* need SymBase and Boundary. Now these are warnings, not errors; perhaps the code would compile fine and run OK anyway (I haven't tried). But surely trying to schedule a routine in a particular group or timebin means that the thorn responsible for such groups/timebins should be an actual dependency? Why are Boundary and SymBase not *required* in the interface.ccl of CartGrid3D? Bernard ------------------------------------------------------ Warnings were generated during execution of the CST ------------------------------------------------------ CST warning: -> Scheduling routine RegisterSymmetryBoundaries from thorn CartGrid3D in non-existent group or timebin SymmetryRegister HINT: If this routine should be scheduled check the spelling of the group or timebin name. Note that scheduling IN must be used to schedule a routine to run in a thorn-defined schedule group, whereas scheduling AT is used for a usual timebin. (Schedule IN may also be used with the usual timebins, but in this case the full name of the bin must be used, e.g. CCTK_EVOL and not EVOL) CST warning: -> Scheduling routine CartGrid3D_ApplyBC from thorn CartGrid3D in non-existent group or timebin BoundaryConditions HINT: If this routine should be scheduled check the spelling of the group or timebin name. Note that scheduling IN must be used to schedule a routine to run in a thorn-defined schedule group, whereas scheduling AT is used for a usual timebin. (Schedule IN may also be used with the usual timebins, but in this case the full name of the bin must be used, e.g. CCTK_EVOL and not EVOL)

3 2

New Thorn: LlamaWaveHyperboloidal
by Michael Jasiulek 17 Jul '13

17 Jul '13

Hello all, here is a new thorn: https://bitbucket.org/mjasiulek/llamawavehyperboloidal <https://bitbucket.org/mjasiulek/llamawavehyperboloidal> =============================================================================== LlamaWaveHyperboloidal =============================================================================== This thorn allows to evolve the scalar wave equation in 3D in Kerr spacetimes using Kerr-Schild coordinates which are horizon penetrating. At some transition radius the user can specify where the Kerr-Schild slicing becomes hyperboloidal, thus, 'bends up' to scri (future null infinity). Therefore, no outer (and inner) boundary conditions have to be set, radiation leaves the spacetime 'naturally' through the inner horizon and scri. The waveform at scri is readily computed during the evolution. It uses Llama's spherical 6-patch coordinates (r,u,v), the method-of-lines with 4th-order Runge-Kutta and 4th-,6th- or 8th-order spatial finite-differencing and matching Kreiss-Oliger dissipation. In http://arxiv.org/abs/1109.2513 <http://arxiv.org/abs/1109.2513> (or http://iopscience.iop.org/0264-9381/29/1/015008/ <http://iopscience.iop.org/0264-9381/29/1/015008/> ) I used this code to study QNMs, late-time tails in Kerr at scri and to test the novel hyperboloidal transformation used to smoothly attach the hyperboloidal slice. If you find this code useful, please cite the paper Class. Quantum Grav. 29 015008 (2012). =============================================================================== Enjoy, Michael Jasiulek .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users July 2013