Users August 2013

users@lists.einsteintoolkit.org

11 participants
22 discussions

Einstein Toolkit Meeting Reminder
by Frank Loeffler 14 May '15

14 May '15

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ I short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. My agenda contains: - Fall workshop - PLEASE let us know if you come [1]! - Hotel booking - Release planning - release date: Mon Oct 24 2011 - freeze date: Mon Sep 26 - the day of the call! - branch name: ET_2011_10 - List of release-concerning bugs [2] - ET paper: proposed "freeze" by Oct 3rd (Mon after this call) As always: feel free to add to this list. Frank Loeffler [1] https://docs.einsteintoolkit.org/et-docs/ET_Workshop_Fall_2011#Fall_Einstei… [2] https://trac.einsteintoolkit.org/query?status=!closed&milestone=ET_2011_11&…

10 31

strong scaling tests on Stampede (TACC)
by Bruno Giacomazzo 19 Sep '13

19 Sep '13

Hi, I did some strong scaling tests on Stampede (https://www.xsede.org/tacc-stampede) with both Whisky and GRHydro (using the development version of ET in both cases). I used a Carpet par file that Roberto DePietri provided me and that he used for similar tests on an Italian cluster (I have attached the GRHydro version, the Whisky one is similar except for using Whisky instead of GRHydro). I used both Intel MPI and Mvapich and I did both pure MPI and MPI/OpenMP runs. I have attached a text file with my results. The first column is the name of the run (if it starts with mvapich it used mvapich otherwise it used Intel MPI), the second one is the number of cores (option --procs in simfactory), the third one the number of threads (--num-threads), the fourth one the time in seconds spent in CCTK_EVOL, and the fifth one the walltime in seconds (i.e., the total time used by the run as measured on the cluster). I have also attached a couple of figures that show CCTK_EVOL vs #cores and walltime vs #cores (only for Intel MPI runs). First of all, in pure MPI runs (--num-threads=1) I was unable to run on more than 1024 cores using Intel MPI (the run was just crashing before iteration zero or hanging up). No problem instead when using --num-threads=8 or --num-threads=16. I also noticed that scaling was particularly bad in pure MPI runs and that a lot of time was spent outside CCTK_EVOL (both with Intel MPI and MVAPICH). After speaking with Roberto, I found out that the problem is due to 1D ASCII output (which is active in that parfile) and that makes the runs particularly slow above ~100 cores on this machine. In plot_scaling_walltime_all.pdf I plot also two pure MPI runs, but without 1D ASCII output and the scaling is much better in this case (the time spent in CCTK_EVOL is identical to the case with 1D output and hence I didn't plot them in the other figure). I didn't try using 1D hdf5 output instead, does anyone use it? According to my tests, --num-threads=16 performs better than --num-threads=8 (which is the current default value in simfactory) and Intel MPI seems to be better than MVAPICH. Is there a particular reason for using 8 instead of 16 threads as the default simfactory value on Stampede? Let me know if you have any comment or suggestion. Cheers, Bruno Dr. Bruno Giacomazzo JILA - University of Colorado 440 UCB Boulder, CO 80309 USA Tel. : +1 303-492-5170 Fax : +1 303-492-5235 email : bruno.giacomazzo(a)jila.colorado.edu web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

6 10

Meeting Minutes
by Roland Haas 12 Sep '13

12 Sep '13

Present: Frank, Roland, Peter, Erik, Matt, Ian, Josh, Maria, Yosef, Philipp git to svn transition: * Ian (and other contributors) created a wiki page outlying ideas for how to organize repositories: https://docs.einsteintoolkit.org/et-docs/Version_control * majority of call participants prefer git over svn, points mentioned are easier management of branches and patches and availability of powerful GUIs * suggest using GUIs even for "advanced" users. For git there is SourceTree for OSX and SmartGit or GitEye (see http://git-scm.com/downloads/guis) Goals to achieve while moving the repositories, some of which might be conflicting, in no particular ordering: * simplify exchanging patches, in particular patches for review where right now we exchange patch files, instead we might want to point to a revision in a branch or private repository * consistency of thorns in toolkit is easier to achieve is strongly dependent thorns are in a single repository * simplify work-cycle for developers, read-only access for end-users is already provided by GetComponents * minimize amount of code downloaded * keep modularity of framework, avoid monolithic repository Opinions: * consistency is given low priority, instead require framework to enforce this (eg via interface.ccl etc). * avoid overly large repository with many unrelated thorns * using branches and cherry picking commits to choose a working set of thorns that mixes cutting edge versions of some thorns and bugfix-only versions of another is hard to do, takes lots of time. * we lack the manpower to keep a always working mainline for all thorns, GRHydro currently follows this paradigm and requires ~0.5 days a week to choose and push patches. The advantage is that Zelmani can do whatever it wants and does not have to worry about "external" users pulling code that contains bugs. This is not in line with the ET philisophy of having trunk branches under development then releases every 6 months. Hosting: * Frank (who will be admin) advertises to host the git repos at CCT, Ian suggests using gitolite as the front-end which uses a plain git repository for admin task (keeps record of admin actions) * desire to be able to automate actions on multiple repositories (assuming we will have multiple/many repositories) * suggestion to consider bitbucket/github as external hoster for speed and/or certificate issues, did not find much support * want to make sure that certificates work and that speed is acceptable * rather spend some money than spend time * bitbucket might be a viable option when using teams and would simplify infrequent tasks, major problem is scriptability * github source code made be available to use even for locally hosted repos if we want a web-based UI, Ian to test this Proposed repository structure: * structure not fully decided on, use wiki page (see above) as location to hash out proper layout Yours, Roland

6 10

Einstein Toolkit Meeting Reminder
by Frank Loeffler 31 Aug '13

31 Aug '13

Hi Please consider joining the weekly Einstein Toolkit phone call at 10 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ In short: the number is (+1) 225-578-4942 or (+1) 866-573-0359 and the conference id is 118682#. As always, we will answer questions and hear comments that come up at the meeting. Please take a look at issues that are under review [1]. We currently have 8, but some of them are already waiting for quite a while. Frank Löffler [1] https://trac.einsteintoolkit.org/query?status=review&col=id&col=summary&col…

1 1

Multigrid elliptic solver released.
by Eloisa Bentivegna 28 Aug '13

28 Aug '13

Dear Cactus and Einstein Toolkit users, I have been working on a solver for elliptic PDEs based on the multigrid paradigm, and developed as a Cactus thorn using Carpet. The solver is now able to tackle some basic problems and could be applied to other cases of interest to this community. You can obtain a copy by checking out the Cosmology arrangement from its repository: git clone git@bitbucket.org:eloisa/cosmology.git There you will also find documentation, license information, and a test suite. There is no tutorial, but I think that what is under docs/ and par/ should suffice to get most people started. For more information, you can also peruse <http://arxiv.org/abs/arXiv:1305.5576>, especially if you are interested in elliptic problems on spaces without boundaries. Please notice that the spirit of the solver is not to replace existing state-of-the-art tools, but to provide a way of solving elliptic PDEs with is both easy to learn and to extend, and gives reasonable answers in a reasonable amount of time. In order to keep this flexibility, the solver is simply unable to compete with other software tailored to specific equations or geometries. In areas where these alternatives exist, they should be used instead. Finally, beware of the lack of explicit support for some of the latest Carpet features, such as multipatch and cell-centered grids. Being unfamiliar with these features, I cannot estimate how much work would be necessary to include them. I hope many of you will find this tool useful, and I look forward to hear your feedback. Eloisa

1 0

Meeting Minutes
by Roland Haas 26 Aug '13

26 Aug '13

Present: Frank, Roland, Matt, Ian, Erik, Philipp, Steve, Peter * please review the outstanding tickets on track [1] * Roland introduced and fixed a bug in CarpetIOHDF5 that prevented recovering from checkpoints from Sun Aug 11 2013 to Sat Aug 17 2013 (fdadb2c to c02dd99) * Philipp will commit the cell centered vector potential code (bugfixes remain) after testing it with a dynamic simulation (TOV works) * Frank will contact Erik to learn how to run Cactus on the Phi processors Yours, Roland

1 0

Meeting Minutes
by Roland Haas 21 Aug '13

21 Aug '13

Present: Frank, Roland, Peter, Erik, Matt, Ian, Josh, Yosef, Philipp, Oleg Cactus tutorial: * Oleg will give a short tutorial on Cactus at Nordita * suggest using current material from last workshop as basis * possibly more detailed tutorial later for interested parties * happy to have telecon with interested parties afterwards Point release: * there was a suggestion to have a point release with the accumulated patches * no support for this right now, instead will keep release branch and consider adding more bugfixes to it once we have a list of current bugs * suggest to add the word "backport" to ticket keywords when a bugfix should be backported to the release * Ian to create a list of bugfixes since the release (from the superrepository) * Ian to send around information on how to sign up for Jenkins (testing system) email Yours, Roland

2 2

Simulation error
by Udayaraj Khanal 19 Aug '13

19 Aug '13

Dear All, I submitted a simulation from Cactus directory as ./simfactory/bin/sim submit static_tov --parfile=par/static_tov.par --procs=32 --walltime=8:0:0 --allocation=loni_cactus08 Then ./simfactory/bin/sim list-simulations gives static_tov [ACTIVE(FINISHED), restart 0000, job id 705851] I found in /scratch/ettest52/simulations/static_tov/output-0000 there is a folder SIMFACTORY and two file static_tov.err and static_tov.out but static_tov directory is not present as mentioned in Tutorial for New Users. File static_tov.err contains {PBS: exec of shell failed} and file static_tov.out contains following details, -------------------------------------- Running PBS prologue script -------------------------------------- User and Job Data: -------------------------------------- Job ID: 705851.qb2 Username: ettest52 Group: lsuusers Date: 17-Aug-2013 13:52 Node: qb031 (11969) -------------------------------------- PBS has allocated the following nodes: qb031 qb030 qb029 qb028 A total of 32 processors on 4 nodes allocated --------------------------------------------- Check nodes and clean them of stray processes --------------------------------------------- Checking node qb031 13:52:20 Checking node qb030 13:52:21 Checking node qb029 13:52:23 Checking node qb028 13:52:25 Done clearing all the allocated nodes ------------------------------------------------------ Concluding PBS prologue script - 17-Aug-2013 13:52:25 ------------------------------------------------------ ------------------------------------------------------ Running PBS epilogue script - 17-Aug-2013 13:52:26 ------------------------------------------------------ Checking node qb031 (MS) Checking node qb028 ok Checking node qb029 ok Checking node qb030 ok Checking node qb031 ok ------------------------------------------------------ Concluding PBS epilogue script - 17-Aug-2013 13:52:33 ------------------------------------------------------ Exit Status: Job ID: 705851.qb2 Username: ettest52 Group: lsuusers Job Name: static_tov-0000 Session Id: 11968 Resource Limits: ncpus=1,nodes=4:ppn=8,walltime=08:00:00 Resources Used: cput=00:00:00,mem=428kb,vmem=3884kb,walltime=00:00:00 Queue Used: checkpt Account String: loni_cactus08 Node: qb031 Process id: 12492 ------------------------------------------------------ I also found a log.txt file in /scratch/ettest52/simulations/static_tov which contains, [LOG:2013-08-17 04:21:54] self.create(simulationName, parfile)::Creating simulation static_tov [LOG:2013-08-17 04:21:54] self.create(simulationName, parfile)::Simulation directory: /scratch/ettest52/simulations/static_tov [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::Simulation Properties: [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile):: [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::[properties] [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::machine = queenbee [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.08.17-04.21.54-5675 [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::sourcedir = /home/ettest52/Cactus [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::configuration = sim [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::buildid = build-sim-qb4.loni.org-ettest52-2013.08.17-09.01.14-32035 [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::testsuite = False [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile):: [LOG:2013-08-17 04:21:56] self.create(simulationName, parfile)::Simulation static_tov created [LOG:2013-08-17 04:21:56] self.submit(submitScript)::Restart for simulation static_tov created with restart id 0, long restart id 0000 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::Prepping for submission [LOG:2013-08-17 04:21:56] self.submit(submitScript)::No previous walltime available to be used, using walltime 8:00:00 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::Defined substituion properties for submission [LOG:2013-08-17 04:21:56] self.submit(submitScript)::{'SIMULATION_ID': 'simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.08.17-04.21.54-5675', 'NODE_PROCS': 1, 'PPN_USED': 8, 'PPN': 8, 'ALLOCATION': 'loni_cactus08', 'WALLTIME_HH': '08', 'CPUFREQ': '2.33', 'USER': 'ettest52', 'RUNDIR': '/scratch/ettest52/simulations/static_tov/output-0000', 'NODES': 4, 'SIMULATION_NAME': 'static_tov', 'WALLTIME': '8:00:00', 'NUM_THREADS': 8, 'EXECUTABLE': '/scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim', 'PROCS_REQUESTED': 32, 'EMAIL': 'urkhanal(a)tucdp.edu.np', 'RESTART_ID': 0, 'CHAINED_JOB_ID': '', 'FROM_RESTART_COMMAND': '', 'NUM_SMT': 1, 'WALLTIME_SECONDS': 28800, 'SIMFACTORY': '/home/ettest52/Cactus/simfactory/bin/sim', 'PROCS': 32, 'SUBMITSCRIPT': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'WALLTIME_HOURS': 8.0, 'WALLTIME_MM': '00', 'PARFILE': '/scratch/ettest52/simulations/static_tov/output-0000/static_tov.par', 'WALLTIME_SS': '00', 'QUEUE': 'checkpt', 'CONFIGURATION': 'sim', 'SOURCEDIR': '/home/ettest52/Cactus', 'HOSTNAME': 'qb3.loni.org', 'NUM_PROCS': 4, 'SCRIPTFILE': '/scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript', 'MEMORY': '8192', 'WALLTIME_MINUTES': 480, 'SHORT_SIMULATION_NAME': 'static_tov-0000'} [LOG:2013-08-17 04:21:56] self.submit(submitScript)::self.Properties: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/properties.ini [LOG:2013-08-17 04:21:56] self.submit(submitScript):: [LOG:2013-08-17 04:21:56] self.submit(submitScript)::[properties] [LOG:2013-08-17 04:21:56] self.submit(submitScript)::machine = queenbee [LOG:2013-08-17 04:21:56] self.submit(submitScript)::simulationid = simulation-static_tov-queenbee-qb3.loni.org-ettest52-2013.08.17-04.21.54-5675 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::sourcedir = /home/ettest52/Cactus [LOG:2013-08-17 04:21:56] self.submit(submitScript)::configuration = sim [LOG:2013-08-17 04:21:56] self.submit(submitScript)::configid = config-sim-qb4.loni.org-home-ettest52-Cactus [LOG:2013-08-17 04:21:56] self.submit(submitScript)::buildid = build-sim-qb4.loni.org-ettest52-2013.08.17-09.01.14-32035 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::testsuite = False [LOG:2013-08-17 04:21:56] self.submit(submitScript)::executable = /scratch/ettest52/simulations/static_tov/SIMFACTORY/exe/cactus_sim [LOG:2013-08-17 04:21:56] self.submit(submitScript)::optionlist = /scratch/ettest52/simulations/static_tov/SIMFACTORY/cfg/OptionList [LOG:2013-08-17 04:21:56] self.submit(submitScript)::submitscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/SubmitScript [LOG:2013-08-17 04:21:56] self.submit(submitScript)::runscript = /scratch/ettest52/simulations/static_tov/SIMFACTORY/run/RunScript [LOG:2013-08-17 04:21:56] self.submit(submitScript)::parfile = /scratch/ettest52/simulations/static_tov/SIMFACTORY/par/static_tov.par [LOG:2013-08-17 04:21:56] self.submit(submitScript)::chainedjobid = -1 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::ppn = 8 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::procsrequested = 32 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::allocation = loni_cactus08 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::user = ettest52 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::numsmt = 1 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::walltime = 8:00:00 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::numprocs = 4 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::nodeprocs = 1 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::numthreads = 8 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::hostname = qb3.loni.org [LOG:2013-08-17 04:21:56] self.submit(submitScript)::ppnused = 8 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::queue = checkpt [LOG:2013-08-17 04:21:56] self.submit(submitScript)::cpufreq = 2.33 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::procs = 32 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::memory = 8192 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::nodes = 4 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::pbsSimulationName= static_tov-0000 [LOG:2013-08-17 04:21:56] self.submit(submitScript):: [LOG:2013-08-17 04:21:56] self.submit(submitScript)::saving substituted submitscript contents to: /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript [LOG:2013-08-17 04:21:56] self.submit(submitScript)::Executing submission command: qsub /scratch/ettest52/simulations/static_tov/output-0000/SIMFACTORY/SubmitScript [LOG:2013-08-17 04:21:56] self.makeActive()::Simulation static_tov with restart-id 0 has been made active [LOG:2013-08-17 04:21:56] job_id = self.extractJobId(output)::received raw output: 705851.qb2 [LOG:2013-08-17 04:21:56] job_id = self.extractJobId(output):: [LOG:2013-08-17 04:21:56] job_id = self.extractJobId(output)::using submitRegex: (\d+)[.]qb2 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::After searching raw output, it was determined that the job_id is: 705851 [LOG:2013-08-17 04:21:56] self.submit(submitScript)::Simulation static_tov, with restart id 0, and job id 705851 has been submitted [LOG:2013-08-17 04:23:24] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 [LOG:2013-08-17 11:46:32] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 [LOG:2013-08-18 10:18:50] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 [LOG:2013-08-18 11:40:56] ret = restart.load(sim, activeId)::For simulation static_tov, loaded restart id 0, long restart id 0000 How can I complete the simulation? Udayaraj Khanal

2 1

Test status
by Ian Hinder 19 Aug '13

19 Aug '13

Hi, If you want to keep up with the current status of the automated build and test of the Einstein Toolkit, there are several options. 1. You can browse the test system at https://build.barrywardell.net/ 2. Follow the RSS feeds advertised on that page, and on the job-specific page. For example, the RSS feed for the EinsteinToolkit trunk is https://build.barrywardell.net/job/EinsteinToolkit/rssAll for all test runs, and https://build.barrywardell.net/job/EinsteinToolkit/rssFailed if you just want to know about failures. 3. Subscribe to the test mailing list at http://lists.einsteintoolkit.org/mailman/listinfo/test Note that occasionally you get some inaccurate items in the feed, for example if there is a problem with the test system. The test mailing list is very new, and we are still experimenting a bit with it. We currently have three "jobs" which are tested; the EinsteinToolkit trunk, EinsteinToolkitProposed, which includes some additional thorns and tests, and EinsteinToolkitRelease, which corresponds to the release branch. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

1 0

Changes to release branch since the release
by Ian Hinder 19 Aug '13

19 Aug '13

Hi, The list of changes to the release branch since the release can be queried using the following URL: http://git.barrywardell.net/EinsteinToolkit.git/log/dac2e71fc0601829d992f79… Currently the list is: * simfactory c04079f...467325a (1): > supermuc-ibmmpi.sub: Change job class to 'micro' for jobs on <= 32 nodes * repos/CRL b11ce17...d9b0f3d (1): > fix uninitialized variable (typo), closes #1421 * flesh 85dd567...56632c4 (1): > fix implementation of 'complex' for c++0x, see #1374 * repos/CRL 8a48ebf...b11ce17 (1): > use 'old' git command syntax to change to different branch to be able to use old git versions We should probably go through all the changes since the release and decide if any are important enough to be backported. -- Ian Hinder http://numrel.aei.mpg.de/people/hinder

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users August 2013