Users May 2017

users@lists.einsteintoolkit.org

20 participants
33 discussions

reminder: please voice your preferences for the ET school topics at NCSA in July/August
by Roland Haas 15 May '17

15 May '17

Hello all, please remember to fill out this: https://docs.google.com/forms/d/e/1FAIpQLSfcMSNB2TW1uNM-7ktxG_X4Jya7ah_Fb7V… Google form in the next week (until including Monday May 15th) so that we can take the results into account when settling the details of the school program (which we hope to have set by the end of next week). This is for the school in the US (July/August), not the one in the EU (October). Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 3

meeting minutes for 2017-05-15
by Roland Haas 15 May '17

15 May '17

Present: Andreas , Bhavesh, Eliu, Qian, Erik, Frank, Roberto, Vassili, Roland, Steve, Ian Hangout limit: * reached again, will try alternatives at increased reate ** could try webex but no Linux client and using a phone is not convenient (eg one needs to use specific headset) ** will look into zoom, other options New thorns: LORENE: * needs to be reviews Failing tests: * Roland has not looked into SphericalHarmoncRecon test * Frank has not looked into GRHydro ** older tests by Roland indicate that the NaNs go away and the test passed if one disables the assoicative math option of -Ofast ** right now the exact file / code that is affected by this is not known now whether this is a compiler bug or not KNL performance: * is highly depended on MPI and OpenMP settings * best situation is often pure MPI since our MPI parallelization is more efficient than our OpenMP parallelization * comparison per node on NERSC Cori Erik finds that Haswell nodes are 5 times as fast as KNL. This is due to our code generator being optimized for Haswell. For just a kernel Erik got 5% of peak on Haswell and 10% on KNL. So we should be able to better but it will require changes to the code generator * the crucial change seems to be to introduce tile based thread parallelization. LoopControl does not quite work yet, Kranc already does this tiling but its overhead is too large. * plus local stuff like vectorization, avoid divisions, sqrt Vectorization in hydro code: * the hydro code in CactusExamples performs better than GRHydro * better able to use vectorization * Erik would like to give a presentation on this in the near future KNL on stampede: * will only be able to provide simfactory files for KNL machines but not optimized code * currently works on Cori and Marconi but not Stampede * removing tacc_affinity helps a bit but now it crashes in TwoPunctures after interpolation * Ian will provide instructions via email on how to obtain a core dump file new user tutorial: * add mention in tutorial that this file is tuned for single workstation and won't work on cluster with many nodes due to resolution being too low * add some instructions on how to use ET as a black box ** diagnostics output in example parfiles ** env variables to set ** how to set up ET on new machine RNSID: * proposal to introduce RNSID into ET * https://bitbucket.org/GravityPR/prthorns.git * ask for inclusion process to be started just after the next release which should be about a month from now * guidelines for contribution are: http://einsteintoolkit.org/contribute.html Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.

1 0

Re: [Users] Einstein Toolkit Meeting Reminder
by Frank Loeffler 15 May '17

15 May '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ Tomorrow, we might use something else than a hangout. I will post instructions as usual, on this list, about 15 minutes before the meeting, and they will be also posted on https://docs.einsteintoolkit.org/et-docs/Main_Page Some background: google hangouts have a limit of 10 participants, and even for registered organizations the hard limit is 15. We have hit 10 more than we liked in the recent past, and can see 15 be a problem in the near future as well. So, we seek to switch to another solution. Like GPU development is to quite an extent driven by the gaming industry, voip seems to be too, at least recently. Some of the developers will test "discord" tomorrow morning - let me know if you are also interested, and available ~1h before the meeting. It's advantage over the other two "big" methods (Mumble and TeamSpeak) is that discord provides a web-client, and thus, doesn't require (but allows) dedicated software installation. Frank Löffler

2 3

Tracer particles: Processor distribution of array
by Jens Mahlmann 10 May '17

10 May '17

Dear all, I am currently working on the implementation of tracer particles into an evolution code. As for this, my orientation is the particle implementation of GRHydro. The number of particles is, in this case, is distributed equally across the processors by the DISTRIB=DEFAULT option. Is there any experience out there with distributing arrays according to a transfer function (e.g., some sort of density), such that higher density regions have more traced particles? In the documentation I only found the DISTRIB=CONSTANT option - which would not serve the purpose. Thank you very much and my best! Jens

3 2

*European* Einstein Toolkit Workshop 2017 (October 11-14, Palma de Mallorca)
by Bruno Giacomazzo 10 May '17

10 May '17

Dear All, this is to announce that, as discussed during the latest EU Einstein Toolkit workshop in Trento, we are organizing also this year also an *EU edition* of the Einstein Toolkit workshop in Europe. The idea of having a workshop also in Europe is as usual to make it easier for ET users in Europe or nearby countries to participate, but also to help the "European" community of users and developers to grow. We will not have a proper school, as we did instead in Trento last year, but we will have anyway one afternoon with a tutorial for new users on the first day of the meeting. The workshop will take place in *Palma de Mallorca *from October 11 to 13 and it will be followed by a special event on Saturday October 14 to celebrate Prof. Ed Seidel (Prof. Bernard Schutz and Prof. Bill Gropp have already confirmed their participation to this special event and we expect many others to attend). At this stage we would like to know how many of you are potentially interested in participating to the EU ET workshop and if you have any special topic you would like to discuss. So, please fill the following form by *May 15*: https://goo.gl/forms/wHHUAha4lMOMSwoj2 So that we can start getting an idea of how many people may come and start booking the hotel. A proper web page and announcement for the workshop will be circulated once we have also fixed the exact location (we are thinking of having it in a beach resort near Palma) and a more detailed program. Please do not hesitate to contact me, or any of the other organizers in cc to this email, if you have any question or suggestion. Thanks, Bruno -- Prof. Bruno Giacomazzo Department of Physics University of Trento via Sommarive 14 38123 Trento Italy Tel. : +39 0461281631 email : bruno.giacomazzo(a)unitn.it web: http://www.brunogiacomazzo.org ---------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ----------------------------------------------------------------------

1 0

dates for Einstein Toolkit workshop and school at NCSA
by Roland Haas 08 May '17

08 May '17

Hello all, please remember to fill out the doodle poll http://doodle.com/poll/p7deb9npmgwwfbmv to ndicate your preferred dates by checking at least 5 consecutive days in in this doodle poll (Mo starting times will be prioritized, you can use ifneedsbe for secondary choices). This will also serve as a very preliminary count of the interested participants. Please respond no later than *tomorrow*, Thursday as well will finalize room bookings for the lecture halls on Friday. If you would like to serve as an instructor/TA please let us (Roland Haas, Eliu Huerta, Gabrielle Allen or just list) know as well. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 2

meeting minutes for 2017-05-08
by Roland Haas 08 May '17

08 May '17

Present: Bhavesh, Eloisa, Frank. Gabrielle, Peter, Steve, Gela Hammerling, Ian, Roberto, Vassili, Zach, Roland, Qian Report on ET workshop planning: * planned for week starting July 31st, 3 days of school, 2 days of workshop * themes not fully decided yet, will sent out poll soon as possible (today) to query prior knowledge, which will let us asses where to pick people up. Topics will be a mix of lectures on NR and numerical methods plus hands on session incl. visualization, ET+LIGO * should be able to broadcast and record the lectures since NCSA has rooms equipped with cameras and trained staff to handle * workshop will likely dedicate one day to actual work and collaboration, will likely ask people to announce what they would like to work on at the time or registration * suggestion to ask for example Bill Gropp on a keynote like talk for the workshop ET release: * postponed a bit this year * request for review of new LORENE thorn * Piraha parser needs to have the tickets in review applied, debug output removed * failing test suites must be fixed, we will not release with failing tests: https://build-test.barrywardell.net/job/EinsteinToolkit/lastCompletedBuild/… * Frank will try and look at the GRHydro test * Roland will look at SphericalHarmonicReconGen GiRaFFE: * Zach gave a quick overview about it, https://arxiv.org/pdf/1704.00599.pdf * not for inclusion in this release yet Survey on ET data vault: https://docs.google.com/forms/d/e/1FAIpQLSdLrcXVJXKZBv3PAIaXtgj6YMylPVxwx-c… * there was discussion on the proper licenses to offer and how indexing by search engines would be implemented Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 2

Einstein Toolkit Meeting Reminder
by Frank Loeffler 08 May '17

08 May '17

Hi Please consider joining the weekly Einstein Toolkit phone call at 9:30 am US central time on Mondays. As usual, you can find instructions how to join on the following web site: http://einsteintoolkit.org/community/support/ We use a Google hangout, with an URL announced on this list about 15 minutes before the meeting, and will also posted on https://docs.einsteintoolkit.org/et-docs/Main_Page Tomorrow, representatives from the NSF grant sites will give a short summary (not more than 5 minutes each) of the current states of their sub-projects. We will continue to do this roughly monthly from now on. Other projects involving the Einstein Toolkit are encouraged to chip in and report about their progress as well. Another topic of discussion will be the upcoming Einstein Toolkit workshop, likely to happen end of July at NCSA. This is only in the early stages of planning, so input is now especially valuable. Frank Löffler

4 6

Benchmarking
by Khamesra, Bhavesh 08 May '17

08 May '17

Hi, I have sent the pull request with the optionlist for Stampede - KNL on Bitbucket simfactory repo. I have tested this with a couple of thornlists including the einsteintoolkit.th and GW150914.th. This is still in experimental stage and so would be great if someone could also test it. Working on benchmarking the performance on Stampede KNL, I was able to do some test runs using the GW150914 simulation. However, I have been running into some issues with it. 1. I tried running QC0 simulation on both Stampede SandyBridge and KNL. While it runs fine on Stampede but it crashes on KNL with this error - while executing schedule bin BoundaryConditions, routine RotatingSymmetry180::Rot180_ApplyBC in thorn RotatingSymmetry180, file /work/04082/tg833814/Cactus_ETK_dev/arrangements/CactusNumerical/RotatingSymmetry180/src/rotatingsymmetry180.c:460: -> TAT/Slab can only be used if there is a single local component per MPI process TACC: MPI job exited with code: 134 I looked up at previous tickets and found the solution to increase the number of cores. But if the same simulation can be run on stampede on 64 cores, why does it require higher number of cores on KNL? Or is it some other issue? 2. I was able to run GW150914 on development queue (68 cores) and the speeds on Stampede were around 12.9M while that on KNL goes around 2.4M. To understand the reason for such small speeds, I tried running this on higher number of cores on Stampede (128) and it runs at speed of around 20.9M (tested the run for 12 hours). However, on doing the same in normal queue in KNL, the simulation crashes after a couple of iterations on KNL with some segmentation fault error. Also, before crashing, the speed on KNL is around 4.2M. I have attached the error file of the simulation. Could someone please look at this? Let me know if you need any other information. Thanks ............................. Bhavesh Khamesra Graduate Student Centre of Relativistic Astrophysics Georgia Institute of Technology

4 13

SimulationTools Frequency
by Gwyneth Allwright 06 May '17

06 May '17

Hi Ian, I've been finding SimulationTools very useful for looking at the results of my Cactus simulations. I just have a question about the function called "Frequency", which is used in the context of gravitational waveforms. How is this frequency defined? Is it an angular frequency? I don't know how to interpret the negative values I'm getting. Best, Gwyneth

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users May 2017