Users March 2019

users@lists.einsteintoolkit.org

16 participants
22 discussions

AMD Ryzen Threadripper 2990WX
by Ferguson, Deborah L 27 Mar '19

27 Mar '19

Hey all, I’m building a new work machine and was trying to determine what processor to get to best be able to run Einstein toolkit on it. I’d been looking at the 2990WX because of the number of cores. I wanted to check though if anyone had used it before and encountered any issues before I spend money ordering one. I know there are other processors that are faster per core though so if anyone thinks a different processor would do better running ETK, I’d welcome recommendations. Thanks! Deborah Get Outlook for iOS<https://aka.ms/o0ukef>

1 0

Reminder: First LISA Waveform Working Group Workshop
by Witek, Helvi 25 Mar '19

25 Mar '19

Dear all, This is a reminder that funding is available to support student travel to the Waveform Working Group (WavWG) meeting that will take place at the AEI, Potsdam from the 13th - 15th May 2019. To apply for this financial support please send an email to wav-wg-chairs(a)lisamission.org<mailto:wav-wg-chairs@lisamission.org>, including a (short!) motivation and letter of support from your PhD supervisor before April 1, 2019. Let us also take this opportunity to also encourage those of you thinking of attending to register on the meeting website at https://workshops.aei.mpg.de/lisawavwg/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fworkshops…>. Note that whilst the WavWG is part of the LISA Consortium, you do not need to be in the Consortium to attend. Regards, Deirdre Shoemaker, Maarten van de Meent, Niels Warburton, Helvi Witek (Co-chairs LISA Waveform Working Group) =========================================== Dr. Helvi Witek Royal Society University Research Fellow Theoretical Particle Physics and Cosmology Department of Physics King's College London ===========================================

1 0

Re: [Users] weird behavior with some ET simulations
by Antoni Ramos Buades 21 Mar '19

21 Mar '19

Hi, we are using a version of the EinsteinToolkit of 2017, particularly, !DEFINE ET_RELEASE = ET_2017_06 so it is not one of the newest versions, but as far as I know in the last releases there has not been updates on the Llama thorn. Although we can try to run with older or newer versions if you think it can help to solve the problem. Best Regards, Toni. El mié., 6 mar. 2019 a las 18:29, Zach Etienne (<zachetie(a)gmail.com>) escribió: > Hi Antoni, > > This is a worrisome finding, thanks for sharing. It would be useful if you > would let us know what version of the ET you are using. Further, if you > could attempt the simulation with earlier versions of the ET (e.g., the > first version with Llama included), it would help to determine if this > issue arose due to some recent commit. > > -Zach > > * * * > Zachariah Etienne > Assistant Professor of Mathematics > West Virginia University > *https://math.wvu.edu/~zetienne/ <https://math.wvu.edu/~zetienne/>* > https://blackholesathome.net > > > On Wed, Mar 6, 2019 at 9:40 AM Antoni Ramos Buades < > antoniramosbuades(a)gmail.com> wrote: > >> Hi, >> >> my name is Antoni Ramos Buades, I am PhD student in the University of the >> Balearic Islands working with Sascha Husa. We have been using ET with the >> Llama thorn to produce waveforms. We have observed in some simulations >> strange behaviors in the waveforms. For example, I have attached a plot of >> the amplitude of the psi4 of a non-spinning q1 simulation, for different >> higher order modes, where you can see that the 22, 32 and 44 show weird >> spikes. We have rerun this simulation outputting 2D data in the x-y plane. >> We have plotted the Hamiltonian constraint and made a movie (I cannot >> attach the movie because its size is 42Mb ) but I can attach some snapshots >> where you can observe that there seems to be an outburst of noise when the >> boxes are aligned, and this continues happening in the simulation each time >> the boxes are aligned with each other. Geraint Pratten is also >> investigating this issue adding some new parameters in the parameter file >> (attached to the email) for the carpet mesh refinement like >> freeze_unaligned_parent_levels, freeze_unaligned_levels, ... and running >> some tests. >> >> We would like to ask you if somebidy has ever seen this thing happening >> before and if somebody knows how one could solve this problem, because with >> the same Carpet settings we have observed that for some configurations >> (mass ratio and spins) this does not happen, but for others, like the one >> presented in this email we do. I have additional quantities as a 2D output >> like the curvature, metric, horizon, ... in case you think they could be >> used to check something. >> >> Thanks and best regards, >> Toni. >> _______________________________________________ >> Users mailing list >> Users(a)einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users >> >

7 12

meeting minutes for 2019-03-21
by Haas, Roland 21 Mar '19

21 Mar '19

Present: Roland, Antoni, Bill, Peter, Sam, Yosef ET release * will release in March * have firm commitment on thorns to be included by end of this week (Friday) release Friday next week BNS gallery example * no updates WVU_Diagnostics status * https://bitbucket.org/einsteintoolkit/tickets/issues/1974/add-wvuthorns_dia… * will ask for firm statement by Steve and Zach * Peter will look at code as well Proca status * ready to include * SIMD issue will be commented out for release, fixed afterwards * has documenation Lean status * Erik is reviewer, made comments * not clear if addressed * will contact Helvi, Miguel, Erik for firm statement of inclusion * NPScalars/teukolsky test failing on some cluster ET testsuite status http://einsteintoolkit.org/testsuite_results/index.php * failure on homebre osx not understood * on one on call uses this compbination and could test * will document failure then fix after release ET tutorial server * new server is in place and ready for testing https://etkhub.ndslabs.org/ Python 2 end of life in 2020 * changes: ** print statement ** divides are floating point by default ** byte strings vs ascii strings and encodings, Bill has experience in this ** iterators and list comprehension behave differently ** could start with 2to3 python converter * best for users would be to support both versions * could propose as summer REU project for good student at LSU * collect census of current cluster support for python 3 and python 2 One unanswered question in the ET users list: * Mar 21 06:21:06 CDT 2019 [Users] GlobalDerivative in spherical (multipatch) coordinates (Severin Frank): http://lists.einsteintoolkit.org/pipermail/users/2019-March/006809.html ET US meeting * now inviting speakers, will send official schedule once done * date and place are fixed and already mentioned on the site: https://ccrg.rit.edu/content/events/2019-06-17/north-american-einstein-tool… Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

einsteintoolkit docker organization
by Haas, Roland 18 Mar '19

18 Mar '19

Hello all, there seems to be an einsteintoolkit organization on docker.com: https://hub.docker.com/u/einsteintoolkit I am assuming that is us and could be used to eg hold the docker image for the ET tutorial server. However I do not know who is the administrator of the organization. Does anyone know who that might be? Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

3 3

Conservative to primative conversion fails in TOV collapse
by Shoup, Anthony 16 Mar '19

16 Mar '19

Hi All, I have been working on reproducing the results in the paper: The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics (arXiv: 1111.3344v1 {gr-qc] 14 Nov 2011) and others as a way to teach myself how to use the Einstein Toolkit and to test out my home-built cluster computer. I am currently working on the non-rotating TOV neutron star collapse into a BH. It works well up until it fails trying to convert conserved variables into primitive variables in grhydro (see attached tovCollapse.out file). I have tried adjusting the atmosphere parameters (specifically GRHydro::rho_abs_min = 1.e-8, up from 1.e-10)., as per the suggestions in the grhydro documentation, which has helped, but I think the conversion is failing now very near where the AH will/is forming. I have included my parameter file for the simulation (I lowered evolve K from 98 to 80 just to accelerate the collapse process). Any help anyone can give would be greatly appreciated. Sincerely, Tony... [https://email.osu.edu/owa/attachment.ashx?id=RgAAAAAb%2fHy0wVvTSoHQx8OJXAaL…] Anthony Shoup PhD, Senior Lecturer College of Arts & Sciences, College of Engineering Departments of Physics, Astronomy, EEIC 315 Science Bldg. | 4250 Campus Dr. Lima, OH 45807 419-995-8018 Office | 419-516-2257 Mobile shoup.31(a)osu.edu<https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hl…> osu.edu<https://email.osu.edu/owa/redir.aspx?C=j5WpnJiBk0W5oVlCbtvB-xiCkA_lbdEIi9hl…>

1 1

meeting minutes for 2019-03-14
by Haas, Roland 14 Mar '19

14 Mar '19

Present: Roland, Bill, Chris, Peter, Sam, Steve, Yosef, Wilke van der Schnee, Zach piecewise polytrope with gallery example: * had issues with complex grid structure, caused by Trigger thorn creating the third refined box * Roland suggests that he has seen the Hydro_Analysis_rho_max_origin_distance value being set to 1e-13 or so for a single point before it recovers to "sane" values. This could have triggered the thorn * it should be possible to change the Trigger thorn parameters during a restart to only enable them later in the run * Roland also suggests to try and run with magnetic field evolution turned on for a zero magnetic field, this seems to help since the MHD con2prim routine seems more robust than the non-MHD one WVU_Diagnostics: * Zach provided tests, Jenkins updater needed to be updated * Steve has suggestions about code Lean: * no review yet, needs to be reviewed etc in the very near future or must be dropped (or release postponed) ET release: * testsuite failures we worry about: bluewaters, golub (number of them), OSX failures * missing machines: queenbee, smic * currently testing with intel compilers, gcc compilers. Peter will test PGI on smic. * tickets: please review the release relevant ones, https://bitbucket.org/einsteintoolkit/tickets/issues?milestone=ET_2019_02&m… tutorial server: * make production ready and test out * try out CILogin, if it does not work revert to github ET meeting at RIT: * need to finalize list of speakers, currently more speakers than slots * calling for volunteers for the tutorial session. Typically Steve is handling this * tutorials should target advanced users as all users are expected to have used the ET before * will leverage another conference at RIT to get some more speakers Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Re: [Users] Synchronisation and boundary prolongation in NSNS gallery example with slypp EOS
by Erik Schnetter 12 Mar '19

12 Mar '19

Wilke If you use symmetry_rotating_180, then there cannot be any refinement boundaries that are too close to the symmetry boundary, hence there will be slightly larger fine grids. -erik On Tue, Mar 12, 2019 at 1:46 PM Wilke van der Schee < wilke.van.der.schee(a)cern.ch> wrote: > Dear Erik, > > Thanks for the reply, such might indeed be the case. We use the same > settings as in the gallery, i.e. a gridsize of 396 with dx=18 and then 7 > levels of refinement. Could the symmetry also be of influence here? > (both reflection_z and symmetry_rotating180 are used). Looking at the grid > (it was attached as visit0000.png) it looks quite fine to me, but I don't > have that much experience. > > Thanks again, best, > > Wilke > > On Tue, 12 Mar 2019 at 18:38, Erik Schnetter <schnetter(a)cct.lsu.edu> > wrote: > >> On Tue, Mar 12, 2019 at 11:26 AM Wilke van der Schee < >> wilke.van.der.schee(a)cern.ch> wrote: >> >>> Dear users, >>> >>> We have been trying for a while to get neutron star mergers for >>> different equations of state, with only partial success. While the gallery >>> example with the polytrope runs fine, we already encounter problems with >>> the SLy_tabulated EOS. (we're using the current release, ET_2018_09) >>> >>> One particularly puzzling error we don't understand is the following >>> (full log+par file attached) >>> >>> "INFO (NSTracker): Found star at (-10.125,-5.90625,0) >>> 21256 373.641 | 52.3035497 | 0.0001505 0.0053630 | >>> 0.0014340 | 1.0717264 | 0.0019186 | 3591 >>> INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 >>> INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 >>> WARNING level 1 from host n0086.compute.hpc process 0 >>> while executing schedule bin (none), routine (no thorn)::(no routine) >>> in thorn CarpetLib, file >>> /hpc/local/CentOS7/uu_itp_strings/Cactus/configs/sim/build/CarpetLib/dh.cc:161: >>> -> >>> >>> /hpc/local/CentOS7/uu_itp_strings/Cactus/configs/sim/build/CarpetLib/dh.cc:980: >>> [ml=0 rl=6 c=0] The following grid structure consistency check failed: >>> Synchronisation and boundary prolongation: All points must have been >>> received >>> needrecv.empty() >>> >> >> This is an internal error in Carpet. Most of the time, this indicates >> another error that wasn't caught properly before the details of the grid >> structure are generated. What resolution are you using? If you are using a >> resolution that is too coarse, then the refined regions don't fit into the >> simulation domain, and this could lead to such an error. >> >> -erik >> >> >> >> ml=0 rl=6 >>> baseextent=([756,504,756]:[6408,11784,6408]:[4,4,4]/[189,126,189]:[1602,2946,1602]/[1414,2821,1414]/5640296116)" >>> (shortened log attached, full log here: >>> https://www.dropbox.com/s/avgrshl8chhzp6i/atmostr025full.out?dl=0) >>> >>> and then much more information about the grid structure (see full log). >>> Indeed when looking into visit at this time the grid structure is a bit >>> strange (also attached, at t=373.5, just before the crash). It containts >>> the 3rd central mesh already, which is not really supposed to be triggered, >>> but on the other hand that by itself should not lead to this crash (and I'm >>> not quite sure why the trigger was activated at t=180). >>> >>> The main difference of the parameter file wrt the gallery is of course >>> the equation of state and hence the initial Lorene file. We also needed to >>> go to dtfac=0.25 (otherwise even earlier con2prim errors appear), and for >>> testing and stability purposes we now use an atmosphere density of 10^{-9}. >>> >>> Any ideas how to look for more information or on similar problems would >>> be very welcome, we do have more output files of course. Looking at the >>> source code didn't help me much for now. >>> >>> Many thanks in advance for suggestions, >>> >>> Wilke van der Schee (post-doc Utrecht University) >>> _______________________________________________ >>> Users mailing list >>> Users(a)einsteintoolkit.org >>> http://lists.einsteintoolkit.org/mailman/listinfo/users >>> >> >> >> -- >> Erik Schnetter <schnetter(a)cct.lsu.edu> >> http://www.perimeterinstitute.ca/personal/eschnetter/ >> >> -- Erik Schnetter <schnetter(a)cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

Synchronisation and boundary prolongation in NSNS gallery example with slypp EOS
by Wilke van der Schee 12 Mar '19

12 Mar '19

Dear users, We have been trying for a while to get neutron star mergers for different equations of state, with only partial success. While the gallery example with the polytrope runs fine, we already encounter problems with the SLy_tabulated EOS. (we're using the current release, ET_2018_09) One particularly puzzling error we don't understand is the following (full log+par file attached) "INFO (NSTracker): Found star at (-10.125,-5.90625,0) 21256 373.641 | 52.3035497 | 0.0001505 0.0053630 | 0.0014340 | 1.0717264 | 0.0019186 | 3591 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 WARNING level 1 from host n0086.compute.hpc process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn CarpetLib, file /hpc/local/CentOS7/uu_itp_strings/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /hpc/local/CentOS7/uu_itp_strings/Cactus/configs/sim/build/CarpetLib/dh.cc:980: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ml=0 rl=6 baseextent=([756,504,756]:[6408,11784,6408]:[4,4,4]/[189,126,189]:[1602,2946,1602]/[1414,2821,1414]/5640296116)" (shortened log attached, full log here: https://www.dropbox.com/s/avgrshl8chhzp6i/atmostr025full.out?dl=0) and then much more information about the grid structure (see full log). Indeed when looking into visit at this time the grid structure is a bit strange (also attached, at t=373.5, just before the crash). It containts the 3rd central mesh already, which is not really supposed to be triggered, but on the other hand that by itself should not lead to this crash (and I'm not quite sure why the trigger was activated at t=180). The main difference of the parameter file wrt the gallery is of course the equation of state and hence the initial Lorene file. We also needed to go to dtfac=0.25 (otherwise even earlier con2prim errors appear), and for testing and stability purposes we now use an atmosphere density of 10^{-9}. Any ideas how to look for more information or on similar problems would be very welcome, we do have more output files of course. Looking at the source code didn't help me much for now. Many thanks in advance for suggestions, Wilke van der Schee (post-doc Utrecht University)

2 1

meeting minutes for 2019-03-07
by Haas, Roland 07 Mar '19

07 Mar '19

Present: Roland, Antoni, Bill, Peter, Steve, Yosef noise bursts showing up in BBH simulations (http://lists.einsteintoolkit.org/pipermail/users/2019-March/006783.html) * shows up in higher order modes mostly * will try suggestions by Zach, Ian, Roland. In particular min_fraction=1.0 ET tutorial server * using github is ready to go * Roland suggests using CILogon instead which would allow for a blanked permission for "well known" Universities. This seen as dangerous. * alternative signup form at https://ekohaes8.ncsa.illinois.edu:4443/hub/login may be useful to avoid signings without authorization ET release testing: * Roland has code compile on: - golub - cori - stampede2-skx - stampede2-knl - wheeler - osx-homebrew - osx-macports - bethe - comet * will work on starting the actual testsuites as soon as possible ET ticket emails: * proof of concept setup is available on the ET server in hooks/issue_changed.php * Steve and Roland to make it fully functional Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users March 2019