Users October 2018

users@lists.einsteintoolkit.org

18 participants
31 discussions

how to speed up ET runs on cluster
by Hamilton, Maria 23 Oct '18

23 Oct '18

Hello, I am using the Marshall University BigGreen cluster to run a parameter file with 5 Carpet time levels, on a 200^3 spatial domain, starting with d(x,y,z)_coarse = 1, and dtfac = 0.03125. The total necessary memory looks to be somewhere around 40 GB. My aim is to run it up to t=200, but I have a time limit on cluster of 72 hours. I was thinking, naively, that if I increase the number of processes, the run will go faster. However, this was not the case. I did an experiment, where I used first 48 procs, then 96 procs, and in 72 hours, the 48 procs run stopped at t = 55.25, while the 96 procs run stopped ash 48.50. That obviously did not work, so I tried next to decrease the time refinement. For the same 48 procs, the run with bigger time step dtfac = 0.0625 stopped ash t = 54.17 after 72 hours. I would very much appreciate if I can get an explanation of what is happening, and how to get to speed up the run, so that it will get finished in the time constraint of 72 hours that is imposed to me. I am attaching the parameter file and the SLURM script I am using to run it. thanks, Maria? _______________________ Dr. Maria C. Babiuc Hamilton Department of Physics Marshall University S257 Science Building Huntington, WV, 25755 Phone: (304)696-2754

2 1

fftw and MKL
by Yosef Zlochower 22 Oct '18

22 Oct '18

Hi, I got bitten by an issue regarding the link order. I use fftw3 for a series of transforms of a type not supported by the MKL fftw wrapper (r2r transforms). I have FFTW3 either built, or use a prebuilt version. The issue comes up if mkl precedes fftw3 on the link line. If it does, then rather than using fftw functions, mkl functions are used instead (and these always fail). See r2r transforms here: https://software.intel.com/en-us/mkl-developer-reference-c-using-fftw3-wrap… Since MKL does not support the functions I need, is there a way to force -mkl to always occur only after -lfftw3 on the link line? -- Dr. Yosef Zlochower Center for Computational Relativity and Gravitation Associate Professor School of Mathematical Sciences Rochester Institute of Technology 85 Lomb Memorial Drive Rochester, NY 14623 Office:74-2067 Phone: +1 585-475-6103 yosef(a)astro.rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

2 1

meeting minutes for 2018-10-22
by Roland Haas 22 Oct '18

22 Oct '18

Present: Roland, Chris, Helvi, Sam, Steve, Zach, Yosef, Peter, Bill, * update ET to LORENE2 ** need to check that files can still be read ** old ticket stated that there would be an abort for some files writen by old an read by new LORENE or vice versa * failing tests: ** comet and cori seem like a (temporary) machine issue ** lost failing runs on OSX VMs * WVU_Diagnostics ** Zach back on working on this ** produce small LORENE file for tests * Proca / Lean ** Helvi and Miguel are back to working on it * Cactus infrastructure and single points of failure ** currently have too many scattered pieces of infrastructure which are locally mantained ** moving to bitbucket / github / whatnot would help with this ** should present the whole set of infrastructure to the ET users / maintainers ** wiki page for infrastructure: https://docs.einsteintoolkit.org/et-docs/Services ** Roland to update information on VMs for macs ** Roland to add information about bitbucket repos ** need to add information about Jenkins setup C++11 updates for Cactus: * most machines ok, wait for mike and qb update * Zach may have update for tianhe1a * vesta is a BG/Q at Argonne that Roland has to look into. Tutorial server: * still down * Roland will ask to have it moved to Nebula Next week will be Peter hosting. Europe changes daylight savings time next week. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

HDF5 data analysis
by 林家暉 19 Oct '18

19 Oct '18

Hello, I am trying to analyze 2D output data (.h5) from ET. I want to get a 2D array with uniform grid.However HDF5 with AMR contains many layers and each layer breaks into many components. When I read out the HDF5 data , I do not know where each component should be. Is there some suggested way to map HDF5 data into a 2D array ? Thank you. Best regards, Chia-Hui

2 2

Re: [Users] [ET Trac] [Einstein Toolkit] #2207: support all of C++11 in Cactus or the Einstein Toolkit
by Erik Schnetter 16 Oct '18

16 Oct '18

On Oct 15, 2018, at 23:13 , Einstein Toolkit <trac-noreply(a)einsteintoolkit.org> wrote: > > #2207: support all of C++11 in Cactus or the Einstein Toolkit > --------------------------+--------------------------------- > Reporter: Roland Haas | Owner: (none) > Type: enhancement | Status: new > Priority: minor | Milestone: > Component: Cactus | Version: development version > Resolution: | Keywords: > --------------------------+--------------------------------- > > Comment (by Roland Haas): > > I had a look at the (non-personal machines) in simfactory2. > > === unknown status === > * angel.ini This machine is very old; newer Compute Canada systems are available. > * compute1.ini > * compute20.ini > * compute2.ini > * compute3.ini > * compute4.ini > * compute5.ini > * compute.ini > * nvidia1.ini > * nvidia.ini > * pi0005009.ini > * titan.ini These are machines at Perimeter. C++11 is available there, but the configurations might not have been updated. These machines are scheduled to be retired within a month. -erik -- Erik Schnetter <schnetter(a)protonmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

meeting minutes for 2018-10-15
by Roland Haas 15 Oct '18

15 Oct '18

Present: Bhavesh, Roland, Peter, Sam, Antoni Ramos Buades, Bill, Yosef * accumulated changes to Carpet were applied over the weekend * change to remove MPI_UB tag from Carpet helps compile with some versions of OpenMPI that remove the deprecated constant * Roland will propose to support all of C++11 in Cactus / Einstein Toolkit. Will collect information on feature matrix of different Intel, GNU and clang compilers (icpc 15 and g++ 4.8 seem to be required for all the features we may want). Have to check simfactory files that all clusters either already use a new enough compiler or provide it and it can be used with the ET Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

error message MPI
by Hamilton, Maria 15 Oct '18

15 Oct '18

Hello, I'm trying to compile the development Cactus version on my Mac laptop after an update: $repos/CRL/GetComponents --update. Unfortunately, I ran into the following error: ******************************************************************************** Running configuration script for thorn MPI: Found MPI compiler wrapper at /usr/local/bin/mpic++! sh: line 1: 30205 Abort trap: 6 /usr/local/bin/mpic++ -compile_info 2> /dev/null sh: line 1: 30207 Abort trap: 6 /usr/local/bin/mpic++ --showme 2> /dev/null ERROR: MPI could not be configured: neither automatic nor manual configuration succeeded CST error 1: -> Configuration script for thorn MPI returned exit code 5 Error message: 'MPI could not be configured: neither automatic nor manual configuration succeeded' Finished running configuration script for thorn MPI. ******************************************************************************** This persists after real clean, and even after I erase the configuration and reconfigure it. Please help, Maria ? _______________________ Dr. Maria C. Babiuc Hamilton Department of Physics Marshall University S257 Science Building Huntington, WV, 25755 Phone: (304)696-2754

3 16

Re: [Users] separation of simulation domain
by 林家暉 11 Oct '18

11 Oct '18

Dear Erik, Thanks for your reply.It is conceptually helpful. I think I would use other codes to deal with the exterior after the interior is simulated by ET like you said. Best regards, Chia-Hui ________________________________ 寄件者: Erik Schnetter <schnetter(a)protonmail.com> 寄件日期: 2018年10月11日下午 09:14:49 收件者: 林家暉副本: Einstein Toolkit Users 主旨: Re: [Users] separation of simulation domain Chia-Hui I think you are asking a question about physics, not about the Einstein Toolkit. To simulate a system, you need to have a mathematical description to start with. If you want to combine a general relativistic interior with a Newtonian (?) exterior, then you first need to come up with a physically consistent description, for example a consistent set of PDEs with respective interface conditions. I believe this is very difficult. At the least this is a mathematical question, not one about software engineering. What we have been doing in the past is to simple use general relativity everywhere, even in the exterior. If you use adaptive mesh refinement (AMR), then you can have a central region where small length and time scales are resolve, and an exterior region that is much larger and where only coarser length and time scales are resolved. As Roland mentions, using approximately spherical coordinates in the exterior via Llama allows you to have a constant angular resolution there, which is convenient for tracking gravitational waves generated by a point source. If you need to resolve hours and days, then in general the following approximations hold: - from the point of view of the interior, the exterior is constant in time - from the point of view of the exterior, only the temporally averaged interior is relevant People would thus first perform a calculation of the interior, and once this has reached a "steady state", use this to motivate certain inner boundary conditions for a second simulation that treats only the exterior. -erik > On Oct 11, 2018, at 0:29 , 林家暉 <r06222015(a)ntu.edu.tw> wrote: > > Hello, > I have a question about whether it is possible to separate the domain of the simulation. I want to simulate with a larger scale such as (1 million km )^3 , however it is too computation-consuming to simulate the whole domain with GR and it is not necessary to consider GR because GR could be ignored in such large scale. But in the region near center with high density , GR is important . > Therefore I am wondering if it is possible that GR is included in the center region ,say (1000 km)^3, but not included in the other region. However another problem is that timescale is also very different. ET's time scale is about ~ ms but what I need is around hours to days. Is there some suggestion for me to deal with it ? > Thanks, > Best regards, > Chia-Hui -- Erik Schnetter <schnetter(a)protonmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

1 0

separation of simulation domain
by 林家暉 11 Oct '18

11 Oct '18

Hello, I have a question about whether it is possible to separate the domain of the simulation. I want to simulate with a larger scale such as (1 million km )^3 , however it is too computation-consuming to simulate the whole domain with GR and it is not necessary to consider GR because GR could be ignored in such large scale. But in the region near center with high density , GR is important . Therefore I am wondering if it is possible that GR is included in the center region ,say (1000 km)^3, but not included in the other region. However another problem is that timescale is also very different. ET's time scale is about ~ ms but what I need is around hours to days. Is there some suggestion for me to deal with it ? Thanks, Best regards, Chia-Hui

2 2

Postdoctoral Position at PSU
by David Radice 09 Oct '18

09 Oct '18

Dear all: A postdoctoral position has also been opened at PSU. Please find the job announcement below. Best, David Radice ------------------------------------ Post-Doctoral Scholar Position in Multimessenger Astrophysics The Department of Physics at The Pennsylvania State University is seeking to fill one or more Postdoctoral Scholar positions in multimessenger astrophysics with gravitational waves. The successful candidate will participate in the research program led by David Radice on the numerical and analytical modeling of compact binary mergers in general relativity. There will also be ample opportunity for collaboration with other faculty members at the Institute for Gravitation and the Cosmos, which includes, among others, Abhay Ashtekar, Eugenio Bianchi, Chad Hanna, Sarah Shandera, and B.S. Sathyaprakash. These positions require a Ph.D. in physics, astronomy, or a closely related field. Applications must be submitted online at https://psu.jobs/job/83305 and include a cover letter, CV, and a statement of research interests. Applicants should arrange for three recommendation letters to be submitted to institute(a)gravity.psu.edu indicating the appropriate job number in the subject line. Applications received before December 1, 2018 will be given full consideration. The expected start date is September 1, 2019, although alternative start dates will also be considered. For more information, please visit http://gravity.psu.edu. This is a fixed-term appointment funded for one year from date of hire with excellent possibility of re-funding. To review the Annual Security Report which contains information about crime statistics and other safety and security matters and policies, please go to https://police.psu.edu/annual-security-reports, which will also explain how to request a paper copy of the Annual Security Report. Penn State is an equal opportunity, affirmative action employer, and is committed to providing employment opportunities to all qualified applicants without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability or protected veteran status.

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2018