Users October 2019

users@lists.einsteintoolkit.org

19 participants
28 discussions

problem in analysis thorn
by Fabrizio di giovanni 04 Oct '19

04 Oct '19

Hi all. I'm having a problem, probably in the scheduling, but I'm not able to understand why this is happening. I attached two files .txt where I report the schedule.ccl file of my thorn and the grid parameters in my parfile. What happen is the following; I have these variables defined in the interface.ccl like this: CCTK_REAL masscenter_point type = GF Timelevels = 3 tags='Prolongation="none" tensortypealias="Scalar" ' { masscenterX_point masscenterY_point }"mass center to be used in the global reductions" CCTK_REAL global_masscenter type = SCALAR { masscenterX masscenterY }"mass center coordinates" What I want to do is evaluate in every grid point the values for masscenterX_point and masscenterY_point (I do this in the function my_analysis_point_calc) and then integrate them (using CCTK_Reduce) over the entire domain and save the result in the scalar variables masscenterX and masscenterY (and I do it in the function my_analysis_reduce). So far everything seems to work well, if I plot the masscenter it has reasonable values. The problem happens when, in the same thorn, I want to use the values of masscenterX and masscenterY in another function ( my_analysis_modes_point_calc) to evaluate some other variable (let's call it mode_m1) point by point and then integrate it (in my_analysis_modes_reduce). At the initial time I get a reasonable value for mode_m1, but the second time I print it ( after 12 timesteps) I get nan, the third time I print it (after 24 timesteps) I get a reasonable value and after that it seems to be fine for the rest of the simulation. When I print masscenterX and masscenterY they have reasonable values at every iteration. This is the output: " #1:iteration 2:time 3:data #data columns: 3:masscenterX 4:masscenterY 0 0 2.04233950307922e-07 1.43009709532618e-07 12 0.6 1.64952865679123e-07 1.15302233769628e-07 24 1.2 2.59441645250822e-08 1.8122269479532e-08 #1:iteration 2:time 3:data #data columns: 3:mode_m1 0 0 5.09771568069084e-09 12 0.6 -nan 24 1.2 5.98971400034287e-10 " I thought that the problematic line in my code could be this one: phi = atan((yy-*masscenterY)/(xx-*masscenterX)); if(((xx-*masscenterX)<0.0)){ phi = atan((yy-*masscenterY)/(xx-*masscenterX)) + 3.14159265358979323846; } where I try to use the values of masscenterX and masscenterY inside the function "my_analysis_modes_point_calc". If I change that part not using masscenterX and masscenterY I don't get nans anymore in mode_m1. After doing different tests, I actually found out that if I try to plot scalar output every 4 iterations I get also NaN in masscenter at time 0.2; this is the output in this case: " #1:iteration 2:time 3:data #data columns: 3:masscenterX 4:masscenterY 0 0 2.04233950307922e-07 1.43009709532618e-07 4 0.2 -nan -nan 8 0.4 1.76472702563281e-07 1.23428120452789e-07 12 0.6 1.64952865679123e-07 1.15302233769628e-07 16 0.8 1.05057935424561e-07 7.33797284607643e-08 20 1 5.6677119219615e-08 3.93971526043537e-08 24 1.2 2.59441645250822e-08 1.8122269479532e-08 28 1.4 5.57004919317397e-08 3.90090013497981e-08 32 1.6 1.2454179781868e-08 8.57409404496548e-09 36 1.8 7.81397952778142e-08 5.4397631432516e-08 #1:iteration 2:time 3:data #data columns: 3:mode_m1 0 0 5.09771568069084e-09 4 0.2 -nan 8 0.4 -nan 12 0.6 -nan 16 0.8 2.49105795013804e-09 20 1 -nan 24 1.2 5.98971400034287e-10 28 1.4 3.14686031011036e-10 32 1.6 2.57981958462494e-10 36 1.8 6.79137649228778e-10 " I'm not really sure what is going on here, so I hope somebody can explain me where is the mistake. Thank you. Best, Fabrizio

2 1

meeting minutes for 2019-10-03
by Haas, Roland 03 Oct '19

03 Oct '19

Present: Bill, Chris, Maria, Peter, Roland, Yosef, Zach ET release: * gallery examples: Guiseppe and Helvi are running gallery examples, at least two done. Not sure if will also run BBH example, will cross check with Sam * HMNS still under investigation, Maria in process of testing on local machines * machines: ** mike does currently not compile and link, Peter is looking into this ** OSX/Macports fails to even install openmpi-gcc9 ** failures to compile on bethe and wheeler systems at Caltech * status of tests: http://einsteintoolkit.org/testsuite_results/index.php ** CT_MultiLevel tests fail on machines with more than 1 thread * Fishbone-Moncrief: Zach is addressing reviewer comments * Self-force code: Peter on process of securing missing code pieces for effective source terms, has license and readme file, working on issues with scons with Steve in relation to python 3.5 or higher (needs python 2, README now reflects this). other issues found are being documented on the bitbucket README file ET thorns: * Zach added some code to analysis thorn to not require HydroBase anymore * Zach corrected some comments in thorns Unanswered questions: * one lost email: http://lists.einsteintoolkit.org/pipermail/users/2019-September/007029.html Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 2

Fwd: HDF5 Access coordinates corresponding to data
by Kyle Nelli 03 Oct '19

03 Oct '19

Hello Roland, Disregard my previous email. My postdoc just explained to be that in order to calculate derivatives and such, they extend the space into minus z just a bit. This explains the negative z values. All is good. Thank you for the help! Also, when I submitted my question to the user list, I received an automatic email saying I wasn't part of the mailing list. How would I get on this mailing list to avoid this in the future? Best, Kyle Nelli Student of Engineering Physics University of Illinois Urbana-Champaign Cell: 847-494-5028

1 0

Fwd: HDF5 Access coordinates corresponding to data
by Kyle Nelli 03 Oct '19

03 Oct '19

Roland, Thanks for the reply. I thought about using the origin and delta to reconstruct the Cartesian coordinates however I ran into a problem. These are the attributes of one level: MHD_EVOLVE::Bx it=732928 tl=0 rl=0 c=0 carpet_mglevel 0 cctk_bbox [1 0 1 0 1 0] cctk_nghostzones [4 4 4] delta [2.32433839 2.32433839 2.32433839] group_timelevel 0 iorigin [0 0 0] level 0 name b'MHD_EVOLVE::Bx' origin [-116.21691954 -116.21691954 -9.29735356] time 3327.2904064016993 timestep 732928 As you can see, the origin has a negative z value. Our data is only in the positive z range, and I confirmed this by plotting the mesh in VisIt. This is why I didn't think to use these values. Do you know why this may be? Thank you, Kyle Nelli Student of Engineering Physics University of Illinois Urbana-Champaign Cell: 847-494-5028

1 0

The simulation stops suddenly, problem with restarting from checkpoint, and with number of processors
by Hassan Khalvati 03 Oct '19

03 Oct '19

Dear All, I had a simulation running for nearly 5 days and it stops today with no reason, no errors, and no termination. the first thing I need help with is that I can not find the cause that the simulation has been stopped. The last lines during the simulation have been attached as a text file. The second problem is that I can not restart from the checkpoint. there is an error : ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 Error: job id is negative Aborting Simfactory. I looked up in email archives, and I did what Roland has suggested, to add a line for jobid, (jobid = 999999) in the properties.ini file, but I am still getting errors ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 Warning: job status is U Warning: job status is U Assigned restart id: 1 Warning: Too many used cores per node specified: specified ppn-used=56 (ppn is 28) Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $! Submit finished, job id is 8907 I changed the lines in the properties.ini file for procs, and again getting error ./simfactory/bin/sim submit the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par Assigned restart id: 1 Executing submit command: exec nohup /home/cosmo/simulations/the-last-one/output-0001/SIMFACTORY/SubmitScript < /dev/null > /dev/null 2> /dev/null & echo $! Submit finished, job id is 10517 And finally, I am confused about the option for the "ppn, procs, and ..." numbers in the Simfactory. I have attached my CPU information. It is a double 14 core Xeon(R) CPU E5-2680, with 2 threads per core. my submission command was: ./simfactory/bin/sim create-run the-last-one --parfile=par/bbh-2res-1mass-10sep-final.par --procs=56 --ppn-used=56 but in the properties.ini file, it is mentioned that: numprocs = 4 nodeprocs = 4 numthreads = 14 I have also attached the properties.ini file. Is it using only 4 cores? I looked up in the Simfactory docs, and also ET's wiki. I can not get a clear picture of how the option of the number of processors works. However, with the same command line, I have mentioned above, --procs=56 --ppn-used=56, the simulation was performing well, I want to know if it is using total number of processors on my system or not. I would be grateful if anyone could help me with each of these issues. Attachments are: parameter file, properties.ini, simulation-last-lines, CPU info, and the log.txt file. Sincerely, Hassan -- *Hassan KhalvatiSharif University of Technology, TehranHassan.Khalvati(a)physics.sharif.edu <Hassan.Khalvati(a)physics.sharif.edu>Hassan.kh92(a)gmail.com <Hassan.kh92(a)gmail.com>*

3 3

Warning: Cannot output variable because it has no storage (ADMAnalysis OR ADMMass)
by 刘屿 03 Oct '19

03 Oct '19

Hello Cactus users, I am new to Einstein Toolkit. I want to write a simple parameter file, according to my understanding of Thorn. When I use the Thorn in EinsteinAnalysis, I encountered a warning (Level 1) and CarpetIOBasic can't output normally. I couldn't find the reason for the warning, because the warning message is brief. ------------------------ par file ----------------------- ###################################### # Flesh parameters ###################################### Cactus::terminate = "iteration" Cactus::cctk_itlast = 0 ###################################### # Coordinate ###################################### ActiveThorns = "Carpet CarpetLib CoordBase CartGrid3D" Carpet::domain_from_coordbase = yes CartGrid3D::type = "coordbase" CartGrid3D::domain = "full" CoordBase::domainsize = "minmax" CoordBase::xmin = -20.0 CoordBase::ymin = -20.0 CoordBase::zmin = -20.0 CoordBase::xmax = +20.0 CoordBase::ymax = +20.0 CoordBase::zmax = +20.0 CoordBase::dx = 1 CoordBase::dy = 1 CoordBase::dz = 1 driver::ghost_size = 3 ###################################### # Boundary ###################################### ActiveThorns = "boundary symbase" CoordBase::boundary_size_x_lower = 3 CoordBase::boundary_size_y_lower = 3 CoordBase::boundary_size_z_lower = 3 CoordBase::boundary_size_x_upper = 3 CoordBase::boundary_size_y_upper = 3 CoordBase::boundary_size_z_upper = 3 ###################################### # Initial Condition ###################################### ActiveThorns = "ADMBase" ADMBase::initial_data = "Cartesian Minkowski" ADMBase::initial_lapse = "one" ADMBase::initial_shift = "zero" ADMBase::initial_dtlapse = "zero" ADMBase::initial_dtshift = "zero" ###################################### # Analysis ###################################### ActiveThorns = "ADMMass StaticConformal SpaceMask ADMMacros" ADMMass::ADMMass_surface_distance[0] = 12 ###################################### # IO ###################################### ActiveThorns = "IOUtil CarpetIOBasic CarpetReduce CarpetIOASCII CarpetIOHDF5 CarpetIOScalar" IO::out_dir = $parfile IO::out_every = 1 IOBasic::outInfo_vars = "ADMMass::ADMMass_box" ------------------------ Warning -------------------- WARNING[L1,P0] (CarpetIOBasic): Cannot output variable "ADMMASS::ADMMass_box_x_max" because it has no storage I also want to know the difference between output-0000 and output-0000-active in simulation directory. Thanks, Yu Liu --------------------------- Yu Liu PhD Student School of Physics Huazhong University of Science and Technology

2 1

Re: [Users] [External] Checkpointing with parameter change 8%
by Garrison, David 03 Oct '19

03 Oct '19

Hello, I noticed that the ReadInterpolate Thorn only works with Carpet. Is it possible to get something like it for Pugh? What would be the differences? -DG > On Oct 2, 2019, at 12:33 PM, Haas, Roland <rhaas(a)illinois.edu> wrote: > > Hello DG, > >> Is it possible to restart a checkpointed run with a few parameters >> changed? For example, can you run a simulation with checkpoint, then >> restart it from the last checkpoint using different values of maximum >> and minimum grid range values (as well as different grid spacing) if >> you keep the number of grid points the same? > You can change with some parameters changed. The parameters are the > ones that are marked as "STEERABLE=recover" or "STEERABLE=always" in > their respective thorn's param.ccl files. > > Unfortunately the grid range values and the spacing are not among those > (since they would require that data is re-interpolated onto those). > > There is a thorn ReadInterpolate that can do that. See > https://github.com/rhaas80/ReadInterpolate.git (you can find a number of non-ET thorns of interest collected here: https://docs.einsteintoolkit.org/et-docs/Thorns_we_know_of). > > Please see the test file > https://github.com/rhaas80/ReadInterpolate/blob/master/ReadInterpolate/test… for how to use it (it reads data that were written by the parfile tov_write.parfile). > > Please note that you most likely will see small "jump" in your > simulation since ReadInterpolate acts as an initial data reader so it > will only (at least in the "master" version) set the current timelevel > values but not past ones which can affect values in mesh refinement > boundaries. > > Yours, > Roland > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > _______________________________________________ > Users mailing list > Users(a)einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users

2 1

Checkpointing with parameter change
by Garrison, David 02 Oct '19

02 Oct '19

Is it possible to restart a checkpointed run with a few parameters changed? For example, can you run a simulation with checkpoint, then restart it from the last checkpoint using different values of maximum and minimum grid range values (as well as different grid spacing) if you keep the number of grid points the same? -DG

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users October 2019