Users

users@lists.einsteintoolkit.org

3 participants
3984 discussions

problems about accretion disk of binary neutron star
by 黃世杰 08 Nov '18

08 Nov '18

Dear all, I want to add energy into the accretion disk of the binary neutron star merger in gallery code(https://einsteintoolkit.org/gallery/bns/index.html)and calculate the accretion rate (dM/dt) do I need to add some other thorn into the simulation, or could just modify the parameter file of the gallery code? Thank you Gratefully, Jie

2 2

Trac
by Steven R. Brandt 08 Nov '18

08 Nov '18

We have a solution for our Trac woes--and Christmas isn't even here yet. If you wish to have access to Trac, and currently do not have such, please email Richard Raether, CC'd above, and he will create an external user CCT account for you. With this account, you will be able to login to Trac, create tickets, and otherwise have fun. --Steve

2 1

cctk_alignment
by Steven R. Brandt 07 Nov '18

07 Nov '18

I'm debugging a modification to Cactus, and something I've done has tripped an assertion error when running the RK2-central.par test. The message is: cactus_sim: /home/sbrandt/et-tester/Cactus/configs/sim/build/TestMoL/Init.c:40: TestMoL_InitVars: Assertion `cctki0_ialn>0' failed. The variable cctki0_ialn seems to get filled in from cctkGH->cctk_alignment, which is (as far as I can see) only set in mode.cc by carpet. The test in question, however, uses PUGH. Valgrind confirms that just before the assertion error, I am tripping a conditional depends on uninitialized value error. However, I don't see this same problem when running with the unmodified Cactus source--so somehow I've broken something relating to this infrastructure. How and where is cctk_alignment supposed to get set when running PUGH? What should its value be? Thanks. --Steve

2 2

Hydro_RNSID Inherits from GRHydro
by Zach Etienne 05 Nov '18

05 Nov '18

Hello, Hydro_RNSID inherits from GRHydro, which goes against the usual policy that initial data thorns depend only on HydroBase / ADMBase. Is there a plan to fix this? Is a fix even difficult? -Zach * * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University *https://math.wvu.edu/~zetienne/ <https://math.wvu.edu/~zetienne/>* https://blackholesathome.net

3 2

meeting minutes for 2018-11-05
by Roland Haas 05 Nov '18

05 Nov '18

Present: Roland, Antoni, Bill, Bruno, Chris, Federico Cipoletta, Hassan Khalvati, Steve, Zach LORENE2: * switch thornlist to have LORENE2 as the default WVUThorns: * postpone to after SC BNS gallery example * postpone to after SC Possible Carpet memory leak * Steve looked at this with Valgrind * Roland pointed out that Carpet does not (as far as he knows) Trac to bitbucket transfer: * can transfer user name properly * still stuck with creating many tickets (ticket 13 is the one that gets stuck) Restmass computation * Federico added capabilities to Hydro_Analaysis to compute rest mass * tested with a TOV star and GRHydro and IllinoisGRMHD * asked to create pull request on bitbucket Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Problem with reproducing the NsNsToHMNS
by hassan.khalvati＠physics.sharif.edu 29 Oct '18

29 Oct '18

Dear all, I need some guidance on reproducing the NsNsToHMNS which is provided in the gallery section of ET's website. but there are a bunch of warnings during the simulation. I would be grateful if anyone could possibly tell me what is the minimum limit on core numbers for the NsNs and also check the warnings below. I tried with 4 procs at first , but now I am trying on 28 procs. but there are same warnings. I don't know if I have to take them serious or not ! Because even with these warnings, the simulation is still running... Gratefully, Hassan *****Here are all warnings***** : Warning: thorn CoordBase already scheduled for activation WARNING level 3 from host compute-0-46.local process 0 while executing schedule bin (none), routine (no thorn)::(no routine) in thorn IOUtil, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/IOUtil/CheckpointRecovery.cc:730: -> No HDF5 checkpoint files with basefilename 'checkpoint.chkpt' found in recovery directory 'NsNs' Formaline::Formaline_RegisterWarnings: [meta] Register to receive warnings and info messages from the flesh WARNING level 3 from host compute-0-46.local process 0 while executing schedule bin CCTK_STARTUP, routine TerminationTrigger::TerminationTrigger_StartSignalHandler in thorn TerminationTrigger, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/TerminationTrigger/signal.c:64: -> Could not register parameter change monitor for 'signal_numbers' INFO (Vectors): Using vector size 2 for architecture SSE2 (64-bit precision) INFO (Formaline): Writing tarballs with the Cactus sources into the directory "NsNs/cactus-source" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:15: -> Forcing ML_BSSN::initial_boundary_condition="extrapolate-gammas" because ML_BSSN::my_initial_boundary_condition="extrapolate-gammas" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:28: -> Forcing ML_BSSN::rhs_boundary_condition="NewRad" because ML_BSSN::my_rhs_boundary_condition="NewRad" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:47: -> Forcing ML_BSSN::epsDiss=0.0 because ML_BSSN::apply_dissipation="never" WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:116: -> Forcing ML_BSSN::advectLapse=1 because ML_BSSN::LapseAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_WRAGH, routine ML_BSSN_Helper::ML_BSSN_ParamCompat in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:134: -> Forcing ML_BSSN::advectShift=1 because ML_BSSN::ShiftAdvectionCoeff=1.0 INFO (MoL): Using Runge-Kutta 4 as the time integrator. INFO (SymBase): Symmetry on lower x-face: rotating_symmetry_180 INFO (SymBase): Symmetry on lower z-face: reflection_symmetry INFO (MoL): The maximum number of evolved variables is 1250. 30 are registered. INFO (MoL): The maximum number of slow evolved variables is 1250. 0 are registered. INFO (MoL): The maximum number of constrained variables is 1250. 37 are registered. INFO (MoL): The maximum number of SandR variables is 1250. 0 are registered. INFO (MoL): The maximum number of evolved array variables is 1250. 0 are registered. INFO (MoL): The maximum number of constrained array variables is 1250. 0 are registered. INFO (MoL): The maximum number of SandR array variables is 1250. 0 are registered. INFO (MoL): The maximum size of any array variables is 0. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 0 WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:150: -> Parameter ML_BSSN::my_initial_data is outdated; please update the parameter file. Do not use this parameter, and set up initial conditions via ADMBase as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:157: -> Parameter ML_BSSN::my_initial_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up initial boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:164: -> Parameter ML_BSSN::my_rhs_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:171: -> Parameter ML_BSSN::my_boundary_condition is outdated; please update the parameter file. Do not use this parameter, and set up RHS boundary conditions as usual. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:198: -> Parameter ML_BSSN::LapseAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectLapse. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin CCTK_PARAMCHECK, routine ML_BSSN_Helper::ML_BSSN_ParamCheck in thorn ML_BSSN_Helper, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/ML_BSSN_Helper/Parameters.c:204: -> Parameter ML_BSSN::ShiftAdvectionCoeff is outdated; please update the parameter file. Instead of using this parameter, you should set ML_BSSN::advectShift. WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). WARNING level 1 from host compute-0-46.local process 1 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). WARNING level 1 from host compute-0-46.local process 0 while executing schedule bin Hydro_Analysis_LocationSearch, routine Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine in thorn Hydro_Analysis, file /home/hassan.khalvati.physics.sharif/NsNs/Cactus/configs/sim/build/Hydro_Analysis/Hydro_Analysis.c:199: -> Found more than one (2) identical maximum, using average location (15.1875,0,0). INFO (NSTracker): Found star at (15.1875,0,0) INFO (CarpetIOHDF5): --------------------------------------------------------- INFO (CarpetIOHDF5): Dumping initial checkpoint at iteration 0, simulation time 0 INFO (CarpetIOHDF5): --------------------------------------------------------- WARNING level 2 from host compute-0-46.local process 0 while executing schedule bin CCTK_ANALYSIS, routine QuasiLocalMeasures::qlm_calculate in thorn QuasiLocalMeasures, file qlm_calculate.F90:130: -> There were not enough past time levels available for accurate calculations INFO (TimerReport): Timer Report at iteration 0 time 0 INFO (TimerReport): Top timers at iteration 0 time 0 ----------------------------------- Hassan Khalvati Master Student Department of Physics Sharif University of Technology Tehran, Azadi Hassan.khalvati(a)physics.sharif.edu

3 5

meeting minutes for 2018-10-29
by Roland Haas 29 Oct '18

29 Oct '18

Present: Bill, Peter, Sam, Roland, Zach BNS gallery example: * no further progresss, should have time two weeks from now ET infrasstructure: * will have presentation for Helvi, Bill, Zach on how the pieces of ET infrastructure fit together * Zach is planning to have a longer term project of more automated code generation from NRPy+ code http://blackholesathome.net C++11 support: * Zach states that intel 15 exists on tianhe, currently intel 14 in use The US will not yet have changed times next week. Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

1 0

Question about GW150914.rpar simulation
by Benjamin Chardi Marco 24 Oct '18

24 Oct '18

Dear friends, We are trying to use the EinsteinToolKit GW150914.rpar binary balckhole merge simulation as use case to test that our container orchestration product OpenShift can be used for HPC. Our test environment only has 30 CPUs so we need to execute that simulation in a reasonable time. Please can you tell us how to modify GW150914.rpar in order to get a less precise simulation executed in a 30 CPUs cluster in a reasonable time (~ few days). Now we can run the simulation GW150914.rpar using OpenMPI + EinsteinToolKit, but it takes so long to be executed (~ weeks). We believe that GW150914.rpar EinsteinToolKit is a great use case to test OpenShift for HPC, and of course we will reference to EinsteinToolKit is our final report as a use case for Openshift in HPC mode. Many thanks in advance for your help, Benja -- Benjamín Chardí Marco Senior Red Hat Consultant RHCE #100-107-341 bchardim(a)redhat.com Mobile: 0034 654 344 878

3 4

Re: [Users] Question about GW150914.rpar simulation
by Roland Haas 24 Oct '18

24 Oct '18

Hello Benja, you will still see both black holes. Half this size is really "reduce the radius of the sphere by half" and not showing only one half of the domain. Eg the original rpar file may have had a domain that goes out to a radius of 2000M while the new one goes only to 1000M. The black holes are never further apart than about 20M or so though so they are always included. Yours, Roland > Hi Ronal, > > Many thanks for your help, we are going to try the suggested rpar. > Just one question, if we use "domain half its current size" as you > suggested, when we will plot phi.*.xy.h5 result files we will see the two > "black holes" as figure " > https://docs.einsteintoolkit.org/et-docs/File:vt-5.png" > or just one ? > > Cheers, > Benja > > > On Tue, Oct 23, 2018 at 7:47 PM Roland Haas <rhaas(a)illinois.edu> wrote: > > > Hello Benja, > > > > attached please find a modified rpar file where I made two changes: > > > > * changed the boundary condition to be of Robin type instead of > > Dirichlet type, which reduces reflections on the boundary (the line > > NewRad::z_is_radial = "yes") > > * made the domain half its current size which reduces memory footprint > > and runtime but will induce some reflections off the boundary, this > > makes the simulation smaller so that is uses less memory > > * then ran with very low resolution (N=24 instead of N=28) this makes > > the simulation runs faster > > > > I gave it a test run on my workstation (12 cores, 96GB of RAM) and it > > runs at ~4.1 M/hour. Since the full simulation is > > about 1000 M this will finish in 10 days. > > > > If this is too slow (which is may well be) then you can try and reduce > > the finite difference order to 6 from 8 by changing the lines (they > > are not consecutive in the file): > > > > Driver::ghost_size = 5 > > Coordinates::patch_boundary_size = 5 > > Coordinates::additional_overlap_size = 3 > > Coordinates::outer_boundary_size = 5 > > ML_BSSN::fdOrder = 8 > > SummationByParts::order = 8 > > Interpolate::interpolator_order = 5 > > WeylScal4::fdOrder = 8 > > to: > > > > Driver::ghost_size = 4 > > Coordinates::patch_boundary_size = 4 > > Coordinates::additional_overlap_size = 3 > > Coordinates::outer_boundary_size = 4 > > ML_BSSN::fdOrder = 6 > > SummationByParts::order = 6 > > Interpolate::interpolator_order = 3 > > WeylScal4::fdOrder = 6 > > > > which gives me a run speed of ~6.9M/hr (so 7 days runtime). > > > > This is the command line to start the simulation: > > > > simfactory/bin/sim create-submit GW150914_24 --define N 24 \ > > --parfile ~/runs/devel/GW150914.rpar --procs 12 --walltime 24:00:00 > > > > Yours, > > Roland > > > > > Dear friends, > > > > > > We are trying to use the EinsteinToolKit GW150914.rpar binary > > > balckhole merge simulation as use case to test that our container > > > orchestration product OpenShift can be used for HPC. > > > Our test environment only has 30 CPUs so we need to execute that > > > simulation in a reasonable time. > > > > > > Please can you tell us how to modify GW150914.rpar in order to get a > > > less precise simulation executed in a 30 CPUs cluster in a reasonable > > > time (~ few days). > > > Now we can run the simulation GW150914.rpar using OpenMPI + > > > EinsteinToolKit, but it takes so long to be executed (~ weeks). > > > > > > We believe that GW150914.rpar EinsteinToolKit is a great use case to > > > test OpenShift for HPC, and of course we will reference to > > > EinsteinToolKit is our final report as a use case for Openshift in > > > HPC mode. > > > > > > Many thanks in advance for your help, > > > Benja > > > > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from http://pgp.mit.edu . > > > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

2 1

how to speed up ET runs on cluster
by Hamilton, Maria 23 Oct '18

23 Oct '18

Hello, I am using the Marshall University BigGreen cluster to run a parameter file with 5 Carpet time levels, on a 200^3 spatial domain, starting with d(x,y,z)_coarse = 1, and dtfac = 0.03125. The total necessary memory looks to be somewhere around 40 GB. My aim is to run it up to t=200, but I have a time limit on cluster of 72 hours. I was thinking, naively, that if I increase the number of processes, the run will go faster. However, this was not the case. I did an experiment, where I used first 48 procs, then 96 procs, and in 72 hours, the 48 procs run stopped at t = 55.25, while the 96 procs run stopped ash 48.50. That obviously did not work, so I tried next to decrease the time refinement. For the same 48 procs, the run with bigger time step dtfac = 0.0625 stopped ash t = 54.17 after 72 hours. I would very much appreciate if I can get an explanation of what is happening, and how to get to speed up the run, so that it will get finished in the time constraint of 72 hours that is imposed to me. I am attaching the parameter file and the SLURM script I am using to run it. thanks, Maria? _______________________ Dr. Maria C. Babiuc Hamilton Department of Physics Marshall University S257 Science Building Huntington, WV, 25755 Phone: (304)696-2754

2 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Users