[Users] [USERS] Problems regarding Binary neutron star merger parameter file available on the Gallery of Einstein Toolkit Website

Wed May 27 08:06:39 CDT 2020

Hello Aryan Sharma,

Q1: The gallery example page indicates that 12 cores are enough to run
the simulation, indicating that at most 12 MPI ranks were. Based on the
partial output that you copied and pasted into the email, namely from
the existence of a file nsnstohmns_from-ET-website/backtrace.63.txt, I
am inferring that you used at least 64 MPI ranks. This quite likely
leads to each MPI rank handling such as small part of the grid that
things fail. My suggestion would be to try running with a MPI rank
count closer to 12. 6 ranks and 2 threads per rank should be enough.

Q2: Thorn HydroAnalaysis and NSTracker are mostly designed, and
certainly used in this parameter file, to handle situations of equal
mass stars with RotatingSymmetry180 is used. HydroAnalysis uses the
point with maximum density as the center of a star. The warning
message that you included indicates that multiple equal maxima were
found (1 for each star). The code, assuming there will be one star,
assumes that these are all clustered around the single star's center
and averages the values, ending up with an averaged location around the
origin (see the text of the warning). This averaging behaviour is
controlled via that parameter
Hydro_Analysis::Hydro_Analysis_average_multiple_maxima_locations in the
parameter file. Taking it out likely makes the code abort if multiple
maxima are found though. Instead (please see param.ccl) you will have
to add (when disabling that option) the option
Hydro_Analysis_rho_max_loc_only_positive_x

Q3: Turning off mesh refinement is no likely to work since the stars
are not resolved enough.

In your case I could glean enough information from the messages that
you had included in the email. It is best, if possible, to include the
full out and err files as attachments in the email as there is
sometimes extra information in there that is not immediately obvious.

Yours,
Roland

> Hi all,
> These are the questions regarding the binary neutron stars merger parameter
> file which is openly available on the Einstein Toolkit website under the
> Gallery section of the the website. The name of the parameter file is
> nsnstohmns.par. I have* total** three questions* regarding this as
> follows:--
> *Question --1:-*
> So, when I downloaded this parameter file and ran it after I had compiled
> all the necessary thorns; I got the errors that the grid structure is
> inconsistent and it is impossible to continue. Can this be fixed?
> I have attached the error that appeared multiple times in the standard
> error output file as follows:--
> ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> ml=0 rl=0 c=7
> dh::light_dboxes:{
>    exterior:
> ([1536,0,2816]:[5376,2304,5888]:[256,256,256]/[6,0,11]:[21,9,23]/[16,10,13]/2080)
>    owned:
> ([2304,768,3584]:[4608,1536,5120]:[256,256,256]/[9,3,14]:[18,6,20]/[10,4,7]/280)
>    interior:
> ([2304,0,3584]:[4608,1536,5120]:[256,256,256]/[9,0,14]:[18,6,20]/[10,7,7]/490)
>    active_size: 280
> }
> 
>  The grid structure is inconsistent.  It is impossible to continue.
> Writing backtrace to nsnstohmns_from-ET-website/backtrace.42.txt
> Writing backtrace to nsnstohmns_from-ET-website/backtrace.45.txtreceived
> signal 6
> 
> Writing backtrace to nsnstohmns_from-ET-website/backtrace.63.txt
> cactus_sim: /home/
>  with PID 76212cactus_sim:
> /home/sharmaar/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int
> Carpet::Abort(const _cGH *, int): Assertion `0' failed.
> Rank cactus_sim:
> /home/sharmaar/Cactus/arrangements/Carpet/Carpet/src/helpers.cc:275: int
> Carpet::Abort(const _cGH *, int): Assertion `0' failed.
> 19 with PID 7231 received signal 6ml=0 ml=0 rl=0
> dh::level_dboxes:{
>    active:
> bboxset<CCTK_INT4,3>(set<bbox>:{([768,768,768]:[6144,11520,6144]:[256,256,256]/[3,3,3]:[24,45,24]/[22,43,22]/20812)},stride:[256,256,256],offset:[0,0,0])
> }
> ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> ml=0 rl=0
> dh::fast_dboxes:{
>    fast_mg_rest_sendrecv: []
>    fast_mg_prol_sendrecv: []
>    fast_ref_prol_sendrecv: []
>    fast_ref_rest_sendrecv: []
>    fast_sync_sendrecv:
> [(send:(ext:([2304,0,0]:[2816,1536,1536]:[256,256,256]/[9,0,0]:[11,6,6]/[3,7,7]/147),c:1),recv:(ext:([2304,0,0]:[2816,1536,1536]:[256,256,256]/[9,0,0]:[11,6,6]/[3,7,7]/147),c:0)),(send:(ext:([0,0,1792]:[2048,1536,2304]:[256,256,256]/[0,0,7]:[8,6,9]/[9,7,3]/189),c:3),recv:(ext:([0,0,1792]:[2048,1536,2304]:[256,256,256]/[0,0,7]:[8,6,9]/[9,7,3]/189),c:0)),(send:(ext:([2304,0,1792]:[2816,1536,2304]:[256,256,256]/[9,0,7]:[11,6,9]/[3,7,3]/63),c:4),recv:(ext:([2304,0,1792]:[2816,1536,2304]:[256,256,256]/[9,0,7]:[11,6,9]/[3,7,3]/63),c:0)),(send:(ext:([0,1792,0]:[2048,2304,1536]:[256,256,256]/[0,7,0]:[8,9,6]/[9,3,7]/189),c:12),recv:(ext:([0,1792,0]:[2048,2304,1536]:[256,256,256]/[0,7,0]:[8,9,6]/[9,3,7]/189),c:0)),(send:(ext:([2304,1792,0]:[2816,2304,1536]:[256,256,256]/[9,7,0]:[11,9,6]/[3,3,7]/63),c:13),recv:(ext:([2304,1792,0]:[2816,2304,1536]:[256,256,256]/[9,7,0]:[11,9,6]/[3,3,7]/63),c:0)),(send:(ext:([0,1792,1792]:[2048,2304,2304]:[256,256,256]/[0,7,7]:[8,9,9]/[9,3,3]/81),c:15),recv:(ext:([0,1792,1792]:[2048,2304,2304]:[256,256,256]/[0,7,7]:[8,9,9]/[9,3,3]/81),c:0)),(send:(ext:([2304,1792,1792]:[2816,2304,2304]:[256,256,256]/[9,7,7]:[11,9,9]/[3,3,3]/27),c:16),recv:(ext:([2304,1792,1792]:[2816,2304,2304]:[256,256,256]/[9,7,7]:[11,9,9]/[3,3,3]/27),c:0)),(send:(ext:([1536,0,0]:[2048,1536,1536]:[256,256,256]/[6,0,0]:[8,6,6]/[3,7,7]/147),c:0),recv:(ext:([1536,0,0]:[2048,1536,1536]:[256,256,256]/[6,0,0]:[8,6,6]/[3,7,7]/147),c:1)),(send:(ext:([0,0,1024]:[2048,1536,1536]:[256,256,256]/[0,0,4]:[8,6,6]/[9,7,3]/189),c:0),recv:(ext:([0,0,1024]:[2048,1536,1536]:[256,256,256]/[0,0,4]:[8,6,6]/[9,7,3]/189),c:3)),(send:(ext:([1536,0,1024]:[2048,1536,1536]:[256,256,256]/[6,0,4]:[8,6,6]/[3,7,3]/63),c:0),recv:(ext:([1536,0,1024]:[2048,1536,1536]:[256,256,256]/[6,0,4]:[8,6,6]/[3,7,3]/63),c:4)),(send:(ext:([0,1024,0]:[2048,1536,1536]:[256,256,256]/[0,4,0]:[8,6,6]/[9,3,7]/189),c:0),recv:(ext:([0,1024,0]:[2048,1536,1536]:[256,256,256]/[0,4,0]:[8,6,6]/[9,3,7]/189),c:12)),(send:(ext:([1536,1024,0]:[2048,1536,1536]:[256,256,256]/[6,4,0]:[8,6,6]/[3,3,7]/63),c:0),recv:(ext:([1536,1024,0]:[2048,1536,1536]:[256,256,256]/[6,4,0]:[8,6,6]/[3,3,7]/63),c:13)),(send:(ext:([0,1024,1024]:[2048,1536,1536]:[256,256,256]/[0,4,4]:[8,6,6]/[9,3,3]/81),c:0),recv:(ext:([0,1024,1024]:[2048,1536,1536]:[256,256,256]/[0,4,4]:[8,6,6]/[9,3,3]/81),c:15)),(send:(ext:([1536,1024,1024]:[2048,1536,1536]:[256,256,256]/[6,4,4]:[8,6,6]/[3,3,3]/27),c:0),recv:(ext:([1536,1024,1024]:[2048,1536,1536]:[256,256,256]/[6,4,4]:[8,6,6]/[3,3,3]/27),c:16))]
>    fast_ref_bnd_prol_sendrecv: []
>    fast_ref_refl_sendrecv_0_0: []
>    fast_ref_refl_sendrecv_0_1: []
>    fast_ref_refl_sendrecv_1_0: []
>    fast_ref_refl_sendrecv_1_1: []
>    fast_ref_refl_sendrecv_2_0: []
>    fast_ref_refl_sendrecv_2_1: []
>    fast_ref_refl_prol_sendrecv_0_0: []
> WARNING level 0 from host nid00272 process 0
>   in thorn CarpetLib, file
> /home/sharmaar/Cactus/arrangements/Carpet/CarpetLib/src/dh.cc:2123:
>   -> The grid structure is inconsistent.  It is impossible to continue.=0  
> c=8
> dh::light_dboxes:{
>    exterior:
> ([4096,0,2816]:[6912,2304,5888]:[256,256,256]/[16,0,11]:[27,9,23]/[12,10,13]/1560)
>    owned:
> ([4864,768,3584]:[6144,1536,5120]:[256,256,256]/[19,3,14]:[24,6,20]/[6,4,7]/168)
>    interior:
> ([4864,0,3584]:[6912,1536,5120]:[256,256,256]/[19,0,14]:[27,6,20]/[9,7,7]/441)
>    active_size: 168WARNING level 1 from host nid00805 process 0
> '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> *Question-- 2:-*
> So, When I saw this I changed the grid structure as below and removed
> rotating symmetry180 and made it a simple grid structure and it worked. But
> now the problem is with the thorn NSTracker which calls the
> Hydrobase_analysis to calculate the maximum rho and it's location but I
> keep getting the below mentioned warnings and thus the refinement regions
> centered on the neutron stars are not moving. Could you please help me in
> this? Like is it possible to do it work correctly?
> '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> Changes to grid structure:--
> OLD
> CoordBase::xmin = 0.00
> CoordBase::ymin = -396.00
> CoordBase::zmin = 0.00
> CoordBase::xmax =  396.00
> CoordBase::ymax =  396.00
> CoordBase::zmax =  396.00
> ActiveThorns = "RotatingSymmetry180 ReflectionSymmetry"
> ReflectionSymmetry::reflection_z   = "yes"
> NEW
> CoordBase::xmin = -396.00
> CoordBase::ymin = -396.00
> CoordBase::zmin = -396.00
> CoordBase::xmax =  396.00
> CoordBase::ymax =  396.00
> CoordBase::zmax =  396.00
> ActiveThorns = "ReflectionSymmetry"  ##RotatingSymmetry180
> ReflectionSymmetry::reflection_z   = "no"
>  ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> Warnings after this:--
> Found more than one (3) identical maximum, using average location
> (15.1875,0,0).
>   while executing schedule bin Hydro_Analysis_LocationSearch, routine
> Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine
>   in thorn Hydro_Analysis, file
> /home/sharmaar/Cactus/arrangements/EinsteinAnalysis/Hydro_Analysis/src/Hydro_Analysis.c:199:
>   -> Found more than one (3) identical maximum, using average location  
> (15.1875,0,0).
> }
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> WARNING level 1 from host nid00805 process 0
>   while executing schedule bin Hydro_Analysis_LocationSearch, routine
> Hydro_Analysis::Hydro_Analysis_LocationSearch_Combine
>   in thorn Hydro_Analysis, file
> /home/sharmaar/Cactus/arrangements/EinsteinAnalysis/Hydro_Analysis/src/Hydro_Analysis.c:228:
>   -> Could not find grid point with maximum density 0.000956964 on grid.  
> Using old location  (15.1875,0,0).
> Could not find grid point with maximum density 0.000956964 on grid. Using
> old location  (15.1875,0,0).
> '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> *Question--3:-*
> So, this is my final query. Seeing this I thought of trying to remove the
> refinement regions centered at the stars and now what I did was to remove
> the active thorns like CarpetTracker, NSTracker and Hydro_Analysis that
> were used mainly for the refinement regridding at the center of the stars
> and I also changed the num_centers to 1 centered at origin. Now the problem
> is that the evolution is happening but the reductions(max,min) obtained by
> Carpet IOScalar for hydrobase variables like rho is wrongly shown in the
> .asc files. They are showing for rho the value 1e-11 for max and min both
> which I think is the value of rho at the origin i.e the center of the
> refinement levels. The fact that I know that the initial data was correctly
> calculated and that the evolution does not have all rho values as 1e-11 is
> that I visualized the initial data (IOHDF5 3D data) using VisIt and it
> showed me the stars with the correct values of rho and other variables.
> Also the evolved values of rho along x,y,z axis seem to be correctly
> evolving as the data in these .asc files is also correct.
> I also checked the reductions of GRHydro::dens variable which seems to be
> correct. Thus there maybe problem in the reductions. Could you please help
> me in this and give me some insight into it if possible?
> I have attached the Carpet IOScalar parameters that are set.
> '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> IOBasic::outInfo_reductions = "maximum"
> IOBasic::outInfo_vars  = "
>  Carpet::physical_time_per_hour
>  HydroBase::rho
>  ML_ADMConstraints::ML_Ham
>  SystemStatistics::maxrss_mb
>  GRHydro::dens{reductions = 'sum maximum'}
>  HydroBase::w_lorentz
> "
> 
> IOScalar::outScalar_every      = 256 # every_coarse
> #IOScalar::all_reductions_in_one_file = "yes"
> IOScalar::one_file_per_group   = "no"
> IOScalar::outScalar_reductions = "minimum maximum average norm1 norm2"
> IOScalar::outScalar_vars       = "
>  ADMBase::lapse
>  ADMBase::shift
>  ADMBase::metric
>  ADMBase::curv
>  HydroBase::rho
>  HydroBase::vel
>  HydroBase::w_lorentz
>  GRHydro::dens{reductions = 'minimum maximum average norm1 norm2 sum'}
>  SystemStatistics::process_memory_mb
>  SphericalSurface::sf_radius
>  ML_ADMConstraints::ML_Ham
> "
> ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
> Finally I have also attached the final parameter file which I am using
> which have a few more changes that I did apart from the above mentioned
> ones.
> I thank you for taking out the time to read through this long mail. I
> really hope you could help me out in this.
> 
> Thanks and regards
> Aryan Sharma
> ᐧ

-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20200527/b8ca8cd0/attachment.bin 