[Users] Question about GW150914.rpar simulation

[Roland Haas]

Hello Benja,

attached please find a modified rpar file where I made two changes:

* changed the boundary condition to be of Robin type instead of
  Dirichlet type, which reduces reflections on the boundary (the line
  NewRad::z_is_radial = "yes")
* made the domain half its current size which reduces memory footprint
  and runtime but will induce some reflections off the boundary, this
  makes the simulation smaller so that is uses less memory
* then ran with very low resolution (N=24 instead of N=28) this makes
  the simulation runs faster

I gave it a test run on my workstation (12 cores, 96GB of RAM) and it
runs at ~4.1 M/hour. Since the full simulation is
about 1000 M this will finish in 10 days. 

If this is too slow (which is may well be) then you can try and reduce
the finite difference order to 6 from 8 by changing the lines (they
are not consecutive in the file):

Driver::ghost_size                      = 5
Coordinates::patch_boundary_size        = 5
Coordinates::additional_overlap_size    = 3
Coordinates::outer_boundary_size        = 5
ML_BSSN::fdOrder                        = 8
SummationByParts::order                 = 8
Interpolate::interpolator_order         = 5
WeylScal4::fdOrder                      = 8
to:

Driver::ghost_size                      = 4
Coordinates::patch_boundary_size        = 4
Coordinates::additional_overlap_size    = 3
Coordinates::outer_boundary_size        = 4
ML_BSSN::fdOrder                        = 6
SummationByParts::order                 = 6
Interpolate::interpolator_order         = 3
WeylScal4::fdOrder                      = 6

which gives me a run speed of ~6.9M/hr (so 7 days runtime).

This is the command line to start the simulation:

simfactory/bin/sim create-submit GW150914_24 --define N 24 \
--parfile ~/runs/devel/GW150914.rpar --procs 12 --walltime 24:00:00

Yours,
Roland

> Dear friends,
> 
> We are trying to use the EinsteinToolKit GW150914.rpar binary
> balckhole merge simulation  as use case to test that our container
> orchestration product OpenShift can be used for HPC.
> Our test environment only has 30 CPUs so we need to execute that
> simulation in a reasonable time.
> 
> Please can you tell us how to modify  GW150914.rpar in order to get a
> less precise simulation executed in a 30 CPUs cluster in a reasonable
> time (~ few days).
> Now we can run the simulation  GW150914.rpar using OpenMPI +
> EinsteinToolKit, but it takes so long to be executed (~ weeks).
> 
> We believe that  GW150914.rpar EinsteinToolKit is a great use case to
> test OpenShift for HPC, and of course we will reference to
> EinsteinToolKit is our final report as a use case for Openshift in
> HPC mode.
> 
> Many thanks in advance for your help,
> Benja
> 



-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GW150914.rpar
Type: application/octet-stream
Size: 26515 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20181023/2f3df6cf/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20181023/2f3df6cf/attachment-0001.bin