[Users] Question about GW150914.rpar simulation

[Roland Haas]

Hello Benja,

you will still see both black holes. Half this size is really "reduce
the radius of the sphere by half" and not showing only one half of the
domain.

Eg the original rpar file may have had a domain that goes out to a
radius of 2000M while the new one goes only to 1000M. The black holes
are never further apart than about 20M or so though so they are always
included.

Yours,
Roland

> Hi Ronal,
> 
> Many thanks for your help, we are going to try the suggested rpar.
> Just one question, if we use "domain half its current size" as you
> suggested, when we will plot phi.*.xy.h5 result files we will see the two
> "black holes" as figure "
> https://docs.einsteintoolkit.org/et-docs/File:vt-5.png"
> or just one ?
> 
> Cheers,
> Benja
> 
> 
> On Tue, Oct 23, 2018 at 7:47 PM Roland Haas <rhaas at illinois.edu> wrote:
> 
> > Hello Benja,
> >
> > attached please find a modified rpar file where I made two changes:
> >
> > * changed the boundary condition to be of Robin type instead of
> >   Dirichlet type, which reduces reflections on the boundary (the line
> >   NewRad::z_is_radial = "yes")
> > * made the domain half its current size which reduces memory footprint
> >   and runtime but will induce some reflections off the boundary, this
> >   makes the simulation smaller so that is uses less memory
> > * then ran with very low resolution (N=24 instead of N=28) this makes
> >   the simulation runs faster
> >
> > I gave it a test run on my workstation (12 cores, 96GB of RAM) and it
> > runs at ~4.1 M/hour. Since the full simulation is
> > about 1000 M this will finish in 10 days.
> >
> > If this is too slow (which is may well be) then you can try and reduce
> > the finite difference order to 6 from 8 by changing the lines (they
> > are not consecutive in the file):
> >
> > Driver::ghost_size                      = 5
> > Coordinates::patch_boundary_size        = 5
> > Coordinates::additional_overlap_size    = 3
> > Coordinates::outer_boundary_size        = 5
> > ML_BSSN::fdOrder                        = 8
> > SummationByParts::order                 = 8
> > Interpolate::interpolator_order         = 5
> > WeylScal4::fdOrder                      = 8
> > to:
> >
> > Driver::ghost_size                      = 4
> > Coordinates::patch_boundary_size        = 4
> > Coordinates::additional_overlap_size    = 3
> > Coordinates::outer_boundary_size        = 4
> > ML_BSSN::fdOrder                        = 6
> > SummationByParts::order                 = 6
> > Interpolate::interpolator_order         = 3
> > WeylScal4::fdOrder                      = 6
> >
> > which gives me a run speed of ~6.9M/hr (so 7 days runtime).
> >
> > This is the command line to start the simulation:
> >
> > simfactory/bin/sim create-submit GW150914_24 --define N 24 \
> > --parfile ~/runs/devel/GW150914.rpar --procs 12 --walltime 24:00:00
> >
> > Yours,
> > Roland
> >  
> > > Dear friends,
> > >
> > > We are trying to use the EinsteinToolKit GW150914.rpar binary
> > > balckhole merge simulation  as use case to test that our container
> > > orchestration product OpenShift can be used for HPC.
> > > Our test environment only has 30 CPUs so we need to execute that
> > > simulation in a reasonable time.
> > >
> > > Please can you tell us how to modify  GW150914.rpar in order to get a
> > > less precise simulation executed in a 30 CPUs cluster in a reasonable
> > > time (~ few days).
> > > Now we can run the simulation  GW150914.rpar using OpenMPI +
> > > EinsteinToolKit, but it takes so long to be executed (~ weeks).
> > >
> > > We believe that  GW150914.rpar EinsteinToolKit is a great use case to
> > > test OpenShift for HPC, and of course we will reference to
> > > EinsteinToolKit is our final report as a use case for Openshift in
> > > HPC mode.
> > >
> > > Many thanks in advance for your help,
> > > Benja
> > >  
> >
> >
> >
> > --
> > My email is as private as my paper mail. I therefore support encrypting
> > and signing email messages. Get my PGP key from http://pgp.mit.edu .
> >  
> 
> 



-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20181024/0a67b807/attachment.bin