[Users] Question about GW150914.rpar simulation

[Benjamin Chardi Marco]

Hi Ronal,

Thanks for the clarification.

Cheers,
Benja

On Wed, Oct 24, 2018 at 5:48 PM Roland Haas <rhaas at illinois.edu> wrote:

> Hello Benja,
>
> you will still see both black holes. Half this size is really "reduce
> the radius of the sphere by half" and not showing only one half of the
> domain.
>
> Eg the original rpar file may have had a domain that goes out to a
> radius of 2000M while the new one goes only to 1000M. The black holes
> are never further apart than about 20M or so though so they are always
> included.
>
> Yours,
> Roland
>
> > Hi Ronal,
> >
> > Many thanks for your help, we are going to try the suggested rpar.
> > Just one question, if we use "domain half its current size" as you
> > suggested, when we will plot phi.*.xy.h5 result files we will see the two
> > "black holes" as figure "
> > https://docs.einsteintoolkit.org/et-docs/File:vt-5.png"
> > or just one ?
> >
> > Cheers,
> > Benja
> >
> >
> > On Tue, Oct 23, 2018 at 7:47 PM Roland Haas <rhaas at illinois.edu> wrote:
> >
> > > Hello Benja,
> > >
> > > attached please find a modified rpar file where I made two changes:
> > >
> > > * changed the boundary condition to be of Robin type instead of
> > >   Dirichlet type, which reduces reflections on the boundary (the line
> > >   NewRad::z_is_radial = "yes")
> > > * made the domain half its current size which reduces memory footprint
> > >   and runtime but will induce some reflections off the boundary, this
> > >   makes the simulation smaller so that is uses less memory
> > > * then ran with very low resolution (N=24 instead of N=28) this makes
> > >   the simulation runs faster
> > >
> > > I gave it a test run on my workstation (12 cores, 96GB of RAM) and it
> > > runs at ~4.1 M/hour. Since the full simulation is
> > > about 1000 M this will finish in 10 days.
> > >
> > > If this is too slow (which is may well be) then you can try and reduce
> > > the finite difference order to 6 from 8 by changing the lines (they
> > > are not consecutive in the file):
> > >
> > > Driver::ghost_size                      = 5
> > > Coordinates::patch_boundary_size        = 5
> > > Coordinates::additional_overlap_size    = 3
> > > Coordinates::outer_boundary_size        = 5
> > > ML_BSSN::fdOrder                        = 8
> > > SummationByParts::order                 = 8
> > > Interpolate::interpolator_order         = 5
> > > WeylScal4::fdOrder                      = 8
> > > to:
> > >
> > > Driver::ghost_size                      = 4
> > > Coordinates::patch_boundary_size        = 4
> > > Coordinates::additional_overlap_size    = 3
> > > Coordinates::outer_boundary_size        = 4
> > > ML_BSSN::fdOrder                        = 6
> > > SummationByParts::order                 = 6
> > > Interpolate::interpolator_order         = 3
> > > WeylScal4::fdOrder                      = 6
> > >
> > > which gives me a run speed of ~6.9M/hr (so 7 days runtime).
> > >
> > > This is the command line to start the simulation:
> > >
> > > simfactory/bin/sim create-submit GW150914_24 --define N 24 \
> > > --parfile ~/runs/devel/GW150914.rpar --procs 12 --walltime 24:00:00
> > >
> > > Yours,
> > > Roland
> > >
> > > > Dear friends,
> > > >
> > > > We are trying to use the EinsteinToolKit GW150914.rpar binary
> > > > balckhole merge simulation  as use case to test that our container
> > > > orchestration product OpenShift can be used for HPC.
> > > > Our test environment only has 30 CPUs so we need to execute that
> > > > simulation in a reasonable time.
> > > >
> > > > Please can you tell us how to modify  GW150914.rpar in order to get a
> > > > less precise simulation executed in a 30 CPUs cluster in a reasonable
> > > > time (~ few days).
> > > > Now we can run the simulation  GW150914.rpar using OpenMPI +
> > > > EinsteinToolKit, but it takes so long to be executed (~ weeks).
> > > >
> > > > We believe that  GW150914.rpar EinsteinToolKit is a great use case to
> > > > test OpenShift for HPC, and of course we will reference to
> > > > EinsteinToolKit is our final report as a use case for Openshift in
> > > > HPC mode.
> > > >
> > > > Many thanks in advance for your help,
> > > > Benja
> > > >
> > >
> > >
> > >
> > > --
> > > My email is as private as my paper mail. I therefore support encrypting
> > > and signing email messages. Get my PGP key from http://pgp.mit.edu .
> > >
> >
> >
>
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>


-- 
Benjamín Chardí Marco
Senior Red Hat Consultant
RHCE #100-107-341
bchardim at redhat.com
Mobile: 0034 654 344 878
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20181024/3852aff8/attachment-0001.html