[Users] Possible performance issue
Vaishak P
vaishak at iucaa.in
Sat Oct 5 08:41:56 CDT 2019
Dear Sir,
I am a little worried about the performance because this is a new cluster
we have and it is supposed to be performing well. I am inclined to think
that some libraries/compiler options / settings might be the bottleneck.
I am presently running two simulations, both using the same parameter file
GW150914.rpar.
The first one is using mpich-3.3.1, the same as in the simulation mentioned
in the previous thread. I am using one node consisting of 2*16 cores, and
32 mpiprocs.
The second one is using openmpi-3.1.2 with openmp. It uses 128 procs,
distributed among 16 mpiprocs and 8 openmp threads per mpiproc. Since I
have 32 PPN, it is launching 4 mpiprocs per node.
I am herewith attaching the carpet-timing..asc file from both these runs.
Thanking you
Regards,
Vaishak
On Fri, Oct 4, 2019 at 8:05 PM Haas, Roland <rhaas at illinois.edu> wrote:
> Hello Vaishak,
>
> I do not see anything obviously wrong with the setup.
>
> It uses 128 MPI ranks for the 4 nodes which fits with there being 2x16
> cores per node.
>
> Lookin at the timer tree output at iteration 1024 (search for
> "gettimeof " and you will find the spot) out of 5977s spend during
> Evolve about 2143s were spent in "syncs" which is communication and
> about the same amount of time in "thorns" that is doing computation.
> While this ratio is not great (spending more time sending data than
> doing computation) it is also not unheard of.
>
> Getting the original output files for the gallery data from Zenodo
> (link is on the gallery page):
>
> wget
> https://zenodo.org/record/155394/files/GW150914_28.tar.xz?download=1
>
> you can see (in GW150914_28/output-0000/GW150914_28.out) that that one
> took about 137s for syncs and 198s for thorns, so the same ratio but
> about a factor of 10 faster.
>
> I am reaching for straws here, but sometimes having too many MPI ranks
> can be detrimental if there is not enough work to split up (OpenMP can
> be a bit more forgiving in that respect, the original gallery run
> used 120 cores on 10 nodes using 6 OpenMP threads per MPI rank).
>
> Since each node has lots of RAM (more than the 96GB required to run the
> simulation), can you try and see what would happen if you were to run
> on only a single node?
>
> Also if you could add the parameter:
>
> Carpet::output_timers_every = 1024
>
> then provide the files carpet-timing-statistics*.asc that would let us
> know in even more detail where the time is spent.
>
> Running for a short time (2048 iterations) is enough to get data to
> compare.
>
> Yours,
> Roland
>
> > Dear All,
> >
> > I am running the simulation GW150914 using the parameter file available
> at
> > the ETK gallery at (GW150914-ETK gallery
> > <https://einsteintoolkit.org/gallery/bbh/index.html>) using 128 cores.
> >
> > Each compute node consists of 2 X 16 Cores Intel SkyLake ( Intel(R)
> Xeon(R)
> > Gold 6142 CPU @ 2.60GHz) and 384 GB RAM . I have compiled and am running
> > Einstein Toolkit without OpenMP and using mpich-3.3.1.
> >
> >
> > The issue is that the simulation seems to be running at a very slow pace.
> > The number of physical time per hour that it is completing is only about
> > 1.3 units. At this rate to complete 1700 units, it would take about 54
> > days, in contrast to 2.8 days on (Intel(R) Xeon(R) CPU E5-2630 v3 @
> > 2.40GHz) as per the details at the example run of GW150914 available at
> the
> > gallery (GW150914-ETK gallery
> > <https://einsteintoolkit.org/gallery/bbh/index.html>).
> >
> > I have also tried using intel mpi (impi) but with simular results.
> >
> > I am also attaching the out file from the simulation.
> >
> > Looking forward to your inputs.
> >
> >
> > Thanks and regards,
> >
> >
> >
> >
> >
> > Vaishak P
> >
> > PhD Scholar,
> > Shyama Prasad Mukherjee Fellow
> > Inter-University Center for Astronomy and Astrophysics (IUCAA)
> > Pune, India
>
>
>
> --
> My email is as private as my paper mail. I therefore support encrypting
> and signing email messages. Get my PGP key from http://pgp.mit.edu .
>
--
Regards,
Vaishak P
PhD Scholar,
Shyama Prasad Mukherjee Fellow
Inter-University Center for Astronomy and Astrophysics (IUCAA)
Pune, India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.einsteintoolkit.org/pipermail/users/attachments/20191005/2bec72b8/attachment-0001.html
-------------- next part --------------
# 0D ASCII output created by CarpetIOASCII
# created on hpc002 by cvaishak on Oct 05 2019 at 11:13:11+0530
# parameter filename: "/mnt/home/visitor/cvaishak/simulations/GW150914_28_test/output-0000/GW150914.par"
# Build ID: build-sim-ln2.iucaa.hpc.com-cvaishak-2019.09.11-19.49.15-127232
# Simulation ID: run-GW150914-hpc002-cvaishak-2019.10.05-05.27.03-128703
# Run ID: run-GW150914-hpc002-cvaishak-2019.10.05-05.27.03-128703
#
# CARPET::TIMING (carpet-timing)
#
# iteration 0 time 0
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
0 0 0 0 0 0 0 0 0 0 0 0 0 0 935.009733915329 0 866.102562427521 0 68.9071714878082 0 0 0 0 0 0 0 0 0 10.7535976880303 7158551.69134718 63.4621898647185 7158488.22915732 741 493275549 4373 493271176 0 0 0 0 928.20902633667
# iteration 256 time 0.550681318682
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
256 0 0 0 0 0 0 0 0.550681318682 0 0 0 0.606811857659008 0.584645774616121 4205.3013420105 3266.99737691879 4133.18746757507 0 72.1138744354248 128 5270.03300389487 178200.8948379 17217184 582181856 4146.34676345402 149664.379731197 13546104 488953136 11.6066430565938 6871827.60432252 186.607641114197 6871640.9966814 837 495554113 13457 495540656 0 0 0 0 4180.10276532173
# iteration 512 time 1.101362637364
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
512 0 0 0 0 0 0 0 1.101362637364 0 0 0 0.598256838240427 0.589223437770107 7565.73426103592 6627.43029594421 7493.61586046219 0 72.1184005737305 256 5204.27110657164 175591.310060577 34490944 1163719168 4094.78527697343 147465.078372546 27137904 977314528 11.6891111463039 6871486.07092807 276.337243220273 6871209.73368485 843 495560585 19929 495540656 0 0 0 0 7497.96553254128
# iteration 768 time 1.652043956046
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
768 0 0 0 0 0 0 0 1.652043956046 0 0 0 0.593696635932741 0.587859946800288 10955.8076560497 10017.503690958 10883.6851673126 0 72.1224887371063 384 5161.82889422573 174211.702020656 51708640 1745166368 4061.29523433289 146303.407836313 40684040 1465594928 11.7716403699641 6871186.25102347 366.002344930438 6870820.24867854 849 495567053 26397 495540656 0 0 0 0 10845.6465444565
# iteration 1024 time 2.202725274728
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1024 0 0 0 0 0 0 0 2.202725274728 0 0 0 0.590428156982431 0.586771941019566 14368.9151160717 13430.61115098 14281.2087128162 0 87.7064032554626 512 5139.26158862574 173371.055555435 69023424 2328479232 4043.65694081146 145605.295992603 54308784 1955568112 15.9053381306355 6097430.33746731 374.750289374714 6097055.58717794 1395 534783684 32868 534750816 0 0 0 0 14215.7185053825
# iteration 1280 time 2.75340659341
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1280 0 0 0 0 0 0 0 2.75340659341 0 0 0 0.589439055594078 0.58549728649031 17754.7392559052 16816.4352908134 17664.353344202 0 90.3859117031097 640 5133.30004291441 173149.430402212 86323808 2911756192 4039.03008131008 145423.785107177 67922088 2445509672 16.4959336240086 5941833.88628167 435.21163042765 5941398.67465124 1491 537058073 39337 537018736 0 0 0 0 17559.3677191734
# iteration 1536 time 3.304087912092
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1536 0 0 0 0 0 0 0 3.304087912092 0 0 0 0.588901083842537 0.58590470043727 21136.4608540535 20198.1568889618 21046.0705792904 0 90.3902747631073 768 5127.65994290302 173023.983683871 103569280 3494765568 4034.57802848014 145320.69357299 81491040 2935210168 16.5615162020837 5941618.65762137 506.769120019271 5941111.88850135 1497 537064543 45807 537018736 0 0 0 0 20898.1933357716
# iteration 1792 time 3.854769230774
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1792 0 0 0 0 0 0 0 3.854769230774 0 0 0 0.589625521647447 0.586723094007292 24473.8680160046 23535.5640509129 24383.4736070633 0 90.3944089412689 896 5136.81982460543 173265.375207434 120897952 4077898336 4041.84882904093 145525.270972512 95127192 3425019336 16.6271345496216 5941418.67058333 578.498168332245 5940840.172415 1503 537071029 52293 537018736 0 0 0 0 24193.4548225403
# iteration 2048 time 4.405450549456
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
2048 0 0 0 0 0 0 0 4.405450549456 0 0 0 0.589315624076797 0.587383669912373 27850.2359249592 26911.9319598675 27743.7824101448 0 106.453514814377 1024 5138.29836543184 173263.099169301 138281536 4662844736 4043.10237415359 145529.409848407 108807696 3916477576 19.2478379278772 5415732.40681893 552.109529708665 5415180.29728922 2049 576523750 58774 576464976 0 0 0 0 27527.2912971973
-------------- next part --------------
# 0D ASCII output created by CarpetIOASCII
# created on hpc005 by cvaishak on Oct 05 2019 at 12:16:04+0530
# parameter filename: "/mnt/home/visitor/cvaishak/simulations/GW150914_28_t2_2/output-0000/GW150914.par"
# Build ID: build-sim-ln2.iucaa.hpc.com-cvaishak-2019.10.05-05.37.46-120159
# Simulation ID: run-GW150914-hpc005-cvaishak-2019.10.05-06.22.20-302377
# Run ID: run-GW150914-hpc005-cvaishak-2019.10.05-06.22.20-302377
#
# CARPET::TIMING (carpet-timing)
#
# iteration 0 time 0
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
0 0 0 0 0 0 0 0 0 0 0 0 0 0 1397.36106491089 0 1343.24657058716 0 54.1144943237305 0 0 0 0 0 0 0 0 0 12.5105114343297 8929281.5730536 80.8101425440529 8929200.76291105 677 483203557 4373 483199184 0 0 0 0 1389.43537902832
# iteration 256 time 0.550681318682
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
256 0 0 0 0 0 0 0 0.550681318682 0 0 0 2.57107125470139 2.19331225583527 2170.29773402214 771.060990095139 2114.46202731133 0 55.8357067108154 128 45780.4096607773 755039.955954932 35299488 582181856 39516.4693732469 634130.29874546 30469608 488953136 13.8441876271671 8685175.75163181 241.100199012837 8684934.6514328 773 484942926 13462 484929464 0 0 0 0 2145.29657387733
# iteration 512 time 1.101362637364
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
512 0 0 0 0 0 0 0 1.101362637364 0 0 0 2.50633765451786 2.37190085798674 2981.18859601021 1581.95185208321 2925.3483915329 0 55.8402044773102 256 44627.7653185406 735622.36832148 70598976 1163719168 38521.5364928785 617790.311830929 60939216 977314528 13.950521981282 8684592.24924672 357.144107667148 8684235.10513905 779 484949407 19943 484929464 0 0 0 0 2916.73786568642
# iteration 768 time 1.652043956046
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
768 0 0 0 0 0 0 0 1.652043956046 0 0 0 2.48733289282551 2.42165329289011 3790.29515790939 2391.05841398239 3734.45096445084 0 55.8441934585571 384 44274.0032535147 729871.908521618 105861728 1745166368 38216.8195748115 612948.19040368 91378648 1465594928 14.0569672759722 8684087.81586028 473.030379059978 8683614.78548121 785 484955880 26416 484929464 0 0 0 0 3686.49597477913
# iteration 1024 time 2.202725274728
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1024 0 0 0 0 0 0 0 2.202725274728 0 0 0 2.46436693799147 2.43324745342002 4617.02499890327 3217.78825497627 4552.59219574928 0 64.4328031539917 512 43898.2769551571 723627.239424171 141255360 2328479232 37894.3020291741 607736.730027445 121935840 1955568112 20.6571798035694 8126538.80273656 510.469797835613 8126028.33293872 1331 523615675 32891 523582784 0 0 0 0 4473.87265825272
# iteration 1280 time 2.75340659341
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1280 0 0 0 0 0 0 0 2.75340659341 0 0 0 2.45596304877473 2.42951327043521 5435.23547005653 4035.99872612953 5369.07863354683 0 66.1568365097046 640 43772.0931020755 721446.26140463 176664112 2911756192 37786.2983485752 605924.292336239 152505452 2445509672 21.5699552047153 7941354.14747246 595.01031301921 7940759.13715944 1427 525374868 39364 525335504 0 0 0 0 5252.62827634811
# iteration 1536 time 3.304087912092
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1536 0 0 0 0 0 0 0 3.304087912092 0 0 0 2.4521881620085 2.42931805584786 6249.89081192017 4850.65406799316 6183.72989225388 0 66.1609196662903 768 43695.0788551468 720473.057656299 211949712 3494765568 37719.3775180296 605116.367165381 182963652 2935210168 21.6593119809689 7940961.80418858 692.735231480646 7940269.0689571 1433 525381336 45832 525335504 0 0 0 0 6027.93404364586
# iteration 1792 time 3.854769230774
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
1792 0 0 0 0 0 0 0 3.854769230774 0 0 0 2.45048113529369 2.430291243291 7062.2753329277 5663.0385890007 6996.10998630524 0 66.165346622467 896 43681.4964461775 720090.155119283 247370000 4077898336 37708.1555500546 604802.401073587 213542740 3425019336 21.7485447210122 7940528.43095966 790.625344993462 7939737.80561466 1439 525387816 52312 525335504 0 0 0 0 6801.00533056259
# iteration 2048 time 4.405450549456
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
2048 0 0 0 0 0 0 0 4.405450549456 0 0 0 2.44490704918211 2.43094708392979 7886.03630685806 6486.79956293106 7811.34801006317 0 74.6882967948914 1024 43620.018971597 718820.535575959 282954320 4662844736 37655.9839147594 603761.151859969 244266820 3916477576 26.5771223228078 7552425.65711557 787.071101131615 7551638.58601444 1985 564077809 58785 564019024 0 0 0 0 7585.59492087364
# iteration 2304 time 4.956131868138
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
2304 0 0 0 0 0 0 0 4.956131868138 0 0 0 2.44666811264019 2.43057507762898 8691.63354802132 7292.39680409431 8615.2381207943 0 76.3954272270203 1152 43680.925292101 719625.503243821 318538640 5247794720 37709.2617677643 604456.833386406 274990900 4407939080 27.2398502833946 7409273.9519336 854.30505946456 7408419.64687413 2081 566034649 65265 565969384 0 0 0 0 8352.03047513962
# iteration 2560 time 5.50681318682
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
2560 0 0 0 0 0 0 0 5.50681318682 0 0 0 2.4469141797169 2.4321128784869 9501.0850379467 8101.8482940197 9424.68562150002 0 76.3994164466858 1280 43679.7534534457 719632.168415646 353886736 5830350656 37707.4989450867 604458.40483274 305500436 4897230296 27.3169625772781 7408971.85248795 939.038586113601 7408032.81390184 2087 566041126 71742 565969384 0 0 0 0 9122.02516961098
# iteration 2816 time 6.057494505502
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
2816 0 0 0 0 0 0 0 6.057494505502 0 0 0 2.4462578659855 2.43296272941235 10313.660902977 8914.42415904999 10237.2574703693 0 76.4034326076508 1408 43666.2749107379 719477.849109158 389259696 6413730720 37695.1320696199 604332.441880814 336030396 5387275720 27.3940571590289 7408667.13288636 1023.72363819905 7407643.40924816 2093 566047600 78216 565969384 0 0 0 0 9895.13613033295
# iteration 3072 time 6.608175824184
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
3072 0 0 0 0 0 0 0 6.608175824184 0 0 0 2.44238694639426 2.43352265021263 11139.4758989811 9740.23915505409 11054.5536875725 0 84.9222114086151 1536 43596.4703987437 718466.537894894 424640048 6998035904 37634.232811398 603492.478000388 366566428 5878161064 31.0754978730132 7128439.40305808 997.31270058999 7127442.09035749 2639 605362838 84694 605278144 0 0 0 0 10681.6905515194
# iteration 3328 time 7.158857142866
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
3328 0 0 0 0 0 0 0 7.158857142866 0 0 0 2.44158795664016 2.43166096577901 11954.6155269146 10555.3787829876 11868.0235681534 0 86.5919587612152 1664 43585.7904731471 718502.333637089 460064528 7584064288 37624.2971630805 603541.778933381 397138708 6370612088 31.5849189592997 7011409.48519502 1052.94996561319 7010356.5352294 2735 607131681 91177 607040504 0 0 0 0 11457.6590619087
# iteration 3584 time 7.709538461548
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
3584 0 0 0 0 0 0 0 7.709538461548 0 0 0 2.44055908448646 2.43142204858527 12771.3598930836 11372.1231491566 12684.7640500069 0 86.5958430767059 1792 43592.8800187824 718477.020063398 495743600 8170609152 37630.6705781443 603539.548594234 427940620 6863526072 31.652789586817 7011169.82557929 1127.74466452732 7010042.08091476 2741 607138162 97658 607040504 0 0 0 0 12235.279665947
# iteration 3840 time 8.26021978023
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
3840 0 0 0 0 0 0 0 8.26021978023 0 0 0 2.44030795441848 2.43130648902067 13584.9082920551 12185.6715481281 13498.308454752 0 86.5998373031616 1920 43600.0079192692 718544.360351237 531295376 8755945568 37636.6385872637 603605.897217037 458627716 7355343208 31.7206138665541 7010921.25467347 1202.49648547779 7009718.75818799 2747 607144640 104136 607040504 0 0 0 0 13009.7387816906
# iteration 4096 time 8.810901098912
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
4096 0 0 0 0 0 0 0 8.810901098912 0 0 0 2.43733782984349 2.43079549621565 14413.1257369518 13013.8889930248 14317.5855305195 0 95.5402064323425 2048 43522.0272974184 717632.510082544 566390832 9339189824 37574.0112937866 602835.486625472 488984012 7845234104 34.4671643799719 6770012.17762746 1157.80574619476 6768854.37188126 3293 646808361 110617 646697744 0 0 0 0 13797.8657853603
# iteration 4352 time 9.361582417594
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
4352 0 0 0 0 0 0 0 9.361582417594 0 0 0 2.43658197394115 2.42907852487013 15230.7831740379 13831.5464301109 15133.6096816063 0 97.1734924316406 2176 43483.4073716207 717327.464295721 601442768 9921748128 37550.8256162384 602570.054629326 519385988 8334475688 34.8757661703276 6672046.79770151 1205.11259881269 6670841.6851027 3389 648346089 117105 648228984 0 0 0 0 14574.8563644886
# iteration 4608 time 9.912263736276
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
4608 0 0 0 0 0 0 0 9.912263736276 0 0 0 2.43493812365715 2.42811751915806 16054.2906460762 14655.0539021492 15957.1131465435 0 97.1774995326996 2304 43432.7633490761 716653.8149996 636509488 10502600288 37516.0754556741 601987.62740178 549800108 8822161128 34.9360707604707 6671838.44117983 1271.82733240025 6670566.61384743 3395 648352577 123593 648228984 0 0 0 0 15357.682986021
# iteration 4864 time 10.462945054958
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
4864 0 0 0 0 0 0 0 10.462945054958 0 0 0 2.43356870569624 2.42712326231203 16877.1651499271 15477.9284060001 16779.9836614132 0 97.1814885139465 2432 43399.692153867 716287.771669894 671737328 11086650848 37495.9834918876 601679.759959985 580360148 9312756248 34.9963769027053 6671631.30462808 1338.49565374102 6670292.80897434 3401 648359061 130077 648228984 0 0 0 0 16139.9402763844
# iteration 5120 time 11.01362637364
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
5120 0 0 0 0 0 0 0 11.01362637364 0 0 0 2.43069455028739 2.42602187292591 17711.0579640865 16311.8212201595 17604.9501922131 0 106.107771873474 2560 43349.2941380512 715244.179453182 707105936 11666935184 37460.4201304521 600786.963744331 611047676 9799929544 37.198029233019 6485213.81469161 1287.03107782569 6483926.78361378 3947 688131588 136564 687995024 0 0 0 0 16933.1169030666
# iteration 5376 time 11.564307692322
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
5376 0 0 0 0 0 0 0 11.564307692322 0 0 0 2.42731432769686 2.42261202186732 18550.5002689362 17151.2635250092 18442.7908468246 0 107.709422111511 2688 43239.4956160878 714230.951331235 741611984 12249963264 37370.6428721937 599932.321545652 640953744 10289597344 37.5361776225509 6402979.63242246 1328.12893427187 6401651.50348819 4043 689661236 143052 689518184 0 0 0 0 17731.8804545403
# iteration 5632 time 12.114989011004
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
5632 0 0 0 0 0 0 0 12.114989011004 0 0 0 2.42589504110346 2.4210861628813 19377.73939991 17978.502655983 19270.0255396366 0 107.713860273361 2816 43172.2831902078 713791.156780757 776173008 12832896208 37317.3083897914 599559.245519974 670909328 10779177488 37.5903341475671 6402776.13530635 1388.40070925381 6401387.7345971 4049 689667734 149550 689518184 0 0 0 0 18518.7440738678
# iteration 5888 time 12.665670329686
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
5888 0 0 0 0 0 0 0 12.665670329686 0 0 0 2.42312619987373 2.41872424586661 20216.4214160442 18817.1846721172 20108.7035660744 0 107.717849969864 2944 43092.0428389867 712950.534618446 810870928 13415721872 37252.5252961302 598849.372440798 700987648 11268659232 37.6446429364721 6402599.2460205 1448.60856435266 6401150.63745615 4055 689674225 156041 689518184 0 0 0 0 19316.9510550499
# iteration 6144 time 13.216351648368
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
6144 0 0 0 0 0 0 0 13.216351648368 0 0 0 2.42039052795496 2.41706439295162 21056.7529110909 19657.5161671638 20940.164863348 0 116.588047742844 3072 43015.0401917231 712060.790232559 845568848 13997346496 37190.1496497905 598094.30632132 731065968 11757048496 39.4637365414022 6255803.13008345 1394.05370573225 6254409.07637772 4601 729351874 162530 729189344 0 0 0 0 20116.8596639633
# iteration 6400 time 13.76703296705
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
6400 0 0 0 0 0 0 0 13.76703296705 0 0 0 2.41818811498013 2.41429160022105 21894.4655370712 20495.2287931442 21776.2947723866 0 118.170764684677 3200 42963.9936634692 711399.436188641 880556880 14580294208 37150.4298724733 597536.088599146 761406560 12246638848 39.7475637272325 6182073.23909047 1430.28608176482 6180642.95300871 4697 730540322 169018 730371304 0 0 0 0 20914.2254683971
# iteration 6656 time 14.317714285732
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
6656 0 0 0 0 0 0 0 14.317714285732 0 0 0 2.41750605520147 2.41316714875305 22720.2883780003 21321.0516340733 22602.11348629 0 118.174891710281 3328 42934.9185823962 711214.719623238 915417616 15163845760 37129.1411693262 597380.387168394 791632336 12736778080 39.7969478282233 6181912.28633418 1485.18012125862 6180427.10621292 4703 730546815 175511 730371304 0 0 0 0 21699.8528726101
# iteration 6912 time 14.868395604414
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
6912 0 0 0 0 0 0 0 14.868395604414 0 0 0 2.41587081910733 2.41167681474257 23555.3155179024 22156.0787739754 23437.1361203194 0 118.179397583008 3456 42900.1702736524 710744.994213366 950499552 15747322080 37103.0903250621 596985.023159257 822058992 13226847200 39.8462007448672 6181731.5364708 1540.07385146943 6180191.46261933 4709 730553309 182005 730371304 0 0 0 0 22494.4488084316
# iteration 7168 time 15.419076923096
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
7168 0 0 0 0 0 0 0 15.419076923096 0 0 0 2.41385545875964 2.41056682074924 24395.0945620537 22995.8578181267 24268.0435137749 0 127.051048278809 3584 42867.1914653673 710184.619906926 985767840 16331304544 37078.5802705689 596514.979892908 852653760 13717373664 41.3613273655807 6062633.66131136 1483.6398640832 6061150.02144728 5255 770263962 188498 770075464 0 0 0 0 23293.7845029831
# iteration 7424 time 15.969758241778
# time level 0
# refinement level 0 multigrid level 0 map 0 component 0
# column format: 1:it 2:tl 3:rl 4:c 5:ml 6:ix 7:iy 8:iz 9:time 10:x 11:y 12:z 13:data
# data columns: 13:physical_time_per_hour 14:current_physical_time_per_hour 15:time_total 16:time_evolution 17:time_computing 18:time_communicating 19:time_io 20:evolution_steps_count 21:local_grid_points_per_second 22:total_grid_points_per_second 23:local_grid_point_updates_count 24:total_grid_point_updates_count 25:local_interior_points_per_second 26:total_interior_points_per_second 27:local_interior_point_updates_count 28:total_interior_point_updates_count 29:io_per_second 30:io_bytes_per_second 31:io_bytes_ascii_per_second 32:io_bytes_binary_per_second 33:io_count 34:io_bytes_count 35:io_bytes_ascii_count 36:io_bytes_binary_count 37:comm_per_second 38:comm_bytes_per_second 39:comm_count 40:comm_bytes_count 41:time_levels
7424 0 0 0 0 0 0 0 15.969758241778 0 0 0 2.41322334710551 2.40896086303049 25222.6137800217 23823.3770360947 25093.9796864986 0 128.634093523026 3712 42856.6545562832 710014.830490751 1020990240 16914951008 37073.1420092902 596372.060370541 883207440 14207596448 41.5986139712033 5997262.4587424 1515.89671648827 5995746.56202591 5351 771452420 194996 771257424 0 0 0 0 24080.9235186577
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GW150914_28_1_32.o8047
Type: application/octet-stream
Size: 386397 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20191005/2bec72b8/attachment-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GW150914_28_16_128.o8054
Type: application/octet-stream
Size: 933529 bytes
Desc: not available
Url : http://lists.einsteintoolkit.org/pipermail/users/attachments/20191005/2bec72b8/attachment-0003.obj
More information about the Users
mailing list