[ET Trac] [Einstein Toolkit] #1743: Reduce number of output files per directory

[Einstein Toolkit]

#1743: Reduce number of output files per directory
-----------------------+----------------------------------------------------
  Reporter:  eschnett  |       Owner:                     
      Type:  defect    |      Status:  new                
  Priority:  unset     |   Milestone:                     
 Component:  Other     |     Version:  development version
Resolution:            |    Keywords:                     
-----------------------+----------------------------------------------------

Comment (by rhaas):

 As long as this grouping is off by default and can be turned on via a
 parameter: fine by me. How would we handle thorns that themselves look at
 {{{IO::out_dir}}}? Would we actually have

 {{{
 out_dir/proc00nnnn/proc0000nn/out_dir/files...
 }}}

 and have IOUtil create the directories and chdir() into
 out_dir/proc00nnnn/proc0000nn at the beginning of the run? This would
 require no (I think) change to current code that uses {{{out_dir}}} but
 leads to an awkward directory layout. An alternative would be to steer
 {{{out_dir}}} to a different value on each process. Finally one could try
 and use piraha's advanced parameter setting options to make {{{out_dir}}}
 assume different values on different processors, something like:

 {{{
 IO::out_dir = "proc"+str(int($MyProc / 10000))+"/proc"+str($MyProc)
 }}}

 which would require the predefined variable MyProc in piraha (same as
 $parfile) and would not (as is) provide zero padding for the process
 numbers. This is similar to steering the parameter.

 Finally there are options in IOUtil to define output groups which are not
 designed for exactly this purpose but which currently can be used to  have
 multiple processes write into a single common hdf5 file (but do nothing
 for ASCII output eg in TimerReport). They may be slow though since they
 involve a serialization of output inside each output group.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1743#comment:3>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit