[ET Trac] [Einstein Toolkit] #1743: Reduce number of output files per directory

[Einstein Toolkit]

#1743: Reduce number of output files per directory
-----------------------+----------------------------------------------------
  Reporter:  eschnett  |       Owner:                     
      Type:  defect    |      Status:  new                
  Priority:  unset     |   Milestone:                     
 Component:  Other     |     Version:  development version
Resolution:            |    Keywords:                     
-----------------------+----------------------------------------------------

Comment (by eschnett):

 My idea was to group things my process id, since this reduces I/O traffic:
 each directory would only be accessed from a subset of all processes.
 Whether a process writes one or multiple files into a directory is likely
 a secondary effect due to caching.

 Having one directory per process may not be enough, since running on 10k
 processes then still creates 10k sub-directories in one directory. I was
 thinking of a hierarchical approach:

 {{{
 proc00nnnn/proc0000nn/files...
 proc00nnnn/proc0001nn/files...
 ...
 proc01nnnn/proc0000nn/files...
 }}}
 where proc00nnnn is used by the first 10k processes, proc0000nn is used by
 the first 100 processes, etc.

-- 
Ticket URL: <https://trac.einsteintoolkit.org/ticket/1743#comment:2>
Einstein Toolkit <http://einsteintoolkit.org>
The Einstein Toolkit