#1743: Reduce number of output files per directory -----------------------+---------------------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: -----------------------+----------------------------------------------------
Comment (by rhaas):
As long as this grouping is off by default and can be turned on via a parameter: fine by me. How would we handle thorns that themselves look at {{{IO::out_dir}}}? Would we actually have
{{{ out_dir/proc00nnnn/proc0000nn/out_dir/files... }}}
and have IOUtil create the directories and chdir() into out_dir/proc00nnnn/proc0000nn at the beginning of the run? This would require no (I think) change to current code that uses {{{out_dir}}} but leads to an awkward directory layout. An alternative would be to steer {{{out_dir}}} to a different value on each process. Finally one could try and use piraha's advanced parameter setting options to make {{{out_dir}}} assume different values on different processors, something like:
{{{ IO::out_dir = "proc"+str(int($MyProc / 10000))+"/proc"+str($MyProc) }}}
which would require the predefined variable MyProc in piraha (same as $parfile) and would not (as is) provide zero padding for the process numbers. This is similar to steering the parameter.
Finally there are options in IOUtil to define output groups which are not designed for exactly this purpose but which currently can be used to have multiple processes write into a single common hdf5 file (but do nothing for ASCII output eg in TimerReport). They may be slow though since they involve a serialization of output inside each output group.