#1743: Reduce number of output files per directory -----------------------+---------------------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: review Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: -----------------------+----------------------------------------------------
Comment (by rhaas):
I think I would like to get back to this. Though I would try and attempt a different solution that is more likely to capture all output files.
As far as I can tell almost all thorns producing output base the output path on IOUtil::out_dir (the exception being the "real" output thorns in Carpet and PUGH which have their own out_dir options that can optionally be used).
So I would try and make IOUtil set {{{out_dir}}} to the directory in the hierarchy. Unfortunately the parameter is currently steerable on recovery only (I suspect since IOUtil will only attempt to create the directory at simulation startup).
So a way to make this work is to add a special variable or so to piraha that expands to the proper directory hierarchy, so that eg:
{{{ IOUtil::out_dir = "$parfile/$MPI_hierarchy" }}}
is expanded to:
{{{ IOUtil::out_dir = "parfile-name/1/4/" }}}
on MPI rank 140 to 149 and similar for the others, having groups of 10 in each intermediate directory. If we want more files per output directory we'd use some other factor to bundle up the runs. Eg the equivalent of this perl code: {{{#!perl $size = 120000; $rank = 78923; $lumpsize = 100; @path = (); # put $lumpsize ranks in the same output directory $size = int($size / $lumpsize); $rank = int($rank / $lumpsize); while($size) { $idx = $rank % $lumpsize; $rank = int($rank / $lumpsize); $size = int($size / $lumpsize); push path,sprintf("%02d",$idx); } print join("/", reverse @path),"\n"; }}}
We can also decide to not put multiple ranks in the same directory by remove the two lines before the loop.