#1743: Reduce number of output files per directory -----------------------+---------------------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: -----------------------+----------------------------------------------------
Comment (by eschnett):
My idea was to group things my process id, since this reduces I/O traffic: each directory would only be accessed from a subset of all processes. Whether a process writes one or multiple files into a directory is likely a secondary effect due to caching.
Having one directory per process may not be enough, since running on 10k processes then still creates 10k sub-directories in one directory. I was thinking of a hierarchical approach:
{{{ proc00nnnn/proc0000nn/files... proc00nnnn/proc0001nn/files... ... proc01nnnn/proc0000nn/files... }}} where proc00nnnn is used by the first 10k processes, proc0000nn is used by the first 100 processes, etc.