#1743: Reduce number of output files per directory -----------------------+---------------------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: review Priority: unset | Milestone: Component: Other | Version: development version Resolution: | Keywords: -----------------------+----------------------------------------------------
Comment (by eschnett):
Replying to [comment:14 knarf]:
Replying to [comment:13 eschnett]:
Replying to [comment:12 knarf]:
This is a special case, since {{{ilog(0)}}} is not defined.
We could define it. The "Number of processes that can access the
same directory" being 1 doesn't seem to be unreasonable to me.
It's "log of the number of processes minus one", and "log of zero" is
undefined. Running on one process is truly a special case.
I meant 'we can (re)define what ilog(0) is, different from the
definition for values >0. If I read the comment for the parameter correctly, restricting the number of processes that can access the same directory to 1 isn't something so special. Each process would write into it's own directory. That is different from running on one process.
The fact that you cannot have just one file per directory is
different. The subdirectories form a tree; if each tree branch has just 1 child, then the tree would be infinite. That's a limit on the use parameter {{{processes_per_directory}}}.
I am afraid, but I think I don't understand the meaning of the
parameter. I assumed: For a value of '1' I would expect each process to write into a different directory. For a value of '2' I would expect 2 processes to share one directory, and so on. A value of '0' would be special, as all processes could write to the same directory in that case. Isn't that how I should interpret the parameter?
We do not only create one level of subdirectories -- we create a tree of subdirectories, ensuring that each directory hold no more than N entries. That's why we need the log, and not just divide nprocs by the max number of files.
You cannot have a tree with just one entry per branch. That would not be a tree. It's not possible to have just 1 entry per directory.
If you let each process write to its own directory, then you need {{{nprocs}}} directories, instead of the {{{nprocs}}} files that you would normally have. You don't gain anything from this, the directories are still holding too many entries, and Lustre will still crash.
I do understand that a wrong usage of both strncat and strncpy can
lead to problems much like the usage of strcat and strcpy. I don't understand why that should prevent us from using them correctly. Right now, with asserts possibly doing nothing, both strcat and strcpy could write into memory they shouldn't touch.
I refuse to do that. I'll convert the file to C++ instead.
That would be fine too, of course. string handling in C is painful even
if done correctly.
Sure. I'm actually with you on this -- I don't like the style, and I'm
serious about changing the style with an automated tool.
I don't have enough experience with tools like this to have an informed
opinion. I didn't quite like what a quick try using clang-format did to my source, but that is more my lack of configuration than anything else.
I assume you didn't like it because you have a very firm opinion on how source code "should" be formatted. The problem is that, in a collaboration, everybody has his or her own firm opinion. What clang- format does is a reasnoable compromise, decided essentially by the LLVM developers who are expert software engineers. I stopped caring about the details of where braces go and how lines are broken, as long as I can read the code, and as long as the style is consistent within a group of source files. clang-format does something very reasonable, although it looks different from how I would format the code. But then, the way you write code looks different from what I would format it. So it doesn't matter. Trust clang-format, let it do its job, give it the benefit of the doubt for a few days, and then enjoy your newfound freedom from worrying about formatting.
I stopped even hitting "enter" when I make a small change to code, let alone worry about spaces, or indentation. I just write the essential characters, and a few seconds later the code is nicely formatted.