#765: NaNChecker should not reduce in local mode -----------------------------------+---------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Keywords: | -----------------------------------+---------------------------------------- The NaNChecker calls reduction operations in local mode. This does not quite work if a process has more than one component of the grid, and deadlocks if processes have different numbers of components.
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by knarf):
I use it for simulations on one MPI processor but multiple levels and components there. What is the symptom I should see in that case? False positives?
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by eschnett):
In your case (same number of components on each process), the bit mask seems to be reset for each component.
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by knarf):
That would mean potentially missing NaNs, right?
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by eschnett):
If there are nans, the NaNChecker will find some nans. However, the bit mask in the output nanmask may be inconsistent.
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: new Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by rhaas):
The attached patch splits NaNCheckers operation into a local search routine followed by a per-level reduction. It does not use ENTER_LEVEL_MODE to not rely on Carpet's code (ie. to compile without Carpet present). Note that NaNChecker does not trigger the warning in CarpetReduce since it uses reducelocalscalar so does not reduce a grid function (it seems to end up in ReduceArrays inside of CarpetReduce). I attach a test for NaNChecker (using PUGH, which is likely fragily since it relies on IEEE floating point math to create NaNs on the grid).
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: review Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+--------------------------------------- Changes (by rhaas):
* status: new => review
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: review Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by eschnett):
Please apply.
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: review Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: | Keywords: ------------------------------------+---------------------------------------
Comment (by rhaas):
It's in cacutscode, I cannot apply it myself.
#765: NaNChecker should not reduce in local mode ------------------------------------+--------------------------------------- Reporter: eschnett | Owner: Type: defect | Status: closed Priority: major | Milestone: Component: EinsteinToolkit thorn | Version: Resolution: fixed | Keywords: ------------------------------------+--------------------------------------- Changes (by rhaas):
* status: review => closed * resolution: => fixed
Comment:
I knew there was something that I had outstanding to commit :-) Though I had thought it was the change to IOJpeg. Applied.
trac@lists.einsteintoolkit.org