Hello.
I am working on writing a diagnostic that reads in ADMBase variables (e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context.
Call the gridfunction quantity computed by this diagnostic, diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
Setting regrid_every=64, I found that the diagnostic outputs reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
Here is what I want to happen: 1) every 64 iterations, the diagnostic computes diag_data_gf at all gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this). 2) After CCTK_ANALYSIS, diag_data_gf should be output to files
So why does my scheduling choice yield obviously wrong values near AMR boundaries after the grids move.
******** Now here is where the situation becomes very weird: ******** When I set diag_data_gf to be computed at *every* iteration, the undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!
Upon further analysis, even at iteration 64 the diagnostic yields inconsistent results at all refinement levels except the finest one.
******** I have created a very simple thorn called ADMBaseMcLachlanTester ( math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
In the thorn's par/ subdirectory, you'll find 2 parfiles: qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
What is causing this weird behavior? Have I uncovered a bug?
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On 2 Jun 2015, at 16:27, Zach Etienne zachetie@gmail.com wrote:
Hello.
I am working on writing a diagnostic that reads in ADMBase variables (e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context.
Call the gridfunction quantity computed by this diagnostic, diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
Setting regrid_every=64, I found that the diagnostic outputs reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
Here is what I want to happen:
- every 64 iterations, the diagnostic computes diag_data_gf at all gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this).
- After CCTK_ANALYSIS, diag_data_gf should be output to files
So why does my scheduling choice yield obviously wrong values near AMR boundaries after the grids move.
Now here is where the situation becomes very weird:
When I set diag_data_gf to be computed at *every* iteration, the undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!
Upon further analysis, even at iteration 64 the diagnostic yields inconsistent results at all refinement levels except the finest one.
I have created a very simple thorn called ADMBaseMcLachlanTester (math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
In the thorn's par/ subdirectory, you'll find 2 parfiles: qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
What is causing this weird behavior? Have I uncovered a bug?
If you regrid level L, Carpet will do time prolongation to fill the points new to the level if level L-1 (the coarser one) does not exist at the regridding time. It needs to do this, since it doesn't have any grid points to do a purely-spatial interpolation. Think about the contents of the three timelevels of myadmbaselapse. Timelevels are cycled every iteration, whether you have set them or not. Since you are setting timelevel 0 only every 64 iterations, the past two timelevels do not contain valid data for the previous two timesteps, so the points which need to be filled by time prolongation will contain bad data. If you can partition your grids into "moving" and "fixed", and then ensure that you only regrid when the fixed grids exist, then you can avoid time interpolation when regridding, and you won't need valid data on the past timelevels. There is also a parameter
Carpet::time_interpolation_during_regridding (https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c...)
but I'm not sure exactly what it does. A quick grep of the Carpet source should reveal this. It might be helpful. But the best option is probably to just arrange your regridding frequency so that you don't need to interpolate in time. i.e., if your first fixed grid exists every 32 iterations, then only regrid on multiples of 32 iterations.
There is another parameter, if you are using CarpetRegrid2,
CarpetRegrid2::freeze_unaligned_parent_levels Do not change refinement levels where the parent does not exist at this time
(https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c...)
but I think this will stop the regridding from actually happening, so you would have to be careful that you do eventually regrid on an iteration when the grid is aligned with its parent.
Thanks for your quick feedback, Ian!
Think about the contents of the three timelevels of myadmbaselapse.
myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
Just prior to the output at, e.g., iteration 192 (the iteration when the grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not matter how often I choose to update myadmbaselapse, since the value just prior to the file output (at iteration 192, for instance) is updated in a GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I choose to update myadmbaselapse every cctk_iteration versus every 64?
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:02 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 16:27, Zach Etienne zachetie@gmail.com wrote:
Hello.
I am working on writing a diagnostic that reads in ADMBase variables
(e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context.
Call the gridfunction quantity computed by this diagnostic,
diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
Setting regrid_every=64, I found that the diagnostic outputs
reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
Here is what I want to happen:
- every 64 iterations, the diagnostic computes diag_data_gf at all
gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this).
- After CCTK_ANALYSIS, diag_data_gf should be output to files
So why does my scheduling choice yield obviously wrong values near AMR
boundaries after the grids move.
Now here is where the situation becomes very weird:
When I set diag_data_gf to be computed at *every* iteration, the
undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!
Upon further analysis, even at iteration 64 the diagnostic yields
inconsistent results at all refinement levels except the finest one.
I have created a very simple thorn called ADMBaseMcLachlanTester (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
In the thorn's par/ subdirectory, you'll find 2 parfiles:
qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
What is causing this weird behavior? Have I uncovered a bug?
If you regrid level L, Carpet will do time prolongation to fill the points new to the level if level L-1 (the coarser one) does not exist at the regridding time. It needs to do this, since it doesn't have any grid points to do a purely-spatial interpolation. Think about the contents of the three timelevels of myadmbaselapse. Timelevels are cycled every iteration, whether you have set them or not. Since you are setting timelevel 0 only every 64 iterations, the past two timelevels do not contain valid data for the previous two timesteps, so the points which need to be filled by time prolongation will contain bad data. If you can partition your grids into "moving" and "fixed", and then ensure that you only regrid when the fixed grids exist, then you can avoid time interpolation when regridding, and you won't need valid data on the past timelevels. There is also a parameter
Carpet::time_interpolation_during_regridding (https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c... )
but I'm not sure exactly what it does. A quick grep of the Carpet source should reveal this. It might be helpful. But the best option is probably to just arrange your regridding frequency so that you don't need to interpolate in time. i.e., if your first fixed grid exists every 32 iterations, then only regrid on multiples of 32 iterations.
There is another parameter, if you are using CarpetRegrid2,
CarpetRegrid2::freeze_unaligned_parent_levels Do not change refinement levels where the parent does not exist atthis time
( https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c... )
but I think this will stop the regridding from actually happening, so you would have to be careful that you do eventually regrid on an iteration when the grid is aligned with its parent.
-- Ian Hinder http://members.aei.mpg.de/ianhin
Forgot to mention something: This run has only 7 refinement levels in total (i.e., the coarsest level, plus 6 additional levels of refinement), so every 64 timesteps, all levels should be updated.
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:11 AM, Zach Etienne zachetie@gmail.com wrote:
Thanks for your quick feedback, Ian!
Think about the contents of the three timelevels of myadmbaselapse.
myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
Just prior to the output at, e.g., iteration 192 (the iteration when the grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not matter how often I choose to update myadmbaselapse, since the value just prior to the file output (at iteration 192, for instance) is updated in a GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I choose to update myadmbaselapse every cctk_iteration versus every 64?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:02 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 16:27, Zach Etienne zachetie@gmail.com wrote:
Hello.
I am working on writing a diagnostic that reads in ADMBase variables
(e.g., alp, betax, etc.) from a binary black hole simulation using McLachlan. The diagnostic is computed at CCTK_ANALYSIS, in a GLOBAL,LOOP-LOCAL context.
Call the gridfunction quantity computed by this diagnostic,
diag_data_gf. diag_data_gf is rather expensive to compute in general, so I only want it computed and output (to, e.g., IOASCII 2D data files) every 64 iterations.
Setting regrid_every=64, I found that the diagnostic outputs
reasonable-looking data every 64 iterations until the first actual AMR grid movement (at iteration 192). At this iteration, a large number of zones near AMR refinement boundaries in the IOASCII 2D output of diag_data_gf are set to undefined values (I have set IOASCII::out3D_ghosts=no). This is undesirable behavior!
Here is what I want to happen:
- every 64 iterations, the diagnostic computes diag_data_gf at all
gridpoints on all refinement levels at CCTK_ANALYSIS (I believe GLOBAL,LOOP-LOCAL should do this).
- After CCTK_ANALYSIS, diag_data_gf should be output to files
So why does my scheduling choice yield obviously wrong values near AMR
boundaries after the grids move.
Now here is where the situation becomes very weird:
When I set diag_data_gf to be computed at *every* iteration, the
undefined value problem disappears! Remember, it is being called at CCTK_ANALYSIS in a GLOBAL,LOOP-LOCAL context, so diag_data_gf should be recomputed at all points on all levels prior to file output. Where are these mysterious undefined values coming from?!
Upon further analysis, even at iteration 64 the diagnostic yields
inconsistent results at all refinement levels except the finest one.
I have created a very simple thorn called ADMBaseMcLachlanTester (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) that reproduces this problem (in both ET 2014 11 and ET 2015 05 releases) with a minimum of coding. All the thorn does is set a gridfunction called "myadmbaselapse", which appropriately enough, is set to ADMBase::alp at all gridpoints.
In the thorn's par/ subdirectory, you'll find 2 parfiles:
qc0-mclachlan-setlapseevery1.par and qc0-mclachlan-setlapseevery64.par. The former sets myadmbaselapse=alp at every cctk_iteration in CCTK_ANALYSIS, and the latter every 64 cctk_iteration's. You will notice that the former parfile yields reasonable data at iteration 192 in the admbasemclachlantester::admbasemclachlantestergfs.*.asc files. However, many nan's are produced at iteration 192 (corresponding to the first AMR grid movement) when using the latter parfile.
What is causing this weird behavior? Have I uncovered a bug?
If you regrid level L, Carpet will do time prolongation to fill the points new to the level if level L-1 (the coarser one) does not exist at the regridding time. It needs to do this, since it doesn't have any grid points to do a purely-spatial interpolation. Think about the contents of the three timelevels of myadmbaselapse. Timelevels are cycled every iteration, whether you have set them or not. Since you are setting timelevel 0 only every 64 iterations, the past two timelevels do not contain valid data for the previous two timesteps, so the points which need to be filled by time prolongation will contain bad data. If you can partition your grids into "moving" and "fixed", and then ensure that you only regrid when the fixed grids exist, then you can avoid time interpolation when regridding, and you won't need valid data on the past timelevels. There is also a parameter
Carpet::time_interpolation_during_regridding (https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c... )
but I'm not sure exactly what it does. A quick grep of the Carpet source should reveal this. It might be helpful. But the best option is probably to just arrange your regridding frequency so that you don't need to interpolate in time. i.e., if your first fixed grid exists every 32 iterations, then only regrid on multiples of 32 iterations.
There is another parameter, if you are using CarpetRegrid2,
CarpetRegrid2::freeze_unaligned_parent_levels Do not change refinement levels where the parent does not existat this time
( https://bitbucket.org/eschnett/carpet/src/59e2074462e38e9a1574d1479d96b2701c... )
but I think this will stop the regridding from actually happening, so you would have to be careful that you do eventually regrid on an iteration when the grid is aligned with its parent.
-- Ian Hinder http://members.aei.mpg.de/ianhin
On Tue, Jun 02, 2015 at 11:11:15AM -0400, Zach Etienne wrote:
myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
You use the ADMBase variables, right? They should, by default, be prolongated, and I don't see a parameter that would disable that in your parfile. Could you output the variables your analysis depends on, and see if the nan values come from there? It does, like Ian suggested, look like something you depend on isn't properly prolonged by regridding.
Frank
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn ( math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as 1) there's nothing complicated about the ADMBaseMcLachlanTester thorn at all (67 lines of code, including ccl files, includes, and whitespace) 2) the included parfiles run on a single desktop computer needing only 5GB of RAM 3) it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if the
nan values come from there? Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:22 AM, Frank Loeffler knarf@cct.lsu.edu wrote:
On Tue, Jun 02, 2015 at 11:11:15AM -0400, Zach Etienne wrote:
myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
You use the ADMBase variables, right? They should, by default, be prolongated, and I don't see a parameter that would disable that in your parfile. Could you output the variables your analysis depends on, and see if the nan values come from there? It does, like Ian suggested, look like something you depend on isn't properly prolonged by regridding.
Frank
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn at all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only 5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)
That tells you that alp is defined at the point it is output, but it
doesn't tell you what its state was at the time myadmbaselapse used it.
You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.
Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn at
all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only
5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if
the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting
myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
-- Ian Hinder http://members.aei.mpg.de/ianhin
On 2 Jun 2015, at 18:19, Zach Etienne zachetie@gmail.com wrote:
Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it.
You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.
Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...
Could it be that restriction is happening after the variable is set in ANALYSIS?
Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn at all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only 5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
-- Ian Hinder http://members.aei.mpg.de/ianhin
Hi Ian,
Can you experiment to see what happens if you schedule your copying
routine in POSTRESTRICT? And in POSTREGRID?
Yes. I just performed the following experiments: 1) Schedule myadmbaselapse=alp function in POSTRESTRICT (GLOBAL, LOOP-LOCAL), every 64 iterations 2) Schedule myadmbaselapse=alp function in POSTREGRID (GLOBAL, LOOP-LOCAL), every 64 iterations 3) Schedule myadmbaselapse=alp function in both POSTRESTRICT and POSTRESTRICT (both GLOBAL, LOOP-LOCAL), every 64 iterations
(1) and (3) resulted in no undefined values! In (2), undefined values remained.
There remains the mystery of why calling my function in CCTK_ANALYSIS at *every* iteration results in no undefined values...
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 12:30 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 18:19, Zach Etienne zachetie@gmail.com wrote:
Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)
That tells you that alp is defined at the point it is output, but it
doesn't tell you what its state was at the time myadmbaselapse used it.
You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.
Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...
Could it be that restriction is happening after the variable is set in ANALYSIS?
Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn
at all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only
5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if
the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester, setting
myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
-- Ian Hinder http://members.aei.mpg.de/ianhin
-- Ian Hinder http://members.aei.mpg.de/ianhin
Oh, and just to be clear, all of these three experiments kept the scheduling in CCTK_ANALYSIS intact and unchanged.
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 2:09 PM, Zach Etienne zachetie@gmail.com wrote:
Hi Ian,
Can you experiment to see what happens if you schedule your copying
routine in POSTRESTRICT? And in POSTREGRID?
Yes. I just performed the following experiments:
- Schedule myadmbaselapse=alp function in POSTRESTRICT (GLOBAL,
LOOP-LOCAL), every 64 iterations 2) Schedule myadmbaselapse=alp function in POSTREGRID (GLOBAL, LOOP-LOCAL), every 64 iterations 3) Schedule myadmbaselapse=alp function in both POSTRESTRICT and POSTRESTRICT (both GLOBAL, LOOP-LOCAL), every 64 iterations
(1) and (3) resulted in no undefined values! In (2), undefined values remained.
There remains the mystery of why calling my function in CCTK_ANALYSIS at *every* iteration results in no undefined values...
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 12:30 PM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 18:19, Zach Etienne zachetie@gmail.com wrote:
Thanks for getting back to me, Ian, and thanks for agreeing this does not appear to be a trivial problem. (I've been battling it on and off for days.)
That tells you that alp is defined at the point it is output, but it
doesn't tell you what its state was at the time myadmbaselapse used it.
You are absolutely right. I just added a printf() statement outputting ADMBase::alp at iteration 192, at all points on the z=0 plane, inside the ADMBaseMcLachlanTester loop that sets myadmbaselapse=alp. Then I re-did the run with ADMBaseMcLachlanTester. After analyzing the output, I can confirm that there are no undefined (nan) values at iteration 192, and alp values from this printf() statement lie in the very reasonable range of 0.25 < alp < 0.995 for all points. Again, this is inconsistent with the data I observe in the myadmbaselapse IOASCII 2D file in the z=0 plane, where many undefined values are seen.
Thus I conclude that something bad is happening to myadmbaselapse after being set at CCTK_ANALYSIS in GLOBAL,LOOP-LOCAL mode at iteration 192, but before the data are written to file at the same iteration, causing perfectly reasonable values to suddenly become undefined. For some reason, the same behavior is not observed in ADMBase::alp...
Could it be that restriction is happening after the variable is set in ANALYSIS?
Can you experiment to see what happens if you schedule your copying routine in POSTRESTRICT? And in POSTREGRID?
-Zach
Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 11:46 AM, Ian Hinder ian.hinder@aei.mpg.de wrote:
On 2 Jun 2015, at 17:40, Zach Etienne zachetie@gmail.com wrote:
Hi Frank,
Thanks for your feedback.
You use the ADMBase variables, right?
My ADMBaseMcLachlanTester thorn (
math.wvu.edu/~zetienne/ADMBaseMcLachlanTester.tar.gz) reproduces this issue, and uses *only* ADMBase::alp as input. I would encourage everyone interested to take a look at this thorn, as
- there's nothing complicated about the ADMBaseMcLachlanTester thorn
at all (67 lines of code, including ccl files, includes, and whitespace)
- the included parfiles run on a single desktop computer needing only
5GB of RAM
- it takes only ~15 mins to evolve to timestep 192
Could you output the variables your analysis depends on, and see if
the nan values come from there?
Absolutely! I just performed a run with ADMBaseMcLachlanTester,
setting myadmbaselapse to alp every 64 iterations, and confirmed that there are *no* undefined (nan) values in the (IOASCII 2D output of) ADMBase::alp (at any iteration, upto and including iteration 192), while again, there are a very large number of undefined values in myadmbaselapse at iteration 192...
So myadmbaselapse is not undefined because ADMBase::alp is undefined.
That tells you that alp is defined at the point it is output, but it doesn't tell you what its state was at the time myadmbaselapse used it. I think you can add a call to CCTK_OutputVarAs or something in the routine, and you will get an output at that point. http://einsteintoolkit.org/documentation/ReferenceManual/ReferenceManualch2....
So maybe something like
CCTK_OutputVarAs(cctkGH, "ADMBaseMcLachlanTester::myadmbaselapse", "myadmbaselapse_when_set")
and you should get your 2D output at the point the variable is set. You could do the same for alp, to see what it is being set from. Or you could use printf.
-- Ian Hinder http://members.aei.mpg.de/ianhin
-- Ian Hinder http://members.aei.mpg.de/ianhin
On 2 Jun 2015, at 17:11, Zach Etienne zachetie@gmail.com wrote:
Thanks for your quick feedback, Ian!
Think about the contents of the three timelevels of myadmbaselapse.
myadmbaselapse has only one timelevel, and tags are set 'InterpNumTimelevels=1 prolongation="none" Checkpoint="no"'. Thus prolongation should be disabled on myadmbaselapse...
Just prior to the output at, e.g., iteration 192 (the iteration when the grid movement occurs), myadmbaselapse is set to ADMBase::alp, at all refinement levels (due to GLOBAL,LOOP-LOCAL). Therefore, it should not matter how often I choose to update myadmbaselapse, since the value just prior to the file output (at iteration 192, for instance) is updated in a GLOBAL,LOOP-LOCAL mode. So why do I see vastly different results when I choose to update myadmbaselapse every cctk_iteration versus every 64?
Pfft. And I thought it was going to be easy... Have you checked the logic for "updating every 64"? Depending on where you schedule, cctk_iteration might be one different to a multiple of 64. Though I thought that this is only off-by-one in EVOL, and since you are scheduling in ANALYSIS, it should be correct. If you add a print statement to the updating routine, can you verify that it is being called on the correct refinement level at the iterations you are expecting?
This off-by-one has always confused me, but Erik explained it at one point and it made sense. I think it was needed for having a monotonically-increasing iteration counter.
Another thought: you said that the finest grid looked ok, but the coarser grids had problems. This points in the direction of restriction, which is not called on the finest grid. Which region of the coarse grid looks wrong? Could it be the points which are restricted, or are within a few points of the refinement boundary? I believe restriction is also not called for routines in ANALYSIS.
What happens if you schedule your calculation routine in POSTREGRID? And maybe POSTRESTRICT?
From the schedule output, it looks like regridding happens before evolution. You have told it not to prolongate, so maybe this has the effect of not prolongating during regridding. Hence the values will be uninitialised after regridding. After REGRID, POSTREGRID is run, in which McLachlan updates ADMBase::alp from ML_BSSN::alpha. I believe that if you have regrid_every = 64, then the grids are modified when performing the EVOL step which produces the data for iteration 64. Could it be that your routine is called when the ADMBase data is wrong? It looks like it would be helpful to find out whether your variable is correct when you set it, and then corrupted afterwards, or whether it is being set wrongly in the first place, e.g. because the ADMBase lapse is bad at that time.
I'm not really sure what the problem is, but maybe some of the ideas above will help? Maybe this is related to the order in which refinement levels are traversed in POSTRESTRICT. We had trouble with that before. I'm not an expert in the scheduling modes. What exactly does GLOBAL,LOOP-LOCAL mean?
Hello all,
Another thought: you said that the finest grid looked ok, but the coarser grids had problems. This points in the direction of restriction, which is not called on the finest grid. Which region of the coarse grid looks wrong? Could it be the points which are restricted, or are within a few points of the refinement boundary? I believe restriction is also not called for routines in ANALYSIS.
Not sure if this is what happens to you but I got caught by this once:
If you are scheduling everything in GLOBAL (which is the same as GLOBAL-LATE in ANALYSIS) then your OUTPUT will be incorrect since OUTPUT is happening in LEVEL mode thus refinement level 3 is output before the GLOBAL routine (which runs along with level 0) is executed. So your output is wrong but the data that the simulation sees would actually correct. An easy fix is to use GLOBAL-EARLY instead of GLOBAL.
So I would add print statements or the like (or Ian's suggestion of calling OutputVarAsByMethod) to get output directly out of the loop-local routine (actually it may have to be loop-level though I am not sure).
Yours, Roland
Hello all,
If you are scheduling everything in GLOBAL (which is the same as GLOBAL-LATE in ANALYSIS) then your OUTPUT will be incorrect since OUTPUT is happening in LEVEL mode thus refinement level 3 is output before the GLOBAL routine (which runs along with level 0) is executed.
Oha, obvious not the right statement. GLOBAL-LATE will happen last but this happens to be the highest refinement level. Still the effect is the same OUTPUT happens before GLOBAL-LATE for all but one refinement level (that refinement level being the finest one).
This can be seen in Carpet's Evolve.cc file in the CallAnalysis routine.
Yours, Roland
Hi Roland,
Thanks for your feedback! I just confirmed that simply changing the scheduling from GLOBAL,LOOP-LOCAL to GLOBAL-EARLY,LOOP-LOCAL every 64 iterations fixes the problem. The output also seems to agree with the case in which the myadmbaselapse=alp function is scheduled in CCTK_POSTRESTRICT (and CCTK_ANALYSIS), every 64 iterations.
I think I'll use this strategy to fix my problem, since it does not involve scheduling an additional function inside schedule.ccl.
Is it obvious to you why I do not see any undefined values when scheduling in CCTK_ANALYSIS at every iteration?
-Zach
* * * Zachariah Etienne Assistant Professor of Mathematics West Virginia University
On Tue, Jun 2, 2015 at 1:46 PM, Roland Haas rhaas@aei.mpg.de wrote:
Hello all,
If you are scheduling everything in GLOBAL (which is the same as GLOBAL-LATE in ANALYSIS) then your OUTPUT will be incorrect since OUTPUT is happening in LEVEL mode thus refinement level 3 is output before the GLOBAL routine (which runs along with level 0) is executed.
Oha, obvious not the right statement. GLOBAL-LATE will happen last but this happens to be the highest refinement level. Still the effect is the same OUTPUT happens before GLOBAL-LATE for all but one refinement level (that refinement level being the finest one).
This can be seen in Carpet's Evolve.cc file in the CallAnalysis routine.
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
users@lists.einsteintoolkit.org
-
Frank Loeffler -
Ian Hinder -
Roland Haas -
Zach Etienne