Hi. I doing runs on Nautilus with some rather large grids, which are not being broken up on more than just a few (2 to 8) MPI processes.
I'm noticing that there's a grid size of about 385^3 which I can't exceed, without Cactus/Carpet beginning to behave strangely, e.g. AHFinderDirect no longer works, and memory errors abound.
Obviously there's the possibility that I have errors in the way I specify the grids in my parameter files. But I'm wondering:
Have other people had success doing "very" large runs on, say, a shared memory architecture where MPI / domain decomposition are not employed? If so, than I've probably just made a mistake in my par file. If not...
I'm wondering if this has to do with trouble address memory locations, and I notice that the type "int" is used in places, e.g. in the definition of CFINDEX3D,..
static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k); static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k) { return (i + GH->cctk_lsh[0]*(j + GH->cctk_lsh[1]*k)); }
Should these "int" declarations be changed to "long" or "unsigned int" in order to access larger numbers?
Thanks.
Scott
One critical number is the number of grid points per MPI process. This number cannot exceed about 1200^3. Another critical number is the linear size of the coarse grid multiplied by 2^(maxreflevels). This cannot exceed about 1,000,000,000, but since this includes a 2^n term, it can easily grow very large. Cactus and Carpet contain safeguards that are supposed to catch any such problems. Are you building with debug options? Those may slow down execution, but may also help find such errors.
What is your Carpet::maxreflevels setting?
-erik
On Wed, Apr 11, 2012 at 2:12 AM, Scott Hawley scott.hawley@belmont.edu wrote:
Hi. I doing runs on Nautilus with some rather large grids, which are not being broken up on more than just a few (2 to 8) MPI processes.
I'm noticing that there's a grid size of about 385^3 which I can't exceed, without Cactus/Carpet beginning to behave strangely, e.g. AHFinderDirect no longer works, and memory errors abound.
Obviously there's the possibility that I have errors in the way I specify the grids in my parameter files. But I'm wondering:
Have other people had success doing "very" large runs on, say, a shared memory architecture where MPI / domain decomposition are not employed? If so, than I've probably just made a mistake in my par file. If not...
I'm wondering if this has to do with trouble address memory locations, and I notice that the type "int" is used in places, e.g. in the definition of CFINDEX3D,..
static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k); static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k) { return (i + GH->cctk_lsh[0]*(j + GH->cctk_lsh[1]*k)); }
Should these "int" declarations be changed to "long" or "unsigned int" in order to access larger numbers?
Thanks.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik, I'm not setting maxreflevels. What is the default? Seems like it's something like 9, based on my unigrid runs.
I notice that in Carpet/Carpet/param.ccl, it looks like there's a default of 1...
CCTK_INT max_refinement_levels "Maximum number of refinement levels (including the base level)" { 1:* :: "must be positive" } 1
Šbut this doesn't explain why there's a stride of 512 for a unigrid run I'm trying to doŠ
Thanks, Scott
On 4/11/12 7:38 AM, "Erik Schnetter" schnetter@cct.lsu.edu wrote:
Scott
One critical number is the number of grid points per MPI process. This number cannot exceed about 1200^3. Another critical number is the linear size of the coarse grid multiplied by 2^(maxreflevels). This cannot exceed about 1,000,000,000, but since this includes a 2^n term, it can easily grow very large. Cactus and Carpet contain safeguards that are supposed to catch any such problems. Are you building with debug options? Those may slow down execution, but may also help find such errors.
What is your Carpet::maxreflevels setting?
-erik
On Wed, Apr 11, 2012 at 2:12 AM, Scott Hawley scott.hawley@belmont.edu wrote:
Hi. I doing runs on Nautilus with some rather large grids, which are not being broken up on more than just a few (2 to 8) MPI processes.
I'm noticing that there's a grid size of about 385^3 which I can't exceed, without Cactus/Carpet beginning to behave strangely, e.g. AHFinderDirect no longer works, and memory errors abound.
Obviously there's the possibility that I have errors in the way I specify the grids in my parameter files. But I'm wondering:
Have other people had success doing "very" large runs on, say, a shared memory architecture where MPI / domain decomposition are not employed? If so, than I've probably just made a mistake in my par file. If not...
I'm wondering if this has to do with trouble address memory locations, and I notice that the type "int" is used in places, e.g. in the definition of CFINDEX3D,..
static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k); static inline int CCTK_GFINDEX3D (const cGH *GH, int i, int j, int k) { return (i + GH->cctk_lsh[0]*(j + GH->cctk_lsh[1]*k)); }
Should these "int" declarations be changed to "long" or "unsigned int" in order to access larger numbers?
Thanks.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Scott Hawley