My code works totally ok with MPI on my linux machine, but as I'm porting to my Mac, I find that MPI isn't working right. I'm using the OpenMPI version from fink) A simple "hello world" MPI program works fine (proof at the bottom of this message), but Cactus does not.
That is to say, e.g. an 8-processor run ("/sw/bin om-mpirun -np 8 ....") results in 8 processes, but each processes thinks it's processor 0, and that there are no other processors.
Any ideas why this might happen? Thanks, Scott
I'm using Cactus version 4.0.b17 (Compile date: May 20 2011 (11:13:49)).
Cactus run:
mpirun -np 8 ~/Cactus/exe/cactus_texmex sep20a0.5fmrbbh97_cactus.par
--------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) --------------------------------------------------------------------------------
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
--------------------------------------------------------------------------------
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6067) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par -------------------------------------------------------------------------------- Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6071) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par --------------------------------------------------------------------------------
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6070) Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6074) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par --------------------------------------------------------------------------------
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6073) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par -------------------------------------------------------------------------------- Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par --------------------------------------------------------------------------------
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6068) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6069) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6072) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
....and so on.....
Yet a imple program works: The simple example which works fine is...
cat hello.c
#include <stdio.h> #include "mpi.h"
int main(argc, argv) int argc; char **argv; { int rank, size; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am %d of %d\n",rank,size); MPI_Finalize(); return 0; }
mpicc -o hello hello.c which mpirun
/sw/bin/mpirun
ls -l /sw/bin/mpirun
lrwxr-xr-x 1 root shawley 9 May 20 10:55 /sw/bin/mpirun@ -> om-mpirun
mpirun -np 8 ./hello
Hello world! I am 0 of 8 Hello world! I am 6 of 8 Hello world! I am 3 of 8 Hello world! I am 1 of 8 Hello world! I am 2 of 8 Hello world! I am 4 of 8 Hello world! I am 5 of 8 Hello world! I am 7 of 8
Scott
This usually means that the libraries used to build the executable and the mpirun command don't belong together. In other words, either the libraries used for building, or the shared libraries used at run time (if you use shared libraries), or the mpirun executable are not what you think they are.
make texmex SILENT=no shows you (at the end) which libraries are used during the build.
ldd exe/cactus_texmex (or otool -L exe/cactus_texmex) shows which shared libraries will be used. Certain environment variables influence this; make sure you use the same settings that you use when you call mpirun.
"which mpirun" shows which mpirun will be used.
-erik
On Fri, May 20, 2011 at 12:36 PM, Scott Hawley scott.hawley@belmont.edu wrote:
My code works totally ok with MPI on my linux machine, but as I'm porting to my Mac, I find that MPI isn't working right. I'm using the OpenMPI version from fink) A simple "hello world" MPI program works fine (proof at the bottom of this message), but Cactus does not.
That is to say, e.g. an 8-processor run ("/sw/bin om-mpirun -np 8 ....") results in 8 processes, but each processes thinks it's processor 0, and that there are no other processors.
Any ideas why this might happen? Thanks, Scott
I'm using Cactus version 4.0.b17 (Compile date: May 20 2011 (11:13:49)).
Cactus run:
mpirun -np 8 ~/Cactus/exe/cactus_texmex sep20a0.5fmrbbh97_cactus.par
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6067) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6071) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6070) Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6074) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6073) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6068) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6069) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6072) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
....and so on.....
Yet a imple program works: The simple example which works fine is...
cat hello.c
#include <stdio.h> #include "mpi.h"
int main(argc, argv) int argc; char **argv; { int rank, size; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am %d of %d\n",rank,size); MPI_Finalize(); return 0; }
mpicc -o hello hello.c which mpirun
/sw/bin/mpirun
ls -l /sw/bin/mpirun
lrwxr-xr-x 1 root shawley 9 May 20 10:55 /sw/bin/mpirun@ -> om-mpirun
mpirun -np 8 ./hello
Hello world! I am 0 of 8 Hello world! I am 6 of 8 Hello world! I am 3 of 8 Hello world! I am 1 of 8 Hello world! I am 2 of 8 Hello world! I am 4 of 8 Hello world! I am 5 of 8 Hello world! I am 7 of 8
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Got it to build and run! See below for config "options" Yes, it is complicated on the Mac because Apple ships its own messed up version of MPI which you have to be careful not to use, and Apple uses the gnu v4.2.1 compiler but in order to get a fortran 90 compiler you need the straight-GNU compiler which for me is v4.4
This, then, is my "options.mac" file, which produces a complete build of my thorn and runs properly with MPI. Consider using it for a "known architectures" file. The /sw directory is where fink puts things. One needs to get gfortran and openmpi from fink. Some of the libraries I'm using will not be necessary for all people, but I'll let you take them out. This works for me! ...Oh, that is to say, with all the "CarpetIO..." thorns not activated. Haven't gotten those working yet.
% cat options.mac CC = /sw/bin/mpicc CXX = /sw/bin/mpic++ F77 = /sw/bin/mpif77 F90 = gfortran CFLAGS = -m64 -std=c99 CXXFLAGS = -m64 FFLAGS = -m64 F90FLAGS = -m64 LIBS = texmex bbhutil sv mfhdf df jpeg z gfortran LDFLAGS = -L/sw/lib -L/sw/lib/gcc4.4/lib/ -L/Users/shawley/lib -L/Users/shawley/texmex/lib MPI = OPENMPI OPENMPI_DIR = /sw TEXMEX_DIR = /Users/shawley/texmex
On May 20, 2011, at 11:43 AM, Erik Schnetter wrote:
Scott
This usually means that the libraries used to build the executable and the mpirun command don't belong together. In other words, either the libraries used for building, or the shared libraries used at run time (if you use shared libraries), or the mpirun executable are not what you think they are.
make texmex SILENT=no shows you (at the end) which libraries are used during the build.
ldd exe/cactus_texmex (or otool -L exe/cactus_texmex) shows which shared libraries will be used. Certain environment variables influence this; make sure you use the same settings that you use when you call mpirun.
"which mpirun" shows which mpirun will be used.
-erik
On Fri, May 20, 2011 at 12:36 PM, Scott Hawley scott.hawley@belmont.edu wrote:
My code works totally ok with MPI on my linux machine, but as I'm porting to my Mac, I find that MPI isn't working right. I'm using the OpenMPI version from fink) A simple "hello world" MPI program works fine (proof at the bottom of this message), but Cactus does not.
That is to say, e.g. an 8-processor run ("/sw/bin om-mpirun -np 8 ....") results in 8 processes, but each processes thinks it's processor 0, and that there are no other processors.
Any ideas why this might happen? Thanks, Scott
I'm using Cactus version 4.0.b17 (Compile date: May 20 2011 (11:13:49)).
Cactus run:
mpirun -np 8 ~/Cactus/exe/cactus_texmex sep20a0.5fmrbbh97_cactus.par
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6067) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6071) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6070) Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6074) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6073) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6068) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6069) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6072) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
....and so on.....
Yet a imple program works: The simple example which works fine is...
cat hello.c
#include <stdio.h> #include "mpi.h"
int main(argc, argv) int argc; char **argv; { int rank, size; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am %d of %d\n",rank,size); MPI_Finalize(); return 0; }
mpicc -o hello hello.c which mpirun
/sw/bin/mpirun
ls -l /sw/bin/mpirun
lrwxr-xr-x 1 root shawley 9 May 20 10:55 /sw/bin/mpirun@ -> om-mpirun
mpirun -np 8 ./hello
Hello world! I am 0 of 8 Hello world! I am 6 of 8 Hello world! I am 3 of 8 Hello world! I am 1 of 8 Hello world! I am 2 of 8 Hello world! I am 4 of 8 Hello world! I am 5 of 8 Hello world! I am 7 of 8
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Well, there does happen to be an MPI_Finalize "train wreck" at the end, but by that point I've already got my data:
[0] Now copying physical spacetime into Cactus data structure INFO (TexMEx): Done with TexMEx INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (ADMMacros): Spatial finite differencing order: 4 INFO (AHFinderDirect): proc 0: searching for horizon 1/2 INFO (AHFinderDirect): AH 1/2: r=1.99593 at (-4.980572,0.000204,0.000000) INFO (AHFinderDirect): AH 1/2: area=54.13981184 m_irreducible=1.037823364 INFO (AHFinderDirect): AH 2/2: r=1.99593 at (4.980572,-0.000204,-0.000000) INFO (AHFinderDirect): AH 2/2: area=54.13981184 m_irreducible=1.037823364 =================================================================================================== Thorn | Scheduled routine in time bin | gettimeofday [secs] | getrusage [secs] =================================================================================================== *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) Carpet | Multi-model Startup routine | 0.00026100 | 0.00015600 Carpet | Startup routine | 0.00001000 | 0.00000900 AEILocalInterp | register CCTK_InterpLocalUniform() inter| 0.00000500 | 0.00000300 CarpetInterp | Startup routine | 0.00000100 | 0.00000100 CarpetReduce | Startup routine | 0.00004000 | 0.00003000 CartGrid3D | Register GH Extension for GridSymmetry | 0.00000500 | 0.00000200 CoordBase | Register a GH extension to store the coo| 0.00000200 | 0.00000200 IOUtil | Startup routine | 0.00000500 | 0.00000200 LocalInterp | register LocalInterp's interpolation ope| 0.00000200 | 0.00000100 [provision.local:91413] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! Slab | Create MPI datatypes for complex variabl| 0.00000300 | 0.00000300 SymBase | Register GH Extension for SymBase | 0.00000200 | 0.00000100 --------------------------------------------------------------------------------------------------- | Total time for CCTK_STARTUP | 0.00033600 | 0.00021000 =================================================================================================== ADMAnalysis | Register symmetry of Ricci tensor and sc| 0.00008000 | 0.00007100 ADMBase | Set up GF symmetries | 0.00008000 | 0.00007700 ADMConstraints | Register GF symmetries for ADM Constrain| 0.00023600 | 0.00022700 Boundary | Register boundary conditions that this t| 0.00001700 | 0.00001100 CartGrid3D | Register coordinates for the Cartesian g| 0.00010700 | 0.00009800 CoordGauge | Initialize slicing, setup priorities for| 0.00001300 | 0.00000800 CoordGauge | Identify the slicing for the next iterat| 0.00000900 | 0.00000600 Slab | Initialise timers | 0.00003400 | 0.00002400 SpaceMask | Set grid symmetries for mask | 0.00001500 | 0.00001000 SpaceMask | Set grid symmetries for emask (compatibi| 0.00001100 | 0.00000800 CartGrid3D | Register symmetry boundaries | 0.00006500 | 0.00006000 SymBase | Print symmetry boundary face description| 0.00000900 | 0.00000500 --------------------------------------------------------------------------------------------------- | Total time for CCTK_WRAGH | 0.00067600 | 0.00060500 =================================================================================================== ADMAnalysis | Check that the metric_type is recognised| 0.00111600 | 0.00003700 ADMBase | Check consistency of parameters | 0.00002600 | 0.00002200 ADMConstraints | Check that we can deal with this metric_| 0.00004200 | 0.00003700 Boundary | Check dimension of grid variables | 0.00000600 | 0.00000300 CarpetRegrid | Check Parameters | 0.00001100 | 0.00000700 CartGrid3D | Check coordinates for CartGrid3D | 0.00000900 | 0.00000600 --------------------------------------------------------------------------------------------------- | Total time for CCTK_PARAMCHECK | 0.00121000 | 0.00011200 =================================================================================================== ADMBase | Set the shift_state variable to 0 | 0.00042900 | 0.00040700 ADMBase | Set the dtlapse_state variable to 0 | 0.00019400 | 0.00017600 ADMBase | Set the dtshift_state variable to 0 | 0.00019000 | 0.00017400 ADMMacros | Initialize the local_spatial_order | 0.00021500 | 0.00011600 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) CartGrid3D | Set up ranges for spatial 3D Cartesian c| 0.00033800 | 0.00018600 CarpetReduce | Initialise the weight function | 0.00680000 | 0.00395200 CarpetReduce | Set up the outer boundaries of the weigh| 0.00042400 | 0.00040600 CarpetReduce | Set up the weight function for the restr| 0.03050000 | 0.02837400 CartGrid3D | Set up spatial 3D Cartesian coordinates | 0.06041500 | 0.03934700 AHFinderDirect | setup data structures | 0.00344000 | 0.00232800 SpaceMask | Initialise mask to zero | 0.00354400 | 0.00203900 SpaceMask | Set old style mask to one | 0.01313800 | 0.00262400 SphericalSurface| Set surface resolution automatically | 0.00100600 | 0.00095500 SphericalSurface| Calculate surface coordinate descriptors| 0.00006900 | 0.00005600 SphericalSurface| Set surface radii to be used for initial| 0.00006100 | 0.00005100 SphericalSurface| Test the state of the spherical surfaces| 0.00006000 | 0.00005000 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91410] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! SymBase | Check whether the driver set up the grid| 0.00029900 | 0.00028200 --------------------------------------------------------------------------------------------------- | Total time for CCTK_BASEGRID | 0.12112200 | 0.08152300 =================================================================================================== StaticConformal | Set the conformal_state variable to 0 | 0.00016800 | 0.00014800 *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) TexMEx | Calculate initial data with TexMEx | 110.10679400 | 106.53449800 ADMBase | Set the lapse to 1 at all points | 0.01679100 | 0.00792400 --------------------------------------------------------------------------------------------------- | Total time for CCTK_INITIAL | 110.12375300 | 106.54257000 =================================================================================================== ADMConstraints | Set conformal_state to 0 for physical, o| 0.00057300 | 0.00055000 LegoExcision | Excise a fixed lego sphere | 0.02932700 | 0.02850400 Boundary | Apply all requested local physical bound| 0.00040000 | 0.00038300 CartGrid3D | Apply symmetry boundary conditions | 0.00015300 | 0.00013100 Boundary | Unselect all grid variables for boundary| 0.00002800 | 0.00001100 --------------------------------------------------------------------------------------------------- | Total time for CCTK_POSTINITIAL | 0.03048100 | 0.02957900 =================================================================================================== AHFinderDirect | import the excision mask | 0.01030700 | 0.00921100 AHFinder[provision.local:91409] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! [provision.local:91412] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! Direct | find apparent horizon(s) after this time| 1.25908800 | 1.24079600 AHFinderDirect | store apparent horizon(s) into spherical| 0.00631300 | 0.00625600 AHFinderDirect | save apparent horizon(s) into Cactus var| 0.00028400 | 0.00025300 AHFinderDirect | set mask(s) based on apparent horizon po| 0.00053800 | 0.00051900 SpaceMask | Ensure that all mask values are legal | 0.00377300 | 0.00367000 SphericalSurface| Set surface radii | 0.00006300 | 0.00005100 SphericalSurface| Test the state of the spherical surfaces| 0.00006200 | 0.00004900 --------------------------------------------------------------------------------------------------- | Total time for CCTK_POSTSTEP | 1.28042800 | 1.26080500 =================================================================================================== CarpetLib [provision.local:91411] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91415] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! | Print timing statistics if desired | 0.00003200 | 0.00001700 CarpetLib | Print memory statistics if desired | 0.00002800 | 0.00001400 --------------------------------------------------------------------------------------------------- | Total time for CCTK_ANALYSIS | 0.00006000 | 0.00003100 =================================================================================================== --------------------------------------------------------------------------------------------------- | Total time for simulation | 112.41905900 | 108.55538600 =================================================================================================== -------------------------------------------------------------------------------- Done. *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91408] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed! *** An error occurred in MPI_Finalize *** after MPI was finalized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [provision.local:91414] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 20 May 2011, at 19:08, Scott Hawley wrote:
Got it to build and run! See below for config "options" Yes, it is complicated on the Mac because Apple ships its own messed up version of MPI which you have to be careful not to use, and Apple uses the gnu v4.2.1 compiler but in order to get a fortran 90 compiler you need the straight-GNU compiler which for me is v4.4
I wrote a simfactory optionlist for using 64 bit fink on Mac OS with Cactus. See
https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010/mdb/opti...
This seems to only be in the python version of simfactory - we could put it in the perl version as well. There are full instructions at the top of the file for the commands needed to get it working, and you don't need simfactory to use the optionlist. Just replace the @..@ tokens with something sensible.
This, then, is my "options.mac" file, which produces a complete build of my thorn and runs properly with MPI. Consider using it for a "known architectures" file. The /sw directory is where fink puts things. One needs to get gfortran and openmpi from fink. Some of the libraries I'm using will not be necessary for all people, but I'll let you take them out. This works for me! ...Oh, that is to say, with all the "CarpetIO..." thorns not activated. Haven't gotten those working yet.
% cat options.mac CC = /sw/bin/mpicc CXX = /sw/bin/mpic++ F77 = /sw/bin/mpif77 F90 = gfortran CFLAGS = -m64 -std=c99 CXXFLAGS = -m64 FFLAGS = -m64 F90FLAGS = -m64 LIBS = texmex bbhutil sv mfhdf df jpeg z gfortran LDFLAGS = -L/sw/lib -L/sw/lib/gcc4.4/lib/ -L/Users/shawley/lib -L/Users/shawley/texmex/lib MPI = OPENMPI OPENMPI_DIR = /sw TEXMEX_DIR = /Users/shawley/texmex
On May 20, 2011, at 11:43 AM, Erik Schnetter wrote:
Scott
This usually means that the libraries used to build the executable and the mpirun command don't belong together. In other words, either the libraries used for building, or the shared libraries used at run time (if you use shared libraries), or the mpirun executable are not what you think they are.
make texmex SILENT=no shows you (at the end) which libraries are used during the build.
ldd exe/cactus_texmex (or otool -L exe/cactus_texmex) shows which shared libraries will be used. Certain environment variables influence this; make sure you use the same settings that you use when you call mpirun.
"which mpirun" shows which mpirun will be used.
-erik
On Fri, May 20, 2011 at 12:36 PM, Scott Hawley scott.hawley@belmont.edu wrote:
My code works totally ok with MPI on my linux machine, but as I'm porting to my Mac, I find that MPI isn't working right. I'm using the OpenMPI version from fink) A simple "hello world" MPI program works fine (proof at the bottom of this message), but Cactus does not.
That is to say, e.g. an 8-processor run ("/sw/bin om-mpirun -np 8 ....") results in 8 processes, but each processes thinks it's processor 0, and that there are no other processors.
Any ideas why this might happen? Thanks, Scott
I'm using Cactus version 4.0.b17 (Compile date: May 20 2011 (11:13:49)).
Cactus run:
mpirun -np 8 ~/Cactus/exe/cactus_texmex sep20a0.5fmrbbh97_cactus.par
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6067) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6071) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6070) Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6074) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6073) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6068) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6069) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6072) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
....and so on.....
Yet a imple program works: The simple example which works fine is...
cat hello.c
#include <stdio.h> #include "mpi.h"
int main(argc, argv) int argc; char **argv; { int rank, size; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am %d of %d\n",rank,size); MPI_Finalize(); return 0; }
mpicc -o hello hello.c which mpirun
/sw/bin/mpirun
ls -l /sw/bin/mpirun
lrwxr-xr-x 1 root shawley 9 May 20 10:55 /sw/bin/mpirun@ -> om-mpirun
mpirun -np 8 ./hello
Hello world! I am 0 of 8 Hello world! I am 6 of 8 Hello world! I am 3 of 8 Hello world! I am 1 of 8 Hello world! I am 2 of 8 Hello world! I am 4 of 8 Hello world! I am 5 of 8 Hello world! I am 7 of 8
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
- -- Ian Hinder ian.hinder@aei.mpg.de
Great. Thanks Ian. I haven't tried simfactory yet -- I've been a bit out of the loop --- I was just noting that the latest version of Cactus I downloaded a couple days ago said that I was on a new architechture
I'll take a look at simfactory.
-- Scott H. Hawley, Ph.D. Asst. Prof. of Physics Chemistry & Physics Dept Office: Hitch 100D Belmont University Tel: +1-615-460-6206 Nashville, TN 37212 USA Fax: +1-615-460-5458 PGP Key at http://sks-keyservers.net
On May 21, 2011, at 4:51 AM, Ian Hinder wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 20 May 2011, at 19:08, Scott Hawley wrote:
Got it to build and run! See below for config "options" Yes, it is complicated on the Mac because Apple ships its own messed up version of MPI which you have to be careful not to use, and Apple uses the gnu v4.2.1 compiler but in order to get a fortran 90 compiler you need the straight-GNU compiler which for me is v4.4
I wrote a simfactory optionlist for using 64 bit fink on Mac OS with Cactus. See
https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010/mdb/optionlists/macos-fink-gcc.cfgThis seems to only be in the python version of simfactory - we could put it in the perl version as well. There are full instructions at the top of the file for the commands needed to get it working, and you don't need simfactory to use the optionlist. Just replace the @..@ tokens with something sensible.
This, then, is my "options.mac" file, which produces a complete build of my thorn and runs properly with MPI. Consider using it for a "known architectures" file. The /sw directory is where fink puts things. One needs to get gfortran and openmpi from fink. Some of the libraries I'm using will not be necessary for all people, but I'll let you take them out. This works for me! ...Oh, that is to say, with all the "CarpetIO..." thorns not activated. Haven't gotten those working yet.
% cat options.mac CC = /sw/bin/mpicc CXX = /sw/bin/mpic++ F77 = /sw/bin/mpif77 F90 = gfortran CFLAGS = -m64 -std=c99 CXXFLAGS = -m64 FFLAGS = -m64 F90FLAGS = -m64 LIBS = texmex bbhutil sv mfhdf df jpeg z gfortran LDFLAGS = -L/sw/lib -L/sw/lib/gcc4.4/lib/ -L/Users/shawley/lib -L/Users/shawley/texmex/lib MPI = OPENMPI OPENMPI_DIR = /sw TEXMEX_DIR = /Users/shawley/texmex
On May 20, 2011, at 11:43 AM, Erik Schnetter wrote:
Scott
This usually means that the libraries used to build the executable and the mpirun command don't belong together. In other words, either the libraries used for building, or the shared libraries used at run time (if you use shared libraries), or the mpirun executable are not what you think they are.
make texmex SILENT=no shows you (at the end) which libraries are used during the build.
ldd exe/cactus_texmex (or otool -L exe/cactus_texmex) shows which shared libraries will be used. Certain environment variables influence this; make sure you use the same settings that you use when you call mpirun.
"which mpirun" shows which mpirun will be used.
-erik
On Fri, May 20, 2011 at 12:36 PM, Scott Hawley scott.hawley@belmont.edu wrote:
My code works totally ok with MPI on my linux machine, but as I'm porting to my Mac, I find that MPI isn't working right. I'm using the OpenMPI version from fink) A simple "hello world" MPI program works fine (proof at the bottom of this message), but Cactus does not.
That is to say, e.g. an 8-processor run ("/sw/bin om-mpirun -np 8 ....") results in 8 processes, but each processes thinks it's processor 0, and that there are no other processors.
Any ideas why this might happen? Thanks, Scott
I'm using Cactus version 4.0.b17 (Compile date: May 20 2011 (11:13:49)).
Cactus run:
mpirun -np 8 ~/Cactus/exe/cactus_texmex sep20a0.5fmrbbh97_cactus.par
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49)
101 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101
Cactus version: 4.0.b17 Compile date: May 20 2011 (11:13:49) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6067) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6071) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6070) Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6074) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6073) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
Activating thorn Cactus...Success -> active implementation Cactus
Activating thorn Cactus...Success -> active implementation Cactus Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6068) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6069) Run date: May 20 2011 (11:24:38-0500) Run host: provision.local (pid=6072) Working directory: /Users/shawley/runs/cactus Executable: /Users/shawley/Cactus/exe/cactus_texmex Parameter file: sep20a0.5fmrbbh97_cactus.par
....and so on.....
Yet a imple program works: The simple example which works fine is...
cat hello.c
#include <stdio.h> #include "mpi.h"
int main(argc, argv) int argc; char **argv; { int rank, size; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am %d of %d\n",rank,size); MPI_Finalize(); return 0; }
mpicc -o hello hello.c which mpirun
/sw/bin/mpirun
ls -l /sw/bin/mpirun
lrwxr-xr-x 1 root shawley 9 May 20 10:55 /sw/bin/mpirun@ -> om-mpirun
mpirun -np 8 ./hello
Hello world! I am 0 of 8 Hello world! I am 6 of 8 Hello world! I am 3 of 8 Hello world! I am 1 of 8 Hello world! I am 2 of 8 Hello world! I am 4 of 8 Hello world! I am 5 of 8 Hello world! I am 7 of 8
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Ian Hinder ian.hinder@aei.mpg.de
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAk3XiwYACgkQF1LN8Zj+Ceh5qwCfSuHFBt55qB7WGsLC+JCE6sdU PIAAn2+G3xAfON9rzfv7UTlcCu7IZ+0j =z2SW -----END PGP SIGNATURE-----
On Sat, May 21, 2011 at 4:12 PM, Scott Hawley scott.hawley@belmont.eduwrote:
Great. Thanks Ian. I haven't tried simfactory yet -- I've been a bit out of the loop --- I was just noting that the latest version of Cactus I downloaded a couple days ago said that I was on a new architechture
I'll take a look at simfactory.
If you prefer to use MacPorts, there is also an optionlist for that included with SimFactory. You can get it from:
https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010/mdb/opti...
Both this and the Fink optionlist can be used independently of SimFactory, although overall using SimFactory makes life easier.
Note that both also use the version of OpenMPI bundled with OSX which should be fine for most use cases, the most notable exception being if you need a Fortran interface to OpenMPI.
Barry
On 21 May 2011, at 18:44, Barry Wardell wrote:
On Sat, May 21, 2011 at 4:12 PM, Scott Hawley scott.hawley@belmont.edu wrote: Great. Thanks Ian. I haven't tried simfactory yet -- I've been a bit out of the loop --- I was just noting that the latest version of Cactus I downloaded a couple days ago said that I was on a new architechture
I'll take a look at simfactory.
If you prefer to use MacPorts, there is also an optionlist for that included with SimFactory. You can get it from:
https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010/mdb/opti...
Both this and the Fink optionlist can be used independently of SimFactory, although overall using SimFactory makes life easier.
Note that both also use the version of OpenMPI bundled with OSX which should be fine for most use cases, the most notable exception being if you need a Fortran interface to OpenMPI.
I would also recommend against using the python version of simfactory. In my opinion it is not yet ready for "prime-time".
In my opinion it is not yet ready for "prime-time".
Seems like. I followed the instructions for downloading the python version, and according to Eric's post, it seems to have given me the perl version instead.
%Sent from my iPhone
On May 21, 2011, at 12:35 PM, "Ian Hinder" <ian.hinder@aei.mpg.demailto:ian.hinder@aei.mpg.de> wrote:
In my opinion it is not yet ready for "prime-time".
users@lists.einsteintoolkit.org
-
Barry Wardell -
Erik Schnetter -
Ian Hinder -
Scott Hawley