git config --global core.preloadIndex false

This allowed me to run GetComponents without error

./GetComponents --verbose https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolkit.th

When I build

./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist ../einsteintoolkit.th

all looks well until, I think, the very end when some sort of git clean up seems to be happening.

Done creating cactus_sim.

fatal: bad revision 'HEAD'

Formaline: Committing source tree to git repository...

Formaline: Created git tag build-sim-comet-ln2.sdsc.edu-jhellmer-2018.09.03-17.30.20-18591

Formaline: Updated git branch config-sim-comet-ln2.sdsc.edu-home-jhellmer-etk-Cactus

Formaline: Optimising git repository (slow only the first time)...

Formaline: ERROR during: Garbage collecting git repo

Command was: /usr/bin/git --git-dir='/home/jhellmer/etk/Cactus/configs/sim/configjar.git/.git' gc at /home/jhellmer/etk/Cactus/configs/sim/scratch/Formaline/bin/update-git-repo.pl line 60, <STDIN> line 5090.

make[1]: *** [/home/jhellmer/etk/Cactus/configs/sim/build/Formaline/git-store-source] Error 2

make: *** [sim] Error 2

So I attempted to run the HelloWorld app, and it looks to be trying to submit a job but apparently fails

[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-submit helloworld --parfile arrangements/CactusExamples/HelloWorld/par/HelloWorld.par --walltime 0:5:0

Parameter file: /home/jhellmer/comet/Cactus/arrangements/CactusExamples/HelloWorld/par/HelloWorld.par

Skeleton Created

Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld"

Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim"

Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/cfg/OptionList"

Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/SubmitScript"

Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/RunScript"

Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/par/HelloWorld.par"

Assigned restart id: 0

Warning: Too many threads per process specified: specified num-threads=24 (ppn-used is 24)

Warning: Total number of threads and number of threads per process are inconsistent: procs=1, num-threads=24 (procs*num-smt must be an integer multiple of num-threads)

Warning: Total number of threads and number of cores per node are inconsistent: procs=1, ppn-used=24 (procs must be an integer multiple of ppn-used)

Executing submit command: sbatch /oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/output-0000/SIMFACTORY/SubmitScript; sleep 5

Warning: submit either failed or could not determine job id, output:

Warning: - Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

intel/2013_sp1.2.144 2017/08/21 20:08:02

mvapich2_ib/2.1 2016/01/20 0:58:02

gnutools/2.69 2016/01/16 0:30:50

hdf5/1.8.14 2016/03/16 19:35:34

netcdf/4.3.2 2016/01/21 22:17:20

ncview/2.1.7 2016/03/31 4:18:21

Unloading compiler-dependent module netcdf/4.3.2

must specify one of the following mpi modules: mvapich2_ib openmpi_ib

- Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

.intel/hdf5/1.8.14 2016/03/16 19:35:34

ncview/2.1.7 2016/03/31 4:18:21

- Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

hdf5/1.8.14 2016/03/16 19:35:34

ncview/2.1.7 2016/03/31 4:18:21

gnutools/2.69 2016/01/16 0:30:50

intel/2015.2.164 2017/08/21 20:08:02

mvapich2_ib/2.1 2016/01/20 0:58:02

fftw/3.3.4 2016/01/20 21:29:46

gsl/2.1 2016/02/25 5:15:41

papi/5.4.1 2016/01/22 0:20:41

scalapack/2.0.2 2016/02/04 8:48:43

parmetis/4.0.3 2016/02/04 7:58:37

superlu/4.2 2016/02/04 7:45:25

petsc/3.6.3 2016/02/04 8:48:36

sbatch: error: bank_limit plugin: expired user, can't submit job

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

When I try to execute the TOV I get something similar.

What is the command just to execute without submitting a job? Alternatvely, how do I configure the ETK to submit successfully on Slurm for the Cluster (Comet) I’m using?

Thanks for all your help!

[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov.par --procs=2 --num-threads=1 --ppn-used=2 --walltime=8:0:0 | cat

Parameter file: /home/jhellmer/comet/Cactus/par/static_tov.par

Skeleton Created

Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov"

Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim"

Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/cfg/OptionList"

Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/SubmitScript"

Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/RunScript"

Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/par/static_tov.par"

Simulation name: static_tov

Assigned restart id: 0

Running simulation static_tov

- Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

intel/2013_sp1.2.144 2017/08/21 20:08:02

mvapich2_ib/2.1 2016/01/20 0:58:02

gnutools/2.69 2016/01/16 0:30:50

hdf5/1.8.14 2016/03/16 19:35:34

netcdf/4.3.2 2016/01/21 22:17:20

ncview/2.1.7 2016/03/31 4:18:21

Unloading compiler-dependent module netcdf/4.3.2

must specify one of the following mpi modules: mvapich2_ib openmpi_ib

- Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

.intel/hdf5/1.8.14 2016/03/16 19:35:34

ncview/2.1.7 2016/03/31 4:18:21

- Package -----------------------------+- Versions -+- Last mod. ------

Currently Loaded Modulefiles:

hdf5/1.8.14 2016/03/16 19:35:34

ncview/2.1.7 2016/03/31 4:18:21

gnutools/2.69 2016/01/16 0:30:50

intel/2015.2.164 2017/08/21 20:08:02

mvapich2_ib/2.1 2016/01/20 0:58:02

fftw/3.3.4 2016/01/20 21:29:46

gsl/2.1 2016/02/25 5:15:41

papi/5.4.1 2016/01/22 0:20:41

scalapack/2.0.2 2016/02/04 8:48:43

parmetis/4.0.3 2016/02/04 7:58:37

superlu/4.2 2016/02/04 7:45:25

petsc/3.6.3 2016/02/04 8:48:36

Preparing:

+ set -e

+ cd /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active

+ echo Checking:

+ pwd

Checking:

/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active

+ hostname

comet-ln2.sdsc.edu

+ date

Mon Sep 3 11:32:34 PDT 2018

+ echo Environment:

Environment:

+ export CACTUS_NUM_PROCS=2

+ CACTUS_NUM_PROCS=2

+ export CACTUS_NUM_THREADS=1

+ CACTUS_NUM_THREADS=1

+ export GMON_OUT_PREFIX=gmon.out

+ GMON_OUT_PREFIX=gmon.out

+ export KMP_AFFINITY=norespect,compact

+ KMP_AFFINITY=norespect,compact

+ export OMP_NUM_THREADS=1

+ OMP_NUM_THREADS=1

+ sort

+ env

+ echo

+ echo Starting:

Starting:

++ date +%s

+ export CACTUS_STARTTIME=1535999554

+ CACTUS_STARTTIME=1535999554

+ ibrun --npernode 2 -v /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/exe/cactus_sim -L 3 /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000/static_tov.par

No SLURM_NODELIST given, run generate_pbs_nodefile inside a Slurm allocation or batch script.

IBRUN ERROR: Unable to obtain number of nodes from resource manager and -N not given.

Contact help@xsede.org for additional assistance.

Mon Sep 3 11:32:34 PDT 2018

Simfactory Done at date: 0

Hello Joe,

Summary of Warnings:

Could not update einsteininitialdata. Could not stash local changes. Error message was 'fatal: unable to create threaded lstat
fatal: unable to create threaded lstat
fatal: unable to create threaded lstat
fatal: unable to create threaded lstat
fatal: unable to create threaded lstat
Cannot save the current worktree state
'.
Time Elapsed: 0 minutes, 27 seconds

This home directory is on a CentOS 6.10 server. I was able to do this successfully on an Ubuntu 16 VM.
Oha. That does not sound good. The threaded lstat thing is an almost
internal git error message that could mean that there are issues with
the file system hosting your $HOME directory. I have seen that message
once before but the exact context escapes me now. It could have been in
relation to a BeeGFS file system issue (you can see the file system in
$HOME by doing "mount' command and looking for the line that
corresponds to $HOME).

In any case, git lets you avoid the threaded lstat if you set (see
):

git config --global core.preloadIndex false

which actually makes my BeeGFS hunch less likely and points to a
resource limit that the admins set.

Yours,
Roland

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .