<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">OK, I did<div class=""><br class=""></div><div class=""><pre class=""><code class="">git config --global core.preloadIndex false</code></pre><div class=""><br class=""></div><div class="">This allowed me to run GetComponents without error</div><div class=""><br class=""></div><div class=""><font face="Menlo" class="">./GetComponents --verbose <a href="https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolkit.th" class="">https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolkit.th</a></font></div><div class=""><br class=""></div><div class="">When I build</div><div class=""><br class=""></div><div class=""><div class=""><font face="Menlo" class="">./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist ../<a href="http://einsteintoolkit.th" class="">einsteintoolkit.th</a></font></div></div><div class=""><br class=""></div><div class="">all looks well until, I think, the very end when some sort of git clean up seems to be happening.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><font face="Courier" class="">Done creating cactus_sim.</font></div><div class=""><font face="Courier" class="">fatal: bad revision 'HEAD'</font></div><div class=""><font face="Courier" class="">Formaline: Committing source tree to git repository...</font></div><div class=""><font face="Courier" class="">Formaline: Created git tag <a href="http://build-sim-comet-ln2.sdsc.edu" class="">build-sim-comet-ln2.sdsc.edu</a>-jhellmer-2018.09.03-17.30.20-18591</font></div><div class=""><font face="Courier" class="">Formaline: Updated git branch <a href="http://config-sim-comet-ln2.sdsc.edu" class="">config-sim-comet-ln2.sdsc.edu</a>-home-jhellmer-etk-Cactus</font></div><div class=""><font face="Courier" class="">Formaline: Optimising git repository (slow only the first time)...</font></div><div class=""><font face="Courier" class="">Formaline: ERROR during: Garbage collecting git repo</font></div><div class=""><font face="Courier" class="">Command was: /usr/bin/git --git-dir='/home/jhellmer/etk/Cactus/configs/sim/configjar.git/.git' gc at /home/jhellmer/etk/Cactus/configs/sim/scratch/Formaline/bin/update-git-repo.pl line 60, <STDIN> line 5090.</font></div><div class=""><font face="Courier" class="">make[1]: *** [/home/jhellmer/etk/Cactus/configs/sim/build/Formaline/git-store-source] Error 2</font></div><div class=""><font face="Courier" class="">make: *** [sim] Error 2</font></div></div><div class=""><br class=""></div><div class="">So I attempted to run the HelloWorld app, and it looks to be trying to submit a job but apparently fails</div><div class=""><br class=""></div><div class=""><div class=""><div class=""><font face="Courier" style="font-size: 9px;" class="">[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-submit helloworld --parfile arrangements/CactusExamples/HelloWorld/par/HelloWorld.par --walltime 0:5:0</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Parameter file: /home/jhellmer/comet/Cactus/arrangements/CactusExamples/HelloWorld/par/HelloWorld.par</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Skeleton Created</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/cfg/OptionList"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/SubmitScript"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/RunScript"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/par/HelloWorld.par"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Assigned restart id: 0</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Warning: Too many threads per process specified: specified num-threads=24 (ppn-used is 24)</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Warning: Total number of threads and number of threads per process are inconsistent: procs=1, num-threads=24 (procs*num-smt must be an integer multiple of num-threads)</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Warning: Total number of threads and number of cores per node are inconsistent: procs=1, ppn-used=24 (procs must be an integer multiple of ppn-used)</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Executing submit command: sbatch /oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/output-0000/SIMFACTORY/SubmitScript; sleep 5</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Warning: submit either failed or could not determine job id, output:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Warning: - Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">intel/2013_sp1.2.144 2017/08/21 20:08:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">mvapich2_ib/2.1 2016/01/20 0:58:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gnutools/2.69 2016/01/16 0:30:50</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">netcdf/4.3.2 2016/01/21 22:17:20</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Unloading compiler-dependent module netcdf/4.3.2</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">must specify one of the following mpi modules: mvapich2_ib openmpi_ib</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">- Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">.intel/hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">- Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gnutools/2.69 2016/01/16 0:30:50</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">intel/2015.2.164 2017/08/21 20:08:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">mvapich2_ib/2.1 2016/01/20 0:58:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">fftw/3.3.4 2016/01/20 21:29:46</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gsl/2.1 2016/02/25 5:15:41</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">papi/5.4.1 2016/01/22 0:20:41</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">scalapack/2.0.2 2016/02/04 8:48:43</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">parmetis/4.0.3 2016/02/04 7:58:37</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">superlu/4.2 2016/02/04 7:45:25</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">petsc/3.6.3 2016/02/04 8:48:36</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">sbatch: error: bank_limit plugin: expired user, can't submit job</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified</font></div></div><div><blockquote type="cite" class=""></blockquote></div></div><div class=""><br class=""></div><div class="">When I try to execute the TOV I get something similar.</div><div class=""><br class=""></div><div class="">What is the command just to execute without submitting a job? Alternatvely, how do I configure the ETK to submit successfully on Slurm for the Cluster (Comet) I’m using?</div><div class=""><br class=""></div><div class="">Thanks for all your help!</div><div class=""><br class=""></div><div class=""><font face="Courier" style="font-size: 9px;" class="">[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov.par --procs=2 --num-threads=1 --ppn-used=2 --walltime=8:0:0 | cat</font></div><div class=""><div class=""><font face="Courier" style="font-size: 9px;" class="">Parameter file: /home/jhellmer/comet/Cactus/par/static_tov.par</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Skeleton Created</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/cfg/OptionList"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/SubmitScript"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/RunScript"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/par/static_tov.par"</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Simulation name: static_tov</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Assigned restart id: 0</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Running simulation static_tov</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">- Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">intel/2013_sp1.2.144 2017/08/21 20:08:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">mvapich2_ib/2.1 2016/01/20 0:58:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gnutools/2.69 2016/01/16 0:30:50</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">netcdf/4.3.2 2016/01/21 22:17:20</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Unloading compiler-dependent module netcdf/4.3.2</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">must specify one of the following mpi modules: mvapich2_ib openmpi_ib</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">- Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">.intel/hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">- Package -----------------------------+- Versions -+- Last mod. ------</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Currently Loaded Modulefiles:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">hdf5/1.8.14 2016/03/16 19:35:34</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">ncview/2.1.7 2016/03/31 4:18:21</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gnutools/2.69 2016/01/16 0:30:50</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">intel/2015.2.164 2017/08/21 20:08:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">mvapich2_ib/2.1 2016/01/20 0:58:02</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">fftw/3.3.4 2016/01/20 21:29:46</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">gsl/2.1 2016/02/25 5:15:41</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">papi/5.4.1 2016/01/22 0:20:41</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">scalapack/2.0.2 2016/02/04 8:48:43</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">parmetis/4.0.3 2016/02/04 7:58:37</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">superlu/4.2 2016/02/04 7:45:25</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">petsc/3.6.3 2016/02/04 8:48:36</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Preparing:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ set -e</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ cd /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ echo Checking:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ pwd</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Checking:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ hostname</font></div><div class=""><font face="Courier" style="font-size: 9px;" class=""><a href="http://comet-ln2.sdsc.edu" class="">comet-ln2.sdsc.edu</a></font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ date</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Mon Sep 3 11:32:34 PDT 2018</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ echo Environment:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Environment:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export CACTUS_NUM_PROCS=2</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ CACTUS_NUM_PROCS=2</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export CACTUS_NUM_THREADS=1</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ CACTUS_NUM_THREADS=1</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export GMON_OUT_PREFIX=gmon.out</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ GMON_OUT_PREFIX=gmon.out</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export KMP_AFFINITY=norespect,compact</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ KMP_AFFINITY=norespect,compact</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export OMP_NUM_THREADS=1</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ OMP_NUM_THREADS=1</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ sort</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ env</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ echo</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ echo Starting:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Starting:</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">++ date +%s</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ export CACTUS_STARTTIME=1535999554</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ CACTUS_STARTTIME=1535999554</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">+ ibrun --npernode 2 -v /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/exe/cactus_sim -L 3 /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000/static_tov.par</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">No SLURM_NODELIST given, run generate_pbs_nodefile inside a Slurm allocation or batch script.</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">IBRUN ERROR: Unable to obtain number of nodes from resource manager and -N not given.</font></div><div class=""><font face="Courier" style="font-size: 9px;" class=""><br class=""></font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Contact <a href="mailto:help@xsede.org" class="">help@xsede.org</a> for additional assistance.</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Mon Sep 3 11:32:34 PDT 2018</font></div><div class=""><font face="Courier" style="font-size: 9px;" class="">Simfactory Done at date: 0 </font></div></div><div><blockquote type="cite" class=""><div class=""><br class=""></div><br class="Apple-interchange-newline"><div class=""><div class="">Hello Joe,<br class=""><br class=""><blockquote type="cite" class="">Summary of Warnings:<br class=""><br class="">Could not update einsteininitialdata. Could not stash local changes. Error message was 'fatal: unable to create threaded lstat<br class="">fatal: unable to create threaded lstat<br class="">fatal: unable to create threaded lstat<br class="">fatal: unable to create threaded lstat<br class="">fatal: unable to create threaded lstat<br class="">Cannot save the current worktree state<br class="">'.<br class=""> Time Elapsed: 0 minutes, 27 seconds<br class=""><br class="">This home directory is on a CentOS 6.10 server. I was able to do this successfully on an Ubuntu 16 VM.<br class=""></blockquote>Oha. That does not sound good. The threaded lstat thing is an almost<br class="">internal git error message that could mean that there are issues with<br class="">the file system hosting your $HOME directory. I have seen that message<br class="">once before but the exact context escapes me now. It could have been in<br class="">relation to a BeeGFS file system issue (you can see the file system in<br class="">$HOME by doing "mount' command and looking for the line that<br class="">corresponds to $HOME).<br class=""><br class="">In any case, git lets you avoid the threaded lstat if you set (see<br class="">):<br class=""><br class="">git config --global core.preloadIndex false<br class=""><br class="">which actually makes my BeeGFS hunch less likely and points to a<br class="">resource limit that the admins set.<br class=""><br class="">Yours,<br class="">Roland<br class=""><br class="">-- <br class="">My email is as private as my paper mail. I therefore support encrypting<br class="">and signing email messages. Get my PGP key from <a href="http://pgp.mit.edu" class="">http://pgp.mit.edu</a> .<br class=""></div></div></blockquote></div><br class=""></div></body></html>