I’m not sure why put I’m not getting that error anymore.

Now I get this

Unable to process EinsteinInitialData/DistortedBHIVP

Summary of Warnings:

Could not checkout EinsteinInitialData/DistortedBHIVP, unable to switch to tag . Any existing symlinks to EinsteinInitialData/DistortedBHIVP will be broken Time Elapsed: 1 minutes, 9 seconds

Am I getting this because I’ve already tried to install once?

Joe Hellmers

On Aug 6, 2018, at 7:56 AM, Roland Haas rhaas@illinois.edu wrote:

Hello Joe,

sorry for the very long delay in responding. Is this issue still occuring?

If have seen issues like this caused by old versions of OpenSSL installed on clusters since github's https server by now requires at least TLS v1.2 and some old clusters only support v1.1.

You can check this by running git manually like so:

git clone https://github.com/barrywardell/EinsteinExact.git/ https://github.com/barrywardell/EinsteinExact.git/

and if it responds like this:

error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing...

then your version of OpenSSL is too old.

In that case I suggest to try and see if the cluster offers a newer version of git or OpenSSL through its module system or, alternatively, check out the code using GetComponents on your laptop/workstation and transfer to the cluster using simfactory's sync command eg:

cd Cactus simfactory/bin/sim sync bluewaters

Yours, Roland

...

I’m loading the toolkit on the Comet Supercomputer at the San Diego Super Computing facility (SDSC).

When running the command

./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolki...

I’m getting the error below at the end. I’m not sure if these errors are important or not. If someone would be so kind as to let me know, it would be greatly appreciated.

Thank, Joe Hellmers

Unable to process EinsteinExact/doc Unable to process EinsteinExact/m Unable to process EinsteinExact/tests Unable to process EinsteinExact/EinsteinExact_Test Unable to process EinsteinExact/GaugeWave Unable to process EinsteinExact/KerrSchild Unable to process EinsteinExact/Minkowski Unable to process KrancNumericalTools/GenericFD Unable to process EinsteinExact/ModifiedSchwarzschildBL Unable to process EinsteinExact/ShiftedGaugeWave Unable to process EinsteinExact/Vaidya2 Unable to process EinsteinInitialData/DistortedBHIVP Unable to process GetComponents

Summary of Warnings:

Could not checkout module EinsteinExact/doc

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/m

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/tests

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/EinsteinExact_Test

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/GaugeWave

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/KerrSchild

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/Minkowski

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module KrancNumericalTools/GenericFD

fatal: unable to access 'https://github.com/ianhinder/Kranc.git/': SSL connect error

Could not checkout module EinsteinExact/ModifiedSchwarzschildBL

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/ShiftedGaugeWave

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/Vaidya2

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout EinsteinInitialData/DistortedBHIVP, unable to switch to tag . Any existing symlinks to EinsteinInitialData/DistortedBHIVP will be broken Could not checkout module GetComponents

fatal: unable to access 'https://github.com/gridaphobe/CRL.git/': SSL connect error

Time Elapsed: 1 minutes, 48 seconds

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu http://pgp.mit.edu/ .

I see an additional message

[jhellmer@comet-ln2 etk]$ ./GetComponents --verbose https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolki... > GetComponents.log

Use of uninitialized value $branch in concatenation (.) or string at ./GetComponents line 1559.

I was able to git clone

git clone https://bitbucket.org/einsteintoolkit/einsteininitialdata.git Cactus/repos/einsteininitialdata

Successfully. I tried to run GetComponents again to see if it would move the initialdata folder to the arrangements folder and create the symlinks, but I got the following

1 components checked out. 252 components updated.

Unable to process EinsteinInitialData/DistortedBHIVP

Summary of Warnings:

Could not update einsteininitialdata. Could not stash local changes. Error message was 'fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat Cannot save the current worktree state '. Time Elapsed: 0 minutes, 27 seconds

This home directory is on a CentOS 6.10 server. I was able to do this successfully on an Ubuntu 16 VM.

Joe

On Sep 1, 2018, at 3:27 PM, Roland Haas rhaas@illinois.edu wrote:

Hello Joe,

this is probably an issue with the incomplete checkout before. You can try and see if adding the --verbose option to GetComponentns gives any useful help (in particular you would look out for error messages from git).

Personally I would remove the directories repos/einsteininitialdata and arrangements/EinsteinInitialData and re-run GetComponents (or the git line that I suggested).

Yours, Roland

...

I’m not sure why put I’m not getting that error anymore.

Now I get this

Unable to process EinsteinInitialData/DistortedBHIVP

Summary of Warnings:

Could not checkout EinsteinInitialData/DistortedBHIVP, unable to switch to tag . Any existing symlinks to EinsteinInitialData/DistortedBHIVP will be broken Time Elapsed: 1 minutes, 9 seconds

Am I getting this because I’ve already tried to install once?

Joe Hellmers

...

On Aug 6, 2018, at 7:56 AM, Roland Haas rhaas@illinois.edu wrote:

Hello Joe,

sorry for the very long delay in responding. Is this issue still occuring?

If have seen issues like this caused by old versions of OpenSSL installed on clusters since github's https server by now requires at least TLS v1.2 and some old clusters only support v1.1.

You can check this by running git manually like so:

git clone https://github.com/barrywardell/EinsteinExact.git/ https://github.com/barrywardell/EinsteinExact.git/

and if it responds like this:

error: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version while accessing...

then your version of OpenSSL is too old.

In that case I suggest to try and see if the cluster offers a newer version of git or OpenSSL through its module system or, alternatively, check out the code using GetComponents on your laptop/workstation and transfer to the cluster using simfactory's sync command eg:

cd Cactus simfactory/bin/sim sync bluewaters

Yours, Roland

...

I’m loading the toolkit on the Comet Supercomputer at the San Diego Super Computing facility (SDSC).

When running the command

./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolki...

I’m getting the error below at the end. I’m not sure if these errors are important or not. If someone would be so kind as to let me know, it would be greatly appreciated.

Thank, Joe Hellmers

Unable to process EinsteinExact/doc Unable to process EinsteinExact/m Unable to process EinsteinExact/tests Unable to process EinsteinExact/EinsteinExact_Test Unable to process EinsteinExact/GaugeWave Unable to process EinsteinExact/KerrSchild Unable to process EinsteinExact/Minkowski Unable to process KrancNumericalTools/GenericFD Unable to process EinsteinExact/ModifiedSchwarzschildBL Unable to process EinsteinExact/ShiftedGaugeWave Unable to process EinsteinExact/Vaidya2 Unable to process EinsteinInitialData/DistortedBHIVP Unable to process GetComponents

Summary of Warnings:

Could not checkout module EinsteinExact/doc

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/m

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/tests

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/EinsteinExact_Test

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/GaugeWave

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/KerrSchild

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/Minkowski

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module KrancNumericalTools/GenericFD

fatal: unable to access 'https://github.com/ianhinder/Kranc.git/': SSL connect error

Could not checkout module EinsteinExact/ModifiedSchwarzschildBL

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/ShiftedGaugeWave

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout module EinsteinExact/Vaidya2

fatal: unable to access 'https://github.com/barrywardell/EinsteinExact.git/': SSL connect error

Could not checkout EinsteinInitialData/DistortedBHIVP, unable to switch to tag . Any existing symlinks to EinsteinInitialData/DistortedBHIVP will be broken Could not checkout module GetComponents

fatal: unable to access 'https://github.com/gridaphobe/CRL.git/': SSL connect error

Time Elapsed: 1 minutes, 48 seconds

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu http://pgp.mit.edu/ .

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

OK, I did

git config --global core.preloadIndex false

This allowed me to run GetComponents without error

./GetComponents --verbose https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_02/einsteintoolki...

When I build

./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist ../einsteintoolkit.th

all looks well until, I think, the very end when some sort of git clean up seems to be happening.

Done creating cactus_sim. fatal: bad revision 'HEAD' Formaline: Committing source tree to git repository... Formaline: Created git tag build-sim-comet-ln2.sdsc.edu-jhellmer-2018.09.03-17.30.20-18591 Formaline: Updated git branch config-sim-comet-ln2.sdsc.edu-home-jhellmer-etk-Cactus Formaline: Optimising git repository (slow only the first time)... Formaline: ERROR during: Garbage collecting git repo Command was: /usr/bin/git --git-dir='/home/jhellmer/etk/Cactus/configs/sim/configjar.git/.git' gc at /home/jhellmer/etk/Cactus/configs/sim/scratch/Formaline/bin/update-git-repo.pl line 60, <STDIN> line 5090. make[1]: *** [/home/jhellmer/etk/Cactus/configs/sim/build/Formaline/git-store-source] Error 2 make: *** [sim] Error 2

So I attempted to run the HelloWorld app, and it looks to be trying to submit a job but apparently fails

[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-submit helloworld --parfile arrangements/CactusExamples/HelloWorld/par/HelloWorld.par --walltime 0:5:0 Parameter file: /home/jhellmer/comet/Cactus/arrangements/CactusExamples/HelloWorld/par/HelloWorld.par Skeleton Created Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld" Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim" Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/cfg/OptionList" Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/SubmitScript" Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/run/RunScript" Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/SIMFACTORY/par/HelloWorld.par" Assigned restart id: 0 Warning: Too many threads per process specified: specified num-threads=24 (ppn-used is 24) Warning: Total number of threads and number of threads per process are inconsistent: procs=1, num-threads=24 (procs*num-smt must be an integer multiple of num-threads) Warning: Total number of threads and number of cores per node are inconsistent: procs=1, ppn-used=24 (procs must be an integer multiple of ppn-used) Executing submit command: sbatch /oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/output-0000/SIMFACTORY/SubmitScript; sleep 5 Warning: submit either failed or could not determine job id, output: Warning: - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: intel/2013_sp1.2.144 2017/08/21 20:08:02 mvapich2_ib/2.1 2016/01/20 0:58:02 gnutools/2.69 2016/01/16 0:30:50 hdf5/1.8.14 2016/03/16 19:35:34 netcdf/4.3.2 2016/01/21 22:17:20 ncview/2.1.7 2016/03/31 4:18:21 Unloading compiler-dependent module netcdf/4.3.2 must specify one of the following mpi modules: mvapich2_ib openmpi_ib - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: .intel/hdf5/1.8.14 2016/03/16 19:35:34 ncview/2.1.7 2016/03/31 4:18:21 - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: hdf5/1.8.14 2016/03/16 19:35:34 ncview/2.1.7 2016/03/31 4:18:21 gnutools/2.69 2016/01/16 0:30:50 intel/2015.2.164 2017/08/21 20:08:02 mvapich2_ib/2.1 2016/01/20 0:58:02 fftw/3.3.4 2016/01/20 21:29:46 gsl/2.1 2016/02/25 5:15:41 papi/5.4.1 2016/01/22 0:20:41 scalapack/2.0.2 2016/02/04 8:48:43 parmetis/4.0.3 2016/02/04 7:58:37 superlu/4.2 2016/02/04 7:45:25 petsc/3.6.3 2016/02/04 8:48:36 sbatch: error: bank_limit plugin: expired user, can't submit job sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

When I try to execute the TOV I get something similar.

What is the command just to execute without submitting a job? Alternatvely, how do I configure the ETK to submit successfully on Slurm for the Cluster (Comet) I’m using?

Thanks for all your help!

[jhellmer@comet-ln2 Cactus]$ ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov.par --procs=2 --num-threads=1 --ppn-used=2 --walltime=8:0:0 | cat Parameter file: /home/jhellmer/comet/Cactus/par/static_tov.par Skeleton Created Job directory: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov" Executable: "/home/jhellmer/comet/Cactus/exe/cactus_sim" Option list: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/cfg/OptionList" Submit script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/SubmitScript" Run script: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/run/RunScript" Parameter file: "/oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/par/static_tov.par" Simulation name: static_tov Assigned restart id: 0 Running simulation static_tov - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: intel/2013_sp1.2.144 2017/08/21 20:08:02 mvapich2_ib/2.1 2016/01/20 0:58:02 gnutools/2.69 2016/01/16 0:30:50 hdf5/1.8.14 2016/03/16 19:35:34 netcdf/4.3.2 2016/01/21 22:17:20 ncview/2.1.7 2016/03/31 4:18:21 Unloading compiler-dependent module netcdf/4.3.2 must specify one of the following mpi modules: mvapich2_ib openmpi_ib - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: .intel/hdf5/1.8.14 2016/03/16 19:35:34 ncview/2.1.7 2016/03/31 4:18:21 - Package -----------------------------+- Versions -+- Last mod. ------ Currently Loaded Modulefiles: hdf5/1.8.14 2016/03/16 19:35:34 ncview/2.1.7 2016/03/31 4:18:21 gnutools/2.69 2016/01/16 0:30:50 intel/2015.2.164 2017/08/21 20:08:02 mvapich2_ib/2.1 2016/01/20 0:58:02 fftw/3.3.4 2016/01/20 21:29:46 gsl/2.1 2016/02/25 5:15:41 papi/5.4.1 2016/01/22 0:20:41 scalapack/2.0.2 2016/02/04 8:48:43 parmetis/4.0.3 2016/02/04 7:58:37 superlu/4.2 2016/02/04 7:45:25 petsc/3.6.3 2016/02/04 8:48:36 Preparing: + set -e + cd /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active + echo Checking: + pwd Checking: /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000-active + hostname comet-ln2.sdsc.edu + date Mon Sep 3 11:32:34 PDT 2018 + echo Environment: Environment: + export CACTUS_NUM_PROCS=2 + CACTUS_NUM_PROCS=2 + export CACTUS_NUM_THREADS=1 + CACTUS_NUM_THREADS=1 + export GMON_OUT_PREFIX=gmon.out + GMON_OUT_PREFIX=gmon.out + export KMP_AFFINITY=norespect,compact + KMP_AFFINITY=norespect,compact + export OMP_NUM_THREADS=1 + OMP_NUM_THREADS=1 + sort + env + echo + echo Starting: Starting: ++ date +%s + export CACTUS_STARTTIME=1535999554 + CACTUS_STARTTIME=1535999554 + ibrun --npernode 2 -v /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/SIMFACTORY/exe/cactus_sim -L 3 /oasis/scratch/comet/jhellmer/temp_project/simulations/static_tov/output-0000/static_tov.par No SLURM_NODELIST given, run generate_pbs_nodefile inside a Slurm allocation or batch script. IBRUN ERROR: Unable to obtain number of nodes from resource manager and -N not given.

Contact help@xsede.org for additional assistance. Mon Sep 3 11:32:34 PDT 2018 Simfactory Done at date: 0

Hello Joe,

...

Summary of Warnings:

Could not update einsteininitialdata. Could not stash local changes. Error message was 'fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat fatal: unable to create threaded lstat Cannot save the current worktree state '. Time Elapsed: 0 minutes, 27 seconds

This home directory is on a CentOS 6.10 server. I was able to do this successfully on an Ubuntu 16 VM.

Oha. That does not sound good. The threaded lstat thing is an almost internal git error message that could mean that there are issues with the file system hosting your $HOME directory. I have seen that message once before but the exact context escapes me now. It could have been in relation to a BeeGFS file system issue (you can see the file system in $HOME by doing "mount' command and looking for the line that corresponds to $HOME).

In any case, git lets you avoid the threaded lstat if you set (see ):

git config --global core.preloadIndex false

which actually makes my BeeGFS hunch less likely and points to a resource limit that the admins set.

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

For some reason the allocation code I use in the defs.local.ini isn’t being picked when the SubmitScript is created.

The value in the script is set to NO_ALLOCATION. When I set it correctly then I can submit the job, and it runs nicely.

If you can point me to the area in the code where allocation value is being read and set I could probably debug the issue.

Joe Hellmers

On Sep 3, 2018, at 12:08 PM, Roland Haas rhaas@illinois.edu wrote:

Hello Joe,

...

Formaline: Optimising git repository (slow only the first time)... Formaline: ERROR during: Garbage collecting git repo Command was: /usr/bin/git --git-dir='/home/jhellmer/etk/Cactus/configs/sim/configjar.git/.git' gc at /home/jhellmer/etk/Cactus/configs/sim/scratch/Formaline/bin/update-git-repo.pl line 60, <STDIN> line 5090. make[1]: *** [/home/jhellmer/etk/Cactus/configs/sim/build/Formaline/git-store-source] Error 2 make: *** [sim] Error 2

Yes that looks like some error in cleanup. This error will not prevent you from compiling Cactus, as the repository is not used for anything else but keeping track of the code you compiled. Oddly I do not remember having seen that error on Comet before (and executing that line in my own checkout I do not see the same error which may be due to subtle differences in our environment). It may be another fallout of the failed attempts before (you could run the command it gives "/usr/bin/git --git-dir='/home/jhellmer/etk/Cactus/configs/sim/configjar.git/.git' gc" manually and see if git produces a useful error message).

...

sbatch: error: bank_limit plugin: expired user, can't submit job sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

The error code seems to indicate that your user account is not allowed to submit jobs. Did you make sure to select an existing allocation? Either via an "allocation =" line in simfactory/etc/defs.local.ini (in the [comet] section) or via an --allocation option to the "sim submit" command?

...

What is the command just to execute without submitting a job? Alternatvely, how do I configure the ETK to submit successfully on Slurm for the Cluster (Comet) I’m using?

That command would be "simfactory/bin/sim create-run static_tov_test01 --parfile static_tov.par --procs 1 --num-threads 1 --walltime 24:00:00" which you tried and found to fail due to ibrun in there expecting SLURM to be present.

You can do a manual run of the executable (just to see if this even works) like so:

exe/cactus_sim par/static_tov.par 2>&1 | tee static_tov.log

which will run on the *login node* (not a good idea if you want the admins to like you). Note that this may well fail if the admins have set up their MPI stack such that you can only run MPI enabled executables in a job.

You can look at the exact SubmitScript that was passed to sbatch in /oasis/scratch/comet/jhellmer/temp_project/simulations/helloworld/output-0000/SIMFACTORY/SubmitScript

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

Actually that explains a lot!!

Things seem to be working fine now.

So, why am I doing this anyway?

The project I am contemplating is to try to use a Mimetic approach to solving some of the PDEs in ETK.

Can you point me to the places in the code where PDE solutions are being performed?

Joe Hellmers

On Sep 3, 2018, at 4:40 PM, Roland Haas rhaas@illinois.edu wrote:

Hello Joe,

...

For some reason the allocation code I use in the defs.local.ini isn’t being picked when the SubmitScript is created.

The value in the script is set to NO_ALLOCATION. When I set it correctly then I can submit the job, and it runs nicely.

If you can point me to the area in the code where allocation value is being read and set I could probably debug the issue.

I would check that you are setting the allocation in the correct section, namely the [comet] section (sim setup does not initialize that section for you). Just setting in the [default] section is not sufficient since comet.ini sets the allocation to NO_ALLOCATION.

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .