[Users] installation of ET on a cluster

[IOSIF PANAGIOTIS]

Dear Roland,

thank you for gathering these links, they are indeed useful to point me in the right direction.

I also found this seminar talk by Erik Schnetter on using Spack to install the ET dependencies.
I also copy it in this thread, so that it is grouped with the other resources you gathered:

https://einsteintoolkit.org/seminars/2022_05_19/index.html

I will work my way through this material and let you know if I encounter other problems.

Best,
Panagiotis


________________________________
From: Roland Haas
Sent: Friday, February 7, 2025 6:41 PM
To: IOSIF PANAGIOTIS
Cc: users at einsteintoolkit.org
Subject: Re: [Users] installation of ET on a cluster

Hello Panagiotis,

Thank you for your inquiry.

This may help:

http://einsteintoolkit.org/seminars/2022_02_24/index.html

https://youtu.be/8dnOfQIk32c?feature=shared

whose slides are:

https://github.com/rhaas80/2024_LSU/blob/main/CactusBuildSytemTour.ipynb

There's also some advanced simulation factory docs here:

https://docs.einsteintoolkit.org/et-docs/Configuring_a_new_machine

(linked on
https://docs.einsteintoolkit.org/et-docs/Main_Page#Documentation)

> The CactusTutorial jupyter notebook for new users does not tackle
> this scenario in detail. Understandably, it would be unrealistic to
> provide generic instructions that cover all cluster configurations.
> How about guidelines that cover some common scenarios though? (e.g. a
> cluster that organizes libraries and packages as modules and also
> uses the Spack manager)

See above. Spack is tricky and often does not work out of the box
unless one really builds everything from scratch. Some of the ET users
have experience and they may be able to provide help.

> Is there some documentation that gives basic guidelines as to how
> setup ET on a cluster aimed at a user profile that has (obviously) no
> previous experience with ET, simfactory etc and (quite possibly) also
> limited experience on how HPC clusters are setup, how packages are
> installed etc?

The recording of the seminar (and the slides therein) tries to do that.
I would give it a try.

> This request stems from the fact that I have been discovering
> little-by-little and through trial-and-error that e.g. running
> `./simfactory/bin/sim setup-silent` as outlined in the
> CactusTutorial,  makes no sense for the case of a cluster. In my
> opinion, and if I may offer a suggestion, it would be a good idea to
> underline this distinction in a future revision of the tutorial.

Unfortunately each cluster is different and often one needs specific
detailed workarounds. A lot of that unfortuntely comes down to
experience.

>
> Furthermore, another crucial step to setup ET on a cluster seems to
> be that the user must create "manually" their respective
> "<my_cluster>.cfg" and "<my_cluster>.ini" files and possibly the
> corresponding run and submit scripts. Could you confirm that this is
> indeed the case? What is the recommended course of action to do this?
> Maybe one should start by copying a sample ".cfg" file and amend it
> with information found from their cluster's technical documentation?

I'd start with an existing cluster that uses the same compiler / queuing
system. Queueing system these days should always be SLURM and one can
then try and see if the cluster supports the GNU compilers which should
make things easy.

The only libraries that one really needs from the cluster are MPI.
Everything else can be built by the EinsteinToolkit if needed (by
setting the various FOO_DIR variable to the string BUILD).

> I can follow up with more specific questions about problems I
> encountered during my so far unsuccessful attempts to install ET on a
> cluster, but to avoid people having to repeat explanations already
> given in the past, I would like first to make sure that I am not
> missing some source of information/documentation already available.

There is no real good docs unfortunately. Even the ones that I had
linked above are (were, they are now) not collected in a single place.

> Also, in the long shot that someone has already installed ET on one
> of the CINECA clusters in Italy (I am trying to install ET on the
> Leonardo cluster), I would be grateful if you could share your
> insights!

Bruno Giacommazzo installed the ET on some of the CINECA clusters in the
past. I would try and reach out to him and his group at Milan. The
newest machine still in simfactory is marconiA3 which however is
already quite old. He also is subscribed to this mailing list.

Yours,
Roland

--
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.einsteintoolkit.org/pipermail/users/attachments/20250211/987fc053/attachment.htm>