[Carpetx-developers] Fwd: Building CarpetX on a laptop

[Roland Haas]

Hello all,

since this was originally shared only with a subset of the devs, here
it is once more. These are the original (tested) instructions on how to
compile CarpetX for CUDA.

The ones on
https://bitbucket.org/eschnett/cactusamrex/wiki/Getting%20Started#markdown-header-cuda-support are a derivative and also use slightly different code branches (in an attempt to support building ExternalLibraries).

Yours,
Roland

Begin forwarded message:

Date: Wed, 4 Aug 2021 11:32:11 -0400
From: Erik Schnetter <eschnetter at perimeterinstitute.ca>
To: Bruno Giacomazzo <bruno.giacomazzo at unimib.it>
Cc: Lorenzo Ennoggi <lorenzo.ennoggi at gmail.com>,        Lucas Timotheo
Sanches <lucas.t.s.carneiro at gmail.com>,        Jay Vijay Kalinani
<jayvijay.kalinani at phd.unipd.it>,        Annamalai P S
<annamalaips.97 at gmail.com>,        Roland Haas <rhaas at illinois.edu>,
    Soham Mukherjee <smukherjee at perimeterinstitute.ca> Subject: Re:
Building CarpetX on a laptop


To run on a GPU, you need to install AMReX differently, probably into
a different directory: You need to configure it with additional cmake
options such as

-DAMReX_GPU_BACKEND=CUDA
-DAMReX_CUDA_ERROR_CAPTURE_THIS=ON
-DAMReX_CUDA_ERROR_CROSS_EXECUTION_SPACE_CALL=ON
-DAMReX_CUDA_ARCH=75

where "75" stands for the CUDA compute capability 7.5. Choose other
values there, depending on the hardware on which you will run.

Then continue as before. Create a different option list (which points
to the new AMReX library) and, which uses nvcc as C++ compiler. For
me, the respective modified or new options are:

CXX = /home/eschnetter/Cactus/view-cuda/bin/nvcc --compiler-bindir
/home/eschnetter/Cactus/view-cuda-compilers/bin/g++ -x cu
CXXFLAGS = -pipe -g --compiler-options -march=native -std=c++17
--compiler-options -std=gnu++17 --expt-relaxed-constexpr
--extended-lambda --gpu-architecture sm_75
--forward-unknown-to-host-compiler --Werror cross-execution-space-call
--Werror ext-lambda-captures-this --relocatable-device-code=true
--objdir-as-tempdir

DISABLE_INT16 = yes
DISABLE_REAL16 = yes
VECTORISE = no

CUDA_DIR = /home/eschnetter/Cactus/view-cuda
CUDA_INC_DIRS =
/home/eschnetter/Cactus/view-cuda/targets/x86_64-linux/include
CUDA_LIB_DIRS =
/home/eschnetter/Cactus/view-cuda/targets/x86_64-linux/lib

Then create a new Cactus configuration (executable). This executable
will only run when you have a GPU available; AMReX will check this at
startup.

Depending on how you write your loops, they will either still run on a
CPU, or will now run on a GPU (if you use "loop_device", as in
HydroToyGPU).

I left out the steps where you get lots of errors from the compiler,
and where you sprinkle the code with macros such as CCTK_DEVICE,
CCTK_HOST, or CCTK_ATTRIBUTE_ALWAYS_INLINE, and where you change C++
lambda expressions to use [=] to ensure the captured variables are
copied to the GPU instead of leading to compiler errors (good, because
they come with line numbers) or run-time errors (bad, because they
just say "something went wrong").

And once you're done, you would start looking at performance, trying
to beat our well-optimized CPU gode.

-erik
-- 
My email is as private as my paper mail. I therefore support encrypting
and signing email messages. Get my PGP key from http://pgp.mit.edu .
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
Url : http://lists.einsteintoolkit.org/pipermail/carpetx-developers/attachments/20210831/9684cb9f/attachment.bin