Hello all,
since this was originally shared only with a subset of the devs, here it is once more. These are the original (tested) instructions on how to compile CarpetX for CUDA.
The ones on https://bitbucket.org/eschnett/cactusamrex/wiki/Getting%20Started#markdown-h... are a derivative and also use slightly different code branches (in an attempt to support building ExternalLibraries).
Yours, Roland
Begin forwarded message:
Date: Wed, 4 Aug 2021 11:32:11 -0400 From: Erik Schnetter eschnetter@perimeterinstitute.ca To: Bruno Giacomazzo bruno.giacomazzo@unimib.it Cc: Lorenzo Ennoggi lorenzo.ennoggi@gmail.com, Lucas Timotheo Sanches lucas.t.s.carneiro@gmail.com, Jay Vijay Kalinani jayvijay.kalinani@phd.unipd.it, Annamalai P S annamalaips.97@gmail.com, Roland Haas rhaas@illinois.edu, Soham Mukherjee smukherjee@perimeterinstitute.ca Subject: Re: Building CarpetX on a laptop
To run on a GPU, you need to install AMReX differently, probably into a different directory: You need to configure it with additional cmake options such as
-DAMReX_GPU_BACKEND=CUDA -DAMReX_CUDA_ERROR_CAPTURE_THIS=ON -DAMReX_CUDA_ERROR_CROSS_EXECUTION_SPACE_CALL=ON -DAMReX_CUDA_ARCH=75
where "75" stands for the CUDA compute capability 7.5. Choose other values there, depending on the hardware on which you will run.
Then continue as before. Create a different option list (which points to the new AMReX library) and, which uses nvcc as C++ compiler. For me, the respective modified or new options are:
CXX = /home/eschnetter/Cactus/view-cuda/bin/nvcc --compiler-bindir /home/eschnetter/Cactus/view-cuda-compilers/bin/g++ -x cu CXXFLAGS = -pipe -g --compiler-options -march=native -std=c++17 --compiler-options -std=gnu++17 --expt-relaxed-constexpr --extended-lambda --gpu-architecture sm_75 --forward-unknown-to-host-compiler --Werror cross-execution-space-call --Werror ext-lambda-captures-this --relocatable-device-code=true --objdir-as-tempdir
DISABLE_INT16 = yes DISABLE_REAL16 = yes VECTORISE = no
CUDA_DIR = /home/eschnetter/Cactus/view-cuda CUDA_INC_DIRS = /home/eschnetter/Cactus/view-cuda/targets/x86_64-linux/include CUDA_LIB_DIRS = /home/eschnetter/Cactus/view-cuda/targets/x86_64-linux/lib
Then create a new Cactus configuration (executable). This executable will only run when you have a GPU available; AMReX will check this at startup.
Depending on how you write your loops, they will either still run on a CPU, or will now run on a GPU (if you use "loop_device", as in HydroToyGPU).
I left out the steps where you get lots of errors from the compiler, and where you sprinkle the code with macros such as CCTK_DEVICE, CCTK_HOST, or CCTK_ATTRIBUTE_ALWAYS_INLINE, and where you change C++ lambda expressions to use [=] to ensure the captured variables are copied to the GPU instead of leading to compiler errors (good, because they come with line numbers) or run-time errors (bad, because they just say "something went wrong").
And once you're done, you would start looking at performance, trying to beat our well-optimized CPU gode.
-erik