Hello,

It still looks like the Vectors Thorn is required to Carpet to work. Is there a way to not compile this Thorn and still have Carpet work?

-DG

On Jan 25, 2022, at 9:55 AM, Roland Haas <rhaas@illinois.edumailto:rhaas@illinois.edu> wrote:

Hello David,

I doubt it will work with vectorization since we do not implement vectors with 16 byte floating point values.

You can try setting

VECTORISE=no

in your option list to disable vectorization which may work.

You can find the available options (not documented unfortunately) here:

https://bitbucket.org/cactuscode/cactusutils/src/master/Vectors/configuratio...

Yours, Roland

Hello,

Has anyone successfully compiled Cactus with REAL_PRECISION = 16? I get the following error which seems to involve the Vectors Thorn which is required for Carpet.

Checking status of thorn Vectors COMPILING CactusUtils/Vectors/src/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... > In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(249): error: #error directive: "Unknown CCTK_REAL_PRECISION" #error "Unknown CCTK_REAL_PRECISION" ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(271): error: #error directive: "Unsupported vector size" #error "Unsupported vector size" ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(811): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 template <typename = void> constexpr vectype &operator=(vectype const &x) { ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(893): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr vectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(897): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr vectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(901): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr vectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(905): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr vectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(990): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 constexpr bvectype &operator=(bvectype const &x) { ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1035): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr bvectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1039): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr bvectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1043): warning #3699: constexpr non-static member function will not be implicitly 'const' in C++14 inline CCTK_ATTRIBUTE_ALWAYS_INLINE constexpr bvectype & ^

In file included from /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(1): /home/garrison/Cactus/arrangements/CactusUtils/Vectors/src/vectors.h(1287): error: right operand of "%" is zero #if CCTK_REAL_CACHELINE_SIZE % CCTK_REAL_VEC_SIZE != 0 ^

/home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(9): error: identifier "CCTK_REAL_VEC_SIZE" is undefined CCTK_REAL_VEC_SIZE, vec_architecture); ^

/home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... >(9): error: identifier "vec_architecture" is undefined CCTK_REAL_VEC_SIZE, vec_architecture); ^

compilation aborted for /home/garrison/Cactus/configs/specgrmhd/build/Vectors/vectors.cc<https://urldefense.com/v3/__http://vectors.cc/__;!!DZ3fjg!ozhila849qlggIwIhM... > (code 2) make[4]: *** [vectors.cc.o] Error 2 make[3]: *** [make.checked] Error 2 make[2]: *** [/home/garrison/Cactus/configs/specgrmhd/lib/libthorn_Vectors.a] Error 2 make[1]: *** [specgrmhd] Error 2 make: *** [specgrmhd-rebuild] Error 2

-DG

-- David Garrison, Ph.D. Interim Associate Dean for the College of Science and Engineering, Professor and Former Chair of Physics University of Houston-Clear Lake Bayou 3611 Houston, TX 77058

Tel: 281-283-3796 Fax: 281-283-3709 https://urldefense.com/v3/__http://sce.uhcl.edu/garrison__;!!DZ3fjg!ozhila84... https://urldefense.com/v3/__http://www.uhcl.edu/physics__;!!DZ3fjg!ozhila849...

"If we knew what it was we were doing, it would not be called research, would it?" ‹ Albert Einstein.

-- David Garrison, Ph.D. Interim Associate Dean for the College of Science and Engineering, Professor and Former Chair of Physics University of Houston-Clear Lake Bayou 3611 Houston, TX 77058

Tel: 281-283-3796 Fax: 281-283-3709 https://urldefense.com/v3/__http://sce.uhcl.edu/garrison__;!!DZ3fjg!ozhila84... https://urldefense.com/v3/__http://www.uhcl.edu/physics__;!!DZ3fjg!ozhila849...

"If we knew what it was we were doing, it would not be called research, would it?" ‹ Albert Einstein.

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

-- David Garrison, Ph.D. Interim Associate Dean for the College of Science and Engineering, Professor and Former Chair of Physics University of Houston-Clear Lake Bayou 3611 Houston, TX 77058

Tel: 281-283-3796 Fax: 281-283-3709 http://sce.uhcl.edu/garrison http://www.uhcl.edu/physics

"If we knew what it was we were doing, it would not be called research, would it?" ‹ Albert Einstein.

That makes sense. It will take me a little time to change the source code but I can try it out. I think this is something that should be included in future versions of Cactus so extra code modifications aren’t needed.

-DG

On Jan 25, 2022, at 10:53 AM, Roland Haas <rhaas@illinois.edumailto:rhaas@illinois.edu> wrote:

Hello David,

It still looks like the Vectors Thorn is required to Carpet to work. Is there a way to not compile this Thorn and still have Carpet work? Hmm, I have not tried this out actually.

VECTORISE=no

is an option to the vector's thorn which tells it to not use any simd vectorization but just plain scalars. So the thorn is present, making CarpetLib happy. But Vectors (see below) may not compile.

When using VECTORISE=no it should be possible to compile even with CCTK_REAL being extended precision (16 bytes), though it is possible that there are explicit checks for the CCTK_REAL width even in that case in the code.

The good news is that, in that case only, one should be able to change the source code to make things compile by (worst case) add a specific sizeof(CCTK_REAL) == 16 case and using "long double" instead of "double" for the type of variable.

Looking at the source code of Vectors one probably has to duplicate vectors-8-default.h into vectors-16-default.h and add a branch for CCTK_REAL_PRECISION_16 to vectors.h.

Does this explanation make sense to you and gives you an idea how you could change the code?

Yours, Roland

-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu .

-- David Garrison, Ph.D. Interim Associate Dean for the College of Science and Engineering, Professor and Former Chair of Physics University of Houston-Clear Lake Bayou 3611 Houston, TX 77058

Tel: 281-283-3796 Fax: 281-283-3709 http://sce.uhcl.edu/garrison http://www.uhcl.edu/physics

"If we knew what it was we were doing, it would not be called research, would it?" ‹ Albert Einstein.

On Tue, Jan 25, 2022 at 12:22 PM Roland Haas rhaas@illinois.edu wrote:

Hello David,

...

That makes sense. It will take me a little time to change the source code but I can try it out. I think this is something that should be included in future versions of Cactus so extra code modifications aren’t needed.

Agreed. If you add such code and provide a pull request then I, for my part, would be in favour of including it, even if only for the VECTORISE=no case.

Note that just because the ET compiles does not guarantee that all places in Cactus / Carpet are actually save for the case where CCTK_REAL is "better" than a "double" (they should work fine for CCTK_REAL == float since that has been used in the past). You are mostly in uncharted areas I would say (though others may have used such a setup in the past, I just do not myself recall having seen this being done).

David

To actually use CCTK_REAL16, a lot of details need to be sorted out.

In many places, people use constants such as "1.0/3.0". With CCTK_REAL4, this will automatically switch to double precision (and run much slower e.g. on GPUs). With CCTK_REAL16, this will lose a lot of precision. You will need to go through the code and manually replace all such terms with e.g. "1 / CCTK_REAL(3)". You will also need to replace M_PI by e.g. "acos(-CCTK_REAL(1))" etc. In C++ and Fortran, calls to "log" or "exp" will automatically do the right thing; in C they won't (use <tgmath.h>; see e.g. https://en.cppreference.com/w/c/numeric/math/sin).

Having done that, you will also need to update calls to printf or CCTK_VINFO to ensure that all its arguments are converted to double, or update the format specifiers (see e.g. https://en.cppreference.com/w/cpp/io/c/fprintf). Finally, the HDF5 output most likely doesn't support CCTK_REAL16, and neither will most visualization tools.

-erik