From trac-noreply at einsteintoolkit.org Wed Feb 7 03:12:08 2024 From: trac-noreply at einsteintoolkit.org (José Ferreira) Date: Wed, 07 Feb 2024 09:12:08 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: submitted Milestone: Version: ET_2023_11 Type: bug Priority: critical Component: Hello, I am facing an issue where running a simulation with the toolkit, using the default thornlist, can either run as expected, or crash right in the beginning of the simulation. This took place with `ET_TOV.par`, that ships with the toolkit, and a thornfile that evolves a constant scalar field, that ships with the Scalar thorn also included in the toolkit. I believe that the culprit is OpenMP because if I disable threading during run-time, by setting `OMP_NUM_THREADS=1`, or if I disable OpenMP during compile-time, the simulations run as expected. This bug takes place both in my laptop and in my desktop, which share similar operating systems and software stack. ? In the following sections, I will write, line by line, the steps that I have performed in order to reproduce the bug, and my attempt of tracking it down. ? # Installing and Compiling the Toolkit To avoid compiling CarpetX thorns that fail to compile in my system, for some reason, I start by downloading the previous version of the toolkit with ```shell $ curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2023_05/GetComponents $ chmod a+x GetComponents $ ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2023_05/einsteintoolkit.th ``` and then change to the Cactus root directory ```shell $ cd Cactus ``` I create the options file `arch.cfg` , which should already be present in the parent directory, with the following written in it ``` # Cactus configuration for Arch and Arch-based distros ## Decide which flags will be used at compile-time OPTIMISE = yes WARN = yes DEBUG = no PROFILE = no OPENMP = yes ## Compilers CPP = cpp FPP = cpp CC = gcc CXX = g++ F77 = gfortran F90 = gfortran ## Default flags CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK FPPFLAGS = -traditional CFLAGS = -g3 -march=native -std=gnu99 CXXFLAGS = -g3 -march=native -std=gnu++0x F77FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none -fno-range-check F90FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none -fno-range-check LDFLAGS = -rdynamic ## Optimization flags CPP_OPTIMISE_FLAGS = -DKRANC_VECTORS # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -Ofast CXX_OPTIMISE_FLAGS = -Ofast F77_OPTIMISE_FLAGS = -Ofast F90_OPTIMISE_FLAGS = -Ofast ## Warning flags CPP_WARN_FLAGS = -Wall FPP_WARN_FLAGS = -Wall C_WARN_FLAGS = -Wall CXX_WARN_FLAGS = -Wall F77_WARN_FLAGS = -Wall F90_WARN_FLAGS = -Wall ## Debug flags CPP_DEBUG_FLAGS = -DCARPET_DEBUG -fsanitize=undefined -fsanitize=thread FPP_DEBUG_FLAGS = -DCARPET_DEBUG -fsanitize=undefined -fsanitize=thread C_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread CXX_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread F77_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread F90_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread ## Code profiling flags CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg ## OpenMP CPP_OPENMP_FLAGS = -fopenmp FPP_OPENMP_FLAGS = -fopenmp C_OPENMP_FLAGS = -fopenmp CXX_OPENMP_FLAGS = -fopenmp F77_OPENMP_FLAGS = -fopenmp F90_OPENMP_FLAGS = -fopenmp ## Libraries location LIBDIRS = MPI_DIR = /usr HDF5_DIR = /usr PTHREADS_DIR = NO_BUILD LIBS = gfortran open-pal z C_LINE_DIRECTIVES = yes F_LINE_DIRECTIVES = yes ``` I then create the configuration folder with ```shell $ make base-config options=../arch.cfg THORNLIST=thornlists/einsteintoolkit.th ``` which reveals no errors, with the terminal output attached in file `make-config.out`, and then make the binary with ```shell $ make -j $(nproc) base ``` that also reveals no errors, and the output is attached in `make-binary.out`. ? To ensure full reproducibility, instead of doing this manually I created a very simple script, attached as `run.sh`, that reproduces the steps laid out in this section. ? # Running the Toolkit and Finding the Bug With no errors so far, and the binary `exe/cactus_base` in place, I will run one of the par files provided by default in the toolkit, in this case, `par/ET_TOV.par` ```shell $ exe/cactus_base par/tov_ET.par ``` which crashes, producing something that says ``` Rank 0 with PID 2329840 received signal 11 Writing backtrace to tov_ET/backtrace.0.txt [1] 2329840 segmentation fault (core dumped) ``` The full output for this simulation is sent in the file `run.out` that is being sent as an attachment, along with `backtrace.0.txt`. ? Interestingly, if insist on running this simulation for a few times, I will eventually find one where the bug doesn?t take place, and the simulation seems to be running as expected. Therefore, this is not just a classical bug, it?s a Heisenbug! ? # Tracking the Bug If I am to run the binary disabling OpenMP at runtime, i.e. ```shell $ OMP_NUM_THREADS=1 exe/cactus_base par/ET_TOV.par ``` then I consistently get no errors, even after running the code a few dozens of time. This is why I am lead to believe that the culprit of the Heisenbug is OpenMP. ? I created a new configuration of the toolkit with `DEBUG=yes` and `OPTIMIZE=no` in the options file above, which created the binary `exe/cactus_base-debug`. For completeness, the output of the making of the configuration is sent in `make-config-debug.out` and for the binary in `make-binary-debug.out`, although no errors were produced. I don?t have that much experience in debugging low-level code, so I decided to disable optimizations and add `-fsanitize=undefined` and `-fsanitize=thread` to GCC , which looked reasonable to me. ? Running the debug version of the binary with the same parfile ```shell $ exe/cactus_base-debug par/ET_TOV.par ``` reveals the same error on loop, and the program never terminates \(and by never I mean in around one minute\), and the errors look something like ``` ================== WARNING: ThreadSanitizer: data race (pid=2419680) Read of size 8 at 0x7ffe815ca630 by thread T5: #0 grhydro_atmospherereset_._omp_fn.0 /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:323 (cactus_base-debug+0xd14fbb9) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #1 gomp_thread_start /usr/src/debug/gcc/gcc/libgomp/team.c:129 (libgomp.so.1+0x20c95) (BuildId: 919d8c8c3093e63652b89795375dcf12dd9cb1d4) Previous write of size 8 at 0x7ffe815ca630 by main thread: #0 grhydro_atmospherereset_ /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:321 (cactus_base-debug+0xd11394a) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #1 CCTKi_BindingsFortranWrapperGRHydro /home/undercover/misc/tmp/Cactus/configs/base-debug/bindings/Variables/GRHydro.c:37 (cactus_base-debug+0x155440a4) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #2 CCTK_CallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:323 (cactus_base-debug+0x152bc047) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #3 CallScheduledFunction /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:440 (cactus_base-debug+0xa36120b) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #4 Carpet::CallFunction(void*, cFunctionData*, void*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:373 (cactus_base-debug+0xa35f44b) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #5 CCTKi_ScheduleCallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:3096 (cactus_base-debug+0x152c6b74) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #6 ScheduleTraverseFunction /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:595 (cactus_base-debug+0x152d6b94) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #7 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:369 (cactus_base-debug+0x152d5be6) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #8 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:385 (cactus_base-debug+0x152d673d) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #9 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:385 (cactus_base-debug+0x152d673d) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #10 CCTKi_DoScheduleTraverse /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:159 (cactus_base-debug+0x152d4d7e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #11 ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:1400 (cactus_base-debug+0x152bedf0) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #12 CCTK_ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:919 (cactus_base-debug+0x152bcedf) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #13 ScheduleTraverse /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:1393 (cactus_base-debug+0xa3f8929) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #14 CallRestrict /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:529 (cactus_base-debug+0xa3e8be8) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #15 Carpet::Initialise(tFleshConfig*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:121 (cactus_base-debug+0xa3ddc37) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #16 main /home/undercover/misc/tmp/Cactus/src/main/flesh.cc:80 (cactus_base-debug+0x1528f2e7) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) Location is stack of main thread. Location is global '' at 0x000000000000 ([stack]+0xf7630) Thread T5 (tid=2419694, running) created by main thread at: #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e) #1 gomp_team_start /usr/src/debug/gcc/gcc/libgomp/team.c:858 (libgomp.so.1+0x212df) (BuildId: 919d8c8c3093e63652b89795375dcf12dd9cb1d4) #2 CarpetLib::dist::pseudoinit(ompi_communicator_t*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/CarpetLib/src/dist.cc:200 (cactus_base-debug+0xac0eac8) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #3 CarpetMultiModelStartup /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CarpetStartup.cc:29 (cactus_base-debug+0xa3733e3) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #4 CCTK_CallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:309 (cactus_base-debug+0x152bbf51) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #5 CCTKi_ScheduleCallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:3096 (cactus_base-debug+0x152c6b74) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #6 ScheduleTraverseFunction /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:595 (cactus_base-debug+0x152d6b94) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #7 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:369 (cactus_base-debug+0x152d5be6) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #8 CCTKi_DoScheduleTraverse /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:159 (cactus_base-debug+0x152d4d7e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #9 ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:1400 (cactus_base-debug+0x152bedf0) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #10 CCTK_ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:919 (cactus_base-debug+0x152bcedf) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #11 CCTKi_CallStartupFunctions /home/undercover/misc/tmp/Cactus/src/main/CallStartupFunctions.c:50 (cactus_base-debug+0x1527f958) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #12 CCTKi_InitialiseCactus /home/undercover/misc/tmp/Cactus/src/main/InitialiseCactus.c:117 (cactus_base-debug+0x152a179e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #13 main /home/undercover/misc/tmp/Cactus/src/main/flesh.cc:64 (cactus_base-debug+0x1528f271) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) SUMMARY: ThreadSanitizer: data race /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:323 in grhydro_atmospherereset_._omp_fn.0 ================== ``` The full output of this run until I stopped is attached in `run-debug.out` \(it?s rather large for a text file, sorry\). ? Once again, by disabling OpenMP at runtime everything seems fine, with the exception of the error ``` Cactus/arrangements/CactusNumerical/MoL/src/Operators.c:332:31: runtime error: variable length array bound evaluates to non-positive value 0 ``` which I don?t think is problematic, but I?ve decided to share anyways. ? If you have any general or specific tips, tricks or hacks to more accurately track down this bug, or on how to interpret the output of the previous tracebacks, would be much appreciated. ? # Machine information I?ve witness this behavior in two different machines, with similar OS?s: * Legion * Type: Laptop * Processor: Quad-core Intel\(R\) Core\(TM\) i5-7300HQ CPU @ 2.50GHz \(no hyper-threading\) * GPU: Integrated \+ Nvidia 1050 Ti Mobile * OS: Manjaro \(x86\_64\) * Kernel: Linux LTS 5.10.206 * GCC: 13.2.1 * OpenMP: 16.0.6 * OpenBLAS: 0.3.26 * hwloc: 2.10.0 * Gravitino * Type: Desktop * Processor: Octa-core Intel\(R\) Core\(TM\) i7-9700 @ 3.00GHz \(no hyper-threading\) * GPU: Nvidia 1050 Ti * OS: Arch Linux \(x86\_64\) * Kernel: Linux LTS 6.6.15 * GCC: 13.2.1 * OpenMP: 16.0.6 * OpenBLAS: 0.3.26 * hwloc: 2.10.0 There are no virtual environments, everything is managed by the global package manager using the latest releases in their corresponding repositories, and all binaries are linked against system libraries. ? If you need any more information about any of the machines, or any of the steps provided above, do no hesitate in replying to this thread. Thank you! attachment: backtrace.0.txt (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/backtrace.0.txt) attachment: make-binary.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/make-binary.out) attachment: make-binary-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/make-binary-debug.out) attachment: make-config.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/make-config.out) attachment: make-config-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/make-config-debug.out) attachment: run.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/run.out) attachment: run.sh (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/run.sh) attachment: run-debug.out (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2774/attachments/run-debug.out) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 7 03:13:35 2024 From: trac-noreply at einsteintoolkit.org (José Ferreira) Date: Wed, 07 Feb 2024 09:13:35 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: submitted Milestone: Version: ET_2023_11 Type: bug Priority: critical Component: Changes (by Jos? Ferreira): Hello, I am facing an issue where running a simulation with the toolkit, using the default thornlist, can either run as expected, or crash right in the beginning of the simulation. This took place with `ET_TOV.par`, that ships with the toolkit, and a thornfile that evolves a constant scalar field, that ships with the Scalar thorn also included in the toolkit. I believe that the culprit is OpenMP because if I disable threading during run-time, by setting `OMP_NUM_THREADS=1`, or if I disable OpenMP during compile-time, the simulations run as expected. This bug takes place both in my laptop and in my desktop, which share similar operating systems and software stack. ? In the following sections, I will write, line by line, the steps that I have performed in order to reproduce the bug, and my attempt of tracking it down. ? # Installing and Compiling the Toolkit To avoid compiling CarpetX thorns that fail to compile in my system, for some reason, I start by downloading the previous version of the toolkit with ```shell $ curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2023_05/GetComponents $ chmod a+x GetComponents $ ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2023_05/einsteintoolkit.th ``` and then change to the Cactus root directory ```shell $ cd Cactus ``` I create the options file `arch.cfg` , which should already be present in the parent directory, with the following written in it ```text # Cactus configuration for Arch and Arch-based distros ## Decide which flags will be used at compile-time OPTIMISE = yes WARN = yes DEBUG = no PROFILE = no OPENMP = yes ## Compilers CPP = cpp FPP = cpp CC = gcc CXX = g++ F77 = gfortran F90 = gfortran ## Default flags CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK FPPFLAGS = -traditional CFLAGS = -g3 -march=native -std=gnu99 CXXFLAGS = -g3 -march=native -std=gnu++0x F77FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none -fno-range-check F90FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none -fno-range-check LDFLAGS = -rdynamic ## Optimization flags CPP_OPTIMISE_FLAGS = -DKRANC_VECTORS # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -Ofast CXX_OPTIMISE_FLAGS = -Ofast F77_OPTIMISE_FLAGS = -Ofast F90_OPTIMISE_FLAGS = -Ofast ## Warning flags CPP_WARN_FLAGS = -Wall FPP_WARN_FLAGS = -Wall C_WARN_FLAGS = -Wall CXX_WARN_FLAGS = -Wall F77_WARN_FLAGS = -Wall F90_WARN_FLAGS = -Wall ## Debug flags CPP_DEBUG_FLAGS = -DCARPET_DEBUG -fsanitize=undefined -fsanitize=thread FPP_DEBUG_FLAGS = -DCARPET_DEBUG -fsanitize=undefined -fsanitize=thread C_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread CXX_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread F77_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread F90_DEBUG_FLAGS = -O0 -fsanitize=undefined -fsanitize=thread ## Code profiling flags CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg ## OpenMP CPP_OPENMP_FLAGS = -fopenmp FPP_OPENMP_FLAGS = -fopenmp C_OPENMP_FLAGS = -fopenmp CXX_OPENMP_FLAGS = -fopenmp F77_OPENMP_FLAGS = -fopenmp F90_OPENMP_FLAGS = -fopenmp ## Libraries location LIBDIRS = MPI_DIR = /usr HDF5_DIR = /usr PTHREADS_DIR = NO_BUILD LIBS = gfortran open-pal z C_LINE_DIRECTIVES = yes F_LINE_DIRECTIVES = yes ``` I then create the configuration folder with ```shell $ make base-config options=../arch.cfg THORNLIST=thornlists/einsteintoolkit.th ``` which reveals no errors, with the terminal output attached in file `make-config.out`, and then make the binary with ```shell $ make -j $(nproc) base ``` that also reveals no errors, and the output is attached in `make-binary.out`. ? To ensure full reproducibility, instead of doing this manually I created a very simple script, attached as `run.sh`, that reproduces the steps laid out in this section. ? # Running the Toolkit and Finding the Bug With no errors so far, and the binary `exe/cactus_base` in place, I will run one of the par files provided by default in the toolkit, in this case, `par/ET_TOV.par` ```shell $ exe/cactus_base par/tov_ET.par ``` which crashes, producing something that says ```text Rank 0 with PID 2329840 received signal 11 Writing backtrace to tov_ET/backtrace.0.txt [1] 2329840 segmentation fault (core dumped) ``` The full output for this simulation is sent in the file `run.out` that is being sent as an attachment, along with `backtrace.0.txt`. ? Interestingly, if insist on running this simulation for a few times, I will eventually find one where the bug doesn?t take place, and the simulation seems to be running as expected. Therefore, this is not just a classical bug, it?s a Heisenbug! ? # Tracking the Bug If I am to run the binary disabling OpenMP at runtime, i.e. ```shell $ OMP_NUM_THREADS=1 exe/cactus_base par/ET_TOV.par ``` then I consistently get no errors, even after running the code a few dozens of time. This is why I am lead to believe that the culprit of the Heisenbug is OpenMP. ? I created a new configuration of the toolkit with `DEBUG=yes` and `OPTIMIZE=no` in the options file above, which created the binary `exe/cactus_base-debug`. For completeness, the output of the making of the configuration is sent in `make-config-debug.out` and for the binary in `make-binary-debug.out`, although no errors were produced. I don?t have that much experience in debugging low-level code, so I decided to disable optimizations and add `-fsanitize=undefined` and `-fsanitize=thread` to GCC , which looked reasonable to me. ? Running the debug version of the binary with the same parfile ```shell $ exe/cactus_base-debug par/ET_TOV.par ``` reveals the same error on loop, and the program never terminates \(and by never I mean in around one minute\), and the errors look something like ```text ================== WARNING: ThreadSanitizer: data race (pid=2419680) Read of size 8 at 0x7ffe815ca630 by thread T5: #0 grhydro_atmospherereset_._omp_fn.0 /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:323 (cactus_base-debug+0xd14fbb9) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #1 gomp_thread_start /usr/src/debug/gcc/gcc/libgomp/team.c:129 (libgomp.so.1+0x20c95) (BuildId: 919d8c8c3093e63652b89795375dcf12dd9cb1d4) Previous write of size 8 at 0x7ffe815ca630 by main thread: #0 grhydro_atmospherereset_ /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:321 (cactus_base-debug+0xd11394a) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #1 CCTKi_BindingsFortranWrapperGRHydro /home/undercover/misc/tmp/Cactus/configs/base-debug/bindings/Variables/GRHydro.c:37 (cactus_base-debug+0x155440a4) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #2 CCTK_CallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:323 (cactus_base-debug+0x152bc047) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #3 CallScheduledFunction /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:440 (cactus_base-debug+0xa36120b) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #4 Carpet::CallFunction(void*, cFunctionData*, void*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CallFunction.cc:373 (cactus_base-debug+0xa35f44b) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #5 CCTKi_ScheduleCallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:3096 (cactus_base-debug+0x152c6b74) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #6 ScheduleTraverseFunction /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:595 (cactus_base-debug+0x152d6b94) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #7 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:369 (cactus_base-debug+0x152d5be6) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #8 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:385 (cactus_base-debug+0x152d673d) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #9 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:385 (cactus_base-debug+0x152d673d) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #10 CCTKi_DoScheduleTraverse /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:159 (cactus_base-debug+0x152d4d7e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #11 ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:1400 (cactus_base-debug+0x152bedf0) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #12 CCTK_ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:919 (cactus_base-debug+0x152bcedf) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #13 ScheduleTraverse /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:1393 (cactus_base-debug+0xa3f8929) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #14 CallRestrict /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:529 (cactus_base-debug+0xa3e8be8) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #15 Carpet::Initialise(tFleshConfig*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/Initialise.cc:121 (cactus_base-debug+0xa3ddc37) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #16 main /home/undercover/misc/tmp/Cactus/src/main/flesh.cc:80 (cactus_base-debug+0x1528f2e7) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) Location is stack of main thread. Location is global '' at 0x000000000000 ([stack]+0xf7630) Thread T5 (tid=2419694, running) created by main thread at: #0 pthread_create /usr/src/debug/gcc/gcc/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x44219) (BuildId: 7e8fcb9ed0a63b98f2293e37c92ac955413efd9e) #1 gomp_team_start /usr/src/debug/gcc/gcc/libgomp/team.c:858 (libgomp.so.1+0x212df) (BuildId: 919d8c8c3093e63652b89795375dcf12dd9cb1d4) #2 CarpetLib::dist::pseudoinit(ompi_communicator_t*) /home/undercover/misc/tmp/Cactus/arrangements/Carpet/CarpetLib/src/dist.cc:200 (cactus_base-debug+0xac0eac8) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #3 CarpetMultiModelStartup /home/undercover/misc/tmp/Cactus/arrangements/Carpet/Carpet/src/CarpetStartup.cc:29 (cactus_base-debug+0xa3733e3) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #4 CCTK_CallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:309 (cactus_base-debug+0x152bbf51) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #5 CCTKi_ScheduleCallFunction /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:3096 (cactus_base-debug+0x152c6b74) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #6 ScheduleTraverseFunction /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:595 (cactus_base-debug+0x152d6b94) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #7 ScheduleTraverseGroup /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:369 (cactus_base-debug+0x152d5be6) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #8 CCTKi_DoScheduleTraverse /home/undercover/misc/tmp/Cactus/src/schedule/ScheduleTraverse.c:159 (cactus_base-debug+0x152d4d7e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #9 ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:1400 (cactus_base-debug+0x152bedf0) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #10 CCTK_ScheduleTraverse /home/undercover/misc/tmp/Cactus/src/main/ScheduleInterface.c:919 (cactus_base-debug+0x152bcedf) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #11 CCTKi_CallStartupFunctions /home/undercover/misc/tmp/Cactus/src/main/CallStartupFunctions.c:50 (cactus_base-debug+0x1527f958) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #12 CCTKi_InitialiseCactus /home/undercover/misc/tmp/Cactus/src/main/InitialiseCactus.c:117 (cactus_base-debug+0x152a179e) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) #13 main /home/undercover/misc/tmp/Cactus/src/main/flesh.cc:64 (cactus_base-debug+0x1528f271) (BuildId: 9f545ec6a5fce94a14678be3027bcefa0e2d6645) SUMMARY: ThreadSanitizer: data race /home/undercover/misc/tmp/Cactus/arrangements/EinsteinEvolve/GRHydro/src/GRHydro_UpdateMask.F90:323 in grhydro_atmospherereset_._omp_fn.0 ================== ``` The full output of this run until I stopped is attached in `run-debug.out` \(it?s rather large for a text file, sorry\). ? Once again, by disabling OpenMP at runtime everything seems fine, with the exception of the error ```text Cactus/arrangements/CactusNumerical/MoL/src/Operators.c:332:31: runtime error: variable length array bound evaluates to non-positive value 0 ``` which I don?t think is problematic, but I?ve decided to share anyways. ? If you have any general or specific tips, tricks or hacks to more accurately track down this bug, or on how to interpret the output of the previous tracebacks, would be much appreciated. ? # Machine information I?ve witness this behavior in two different machines, with similar OS?s: * Legion * Type: Laptop * Processor: Quad-core Intel\(R\) Core\(TM\) i5-7300HQ CPU @ 2.50GHz \(no hyper-threading\) * GPU: Integrated \+ Nvidia 1050 Ti Mobile * OS: Manjaro \(x86\_64\) * Kernel: Linux LTS 5.10.206 * GCC: 13.2.1 * OpenMP: 16.0.6 * OpenBLAS: 0.3.26 * hwloc: 2.10.0 * Gravitino * Type: Desktop * Processor: Octa-core Intel\(R\) Core\(TM\) i7-9700 @ 3.00GHz \(no hyper-threading\) * GPU: Nvidia 1050 Ti * OS: Arch Linux \(x86\_64\) * Kernel: Linux LTS 6.6.15 * GCC: 13.2.1 * OpenMP: 16.0.6 * OpenBLAS: 0.3.26 * hwloc: 2.10.0 There are no virtual environments, everything is managed by the global package manager using the latest releases in their corresponding repositories, and all binaries are linked against system libraries. ? If you need any more information about any of the machines, or any of the steps provided above, do no hesitate in replying to this thread. Thank you! -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 7 10:38:16 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Wed, 07 Feb 2024 16:38:16 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: open Milestone: Version: ET_2023_11 Type: bug Priority: critical Component: Changes (by Roland Haas): status: open (was submitted) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 7 10:39:01 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Wed, 07 Feb 2024 16:39:01 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: open Milestone: ET_2024_05 Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): responsible: [] (was ) milestone: ET_2024_05 (was ) component: EinsteinToolkit thorn (was ) priority: major (was critical) assignee: Roland Haas (was ) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Fri Feb 9 03:25:02 2024 From: trac-noreply at einsteintoolkit.org (José Ferreira) Date: Fri, 09 Feb 2024 09:25:02 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: open Milestone: ET_2024_05 Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Jos? Ferreira): I?ve been trying to disable/enable thorns in the compilation procedure to see whether I could get something meaningful. After disabling Formaline during the compilation and in the par files, the incidence of the Heisenbug reduced significantly. Whereas before it would happen consistently on the par file that evolved a constant scalar field on Minkowski spacetime, that also ships with the toolkit in the Scalar thorn, meaning that I couldn?t run this simulation at all, now it happens very rarely. I made a Bash script to run the simulation on loop \(just for a single time step\) and it took a number of successful simulations in the order to the dozens to replicate the error. I would like to stress than the error always took place during the initialization procedure and never during the simulation. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 04:36:07 2024 From: trac-noreply at einsteintoolkit.org (José Ferreira) Date: Mon, 12 Feb 2024 10:36:07 +0000 Subject: [ET Trac] #2774: OpenMP Heisenbug with Default Thornfile Message-ID: #2774: OpenMP Heisenbug with Default Thornfile Reporter: Jos? Ferreira Status: open Milestone: ET_2024_05 Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Jos? Ferreira): I found a situation where the bug manifests itself seemingly every time: on trying to recover from checkpoints. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2774/openmp-heisenbug-with-default-thornfile -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 09:30:32 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 15:30:32 +0000 Subject: [ET Trac] #2755: bugs in POWER code Message-ID: #2755: bugs in POWER code Reporter: Anuj Kankani Status: open Milestone: Version: Type: bug Priority: major Component: Other Comment (by Roland Haas): Applied as git hash [bbcc63e](https://git.ncsa.illinois.edu/elihu/Gravitational_Waveform_Extractor/commits/bbcc63ea64be72c997871d28e7d047d6c230a19d) "POWER: correct expression for angular momentum once more" of [Gravitational\_Waveform\_Extractor](https://git.ncsa.illinois.edu/elihu/Gravitational_Waveform_Extractor) Backported to 2023\_11 as git hash [3db0d59](https://git.ncsa.illinois.edu/elihu/Gravitational_Waveform_Extractor/commits/3db0d59050934c648ad89ab07f092eb88ee7be94) "POWER: correct expression for angular momentum once more" of [Gravitational\_Waveform\_Extractor](https://git.ncsa.illinois.edu/elihu/Gravitational_Waveform_Extractor) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2755/bugs-in-power-code -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 09:31:38 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 15:31:38 +0000 Subject: [ET Trac] #2755: bugs in POWER code Message-ID: #2755: bugs in POWER code Reporter: Anuj Kankani Status: open Milestone: Version: Type: bug Priority: major Component: Other Comment (by Roland Haas): @{61d746178534980073cab4af} thank you for reviewing. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2755/bugs-in-power-code -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 09:31:49 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 15:31:49 +0000 Subject: [ET Trac] #2755: bugs in POWER code Message-ID: #2755: bugs in POWER code Reporter: Anuj Kankani Status: resolved Milestone: Version: Type: bug Priority: major Component: Other Changes (by Roland Haas): status: resolved (was open) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2755/bugs-in-power-code -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 09:53:34 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 15:53:34 +0000 Subject: [ET Trac] #2771: do not use non-standard include file "endian.h" in SGRID Message-ID: #2771: do not use non-standard include file "endian.h" in SGRID Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): responsible: [] (was ) assignee: Roland Haas (was ) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2771/do-not-use-non-standard-include-file -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 09:54:57 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 15:54:57 +0000 Subject: [ET Trac] #2771: do not use non-standard include file "endian.h" in SGRID Message-ID: #2771: do not use non-standard include file "endian.h" in SGRID Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): To be backported. @{557058:290a1cbd-99fe-4ac1-bd14-fd6b46a9a019} already has a better \(but larger\) fix in master -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2771/do-not-use-non-standard-include-file -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 10:12:08 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 16:12:08 +0000 Subject: [ET Trac] #2771: do not use non-standard include file "endian.h" in SGRID Message-ID: #2771: do not use non-standard include file "endian.h" in SGRID Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Better fix as git hash [93d909e](https://github.com/EinsteinToolkit/ExternalLibraries-SGRID/commits/93d909e08f2a9e488cf0a6be9ff8eee2d639b792) "update sgrid.tar to a4fe199d, that should work on MacOS" of [ExternalLibraries-SGRID](https://github.com/EinsteinToolkit/ExternalLibraries-SGRID) Backported as git hash [041bc1f](https://github.com/EinsteinToolkit/ExternalLibraries-SGRID/commits/041bc1fb29014f83f4877fc96c3f91784bf224ff) "SGRID: detect endianness at runtime" of [ExternalLibraries-SGRID](https://github.com/EinsteinToolkit/ExternalLibraries-SGRID) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2771/do-not-use-non-standard-include-file -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 10:12:43 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 16:12:43 +0000 Subject: [ET Trac] #2771: do not use non-standard include file "endian.h" in SGRID Message-ID: #2771: do not use non-standard include file "endian.h" in SGRID Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): @{557058:1671c5c3-29cc-4e83-9850-a152d33a6235} -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2771/do-not-use-non-standard-include-file -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 10:12:52 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 16:12:52 +0000 Subject: [ET Trac] #2771: do not use non-standard include file "endian.h" in SGRID Message-ID: #2771: do not use non-standard include file "endian.h" in SGRID Reporter: Roland Haas Status: resolved Milestone: Version: ET_2023_11 Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was open) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2771/do-not-use-non-standard-include-file -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 12 10:53:41 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 12 Feb 2024 16:53:41 +0000 Subject: [ET Trac] #2770: make ADIOS2 compile on macOS Message-ID: #2770: make ADIOS2 compile on macOS Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: Comment (by Roland Haas): This is fixed in ADIOS 2.9.2 so will not need to be fixed in master anymore.? This should be backported to the release though. So there?s two options: 1\. apply the patch here to the release version \(2023\_11\) only 2\. bump the release?s included tar file to 2.9.2 which fixes the issue but brings in other changes as well In SGRID #2771 I applied the patch to the release even after master received a fix to the tar file \(deeming the changes in the tar file to be more extensive than a minimal fix desired for the backport\). By the same argument one would apply the patch here, but the patch here is less ?nice? and changes files that I doubt are actually used on macOS \(being in directories named ?windows?\). Comments? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2770/make-adios2-compile-on-macos -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 15 15:44:05 2024 From: trac-noreply at einsteintoolkit.org (tootle) Date: Thu, 15 Feb 2024 21:44:05 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by tootle): @{557058:59e031ba-9bb5-4298-a472-7b99d0ae6f22} any updates on whether or not this will be possible to do? Until this is resolved there's no point in pushing further updates to the importer as MKL is prevalent on HPC systems. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 19 16:07:00 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 19 Feb 2024 22:07:00 +0000 Subject: [ET Trac] #2770: make ADIOS2 compile on macOS Message-ID: #2770: make ADIOS2 compile on macOS Reporter: Roland Haas Status: open Milestone: Version: ET_2023_11 Type: bug Priority: major Component: Comment (by Roland Haas): Comments in the ET call on Feb 15th say to backport the tar file. Done in git hash [4caaaaa](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/commits/4caaaaa4820c01866dfc874808fbfcf173dd7329) "ADIOS2: update to 2.9.2" of [ExternalLibraries-ADIOS2](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2770/make-adios2-compile-on-macos -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 19 16:07:08 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 19 Feb 2024 22:07:08 +0000 Subject: [ET Trac] #2770: make ADIOS2 compile on macOS Message-ID: #2770: make ADIOS2 compile on macOS Reporter: Roland Haas Status: resolved Milestone: Version: ET_2023_11 Type: bug Priority: major Component: Changes (by Roland Haas): status: resolved (was open) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2770/make-adios2-compile-on-macos -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Feb 20 08:14:27 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 20 Feb 2024 14:14:27 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): How about adding \`$\{FFTW3\_LIBS\}\` to KadathThorn?s LIBS output in detect.sh ? Like HDF5 does: ``` HDF5_LIB_DIRS="$HDF5_LIB_DIRS $MPI_LIB_DIRS" HDF5_INC_DIRS="$HDF5_INC_DIRS $MPI_INC_DIRS" HDF5_LIBS="$HDF5_LIBS $MPI_LIBS" ``` That lets you ensure that FFTW3\_LIBS is right after Kadath?s. This should work for static linking. For dynamic linking adding it to `LD` should work. So it?s a cluster specific solution \(at least for `LD`\) but so is this issue you are encountering. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Feb 20 10:04:24 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 20 Feb 2024 16:04:24 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Testing this I can nudge things from \(the not correct\): ``` g++ -o "/data/rhaas/postdoc/gr/cactus/ET_trunk/exe/cactus_fuka" -fopenmp -Wall -g -std=gnu++17 -O2 -fopenmp -Wall -rdynamic "/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/datestamp.o" -Wl,--whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/lib -lthorn_ADMBase -lthorn_ADMCoupling -lthorn_ADMMacros -lthorn_AEILocalInterp -lthorn_AHFinderDirect -lthorn_BLAS -lthorn_Boost -lthorn_Boundary -lthorn_Carpet -lthorn_CarpetIOHDF5 -lthorn_CarpetIOScalar -lthorn_CarpetInterp -lthorn_CarpetLib -lthorn_CarpetMask -lthorn_CarpetReduce -lthorn_CarpetSlab -lthorn_CartGrid3D -lthorn_CoordBase -lthorn_CoordGauge -lthorn_CycleClock -lthorn_FFTW3 -lthorn_GSL -lthorn_HDF5 -lthorn_HydroBase -lthorn_IOASCII -lthorn_IOBasic -lthorn_IOUtil -lthorn_InitBase -lthorn_KadathImporter -lthorn_KadathThorn -lthorn_LAPACK -lthorn_LocalInterp2 -lthorn_LocalReduce -lthorn_LoopControl -lthorn_MPI -lthorn_MoL -lthorn_NaNChecker -lthorn_PUGH -lthorn_SpaceMask -lthorn_SphericalSurface -lthorn_StaticConformal -lthorn_SymBase -lthorn_SystemTopology -lthorn_Time -lthorn_Timers -lthorn_TmunuBase -lthorn_Vectors -lthorn_hwloc -lthorn_zlib -lthorn_Cactus -lthorn_CactusBindings -Wl,--no-whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/13 -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/hdf5/openmpi -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -lkadath -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lgsl -lgslcblas -lm -lfftw3 -lboost_filesystem -lboost_system -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lmpi -lhwloc -levent_core -levent_pthreads -lm -lz -lsz -lz -lmpi_cxx -lmpi -lm -lz -lmpi_cxx -lmpi -lhwloc -lm -ludev -lpthread -lz -lm -lm -lm -lm -lm -lm -lm -lm -lcrypt -lgfortran ``` to the \(more correct\): ``` g++ -o "/data/rhaas/postdoc/gr/cactus/ET_trunk/exe/cactus_fuka" -fopenmp -Wall -g -std=gnu++17 -O2 -fopenmp -Wall -rdynamic "/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/datestamp.o" -Wl,--whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/lib -lthorn_ADMBase -lthorn_ADMCoupling -lthorn_ADMMacros -lthorn_AEILocalInterp -lthorn_AHFinderDirect -lthorn_BLAS -lthorn_Boost -lthorn_Boundary -lthorn_Carpet -lthorn_CarpetIOHDF5 -lthorn_CarpetIOScalar -lthorn_CarpetInterp -lthorn_CarpetLib -lthorn_CarpetMask -lthorn_CarpetReduce -lthorn_CarpetSlab -lthorn_CartGrid3D -lthorn_CoordBase -lthorn_CoordGauge -lthorn_CycleClock -lthorn_FFTW3 -lthorn_GSL -lthorn_HDF5 -lthorn_HydroBase -lthorn_IOASCII -lthorn_IOBasic -lthorn_IOUtil -lthorn_InitBase -lthorn_KadathImporter -lthorn_KadathThorn -lthorn_LAPACK -lthorn_LocalInterp2 -lthorn_LocalReduce -lthorn_LoopControl -lthorn_MPI -lthorn_MoL -lthorn_NaNChecker -lthorn_PUGH -lthorn_SpaceMask -lthorn_SphericalSurface -lthorn_StaticConformal -lthorn_SymBase -lthorn_SystemTopology -lthorn_Time -lthorn_Timers -lthorn_TmunuBase -lthorn_Vectors -lthorn_hwloc -lthorn_zlib -lthorn_Cactus -lthorn_CactusBindings -Wl,--no-whole-archive -L/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -L/usr/lib/gcc/x86_64-linux-gnu/13 -Wl,-rpath,/data/rhaas/postdoc/gr/cactus/ET_trunk/configs/fuka/scratch/external/KadathThorn/lib -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/opt/intel/oneapi/mkl/2021.3.0/lib/intel64 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/hdf5/openmpi -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/13 -lkadath -lfftw3 -lgsl -lgslcblas -lm -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lboost_filesystem -lboost_system -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_scalapack_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lgsl -lgslcblas -lm -lfftw3 -lboost_filesystem -lboost_system -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lmpi -lhwloc -levent_core -levent_pthreads -lm -lz -lsz -lz -lmpi_cxx -lmpi -lm -lz -lmpi_cxx -lmpi -lhwloc -lm -ludev -lpthread -lz -lm -lm -lm -lm -lm -lm -lm -lm -lcrypt -lgfortran ``` Change to KadathThorn: ```diff diff --git a/src/detect.sh b/src/detect.sh index 9f80ed6..d5be1a0 100755 --- a/src/detect.sh +++ b/src/detect.sh @@ -89,9 +89,10 @@ echo "END MESSAGE" # Pass options to Cactus echo "BEGIN MAKE_DEFINITION" echo "KADATH_DIR = ${KADATH_DIR}" -echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS}" -echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}" -echo "KADATH_LIBS = ${KADATH_LIBS}" +echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS} ${FFTW3_INC_DIRS} ${GSL_INC_DIRS} ${LAPACK_INC_DIRS} ${BOOST_INC_DIRS}" +echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}" ${FFTW3_LIB_DIRS} ${GSL_LIB_DIRS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS} +# keep the order like this, since MKL (LAPACK) can contain a (partial) FFTW3 which conflicts +echo "KADATH_LIBS = ${KADATH_LIBS} ${FFTW3_LIBS} ${GSL_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}" echo "END MAKE_DEFINITION" echo 'INCLUDE_DIRECTORY $(KADATH_INC_DIRS)' ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Feb 20 10:05:07 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 20 Feb 2024 16:05:07 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: open (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Feb 20 10:05:20 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 20 Feb 2024 16:05:20 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Please review. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 00:45:01 2024 From: trac-noreply at einsteintoolkit.org (tootle) Date: Thu, 22 Feb 2024 06:45:01 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by tootle): @{557058:59e031ba-9bb5-4298-a472-7b99d0ae6f22} Thanks! Indeed that was enough \(with slight modification\) to clean up compiling! I have pushed the latest to KadathThorn master so it will be ready for the next rel ```diff diff --git a/src/detect.sh b/src/detect.sh index 9f80ed6..5347190 100755 --- a/src/detect.sh +++ b/src/detect.sh @@ -89,9 +89,10 @@ echo "END MESSAGE" # Pass options to Cactus echo "BEGIN MAKE_DEFINITION" echo "KADATH_DIR = ${KADATH_DIR}" -echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS}" -echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}" -echo "KADATH_LIBS = ${KADATH_LIBS}" +echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS} ${FFTW3_INC_DIRS} ${GSL_INC_DIRS} ${LAPACK_INC_DIRS} ${BOOST_INC_DIRS}" +echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS} ${FFTW3_LIB_DIRS} ${GSL_LIB_DIRS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}" +# keep the order like this, since MKL (LAPACK) can contain a (partial) FFTW3 which conflicts +echo "KADATH_LIBS = ${KADATH_LIBS} ${FFTW3_LIBS} ${GSL_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}" echo "END MAKE_DEFINITION" echo 'INCLUDE_DIRECTORY $(KADATH_INC_DIRS)' ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 01:04:11 2024 From: trac-noreply at einsteintoolkit.org (tootle) Date: Thu, 22 Feb 2024 07:04:11 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by tootle): @{557058:59e031ba-9bb5-4298-a472-7b99d0ae6f22} Indeed that worked! Here is the final version with minor modification: ```diff diff --git a/src/detect.sh b/src/detect.sh index 9f80ed6..3fe691b 100755 --- a/src/detect.sh +++ b/src/detect.sh @@ -82,16 +82,18 @@ KADATH_LIBS="kadath" echo "BEGIN MESSAGE" echo "KadathThorn: Detect settings - KADATH_DIR : ${KADATH_DIR}" -echo "KadathThorn: Detect settings - KADATH_LIB_DIRS : ${KADATH_LIB_DIRS}" -echo "KadathThorn: Detect settings - KADATH_LIBS : ${KADATH_LIBS}" +echo "KadathThorn: Detect settings - KADATH_INC_DIRS : ${KADATH_INC_DIRS}" +echo "KadathThorn: Detect settings - KADATH_LIB_DIRS : ${KADATH_LIB_DIRS} ${GSL_LIB_DIRS} ${FFTW3_LIB_DIRS} ${BLAS_LIB_DIRSS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}" +echo "KadathThorn: Detect settings - KADATH_LIBS : ${KADATH_LIBS} ${GSL_LIBS} ${FFTW3_LIBS} ${BLAS_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}" echo "END MESSAGE" # Pass options to Cactus echo "BEGIN MAKE_DEFINITION" echo "KADATH_DIR = ${KADATH_DIR}" echo "KADATH_INC_DIRS = ${KADATH_INC_DIRS}" -echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS}" -echo "KADATH_LIBS = ${KADATH_LIBS}" +echo "KADATH_LIB_DIRS = ${KADATH_LIB_DIRS} ${GSL_LIB_DIRS} ${FFTW3_LIB_DIRS} ${BLAS_LIB_DIRSS} ${LAPACK_LIB_DIRS} ${BOOST_LIB_DIRS}" +# keep the order like this, since MKL (LAPACK) can contain a (partial) FFTW3 which conflicts +echo "KADATH_LIBS = ${KADATH_LIBS} ${GSL_LIBS} ${FFTW3_LIBS} ${BLAS_LIBS} ${LAPACK_LIBS} ${BOOST_LIBS}" echo "END MAKE_DEFINITION" echo 'INCLUDE_DIRECTORY $(KADATH_INC_DIRS)' ``` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 08:32:20 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 14:32:20 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): I did some more tests and incorporated GRHydro?s PPM \( Ian Hawke, Toni Font, Luca Baiotti, Frank Loeffler\) to my test and find that PPM does reconstruction not interpolation. So WENO \(which does interpolation\) should be changed. Attached please find the test code and test output. ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 08:33:20 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 14:33:20 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Code and output. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 08:33:20 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 14:33:20 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Code and output. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 08:43:56 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 14:43:56 +0000 Subject: [ET Trac] #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Message-ID: #2772: Issues link against Kadath thorns and Intel MKL due to link ordering Reporter: tootle Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Looks good to me. Please apply. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2772/issues-link-against-kadath-thorns-and -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 14:58:04 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 20:58:04 +0000 Subject: [ET Trac] #2775: FLRWSolver: work around Cactus include search path mangling Message-ID: #2775: FLRWSolver: work around Cactus include search path mangling Reporter: Roland Haas Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Cactus' ExternalLibraries system removes the "system" locations from any INC\_DIRS, but not all compilers look at the "well known" locations on their own. So the machine definition files \(in the Einstein Toolkit\) try to work around by setting CPATH which is however not used by gfortran, only by cpp. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2775/flrwsolver-work-around-cactus-include -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 14:58:21 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 20:58:21 +0000 Subject: [ET Trac] #2775: FLRWSolver: work around Cactus include search path mangling Message-ID: #2775: FLRWSolver: work around Cactus include search path mangling Reporter: Roland Haas Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Pull request is here: [https://github.com/hayleyjm/FLRWSolver\_public/pull/3](https://github.com/hayleyjm/FLRWSolver_public/pull/3) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2775/flrwsolver-work-around-cactus-include -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 14:58:42 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 20:58:42 +0000 Subject: [ET Trac] #2775: FLRWSolver: work around Cactus include search path mangling Message-ID: #2775: FLRWSolver: work around Cactus include search path mangling Reporter: Roland Haas Status: new Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Please review -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2775/flrwsolver-work-around-cactus-include -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 22 14:58:54 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Thu, 22 Feb 2024 20:58:54 +0000 Subject: [ET Trac] #2775: FLRWSolver: work around Cactus include search path mangling Message-ID: #2775: FLRWSolver: work around Cactus include search path mangling Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: open (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2775/flrwsolver-work-around-cactus-include -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Sun Feb 25 08:26:06 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Sun, 25 Feb 2024 14:26:06 +0000 Subject: [ET Trac] #2776: zlib: also use pkg-config when searching for installed version Message-ID: #2776: zlib: also use pkg-config when searching for installed version Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn this helps on macOS using the system installed zlib -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2776/zlib-also-use-pkg-config-when-searching -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Sun Feb 25 08:26:22 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Sun, 25 Feb 2024 14:26:22 +0000 Subject: [ET Trac] #2776: zlib: also use pkg-config when searching for installed version Message-ID: #2776: zlib: also use pkg-config when searching for installed version Reporter: Roland Haas Status: open Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Changes (by Roland Haas): status: open (was new) Comment (by Roland Haas): Pull request is here: https://github.com/EinsteinToolkit/ExternalLibraries-zlib/pull/2 -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2776/zlib-also-use-pkg-config-when-searching -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Sun Feb 25 08:26:34 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Sun, 25 Feb 2024 14:26:34 +0000 Subject: [ET Trac] #2776: zlib: also use pkg-config when searching for installed version Message-ID: #2776: zlib: also use pkg-config when searching for installed version Reporter: Roland Haas Status: open Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): Please review. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2776/zlib-also-use-pkg-config-when-searching -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Sun Feb 25 14:15:30 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Sun, 25 Feb 2024 20:15:30 +0000 Subject: [ET Trac] #2526: McLachlan has incorrect READ declarations for presync. Message-ID: #2526: McLachlan has incorrect READ declarations for presync. Reporter: Peter Diener Status: new Milestone: ET_2023_05 Version: development version Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Applied as git hash [b4b2b40](https://github.com/ianhinder/Kranc/commits/b4b2b40103a706a29f8f6b3910110a30afff75aa) "Schedule.m: remove unused Shorthands when creating READS/WRITES" of [Kranc](https://github.com/ianhinder/Kranc) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2526/mclachlan-has-incorrect-read-declarations -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Sun Feb 25 14:15:43 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Sun, 25 Feb 2024 20:15:43 +0000 Subject: [ET Trac] #2526: McLachlan has incorrect READ declarations for presync. Message-ID: #2526: McLachlan has incorrect READ declarations for presync. Reporter: Peter Diener Status: resolved Milestone: ET_2023_05 Version: development version Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2526/mclachlan-has-incorrect-read-declarations -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 26 14:44:34 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 26 Feb 2024 20:44:34 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): maple worksheet showing explicitly how Aminus is computed. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 26 14:45:00 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 26 Feb 2024 20:45:00 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): PDF version -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 26 14:47:11 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 26 Feb 2024 20:47:11 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Mon Feb 26 14:47:49 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Mon, 26 Feb 2024 20:47:49 +0000 Subject: [ET Trac] #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Message-ID: #2647: incorrect WENO coefficient in GRHydro WENO reconstruction code Reporter: Roland Haas Status: open Milestone: Version: Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2647/incorrect-weno-coefficient-in-grhydro-weno -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Tue Feb 27 08:19:04 2024 From: trac-noreply at einsteintoolkit.org (Roland Haas) Date: Tue, 27 Feb 2024 14:19:04 +0000 Subject: [ET Trac] #2776: zlib: also use pkg-config when searching for installed version Message-ID: #2776: zlib: also use pkg-config when searching for installed version Reporter: Roland Haas Status: open Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit thorn Comment (by Roland Haas): @{557058:1671c5c3-29cc-4e83-9850-a152d33a6235} suggested to fully move zlib over to using `bash_utils` instead. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2776/zlib-also-use-pkg-config-when-searching -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:09:25 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:09:25 +0000 Subject: [ET Trac] #2777: Upgrade IllinoisGRMHD to use GRHayL Message-ID: #2777: Upgrade IllinoisGRMHD to use GRHayL Reporter: Samuel Cupp Status: new Milestone: Version: Type: enhancement Priority: major Component: This major overhaul of the `IllinoisGRMHD` thorn uses `GRHayL` to implement its original features. In addition, it introduces entropy evolution, tabulated equation of state, and piecewise polytrope support. All con2prim routines provided by `GRHayL` are also automatically supported by the thorn, with control of this behavior governed by the `GRHayLib` thorn. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2777/upgrade-illinoisgrmhd-to-use-grhayl -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:12:19 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:12:19 +0000 Subject: [ET Trac] #2778: GRHayLHD(X) update for tabulated EOS support Message-ID: #2778: GRHayLHD(X) update for tabulated EOS support Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: enhancement Priority: major Component: Previous release only included hybrid EOS for `GRHayLHD` and `GRHayLHDX`, as the tabulated code was still being debugged. This update includes full tabulated support, as well as code simplification/cleanup. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2778/grhaylhd-x-update-for-tabulated-eos -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:12:47 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:12:47 +0000 Subject: [ET Trac] #2777: Upgrade IllinoisGRMHD to use GRHayL Message-ID: #2777: Upgrade IllinoisGRMHD to use GRHayL Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: enhancement Priority: major Component: Changes (by Samuel Cupp): milestone: ET_2024_05 (was ) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2777/upgrade-illinoisgrmhd-to-use-grhayl -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:23:42 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:23:42 +0000 Subject: [ET Trac] #2779: NRPy 2.0 update for Baikal(Vacuum) Message-ID: #2779: NRPy 2.0 update for Baikal(Vacuum) Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: `NRPy+` has been significantly updated during the past year, with the 2.0 release and subsequent updates. This PR changes `Baikal` and `BaikalVacuum` to be generated using the new `NRPy+`. Other than accumulated improvements/changes to code generation, this update has two notable changes: 1. parameters currently don?t restrict their options, as the autogeneration doesn?t yet support this capability 2. `NewRad`'s radpower parameter was changed from `2.0` to `1.0` for `aDD` and `trK` ? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2779/nrpy-20-update-for-baikal-vacuum -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:24:44 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:24:44 +0000 Subject: [ET Trac] #2779: NRPy 2.0 update for Baikal(Vacuum) Message-ID: #2779: NRPy 2.0 update for Baikal(Vacuum) Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: Comment (by Samuel Cupp): Terrence Pierre Jacques is the reviewer. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2779/nrpy-20-update-for-baikal-vacuum -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:25:34 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:25:34 +0000 Subject: [ET Trac] #2779: NRPy 2.0 update for Baikal(Vacuum) Message-ID: #2779: NRPy 2.0 update for Baikal(Vacuum) Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: Comment (by Samuel Cupp): The associated PR is found [here](https://bitbucket.org/zach_etienne/wvuthorns/pull-requests/15). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2779/nrpy-20-update-for-baikal-vacuum -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Wed Feb 28 15:57:55 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Wed, 28 Feb 2024 21:57:55 +0000 Subject: [ET Trac] #2780: Include BaikalX and BaikalVacuumX in ET_2024_05 release Message-ID: #2780: Include BaikalX and BaikalVacuumX in ET_2024_05 release Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: enhancement Priority: major Component: `BaikalX` and `BaikalVacuumX` provide the same features of `Baikal` and `BaikalVacuum` for the `CarpetX` driver and are generated using `NRPy+` 2.0. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2780/include-baikalx-and-baikalvacuumx-in -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 29 15:01:21 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Thu, 29 Feb 2024 21:01:21 +0000 Subject: [ET Trac] #2781: Fix bug in debug mode with presync_mode = "off" Message-ID: #2781: Fix bug in debug mode with presync_mode = "off" Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: Cactus Currently, `CCTK_HasAccess` only checks access if `presync_mode` is set to `presync_only`. However, with debug on \(`CCTK_DEBUG` is defined\), it instead always checks access. ? The first part of this code section asserts that the function has the attribute RDWR: ``` assert(current_function->RDWR); ``` This is incompatible with the form of `CCTKi_CreateRDWRData`, which immediately returns if this parameter is set to `off`, meaning that this attribute is never created. As such, `presync_mode="off"` cannot run in debug mode, which is especially unacceptable for the default parameter setting. If presync is completely off, then it should never enter this section of the code. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2781/fix-bug-in-debug-mode-with-presync_mode -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 29 15:04:05 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Thu, 29 Feb 2024 21:04:05 +0000 Subject: [ET Trac] #2781: Fix bug in debug mode with presync_mode = "off" Message-ID: #2781: Fix bug in debug mode with presync_mode = "off" Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: Cactus Changes (by Samuel Cupp): responsible: [] (was ) assignee: Steven R. Brandt (was ) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2781/fix-bug-in-debug-mode-with-presync_mode -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac-noreply at einsteintoolkit.org Thu Feb 29 15:04:34 2024 From: trac-noreply at einsteintoolkit.org (Samuel Cupp) Date: Thu, 29 Feb 2024 21:04:34 +0000 Subject: [ET Trac] #2781: Fix bug in debug mode with presync_mode = "off" Message-ID: #2781: Fix bug in debug mode with presync_mode = "off" Reporter: Samuel Cupp Status: new Milestone: ET_2024_05 Version: Type: bug Priority: major Component: Cactus Comment (by Samuel Cupp): The associated PR: [https://bitbucket.org/cactuscode/cactus/pull-requests/159](https://bitbucket.org/cactuscode/cactus/pull-requests/159) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2781/fix-bug-in-debug-mode-with-presync_mode -------------- next part -------------- An HTML attachment was scrubbed... URL: