#2902: CarpetX: interpolation with z-reflection symmetry only works for Psi4
Reporter: Miren Radia
Status: submitted
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Currently the end of `CarpetX::interpolate` looks like this:
```c++
// Apply symmetries to interpolated values
assert(!reflection_x);
assert(!reflection_y);
assert(!reflection_upper_x);
assert(!reflection_upper_y);
assert(!reflection_upper_z);
if (reflection_z) {
// The code below is only valid for Psi4
assert(nvars == 2);
assert(varinds[0] == CCTK_VarIndex("Weyl::Psi4re"));
assert(varinds[1] == CCTK_VarIndex("Weyl::Psi4im"));
// l^a = et^a + er^a
// n^a = et^a - er^a
// m^a = etheta^a + i ephi^a
// Psi4 = C_abcd m-bar^b n^b m-bar^c n^d
for (int n = 0; n < npoints; ++n) {
if (symmetry_reflected_z[n]) {
resultptrs[0][n] = -resultptrs[0][n];
resultptrs[1][n] = +resultptrs[1][n];
}
}
}
```
\([link to source](https://github.com/EinsteinToolkit/CarpetX/blob/2cba1e969badbe8e6a1…)
These assertions are failed when you have z-reflection symmetry and try to interpolate different variables.
Note that has already been reported in [an issue in the CarpetX GitHub repo](https://github.com/EinsteinToolkit/CarpetX/issues/146) but I thought I’d open a ticket here given the lack of activity there.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2902/carpetx-interpola…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Here's the parameter file I mentioned in the previous comment.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Miren Radia):
Liwei, I’m not sure if your [puncture tracker PR](https://github.com/EinsteinToolkit/SpacetimeX/pull/59) has fully resolved the issues with poison for me. I cherry-picked the commits onto your [subcycling development branch](https://github.com/lwJi/SpacetimeX/tree/development) and received the following error when I tried to run with the `qc0-SC.par` parameter file I’ll try to attach after I’ve posted this comment.
```
INFO (PunctureTracker): Puncture #0 is at (1.16864,0,0)
INFO (PunctureTracker): Puncture #1 is at (-1.16864,0,0)
INFO (PunctureTracker): Shift at puncture #0 is at (0,0,0)
INFO (PunctureTracker): Shift at puncture #1 is at (0,0,0)
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_POSTSTEP
INFO (CarpetX): CallFunction iteration 1 CCTK_POSTSTEP: BoxInBox::EstimateError
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_CHECKPOINT
INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: CarpetX::CarpetX_Checkpoint
INFO (CarpetX): CallFunction iteration 1 CCTK_CHECKPOINT: TimerReport::zzz_TimerReport_Checkpoint
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_ANALYSIS
INFO (CarpetX): CallFunction iteration 1 CCTK_ANALYSIS: TimerReport::zzz_TimerReport_Output
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_AnalysisGroup: Z4cowGPU::Z4cowGPU_Constraints
INFO (CarpetX): OutputGH: iteration 1, time 0.416667, run time 6 s
INFO (CarpetX): OutputGH done.
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_PRESTEP
INFO (CarpetX): ScheduleTraverseGH iteration 1 CCTK_EVOL
INFO (CarpetX): CallFunction iteration 1 CCTK_EVOL: ODESolvers::ODESolvers_Solve_Subcycling
INFO (ODESolvers): Integrator is RK4
INFO (ODESolvers): Integrating 22 variables
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_OLD, Z4COWGPU::GAMMA_TILDE_OLD, Z4COWGPU::K_HAT_OLD, Z4COWGPU::A_TILDE_OLD, Z4COWGPU::GAM_TILDE_OLD, Z4COWGPU::THETA_OLD, Z4COWGPU::ALPHAG_OLD, Z4COWGPU::BETAG_OLD
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K1, Z4COWGPU::GAMMA_TILDE_K1, Z4COWGPU::K_HAT_K1, Z4COWGPU::A_TILDE_K1, Z4COWGPU::GAM_TILDE_K1, Z4COWGPU::THETA_K1, Z4COWGPU::ALPHAG_K1, Z4COWGPU::BETAG_K1
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K2, Z4COWGPU::GAMMA_TILDE_K2, Z4COWGPU::K_HAT_K2, Z4COWGPU::A_TILDE_K2, Z4COWGPU::GAM_TILDE_K2, Z4COWGPU::THETA_K2, Z4COWGPU::ALPHAG_K2, Z4COWGPU::BETAG_K2
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K3, Z4COWGPU::GAMMA_TILDE_K3, Z4COWGPU::K_HAT_K3, Z4COWGPU::A_TILDE_K3, Z4COWGPU::GAM_TILDE_K3, Z4COWGPU::THETA_K3, Z4COWGPU::ALPHAG_K3, Z4COWGPU::BETAG_K3
INFO (CarpetX): SyncGroupsProlongateOnly Z4COWGPU::W_K4, Z4COWGPU::GAMMA_TILDE_K4, Z4COWGPU::K_HAT_K4, Z4COWGPU::A_TILDE_K4, Z4COWGPU::GAM_TILDE_K4, Z4COWGPU::THETA_K4, Z4COWGPU::ALPHAG_K4, Z4COWGPU::BETAG_K4
INFO (ODESolvers): Set interior old state at t=0, to be prolongated later
INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #1 at t=0
INFO (ODESolvers): Calculating RHS #1 at t=0
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_Apply_NewRadX_BC
INFO (ODESolvers): Set interior Ks for stage #1 at t=0, to be prolongated later
INFO (ODESolvers): Calculated new state #1 at t=0
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Sync
INFO (CarpetX): SyncGroups Z4COWGPU::W, Z4COWGPU::GAMMA_TILDE, Z4COWGPU::K_HAT, Z4COWGPU::A_TILDE, Z4COWGPU::GAM_TILDE, Z4COWGPU::THETA, Z4COWGPU::ALPHAG, Z4COWGPU::BETAG
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_Enforce
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_PostStepGroup: Z4cowGPU::Z4cowGPU_ADM
INFO (CarpetX): CallFunction iteration 1 TmunuBaseX_SetTmunuVars: TmunuBaseX::TmunuBaseX_ZeroTmunu
INFO (ODESolvers): Fill refinement boundary ghost zones using Ys for stage #2 at t=0.104167
INFO (ODESolvers): Calculating RHS #2 at t=0.104167
INFO (CarpetX): CallFunction iteration 1 Z4cowGPU_RHSGroup: Z4cowGPU::Z4cowGPU_RHS
ERROR from host tu-c0r0n72 process 0
in thorn CarpetX, file /mnt/lustre/tursafs1/home/dp415/dp415/dc-radi1/ETK/Cactus-CarpetX-subcycling-spack/arrangements/CarpetX/CarpetX/src/valid.cxx:552:
-> CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track checking output: Grid array "PUNCTURETRACKER::pt_loc_t[0]" has 1 nans on time level 0; expected valid
The interior is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
The outer boundary is valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
The ghost zones are valid because: CallFunction iteration 1 CCTK_EVOL: PunctureTracker::PunctureTracker_Track: Mark output variables as valid.
```
I guess it could be related to subcycling? Do you have any idea?
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2892: SpacetimeX: GPU error in PunctureTracker::PunctureContainer::interpolate
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: major
Component: CarpetX
Comment (by Liwei Ji):
Thanks @{61d89c2368926d0068827aa3} , for figuring the issue out
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2892/spacetimex-gpu-er…
#2901: ExternalLibraries-ADIOS2: detect.sh adds nonexistent MPI libraries if ADIOS2_ENABLE_SST == yes when using external ADIOS2
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: minor
Component: EinsteinToolkit thorn
Comment (by Miren Radia):
My hacky fix was this change:
```diff
diff --git a/src/detect.sh b/src/detect.sh
index 7fce31d..087c260 100755
--- a/src/detect.sh
+++ b/src/detect.sh
@@ -45,6 +45,7 @@ ADIOS2_REQ_LIBS="adios2_cxx11 adios2_c adios2_core"
if [ "$(echo ${ADIOS2_ENABLE_FORTRAN} | tr '[:upper:]' '[:lower:]')" = 'yes' ]; then
ADIOS2_REQ_LIBS="adios2_fortran $ADIOS2_REQ_LIBS"
fi
+ADIOS2_MPI_LIBS="$ADIOS2_REQ_LIBS"
if [ "$(echo ${ADIOS2_ENABLE_SST} | tr '[:upper:]' '[:lower:]')" = 'yes' ] ; then
# core depends on adios2_evpath
ADIOS2_REQ_LIBS="$ADIOS2_REQ_LIBS adios2_evpath adios2_atl adios2_enet adios2_ffs adios2_dill"
@@ -82,7 +83,7 @@ if [ -z "${ADIOS2_BUILD}" -a -z "${ADIOS2_INC_DIRS}" -a -z "${ADIOS2_LIB_DIRS}"
fi
if [ $test_mpi -eq 0 ]; then
mpi_libs="" # need to prepend MPI libs
- for lib in $ADIOS2_REQ_LIBS ; do
+ for lib in $ADIOS2_MPI_LIBS ; do
mpi_libs="$mpi_libs ${lib}_mpi"
done
ADIOS2_LIBS="$mpi_libs $ADIOS2_LIBS"
```
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2901/externallibraries…
#2901: ExternalLibraries-ADIOS2: detect.sh adds nonexistent MPI libraries if ADIOS2_ENABLE_SST == yes when using external ADIOS2
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: minor
Component: EinsteinToolkit thorn
Changes (by Roland Haas):
responsible: [] (was )
assignee: Roland Haas (was )
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2901/externallibraries…
#2901: ExternalLibraries-ADIOS2: detect.sh adds nonexistent MPI libraries if ADIOS2_ENABLE_SST == yes when using external ADIOS2
Reporter: Miren Radia
Status: open
Milestone:
Version: development version
Type: bug
Priority: minor
Component: EinsteinToolkit thorn
Changes (by Roland Haas):
status: open (was submitted)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2901/externallibraries…
#2901: ExternalLibraries-ADIOS2: detect.sh adds nonexistent MPI libraries if ADIOS2_ENABLE_SST == yes when using external ADIOS2
Reporter: Miren Radia
Status: submitted
Milestone:
Version: development version
Type: bug
Priority: minor
Component: EinsteinToolkit thorn
Changes (by Miren Radia):
I opened [an issue in the GitHub repository](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/iss… but am opening this ticket here as suggested by Roland Haas. The GitHub issue description is below
In the case `ADIOS2_ENABLE_SST == yes`, after [this line](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b…
```shell
ADIOS2_REQ_LIBRARIES="adios2_cxx11 adios2_c adios2_core adios2_evpath adios2_atl adios2_enet adios2_ffs adios2_dill"
```
\(maybe with a fortran one too depending on that config option\). Then [later on](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b71…, all of these libraries are appended by `_mpi` so the full library list looks something like
```shell
ADIOS2_LIBS="adios2_cxx11_mpi adios2_c_mpi adios2_core_mpi adios2_evpath_mpi adios2_atl_mpi adios2_enet_mpi adios2_ffs_mpi adios2_dill_mpi adios2_cxx11 adios2_c adios2_core adios2_evpath adios2_atl adios2_enet adios2_ffs adios2_dill"
```
Unfortunately, looking at my Spack build of ADIOS2, I only have the following in the `lib64` directory \(actually it's the same as the one built by Cactus using the `build.sh` script\):
```
cmake libadios2_c.a libadios2_core.a libadios2_cxx11.a libadios2_dill.a libadios2_evpath.a libadios2_atl.a libadios2_c_mpi.a libadios2_core_mpi.a libadios2_cxx11_mpi.a libadios2_enet.a libadios2_ffs.a
```
I think only `adios2_cxx11 adios2_c adios2_core` \(and `adios2_fortran` if enabled\) should be appended by `_mpi`.
Note that it seems to be correct in the case ADIOS2 is built rather than using an external library \(see [here](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b…).
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2901/externallibraries…
#2901: ExternalLibraries-ADIOS2: detect.sh adds nonexistent MPI libraries if ADIOS2_ENABLE_SST == yes when using external ADIOS2
Reporter: Miren Radia
Status: submitted
Milestone:
Version: development version
Type: bug
Priority: minor
Component: EinsteinToolkit thorn
I opened [an issue in the GitHub repository](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/iss… but am opening this ticket here as suggested by Roland Haas. The GitHub issue description is below
In the case \`ADIOS2\_ENABLE\_SST == yes\`, after [this line](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b…
\`\`\`bash
ADIOS2\_REQ\_LIBRARIES="adios2\_cxx11 adios2\_c adios2\_core adios2\_evpath adios2\_atl adios2\_enet adios2\_ffs adios2\_dill"
\`\`\`
\(maybe with a fortran one too depending on that config option\). Then [later on](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b71…, all of these libraries are appended by \`\_mpi\` so the full library list looks something like
\`\`\`
ADIOS2\_LIBS="adios2\_cxx11\_mpi adios2\_c\_mpi adios2\_core\_mpi adios2\_evpath\_mpi adios2\_atl\_mpi adios2\_enet\_mpi adios2\_ffs\_mpi adios2\_dill\_mpi adios2\_cxx11 adios2\_c adios2\_core adios2\_evpath adios2\_atl adios2\_enet adios2\_ffs adios2\_dill"
\`\`\`
Unfortunately, looking at my Spack build of ADIOS2, I only have the following in the \`lib64\` directory \(actually it's the same as the one built by Cactus using the \`build.sh\` script\):
\`\`\`
cmake libadios2\_c.a libadios2\_core.a libadios2\_cxx11.a libadios2\_dill.a libadios2\_evpath.a libadios2\_atl.a libadios2\_c\_mpi.a libadios2\_core\_mpi.a libadios2\_cxx11\_mpi.a libadios2\_enet.a libadios2\_ffs.a
\`\`\`
I think only \`adios2\_cxx11 adios2\_c adios2\_core\` \(and \`adios2\_fortran\` if enabled\) should be appended by \`\_mpi\`.
Note that it seems to be correct in the case ADIOS2 is built rather than using an external library \(see [here](https://github.com/EinsteinToolkit/ExternalLibraries-ADIOS2/blob/c45b…).
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2901/externallibraries…
#2900: Support for Lonestar
Reporter: Hector Iglesias
Status: submitted
Milestone:
Version:
Type: bug
Priority: blocker
Component:
Hello,
I was wondering if there was support for the Lonestar cluster. I don’t see any files for it in `simfactory/mdb` in the latest ET release, but [#256](https://bitbucket.org/einsteintoolkit/tickets/issues/256/add-support-for-the-new-lonestar) seems to indicate that there was a time where there was support.
Thanks.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2900/support-for-lones…
#2899: BHNS with Elliptica: Assertion `all(offset == other.offset)' failed
Reporter: Alejandra Gonzalez
Status: new
Milestone:
Version:
Type: bug
Priority: major
Component:
Hello,
I’m testing the Elliptica\_ID\_Reader thorn to evolve black hole neutron star binaries. Right after iteration 0 I get the following error:
```
cactus_sim_ellip: /gpfs/home/uib/uib416720/ETK2024/Elliptica/Cactus/arrangements/Carpet/CarpetLib/src/bboxset2.hh:263: CarpetLib::bboxset2::bboxset<T, D> CarpetLib::bboxset2::bboxset<T, D>::binary_operator(const F&, const CarpetLib::bboxset2::bboxset<T, D>&) const [with F = CarpetLib::bboxset2::bboxset<int, 3>::operator|(const CarpetLib::bboxset2::bboxset<int, 3>&) const::<lambda(const bboxset1&, const bboxset1&)>; T = int; int D = 3]: Assertion `all(offset == other.offset)' failed.
```
I understand there’s something wrong with the box covering either of the two objects so I would appreciate suggestions as to which parameters to modify and how to set this properly.
I’m attaching the parfile I’m using, maybe I’m missing something there.
attachment: bhns_gamma2.0_k92.1_n128.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2899/bhns-with-ellipti…
#2898: CarpetX: Interpolator caches
Reporter: Lucas Timotheo Sanches
Status: new
Milestone:
Version:
Type: enhancement
Priority: major
Component:
Comment (by Roland Haas):
Having a hidden cache is dangerous since it will accumulate cache entries that may no longer be required \(eg a run that never regrids but updates coordinates\). Something like this happened to the HTTP thorn
Better maybe to let the user code control cache entries.
Eg. Slab uses \(in `Slab/src/slab.h` \(though apparently there’s no LaTeXed docs for this\) `Slab_MultiTransfer_Init` and `Slab_MultiTransfer_Finalize` to manage lifetime of its control structures. CarpetInterp2 \(being C\+\+ only\) uses the `fasterp_setup_t`class \(`CarpetInterp2/src/fasterp.hh`\).
This also avoids the need to have to hash the input coordinates etc. or otherwise find out which cached entry to use.
This can be added to the regular Cactus interpolator \(could be in Carpet as well, though apparently I never created code for this\) by having a handle based interface and storing the \(int\) handle is a parameter in the interpolator parameter table \(in fact the interpolator is free to modify the table, so it could actually allocate the cache entry is an “emtpy” field for the cache handle is found in the table, which would require only call to deallocate the handle object\).
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2898/carpetx-interpola…
#2886: support cell centered directions when calling Fortran scheduled functions in CarpetX
Reporter: Roland Haas
Status: open
Milestone:
Version:
Type: enhancement
Priority: major
Component: Cactus
Changes (by Roland Haas):
title: support cell centered directions when calling Fortran scheduled functions in CarpetX (was support cell centered directions when calling Fortran scheduled functions)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2886/support-cell-cent…
#2172: Test "Binary black hole GW150914" example
Reporter: Roland Haas
Status: resolved
Milestone: ET_2025_05
Version: development version
Type: task
Priority: major
Component: EinsteinToolkit website
Changes (by Roland Haas):
status: resolved (was open)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2172/test-binary-black…
#2172: Test "Binary black hole GW150914" example
Reporter: Roland Haas
Status: open
Milestone: ET_2025_05
Version: development version
Type: task
Priority: major
Component: EinsteinToolkit website
Comment (by Roland Haas):
All applied. Thank you!
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2172/test-binary-black…
#2897: Reading nuceos table from initial data reader to EOS_Omni
Reporter: Alejandra Gonzalez
Status: open
Milestone:
Version: ET_2025_05
Type: bug
Priority: major
Component: EinsteinToolkit thorn
Comment (by Alejandra Gonzalez):
Hi Roland, unfortunately the resulting backtrace when I run the parfile I share turns out empty, that’s why I ran gdb to pinpoint the issue.
I will try out the `GRHydro` and `HydroBase` settings to evolve temperature and neutrinos since I’m suspecting that’s where the problem comes from.
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2897/reading-nuceos-ta…