November 2022 - Trac - lists.einsteintoolkit.org

#2670: Refinement prolongation: All points must have been received
by Artectek 30 Nov '22

30 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): I try what you said,the threads are now correct,but the problem still breaks down,_exactly_ as before. is there any other suggestion regarding this problem? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): Words can’t not express my gratitude,thank you,I will try that right away. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): The variable that \(usually\) controls how many threads are used is called `OMP_NUM_THREADS` . `CACTUS_NUM_THREADS` is a copy that records the number of threads requested from simfactory. In case something interferes with simfactory \(mpirun sometimes does this, or SLURM, or Cray’s alps system\) then Cactus / Carpet can used `CACTUS_NUM_THREADS` to detect this. More or less what you see \(only it would produce an error instead of just a warning\). If _neither_ `OMP_NUM_THREADS` nor `CACTUS_NUM_THREADS` is set then OpenMP would default to the total number of cores for the number of threads \(ie 24\) and Carpet would output such an error message. Some MPI stacks \(eg OpenMPI, note the “I” at the end\) can be configured to not pass environment variables to the Cactus executable. They may need to be given an explicit list of environment variables to pass to Cactus using the `-x` option. See eg: [https://bitbucket.org/simfactory/simfactory2/src/master/mdb/runscripts/cygw… where you can see the various `-x` options in the mpirun line. I would give adding those a try. ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): indeed it set ``` CACTUS_NUM_THREADS=8 ``` but he info\(Carpet\) shows : ``` INFO (Carpet): There are 24 threads per process ``` shouldnt it be 8 threads per process? worth to mention that ``` Although OpenMP is enabled, the environment variable CACTUS_NUM_THREADS is not set. ``` i submited this job on a login node,but openmpi ran it on a compute node,could this be the problem? -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): Ok, in some sense this is actually good news \(at least it’s not obviously a race condition\). Being paranoid, did you check in RunScript that it does indeed set `OMP_NUM_THREADS=8` \(Carpet also prints out the number of threads used at the top of the out file, in a line `INFO (Carpet): There are 24 threads per process`\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): I’m sorry,but the result is the same. ``` INFO (NSTracker): Found star at (0,-12.75,0) 10248 192.150 | 111.3619840 | 0.0003162 0.0026300 | 0.0010288 | 1.0506992 | 0.0011846 | 2909 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (0,-12.75,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (-0,12.75,0) INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 ==> projectdns_maxwell_65_300_1_maxrho=10_11291630_threads=8_core64.err <== box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,8408,768]:[1048,8664,920]:[4,4,4]/[192,2102,192]:[262,2166,230]/[71box.active=bboxset<CCTK_INT4,3>(set<bbox>:,65,39]/179985)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,8408,768]:[1048{([768,8408,924]:[1048,8664,1048]:[4,4,4]/[192,2102,231]:[262,2166,262]/[71,65,32]/147680)},stride:[4,4,4],8516,920]:[4,4,4]/[192,2102,192]:[262,2129,230]/[71,28,39]/77532)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> ,offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,8408,924]:[1048,8516,1048]:[4,4,4] ==> projectdns_maxwell_65_300_1_maxrho=10_11291630_threads=8_core64.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_maxrho=10_11291630_threads=8_core64.err <== /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() /[192,2102,231]:[262,2129,262]/[71,28,32]/63616)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9408,768]:[1048,9512,1048]:[4,4,4]/[192,2352,192]:[262,2378,262]/[71,27,71]/136107)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9408,768]:[1048,9512,1048]:[4,4,4]/[192,2352,192]:[262,2378,262]/[71,27,71]/136107)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9260,768]:[1048,9404,1048]:[4,4,4]/[192,2315,192]:[262,2351,262]/[71,37,71]/186517)},stride:[4,4,4] ==> projectdns_maxwell_65_300_1_maxrho=10_11291630_threads=8_core64.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_maxrho=10_11291630_threads=8_core64.err <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ,offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9404,768]:[1048,9404,1048]:[4,4,4]/[192,2351,192]:[262,2351,262]/[71,1,71]/5041)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 6 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=6] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 0 with PID 768170 received signal 6 Writing backtrace to dns/backtrace.0.txt WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 1 with PID 768171 received signal 6 Writing backtrace to dns/backtrace.1.txt WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() WARNING level 1 from host node2 process 6 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=6] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 7 with PID 768177 received signal 6 Writing backtrace to dns/backtrace.7.txt cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 6 with PID 768176 received signal 6 Writing backtrace to dns/backtrace.6.txt ``` although it speeded up a lot.but ended the same. ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): yes,we have 8 cpus per node,24cores per cpu,so that make it 192 cores per node. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): Just to be clear, by 8 CPU you mean 8 CPU sockets \(which would make this a very large node indeed\)? Or 8 cores \(which would make this almost small by today’s standards\)? Basically if you run `cat /proc/cpuinfo`, what is the highest value for `processor` that you see? I would guess either 7 or 191. You can also try to see if `lscpu` exists which will give output in a bit of a nicer form than the raw cpuinfo output. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): yes,its only one node with 8 cpus,i will try your solution right away,thank you very much! -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 29 Nov '22

29 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): Hmm, one thing I would suggest changing would be to use fewer threads per MPI rank. Right now you have set: ``` export OMP_NUM_THREADS=24 ``` which gives you 24 OpenMP thtreads per MPI rank. The SubmitScipt you use is \(essentially, up to comments\) `generic.sub` as far as I can tell, so really only designed for a non-cluster environment \(it may work on a cluster, but that would be kind of accidental\). Right now it seems that you are using 8 MPI ranks each with 24 OpenMP threads. So this is a total of 192 cores. So this should be somewhere between 4 and 8 nodes, yes? My suggestion would be to try and use only about 8 OpenMP threads and correspondingly more MPI ranks, so use `--cores 192 --num-threads 8` instead of `--cores 192 --num-threads 24` which is what you seem to have used. In principle, more threads should of course not make things fail \(this would indeed be a bug\), though it may be quite hard to reproduce since it would, most likely, be a race condition that only shows up with large thread counts. Also not that multi-threading in Cactus/Carpet tends to not be extremely efficient \(since it was added to an existing MPI parallel code instead of being integrated from the beginning\), which is why I suggest to use fewer threads. Usually you want to use as many MPI ranks and as few threads as you can get away with before you are limited by added communication overhead \(which scales as the number of MPI ranks and is constant with the number of OpenMP threads\). -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): here are the files you requested. if you can shed some light on this matter,I would be very grateful. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): here are the files you requested. if you can shed some light on this matter,I would be very grateful. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): <div class="preview-container wiki-content"></div> <div class="mask"></div> </div> </div> </form> -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): <div class="preview-container wiki-content"></div> <div class="mask"></div> </div> </div> </form> -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: resolved Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: new Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Applied as git hash [33a5f20](https://bitbucket.org/einsteintoolkit/einsteinevolve/commits/33a5f… "GRHydro: work around Intel 19 compiler issue" of [einsteinevolve](https://bitbucket.org/einsteintoolkit/einsteinevolve) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: new Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): @{557058:56049c54-f8c2-4b6c-9b88-ab697c967495} says \(private conversation\): please apply. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2671: add instuctions for what information to provide when asking for help with a failed run
by Roland Haas 28 Nov '22

28 Nov '22

#2671: add instuctions for what information to provide when asking for help with a failed run Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit website Currently the how to ask for help section [https://www.einsteintoolkit.org/support.html#general-guidelines-for-questio… only lists information for compilation issues. It should be extended to included information on runtime errors. Needed are \(at least\): 1. full parfile 2. full `out` and `err` files for all segments 3. SubmitScript, RunScript as found in `output-NNNN/SIMFACTORY` ie. the ones with everything substituted 4. name of cluster if public, option list otherwise 5. ET release being used 6. if a Carpet regidding error then all `carpet-grid.asc` files for all segments ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2671/add-instuctions-f…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): These errors are usually caused by issues with the grid setup that develop over time in the simulation. The error is unfortunately only detectable deep inside of Carpet where the original cause is no longer obvious. With just the error message it is not really possible to help diagnose it. To help with this, more information is needed: 1. the `.out` and `.err` files produced by the simulation 2. the RunScript as found in output-NNNN/SIMFACTORY 3. the SubmitScript as found in output-NNNN/SIMFACTORY 4. if a public cluster, the name of the cluster used, otherwise the option list used With those it may be possible to provide meaningful advise. Otherwise shot in the dark suggestion would be: 1. do not use too many MPI ranks \(no more than 128 for a mid res BNS run\), this avoids the domain parts assigned to each MPI rank becomming too small 2. check if your box locations have moved out of the grid \(ie something went wrong with tracking\) 3. check for NaNs in evolution variables ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Anonymous 27 Nov '22

27 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet ``` ----------------------------------------------------------------------------------------------------------------------- Iteration Time | *me_per_hour | GRHYDRO::dens | *ROBASE::rho | *::w_lorentz | *STRAINTS::H | *axrss_mb | | sum maximum | maximum | maximum | maximum | maximum ----------------------------------------------------------------------------------------------------------------------- 9840 184.500 | 98.5293558 | 0.0004144 0.0038553 | 0.0011696 | 1.2604477 | 0.0020844 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0) INFO (NSTracker): Found star at (-0.375,12.5625,0) 9844 184.575 | 98.5440274 | 0.0004144 0.0038556 | 0.0011696 | 1.2596245 | 0.0020791 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0) INFO (NSTracker): Found star at (-0.5625,12.5625,0) 9848 184.650 | 98.5682477 | 0.0004144 0.0038559 | 0.0011696 | 1.2568424 | 0.0020807 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.5625,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.5625,-12.5625,0) INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 ==> projectdns_maxwell_65_300_1_11251505_.err <== box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([box.active=bboxset<CCTK_INT4,3>928,8416,768]:[1056,8668,1048]:[4,4,4]/[232,2104,192]:[264,(set<bbox>:{([768,8416,768]:[924,8668,1048]:[4,4,4]/[192,2104,192]:[2312167,262]/[33,64,71]/149952)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>,2167,262]/[40,64,71]/181760)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([928,8416,768]:[1056,8516,1048]:[4,4,4]/[232,2104,192]:[264,2129(set<bbox>:{([768,8416,768]:[924,8516,1048]:[4,4,4]/[192,2104,192]:[231,2129,262]/[40,26,262]/[33,26,71]/60918)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: ->,71]/73840)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.err <== box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9400,768]:[1056,9504,1048]:[4,4,4]/[192,2350,192]:[264,2376,262]/[73,27,71]/139941)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9404,768]:[1056,9504,1048]:[4,4,4]/[192,2351,192]:[264,2376,262]/[73,26,71]/134758)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.err <== WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 0 with PID 285012 received signal 6 Writing backtrace to dns/backtrace.0.txt cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 1 with PID 285013 received signal 6 Writing backtrace to dns/backtrace.1.txt WARNING level 1 from host node2 process 6 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=6] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 6 with PID 285018 received signal 6 Writing backtrace to dns/backtrace.6.txt cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 7 with PID 285019 received signal 6 Writing backtrace to dns/backtrace.7.txt ==> projectdns_maxwell_65_300_1_11251505_.out <== ``` can some any point some direction on how to solve this problem? thanks so much. attachment: dns.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2666: Testsuite system does not recognize "Cactus" thorn
by Roland Haas 25 Nov '22

25 Nov '22

#2666: Testsuite system does not recognize "Cactus" thorn Reporter: Roland Haas Status: new Milestone: Version: development version Type: bug Priority: major Component: Cactus Changes (by Roland Haas): responsible: [] (was []) component: Cactus (was EinsteinToolkit thorn) assignee: Roland Haas (was Zach Etienne) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2666/testsuite-system-…

1 0

#2666: Testsuite system does not recognize "Cactus" thorn
by Roland Haas 25 Nov '22

25 Nov '22

#2666: Testsuite system does not recognize "Cactus" thorn Reporter: Roland Haas Status: new Milestone: Version: development version Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): to be clear: this is a bug in the testsuite system which will be fixed after the release. However right now this means that the NRPyEllitpticET test did not actually run. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2666/testsuite-system-…

1 0

#2669: math is not rendered correctly in online docs
by Roland Haas 24 Nov '22

24 Nov '22

#2669: math is not rendered correctly in online docs Reporter: Roland Haas Status: new Milestone: Version: development version Type: bug Priority: major Component: EinsteinToolkit website Changes (by Roland Haas): The mathjax \(so one hopes\) rendered math in thorndocs eg in [http://einsteintoolkit.org/thornguide/CactusNumerical/LocalReduce/documenta… is not being shown correctly. I \(firefox 102.5.0esr, Debian Linux/GNU\) am shown: ‌ ![](https://bitbucket.org/repo/8zBLME5/images/3580170200-incorrect.png) ‌ instead of the correct \(when I run `make LocalReduce-ThornDocHTML` on my laptop\): ‌ ![](https://bitbucket.org/repo/8zBLME5/images/2946320859-correct.png) ‌ This most likely is due to different tex4html engines in Ubuntu 20.04 \(the github documentation runner\) and Debian bullseye. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2669/math-is-not-rende…

1 0

#2669: math is not rendered correctly in online docs
by Roland Haas 24 Nov '22

24 Nov '22

#2669: math is not rendered correctly in online docs Reporter: Roland Haas Status: new Milestone: Version: development version Type: bug Priority: major Component: EinsteinToolkit website The mathjax \(so one hopes\) rendered math in thorndocs eg in [http://einsteintoolkit.org/thornguide/CactusNumerical/LocalReduce/documenta… is not being shown correctly. I \(firefox 102.5.0esr, Debian Linux/GNU\) am shown: ![](https://bitbucket.org/repo/8zBLME5/images/3580170200-incorrect.png) instead of the correct \(when I run `make LocalReduce-ThornDocHTML` on my laptop\): ![](https://bitbucket.org/repo/8zBLME5/images/2946320859-correct.png) This most likely is due to different tex4html engines in Ubuntu 20.04 \(the github documentation runner\) and Debian bullseye. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2669/math-is-not-rende…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 24 Nov '22

24 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: new Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Please review. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac November 2022