Trac November 2022

trac@lists.einsteintoolkit.org

1 participants
71 discussions

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): here are the files you requested. if you can shed some light on this matter,I would be very grateful. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): here are the files you requested. if you can shed some light on this matter,I would be very grateful. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): <div class="preview-container wiki-content"></div> <div class="mask"></div> </div> </div> </form> -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Artectek 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Artectek): <div class="preview-container wiki-content"></div> <div class="mask"></div> </div> </div> </form> -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: resolved Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Changes (by Roland Haas): status: resolved (was new) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: new Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): Applied as git hash [33a5f20](https://bitbucket.org/einsteintoolkit/einsteinevolve/commits/33a5f… "GRHydro: work around Intel 19 compiler issue" of [einsteinevolve](https://bitbucket.org/einsteintoolkit/einsteinevolve) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2668: GRHydro: work around Intel 19 compiler issue
by Roland Haas 28 Nov '22

28 Nov '22

#2668: GRHydro: work around Intel 19 compiler issue Reporter: Roland Haas Status: new Milestone: ET_2022_11 Version: ET_2021_05 Type: bug Priority: major Component: EinsteinToolkit thorn Comment (by Roland Haas): @{557058:56049c54-f8c2-4b6c-9b88-ab697c967495} says \(private conversation\): please apply. -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2668/grhydro-work-arou…

1 0

#2671: add instuctions for what information to provide when asking for help with a failed run
by Roland Haas 28 Nov '22

28 Nov '22

#2671: add instuctions for what information to provide when asking for help with a failed run Reporter: Roland Haas Status: new Milestone: Version: Type: enhancement Priority: minor Component: EinsteinToolkit website Currently the how to ask for help section [https://www.einsteintoolkit.org/support.html#general-guidelines-for-questio… only lists information for compilation issues. It should be extended to included information on runtime errors. Needed are \(at least\): 1. full parfile 2. full `out` and `err` files for all segments 3. SubmitScript, RunScript as found in `output-NNNN/SIMFACTORY` ie. the ones with everything substituted 4. name of cluster if public, option list otherwise 5. ET release being used 6. if a Carpet regidding error then all `carpet-grid.asc` files for all segments ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2671/add-instuctions-f…

1 0

#2670: Refinement prolongation: All points must have been received
by Roland Haas 28 Nov '22

28 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet Comment (by Roland Haas): These errors are usually caused by issues with the grid setup that develop over time in the simulation. The error is unfortunately only detectable deep inside of Carpet where the original cause is no longer obvious. With just the error message it is not really possible to help diagnose it. To help with this, more information is needed: 1. the `.out` and `.err` files produced by the simulation 2. the RunScript as found in output-NNNN/SIMFACTORY 3. the SubmitScript as found in output-NNNN/SIMFACTORY 4. if a public cluster, the name of the cluster used, otherwise the option list used With those it may be possible to provide meaningful advise. Otherwise shot in the dark suggestion would be: 1. do not use too many MPI ranks \(no more than 128 for a mid res BNS run\), this avoids the domain parts assigned to each MPI rank becomming too small 2. check if your box locations have moved out of the grid \(ie something went wrong with tracking\) 3. check for NaNs in evolution variables ‌ -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

#2670: Refinement prolongation: All points must have been received
by Anonymous 27 Nov '22

27 Nov '22

#2670: Refinement prolongation: All points must have been received Reporter: Status: new Milestone: ET_2022_05 Version: ET_2022_05 Type: bug Priority: major Component: Carpet ``` ----------------------------------------------------------------------------------------------------------------------- Iteration Time | *me_per_hour | GRHYDRO::dens | *ROBASE::rho | *::w_lorentz | *STRAINTS::H | *axrss_mb | | sum maximum | maximum | maximum | maximum | maximum ----------------------------------------------------------------------------------------------------------------------- 9840 184.500 | 98.5293558 | 0.0004144 0.0038553 | 0.0011696 | 1.2604477 | 0.0020844 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0) INFO (NSTracker): Found star at (-0.375,12.5625,0) 9844 184.575 | 98.5440274 | 0.0004144 0.0038556 | 0.0011696 | 1.2596245 | 0.0020791 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0) INFO (NSTracker): Found star at (-0.5625,12.5625,0) 9848 184.650 | 98.5682477 | 0.0004144 0.0038559 | 0.0011696 | 1.2568424 | 0.0020807 | 2839 INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.5625,12.5625,0) INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.5625,-12.5625,0) INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0 INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1 ==> projectdns_maxwell_65_300_1_11251505_.err <== box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([box.active=bboxset<CCTK_INT4,3>928,8416,768]:[1056,8668,1048]:[4,4,4]/[232,2104,192]:[264,(set<bbox>:{([768,8416,768]:[924,8668,1048]:[4,4,4]/[192,2104,192]:[2312167,262]/[33,64,71]/149952)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>,2167,262]/[40,64,71]/181760)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([928,8416,768]:[1056,8516,1048]:[4,4,4]/[232,2104,192]:[264,2129(set<bbox>:{([768,8416,768]:[924,8516,1048]:[4,4,4]/[192,2104,192]:[231,2129,262]/[40,26,262]/[33,26,71]/60918)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: ->,71]/73840)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.err <== box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9400,768]:[1056,9504,1048]:[4,4,4]/[192,2350,192]:[264,2376,262]/[73,27,71]/139941)},stride:[4,4,4],offset:[0,0,0]) needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9404,768]:[1056,9504,1048]:[4,4,4]/[192,2351,192]:[264,2376,262]/[73,26,71]/134758)},stride:[4,4,4],offset:[0,0,0]) WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Refinement prolongation: All points must have been received needrecv.empty() WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.out <== WARNING level 1 from host node2 process 0 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=0] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() ==> projectdns_maxwell_65_300_1_11251505_.err <== WARNING level 1 from host node2 process 1 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=1] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 0 with PID 285012 received signal 6 Writing backtrace to dns/backtrace.0.txt cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 1 with PID 285013 received signal 6 Writing backtrace to dns/backtrace.1.txt WARNING level 1 from host node2 process 6 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=6] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() WARNING level 1 from host node2 process 7 in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161: -> /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971: [ml=0 rl=6 c=7] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty() cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 6 with PID 285018 received signal 6 Writing backtrace to dns/backtrace.6.txt cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed. Rank 7 with PID 285019 received signal 6 Writing backtrace to dns/backtrace.7.txt ==> projectdns_maxwell_65_300_1_11251505_.out <== ``` can some any point some direction on how to solve this problem? thanks so much. attachment: dns.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/2…) -- Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolon…

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Trac November 2022