#2670: Refinement prolongation: All points must have been received - Trac

27 Nov 2022


      #2670: Refinement prolongation: All points must have been received
Reporter: 
   Status: new
Milestone: ET_2022_05
  Version: ET_2022_05
     Type: bug
 Priority: major
Component: Carpet
```
-----------------------------------------------------------------------------------------------------------------------
Iteration      Time | *me_per_hour |             GRHYDRO::dens | *ROBASE::rho | *::w_lorentz | *STRAINTS::H | *axrss_mb
                    |              |          sum      maximum |      maximum |      maximum |      maximum |   maximum
-----------------------------------------------------------------------------------------------------------------------
     9840   184.500 |   98.5293558 |    0.0004144    0.0038553 |    0.0011696 |    1.2604477 |    0.0020844 |      2839
INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0)
INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0)
INFO (NSTracker): Found star at (-0.375,12.5625,0)
     9844   184.575 |   98.5440274 |    0.0004144    0.0038556 |    0.0011696 |    1.2596245 |    0.0020791 |      2839
INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.375,12.5625,0)
INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.375,-12.5625,0)
INFO (NSTracker): Found star at (-0.5625,12.5625,0)
     9848   184.650 |   98.5682477 |    0.0004144    0.0038559 |    0.0011696 |    1.2568424 |    0.0020807 |      2839
INFO (CarpetTracker): Setting position of refined region #1 from surface #0 to (-0.5625,12.5625,0)
INFO (CarpetTracker): Setting position of refined region #2 from surface #1 to (0.5625,-12.5625,0)
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 0
INFO (CarpetRegrid2): Enforcing grid structure properties, iteration 1
==> projectdns_maxwell_65_300_1_11251505_.err <==
box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([box.active=bboxset<CCTK_INT4,3>928,8416,768]:[1056,8668,1048]:[4,4,4]/[232,2104,192]:[264,(set<bbox>:{([768,8416,768]:[924,8668,1048]:[4,4,4]/[192,2104,192]:[2312167,262]/[33,64,71]/149952)},stride:[4,4,4],offset:[0,0,0])
needrecv=bboxset<CCTK_INT4,3>,2167,262]/[40,64,71]/181760)},stride:[4,4,4],offset:[0,0,0])
needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([928,8416,768]:[1056,8516,1048]:[4,4,4]/[232,2104,192]:[264,2129(set<bbox>:{([768,8416,768]:[924,8516,1048]:[4,4,4]/[192,2104,192]:[231,2129,262]/[40,26,262]/[33,26,71]/60918)},stride:[4,4,4],offset:[0,0,0])
WARNING level 1 from host node2 process 1
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->,71]/73840)},stride:[4,4,4],offset:[0,0,0])
WARNING level 1 from host node2 process 0
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829:
   [ml=0 rl=6 c=1] The following grid structure consistency check failed:
   Refinement prolongation: All points must have been received
   needrecv.empty()
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829:
   [ml=0 rl=6 c=0] The following grid structure consistency check failed:
   Refinement prolongation: All points must have been received
   needrecv.empty()
==> projectdns_maxwell_65_300_1_11251505_.out <==
WARNING level 1 from host node2 process 0
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829:
   [ml=0 rl=6 c=0] The following grid structure consistency check failed:
   Refinement prolongation: All points must have been received
   needrecv.empty()
==> projectdns_maxwell_65_300_1_11251505_.err <==
box.active=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9400,768]:[1056,9504,1048]:[4,4,4]/[192,2350,192]:[264,2376,262]/[73,27,71]/139941)},stride:[4,4,4],offset:[0,0,0])
needrecv=bboxset<CCTK_INT4,3>(set<bbox>:{([768,9404,768]:[1056,9504,1048]:[4,4,4]/[192,2351,192]:[264,2376,262]/[73,26,71]/134758)},stride:[4,4,4],offset:[0,0,0])
WARNING level 1 from host node2 process 7
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:829:
   [ml=0 rl=6 c=7] The following grid structure consistency check failed:
   Refinement prolongation: All points must have been received
   needrecv.empty()
WARNING level 1 from host node2 process 0
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971:
   [ml=0 rl=6 c=0] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
==> projectdns_maxwell_65_300_1_11251505_.out <==
WARNING level 1 from host node2 process 0
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971:
   [ml=0 rl=6 c=0] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
==> projectdns_maxwell_65_300_1_11251505_.err <==
WARNING level 1 from host node2 process 1
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971:
   [ml=0 rl=6 c=1] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed.
Rank 0 with PID 285012 received signal 6
Writing backtrace to dns/backtrace.0.txt
cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed.
Rank 1 with PID 285013 received signal 6
Writing backtrace to dns/backtrace.1.txt
WARNING level 1 from host node2 process 6
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971:
   [ml=0 rl=6 c=6] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
WARNING level 1 from host node2 process 7
  in thorn CarpetLib, file /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:161:
  ->
/public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:971:
   [ml=0 rl=6 c=7] The following grid structure consistency check failed:
   Synchronisation and boundary prolongation: All points must have been received
   needrecv.empty()
cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed.
Rank 6 with PID 285018 received signal 6
Writing backtrace to dns/backtrace.6.txt
cactus_sim: /public/home/guolj/nr/etk/Cactus/configs/sim/build/CarpetLib/dh.cc:1036: void CarpetLib::dh::regrid(bool): Assertion `(allrestricted & obox.buffers).empty()' failed.
Rank 7 with PID 285019 received signal 6
Writing backtrace to dns/backtrace.7.txt
==> projectdns_maxwell_65_300_1_11251505_.out <==
```
can some any point some direction on how to solve this problem? 
thanks so much.
attachment: dns.par (https://api.bitbucket.org/2.0/repositories/einsteintoolkit/tickets/issues/26...)
--
Ticket URL: https://bitbucket.org/einsteintoolkit/tickets/issues/2670/refinement-prolong...