Dear all,
in order to decrease the necessary calculations for a simulation, I tried to lower the resolution and limit the domain size of the simulation. However, I get the error that the communicated region has to be contained within the active part of the domain; I don't see where the issue is in my parameter file, though. I paid attention that the region size is an integer multiple of the resolution and to not extract e.g. gravitational waves from surfaces at a larger radius than the region size. I have attached both the error file, the output file and the parameter file. (Note that I also use some non-standard thorns, e.g. for seeding magnetic fields.)
Sincerely, Dominik Rhiem
Dominik
Carpet calculates these region sizes automatically based on the parameters. This calculation can fail (and will then sometimes lead to strange errors such as the one you reported) if either: - the coarse grid has too few points (since Carpet cannot increase the coarse grid size) - there are too many processes used (since Carpet is not always clever enough to leave some of the processes unused)
It is sometimes surprising how many grid points are needed for a self-consistent grid setup. For example, with 3 ghost zones and an RK4 operator, the coarse grid needs to have more than 30 grid points or so at least to contain all ghost and buffer zones around the fine grid. If there is only a single grid, arbitrarily small domains should be possible (if you set up the boundaries correctly).
The error messages should probably be improved.
As a remedy, I recommend using fewer MPI processes and/or increasing the resolution of the coarse grid.
-erik
On Thu, Nov 28, 2019 at 3:41 AM Dominik Guido Rhiem s6dorhie@uni-bonn.de wrote:
Dear all,
in order to decrease the necessary calculations for a simulation, I tried to lower the resolution and limit the domain size of the simulation. However, I get the error that the communicated region has to be contained within the active part of the domain; I don't see where the issue is in my parameter file, though. I paid attention that the region size is an integer multiple of the resolution and to not extract e.g. gravitational waves from surfaces at a larger radius than the region size. I have attached both the error file, the output file and the parameter file. (Note that I also use some non-standard thorns, e.g. for seeding magnetic fields.)
Sincerely, Dominik Rhiem
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Dear all,
I'm calculating radial oscillations on a Neutron Star within a multipatch (Llama) environment. The coordinate system is Thornburg04. For my time evolution of the fluid element displacement equations, I'm scheduling routines for my boundary conditions in MoL_PostStep. The conditions are with respect to the (changing) surface of the star TOV_surface. I want to loop over all grid points to apply the conditions on certain grid points (close to the surface) in Fortran.
!$OMP PARALLEL DO private(i,j,k) do k = 1, cctk_lsh(3) do j = 1, cctk_lsh(2) do i = 1, cctk_lsh(1) if (grid_r(i,j,k) >= (TOV_surface-rprec) .AND. grid_r(i,j,k) <= (TOV_surface+ rprec)) then drXi(i,j,k) = 0 Pidot(i,j,k) = Wrinv(i,j,k) * P(i,j,k) * drrXi(i,j,k) + Qr(i,j,k) * Wrinv(i,j,k) * Xi(i,j,k) else if (grid_r(i,j,k) > (TOV_surface + rprec)) then drXi(i,j,k) = 0 drrXi(i,j,k) = 0 Xi(i,j,k) = 0 Pi(i,j,k) = 0 Xidot(i,j,k) = 0 Pidot(i,j,k) = 0 Xeta(i,j,k) = 0 end if end do end do end do !$OMP END PARALLEL DO
I attached the specific files of my thorn, the parameter file, as well as some screenshots of the evolved data.
As you can see, the boundary conditions do not affect any part along the x axis. Does any one have an idea why my loop seems to "miss" some if-cases?
Thanks a lot and best regards,
Severin
Severin
A general comment: Your variable pio has only 7 digits of accuracy. You typed more digits, but since this is a single precision constant, it is rounded to 7 digits. I recommend using "acos(-1.0d0)" instead.
Your loops look correct. If you want to test your code, then you could initialize a grid function to zero before your loop, and set it to one or two inside the loop. This will test whether your loop executes for all points, and which if branch is taken.
There might be another function called after your loop runs. For example, interpolation from a neighbouring patch might overwrite some data. You could output a few grid point values after your loop and compare with what you see in your images to check this.
-erik
On Mon, Dec 2, 2019 at 10:43 AM Severin Frank severin.frank@uni-tuebingen.de wrote:
Dear all,
I'm calculating radial oscillations on a Neutron Star within a multipatch (Llama) environment. The coordinate system is Thornburg04. For my time evolution of the fluid element displacement equations, I'm scheduling routines for my boundary conditions in MoL_PostStep. The conditions are with respect to the (changing) surface of the star TOV_surface. I want to loop over all grid points to apply the conditions on certain grid points (close to the surface) in Fortran.
!$OMP PARALLEL DO private(i,j,k) do k = 1, cctk_lsh(3) do j = 1, cctk_lsh(2) do i = 1, cctk_lsh(1) if (grid_r(i,j,k) >= (TOV_surface-rprec) .AND. grid_r(i,j,k) <= (TOV_surface+ rprec)) then drXi(i,j,k) = 0 Pidot(i,j,k) = Wrinv(i,j,k) * P(i,j,k) * drrXi(i,j,k) + Qr(i,j,k) * Wrinv(i,j,k) * Xi(i,j,k) else if (grid_r(i,j,k) > (TOV_surface + rprec)) then drXi(i,j,k) = 0 drrXi(i,j,k) = 0 Xi(i,j,k) = 0 Pi(i,j,k) = 0 Xidot(i,j,k) = 0 Pidot(i,j,k) = 0 Xeta(i,j,k) = 0 end if end do end do end do !$OMP END PARALLEL DO
I attached the specific files of my thorn, the parameter file, as well as some screenshots of the evolved data.
As you can see, the boundary conditions do not affect any part along the x axis. Does any one have an idea why my loop seems to "miss" some if-cases?
Thanks a lot and best regards,
Severin
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Thank you Erik for your last reply and your helpful advice!
In the last days I changed a lot of my code and I noticed another (different?) issue:
I'm calculating spatial derivatives in r-direction for different grid functions by using Llama GlobalDerivative thorn, such as this example:
call globalDiff_gv (cctkGH, 0, PHI, param_dx, J11, J21, J31, & J12, J22, J32, J13, J23, J33, -1_ik) call globalDiff_gv (cctkGH, 1, PHI, param_dy, J11, J21, J31, & J12, J22, J32, J13, J23, J33, -1_ik) call globalDiff_gv (cctkGH, 2, PHI, param_dz, J11, J21, J31, & J12, J22, J32, J13, J23, J33, -1_ik)
param_dr = dxdr * param_dx + dydr * param_dy + dzdr * param_dz
, where dxdr, dydr, dzdr are the specific coordinate transformations and my coordinates are again Thornburg04 multipatch.
I noticed that I receive very odd (and wrong) zero-values on specific grid points in the positive x-sphere. Not only for param_dr, but also for param_dx, param_dy as well as param_dz. I should mention, that I did not change the GlobalDerivative thorn and this problem also appears, when calculating the derivatives of other variables then PHI. Also PHI itself is continuous in this area. I also initiate all "param"-variables with 1 so that the call of GlobalDerivative is actively changing them (to 0).
The same error also appears, when calling GlobalDiff directly with the inverse Jacobians like:
call globalDiff_gv (cctkGH, 2, PHI, param_dr, J11, J21, J31, J12, J22, J32, iJ13, iJ23, iJ33, -1_ik).
Furthermore it makes no difference whether this is scheduled at initial or in MoL_CalcRHS.
I attached plots from PHI, param_dx and param_dr so that you can see where those points are on the x-z plane. These are for an angular resolution of Coordinates::n_angular = 9. If I increase the angular resolution this affects also more grid points along the same radius (for example 8 points in the x-z plane for n_angular = 15).
Since I'm assuming, that the GobalDerivative thorn is working correct I don't know where this issue might come from. Does anyone have an idea for this?
Thank you all a lot!
Best regards, Severin
users@lists.einsteintoolkit.org
-
Dominik Guido Rhiem -
Erik Schnetter -
Severin Frank