Scott
Thanks for showing the code.
I can think of several things that could go wrong:
1. It takes some time to start up and shut down parallel threads. Therefore, people usually parallelise the outermost loop, i.e. the k loop in your case. Parallelising the j loop requires starting and stopping threads nz times, which adds overhead.
2. I notice that you don't declare private variables. Variables are shared by default, and any local variables that you use inside the parallel region (and which are not arrays where you only access one element) need to be declared as private. In your case, these are probably x, y, z, gxx, gxy, gxz, etc. Did you compare results between serial and parallel runs? I would expect the results to differ, i.e. the current parallel code seems to have a serious error.
3. How much computational work is done inside this loop? If most of the time is spent in memory access writing to the gij and Kij arrays, then OpenMP won't be able to help. Only if there is sufficient computation going on will you see a benefit.
4. Since you say that you have 24 cores, I assume you have an AMD system. In this case, your machine consists of 4 subsystems that have 6 cores each, and communication between these 4 subsystems will be much slower than within each of these subsystems. People usually recommend to use not more than 6 OpenMP threads, and to ensure that these run within one of these subsystems. You can try setting the environment variable GOMP_CPU_AFFINITY='0-5' to force your threads to run on cores 0 to 5.
-erik
On Tue, May 17, 2011 at 1:00 PM, Scott Hawley scott.hawley@belmont.edu wrote:
No doubt someone will ask for the code itself. The relevant part is given below. ( In old-school Fortran77) Specifically, the 'problem' I'm noticing is that the "% done" messages appear with less frequency and with lesser increment per wall clock time with OMP_NUM_THREADS > 1 than for OMP_NUM_THREADS = 1. The cpus get used alot more --- 'top' shows up to 2000% cpu usage for 24 threads --- but the wallclock time doesn't decrease at all. Also note that whether I use the long OMP directive shown (with the 'shared' declarations and schedule, etc) and the 'END PARALLEL DO' at the end, or if I just use a simple '!$OMP PARALLEL DO' and *nothing else*, the execution time is *identical*. Thanks again! -Scott
chunk = 8 do k = 1, nz write(msg,*)'[1F setbkgrnd: ' // & 'nz = ',nz,', % done = ',int(k*1.0d2/nz),' ' call writemessage(msg)
!$OMP PARALLEL DO SHARED(mask,agxx,agxy,agxz,agyy,agyz,agzz, !$OMP& aKxx,aKxy,aKxz,aKyy,aKyz,aKzz,ax,ay,az), !$OMP& SCHEDULE(STATIC,chunk) PRIVATE(j) do j = 1, ny do i = 1, nx c if (ltrace) then c write(msg,*) '---------------' c call writemessage(msg) c endif
if (mask(i,j,k) .ne. m_ex .and. & (ibonly .eq. 0 .or. & mask(i,j,k) .eq. m_ib)) then x = ax(i) y = ay(j) z = az(k) c the following two include files just perform many pointwise calc's include 'gd.inc' include 'kd.inc' agxx(i,j,k) = gxx agxy(i,j,k) = gxy agxz(i,j,k) = gxz agyy(i,j,k) = gyy agyz(i,j,k) = gyz agzz(i,j,k) = gzz
aKxx(i,j,k) = Kxx aKxy(i,j,k) = Kxy aKxz(i,j,k) = Kxz aKyy(i,j,k) = Kyy aKyz(i,j,k) = Kyz aKzz(i,j,k) = Kzz
else if (mask(i,j,k) .eq. m_ex) then c Excised points agxx(i,j,k) = exval agxy(i,j,k) = exval agxz(i,j,k) = exval agyy(i,j,k) = exval agyz(i,j,k) = exval agzz(i,j,k) = exval
aKxx(i,j,k) = exval aKxy(i,j,k) = exval aKxz(i,j,k) = exval aKyy(i,j,k) = exval aKyz(i,j,k) = exval aKzz(i,j,k) = exval endif enddo enddo !$OMP END PARALLEL DO enddo
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users