Erik, Thanks for your ideas. #3 may be the most significant.
1. I switched the OMP directives to the outer loop, with the main result being, of course, that the "% done" line skips around, but NO change in execution speed.
2. I also increased the number of private variables as shown below. Again no change in speed. And by this I mean: 1 thread - the routine takes 11.3 seconds 2 threads - the routine takes 47.7 seconds 4 threads - the routine takes 40.1 seconds
These results are using code at the beginning of the loops which now reads... !$OMP PARALLEL DO SHARED(mask,agxx,agxy,agxz,agyy,agyz,agzz, !$OMP& aKxx,aKxy,aKxz,aKyy,aKyz,aKzz,ax,ay,az), !$OMP& SCHEDULE(STATIC,chunk) PRIVATE(k,j,i,gxx,gxy,gxz,gyy,gyz,gzz, !$OMP& x, y, z) do k = 1, nz write(msg,*)'[1F setbkgrnd: ' // & 'nz = ',nz,', % done = ',int(k*1.0d2/nz),' ' call writemessage(msg)
do j = 1, ny do i = 1, nx ...
3. There is ALOT of computational work 'per point' done, at each value of i,j,k; there are long formulas in the include files include 'gd.inc' include 'kd.inc' Wait...these include files contain *tons* of temporary variables that maybe should be private -- they were generated by maple, variables like 't1' thru 't250'. Do these all be need to be declared as private? i certainly don't want the various processors overwriting each others' work, which might be what they're doing. -- maybe they're even generating NaNs which would slow things down a bit!
4. Yea, even 2 threads vs one thread is a significant slowdown, as noted above.
5. Earlier I misspoke: "It works fine on my mac" means OpenMP works fine on my Mac, for a *different* program I wrote. This program *also* works fine on my linux box. So it's *very likely* that my current issue is "user error", and not a misconfigured OpenMP. lol.
Thanks, Scott
-- Scott H. Hawley, Ph.D. Asst. Prof. of Physics Chemistry & Physics Dept Office: Hitch 100D Belmont University Tel: +1-615-460-6206 Nashville, TN 37212 USA Fax: +1-615-460-5458 PGP Key at http://sks-keyservers.net
On May 17, 2011, at 12:46 PM, Erik Schnetter wrote:
Scott
Thanks for showing the code.
I can think of several things that could go wrong:
- It takes some time to start up and shut down parallel threads.
Therefore, people usually parallelise the outermost loop, i.e. the k loop in your case. Parallelising the j loop requires starting and stopping threads nz times, which adds overhead.
- I notice that you don't declare private variables. Variables are
shared by default, and any local variables that you use inside the parallel region (and which are not arrays where you only access one element) need to be declared as private. In your case, these are probably x, y, z, gxx, gxy, gxz, etc. Did you compare results between serial and parallel runs? I would expect the results to differ, i.e. the current parallel code seems to have a serious error.
- How much computational work is done inside this loop? If most of
the time is spent in memory access writing to the gij and Kij arrays, then OpenMP won't be able to help. Only if there is sufficient computation going on will you see a benefit.
- Since you say that you have 24 cores, I assume you have an AMD
system. In this case, your machine consists of 4 subsystems that have 6 cores each, and communication between these 4 subsystems will be much slower than within each of these subsystems. People usually recommend to use not more than 6 OpenMP threads, and to ensure that these run within one of these subsystems. You can try setting the environment variable GOMP_CPU_AFFINITY='0-5' to force your threads to run on cores 0 to 5.
-erik
On Tue, May 17, 2011 at 1:00 PM, Scott Hawley scott.hawley@belmont.edu wrote:
No doubt someone will ask for the code itself. The relevant part is given below. ( In old-school Fortran77) Specifically, the 'problem' I'm noticing is that the "% done" messages appear with less frequency and with lesser increment per wall clock time with OMP_NUM_THREADS > 1 than for OMP_NUM_THREADS = 1. The cpus get used alot more --- 'top' shows up to 2000% cpu usage for 24 threads --- but the wallclock time doesn't decrease at all. Also note that whether I use the long OMP directive shown (with the 'shared' declarations and schedule, etc) and the 'END PARALLEL DO' at the end, or if I just use a simple '!$OMP PARALLEL DO' and *nothing else*, the execution time is *identical*. Thanks again! -Scott
chunk = 8 do k = 1, nz write(msg,*)'[1F setbkgrnd: ' // & 'nz = ',nz,', % done = ',int(k*1.0d2/nz),' ' call writemessage(msg)!$OMP PARALLEL DO SHARED(mask,agxx,agxy,agxz,agyy,agyz,agzz, !$OMP& aKxx,aKxy,aKxz,aKyy,aKyz,aKzz,ax,ay,az), !$OMP& SCHEDULE(STATIC,chunk) PRIVATE(j) do j = 1, ny do i = 1, nx c if (ltrace) then c write(msg,*) '---------------' c call writemessage(msg) c endif
if (mask(i,j,k) .ne. m_ex .and. & (ibonly .eq. 0 .or. & mask(i,j,k) .eq. m_ib)) then x = ax(i) y = ay(j) z = az(k)c the following two include files just perform many pointwise calc's include 'gd.inc' include 'kd.inc' agxx(i,j,k) = gxx agxy(i,j,k) = gxy agxz(i,j,k) = gxz agyy(i,j,k) = gyy agyz(i,j,k) = gyz agzz(i,j,k) = gzz
aKxx(i,j,k) = Kxx aKxy(i,j,k) = Kxy aKxz(i,j,k) = Kxz aKyy(i,j,k) = Kyy aKyz(i,j,k) = Kyz aKzz(i,j,k) = Kzz else if (mask(i,j,k) .eq. m_ex) thenc Excised points agxx(i,j,k) = exval agxy(i,j,k) = exval agxz(i,j,k) = exval agyy(i,j,k) = exval agyz(i,j,k) = exval agzz(i,j,k) = exval
aKxx(i,j,k) = exval aKxy(i,j,k) = exval aKxz(i,j,k) = exval aKyy(i,j,k) = exval aKyz(i,j,k) = exval aKzz(i,j,k) = exval endif enddo enddo!$OMP END PARALLEL DO enddo
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.cct.lsu.edu/~eschnett/