<div dir="ltr">Hi Erik,<div><br></div><div>Thanks for your reassuring reply. </div><div><br></div><div>> Did you look at the generated code? Try calling the compiler with the "-S" option to get assembler output, or use "objdump -d" to disassemble the object file. You should see lots of "ymms" mentioned in your memory access and arithmetic operations.</div><div><br></div><div>Yep, as I mentioned, the `-S -g` commented assembler indeed did output lots of "ymms" in the innermost loop. Also, the annotated assembler on the innermost loop gave precisely the same remark about not being able to vectorize an operation:</div><div><br></div><div> # optimization report<br> # LOOP WITH USER VECTOR INTRINSICS<br> # %s was not vectorized: operation cannot be vectorized<br> # VECTOR TRIP COUNT IS ESTIMATED CONSTANT<br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px">-Zach</div><div style="font-size:12.8px"><br></div><span style="font-size:12.8px">* * *</span><br style="font-size:12.8px"><span style="font-size:12.8px">Prof. Zachariah Etienne</span><br style="font-size:12.8px"><div style="font-size:12.8px">West Virginia University</div><div><font color="#0000ee"><u><a href="https://math.wvu.edu/~zetienne/" target="_blank">https://math.wvu.edu/~zetienne/</a></u></font></div><div><a href="https://blackholesathome.net" style="font-size:12.8px" target="_blank">https://blackholesathome.net</a><br></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 23, 2019 at 11:08 AM Erik Schnetter <<a href="mailto:schnetter@gmail.com">schnetter@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Zach<br>
<br>
The compiler is correct: The intrinsic _mm256_loadu_pd cannot be<br>
vectorized because ... it is already vectorized! If you are using<br>
intrinsics for vectorization, then the compiler does not need to<br>
perform any work.<br>
<br>
Did you look at the generated code? Try calling the compiler with the<br>
"-S" option to get assembler output, or use "objdump -d" to<br>
disassemble the object file. You should see lots of "ymms" mentioned<br>
in your memory access and arithmetic operations.<br>
<br>
-erik<br>
<br>
On Tue, Jul 23, 2019 at 9:35 AM Zach Etienne <<a href="mailto:zachetie@gmail.com" target="_blank">zachetie@gmail.com</a>> wrote:<br>
><br>
> Hi all,<br>
><br>
> I used NRPy+ to create a "minimal example" SIMD-intrinsics-enabled PDE solver kernel -- solving the scalar wave equation in 3 spatial dimensions.<br>
><br>
> With AVX256+FMA intrinsics, neither Intel nor GNU compilers report success at fully vectorizing the RHS eval loop. E.g., the Intel compiler yields the cryptic message when compiling the innermost loop:<br>
><br>
> remark #15310: loop was not vectorized: operation cannot be vectorized [ ScalarWave/ScalarWave_RHSs-SIMD.h(31,52) ]<br>
><br>
> The line it's referring to has to do with loading data from memory _mm256_loadu_pd(&a).<br>
><br>
> The entire source code is attached to this email, and I've been compiling using<br>
><br>
> icc -restrict -align -qopenmp -xHost -O2 -qopt-report=5 -qopt-report-phase ipo -qopt-report-phase vec -vec-threshold1 -qopt-prefetch=4 ScalarWave_Playground-SIMD.c -o ScalarWave_Playground-SIMD<br>
><br>
> for Intel 19, and for GNU (gcc 9):<br>
><br>
> gcc -fsimd-cost-model=unlimited -Ofast -fopenmp -march=native ScalarWave_Playground-SIMD.c -fopt-info-vec-optimized-missed -o ScalarWave_Playground-SIMD -lm<br>
><br>
> When I look at the Intel-generated annotated assembly of the innermost RHS loop (using icc -S -g -restrict -align -qopenmp -xHost -O2 -qopt-report=5 -qopt-report-phase ipo -qopt-report-phase vec -vec-threshold1 -qopt-prefetch=4 ScalarWave_Playground-SIMD.c), I see many 256-bit "ymmX"'s and corresponding instructions that seem to be consistent with the *math* intrinsics. I can't decipher much beyond that, though. Notably I didn't see any assembler instructions that look like _mm256_loadu_pd().<br>
><br>
> I fiddled around a bit with what goes inside the _mm256_loadu_pd(), just to see what might be causing the cryptic remark above. I found that if I remove dependence on the innermost loop variable "i0" on certain (obviously this would break the functionality, but the compiler doesn't care), then it is capable of vectorizing that loop.<br>
><br>
> Note that the version of the code that does not use intrinsics is about 1.8x slower with either compiler, so I think intrinsics are providing some good benefit. However, I am discouraged by the compiler telling me that the inner loop cannot be fully vectorized.<br>
><br>
> Any tips would be greatly appreciated!<br>
><br>
> -Zach<br>
><br>
> * * *<br>
> Prof. Zachariah Etienne<br>
> West Virginia University<br>
> <a href="https://math.wvu.edu/~zetienne/" rel="noreferrer" target="_blank">https://math.wvu.edu/~zetienne/</a><br>
> <a href="https://blackholesathome.net" rel="noreferrer" target="_blank">https://blackholesathome.net</a><br>
> _______________________________________________<br>
> performanceoptimization-wg mailing list<br>
> <a href="mailto:performanceoptimization-wg@einsteintoolkit.org" target="_blank">performanceoptimization-wg@einsteintoolkit.org</a><br>
> <a href="http://lists.einsteintoolkit.org/mailman/listinfo/performanceoptimization-wg" rel="noreferrer" target="_blank">http://lists.einsteintoolkit.org/mailman/listinfo/performanceoptimization-wg</a><br>
<br>
<br>
<br>
-- <br>
Erik Schnetter <<a href="mailto:schnetter@gmail.com" target="_blank">schnetter@gmail.com</a>><br>
<a href="http://www.perimeterinstitute.ca/personal/eschnetter/" rel="noreferrer" target="_blank">http://www.perimeterinstitute.ca/personal/eschnetter/</a><br>
</blockquote></div>