Hi everyone,
I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library:
-- In the local cluster, I had the option
LAPACK_LIBS = lapack lapacke in my .ini file
and used #include "lapacke.h" in the code.
-- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg
LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64
LAPACK_LIBS = -mkl
which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h".
Question: Is this the right way to generalise to mkl lapacke?
Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit.
Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like
LAPACKE_dgels -> MKL_LAPACKE_dgels
Thanks!
Chris
Chris
Can you say what is actually going wrong? We need to see an error message or similar to help out.
-erik
On Sat, Mar 24, 2018 at 03:38 Chris Stevens c.stevens@ru.ac.za wrote:
Hi everyone,
I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library:
-- In the local cluster, I had the option
LAPACK_LIBS = lapack lapacke in my .ini file
and used #include "lapacke.h" in the code.
-- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg
LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64
LAPACK_LIBS = -mkl
which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h".
Question: Is this the right way to generalise to mkl lapacke?
Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit.
Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like
LAPACKE_dgels -> MKL_LAPACKE_dgels
Thanks!
Chris
Dr Chris Stevens
Claude Leon Postdoctoral Fellow
Department of Mathematics
Rhodes University
Room 5
Ph: +27 46 603 8932
Web: www.chrisdoesmaths.com _______________________________________________ Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi Erik,
thanks for your quick reply. I'm not so worried about the error, it actually seems to be an MPI issue, even though MPI worked fine on the smaller cluster. Hopefully it isn't a pain to fix.
What I really want to know is the difference between the standard lapacke and intel's mkl one. Is the only difference between the two, if all I used before was functions defined in lapacke.h, to replace this with mkl_lapacke.h? The code seems to run fine with only this change, but I remember seeing somewhere standard lapack function calls but with mkl__ appended at the front. I want to know if I should also modify these function calls as well to make proper use of the mkl. The mkl_lapacke.h header file itself doesn't seem very insightful.
I can't seem to find much useful documentation on this through google...
Cheers,
Chris
On 03/24/2018 04:50 PM, Erik Schnetter wrote:
Chris
Can you say what is actually going wrong? We need to see an error message or similar to help out.
-erik
On Sat, Mar 24, 2018 at 03:38 Chris Stevens <c.stevens@ru.ac.za mailto:c.stevens@ru.ac.za> wrote:
Hi everyone, I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library: -- In the local cluster, I had the option LAPACK_LIBS = lapack lapacke in my .ini file and used #include "lapacke.h" in the code. -- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64 LAPACK_LIBS = -mkl which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h". Question: Is this the right way to generalise to mkl lapacke? Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit. Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like LAPACKE_dgels -> MKL_LAPACKE_dgels Thanks! Chris -- Dr Chris Stevens Claude Leon Postdoctoral Fellow Department of Mathematics Rhodes University Room 5 Ph: +27 46 603 8932 Web: www.chrisdoesmaths.com <http://www.chrisdoesmaths.com> _______________________________________________ Users mailing list Users@einsteintoolkit.org <mailto:Users@einsteintoolkit.org> http://lists.einsteintoolkit.org/mailman/listinfo/users-- Erik Schnetter <schnetter@cct.lsu.edu mailto:schnetter@cct.lsu.edu> http://www.perimeterinstitute.ca/personal/eschnetter/
Chris
I don't think the "mkl_" prefix is significant. I would look out for two things:
(1) I would compare the names, types, and order (!) of the arguments to the functions you are calling, ensuring that they are what you are expect. The Fortran versions of these routines are well standardized, but the C versions might have a different convention. I don't expect any difference there.
(2) I would check that the size of an int (32 or 64) is what you expect. This can usually be chosen at compile time. I know that MKL supports both, so you need to choose the right library at link time that corresponds to the choice you made at compile time. I don't recall how that choice is made at compile time -- it could be the include path, the include file name, or a preprocessor flag that you set. Of course, this needs to correspond to the actual int size that you use in your code. Most likely, your int has 32 bit, which is insufficient if you have really large vectors (with more than 2e9 elements). If you get this wrong, you either get completely nonsensical results or a segfault.
-erik
On Sat, Mar 24, 2018 at 1:47 PM, Chris Stevens c.stevens@ru.ac.za wrote:
Hi Erik,
thanks for your quick reply. I'm not so worried about the error, it actually seems to be an MPI issue, even though MPI worked fine on the smaller cluster. Hopefully it isn't a pain to fix.
What I really want to know is the difference between the standard lapacke and intel's mkl one. Is the only difference between the two, if all I used before was functions defined in lapacke.h, to replace this with mkl_lapacke.h? The code seems to run fine with only this change, but I remember seeing somewhere standard lapack function calls but with mkl__ appended at the front. I want to know if I should also modify these function calls as well to make proper use of the mkl. The mkl_lapacke.h header file itself doesn't seem very insightful.
I can't seem to find much useful documentation on this through google...
Cheers,
Chris
On 03/24/2018 04:50 PM, Erik Schnetter wrote:
Chris
Can you say what is actually going wrong? We need to see an error message or similar to help out.
-erik
On Sat, Mar 24, 2018 at 03:38 Chris Stevens c.stevens@ru.ac.za wrote:
Hi everyone,
I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library:
-- In the local cluster, I had the option
LAPACK_LIBS = lapack lapacke in my .ini file
and used #include "lapacke.h" in the code.
-- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg
LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64
LAPACK_LIBS = -mkl
which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h".
Question: Is this the right way to generalise to mkl lapacke?
Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit.
Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like
LAPACKE_dgels -> MKL_LAPACKE_dgels
Thanks!
Chris
-- Dr Chris Stevens
Claude Leon Postdoctoral Fellow
Department of Mathematics
Rhodes University
Room 5
Ph: +27 46 603 8932
Web: www.chrisdoesmaths.com
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Dr Chris Stevens
Claude Leon Postdoctoral Fellow
Department of Mathematics
Rhodes University
Room 5
Ph: +27 46 603 8932
Web: www.chrisdoesmaths.com
Hi Erik,
thanks for the reply.
It sounds like it will be ok, as my output is not crazy, although I will double check the arguments of the functions in the mkl header.
Thanks,
Chris
On 03/25/2018 01:57 AM, Erik Schnetter wrote:
Chris
I don't think the "mkl_" prefix is significant. I would look out for two things:
(1) I would compare the names, types, and order (!) of the arguments to the functions you are calling, ensuring that they are what you are expect. The Fortran versions of these routines are well standardized, but the C versions might have a different convention. I don't expect any difference there.
(2) I would check that the size of an int (32 or 64) is what you expect. This can usually be chosen at compile time. I know that MKL supports both, so you need to choose the right library at link time that corresponds to the choice you made at compile time. I don't recall how that choice is made at compile time -- it could be the include path, the include file name, or a preprocessor flag that you set. Of course, this needs to correspond to the actual int size that you use in your code. Most likely, your int has 32 bit, which is insufficient if you have really large vectors (with more than 2e9 elements). If you get this wrong, you either get completely nonsensical results or a segfault.
-erik
On Sat, Mar 24, 2018 at 1:47 PM, Chris Stevens c.stevens@ru.ac.za wrote:
Hi Erik,
thanks for your quick reply. I'm not so worried about the error, it actually seems to be an MPI issue, even though MPI worked fine on the smaller cluster. Hopefully it isn't a pain to fix.
What I really want to know is the difference between the standard lapacke and intel's mkl one. Is the only difference between the two, if all I used before was functions defined in lapacke.h, to replace this with mkl_lapacke.h? The code seems to run fine with only this change, but I remember seeing somewhere standard lapack function calls but with mkl__ appended at the front. I want to know if I should also modify these function calls as well to make proper use of the mkl. The mkl_lapacke.h header file itself doesn't seem very insightful.
I can't seem to find much useful documentation on this through google...
Cheers,
Chris
On 03/24/2018 04:50 PM, Erik Schnetter wrote:
Chris
Can you say what is actually going wrong? We need to see an error message or similar to help out.
-erik
On Sat, Mar 24, 2018 at 03:38 Chris Stevens c.stevens@ru.ac.za wrote:
Hi everyone,
I have just ported my code from a local cluster to larger one which uses intel mkl and wanted to clarify the changes I've made are the correct ones wrt the lapacke library:
-- In the local cluster, I had the option
LAPACK_LIBS = lapack lapacke in my .ini file
and used #include "lapacke.h" in the code.
-- In the bigger cluster, I now have to use intel mkl, and I have the following options in my .cfg
LAPACK_DIR = /apps/compilers/intel/parallel_studio_xe_2016/mkl/lib/intel64
LAPACK_LIBS = -mkl
which will only compile my code if I replace #include "lapacke.h" with #include "mkl_lapacke.h".
Question: Is this the right way to generalise to mkl lapacke?
Problem: The code runs fine on the local cluster, but breaks (not obvious where) on the larger one with the above changes. The lapacke functions seem to work OK at a glance, but seeing as this is the only thing that has changed this may still be the culprit.
Question: Do I perhaps need to change the lapacke function calls also? The mkl_lapacke.h header file doesn't seem to give much insight... I think I remember seeing somewhere calls replaced something like
LAPACKE_dgels -> MKL_LAPACKE_dgels
Thanks!
Chris
-- Dr Chris Stevens
Claude Leon Postdoctoral Fellow
Department of Mathematics
Rhodes University
Room 5
Ph: +27 46 603 8932
Web: www.chrisdoesmaths.com
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
-- Erik Schnetter schnetter@cct.lsu.edu http://www.perimeterinstitute.ca/personal/eschnetter/
-- Dr Chris Stevens
Claude Leon Postdoctoral Fellow
Department of Mathematics
Rhodes University
Room 5
Ph: +27 46 603 8932
Web: www.chrisdoesmaths.com
users@lists.einsteintoolkit.org
-
Chris Stevens -
Erik Schnetter