Hello all,
I'd like to use PBS's -V switch (propagates environment settings to the qsub script) on Kraken (through simfactory2). Simfactory currently does not use this swich on Kraken. In fact NICS recommends against using -V (http://www.nics.tennessee.edu/computing-resources/kraken/running-jobs#Batch%...): --8<-- Note: Please do not use the PBS -V option. This can propagate large numbers of environment variable settings from the submitting shell into a job which may cause problems for the batch environment. Instead of using PBS -V, please pass only necessary environment variables using -v <comma_separated_list_of_ needed_envars>. You can also include module load statements in the job script. --8<-- TACC on the other hand actually requires you to have the flag present (for the modules I presume). Did anyone every actually experience any issues with -V on Kraken?
Mostly I would like to use the flag so that Cactus' $ENV{'...'} mechanism works without me having to modify the run/submission scripts by hand. Is there maybe a simfactory option to define extra environment variables (I only need one but would rather not hard-code it into the submission script and instead could add it to the submit call)?
Yours, Roland
Roland
Which variable do you want to set? Did you try -V on Kraken? If it works, then let's use it. Modifying the submit script or run script would also work; no need to provide a generic solution if there's only a single use case.
-erik
On Tue, Feb 21, 2012 at 7:45 PM, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all,
I'd like to use PBS's -V switch (propagates environment settings to the qsub script) on Kraken (through simfactory2). Simfactory currently does not use this swich on Kraken. In fact NICS recommends against using -V (http://www.nics.tennessee.edu/computing-resources/kraken/running-jobs#Batch%...): --8<-- Note: Please do not use the PBS -V option. This can propagate large numbers of environment variable settings from the submitting shell into a job which may cause problems for the batch environment. Instead of using PBS -V, please pass only necessary environment variables using -v <comma_separated_list_of_ needed_envars>. You can also include module load statements in the job script. --8<-- TACC on the other hand actually requires you to have the flag present (for the modules I presume). Did anyone every actually experience any issues with -V on Kraken?
Mostly I would like to use the flag so that Cactus' $ENV{'...'} mechanism works without me having to modify the run/submission scripts by hand. Is there maybe a simfactory option to define extra environment variables (I only need one but would rather not hard-code it into the submission script and instead could add it to the submit call)?
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hi,
I've used -V on kraken using another code successfully, but not ET (haven't tried).
One thing you could do as a compromise is limit your environment variables before submission, only setting the ones you need.
I hope this helps.
cheers, scott n.
On 02/22/2012 12:03 PM, Erik Schnetter wrote:
Roland
Which variable do you want to set? Did you try -V on Kraken? If it works, then let's use it. Modifying the submit script or run script would also work; no need to provide a generic solution if there's only a single use case.
-erik
On Tue, Feb 21, 2012 at 7:45 PM, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all,
I'd like to use PBS's -V switch (propagates environment settings to the qsub script) on Kraken (through simfactory2). Simfactory currently does not use this swich on Kraken. In fact NICS recommends against using -V (http://www.nics.tennessee.edu/computing-resources/kraken/running-jobs#Batch%...): --8<-- Note: Please do not use the PBS -V option. This can propagate large numbers of environment variable settings from the submitting shell into a job which may cause problems for the batch environment. Instead of using PBS -V, please pass only necessary environment variables using -v <comma_separated_list_of_ needed_envars>. You can also include module load statements in the job script. --8<-- TACC on the other hand actually requires you to have the flag present (for the modules I presume). Did anyone every actually experience any issues with -V on Kraken?
Mostly I would like to use the flag so that Cactus' $ENV{'...'} mechanism works without me having to modify the run/submission scripts by hand. Is there maybe a simfactory option to define extra environment variables (I only need one but would rather not hard-code it into the submission script and instead could add it to the submit call)?
Yours, Roland
-- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net.
Users mailing list Users@einsteintoolkit.org http://lists.einsteintoolkit.org/mailman/listinfo/users
Hello Erik, Scott,
I've used -V on kraken using another code successfully, but not ET (haven't tried).
One thing you could do as a compromise is limit your environment variables before submission, only setting the ones you need.
I hope this helps.
At least it tells me that I am not the only one who does what NICS recommends against :-). Thank you. I am not sure how to limit my environment variables before submission though. They are already set at that point. I don't add extra ones via (something equivalent to): INITIAL_DATA=blah qsub Submit.qsub rather they are set in .bashrc (otherwise I tended to forget to set them and have the run abort after waiting for 24hr in queue).
Which variable do you want to set? Did you try -V on Kraken? If it works, then let's use it. Modifying the submit script or run script would also work; no need to provide a generic solution if there's only a single use case.
I have used -V on kraken before (always did in the GT submit script and for maybe 4 runs using a modified simfactory submit script). I never had problems that I would blame on the environment variable.
The variable I would like to pass in is $INITIAL_DATA which is actually a path to a directory containing ID files with LORENE data files. So it is not a "general use" variable that is always defined. Expanding the path by hand in the parameter file is awkward since it would likely hard-code information about the user name and/or the cluster and whether simfactory is used in the parameter file.
NICS suggests to use the -v (lowercase V) switch to pass only individual environment variables (which is close to what you suggested, Scott, isn't it?).
So the simfactory submit script does not not include "-V" because the original author wanted to do what NICS suggests but he omission is without explicit intent?
Yours, Roland
Shouldn't the location of the initial data be in the parameter file? Either this, or it should be compiled into the executable (similar to the location of a shared library).
You could add an option to the thorn calling Lorene to use setenv before calling Lorene.
-erik
On Wed, Feb 22, 2012 at 1:10 PM, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello Erik, Scott,
I've used -V on kraken using another code successfully, but not ET (haven't tried).
One thing you could do as a compromise is limit your environment variables before submission, only setting the ones you need.
I hope this helps.
At least it tells me that I am not the only one who does what NICS recommends against :-). Thank you. I am not sure how to limit my environment variables before submission though. They are already set at that point. I don't add extra ones via (something equivalent to): INITIAL_DATA=blah qsub Submit.qsub rather they are set in .bashrc (otherwise I tended to forget to set them and have the run abort after waiting for 24hr in queue).
Which variable do you want to set? Did you try -V on Kraken? If it works, then let's use it. Modifying the submit script or run script would also work; no need to provide a generic solution if there's only a single use case.
I have used -V on kraken before (always did in the GT submit script and for maybe 4 runs using a modified simfactory submit script). I never had problems that I would blame on the environment variable.
The variable I would like to pass in is $INITIAL_DATA which is actually a path to a directory containing ID files with LORENE data files. So it is not a "general use" variable that is always defined. Expanding the path by hand in the parameter file is awkward since it would likely hard-code information about the user name and/or the cluster and whether simfactory is used in the parameter file.
NICS suggests to use the -v (lowercase V) switch to pass only individual environment variables (which is close to what you suggested, Scott, isn't it?).
So the simfactory submit script does not not include "-V" because the original author wanted to do what NICS suggests but he omission is without explicit intent?
Yours, Roland
Hi,
On Wed, Feb 22, 2012 at 01:35:42PM -0500, Erik Schnetter wrote:
Shouldn't the location of the initial data be in the parameter file?
The location of the initial data is system-dependent, while the parameter file should not.
Either this, or it should be compiled into the executable (similar to the location of a shared library).
It should be possible to set that location at start time. The current mechanism does that: it instructs Cactus to look at an environment variable. For most cases it would work to compile this in, but that would mean to recompile to change just this, where a recompilation isn't strictly necessary and could be avoided.
Frank
If you consider the location to be machine dependent, then it should be part of Simfactory. Simfactory should set this environment variable as specified in its mdb. This will allow your collaborators/students to use the same setup as you, without you having to explain them how to set up their accounts on the various machines you are using.
If you don't think that modifying .bashrc is a problem, then modifying kraken.sub shouldn't be much of a problem either. (Version control systems are good at managing small local changes.)
Alternatively, if you think that Simfactory should help, then you could e.g. source ~/.bashrc in kraken.run -- this would solve similar problems as well. Or you can introduce a file ~/.environment that is sourced by both ~/.bashrc and by kraken.run.
You could also augment Simfactory's MDB to have a notion of a "read-only input data directory" for each machine. Lorene data would go there, and other input data as well. (One could argue that $sourcebasedir is a good location for this, but I believe there are systems where $sourcebasedir is not accessible from compute nodes.) Lorene's input data would be located in $inputdir/$name, where $name is specified in the parameter file (e.g. "Lorene/BNS-08-15").
If you do the latter, then extending the "sync" command to sync the input data directory (or a portion thereof) would also make sense.
-erik
2012/2/22 Frank Loeffler knarf@cct.lsu.edu:
Hi,
On Wed, Feb 22, 2012 at 01:35:42PM -0500, Erik Schnetter wrote:
Shouldn't the location of the initial data be in the parameter file?
The location of the initial data is system-dependent, while the parameter file should not.
Either this, or it should be compiled into the executable (similar to the location of a shared library).
It should be possible to set that location at start time. The current mechanism does that: it instructs Cactus to look at an environment variable. For most cases it would work to compile this in, but that would mean to recompile to change just this, where a recompilation isn't strictly necessary and could be avoided.
Frank
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIcBAEBCAAGBQJPRTcGAAoJEOkzpip+I59kQmwP/iYBaZm/5Wq8yIrTY8ouG3uQ SD9gZAkx3c5CqFCJPJIKFmS0QQYj0MLEWJxl7B7+/cCBYA5aZts0hSq9uAmn/ToB AIqZ1goKvdEE61/36FDGYR9MFEIM77+Pyk0djxksmMCE7CwieGJSB7TIPwDU3O+C QNX6trxKD46wswU+cGNKsfpc3STWuVnAlEzbEgrImBdQOo54SrFi2hP0U70v72Xc 0QlT6t9qxoBg7myKGJBDnqOMWjdPqhTgj9jhoMYpt3+bywl/rQY1Nzzug7Z6Be2I pPRHaXAY6t5ZFZiGM6u/CEKCFG/Aj5/rwRv2/u+KTIHbICgcOfUTwbmHSWbdxV5I y9QffiIsL9JrfJfh/sJia7NEiQn9uLvjd7JFFd+eTh81dyJRp9X/WfMtfMBbVG2o OiTlc9oyTI0SBwSKQxBomLl6lIZqy+jxxYikZufGEItztUKqOxLqnG0xX8NjzfWf Sqp8ESqopsHC9BZs4l2ZlQVGZwG6pBpl4xttd1UoDlPzfMztTCzjo4pKdWI6Cjlr +gjdX5KsQLLtiezzDcw/eZmlK36Doyy2sr83NOFzb6qLhBPxZsNOkYZqwcEq/rPn dJCv0muiXdIaxLfPQCc+dY5MuajywWiDEijIorfiKl4HQOQDGU9yrIsO/8VziFdT CUpNczGpuTN2PGASj3QA =XfGh -----END PGP SIGNATURE-----
Hello all,
If you consider the location to be machine dependent, then it should be part of Simfactory. Simfactory should set this environment variable as specified in its mdb. This will allow your collaborators/students to use the same setup as you, without you having to explain them how to set up their accounts on the various machines you are using.
Having a @DATADIR@ (or similar) directive does seem to make sense. It could point to either a project directory or $WORK or if nothing else is available $SCRATCH.
If you don't think that modifying .bashrc is a problem, then modifying kraken.sub shouldn't be much of a problem either. (Version control systems are good at managing small local changes.)
Well yes. Part of the question was whether to propagate my (local) change to kraken.sub to trunk. It seems as if this is not favoured solution.
Alternatively, if you think that Simfactory should help, then you could e.g. source ~/.bashrc in kraken.run -- this would solve similar problems as well. Or you can introduce a file ~/.environment that is sourced by both ~/.bashrc and by kraken.run.
Doable I think (and what SpEC does). It would have the definite advantage of working for any use of $ENV{'...'} rather than just $INITIAL_DATA. The disadvantage is that -- unless I make a copy of the environment file at submimsion time -- a simulation directory is no longer self contained (using -V it is, since it makes copy of the current env settings).
You could also augment Simfactory's MDB to have a notion of a "read-only input data directory" for each machine. Lorene data would go there, and other input data as well. (One could argue that $sourcebasedir is a good location for this, but I believe there are systems where $sourcebasedir is not accessible from compute nodes.)
Yes. Kraken is one those I think ($HOME is not visible, not sure about the projects directories).
If you do the latter, then extending the "sync" command to sync the input data directory (or a portion thereof) would also make sense.
Thank you all for your suggestions.
So consensus seems to "be nice" and not do what NICS says not to do. I'll play around with the solutions
On Wed, Feb 22, 2012 at 4:58 PM, Roland Haas roland.haas@physics.gatech.edu wrote:
Hello all,
If you consider the location to be machine dependent, then it should be part of Simfactory. Simfactory should set this environment variable as specified in its mdb. This will allow your collaborators/students to use the same setup as you, without you having to explain them how to set up their accounts on the various machines you are using.
Having a @DATADIR@ (or similar) directive does seem to make sense. It could point to either a project directory or $WORK or if nothing else is available $SCRATCH.
DATADIR (better: INPUTDIR, because it's shared between simulations, and simulations shouldn't really write there) would be set in the mdb. A future feature could be to copy data from INPUTDIR to the simulation's directory, if desired. INPUTDIR should be defined in the MDB.
-erik
Hello all,
DATADIR (better: INPUTDIR, because it's shared between simulations, and simulations shouldn't really write there) would be set in the mdb. A future feature could be to copy data from INPUTDIR to the simulation's directory, if desired. INPUTDIR should be defined in the MDB.
NICS seems to have updated their system since I last used it. Now one receives:
---8<--- Warning: Your job requests that all of your current shell environment settings (19410 bytes) be exported to it. This is not recommended, as it may cause problems for the batch environment in some cases. You are strongly encouraged to export individual environment variables with the -v variable_list directive instead; see the qsub man page for details
Please contact help@xsede.org if you need assistance. ---8<---
So I would say that the "-V" option is definitely out.
Yours, Roland
users@lists.einsteintoolkit.org
-
Erik Schnetter -
Frank Loeffler -
Roland Haas -
Scott Noble