I applied it as a PBS flag at the top of my script. Beforehand I had "#PBS -j oe" for tying together the STDERR and STDOUT. This time, I did "#PBS -j roe", and PBS refused to submit, saying it didn't recognise the syntax: "qsub: illegal -j value".

Perhaps I could have put it in the mpiexec line itself? I haven't experimented with this ... Here's what my normal mpiexec line looks like:

mpiexec -np 64 bin/cactus_hahndol parfiles_Cactus/X2_DU_d10.0_1o96.par | tee X2_DU_d10_1o96.log

Bernard

----------------------------------------------------------------------------- 
Bernard Kelly -- CRESST Research Associate, NASA/GSFC
  
Phone: +1 (301) 286-7243
E-Mail: bernard.j.kelly@nasa.gov
Web: http://science.gsfc.nasa.gov/sed/index.cfm?fuseAction=people.jumpBio&iphonebookid=13052
-----------------------------------------------------------------------------


From: Ian Hinder <ian.hinder@aei.mpg.de>
Date: Tue, 14 Feb 2012 08:56:51 -0600
To: Bernard Kelly <bernard.j.kelly@nasa.gov>
Cc: Erik Schnetter <schnetter@cct.lsu.edu>, "users@einsteintoolkit.org" <users@einsteintoolkit.org>
Subject: Re: [Users] changing CarpetRegrid2 parameters for ETK Maxwell + mercurial Carpet?


On 14 Feb 2012, at 15:39, Kelly, Bernard J. (GSFC-660.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY] wrote:

Hi Erik.

I tried standard output the way you originally outlined, but found I
couldn't get any output from it.

Specifically, the machine wouldn't accept the "-roe" flag, so I couldn't
get full output from all nodes. Hence my trying the CCTK_Info route.

Anyway, I can probably replicate this problem on a different machine (like
Kraken) and use "-roe" there. More coming ...

Out of curiosity, what was the problem with the -roe flag on the original machine?  Did it give an error message?  What was the exact command you ran?