From rhaas at illinois.edu Thu Jun 1 09:56:27 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 1 Jun 2023 09:56:27 -0500 Subject: [Users] meeting minutes for 2023-06-01 Message-ID: <20230601095627.55421d9e@ekohaes8> Present: Zach, Keith, Roland, Steve, Leo, Peter SPECv8 contribution =================== * no specific news, progressing ET release coordinator search ============================= * had planned to include CarpetX and AsterX * no documentation exists for it yet * Roland estimates that CarpetX is not closer to inclusion ready than 6 months ago. CarpetX reviewers * paired coordinator approach was useful for Leo in the 2023_05 release * will pick this up again next week due to small number of participants Unanswered question =================== * https://lists.einsteintoolkit.org/pipermail/users/2023-May/008930.html Roland will forward similar answer about grid structure consistency * https://lists.einsteintoolkit.org/pipermail/users/2023-May/008935.html Roland will respond suggesting that a full C++11 stack is required Error message shown about missing http://lists.einsteintoolkit.org/pipermail/users/2023-June/date.html is likely due there not yet being a posting to the mailing list for June and the script cannot distinguish "no posting yet" from "server is down or cannot access the file". Galley examples =============== * Steve will update the BBH example (https://www.einsteintoolkit.org/gallery/bbh/index.html) ** Zach mentioned ticket https://bitbucket.org/einsteintoolkit/tickets/issues/2706/increase-twopunctures-resolution-in-bbh * Peter had added convergence plot to TOV example (https://www.einsteintoolkit.org/gallery/ns/index.html). Convergence order is between 1 and 2. Zach brought up the issue that gallery examples are typically very low resolution and one often has to increase resolution significantly observed for example good convergence. This typically clashes with the desire to keep the gallery examples small enough so that they can be easily run. Zach noted that the TOV example is not featured in the cycler on the main page. Steve and Zach had a discussion about how one could create a more visually interesting image using VisIt. Steve brought up work on openPMD reader in Python. Roland will work with summer student in summer to add mesh refined openPMD support to VisIt. Will meet with Steve and (NCSA) local visualization expert. Updated tickets =============== * https://bitbucket.org/einsteintoolkit/tickets/issues/2734/consider-adding-if-statement-functionality ** Steve mentioned current situation in CarpetX where storage is always one. ** Steve considers that he may be able to add auto-allocation of storage to the WRITE statements, READ would result in invalid data to read from * https://bitbucket.org/einsteintoolkit/tickets/issues/2733/twopunctures-contains-globally-visible ** we should fix this Open topics =========== * Clang and Flang support. Zach mentioned benchmarks that have clang outperform gcc, asks about support for clang and flang in simfactory and the Einstein Toolkit. ** Roland says no clang support in the past due to lack of Fortran compiler. Steve and Zach point out that gfortran can be used and the there is now also flang. ** Roland mentions that there are "old" flang based on PGI and "new" flang based on llvm. Also AMD compiler and Cray compiler on Crusher / Frontier are llvm based, so are modern version of the Intel compiler suite. ** currently have not knowledge about clang / flang in the Linux known-architecture files, only in Darwin files for (older) clang Next Chair: Steve Next Minutes Taker: Leo Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Jun 1 15:01:28 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 1 Jun 2023 15:01:28 -0500 Subject: [Users] Compilation Error in cluster ParamGanga In-Reply-To: References: <74397676-cd6a-551e-ef18-b5df77edfc71@cct.lsu.edu> <4449ed77-ac78-8774-896b-f86737ea9bf0@cct.lsu.edu> <20230511135018.1d31cfb1@ekohaes8> <20230518090147.4248b227@ekohaes8> Message-ID: <20230601150128.22b31a04@ekohaes8> Hello Shamim Haque, This most likely is caused by not having a full C++11 software stack enabled. The Intel compiler always uses the C++ STL library of the GNU g++ compiler and must be matched up with a compatible g++ compiler's STL to correctly support C++11. This ticket (the last of many, hopefully) contains details: https://bitbucket.org/einsteintoolkit/tickets/issues/2663/add-an-notice-about-minimum-intel-gcc Basically, the fix is to find a gcc compiler at least version 6 on your cluster, then pass the option -gxx-name PATH-TO-GXX to icpc in the CXXFLAGS option list variable (see eg https://bitbucket.org/simfactory/simfactory2/src/7e2592d264b1afe9ca2cb54bcab8dd9e4165cf98/mdb/optionlists/smic-openmpi.cfg?at=master#smic-openmpi.cfg-15:16). If there is no gcc compiler version 6 or newer installed in the system, you will have to contact the administrators and ask them about it. The demo code in the description of https://bitbucket.org/cactuscode/cactus/pull-requests/152 may help demonstrate to the admins what the issue is. Yours, Roland > Hello All, > > I have made little progress on compiling ETK on this cluster, as the admins > allowed me to compile ETK on the login node. I am finding new errors and am > unable to debug the issue. I am trying to configure ETK using Intel19, and > the error says the following: > > > > > > > > > > > > */home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160): > error: more than one instance of overloaded function "CarpetMask::isnan" > matches the argument list: function "isnan(double)" > function "std::isnan(double)" argument types are: (CCTK_REAL8) > assert(not isnan(phi)); ^compilation aborted for > /home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc > (code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: *** [make.checked] > Error 2make[1]: *** > [/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a] > Error 2make: *** [sim] Error 2* > > I have attached the machinefile, optionlist, config log and full output > file for reference. I configured the scripts using the available example > optionlist and machinefile for supermuc_ng and stampede2. Please let me > know if more info is required, and thanks in advance for your help. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ? > > On Fri, May 19, 2023 at 2:32?PM Shamim Haque 1910511 > wrote: > > > Hello Roland, > > > > Thank you for the reply. Yes, I am trying to compile ETK in the computing > > node via sbatch queue. I am in touch with the system admin to resolve > > the issue with the compilers. > > > > Regards > > Shamim Haque > > Senior Research Fellow (SRF) > > Department of Physics > > IISER Bhopal > > > > ? > > > > On Thu, May 18, 2023 at 7:32?PM Roland Haas wrote: > > > >> Hello Shamin, > >> > >> A missing math.h would point to a faulty compiler installation. Since > >> you are trying to compile (are you?) ona compute node in a running job, > >> my guess would be that the cluster admins are not providing compilers > >> on the compute nodes but only on the login nodes. > >> > >> This would be something to contact your cluster's help desk about. > >> > >> Yours, > >> Roland > >> > >> > Hello Roland, > >> > > >> > I did try that fix; the error remains the same. Actually, we landed on > >> an > >> > even more primitive problem, where Steve suggested a basic test, that > >> is to > >> > compile the following code: > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > *#include #include int main() { printf("%f\n", > >> > M_PI); return 0;}* > >> > > >> > This program compiles just fine in the login node but gives out the > >> > following error while compiling it via jobscript submission in the > >> sbatch > >> > queue: > >> > > >> > prog.cpp:1:18: fatal error: math.h: No such file or directory > >> > #include > >> > ^ > >> > compilation terminated. > >> > > >> > ----OR----- > >> > > >> > > >> /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > >> > fatal error: features.h: No such file or directory > >> > 39 | #include > >> > | ^~~~~~~~~~~~ > >> > compilation terminated. > >> > > >> > I have reported this problem to the HPC admin and await their response. > >> > > >> > Regards > >> > Shamim Haque > >> > Senior Research Fellow (SRF) > >> > Department of Physics > >> > IISER Bhopal > >> > > >> > ? > >> > > >> > On Fri, May 12, 2023 at 12:20?AM Roland Haas > >> wrote: > >> > > >> > > Hello all, > >> > > > >> > > Since the error is a missing M_PI and the configure script already > >> > > gives a suggested fix: > >> > > > >> > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > >> > > > >> > > have you tried adding: > >> > > > >> > > CPPFLAGS=-D_XOPEN_SOURCE > >> > > > >> > > to your option list file before compiling? > >> > > > >> > > Yours, > >> > > Roland > >> > > > >> > > > Thanks Steve. I have raised complaint to the admin about this issue. > >> > > > > >> > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > >> sbrandt at cct.lsu.edu> > >> > > wrote: > >> > > > > >> > > > > I would talk to the system administrators about that. There is > >> > > definitely > >> > > > > something wrong with your compiler install if you can't find > >> math.h. > >> > > > > > >> > > > > --Steve > >> > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > >> > > > > > >> > > > > Dear Steve, > >> > > > > > >> > > > > I just rechecked the cpp program to output M_PI in the computing > >> nodes, > >> > > > > and that is not working right now, which was somehow working fine > >> a few > >> > > > > days ago. I also tried different gcc versions (4.8, 10.2, 11.1, > >> > > gnu8/8.3) > >> > > > > available in the cluster. All report similar issue: > >> > > > > > >> > > > > prog.cpp:1:18: fatal error: math.h: No such file or directory > >> > > > > #include > >> > > > > ^ > >> > > > > compilation terminated. > >> > > > > > >> > > > > ----OR----- > >> > > > > > >> > > > > > >> > > > >> /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > >> > >> > > > > fatal error: features.h: No such file or directory > >> > > > > 39 | #include > >> > > > > | ^~~~~~~~~~~~ > >> > > > > compilation terminated. > >> > > > > > >> > > > > The cpp code seems to work fine in login node, as it gives out > >> the > >> > > value > >> > > > > of pi upon execution. I am contacting the hpc admin to resolve > >> this > >> > > issue. > >> > > > > > >> > > > > I am attaching the test code (prog.cpp), the sbatch jobscript > >> (prog.sh) > >> > > > > and the output file (slurn-117473.out), if you would like to have > >> a > >> > > look at > >> > > > > it. > >> > > > > > >> > > > > Please let me know if I should try something different to get > >> this > >> > > working. > >> > > > > > >> > > > > Regards > >> > > > > Shamim Haque > >> > > > > Senior Research Fellow (SRF) > >> > > > > Department of Physics > >> > > > > IISER Bhopal > >> > > > > > >> > > > > ? > >> > > > > > >> > > > > On Mon, May 8, 2023 at 7:25?PM Shamim Haque 1910511 < > >> > > shamims at iiserb.ac.in> > >> > > > > wrote: > >> > > > > > >> > > > >> Thanks Steve. I'll make the changes and try again. I'll let you > >> know > >> > > the > >> > > > >> outcome. > >> > > > >> > >> > > > >> Regards > >> > > > >> Shamim Haque > >> > > > >> Senior Research Fellow (SRF) > >> > > > >> Department of Physics > >> > > > >> IISER Bhopal > >> > > > >> > >> > > > >> ? > >> > > > >> > >> > > > >> On Mon, May 8, 2023 at 7:21?PM Steven R. Brandt < > >> sbrandt at cct.lsu.edu> > >> > > > >> wrote: > >> > > > >> > >> > > > >>> That tends to make me think you aren't using the compiler you > >> think > >> > > > >>> you're using. You could put full paths in the .cfg files and > >> > > recompile from > >> > > > >>> scratch. > >> > > > >>> > >> > > > >>> i.e. > >> > > > >>> > >> > > > >>> CXX = /full/path/to/icc > >> > > > >>> > >> > > > >>> --Steve > >> > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > >> > > > >>> > >> > > > >>> Yes, this gives the value of Pi successfully, before starting > >> the > >> > > > >>> compilation: > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > >> paramganga3 > >> > > Using > >> > > > >>> configuration: sim Reconfiguring sim Writing configuration to: > >> > > > >>> /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList > >> Cactus - > >> > > > >>> version: 4.12.0 Reconfiguring sim. * > >> > > > >>> *.* > >> > > > >>> *.* > >> > > > >>> *.* > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> *checking for M_PI... no configure: error: M_PI not defined. > >> Try > >> > > adding > >> > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring sim-config > >> make: *** > >> > > > >>> [sim-config] Error 2 * > >> > > > >>> > >> > > > >>> > >> > > > >>> Shamim Haque > >> > > > >>> Senior Research Fellow (SRF) > >> > > > >>> Department of Physics > >> > > > >>> IISER Bhopal > >> > > > >>> > >> > > > >>> ? > >> > > > >>> > >> > > > >>> On Thu, May 4, 2023 at 12:29?AM Steven Brandt < > >> sbrandt at cct.lsu.edu> > >> > > > >>> wrote: > >> > > > >>> > >> > > > >>>> So, sanity check, can you compile the following program > >> outside of > >> > > > >>>> Cactus? > >> > > > >>>> > >> > > > >>>> #include > >> > > > >>>> #include > >> > > > >>>> > >> > > > >>>> int main() { > >> > > > >>>> printf("%f\n", M_PI); > >> > > > >>>> return 0; > >> > > > >>>> } > >> > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > >> > > > >>>> > >> > > > >>>> Dear Steve, > >> > > > >>>> > >> > > > >>>> I tried gcc (10 and 11) as well and I am stuck with the > >> following > >> > > error: > >> > > > >>>> *configure: error: M_PI not defined. Try adding > >> -D_XOPEN_SOURCE to > >> > > > >>>> CPPFLAGS.* > >> > > > >>>> > >> > > > >>>> This also happened for Intel (2020 and 2019). I have attached > >> both > >> > > the > >> > > > >>>> output files here. > >> > > > >>>> > >> > > > >>>> Please ignore the out file in the first email, I have > >> uploaded > >> > > another > >> > > > >>>> out file by mistake. The rest of the files are correct in the > >> first > >> > > mail. > >> > > > >>>> > >> > > > >>>> Regards > >> > > > >>>> Shamim Haque > >> > > > >>>> Senior Research Fellow (SRF) > >> > > > >>>> Department of Physics > >> > > > >>>> IISER Bhopal > >> > > > >>>> > >> > > > >>>> ? > >> > > > >>>> > >> > > > >>>> On Wed, May 3, 2023 at 8:04?PM Steven Brandt < > >> sbrandt at cct.lsu.edu> > >> > > > >>>> wrote: > >> > > > >>>> > >> > > > >>>>> I notice from the output that your compiler claims "limits.h" > >> is > >> > > > >>>>> missing. Possibly there's something wrong with your icc > >> setup? > >> > > Maybe you > >> > > > >>>>> should try using gcc / gfortran instead. > >> > > > >>>>> > >> > > > >>>>> --Steve > >> > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > >> > > > >>>>> > >> > > > >>>>> Dear Frank, > >> > > > >>>>> > >> > > > >>>>> I added the alias pattern for the computing nodes and now ETK > >> can > >> > > > >>>>> detect the cluster (checked by asking "whoami") during the > >> batch > >> > > queuing. > >> > > > >>>>> I'll get in touch with the admins to include 'patch'. > >> > > > >>>>> > >> > > > >>>>> However, the problem still persists, having assured that the > >> > > correct > >> > > > >>>>> machine file and option list are being used during the > >> compilation. > >> > > > >>>>> > >> > > > >>>>> Regards > >> > > > >>>>> Shamim Haque > >> > > > >>>>> Senior Research Fellow (SRF) > >> > > > >>>>> Department of Physics > >> > > > >>>>> IISER Bhopal > >> > > > >>>>> > >> > > > >>>>> ? > >> > > > >>>>> > >> > > > >>>>> On Wed, May 3, 2023 at 3:49?PM Frank Loeffler < > >> > > > >>>>> frank.loeffler at uni-jena.de> wrote: > >> > > > >>>>> > >> > > > >>>>>> Hi Shamim > >> > > > >>>>>> > >> > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim Haque > >> 1910511 > >> > > wrote: > >> > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT Roorkee. > >> The > >> > > > >>>>>> >compilation > >> > > > >>>>>> >stops at a very initial step and gives out the error: > >> > > > >>>>>> > > >> > > > >>>>>> >*checking whether the Fortran compiler (gfortran ) > >> works... > >> > > > >>>>>> yeschecking > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > >> > > cross-compiler... > >> > > > >>>>>> nochecking > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) supports TYPE(*) > >> for > >> > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > >> preprocessor... > >> > > > >>>>>> cppchecking > >> > > > >>>>>> >for ANSI C header files... nochecking for C99 features... > >> > > > >>>>>> yeschecking for > >> > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try adding > >> > > > >>>>>> -D_XOPEN_SOURCE to > >> > > > >>>>>> >CPPFLAGS.* > >> > > > >>>>>> > > >> > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but it does > >> not > >> > > help. > >> > > > >>>>>> This > >> > > > >>>>>> >error is consistent if we set up using Intel Compiler (2020 > >> or > >> > > > >>>>>> 2019), or > >> > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error file, > >> > > machine > >> > > > >>>>>> script, > >> > > > >>>>>> >and option lists for both cases (Intel and gcc) for any > >> > > reference. > >> > > > >>>>>> > > >> > > > >>>>>> >Secondly, we are not allowed to compile ETK on login nodes. > >> So > >> > > we are > >> > > > >>>>>> >queuing a job script, which upon execution, enters the > >> Cactus > >> > > > >>>>>> directory, > >> > > > >>>>>> >does "setup-silent", and then executes the build command. > >> Is > >> > > this > >> > > > >>>>>> the right > >> > > > >>>>>> >way to compile ETK using the batch script? Or should we do > >> > > something > >> > > > >>>>>> >different? I have attached the shell script (compile.sh) > >> used > >> > > for > >> > > > >>>>>> sbatch > >> > > > >>>>>> >queue for reference. > >> > > > >>>>>> > >> > > > >>>>>> I might be wrong, but setup-silent will not use your machine > >> file > >> > > > >>>>>> which > >> > > > >>>>>> contains all the compiler flags, but will setup something > >> "from > >> > > > >>>>>> scratch" > >> > > > >>>>>> instead. According to the logs, this will be, e.g., in > >> > > > >>>>>> > >> > > > >>>>>> > >> > > > >>>>>> > >> > > > >> /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini > >> > >> > > > >>>>>> > >> > > > >>>>>> My guess would be that this misses the compiler flags > >> necessary to > >> > > > >>>>>> have > >> > > > >>>>>> M_PI defined. The best way to get this working might be to > >> let > >> > > > >>>>>> simfactory also detect the cluster configuration on the > >> compute > >> > > nodes > >> > > > >>>>>> (where you compile), such that setup-silent is not needed, > >> but > >> > > > >>>>>> instead > >> > > > >>>>>> the correct cluster configuration is found and used > >> > > automatically. In > >> > > > >>>>>> order to do that, look at examples of 'aliaspattern' in the > >> > > > >>>>>> mdb/machines > >> > > > >>>>>> directory. An alternative would be to tell simfactory > >> specifically > >> > > > >>>>>> that > >> > > > >>>>>> you want to use your machine configuration file. > >> > > > >>>>>> > >> > > > >>>>>> As a side-note: I noticed the command 'patch' is missing > >> too. > >> > > This is > >> > > > >>>>>> a > >> > > > >>>>>> tool so common that it should be installed everywhere. Your > >> admin > >> > > is > >> > > > >>>>>> probably the best person to ask for advise here. It should > >> not be > >> > > too > >> > > > >>>>>> hard to install yourself, but that should not be necessary > >> either, > >> > > > >>>>>> especially when you are told to compile on compute nodes. > >> > > > >>>>>> > >> > > > >>>>>> Frank > >> > > > >>>>>> > >> > > > >>>>>> > >> > > > >>>>> _______________________________________________ > >> > > > >>>>> Users mailing listUsers at einsteintoolkit.orghttp:// > >> > > lists.einsteintoolkit.org/mailman/listinfo/users > >> > > > >>>>> > >> > > > >>>>> _______________________________________________ > >> > > > >>>>> Users mailing list > >> > > > >>>>> Users at einsteintoolkit.org > >> > > > >>>>> > >> > > > >> https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4xAKSrBYUo_v8jHEbenNg6aQ99Oq9QyeXsjq766lVDVwHAhFJODYjSU6XryKp264bQqDmBcTIA4Srkc4wOCn$ > >> > >> > > > >>>>> > >> > > > >>>> > >> > > > >> > > > >> > > Yours, > >> > > Roland > >> > > > >> > > -- > >> > > My email is as private as my paper mail. I therefore support > >> encrypting > >> > > and signing email messages. Get my PGP key from > >> https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xbeLFN1po4t2MOETlRoEnO3fr4pBXNdsn_VXBbuywp1tgk0zKYy_oBKphC1Ny-KY0Vh9$ > >> . > >> > > > >> > >> -- > >> My email is as private as my paper mail. I therefore support encrypting > >> and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6bdaoR4iJf6WXHxF1UsHWThqJwM8QxTwHKDWcVgn0kphw4Lmahcfk5a8gMIPSpV8TMD$ . > >> > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Thu Jun 1 16:53:51 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 1 Jun 2023 16:53:51 -0500 Subject: [Users] Thorn setup taking too much time in cluster In-Reply-To: References: <5e5208b8-c440-685a-ffd3-66cd03c3630a@cct.lsu.edu> Message-ID: <20230601165351.225be9bd@ekohaes8> Hello Shamim Haque, "the grid structure inconsistent. Impossible to continue" is a fairly generic error that Carpet outputs if it detects that the grid structure has become inconsistent and it is impossible to continue the simulation. This condition is detected by a low level routine in Carpet that no longer has access to the higher level information that was passed to Carpet and that is actually the cause for the inconsistent grid structure. The mailing list has a couple of similar reports that you could take a look at and that may point to a possible solution. For this sort of error, more details would be needed to do any useful diagnosis. At least the full log file (stdout and stderr) of the simulation up to the point when it aborted would be needed along with possibly some more log files that record the grid structure. Here are some possible relevant email threads: https://lists.einsteintoolkit.org/pipermail/users/2023-March/008881.html https://lists.einsteintoolkit.org/pipermail/users/2021-February/007792.html maybe also: https://bitbucket.org/einsteintoolkit/tickets/issues/2599/nsnstohmns-cannot-be-reproduced-using https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different Yours, Roland > Dear Steve, > > Thank you for your reply. I tried the same simulation with a finer grid, > and the simulation started working fine, even though very slow (looks like > due to slow inter-node communication), but it did work out. I could see a > few iterations towards the final couple of hours from the wall time. > > Turns out, a simulation with GRhydro, in such cases (where the grid needs > to be finer), would end with an error saying, "*the grid structure > inconsistent. Impossible to continue*". On the other hand, a simulation > with IllinoisGRMHD stops abruptly during the thorn setup (somewhere around > the SpaceMask and AHFinderDirect setup). > > Later I tried to see if I can pace up the simulation, but looks like the > inter-node communication is very slow in the HPC, which may be an inherent > problem with the HPC since it is a very old one. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ? > > On Tue, May 23, 2023 at 10:08?PM Steven R. Brandt > wrote: > > > Sorry that no one has replied to you in a while. Are you still > > experiencing this difficulty? > > > > --Steve > > On 4/4/2023 3:08 AM, Shamim Haque 1910511 wrote: > > > > Dear Steven, > > > > I assure you that I submitted the simulation for the first time only. I > > used "sim create-submit" to submit the simulation, which would not submit > > the job if the same name was executed earlier. > > > > Secondly, I found this same message appearing in the output files from > > debug queue (1 node, with GRHydro) and high memory node (3 nodes, with > > IllinoisGRMHD), here the simulation ran successfully. I have attached the > > output files for reference. > > > > Regards > > Shamim Haque > > Senior Research Fellow (SRF) > > Department of Physics > > IISER Bhopal > > > > ? > > > > On Tue, Apr 4, 2023 at 12:35?AM Steven R. Brandt > > wrote: > > > >> I see this error message in your output: > >> > >> -> [0m No HDF5 checkpoint files with basefilename 'checkpoint.chkpt' > >> and file extension '.h5' found in recovery directory > >> 'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25' > >> > >> I suspect you did a "sim submit" for a job, got a failure, and did a > >> second "sim submit" without purging. That immediately triggered the error. > >> Then, for some reason, MPI didn't shut down cleanly and the processes hung > >> doing nothing until they used up the walltime. > >> > >> --Steve > >> On 4/2/2023 5:16 AM, Shamim Haque 1910511 wrote: > >> > >> Hello, > >> > >> I am trying to run BNSM using IllinoisGRMHD on HPC Kanad at IISER Bhopal. > >> While I have tested the parfile to be running fine on debug queue (1 node) > >> and high memory queue (3 nodes), I am unable to run the simulation in a > >> queue with 9 nodes (144 cores). > >> > >> The output file suggests that the setup of listed thorns is not complete > >> within 24 hours, which is the max walltime for this queue. > >> > >> Is there a way to sort out this issue? I have attached the parfile and > >> outfile for reference. > >> > >> Regards > >> Shamim Haque > >> Senior Research Fellow (SRF) > >> Department of Physics > >> IISER Bhopal > >> ? > >> > >> _______________________________________________ > >> Users mailing listUsers at einsteintoolkit.orghttp://lists.einsteintoolkit.org/mailman/listinfo/users > >> > >> _______________________________________________ > >> Users mailing list > >> Users at einsteintoolkit.org > >> https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!_x567GYN6TSCHGzd9qNq7I2vnukVIdWuWrpvklLkBiR2voNBEMX99OkQxtvGmuazb6nd9jcdqRNh8C_eiuyn$ > >> > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From shamims at iiserb.ac.in Tue Jun 6 03:04:45 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Tue, 6 Jun 2023 13:34:45 +0530 Subject: [Users] Thorn setup taking too much time in cluster In-Reply-To: <20230601165351.225be9bd@ekohaes8> References: <5e5208b8-c440-685a-ffd3-66cd03c3630a@cct.lsu.edu> <20230601165351.225be9bd@ekohaes8> Message-ID: Hello Roland, Thanks for pointing me to the links. I'll check out the resources. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? On Fri, Jun 2, 2023 at 3:24?AM Roland Haas wrote: > Hello Shamim Haque, > > "the grid structure inconsistent. Impossible to continue" is a fairly > generic error that Carpet outputs if it detects that the grid structure > has become inconsistent and it is impossible to continue the > simulation. > > This condition is detected by a low level routine in Carpet that no > longer has access to the higher level information that was passed to > Carpet and that is actually the cause for the inconsistent grid > structure. > > The mailing list has a couple of similar reports that you could take a > look at and that may point to a possible solution. > > For this sort of error, more details would be needed to do any useful > diagnosis. At least the full log file (stdout and stderr) of the > simulation up to the point when it aborted would be needed along with > possibly some more log files that record the grid structure. > > Here are some possible relevant email threads: > > https://lists.einsteintoolkit.org/pipermail/users/2023-March/008881.html > > https://lists.einsteintoolkit.org/pipermail/users/2021-February/007792.html > > maybe also: > > > https://bitbucket.org/einsteintoolkit/tickets/issues/2599/nsnstohmns-cannot-be-reproduced-using > > > https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different > > Yours, > Roland > > > Dear Steve, > > > > Thank you for your reply. I tried the same simulation with a finer grid, > > and the simulation started working fine, even though very slow (looks > like > > due to slow inter-node communication), but it did work out. I could see a > > few iterations towards the final couple of hours from the wall time. > > > > Turns out, a simulation with GRhydro, in such cases (where the grid needs > > to be finer), would end with an error saying, "*the grid structure > > inconsistent. Impossible to continue*". On the other hand, a simulation > > with IllinoisGRMHD stops abruptly during the thorn setup (somewhere > around > > the SpaceMask and AHFinderDirect setup). > > > > Later I tried to see if I can pace up the simulation, but looks like the > > inter-node communication is very slow in the HPC, which may be an > inherent > > problem with the HPC since it is a very old one. > > > > Regards > > Shamim Haque > > Senior Research Fellow (SRF) > > Department of Physics > > IISER Bhopal > > > > ? > > > > On Tue, May 23, 2023 at 10:08?PM Steven R. Brandt > > wrote: > > > > > Sorry that no one has replied to you in a while. Are you still > > > experiencing this difficulty? > > > > > > --Steve > > > On 4/4/2023 3:08 AM, Shamim Haque 1910511 wrote: > > > > > > Dear Steven, > > > > > > I assure you that I submitted the simulation for the first time only. I > > > used "sim create-submit" to submit the simulation, which would not > submit > > > the job if the same name was executed earlier. > > > > > > Secondly, I found this same message appearing in the output files from > > > debug queue (1 node, with GRHydro) and high memory node (3 nodes, with > > > IllinoisGRMHD), here the simulation ran successfully. I have attached > the > > > output files for reference. > > > > > > Regards > > > Shamim Haque > > > Senior Research Fellow (SRF) > > > Department of Physics > > > IISER Bhopal > > > > > > ? > > > > > > On Tue, Apr 4, 2023 at 12:35?AM Steven R. Brandt > > > wrote: > > > > > >> I see this error message in your output: > > >> > > >> -> [0m No HDF5 checkpoint files with basefilename > 'checkpoint.chkpt' > > >> and file extension '.h5' found in recovery directory > > >> 'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25' > > >> > > >> I suspect you did a "sim submit" for a job, got a failure, and did a > > >> second "sim submit" without purging. That immediately triggered the > error. > > >> Then, for some reason, MPI didn't shut down cleanly and the processes > hung > > >> doing nothing until they used up the walltime. > > >> > > >> --Steve > > >> On 4/2/2023 5:16 AM, Shamim Haque 1910511 wrote: > > >> > > >> Hello, > > >> > > >> I am trying to run BNSM using IllinoisGRMHD on HPC Kanad at IISER > Bhopal. > > >> While I have tested the parfile to be running fine on debug queue (1 > node) > > >> and high memory queue (3 nodes), I am unable to run the simulation in > a > > >> queue with 9 nodes (144 cores). > > >> > > >> The output file suggests that the setup of listed thorns is not > complete > > >> within 24 hours, which is the max walltime for this queue. > > >> > > >> Is there a way to sort out this issue? I have attached the parfile and > > >> outfile for reference. > > >> > > >> Regards > > >> Shamim Haque > > >> Senior Research Fellow (SRF) > > >> Department of Physics > > >> IISER Bhopal > > >> ? > > >> > > >> _______________________________________________ > > >> Users mailing listUsers at einsteintoolkit.orghttp:// > lists.einsteintoolkit.org/mailman/listinfo/users > > >> > > >> _______________________________________________ > > >> Users mailing list > > >> Users at einsteintoolkit.org > > >> > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!_x567GYN6TSCHGzd9qNq7I2vnukVIdWuWrpvklLkBiR2voNBEMX99OkQxtvGmuazb6nd9jcdqRNh8C_eiuyn$ > > >> > > > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Tue Jun 6 08:25:43 2023 From: rhaas at illinois.edu (Roland Haas) Date: Tue, 6 Jun 2023 08:25:43 -0500 Subject: [Users] Compilation Error in cluster ParamGanga In-Reply-To: References: <74397676-cd6a-551e-ef18-b5df77edfc71@cct.lsu.edu> <4449ed77-ac78-8774-896b-f86737ea9bf0@cct.lsu.edu> <20230511135018.1d31cfb1@ekohaes8> <20230518090147.4248b227@ekohaes8> <20230601150128.22b31a04@ekohaes8> Message-ID: <20230606082543.3131959b@ekohaes8> Hello Shamim Haque, "icpc unable to checkout FLEXlm license" is caused by the compiler being unable to contact the license server. This is unrelated to the C++ compilation issue really. To be sure: * the only thing you did was add "-gxx-name=/opt/ohpc/pub/compiler/gcc/8.3.0/bin/g++" was it? In particular you did not add / remove any "module load" commands (e.g., added a module load gcc after an existing module load intel)? I am asking b/c module load gcc may unload the intel compiler module and unset the environment variable that the Intel compiler uses to determine how to contact the license server. * you may also try to compile with just "make -j1" or "simfactory/bin/sim build -j1" to use only a single compiler instance in case the number of compiler licenses available is limited or the license server overwhelmed by too many concurrent compilations (by you or other users of the cluster) The issue with fftw3 and using gcc is that fftw3 (you are using a system provided fftw3? Either from a module or otherwise provided by the admin) has been compiled with the Intel compiler and thus cannot be used with the GNU compiler. You an set "FFTW_DIR = BUILD" to make the Einstein Toolkit build a copy from scratch using an included tar file. Not that this may be the case for other libraries as well. In particular you usually need to ensure that the MPI library used was compiled with the same compiler you use to compile the simulation code. Yours, Roland > Hello Roland, > > Thank you for the reply. > > I pointed -gxx-name to gcc/8.3.0, which resulted in an error related to the > "icpc unable to checkout FLEXlm license". I also tried to use a > different intel compiler (and point -gxx-name to gcc/11.1.0) to check if > the error can go away, but it remains the same. I communicated this to the > HPC admin and awaiting their response; please let me know if I should fix > something in my configuration. I have attached the out file > (intel_compile.out), config log (intel_config.log), > and optionlist (intel.cfg) for reference. > > Meanwhile, I also attempted to configure ET with gcc/11.1.0, which resulted > in some issues regarding the FFTW3. I have contacted the admins regarding > this, but please let me know if it is something to be fixed within the > optionlist. I have attached the out file (gcc_compile.out), config log > (gcc_config.log), and optionlist (gcc.cfg) for reference. > > The error in the output file says the following: > > > > > > > > > > > > *Formaline: is still running, e.g. the final link stage.)/usr/bin/ld: > warning: libimf.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, > not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: > libsvml.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found > (try using -rpath or -rpath-link)/usr/bin/ld: warning: libirng.so, needed > by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath > or -rpath-link)/usr/bin/ld: warning: libintlc.so.5, needed by > /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or > -rpath-link)/usr/bin/ld: warning: libgfortran.so.3, needed by > /lib/../lib64/liblapack.so, may conflict with > libgfortran.so.5/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > reference to > `__intel_cpu_features_init_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > undefined reference to > `__svml_sincos2'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > reference to > `_intel_fast_memset'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > reference to > `__intel_cpu_feature_indicator_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > undefined reference to > `_intel_fast_memmove'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > reference to `_intel_fast_memcpy'* > > Thanks in advance for all the help. > > Regards > Shamim > > ? > > On Fri, Jun 2, 2023 at 1:31?AM Roland Haas wrote: > > > Hello Shamim Haque, > > > > This most likely is caused by not having a full C++11 software stack > > enabled. > > > > The Intel compiler always uses the C++ STL library of the GNU g++ > > compiler and must be matched up with a compatible g++ compiler's STL > > to correctly support C++11. > > > > This ticket (the last of many, hopefully) contains details: > > > > > > https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/issues/2663/add-an-notice-about-minimum-intel-gcc__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlRGRwlyl$ > > > > Basically, the fix is to find a gcc compiler at least version 6 on your > > cluster, then pass the option > > > > -gxx-name PATH-TO-GXX > > > > to icpc in the CXXFLAGS option list variable (see eg > > > > https://urldefense.com/v3/__https://bitbucket.org/simfactory/simfactory2/src/7e2592d264b1afe9ca2cb54bcab8dd9e4165cf98/mdb/optionlists/smic-openmpi.cfg?at=master*smic-openmpi.cfg-15:16__;Iw!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlSBbGTFf$ > > ). > > > > If there is no gcc compiler version 6 or newer installed in the system, > > you will have to contact the administrators and ask them about it. > > > > The demo code in the description of > > https://urldefense.com/v3/__https://bitbucket.org/cactuscode/cactus/pull-requests/152__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlUmecdAQ$ may help > > demonstrate to the admins what the issue is. > > > > Yours, > > Roland > > > > > Hello All, > > > > > > I have made little progress on compiling ETK on this cluster, as the > > admins > > > allowed me to compile ETK on the login node. I am finding new errors and > > am > > > unable to debug the issue. I am trying to configure ETK using Intel19, > > and > > > the error says the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > */home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160): > > > error: more than one instance of overloaded function "CarpetMask::isnan" > > > matches the argument list: function "isnan(double)" > > > function "std::isnan(double)" argument types are: (CCTK_REAL8) > > > assert(not isnan(phi)); ^compilation aborted for > > > > > /home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc > > > (code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: *** > > [make.checked] > > > Error 2make[1]: *** > > > > > [/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a] > > > Error 2make: *** [sim] Error 2* > > > > > > I have attached the machinefile, optionlist, config log and full output > > > file for reference. I configured the scripts using the available example > > > optionlist and machinefile for supermuc_ng and stampede2. Please let me > > > know if more info is required, and thanks in advance for your help. > > > > > > Regards > > > Shamim Haque > > > Senior Research Fellow (SRF) > > > Department of Physics > > > IISER Bhopal > > > > > > ? > > > > > > On Fri, May 19, 2023 at 2:32?PM Shamim Haque 1910511 < > > shamims at iiserb.ac.in> > > > wrote: > > > > > > > Hello Roland, > > > > > > > > Thank you for the reply. Yes, I am trying to compile ETK in the > > computing > > > > node via sbatch queue. I am in touch with the system admin to resolve > > > > the issue with the compilers. > > > > > > > > Regards > > > > Shamim Haque > > > > Senior Research Fellow (SRF) > > > > Department of Physics > > > > IISER Bhopal > > > > > > > > ? > > > > > > > > On Thu, May 18, 2023 at 7:32?PM Roland Haas > > wrote: > > > > > > > >> Hello Shamin, > > > >> > > > >> A missing math.h would point to a faulty compiler installation. Since > > > >> you are trying to compile (are you?) ona compute node in a running > > job, > > > >> my guess would be that the cluster admins are not providing compilers > > > >> on the compute nodes but only on the login nodes. > > > >> > > > >> This would be something to contact your cluster's help desk about. > > > >> > > > >> Yours, > > > >> Roland > > > >> > > > >> > Hello Roland, > > > >> > > > > >> > I did try that fix; the error remains the same. Actually, we landed > > on > > > >> an > > > >> > even more primitive problem, where Steve suggested a basic test, > > that > > > >> is to > > > >> > compile the following code: > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > *#include #include int main() { printf("%f\n", > > > >> > M_PI); return 0;}* > > > >> > > > > >> > This program compiles just fine in the login node but gives out the > > > >> > following error while compiling it via jobscript submission in the > > > >> sbatch > > > >> > queue: > > > >> > > > > >> > prog.cpp:1:18: fatal error: math.h: No such file or directory > > > >> > #include > > > >> > ^ > > > >> > compilation terminated. > > > >> > > > > >> > ----OR----- > > > >> > > > > >> > > > > >> > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > > > >> > fatal error: features.h: No such file or directory > > > >> > 39 | #include > > > >> > | ^~~~~~~~~~~~ > > > >> > compilation terminated. > > > >> > > > > >> > I have reported this problem to the HPC admin and await their > > response. > > > >> > > > > >> > Regards > > > >> > Shamim Haque > > > >> > Senior Research Fellow (SRF) > > > >> > Department of Physics > > > >> > IISER Bhopal > > > >> > > > > >> > ? > > > >> > > > > >> > On Fri, May 12, 2023 at 12:20?AM Roland Haas > > > >> wrote: > > > >> > > > > >> > > Hello all, > > > >> > > > > > >> > > Since the error is a missing M_PI and the configure script already > > > >> > > gives a suggested fix: > > > >> > > > > > >> > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > > >> > > > > > >> > > have you tried adding: > > > >> > > > > > >> > > CPPFLAGS=-D_XOPEN_SOURCE > > > >> > > > > > >> > > to your option list file before compiling? > > > >> > > > > > >> > > Yours, > > > >> > > Roland > > > >> > > > > > >> > > > Thanks Steve. I have raised complaint to the admin about this > > issue. > > > >> > > > > > > >> > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > > > >> sbrandt at cct.lsu.edu> > > > >> > > wrote: > > > >> > > > > > > >> > > > > I would talk to the system administrators about that. There > > is > > > >> > > definitely > > > >> > > > > something wrong with your compiler install if you can't find > > > >> math.h. > > > >> > > > > > > > >> > > > > --Steve > > > >> > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > > > >> > > > > > > > >> > > > > Dear Steve, > > > >> > > > > > > > >> > > > > I just rechecked the cpp program to output M_PI in the > > computing > > > >> nodes, > > > >> > > > > and that is not working right now, which was somehow working > > fine > > > >> a few > > > >> > > > > days ago. I also tried different gcc versions (4.8, 10.2, > > 11.1, > > > >> > > gnu8/8.3) > > > >> > > > > available in the cluster. All report similar issue: > > > >> > > > > > > > >> > > > > prog.cpp:1:18: fatal error: math.h: No such file or directory > > > >> > > > > #include > > > >> > > > > ^ > > > >> > > > > compilation terminated. > > > >> > > > > > > > >> > > > > ----OR----- > > > >> > > > > > > > >> > > > > > > > >> > > > > > >> > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > >> > > > >> > > > > fatal error: features.h: No such file or directory > > > >> > > > > 39 | #include > > > >> > > > > | ^~~~~~~~~~~~ > > > >> > > > > compilation terminated. > > > >> > > > > > > > >> > > > > The cpp code seems to work fine in login node, as it gives > > out > > > >> the > > > >> > > value > > > >> > > > > of pi upon execution. I am contacting the hpc admin to > > resolve > > > >> this > > > >> > > issue. > > > >> > > > > > > > >> > > > > I am attaching the test code (prog.cpp), the sbatch > > jobscript > > > >> (prog.sh) > > > >> > > > > and the output file (slurn-117473.out), if you would like to > > have > > > >> a > > > >> > > look at > > > >> > > > > it. > > > >> > > > > > > > >> > > > > Please let me know if I should try something different to > > get > > > >> this > > > >> > > working. > > > >> > > > > > > > >> > > > > Regards > > > >> > > > > Shamim Haque > > > >> > > > > Senior Research Fellow (SRF) > > > >> > > > > Department of Physics > > > >> > > > > IISER Bhopal > > > >> > > > > > > > >> > > > > ? > > > >> > > > > > > > >> > > > > On Mon, May 8, 2023 at 7:25?PM Shamim Haque 1910511 < > > > >> > > shamims at iiserb.ac.in> > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > >> Thanks Steve. I'll make the changes and try again. I'll let > > you > > > >> know > > > >> > > the > > > >> > > > >> outcome. > > > >> > > > >> > > > >> > > > >> Regards > > > >> > > > >> Shamim Haque > > > >> > > > >> Senior Research Fellow (SRF) > > > >> > > > >> Department of Physics > > > >> > > > >> IISER Bhopal > > > >> > > > >> > > > >> > > > >> ? > > > >> > > > >> > > > >> > > > >> On Mon, May 8, 2023 at 7:21?PM Steven R. Brandt < > > > >> sbrandt at cct.lsu.edu> > > > >> > > > >> wrote: > > > >> > > > >> > > > >> > > > >>> That tends to make me think you aren't using the compiler > > you > > > >> think > > > >> > > > >>> you're using. You could put full paths in the .cfg files > > and > > > >> > > recompile from > > > >> > > > >>> scratch. > > > >> > > > >>> > > > >> > > > >>> i.e. > > > >> > > > >>> > > > >> > > > >>> CXX = /full/path/to/icc > > > >> > > > >>> > > > >> > > > >>> --Steve > > > >> > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > > > >> > > > >>> > > > >> > > > >>> Yes, this gives the value of Pi successfully, before > > starting > > > >> the > > > >> > > > >>> compilation: > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > > > >> paramganga3 > > > >> > > Using > > > >> > > > >>> configuration: sim Reconfiguring sim Writing configuration > > to: > > > >> > > > >>> > > /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList > > > >> Cactus - > > > >> > > > >>> version: 4.12.0 Reconfiguring sim. * > > > >> > > > >>> *.* > > > >> > > > >>> *.* > > > >> > > > >>> *.* > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> *checking for M_PI... no configure: error: M_PI not > > defined. > > > >> Try > > > >> > > adding > > > >> > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring > > sim-config > > > >> make: *** > > > >> > > > >>> [sim-config] Error 2 * > > > >> > > > >>> > > > >> > > > >>> > > > >> > > > >>> Shamim Haque > > > >> > > > >>> Senior Research Fellow (SRF) > > > >> > > > >>> Department of Physics > > > >> > > > >>> IISER Bhopal > > > >> > > > >>> > > > >> > > > >>> ? > > > >> > > > >>> > > > >> > > > >>> On Thu, May 4, 2023 at 12:29?AM Steven Brandt < > > > >> sbrandt at cct.lsu.edu> > > > >> > > > >>> wrote: > > > >> > > > >>> > > > >> > > > >>>> So, sanity check, can you compile the following program > > > >> outside of > > > >> > > > >>>> Cactus? > > > >> > > > >>>> > > > >> > > > >>>> #include > > > >> > > > >>>> #include > > > >> > > > >>>> > > > >> > > > >>>> int main() { > > > >> > > > >>>> printf("%f\n", M_PI); > > > >> > > > >>>> return 0; > > > >> > > > >>>> } > > > >> > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > > > >> > > > >>>> > > > >> > > > >>>> Dear Steve, > > > >> > > > >>>> > > > >> > > > >>>> I tried gcc (10 and 11) as well and I am stuck with the > > > >> following > > > >> > > error: > > > >> > > > >>>> *configure: error: M_PI not defined. Try adding > > > >> -D_XOPEN_SOURCE to > > > >> > > > >>>> CPPFLAGS.* > > > >> > > > >>>> > > > >> > > > >>>> This also happened for Intel (2020 and 2019). I have > > attached > > > >> both > > > >> > > the > > > >> > > > >>>> output files here. > > > >> > > > >>>> > > > >> > > > >>>> Please ignore the out file in the first email, I have > > > >> uploaded > > > >> > > another > > > >> > > > >>>> out file by mistake. The rest of the files are correct in > > the > > > >> first > > > >> > > mail. > > > >> > > > >>>> > > > >> > > > >>>> Regards > > > >> > > > >>>> Shamim Haque > > > >> > > > >>>> Senior Research Fellow (SRF) > > > >> > > > >>>> Department of Physics > > > >> > > > >>>> IISER Bhopal > > > >> > > > >>>> > > > >> > > > >>>> ? > > > >> > > > >>>> > > > >> > > > >>>> On Wed, May 3, 2023 at 8:04?PM Steven Brandt < > > > >> sbrandt at cct.lsu.edu> > > > >> > > > >>>> wrote: > > > >> > > > >>>> > > > >> > > > >>>>> I notice from the output that your compiler claims > > "limits.h" > > > >> is > > > >> > > > >>>>> missing. Possibly there's something wrong with your icc > > > >> setup? > > > >> > > Maybe you > > > >> > > > >>>>> should try using gcc / gfortran instead. > > > >> > > > >>>>> > > > >> > > > >>>>> --Steve > > > >> > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > > > >> > > > >>>>> > > > >> > > > >>>>> Dear Frank, > > > >> > > > >>>>> > > > >> > > > >>>>> I added the alias pattern for the computing nodes and now > > ETK > > > >> can > > > >> > > > >>>>> detect the cluster (checked by asking "whoami") during > > the > > > >> batch > > > >> > > queuing. > > > >> > > > >>>>> I'll get in touch with the admins to include 'patch'. > > > >> > > > >>>>> > > > >> > > > >>>>> However, the problem still persists, having assured that > > the > > > >> > > correct > > > >> > > > >>>>> machine file and option list are being used during the > > > >> compilation. > > > >> > > > >>>>> > > > >> > > > >>>>> Regards > > > >> > > > >>>>> Shamim Haque > > > >> > > > >>>>> Senior Research Fellow (SRF) > > > >> > > > >>>>> Department of Physics > > > >> > > > >>>>> IISER Bhopal > > > >> > > > >>>>> > > > >> > > > >>>>> ? > > > >> > > > >>>>> > > > >> > > > >>>>> On Wed, May 3, 2023 at 3:49?PM Frank Loeffler < > > > >> > > > >>>>> frank.loeffler at uni-jena.de> wrote: > > > >> > > > >>>>> > > > >> > > > >>>>>> Hi Shamim > > > >> > > > >>>>>> > > > >> > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim Haque > > > >> 1910511 > > > >> > > wrote: > > > >> > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT > > Roorkee. > > > >> The > > > >> > > > >>>>>> >compilation > > > >> > > > >>>>>> >stops at a very initial step and gives out the error: > > > >> > > > >>>>>> > > > > >> > > > >>>>>> >*checking whether the Fortran compiler (gfortran ) > > > >> works... > > > >> > > > >>>>>> yeschecking > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > > > >> > > cross-compiler... > > > >> > > > >>>>>> nochecking > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) supports > > TYPE(*) > > > >> for > > > >> > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > > > >> preprocessor... > > > >> > > > >>>>>> cppchecking > > > >> > > > >>>>>> >for ANSI C header files... nochecking for C99 > > features... > > > >> > > > >>>>>> yeschecking for > > > >> > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try > > adding > > > >> > > > >>>>>> -D_XOPEN_SOURCE to > > > >> > > > >>>>>> >CPPFLAGS.* > > > >> > > > >>>>>> > > > > >> > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but it > > does > > > >> not > > > >> > > help. > > > >> > > > >>>>>> This > > > >> > > > >>>>>> >error is consistent if we set up using Intel Compiler > > (2020 > > > >> or > > > >> > > > >>>>>> 2019), or > > > >> > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error > > file, > > > >> > > machine > > > >> > > > >>>>>> script, > > > >> > > > >>>>>> >and option lists for both cases (Intel and gcc) for > > any > > > >> > > reference. > > > >> > > > >>>>>> > > > > >> > > > >>>>>> >Secondly, we are not allowed to compile ETK on login > > nodes. > > > >> So > > > >> > > we are > > > >> > > > >>>>>> >queuing a job script, which upon execution, enters the > > > >> Cactus > > > >> > > > >>>>>> directory, > > > >> > > > >>>>>> >does "setup-silent", and then executes the build > > command. > > > >> Is > > > >> > > this > > > >> > > > >>>>>> the right > > > >> > > > >>>>>> >way to compile ETK using the batch script? Or should we > > do > > > >> > > something > > > >> > > > >>>>>> >different? I have attached the shell script > > (compile.sh) > > > >> used > > > >> > > for > > > >> > > > >>>>>> sbatch > > > >> > > > >>>>>> >queue for reference. > > > >> > > > >>>>>> > > > >> > > > >>>>>> I might be wrong, but setup-silent will not use your > > machine > > > >> file > > > >> > > > >>>>>> which > > > >> > > > >>>>>> contains all the compiler flags, but will setup > > something > > > >> "from > > > >> > > > >>>>>> scratch" > > > >> > > > >>>>>> instead. According to the logs, this will be, e.g., in > > > >> > > > >>>>>> > > > >> > > > >>>>>> > > > >> > > > >>>>>> > > > >> > > > > > >> > > /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini > > > >> > > > >> > > > >>>>>> > > > >> > > > >>>>>> My guess would be that this misses the compiler flags > > > >> necessary to > > > >> > > > >>>>>> have > > > >> > > > >>>>>> M_PI defined. The best way to get this working might be > > to > > > >> let > > > >> > > > >>>>>> simfactory also detect the cluster configuration on the > > > >> compute > > > >> > > nodes > > > >> > > > >>>>>> (where you compile), such that setup-silent is not > > needed, > > > >> but > > > >> > > > >>>>>> instead > > > >> > > > >>>>>> the correct cluster configuration is found and used > > > >> > > automatically. In > > > >> > > > >>>>>> order to do that, look at examples of 'aliaspattern' in > > the > > > >> > > > >>>>>> mdb/machines > > > >> > > > >>>>>> directory. An alternative would be to tell simfactory > > > >> specifically > > > >> > > > >>>>>> that > > > >> > > > >>>>>> you want to use your machine configuration file. > > > >> > > > >>>>>> > > > >> > > > >>>>>> As a side-note: I noticed the command 'patch' is > > missing > > > >> too. > > > >> > > This is > > > >> > > > >>>>>> a > > > >> > > > >>>>>> tool so common that it should be installed everywhere. > > Your > > > >> admin > > > >> > > is > > > >> > > > >>>>>> probably the best person to ask for advise here. It > > should > > > >> not be > > > >> > > too > > > >> > > > >>>>>> hard to install yourself, but that should not be > > necessary > > > >> either, > > > >> > > > >>>>>> especially when you are told to compile on compute nodes. > > > >> > > > >>>>>> > > > >> > > > >>>>>> Frank > > > >> > > > >>>>>> > > > >> > > > >>>>>> > > > >> > > > >>>>> _______________________________________________ > > > >> > > > >>>>> Users mailing listUsers at einsteintoolkit.orghttp:// > > > >> > > lists.einsteintoolkit.org/mailman/listinfo/users > > > >> > > > >>>>> > > > >> > > > >>>>> _______________________________________________ > > > >> > > > >>>>> Users mailing list > > > >> > > > >>>>> Users at einsteintoolkit.org > > > >> > > > >>>>> > > > >> > > > > > >> > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4xAKSrBYUo_v8jHEbenNg6aQ99Oq9QyeXsjq766lVDVwHAhFJODYjSU6XryKp264bQqDmBcTIA4Srkc4wOCn$ > > > >> > > > >> > > > >>>>> > > > >> > > > >>>> > > > >> > > > > > >> > > > > > >> > > Yours, > > > >> > > Roland > > > >> > > > > > >> > > -- > > > >> > > My email is as private as my paper mail. I therefore support > > > >> encrypting > > > >> > > and signing email messages. Get my PGP key from > > > >> > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xbeLFN1po4t2MOETlRoEnO3fr4pBXNdsn_VXBbuywp1tgk0zKYy_oBKphC1Ny-KY0Vh9$ > > > >> . > > > >> > > > > > >> > > > >> -- > > > >> My email is as private as my paper mail. I therefore support > > encrypting > > > >> and signing email messages. Get my PGP key from > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6bdaoR4iJf6WXHxF1UsHWThqJwM8QxTwHKDWcVgn0kphw4Lmahcfk5a8gMIPSpV8TMD$ > > . > > > >> > > > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlS57nxFF$ . > > Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From rhaas at illinois.edu Tue Jun 6 08:26:23 2023 From: rhaas at illinois.edu (Roland Haas) Date: Tue, 6 Jun 2023 08:26:23 -0500 Subject: [Users] Thorn setup taking too much time in cluster In-Reply-To: References: <5e5208b8-c440-685a-ffd3-66cd03c3630a@cct.lsu.edu> <20230601165351.225be9bd@ekohaes8> Message-ID: <20230606082623.7884bcbe@ekohaes8> Hello Shamim Haque, Ok. Yours, Roland > Hello Roland, > > Thanks for pointing me to the links. I'll check out the resources. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ? > > On Fri, Jun 2, 2023 at 3:24?AM Roland Haas wrote: > > > Hello Shamim Haque, > > > > "the grid structure inconsistent. Impossible to continue" is a fairly > > generic error that Carpet outputs if it detects that the grid structure > > has become inconsistent and it is impossible to continue the > > simulation. > > > > This condition is detected by a low level routine in Carpet that no > > longer has access to the higher level information that was passed to > > Carpet and that is actually the cause for the inconsistent grid > > structure. > > > > The mailing list has a couple of similar reports that you could take a > > look at and that may point to a possible solution. > > > > For this sort of error, more details would be needed to do any useful > > diagnosis. At least the full log file (stdout and stderr) of the > > simulation up to the point when it aborted would be needed along with > > possibly some more log files that record the grid structure. > > > > Here are some possible relevant email threads: > > > > https://urldefense.com/v3/__https://lists.einsteintoolkit.org/pipermail/users/2023-March/008881.html__;!!DZ3fjg!7i12qoKBFzgYKbEGMk2b6YLNEpw3ozT2F8ckbK4GcAz2DyvjiD9kRk44tlGqfH28gU8BGYKS0zxiiEfa86nK$ > > > > https://urldefense.com/v3/__https://lists.einsteintoolkit.org/pipermail/users/2021-February/007792.html__;!!DZ3fjg!7i12qoKBFzgYKbEGMk2b6YLNEpw3ozT2F8ckbK4GcAz2DyvjiD9kRk44tlGqfH28gU8BGYKS0zxiiH9Bokui$ > > > > maybe also: > > > > > > https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/issues/2599/nsnstohmns-cannot-be-reproduced-using__;!!DZ3fjg!7i12qoKBFzgYKbEGMk2b6YLNEpw3ozT2F8ckbK4GcAz2DyvjiD9kRk44tlGqfH28gU8BGYKS0zxiiDTjn80-$ > > > > > > https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/issues/2516/different-simulation-result-with-different__;!!DZ3fjg!7i12qoKBFzgYKbEGMk2b6YLNEpw3ozT2F8ckbK4GcAz2DyvjiD9kRk44tlGqfH28gU8BGYKS0zxiiDGaf7Zj$ > > > > Yours, > > Roland > > > > > Dear Steve, > > > > > > Thank you for your reply. I tried the same simulation with a finer grid, > > > and the simulation started working fine, even though very slow (looks > > like > > > due to slow inter-node communication), but it did work out. I could see a > > > few iterations towards the final couple of hours from the wall time. > > > > > > Turns out, a simulation with GRhydro, in such cases (where the grid needs > > > to be finer), would end with an error saying, "*the grid structure > > > inconsistent. Impossible to continue*". On the other hand, a simulation > > > with IllinoisGRMHD stops abruptly during the thorn setup (somewhere > > around > > > the SpaceMask and AHFinderDirect setup). > > > > > > Later I tried to see if I can pace up the simulation, but looks like the > > > inter-node communication is very slow in the HPC, which may be an > > inherent > > > problem with the HPC since it is a very old one. > > > > > > Regards > > > Shamim Haque > > > Senior Research Fellow (SRF) > > > Department of Physics > > > IISER Bhopal > > > > > > ? > > > > > > On Tue, May 23, 2023 at 10:08?PM Steven R. Brandt > > > wrote: > > > > > > > Sorry that no one has replied to you in a while. Are you still > > > > experiencing this difficulty? > > > > > > > > --Steve > > > > On 4/4/2023 3:08 AM, Shamim Haque 1910511 wrote: > > > > > > > > Dear Steven, > > > > > > > > I assure you that I submitted the simulation for the first time only. I > > > > used "sim create-submit" to submit the simulation, which would not > > submit > > > > the job if the same name was executed earlier. > > > > > > > > Secondly, I found this same message appearing in the output files from > > > > debug queue (1 node, with GRHydro) and high memory node (3 nodes, with > > > > IllinoisGRMHD), here the simulation ran successfully. I have attached > > the > > > > output files for reference. > > > > > > > > Regards > > > > Shamim Haque > > > > Senior Research Fellow (SRF) > > > > Department of Physics > > > > IISER Bhopal > > > > > > > > ? > > > > > > > > On Tue, Apr 4, 2023 at 12:35?AM Steven R. Brandt > > > > wrote: > > > > > > > >> I see this error message in your output: > > > >> > > > >> -> [0m No HDF5 checkpoint files with basefilename > > 'checkpoint.chkpt' > > > >> and file extension '.h5' found in recovery directory > > > >> 'nsns_toy1.2_DDME2BPS_quark_1.2vs1.6M_40km_g25' > > > >> > > > >> I suspect you did a "sim submit" for a job, got a failure, and did a > > > >> second "sim submit" without purging. That immediately triggered the > > error. > > > >> Then, for some reason, MPI didn't shut down cleanly and the processes > > hung > > > >> doing nothing until they used up the walltime. > > > >> > > > >> --Steve > > > >> On 4/2/2023 5:16 AM, Shamim Haque 1910511 wrote: > > > >> > > > >> Hello, > > > >> > > > >> I am trying to run BNSM using IllinoisGRMHD on HPC Kanad at IISER > > Bhopal. > > > >> While I have tested the parfile to be running fine on debug queue (1 > > node) > > > >> and high memory queue (3 nodes), I am unable to run the simulation in > > a > > > >> queue with 9 nodes (144 cores). > > > >> > > > >> The output file suggests that the setup of listed thorns is not > > complete > > > >> within 24 hours, which is the max walltime for this queue. > > > >> > > > >> Is there a way to sort out this issue? I have attached the parfile and > > > >> outfile for reference. > > > >> > > > >> Regards > > > >> Shamim Haque > > > >> Senior Research Fellow (SRF) > > > >> Department of Physics > > > >> IISER Bhopal > > > >> ? > > > >> > > > >> _______________________________________________ > > > >> Users mailing listUsers at einsteintoolkit.orghttp:// > > lists.einsteintoolkit.org/mailman/listinfo/users > > > >> > > > >> _______________________________________________ > > > >> Users mailing list > > > >> Users at einsteintoolkit.org > > > >> > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!_x567GYN6TSCHGzd9qNq7I2vnukVIdWuWrpvklLkBiR2voNBEMX99OkQxtvGmuazb6nd9jcdqRNh8C_eiuyn$ > > > >> > > > > > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!7i12qoKBFzgYKbEGMk2b6YLNEpw3ozT2F8ckbK4GcAz2DyvjiD9kRk44tlGqfH28gU8BGYKS0zxiiHEsY4GE$ . > > Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From physicsbeany at gmail.com Wed Jun 7 09:46:19 2023 From: physicsbeany at gmail.com (Bernard Kelly) Date: Wed, 7 Jun 2023 10:46:19 -0400 Subject: [Users] ET_2023_05: Hydro_RNSID evolution example parfiles should drop ADMMacros Message-ID: Hi. I was just looking at the Hydro_RNSID thorn in the latest ET release, and the test cases in par (all Perl-generated), which use GRHydro. Running one of these (e.g. speedA0_ppm_r12.par), I immediately hit an error: ------- PARAMETER ERROR (GRHydro): ADMMacros::spatial_order (4) differs from GRHydro::sources_spatial_order (2), this is almost certainly incorrect since GRHydro no longer uses ADMMacros PARAMETER ERROR (GRHydro): ADMMacros::spatial_order (4) differs from GRHydro::sources_spatial_order (2), this is almost certainly incorrect since GRHydro no longer uses ADMMacros -------- So can ADMMacros be explicitly removed from these Hydro_RNSID parameter files? Thanks, Bernard -- ------------------------------------------------------------------ Bernard Kelly -- CRESST Assistant Research Scientist, NASA/GSFC Gravitational Astrophysics Laboratory --- Code 663 Phone: +1 (301) 286-7243 *** Fax: +1 (301) 286-2226 Web: http://science.gsfc.nasa.gov/sed/bio/bernard.j.kelly ORCID: orcid.org/0000-0002-3326-4454 ------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruno.giacomazzo at unimib.it Thu Jun 8 04:16:16 2023 From: bruno.giacomazzo at unimib.it (Bruno Giacomazzo) Date: Thu, 8 Jun 2023 11:16:16 +0200 Subject: [Users] 2-year postdoc position in numerical relativity in Milan, Italy Message-ID: At the University of Milano-Bicocca (Milan, Italy), we have an opening for two postdoc positions "on the development of algorithms, codes and computational strategies for theoretical physics, theoretical astrophysics, and high-energy physics". Candidates with experience in *numerical relativity* are very strongly encouraged to apply. Candidates at the end of their PhD can also apply. *The deadline is June 28*. The candidate is expected to start by October. The salary will be around 2000 EUR per month (after taxes) and the position is for 2 years. Reference letters are not needed and the application has to be submitted on line at https://pica.cineca.it/unimib/23a058/ See the call document at https://www.unimib.it/ateneo/gare-e-concorsi/cod-23a058 for the instructions and the details of the selection procedure. Make sure to attach also the pdf of the published version (journal version) of all your papers with the application. Candidates that will be selected for the interview will be interviewed on 05/07/2023 at 10:00 (Rome Time). When submitting their application, candidates shall enter their e-mail/telematic contact (Skype, Cisco Webex Meetings, Google Hangouts or Google Meet) for the configuration of the online interview. The astrophysics group at Milano-Bicocca provides a vibrant environment with expertise covering all aspects of gravitational-wave astronomy, relativistic astrophysics, galactic dynamics, and numerical relativity. This is embedded in a wider astronomical context including both observational and experimental activities. Our group has tight connections with the LISA Consortium, the Virgo Collaboration, the Einstein Telescope Science Board, the European Pulsar Timing Array, and the Italian National Institute for Nuclear Physics (INFN) via the TEONGRAV national initiative. Staff members in Gravitational Waves include Colpi, Dotti, Gerosa, Giacomazzo , and Sesana. Milan is a beautiful, international city in the north of Italy. Mountains and lakes are just around the corner. Art, culture, food, and nightlife are outstanding. The city hosts three international airports with worldwide connections. Please forward this email to potential candidates. Thanks, Bruno -- Prof. Bruno Giacomazzo Department of Physics University of Milano-Bicocca Piazza della Scienza 3 20126 Milano Italy email: bruno.giacomazzo at unimib.it phone: (+39) 02 6448 2321 web: http://www.brunogiacomazzo.org --------------------------------------------------------------------- There are only 10 types of people in the world: Those who understand binary, and those who don't ---------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachetie at gmail.com Thu Jun 8 09:32:50 2023 From: zachetie at gmail.com (Zach Etienne) Date: Thu, 8 Jun 2023 07:32:50 -0700 Subject: [Users] ET_2023_05: Hydro_RNSID evolution example parfiles should drop ADMMacros In-Reply-To: References: Message-ID: Hi Beany, Simply set GRHydro::sources_spatial_order = 4 and remove ADMMacros. Sorry it's not so explicit. -Zach * * * Zachariah Etienne Assoc. Prof. of Physics, U. of Idaho Adjunct Assoc. Prof. of Physics & Astronomy, West Virginia U. https://etienneresearch.com https://blackholesathome.net On Wed, Jun 7, 2023 at 7:46?AM Bernard Kelly wrote: > Hi. > > I was just looking at the Hydro_RNSID thorn in the latest ET release, and > the test cases in par (all Perl-generated), which use GRHydro. Running one > of these (e.g. speedA0_ppm_r12.par), I immediately hit an error: > > ------- > PARAMETER ERROR (GRHydro): ADMMacros::spatial_order (4) differs from > GRHydro::sources_spatial_order (2), this is almost certainly incorrect > since GRHydro no longer uses ADMMacros > PARAMETER ERROR (GRHydro): ADMMacros::spatial_order (4) differs from > GRHydro::sources_spatial_order (2), this is almost certainly incorrect > since GRHydro no longer uses ADMMacros > -------- > > So can ADMMacros be explicitly removed from these Hydro_RNSID parameter > files? > > Thanks, Bernard > > -- > ------------------------------------------------------------------ > Bernard Kelly -- CRESST Assistant Research Scientist, NASA/GSFC > Gravitational Astrophysics Laboratory --- Code 663 > > Phone: +1 (301) 286-7243 *** Fax: +1 (301) 286-2226 > Web: http://science.gsfc.nasa.gov/sed/bio/bernard.j.kelly > ORCID: orcid.org/0000-0002-3326-4454 > ------------------------------------------------------------------ > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wernecklr at gmail.com Thu Jun 8 09:37:25 2023 From: wernecklr at gmail.com (Leonardo Rosa Werneck) Date: Thu, 8 Jun 2023 07:37:25 -0700 Subject: [Users] Meeting minutes for 2023-06-08 Message-ID: <76BEA626-7C59-46E0-BD64-55A3D704DB12@gmail.com> Hi all, Here are the minutes for today?s meeting: Chair : Steve Minutes: Leo Present: Steve Brandt, Peter Diener, Zach Etienne, Roland Haas, Leo Werneck, Yosef Zlochower Nothing specific added to the agenda. * Next ET release manager search: - Sam Cupp is currently expected to be the lead release manager of the ET_11_2023 release, but he was not in today's telecon. Leo will ask him in person. * ET Summer School and Workshop: - Yosef said a discount code that allows folks to book their room at the RIT Inn has been sent to everyone who already registered. Participants are encouraged to book their rooms as soon as possible. As a reminder, the RIT Inn is the only hotel that offers free ground transportation to/from the RIT. - If someone booked their hotel earlier without using their code, they should reach out to Yosef so that they can see what can be done. - There are people who began their registration but never finished. Unfortunately the organizers do not have access to any information that identifies these people. They are encouraging everyone who wants to register to do so by completing the registration forms all the way through. * NRPy+ modernization: - Zach wants to announce that NRPy+ is undergoing an extensive modernization that will allow it to be pip installed. - Steve suggests all core NRPy+ modules should be moved inside a nrpy/ directory as soon as possible to avoid future issues. - Steve suggests a 'project class' that would hold all global variables (e.g., gridfunction list), but Zach does not think it is necessary since modules containing global variables contain very few of them. - This new version of NRPy+ (still under development) requires Python >=3.6, and thus supports f-strings and type hinting/mypy. Support for Python 2 will be dropped once this new version becomes the official version. * Unanswered question on the mailing list: - Bernard Kelly asked a question regarding whether or not ADMMacros should be removed from parameter files that use GRHydro for the MHD evolution. Steve is writing a reply to that question, which likely involves removing the ADMMacros thorn from the parameter file but updating the value of a new GRHydro parameter that replaces the one from ADMMacros. Next week chair : Peter Next week minutes: Roland Cheers, Leo ------ Leonardo R. Werneck, Ph.D. Postdoctoral researcher Office EP 314 | Department of Physics | University of Idaho 875 Perimeter Dr. MS 0903 Moscow, ID 83844-0903, USA leonardo at uidaho.edu https://leowerneck.github.io -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Thu Jun 15 10:05:33 2023 From: rhaas at illinois.edu (Roland Haas) Date: Thu, 15 Jun 2023 10:05:33 -0500 Subject: [Users] meeting minutes for 2023-06-15 Message-ID: <20230615100533.2657a7f7@ekohaes8> Present: Peter, Roland, Leo, Sam, Steve, Yosef open tickets: Ticket 2735 =========== * proposed simplification of schedule.ccl for ADMBase, HydroBase, and TmunuBase * Sam simplified the schedule making use of more modern STORAGE foo[timelevels] constructs * there is also checks if intial_FOO is "none" in which case no storage is allocated for FOO * Roland says that these should be retained Ticket 2742 =========== * proposed restructuring of Seed_Magnetic_Fields * Sam described the two Seed_Magnetic_Fields thorns in WVUThorns and WVUThorns_Diagnostics and desire to merge them into a single thorn * added schedule statements to make thorns use with just HydroBase rather than IllinoisGRMHD * PR is about to be created * plan is to deprecated Seed_Magnetic_Fields_BNS in the next release and remove the release after next ticket 2627 =========== * announce removal of TmunuBase inherting from ADMBase and StaticConformal * would remove more uses of StaticConformal and simplify code * will proceed with deprecation warning ticket 2497 =========== * Sam and Leo are in conversation with Vikram Manikantan about issues they face running IllinoisGRMHD code * Leo suggests to use magnetized TOV test as parfile to study this ticket 2741 =========== * some thorns seem to incorrectly use CCTK_GFINDEX4D * Sam replace some more instances of CCTK_GFINDEX4D with CCTK_VECTGFINDEX3D * Sam will close ticket ticket 2526 =========== * McLachlan has incorrect READ declarations for presync. * Ian is busy right now and is asking for others that could volunteer to review change to Kranc ET release: gallery examples ================ * BBH gallery example updated by Steve * Peter will update gallery example GRHayL library inclusion proposal in ET_2023_11 =============================================== * Ticket 2497, as scheduling doesn't mix con2prim and updating Tmunu * IllinoisGRMHD would no longer see feature updates, be marked as deprecated and eventually archived. * Peter suggests that a "transition guide" from IllinoisGRMHD to GRHayL be provided when IllinoisGRMHD is no longer updated. Sam has a draft of such a document written already. * Roland mentioned to carry IllinoisGRMHD along in the ET even after it no longer sees updates until maintaining it becomes prohibitive. At which point it would move to a archivedthorns-IllinoisGRMHD repository * this will not affect GiRaFFE process to deprecate features ============================= * Sam had asked about procedure * Release announcement will be official notification * process https://docs.einsteintoolkit.org/et-docs/Policies_to_retire_functionality release process updates ideas ============================= * Sam reported on issues he encountered some issues when creating the new release * Sam will update "unshallow" commands to download instructions * notes added to https://docs.einsteintoolkit.org/et-docs/Release_Process_Updates timing for ET_2023_11 meeting ============================= * schedule: https://docs.einsteintoolkit.org/et-docs/Release_Details * proposal deadline for new features would be weeks of RIT meeting * Sam (release manager) will shift choose features date back by one week, keeping other dates the same ET meetings: EU ET meeting ============= * Steve has been working on making the tutorial server a multi-node setup, will test after meeting * will skip ET call next week US ET meeting ============= * please check that you have received registration confirmation email if you registered. If no email was received then the registration is not complete. * RIT Inn reservation code sent at registration, if you already reserved the room without the reservation code, contact the organizers to have the code applied (and be able to check out on Saturday if so desired) * the reserved block will go away soon, please reserve your room soon * Yosef will check if hotel shuttle runs to the airport * Shuttle from RIT Inn to RIT is provided during extended business hours, shuttle runs by request chair week after next (June 29): Leo minutes week after next (June 29): Sam Yours, Roland -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://pgp.mit.edu . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From shamims at iiserb.ac.in Mon Jun 19 09:21:41 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Mon, 19 Jun 2023 19:51:41 +0530 Subject: [Users] Compilation Error in cluster ParamGanga In-Reply-To: <20230606082543.3131959b@ekohaes8> References: <74397676-cd6a-551e-ef18-b5df77edfc71@cct.lsu.edu> <4449ed77-ac78-8774-896b-f86737ea9bf0@cct.lsu.edu> <20230511135018.1d31cfb1@ekohaes8> <20230518090147.4248b227@ekohaes8> <20230601150128.22b31a04@ekohaes8> <20230606082543.3131959b@ekohaes8> Message-ID: Dear Roland, Steve and Frank, I am able to compile ETK properly with GNU compiler 8.3.0. The simulations are also running successfully. For Intel, the licence issue was cleared after using -j1, but there are a few issues with other libraries, so I have stopped exploring this route right now, maybe I'll try it later. Thank you for all the help. Regards Shamim Haque Senior Research Fellow (SRF) Department of Physics IISER Bhopal ? On Tue, Jun 6, 2023 at 6:56?PM Roland Haas wrote: > Hello Shamim Haque, > > "icpc unable to checkout FLEXlm license" is caused by the compiler > being unable to contact the license server. This is unrelated to the > C++ compilation issue really. > > To be sure: > > * the only thing you did was add > "-gxx-name=/opt/ohpc/pub/compiler/gcc/8.3.0/bin/g++" was it? In > particular you did not add / remove any "module load" commands (e.g., > added a module load gcc after an existing module load intel)? I am > asking b/c module load gcc may unload the intel compiler module and > unset the environment variable that the Intel compiler uses to > determine how to contact the license server. > > * you may also try to compile with just "make -j1" or > "simfactory/bin/sim build -j1" to use only a single compiler instance > in case the number of compiler licenses available is limited or the > license server overwhelmed by too many concurrent compilations (by > you or other users of the cluster) > > The issue with fftw3 and using gcc is that fftw3 (you are using a > system provided fftw3? Either from a module or otherwise provided by > the admin) has been compiled with the Intel compiler and thus cannot be > used with the GNU compiler. You an set "FFTW_DIR = BUILD" to make the > Einstein Toolkit build a copy from scratch using an included tar file. > Not that this may be the case for other libraries as well. In > particular you usually need to ensure that the MPI library used was > compiled with the same compiler you use to compile the simulation code. > > Yours, > Roland > > > Hello Roland, > > > > Thank you for the reply. > > > > I pointed -gxx-name to gcc/8.3.0, which resulted in an error related to > the > > "icpc unable to checkout FLEXlm license". I also tried to use a > > different intel compiler (and point -gxx-name to gcc/11.1.0) to check if > > the error can go away, but it remains the same. I communicated this to > the > > HPC admin and awaiting their response; please let me know if I should fix > > something in my configuration. I have attached the out file > > (intel_compile.out), config log (intel_config.log), > > and optionlist (intel.cfg) for reference. > > > > Meanwhile, I also attempted to configure ET with gcc/11.1.0, which > resulted > > in some issues regarding the FFTW3. I have contacted the admins regarding > > this, but please let me know if it is something to be fixed within the > > optionlist. I have attached the out file (gcc_compile.out), config log > > (gcc_config.log), and optionlist (gcc.cfg) for reference. > > > > The error in the output file says the following: > > > > > > > > > > > > > > > > > > > > > > > > *Formaline: is still running, e.g. the final link stage.)/usr/bin/ld: > > warning: libimf.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, > > not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: > > libsvml.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not > found > > (try using -rpath or -rpath-link)/usr/bin/ld: warning: libirng.so, needed > > by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using > -rpath > > or -rpath-link)/usr/bin/ld: warning: libintlc.so.5, needed by > > /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath > or > > -rpath-link)/usr/bin/ld: warning: libgfortran.so.3, needed by > > /lib/../lib64/liblapack.so, may conflict with > > libgfortran.so.5/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > reference to > > `__intel_cpu_features_init_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > > undefined reference to > > `__svml_sincos2'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > reference to > > `_intel_fast_memset'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > reference to > > > `__intel_cpu_feature_indicator_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > > undefined reference to > > `_intel_fast_memmove'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > undefined > > reference to `_intel_fast_memcpy'* > > > > Thanks in advance for all the help. > > > > Regards > > Shamim > > > > ? > > > > On Fri, Jun 2, 2023 at 1:31?AM Roland Haas wrote: > > > > > Hello Shamim Haque, > > > > > > This most likely is caused by not having a full C++11 software stack > > > enabled. > > > > > > The Intel compiler always uses the C++ STL library of the GNU g++ > > > compiler and must be matched up with a compatible g++ compiler's STL > > > to correctly support C++11. > > > > > > This ticket (the last of many, hopefully) contains details: > > > > > > > > > > https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/issues/2663/add-an-notice-about-minimum-intel-gcc__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlRGRwlyl$ > > > > > > Basically, the fix is to find a gcc compiler at least version 6 on your > > > cluster, then pass the option > > > > > > -gxx-name PATH-TO-GXX > > > > > > to icpc in the CXXFLAGS option list variable (see eg > > > > > > > https://urldefense.com/v3/__https://bitbucket.org/simfactory/simfactory2/src/7e2592d264b1afe9ca2cb54bcab8dd9e4165cf98/mdb/optionlists/smic-openmpi.cfg?at=master*smic-openmpi.cfg-15:16__;Iw!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlSBbGTFf$ > > > ). > > > > > > If there is no gcc compiler version 6 or newer installed in the system, > > > you will have to contact the administrators and ask them about it. > > > > > > The demo code in the description of > > > > https://urldefense.com/v3/__https://bitbucket.org/cactuscode/cactus/pull-requests/152__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlUmecdAQ$ > may help > > > demonstrate to the admins what the issue is. > > > > > > Yours, > > > Roland > > > > > > > Hello All, > > > > > > > > I have made little progress on compiling ETK on this cluster, as > the > > > admins > > > > allowed me to compile ETK on the login node. I am finding new errors > and > > > am > > > > unable to debug the issue. I am trying to configure ETK using > Intel19, > > > and > > > > the error says the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > */home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160): > > > > > error: more than one instance of overloaded function > "CarpetMask::isnan" > > > > matches the argument list: function "isnan(double)" > > > > function "std::isnan(double)" argument types are: > (CCTK_REAL8) > > > > assert(not isnan(phi)); ^compilation > aborted for > > > > > > > > /home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc > > > > > (code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: *** > > > [make.checked] > > > > Error 2make[1]: *** > > > > > > > > [/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a] > > > > > Error 2make: *** [sim] Error 2* > > > > > > > > I have attached the machinefile, optionlist, config log and full > output > > > > file for reference. I configured the scripts using the available > example > > > > optionlist and machinefile for supermuc_ng and stampede2. Please let > me > > > > know if more info is required, and thanks in advance for your help. > > > > > > > > Regards > > > > Shamim Haque > > > > Senior Research Fellow (SRF) > > > > Department of Physics > > > > IISER Bhopal > > > > > > > > ? > > > > > > > > On Fri, May 19, 2023 at 2:32?PM Shamim Haque 1910511 < > > > shamims at iiserb.ac.in> > > > > wrote: > > > > > > > > > Hello Roland, > > > > > > > > > > Thank you for the reply. Yes, I am trying to compile ETK in the > > > computing > > > > > node via sbatch queue. I am in touch with the system admin to > resolve > > > > > the issue with the compilers. > > > > > > > > > > Regards > > > > > Shamim Haque > > > > > Senior Research Fellow (SRF) > > > > > Department of Physics > > > > > IISER Bhopal > > > > > > > > > > ? > > > > > > > > > > On Thu, May 18, 2023 at 7:32?PM Roland Haas > > > wrote: > > > > > > > > > >> Hello Shamin, > > > > >> > > > > >> A missing math.h would point to a faulty compiler installation. > Since > > > > >> you are trying to compile (are you?) ona compute node in a > running > > > job, > > > > >> my guess would be that the cluster admins are not providing > compilers > > > > >> on the compute nodes but only on the login nodes. > > > > >> > > > > >> This would be something to contact your cluster's help desk about. > > > > >> > > > > >> Yours, > > > > >> Roland > > > > >> > > > > >> > Hello Roland, > > > > >> > > > > > >> > I did try that fix; the error remains the same. Actually, we > landed > > > on > > > > >> an > > > > >> > even more primitive problem, where Steve suggested a basic > test, > > > that > > > > >> is to > > > > >> > compile the following code: > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > *#include #include int main() { > printf("%f\n", > > > > >> > M_PI); return 0;}* > > > > >> > > > > > >> > This program compiles just fine in the login node but gives out > the > > > > >> > following error while compiling it via jobscript submission in > the > > > > >> sbatch > > > > >> > queue: > > > > >> > > > > > >> > prog.cpp:1:18: fatal error: math.h: No such file or directory > > > > >> > #include > > > > >> > ^ > > > > >> > compilation terminated. > > > > >> > > > > > >> > ----OR----- > > > > >> > > > > > >> > > > > > >> > > > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > > > > > >> > fatal error: features.h: No such file or directory > > > > >> > 39 | #include > > > > >> > | ^~~~~~~~~~~~ > > > > >> > compilation terminated. > > > > >> > > > > > >> > I have reported this problem to the HPC admin and await their > > > response. > > > > >> > > > > > >> > Regards > > > > >> > Shamim Haque > > > > >> > Senior Research Fellow (SRF) > > > > >> > Department of Physics > > > > >> > IISER Bhopal > > > > >> > > > > > >> > ? > > > > >> > > > > > >> > On Fri, May 12, 2023 at 12:20?AM Roland Haas < > rhaas at illinois.edu> > > > > >> wrote: > > > > >> > > > > > >> > > Hello all, > > > > >> > > > > > > >> > > Since the error is a missing M_PI and the configure script > already > > > > >> > > gives a suggested fix: > > > > >> > > > > > > >> > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > > > >> > > > > > > >> > > have you tried adding: > > > > >> > > > > > > >> > > CPPFLAGS=-D_XOPEN_SOURCE > > > > >> > > > > > > >> > > to your option list file before compiling? > > > > >> > > > > > > >> > > Yours, > > > > >> > > Roland > > > > >> > > > > > > >> > > > Thanks Steve. I have raised complaint to the admin about > this > > > issue. > > > > >> > > > > > > > >> > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > > > > >> sbrandt at cct.lsu.edu> > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > > I would talk to the system administrators about that. > There > > > is > > > > >> > > definitely > > > > >> > > > > something wrong with your compiler install if you can't > find > > > > >> math.h. > > > > >> > > > > > > > > >> > > > > --Steve > > > > >> > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > > > > >> > > > > > > > > >> > > > > Dear Steve, > > > > >> > > > > > > > > >> > > > > I just rechecked the cpp program to output M_PI in the > > > computing > > > > >> nodes, > > > > >> > > > > and that is not working right now, which was somehow > working > > > fine > > > > >> a few > > > > >> > > > > days ago. I also tried different gcc versions (4.8, > 10.2, > > > 11.1, > > > > >> > > gnu8/8.3) > > > > >> > > > > available in the cluster. All report similar issue: > > > > >> > > > > > > > > >> > > > > prog.cpp:1:18: fatal error: math.h: No such file or > directory > > > > >> > > > > #include > > > > >> > > > > ^ > > > > >> > > > > compilation terminated. > > > > >> > > > > > > > > >> > > > > ----OR----- > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > > > >> > > > > >> > > > > fatal error: features.h: No such file or directory > > > > >> > > > > 39 | #include > > > > >> > > > > | ^~~~~~~~~~~~ > > > > >> > > > > compilation terminated. > > > > >> > > > > > > > > >> > > > > The cpp code seems to work fine in login node, as it > gives > > > out > > > > >> the > > > > >> > > value > > > > >> > > > > of pi upon execution. I am contacting the hpc admin to > > > resolve > > > > >> this > > > > >> > > issue. > > > > >> > > > > > > > > >> > > > > I am attaching the test code (prog.cpp), the sbatch > > > jobscript > > > > >> (prog.sh) > > > > >> > > > > and the output file (slurn-117473.out), if you would like > to > > > have > > > > >> a > > > > >> > > look at > > > > >> > > > > it. > > > > >> > > > > > > > > >> > > > > Please let me know if I should try something different > to > > > get > > > > >> this > > > > >> > > working. > > > > >> > > > > > > > > >> > > > > Regards > > > > >> > > > > Shamim Haque > > > > >> > > > > Senior Research Fellow (SRF) > > > > >> > > > > Department of Physics > > > > >> > > > > IISER Bhopal > > > > >> > > > > > > > > >> > > > > ? > > > > >> > > > > > > > > >> > > > > On Mon, May 8, 2023 at 7:25?PM Shamim Haque 1910511 < > > > > >> > > shamims at iiserb.ac.in> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > >> Thanks Steve. I'll make the changes and try again. I'll > let > > > you > > > > >> know > > > > >> > > the > > > > >> > > > >> outcome. > > > > >> > > > >> > > > > >> > > > >> Regards > > > > >> > > > >> Shamim Haque > > > > >> > > > >> Senior Research Fellow (SRF) > > > > >> > > > >> Department of Physics > > > > >> > > > >> IISER Bhopal > > > > >> > > > >> > > > > >> > > > >> ? > > > > >> > > > >> > > > > >> > > > >> On Mon, May 8, 2023 at 7:21?PM Steven R. Brandt < > > > > >> sbrandt at cct.lsu.edu> > > > > >> > > > >> wrote: > > > > >> > > > >> > > > > >> > > > >>> That tends to make me think you aren't using the > compiler > > > you > > > > >> think > > > > >> > > > >>> you're using. You could put full paths in the .cfg > files > > > and > > > > >> > > recompile from > > > > >> > > > >>> scratch. > > > > >> > > > >>> > > > > >> > > > >>> i.e. > > > > >> > > > >>> > > > > >> > > > >>> CXX = /full/path/to/icc > > > > >> > > > >>> > > > > >> > > > >>> --Steve > > > > >> > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > > > > >> > > > >>> > > > > >> > > > >>> Yes, this gives the value of Pi successfully, before > > > starting > > > > >> the > > > > >> > > > >>> compilation: > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > > > > >> paramganga3 > > > > >> > > Using > > > > >> > > > >>> configuration: sim Reconfiguring sim Writing > configuration > > > to: > > > > >> > > > >>> > > > /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList > > > > >> Cactus - > > > > >> > > > >>> version: 4.12.0 Reconfiguring sim. * > > > > >> > > > >>> *.* > > > > >> > > > >>> *.* > > > > >> > > > >>> *.* > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> *checking for M_PI... no configure: error: M_PI not > > > defined. > > > > >> Try > > > > >> > > adding > > > > >> > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring > > > sim-config > > > > >> make: *** > > > > >> > > > >>> [sim-config] Error 2 * > > > > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> Shamim Haque > > > > >> > > > >>> Senior Research Fellow (SRF) > > > > >> > > > >>> Department of Physics > > > > >> > > > >>> IISER Bhopal > > > > >> > > > >>> > > > > >> > > > >>> ? > > > > >> > > > >>> > > > > >> > > > >>> On Thu, May 4, 2023 at 12:29?AM Steven Brandt < > > > > >> sbrandt at cct.lsu.edu> > > > > >> > > > >>> wrote: > > > > >> > > > >>> > > > > >> > > > >>>> So, sanity check, can you compile the following > program > > > > >> outside of > > > > >> > > > >>>> Cactus? > > > > >> > > > >>>> > > > > >> > > > >>>> #include > > > > >> > > > >>>> #include > > > > >> > > > >>>> > > > > >> > > > >>>> int main() { > > > > >> > > > >>>> printf("%f\n", M_PI); > > > > >> > > > >>>> return 0; > > > > >> > > > >>>> } > > > > >> > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > > > > >> > > > >>>> > > > > >> > > > >>>> Dear Steve, > > > > >> > > > >>>> > > > > >> > > > >>>> I tried gcc (10 and 11) as well and I am stuck with > the > > > > >> following > > > > >> > > error: > > > > >> > > > >>>> *configure: error: M_PI not defined. Try adding > > > > >> -D_XOPEN_SOURCE to > > > > >> > > > >>>> CPPFLAGS.* > > > > >> > > > >>>> > > > > >> > > > >>>> This also happened for Intel (2020 and 2019). I have > > > attached > > > > >> both > > > > >> > > the > > > > >> > > > >>>> output files here. > > > > >> > > > >>>> > > > > >> > > > >>>> Please ignore the out file in the first email, I have > > > > >> uploaded > > > > >> > > another > > > > >> > > > >>>> out file by mistake. The rest of the files are correct > in > > > the > > > > >> first > > > > >> > > mail. > > > > >> > > > >>>> > > > > >> > > > >>>> Regards > > > > >> > > > >>>> Shamim Haque > > > > >> > > > >>>> Senior Research Fellow (SRF) > > > > >> > > > >>>> Department of Physics > > > > >> > > > >>>> IISER Bhopal > > > > >> > > > >>>> > > > > >> > > > >>>> ? > > > > >> > > > >>>> > > > > >> > > > >>>> On Wed, May 3, 2023 at 8:04?PM Steven Brandt < > > > > >> sbrandt at cct.lsu.edu> > > > > >> > > > >>>> wrote: > > > > >> > > > >>>> > > > > >> > > > >>>>> I notice from the output that your compiler claims > > > "limits.h" > > > > >> is > > > > >> > > > >>>>> missing. Possibly there's something wrong with your > icc > > > > >> setup? > > > > >> > > Maybe you > > > > >> > > > >>>>> should try using gcc / gfortran instead. > > > > >> > > > >>>>> > > > > >> > > > >>>>> --Steve > > > > >> > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > > > > >> > > > >>>>> > > > > >> > > > >>>>> Dear Frank, > > > > >> > > > >>>>> > > > > >> > > > >>>>> I added the alias pattern for the computing nodes and > now > > > ETK > > > > >> can > > > > >> > > > >>>>> detect the cluster (checked by asking "whoami") > during > > > the > > > > >> batch > > > > >> > > queuing. > > > > >> > > > >>>>> I'll get in touch with the admins to include 'patch'. > > > > >> > > > >>>>> > > > > >> > > > >>>>> However, the problem still persists, having assured > that > > > the > > > > >> > > correct > > > > >> > > > >>>>> machine file and option list are being used during > the > > > > >> compilation. > > > > >> > > > >>>>> > > > > >> > > > >>>>> Regards > > > > >> > > > >>>>> Shamim Haque > > > > >> > > > >>>>> Senior Research Fellow (SRF) > > > > >> > > > >>>>> Department of Physics > > > > >> > > > >>>>> IISER Bhopal > > > > >> > > > >>>>> > > > > >> > > > >>>>> ? > > > > >> > > > >>>>> > > > > >> > > > >>>>> On Wed, May 3, 2023 at 3:49?PM Frank Loeffler < > > > > >> > > > >>>>> frank.loeffler at uni-jena.de> wrote: > > > > >> > > > >>>>> > > > > >> > > > >>>>>> Hi Shamim > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim > Haque > > > > >> 1910511 > > > > >> > > wrote: > > > > >> > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT > > > Roorkee. > > > > >> The > > > > >> > > > >>>>>> >compilation > > > > >> > > > >>>>>> >stops at a very initial step and gives out the > error: > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> >*checking whether the Fortran compiler (gfortran > ) > > > > >> works... > > > > >> > > > >>>>>> yeschecking > > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > > > > >> > > cross-compiler... > > > > >> > > > >>>>>> nochecking > > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) supports > > > TYPE(*) > > > > >> for > > > > >> > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > > > > >> preprocessor... > > > > >> > > > >>>>>> cppchecking > > > > >> > > > >>>>>> >for ANSI C header files... nochecking for C99 > > > features... > > > > >> > > > >>>>>> yeschecking for > > > > >> > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try > > > adding > > > > >> > > > >>>>>> -D_XOPEN_SOURCE to > > > > >> > > > >>>>>> >CPPFLAGS.* > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but > it > > > does > > > > >> not > > > > >> > > help. > > > > >> > > > >>>>>> This > > > > >> > > > >>>>>> >error is consistent if we set up using Intel > Compiler > > > (2020 > > > > >> or > > > > >> > > > >>>>>> 2019), or > > > > >> > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error > > > file, > > > > >> > > machine > > > > >> > > > >>>>>> script, > > > > >> > > > >>>>>> >and option lists for both cases (Intel and gcc) > for > > > any > > > > >> > > reference. > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> >Secondly, we are not allowed to compile ETK on > login > > > nodes. > > > > >> So > > > > >> > > we are > > > > >> > > > >>>>>> >queuing a job script, which upon execution, enters > the > > > > >> Cactus > > > > >> > > > >>>>>> directory, > > > > >> > > > >>>>>> >does "setup-silent", and then executes the build > > > command. > > > > >> Is > > > > >> > > this > > > > >> > > > >>>>>> the right > > > > >> > > > >>>>>> >way to compile ETK using the batch script? Or > should we > > > do > > > > >> > > something > > > > >> > > > >>>>>> >different? I have attached the shell script > > > (compile.sh) > > > > >> used > > > > >> > > for > > > > >> > > > >>>>>> sbatch > > > > >> > > > >>>>>> >queue for reference. > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> I might be wrong, but setup-silent will not use > your > > > machine > > > > >> file > > > > >> > > > >>>>>> which > > > > >> > > > >>>>>> contains all the compiler flags, but will setup > > > something > > > > >> "from > > > > >> > > > >>>>>> scratch" > > > > >> > > > >>>>>> instead. According to the logs, this will be, e.g., > in > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> > > > > >> > > > > > > >> > > > > /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini > > > > > >> > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> My guess would be that this misses the compiler > flags > > > > >> necessary to > > > > >> > > > >>>>>> have > > > > >> > > > >>>>>> M_PI defined. The best way to get this working might > be > > > to > > > > >> let > > > > >> > > > >>>>>> simfactory also detect the cluster configuration on > the > > > > >> compute > > > > >> > > nodes > > > > >> > > > >>>>>> (where you compile), such that setup-silent is not > > > needed, > > > > >> but > > > > >> > > > >>>>>> instead > > > > >> > > > >>>>>> the correct cluster configuration is found and used > > > > >> > > automatically. In > > > > >> > > > >>>>>> order to do that, look at examples of 'aliaspattern' > in > > > the > > > > >> > > > >>>>>> mdb/machines > > > > >> > > > >>>>>> directory. An alternative would be to tell > simfactory > > > > >> specifically > > > > >> > > > >>>>>> that > > > > >> > > > >>>>>> you want to use your machine configuration file. > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> As a side-note: I noticed the command 'patch' is > > > missing > > > > >> too. > > > > >> > > This is > > > > >> > > > >>>>>> a > > > > >> > > > >>>>>> tool so common that it should be installed > everywhere. > > > Your > > > > >> admin > > > > >> > > is > > > > >> > > > >>>>>> probably the best person to ask for advise here. It > > > should > > > > >> not be > > > > >> > > too > > > > >> > > > >>>>>> hard to install yourself, but that should not be > > > necessary > > > > >> either, > > > > >> > > > >>>>>> especially when you are told to compile on compute > nodes. > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> Frank > > > > >> > > > >>>>>> > > > > >> > > > >>>>>> > > > > >> > > > >>>>> _______________________________________________ > > > > >> > > > >>>>> Users mailing listUsers at einsteintoolkit.orghttp:// > > > > >> > > lists.einsteintoolkit.org/mailman/listinfo/users > > > > >> > > > >>>>> > > > > >> > > > >>>>> _______________________________________________ > > > > >> > > > >>>>> Users mailing list > > > > >> > > > >>>>> Users at einsteintoolkit.org > > > > >> > > > >>>>> > > > > >> > > > > > > >> > > > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4xAKSrBYUo_v8jHEbenNg6aQ99Oq9QyeXsjq766lVDVwHAhFJODYjSU6XryKp264bQqDmBcTIA4Srkc4wOCn$ > > > > > >> > > > > >> > > > >>>>> > > > > >> > > > >>>> > > > > >> > > > > > > >> > > > > > > >> > > Yours, > > > > >> > > Roland > > > > >> > > > > > > >> > > -- > > > > >> > > My email is as private as my paper mail. I therefore support > > > > >> encrypting > > > > >> > > and signing email messages. Get my PGP key from > > > > >> > > > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xbeLFN1po4t2MOETlRoEnO3fr4pBXNdsn_VXBbuywp1tgk0zKYy_oBKphC1Ny-KY0Vh9$ > > > > > >> . > > > > >> > > > > > > >> > > > > >> -- > > > > >> My email is as private as my paper mail. I therefore support > > > encrypting > > > > >> and signing email messages. Get my PGP key from > > > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6bdaoR4iJf6WXHxF1UsHWThqJwM8QxTwHKDWcVgn0kphw4Lmahcfk5a8gMIPSpV8TMD$ > > > . > > > > >> > > > > > > > > > > > > > > -- > > > My email is as private as my paper mail. I therefore support encrypting > > > and signing email messages. Get my PGP key from > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlS57nxFF$ > . > > > > > > Yours, > Roland > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhaas at illinois.edu Mon Jun 19 11:13:10 2023 From: rhaas at illinois.edu (Roland Haas) Date: Mon, 19 Jun 2023 11:13:10 -0500 Subject: [Users] Compilation Error in cluster ParamGanga In-Reply-To: References: <4449ed77-ac78-8774-896b-f86737ea9bf0@cct.lsu.edu> <20230511135018.1d31cfb1@ekohaes8> <20230518090147.4248b227@ekohaes8> <20230601150128.22b31a04@ekohaes8> <20230606082543.3131959b@ekohaes8> Message-ID: <20230619111310.3ea06bed@fdea4908.spdns.org> Hello Shamim Haque, Great. Glad that we could help you. Yours, Roland > Dear Roland, Steve and Frank, > > I am able to compile ETK properly with GNU compiler 8.3.0. The simulations > are also running successfully. > > For Intel, the licence issue was cleared after using -j1, but there are a > few issues with other libraries, so I have stopped exploring this > route right now, maybe I'll try it later. > > Thank you for all the help. > > Regards > Shamim Haque > Senior Research Fellow (SRF) > Department of Physics > IISER Bhopal > > ? > > On Tue, Jun 6, 2023 at 6:56?PM Roland Haas wrote: > > > Hello Shamim Haque, > > > > "icpc unable to checkout FLEXlm license" is caused by the compiler > > being unable to contact the license server. This is unrelated to the > > C++ compilation issue really. > > > > To be sure: > > > > * the only thing you did was add > > "-gxx-name=/opt/ohpc/pub/compiler/gcc/8.3.0/bin/g++" was it? In > > particular you did not add / remove any "module load" commands (e.g., > > added a module load gcc after an existing module load intel)? I am > > asking b/c module load gcc may unload the intel compiler module and > > unset the environment variable that the Intel compiler uses to > > determine how to contact the license server. > > > > * you may also try to compile with just "make -j1" or > > "simfactory/bin/sim build -j1" to use only a single compiler instance > > in case the number of compiler licenses available is limited or the > > license server overwhelmed by too many concurrent compilations (by > > you or other users of the cluster) > > > > The issue with fftw3 and using gcc is that fftw3 (you are using a > > system provided fftw3? Either from a module or otherwise provided by > > the admin) has been compiled with the Intel compiler and thus cannot be > > used with the GNU compiler. You an set "FFTW_DIR = BUILD" to make the > > Einstein Toolkit build a copy from scratch using an included tar file. > > Not that this may be the case for other libraries as well. In > > particular you usually need to ensure that the MPI library used was > > compiled with the same compiler you use to compile the simulation code. > > > > Yours, > > Roland > > > > > Hello Roland, > > > > > > Thank you for the reply. > > > > > > I pointed -gxx-name to gcc/8.3.0, which resulted in an error related to > > the > > > "icpc unable to checkout FLEXlm license". I also tried to use a > > > different intel compiler (and point -gxx-name to gcc/11.1.0) to check if > > > the error can go away, but it remains the same. I communicated this to > > the > > > HPC admin and awaiting their response; please let me know if I should fix > > > something in my configuration. I have attached the out file > > > (intel_compile.out), config log (intel_config.log), > > > and optionlist (intel.cfg) for reference. > > > > > > Meanwhile, I also attempted to configure ET with gcc/11.1.0, which > > resulted > > > in some issues regarding the FFTW3. I have contacted the admins regarding > > > this, but please let me know if it is something to be fixed within the > > > optionlist. I have attached the out file (gcc_compile.out), config log > > > (gcc_config.log), and optionlist (gcc.cfg) for reference. > > > > > > The error in the output file says the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Formaline: is still running, e.g. the final link stage.)/usr/bin/ld: > > > warning: libimf.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, > > > not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: > > > libsvml.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not > > found > > > (try using -rpath or -rpath-link)/usr/bin/ld: warning: libirng.so, needed > > > by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using > > -rpath > > > or -rpath-link)/usr/bin/ld: warning: libintlc.so.5, needed by > > > /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath > > or > > > -rpath-link)/usr/bin/ld: warning: libgfortran.so.3, needed by > > > /lib/../lib64/liblapack.so, may conflict with > > > libgfortran.so.5/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > > reference to > > > `__intel_cpu_features_init_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > > > undefined reference to > > > `__svml_sincos2'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > > reference to > > > `_intel_fast_memset'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined > > > reference to > > > > > `__intel_cpu_feature_indicator_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > > > undefined reference to > > > `_intel_fast_memmove'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: > > undefined > > > reference to `_intel_fast_memcpy'* > > > > > > Thanks in advance for all the help. > > > > > > Regards > > > Shamim > > > > > > ? > > > > > > On Fri, Jun 2, 2023 at 1:31?AM Roland Haas wrote: > > > > > > > Hello Shamim Haque, > > > > > > > > This most likely is caused by not having a full C++11 software stack > > > > enabled. > > > > > > > > The Intel compiler always uses the C++ STL library of the GNU g++ > > > > compiler and must be matched up with a compatible g++ compiler's STL > > > > to correctly support C++11. > > > > > > > > This ticket (the last of many, hopefully) contains details: > > > > > > > > > > > > > > https://urldefense.com/v3/__https://bitbucket.org/einsteintoolkit/tickets/issues/2663/add-an-notice-about-minimum-intel-gcc__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlRGRwlyl$ > > > > > > > > Basically, the fix is to find a gcc compiler at least version 6 on your > > > > cluster, then pass the option > > > > > > > > -gxx-name PATH-TO-GXX > > > > > > > > to icpc in the CXXFLAGS option list variable (see eg > > > > > > > > > > https://urldefense.com/v3/__https://bitbucket.org/simfactory/simfactory2/src/7e2592d264b1afe9ca2cb54bcab8dd9e4165cf98/mdb/optionlists/smic-openmpi.cfg?at=master*smic-openmpi.cfg-15:16__;Iw!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlSBbGTFf$ > > > > ). > > > > > > > > If there is no gcc compiler version 6 or newer installed in the system, > > > > you will have to contact the administrators and ask them about it. > > > > > > > > The demo code in the description of > > > > > > https://urldefense.com/v3/__https://bitbucket.org/cactuscode/cactus/pull-requests/152__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlUmecdAQ$ > > may help > > > > demonstrate to the admins what the issue is. > > > > > > > > Yours, > > > > Roland > > > > > > > > > Hello All, > > > > > > > > > > I have made little progress on compiling ETK on this cluster, as > > the > > > > admins > > > > > allowed me to compile ETK on the login node. I am finding new errors > > and > > > > am > > > > > unable to debug the issue. I am trying to configure ETK using > > Intel19, > > > > and > > > > > the error says the following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > */home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160): > > > > > > > error: more than one instance of overloaded function > > "CarpetMask::isnan" > > > > > matches the argument list: function "isnan(double)" > > > > > function "std::isnan(double)" argument types are: > > (CCTK_REAL8) > > > > > assert(not isnan(phi)); ^compilation > > aborted for > > > > > > > > > > > /home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc > > > > > > > (code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: *** > > > > [make.checked] > > > > > Error 2make[1]: *** > > > > > > > > > > > [/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a] > > > > > > > Error 2make: *** [sim] Error 2* > > > > > > > > > > I have attached the machinefile, optionlist, config log and full > > output > > > > > file for reference. I configured the scripts using the available > > example > > > > > optionlist and machinefile for supermuc_ng and stampede2. Please let > > me > > > > > know if more info is required, and thanks in advance for your help. > > > > > > > > > > Regards > > > > > Shamim Haque > > > > > Senior Research Fellow (SRF) > > > > > Department of Physics > > > > > IISER Bhopal > > > > > > > > > > ? > > > > > > > > > > On Fri, May 19, 2023 at 2:32?PM Shamim Haque 1910511 < > > > > shamims at iiserb.ac.in> > > > > > wrote: > > > > > > > > > > > Hello Roland, > > > > > > > > > > > > Thank you for the reply. Yes, I am trying to compile ETK in the > > > > computing > > > > > > node via sbatch queue. I am in touch with the system admin to > > resolve > > > > > > the issue with the compilers. > > > > > > > > > > > > Regards > > > > > > Shamim Haque > > > > > > Senior Research Fellow (SRF) > > > > > > Department of Physics > > > > > > IISER Bhopal > > > > > > > > > > > > ? > > > > > > > > > > > > On Thu, May 18, 2023 at 7:32?PM Roland Haas > > > > wrote: > > > > > > > > > > > >> Hello Shamin, > > > > > >> > > > > > >> A missing math.h would point to a faulty compiler installation. > > Since > > > > > >> you are trying to compile (are you?) ona compute node in a > > running > > > > job, > > > > > >> my guess would be that the cluster admins are not providing > > compilers > > > > > >> on the compute nodes but only on the login nodes. > > > > > >> > > > > > >> This would be something to contact your cluster's help desk about. > > > > > >> > > > > > >> Yours, > > > > > >> Roland > > > > > >> > > > > > >> > Hello Roland, > > > > > >> > > > > > > >> > I did try that fix; the error remains the same. Actually, we > > landed > > > > on > > > > > >> an > > > > > >> > even more primitive problem, where Steve suggested a basic > > test, > > > > that > > > > > >> is to > > > > > >> > compile the following code: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > *#include #include int main() { > > printf("%f\n", > > > > > >> > M_PI); return 0;}* > > > > > >> > > > > > > >> > This program compiles just fine in the login node but gives out > > the > > > > > >> > following error while compiling it via jobscript submission in > > the > > > > > >> sbatch > > > > > >> > queue: > > > > > >> > > > > > > >> > prog.cpp:1:18: fatal error: math.h: No such file or directory > > > > > >> > #include > > > > > >> > ^ > > > > > >> > compilation terminated. > > > > > >> > > > > > > >> > ----OR----- > > > > > >> > > > > > > >> > > > > > > >> > > > > > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > > > > > > > >> > fatal error: features.h: No such file or directory > > > > > >> > 39 | #include > > > > > >> > | ^~~~~~~~~~~~ > > > > > >> > compilation terminated. > > > > > >> > > > > > > >> > I have reported this problem to the HPC admin and await their > > > > response. > > > > > >> > > > > > > >> > Regards > > > > > >> > Shamim Haque > > > > > >> > Senior Research Fellow (SRF) > > > > > >> > Department of Physics > > > > > >> > IISER Bhopal > > > > > >> > > > > > > >> > ? > > > > > >> > > > > > > >> > On Fri, May 12, 2023 at 12:20?AM Roland Haas < > > rhaas at illinois.edu> > > > > > >> wrote: > > > > > >> > > > > > > >> > > Hello all, > > > > > >> > > > > > > > >> > > Since the error is a missing M_PI and the configure script > > already > > > > > >> > > gives a suggested fix: > > > > > >> > > > > > > > >> > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > > > > >> > > > > > > > >> > > have you tried adding: > > > > > >> > > > > > > > >> > > CPPFLAGS=-D_XOPEN_SOURCE > > > > > >> > > > > > > > >> > > to your option list file before compiling? > > > > > >> > > > > > > > >> > > Yours, > > > > > >> > > Roland > > > > > >> > > > > > > > >> > > > Thanks Steve. I have raised complaint to the admin about > > this > > > > issue. > > > > > >> > > > > > > > > >> > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > > > > > >> sbrandt at cct.lsu.edu> > > > > > >> > > wrote: > > > > > >> > > > > > > > > >> > > > > I would talk to the system administrators about that. > > There > > > > is > > > > > >> > > definitely > > > > > >> > > > > something wrong with your compiler install if you can't > > find > > > > > >> math.h. > > > > > >> > > > > > > > > > >> > > > > --Steve > > > > > >> > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > > > > > >> > > > > > > > > > >> > > > > Dear Steve, > > > > > >> > > > > > > > > > >> > > > > I just rechecked the cpp program to output M_PI in the > > > > computing > > > > > >> nodes, > > > > > >> > > > > and that is not working right now, which was somehow > > working > > > > fine > > > > > >> a few > > > > > >> > > > > days ago. I also tried different gcc versions (4.8, > > 10.2, > > > > 11.1, > > > > > >> > > gnu8/8.3) > > > > > >> > > > > available in the cluster. All report similar issue: > > > > > >> > > > > > > > > > >> > > > > prog.cpp:1:18: fatal error: math.h: No such file or > > directory > > > > > >> > > > > #include > > > > > >> > > > > ^ > > > > > >> > > > > compilation terminated. > > > > > >> > > > > > > > > > >> > > > > ----OR----- > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > > > > > >> > > > > > >> > > > > fatal error: features.h: No such file or directory > > > > > >> > > > > 39 | #include > > > > > >> > > > > | ^~~~~~~~~~~~ > > > > > >> > > > > compilation terminated. > > > > > >> > > > > > > > > > >> > > > > The cpp code seems to work fine in login node, as it > > gives > > > > out > > > > > >> the > > > > > >> > > value > > > > > >> > > > > of pi upon execution. I am contacting the hpc admin to > > > > resolve > > > > > >> this > > > > > >> > > issue. > > > > > >> > > > > > > > > > >> > > > > I am attaching the test code (prog.cpp), the sbatch > > > > jobscript > > > > > >> (prog.sh) > > > > > >> > > > > and the output file (slurn-117473.out), if you would like > > to > > > > have > > > > > >> a > > > > > >> > > look at > > > > > >> > > > > it. > > > > > >> > > > > > > > > > >> > > > > Please let me know if I should try something different > > to > > > > get > > > > > >> this > > > > > >> > > working. > > > > > >> > > > > > > > > > >> > > > > Regards > > > > > >> > > > > Shamim Haque > > > > > >> > > > > Senior Research Fellow (SRF) > > > > > >> > > > > Department of Physics > > > > > >> > > > > IISER Bhopal > > > > > >> > > > > > > > > > >> > > > > ? > > > > > >> > > > > > > > > > >> > > > > On Mon, May 8, 2023 at 7:25?PM Shamim Haque 1910511 < > > > > > >> > > shamims at iiserb.ac.in> > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> > > > >> Thanks Steve. I'll make the changes and try again. I'll > > let > > > > you > > > > > >> know > > > > > >> > > the > > > > > >> > > > >> outcome. > > > > > >> > > > >> > > > > > >> > > > >> Regards > > > > > >> > > > >> Shamim Haque > > > > > >> > > > >> Senior Research Fellow (SRF) > > > > > >> > > > >> Department of Physics > > > > > >> > > > >> IISER Bhopal > > > > > >> > > > >> > > > > > >> > > > >> ? > > > > > >> > > > >> > > > > > >> > > > >> On Mon, May 8, 2023 at 7:21?PM Steven R. Brandt < > > > > > >> sbrandt at cct.lsu.edu> > > > > > >> > > > >> wrote: > > > > > >> > > > >> > > > > > >> > > > >>> That tends to make me think you aren't using the > > compiler > > > > you > > > > > >> think > > > > > >> > > > >>> you're using. You could put full paths in the .cfg > > files > > > > and > > > > > >> > > recompile from > > > > > >> > > > >>> scratch. > > > > > >> > > > >>> > > > > > >> > > > >>> i.e. > > > > > >> > > > >>> > > > > > >> > > > >>> CXX = /full/path/to/icc > > > > > >> > > > >>> > > > > > >> > > > >>> --Steve > > > > > >> > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > > > > > >> > > > >>> > > > > > >> > > > >>> Yes, this gives the value of Pi successfully, before > > > > starting > > > > > >> the > > > > > >> > > > >>> compilation: > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > > > > > >> paramganga3 > > > > > >> > > Using > > > > > >> > > > >>> configuration: sim Reconfiguring sim Writing > > configuration > > > > to: > > > > > >> > > > >>> > > > > /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList > > > > > >> Cactus - > > > > > >> > > > >>> version: 4.12.0 Reconfiguring sim. * > > > > > >> > > > >>> *.* > > > > > >> > > > >>> *.* > > > > > >> > > > >>> *.* > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> *checking for M_PI... no configure: error: M_PI not > > > > defined. > > > > > >> Try > > > > > >> > > adding > > > > > >> > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring > > > > sim-config > > > > > >> make: *** > > > > > >> > > > >>> [sim-config] Error 2 * > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> Shamim Haque > > > > > >> > > > >>> Senior Research Fellow (SRF) > > > > > >> > > > >>> Department of Physics > > > > > >> > > > >>> IISER Bhopal > > > > > >> > > > >>> > > > > > >> > > > >>> ? > > > > > >> > > > >>> > > > > > >> > > > >>> On Thu, May 4, 2023 at 12:29?AM Steven Brandt < > > > > > >> sbrandt at cct.lsu.edu> > > > > > >> > > > >>> wrote: > > > > > >> > > > >>> > > > > > >> > > > >>>> So, sanity check, can you compile the following > > program > > > > > >> outside of > > > > > >> > > > >>>> Cactus? > > > > > >> > > > >>>> > > > > > >> > > > >>>> #include > > > > > >> > > > >>>> #include > > > > > >> > > > >>>> > > > > > >> > > > >>>> int main() { > > > > > >> > > > >>>> printf("%f\n", M_PI); > > > > > >> > > > >>>> return 0; > > > > > >> > > > >>>> } > > > > > >> > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > > > > > >> > > > >>>> > > > > > >> > > > >>>> Dear Steve, > > > > > >> > > > >>>> > > > > > >> > > > >>>> I tried gcc (10 and 11) as well and I am stuck with > > the > > > > > >> following > > > > > >> > > error: > > > > > >> > > > >>>> *configure: error: M_PI not defined. Try adding > > > > > >> -D_XOPEN_SOURCE to > > > > > >> > > > >>>> CPPFLAGS.* > > > > > >> > > > >>>> > > > > > >> > > > >>>> This also happened for Intel (2020 and 2019). I have > > > > attached > > > > > >> both > > > > > >> > > the > > > > > >> > > > >>>> output files here. > > > > > >> > > > >>>> > > > > > >> > > > >>>> Please ignore the out file in the first email, I have > > > > > >> uploaded > > > > > >> > > another > > > > > >> > > > >>>> out file by mistake. The rest of the files are correct > > in > > > > the > > > > > >> first > > > > > >> > > mail. > > > > > >> > > > >>>> > > > > > >> > > > >>>> Regards > > > > > >> > > > >>>> Shamim Haque > > > > > >> > > > >>>> Senior Research Fellow (SRF) > > > > > >> > > > >>>> Department of Physics > > > > > >> > > > >>>> IISER Bhopal > > > > > >> > > > >>>> > > > > > >> > > > >>>> ? > > > > > >> > > > >>>> > > > > > >> > > > >>>> On Wed, May 3, 2023 at 8:04?PM Steven Brandt < > > > > > >> sbrandt at cct.lsu.edu> > > > > > >> > > > >>>> wrote: > > > > > >> > > > >>>> > > > > > >> > > > >>>>> I notice from the output that your compiler claims > > > > "limits.h" > > > > > >> is > > > > > >> > > > >>>>> missing. Possibly there's something wrong with your > > icc > > > > > >> setup? > > > > > >> > > Maybe you > > > > > >> > > > >>>>> should try using gcc / gfortran instead. > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> --Steve > > > > > >> > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> Dear Frank, > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> I added the alias pattern for the computing nodes and > > now > > > > ETK > > > > > >> can > > > > > >> > > > >>>>> detect the cluster (checked by asking "whoami") > > during > > > > the > > > > > >> batch > > > > > >> > > queuing. > > > > > >> > > > >>>>> I'll get in touch with the admins to include 'patch'. > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> However, the problem still persists, having assured > > that > > > > the > > > > > >> > > correct > > > > > >> > > > >>>>> machine file and option list are being used during > > the > > > > > >> compilation. > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> Regards > > > > > >> > > > >>>>> Shamim Haque > > > > > >> > > > >>>>> Senior Research Fellow (SRF) > > > > > >> > > > >>>>> Department of Physics > > > > > >> > > > >>>>> IISER Bhopal > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> ? > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> On Wed, May 3, 2023 at 3:49?PM Frank Loeffler < > > > > > >> > > > >>>>> frank.loeffler at uni-jena.de> wrote: > > > > > >> > > > >>>>> > > > > > >> > > > >>>>>> Hi Shamim > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim > > Haque > > > > > >> 1910511 > > > > > >> > > wrote: > > > > > >> > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT > > > > Roorkee. > > > > > >> The > > > > > >> > > > >>>>>> >compilation > > > > > >> > > > >>>>>> >stops at a very initial step and gives out the > > error: > > > > > >> > > > >>>>>> > > > > > > >> > > > >>>>>> >*checking whether the Fortran compiler (gfortran > > ) > > > > > >> works... > > > > > >> > > > >>>>>> yeschecking > > > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > > > > > >> > > cross-compiler... > > > > > >> > > > >>>>>> nochecking > > > > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) supports > > > > TYPE(*) > > > > > >> for > > > > > >> > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > > > > > >> preprocessor... > > > > > >> > > > >>>>>> cppchecking > > > > > >> > > > >>>>>> >for ANSI C header files... nochecking for C99 > > > > features... > > > > > >> > > > >>>>>> yeschecking for > > > > > >> > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try > > > > adding > > > > > >> > > > >>>>>> -D_XOPEN_SOURCE to > > > > > >> > > > >>>>>> >CPPFLAGS.* > > > > > >> > > > >>>>>> > > > > > > >> > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but > > it > > > > does > > > > > >> not > > > > > >> > > help. > > > > > >> > > > >>>>>> This > > > > > >> > > > >>>>>> >error is consistent if we set up using Intel > > Compiler > > > > (2020 > > > > > >> or > > > > > >> > > > >>>>>> 2019), or > > > > > >> > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error > > > > file, > > > > > >> > > machine > > > > > >> > > > >>>>>> script, > > > > > >> > > > >>>>>> >and option lists for both cases (Intel and gcc) > > for > > > > any > > > > > >> > > reference. > > > > > >> > > > >>>>>> > > > > > > >> > > > >>>>>> >Secondly, we are not allowed to compile ETK on > > login > > > > nodes. > > > > > >> So > > > > > >> > > we are > > > > > >> > > > >>>>>> >queuing a job script, which upon execution, enters > > the > > > > > >> Cactus > > > > > >> > > > >>>>>> directory, > > > > > >> > > > >>>>>> >does "setup-silent", and then executes the build > > > > command. > > > > > >> Is > > > > > >> > > this > > > > > >> > > > >>>>>> the right > > > > > >> > > > >>>>>> >way to compile ETK using the batch script? Or > > should we > > > > do > > > > > >> > > something > > > > > >> > > > >>>>>> >different? I have attached the shell script > > > > (compile.sh) > > > > > >> used > > > > > >> > > for > > > > > >> > > > >>>>>> sbatch > > > > > >> > > > >>>>>> >queue for reference. > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> I might be wrong, but setup-silent will not use > > your > > > > machine > > > > > >> file > > > > > >> > > > >>>>>> which > > > > > >> > > > >>>>>> contains all the compiler flags, but will setup > > > > something > > > > > >> "from > > > > > >> > > > >>>>>> scratch" > > > > > >> > > > >>>>>> instead. According to the logs, this will be, e.g., > > in > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > > > > > >> > > > > > > /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini > > > > > > > >> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> My guess would be that this misses the compiler > > flags > > > > > >> necessary to > > > > > >> > > > >>>>>> have > > > > > >> > > > >>>>>> M_PI defined. The best way to get this working might > > be > > > > to > > > > > >> let > > > > > >> > > > >>>>>> simfactory also detect the cluster configuration on > > the > > > > > >> compute > > > > > >> > > nodes > > > > > >> > > > >>>>>> (where you compile), such that setup-silent is not > > > > needed, > > > > > >> but > > > > > >> > > > >>>>>> instead > > > > > >> > > > >>>>>> the correct cluster configuration is found and used > > > > > >> > > automatically. In > > > > > >> > > > >>>>>> order to do that, look at examples of 'aliaspattern' > > in > > > > the > > > > > >> > > > >>>>>> mdb/machines > > > > > >> > > > >>>>>> directory. An alternative would be to tell > > simfactory > > > > > >> specifically > > > > > >> > > > >>>>>> that > > > > > >> > > > >>>>>> you want to use your machine configuration file. > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> As a side-note: I noticed the command 'patch' is > > > > missing > > > > > >> too. > > > > > >> > > This is > > > > > >> > > > >>>>>> a > > > > > >> > > > >>>>>> tool so common that it should be installed > > everywhere. > > > > Your > > > > > >> admin > > > > > >> > > is > > > > > >> > > > >>>>>> probably the best person to ask for advise here. It > > > > should > > > > > >> not be > > > > > >> > > too > > > > > >> > > > >>>>>> hard to install yourself, but that should not be > > > > necessary > > > > > >> either, > > > > > >> > > > >>>>>> especially when you are told to compile on compute > > nodes. > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> Frank > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>>> > > > > > >> > > > >>>>> _______________________________________________ > > > > > >> > > > >>>>> Users mailing listUsers at einsteintoolkit.orghttp:// > > > > > >> > > lists.einsteintoolkit.org/mailman/listinfo/users > > > > > >> > > > >>>>> > > > > > >> > > > >>>>> _______________________________________________ > > > > > >> > > > >>>>> Users mailing list > > > > > >> > > > >>>>> Users at einsteintoolkit.org > > > > > >> > > > >>>>> > > > > > >> > > > > > > > >> > > > > > > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4xAKSrBYUo_v8jHEbenNg6aQ99Oq9QyeXsjq766lVDVwHAhFJODYjSU6XryKp264bQqDmBcTIA4Srkc4wOCn$ > > > > > > > >> > > > > > >> > > > >>>>> > > > > > >> > > > >>>> > > > > > >> > > > > > > > >> > > > > > > > >> > > Yours, > > > > > >> > > Roland > > > > > >> > > > > > > > >> > > -- > > > > > >> > > My email is as private as my paper mail. I therefore support > > > > > >> encrypting > > > > > >> > > and signing email messages. Get my PGP key from > > > > > >> > > > > > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xbeLFN1po4t2MOETlRoEnO3fr4pBXNdsn_VXBbuywp1tgk0zKYy_oBKphC1Ny-KY0Vh9$ > > > > > > > >> . > > > > > >> > > > > > > > >> > > > > > >> -- > > > > > >> My email is as private as my paper mail. I therefore support > > > > encrypting > > > > > >> and signing email messages. Get my PGP key from > > > > > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6bdaoR4iJf6WXHxF1UsHWThqJwM8QxTwHKDWcVgn0kphw4Lmahcfk5a8gMIPSpV8TMD$ > > > > . > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > My email is as private as my paper mail. I therefore support encrypting > > > > and signing email messages. Get my PGP key from > > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!4Z697XBz_Q8JIQSrOykhIqb_thU0tH0XLyTUT1zDMwhZlk1earaaKDJXweGCrJ3LGb2HvWAHbeRBlS57nxFF$ > > . > > > > > > > > > > Yours, > > Roland > > > > -- > > My email is as private as my paper mail. I therefore support encrypting > > and signing email messages. Get my PGP key from https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!74UYivLDsgDyPF-Kn9OePqrK6mQFaU6DJIWuNHV41ASctcTTwQkuwK5QhQdt4fQakEt4WOp6BDVKkL7EO8fH$ . > > -- My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from http://keys.gnupg.net. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From wernecklr at gmail.com Wed Jun 28 14:49:44 2023 From: wernecklr at gmail.com (Leonardo Werneck) Date: Wed, 28 Jun 2023 12:49:44 -0700 Subject: [Users] Fwd: SphericalSurfaces + QuasiLocalMeasures + AHFinderDirect In-Reply-To: References: Message-ID: Dear all, I am trying to create a simple parfile to help a PhD student. The goal is to set up puncture initial data with NRPyEllipticET and then use QuasiLocalMeasures/AHFinderDirect to compute properties of the punctures and compare with the expected results. No evolution thorn is expected/required and the simulation ends after the initial data has been set and the first round of analysis has finished. The parfile attached partially works. The puncture initial data is set up and NRPyEllipticET's output is what I'd expect. However, the diagnostics are clearly wrong since: 1) AHFinderDirect fails to find the puncture (in this example I am setting the mass of one of the punctures to zero); and 2) the QuasiLocalMeasures diagnostics is never triggered, despite being scheduled. Could someone provide some guidance on how to set up these thorns correctly, specifically SphericalSurfaces, QuasiLocalMeasures, and AHFinderDirect? Am I missing additional thorns? Also, if someone has a working TwoPuncture example parfile instead, please share it with me as I'd be happy to adapt it! Thanks in advance! Leo -- *------------------------------* *Dr. Leonardo R. Werneck, Ph.D.* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bbh_conformally_flat.par Type: application/octet-stream Size: 9085 bytes Desc: not available URL: From schnetter at gmail.com Wed Jun 28 15:49:05 2023 From: schnetter at gmail.com (Erik Schnetter) Date: Wed, 28 Jun 2023 16:49:05 -0400 Subject: [Users] Fwd: SphericalSurfaces + QuasiLocalMeasures + AHFinderDirect In-Reply-To: References: Message-ID: Leonardo I've briefly looked at the parameter file. The horizon finder needs to know an initial guess for both position and shape of the horizon. You are setting this location to 0, but are placing the puncture at x=5. The horizon finder then fails. QuasiLocalMeasures then sees that there is no horizon and does not run. -erik On Wed, Jun 28, 2023 at 3:50?PM Leonardo Werneck wrote: > > Dear all, > > I am trying to create a simple parfile to help a PhD student. The goal is to set up puncture initial data with NRPyEllipticET and then use QuasiLocalMeasures/AHFinderDirect to compute properties of the punctures and compare with the expected results. No evolution thorn is expected/required and the simulation ends after the initial data has been set and the first round of analysis has finished. > > The parfile attached partially works. The puncture initial data is set up and NRPyEllipticET's output is what I'd expect. However, the diagnostics are clearly wrong since: 1) AHFinderDirect fails to find the puncture (in this example I am setting the mass of one of the punctures to zero); and 2) the QuasiLocalMeasures diagnostics is never triggered, despite being scheduled. > > Could someone provide some guidance on how to set up these thorns correctly, specifically SphericalSurfaces, QuasiLocalMeasures, and AHFinderDirect? Am I missing additional thorns? > > Also, if someone has a working TwoPuncture example parfile instead, please share it with me as I'd be happy to adapt it! > > Thanks in advance! > Leo > > -- > ------------------------------ > Dr. Leonardo R. Werneck, Ph.D. > _______________________________________________ > Users mailing list > Users at einsteintoolkit.org > http://lists.einsteintoolkit.org/mailman/listinfo/users -- Erik Schnetter http://www.perimeterinstitute.ca/personal/eschnetter/ From tootle at itp.uni-frankfurt.de Wed Jun 28 15:56:25 2023 From: tootle at itp.uni-frankfurt.de (Samuel Tootle) Date: Wed, 28 Jun 2023 22:56:25 +0200 Subject: [Users] Fwd: SphericalSurfaces + QuasiLocalMeasures + AHFinderDirect In-Reply-To: References: Message-ID: <591187eb-cdea-5594-4d04-92e58eeae1b6@itp.uni-frankfurt.de> Hi Leo, I actually have the same test for the FUKA importer using a single BH.? The par can be found with the KadathImporter thorn or directly here: https://bitbucket.org/fukaws/kadathimporter/src/master/test/bhtest.par Cheers, Samuel -------------------------------- Postdoctoral Researcher Institue for Theoretical Physics Goethe University Frankfurt https://relastro.uni-frankfurt.de/ On 6/28/23 10:49 PM, Erik Schnetter wrote: > Leonardo > > I've briefly looked at the parameter file. The horizon finder needs to > know an initial guess for both position and shape of the horizon. You > are setting this location to 0, but are placing the puncture at x=5. > The horizon finder then fails. QuasiLocalMeasures then sees that there > is no horizon and does not run. > > -erik > > On Wed, Jun 28, 2023 at 3:50?PM Leonardo Werneck wrote: >> Dear all, >> >> I am trying to create a simple parfile to help a PhD student. The goal is to set up puncture initial data with NRPyEllipticET and then use QuasiLocalMeasures/AHFinderDirect to compute properties of the punctures and compare with the expected results. No evolution thorn is expected/required and the simulation ends after the initial data has been set and the first round of analysis has finished. >> >> The parfile attached partially works. The puncture initial data is set up and NRPyEllipticET's output is what I'd expect. However, the diagnostics are clearly wrong since: 1) AHFinderDirect fails to find the puncture (in this example I am setting the mass of one of the punctures to zero); and 2) the QuasiLocalMeasures diagnostics is never triggered, despite being scheduled. >> >> Could someone provide some guidance on how to set up these thorns correctly, specifically SphericalSurfaces, QuasiLocalMeasures, and AHFinderDirect? Am I missing additional thorns? >> >> Also, if someone has a working TwoPuncture example parfile instead, please share it with me as I'd be happy to adapt it! >> >> Thanks in advance! >> Leo >> >> -- >> ------------------------------ >> Dr. Leonardo R. Werneck, Ph.D. >> _______________________________________________ >> Users mailing list >> Users at einsteintoolkit.org >> http://lists.einsteintoolkit.org/mailman/listinfo/users > > From wernecklr at gmail.com Wed Jun 28 15:57:52 2023 From: wernecklr at gmail.com (Leonardo Werneck) Date: Wed, 28 Jun 2023 13:57:52 -0700 Subject: [Users] Fwd: SphericalSurfaces + QuasiLocalMeasures + AHFinderDirect In-Reply-To: References: Message-ID: He Erik, Thanks for the response! The puncture is placed at x=5 because of the nature of the grid used by the elliptic solver. However, when the solution is interpolated to the carpet grid the position of the puncture is shifted to x=0 (this is controlled by the parameter NRPyEllipticET::position_shift[0]; please see the attached plot of the lapse function at t=0). Cheers, Leo [image: image.png] On Wed, Jun 28, 2023 at 1:49?PM Erik Schnetter wrote: > Leonardo > > I've briefly looked at the parameter file. The horizon finder needs to > know an initial guess for both position and shape of the horizon. You > are setting this location to 0, but are placing the puncture at x=5. > The horizon finder then fails. QuasiLocalMeasures then sees that there > is no horizon and does not run. > > -erik > > On Wed, Jun 28, 2023 at 3:50?PM Leonardo Werneck > wrote: > > > > Dear all, > > > > I am trying to create a simple parfile to help a PhD student. The goal > is to set up puncture initial data with NRPyEllipticET and then use > QuasiLocalMeasures/AHFinderDirect to compute properties of the punctures > and compare with the expected results. No evolution thorn is > expected/required and the simulation ends after the initial data has been > set and the first round of analysis has finished. > > > > The parfile attached partially works. The puncture initial data is set > up and NRPyEllipticET's output is what I'd expect. However, the diagnostics > are clearly wrong since: 1) AHFinderDirect fails to find the puncture (in > this example I am setting the mass of one of the punctures to zero); and 2) > the QuasiLocalMeasures diagnostics is never triggered, despite being > scheduled. > > > > Could someone provide some guidance on how to set up these thorns > correctly, specifically SphericalSurfaces, QuasiLocalMeasures, and > AHFinderDirect? Am I missing additional thorns? > > > > Also, if someone has a working TwoPuncture example parfile instead, > please share it with me as I'd be happy to adapt it! > > > > Thanks in advance! > > Leo > > > > -- > > ------------------------------ > > Dr. Leonardo R. Werneck, Ph.D. > > _______________________________________________ > > Users mailing list > > Users at einsteintoolkit.org > > http://lists.einsteintoolkit.org/mailman/listinfo/users > > > > -- > Erik Schnetter > http://www.perimeterinstitute.ca/personal/eschnetter/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 12840 bytes Desc: not available URL: From schnetter at gmail.com Wed Jun 28 17:13:32 2023 From: schnetter at gmail.com (Erik Schnetter) Date: Wed, 28 Jun 2023 18:13:32 -0400 Subject: [Users] Fwd: SphericalSurfaces + QuasiLocalMeasures + AHFinderDirect In-Reply-To: References: Message-ID: Leonardo In this case the initial guess for the horizon radius might be off. As a rule of thumb, the surface where \alpha = 0.3 is the horizon. Your guess of r = 1 might be off. -erik On Wed, Jun 28, 2023 at 4:58?PM Leonardo Werneck wrote: > He Erik, > > Thanks for the response! > > The puncture is placed at x=5 because of the nature of the grid used by > the elliptic solver. However, when the solution is interpolated to the > carpet grid the position of the puncture is shifted to x=0 (this is > controlled by the parameter NRPyEllipticET::position_shift[0]; please see > the attached plot of the lapse function at t=0). > > Cheers, > Leo > > [image: image.png] > > > > On Wed, Jun 28, 2023 at 1:49?PM Erik Schnetter > wrote: > >> Leonardo >> >> I've briefly looked at the parameter file. The horizon finder needs to >> know an initial guess for both position and shape of the horizon. You >> are setting this location to 0, but are placing the puncture at x=5. >> The horizon finder then fails. QuasiLocalMeasures then sees that there >> is no horizon and does not run. >> >> -erik >> >> On Wed, Jun 28, 2023 at 3:50?PM Leonardo Werneck >> wrote: >> > >> > Dear all, >> > >> > I am trying to create a simple parfile to help a PhD student. The goal >> is to set up puncture initial data with NRPyEllipticET and then use >> QuasiLocalMeasures/AHFinderDirect to compute properties of the punctures >> and compare with the expected results. No evolution thorn is >> expected/required and the simulation ends after the initial data has been >> set and the first round of analysis has finished. >> > >> > The parfile attached partially works. The puncture initial data is set >> up and NRPyEllipticET's output is what I'd expect. However, the diagnostics >> are clearly wrong since: 1) AHFinderDirect fails to find the puncture (in >> this example I am setting the mass of one of the punctures to zero); and 2) >> the QuasiLocalMeasures diagnostics is never triggered, despite being >> scheduled. >> > >> > Could someone provide some guidance on how to set up these thorns >> correctly, specifically SphericalSurfaces, QuasiLocalMeasures, and >> AHFinderDirect? Am I missing additional thorns? >> > >> > Also, if someone has a working TwoPuncture example parfile instead, >> please share it with me as I'd be happy to adapt it! >> > >> > Thanks in advance! >> > Leo >> > >> > -- >> > ------------------------------ >> > Dr. Leonardo R. Werneck, Ph.D. >> > _______________________________________________ >> > Users mailing list >> > Users at einsteintoolkit.org >> > http://lists.einsteintoolkit.org/mailman/listinfo/users >> >> >> >> -- >> Erik Schnetter >> http://www.perimeterinstitute.ca/personal/eschnetter/ >> > -- Erik Schnetter http://www.perimeterinstitute.ca/personal/eschnetter/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 12840 bytes Desc: not available URL: From scupp1 at my.apsu.edu Thu Jun 29 09:38:57 2023 From: scupp1 at my.apsu.edu (Cupp, Samuel D.) Date: Thu, 29 Jun 2023 14:38:57 +0000 Subject: [Users] Meeting Minutes for 23-06-29 Message-ID: Chair: Leo Minutes: Sam Present: Leo Sam Roland Yosef ET_2023_11 release People can begin considering possible inclusions for the ET_2023_11 release. There is still ~1 month before finalizing this list, but it would be good to start proposing these sooner rather than later. Roland: Will the Elliptica reader be proposed again? Leo sent email during previous release detailing goals for inclusion. We don't know if they plan to propose it again for this release. Ticket 2497 Leo and Sam made some suggestions, but it is still unclear what the desired behavior is. Because of how TmunuBase schedules things, it appears that setting stress_energy_at_RHS to "no" will never set the Tmunu variables, meaning they will be nan. SetTmunu should probably also be scheduled in CCTK_INITIAL to allow for fixed Tmunu. The scheduling would likely be similar to HydroBase_Prim2ConInitial. Roland will propose a possible solution in the ticket. Ticket 2738 Canuda tests are now working ET US Meeting: RIT Inn answered ambiguously about whether they provide a shuttle to/from airport or just to/from university. Yosef will clarify this with the hotel. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shamims at iiserb.ac.in Tue Jun 6 03:01:56 2023 From: shamims at iiserb.ac.in (Shamim Haque 1910511) Date: Tue, 06 Jun 2023 08:01:56 -0000 Subject: [Users] Compilation Error in cluster ParamGanga In-Reply-To: <20230601150128.22b31a04@ekohaes8> References: <74397676-cd6a-551e-ef18-b5df77edfc71@cct.lsu.edu> <4449ed77-ac78-8774-896b-f86737ea9bf0@cct.lsu.edu> <20230511135018.1d31cfb1@ekohaes8> <20230518090147.4248b227@ekohaes8> <20230601150128.22b31a04@ekohaes8> Message-ID: Hello Roland, Thank you for the reply. I pointed -gxx-name to gcc/8.3.0, which resulted in an error related to the "icpc unable to checkout FLEXlm license". I also tried to use a different intel compiler (and point -gxx-name to gcc/11.1.0) to check if the error can go away, but it remains the same. I communicated this to the HPC admin and awaiting their response; please let me know if I should fix something in my configuration. I have attached the out file (intel_compile.out), config log (intel_config.log), and optionlist (intel.cfg) for reference. Meanwhile, I also attempted to configure ET with gcc/11.1.0, which resulted in some issues regarding the FFTW3. I have contacted the admins regarding this, but please let me know if it is something to be fixed within the optionlist. I have attached the out file (gcc_compile.out), config log (gcc_config.log), and optionlist (gcc.cfg) for reference. The error in the output file says the following: *Formaline: is still running, e.g. the final link stage.)/usr/bin/ld: warning: libimf.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: libsvml.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: libirng.so, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: libintlc.so.5, needed by /home/apps/fftw3/fftw3.3.8/lib/libfftw3.so, not found (try using -rpath or -rpath-link)/usr/bin/ld: warning: libgfortran.so.3, needed by /lib/../lib64/liblapack.so, may conflict with libgfortran.so.5/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `__intel_cpu_features_init_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `__svml_sincos2'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `_intel_fast_memset'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `__intel_cpu_feature_indicator_x'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `_intel_fast_memmove'/home/apps/fftw3/fftw3.3.8/lib/libfftw3.so: undefined reference to `_intel_fast_memcpy'* Thanks in advance for all the help. Regards Shamim ? On Fri, Jun 2, 2023 at 1:31?AM Roland Haas wrote: > Hello Shamim Haque, > > This most likely is caused by not having a full C++11 software stack > enabled. > > The Intel compiler always uses the C++ STL library of the GNU g++ > compiler and must be matched up with a compatible g++ compiler's STL > to correctly support C++11. > > This ticket (the last of many, hopefully) contains details: > > > https://bitbucket.org/einsteintoolkit/tickets/issues/2663/add-an-notice-about-minimum-intel-gcc > > Basically, the fix is to find a gcc compiler at least version 6 on your > cluster, then pass the option > > -gxx-name PATH-TO-GXX > > to icpc in the CXXFLAGS option list variable (see eg > > https://bitbucket.org/simfactory/simfactory2/src/7e2592d264b1afe9ca2cb54bcab8dd9e4165cf98/mdb/optionlists/smic-openmpi.cfg?at=master#smic-openmpi.cfg-15:16 > ). > > If there is no gcc compiler version 6 or newer installed in the system, > you will have to contact the administrators and ask them about it. > > The demo code in the description of > https://bitbucket.org/cactuscode/cactus/pull-requests/152 may help > demonstrate to the admins what the issue is. > > Yours, > Roland > > > Hello All, > > > > I have made little progress on compiling ETK on this cluster, as the > admins > > allowed me to compile ETK on the login node. I am finding new errors and > am > > unable to debug the issue. I am trying to configure ETK using Intel19, > and > > the error says the following: > > > > > > > > > > > > > > > > > > > > > > > > > */home/shamims.iiserb/ET_debug/Cactus/arrangements/Carpet/CarpetMask/src/mask_surface.cc(160): > > error: more than one instance of overloaded function "CarpetMask::isnan" > > matches the argument list: function "isnan(double)" > > function "std::isnan(double)" argument types are: (CCTK_REAL8) > > assert(not isnan(phi)); ^compilation aborted for > > > /home/shamims.iiserb/ET_debug/Cactus/configs/sim/build/CarpetMask/mask_surface.cc > > (code 2)make[3]: *** [mask_surface.cc.o] Error 2make[2]: *** > [make.checked] > > Error 2make[1]: *** > > > [/home/shamims.iiserb/ET_debug/Cactus/configs/sim/lib/libthorn_CarpetMask.a] > > Error 2make: *** [sim] Error 2* > > > > I have attached the machinefile, optionlist, config log and full output > > file for reference. I configured the scripts using the available example > > optionlist and machinefile for supermuc_ng and stampede2. Please let me > > know if more info is required, and thanks in advance for your help. > > > > Regards > > Shamim Haque > > Senior Research Fellow (SRF) > > Department of Physics > > IISER Bhopal > > > > ? > > > > On Fri, May 19, 2023 at 2:32?PM Shamim Haque 1910511 < > shamims at iiserb.ac.in> > > wrote: > > > > > Hello Roland, > > > > > > Thank you for the reply. Yes, I am trying to compile ETK in the > computing > > > node via sbatch queue. I am in touch with the system admin to resolve > > > the issue with the compilers. > > > > > > Regards > > > Shamim Haque > > > Senior Research Fellow (SRF) > > > Department of Physics > > > IISER Bhopal > > > > > > ? > > > > > > On Thu, May 18, 2023 at 7:32?PM Roland Haas > wrote: > > > > > >> Hello Shamin, > > >> > > >> A missing math.h would point to a faulty compiler installation. Since > > >> you are trying to compile (are you?) ona compute node in a running > job, > > >> my guess would be that the cluster admins are not providing compilers > > >> on the compute nodes but only on the login nodes. > > >> > > >> This would be something to contact your cluster's help desk about. > > >> > > >> Yours, > > >> Roland > > >> > > >> > Hello Roland, > > >> > > > >> > I did try that fix; the error remains the same. Actually, we landed > on > > >> an > > >> > even more primitive problem, where Steve suggested a basic test, > that > > >> is to > > >> > compile the following code: > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > *#include #include int main() { printf("%f\n", > > >> > M_PI); return 0;}* > > >> > > > >> > This program compiles just fine in the login node but gives out the > > >> > following error while compiling it via jobscript submission in the > > >> sbatch > > >> > queue: > > >> > > > >> > prog.cpp:1:18: fatal error: math.h: No such file or directory > > >> > #include > > >> > ^ > > >> > compilation terminated. > > >> > > > >> > ----OR----- > > >> > > > >> > > > >> > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > > >> > fatal error: features.h: No such file or directory > > >> > 39 | #include > > >> > | ^~~~~~~~~~~~ > > >> > compilation terminated. > > >> > > > >> > I have reported this problem to the HPC admin and await their > response. > > >> > > > >> > Regards > > >> > Shamim Haque > > >> > Senior Research Fellow (SRF) > > >> > Department of Physics > > >> > IISER Bhopal > > >> > > > >> > ? > > >> > > > >> > On Fri, May 12, 2023 at 12:20?AM Roland Haas > > >> wrote: > > >> > > > >> > > Hello all, > > >> > > > > >> > > Since the error is a missing M_PI and the configure script already > > >> > > gives a suggested fix: > > >> > > > > >> > > Try adding -D_XOPEN_SOURCE to CPPFLAGS. > > >> > > > > >> > > have you tried adding: > > >> > > > > >> > > CPPFLAGS=-D_XOPEN_SOURCE > > >> > > > > >> > > to your option list file before compiling? > > >> > > > > >> > > Yours, > > >> > > Roland > > >> > > > > >> > > > Thanks Steve. I have raised complaint to the admin about this > issue. > > >> > > > > > >> > > > On Thu, May 11, 2023, 11:18 PM Steven R. Brandt < > > >> sbrandt at cct.lsu.edu> > > >> > > wrote: > > >> > > > > > >> > > > > I would talk to the system administrators about that. There > is > > >> > > definitely > > >> > > > > something wrong with your compiler install if you can't find > > >> math.h. > > >> > > > > > > >> > > > > --Steve > > >> > > > > On 5/11/2023 2:02 AM, Shamim Haque 1910511 wrote: > > >> > > > > > > >> > > > > Dear Steve, > > >> > > > > > > >> > > > > I just rechecked the cpp program to output M_PI in the > computing > > >> nodes, > > >> > > > > and that is not working right now, which was somehow working > fine > > >> a few > > >> > > > > days ago. I also tried different gcc versions (4.8, 10.2, > 11.1, > > >> > > gnu8/8.3) > > >> > > > > available in the cluster. All report similar issue: > > >> > > > > > > >> > > > > prog.cpp:1:18: fatal error: math.h: No such file or directory > > >> > > > > #include > > >> > > > > ^ > > >> > > > > compilation terminated. > > >> > > > > > > >> > > > > ----OR----- > > >> > > > > > > >> > > > > > > >> > > > > >> > /home/apps/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/os_defines.h:39:10: > > >> > > >> > > > > fatal error: features.h: No such file or directory > > >> > > > > 39 | #include > > >> > > > > | ^~~~~~~~~~~~ > > >> > > > > compilation terminated. > > >> > > > > > > >> > > > > The cpp code seems to work fine in login node, as it gives > out > > >> the > > >> > > value > > >> > > > > of pi upon execution. I am contacting the hpc admin to > resolve > > >> this > > >> > > issue. > > >> > > > > > > >> > > > > I am attaching the test code (prog.cpp), the sbatch > jobscript > > >> (prog.sh) > > >> > > > > and the output file (slurn-117473.out), if you would like to > have > > >> a > > >> > > look at > > >> > > > > it. > > >> > > > > > > >> > > > > Please let me know if I should try something different to > get > > >> this > > >> > > working. > > >> > > > > > > >> > > > > Regards > > >> > > > > Shamim Haque > > >> > > > > Senior Research Fellow (SRF) > > >> > > > > Department of Physics > > >> > > > > IISER Bhopal > > >> > > > > > > >> > > > > ? > > >> > > > > > > >> > > > > On Mon, May 8, 2023 at 7:25?PM Shamim Haque 1910511 < > > >> > > shamims at iiserb.ac.in> > > >> > > > > wrote: > > >> > > > > > > >> > > > >> Thanks Steve. I'll make the changes and try again. I'll let > you > > >> know > > >> > > the > > >> > > > >> outcome. > > >> > > > >> > > >> > > > >> Regards > > >> > > > >> Shamim Haque > > >> > > > >> Senior Research Fellow (SRF) > > >> > > > >> Department of Physics > > >> > > > >> IISER Bhopal > > >> > > > >> > > >> > > > >> ? > > >> > > > >> > > >> > > > >> On Mon, May 8, 2023 at 7:21?PM Steven R. Brandt < > > >> sbrandt at cct.lsu.edu> > > >> > > > >> wrote: > > >> > > > >> > > >> > > > >>> That tends to make me think you aren't using the compiler > you > > >> think > > >> > > > >>> you're using. You could put full paths in the .cfg files > and > > >> > > recompile from > > >> > > > >>> scratch. > > >> > > > >>> > > >> > > > >>> i.e. > > >> > > > >>> > > >> > > > >>> CXX = /full/path/to/icc > > >> > > > >>> > > >> > > > >>> --Steve > > >> > > > >>> On 5/3/2023 3:26 PM, Shamim Haque 1910511 wrote: > > >> > > > >>> > > >> > > > >>> Yes, this gives the value of Pi successfully, before > starting > > >> the > > >> > > > >>> compilation: > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> */home/vkerni.iitr/sam/trial 3.141593 Current machine: > > >> paramganga3 > > >> > > Using > > >> > > > >>> configuration: sim Reconfiguring sim Writing configuration > to: > > >> > > > >>> > /home/vkerni.iitr/sam/ET_debug/Cactus/configs/sim/OptionList > > >> Cactus - > > >> > > > >>> version: 4.12.0 Reconfiguring sim. * > > >> > > > >>> *.* > > >> > > > >>> *.* > > >> > > > >>> *.* > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> *checking for M_PI... no configure: error: M_PI not > defined. > > >> Try > > >> > > adding > > >> > > > >>> -D_XOPEN_SOURCE to CPPFLAGS. Error reconfiguring > sim-config > > >> make: *** > > >> > > > >>> [sim-config] Error 2 * > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> Shamim Haque > > >> > > > >>> Senior Research Fellow (SRF) > > >> > > > >>> Department of Physics > > >> > > > >>> IISER Bhopal > > >> > > > >>> > > >> > > > >>> ? > > >> > > > >>> > > >> > > > >>> On Thu, May 4, 2023 at 12:29?AM Steven Brandt < > > >> sbrandt at cct.lsu.edu> > > >> > > > >>> wrote: > > >> > > > >>> > > >> > > > >>>> So, sanity check, can you compile the following program > > >> outside of > > >> > > > >>>> Cactus? > > >> > > > >>>> > > >> > > > >>>> #include > > >> > > > >>>> #include > > >> > > > >>>> > > >> > > > >>>> int main() { > > >> > > > >>>> printf("%f\n", M_PI); > > >> > > > >>>> return 0; > > >> > > > >>>> } > > >> > > > >>>> On 5/3/2023 1:50 PM, Shamim Haque 1910511 wrote: > > >> > > > >>>> > > >> > > > >>>> Dear Steve, > > >> > > > >>>> > > >> > > > >>>> I tried gcc (10 and 11) as well and I am stuck with the > > >> following > > >> > > error: > > >> > > > >>>> *configure: error: M_PI not defined. Try adding > > >> -D_XOPEN_SOURCE to > > >> > > > >>>> CPPFLAGS.* > > >> > > > >>>> > > >> > > > >>>> This also happened for Intel (2020 and 2019). I have > attached > > >> both > > >> > > the > > >> > > > >>>> output files here. > > >> > > > >>>> > > >> > > > >>>> Please ignore the out file in the first email, I have > > >> uploaded > > >> > > another > > >> > > > >>>> out file by mistake. The rest of the files are correct in > the > > >> first > > >> > > mail. > > >> > > > >>>> > > >> > > > >>>> Regards > > >> > > > >>>> Shamim Haque > > >> > > > >>>> Senior Research Fellow (SRF) > > >> > > > >>>> Department of Physics > > >> > > > >>>> IISER Bhopal > > >> > > > >>>> > > >> > > > >>>> ? > > >> > > > >>>> > > >> > > > >>>> On Wed, May 3, 2023 at 8:04?PM Steven Brandt < > > >> sbrandt at cct.lsu.edu> > > >> > > > >>>> wrote: > > >> > > > >>>> > > >> > > > >>>>> I notice from the output that your compiler claims > "limits.h" > > >> is > > >> > > > >>>>> missing. Possibly there's something wrong with your icc > > >> setup? > > >> > > Maybe you > > >> > > > >>>>> should try using gcc / gfortran instead. > > >> > > > >>>>> > > >> > > > >>>>> --Steve > > >> > > > >>>>> On 5/3/2023 8:20 AM, Shamim Haque 1910511 wrote: > > >> > > > >>>>> > > >> > > > >>>>> Dear Frank, > > >> > > > >>>>> > > >> > > > >>>>> I added the alias pattern for the computing nodes and now > ETK > > >> can > > >> > > > >>>>> detect the cluster (checked by asking "whoami") during > the > > >> batch > > >> > > queuing. > > >> > > > >>>>> I'll get in touch with the admins to include 'patch'. > > >> > > > >>>>> > > >> > > > >>>>> However, the problem still persists, having assured that > the > > >> > > correct > > >> > > > >>>>> machine file and option list are being used during the > > >> compilation. > > >> > > > >>>>> > > >> > > > >>>>> Regards > > >> > > > >>>>> Shamim Haque > > >> > > > >>>>> Senior Research Fellow (SRF) > > >> > > > >>>>> Department of Physics > > >> > > > >>>>> IISER Bhopal > > >> > > > >>>>> > > >> > > > >>>>> ? > > >> > > > >>>>> > > >> > > > >>>>> On Wed, May 3, 2023 at 3:49?PM Frank Loeffler < > > >> > > > >>>>> frank.loeffler at uni-jena.de> wrote: > > >> > > > >>>>> > > >> > > > >>>>>> Hi Shamim > > >> > > > >>>>>> > > >> > > > >>>>>> On Wed, May 03, 2023 at 03:11:49PM +0530, Shamim Haque > > >> 1910511 > > >> > > wrote: > > >> > > > >>>>>> >We are trying to compile ETK on ParamGanga at IIT > Roorkee. > > >> The > > >> > > > >>>>>> >compilation > > >> > > > >>>>>> >stops at a very initial step and gives out the error: > > >> > > > >>>>>> > > > >> > > > >>>>>> >*checking whether the Fortran compiler (gfortran ) > > >> works... > > >> > > > >>>>>> yeschecking > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) is a > > >> > > cross-compiler... > > >> > > > >>>>>> nochecking > > >> > > > >>>>>> >whether the Fortran compiler (gfortran ) supports > TYPE(*) > > >> for > > >> > > > >>>>>> >CCTK_PointerTo... yeschecking how to run the C > > >> preprocessor... > > >> > > > >>>>>> cppchecking > > >> > > > >>>>>> >for ANSI C header files... nochecking for C99 > features... > > >> > > > >>>>>> yeschecking for > > >> > > > >>>>>> >M_PI... noconfigure: error: M_PI not defined. Try > adding > > >> > > > >>>>>> -D_XOPEN_SOURCE to > > >> > > > >>>>>> >CPPFLAGS.* > > >> > > > >>>>>> > > > >> > > > >>>>>> >We tried adding -D_XOPEN_SOURCE to CPPFLAGS, but it > does > > >> not > > >> > > help. > > >> > > > >>>>>> This > > >> > > > >>>>>> >error is consistent if we set up using Intel Compiler > (2020 > > >> or > > >> > > > >>>>>> 2019), or > > >> > > > >>>>>> >gcc (11 or 10). I have attached the outfile, error > file, > > >> > > machine > > >> > > > >>>>>> script, > > >> > > > >>>>>> >and option lists for both cases (Intel and gcc) for > any > > >> > > reference. > > >> > > > >>>>>> > > > >> > > > >>>>>> >Secondly, we are not allowed to compile ETK on login > nodes. > > >> So > > >> > > we are > > >> > > > >>>>>> >queuing a job script, which upon execution, enters the > > >> Cactus > > >> > > > >>>>>> directory, > > >> > > > >>>>>> >does "setup-silent", and then executes the build > command. > > >> Is > > >> > > this > > >> > > > >>>>>> the right > > >> > > > >>>>>> >way to compile ETK using the batch script? Or should we > do > > >> > > something > > >> > > > >>>>>> >different? I have attached the shell script > (compile.sh) > > >> used > > >> > > for > > >> > > > >>>>>> sbatch > > >> > > > >>>>>> >queue for reference. > > >> > > > >>>>>> > > >> > > > >>>>>> I might be wrong, but setup-silent will not use your > machine > > >> file > > >> > > > >>>>>> which > > >> > > > >>>>>> contains all the compiler flags, but will setup > something > > >> "from > > >> > > > >>>>>> scratch" > > >> > > > >>>>>> instead. According to the logs, this will be, e.g., in > > >> > > > >>>>>> > > >> > > > >>>>>> > > >> > > > >>>>>> > > >> > > > > >> > /home/shamims.iiserb/ET_debug/Cactus/repos/simfactory2/mdb/machines/cn139.iitr.ac.in.ini > > >> > > >> > > > >>>>>> > > >> > > > >>>>>> My guess would be that this misses the compiler flags > > >> necessary to > > >> > > > >>>>>> have > > >> > > > >>>>>> M_PI defined. The best way to get this working might be > to > > >> let > > >> > > > >>>>>> simfactory also detect the cluster configuration on the > > >> compute > > >> > > nodes > > >> > > > >>>>>> (where you compile), such that setup-silent is not > needed, > > >> but > > >> > > > >>>>>> instead > > >> > > > >>>>>> the correct cluster configuration is found and used > > >> > > automatically. In > > >> > > > >>>>>> order to do that, look at examples of 'aliaspattern' in > the > > >> > > > >>>>>> mdb/machines > > >> > > > >>>>>> directory. An alternative would be to tell simfactory > > >> specifically > > >> > > > >>>>>> that > > >> > > > >>>>>> you want to use your machine configuration file. > > >> > > > >>>>>> > > >> > > > >>>>>> As a side-note: I noticed the command 'patch' is > missing > > >> too. > > >> > > This is > > >> > > > >>>>>> a > > >> > > > >>>>>> tool so common that it should be installed everywhere. > Your > > >> admin > > >> > > is > > >> > > > >>>>>> probably the best person to ask for advise here. It > should > > >> not be > > >> > > too > > >> > > > >>>>>> hard to install yourself, but that should not be > necessary > > >> either, > > >> > > > >>>>>> especially when you are told to compile on compute nodes. > > >> > > > >>>>>> > > >> > > > >>>>>> Frank > > >> > > > >>>>>> > > >> > > > >>>>>> > > >> > > > >>>>> _______________________________________________ > > >> > > > >>>>> Users mailing listUsers at einsteintoolkit.orghttp:// > > >> > > lists.einsteintoolkit.org/mailman/listinfo/users > > >> > > > >>>>> > > >> > > > >>>>> _______________________________________________ > > >> > > > >>>>> Users mailing list > > >> > > > >>>>> Users at einsteintoolkit.org > > >> > > > >>>>> > > >> > > > > >> > https://urldefense.com/v3/__http://lists.einsteintoolkit.org/mailman/listinfo/users__;!!DZ3fjg!4xAKSrBYUo_v8jHEbenNg6aQ99Oq9QyeXsjq766lVDVwHAhFJODYjSU6XryKp264bQqDmBcTIA4Srkc4wOCn$ > > >> > > >> > > > >>>>> > > >> > > > >>>> > > >> > > > > >> > > > > >> > > Yours, > > >> > > Roland > > >> > > > > >> > > -- > > >> > > My email is as private as my paper mail. I therefore support > > >> encrypting > > >> > > and signing email messages. Get my PGP key from > > >> > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!5ZGhPGL_16Uj1eN8xbeLFN1po4t2MOETlRoEnO3fr4pBXNdsn_VXBbuywp1tgk0zKYy_oBKphC1Ny-KY0Vh9$ > > >> . > > >> > > > > >> > > >> -- > > >> My email is as private as my paper mail. I therefore support > encrypting > > >> and signing email messages. Get my PGP key from > https://urldefense.com/v3/__http://pgp.mit.edu__;!!DZ3fjg!9Rc70AgC5_rUJApjF6bdaoR4iJf6WXHxF1UsHWThqJwM8QxTwHKDWcVgn0kphw4Lmahcfk5a8gMIPSpV8TMD$ > . > > >> > > > > > > -- > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from http://pgp.mit.edu . > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: intel_compile.out Type: application/octet-stream Size: 286535 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: intel_config.log Type: text/x-log Size: 53512 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: intel.cfg Type: application/octet-stream Size: 2798 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gcc_compile.out Type: application/octet-stream Size: 564990 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gcc_config.log Type: text/x-log Size: 58849 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gcc.cfg Type: application/octet-stream Size: 2269 bytes Desc: not available URL: From vinod.pisharody22 at gmail.com Fri Jun 23 05:30:20 2023 From: vinod.pisharody22 at gmail.com (Vinod Pisharody) Date: Fri, 23 Jun 2023 16:00:20 +0530 Subject: [Users] Only getting partial Waveform while running that parameter file in the gallery example for BBH GW150914 Message-ID: Hello, I have been trying to run the simulation of GW150914 given in gallery .The simulation ran succesfully .However I am only getting a partial waveform .The merger and the ringdown phase is not being captured .I am attaching both the waveform and a gnuplot showing the completing of merger . There is also another doubt I have .In the gallery while creating the job for this simulation there is a command called --define N which is set to 28 .Could someone tell me whats the purpose of that and help me understand whats going wrong ? Thanks and Regards, Vinod -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tracker.png Type: image/png Size: 13059 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: strain_22_det-1.png Type: image/png Size: 57892 bytes Desc: not available URL: From nils.vu at black-holes.org Mon Jun 5 06:00:05 2023 From: nils.vu at black-holes.org (Nils Vu) Date: Mon, 05 Jun 2023 11:00:05 -0000 Subject: [Users] Today: NR community call at 9am Pacific / noon NYC / 5pm London / 6pm Berlin Message-ID: <2B9477B6-BC16-45E7-96BF-ED469FB3421F@black-holes.org> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: zoom.ico Type: image/x-icon Size: 4286 bytes Desc: not available URL: From wsun3 at nd.edu Tue Jun 20 15:01:42 2023 From: wsun3 at nd.edu (Wei Sun) Date: Tue, 20 Jun 2023 20:01:42 -0000 Subject: [Users] Failure to run static_tov simulation using more than 1 node on cluster Message-ID: Hello, I am trying to run the static_tov simulation on my university's cluster (CRC Notre Dame ). However, I got this error when acquiring 48 cores: Error: Too many nodes specified: nodes=2 (maxnodes is 1) I am assuming this is due to the *#Source tree management* module in the Cactus/simfactory/mdb/machines/.ini file. In my case, the automatic *.ini file I got has these configuration: ppn = 24 max-num-threads = 24 num-threads = 12 nodes = 1 However, the machine I use has 24 cores for each node, and the number of threads for each core is 1. That means if I run the simulation using 48 cores, I need 2 nodes. Although I didn't get 2 hosts successfully(I got only 1), I am wondering if I have to edit the .ini file after the command ./simfactory/bin/sim setup-silent by hand, for example: num-threads=1, nodes=2? Or is there any other correct way for getting more than 1 node? I would be appreciate it if you could give me some ideas on this problem. Best regards, Wei -------------- next part -------------- An HTML attachment was scrubbed... URL: From cuosakwe at ucalgary.ca Thu Jun 29 14:58:01 2023 From: cuosakwe at ucalgary.ca (CJ Osakwe) Date: Thu, 29 Jun 2023 19:58:01 +0000 Subject: [Users] running TOVSolver standalone Message-ID: Hello, I am a Ph.D. student working at the University of Calgary. I recently started using the Einstein Toolkit to conduct numerical relativity simulations for my thesis. As of now I am familiarizing myself with the software. In particular, I am trying to model a perturbed TOV neutron star. To this end, I used the TOVSolver thorn for my initial data (based off of the single, stable neutron star example). Where does the TOVSolver thorn output its data? I have so far been unable to determine this myself. I want to run this thorn by itself and am not sure how. Any help would be greatly appreciated. Cheers, CJ Osakwe -------------- next part -------------- An HTML attachment was scrubbed... URL: