[Einstein Toolkit] #1854: Very large global grid points leading termination

List overview All Threads
Download

newer

older

[Einstein Toolkit] #1863: Carpet...

[Einstein Toolkit] #1860: handle...

Einstein Toolkit

5 Feb 2016 5 Feb '16

6:37 a.m.

#1854: Very large global grid points leading termination -------------------------------------------+-------------------------------- Reporter: himanshukharkwal765@… | Owner: himanshukharkwal765@… Type: defect | Status: new Priority: major | Milestone: ET_2015_11 Component: Cactus | Version: development version Keywords: grid global terminate | -------------------------------------------+-------------------------------- In WaveMol/gaussian.par whenever we increase our global grid points to more than 350 the program terminates abruptly. I am attaching gaussian.par file and hn1.lnmiit.ac.in.ini (simfactory/mdb/machines/hn1.lnmiit.ac.in.ini).

HPC Details- Node: 6 Processor: http://ark.intel.com/products/75277/Intel-Xeon- Processor-E5-2680-v2-25M-Cache-2_80-GHz

Q1. Is there anything wrong inside .par or .ini file? Q2. Should i use Carpet instead of PUGH?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Show replies by date

Einstein Toolkit

5 Feb 5 Feb

7:50 a.m.

#1854: Very large global grid points leading termination --------------------------------------------+------------------------------- Reporter: himanshukharkwal765@… | Owner: himanshukharkwal765@… Type: defect | Status: new Priority: major | Milestone: ET_2015_11 Component: Cactus | Version: development version Resolution: | Keywords: grid global terminate --------------------------------------------+-------------------------------

Comment (by hinder):

If you increase the number of grid points, this increases the amount of memory needed for the run. Is your run terminating because it runs out of memory? If you can run "top" on the node, you should see if this is happening. You can also activate the thorn SystemStatistics and add the following into your parameter file to get output of the memory usage ("maxrss"):

{{{ IOBasic::outInfo_reductions = "maximum" IOBasic::outInfo_vars = "SystemStatistics::maxrss_mb SystemStatistics::swap_used_mb" }}}

This will add two new columns to the info output: the memory used by the process, and the swap space allocated, in MB. You can then compare this to the memory available on the node, and see if there is swapping.

Can you provide the standard output and standard error files so we can see what is happening?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:1 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

1:27 p.m.

Comment (by knarf):

The attached par file needs about 55GB of RAM on my workstation (but runs out of memory during the first step). Memory usage goes roughly like N^3, so ~368 would be the maximum for 64GB of RAM - but even in the current form, 64GB aren't enough for this parameter file. How much memory do you have available when you run it?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:2 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

7 Feb 7 Feb

2:54 a.m.

Comment (by himanshukharkwal765@…):

Replying to [comment:1 hinder]:

1. The results for global grid points = 350 (when it runs fine) a. top command output - top1.png b. output1.out c. output1.err

I stopped it at 64th iteration due to which termination error 15 is occuring.

2. The results for global grid points = 700 (greater than 350, which terminates it automatically) a. top command output - top2.png b. output2.out c. output2.err

It stopped all cactus_sim tasks as soon as it used all its swap memory and gave termination error 9.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:3 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8 Feb 8 Feb

4:23 a.m.

Comment (by hinder):

Thanks for the output. The file "top1.png", from n = 350, already shows that simulation is using much more memory than is available on the node. You see that there is 99 GB total memory, and 99 GB of this are "used". Similarly, the swap usage is very high, at 8 GB. As far as I can see, there is no evidence of there being anything wrong with this simulation; it is just too large to fit on the machine. Note that "350 global grid points" means 350^3 because this is 3D. When you increase that to 700, the memory requirements increase by a factor of 2^3 = 8. So if the n=350 run doesn't fit, the n=700 run has no chance.

According to the output1.out file, SystemStatistics is reporting that each process is using about 15 GB (this means that this amount of data is resident in physical memory currently; it's very possible that the run needs even more than this, and the rest is in swap). You have 6 processes, so the total memory usage is 90 GB, which is about the same as the total memory available (accounting for other processes, overheads, and general subtleties in measuring memory usage). I'm a bit confused about why SystemStatistics is reporting a swap usage of 0, when the "top" output shows 8 GB. Was the "top" output recorded at the same time as the simulation which generated "output1.out"?

To summarise, the machine on which you are running does not have enough memory for the size of job you are trying to run. You either need to restrict to a smaller number of grid points, or run on more nodes.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:4 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4:40 a.m.

Comment (by anonymous):

Replying to [comment:4 hinder]: Thanks. I have attached two outputs of "top". top2.png represents when the termination is occurring while top1.png is the output when everything is running fine and it is showing SystemStatistics.

This "top" is showing output for master node of our HPC. We have 6 slave nodes also.

I use following command to run: ./simfactory/bin/sim submit <simulation folder name> --parfile=arrangements/CactusExamples/WaveMoL/par/gaussian.par --procs=60 --walltime=8:0:0 This runs 6 cactus_sim tasks mentioned in top. I have also attached .ini file containing hpc details.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:5 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

4:56 a.m.

Comment (by hinder):

There is something wrong; the output says that the evolution is running on 6 MPI processes, and your ini file says that you want 10 threads per process, but I can see 6 processes on the first node. Can you describe exactly how your "HPC" is set up? It sounds like you have 7 nodes, one "master" and 6 "slaves". How many processors, and how many cores, does each node have? Are you intending to run using OpenMP + MPI, or pure MPI?

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:6 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:17 a.m.

Comment (by anonymous):

Replying to [comment:6 hinder]: HPC details:

1 node: 2 processor Intel Xeon Processor E5-2680 processor details - http://ark.intel.com/products/75277/Intel-Xeon- Processor-E5-2680-v2-25M-Cache-2_80-GHz 1 node: 20 core 1 node: 96GB

master: 1 node slave: 6 nodes total: 7 nodes

OpenMP + MPI

I think right now i am only running everything on one node(master). I need to submit jobs through job.sh which will devide workload on slave nodes. If that is correct then can you please tell me where to make changes.

PFA: job.sh hostfile OptionList SubmitScript RunScript

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:7 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

6:39 a.m.

Comment (by hinder):

job.sh looks like a PBS submission script. Do you have PBS, or a similar queueing system, running on this machine? If so, then you will need to split the content of this script into a "submission script", containing the #PBS directives, and a "run script" containing the mpirun_rsh commands. If you do not have a queueing system, then you only need the run script. Take a look in Cactus/simfactory/mdb, in the submitscripts and runscripts directories for examples. You need to create a submit script and a run script for your machine, and then edit your machine ini file to point to these. You are using a host file in your mpirun command, but if you are using a queueing system, then OpenMPI should instead know automatically which hosts to run on.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:8 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

1 Mar 1 Mar

7:45 a.m.

Comment (by anonymous):

I tried to work upon the pbs script but i am still getting some errors. After run command i did: qstat -r :

hn1.lnmiit.ac.in:

Job ID Username Queue Jobname SessID NDS TSK Req'd-Memory Req'd-Time S Elap-Time ----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - --------- 207.hn1.lnmiit.ac.in himanshu batch ModernFamily 36422 6 120 -- 720:00:00 R 00:14:57

There were no results(tables). But got some errors inside .err and .out files. Can you please look at the ouput3.err and output3.out.

new attached files: old names - new names

.ini - hn1.lnmiit.ac.in1.ini RunScript - RunScript1 SubmitScript - SubmitScript1 .out - output3.out .err - output3.err

Sorry for the late reply

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:9 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

8:44 a.m.

Comment (by hinder):

There should not be an mpiexec in the submission script. Look at the other submission scripts for examples. You call simfactory from the submission script, and simfactory calls the run script. The run script should then contain the mpiexec call.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:10 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

2 Mar 2 Mar

1:05 a.m.

Comment (by anonymous):

I removed mpiexec from submission script but it is still showing some error:

/home/himanshu/simulations/test_13/SIMFACTORY/exe/cactus_sim: error while loading shared libraries: libhdf5hl_fortran.so.7: cannot open shared object file: No such file or directory

PFA:

.out - output4.out .err - output4.err

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:11 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

3:16 p.m.

Comment (by hinder):

You are using an optionlist for Fedora, which expects to find HDF5 installed in /usr. The problem is that it is not able to find the Fortran HDF5 library at runtime. Is this installed? Are the compute nodes exactly the same as the head node where you compiled Cactus? I suspect that the file libhdf5hl_fortran.so.7 might be present on the head node but not on the compute nodes. Can you check? {{{ find /usr -name libhdf5hl_fortran.so.7 }}}

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:12 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

Einstein Toolkit

15 Mar 15 Mar

5:46 a.m.

#1854: Very large global grid points leading termination --------------------------------------------+------------------------------- Reporter: himanshukharkwal765@… | Owner: himanshukharkwal765@… Type: defect | Status: closed Priority: major | Milestone: ET_2015_11 Component: Cactus | Version: development version Resolution: fixed | Keywords: grid global terminate --------------------------------------------+------------------------------- Changes (by anonymous):

* status: new => closed * resolution: => fixed

Comment:

Thanks for being patient and helpful.

-- Ticket URL: https://trac.einsteintoolkit.org/ticket/1854#comment:13 Einstein Toolkit http://einsteintoolkit.org The Einstein Toolkit

3860

Age (days ago)

3899

Last active (days ago)

trac@lists.einsteintoolkit.org

13 comments

1 participants

tags (0)

participants (1)

Einstein Toolkit