[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: feff8 & MPI

Dear Feffusers,

 Michel Jaouen asks for some scripts to run the special
MPI version of FEFF8 for parallel calculations. This version
is available to licensed users of FEFF8 who have access to
parallel computers running MPI.

The PBS installation is not as friendly as some MPI installations like
LAM and MPICH due to the need to invoke the job scheduler.

Attached are some PBS scripts from our chemistry cluster for FEFF8MPI 
that Alex made. They have to be modified for the environment of your
cluster, of course.


 Cheers,
 John


---------- Forwarded message ----------
Date: Fri, 21 Jun 2002 11:57:38 -0700
From: Alex Ankudinov <alex@leonardo.phys.washington.edu>
To: John J. Rehr <jjr@leonardo.phys.washington.edu>
Subject: parallel runs

Hi John,

Here is how I did it. I put put source modules in Feff8 directory and
I use compile spcript 
 pgf77 -fast -I/usr/pgi/linux86/include/ rdinp_tot.f -o rdinp -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ pot_tot.f -o ffmod1 -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ ldos_tot.f -o ldos -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ xsph_tot.f -o ffmod2 -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ fms_tot.f -o ffmod3 -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ path_tot.f -o ffmod4 -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ genfmt_tot.f -o ffmod5 -lmpich
 pgf77 -fast -I/usr/pgi/linux86/include/ ff2x_tot.f -o ffmod6 -lmpich
to compile the code, where second option tells compiler where to get
that mpi.inc file.

Then you have to prepare your batch job file for PBS job scheduler.
This file should be in the same directory as your feff.inp.
It usually has extension .pbs. For example, I have feff8.pbs
### Get the current environment:
#PBS -V

### Fixed output files
#PBS -e feff8.err
#PBS -o feff8.log

### Ask for number of nodes and processors per node
#PBS -l nodes=5:ppn=2
### Default queue:
#PBS -q default
### Go to working directory
cd $PBS_O_WORKDIR

### The number of processors assigned
export NPROCS=`wc -l < $PBS_NODEFILE`
### Print assigned nodes and processors
echo Job: $PBS_JOBID CPUs: $NPROCS on Nodes...  
export PATH=/usr/pgi/linux86/bin/":"$PATH
cat $PBS_NODEFILE

### Time my MPI job which description is in second script
time $PBS_O_WORKDIR/mpijob

I needed second script (mpijob), since I wanted to time the whole job
for benchmarking. Currently it is set to be in working directory,
but it can be moved to the Feff8 source directory, with
appropriate change in .pbs file.

Finally submit the job usnig:
  qsub feff8.pbs
or you can increase change the number of nodes withour rewriting .pbs
  qsub -l nodes=10:ppn=1 feff8.pbs
Watch the job:
  qstat -n
  qstat -a
  qstat -Bf
  qstat -Qf
  qstat -f JobID
  qstat -s JobID
The error messages and total time will be in feff8.err file,
while the feff8 screen output will be in feff8.log file.
Both files are specified in feff8.pbs.

The above is for MPICH job. To submit MPI LAM job you probably
will need to change only the compile script and mpijob script.
Try first the MPICH.

Let me know if you have any problems.

Best wishes
Alex
------


---------- Forwarded message ----------
Date: Fri, 21 Jun 2002 12:14:51 -0700
From: Alex Ankudinov <alex@leonardo.phys.washington.edu>
To: John J. Rehr <jjr@leonardo.phys.washington.edu>
Subject: mpijob script

The mpijob script is
#!/bin/sh
export PATH=/usr/pgi/linux86/bin/":"$PATH
mpirun  -np 1 -machinefile $PBS_NODEFILE ~/Feff8/rdinp
mpirun  -np $NPROCS -machinefile $PBS_NODEFILE ~/Feff8/ffmod1
mpirun  -np $NPROCS -machinefile $PBS_NODEFILE ~/Feff8/ldos
mpirun  -np $NPROCS -machinefile $PBS_NODEFILE ~/Feff8/ffmod2
mpirun  -np $NPROCS -machinefile $PBS_NODEFILE ~/Feff8/ffmod3
mpirun  -np 1 -machinefile $PBS_NODEFILE ~/Feff8/ffmod4
mpirun  -np 1 -machinefile $PBS_NODEFILE ~/Feff8/ffmod5
mpirun  -np 1 -machinefile $PBS_NODEFILE ~/Feff8/ffmod6

Alex

 
On Tue, 14 Oct 2003, Michel Jaouen wrote:

> 	Hi guys,
> 
> 	I'm currently running the parallel feff8 code under MPI on a 
> cluster (20 nodes, 2 processors pernode). The administrator has 
> defined various queues for job sumitting in batch and thus one for 
> parallel jobs (8 nodes available for, e.g. 16 processors). The 
> queuing system is running under PBS which hase some kind of nodes 
> files (or environment
> variables, which actually contains the names of nodes assigned to the 
> job). But it seems that MPI doesn't take it into account and it 
> happens that my jobs are waiting in some cases (when MPI asks for a 
> node which is active for a job from an other defined queue, even it 
> exists enough free processors).
> Therefore it seems I need a small "wrapper" which pastes this 
> information (the node file defined for PBS) into MPI. As anybody a 
> little shell script for doing that? (I'm not an expert at all with 
> MPI management).
> 
> 	Thanks for advice
> 
> 	M. Jaouen
>