FeffMPI

[James S Sims <jim.sims@nist.gov>]

I am pleased to announce that there is a new version of Feff available, Feff 8.2.4,
which uses the Message Passing Interface (MPI) to run on multiple processors on shared
or distributed memory machines (hence the alias FeffMPI). John Rehr and Chuck Bouldin 
have been reporting speedups of over 30 on IBM SP2 distributed memory clusters and on
a Linux cluster, respectively, so we think this is a significant advance over the
sequential version of the code.

The new version, 8.2.4, is CVS committed on tiziano.phys.washington.edu, and the 
most recent version is also copied to fefflib@leonardo.phys.washington.edu both as
a complete directory tree Feff824 and compressed, Feff824.tar.gz. The source is also
on gseaborg.nersc.gov under /u3/sims/Feff824.tar.gz and also in the ~sims home directory
on hudson.nist.gov.

I have attached a Readme (also in the distribution) which explains how to compile and
run both a sequential version and the new parallel version. To run in parallel, you need
multiple processors, of course, either on a single machine or distributed across many
machines, and you need the MPI library. On multiprocessor SGIs running IRIX, IBM SP2s,
and Linux clusters it is probably already installed. A good web site for MPI is

http://www-unix.mcs.anl.gov/mpi/

The library we use on our Linux cluster is LAM, which is part of RedHat Linux
and also available from

http://www.mpi.nd.edu/lam/6.4/

I will be happy to help with any problems you may have with installing and running
this version in parallel, whether it be the code itself or with MPI.

jim.sims@nist.gov


This directory contains source for making both
parallel and sequential versions of FEFF8.24,
based on dummy MPI codes contributed by J. Sims (NIST).


Two Makefiles are included,

Makefile.parallel   for parallel versions
Makefile.sequential for sequential versions

Use the the commands

make -f Makefile.sequential src 

or 

make -f Makefile.parallel src 

to generate sequential or parallel versions of source
modules 

ff2x_tot.f    genfmt_tot.f  path_tot.f    rdinp_tot.f
fms_tot.f     ldos_tot.f    pot_tot.f     xsph_tot.f

For single processor runs, these may be copied, for example,
into the Test/mod subdirectory

The modules can be compiled using for various architectures using
the feff_comp.ARCH scripts, also in the Test/mod subdirectory.
These scripts will output executables into the ../bin
subdirectory, i.e., if you run a script in the Test/mod
directory, look for executables in the Test/bin subdirectory.
Sample input and output files are in the Test/Cu GaN and Gecl4 
subdirectories.

Analogously, for parallel execution, the source modules generated 
from

make -f Makefile.parallel src

may be copied, for example, into the TestMPI/mod subdirectory,
where there are feff_comp.ARCH scripts for ARCH=aix and
ARCH=sgi, and a Makefile for the more complicated case of a
Linux cluster using the Portland Group compilers and the public
domain LAM MPI library.

There also is a TestMPI/run.scripts directory to illustrate
how to run parallel jobs. In the simple case of an interactive
run, the script feffmpi.sgi is

#!/bin/csh -f
mpirun -np $1 ../bin/rdinp
mpirun -np $1 ../bin/ffmod1
mpirun -np $1 ../bin/ldos
mpirun -np $1 ../bin/ffmod2
mpirun -np $1 ../bin/ffmod3
mpirun -np $1 ../bin/ffmod4
mpirun -np $1 ../bin/ffmod5
mpirun -np $1 ../bin/ffmod6

and to run the job all one has to do is use

feffmpi.sgi <Number_of_processors>

This is pretty straightforward, i.e, its just 

mpirun -np <Number_of_processors> Module_to_run
for each module, and the script is there for convenience.

The feffmpi.aix and feffmpi.lam (Linux) scripts are
analogous, where AIX uses poe (Parallel Operating Environment)
instead of mpirun and Lam uses mpirun with a different
collection of parameters.

An additional complication arises for batch queues, where it
is probably best to consult your local batch queue expert.
Sample batch queuing scripts are provided for 16 processor runs
using NQS on an SGI (script.qsub.16), using IBM's LoadLeveler
(script.loadleveler.16), and using the free Linux PBS system
for a Linux cluster (script.pbs.16).


1/26/01  J. Rehr
2/08/01  J. Sims