UMBC logo
UMBC High Performance Computing Facility
Please note that this page is under construction. We are documenting the 240-node cluster maya that will be available after Summer 2014. Currently, the 84-node cluster tara still operates independently, until it becomes part of maya at the end of Summer 2014. Please see the 2013 Resources Pages under the Resources tab for tara information.
How to use PETSc on maya

Introduction

The PETSc (Portable, Extensible Toolkit for Scientific Computation) library "is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations." It provides a useful framework of parallel data structures, solvers, preconditioners, etc, so that you can focus on solving your PDE problem. In this tutorial, we'll show how to build and run one of the examples that comes with PETSc, which solves a "variable Poisson" problem. We will not discuss the details of problem here, we will just focus on getting the example to run on maya. Before you begin, make sure to read the tutorial for compiling C programs.

Where to find documentation

PETSC is thoroughly documented by its authors at the PETSc home page. On maya, PETSc is installed in the following locations (corresponding to the currently available MPI/compiler combinations): Each location has an "include" directory with PETSc header files which you can browse.

Example

The example we'll use is ex13.c . An easy way to get the example to your maya account is with wget. Notice that we are retrieving the plain C file, and not the HTML-ized version.
[araim1@maya-usr1 petsc-poisson]$ wget http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/
src/ksp/ksp/examples/tutorials/ex13.c
(The command should be one continuous line). The next thing we'll need is a Makefile
EXECUTABLE := petsc_ex13
OBJS := ex13.o

CFLAGS := -O3 -c99 -Minform=warn -fastsse

INCLUDES := -I$(PETSC_DIR)/include
LIBLOCS := -L$(PETSC_DIR)/lib
LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm

CC := mpicc $(INCLUDES)

%.o: %.c %.h
    $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) -c $< -o $@

$(EXECUTABLE): $(OBJS)
    $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) $(OBJS) -o $@ $(LIBLOCS) $(LDFLAGS)

clean:
    -rm -f *.o $(EXECUTABLE)


Download: ../code/petsc-poisson/Makefile
The most important lines, which will allow us to use the PETSc library are:
INCLUDES := -I$(PETSC_DIR)/include
LIBLOCS := -L$(PETSC_DIR)/lib
LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm
Notice that the environment variable PETSC_DIR is used, but not defined in this file. We will define it in our session before proceeding (if you forget to do this, compiling and execution will fail!). We will assume you're using the default PGI+MVAPICH2 switcher setting, so execute the following commands.
[araim1@maya-usr1 petsc-poisson]$ export PETSC_ARCH=cxx-opt
[araim1@maya-usr1 petsc-poisson]$ export PETSC_DIR=/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt
[araim1@maya-usr1 petsc-poisson]$ export LD_LIBRARY_PATH=:/lib
[araim1@maya-usr1 petsc-poisson]$ 
Note that you need to set the environment variables PETSC_ARCH, PETSC_DIR, and LD_LIBRARY_PATH in each maya session, any time you want to use PETSc. To automate this, add the three above lines into

/home/<your_username>/.bash_profile

Now we should be able to compile the example
[araim1@maya-usr1 petsc-poisson]$ make
mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
-O3 -c99 -Minform=warn -fastsse   -c -o ex13.o ex13.c
mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
-O3 -c99 -Minform=warn -fastsse  
-I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include 
ex13.o -o petsc_ex13 -L/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib 
-lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm
[araim1@maya-usr1 petsc-poisson]$ 
Since this is a small example, we can try to run it directly on the command line
[araim1@maya-usr1 petsc-poisson]$ ./petsc_ex13 
m 6 n 7 error norm 0.000925256
m 6 n 7 error norm 0.000925256
[araim1@maya-usr1 petsc-poisson]$ 
The example ran, but all we got at the end was the error norm. Let's see what other kinds of things we can do with the example by specifying the "-help' flag.
[araim1@maya-usr1 petsc-poisson]$ ./petsc_ex13 -help
Solves a variable Poisson problem with KSP.

--------------------------------------------------------------------------
Petsc Release Version 3.0.0, Patch 9, Fri Oct 23 13:37:28 CDT 2009
       The PETSc Team
    petsc-maint@mcs.anl.gov
 http://www.mcs.anl.gov/petsc/
See docs/copyright.html for copyright information
See docs/changes/index.html for recent updates.
See docs/troubleshooting.html for problems.
See docs/manualpages/index.html for help. 
Libraries linked from /usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib
--------------------------------------------------------------------------
Options for all PETSc programs:
... (tons of options) ...
[araim1@maya-usr1 petsc-poisson]$ 
To illustrate one option, let's try "-ksp_view" to view the parameters of the linear solver
[araim1@maya-usr1 petsc-poisson]$ ./petsc_ex13 -ksp_view
KSP Object:
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
  left preconditioning
PC Object:
  type: lu
    LU: out-of-place factorization
      matrix ordering: nd
    LU: tolerance for zero pivot 1e-12
    LU: factor fill ratio needed 2.38043
         Factored matrix follows
        Matrix Object:
          type=seqaij, rows=42, cols=42
          package used to perform factorization: petsc
          total: nonzeros=438, allocated nonzeros=438
            not using I-node routines
  linear system matrix = precond matrix:
  Matrix Object:
    type=seqaij, rows=42, cols=42
    total: nonzeros=184, allocated nonzeros=210
      not using I-node routines
m 6 n 7 error norm 0.000925256
...
[araim1@maya-usr1 petsc-poisson]$ 
Running the example through the batch system requires no special effort. As long as your environment variables are set correctly, your executable should be able to find the PETSc runtime libraries. As a quick example, the following batch script should be sufficient.
#!/bin/bash
#SBATCH --job-name=petsc_ex13
#SBATCH --output=slurm.out
#SBATCH --error=slurm.err
#SBATCH --partition=develop
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

srun ./petsc_ex13

Download: ../code/petsc-poisson/mvapich2.slurm