UMBC High Performance Computing Facility

Please note that this page is under construction. We are documenting the
240-node cluster maya that will be available after Summer 2014.
Currently, the 84-node cluster tara still operates independently,
until it becomes part of maya at the end of Summer 2014.
Please see the 2013 Resources Pages under the Resources tab for tara information.

How to use PETSc on tara

- /usr/cluster/contrib/gcc-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/
- /usr/cluster/contrib/gcc-openmpi-1.3.3/petsc-3.0.0-p9/cxx-opt/
- /usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/
- /usr/cluster/contrib/pgi-9.0-openmpi-1.3.3/petsc-3.0.0-p9/cxx-opt/

[araim1@tara-fe1 petsc-poisson]$ wget http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-current/ src/ksp/ksp/examples/tutorials/ex13.c

EXECUTABLE := petsc_ex13 OBJS := ex13.o CFLAGS := -O3 -c99 -Minform=warn -fastsse INCLUDES := -I$(PETSC_DIR)/include LIBLOCS := -L$(PETSC_DIR)/lib LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm CC := mpicc $(INCLUDES) %.o: %.c %.h $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) -c $< -o $@ $(EXECUTABLE): $(OBJS) $(CC) $(CFLAGS) $(DEFS) $(INCLUDES) $(OBJS) -o $@ $(LIBLOCS) $(LDFLAGS) clean: -rm -f *.o $(EXECUTABLE)

Download: ../code-2010/petsc-poisson/Makefile

INCLUDES := -I$(PETSC_DIR)/include LIBLOCS := -L$(PETSC_DIR)/lib LDFLAGS := -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm

[araim1@tara-fe1 petsc-poisson]$ export PETSC_ARCH=cxx-opt [araim1@tara-fe1 petsc-poisson]$ export PETSC_DIR=/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt [araim1@tara-fe1 petsc-poisson]$ export LD_LIBRARY_PATH=:/lib [araim1@tara-fe1 petsc-poisson]$

Note that you need to set the environment variables PETSC_ARCH, PETSC_DIR,
and LD_LIBRARY_PATH in each tara session, any time you want to use PETSc.
To automate this, add the three above lines into

Now we should be able to compile the example
/home/<your_username>/.bash_profile

[araim1@tara-fe1 petsc-poisson]$ make mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include -O3 -c99 -Minform=warn -fastsse -c -o ex13.o ex13.c mpicc -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include -O3 -c99 -Minform=warn -fastsse -I/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/include ex13.o -o petsc_ex13 -L/usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib -lpetsc -lpetscvec -lpetscmat -lpetscksp -lpetscdm [araim1@tara-fe1 petsc-poisson]$

[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 m 6 n 7 error norm 0.000925256 m 6 n 7 error norm 0.000925256 [araim1@tara-fe1 petsc-poisson]$

[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 -help Solves a variable Poisson problem with KSP. -------------------------------------------------------------------------- Petsc Release Version 3.0.0, Patch 9, Fri Oct 23 13:37:28 CDT 2009 The PETSc Team petsc-maint@mcs.anl.gov http://www.mcs.anl.gov/petsc/ See docs/copyright.html for copyright information See docs/changes/index.html for recent updates. See docs/troubleshooting.html for problems. See docs/manualpages/index.html for help. Libraries linked from /usr/cluster/contrib/pgi-9.0-mvapich2-1.4rc2/petsc-3.0.0-p9/cxx-opt/lib -------------------------------------------------------------------------- Options for all PETSc programs: ... (tons of options) ... [araim1@tara-fe1 petsc-poisson]$

[araim1@tara-fe1 petsc-poisson]$ ./petsc_ex13 -ksp_view KSP Object: type: gmres GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement GMRES: happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000 left preconditioning PC Object: type: lu LU: out-of-place factorization matrix ordering: nd LU: tolerance for zero pivot 1e-12 LU: factor fill ratio needed 2.38043 Factored matrix follows Matrix Object: type=seqaij, rows=42, cols=42 package used to perform factorization: petsc total: nonzeros=438, allocated nonzeros=438 not using I-node routines linear system matrix = precond matrix: Matrix Object: type=seqaij, rows=42, cols=42 total: nonzeros=184, allocated nonzeros=210 not using I-node routines m 6 n 7 error norm 0.000925256 ... [araim1@tara-fe1 petsc-poisson]$

#!/bin/bash #SBATCH --job-name=petsc_ex13 #SBATCH --output=slurm.out #SBATCH --error=slurm.err #SBATCH --partition=develop #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 srun ./petsc_ex13

Download: ../code-2010/petsc-poisson/mvapich2.slurm