UMBC High Performance Computing Facility
System Description for tara
The cluster tara is an IBM Server x iDataPlex which
was deployed in November 2009, with the release to HPCF users
following in January 2010. Read on for a hardware-level description of this
system. For more information about using your account on the system, see
The name tara originates from a
and means "star". tara is the second HPCF cluster named from Hindu
scripture - the first was
A few photos of the cluster
tara has a total of 86 nodes which fall into four different categories. There
is one front end node, one management node (for admins only), two development
nodes, and 82 compute nodes. Each node features two (2) quad core Intel
Nehalem X5550 processors (2.66 GHz, 8192 kB cache), 24 GB of memory, and a
120 GB local hard drive. All nodes run the standard UMBC distribution of the
Linux operating system, Redhat Enterprise Linux 5. Attached to the cluster is
160 TB of central storage.
The Intel Nehalem processor series that is used in tara features some
innovations that should be very beneficial to scientific computations.
Specifically, each processor on a node has its own memory channels to
its dedicated local memory, which should improve the speed of memory access.
Other improvements by Intel aim at optimizing cache and loop performance
and at improving head efficiency;
for more information about Nehalem see
provides a useful comparison between the various models of the chip that
Our own initial tests bear out the performance improvements of the
processors, since they demonstrate that all cores on a node can be used
simultaneously with profitable performance;
see the Technical Report
More details on the different types of nodes in tara:
Management node -
There is one management node, which is reserved for administration of the
cluster. It is not visible to users.
Front End node -
This is the node users of the cluster interact with directly.
Users access all their files, edit code, compile jobs, and submit jobs to
the scheduler from this node. It is reached
through SSH by logging in to tara.rs.umbc.edu, and is the only node directly
accessible from the outside network (that is, from outside of the cluster).
Once logged into the front end node, it's hostname will appear as
Compute nodes -
These nodes are where the majority of the computing will take place. You do
not normally interact with these nodes directly. Instead, you submit jobs
to the scheduler while logged onto the front end node.
In turn, the scheduler will allocate your job to available processors
on the compute nodes.
The hostnames of the compute nodes are
n3, ..., n84.
you could connect to them from the front end node (e.g. "ssh n84").
However, SSH access to the compute nodes is disabled to help maintain
stability of the cluster.
In general, any use of resources on the compute nodes should be allocated
by the scheduler. Advanced users should be very careful when doing things
such as spawning child processes or threads - you must ensure that you
are only using the resources allocated to you by the scheduler.
Development nodes -
These are special nodes which are dedicated to running code that's
in development. This allows users to test their programs, and not interfere
with production runs. These nodes can be access through the 'develop' queue.
Development jobs are expected to be small in scale, but rerun frequently as
you work on your code.
The availability of two development nodes allows you to try several
useful configurations: single core, several cores on one processor,
several cores on both processors of one machine, all cores on both machines,
etc. The hostnames of the development nodes are n1 and n2.
Two networks connect all components of the system:
For communications among the processes of a parallel job, a high
performance InfiniBand interconnect with low latency and wide bandwidth
(quad data rate) connects all nodes as well as connects to the central
storage. This amounts to having a parallel file system available during
computations. Ideally the system can achieve a latency of 1.2usec to transfer
a message between two nodes, and can support a bandwidth of up to 3.5 gigabytes
per second. For more information about the network components, see:
A conventional Ethernet network connects all nodes and is used for
operating system and other connections from the front end node to the compute
nodes that do not require high performance. It also connects the long-term
storage (see below) to the cluster.
There are a few special storage systems attached to the cluster, in
addition to the standard Unix filesystem. Here we descibe the areas
which are relevant to users of the cluster. See
Using Your Account
for more information about how to access the space.
Home directory -
Each user has a home directory on the /home partition. This partition is
200 GB and its data is backed up by DoIT. Since the partition is fairly
small, users can only store 100 MB of data in their home directory.
Scratch Space -
All compute nodes have a local /scratch directory and it is generally about
100 GB in size. Files in /scratch exist only for the duration of the job that
created them. This space is shared between all users, but your data is accessible
only to you. This space is safer to use for jobs than the usual /tmp directory,
since critical system processes also require use of /tmp
Tmp Space -
All nodes (including the front end node) have a local /tmp directory and
it is generally 40 GB in size. Any files in a machines /tmp
directory are only visible on that machine and the system deletes them
once in a while. Furthermore, the space is shared between all users. It is
preferred that users make use of Scratch Space over /tmp whenever possible.
Central Storage -
Users are also given space on this storage area which can be accessed from
anywhere on the cluster. Spaces in central storage area are available for
personal workspace as well as group workspace. Full backups are maintained for
some of these spaces, while others have nightly snapshots taken which are kept
for 10 days.
UMBC AFS Storage on tara -
Your AFS partition is the directory where your personal files are stored
when you use the DoIT computer labs or the gl.umbc.edu login nodes. The
UMBC-wide /afs can be accessed from tara.