OIT SysCore

"license1.umbc.edu" has been upgraded to (a variant of) Redhat Enterprise 3. Meaning, the magical 'gendisk' system has been upgraded to generate a working
disk image of RHEL3. There are, however, some caveats.

RedHat is nice to put a lot of odd patches into their kernel -- significantly, the
"audit" subsystem and a backport of Thread Local Storage. As we use stock kernels
on our servers, and there aren't very good patches for these around, we don't have them built into our kernel.
The audit subsystem ain't no thang, as a link (created by devfs, of course) to /dev/null for /dev/audit seems to satisfy their libraries instances in being able to write audit records when people log in. TLS, as well, is not a problem, as moving aside the TLS-aware libc/pthread libraries seems to clear this up.

In the end, it's suffice to say that this admin is now officially only using redhat under duress. If your linux distro can't handle users putting a "stock"
kernel on it elegantly, your linux distro ain't my distro.

BB6.umbc.edu now points to the virthost webserver, which generates a redirect to bb-app6.umbc.edu. This replaces it's previous behaviour, as browsers have problems setting cookies...

The mail delivery system has been altered to "bounce" messages to accounts that are inactive, deactivated, or scheduled to be deleted. This change has been made both because of user requests, and an interest to stop filling up our disks with spam that will never be read.

While examining the impact of the mail delivery changes made yesterday, it became clear that we could do less. Meaning, as the determination of an account's status was being made by the MDA at the time of delivery, the a bounce message had to be generated, and sent, by our mail servers to the user. This determination would better be done at the MTA level, where the user's account state could be determined at address resolution time, and returned as an error state to the sending MTA during the SMTP transaction, leaving it with the job to notify the sending user...

We've enabled "TypeKey" authentication for comments on this site. TypeKey-authenticated users will have their comments auto-approved. Discussion is a good thing.

We'd like to webauth-enable the comments portion of MovableType, but it might take some work. One thing at a time :)

Our first production Solaris x86 server is on line, and it's not a fileserver as we'd originally planned.

"mr4.umbc.edu", a 2x Xeon Dell 2650 with 3G of RAM was jumpstarted to Solaris 10 yesterday afternoon, and is currently serving imap/pop service to over 400 users as part of the imap/pop service cluster. The other machines in the cluster (mr5 - 8) with similar hardware are currently running Linux.

So, if you're reading mail right now, there's a 1/5 chance you're using it...

The content for disabled accounts will no longer display on userpages.umbc.edu. This means that if you're account disabled, so is your free web hosting.

Very simple mod_perl module to do this:

package UMBC::HomePageAccess;

use Apache;
use Apache::Constants qw(:common HTTP_MOVED_TEMPORARILY DECLINE_CMD);
use Apache::Log;

sub handler {

my($r) = shift(@_);

my $uri = $r->uri;

if ( $uri =~ /^\/\~([^\/]+)\// ) {
my ($name, $passwd, $uid, $gid, $quota, $gcos, $dir, $shell) =
getpwnam($1);
if ( defined($passwd) ) {
if ( $passwd =~ /^\*(DEACTIVATED|INACTIVE|DELETED)\*$/ ) {
$r->custom_response(404, "/error_disabled.html");
return 404;

}
}
}

return DECLINED;

}

1;

Today we tried to upgrade our serial console server from the home-built rack mount machine running RedHat 6.1 that we put together something like 5 years ago to a reliable Dell 2450 we had around (old, but reliable) running RH Enterprise. This was "aborted", well, because it didn't really work out.

As it turns out, however, our Cyclades Y-series cards seemd to have an issue with running in the 2450. We have two different "style" cards, you see, and old-style full-height PCI card, and newer ones are a bit smaller. While they are supposed to be the same card, the "large" cards do not seem to work well in the presence of the "small" ones while in the 2450. Ended up going back to the old setup, and will see what Cyclades has to say about it.

We've started a process to purge "really old" accounts from our system; meaning, we're clearing out the files of accounts that have been deactivated for a really long time....

We periodically run a process which examines the space usage on our AFS home directory servers, and moves user volumes from one server to another in an attempt to balance the usage. However, just balancing on usage isn't enough. Recently, we've made some changes in the process to take into account other factors, such as the volume's average activity, in order to have servers with a load profile that is more even.

Ok, so it's this virus that's mutating at an alarming rate. It's that one that tells you that your account is being suspended.

Anyhow, I wrote up some rules that should atleast get the thing caught as Spam, so it won't be as annoying.

We've set up a Jabber server @umbc. It's in TESTING, not production, so there will probably be some changes, downtimes, all that fun stuff.

However, if you'd like to use it, feel free.

Jabber ID: username@umbc.edu
Password: Your MyUMBC/Kerberos Password
Connect Server: jabber.umbc.edu

Choose "Use TLS Encryption" in your client. If you don't, you won't be allowed to authenticate. "old style" straight SSL is currently disabled, because I can't seem to get it authenticating for some reason.

We're configured with appropriate message routing, so you can chat with folks in other Jabber realms.

When we go "production" with this service, we'll be mucking with the Jabber ID portion of the system. You might have to "choose" your JabberID from MyUMBC before you use Jabber for the first time, or something like that. We'll let ya know...

This is UMBC's internal CA certificate, currently just used internally, but not for long.

To all ya'll console server users:

In the past week, there have been some significant changes going on with the console server, so here's a quick update, and what to expect in the ensuing couple of weeks.

(read more!)

A major upgrade to myUMBC went into production today. The intention, and hope, is that this change will be transparent to users. Please let us know if you encounter any strange behavior.

This position is open to UMBC STUDENTS only!

Position Name: Junior Systems Administrator
Department: UMBC Office of Information Technology, Core Systems
Location: On-campus, ECS201
Hours: Up to 20 hours per week
Compensation: Commensurate with experience

Duty Profile:
The Junior Systems Administrator covers a wide-range of applications and duties in support of the day-to-day operations of the Core Systems (Syscore) group. These duties include:

• Processing and resolving student, staff, and faculty requests and issues pertaining to the systems and services operated by Syscore.
• Programming (Bourne Shell, Perl)
• Installation and maintenance of software packages.
• Installation and maintenance of Syscore server hardware.

The preferred candidate for this position should be able to demonstrate, at the minimum, basic working knowledge in the following areas:

• UNIX OS administration (Linux, Solaris)
• System services (SSH, Sendmail, Apache Web Server)
• Basic concepts of AFS file services
• Security
• Hardware handling
• Problem solving
• Ethics

Please submit your resumé with cover letter in PDF or Word format to Dale Ghent for consideration. Applications will be accepted until the position is filled.

We're currently in the process of upgrading user quotas. Quotas will be bumped to a minimum of 250Mb for faculty, staff, and graduate students. Everyone else is being bumped to a 100Mb minimum.

I moved the listproc Apache server from the local one to the "Uber" build 003. This was done to bring listproc in-line with the rest of our web servers and also because I needed FastCGI for testing Sympa, which is a possible replacement for the old CREN ListProc software currently in use for UMBC mailing lists.

Our scheduled outage to upgrade our LDAP directory service was a complete success. Well within our downtime window, the master directory server was upgraded from Sun iPlanet 4.16 to Sun Java Directory Server 5.2, and moved to new hardware. Our directory replicas were also upgraded at the same time; these are still running on their old hardware, as we have a planned migration of these to newer systems later this year.

Tidbit One

The new version of the Sun One Directory Server defaults to using the {SSHA} password hash format, as opposed to the old {SHA} hash. Any code that directly verifies password hashes needs to be modified appropriately. Some code to do it is included in the extended article if you need it.

Tidbit Two

With that, the KDC/LDAP password sync has been re-enabled. Had disabled it for the DS upgrade just incase we seriously horked something :)

We're currently testing "copying" the /usr/afsws & /usr/k5 trees, via cfengine, to local drives instead of linking to them in AFS. This is a precursor to taking advantage of using shared Kerberos & AFS libraries instead of always statically linking these apps. This will make updates to Kerberos for bugs & security much easier, as applications which use them will not need to be rebuilt. In case of AFS service troubles, applications which make use of these libraries will continue to be available -- as their dependant libraries are not mounted in AFS.

A small (hopefully unnoticable) change was made to the cfengine configuration, moving the disable action before the copy. This allows the removal of the /usr/afsws & /usr/k5 symlinks before the recursive copy operation. This allows for a continued "simple" afs client installation and configuration (by just creating the links), and allows cfengine to do the heavy lifting once the machine is finally configured.

Right now I'm currently testing this under linux only -- I'll expand it to the platforms we care about (solaris sparc & solaris x86) next week.

update

The Weather function in Webmail has been disabled. It pulls its data from a NOAA website which apparently underwent a large change recently, preventing the Weather function from collecting data correctly. Until further notice, this function has been deemed superfluous to Webmail's purpose and has been disabled.

It may return at a later date, but there is no guarantee of that happening. It's a very low-priority thing, especially at this time of year.

Added server-side TLS on the mxin.umbc.edu machines. Encrypting mail is good, ya.

(now we just need to toss that out to all of the clients! :) )

www.umbc.edu can now serve content via https. Currently, access to https URLs will simply return a redirect back to the non-https content. Areas can be configured so that they accept https connections in the 'umbc_ssl.conf' configuration file (it's obvious, see examples...)

Update
Also enabled the php accellerator, which as installed, but never turned on for www.umbc.edu's sites.

Planning on making the following changes to the mxout (smtp.umbc.edu) server cluster.

• Enabling multiple queues based on intended delivery address; filtering stuff destined for the "central" OIT mail servers and stuff destined elsewhere


• Switching to using persistant queue runners. There will be two configurations; one for the "external" mail that takes advantage of host connection statistics, and one for the internal that doesn't.
  

This is a presentation given at a NERCOMP Identity Managment SIG regarding the basics of UMBC's SSN migration plans over the next year or so. It overly simplifies some things as to not get mired down into details, but the basic outline of the problem and intended solution is there.

As with everything posted on this blog, this is not official OIT communication. This was a presentation given as a case study, and not yet officially OIT's plans. Anything you read is subject to change without notice.

Go Go Powerpoint, Away!

We've taken our "old and crufty" web presence (www.gl.umbc.edu), and replaced it with a new and less crufty wiwi. (www.umbc.edu/oit/iss/syscore/wiki)

Thanks to Jason for getting MediaWiki online.

The syslog daemon on loghost has been switched to syslogng.

Syslogng totally rocks.

A couple things have changed:

* Current log files will be named /var/syslog/messages., where Day is like "Mon" or "Tue" or whatever. It'll automatically reset the logfile to the next day when it rolls around, and empty out last week's day-log when it's time to.

* Archives are now in /var/syslog/backup//..
They'll be in plain text for 10 days or so, then bzip'ed once they've gotten old enough. These are written in real-time with the current log file

* Playing with the idea of filtering out logs for various services (such as mail transport) to separate files. Look at /var/syslog/services/ for an example.

* The log format line has changed. It is now sane.
2005 10 27 16:36:01 -0500 mr6.umbc.edu [notice] imapd[12753]: maildir_open: /afs
/umbc.edu/users/t/g/tgindlin/Mail///inbox/cur

The date is the date of arrival to the syslog server, /not/ the date that the sending host decided to "send." Notice, we've got year and GMT offset!

However, on the other hand, syslog-ng takes up more CPU to do all of this coolness. So, I've ordered a new syslog server out of our "maintenance funds". A shiny, sparkly, new Sun X2100. "X" stands for X-treme.

Been making some changes today to our Kerberos configuration -- adding support for some encryption types *other* than the slightly out-of-date DES ;)

"out of date DES" is still the default until I've verified that all of the older 'aklog' binaries have been updated to support the new encryption types; the kerberos libraries that the builds were linked against were rather moldy and oldy, so they don't contain support for the newer enctypes that they'll be seeing. In addition, the newer builds of aklog will support addressless tickets, which is the flavor of the month in KRB5 land.

Kendrick Hernandez joined the Syscore team this week! Kendrick comes to us from the Help Desk and will help with ticket resolution and administration of the servers. Welcome, Kendrick!

We made some memory upgrades/configuration changes on our mail readers to increase their capacity and snappiness on Tuesday & Wednesday (12/13 & 12/14).

The syscore wiki, 'http://www.umbc.edu/oit/iss/syscore/wiki/', has been upgraded to mediawiki version 1.5.5 from 1.5.0. This was a security update, so there shouldn't be any noticeable changes. If you notice anything broken, please let me know.

Jason

We're in the process of upgrading our ''ifs'' AFS fileservers with the hardware that used to make up HFS8 & HFS9. These are Dell 2650 servers with direct-attached PowerVault arrays.

This upgrade moves us away from the home-built Kingston drive arrays that have been used in the past, doubles our IFS storage capacity, and moves to a faster hardware platform. (the old IFS servers consisted of 2 sun Netra T1 machines, and a home-built Linux box)

The new "ifs2" is currently online, and we're bringing online the new "ifs1" online right now as "ifs5".

Our Spam mail filter has been modified to correctly support the whitelisting of SMTP-AUTH authenticated mail on our servers. Therefore, sending mail using 'smtp.umbc.edu' and authenticating to it using SMTP AUTH will allow your mail to bypass our spam flagging (and filtering.) This was the behaviour in our pre-January mail environment, and was an oversight when converting to our new system.

After doing some optimization to get our NIS map "refresh" process to take less than 3 hours -- it takes 15 minutes now -- we're now refreshing the contents of our maps nightly.

This means that accounts that are supposed to be "deactivated" will actually be deactivated when they're supposed to be, and "leavins" left from account renames and other odd operations will also get cleaned up.

The problems causing the mail servers to periodically "spaz" and cause one or two of them to not accept mail correctly might actually be fixed now. There was an incorrect usage of a mutex around some socket code in the Spam & AntiVirus filtering modules which caused various threads to trounce on themselves.

Cacti is now back in operation on a beefier box (Sun X4100).

http://stats.umbc.edu/syscore/cacti/graph_view.php

www1 and www3 serve www.umbc.edu
www4 serves userpages.umbc.edu
uportal1 & 2 serve myUMBC
grimm is the MySQL db server

Not all hosts we have are listed there, namely Linux servers because of problems with Net-SNMP on that platform.

UMBC now has a Shibboleth IDP, currently part of the "InQueue" federation.

It is currently in the testing stages (could go down without notice) -- there's still some configuration and stuff to be done.

To experience it, visit "http://wayf.internet2.edu/InQueue/sample.jsp"
and choose Univeristy of MD, Baltimore County from the "where are you from" list. You'll log in with the familiar MyUMBC login screen -- and your assertion as a member of the UMBC community will be accepted by the remote application.

Doesn't sound too impressive when it just works; see http://shibboleth.internet2.edu/ for more information.

console.umbc.edu now has console entries for the mgmt interface, and serial console interfaces on all of the Sun V20z servers we currently have in production. To access the managment interface, do a 'sudo console SERVER.cmgmt'.

Please, when putting any new server on line, please make sure you can get the console by typing "console SERVERNAME" on the console server; this single-point of interface makes managing our servers much easier, and much less of a headache than trying ot figure out which random managment interface you need to connect to, and what magic incantations you need to type to get access to the console output. It also make sure our console output is logged in one place, so that when a server crashes it's easy to troubleshoot why.

[UPDATE]
Also have the consoles working for the X4100s as well. Currently they're running over the serial line, however, they can be converted to use the ssh-based managment port at some point, it's just a matter of figuring out how to add the ssh keys in the totally weird CLI they've created.

We've had a rash of machines on the wireless network infected with spam bots and such. We've change our mail relaying ACLs to require that folks sending mail through our smtp.umbc.edu relay from the wireless be "authenticated" in the same way that external IP addresses are, this means they must either:

• Use SMTP AUTH

or

• Have used IMAP or POP to read mail through the imap/pop.umbc.edu servers within an hour before attempting to send.
  

A drive in one of our AFS fibre channel RAID arrays crapped out recently, but due to the highly redundant hardware setup of our SAN and the individual AFS file servers, there was zero impact to users. In fact the drive in question, one of 14 400GB drives in the array, has been dead for a few weeks before we got a replacement in and did the swap this evening.

ff1-raid1 is in the PUP building, so I swung by there on my way home. It was a simple matter of pulling the bad drive, sticking in the new one, and walking back to my car. The RAID array automatically picked up on the new good drive and assimilated it without requiring any input from me.

I've been silent on the blog ( a lot of us have ) for the past few weeks as we've been caught up in the Campus Card project and a lot of other work relating to the campus identity management system (IDMS) in support of these goings on.

The work that was done involved integrating CSGold system that is ran by Communications Services, the Identity Management sytem, and the two campus door access systems ran by Comm Services & Lenel. The Campus IDMS feeds identity information into the CSGold system, but also consumes ISO ID (Magstripe) and LIMS # assignments -- the Campus Card system assigns a new, unique ISO ID & LIMS ID whenever a new card is printed for an individual. It pushes identiy information and ISO ID changes out to the campus door access systems in near-real-time -- so, if you get a new card, you'll still be able to open all of the doors you were able to.

Since the campus IDMS is now the source of identity information for the ID Card production, some extensions had to be made to it's management tools so that various groups on campus could be extended the ability to assert affiliations, or, add identities to the system, for those individuals that need to receive these cards. We're currently in the process of documenting the procedures for the folks that administer these "affiliated programs" to manage their consituent's data in the IDMS. The goal is that when someone shows up at the Campus Card office, their data already exists in the system, and their card can just be printed.

Mail filters have been modified to 'whitelist' internally originating email; meaning mail that is originating from a 130.85.* address where our mail exchangers are the first hop.

The new password construction rules for MyUMBC accounts are now in effect, in preparation for the upcoming forced password change for administrative users. As you may, or may not know, this is in response to the state audit finding that UMBC's password authentication practices did not meet the requirements of the State of MD IT Security Policy. These password rules are enforced for changes made through the MyUMBC web interface, and through Kerberos' password change protocol.

Note: We are not yet synching MyUMBC account password changes to active directory -- that will be forthcoming.
New password rules.

Must be at least 8 characters in length
May only contain printable characters (alphanumeric and symbols -- no spaces or "control" characters)
Must include at least one upper case, lower case, and one non-alphabetic character
Must not contain a sequence of three or more recurring characters. (such as 'sss')
Must not contain a sequence of three or more characters in the order that they appear on standard keyboard layouts.
Passwords must not be primarily constructed of dictionary words.
Passwords must not contain personally identifiable information that is known to UMBC. This includes such elements as the individual's username, CampusID, Social Security Number, Date of Birth, or elements of other directory information such as address, office number, etc.
Passwords must not match the individual's password history, which comprises of the passwords used for this account over the past year, or, the last three passwords used for this account.

Last week, as many of you know, OIT's primary datacenter suffered a very brief power outage taking many of our services off-line. SysCore managed systems weathered this outage rather well, however, there were some lessons both good and bad that we took away from the experience that may help us in the future.