OIT SysCore

As many of the campus web developers and content owners already know, OIT made some major changes on the weekend of February 24, 2007 regarding the configuration of it's central web serving environment as it relates to how various websites access the filesystem, and persmissions that they use. We implemented a security regime based around an Apache module called mod_waklog, which was originally developed at the University of Michigan, and has been twisted by us at UMBC for our own purposes.

The changes made probably seemed drastic, heavy-handed, without planning -- however, we had been planning on making these changes over a long time frame, working with various content owners and web developers to convert their sites to the new security regime as time allowed. Due to an immediate security concern, however, we had to accelerate our time table on the site conversions, and do them all in one swoop as soon as we technically and humanly could.

As of this evening, both ifs1.afs & ifs2.afs have been upgraded to Solaris 10 11/06, and now have their AFS filesystems hosted on ZFS.

~12:30PM on 3/12 the server App2.umbc.edu's JVM seemed to freeze up due to the JVM's memory overflow problem. App1 remained operational during this time. Bob and I took this opportunity to verify that assessments were working correctly while failing over the load balanced servers. Since this works, I'm going to set up a daily automatic JVM restart job that should
prevent these "memory leak overflows" from affecting us any more.

A few bugfixes were made to mod_waklog today. The most important was dealing with pathinfo'd applications, such as:

hughadhusdfasf/something/test.cgi/stuff/here/is/cool

Where test.cgi is the actual script that gets ran, and stuff/here/is/cool is extra data.

Apache doesn't always pass full configuration data to subrequests (which happen in this case), so you have to go looking for them in the original request.

(update: also had to fix another little bug that this fix ended up exposing regarding internal redirects.)

The mod_waklog source is now available from the mod_waklog wiki page:
http://www.umbc.edu/oit/iss/syscore/wiki/Mod_waklog

Enjoy, ya'll.

There was a brief down time on blackboard yesterday, 3/14 at ~3:07 PM
till ~3:13 This was due to the auto-restart script restarting the server.(app2)
This should have not had any effect on users. However, the other server
was not working at the time and this created the problem. I will be trying to figure out why the other server was down and add some additional logic
to the restart scripts to avoid this problem.

I've updated the SNMP configuraition of all of our hosts and santized their Cacti entries as well, so all Syscore servers should have stats collected on them.

I also got SNMP monitoring of our Qlogic fibre channel switches finally working, and so we're collecting errors and bytes in/out on a per-port basis now. Only switches 1-4 are being monitored at the moment. Switches 5-8 need to get a network cable hooked up to them so they can service SNMP queries.

Click the URL below to visit our public Cacti site:
http://stats.umbc.edu/syscore/cacti/graph_view.php

Last night Jason and I worked to upgrade our master LDAP server from Solaris 9 to Solaris 10. This is a special event because it means that all of our major core infrastructure servers are now running Solaris 10. There are a few servers which still run older Solaris versions, but they're slated to be discontinued, consolidated, or replaced with new services over the next several months.

Sudo, for all supported platforms (and IRIX -- ew), has been modified to include a shell script wrapper to call a "local" copy of sudo instead of the AFS-housed copy when one is available.

This change has been made to support some AFS client configuration changes that will be implemented on core systems in upcoming weeks.