OIT SysCore

Miltermonkey on the milter machines has been upgraded.

In my development, it was found that there were some mistakes made in shared variable locking when running it under a new version of perl (5.8.7) that weren't caught by warnings under 5.8.0. Since we now have a standard perl distribution
with threading enabled (5.8.6), fixes had to be made to the miltermonkey code to use it. Miltermonkey is now running on the standard /usr/local/bin/perl5.8.6 with the threading fixes that were found under 5.8.7.

As part of this, the sendmail binary on the mxout machines (smtp.umbc.edu) was upgraded to 8.13.1. It seems that the wire-level milter protocol was incompatible with the newer version of the Sendmail:Milter perl module -- which was done against the 8.13.1 milter library. No big deal, it had already been tested awhile back and was ready to roll out anyhow.

With the miltermonkey codebase upgraded, we'll be installing and running the open source virus scanner ClamAV to see how it performs up against our current (commercial) scanning solution -- which is slow, and never gets a signature until the worm has been around for days ;)

After a few students started having the session switching issue, which is really due to misconfigured webcach at the students ISP, I decided to double check our blackboard patch level, etc.

Turns out the blackboard app reported we were quite a bit behind. 6.2.3-6 We should be at 6.2.3-23

Darn! I could have sworn I did that upgrade at the end of summer.
Darn! Its even listed on my board and checked off.
Darn! Its even listed in this blog as being done on 8/19...
Wait a minute...

It seems the version number comes from a file that was
replaced with the old version, causing blackboard to display the old version number.

I'm still working on verifying that the SP3 upgrade was done correctly.

Blackboard is now undisputedly at 6.2.3.23 It was
rather nebulas as to just how much of the SP3 update
had taken place so the patch was applied again last night. (10/14)
Started backup stuff at 12:30 AM, slept part of the night, then applied the upgrade around 6:30 AM

We've added ClamAV to our production mail filters.

After running it for a few days on one of our filters, it didn't miss *one* email that our current virus scanner (McAffee) picked up. That, and it's faster. And free. And releases signatures faster then the commercial vendors. Here here!

We've taken our "old and crufty" web presence (www.gl.umbc.edu), and replaced it with a new and less crufty wiwi. (www.umbc.edu/oit/iss/syscore/wiki)

Thanks to Jason for getting MediaWiki online.

We've completely switched off McAffee on our mail servers, and are now exclusively using ClamAV...because...well...it totally rocks.

Our beloved blackboard test server is now running BB Version 7.0
This is mostly a backend upgrade to prepare the way for some really nice features to be release in 7.1 (~Decemeber release)

From the little time I've had to work with it, it still seems really nice. Security is improved. File sharing is set to really work allowing us load balance.

I think I got the collabaration tools working again
on our test BB 7.0 instance. Seems blackboard deleted
all of our custom auth files and cleaned out its libraries also.

We had an outage of hfs10,11 & 12 during the morning of 10/25. These three fileservers experienced the "thread starvation" problem which causes all clients that are accessing them to "hang". This was very quickly identified, and the fileserver processess were forcibly restarted. Unfortunatly,this meant the salvager had to run on all of the volumes -- which took between 30-45 minutes per server to complete. Service was restored to the affected volumes by 12:45pm.

Tonight I split the mirrors on the production AFS servers to upgrade the second of the two Apple XRAIDs's firmware to version 1.5. While doing that I also put in the proper LUN Masking for the three new AFS server that I built last night - bfs1, hfs1 and hfs2.

BFS1 will replace two very... um... mature AFS servers which serve out things such as everyone's web content and departmental spaces in AFS. It'll have a total of 1TB of mirrored disk space to allocate out.

HFS1 and HFS2 will replace the last of the old direct-attached SCSI AFS servers which serve everyone's home volumes. When these two come online, the HFS* servers will be 100% on the fabric.

I also spent part of the morning updating the zoning on the FC switches and getting them ready for the pending expansion into the PP building. This also entailed the installation of longwave SFP tranceivers into the switches here in ECS to make the distance across campus.

When the mirrors on the production AFS servers are done syncing tonight, I'll join the mirrors on the three new servers and then we'll plan the moving of volumes to BFS1, and also kick off the volume balancer script to move the home volumes off the old servers and on to the two new ones, and then go down and decomission them.

The syslog daemon on loghost has been switched to syslogng.

Syslogng totally rocks.

A couple things have changed:

* Current log files will be named /var/syslog/messages., where Day is like "Mon" or "Tue" or whatever. It'll automatically reset the logfile to the next day when it rolls around, and empty out last week's day-log when it's time to.

* Archives are now in /var/syslog/backup//..
They'll be in plain text for 10 days or so, then bzip'ed once they've gotten old enough. These are written in real-time with the current log file

* Playing with the idea of filtering out logs for various services (such as mail transport) to separate files. Look at /var/syslog/services/ for an example.

* The log format line has changed. It is now sane.
2005 10 27 16:36:01 -0500 mr6.umbc.edu [notice] imapd[12753]: maildir_open: /afs
/umbc.edu/users/t/g/tgindlin/Mail///inbox/cur

The date is the date of arrival to the syslog server, /not/ the date that the sending host decided to "send." Notice, we've got year and GMT offset!

However, on the other hand, syslog-ng takes up more CPU to do all of this coolness. So, I've ordered a new syslog server out of our "maintenance funds". A shiny, sparkly, new Sun X2100. "X" stands for X-treme.

We had some more oddness on the three AFS servers that were mentioned on Tuesday(?). Basically, we were able to watch them "busy out", all pretty much at the same time -- by busy, we really don't mean busy, they were busy sitting there doing "nothing" except for waiting for something to happen.

After a few minutes, that something would happen, and they'd go about their business. Then 15 minutes or so later, the same thing would happen.

We've never noticed anything like this on our OpenAFS 1.2.13 servers, which were running fine during this time. To test out the theory that this could be a problem, we did a "special build" of the fileserver & volserver processes for Solaris 10, and installed them on hfs12. We'll be watching hfs12 to see if it exhibits the same wankiness as it did before (we expect hfs10 & 11 will continue weirding out.) If it's fine while the others aren't, we'll 'upgrade' them to the version of the code we're running on hfs12.

Our restart of hfs12 to install the new software took much longer than expected because it didn't (or wouldn't) shut down cleanly. This could very well be related to whatever is going wrong...