OIT SysCore

The KVM switches in racks A3, C1, E3, E4 are currently not functioning. This covers mostly windows servers. There are problems with the pins in some of the cables.
New cables have been ordered and the firmware has been upgraded in the master switch. The cables should arrive by the end of this week.

Here it is! A slightly-belated, much anticipated explanation of myUMBC's issues last week. Short story: myUMBC didn't exactly "break" per se, but it was acting a little flaky due to a couple of separate, unrelated circumstances. The symptoms were: sporadic session timeouts, where users would click on a link and get reprompted for username/password, even if they had not been idle; and also on Friday, there was some weirdness with registration, where myUMBC was not allowing students to register even though their appointments had already passed, making them allegedly eligible.

Both of these issues were caused by time inconsistencies on separate back-end systems. The first issue was due to the Oracle database server time being off 1 hour (not accounting for daylight savings time). The second was the fault of the HP 3000 mainframe, our back-end system of record for all student data. The time on this server was off by 22 minutes, causing the HP to report incorrect student registration appointment times.

There was no correlation between these two separate incidents. Both should now be resolved. I'm writing it off to those wacky leprechauns who live under the floor in the computer room.

Some users also noticed flakiness with the Degree Navigation application last week. The app was giving sporadic "internal server error" pages. This was related to incident #1 (the time discrepancy on the Oracle database server). The Degree Navigation code had a bug which caused it to barf out when it encountered a session timeout condition. This has now been fixed as well.

Hope that all made sense.. It's Monday and I'm still half awake!

We've brought hfs10, one of the new Sun V20z fileservers on-line, and are starting to migrate users to it for some good old-fashioned live load testing. Perhaps you're lucky enough to be one of them...

We've brought the second of our Sun V20z/Xraid backed fileservers on line. The first, hfs10, has been performing quite well -- so we figured we'd fire up the second.

Update: Just a reminder -- *someone* configured an alternate interface on this server (an internal, 192.* address). However, this causes the AFS fileserver to register that address as a valid address in the VLDB. An entry in the server's /usr/afs/local/NetRestrict file had to be added, and the server restarted.

Our AFS backup system has been moved to the new Public Policy building datacenter! With the exception of the network gear (and our serial-port server installed last week), it's the first production system located there!

Thanks to Steve & Dale for trudging the gear over there!

The SGI Indy, cgi.umbc.edu has been retired. The "cgi.umbc.edu" virtual host is serving specific redirects and such as required for backwards compatibility. No longer is the server "cgi.umbc.edu" being used to run, well, cgis. Those are done from your user account ;)

If there's any wackyness, please let us know.

The linux login servers are in the process of being upgraded to our wacky build of RedHat Enterprise Linux 3. Linux1 & 2 are complete, linux3 is currently having it's disk built and will be on-line soon. Please read on for caveats and whathaveyous.

We've completely switched off McAffee on our mail servers, and are now exclusively using ClamAV...because...well...it totally rocks.

Today I moved the volumes being served by bfs5.afs.umbc.edu to bfs1.afs, a new Sun V20z which is connected to our Core Storage Fabric which has 1TB of mirrored disk online. This removes one more of the old Gen 2 linux AFS servers from our machine room.

Next week, the contents of bfs3.afs will be moved to bfs1, allowing us to turn off bfs3 and its multiple A5200 JBOD arrays in an effort to conserve power and reduce the cooling load for the ECS data center.

curly.umbc.edu has been retired, and replaced with grinder.umbc.edu, a Sun X2100.

Last weekend, I learned that myUMBC (among other things) was going down every night from roughly 9:05pm till 9:20pm. I had no idea this had been happening, so the night before last I signed on around 9:00pm to see what was going on. Sure enough, at 9:05 sharp, it hung. I got stack traces from all of the myUMBC jobs, and found they were all hanging on calls to the Oracle DB. The next morning I talked to one of our DBAs. Turns out there was a backup job running during that time. The DBAs shuffled the backup jobs around, and last night there were no problems. Dunno how they fixed it, but if it works now, that's all I really care about :-) I plan on keeping an eye on things for awhile, to see if the problem turns up again, but for now I'm calling it fixed.

We've moved the following servers to the Public Policy datacenter:

* webauth3.umbc.edu
* ifs2.afs.umbc.edu
* kerberos2.umbc.edu
* db3.afs.umbc.edu
* ldap-rep2.umbc.edu (will be an LDAP directory replica)

We have brought online a new www.umbc.edu server -- www3.umbc.edu -- to replace the www2.umbc.edu server (an old Sun E250). www3 is a Sun V210 that used to be milter4.umbc.edu, and is the same configuration as www1.umbc.edu.

This part of the www.umbc.edu loadbalanced cluster is housed in the Public Policy datacenter.

We've been tracking the status of our AFS servers since their "problems" at the end of last week, and have nailed the issue down to a redux of some of the problems we had in the fall due to thread resource contention. While we had fixed 99% of the problems last semester by working around the code that was causing the contention, there still seems to be some instances where a state can be reached which causes thread deadlock. We're working with some of the OpenAFS folks on things to look at to further reduce the instances of deadlock.

The user web server, userpages.umbc.edu, was upgraded today to a Sun 280R which used to be milter1. It is now running on www4 and the old server, www9, has been decommissioned.

Tapestud has been upgraded to Solaris 10. The old system disk is still in it as c0t0, will set up mirroring to a new disk in a couple days after everything has worked fine.

Titan lost another 4 processors today. Sad, so sad. Really. I'm sad.

Our two LDAP server replicas (ig88 & dengar), both Sun E250's have been replaced with two Sun E280R systems (ldap-rep1 & ldap-rep2).

The data for ldap-master is now housed on our SAN, mirrored between the ECS and Public Policy datacenters.

The calendar.umbc.edu service was moved last night around 8:30pm from a Sun Netra X1 to a Sun E280R.

4x the memory, twice the CPUs (now with "speed!"), and it's databases have been moved to our bi-located mirrored storage.

All of our Dell 2650's (with the exception of ifs2) have had their DRAC units configured on our managment network.

Yesterday we upgraded grimm.umbc.edu (a Sun V210) from 2GB to 4GB of RAM. This is in preparation for moving the Oracle "GL" instance from the old threepio.umbc.edu server (a Sun E250) to this new server. grimm already runs the MySQL database Syscore maintains for user and departmental use.

* hfs11 has been moved to the Public Policy CR
(figured, we didn't have anything over there that could actually
serve the files that were over there if we needed it to)

* One of the spare v100's has been relocated to the PUPCR, and will
serve as corens3 (doing caching DNS, ntp, and NIS)

* Webmail2.umbc.edu has been rebadged as sympa.umbc.edu, so it can be
recommissioned to do...well, that.

* Uportal2's managment network line has been fixed, so it can be
managed yet again.

Remedy has been moved to yet another E250, after the one that it was on started having memory errors (not related to a specific piece of memory...)

www5 (an e220r) died in an odd way -- as in completely died -- this morning. it's been replaced with the e220r that was formally known as www2. Inventory has been updated, etc.

www5 houses "webadmin", our IDMS & account administration application.

Weblock, our blackboard development server is now...
dead. The hardward won't boot thru the bios load and
such. I should probably try and tranfer the system to
a dell 2650. The dead hardware is a 2550 that is ~5 years old.

Our 'webauth' servers (umbc's web single signon software) have been migraded from Sun Netra T1 servers (really old boxes) to Dell 2650's running Solaris 10.

They will also be our Shibboleth IDP & AA servers as soon as that functionality is tested.

db1.afs db2.afs & db3.afs are all running on Dell 2650s now running Solaris 10x86. db2.afs has been renamed chablis, and is running the adm server because I'm too lazy to port adm to something modern so I can recompile it. Good 'ole adm.

Our Shibboleth IDP/AA functions have been moved to the loadbalanced webauth.umbc.edu server cluster and are officially in production as part of the
InCommon federation.

The two Sun V210 server which serve www.umbc.edu and many other virtual hosts were upgraded from Solaris 9 to Solaris 10 last night. All campus web services maintained by Syscore are now operating on the Solaris 10 operating system.

Our only "public" shell server running Solaris, solaris.gl.umbc.edu (aka solaris1.gl.umbc.edu) has been upgraded from Solaris 8 to Solaris 10. From the user's perspective, there isn't much difference, however if you run ppriv $$, you will notice that aside from basic priviledges, you also have privs which allow you to use DTrace.

All of our production hfs & bfs AFS servers have been upgraded to Solaris 10 11/06 levels, and are using ZFS pools for their data storage. They are running OpenAFS 1.4.2 w/ the fsync() patches.