OIT SysCore

First problem I heard was around the date of 8/13 while I was still away on vacation. Kip Canfield could not access his assessments and students could not take tests. I did some testing and worked with Jeffery to prove that this was not system wide. I also found a possible work around.
I then tried rebooting the servers to see if the problem would clear. This caused the access problems to happen to not only assessments, but to every course, systemwide. I pushed out a reconfig and this caused even more problems as it would not complete cleanly. One instance of the problems was the tomcat service would not delete, so it could not be installed correctly. When I finally got the system to "config" without errors, blackboard started working again.

Then I found one really weird network issue. The database server could not reach the fileserver/collabserver. As in "ping 130.85.29.12" (from 130.85.29.13) returned "unreachable" I fixed that by removing some of the broadcom advanced features I had enabled on the database server.
The features have been running for over a year. So, my guess is that some of the recent network changes in the computer room caused this problem to show up now.
This was probably not cause of the access problems, although it would have broken the collab service.

Todays updates did not go well. I'm finishing the process of rolling back at this point.
Blackboard is up and running in the state it was before the upgrade.

There were a number of problems but the short version is that the upgrade was taking way too long and finally errored out. I started at about 6:30 and started the rollback at about 11:45.
I have a few ideas on how to make the update work that involve the
database. I'm also submitting some errors to blackboard in a few minutes.
If I can get the mirrors rebuild I can try the update again this Sunday night at
10PM. Otherwise I can try it some night this coming week

The first unexpected "got ya" that hit me was that upgrade was unable to modify the
database due to the replication subscriptions. We are replicating our blackboard databases to a special database for access by our umbc apps. I had to remove the replication subscriptions to remove the publications, to alow the updater to modify the database.

What this means is that I'll need to completely reconfigure the replication
setup next week. This should be a good thing. When I first set up the replication,
I felt like there was quite a bit of pressure to get it working very quickly. So, I never really had time to figure out what I was doing. Nor was I able to document "how to set up replication" for the blackboard database. Now hopefully I should be able to set this up better and document the procedure properly.

The second problem was that the updater had not finished the first server after 2 hours
and it normally takes 30 minutes. I've seen this before in test, and it didn't finish after a
full day. So I had to start over. The next try I let the updater run for 2.5 hours and it finished with a critical error. Hence the rollback.

Netsaint is broken. Going to netsaint.umbc.edu gives a PGP page.