We had some more oddness on the three AFS servers that were mentioned on Tuesday(?). Basically, we were able to watch them "busy out", all pretty much at the same time -- by busy, we really don't mean busy, they were busy sitting there doing "nothing" except for waiting for something to happen.
After a few minutes, that something would happen, and they'd go about their business. Then 15 minutes or so later, the same thing would happen.
We've never noticed anything like this on our OpenAFS 1.2.13 servers, which were running fine during this time. To test out the theory that this could be a problem, we did a "special build" of the fileserver & volserver processes for Solaris 10, and installed them on hfs12. We'll be watching hfs12 to see if it exhibits the same wankiness as it did before (we expect hfs10 & 11 will continue weirding out.) If it's fine while the others aren't, we'll 'upgrade' them to the version of the code we're running on hfs12.
Our restart of hfs12 to install the new software took much longer than expected because it didn't (or wouldn't) shut down cleanly. This could very well be related to whatever is going wrong...