We've been tracking the status of our AFS servers since their "problems" at the end of last week, and have nailed the issue down to a redux of some of the problems we had in the fall due to thread resource contention. While we had fixed 99% of the problems last semester by working around the code that was causing the contention, there still seems to be some instances where a state can be reached which causes thread deadlock. We're working with some of the OpenAFS folks on things to look at to further reduce the instances of deadlock.
One recommendation is to further increase the size of the AFS callback tables on the fileservers to an "insane" amount; even more insane than what we set it to on Friday night/Saturday morning.
The second recommendation involves tracking down AFS clients in the wild which are not allowing AFS Cache Manager Callbacks -- meaning, they're network -- or host-based firewall -- is blocking incoming port 7001/UDP traffic from the AFS fileservers. I'm going to be writing a script to query the AFS fileservers for active clients and test them with cmdebug to get a better idea of how widespread the problem really is.