UMBC logo

« Update on Windows XP Problems | Main | Major Blackboard Upgrade Scheduled in Late May »

April 26, 2010

Summary of McAfee and Windows XP Events

Number of Known Machines Affected Campus Wide
(Including DIT Managed Spaces)

These numbers are self-reported from Departmental IT (DIT) staff and Division of Information Technology (DoIT) staff. The numbers are likely higher than these due to staff fixing the problem themselves using DoITs online documentation and us not receiving reports back from everyone on campus.

Campus Labs (DoIT and Departmental)
679 PCs

Faculty and Staff Desktops
330 PCs

TOTAL Number of Machines Affected
1,009 PCs

DAY ONE Wednesday, April 21st
12:15 p.m.
On Wed, April 21, 2010, beginning at 12:15 p.m. the UMBC help desk began receiving calls from across campus reporting Windows machines crashing. A quick on-site assessment of the situation along with some Twitter posts led us to believe the cause was not a virus but more likely an errant DAT file pushed out my McAfee. A DAT file contains new virus definitions which allow McAfee to block new viruses and malware. These DAT files are pushed out frequently to ensure protection against newly discovered viruses and malware. We immediately removed the bad DAT file from the on-campus McAfee repository.

DoIT staff spent the next 45 minutes testing various combinations of operating systems and McAfee versions on virtual machines. The use of virtual machines allowed us to test multiple permutations in minutes compared to hours had we used physical machines. Our testing revealed that the problem was limited to Windows XP Service Pack 3 and McAfee 8.7i.

1 p.m.
An emergency meeting of DoIT staff including Desktop Support, Help Desk, Security, Networks and senior management convened at 1 p.m. A careful review of all available information showed that McAfee still had the errant DAT file available from their web site. During this meeting we blocked access to the McAfee repository from on campus to prevent further damage to unaffected machines. At this time there was no fix available from McAfee. We split the staff into three teams.

Labs Team
We knew that re-imaging would fix the labs machines because our image did not contain the bad McAfee DAT file. The first team dealt with restoring the labs. Our first priority was to get these spaces up and running to minimize impact to classes. Two staff members re-imaged all of the DoIT Labs in ENG and AcIV. There were 8 DoIT labs affected, the last lab was re-imaged and available for teaching by 6 p.m. on Wednesday.

Communications Team
2:30 p.m.

A quick message was sent to the Departmental IT (DIT) staff on campus letting them know briefly what we were seeing. At the same time individual calls were made to several departmental IT staff to ensure they were aware of the problem.

3:05 p.m.
A campus wide news announcement was sent via email letting the campus know what we were seeing and we are working on developing a solution to the problem. We recognized that those with affected machines would not be able to view this message but there were still a large number of unaffected machines on campus. Our hope was that those who could read the message would share it with their neighbors.

3:15 p.m.
A DoIT News post was made to the Web site echoing our email announcement.

4 p.m.
A myUMBC spotlight was created regarding the McAfee and Windows XP issue. It linked back to the more detailed DoIT News post.

4:13 p.m.
A text message was sent to the 4,900 registered E2Campus users reporting the problem.

6 p.m.
The DoIT News post was updated with the solution developed by DoIT staff and the files were uploaded to our web site.

6:28 p.m.
Second text message was sent to the campus letting them know a fix was available and referenced the DoIT Website.

6:54 p.m.
An e-mail was sent to the campus departmental IT staff letting them know of the posted solution and DoIT plans for remediation the following day.

10:22 p.m.
A second campus wide e-mail was sent detailing our solution and letting the campus know we would begin remediation the following day at 7 a.m.

Solutions Team
2 to 6 p.m.

DoIT staff worked to identify the file that McAfee was blocking on Windows XP machines. At the same time more information was filtering out from both McAfee and other tech sites around the world. Once the we knew the exact nature of the problem, DoIT staff began developing a script to replace the affected file, update McAfee with a working DAT file and test the solution. The solution was tested on several affected machines around the campus to ensure it worked correctly.

6:30 p.m.
50 CDs were created with the fix script. Copies were made available at the help desk and the remaining CDs were used by DoIT staff the following day.


DAY TWO Thursday, April 22nd
7 a.m
.
Command center setup in office suite with white boards and conference phone. Staff were assigned to various buildings across campus and were re-tasked as they completed a building sweep.

10:45 a.m.
Email sent to campus departmental IT staff requesting an update on their status.

11:45 a.m.
First sweep and resolution of affected PCs in all UMBC buildings completed.

11:45 a.m. to 1:30 p.m.
Patching of departmental non-teaching labs (economics, physics, public policy, athletics, sociology, psychology, chemistry, biology)

2 p.m.
Review meeting was held to recap what had been completed. A second sweep of all UMBC buildings was conducted by DoIT staff to catch those faculty and staff not in their offices during the morning sweep.

4:30 p.m.
Second on-site sweep by DoIT staff of the campus buildings completed.

|