problem, all workers had to log off of the email server while someone moved
emails from the queue file by file.
Further, the surge of disaster victim registrations resulting from the Florida
hurricanes overloaded NEMIS' main server, pushing the system beyond its
limit. Originally designed to handle a maximum of 20,000 disaster victim
registrations a day, during a four month period from August to December
2004, NEMIS registrations far exceeded these limits during peak periods,
reaching over 40,000 on some days. The total number of disaster victim
registrations processed during the four month period of the Florida hurricanes
was 1,745,183. The volume of transactions and the number of personnel
managing these registrations significantly slowed down the system or made it
unavailable for use during peak operations.
Although EP&R CIO staff worked to keep the system operational by
increasing system memory, NEMIS' main server became overloaded, the
system froze, and unplanned system restarts were necessary. Users were
unable to perform their jobs in the system and consequently reverted to paper
based methods. When NEMIS' main server went down, approximately 2,000
IT users were kicked out of the system for as long as 20 to 30 minutes at a
time. FEMA personnel accepting victim registrations had to record the
information manually and wait to register the victims in NEMIS when the
system was functional again. Additionally, FEMA personnel lacked the up
to date NEMIS information needed to answer disaster victim inquiries when
they called the National Processing Service Center for assistance.
As part of our review, we requested system performance reports from the
EP&R CIO's office to determine how the systems performed during the
hurricane response. However, the CIO office did not have a standard process
in place to produce system performance reports. Instead, it had to complete a
manual analysis of raw data to provide performance data for only one of
NEMIS' key servers. The CIO office stated that it would take several months
to supply performance information for all of the other servers. Although the
CIO office can monitor central processing unit and hard drive space
availability on a real time basis, it does not have a tool that can show system
performance over time. Without such a tool, it is difficult for FEMA to
identify system performance problems and take corrective actions to address
them.
Given the problems experienced during the 2004 hurricane response and
recovery season, a number of FEMA officials expressed concern about not
only NEMIS' current capabilities, but also its capacity to support future
Emergency Preparedness and Response Could Better Integrate Information Technology
with Incident Response and Recovery
Page 24
New Page 1