ST. PAUL - Human error caused the hourlong 911 outage in Minnesota, North Dakota and North Carolina on Aug. 1, according to the Minnesota Department of Public Safety.
CenturyLink, the contracted 911 service provider in Minnesota and North Dakota, said an error by an employee of a third-party vendor is to blame.
In the late afternoon incident, CenturyLink said an employee of West Safety Services made a mistake while making a network configuration change that prevented 911 calls from being accepted in the three states.
Further investigations are underway and both CenturyLink and West Safety Services could face fines.
In its report released by the DPS's emergency communication networks division on Wednesday, Aug. 15, CenturyLink provided a timeline for the event:
• 3:47 p.m. - The outage began.
• 3:58 p.m. - 911 dispatch centers alerted CenturyLink to the call failures.
• 4:02 p.m. - CenturyLink contacted West Safety Services about the issue, who noted they were aware of the problem and attempting to correct it.
• 4:24 p.m. - CenturyLink notified all 911 dispatch centers in Minnesota, North Carolina and
North Dakota of the outage.
• 4:52 p.m. - West Safety Services rolled back the configuration change which restored 911
CenturyLink said 693 calls to more than 50 Minnesota 911 dispatch centers failed to be routed during the outage. During that same time, 356 calls to 911 were successfully routed to Minnesota 911 dispatch centers through a redundant router.
In North Dakota, where only the 911 centers in Fargo and Grand Forks were affected, 46 calls failed to be routed, while 25 were successfully routed, according to Jason Horning, 911 program manager for the North Dakota Association of Counties.
Horning said the state's 20 other 911 centers weren't affected by the outage.
He made the point that some of those 911 calls during the outage perhaps weren't emergencies, but instead people calling in to see if the system was operational. He said that's one way they test outages themselves..
The Red River Regional 911 Center in Fargo is tied into both the Minnesota and North Dakota networks, Horning said. Grand Forks, however, isn't.
Meanwhile, CenturyLink said West Safety Services has agreed to stop work on its network through the end of August while it reviews the reason for the outage. They are also looking into changing processes and procedures along with enhancing software and safety mechanisms to prevent future errors from occurring.
Minnesota's DPS is in the second year of a five-year, $29.5 million contract with CenturyLink.
"We are committed to holding all service providers and their vendors accountable for any failures of the system," said DPS's emergency communication networks division director Dana Wahlberg. "We will continue to work towards providing dependable, state-of-the-art 911 services for all Minnesotans in an emergency."
The Minnesota Public Utilities Commission is the regulatory authority over CenturyLink and West Safety Services. The Federal Communications Commission will conduct its own investigation into the outage, which could result in fines for both companies.
Minnesota has 102 dispatch centers statewide that have received an average of 7,817 calls per day this year.
Horning didn't have numbers for North Dakota's 22 centers' calls per day this year, but said last year there were 230,000 911 calls in the state.