A number of factors and failings came together to make the August 14th northeastern blackout the worst outage in North American history. One of them was buried in a massive piece of software compiled from four million lines of C code and running on an energy management computer in Ohio.

To nobody's surprise, the final report on the blackout released by a US-Canadian task force Monday puts most of blame for the outage on Ohio-based FirstEnergy Corp., faulting poor communications, inadequate training, and the company's failure to trim back trees encroaching on high-voltage power lines. But over a dozen of task force's 46 recommendations for preventing future outages across North America are focused squarely on cyberspace.


That may have something to do with the timing of the blackout, which came three days after the relentless Blaster worm began wreaking havoc around the Internet - a coincidence that prompted speculation at the time that the worm, or the traffic it was generating in its efforts to spread, might have triggered or exacerbated the event. When US and Canadian authorities assembled their investigative teams, they included a computer security contingent tasked with looking specifically at any cybersecurity angle on the outage.

In the end, it turned out that a computer snafu actually played a significant role in the cascading blackout - though it had nothing to do with viruses or cyber terrorists. A silent failure of the alarm function in FirstEnergy's computerized Energy Management System (EMS) is listed in the final report as one of the direct causes of a blackout that eventually cut off electricity to 50 million people in eight states and Canada.

The alarm system failed at the worst possible time: in the early afternoon of August 14th, at the critical moment of the blackout's earliest events. The glitch kept FirstEnergy's control room operators in the dark while three of the company's high voltage lines sagged into unkempt trees and "tripped" off. Because the computerized alarm failed silently, control room operators didn't know they were relying on outdated information; trusting their systems, they even discounted phone calls warning them about worsening conditions on their grid, according to the blackout report.

"Without a functioning alarm system, the [FirstEnergy] control area operators failed to detect the tripping of electrical facilities essential to maintain the security of their control area," reads the report. "Unaware of the loss of alarms and a limited EMS, they made no alternate arrangements to monitor the system."

With the FirstEnergy control room blind to events, operators failed to take actions that could have prevented the blackout from cascading out of control.

In the aftermath, investigators quickly zeroed in on the Ohio line-tripping as a root cause. But the reason for the alarm failure remained a mystery. Solving that mystery fell squarely on the corporate shoulders of GE Energy, makers of the XA/21 EMS in use at FirstEnergy's control center. According to interviews, a half-a-dozen workers at GE Energy began working feverishly with the utility and with energy consultants from KEMA Inc. to figure out what went wrong.

The XA/21 isn't based on Windows, so it couldn't have been infected by Blaster, but the company didn't immediately rule out the possibility that the worm somehow played a role in the alarm failure. "In the initial stages, nobody really knew what the root cause was," says Mike Unum, manager of commercial solutions at GE Energy. "We spent a considerable amount of time analyzing that, trying to understand if it was a software problem, or if - like some had speculated - something different had happened."

Source