When the power went out in Manhattan late Thursday afternoon (Friday morning NZ time), the stock markets had already closed. But the crucial trade-settlement system that uses thousands of batch-processing computers around New York City to clear billions of dollars in trades had just come to life.
Diesel generators at brokerage, bank and clearinghouse data centers around Manhattan and New Jersey kicked in, and IT departments said that they were far better prepared for what most called a simple power outage than they were on September 11, 2001.
The New York Stock Exchange says no data was lost from Thursday's trading as a result of the blackout. "In addition, the Securities Industry Automation, which is our data processing and technology operations arm, is operating at normal capacity on generator power," a spokeswoman said on Thursday night.
Russ Lewis, CIO at GFI Group, says the Wall Street-based online brokerage took a "hard hit around 4.12pm. . . . and we went right into disaster recovery mode."
"All the systems did come down. We immediately went on generator backup for both our data centre and our trading floor," Lewis said on Friday morning. "Our systems all flipped over as well. Asia and London were unaffected because the systems flipped over properly."
As a precautionary measure, Lewis said he performed end-of-week backups Thursday night and sent them via the company's virtual private network to London, "in case we weren't able to get power into the New York office today and we had to shut the office down."
Lari Sue Taylor, director of enterprise information security and recovery at FleetBoston Financial in New York, says a 62-member crisis management team that was created after 9/11 began assessing the situation within an hour of the initial blackout.
FleetBoston, which has several offices in Manhattan, was forced to move workers to SunGard Data Systems' facilities in Carlstadt, New Jersey. Taylor says the bank also had to transfer network operations for its Quick & Reilly online brokerage service to those facilities.
Diesel generators at Merrill Lynch & Co. in lower Manhattan revved up as the power went out, and computer systems in the Manhattan and New Jersey data centres didn't skip a beat, says spokeswoman Selena Morris.
"We were obviously prepared if something like this happens," Morris says.
At Case Western Reserve University in Cleveland Friday morning, CIO Lev Gonick was running on two hours' sleep after having worked on recovering core systems, including email, course management systems and enterprise systems, throughout the night.
Power was still out Friday morning, and nearly 1000 students were due to move into the university for the new school year on Saturday.
Gonick says school officials were "desperately concerned" about losing data on returning students' tuition payments and course information, but a storage-area network Gonick implemented after September 11 took automatic snapshots of data sets as the power began flickering at 4.07pm EDT on August 14. He said on Friday that he lost only a "fraction of a second" worth of data.
"When we got hit, we got hit with a double surge. It was on the second surge that some backplanes and some network routers got hit pretty badly. We also think the second surge may have hurt some of our large servers as well," Gonick said. "We've got a couple of servers that are a bit cranky coming up. As soon as the system came up, we had to go back and match the last save. It's not been flawless. But it's been as close as I can imagine."
Similarly, Alan Winchester, a technology attorney at Harris Beach in New York, says all of the law firm's financial records are replicated in real time to its Rochester, New York, office, which has a generator.
Winchester says disaster recovery lessons learned after September 11 were quickly implemented at Harris Beach after the lights went out.
IT staff members left the building with backup tapes for Tuesday through Friday, he says. "We can always restore it if something crazy happens to the building," he notes. "We can also restore it if we need to get the information to a server in a part of the country that's not affected." The law firm has offices in several other locations, including Washington and California, as well as connections with other law firms that would help if needed, Winchester says.
FedEx says the lack of power at its hubs and stations in the blackout areas delayed the processing of package information because drivers couldn't download data from bar-code scanners into the FedEx network.
Bob Brewin, Linda Rosencrance and Todd R Weiss contributed to this story.