Power Up

Hurricane Katrina proves why it most definitely pays to have a disaster recovery plan in place

"Monday was not a good day." That's how Ray Johnson, not one for hyperbole, remembers August 29, 2005 - the day Hurricane Katrina roared ashore on the US Gulf Coast. But there's a reason for the understatement. Johnson, CIO of New Orleans-based Entergy, would see darker days to follow.

It's now two and a half weeks after the killer storm left a devastated landscape in its wake. Johnson has had little time to reflect on his company's efforts not only to restore power to its customers affected by the hurricane but also to prevent severe business interruption despite a virtual attack on its own physical and human infrastructure. But he spoke to CIO on September 15 to discuss Entergy's preparation for and response to the worst disaster in the company's history.

Entergy's recovery efforts can be traced back to long before Katrina hit. The $US10 billion energy company has a disaster recovery plan that's tested and revised once a year. Last year, that "test" came in the form of Hurricane Ivan. Entergy activated its storm command centre and disaster recovery processes in response to the approaching storm which ultimately made landfall further east in Pensacola, Florida. And just this April, the company conducted an extensive storm drill that simulated what corporate executives thought of as a worst case scenario - a major hurricane followed by extreme flooding.

"This is not a unique event for us," explains Johnson, "I wish it was. But we've got our disaster plan nailed. We review it at least once a year and either we conduct a drill or get to test it when a hurricane threatens us and then misses."

On Friday, August 26, it looked like Johnson would have another opportunity to test the disaster recovery plan. Katrina was crossing over Florida and beginning to churn in the Gulf. But early in the day, the hurricane was predicted to hit several hundred miles east of New Orleans.

As Friday evening wore on, the situation changed. This was not a test. "We started making our storm calls and made the decision to activate our disaster recovery plan," which calls for some preliminary actions to be taken beginning 72 hours before a category 3 hurricane is scheduled to make landfall," recalls Johnson.

Saturday morning 5am, Johnson sent his first "away" team to the active disaster recovery site Entergy has in Little Rock, Arkansas. Entergy's primary data centre is located in Gretna, Louisiana, across the river from Entergy's corporate headquarters. The backup generators supporting that data centre had never failed before, but Johnson knew there could be one of many gloomy firsts he might encounter this time out. "By late in the day [on Saturday] it became more and more obvious as Katrina tracked closer and closer that New Orleans was going to be a primary target." Once the first team arrived safely, a second was deployed.

Around that time the decision was made to do a full implementation of the disaster recovery plan and Entergy's core restoration team flew into action, led by Randy Helmick, vice president of customer service and support during normal operations but given the title of "storm boss" when an emergency is declared.

Page Break

Power House Up

Johnson made it to Entergy's storm command centre - aka "The Power House" - in Jackson, Mississippi, around 4am on Sunday morning. Katrina had strengthened from category three to category five. "The news reports were alarming," Johnson says. "The potential implications for the city went up dramatically."

As outlined in the disaster recovery plan, Johnson's team prepared the company's systems that would be most critical in the restoration of electricity - its outage recording and management applications - to run off the Little Rock data centre in case something happened in Gretna.

Sure enough, by 3am Monday Gretna not only lost commercial power but the backup generator was sustaining serious damage from wind and debris. It wasn't a good sign. "We've had pretty bad storms before where we've lost commercial power and failed over to the generator," explains Johnson. "But we've never lost both." A day later he would find out the Gretna centre suffered roof and water damage as well. As the morning began, Johnson declared an emergency with its vendor SunGard to reserve capacity at its hot site facility should they not be able to replicate systems in Little Rock.

Tuesday, the electricity was out everywhere - even in Jackson. "Tuesday was a pretty rough day," Johnson says. "We didn't even have power at the Power House." That evening was the first chance for Johnson to send an expeditionary force to the Gretna data centre where he discovered the extent of the damage.

A significant portion of Jackson's IT staff actually works for Science Applications International Corporation (SAIC) as part of a major outsourcing relationship dating back to 1999. "When we sent people out to the data centres in those first few days, when there was no food or water yet, you couldn't tell who wore what badge. It didn't matter, we were all working together," Johnson says.

Many vendors went beyond the call of duty. "All of our vendors - and even some we've never worked with - were there within a couple of days saying: 'What do you need? How can we help?'," recalls Johnson.

Although Entergy's most critical applications were successfully brought online in Little Rock from backup tapes sent over the weekend, on Wednesday the team determined they could get the Gretna generator back online, bringing in a generator from another facility as backup. On Thursday, they brought in a contractor to patch up the roof and by Friday had it up and running again. Another storm was brewing - what would become Hurricane Ophelia - so Entergy continued on its path with SunGard in order to "keep all our options open", Johnson says. Ophelia took a different tack, and by Labour Day "we had all but completed our disaster recovery plan", recalls Johnson. All critical and medium-priority applications had been restored at the Gretna data centre and the disaster recovery team worked to continue to get all "normal" systems up and stabilized.

By all accounts, the disaster recovery plan worked well, but some changes had to be made along the way. "We never follow the plan to the letter," explains Johnson. "In the IT space, the plan is very solid in terms of what we have to do. But we're always working - our core IT staff in conjunction with representatives from the business areas - to see if we need to change priorities."

Entergy's natural gas infrastructure below ground typically isn't impacted by hurricanes the way the electrical infrastructure is. But because of the extensive flooding that Katrina wrought, gas leaks followed. "That changed the game a bit," says Johnson. "The applications associated with natural gas facilities mapping and asset tracking systems had to be moved up the list in terms of priority. It became clear that that would be much bigger part of the restoration effort." Entergy New Orleans Gas Operations is currently working to find, control and repair those gas leaks where the water has receded.

As crews were dispatched to restore power, problems arose with the two-way radio systems used in the field. Johnson's staff worked to figure out where the problem was - Was there power to the transmitting equipment? Was there a tower down? Was there a problem with the fibre link? - and resolve it.

Page Break

Endurance Test

The weeks following hurricane Katrina and the way Johnson's team was able to respond to the evolving disaster is a testament to the importance of disaster recovery planning. But what's most remarkable is not how well a frequently-practised and well-executed plan worked, but what the people executing that plan had to endure as they carried it out.

Johnson is the first to tell you he's one of the lucky ones - the damage to his house on the west bank of Lake Ponchatrain in Jefferson Parish can be fixed. Not everyone was so fortunate. The first few days after Katrina, Entergy was also focused on locating its 14,000 employees, including the 700 men and women working in IT. They were scattered to the four winds. "One of the first things we did was to try to contact everyone using our network of supervisors beginning on Tuesday to find out such things as: Are you OK? Do you know the status of your home? Do you have any urgent needs that aren't being met?," says Johnson. "There were quite a few people we couldn't find at first." Ultimately everyone was located and there were no fatalities.

Nonetheless tragedy permeated the staff. "What's unique about this story is the fact that so many people involved in the core restoration that relocated here to Jackson were on the job working hard even though they knew they had no home to return to," Johnson says, including a senior member of the business continuity team. "People up to the senior management level couldn't get in touch with family members. And these people were working with us twenty hours a day. It was a testament to their dedication."

Beyond disaster recovery, some Entergy employees faced with dealing with their own personal losses had to work on the business continuity plan as well. That commenced the Sunday following the storm. Entergy headquarters located in the heart of the central business district a stone's throw from the Superdome, high and dry just after Katrina blew through was now surrounded by six feet of water. The building where 70 percent of Entergy's employees reported to work every day was unreachable. "We had to make the assumption that because of the substantial damage downtown, we would not be returning to normal operations there in a week or two," says Johnson. He and other senior executives began to look for a temporary home for the company and a solution was found in the former Worldcom headquarters in Clinton, Mississippi, just outside Jackson.

The IT team was charged with getting the telecom and IT infrastructure in place at interim headquarters, located in a larger office complex. The enterprise business continuity team, having located all employees either by phone or Internet, began to assist those who could return to work in finding temporary housing in the area.

Unlike some other disasters Entergy has weathered, this one had a huge business continuity element to it - and not just in terms of the physical headquarters. "The business continuity plan was a major part of our restoration. As we located people we were able to immediately start looking at how we could best mobilize our employees," Johnson says. "We knew who was where so we could start to look at adding equipment and computers in new locations where necessary and mapping our needs to the resources we had."

On September 9, Entergy deployed an electronic survey on its intranet, asking all employees to sign in and provide updated information on their location and personal situation.

Obviously, not everyone originally working at Entergy headquarters could relocate to the Jackson area. "That gave us a pretty good handle of where everyone is, what their family situation is, and whether they are available to be redeployed," Johnson says. Armed with a spreadsheet of that data, Entergy's business continuity team began to build ad hoc teams based on geography and skills, not job title. "Almost no one is doing the job they had before," Johnson says. "Everyone's job is different." In some cases, employees are beginning to report to Entergy locations in Houston and Little Rock, for example. In other cases, workers are telecommuting.

Page Break

We Will Return

At the time of writing Entergy had stated publicly that the ultimate goal is to return to New Orleans and its headquarters in the CBD. But a lot has to happen within the city before that can happen, much of that not in Entergy's hands. "There's a lot of cleanup to be done and restoration of the basic infrastructure there," says Johnson. The good news thus far has been the word that draining of the city will take less time than originally estimated.

What has been in Entergy's control is the restoration of power to its customers. As of September 15 at 5pm the company has restored electrical service to more than 860,000 of the 1.1 million customers affected by Hurricane Katrina, with 230,000 remaining without power.

The number of employees returning to full-time work at Entergy in some location or another changes every day, says Johnson. Temporary headquarters were slated to be ready for business on Monday, September 19. "We expect we'll be giving a substantial number of employees directions on where they need to report as we start getting them interim housing arrangements based on their situations," Johnson says.

It's an attempt at a return "normalcy", but only in relation to a situation that's been anything but. "We were faced with a combination of major hurricane, followed by flooding event that had levees not failed would not have been nearly as bad, followed by civil and social challenges," says Johnson. "We drill for even really bad hurricanes, and we couldn't have realistically plan for something that bad."

The company has created a specific task force called the Entergy Virtual Office team that has begun to look at what it will take to operate efficiently in Entergy's new incredibly distributed office environment, leveraging existing and new technologies.

Down the Road

Someday things will return back to "normal" normal and then Johnson and other members of the core restoration team will pull together their lessons learned, as they do after every disaster recovery effort and drill, and revise the plan for next year. "The sheer magnitude of this makes it a kind of one-time event and we hope this will never happen to this degree again," says Johnson. "But you can't go through something this significant and not find ways to do better."

One thing that probably won't be on that list - but is as surprising as anything else that's happened - was just how much Johnson, his peers, and his employees could take. "I don't think anyone ever felt overwhelmed. Even in the darkest period, no pun intended, when there was no power at the Power House, there was never a sense of panic," Johnson says. "We had a lot of people worried about their homes and their city, but they went to work collecting facts to figure out where we should start and what we should do."

For the next few weeks, the business continuity work continues for Johnson, who currently calls the Jackson Hilton home along with his wife, and two daughters and his dog - at least for the four or five hours a day he isn't working.

Should he start to lose focus, Johnson stares at his computer screen. It's adorned with a picture taken the Sunday after the hurricane and the subsequent flood swept through New Orleans. It depicts a few office buildings in a small section of the CBD that Entergy was able to restore power to in the early going. "You could see that little piece of the CBD lit up and reflecting off the river. It was just a few days after Katrina and it didn't get much media coverage. But I made it my screensaver," says Johnson. "It's just one small area. But it gives you hope."