Power Up

The weeks following hurricane Katrina and the way Johnson's team was able to respond to the evolving disaster is a testament to the importance of disaster recovery planning

Power House Up

Johnson made it to Entergy's storm command centre - aka "The Power House" - in Jackson, Mississippi, around 4am on Sunday morning. Katrina had strengthened from category three to category five. "The news reports were alarming," Johnson says. "The potential implications for the city went up dramatically."

As outlined in the disaster recovery plan, Johnson's team prepared the company's systems that would be most critical in the restoration of electricity - its outage recording and management applications - to run off the Little Rock data centre in case something happened in Gretna.

Sure enough, by 3am Monday Gretna not only lost commercial power but the backup generator was sustaining serious damage from wind and debris. It wasn't a good sign. "We've had pretty bad storms before where we've lost commercial power and failed over to the generator," explains Johnson. "But we've never lost both." A day later he would find out the Gretna centre suffered roof and water damage as well. As the morning began, Johnson declared an emergency with its vendor SunGard to reserve capacity at its hot site facility should they not be able to replicate systems in Little Rock.

Tuesday, the electricity was out everywhere - even in Jackson. "Tuesday was a pretty rough day," Johnson says. "We didn't even have power at the Power House." That evening was the first chance for Johnson to send an expeditionary force to the Gretna data centre where he discovered the extent of the damage.

A significant portion of Jackson's IT staff actually works for Science Applications International Corporation (SAIC) as part of a major outsourcing relationship dating back to 1999. "When we sent people out to the data centres in those first few days, when there was no food or water yet, you couldn't tell who wore what badge. It didn't matter, we were all working together," Johnson says.

Many vendors went beyond the call of duty. "All of our vendors - and even some we've never worked with - were there within a couple of days saying: 'What do you need? How can we help?'," recalls Johnson.

Although Entergy's most critical applications were successfully brought online in Little Rock from backup tapes sent over the weekend, on Wednesday the team determined they could get the Gretna generator back online, bringing in a generator from another facility as backup. On Thursday, they brought in a contractor to patch up the roof and by Friday had it up and running again. Another storm was brewing - what would become Hurricane Ophelia - so Entergy continued on its path with SunGard in order to "keep all our options open", Johnson says. Ophelia took a different tack, and by Labour Day "we had all but completed our disaster recovery plan", recalls Johnson. All critical and medium-priority applications had been restored at the Gretna data centre and the disaster recovery team worked to continue to get all "normal" systems up and stabilized.

By all accounts, the disaster recovery plan worked well, but some changes had to be made along the way. "We never follow the plan to the letter," explains Johnson. "In the IT space, the plan is very solid in terms of what we have to do. But we're always working - our core IT staff in conjunction with representatives from the business areas - to see if we need to change priorities."

Entergy's natural gas infrastructure below ground typically isn't impacted by hurricanes the way the electrical infrastructure is. But because of the extensive flooding that Katrina wrought, gas leaks followed. "That changed the game a bit," says Johnson. "The applications associated with natural gas facilities mapping and asset tracking systems had to be moved up the list in terms of priority. It became clear that that would be much bigger part of the restoration effort." Entergy New Orleans Gas Operations is currently working to find, control and repair those gas leaks where the water has receded.

As crews were dispatched to restore power, problems arose with the two-way radio systems used in the field. Johnson's staff worked to figure out where the problem was - Was there power to the transmitting equipment? Was there a tower down? Was there a problem with the fibre link? - and resolve it.

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about BillionBossEntergyHISRockScience Applications InternationalSUNGARDVirtual OfficeWorldCom

Show Comments
[]