The improbable story of the datacentre saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.
Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying fuel up 17 floors to deposit into a rooftop generator. There was the sleeping on floors and the lack of showers, and the unrelenting stress that their efforts could suddenly unravel and fail.
The Peer 1 bucket brigade carries fuel to the rooftop generator
Then there was the broader weight on all involved about the impact that Hurricane Sandy was also having on their friends, family and on New York City itself.
The effort to save the datacentre during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the datacentre owner, produced and showed Tuesday to a group of customers and employees.
The film, with photos of datacentre rescue efforts along with interviews of participants, doesn't attempt to romanticize or triumph the events. It just lays them out, and leaves the viewer with a sense of the sheer exhaustion watching the effort of the participants and the constant uncertainty of the outcome.
The idea of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 datacentre, in an interview. "It just didn't see feasible."
Pryor's idea, retold during the film, was to bring some extra pumps from fish tank in his office to the nearby Peer 1 datacentre to see if they might help get fuel to rooftop generator. The proposal drew healthy laughter from the audience, something Pryor now chuckles at as well.
But his thoughts of using fish tank pumps helps explain how the datacentre workers and customers, in the early going, were grappling to come up with ideas for keeping a roof top generator that was burning 40 gallons of fuel an hour out of a 200 gallon capacity tank running.
Fog Creek makes a software management platform used by developers, and the loss of operations at the 5,000 square-foot datacentre at 75 Broad St. would have hurt its customers and its ability to generate revenue.
Michael Mazzei, Peer 1 datacentre manager (Photo: Patrick Thibodeau/Computerworld)
Peer 1 was a short walk from the Fog Creek offices, and as the company grew so did its use of the datacentre. The datacentre provides Fog Creek its network and infrastructure for customer equipment.
There's an area set aside for Peer 1 customers to work on their systems.
As flood waters entered the lobby, Jeffery Burns, a worker in the datacentre, told of hearing the sound of a waterfall in the elevator shaft that eventually flooded the basement and disalbed the fuel pumping systems.
"Everybody in Manhattan has the same problem, how to relocate infrastructure that - for the last 50 years - has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street datacentre.
Despite the damage caused by Hurricane Sandy, datacentres won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.
The Peer 1 datacentre is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterways.
NYC Mayor Michael Bloomberg this month released a multi-billion dollar, multi-decade plan to protect the city from powerful storms, and the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.
Bloomberg's plan has broad IT and telecommunications implications.
It seeks a higher level of reliability and service for telecommunication facilities. The specific recommendations include requiring new hospitals "to increase their IT and telecommunications resiliency by installing two independent points-of-entry for telecom and communication to reduce the risk of outages from a single supplier."
But for a business such as Fog Creek, the hurricane exposed its lack of a back-up plan. The company has been working to fix that weakness in the months since.
For instance, Fog Creek has improved its data replication capabilities and is developing an ability to switch operations, in an emergency, to a separate facility. One tool now runs in Amazon's cloud service, though that migration has been planned prior to the storm.
The storm made clear to Fog Creek that it hadn't paid enough attention to the possibilities of bad things, like a massive storm, happening, said Pryor. That was a positive outcome of the experience, "because it did kick us in the butt and get us to fix a lot of things that were broken."
In his office, Mazzei points to the part of the floor he slept on -- next to his phone.
The bucket brigade wasn't the first move by the group. Initially, they tried moving 55-gallon drums up 17 flights of stairs, but the physical exertion proved too much.
Gradually, though, a realistic plan came into place. The team acquired supplies, pumps, hoses and buckets from Home Depot and other places. Organization, rules and efficiency arrived.
One rule: "Go slowly, work safely, always have your flashlight," said Mazzei.
For 72 hours, the effort continued and the datacentre was able to remain in operation until a more stable fuel supply was established. For the future, one option the building owners are giving consideration to is putting submersible pumps that could continue operating in a flood, said Mazzie.
There were a lot of motivations at work in manning the bucket brigade. For customers, it was about keeping their businesses up. For the datacentre workers, it was about not failing.
No one is really taking credit for saving the datacentre.
Mazzei is very matter-of-fact about the effort, and describes it more as an organic process that developed on its own as ideas and processes emerged collectively.
His co-worker, Burns, a film editor who works part-time at the datacentre, said something particularly flattering about Mazzei. "It was just really important to not let him down, not because he was going to get upset, but because he never let us down," said Burns.
Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, datacentres and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov, or subscribe to Patrick's RSS feed . His email address is pthibodeau@computerworld.com.
Read more about datacentre in Computerworld's datacentre Topic Center.