The New Zealand Red Cross IT team was just returning to some kind of normality following the September earthquake in Christchurch, when the second, more damaging quake struck the city on February 22 shortly before 1pm.
For the next few hours the organisation’s website was put under enormous strain as people around the world logged onto the site seeking information about the disaster.
IT Manager Charles Ranby says at one point there was an unprecedented amount of traffic from people seeking information about the emergency and wishing to make donations. “It was overloaded, and we had to rebuild and put more capacity in to cope with demand. It was very slow to respond for a good four to six hour period, while we were getting the capacity set up for that,” he says.
Immediate response
Web host Netspace had a technician monitoring traffic, but Ranby says there was little he could do “when half the world is visiting the site”.
The second earthquake, unlike the first quake in September, claimed lives and Ranby’s first task was ensuring the website and related IT systems could cope with the influx of queries about missing people.
To cope with the spike in traffic Netspace Services Limited added a reverse proxy to handle the static content delivery, tuned PHP to include an op-code cache and also trimmed the PHP modules to the bare minimum. The Apache daemon was also tuned for high turnover of processes to prevent memory bloat. The web server was put onto a new 64-bit Debian operating system and the database was shifted to its own 64-bit hardware and operating system.
The MySQL database was tuned, with indexing optimisations applied and alterations to cache settings to increase performance while reducing overall load. The presentation code was rewritten to push more work to the SQL database, to optimise queries, and to streamline processing. Within 36 hours of the site getting hit with a wall of traffic ten times larger than the solution was specified to handle, Netspace had reduced processing and memory load on the servers while being able to serve the higher connection demands from the world, and return to business as usual.
With 15,000 phone enquiries in the first day after the quake, the Red Cross’s call centre in Hamilton quickly became overloaded and, on the request of police and with the help of Gen-i, calls relating to the earthquake were transitioned to the Horizons Regional Council centre, which is set up to handle extra call capacity. It would be two weeks before the missing person call enquiries were handed back to the Red Cross centre on March 7.
Volunteers were recruited utilising online tools such Facebook, Google apps and online roster system Shift-Planner. Taking into account they were non-paid helpers, volunteers were rostered onto duties such as data entry in three-hour shifts. Ranby says the support of volunteers and their dedication to dealing with the emergency was hugely appreciated. They logged 50,000 entries into PRIDE (Public Registration and Inquiry Database for Emergencies) and there were 1000 RFL (Restoring Family Links) enquiries.
In order to enable the volunteer workforce Ranby and IT officer Rick Butler – the two permanent staff members in the New Zealand Red Cross IT team – authenticated around 160 users. Up to 60 laptops were either bought or leased and Ranby says vendors told him stocks of laptops were quickly depleted following the quakes.
Ranby was keen to ensure that Red Cross staff and volunteers could work remotely, so that if the Christchurch building where they’d set up the temporary grants processing centre had to be vacated, they could work out of cars and vehicles. Connectivity was therefore enabled by mobile data cards.
In the immediate days following the February 22 earthquake, the focus was on the emergency. New Zealand Red Cross didn’t activate its 0900 donation line until a week after the quake. When it did become available they were flooded with calls.
New grants management system
Managing the disbursement of grants to victims of the quake was another crucial role the New Zealand Red Cross played in both disasters.
Following the September quake Ranby says it quickly became apparent that the paper-based system they’d previously used wouldn’t cope with the enormous task of distributing millions of dollars of grant money.
The society had implemented a Microsoft Dynamics CRM in 2009 and Ranby was keen to leverage the database. He engaged the services of Magnetism to create a solution to manage the application process and distribute funds to applicants.
The criteria for applications is set by an independent Red Cross Earthquake Commission, but it was up to New Zealand Red Cross to ensure the delivery of the grants was managed in way that is both secure and timely.
The system that Magnetism designed took the data from the paper-based applications forms and ensured there was no duplication of applicants’ information, and streamlined the daily payments process. Ranby says they originally thought the grants dispersement’s would take a several weeks, but it turned out to be a job requiring months of work. The biggest disbursement following the September quake — $1.5 million — occurred on Christmas Eve. Following the February quake the largest payment in a day was $4.9 million on March 21.
Despite the delay Ranby is full of praise for Magnetism, which he says follows the agile method of development, which works well with a small IT team.
When the second quake struck, New Zealand Red Cross had a fully developed grant management system to manage disbursements. And, learning the lessons of the September quake, they installed some enhancements so that data entry – much of it hand written – was accurate.
They used NZ Post’s local post code database to cross-reference and lookup addresses and they installed an IRD bank account validation module to ensure applicant’s banking details were correctly entered.
Looking forward
Ranby has worked for the New Zealand Red Cross for ten years and says the Christchurch earthquakes are on a par with the 2004 Boxing Day tsunami in terms of managing the grants process.
Following the dual disaster in Christchurch, Ranby is considering boosting the organisation’s mobile capability with smart phones and tablets such as iPads. The idea is these devices will connect to a portal hosted using Microsoft Azure technology, which would provide scalability to meet demands of a future grants disbursement. The portal would be linked to Microsoft Dynamics CRM database for live data capture.
Ranby says the portal will allow data to be directly input into XRM and this would eliminate the biggest bottleneck of keying information manually.
The Azure platform the portal sits on would be able to cope with a significant number of spontaneous connections and Red Cross would only pay for what processing resources are required to meet the demand, Ranby says.
He has yet to look into the compliance issue around offshoring data, but he says moving to a cloud-based solution would be in line with international practice in the Red Cross; whose global chief information officer Edward Happ, based in Geneva, is championing cloud technology for the humanitarian organisation worldwide. Although there is no direct command structure, various Red Cross societies throughout the world contribute their IT tools and expertise to a shared global database.