Ghost server comes alive

Ohio University says someone has hacked into an alumni database server and may have stolen personal information on more than 300,000 people and organisations — including 137,800 social security numbers.

But that’s not the scary part. It also turns out that, according to security logs, the server was compromised early last year at the latest, and that it was being used for a denial-of-service attack against an external target. In short, it was, as kids say, “owned”. But that’s still not the part that’s so frightening.

Here’s what’s scary: everyone thought this server was off-line.

In fact, it was supposed to have been decommissioned more than a year ago. IT managers thought it had been. Thus, logically enough, it didn’t get any security updates or patches. After all, you don’t patch an out-of-service machine. You don’t waste any budget on it at all. It’s dead.

But this unpatched server was still running and still connected. It was a ghost — officially dead, but still haunting the network. So, it was hacked. And turned into a denial-of-service weapon. And the information on it was exposed to bad guys who could use it for identity theft.

There’s an obvious lesson here, and it’s worth saying early and often: there’s no such thing as a decommissioned server. At least not until it has been unplugged, its disks have been wiped and its carcass has been carted away.

Just unplugging it from the power and the network isn’t enough. It’s too easy to plug it back in.

What about unplugging it, wiping the disks and putting it in storage? Still not enough. Some enterprising systems administrator in a cash-strapped department can easily dust it off, plug it in and restore it from backup tapes. Voila — a functioning server at no incremental cost.

Except that, being off the books, it won’t get the proper security treatment. No patches, no upgrades, no security log reviews.

We don’t know whether that’s what happened at Ohio University, or whether the server was supposed to be shut down and simply never was. But the result is the same either way: a ghost server, ripe and ready to be compromised.

What’s worse, we can be pretty sure that most organisations won’t take that last step and physically dispose of decommissioned IT equipment. A roomful of out-of-service servers is just too handy. They’re good for parts, they’re good for emergency replacement machines, they’re good for skunk-works projects.

And everybody’s happy when IT can magically deliver a working server in almost no time and without spending a dime. That’s the sort of responsiveness we always preach. It delights users and makes IT people feel like the wonder workers we know we are.

Hauling that equipment away is good security policy, but when a good security policy runs against the interests of both users and IT, it’s unenforceable.

So, don’t try to enforce it. If you don’t physically dispose of old servers, assume that they could resurface at any time. That means you have to keep watching for them — constantly scanning your networks for unusual traffic from machines that aren’t supposed to exist.

You have to hunt them down. Otherwise, you can’t lock them down.

Don’t trust your inventories. Don’t trust your network maps. Most of all, don’t trust that as long as you’re careful to wipe personal information, like social security numbers, the worst that can result from a ghost server is an embarrassing news story about how there’s a hackers’ playground on your network.

Remember, a ghost server isn’t just a machine where some intruder can serve up pirated files or launch denial-of-service attacks. It could also be a gateway for attacks in the other direction — on your networks, users and information.

And for IT, that’s one really scary possibility.