Traffic on the network had been increasing all week, but the intrusion-detection system wasn’t picking up anything malicious. Whatever traffic was traversing the network was considered okay.
The servers and desktop systems were all at current patch levels, switches at current IOS levels. Sophos Anti-Virus wasn’t showing any virus activity. We hadn’t installed any new devices on the network either, yet something had changed and was causing problems.
We fired up the sniffer, but we weren’t able to identify any particularly malicious traffic, but we did see a lot of what I call “stupid traffic” — IPX broadcasts from the new Ricoh multifunction devices that the vendor had installed and improperly configured; nothing that would affect outbound traffic.
We found that several users were running something called Netropa, a freely downloadable newsfeed application. As the users professed innocence, we uninstalled the service on each machine. But, Netropa insists the product isn’t a bandwidth hog.
We couldn’t identify any one system on the network that was transmitting unusually large amounts of data. Staffers ran around all afternoon tuning the network (and, in one case, inadvertently brought down a primary server).
We were stumped. I even directed the senior server administrator to run F-Secure’s BlackLight rootkit detector on the Windows servers. No hidden processes were found.
As the week came to an end bandwidth, thankfully, returned to normal but we still didn’t have any idea of the root cause of our network problems. Feeling frustrated, I thought out loud: “This is ridiculous. It shouldn’t take us several days to figure this out.”
What you need to understand, and what I need to be patient about, is the fact that my staff have no real network monitoring or troubleshooting experience, and no network security experience.
I manage IT and security for a small state agency and our resources are very limited. In past jobs, I was always able to hire subject-matter experts, including CCIE-level network engineers and CISSP security engineers. Here, I have to rely on my own skills, which are more management-oriented than deeply technical.
In the past, the agency had relied on the state’s network engineers to help out when network problems arose. This time though I wanted the staff to learn something about how to troubleshoot these types of problems and not turn to out-of-agency resources to get the job done. I was asking for a lot in a short period of time, but sometimes we learn best when under fire, even though I think you could characterise our activity as chickens running around with their heads cut off.
Public sector constraints
When you think about network monitoring, performance comes to mind. But, what if you want to monitor for changes on the network in a holistic way? I wondered if it was possible to baseline the LAN/WAN environment for normal ports and services, and be notified when something changed that was outside “normal”. I could then investigate to find out if the change was desired or not. For instance, if a server began listening on a port that it normally would not be listening on, even for a brief period of time, this could indicate a possible compromise.
The basic problem every security team faces, especially in a defence-in-depth model, is that there are too many events, and too much data and too many places where security data must be reviewed, to manually correlate the data and pinpoint a complex security issue. You just can’t do it.
A couple of years ago, a manager without a security background demanded metrics from me, and I explained that you can’t easily pull metrics from disparate security devices without some way of correlating the data. We had to rely on our own analysis of a situation, decide whether it could be considered an incident and then investigate. That’s not good enough.
There are numerous commercial security monitoring tools that operate at the network and systems levels. Back in the private sector the answer to the metrics problem was to evaluate Protego MARS, a comprehensive security information management system (this was before Protego Networks was acquired by Cisco Systems). In an ideal world, our agency would be able to go that same route. Of course, it’s not an ideal world. We are a state agency with no budget to purchase such commercial products.
The answer, as it so often has been since I entered the public sector, was to turn to open-source. I went on a quest to find an open-source tool that would allow me to be notified when a change on the network occurred, malicious or not.
I was racking my brain for a way to monitor the network and somehow gain insight into potential security incidents via the use of a single open-source tool. I had heard about Nagios, an open-source host, service and network monitoring program, but had never used it. I knew it was primarily used to monitor networked resources. I learned that it can be used with Snort, Nmap, Nessus and some of the more common open-source network security tools and can become an aggregation point. I was a little excited until I downloaded the installation guide — 200 pages. I groaned. If nothing else, I thought, this job is forcing me to become more proficient in Linux.
The other advantage I am finding in seeking out, selecting and implementing open-source alternatives for network and security management on no budget is that I’m becoming more technically able, which is probably a good thing.
“Kelly,” whose name and employer have been disguised for obvious reasons, is a security manager. Contact her at mscjkelly@yahoo.com