The latest tests by security testing specialist NSS Labs of intrusion protection systems show they are improving but are far from perfect in stopping hacker attacks.
Intrusion protection systems (IPS) are often the front-line guard at the door of the internet and used to detect sophisticated attacks designed to steal information or execute fraud. The systems detect attacks against applications or OSes intended to install malicious software such as keyloggers and rootkits.
NSS Labs tested 13 IPS products from 11 vendors. Those vendors voluntarily submitted their products, but nine vendors refused, says Rick Moy, president of NSS Labs. The refusal is not uncommon. Moy says more IPS vendors participated in the 2010 testing than the previous year.
"The vendors who had confidence in their products wanted to volunteer to participate," he says. "At some point, it's a marketing decision whether you participate or not."
NSS Labs measured the average default protection from exploits, which is the method by which malware is delivered. In 2009, products in their default configurations only caught on average 45 percent of attacks, but in 2010 that average was 62 percent.
McAfee's M-8000 and Cisco Systems' IPS 4260 Sensor proved the best in their default configurations at catching attacks designed to target desktop applications, with a 94.5 percent and 91.8 percent effectiveness, respectively.
When engineers from the companies were allowed to "tune" their products, or add more rules designed to catch specific types of attacks, the products upped their detection rates on average by 21 percent.
"There's a big difference between the default and the tuned for many vendors," Moy says.
Vendors also improved their performance on so-called anti-evasion techniques, where attack techniques are combined to get past security products. In 2009, half of the vendors failed to counter basic evasions, and more continue to be discovered.
But this time all but three products from two vendors passed NSS Labs' evasion testing, showing that vendors are paying more attention to the issue.
One vendor, Stonesoft, missed several of the basic evasions in 2009 but has remedied its IPS 1205 and IPS 3205 products. Stonesoft created a stir last October when it said it had discovered new anti-evasion techniques.
Still, there is a wide variance in IPS products, ranging from a 31 percent to 98 percent effectiveness on average over seven years, depending on whether the product had been tuned.
NSS Labs has estimated what each product costs to protect 1MB bps (bit per second) of data over three years, including the cost of the product itself, yearly maintenance and ongoing costs for upkeep and tuning.
It's difficult to compare since no two products deliver the same security effectiveness or throughput. NSS Labs uses a formula that divides the total cost of ownership including labor by the percentage of threats it stops -- what NSS calls the "security effectiveness" -- and then multiples that by the device's throughput.
The results take some interpretation, since the cheapest product doesn't necessarily provide the greatest value if it blocks fewer attacks, according to the report. But the figure does provide perspective on the total cost of individual products.
NSS Labs chose not to publicly reveal the results due to the proprietary research that went into producing the report, but it is for sale for US$1,800.
Five of the products were rated "recommend," meaning the product had above average security and value. The others were rated neutral, which was defined as average security scores but below average value.
Those tested were Check Point's Power-1 11065, Cisco's IPS 4260 Sensor, McAfee's M-8000, Palo Alto Network's PA-4020, Sourcefire's 3D 4500, Fortinet's Fortigate 3801A, IBM's Proventia Network IPS GX6116, Juniper's IDP-8200 and SRX 3600, Stonesoft's IPS 3205 and 1205, NSFocus's NIPS-1200 and Endace's Core-100. None of the tested vendors rated a "caution."
Companies that declined to have their products tested were DeepNines, Enterasys, HP/Tipping Point, Nitro Security, Radware, SAIC/Cloudshield, SecureWorks, StillSecure and TopLayer. Vendors were not charged to participate in the study.