A security testing firm today said a recent report that named Google's Chrome as the most secured browser was flawed -- and part of a campaign by Google to undermine Mozilla's Firefox.
The work done by Denver-based security consultancy Accuvant, which released a report last week naming Chrome as more secured than either Firefox or Microsoft's Internet Explorer (IE), was paid for by Google.
That raised the hackles of NSS Labs, a California company that tests browser security and antivirus software.
"This is a vendor-funded paper, and in these cases, the vendor is going to drive the methodology [of the testing], which appears to be the case here," said Vikram Phatak, the chief technology officer of NSS Labs, in an interview today.
When reminded that NSS Labs has conducted vendor-funded browser security research in the past -- Microsoft sponsored several NSS tests on anti-malware blocking technologies -- Phatak replied, "There's a reason why we don't do that anymore."
Calling Accuvant's testing process "skewed toward Chrome," Phatak argued that the consulting company's researchers ignored some key Firefox security features -- notably "frame poisoning," which blocks exploits of most layout code crashes; didn't give enough weight to such things as frequent security updates; and failed to use real-world anti-browser malware in its testing.
But Phatak and Rick Moy, president of NSS Labs, leveled more serious charges against Google than the allegedly-slanted report.
The two tied the release of the report with two other factors -- the apparent non-renewal of the Google-Mozilla search contract and a recent rise in Chrome's anti-malware blocking effectiveness -- to conclude that Google was running a campaign to knock Firefox out of the market.
"This tells a story, that Google is looking to go it alone now, and examining their position vis-a-vis Mozilla," said Phatak. "Google paid for this report, and it's part of a marketing campaign that's probably aimed at Firefox to cut off Firefox's revenues, cut if off from the SafeBrowsing service, and then put out a report that says Firefox is less secure than Chrome."
"I think there's consistency in the data points," said Moy.
While Mozilla has said it was "in active negotiations" with Google about a new contract, it has declined to announce whether it has reached a deal with its long-time partner. That contract expired last month.
Income from the Google-Mozilla contract accounted for 84% of the $123 million the latter reported in revenue for 2010, the last year Mozilla has made public its finances.
The other factor Phatak and Moy used to bolster their claim of a concerted effort by Google to squash Mozilla's Firefox came from NSS Labs' own testing, which showed a five-fold jump in Chrome's effectiveness blocking malware in an 11-day period from Nov. 22 to Dec. 2.
Chrome, like Firefox and Apple's Safari, uses Google's SafeBrowsing service to block malicious websites and potentially malware-infected downloads.
As Chrome's blocking rate soared from just 8% to 40% in that 11-day period, Firefox and Safari both declined to less than 2%.
NSS Labs' conclusion: Google is keeping some blocking protection from the SafeBrowsing API (application programming interface) tapped by Firefox and Safari.
"It appears Google has purposefully withheld important malware protection from its SafeBrowsing feed coinciding with its break from Firefox and release of the Google-funded report by Accuvant," NSS Labs wrote in a document published Tuesday ( download PDF ). This episode could indicate a more aggressive direction for Google."
NSS Labs also pointed to the release date of the Accuvant paper as another piece of the plot puzzle.
"The report was completed in July, but they held it for some reason," said Phatak.
Accuvant researchers denied that Google had in any way influenced their testing process or the conclusions they reached in their report.
"Google came to us and say, 'Hey, we want a browser security comparison [and] we don't have anything in mind, so come up with something to compare browsers,'" said Chris Valasek, a senior research scientist at Accuvant Labs, and one of six researchers at the firm that contributed to the testing and the ensuing report.
Accuvant approached browser security differently than earlier attempts, which have relied on comparisons of vulnerability counts, the time it takes a vendor to patch or pointing browsers toward known malware sites.
Instead, Accuvant created testing tools that allowed its researchers to investigate the browsers' -- and Windows' own, since it tested only on that OS -- anti-exploit mitigation technologies, and determine exactly what those technologies did or did not do.
Chrome, concluded Accuvant, was the most "secured" -- Ryan Smith, chief scientist in the group, made a point to use that term rather than "secure" -- browser in large part because of its sandbox, technology that isolates the browser and its processes from the rest of the operating system and machine.
Through sandboxing, Chrome ensures that exploit code which does make it onto a system either cannot, or can only in extreme circumstances, wriggle out of the browser itself.
Other anti-exploit technologies that Accuvant tested for Chrome, Firefox and IE included ASLR (address space layout randomization), DEP (data execution prevention) and JIT (just-in-time) JavaScript hardening.
Accuvant downplayed traditional comparison methods -- relying on vulnerability tallies, for instance -- as too variable by vendor, and said that anti-malware blocking was also worthless, noting, "The URL blacklisting services offered by all three browsers will stop fewer attacks than will go undetected."
"We like to think that we've advanced the state of the art in browser security comparison," said Accuvant's Smith.
"This is a better litmus than a shouting match," added Valasek, who pointed out that Accuvant has released not only its paper, but also the testing tools it built and the data those tools collected.
Accuvant's conclusion: Chrome was the most secured browser of the trio, with IE and Firefox in second and third place, respectively.
NSS Labs gave Accuvant credit for the research even as it claimed that the testing was flawed.
"They created some good tests, did some decent work," said Phatak. "It's a good first step, but it needs to be an evenhanded treatment. We think [the test methodology] skews the results in a way that biased the end results."
Phatak and Moy said that NSS Labs had "no contractual relationship with Mozilla" when asked whether it was fronting for the open-source firm.
Google did not reply to a request for comment on NSS Labs' charges. Mozilla declined to directly react to NSS Labs' contention that Firefox was shorted by the Accuvant tests, or comment on the claims that the report was one part of a multi-stage campaign against the company.
Instead, Mozilla repeated a statement it issued last week when Accuvant released its paper.
"Firefox includes a broad array of technologies to eliminate or reduce security threats," said Johnathan Nightingale, director of Firefox engineering, in that statement. "Sandboxing is a useful addition to that toolbox that we are investigating, but no technology is a silver bullet."
Accuvant's browser security report can be downloaded from the company's website ( download PDF ).
Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer , on Google+ or subscribe to Gregg's RSS feed . His e-mail address is gkeizer@computerworld.com .
See more articles by Gregg Keizer .
Read more about browsers in Computerworld's Browsers Topic Center.