Benchmarking Group Withdraws Microsoft SQL Rating

FRAMINGHAM (07/06/2000) - Microsoft Corp.'s SQL Server 2000 database, which is set to ship within a month, suffered a double embarrassment this week. Record TPC-C benchmark numbers, which Microsoft has touted for the past five months, were scratched from the official records after the Transaction Processing Council found the results "noncompliant." And IBM Corp. published numbers twice as high as Microsoft's with its own DB2.

At the launch of Windows 2000 in February, Microsoft Chairman and Chief Software Architect Bill Gates touted the performance numbers, which showed SQL Server 2000 and Windows 2000 processing 227,079.15 transactions per minute (TPM-C), the highest ever measured. The result, which beat high-end RISC servers from IBM and Sun Microsystems Inc., was achieved with a cluster of 12 Compaq ProLiant 8500 servers. The TPC-C numbers have figured highly in presentations by Microsoft executives over the last five months.

But at a meeting of the TPC last Thursday, the benchmark "was found noncompliant to our policies," said TPC chairman Jerrold Buggert. "Given the nature of the problem, the tests need to be redone."

The Compaq/Microsoft benchmark was challenged because the tested configuration didn't allow for the primary key of a distributed database to be updated. TPC declined to identify who challenged the results, citing the confidentiality of the challenge procedure.

Steve Murchie, group product manager for SQL Server at Microsoft, said the TPC's rules are ambiguous on this issue, and claimed previous benchmarks published by Houston-based Compaq Computer Corp.'s Tandem unit and by Oracle Corp. in Redwood City, California, have also failed to implement primary key updates. Officials from those companies weren't available for comment by the deadline for this article.

According to Murchie, Microsoft has now modified the SQL Server 2000 code to include the required feature, and it will be in the product when it ships in the next few weeks. Murchie said the missing feature did not materially affect performance in the TPC-C benchmark. Microsoft will rerun the tests with the newest SQL Server 2000 code and publish a new benchmark in about a month, said Murchie.

It is not unprecedented for benchmark results to be removed in this way, said Jonathan Eunice, an analyst at Illuminata Inc. in Nashua, New Hampshire. TPC-C benchmark results are open to peer review. "You are policed by your competitors," said Eunice. "It's not pretty, but there is a kind of efficiency to it."

Rerunning the benchmarks could easily cost Microsoft and Compaq US$1 million, Eunice estimated.

The SQL Server-on-Windows 2000 benchmark would have lost its first-place ranking this week anyway, with IBM's publication on Wednesday of a record 440,870 TPM-C result for DB2 running on its NetFinity servers.

The IBM benchmark does return Windows 2000 to the top of the TPC-C ranking, since it was run using Windows 2000 Advanced Server.

Eunice said TPC-C benchmarks performed on clusters -- such as the IBM NetFinity and Compaq ProLiant clusters -- aren't as important as single-server numbers.

"From a manageability perspective, people don't like to run a 16-way cluster if they can get a single SMP server (with the same performance)," said Eunice.