Statistical reports on contentious topics are of limited long-term value. Many just get filed away and forgotten about. More are just of limited use for a particular period and business environment.
The report published last week on the performance of IT projects in public and private sectors professes to tackle a more long-term bugbear and put some figures on it for the first time.
But the document that has emerged is hardly likely to lay any ghosts to rest in the debate over failed projects.
The survey produces a figure for government and private sector projects that seems on the face of it very worrying.
On strict criteria of meeting budget and time scale, only around 30% of IT projects ever "succeed".
But success is a slippery concept to define. Many IT ideas may be floated, taken a little way, then abandoned as impractical.
This may actually be a positive. As one commentator put it "better to hit the off-ramp early than persist with something that is not going to meet requirements".
Two or more candidate approaches to solving a problem may be tried, and only one adopted - as a respondent to the 1998 US survey used for comparison purposes tellingly put it: "We have 500 projects this year, 40% will be cancelled." Do we count those 40% as "failures"?
The report, moreover, includes so many caveats as to make figures quoted to within 1% seem suspect at the least and at the worst worthless.
Two of the highest-profile failures, the Police Incis project and the Corrections Department's Inslaw project - are not even factored into the figures. The survey was conducted by voluntary reply and achieved only about a 30% reply rate.
This surely indicates a measure of self-selection, which may have eliminated some of the organisations most prone to project failure.
Furthermore, at least one question was misleadingly phrased, perhaps leading respondents to under-report uncompleted projects.
Certainly it led to ambiguity over the cause of the most prominent anomaly in the survey - the 2% of projects ranked as complete cancellations, as opposed to 28% in the US analysis.
This, sources connected with the survey admit, may have been due to the misphrased question. On the other hand, it may say something genuinely valuable about New Zealand organisations' unwillingness to abandon projects afflicted with problems. We just don't know which.
For purposes of comparison with a US survey, the New Zealand exercise chose to adopt the very tight criteria of "on time, within budget, providing full functionality".
The author of the US survey, the Standish group, provides advice on project management, so it is hardly surprising that their survey concentrated on good project management matters, rather than whether the product of the development was worthwhile. This has been allowed to influence the New Zealand results.
To their credit, the Simpl Group and NZIER, authors of the survey, re-analysed the local figures on a basis that included value to the organisation of the final product.
Under this, the percentage of satisfactory projects was much improved, raising "success" figures to 83% - or 88% for Government.
This leaves us with the question: which definition of "success" do we work to? Or should we realistically estimate that it's somewhere between the two? There's a lot of space between 30% and 88%.
The report has a few lessons, garnered from requests to the participants to identify the causes of success and failure; but there is little new.
"Take small steps in development", "avoid the bleeding edge of technology", "get top management involved" and "adopt professional project management" have been so often mentioned as to be virtual truisms.
The authors are pleased to have previous anecdote confirmed by "evidence", says a Simpl spokesperson; but it is surely doubtful that the opinions of people expressed in interviews are really any closer to "evidence" than the original anecdote.
All in all, the long-awaited report gives us little to chew on - certainly little new - regarding this vexed topic.
It could be another one for that dusty shelf in the corner of the office.
Stephen Bell is a Computerworld journalist based in Wellington. Send email to Stephen Bell. Letters for publication should be sent to Computerworld Letters.