A recent study by Enterprise Strategy Group has found that databases used for daily processing tasks are ballooning at a double-digit rate each year, and may contain piles of inactive data.
Fifty-nine percent of respondents said their primary database instances are growing by between 11% and 30% each year. Overall, 9% reported the annual rate was greater than 51%.
Also, on average, respondents said only 50% of an average primary database instance contains "active" or transactional data, suggesting that the rest is relatively idle, the study found.
"What that says to me is we're just keeping more and more information inside these databases that we don't need, or have to keep for compliance or something else," says Brian Babineau, one of the ESG analysts who worked on the report. "It's not a smart move for IT to keep doing it the way it's been doing."
The way companies handle day-to-day tasks, such as creating database instances for development and testing, is compounding the data-bloat problem, ESG's data suggests.
"Despite the fact that many test and development applications rarely need a full copy of a data set, 61% of secondary instances created for testing purposes are full copies of the primary database instance, as are 65% of those created for development purposes," the study notes. "Most of this data is then retained for four or more years, fueling the growth of storage requirements and costs much faster than actual business requirements demand."
ESG predicts that database archive capacity worldwide will grow from 1,198 petabytes in 2007 to 13,639 petabytes in 2012.
Yet just 6% of respondents said they frequently use a "purpose-built" archiving tool to build secondary databases, according to ESG.
A range of vendors, from smaller players such as Applimation and NEON Enterprise Software to larger vendors including IBM, are selling products targeted at this issue.
Applimation makes products that are used for archiving, securing and creating subsets of Oracle, Siebel, PeopleSoft and SAP application data. NEON makes TITAN Archive. IBM acquired Princeton Softech last year as part of its broad-based Information on Demand strategy.
Small vendors like Applimation are trying to find footing in the market through specialisation. "It's a focus thing," Babineau says. "They focus specifically on integrating with pre-packaged applications."
But given the converging factors at work — the cost of adding storage, along with the need for data to meet privacy and compliance concerns — there is room for a number of players, Babineau says. "I think [the market] is going to get a lot bigger," he says.
ESG surveyed 142 "IT decision-makers" last year for the study.