In 1964, The New Statesman coined the term Information Explosion to describe the deluge of paper-based data, and the problems that created.
Fast-forward to 2015 and organisations of all sizes are facing a deluge of digital data.
Studies from Gartner and IDC show CIOs and IT administrators rank managing data growth at the top of their concerns.
This is not simply a case of working out how to manage the cost of storing and protecting their ever-expanding volumes of data.
IT decision makers need to determine which data are worth preserving, and how to extract value from these data.
This issue extends from small to large organisations, affecting most use cases. Big Data is not only about the speed (velocity) and amount (volume) of data created, it is also about the variety of data types and data storage types, a much harder issue to manage.
The crux of the problem is enabling intelligence and unified management across a variety of data types and storage types.
In a data-centric world, limiting users’ and IT’s ability to globally manage and analyse all types of data leads to greater complexity, the cost of managing this complexity, and importantly, reduces your organisation’s ability to drive better business decisions.
Data stovepipes limit data value
The storage industry sees the rapid growth of data as a very lucrative problem for which they position themselves as the solution. Whatever your requirement, there are plenty of options for storing data to meet the needs of the moment.
However, even within a single organisation different data types and workflows can drive diverse storage requirements, as can the different stages of data’s lifecycle, which have different storage performance requirements.
This leads to a disparate mix of storage infrastructure, representing different storage types, of various ages, and from multiple vendors. Costs rise as data stovepipes emerge.
Stale or persistent data gets stranded in expensive high-performance systems by inertia, or by the complexity of trying to figure out which data should be disposed of, which should be retained, and how to keep track of where these data reside.
Simply building more and better storage containers does not solve the problem of managing data variety. It would be like car manufacturers adding more fuel tanks to vehicles in response to decreased engine efficiency.
Metadata is the Rosetta Stone
The storage industry is focused on the data, particularly capacity and bandwidth, since that is what drives the cost and design decisions for the infrastructure. But within all data are multiple sorts of metadata that hold the keys to solving these problems.