The amount of data being produced and replicated worldwide is doubling every two years, and this is posing a new challenge to data managers, according to a recent study.
Up to 1.8 zetabytes or 1.8 trillion gigabytes of data are expected to be created and replicated in 2011, according to the IDC Digital Universe Study sponsored by EMC.
At this rate, the report said data explosion is far exceeding Moore's Law, formulated by Intel co-founder Gordon Moore. It estimates that the number of transistors on a chip roughly doubles every two years, resulting in smaller chips.
At 1.8 zetabytes, the study estimates that it is equivalent to 57.5 billion 32GB Apple iPads or enough iPads to build the Great iPad Wall of China at twice the average height of the original.
The data explosion has cost enterprises up to US$4 trillion since 2005, according to the fifth annual study, titled 'Extracting Value from Chaos'. The investments have been put in cloud services, hardware, software and staff.
Challenge
The "colossal" growth in data generation, said the report, is posing a challenge to IT departments who, in the next decade, will be responsible for at least 10 times the number of virtual and physical servers, and 50 times the amount of information to be managed.
IDC said "the skills, experience, and resources to manage the deluge of data and resources simply are not keeping pace with all areas of growth". By 2020, the IT departments will have to deal with 1.5 times the number of IT professionals to be able to manage the data explosion.
Still, for every challenge, there are opportunities.
"The chaotic volume of information that continues growing relentlessly presents an endless amount of opportunity-driving transformational societal, technological, scientific, and economic changes," said Jeremy Burton, chief marketing officer, EMC.
"'Big Data' is forcing change in the way businesses manage and extract value from their most important asset - information. EMC is at an ideal crossroad to help our customers-from the world's largest enterprises to governments to small businesses-exploit the hidden value in the digital universe as they continue on their journey to the cloud."
Cloud computing
One area of opportunity is in cloud computing. IDC estimated that by 2015, nearly 20 per cent of the information will be somehow "touched" by cloud computing, or that some of the information will be somehow stored or processed in a cloud environment. As much as 10 per cent of the information will be maintained in the cloud.
The report said cloud computing - whether public or private, or hybrid - can give enterprises the economics of scale, agility and flexibility to manage the growth of data. "Long term, this will be a key tool for dealing with the complexity of the digital universe," the report stated.
