Businesses and public sector organisations around the world are waking up to the possibility of exploiting big data to generate new insights which deliver step changes in operational efficiency and gain competitive advantage. The benefits for both individual organisations and the wider economy could be huge.
Gleaning valuable intelligence from big data requires new tools and techniques. Large enterprises, such as financial institutions and telecommunications providers, now routinely measure their data in petabytes. But size is not the only challenge. This data also has ever-more diverse structures and formats and often is generated in continuous streams – leading to data with levels of scale, speed and complexity that traditional business intelligence tools are not designed to cope with.
As a result, we are witnessing an exciting chapter in the development of data analytics, with a plethora of new tools and innovative start-ups emerging to address the storage, processing and analysis challenges presented by big data.
However, while many major online brands are famous for pioneering the approaches and technologies behind big data – such as Hadoop – organisations in other sectors often face significant challenges when attempting to leverage the same technology and have been slower to adapt.
Exploding data volumes are already driving up spend on data storage and consuming more and more of the IT budget. Integrating a big data solution into the existing IT estate may add further cost and complexity. While many new big data solutions are generating excitement and interest, they are often immature compared to other enterprise platforms, and depend on specialists with deep technical skills to make them work.
Many organisations therefore find themselves with a dilemma: how to demonstrate to their boards the potential value to be realised from big data, without incurring the cost and risk of setting up the capabilities to exploit it, and how to invest in these new capabilities without a proven business case to justify the costs in the first place?
One option is to use a big-data-analytics-as-a-service platform as the initial test bed for understanding the potential value. These platforms are run by external service providers, who have pre-built big data environments and data scientist teams who are experienced in using a range of big data tools. Once a secure data transfer has been arranged to the platform, it is possible to use it to undertake rapid data discovery and analytics on hitherto untapped sources of data, using an experienced team and without having to stand up the infrastructure and technology in-house.
Doing so means organisations get much greater speed-to-value - getting the insights they need quicker - with lower risk and that they are able to experiment to establish what works best for them. As speed-to-value becomes even more critical and businesses increasingly need big data analytics to be swift as well as insightful, we see big data managed platforms as being the key to unlocking value from big data over the next couple of years.