Is there such a thing as too much data? If your ability to handle it is not well developed, the answer is yes. So if you want to take advantage of big data, it’s time to improve your data capabilities, says Christine Andrews, managing director, DQM Group.
Big data is big news. Everyone seems to be talking about it. Indeed, there is little doubt that the volume, variety and velocity of big data present huge opportunities. These range from the ability to recalculate entire risk portfolios in minutes, which would once have taken days, to sending tailored recommendations to mobile devices just when target customers are at the right location, which just couldn’t happen without sizeable amounts of data.
Social media which was not even around five years ago now present a whole new set of unstructured data to analyse and from which to detect new market trends. The stats are mind blowing: 340 million tweets per day, 901 million active Facebook users. Yes, data lovers - we are literally drowning in it!
Indeed, many businesses are concerned that, with the amount of data they are amassing, they can’t see the wood for the trees. So finding the gold is becoming quite a challenge. What if your data volume gets so large and varied you don’t know how to deal with it? Do you store all your data? Do you analyse it all? How can you use this knowledge to your best advantage?
These were just some of the challenges put by the Economist Intelligence Unit (sponsored by SAS) to a number of senior executives in 2011 and again in 2012. What was interesting in the 2011 survey was that 50 per cent of respondents said data had become an important factor for their business, with nearly 10 per cent saying that data had completely changed the way their company worked.
So far, so good. However, the EIU report also found that many companies were still struggling with the basic aspects of data management and with their attempts to exploit data effectively. In the 2012 survey, some more interesting facts emerged. Tempting as it is to think that it’s big data technologies that are transforming businesses, many executives raised concerns that they didn’t have the talent within the organisation. There just weren’t enough analytical thinkers who had IT as well as business skills - 41 per cent of respondents said a lack of skilled staff was hampering their attempts to process data efficiently.
Many companies claimed they were collecting data from web tracking and social media, but were less confident about how they could integrate this unstructured data with their current data warehouses. Interestingly, many of the better-performing companies had a clear data strategy which they had put in place before starting the process of collecting large amounts of data.
So who is doing big data well? A good example is EMI, which spotted something interesting in 2011 - that new artists had a strong following among certain groups of young people, but little recognition in other demographic groups. This knowledge in the past was almost certainly confined to managers who knew a lot about the industry, but not about the data.
EMI decided to build a database from over 1 million customer interviews over 25 countries with each one generating over 100 pieces of unstructured data. The results were then matched to a stream of data from Spotify, which has 15 million tracks and three million paying users. Anonymised data on every track a user listens to was combined with the EMI interviews which enabled EMI to track an artist’s popularity among different demographic groups, so they are now able to target their marketing spend accordingly.
The mobile network operators such as O2 and EE provide another good example of big data in action. For years they have had vast amounts of customer data, transactional data and, more recently, loyalty data. They now want to be able to push relevant offers to customers at the point when they are in the vicinity of an outlet for which the network knows through its profiling activity a customer may be interested in an offer. This capability can only be achieved using the power of big data.
So where does this leave your own big data programme? Well, my take-out from reviewing the EIU survey and looking at good big data users is that there is little doubt that big data can be transformational. But you need to know what you want the data for before collecting it. Without a clear strategy, it is unlikely the big data project will succeed. Essentially, you need to know whether you have the capability within the business to make the best use of the data. In other words - you need to know how data capable your business is.
Therefore, before embarking on a big data project, we strongly recommend companies consider undertaking a data capability assessment - a test, if you like, of how they are currently analysing, maintaining, measuring, governing and coping with their current data.
All too often, the CTO has commissioned a big data initiative without the business really being ready to cope with the volume and variability of data. As the EIU survey highlighted, many companies just don’t have the skill sets within their analysts or “data scientists” to bridge the gap between IT and the business to fully exploit the value of the data. Equally, without a data strategy and clear goals on how you’re planning on using the big data, chances are the initiative will end up being a costly white elephant.
A data capability assessment enables you to get a really good measure of your data maturity across aspects such as people, process, policy, compliance, security and measurement. Using a series of structured face-to-face and telephone interviews, often involving third parties, together with a tailored range of online questionnaires, we build up a quick, but thorough picture of an organisation’s data capability which is fed into our scoring model.
We also ask the executive team specific questions about how data is used to run the business, for example, whether data is more important than the company’s brand. In all, there are over 300 questions that we can use to build up a clear picture of an organisation’s data capability. What is useful about this approach for many executive teams we engage with is the fact that the output from the initial discovery phase covers not just how data is being valued and used with an organisation, but also people, policy, process, compliance and measurement .
The approach enables an objective assessment of the current state and a benchmark for future assessments. What happens after the initial discovery phase is then a prioritisation process assessing which elements of the findings need to be addressed by whom and when so that the company can move up the maturity curve. It provides a roadmap, if you will, covering all the recommendations.
It might be, for example, that third parties are presenting a security risk and a programme to address this is required, or that better data quality metrics are needed, or sharper data processes. Frequently up to 15 work packages emerge which, once the company embarks on them, moves the organisation from the current capability into a new level of skill sets and processes that really allow big data to be fully embraced within the organisation.
So, good luck with your big data programme. But do consider your current data capabilities before you start!