In the halcyon days of my youth, one of my favourite songs was Don McLean’s “Castles in the air”. It highlighted McClean’s emotional dislocation between his desire to lead a tranquil, contented life and having to endure the pressures of the city in which he lived. The title refers to plans which have little or no chance of succeeding. I guess he realised his ambition was little more than a forlorn hope at the time he wrote the song.
A perusal of the media in recent weeks demonstrated to me how so much data exploitation and management also appears focused on building castles in the air. Undoubtedly, data’s potential to revolutionise society and all our lives is an exciting and enticing prospect.
I recently watched an excellent BBC documentary on how data can do this. Entitled “The joy of data” , presented by Dr Hannah Fry, a mathematician at University College, London, it highlighted how data acts as a connecting bridge between the domain of concepts, numbers, abstractions and the real world. Data provides us with the facts and evidence to reduce uncertainty in our lives and society, and can improve our personal and organisational decision making.
In the programme, she went on to provide examples of how big data in particular was going to do this, exemplified in a use case of the city of Bristol, which has bold and ambitious plans to become Britain’s first truly digital city. It is starting to collect, store and analyse large quantities of data gathered from sensors and other Internet of Things (IoT) devices to transform how the city cares for its elderly, ill and vulnerable, controls traffic congestion and generally make the city a more pleasant place to live in. Maybe Don McClean should move there.
But this optimistic, even utopian, view of data and its potential power has contrasted sharply with several data horror stories that have recently appeared in the media. A study by the ECRI Institute found that many US healthcare organisations treat patients incorrectly because they confuse them with other patients with the same or similar names. This happened no less than 7,613 times in the two-year period of the research. Needless to say, the impact of this poor management of data quality can be catastrophic.
Closer to home, the National Health Service (NHS) in England reported that up to 5% of patient data held in General Practitioners lists is inaccurate, obsolete and duplicated. This also increases the chances of treatment errors and, moreover, has financial implications as the NHS pays GPs £136 for each registered patient, money it can ill afford to waste as it is under severe financial pressures.
And it’s not only in health that problems with basic data quality have been exposed. Energy regulator OFGEM found that some energy companies routinely overcharge customers because of billing errors caused by a muddle over imperial and metric gas meter readings, which measure gas consumption in different units. Some people have been overcharged by as much as 130% for their gas over many years. This is bad news for consumers, but also for the suppliers concerned as OFGEM has ordered them to sort out the mess and refund affected clients, a complex and costly undertaking.
But no article about the perils of poor data quality would be complete without another retail blooper. On its UK website, Hewlett Packard (HP) inadvertently reduced the cost of a high-spec workstation from £2,278.30 to the bargain price of £1.58. Needless to say, lots of sharp-eyed bargain hunters ordered them before HP spotted the error and corrected it. Ultimately, HP refused to sell them at that price, citing a “processing error”, but was left red-faced when many would-be buyers contacted the UK press to complain. Lots of bad publicity ensued.
So, what does all this positive and negative media exposure tell us about the data challenge? To reiterate, there is no doubt that data has the potential power to make a real difference in our lives and the wider world in which we live. But the exciting possibilities of big data, IoT, self-service business intelligence, data science, analytics and so on can only be truly realised if they are anchored on proven, root-and-branch data management principles and practices.
This may not be the most thrilling aspect of data work, any more than the task of laying the foundations of a castle, rather than building its soaring walls and turrets, but without it castles fall down and crumble. It’s hard to deliver the vision of the digital world outlined by Fry unless you get the basics right, and it’s equally clear that many organisations are still failing to do that.
Focusing on the foundations is key to data success. There are a number of universal, but essential practices. First, understand what data is really important to the successful operation of your company, whether it relates to your customers, products, suppliers, sales transactions, invoices et al. You cannot rigorously control all the data you use, so put your focus on the things that really matter.
Then make people formally responsible for this data through data governance, set data standards, analyse its quality against these standards, and put business and IT processes in place to address the quality problems. Also develop and implement policies to ensure all people in an organisation adhere to good data practices. Always make all these foundational activities an integral part of any big data and analytics project.
Sound and trustworthy data provides the basis on which the digital world and its many artefacts can and must be built. If you don’t want to build castles in the air, but data edifices that endure and last, get the basics right. Don McClean wanted to get back to basics - organisations should look to do the same with their data.