Back in 1996 I presented a data engineering paper on preparing data for analytics with a “very large dataset” of 60Gb. Tiny by today’s standards.
I’ve worked in a variety of roles in “the real world”, consultancy and technology. The mainstream data work started at Centrica around 20 years ago. We had just completed a build of a customer data warehouse and I joined in order to lead the development of it; that role moved into leading on data topics like governance, architecture and quality.
While the technology changes at a fantastic pace, the challenges have remained the same. Delivering sustainable value from data relies on knowing where and how your data is created, measuring and improving the quality of the data and getting that information to the analysts and data scientists.
In my current role, I lead the team responsible for managing the Co-op’s data assets. Organisationally, we are making sure that our data design, governance and quality experts are working in teams with data engineers and scientists. Embedding our team will allow the Co-op the deliver value from data more quickly than before.
Winning the inaugural DataIQ award for best data ethics initiative ranks pretty highly in recent years. It’s been a massive achievement for the team. The knock-on benefits are fantastic, and it’s helped us to further develop our ethical approach to the data management.
I’ve found Demis Hassabis’ presentations and interviews to be really inspirational from the point of view of how he is driven to solve problems with AI. He has a really optimistic outlook on the future.
In 2019, the hype and scaremongering around the use of data and AI gave way to proper discussion and debate. We are well placed to have that conversation with our members. At the end of the year we asked our members for their views on privacy, data usage and marketing. The results were really positive, with a clear message that people want to have more information about what we are doing with their data.
I’m looking forward to the ICO’s promise of provide more guidance and regulation on the use of data in adtech and the work it is doing with The Alan Turing Institute on explainable AI in automated decisions. I think that more clarity and fairness is going to be a key feature of how data is used this year.
As AI starts to live up to its potential, its use in manufacturing technology will create huge innovations. Applying this tech in renewable energy generation and storage, for example, could be the key to providing part of a solution to our current climate crisis.
Making data lineage explainable to people; businesspeople, data scientists and customers all need to know how data is handled. We tend to take a conservative approach to using data, so being able to track its movement and usage is going to be really important to us.