Love it or loathe, big data is now firmly in the mainstream and forms part of many industry sector's data strategies - from finance, retail and telecoms to insurance, utilities, and government. Drilling down into large data sets is now a crucial business practice.
But the rise of such techniques has triggered fears among both consumer groups and regulators that automated decision making is having a major impact on individuals without them having any awareness or understanding of how those decisions have been made. The UK Parliament's Science & Technology Committee has been taking evidence on the issue and MEPs have just passed a resolution making a similar demand for accountability.
Following an investigation into the issue in the US, the Federal Trade Commission (FTC) has called on companies to check how representative their datasets are, whether their "data model" takes account of biases and how accurate are the predictions they make when based on big data.
It cited the case of how one credit card company - Compucredit Corp - was forced to change its processes after the FTC found that it had failed to disclose its practice of rating consumers as having a greater credit risk because they used their cards to pay for marriage counselling, therapy, or tyre-repair services, based on its experiences with other consumers and their repayment histories.
It is an issue which is very close to home for Blueberry Wave head of operations Steve Mattey. He explains: "I have myself been on the receiving end of a rejection for a credit card application. I queried it. I was told, 'we can't tell you why - it's the scorecard'. As it happens, I know a thing or two about scorecards and I know my financial rating. I'm a good bet financially. I was turned down because I already had one credit card with that particular company and they didn't want me benefitting from deals on the other. Why not just tell me that?”
He adds: ”It was a slight irritation to me, but this sort of automated decisioning could drastically affect other people who desperately need financial help and are instead driven into the hands of deplorable businesses which offer punitive rates of interest."
One solution being considered is to require both public sector and private companies to make their algorithms more transparent. If such a requirement became law, what would be the implications for brands and data companies alike?
Sainsbury’s head of data science and algorithms Enda Ridge, who is also the author of “Guerrilla Analytics – a practical approach to working with data”, is in no doubt that full algorithm transparency could be onerous for some businesses.
He says: "To make an algorithm completely transparent, you need to track many things. For every algorithmic decision, you need to know the input data it received, the rules used to prepare that data, the rules the algorithm used to process that data and the output calculations the algorithm produced. All of these things change over time because new data arrives and algorithm code is further developed to improve and maintain the algorithm.”
Ridge explains: ”Things get more complicated when more sophisticated algorithms are taken through guided training, rather than directly programmed by a human, using machine learning. In these cases, the training data the algorithm is exposed to also influences its final behaviour."
Charles Ping, chief executive of data agency Fuel, goes even further: "[If greater transparency became law] it could remove almost all propriety knowledge and know-how and severely impact investment and development in technology. It has the ability to send the UK right to the back of the queue as either a good place to work/invest if analytical competitive advantage is eroded within a short time. It runs against all the tenets of commercial motivation that have existed since the industrial age started, where being better and smarter in a market can convey financial advantage."
Even so, as Janet Snedden, deputy managing director and strategy director at Amaze One, warns, if a company can not readily explain why its automated data models operate in the best interests of the consumer, they could soon become obsolete. "To avoid ‘discrimination by algorithm’, targeting models may require adjustment, depending on their nature and the data sets used. As the use of artificial intelligence and black box techniques proliferate into everyday use, there will be the added problem in that neither the model nor the outcome may be explainable - while sophisticated machine learning approaches are only as good as the datasets that they are trained upon, they work by integrating multiple layers of inference and not neatly annotatable logic.”
According to Snedden, ”what goes into models or decisioning applications in the first place will therefore be placed under more and more scrutiny and this will naturally attract more attention for what personal data is being collided for what purpose. In situations where the automated outcome favours the business at the expense of the customer, however, brands will risk losing trust and revenues as alternative vendors are sought. Advanced practices in yield optimisation in industries where there are many options for services or bundles, for instance, will need to be modified if brand reputation is to be protected."
Insurance companies are already being monitored, but what other sectors are likely to be affected by the move? Caroline Kimber, data strategy director at Stack, reckons that financial services organisations are likely to be the most affected as automated decision making has the biggest impact on consumers in this sector. "Being turned down for a mortgage or loan clearly disadvantages an individual much more significantly than, say, not being granted a test drive of a £50,000 car," she adds.
But Snedden sees a much wider spread of sectors, including media, where consumers are starting to feel over-targeted - retail banking, where product targeting and pre-screening may limit access; public sector, including healthcare, where the potential for the suspension or withdrawal of services will have huge social implications; and, finally, the recruitment market, where the screening of CVs could become biased. The FTC study has already flagged up the latter - it found that one company determined that employees who live closer to their workplace stay at their jobs longer than those who live farther away and therefore only recruited local talent.
So does automated decision making really impinge on privacy and legal rights in the way claimed?
Kimber is not so sure. She explains: "Automated decision making only impacts on privacy and legal rights if consumers are disadvantaged as a result of the decision making. The reality is that automated decision making is often very advantageous for the consumer, for example, saving consumers time by only displaying products online that they are interested in or saving them money by offering discounts on add-on products related to a previous purchase they have made."
But Mattey reckons there does need to be a safety valve as there are people on the receiving end of those decisions. As a strong believer in the correct ethical use of personal data, he maintains he has never been comfortable with the fact that people's personal information can be used in ways that they are, at worst, totally unaware of and, at best, only vaguely told about due to sleight of hand being applied through the small print.
"There must be a mechanism for transparency in any decision made about people which is automated. Some of these will be low-key and of minimal impact - for example, I will get a different promotional offer than the next person because our behaviours are different. Not really that important in the grand scheme of things - I can moan and I will probably get the better offer if I want to. Some, however, can be life-changing, for example, financial suitability or even allocation of medical resources. Someone must stand up for the rights of people in these situations,” argues Mattey. "Any automated decision should have the right to be queried by a consumer who feels aggrieved and wants to understand why. There should be an appeal ombudsman, too, to bring business to account."
Sainsbury’s Ridge adds: "We need to strike a practical balance for the good of the customer, both the potential impact on them, but not stifling the many benefits and innovations we all receive from hundreds of algorithms in modern life - navigation, hotel booking, product recommendations, Internet search, to name a few."
With GDPR looming - and transparency now becoming the new currency - how can brands get ahead of the game and play this to their advantage? Or should they be making the case this is an unnecessary clampdown and that so-called big data has more benefits than drawbacks?
For Ping, it is a clampdown based on some specific, future concerns and companies would be wise to consider them. He comments: "It is important that brands are aware, but it’s also important that the Government fully understands the key use cases and doesn’t create a knee-jerk response."
The introduction of the GDPR next year will inevitably heighten awareness and sensitivity about personal data use and control, Snedden believes, adding that while getting your house in order for compliance is a must, the more savvy operators are already mobilising their data assets to deliver enhanced services and an improved customer experience. She says: ”Those choosing to view GDPR as an opportunity to hone their market advantage will prosper, such as the likes of Toyota which understands that the future of automotive will not just be about manufacturing vehicles, but owning the relationships with customers who require personal transport solutions."
As many companies have discovered to their cost, in these days of consumer empowerment and social media backlashes, there is nowhere to hide for companies when their "sleight of hand" practices are finally exposed. The regulators will no doubt get there in the end - after all, it took over five years for GDPR to be passed. But if they don't bring you down, consumers certainly will. The question is, are you willing to take that chance?