“Is it just us or has all data gone real-time overnight?” That question was asked - and answered - by Ciaran Dynes, VP products and marketing, Talend during an interview with DataIQ in London late last year. It reflects a rapidly changing market for the company’s ETL services which are increasingly being used to manage what it describes as the “data fabric” underpinning modern enterprises.
Reflecting this rapid change, Dynes noted that there is an important role for educating practitioners into what is possible. “The market is growing through a bit of push and a lot of pull. Clients often ask us where others have got to because they don’t want to be on the ‘bleeding edge’,” he pointed out.
Presentations at conferences are part of this process, as are case studies, such as BetVictor. During major sporting events like Euro 2016, it has a pressing need to integrate data on its 500,000-plus customers worldwide from across devices and touchpoints in order to improve their betting experience as well as to drive up revenues.
As Dynes pointed out: “The gaming industry is really driving performance in a highly-specialised way. We are helping their network to run better.” Gaming is the very definition of real-time business since bets can only be taken while an event is live. Competition is also so intense that ensuring customers can place their bets quickly - and take their winnings - is critical to retention.
But it is not the only sector pushing real-time data integration. “Our blue-chip clients are coming back and want real-time processing, whether it is for self-service BI or connected servces. For example, GE has a gas turbines division whose clients want maximum longevity and return on their investment, which may be £15 million each over ten years, so they are running real-time predictive maintenance based on sensor data.”
Growth in the internet of things, especially from sensors embedded into critical systems, ranging from turbines through to airplane engines. But while IoT is expanding the market for ETL, it is also bringing with it pressures that were previously out of its domain. “Data ownership is another emerging issue. For example, if Airbus is doing predictive maintenance, it has to plan when to carry it out during a plane’s flying schedule. But who owns the data being generated by that plane?” he asked.
This is also forcing companies to rethink the data-value exchange which they are offering to customers. Said Dynes: “Data has become more complicated in a good way. There are big ethical issues that need to be worked out and built into governance. Historically, people have just been giving their data to companies and those companies have used it to create value, which is not what the individuals intended.”
Dynes is well-placed to help Talend benefit from this new market direction. He noted: “I come from a real-time background as opposed to ETL, which has a mindset that is very good at breaking down problems if you have got the time. Real-time is about doubling down and problem-solving in a different way.”
Real-time may be growing fast, but the original market for ETL in batch integration is still important and a major client-base for Talend. One billion-dollar US business is trying to pull together customer information from 39 different ERP systems in order to understand its customers buying patterns. “They need that basic integration,” said Dynes, “but once they have done it, what else could they do?”
If digital platforms are accelerating the pace of data creation and real-time decision making and reporting is increasingly on the corporate agenda, it is not only the underpinning technology which needs to change. An important parallel shift Dynes points to is in the culture of IT departments.
“They used to be perceived as antagonistic towards the business, but IT has now backed off from that mindset and is being co-operative with stakeholders across the organisation,” he noted. It is not just data which is becoming ever more integrated - the functions who manage and exploit it are, too.