Fergus Weldon, director of data science, and his team of eight data scientists have built a price prediction model for the users of the online rail and coach ticketing platform Trainline.
Rail tickets become increasingly expensive the closer you get to the date of the departure, and so the data team built a price predictor that learnt those patterns and predicts when they believe a particular price would expire. “As of June 2018, we’ve saved people who use our feature about £9 million,” he said.
It sounds simple, but Weldon explained that it is far more complicated. “What looks like a really simple feature is actually a complex system and it has taken about two years to get to the point at which we could release it at scale,” said Weldon.
During those two years Weldon has learnt some tough lessons. The first is to take the rough with the smooth. “I find it very, very exciting to be working in data and I feel very privileged to be here at this point in time. But it is hard and you need to be willing to accept that,” he said.
While he stated that if you took the model and boiled it down, you would end up with a “small serialised object,” he also likened the difficulty of creating of the price prediction model to building the pyramids of Giza.
He is also a big advocate of using open source software and platforms. “In order to scale you need to build that capability so look to the cloud, look to open source software packages, and open source frameworks. You are not going to be inventing something that hasn’t been invented before.”
"Building with AWS is like building with Lego blocks."
Weldon uses Kinesis, Elastic Container Service and SQS from Amazon as well as Kafka for real time data capture. The data processing is done with Amazon EMR, Spark and Python, and it then goes into an Amazon S3 Bucket data lake. “Building in AWS is like building with Lego blocks. You are literally putting different pieces in the system together,” he stated.
Weldon also said that a data scientist shouldn’t be the first person you hire. “If you are looking at building data capabilities, don’t think data scientist first. They will just turn into a data engineer. Go find a data person who can help you build your capability.”
He then said that when you do hire data scientists, they need to be put to work on projects and tasks that will reap the most benefit for the business. If not, it may be harder to grow the team in the future. “Data scientists are not cheap so bringing them in and having them work on something that is not the highest ROI for your business is not going to build a business case to build the team out further,” he said.
Lastly, done is better than perfect. “I’ve had the privilege of working with some very, very smart people in my career. Typically, you hire a researcher and give them a problem to solve but they are going to want to research that to perfection. But there’s a balance between deriving value and releasing that value early, and actually writing a PhD paper.
Fergus Weldon was speaking at Chief Disruptor Live.