Data IQ
News

News

Articles

Articles

Reports

Reports

Market Insight

Market Insight

Conferences

Conferences

Awards

Awards

Member Workshops

Member Workshops

Leadership Events

Leadership Events

Recognition
DataIQ 100

DataIQ 100

Advisory Board

Advisory Board

Partners

Partners

New to DataIQ?
Knowledge
News

News

Articles

Articles

Reports

Reports

Market Insight

Market Insight

Events
Conferences

Conferences

Awards

Awards

Member Workshops

Member Workshops

Leadership Events

Leadership Events

Recognition
DataIQ 100

DataIQ 100

Advisory Board

Advisory Board

Partners

Partners

New to DataIQ?

You are viewing 1 of your 3 guest articles for this month.

You can access an unlimited number of articles for free by registering with DataIQ.

Register

Already registered? Login

The making of a simple yet complex price predictor

Fergus Weldon, director of data science, and his team of eight data scientists have built a price prediction model for the users of the online rail and coach ticketing platform Trainline.

Linked InTwitterFacebook
train ticket barrier.jpg

Rail tickets become increasingly expensive the closer you get to the date of the departure, and so the data team built a price predictor that learnt those patterns and predicts when they believe a particular price would expire. “As of June 2018, we’ve saved people who use our feature about £9 million,” he said.

 

It sounds simple, but Weldon explained that it is far more complicated. “What looks like a really simple feature is actually a complex system and it has taken about two years to get to the point at which we could release it at scale,” said Weldon.

 

During those two years Weldon has learnt some tough lessons. The first is to take the rough with the smooth. “I find it very, very exciting to be working in data and I feel very privileged to be here at this point in time. But it is hard and you need to be willing to accept that,” he said.

 

While he stated that if you took the model and boiled it down, you would end up with a “small serialised object,” he also likened the difficulty of creating of the price prediction model to building the pyramids of Giza.

Train station Liverpool Street.jpg

He is also a big advocate of using open source software and platforms. “In order to scale you need to build that capability so look to the cloud, look to open source software packages, and open source frameworks. You are not going to be inventing something that hasn’t been invented before.”

 

 

"Building with AWS is like building with Lego blocks."

 

Weldon uses Kinesis, Elastic Container Service and SQS from Amazon as well as Kafka for real time data capture. The data processing is done with Amazon EMR, Spark and Python, and it then goes into an Amazon S3 Bucket data lake. “Building in AWS is like building with Lego blocks. You are literally putting different pieces in the system together,” he stated.

 

Weldon also said that a data scientist shouldn’t be the first person you hire. “If you are looking at building data capabilities, don’t think data scientist first. They will just turn into a data engineer. Go find a data person who can help you build your capability.”

 

He then said that when you do hire data scientists, they need to be put to work on projects and tasks that will reap the most benefit for the business. If not, it may be harder to grow the team in the future. “Data scientists are not cheap so bringing them in and having them work on something that is not the highest ROI for your business is not going to build a business case to build the team out further,” he said.

 

Lastly, done is better than perfect. “I’ve had the privilege of working with some very, very smart people in my career. Typically, you hire a researcher and give them a problem to solve but they are going to want to research that to perfection. But there’s a balance between deriving value and releasing that value early, and actually writing a PhD paper.

 

Fergus Weldon was speaking at Chief Disruptor Live.

Linked InTwitterFacebook
Add New Comment
If you would like to comment please Login

Did you find this content useful?

Thank you for your input

Thank you for your feedback

Next read

NEW! DataIQ Podcast - 7: Parkinson's and ONS

NEW! DataIQ Podcast - 7: Parkinson's and ONS

Next event
DataIQ Transform

DataIQ Transform

Sign up for DataIQ

Access to all data news and articles
Access to DataIQ 100 profiles - the most influential people in data
Invitations to leading industry events
Everything delivered to your inbox

Next read

Should AI systems be given MOTs to put the brakes on hackers?

Data questions then answers or data answers then questions?

The doctor will text you back now?

You may also be interested in

The importance of data role models

The importance of data role models

C2D3 scales up real-world research problems

C2D3 scales up real-world research problems

What’s in a name? £35,000 apparently

What’s in a name? £35,000 apparently

JCURV's supermarket sweeping transformation

JCURV's supermarket sweeping transformation

Data IQ
Twitter
Linked In
DataIQ is a trading name of IQ Data Group Limited
5th Floor, 10 York Road, London SE1 7ND
Registered in England: 9900834
Tel: [+44] 203-829-1112
Copyright © IQ Data Group Limited 2019