NOTE: This article was originally published at the Revolut Tech publication on Medium on November 14th 2019.

Have you ever experienced the dismay of getting a call from your bank’s security team? The voice on the other end of the line reports that they detected a suspicious transaction.

“Please confirm that it’s you making a payment for $149.99.”

You start nervously recalling your latest purchases:

“Am I losing track of the things I buy, or is someone really trying to rob me?”

Your palms are sweating. It’s not you, so you ask your bank to cancel the payment. For the next few hours you’re overwhelmed with uneasy thoughts:

“How did the perpetrator get my payment data? Is it going to happen again? What did they buy?”

It’s irritating, to say the least, and I’ve been there myself. That’s why when I received an offer to build a fraud prevention solution at Revolut, I was happy to embrace the opportunity. What’s more, I was truly excited about solving this business problem using supervised machine learning methods.

In this article, I’ll tell you:

• about the fraud prevention system we’ve created,
• the technologies we used,
• the results we’ve achieved by now,
• and where we’re heading.

I’ll lead you through all the stages of the system creation, from definition to deployment. But first things first.

Meet Sherlock

Sherlock is our card fraud prevention system based on machine learning. It continuously and autonomously monitors Revolut users’ transactions. While a web store or a terminal at Starbucks is displaying the “Processing Payment” animation, in under 50 ms Sherlock evaluates your transaction.

If Sherlock finds it suspicious, it blocks the purchase and freezes your card. A follow-up push notification prompts you to confirm if it was a fraudulent payment, or not. If you respond that it was legit, the card is unblocked, and you simply repeat the purchase. If, however, you don’t recognise the transaction, your card gets terminated, and you can order a free replacement.

Fraudsters exploit advanced technologies, they learn fast, but so does Sherlock. Every night, Sherlock’s machine learning models are retrained to account for any missed fraudulent and incorrectly declined transactions.

Sherlock in numbers:

• over $3M saved during the year in production
• just 1c out of $100 is lost due to fraud
• 96% of fraudulent transactions are caught
• 30% of Sherlock’s fraud predictions turn out to be correct

What these numbers mean to our clients is the crucial difference between an unforgettable holiday and a holiday marred by being robbed and having to make do in a foreign country.

Defining project goals, metrics, and talent

Above all, we aimed to minimise Revolut fraud losses and the losses caused by incorrectly blocked transactions. Banks have dedicated departments for transaction analysis with hundreds of people calling their clients in order to prevent unauthorised transactions.

It really takes a huge effort to reduce the damage caused by scammers. We wanted to find a way to detect and prevent fraudulent card transactions efficiently with fewer resources. We needed to learn to predict which transactions a user may demand to chargeback. While diving deeper into the problem we realised the importance of catching the first instance in a sequence of fraudulent transactions.

Basically, we’re solving a binary classification problem here, i.e. identifying whether a given transaction is fraudulent or not. Thus, precision, recall, and the number of false-positive detections are our primary metrics. Adding metrics for detecting the first fraud in a row helped us tweak the machine learning performance during its training.

In the early days of our research, we needed an expert to perform data wrangling, data analysis, data visualisation, and machine learning. I took on the responsibilities of a data scientist and a backend engineer to build Sherlock, and later I used the help of my colleagues — mobile developers and platform engineers — to integrate it into the Revolut product and develop UI.

Discovering. Data sources

Having the right data is much more important than choosing a machine learning algorithm. At Revolut, we’ve had a table of transactions reported fraudulent since the early days. It was stored in a PostgreSQL database and was not suitable for doing the analysis.

To simplify the data processing and data visualisation, I set up a nightly delta dump from PostgreSQL into BigQuery — a warehouse database running on Google Cloud. It became easier to join multiple tables and perform an initial analysis of millions of transactions happening daily. Besides, the serverless nature of BigQuery relieved me from the need to maintain the infrastructure. My focus was on the ETL code only.

Design. Envisioning the solution

Having the right data in hand, we moved on to envisioning the end solution. It was important to decide on the architecture at such an early stage of the project to make sure that our solution would fit the pace and conditions of real-time processing.

For the task of detecting fraudulent card transactions, we decided to implement the lambda architecture. We designed a real-time transaction scoring system paired with a nightly batch data processing pipeline. The nightly job is aimed at fixing any inaccuracies that might have appeared throughout the day in user profiles.

We also wanted to increase the accuracy of data scoring and automate the fraud-prevention flow entirely. Therefore we decided to collect users’ feedback on declined suspicious transactions within the mobile app. Besides, this logic would empower our users to take control over their funds and make their own decision — they can either confirm a transaction and unblock their card, or terminate their card quickly and order a new one.

Development

Choosing the language

We’ve chosen Python both for development and production as it’s the most widely used language in the data science community.

Feature engineering

This is the most challenging and creative part of any machine learning project. At this stage, we investigated how we could use what we knew about customer behaviour and merchants to identify fraud patterns in real-time; what kinds of deviations should be considered risky.

Features are attributes that data scientists have to come up with based on the available data. Features are combined into a vector to serve as an input to a machine learning algorithm.

We categorised our features as follows:

1. Features with no or minimum pre-processing:
   - Merchant’s name, city, category, currency, etc.
   - USD amount of the transaction
   - Transaction time of day
   
   
   
2. User-focused features that compare the values of the given transaction against the historical transaction data for the given user:
   - How quickly does the user make the transaction?
   - How much does the USD transaction amount differ from the average spending of this user with the given merchant and the category code, using this payment method, and at this time of day?
   - Is it the first transaction of the user with this merchant? If yes, this merchant is monitored for the next few hours
   
   
   
3. Merchant-focused features that compare the current transaction with all the previous non-fraudulent and fraudulent transactions at this merchant or merchant category:
   - How many users transacted with this merchant before? How many of them reported fraud with this merchant?
   - How much and how often do users transact with this merchant?
   - How long ago have we seen this merchant for the first time at Revolut?
   
   
   

Getting to scale

During the early stages of my research, I used to experiment with feature engineering in the Jupyter Notebook. But on a scale of millions of users, such an approach wouldn’t work.

Besides, I had to construct our training data in a way which would prevent data leakage. In other words, the features for every transaction should only be based on the previous transactions of a user when sorted chronologically. That way, we precisely emulate the production flow.

I turned to the Apache Beam framework using Google Cloud Dataflow as the runner. Beam allowed me to focus only on the logic of feature engineering for an individual user, leaving out the considerations about the parallelisation of the code and hardware infrastructure.

Choosing a machine learning algorithm

I have experimented with a wide range of machine learning algorithms: linear regression using Vowpal Wabbit, gradient boosting using Catboost and XGBoost, scikit implementations of random forest, SVM with linear and RBF kernels, one-class SVM and neural networks using Tensorflow.

In the end, we chose Catboost, an open-source library from Yandex implementing gradient boosting on decision trees, as it outperformed other algorithms on several metrics and the inference speed. This algorithm has proved itself robust on heterogeneous data, data which comes from different sources, containing both numerical features of varying nature and categorical features. Besides, it didn’t require much hyperparameter tuning.

How I dealt with an imbalanced dataset

Training a machine learning model on a highly imbalanced dataset requires a particular approach. The fraud rate in the training data is around 0.03%. At first, I reduced the number of non-fraudulent transactions raising the fraud ratio to about 10%. Then, I assigned weights to individual fraudulent transactions based on the amount of the transaction and several other factors.

I constructed the validation data using all transactions that happened chronologically for five weeks in the future. It’s essential to keep the fraud ratio in the validation dataset the same as in production. That way, we obtain the performance results as close as possible to the ones we can expect in production during live scoring.

Deployment. Our current production flow

Finally, in production, we had to take into account that the card fraud prevention system is a mission-critical application. Therefore we had to implement specific measures to ensure resilience and reliability of the service, reduce the response latency down to 50 ms, and make sure that all the important events were monitored and covered by alerts.

Let’s take a look into our current production flow

It all starts with an offline nightly job that is orchestrated by the Google Cloud Composer, a managed service for Apache Airflow. The job dumps the delta of transactions for the day from our PostgreSQL database into BigQuery. Given that the schema of some tables might have changed, I’m modifying the BigQuery schemas on the fly and dumping the data there.

After that, an Apache Beam job running on Google Cloud Dataflow generates the training data that gets dumped back into BigQuery. At the same time, several Beam jobs create users and merchants profiles that are put into Couchbase — the in-memory NoSQL database.

The training data is then used by several machine learning jobs training Catboost models on Google Cloud AI platform. The trained models are stored in Google Cloud Storage.

What happens in real-time

When you make a card transaction, the Revolut’s processing backend sends it to the Sherlock Flask app deployed on Google Cloud App Engine. The trained machine learning models are pre-loaded into memory.

Upon receiving a transaction via an HTTP POST request, the Sherlock app fetches the corresponding user’s and merchant’s profiles from Couchbase. Then, it generates a feature vector — using the same features as the ones created in the Apache Beam job that produces the training data — and makes a prediction. The prediction is then sent in a JSON response to the processing backend where a corresponding action is taken — all within 50 ms.

In the end

If the probability of fraud is below a certain threshold, the processing backend lets the transaction go through, and the user sees a “Payment Approved” message on a terminal or a website.

If the probability of fraud is above the threshold, the processing backend declines the transaction and sends a push notification to the app.

Performance monitoring

We use Google Cloud Stackdriver and Kibana dashboards to monitor the system performance in real-time. Our areas of interest here are:

• Functional performance, such as monitoring of merchants, number of alerts and frauds, number of true- and false-positives.
• Operational performance, such as latency (how fast the system responds), number of transactions processed per second, and more.

Stackdriver sends us email and SMS alerts in rare cases of any issues.

What’s next?

We’re happy with our current results — the fraud rate is already low, but we’re going to continue improving the prediction quality. Essentially, we see Sherlock as the end product that can be purchased and integrated by other banks and financial institutions.

I often talk to people from banks, and some of them have already asked if we can sell our technology to them. Operations which banks process manually for hours, take just a few minutes at Revolut. If you’re going to sign up millions of users and scale your business to new markets worldwide, you cannot rely on manual work alone.

Nikolay Storonsky, CEO and CO-founder of Revolut.

Join Revolut

Today, Revolut has 9 million customers, and that number continues to grow. If you’re interested in helping us take Sherlock to the next level, check out our vacancies on the Careers page.

P.S. Have you ever received a push notification from Sherlock? How did it make you feel? Let me know in the comments!

When I was a kid, I loved playing Lego. It was magical seeing anything from castles to cars appear out of the same colourful building blocks.

Today, I love tinkering with software frameworks and Google Cloud products, building solutions to problems that I encounter in life. Just like with Lego – there is an infinite number of combinations. It only takes a little bit of imagination and continuous experimentation to put these building blocks to good use.

Last summer, I started recording short videos aiming to practice speaking on camera and build my personal brand. Yes, marketing is also one of those Lego blocks I discovered a couple of years ago.

I noticed that the best videos on social media have captions to catch the attention of people scrolling through their newsfeed with the sound off. It's also a matter of respect to people who can't hear.

I started looking for a solution that would allow me to record a video on the go, send it somewhere and get back a captioned version of it.

Either I wasn't patient googling it or those few web services, like Kapwing and Zubtitle, didn't have their SEO set up, I couldn't find anything. Funnily enough, I started seeing their ads right after I finished working on Captionly!

I decided to build such a service myself out of the available "Lego blocks".

In this article, I'm going to show you how I built Captionly – a web service that allows people to do just that – generate a captioned version of their videos.

What If It Doesn't Work?

But first, have you ever had such moments when you wanted to solve a problem but wasn't sure how? You decide to try an approach, but you aren't sure if it's going to work and if it actually makes sense. Such doubts nag you as you're building your solution, but you carry on nonetheless.

Then, sometime later, you hear someone talking about a similar approach or architecture at a conference. You experience a feeling of relief, knowing that you were right along the way! It boosts your confidence, and you become even more brave experimenting with your ideas.

Do you remember such moments in your life?

On 20-21st November 2019, Google held their annual event in London – Google Cloud Next. At one of the presentations, Bret McGowen showed how to build a serverless online shop – pretty much the same way I made my Captionly – with AppEngine and Cloud Functions. That's when I realised that what I developed made sense!

Building Captionly

Getting a text version of captions wasn't a problem. I knew about Rev.com – a service set up by guys from MIT many years ago. They built a network of professional captioners and throughout the years accumulated a high-quality dataset to train AI models outperforming Google and IBM! Last summer, they launched Rev.ai offering AI-generated captions with quality slightly lower than human captions but cheaper and much faster.

Besides, Rev have a convenient API allowing to automate the process of generating either human- or AI-made captions, transcripts and translations.

To build a service that returns captioned videos, we require three elements:

• a website to let users upload a video
• an integration with Rev to get a text file with captions
• a service that embeds the captions into the video

I decided to try Firebase – a Google service that comes with the Firestore database, Cloud Functions and several other services that help build serverless web and mobile apps.

Firebase also allowed me not to worry about implementing secure user authentication because it provides a way to take care of that very elegantly supporting multiple social media logins.

To build the frontend, I used the React + Material-UI + Firebase boilerplate app that comes with ready-made integrations with Firebase Authentication. I combined React frontend with the Flask backend running on Google Cloud AppEngine Standard Environment.

Firebase Storage, which runs on Google Cloud Storage, provides a JavaScript SDK that I used to let Captionly users upload their videos directly to the Cloud Storage through the web browser. Firebase Storage comes with a way to define security rules making sure that users can read and write only their files.

When a user uploads her video, I create an entry in Firestore capturing the details of the order, such as the Storage path to the uploaded file.

Firestore allows writing Cloud Functions that get triggered automatically whenever a change happens in the database. We write such functions using JavaScript or TypeScript.

Once the user's order status changes to "Video Uploaded" in the database, a Firestore Function gets triggered to create a new order with Rev through their API. The order status gets changed to "Captions Order Submitted".

It takes a while for Rev to process the video and generate captions. Depending on the user's choice at Captionly, it takes from about an hour for high-quality human-made captions to a couple of minutes for AI-made captions.

When Rev completes the order, they trigger an endpoint that I created in Cloud Functions. The function downloads the text file with captions to the corresponding order folder in Cloud Storage. The order status gets changed to "Captions Created", followed by "Rendering Started".

This status change triggers the Firebase function again that sends the order details to my video rendering service.

Video Rendering with FFmpeg

Video rendering is an interesting problem. There are several video editing solutions ranging from paid ones like Adobe Premiere and Apple Final Cut Pro X to free ones. However, I didn't need a user interface to embed captions into videos. I wanted a command-line version to automate the process entirely.

That's how I discovered FFmpeg – an open-source console-only application that allows you to do anything you can imagine with videos as long as you are patient figuring out how to encode what you want to do using the command-line options.

To give you an idea, here's how to ask FFmpeg to embed captions into your videos to get a result like this:

ffmpeg -y -f lavfi -i color=color=#BF0210:size=3840x40 -t 38 -pix_fmt yuv420p dark_red_2000_27.mov && ffmpeg -y -i creative_block.MOV -i dark_red_2000_27.mov -filter_complex "[0:v]pad=w=iw:h=3840:x=0:y=840:color=white[padded];[padded][1:v]overlay=x='-w+(t*w/38)':y=3000[padded_progress];[padded_progress]drawtext=fontfile=/fonts/roboto/Roboto-Bold.ttf: text='OVERCOMING CREATIVE BLOCK': fontcolor=#BF0210: fontsize=200: x=(w-text_w)/2: y=(840-text_h)/2[titled];[titled]subtitles=creative_block.srt: force_style='Fontname=Roboto Bold,PrimaryColour=&H1002BF&,Outline=0,Fontsize=16,MarginV=0020'" -codec:a copy creative_block_padded.mov

I created a service that takes a video file, a corresponding text file with captions and merges them delivering a captioned version of the video.

Video rendering is a memory- and CPU-intensive process, so I must use powerful-enough virtual machines to accomplish the task.

Besides, I wanted my video rendering service to be scalable and automatically spin up necessary computing resources depending on the workload – the number of orders submitted through Captionly.

I decided to leverage the power of Google Cloud Kubernetes and its capability to scale both horizontally and vertically.

I didn't have any experience with Kubernetes when I started this project, so it was a steep learning curve for me understanding the relationships between nodes, pods, containers, deployments, and services.

I created my Kubernetes cluster with a node pool specifying that I want it to be horizontally and vertically scalable. In the minimal configuration, when there is no workload, my cluster runs a little preemptible virtual machine. When video orders start flowing in, Kubernetes provisions additional pods of my rendering service. When the number of pods becomes too large, Kubernetes spins up additional nodes to allocate new pods there. If an order comes in with a lengthy video that requires more computing power and memory, Kubernetes spins up a more powerful VM according to the limits I had predefined.

Such a setup is incredibly cost-effective and powerful to scale pretty much infinitely.

To orchestrate the video rending jobs, I set up Celery using Google Cloud Memorystore – a managed Redis service – as a synchronisation backend.

After the order status in Firestore gets changed to "Rendering Started", the Cloud Function sends the order details to my endpoint in AppEngine. The AppEngine function creates an entry in Celery.

Celery triggers the job in Kubernetes that pulls the video and the captions file from Cloud Storage and launches FFmpeg to render the video. The completed video gets uploaded to Cloud Storage, and the rendering service calls a Cloud Function, which updates the order status to "Rendering Completed" and sends the user a notification email.

The user can watch how the status of the order changes in real-time in her account on the website without refreshing the page. Firestore can notify subscribers – our website in this case – about any changes that happen in the database.

Accepting Payments with Stripe

To accept payments for the Captionly orders, I built an integration with Stripe using their powerful and very flexible Python API and ReactJS elements for the payment form.

I wanted the payment form to look very natural on the website and also support subscriptions, as well as Apple Pay and Google Pay.

It required me to set up an additional endpoint to listen to events sent by Stripe when payments get processed.

Such setup allowed me to stay PCI compliant and satisfy SCA requirements by not storing or processing user payment details at all by myself but rely on Stripe.

It's Your Turn!

This experience of building a fully serverless infrastructure paired with a scalable Kubernetes service made me even more convinced that we have incredible power at our disposal to build anything we can imagine.

The trick is to be able to find problems – that's the hardest bit!

I encourage you to experiment yourself with Cloud Services because that's how you come to realise what is possible to build. It also helps you keep your technical skills at peak and lets you quickly prototype with your ideas.

In the end, is there anything more exciting than seeing your ideas come to life?

If you've read this far, I invite you to give Captionly a try using this 25% discount link valid for any subscriptions that we offer.

If you wonder what to talk about, here's a short video on how to get ideas for your videos! In addition, check out my Instagram @dimileeh to watch videos I created using Captionly. Good luck!