Retention Metrics Explained: What Is Customer Churn? [RS Labs]

Customer churn can be difficult to define. Here at Retention Science, our data science team spends a lot of time thinking deeply about customer retention for commercial businesses. Although what we think retention actually means is a topic for a future post — hint: it’s a bit more complicated than most people initially think! — at its core, our retention research focuses on two fundamental questions:

  1. Can we objectively measure whether customers will stick around (and purchase)
  2. Can we predict these measurements, so that retailers can take action to keep their customers happy, engaged and coming back?

These are brow-furrowing questions we may never solve with 100% accuracy, but with the help of predictive metrics, we can get pretty close.


Figure 2.1: The Retention Cycle

In Figure 2.1 above, we see what we call the retention cycle. On the left of Fig 2.1, customers are acquired (e.g., they register on your eCommerce site), following your marketing funnel. Some of these users then make purchases, moving them into the converted bubble, and some of them never become customers at all. We say that people who stop being (or never became) paying customers have “churned.”

Churn can be a tricky thing to define, because it happens at so many stages of the retention cycle. Some smaller group of paying customers become repeat purchasers, until they don’t, at which point they’ve churned as well. Another percentage of customers only ever make one purchase, in which case they move from the converted to the churned bubble directly. Basically, customers can churn from any bubble in the cycle.

Each transition in this bubble needs to be measured and managed in some way, as each transition to churn represents potentially lost revenue to your business. This blog series is motivated by our work in measuring and predicting aspects of these transitions that are important to your business.

Given the depth of the problems and our thoughts on them, we have broken our writing into a series of posts that will span over the next few months, each on a different metric. You can think of this series as a geek-to-business translator, providing definitions for some of our core measurements and predictions. By influencing these metrics, we believe that companies can improve their business and keep their customers happy. The definitions will include both high-level descriptions as well as a deeper technical discussion.

Without further ado, the first metric we explain is customer churn:

Churn (aka Churn Probability, Churn Score, and More)

Churn is a key retention metric. Essentially, churn represents the probability that someone will stop being a customer. (Here, we make the distinction that a customer is someone who makes a purchase.) This is a key metric in understanding how to retain customers, since without it you wouldn’t know which customers you should be focusing on. For instance, you might target your users that are most likely to churn with a discounted offer to help keep them happy and to prevent them from leaving. Or you might send a note of appreciation to your most loyal/VIP customers (those least likely to churn), which may cause them to evangelize your company even more.

However, people quit being customers for any number of reasons, which makes predicting this value hard. But there is hope! Here at Retention Science, we use machine learning to predict churn.

At its core, machine learning is all about computer programs that adapt themselves to the problem at hand (for instance, machine learning could be used to identify potential VIP customers based on attributes such as website and purchase behavior). Machine learning helps us identify obvious and unobvious attributes of customer behavior (called latent features) that pick up on the less easily measurable influences that cause people to buy or not (for instance, combinations of location, gender, and recent order categories).

In our case, our algorithms account for a large number of features, such as customer information, behavior, order history, and website activity. They then place customers along a continuum from 0 to 1, where 0 represents a customer definitely staying as a customer and 1 represents a customer that will leave the business. Any number in between can be interpreted as how likely it is that the person will quit being a customer.

Modeling (and Defining) Churn

At a deeper level, we model churn using an ensemble of a number of methods. We combine classic RFM (Recency, Frequency, Monetary value) models, linear and non-linear machine learning classifiers to predict churners versus non-churners, and knowledge-based models that take clues from business-specific information. For instance, if your business sells diapers, then the size of the diapers a customer orders is a great proxy for the child’s age, and predicts pretty well when that customer will churn. But it says very little to you if you sell tires!

Our customer churn prediction process is shown in Figure 2.2 below.

churn 2

Figure 2.2: Modeling Whether A Customer Will Churn

For machine learning aficionados, the classifier method is probably the most interesting. In particular, this problem seems to lend itself naturally to a supervised classification approach, where we tell the algorithm explicitly which users have churned and which have not; future predictions are based on what is learned from that data. However, creating these “churn labels” is challenging because we can never really know if someone has churned or not. Someone may cease being a customer for years, only to come back at some arbitrary later time.

In our case, we found windowing our users to be a useful proxy: for instance, if a user in a monthly subscription business keeps postponing her order for six months, that user has likely churned. Of course, then, we spend significant time investigating different temporal windows and their effects on different industries and businesses (for instance, you probably buy diapers more frequently than cars). We often partner with our clients to customize our models to their business needs, and this is one area where customization can have a strong and beneficial impact.

This brings us to an interesting point: Even the notion of churn itself differs across businesses and situations. For example, in a pure subscription model, without postponement, customer churn is simply users who have unsubscribed. This is straightforward, because within this business model, the customer’s only options are to subscribe and pay, or not. This is common for things like cable or internet service.

In ad-hoc purchase models, however, such as most eCommerce sites, customer churn is defined as customers who stop being paying customers. Compared to the yes/no definition of churn for pure subscription companies, this definition is trickier to pin down, and can vary from business to business.

It may involve defining churners statically, like people who haven’t purchased for a time period that is a few standard deviations away from the average purchase time, or someone whose time on the site without purchase far exceeds the average customer lifetime. Or, it may mean making more knowledge-based approaches rooted in commonsense: Customers who only buy apples are likely churners once you stop selling apples. In an even more specific example, a media company may define churn as when people stop watching their videos online.

While we’ll write more deeply on the topic in a later publication, we have some research that demonstrates our churn models for ad-hoc purchases generalize extremely well to predicting when customers will unsubscribe from subscription-only businesses, too. This is a strong result because it demonstrates that our models are truly picking up on signals of purchasing behavior, whether the purchases are made at a timely, prescribed fashion (through subscription companies) or on the fly and on demand (like at traditional eCommerce sites).

Measuring Predicted Churn Accuracy

Given the challenge in defining and measuring customer churn, the next natural question should be: How do we evaluate our measurement of churn? As you might expect, measuring churn prediction accuracy is complicated.

So, we decided to show you some of our own metrics that provide insight into our models’ performance in predicting churn. As far as we know, we are the first data science group to publish our churn validation approach. This is not to say we are the only ones, but as a marketer, this is an important and tough question you should ask whomever is optimizing your campaigns.

Below is a validation report for one our of churn models that combines both a classic RFM churn score and a random forest classifier (a machine learning method that learns different sets of rules that determine churners and non-churners). It shows the results of training a model on data up until 1/1/2015 and then testing that model on data through 7/1/2015. The results demonstrate how well the model performed on that day.

Screen Shot 2015-08-14 at 2.34.40 AM

One of the clearest metrics is our accuracy in predicting customer churn. As you can see in the report above, we are able to identify customers that churn (e.g. non-buyers in the report) with almost 74% accuracy, and we can predict customers that will not churn (e.g. buyers) with 90% accuracy.

Screen Shot 2015-08-14 at 2.35.10 AM

We also include other robust metrics in the report. One measure is the Area Under the Curve of a Receiver Operating Characteristic chart (we summarize this value to AUC). This is an intimidating name, but a useful metric that can be interpreted as how well our algorithm discriminates between churners and non-churners. In this case, the AUC is .88, which is pretty good (1.00 would be perfect).

Finally, we report the Root Mean Square Error of the probabilities we generate for each customer belonging to each class. (For you math whizzes, see Figure 2.3 below). Essentially, we want to know, on average, how well (or not) we did at classifying each churner using the probability we assign that the customer will churn.

churn 3

Figure 2.3: Root Mean Square Error of Churn Class Predicted Probabilities

As a note, our 6-month testing window is simply a parameter for generating these reports; we can expand or contract it as necessary, to better understand our algorithm’s behavior. Also, we can tune the period depending on our client’s business requirements, as some businesses have naturally faster customer churn rates than others.

The Bottom Line: Why Churn Matters

So, what’s the bottom line for customer churn? Well, not only does it predict when your customers will stop purchasing, so you can target them to keep them happy, it also gives you deep insight into the types of customers that represent your greatest champions and the types of customers that are your biggest distractions.

To that end, it is interesting to analyze which aspects of customers tend to be most influential in causing customer churn. Customer churn happens for qualitative reasons that are difficult to quantify even for companies with rich troves of customer data. Correlating these deep qualitative reasons to some quantitative metrics can yield some good modeling results, but we caution that, as always, correlation does not necessarily imply causation.

 churn 4

Figure 2.4: Different Factors that Affect Churn for eCommerce

We hope you found our treatment of churn enlightening and useful. It’s a powerful tool for an increasingly competitive eCommerce landscape. Also, look out for future posts where we build upon churn to define how much those customers are actually worth to your business.

If you are really, really into the types of models discussed above, we would love to hear from you on the data science side. Even if you don’t, but you have questions, we would love to hear from you in the comments below!

Otherwise, catch y’all the next time we come up from the data mines! Next month, we’ll dive deep into defining Customer Future Value (CFV).

About the Author

Vedant Dhandhania is a Machine Learning Engineer at Retention Science. He helps predict customer behavior using advanced machine learning algorithms. His passion lies in the intersection of Signal Processing and Deep Learning.

To learn more about why customers leave, and how to prevent it by using a scientific approach to eliminate customer churn, download  The Scientific Guide to Eliminating Customer Churn