Retention Metrics Explained: Lead Scoring, Pt. 1 [RS Labs]

As the saying goes, you don’t get a second chance to make a first impression. This is true for customers too, and it can have a big impact on a business. At Retention Science, we address this problem by predicting, after a customer first signs up, whether he or she will actually turn into a paying customer. We define this as a customer’s Lead Scoring. Lead Scoring predicts whether or not someone will become a purchasing customer, based solely on sign-up data. Marketers can then use this prediction to create campaigns that will resonate with customers more likely to purchase. In many ways, Lead Scoring is the start of the retention cycle, since without the ability to get paying customers, there is no one to retain in the first place.

In fact, across a number of different businesses, we find that 62% of customers immediately churn. That is, more than half of the customers who sign-up or register never end up making a purchase. The figure below shows the immediate churn rate for companies of various sizes. The x-axis shows the company sizes, ranging from tens of thousands of customers to more than 16 million customers. The immediate churn rate is shown above each bar, as a percentage of the total customers. For instance, the company with 3.4 million customers has an immediate churn rate of almost 84%, representing some 2.5 million of their customers. The immediate churn rate is shown above each bar, as a percentage of the total customers.


 Immediate churn rate for various companies 


On average, more than 60% of eCommerce customers that register with a website fail to make a purchase. That is a significant waste of customer acquisition dollars, and means companies are already fallen behind in terms of retention, since they are losing so many customers from the start. Our Lead Scoring models are meant to address this challenge.

Use Cases for Lead Scoring

The value of Lead Scoring is in its ability to segment users into high and low probability of purchase at time of sign-up, and allow marketers to route different messages to these different groups for different purposes. For instance, it supports marketing teams iterating on the first sets of emails sent to customers (known as the “Welcome Series”). The marketing team may A/B test different templates, change their order, or even use Retention Science to power recommendations to users or to optimize potential incentives, such as coupons, and use WPP as a metric to gauge progress.

Another important aspect of Lead Scoring is that it can produce profiles of likely and unlikely purchasers. For instance, as we describe below, there are certain factors that indicate with strong support that the user will probably purchase (or not). By investigating these features, marketers can build powerful acquisition schemes, tailored to those most likely to buy. For instance, if WPP identifies college-aged females in the western United States as the most likely purchasers, then you can specifically target those users with advertising and content marketing.

Modeling Approach

So, how do we actually do this prediction? Here, our approach is entirely data driven. We create features based on individual customers, and train a classification model (we found that ensemble methods worked well) to use these features and predict whether someone in the past purchased or not. These features span a large range of user description information, such as whether a user registered with the business via Facebook or where the customer is located. In some cases, we have even more detailed specific customer information, which could potentially be quite discriminative. For a specific eCommerce company, we proved that a particular color preference is a strong predictor of whether someone would convert or not (we’ll go over this example in our next post).

Our models yield scores which are aligned with the posterior probability distribution, which allows us to predict, for a future user, if he or she will purchase based on the combination of features that represents that customer. While this method is useful, we found the most utility by simply bucketing users into two cases: likely to purchase and not likely to purchase. To create these buckets, we simply find an empirical point in the distribution of the purchase probabilities, and split customers into purchase or not purchase buckets using this criterion.

By making this approach completely data-driven (e.g., predicting based on the past data), we can update the model every day, as new customers register with our clients and become purchasers. In this way, we can reflect how the Lead Scoring is changing based on specific marketing campaigns.

To evaluate the performance of our Lead Scoring models, we track what each user actually ends up doing in the following months after our models make their predictions. We find that we do quite well in differentiating between good and bad customers, and this identification can make a big difference for clients. For example, for one client, when we compared the 10% most promising and the 10% least promising users (by the model’s scoring), we found that the top 10% spent almost 300% more and converted 40% more often than the bottom 10%. In our next post, we will dive deeper into this and other client examples and discuss some of the features used by the model.

For part 2, go here.


About the Author

Eric Doi is a data scientist at Retention Science.  His goal is to improve every day, just like gradient boosted learners.  He studied Computer Science at UC San Diego and Harvey Mudd College.