The Impact of Churn Labelling Rules on Churn Prediction in Telecommunications

1.1The Related Work

There is a wide variety of methods and their combinations were applied to a telecommunication churn prediction. Some of them will be discussed below.

The extensive comparison of different classifiers was done in the research by Adhikary and Gupta (2020) where authors analysed and compared the performance of over 100 classifiers in churn prediction of a telecom company. However, it is important to mention that in the aforementioned research the tuning of different methods was not analysed nor the set hyperparameters were provided – the details were defined by the selected software automatically. Thus, the result of highest accuracy which was given by the Regularized Random Forest classifier is unreliable.

Bose and Chen (2009) have utilized the clustering techniques to improve the decision tree-based churn prediction – clustering was combined with decision trees in such a way that the unsupervised learning technique aided the performance of a supervised learning technique for the classification task of churn prediction.

The other study by Vafeiadis et al. (2015) compares different standard machine learning techniques, the tuning of the hyperparameters is performed, the accuracy level 97% was achieved on the publicly available test data. Data are artificially based on claims similar to the real world (Vafeiadis et al., 2015). Note that the results are highly dataset-dependent, we suppose that in this case the high performance is due to dataset specificity.

Some studies show the possibility to combine multiple methods, for example, De Caigny et al. (2018) develop a new hybrid classification algorithm that uses a combination of decision trees and logistic regression and that is developed to reduce the weaknesses of DT and LR while maintaining their strengths.

Some research is dedicated to alternative classification methods, such as fuzzy classification methods (Azeem et al., 2017). According to the authors, the achieved level of AUC score of 0.68 is high comparing to other authors, however, achieved values above 0.9 can be found in literature (Ahmad et al., 2019). It is important to note that the results might strongly depend on the dataset specificity, the direct comparison between the results of different researchers is very limited, since different researchers use different datasets. It is important to note that (Azeem et al., 2017) ignore the hyperparameter tuning skipping the opportunity to get better results for every method, leading to incomplete comparison between different methods.

The quality of the results might be strongly conditioned by the data preparation algorithms, according to the research by Coussement et al. (2017), the data-preparation technique can strongly affect churn prediction performance. Authors show the improvements of up to 14.5% in the area under the receiving operating characteristics curve (AUC) and 34% in the top decile lift.

In literature, some research that is based on very specific data which is usually not available can also be found. For example, in research by Ahmad et al. (2019) the outstanding 70 terabyte dataset from Syrian company SyriaTel was analysed using Hadoop Distributed File System and Spark engine. According to the authors, the extensive expansion of traditional attributes by some data that use a lot of memory has let them improve the results that otherwise were very poor. However, it should be noted that a lot of data had details about customers. The European Union (EU) regulations on data privacy might not let perform similar analysis in EU countries, since a lot of private data in mentioned research was used, such as exact information of migration between companies for each person. Another note is that according to the information specificity, the matrix with the data was very sparse, thus, the storage of these data could be greatly improved by using compressed matrices. Adwan et al. (2014) analysed the customer movement from one provider to another. However, nowadays, under EU data privacy regulations the implementation of such research would be hard or nearly impossible. Thus, methods in the aforementioned research are not applicable in practice.

In context of the final goal for business, the reason for churn prediction is the possible customer retention; there are researches in which the churn prediction is accompanied by its further application. Ullah et al. (2019) presented the investigation of the existing techniques in machine learning and data mining and proposed a model for customer churn predictions, to identify churning factors and to provide retention strategies. In study by Ahn et al. (2020), there are main concepts for churn loss evaluation presented: Customer Acquisition Cost (CAC), Customer Lifetime Value (CLV); also the feature engineering was discussed. There are successful attempts to include the survival analysis methods into the churn prediction, interpreting the churn retention as a subject survival (Routh et al., 2021), the risks raised due to competing events are used in modelling by a random survival forest.

Various machine learning methods are used to determine the churn, as can be seen from the overview in the Table 1. The results obtained in the articles show that the results of the several different methods are good enough. Thus, the efficiency of churn prediction for the actual customer retention might not be bound by the actual (technical) classification performance – the main source of inaccuracy might arise from some assumptions, such as sufficient conditions of churn definition or the assumptions used for the artificial data generation. This is the problem which we are going to address.

In general, there are few most commonly used methods that give a reasonably good results in many researches, moreover, often there were no significant differences between these and some other methods that sometimes perform better in other researches. In our case, several commonly used methods were selected and studied, most of them are tree-based methods.

For example, Adhikary and Gupta (2020) have analysed as many as 100 methods for determining the churn. However, as we have already mentioned, the hyperparameters tuning was ignored. It can be assumed that the hyperparameter optimization was not analysed in the aforementioned research. Thus, it can be considered as a drawback of the considered study.

Another important note can be done based on research by Alboukaey et al. (2020) – the RFM (Recency, Frequency, Monetary) features enabled even such method as Random Forests to perform surprisingly well, however, this method was not among the best ones in other researches. The results without RFM features were much worse in aforementioned research. Thus, in this article we compute RFM features as an important part of dataset for all experiments as well, a detailed description is provided in Section 3.1.

1.2Some of Requirements for a Proper Churn Prediction

A big amount of flaws can be noted in other researches, which can be seen as research gaps – here we will try to categorize them by providing the appropriate research requirements. From the overview of the results of other researchers, we highlight these requirements which should be fulfilled:

From Table 2 the lack of the proper attention to the aspects mentioned above can be seen. However, it must be noted that the good results can still be achieved without fulfilling all of these requirements. For example, some of the methods are less prone to the data imbalance problem and might perform relatively well even without balancing, especially if a proper measurement was used (for example, if it is sensitive to the Recall metric such as F-measure). However, according to other authors, these requirements are recommended to be fulfilled and in current research one of the goals is to fulfill these requirements.

1.3Churn and Partial Churn Definitions

Often a churner in the mobile telecommunications is defined as a customer that stops doing revenue generating events during the next 90 days (Alboukaey et al., 2020), while he was active during the observation period. Churn directly affects the profit of the company, thus a customer retention is an important part of company strategy. However, to be able to apply any customer retention, it is necessary to detect the churn event on its early stage, i.e. during the first days of this 90 day period or even before this period has started. In order to estimate the consequences of the churn, it is important to estimate the loss of the profit due to some specific churn event, i.e. churners might have different weights depending on the amount of profit being generated by these customers. However, it is important to note that in most cases the one month period is sufficient to identify the churn case, i.e. the inactivity during 30 days is followed by another 60 day inactivity in most cases. Thus, the definitions of a partial churner as a customer who doesn’t use the services for 30 days might be used in practice. On the one hand, the shorter period lets us label more recent cases, thus it lets a model achieve a faster reaction to the changes in the behavioural patterns. On the other hand, the quality of the possible prediction using shorter time interval might decrease due to possible errors in labels of the churners, i.e. a small portion of partial churners does not fulfill the true churner criteria which means a good machine learning approach will produce a portion of false positives and potentially these false positives could even increase false negatives as well due to the shifts in the model focus while learning from some false data.

The above said implies that it is necessary to find a compromise between relevancy and certainty. This topic has not been investigated before, some authors only mention that in their specific case the churner definition and the partial churner definitions are equivalent in over 80% cases, however, they achieve over 80% accuracy in their predictions using machine learning models, thus it is not clear what is the true accuracy and how does the differences between churner and partial churner affect the final results. On technical level, the aforementioned differences of definitions lead to different labelling rules, thus different training and test data for the classification problem. This is one of the main questions this article is dedicated to.

The main reason behind the churner detection and prediction is the customer retention in order to save the portion of the profit, since in most cases the costs of customer retention are usually much lower than the costs of new user attraction (Barrett, 2003; Lu, 2002). However, the success of retention highly depends on how fast the churn detection was performed. Some authors focus on early churn detection calling it early prediction (Zhang et al., 2016). Therefore, only analysis of retention effectiveness can lead to clues about analysis of suitability and usefulness of different churner definitions, but this is out of scope of the current research. We will assume that the standard definition is the most commonly used definition of 3 months customer activity absence leading to a benchmark rule for labelling the churner. Any other alternative definitions along with their appropriate labelling rules will be considered as approximations of the standard definition, the precision of such approximations is the main topic of the current research.

Fig. 1

The values of quantiles of maximum pauses during 275 days time interval according to Moremins data.

In Fig. 1, the values of quantiles for maximum pauses per customer are presented. This lets us evaluate the difference of partial churn and churn, more specifically, every customer with maximum pause between 1 and 3 months would mean at least one false positive label if we substitute the churn by partial churn in some specific time window. If we define a churner to be a customer leaving the services forever, then it can be seen that 30 day length interval would let us correctly label up to 59.4% of churners, 90 day interval – would be correct for up to 92.2% of cases. It must be noted that these data are derived from the whole interval, however, at some selected moment there will be only a part of these falsely potentially labelled churners. In other words, the provided estimates show the pessimistic lower bound of correctly labelled churners’ percentage. Thus, the three month period can be seen as quite robust and reliable comparing to labelling in 30 day period. It can be concluded that for the considered data churn and partial churn definitions show how different they can be, despite some authors (Alboukaey et al., 2020) claiming otherwise based on the distribution of churners’ activity during the last thirty days – such analysis can be misleading because the churners form a minority and there might be other customers’ minority group that has a similar activity patterns (e.g. with periodic peaks of activity). The provided rough lower bounds do not prove the suitability or unsuitability of the partial churn definition either. In order to compare the performance of different churner definitions, both definitions with according labelling rules can be used for training the classification algorithm, however, for the final measurement the full churner definition can be used to compare the results of different classification algorithms. Also, it is an open question whether we should consider a churner which stopped using the services for 3 months and have returned afterwards, since it can be considered as churn from competitor services.

Summarizing the said above, there is no perfect churner definition itself and appropriate criteria to define a churner might differ depending on many factors. However, the analysis of suitability and usefulness of different churner definitions is out of scope of the current research, thus, as a standard we will use the most commonly used definition of 3 month customer activity absence and use it as a reference labelling rule. For this research we generalize the churner definition by introducing the labelling rule parameter Tc which is equal to the minimal length of the absence time interval which is needed for to proclaim the customer to be a churner.

1.4Paper Roadmap

The remainder of this paper is organized as follows: the datasets used in this research are described in Section 2, Section 3 provides the method-related information for this study, results are provided in Section 4 followed by discussion in Section 5.

3.1Feature Extraction

A proper feature extraction might be the key for the performance of machine learning methods. Here we discuss the features used in the current research.

Raw data. Here we present the most detailed form of data feeding the machine learning methods in this research. For that role we have chosen to aggregate the most important parameters for every day per each customer, according to other researches (De Caigny et al., 2018; Alboukaey et al., 2020), the most significant and usually sufficient information is the one which is directly related to the usage of services (minutes, amount of SMS, etc.) or charges for the services (customer payments), thus, a daily data is used and is described as:

RFM features. As it was mentioned before, the absence threshold for partial churn is 30 days partial, for such labelling in some other researches Alboukaey et al. (2020) RFM-based classification has given one of the best classification results achieving over 90% accuracy.

Compared to usual operators, MVNO clients tend to perform non-regular usage of the services, since clients use services from traditional operators as well. Which means the explicit aggregation of parameters in longer time intervals could give the regularization effect making adding the robustness to the solution.

Recency, Frequency and Monetary (RFM) features are well-known for their suitability for churn modelling (Gupta et al., 2006; Khajvand et al., 2011). These features are the values obtained by aggregation of the considered parameter in the selected time interval in three different ways. Let us assume that we have daily data, i.e. the sums of some parameter for different days, we denote t1, t2 to be integers representing the first and last day of investigated interval, then R, F, M features in interval t=[t1,t2] are obtained using these formulas:

As it was proposed by Alboukaey et al. (2020), we calculate RFM features in intervals t∈[1,90],t∈[1,30],t∈[31,60],t∈[61,90]. Which means there are 4 features calculated per each parameter, we use these parameters:

In order to label the data for some time moment t, some amount of future days must be known. This means that if we use partial churn definition to label the data we can do that only for the moment t that is at least 31 days old (compared to the actual real moment). For the RFM feature usage we need the data aggregation in 5 time windows: 4 for RFM features, and one 30 day window t∈[91,180] for churn labelling. The RFM features application in the full test procedure context looks like that:

Other features. Here we present the list of some simple, yet efficient features suitable for churn prevention, mostly taken from publicly available datasets (Xu et al., 2021a) or based on them:

3.2Classification Algorithm

Here we provide the sequence of procedures defining the classification algorithm. It must be noted that some standard steps were omitted from discussion since the motivation of their usage is well known and these are done in most of other researches. More specifically we include into the algorithm: data normalization, random oversmapling balancing method, dimensionality reduction method PCA (Principal component analysis). Data normalization is important for variance-based PCA, PCA can increase the performance of some methods such as SVM. Thus, the steps to perform the churn prediction are:

Algorithm 1

The algorithm for metrics extraction

In Algorithm 1 some of functions reflect the functionality of Pandas dataframes and scikit-learn library, more specifically:

3.3Classification Methods

Taking into account the results obtained by other authors, we notice that the methods based on the decision tree give a good enough result.

3.4Performance Metrics

We find a lack of description of different metrics’ importance in other researches, thus, here we will fill this gap by discussing different metrics in the context of the churn problem.

Accuracy. Accuracy is the most common metric to measure the performance of classification method. What it lacks is the sensitivity for imbalanced data, i.e. if, for example, there are 10% of churners in the data, and there will be no single churner classified, the accuracy will still show value 0.90 which is usually a good result in the case with balanced data. For churn prediction the focus must be made on churners, the minority of objects, so the accuracy doesn’t show the performance very well.

Recall. Recall or true positive rate (TPR) is calculated as TPR=TP/(TP+FN). If true positives are very important in the context of the research, attention must be paid to recall since it measures that true positive rate among all positives. This metric doesn’t take into account false positive though, so it is unsuitable as a standalone metric, i.e. classifying all objects as churners will give you a perfect recall. In the context of churn prediction this is an important metric, i.e. we want to be sure that if a customer is going to leave, then a retention technique should be applied, even if it will lead to a side effect of applying the retention to some of non-churners falsely classified as churners.

Specificity. Specificity or true negative rate (TNR) is calculated as TNR=TN/(TN+FP). it doesn’t hold very important information if we want to find churners and apply retention techniques to them, however, if we want to limit the amount of retention applied to non-churners, this characteristic measures that indirectly, since you can derive false positive rate from it FPR=1−TNR.

Precision. Precision or positive predictive value (PPV) is calculated as PPV=TP/(TP+FP), it shows how well the positive result is determined. Usually this characteristic is important in cases when it is important to avoid false positives. Thus, in the case of churn prediction this metric is not important, however, the indirect usage of it might be still useful, for example, as a criteria to limit the recall – even if we want recall to be as big as possible, we should control the precision and keep it from being unreasonably small.

Balanced accuracy. Balanced accuracy is a compromise between recall and specificity, more specifically, it is an average of the aforementioned two metrics BA=(TPR+TNR)/2. It means, the rates for classification of minority and majority have the same weights, making balanced accuracy somewhat more suitable for imbalanced data comparing to accuracy metric. This is based on obvious observation that model can be tuned to have either big recall or specificity, so this characteristic seeks to find a compromise. However, this characteristic in some cases do not represent the situation well, a well-known fact is that arithmetic mean has drawbacks when it is applied to the rate-type characteristics, moreover, this metric is sensitive to majority which usually is not important, thus nowadays this metric is rarely used as primary.

F-measure. F-measure has the same motivation behind it as the balanced accuracy has, but the counterpart of recall is selected to be the precision which is far more suitable to describe the success of minority prediction. Moreover, attention should be paid to the fact that we are talking about rates, so the significance of the changes of absolute values depend on the values themselves, e.g. decrements of 0.1 have a more dramatic effect for value 0.11 than for value 0.9. In such situations a more appropriate way to find a compromise is to use harmonic mean instead of arithmetic average. Thus, F-measure is the harmonic mean of precision and recall F1=2·PPV·TPR/(PPV+TPR). This is a very popular metric to measure the churn prediction classification in recent researches, therefore, this metric will be the main measurement instrument in the current research.

AUC. The aforementioned metrics are calculated using a finally tuned and fixed classification method, however, often methods give you the estimations of likelihood of some object being belonged to one or other class, the final classification is performed by selecting discrimination threshold which must be stepped over in order to assign the object to some class. This means, that, for example, you can sacrifice the accuracy for a bigger recall if you shift the discrimination threshold for the final decision. In other words, after the learning process is finished, you can create a series of classification algorithms by choosing different thresholds and draw the ROC curve describing the dependence of TPR on FPR, both values will increase from 0 to 1 forming a curve. The area under ROC curve (AUC) is a good way to estimate the overall machine learning technique independently of discrimination threshold. The closer AUC value to 1 – the better, AUC equal 0.5 means the method’s performance is similar to random classifier.

True metrics. In the context of the current research there is a need to distinguish between the classification metrics and the actual metrics in terms of the true (full) churn definition. The reason is the partial churn definition and generalized definitions which let the churner be labelled based on the inactivity time interval which is less than 3 months. However, it is done with the intention to predict the churner based on the full churner definition assuming that the differences between aforementioned definitions are small.

For correct formulation of classification problem we must remove the entries for which the answer is already known. It is assumed that the definition itself is changed to an alternative one with other labelling rule, so if the object fulfills this definition the prediction is not needed, the answer is known. Such removal of the entries is an exact representation of what is done in other researches, Alboukaey et al. (2020) excluded inactive customers for the last 30 days. I.e. in the training data there should be no customers who are inactive for the last k⩽m days, where m is the number of days of inactivity that must be passed for the customer to be labelled as churner according to one or another generalized churner definition. However, these dropped entries should be still present in data labelled using full churn definition’s labelling rules.

Therefore, to unify the metrics for different churn definitions we introduce the true metrics concept – before the final measurement, the entries, which where excluded due to decreased time interval in labelling rules, must be returned to the dataset, such objects must be labelled as churners.

4.1Data Preparation

As it was mentioned in Sections 1.3 and 3.4, the churn definition, which is used for labelling, affects the dataset. A single parameter can be used to describe different labelling rules for different churn definitions used in the current research – the amount of days of customer absence Tc which was presented already in Section 1.3. In Table 5 the list of label names according to different churn definitions is provided along with the value of the aforementioned parameter Tc.

Note that in order to label the data, additional 90 days from the future were used, expanding the actual time interval used to create the dataset from 275 to 365 days, last 90 of which were used only for labelling.

4.2Hyperparameter Values for Different Methods

The explored hyperparameter ranges in current research are presented in Table 6.

Hyperparameter values for Moremins dataset are presented in the Table 15 and for other datasets in Table 8. The appropriate parameters were selected by trying all possible combinations using the Grid search method, as a comparison criterion F-measure was used.

As it was already presented in Section 2, the imbalance in the considered data is not very high. Thus, metrics for imbalanced data, such as F-measure, should not be blindly considered as the only right way to seek for an optimal solution. It is possible to optimize the F-measure metric at the cost of Accuracy via thresholding techniques or even the usage of special loss functions, however, the optimality of results obtained in that way can be still questionable as there is no single perfect way to measure the performance of the methods. Thus, the approach used in this research is to let methods converge using loss functions which are known to be most suitable for according methods, i.e. default ones, however, to use F-measure to perform the final selection of combination of hyperparameters.

4.3Method Comparison for Different Datasets

Comparison of different methods for different datasets is presented in Tables 9–13.

As it can be seen from Table 7, the achieved accuracy score for dataset BigML is close to the values which were presented for this particular dataset by other authors, which in most cases are between 0.91 and 0.98 (Xu et al., 2021b; Śniegula et al., 2019), as for IBM dataset, its accuracy values can be found in literature to be between 0.69 and 0.80 (Singh et al., 2021; Pamina et al., 2019). However, in some sources there is a lack of information about the F-measure scores which were used in the current research as the criterion to choose parameters, thus, in this research, a bit of accuracy might be sacrificed to achieve a better F-measure. Thus, from now we assume that the achieved results along with the quality of the selected methods are similar to the ones existing in other researches. In Table 8 the used parameters are presented.

The results of different metrics with different churn labelling rules using different methods are provided in Tables 9–14. In these tables we provide the metrics values for different methods with different labelling rules, for each rule there are two different metrics sets presented: the standard one which is used in all other researches and the True metrics (which were introduced in Section 3.4). The general note on the provided tables is that different methods show similar performance in many cases. Some special cases of exceptional values might be related to the specificity of the data in the provided dataset, thus, next, the attempt to provide general insights of tendencies in the results will be performed.

Fig. 2

The dependency of results on the amount of days of customer inactivity Tc used in labelling rules using GBC.

As the Tc of according labels decreases (row numbers in tables increase), the additional entries with positive churner answers according to the considered approach do the following:

In Figs. 2–7 there are presented results from Tables 9–14 for Recall, Accuracy and F-measure metrics. The interesting observation is that the high recall with lower Tc results in a better F-measure, especially it can be seen with label Churner3. As it was mentioned before, we do not optimize the results in terms of F-measure in all possible ways, thus, it is possible that similar result can be achieved without decreasing Tc, but with a proper method optimization via thresholding techniques and special loss functions towards a better F-measure. Thus, the decrease of Tc can lead to indirect methods tuning towards a better F-measure at the cost of Accuracy. However, decreasing Tc below 40 results in worse performance for all considered methods: for GBC 0.088, for XGBoost 0.037, for LGBM 0.025, for RF 0.027, for KNN 0.035, for SVM 0.064. It can be concluded that the 30 day inactivity period Tc is not sufficient to keep the same performance of classification methods as it is achieved using full churn definition labelling rule. However, for considered case the Tc=40 performed surprisingly well. Please note that such conclusion is valid for Moremins data case only.

Fig. 3

The dependency of results on the amount of days of customer inactivity Tc used in churn labelling rules using XGBoost.

Fig. 4

The dependency of results on the amount of days of customer inactivity Tc used in churn labelling rules using LGBM.

Fig. 5

The dependency of results on the amount of days of customer inactivity Tc used in churn labelling rules using RF.

Fig. 6

The dependency of results on the amount of days of customer inactivity Tc used in churn labelling rules using KNN.

Fig. 7

The dependency of results on the amount of days of customer inactivity Tc used in churn labelling rules using SVM.