Prediction of Customer Churn Behavior in the Telecommunication Industry Using Machine Learning Models

Abstract

Customer churn is a significant concern, and the telecommunications industry has the largest annual churn rate of any major industry at over 30%. This study examines the use of ensemble learning models to analyze and forecast customer churn in the telecommunications business. Accurate churn forecasting is essential for successful client retention initiatives to combat regular customer churn. We used innovative and improved machine learning methods, including Decision Trees, Boosted Trees, and Random Forests, to enhance model interpretability and prediction accuracy. The models were trained and evaluated systematically by using a large dataset. The Random Forest model performed best, with 91.66% predictive accuracy, 82.2% precision, and 81.8% recall. Our results highlight how well the model can identify possible churners with the help of explainable AI (XAI) techniques, allowing for focused and timely intervention strategies. To improve the transparency of the decisions made by the classifier, this study also employs explainable artificial intelligence methods such as LIME and SHAP to illustrate the results of the customer churn prediction model. Our results demonstrate how crucial it is for customer relationship managers to implement strong analytical tools to reduce attrition and promote long-term economic viability in fiercely competitive marketplaces. This study indicates that ensemble learning models have strategic implications for improving consumer loyalty and organizational profitability in addition to confirming their performance.

Full text

Citation: Chang, V.; Hall, K.; Xu, Q.A.;
Amao, F.O.; Ganatra, M.A.; Benson, V.
Prediction of Customer Churn
Behavior in the Telecommunication
Industry Using Machine Learning
Models. Algorithms 2024,17, 231.
https://doi.org/10.3390/a17060231
Academic Editors: Mateus Mendes
and Balduíno Mateus
Received: 21 April 2024
Revised: 19 May 2024
Accepted: 22 May 2024
Published: 27 May 2024
Copyright: © 2024 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
algorithms
Article
Prediction of Customer Churn Behavior in the
Telecommunication Industry Using Machine Learning Models
Victor Chang 1,* , Karl Hall 2, Qianwen Ariel Xu 1, Folakemi Ololade Amao 2, Meghana Ashok Ganatra 1
and Vladlena Benson 1
1Department of Operations and Information Management, Aston Business School, Aston University,
Birmingham B4 7ET, UK; meghana.a.ganatra@gmail.com (M.A.G.); v.benson@aston.ac.uk (V.B.)
2School of Computing, Engineering and Digital Technologies, Teesside University,
Middlesbrough TS1 3BX, UK; k.hall@tees.ac.uk (K.H.); amaofolakemi@gmail.com (F.O.A.)
*Correspondence: victorchang.research@gmail.com or v.chang1@aston.ac.uk
Abstract: Customer churn is a significant concern, and the telecommunications industry has the
largest annual churn rate of any major industry at over 30%. This study examines the use of ensem-
ble learning models to analyze and forecast customer churn in the telecommunications business.
Accurate churn forecasting is essential for successful client retention initiatives to combat regular
customer churn. We used innovative and improved machine learning methods, including Decision
Trees, Boosted Trees, and Random Forests, to enhance model interpretability and prediction accuracy.
The models were trained and evaluated systematically by using a large dataset. The Random Forest
model performed best, with 91.66% predictive accuracy, 82.2% precision, and 81.8% recall. Our results
highlight how well the model can identify possible churners with the help of explainable AI (XAI)
techniques, allowing for focused and timely intervention strategies. To improve the transparency of
the decisions made by the classifier, this study also employs explainable artificial intelligence methods
such as LIME and SHAP to illustrate the results of the customer churn prediction model. Our results
demonstrate how crucial it is for customer relationship managers to implement strong analytical
tools to reduce attrition and promote long-term economic viability in fiercely competitive market-
places. This study indicates that ensemble learning models have strategic implications for improving
consumer loyalty and organizational profitability in addition to confirming their performance.
Keywords: customer churn prediction; machine learning; explainable AI; ensemble learning; predictive
analytics
1. Introduction
Electronic commerce dramatically boosted the quantity of information accessible to
customers when it first began. In the digital age, consumers have become more informed
about the products they buy. Armed with access to information about a wide range
of related products or services, the behavior of consumers has shifted towards being
less impulsive. Instead, getting a range of information on a wide range of products
to make a more calculated purchasing decision has become the norm. This change in
attitude makes attracting and retaining new customers one of the most critical difficulties
confronting businesses in marketing today. While new businesses focus on obtaining new
consumers, established businesses are more focused on retaining current customers to
increase cross-selling opportunities. Customer satisfaction is one of the keystone concepts
in strategic marketing [
1
]. Efforts have been made to enhance customer value, such as
the 7Cs framework proposed by [
2
]. In several businesses, customer churn rates are a
significant problem. Many studies have demonstrated that even a minor shift in churn rates
may have a big effect on profits. Understanding client behavior in advance can provide
businesses with a competitive edge.
Algorithms 2024,17, 231. https://doi.org/10.3390/a17060231 https://www.mdpi.com/journal/algorithms

Algorithms 2024,17, 231 2 of 21
Even though mobile phones account for over 75% of all potential phone calls world-
wide, the mobile telephone market is one of the most rapidly growing segments of the
telecom sector. For the same reasons that every other competitive market has witnessed
a movement in the competitive landscape from customer acquisition to customer reten-
tion, retail has also seen a shift in the means of competition from customer acquisition to
customer retention. In the telecom industry, churn refers to a company losing customers
to other service providers [
3
]. Like many businesses that work with long-term clientele,
telecom firms utilize customer churn research as one of their primary business indicators [
4
].
According to a survey presented by [
5
], 30–35% of clients leave their telecom company
annually post-COVID. This churn rate may continue to rise with market growth or the
emergence of new, big telecom players in the future. In addition, for cellular corporations,
client acquisition costs can be comparable to 5–10 times the amount spent on customer
retention or satisfaction costs [6].
Therefore, it is essential to learn the reasons why customers leave to reduce the harm
churn has on a business’s bottom line. Predictive marketing analytics and churn analysis
can help identify factors influencing consumers’ voluntary churn by using advanced
machine learning algorithms [
7
]. By investigating the existing studies, it was discovered
that ensemble learning, including classical machine learning algorithms like Random Forest,
Decision Trees, and Naïve Bayes [
8
–
10
], and deep learning methods, such as particle-
classification-optimization-based BP networks [
11
] and Deep-BP-ANN [
12
], have been
employed to improve the accuracy of churn prediction.
However, the models should not only focus on accuracy in predicting churning [
9
]
but also be comprehensible, which means it should provide reasons for churning so that
experts can validate its results and check that it predicts intuitively and correctly. If the
company had understandable and transparent models to work with, it would increase the
understanding of what was causing the churn and how to enhance customer happiness to
boost retention.
As a result, this research aims to construct an accurate, efficient, responsible, and
explainable prediction model for customer attrition in the telecom industry using ensemble
learning methods. Various data mining methods like Decision trees, Random Forests,
and Logistic Regression were utilized by experts to build the predictive model. The
performance of the models was evaluated using the accuracy measures, area under the
curve, and sensitivity and specificity measures. In addition, this study also aims to improve
the interpretability of customer churn predictive models through the use of LIME and
SHAP to provides decision-makers with an overall explanation of the factors affecting
the customer’s decision to churn, as well as a specific analysis for every single customer.
Following a brief introduction to customer churn amplification in the telecom business, the
existing literature is reviewed and summarized in detail.
The remainder of this paper is structured as follows: Section 2focuses on reviewing
the contribution of predictive marketing to Customer Relationship Management, different
machine learning-based churn analysis models for the telecom industry, as well as the
explainable AI. In Section 3, the most acceptable attributes are determined, and the tech-
niques used in this study are introduced. Section 4is concerned with the research analysis
and findings. Finally, the conclusion and several alternate interpretations are presented in
Section 5.
2. Literature Review
The current section is divided into three distinct components. The first section will
introduce Customer Relationship Management (CRM) with its core ideas. The next part
discusses the most well-known and significant ensemble learning models in CRM, which
were also employed in the model design phase of this examination. Finally, the third
segment discusses current research on the importance of explainability and transparency
of the methods used in this regard.

Algorithms 2024,17, 231 3 of 21
2.1. Customer Relationship Management and Predictive Marketing
Customer Relationship Management (CRM) started gaining popularity in the late 1990s
through work conducted by American tech companies such as IBM and Gartner. There are
four main types of CRM—strategic, operational, analytical, and c
ollaborative—which
all aim
to manage the company’s relationship with both potential and current customers:
•
Strategic CRM, which is the use of customer data through systematic analysis as a
means of marketing management [13];
•
Operational CRM, which supports company operations whereby information on
employees, customers, and leads is stored;
•
Analytical CRM, which is a strategy whereby customer and market information is
analyzed to aid the decision-making process for business management [14];
•
Collaborative CRM, which allows for multi-way communications between compa-
nies and their customers, with the goal of facilitating more profitable retention of
customers [15].
The primary goal of CRM is to maximize customer retention while attracting a steady
stream of new customers to maximize the company’s revenue streams [
16
]. By adopting
certain CRM methodologies, companies can improve their understanding of how to en-
gage and interact with their customers to form mutually beneficial relationships through
collaboration [17].
Utilizing information technology and information systems in CRM is becoming in-
creasingly important. More specifically, modern data analysis techniques combined with
the increased amount of available data have made this approach particularly attractive
to businesses. It is critical, therefore, for companies to adopt such practices to achieve
innovative CRM capabilities. Machine learning has emerged as one of the most influen-
tial and popular tools to increase the effectiveness of any CRM strategy. The main four
dimensions [
18
] across which ML is commonly applied in the context of CRM are customer
identification (target customer analysis and customer segmentation), customer attraction
(marketing), customer retention (loyalty program analysis, one-to-one marketing, com-
plaint, and conflict management), and customer development (customer lifetime cycle
analysis, upselling and cross-selling, the market basket analysis).
Big data analytics is transforming businesses by transferring the focus from products
and channels to the customer, with the aim of maintaining personal relationships and
moving from mass marketing to highly personalized marketing. Predictive analytics, used
to transform raw data into useful information, has great relevance for marketing purposes,
allowing the prediction of customer behavior and allocation to specific groups of customers.
By using innovative instruments, companies can adopt big-data-driven, micro-targeting
marketing practices, which permit the improved precision of segmentation and targeting.
Predictive marketing is a powerful tool that can greatly enhance a company’s CRM
capabilities. One of the predictive marketing methods is cluster analysis. Cluster analysis
algorithms can manage vast amounts of data to identify elements that characterize certain
consumers and find correlations that are difficult to identify manually. Conjoint analysis is
also a method adopted in predictive marketing for CRM. It is a research technique used
to identify customer preferences by identifying relevant attributes for consumers in the
selection and purchase processes. The outputs of conjoint analysis often guide business
decisions on new products and promotions. Moreover, online reviews have a strong impact
on consumer choices, and sentiment analysis or opinion mining can potentially predict
future sales and assist marketing strategy. Sentiment analysis is important for a better
understanding of consumer preferences, which leads to the identification of precise targets
and better advertising strategies. Eachempati et al. [
19
] conducted sentiment analysis to
analyze the effect of the emotions expressed on social media platforms by Indian customers
regarding automobile companies’ stock prices. They found that customer sentiment is a
strong factor contributing to the stock price as well as the corporate value. In addition
to the classification algorithms for cluster analysis, Lamrhari et al. [
20
] also employed
cluster analysis and k-means algorithms to group customers into different clusters for

Algorithms 2024,17, 231 4 of 21
the purpose of extracting insights from social media data. Random Forest outperformed
the other models used with 98.46% accuracy and K-means helped to realize the eWoM
communication balance.
While gaining new clients is necessary for a company to develop, customer retention
should not be disregarded. Companies employing CRM strategies should take particular
care to analyze the factors contributing most to improving customer retention rates. Ac-
cording to [
21
], such key factors involve the interplay between relationship management
practices, such as customer trust, employee commitment, and conflict handling. However,
it is also noted that more research needs to be conducted in these areas. Predictive analytics
not only enhances segmentation techniques but also enables the implementation of churn
analysis, which predicts the likelihood of customers leaving the company for competitors.
This information helps the company take initiative-taking measures to prevent customer
churn. Predictive analytics algorithms can estimate the likelihood of customers switching
to competitors and identify the factors that contribute to that probability.
2.2. Customer Churn Prediction Review
Customer churn prediction (CCP) plays a crucial role in this research, since it can
contribute to the impact of customer retention on business profitability and growth. Well-
known papers have been studied and summarized as follows.
Researchers [
16
] have investigated the understanding of nuanced behaviors leading
to customer churn in the telecommunication industry. Their combined approaches, using
temporal centrality metrics and data analytics, have provided insights into early churn
indicators and the role of customer lifecycle stages in influencing churn tendencies. Their
emphasis on leveraging advanced predictive analytics for proactive churn management
underscores the importance of timely interventions and personalized customer outreach.
De Caigny et al. [
22
] focused on developing their convolution neural network (CNN)
techniques in CCP and compared their approach with current practices of text data analysis
via text mining. They have found that CNNs outperform the current practices and un-
structured data in text data prevent them from achieving a high accuracy. Their approach
can extract the most valuable features from the unstructured textual data, thus achieving
more reliable prediction with higher accuracy. Additionally, they [
23
] have focused on
business-to-business (B2B) customer retention and analyzed 6432 customers. They devel-
oped the uplift Logit Leaf model (LLM), which can achieve better performance in CCP than
competing models. By integrating behavioral economics principles with their LLM model,
they have paved the way for more human-centric churn prediction strategies.
Seymen et al. [
24
] proposes a deep learning model for predicting whether retail
customers will churn in the future. The results showed that the deep learning model
achieved better classification and prediction results than the Logistic Regression model
and an artificial neural network model. Deep learning models have been proved to be
effective in a variety of domains, including agriculture [
25
] and stock price forecasting [
26
].
Researchers [
27
] blended deep learning and natural language processing (NLP) to analyze
25,943 customer survey data. Their blended model can identify patterns, which can lead to
improved decision-making and improve accuracy in their CCP analysis. Therefore, the use
of ML algorithms can be effective in prediction robustness.
2.3. Customer Churn Prediction Model for Telecom Industry
Ensemble learning contains the use of multiple ML models to achieve improved
performance and accuracy [
28
,
29
] and also provides more independent views, so that
decision-making can be better and more accurate [
30
]. Ensemble-based ML classifiers
have recently emerged as a new way of building ML models. Since models have different
strengths and weaknesses, researchers have started building hybrid models consisting of
two or more models. Mishra and Reddy [
31
] took this approach and applied ensemble
models, such as Random Forest, Decision Trees, bagging algorithms, and boosting algo-
rithms, to predict customer churn in the telecom industry. They compared the results of

Algorithms 2024,17, 231 5 of 21
these ensemble models with more classical models such as Naïve Bayes. From their results,
they found ensemble classifiers performed favorably, with Random Forest returning the
highest accuracy of 91.66% and all other ensemble classifiers returning results above 90%.
Researchers [
19
] used a particle classification optimization-based BP network for tele-
com customer churn prediction. The PBCCP algorithm is based on particle classification
optimization and particle fitness calculation. In this case, particles refer to vectors contain-
ing the thresholds and weights within a BP neural network. The particles are classified
into categories using their fitness values and are updated using distinct equations. They
found that increasing the number of layers can improve the performance of the algorithm
at the cost of training time. As a result of this, they opted to use one hidden layer in the
neural network. They used a balanced dataset made up of 50% churn customers and 50%
non-churn customers, resulting in the PBCCP network returning an overall accuracy of
73.3%. By comparison, the PSO-BP network returned an overall accuracy of 69.6% and the
BP model had 63.6%.
Similarly, researchers [
9
] devised the Logit Leaf model, an ensemble model using
aspects of Logistic Regression and Decision Trees, and compared their results with standard
Decision Tree, Logistic Model Tree, Logistic Regression, and Random Forest models. The
AUC performance criteria were one of the metrics used to evaluate performance—a metric
commonly used for evaluating the performance of binary classification systems, such as
customer churn prediction. In this regard, the Logit Leaf model performed the best, slightly
better than Random Forest. The same was true when the models were evaluated using the
TDL (10%) performance criteria. The Logit Leaf model was also more efficient at making
predictions, taking less time than the other models used. When real-time predictions are
required, this can be a crucial factor when deciding which model to use for a given problem.
Finally, scientists [
10
] applied gravitational search algorithms to perform effective
feature selection, resulting in a dimensionality reduction in the dataset used for customer
churn prediction. After this, they applied a selection of ML models for comparison: Logistic
Regression, Naïve Bayes, Support Vector Machine, Decision Trees, Random Forest, and
Extra Tree Classifiers and boosting algorithms such as Adaboost, XGBoost, and CatBoost.
They found that, when comparing the models used, the boosting algorithms performed the
best, with CatBoost achieving the highest accuracy, recall, and precision with scores of 81.8%,
82.2%, and 81.2%, respectively. When comparing their AUC scores, boosting algorithms
performed the best again, with XGBoost and Adaboost scoring the highest at 84%.
One of the most important applications of ML with regard to CRM is to give companies
the ability to predict customer churn. This is particularly important given that retaining
existing customers is significantly more valuable than acquiring new customers, as more
resources are needed for new customer acquisition. It follows, therefore, that customer
retention is one of the key priorities of the CRM strategy [
32
]. The ability to harness ML to
significantly improve customer churn rates is of particular importance.
In this study, we will make use of predictive machine learning models to identify clients
who are likely to churn after categorizing current customers using clear machine learning.
2.4. Explainable Artificial Intelligence (AI) in Churn Analysis
While the application of machine learning algorithms has brought many benefits to
business, it has also revealed several drawbacks. For example, in Customer Relationship
Management, it is essential that every decision is meaningful. Advanced data analytics
should enable decision-makers to understand the reason underlying the model’s prediction
of customer behavior results so that they can tailor their decisions and adjust personalized
marketing strategies accordingly [
4
]. Decision-makers are often wary of or even reject AI
systems because the performance of the predictive model or system is overemphasized
while interpretability or transparency is ignored [33].
Many popular ML models are considered as “black boxes” [
7
,
34
], meaning that the
inner workings of the algorithms and associated decisions or classifications made are
secretive. For example, individual predictions often lack interpretability when predicting a

Algorithms 2024,17, 231 6 of 21
potential customer’s risk of discontinuing a telecommunications service. It is difficult to
relate the predicted probability of customer churn to customer characteristics, which creates
challenges in determining customer retention success and deciding on refinement strategies.
Additionally, from a legislative perspective, the interpretability of AI algorithm conclusions
is itself a right of consumers and regulators have a responsibility to ensure this right.
Studies have discussed data transparency and ethical issues from the perspective of users
or data subjects [
35
]. To address this problem, explainable AI (XAI) models are required to
provide details or interpretations that make the operations of AI understandable.
According to [
36
,
37
], explainable AI enables interested parties to comprehend the
primary influences on model-driven decisions as well as how AI algorithms carry out their
operations, forecast outcomes, and make decisions. The European General Data Protection
Regulation (GDPR) emphasizes that intelligent decisions should be accompanied by relevant
information about the logic involved and the significance and possible effects of their
processing for the data subject. Therefore, the GDPR grants data subjects the right to obtain
pertinent information regarding the basis for smart decision-making [38].
A few XAI approaches have been used for customer churn predictive models in the
existing literature, including global surrogate models, Partial Dependence Plots (PDPs),
Accumulated Local Effects (ALEs) [4], LIME [39], and Shapley values [40,41].
In the work of [
4
], a number of XAI methods were employed to explain the customer
churn predictive model by using a dataset of the telecom industry. The discussed global
explanation methods included Partial Dependence Plots (PDPs), Accumulated Local Effects
(ALEs), global surrogate models, and Shapley Valuea, as well local methods, including
Individual Conditional Expectation (ICE) and the local surrogate model (LIME). The results
show that a thorough understanding of the data, the inner workings of the model, and
the issue can be addressed by the combination of several interpretations and approaches
in a specific way. Scientists [
42
] proposed a novel XAI framework to identify the most
significant factors that influence a customer’s decisions on purchasing or abandoning non-
life insurance coverage. Their framework applied similarity clustering to the Shapley values
from a predictive model created by XGBoost. The results showed that the integration of
the clustering technique and Shapley values effectively grouped the customers, which was
used to forecast the churn decisions of customers. Researchers [
7
] provided an XAI solution
for predicting customer churn and outputting its explanations through visualization. This
system integrates two major methods, the Random Forest algorithm used to train a churn
classifier and Shapley values used to provide global and local explanations for the churn
classifier. Using LIME and Shapley techniques, Ref. [
39
] interpreted customer churn
predictive models constructed using Random Forest and light-gradient-boosting machines
from local and global perspectives, respectively.
Personalized marketing initiatives, especially those related to managing customer
churn, are essential for companies looking to establish and uphold strong customer relation-
ships. The use of predictive analytics methods undoubtedly enhances the efficacy of these
endeavors [
43
]. For better customer churn prediction, various emerging ML algorithms
have been employed in the existing literature and have performed well in terms of accuracy.
Nevertheless, decisions produced via ML-based customer churn predictive models are not
easily understood by managers of customer relationships because of the black-box nature
of artificial intelligence. Further research in this area is nascent. Therefore, the aim of this
research is to contribute to the literature by improving the interpretability of customer
churn predictive models through the use of XAI interpretations to optimize their local and
global interpretability.
3. Research Methodology
The telecommunications sector has long struggled with churn. This research aims to
create and deploy a cost-effective system for predicting client churn in the telecommunica-
tions sector. Addressing this issue is expected to yield a deeper comprehension of churning
customers, enabling the identification of such customers and providing a foundation for

Algorithms 2024,17, 231 7 of 21
future initiatives aimed at reducing the sector’s churn rate. Section 3discusses both the
research approach selected and the machine learning models adopted.
3.1. The Crisp Model
The CRoss-Industry Standard Process for Data Mining (CRISP-DM) model is referred
to as a standardized way of obtaining a good process via data mining across businesses and
industries where data and modeling are a priority [
44
]. Researchers advise that, after twenty
years of developing CRISP-DM, the emphasis is on data science and the methodologies
should accommodate the need for data release, data architecting, data simulation, and
data acquisition [
45
]. Business processes and demands can be centered based on the data-
driven approach. In other words, we can check work progress, evaluate our outputs, and
make decisions in real-time. By doing so, our efficiency and accuracy in our tasks can be
significantly improved.
Thus, CRISP-DM models are an apparent methodological way of directing the research’s
procedure. The process diagram of the CRISP-DM model is depicted in Figure 1below.
Algorithms2024,17,xFORPEERREVIEW7of22

algorithmshavebeenemployedintheexistingliteratureandhaveperformedwellin
termsofaccuracy.Nevertheless,decisionsproducedviaML-basedcustomerchurnpre-
dictivemodelsarenoteasilyunderstoodbymanagersofcustomerrelationshipsbecause
oftheblack-boxnatureofartificialintelligence.Furtherresearchinthisareaisnascent.
Therefore,theaimofthisresearchistocontributetotheliteraturebyimprovingtheinter-
pretabilityofcustomerchurnpredictivemodelsthroughtheuseofXAIinterpretationsto
optimizetheirlocalandglobalinterpretability.
3.ResearchMethodology
Thetelecommunicationssectorhaslongstruggledwithchurn.Thisresearchaimsto
createanddeployacost-effectivesystemforpredictingclientchurninthetelecommuni-
cationssector.Addressingthisissueisexpectedtoyieldadeepercomprehensionof
churningcustomers,enablingtheidentificationofsuchcustomersandprovidingafoun-
dationforfutureinitiativesaimedatreducingthesector’schurnrate.Themethodology
sectiondiscussesboththeresearchapproachselectedandthemachinelearningmodels
adopted.
3.1.TheCrispModel
TheCRoss-IndustryStandardProcessforDataMining(CRISP-DM)modelisreferred
toasastandardizedwayofobtainingagoodprocessviadataminingacrossbusinesses
andindustrieswheredataandmodelingareapriority[44].Researchersadvisethat,after
twentyyearsofdevelopingCRISP-DM,theemphasisisondatascienceandthemethod-
ologiesshouldaccommodatetheneedfordatarelease,dataarchitecting,datasimulation,
anddataacquisition[45].Businessprocessesanddemandscanbecenteredbasedonthe
data-drivenapproach.Inotherwords,wecancheckworkprogress,evaluateouroutputs,
andmakedecisionsinreal-time.Bydoingso,ourefficiencyandaccuracyinourtaskscan
besignificantlyimproved.
Thus,CRISP-DMmodelsareanapparentmethodologicalwayofdirectingthere-
search’sprocedure.TheprocessdiagramoftheCRISP-DMmodelisdepictedinFigure1
below.

Figure1.Crispmodelcycle.
Wefirstdefineaspecificbusinessproblem,i.e.,customerchurninthetelecommuni-
cationindustry,andsetexplicitgoalstomitigatethisproblemthroughpredictive
Figure 1. Crisp model cycle.
We first define a specific business problem, i.e., customer churn in the telecommunica-
tion industry, and set explicit goals to mitigate this problem through predictive modeling.
Following the CRISP-DM framework, we collected and preprocessed customer data from
the telecom industry, focusing on the characteristics that may indicate customer churn.
Throughout the modeling phase, we applied various data mining techniques such as Deci-
sion Trees, Random Forests, and Logistic Regression. After modeling, we evaluated the
performance of the models using rigorous metrics such as accuracy and area under the
curve to ensure that they meet the operational requirements of the business environment.
Finally, we translated the findings from the models into actionable strategies aimed at
customer retention.
3.2. Dataset
The dataset used in this study is a publicly available, large dataset. The aim was to
find a sufficiently large and recent dataset regarding churn within the telecommunication
industry. The dataset in question was selected based on meeting certain search
criteria—it
could meet a representative size and was presumed to contain relevant explanatory vari-
ables such as demographics. Finally, a Telecom CUSTOMER Churn dataset from the Maven
Analytics website platform was selected. It consists of customer activity data (features),

Algorithms 2024,17, 231 8 of 21
along with a churn label specifying whether a customer canceled the subscription, which
we used to develop predictive models. The dataset consists of 7043 rows and 38 attributes.
The attributes consist of different pieces of information about the individual customer,
including the status of the customer’s subscription, which is categorized as churn and
not churn.
A high-quality dataset is required for further analysis; therefore, this study examined
each variable in the dataset for missing values. In order to select a predictive model with
optimal predictive accuracy, the research data was divided into two groups—the Train
(75%) and Test (25%) datasets—by putting into consideration the great ratio of 2:1 (non-
churner: churner) based on Wei and Chiu ’s [
31
] research. After testing data quality and
selecting variables, none of the observations or rows were deleted.
3.3. Machine Learning Algorithms
Different machine learning techniques were adopted to create a customer churn
prediction system. These techniques were adopted to find the best possible way to predict
customer churn through a model with high predictive accuracy. The five techniques used
were the Logistic Regression model, the Random Forest classifier, the Naïve Bayes classifier,
The K-Nearest Neighbor classifier, and the Decision Tree Classifier. All the models and
their functions are described below.
Logistic Regression: Logistical models attempt to create a regression model based on
data with the binary response variable. These models can be used, among other things,
to estimate population groups where a statement may be true or false, such as churn or
non-churn. In Logistic Regression, the logit function is used to determine the probability of
a binary outcome.
K-Nearest Neighbor (KNN) Classifier: KNN is a non-parametric algorithm and a
lazy-learner algorithm, meaning it does not make any assumptions about the underlying
data and does not immediately learn from the training set until new data is obtained. The
KNN algorithm assumes that the new data is similar to the existing data and assigns the
new data to the category that is most similar to the existing category based on the distance
between two points.
Naïve Bayes Classifier: This procedure is based on Bayes’ Theorem and the assump-
tion that the predictors are unrelated. In contrast to other models, the Naive Bayes model
is straightforward to create and is particularly successful when dealing with large datasets.
Additionally, it is easy to use.
Decision Tree: Decision Tree learning is a fundamental technique in decision theory.
These trees comprise a root at the top and knots that are interconnected by branches. Nodes
can be classified as either internal or terminal. At each internal node, a specific attribute
is tested, and the result guides the selection of different branches, eventually leading to a
terminal node. The terminal nodes, or “leaves”, correspond to a classification [46].
Random Forest Classifier: A Random Forest is a combination method that works
with Decision Trees as building blocks. The algorithm generates a predefined number of
trees and takes a cut of the total number of trees and uses it as its predictor [
47
]. In the
task, the Random Forest model can be adjusted to achieve the best possible performance by
adjusting the number of trees and the number of cuts used in the algorithm.
3.4. Model Performance Evaluation
In classification problems, the following measures are used to assess the performance
of ML models.
3.4.1. Confusion Matrix
A Confusion Matrix is a well-known way of examining how a good classification
model achieves the observed classes. The Confusion Matrix is presented in Table 1
as follows.

Algorithms 2024,17, 231 9 of 21
Table 1. Confusion Matrix.
Observed 0 Observed 1
Estimated 0 TN FN
Estimated 1 FP TP
The four boxes in the classification table are all assigned a name: FN,FP,TP, and TN.
•
TN stands for true negative. Here, the customers are observed as not being churners,
and the model has also classified the customers as non-churners.
•
FP stands for false positive. Here, customers are observed as being non-churners, but
the model has classified the customers as churners.
•
FN stands for false negative. Customers are observed as being churners, but the model
has classified the customers as non-churners.
•
TP stands for true positive. Here, customers are both observed and classified
as churners.
The Confusion Matrix is not only a visually advantageous way of measuring the
model’s ability to classify correctly. Several calculations can also be made based on the four
values. These calculations are all targets for more specific evaluations of the model and can,
therefore, be used to identify the model’s strengths and weaknesses.
3.4.2. Four Evaluation Metrics
This section describes four metrics used to assess the performance of churn prediction
models, including accuracy, error rate, specificity, and sensitivity.
Accuracy: Accuracy is the proportion of correctly classified observations out of all
customers classified. This is an overall score, which counts on the overall performance of
the model. Accuracy is determined as follows:
Accuracy =TP +T N
TP +T N +FP +F N . (1)
Error rate: As opposed to accuracy, the error rate is the proportion of incorrectly
classified observations out of all customers classified. The error rate is calculated as follows:
Error rate = (FP +FN)/(TP +TN +FP +FN). (2)
Specificity: This is the true negative rate. The rate, thus, indicates how large a pro-
portion of the customers estimated as non-churners are correctly classified. Specificity is
determined as follows:
Specificity = TN/(TN +FP). (3)
Sensitivity: This is the true positive rate. The rate indicates what percentage of the
customers were estimated as churners and classified correctly. Sensitivity is calculated
as below:
Sensitivity = TP/(TP +FN).
3.4.3. Other Evaluation Metrics
ROC stands for Receiver Operating Characteristic and is a visual representation of the
ability to classify correctly. The graph shows the relationship between the false positive
rate on the x-axis and the true positive rate on the y-axis. Three different ROC curves are
shown in Figure 2.

Algorithms 2024,17, 231 10 of 21
Algorithms2024,17,xFORPEERREVIEW10of22

Specificity=TN/(TN+FP). (3)
Sensitivity:Thisisthetruepositiverate.Therateindicateswhatpercentageofthe
customerswereestimatedaschurnersandclassifiedcorrectly.Sensitivityiscalculatedas
below:
Sensitivity=TP/(TP+FN).
3.4.3.EvaluationMetrics
ROCstandsforReceiverOperatingCharacteristicandisavisualrepresentationof
theabilitytoclassifycorrectly.Thegraphshowstherelationshipbetweenthefalseposi-
tiverateonthex-axisandthetruepositiverateonthey-axis.ThreedifferentROCcurves
areshowninFigure2.

Figure2.ROCcurves.
TheROCcurveinthegraphtotheleftindicatesamodelthatcorrectlypredicts50%
ofthetimeand,therefore,alsoindicatesrandomclassification.Thegraphinthemiddle
indicatesamodelwithimprovedpredictiveabilitybutthatstillleadstomisclassification.
Inthegraphtotheright,theROCcurveindicatesthatthemodelclassifiesclosetoperfect,
asthecurveisveryclosetotheupperleftcorner.
AUCstandsforareaunderthecurveandmeasurestheareaundertheReceiverOp-
eratingCharacteristic(ROC)curve.Itisonewaytoobtainamoreaccuratemeasurement
ofthemodel’sperformance.AperfectmodelwillhaveanAUCof1,afterwhichthe
model’squalitydecreasesastheAUCvaluedecreases.
3.5.ExplainableAITechniques
Duetotheblack-boxnatureofmachinelearningalgorithms,theoperatingprinciples
ofthealgorithmsaredifficulttounderstandandcannotbeeasilyexplainedtodecision-
makersinthetelecomindustry,especiallyiftheydonothaveacomputerscienceback-
ground.Asaresult,althoughMLalgorithmsexcelintermsofaccuracy,theymaynotgain
thetrustofdecision-makersorusersorbeacceptedinreal-worldenterprisemanagement.
Whileaimingtoaddressthisissue,afterevaluatingtheclassifiers,thispaperintroduces
theconceptofexplainableAIinthestudyandillustratestheresultsofthebestcustomer
churnpredictionmodelsthroughtwovisualizationtools,includingLIMEandSHAP.
First,LocalInterpretableModel-AgnosticExplanations(LIMEs)areemployedtoop-
timizethelocalinterpretabilityoftheDecisionTreeclassifiersandRandomForestclassi-
fiers.Itemphasizestraininglocallyinterpretablemodelsthatmaybeusedtoexplainspe-
cificpredictionsandhelpdecision-makersunderstandwhyaparticularclasswaspre-
dictedforacertaininstance[46].
However,thepurelylocalcharacteroftheLIMEexplanation,whichonlypredicts
howthepresentpredictionwillchangeinresponsetominutechangesintheinputvalues,
isoneofitslimitations.Asaresult,ratherthanservingasaninterpretationofwhythe
forecastwasmadeinthefirstplace,itactsmorelikeasensitivitystudy.Therefore,this
Figure 2. ROC curves.
The ROC curve in the graph to the left indicates a model that correctly predicts 50%
of the time and, therefore, also indicates random classification. The graph in the middle
indicates a model with improved predictive ability but that still leads to misclassification.
In the graph to the right, the ROC curve indicates that the model classifies close to perfect,
as the curve is very close to the upper left corner.
AUC stands for area under the curve and measures the area under the Receiver
Operating Characteristic (ROC) curve. It is one way to obtain a more accurate measurement
of the model’s performance. A perfect model will have an AUC of 1, after which the
model’s quality decreases as the AUC value decreases.
3.5. Explainable AI Techniques
Due to the black-box nature of machine learning algorithms, the operating principles of
the algorithms are difficult to understand and cannot be easily explained to decision-makers
in the telecom industry, especially if they do not have a computer science background.
As a result, although ML algorithms excel in terms of accuracy, they may not gain the
trust of decision-makers or users or be accepted in real-world enterprise management.
While aiming to address this issue, after evaluating the classifiers, this paper introduces the
concept of explainable AI in the study and illustrates the results of the best customer churn
prediction models through two visualization tools, including LIME and SHAP.
First, Local Interpretable Model-Agnostic Explanations (LIMEs) are employed to
optimize the local interpretability of the Decision Tree classifiers and Random Forest
classifiers. It emphasizes training locally interpretable models that may be used to explain
specific predictions and help decision-makers understand why a particular class was
predicted for a certain instance [46].
However, the purely local character of the LIME explanation, which only predicts
how the present prediction will change in response to minute changes in the input values,
is one of its limitations. As a result, rather than serving as an interpretation of why the
forecast was made in the first place, it acts more like a sensitivity study. Therefore, this
study also employs another explanation technique, SHAP, to show the overall importance
of the factors.
The SHapley Additive exPlanation (SHAP) is a method of interpreting the results of
any ML model by attributing an importance score to each feature in the data [
48
]. In this
study, summary plots are drawn to allow visualization of the importance of features and
their influence on prediction. The features are ranked according to the sum of the SHAP
values of all samples. The colors indicate the high and low SHAP values of the features,
i.e., blue indicates low importance and red indicates high importance. It makes use of the
SHAP values to show the impact distribution of each feature as well.

Algorithms 2024,17, 231 11 of 21
4. Analysis and Results
This section will present the results from the application of the machine learning model
described in Section 3of this paper to the research dataset, as well as the model evaluation
procedure. The initial step of the analysis section involves the descriptive statistics, which
will help to build an understanding of the data for the subsequent analysis. The second
area consists of, Sections 4.2 and 4.3, showing different but related results and analysis. The
section will conclude with a review of the variable and the machine learning technique that
provides the best algorithm to predict customer churn.
4.1. Descriptive Statistics
To develop a better understanding of the attributes, it is required to construct summary
statistics for all the features contained in the dataset used in this study. Table 2shows
the outcomes.
Table 2. Table of descriptive statistics.
Count Mean Std Min 25% 50% 75% Max
Age 4601 47.89307 17.362 19 33 47 62 80
Number of Dependents 4601 0.380569 0.880541 0 0 0 0 8
Number of Referrals 4601 1.947403 2.957352 0 0 0 3 11
Tenure in Months 4601 34.63117 24.19849 1 12 32 58 72
Avg Monthly
Long-Distance Charges 4601 25.5811 14.26574 1.01 13.02 25.84 37.97 49.99
Avg Monthly GB
Download 4601 26.12889 19.53716 2 13 21 30 85
Monthly Charge 4601 81.20272 21.14967 −10 69.9 83.75 96.2 118.75
Total Charges 4601 3042.595 2391.057 42.9 847.3 2564.3 4968 8684.8
Total Refunds 4601 2.163306 8.286778 0 0 0 0 49.57
Total Extra Data Charges 4601 8.943708 28.62071 0 0 0 0 150
Total Long-Distance
Charges 4601 888.9163 866.5069 1.13 178.89 582 1417.92 3536.64
Total Revenue 4601 3938.291 3054.189 46.92 1119.4 3378.79 6412.05 11979.34
From the over 4601 customer samples used in this research study, the number of
referrals is 1.95, with an SD of 2.96, which means that, on average, each customer referred
this telecom company to two friends. The average download volume in gigabytes to the
end of the second quarter is 26.12 GB, and the customer’s total charges for additional data
downloads for the same quarter is 8.94, with an SD of 28.62. Table 2also shows details
of the customers’ charge history, where the average values are 3042.59, 2.16, 8.94, and
888.91 for total charges, total refunds, total extra data charges, and total long-distance
charges, respectively.
The bar chart presented below was computed to examine the distribution of the target
variable across the customer service call variable. It can be observed from the chart that
the dataset is unbalanced, with almost twice as many churning samples than not churning
samples. See Figure 3.
To examine the inter-correlation among the features, a correlation matrix was com-
puted and displayed in the form of a heatmap to indicate the pairwise relationship among
the call activities and features of the telecom customers. See Figure 4.

Algorithms 2024,17, 231 12 of 21
Algorithms2024,17,xFORPEERREVIEW12of22

Thebarchartpresentedbelowwascomputedtoexaminethedistributionofthetar-
getvariableacrossthecustomerservicecallvariable.Itcanbeobservedfromthechart
thatthedatasetisunbalanced,withalmosttwiceasmanychurningsamplesthannot
churningsamples.SeeFigure3.

Figure3.Barplotforcustomerchurndistribution.
Toexaminetheinter-correlationamongthefeatures,acorrelationmatrixwascom-
putedanddisplayedintheformofaheatmaptoindicatethepairwiserelationshipamong
thecallactivitiesandfeaturesofthetelecomcustomers.SeeFigure4.

Figure4.HeatmapofCorrelationMatrix.

Figure 3. Bar plot for customer churn distribution.
Algorithms2024,17,xFORPEERREVIEW12of22

Thebarchartpresentedbelowwascomputedtoexaminethedistributionofthetar-
getvariableacrossthecustomerservicecallvariable.Itcanbeobservedfromthechart
thatthedatasetisunbalanced,withalmosttwiceasmanychurningsamplesthannot
churningsamples.SeeFigure3.

Figure3.Barplotforcustomerchurndistribution.
Toexaminetheinter-correlationamongthefeatures,acorrelationmatrixwascom-
putedanddisplayedintheformofaheatmaptoindicatethepairwiserelationshipamong
thecallactivitiesandfeaturesofthetelecomcustomers.SeeFigure4.

Figure4.HeatmapofCorrelationMatrix.

Figure 4. Heatmap of Correlation Matrix.
4.2. Results Based on Confusion Matrix
Customer churn is a critical issue for telecom companies, as it can significantly impact
their revenue and profitability. Therefore, accurate prediction of customer churn is an
important task in the industry. This study applied five different ML algorithms to predict
customer churn in the telecom industry, including Logistic Regression, KNN model, Naïve
Bayes, Decision Tree, and Random Forest. Their performance was first evaluated using a
Confusion Matrix, which consisted of values of true negatives (TN) and true positives (TP),
false negatives (FN) and false positives (FP). As stated in Section 3.4.1,TNs are cases where
the actual negative is also predicted to be negative, while TPs are cases where the actual
positive is also predicted to be positive. FNs are cases that are positive but predicted to be
negative, while FPs are cases that are actually negative but predicted to be positive. See
Figure 5.

Algorithms 2024,17, 231 13 of 21
Algorithms2024,17,xFORPEERREVIEW13of22

4.2.ResultsofAnalysisandDiscussion
4.2.1.ResultsBasedonConfusionMatrix
Customerchurnisacriticalissuefortelecomcompanies,asitcansignificantlyimpact
theirrevenueandprofitability.Therefore,accuratepredictionofcustomerchurnisanim-
portanttaskintheindustry.ThisstudyappliedfivedifferentMLalgorithmstopredict
customerchurninthetelecomindustry,includingLogisticRegression,KNNmodel,Na-
ïveBayes,DecisionTree,andRandomForest.Theirperformancewasfirstevaluatedusing
aConfusionMatrix,whichconsistedofvaluesoftruenegatives(TN)andtruepositives
(TP),falsenegatives(FN)andfalsepositives(FP).AsstatedinSection3.4.1,TNsarecases
wheretheactualnegativeisalsopredictedtobenegative,whileTPsarecaseswherethe
actualpositiveisalsopredictedtobepositive.FNsarecasesthatarepositivebutpredicted
tobenegative,whileFPsarecasesthatareactuallynegativebutpredictedtobepositive.
SeeFigure5.

Figure5.ConfusionMatrixvaluesforthefivealgorithms.
Aclusteredbarchartwasdrawntoillustratetheperformanceofthesealgorithmsin
termsofthesefourmetrics.RandomForestappearedtohavethehighestnumberofTP
andTNandthelowestnumberofFNandFP,indicatingthatitmaybethebest-performing
modelofthefive.Itcanbeobservedthat641customersfromthetestdatasetarecorrectly
classifiedasnon-churner,109ofthecustomerswhoarenon-churnersareincorrectlyclas-
sifiedaschurners,88ofthecustomersthatarechurnersaremisclassifiedasnon-churners,
and670ofthecustomersareclassifiedcorrectlyaschurners.Thepredictionaccuracyof
theRandomForestalgorithmwascomputedtobeequalto0.8694.Thisimpliesthat
86.94%ofthecustomersarecorrectlyclassifiedbytheRandomForestmodel.Random
Forestiseffectiveinidentifyingbothcustomersatriskofchurn(TP)andthoseunlikelyto
churn(TN),whileavoidingmisclassificationofnon-churningcustomers,whichcanlead
tocustomerdissatisfaction,orchurningcustomersasnon-churningandmissedopportu-
nitiestotakeactiontoretainthem,leadingtolossofprofit.
IntermsoftheTPmetric,DecisionTreewasthesecond-bestchoiceafterRandom
Forest,indicatingthatitwasexcellentatidentifyingcustomersatriskofchurn(TP:614).
However,itsrelativelylowerTNvaluesof593comparedtotheotheralgorithmsindicate
thatitperformspoorlyinidentifyingnon-churningcustomers.Furthermore,itsrelatively
lowFNvalues(144)andhighFPvalues(157)confirmthattheDecisionTreehasaten-
dencytoclassifycustomersasthoseatriskofchurn.Therefore,telecomcompaniesmay
555
635
617
593
641
174
318
171
144
88
584
440
587
614
670
195
115
133
157
109
0 100 200 300 400 500 600 700 800
LogisticRegression
KNNModel
NaïveBayes
DecisionTree
RandomForest
FP TP FN TN
Figure 5. Confusion Matrix values for the five algorithms.
A clustered bar chart was drawn to illustrate the performance of these algorithms
in terms of these four metrics. Random Forest appeared to have the highest number
of TP and TN and the lowest number of FN and FP, indicating that it may be the best-
performing model of the five. It can be observed that 641 customers from the test dataset
are correctly classified as non-churner, 109 of the customers who are non-churners are
incorrectly classified as churners, 88 of the customers that are churners are misclassified as
non-churners, and 670 of the customers are classified correctly as churners. The prediction
accuracy of the Random Forest algorithm was computed to be equal to 0.8694. This implies
that 86.94% of the customers are correctly classified by the Random Forest model. Random
Forest is effective in identifying both customers at risk of churn (TP) and those unlikely to
churn (TN), while avoiding misclassification of non-churning customers, which can lead to
customer dissatisfaction, or churning customers as non-churning and missed opportunities
to take action to retain them, leading to loss of profit.
In terms of the TP metric, Decision Tree was the second-best choice after Random
Forest, indicating that it was excellent at identifying customers at risk of churn (TP: 614).
However, its relatively lower TN values of 593 compared to the other algorithms indicate
that it performs poorly in identifying non-churning customers. Furthermore, its relatively
low FN values (144) and high FP values (157) confirm that the Decision Tree has a tendency
to classify customers as those at risk of churn. Therefore, telecom companies may want
to carefully assess the accuracy of the algorithm’s positive predictions before taking any
retention action.
Naïve Bayes and Logistic Regression performed similarly with moderate levels of
performance, with high numbers of true negatives (TNNB: 617; TNLR: 555) and true
positives (TPNB: 587; TPLR: 584) as well as high numbers of false negatives (FNNB: 171;
FNLR: 174) and false positives (FPNB: 133; FPLR: 195). This suggests that these algorithms
can effectively identify customers who are likely to churn (TP) and those who are unlikely
to churn (TN), but it may also misclassify some customers as either those who will not
churn (FN) or those who will churn (FP).
The KNN model is the worst performer of all the algorithms. Although it performs
well in identifying non-churning customers, as evidenced by the high number of TNs
(
635 customers
), it performs rather poorly in identifying customers at risk of churning (TP:
440 customers). In addition, it has a high number of FN (318 customers) and a relatively
low number of FP (115 customers) compared to the other models, which could indicate
a tendency to misclassify positive instances (i.e., customers who will churn) as negative

Algorithms 2024,17, 231 14 of 21
(i.e., customers who will not churn). This could potentially lead to missed opportunities for
telecom companies to take retention action.
The area under the Receiver Operating Characteristic (ROC) curve is a metric widely
used to assess the overall performance of binary classification models. The results of
AUC-ROC for all algorithms tested in this study are shown in Figure 6. Random Forest
had the highest score of 0.95, followed closely by Naïve Bayes, with a score of 0.88. These
two models were the most effective at predicting customer churn in the telecom industry in
this study. Logistic Regression also performed well, with an AUC score of 0.84, indicating
that it is a reliable algorithm for predicting customer churn. KNN and Decision Tree had
lower AUC scores of 0.81 and 0.8, respectively, suggesting that they may be less accurate in
distinguishing between the customers at risk of churn and customers who will not churn.
See Figure 6.
Algorithms2024,17,xFORPEERREVIEW15of22


Figure6.ROCcurvesforthefivemodels.
Insummary,fromtheanalysisoftheConfusionMatrixandAUC-ROCcurve,Ran-
domForestwasthemosteffectivemodelforautomaticallydetectingcustomerchurnrisk
fortelecomcompanies.Theperformanceofthefivemodelswillbefurtherevaluatedin
thenextsection.
4.3.ResultsofAnalysisandDiscussion
Thisstudyaimstoidentifythesystemthatproducesgoodpredictionsoncustomer
churnthroughmachinelearningmodels.Therefore,itisnecessarytocomparethemodels
selectedinthisstudyandidentifythemodelwiththebestpredictiveability.Presentedin
Table3arethemodelevaluationcriteriaconsideredinthisstudy.
Tab l e 3.Tableofdescriptivestatistics.
ModelAccuracyAUCSensitivitySpecificity
LogisticRegression0.75530.840.74000.7704
KNN0.71290.810.84670.5805
NaïveBayes0.79840.880.82270.7744
DecisionTree0.80040.800.79070.8100
RandomForest0.86940.950.85470.8839
Theaccuracymeasureistheabilityofamachinelearningmodeltomakeacorrect
prediction.AsobservedfromTable3above,themachinelearningmodelsbasedonNaïve
Bayes,DecisionTree,andRandomForestalgorithmscomputedhighpredictionaccuracy,
astheycomputedwellwitharound80%oraboveaccuracymeasures.TheRandomForest
classifiercomputedthehighestaccuracy(86.94%)measureswitharound7%pointsbeer
(a) Logistic Regression Model. (b) KNN Model. (c) Naïve Bayes Model.
(d) Decision Tree Model. (e) Random Forest Model.
Figure 6. ROC curves for the five models.
In summary, from the analysis of the Confusion Matrix and AUC-ROC curve, Random
Forest was the most effective model for automatically detecting customer churn risk for
telecom companies. The performance of the five models will be further evaluated in the
next section.
4.3. Results of Analysis and Discussion
This study aims to identify the system that produces good predictions on customer
churn through machine learning models. Therefore, it is necessary to compare the models
selected in this study and identify the model with the best predictive ability. Presented in
Table 3are the model evaluation criteria considered in this study.
The accuracy measure is the ability of a machine learning model to make a correct
prediction. As observed from Table 3above, the machine learning models based on Naïve
Bayes, Decision Tree, and Random Forest algorithms computed high prediction accuracy,
as they computed well with around 80% or above accuracy measures. The Random Forest
classifier computed the highest accuracy (86.94%) measures with around 7% points better
than the decision tree model, which computed an 80.04% accuracy measure. Figure 7
depicts a graphical representation of the model evaluation metrics.

Algorithms 2024,17, 231 15 of 21
Table 3. Table of descriptive statistics.
Model Accuracy AUC Sensitivity Specificity
Logistic Regression 0.7553 0.84 0.7400 0.7704
KNN 0.7129 0.81 0.8467 0.5805
Naïve Bayes 0.7984 0.88 0.8227 0.7744
Decision Tree 0.8004 0.80 0.7907 0.8100
Random Forest 0.8694 0.95 0.8547 0.8839
Algorithms2024,17,xFORPEERREVIEW16of22

thanthedecisiontreemodel,whichcomputedan80.04%accuracymeasure.Figure7de-
pictsagraphicalrepresentationofthemodelevaluationmetrics.

Figure7.ModelEvaluation.
4.4.ModelInterpretation
Accordingtothemodelcomparisonresults,thebestcustomerchurnprediction
modelwascreatedfromtherandomforestalgorithm,whichhadapredictionaccuracyof
closeto90%.Afterthat,toaddresswiththeblack-boxnatureofmachinelearningalgo-
rithms,thispaperillustratestheresultsofcustomerchurnpredictionmodelsthroughtwo
explainableAItools,includingLIMEandSHAP.
4.4.1.LocalInterpretableModel-AgnosticExplanations(LIME)
Figure8showsthelocalexplanationsoftheRandomForestclassifierforthefirst
sampleandthesecondsampleusingtheLIMEtechnique.AccordingtoFigure8,thefive
mostsignificantfactorsinRandom-Forest-basedchurndetectionforthefirstcustomerare
the‘Contract’,‘NumberofDependents’,‘OnlineSecurity’,‘PremiumTechSupport’,and
‘NumberofReferrals’.Furthermore,anincreaseinthevalueofthesefivefactorswilldrive
thepredictiontowardnotchurning,whileanincreaseinthevalueof‘PaymentMethod’
and‘MonthlyCharge’willincreasethelikelihoodofchurning.Forthesecondcustomer,
‘MonthlyCharge’wasmoreimportantthan‘PremiumTechSupport’andwasoneofthe
topfivefactorsthatinfluencedacustomer’ssubscriptiondecision.Inaddition,contrary
tothefirstcustomer,LIMEindicatesthatanincreaseinthevalueof‘Contract’,‘Number
ofDependents’,and‘NumberofReferrals’valuescanincreasethelikelihoodofthesecond
customer’schurn.SeeFigure8.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Logisticregression KNN Naïvebayes Decisiontree Randomforest
Accuracy AUC sensitivity Specificity
Figure 7. Model Evaluation.
4.4. Model Interpretation
According to the model comparison results, the best customer churn prediction model
was created from the random forest algorithm, which had a prediction accuracy of close to
90%. After that, to address with the black-box nature of machine learning algorithms, this
paper illustrates the results of customer churn prediction models through two explainable
AI tools, including LIME and SHAP.
4.4.1. Local Interpretable Model-Agnostic Explanations (LIME)
Figure 8shows the local explanations of the Random Forest classifier for the first
sample and the second sample using the LIME technique. According to Figure 8, the five
most significant factors in Random-Forest-based churn detection for the first customer are
the ‘Contract’, ‘Number of Dependents’, ‘Online Security’, ‘Premium Tech Support’, and
‘Number of Referrals’. Furthermore, an increase in the value of these five factors will drive
the prediction toward not churning, while an increase in the value of ‘Payment Method’
and ‘Monthly Charge’ will increase the likelihood of churning. For the second customer,
‘Monthly Charge’ was more important than ‘Premium Tech Support’ and was one of the
top five factors that influenced a customer’s subscription decision. In addition, contrary to
the first customer, LIME indicates that an increase in the value of ‘Contract’, ‘Number of
Dependents’, and ‘Number of Referrals’ values can increase the likelihood of the second
customer’s churn. See Figure 8.

Algorithms 2024,17, 231 16 of 21
Algorithms2024,17,xFORPEERREVIEW17of22


(a)

(b)
Figure8.Localexplanationfor(a)thefirstsample(upper)and(b)thesecondsample(lower)ofthe
RandomForestclassifier.
However,theLIMEexplanationonlypredictshowthepresentpredictionwillchange
inresponsetominutechangesintheinputvalues.Therefore,thisstudyalsoemploysan-
otherexplanationtechnique,SHAP,toshowtheoverallimportanceofthefactors.The
resultsareshowninthefollowingsubsection.
4.4.2.SHapleyAdditiveexPlanation(SHAP)
ThesummaryplotsfortherandomforestclassifiercomputedusingtheSHAPtech-
niqueareshowninFigure9.Intermsoftheoverallimportanceofthefeatures,‘Contract’,
‘NumberofReferrals’,‘TenureinMonths’,‘MonthlyCharge’,and‘OnlineSecurity’are
thefivefactorsthatmakethemostcontributiontothepredictionoutcomesoftheRandom
Forestclassifier.Tobespecific,highvaluesof‘Contract’,‘NumberofReferrals’,‘Tenure
inMonths’,and‘OnlineSecurity’increasethepossibilityofcustomerchurnpredictedby
theclassifiers,whilethelowvalueof‘MonthlyCharge’increasesthepossibility.
Figure 8. Local explanation for (a) the first sample (upper) and (b) the second sample (lower) of the
Random Forest classifier.
However, the LIME explanation only predicts how the present prediction will change
in response to minute changes in the input values. Therefore, this study also employs
another explanation technique, SHAP, to show the overall importance of the factors. The
results are shown in the following subsection.
4.4.2. SHapley Additive exPlanation (SHAP)
The summary plots for the random forest classifier computed using the SHAP tech-
nique are shown in Figure 9. In terms of the overall importance of the features, ‘Contract’,
‘Number of Referrals’, ‘Tenure in Months’, ‘Monthly Charge’, and ‘Online Security’ are the
five factors that make the most contribution to the prediction outcomes of the Random
Forest classifier. To be specific, high values of ‘Contract’, ‘Number of Referrals’, ‘Tenure in
Months’, and ‘Online Security’ increase the possibility of customer churn predicted by the
classifiers, while the low value of ‘Monthly Charge’ increases the possibility.

Algorithms 2024,17, 231 17 of 21
Algorithms2024,17,xFORPEERREVIEW18of22


Figure9.SummaryplotsforRandomForestclassifier.
Inaddition,toprovideaglobalexplanation,SHAPvaluescanalsobeusedtoexplore
thepredictionmadeforanindividualsample.Therefore,thispapercomputestheSHAP
explanationforthesametwosamples,asinvestigatedusingLIMEinSection4.4.1,and
conductsacomparison.
AccordingtoFigure10,themostsignificantfactorsforthefirstcustomersamplethat
pushthepredictiontowardchurningare‘PaymentMethod’,‘Age’,and‘MonthlyCharge’,
andtheonesthatpushthepredictiontowardnotchurningare‘Contract’,‘OnlineSecu-
rity’,‘City’,and‘NumberofReferrals’.Although‘Ag e ’isnotincludedinthetop10groups
ofLIME,LIMEandSHAPareabletocorroborateeachother’sjudgmentaboutthesignif-
icanceanddirectionofthefactorsforthefirstsample.Forthesecondcustomersample,as
showninFigure10,SHAPshowsthatthe‘NumberofReferrals’,‘Contract’,‘Numberof
Dependents’,and‘MonthlyCharge’arethemostsignificantfactorsthatpushthepredic-
tiontowardchurning,whichisconsistentwithLIMEexplanationaswell.

(a)

(b)
Figure10.SHAPexplanationfor(a)thefirstsample(upper)and(b)thesecondsample(lower)of
theRandomForestclassifier.
Figure 9. Summary plots for Random Forest classifier.
In addition, to provide a global explanation, SHAP values can also be used to explore
the prediction made for an individual sample. Therefore, this paper computes the SHAP
explanation for the same two samples, as investigated using LIME in Section 4.4.1, and
conducts a comparison.
According to Figure 10, the most significant factors for the first customer sample
that push the prediction toward churning are ‘Payment Method’, ‘Age’, and ‘Monthly
Charge’, and the ones that push the prediction toward not churning are ‘Contract’, ‘Online
Security’, ‘City’, and ‘Number of Referrals’. Although ‘Age’ is not included in the top 10
groups of LIME, LIME and SHAP are able to corroborate each other’s judgment about
the significance and direction of the factors for the first sample. For the second customer
sample, as shown in Figure 10, SHAP shows that the ‘Number of Referrals’, ‘Contract’,
‘Number of Dependents’, and ‘Monthly Charge’ are the most significant factors that push
the prediction toward churning, which is consistent with LIME explanation as well.
Algorithms2024,17,xFORPEERREVIEW18of22


Figure9.SummaryplotsforRandomForestclassifier.
Inaddition,toprovideaglobalexplanation,SHAPvaluescanalsobeusedtoexplore
thepredictionmadeforanindividualsample.Therefore,thispapercomputestheSHAP
explanationforthesametwosamples,asinvestigatedusingLIMEinSection4.4.1,and
conductsacomparison.
AccordingtoFigure10,themostsignificantfactorsforthefirstcustomersamplethat
pushthepredictiontowardchurningare‘PaymentMethod’,‘Age’,and‘MonthlyCharge’,
andtheonesthatpushthepredictiontowardnotchurningare‘Contract’,‘OnlineSecu-
rity’,‘City’,and‘NumberofReferrals’.Although‘Ag e ’isnotincludedinthetop10groups
ofLIME,LIMEandSHAPareabletocorroborateeachother’sjudgmentaboutthesignif-
icanceanddirectionofthefactorsforthefirstsample.Forthesecondcustomersample,as
showninFigure10,SHAPshowsthatthe‘NumberofReferrals’,‘Contract’,‘Numberof
Dependents’,and‘MonthlyCharge’arethemostsignificantfactorsthatpushthepredic-
tiontowardchurning,whichisconsistentwithLIMEexplanationaswell.

(a)

(b)
Figure10.SHAPexplanationfor(a)thefirstsample(upper)and(b)thesecondsample(lower)of
theRandomForestclassifier.
Figure 10. SHAP explanation for (a) the first sample (upper) and (b) the second sample (lower) of the
Random Forest classifier.

Algorithms 2024,17, 231 18 of 21
In summary, the results of the model explanations presented by the LIME and SHAP
techniques, respectively, are consistent. By showing decision-makers in the telecom indus-
try how customer-related factors affect customer retention through these interpretation
figures, decision-makers can understand how AI makes predictions in a way they under-
stand, even if they do not understand the complex algorithms of AI.
5. Discussion and Recommendations
5.1. Relevance to This Emerging Area and Its Contributions
Due to the black-box nature of machine learning algorithms, it may be difficult for
decision-makers in any industry to comprehend the advanced customer churn system
and may, therefore, lead to the systems being rejected. While aiming to address this issue,
this paper also introduces the concept of explainable ensemble learning, with the use
of ensemble-based machine learning and innovative ensemble learning, into business
analytics and illustrates the results of the best customer churn prediction model through
two visualization tools, including LIME and SHAP.
We implemented multiple classifiers including Logistic Regression, Random Forests,
Naïve Bayes, Decision Trees, and KNN to compare ensemble techniques like bagging and
boosting against individual methods [
49
]. The ensemble approach of combining diverse
models clearly provided superior predictive performance over any individual learner for
this key business analytics challenge.
Consequently, the developed predictive system is sufficient for distinguishing churn-
ers and non-churners and helps business developers in the telecommunication industry
to conduct a more efficient campaign for customer retention, which will help to reduce
marketing costs and churn rate effectively. The use of analytics and explainable ensemble
learning can deliver critical insights effectively to all customers, stakeholders, and employ-
ees. Therefore, it makes decision-making more streamlined and straightforward, and the
employees and other stakeholders can revise their work focus.
5.2. Implications Relevance to This Special Issue
This research aimed to develop a predictive model for customer turnover in the
telecommunications industry that could distinguish between customers who are likely
to churn in the near future and those who are loyal to the company. The value of such a
strategy to a firm is that it avoids wasting money on useless mass marketing approaches
and allows organizations to target true churners by removing customers who are likely
to churn. Furthermore, as explained in Section 1, obtaining a new customer costs eight
times as much as keeping an existing one; as a result, because the churn predictive model
can predict future churners, businesses seeking to keep their clientele can concentrate on
retention strategies rather than acquisition strategies, which are less expensive.
In addition, this study also improves the interpretability of customer churn predictive
models through the use of LIME and SHAP to optimize their local and global interpretabil-
ity. It provides decision-makers with an overall explanation of the factors affecting the
customer’s decision to churn, as well as a specific analysis for every single customer. As
a result, decision-makers can more easily understand the results of advanced predictive
models and use these explanations to develop global strategies and customize strategies
for specific customer segments.
The conclusions of this study have significant consequences for telecommunication
companies. Apart from the notion of establishing a predictive system for predicting
customer turnover in the telecommunication industry, the findings of this study may be
applied to the banking industry in terms of developing a churn prediction model for debit
card customers.
To generalize the previous discussion, we may advocate that companies employ
machine learning approaches to transform existing consumer information in their databases
into meaningful information that can assist them with their efforts to market based on the
research results. They would also benefit from using machine learning to create a projected

Algorithms 2024,17, 231 19 of 21
churn model, which could act as an alert system for organizations and help them to spend
their retention money effectively.
5.3. Limitations of the Study
There are limitations to this study. Some of the constraints were that we could not ac-
cess some types of customer data, such as billing and credit information, due to telecom data
categorization and confidentiality restrictions. This was a significant research constraint.
Another constraint in conducting this study was the lack of demographic information
on the clients. Therefore, it was not possible to include such criteria in the classification
process, which would have improved the classification accuracy and interpretability.
6. Conclusions and Recommendations
The telecommunication industry has been hit the hardest and at a high stake, with an
average annual churn rate of 30%, resulting in a tremendous waste of money and effort. It
is essential to understand why customers leave to minimize the damage to the bottom line
of the business.
Since churn is certain to result in lost revenue, and the cost of customer acquisition
is equivalent to 5–10 times the cost spent on customer retention or satisfaction, churn
forecasting has become a major task for companies to remain competitive. While aiming
to help companies reduce customer churn to survive or grow, customer churn forecasting
can be used in the long term to obtain continuously updated, real-time customer data.
Therefore, the principal aim of this study was to build a prediction system that can help to
identify customers who are likely to leave, specifically in the telecommunication industry.
Various data mining methods like Decision Trees, Random Forests, and Logistic Regression
were utilized by experts to build the predictive model. This study created a comparison
system, since it is a more interpretable and intelligible approach to visualizing several
machine learning algorithms. The performance of the models was evaluated using the
accuracy measures, area under the curve, and sensitivity and specificity measures. The
Random Forest classifier is the best classifier in this study, with an accuracy of around 90%,
and it also performed well in other evaluation metrics.
Moreover, this paper contributes to the field by introducing the concept of explainable
ensemble learning. By integrating various tools, such as LIME and SHAP, this study
provides decision-makers with clear, actionable insights into the factors driving customer
churn. This advancement is particularly significant in addressing the opaque nature of
machine learning algorithms, making advanced analytics accessible and applicable for
strategic decision-making in business settings.
Author Contributions: Conceptualization, V.C. and F.O.A.; methodology, V.C. and F.O.A.; software,
V.C., F.O.A. and M.A.G.; validation, V.C. and Q.A.X.; formal analysis, V.C. and K.H.; investigation,
V.C.; resources, V.C.; data curation, F.O.A. and M.A.G.; writing—original draft preparation, V.C. and
F.O.A.; writing—review and editing, V.C., K.H., Q.A.X. and V.B.; visualization, K.H., Q.A.X., F.O.A.
and M.A.G.; supervision, V.C.; project administration, V.C.; funding acquisition, V.C. All authors
have read and agreed to the published version of the manuscript.
Funding: This research is partly supported by VC Research (VCR 0000183) for Prof. Chang.
Data Availability Statement: The authors do not own the data.
Acknowledgments: Thank you to reviewers and editor for their reviews.
Conflicts of Interest: The authors declare no conflicts of interest.
References
1.
Eklof, J.; Podkorytova, O.; Malova, A. Linking customer satisfaction with financial performance: An empirical study of Scandina-
vian banks. Total Qual. Manag. Bus. Excell. 2020,31, 1684–1702. [CrossRef]
2.
Madhani, P.M. Enhancing Customer Value Creation with Market Culture: Developing 7Cs Framework. IUP J. Manag. Res. 2018,
17, 46–64.

Algorithms 2024,17, 231 20 of 21
3.
Chouiekh, A. Deep convolutional neural networks for customer churn prediction analysis. Int. J. Cogn. Inform. Nat. Intell. 2020,
14, 1–16. [CrossRef]
4.
Duval, A. Explainable Artificial Intelligence (XAI); MA4K9 Scholarly Report; Mathematics Institute, The University of Warwick:
Coventry, UK, 2019; pp. 1–53.
5.
Reilly, J. The Machine Learning Revolution: Telco Customer Churn Prediction. Technical Paper. 2023. Available online:
https://www.akkio.com/post/telecom-customer-churn (accessed on 13 April 2024).
6.
Yulianti, Y.; Saifudin, A. Sequential feature selection in customer churn prediction based on Naive Bayes. IOP Conf. Ser. Mater. Sci.
Eng. 2020,879, 012090. [CrossRef]
7.
Leung, C.K.; Pazdor, A.G.; Souza, J. Explainable artificial intelligence for data science on customer churn. In Proceedings of the
2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), Porto, Portugal, 6–9 October 2021;
pp. 1–10.
8.
Mishra, A.; Reddy, U.S. A comparative study of customer churn prediction in telecom industry using ensemble based classifiers.
In Proceedings of the 2017 International Conference on Inventive Computing and Informatics (ICICI), Coimbatore, India, 23–24
November 2017; pp. 721–725.
9.
De Caigny, A.; Coussement, K.; De Bock, K.W. A new hybrid classification algorithm for customer churn prediction based on
logistic regression and decision trees. Eur. J. Oper. Res. 2018,269, 760–772. [CrossRef]
10.
Lalwani, P.; Mishra, M.K.; Chadha, J.S.; Sethi, P. Customer churn prediction system: A machine learning approach. Computing
2022,104, 271–294. [CrossRef]
11.
Yu, R.; An, X.; Jin, B.; Shi, J.; Move, O.A.; Liu, Y. Particle classification optimization-based BP network for telecommunication
customer churn prediction. Neural Comput. Appl. 2018,29, 707–720. [CrossRef]
12.
Fujo, S.W.; Subramanian, S.; Khder, M.A. Customer churn prediction in telecommunication industry using deep learning. Inf. Sci.
Lett. 2022,11, 24.
13.
Kampani, N.; Jhamb, D. Analyzing the role of e-crm in managing customer relations: A critical review of the literature. J. Crit.
Rev. 2020,7, 221–226.
14. Nelson, C.A.; Walsh, M.F.; Cui, A.P. The role of analytical CRM on salesperson use of competitive intelligence. J. Bus. Ind. Mark.
2020,35, 2127–2137. [CrossRef]
15.
Al-Homery, H.; Asharai, H.; Ahmad, A. The core components and types of CRM. Pak. J. Human. Soc. Sci. 2019,7, 121–145.
[CrossRef]
16.
Calzada-Infante, L.; Óskarsdóttir, M.; Baesens, B. Evaluation of customer behavior with temporal centrality metrics for churn
prediction of prepaid contracts. Expert Syst. Appl. 2020,160, 113553. [CrossRef]
17.
Hendriyani, C.; Auliana, L. Transformation from relationship marketing to electronic customer relationship management: A
literature study. Rev. Integr. Bus. Econ. Res. 2018,7, 116–124.
18.
Chagas, B.N.R.; Viana, J.A.N.; Reinhold, O.; Lobato, F.; Jacob, A.F.; Alt, R. Current applications of machine learning techniques in
CRM: A literature review and practical implications. In Proceedings of the 2018 IEEE/WIC/ACM International Conference on
Web Intelligence (WI), Santiago, Chile, 3–6 December 2018; pp. 452–458.
19.
Eachempati, P.; Srivastava, P.R.; Kumar, A.; de Prat, J.M.; Delen, D. Can customer sentiment impact firm value? An integrated
text mining approach. Technol. Forecast. Soc. Chang. 2022,174, 121265. [CrossRef]
20.
Lamrhari, S.; El Ghazi, H.; Oubrich, M.; El Faker, A. A social CRM analytic framework for improving customer retention,
acquisition, and conversion. Technol. Forecast. Soc. Chang. 2022,174, 121275. [CrossRef]
21.
Mahmoud, M.A.; Hinson, R.E.; Adika, M.K. The effect of trust, commitment, and conflict handling on customer retention: The
mediating role of customer satisfaction. J. Relat. Mark. 2018,17, 257–276. [CrossRef]
22.
De Caigny, A.; Coussement, K.; De Bock, K.W.; Lessmann, S. Incorporating textual information in customer churn prediction
models based on a convolutional neural network. Int. J. Forecast. 2020,36, 1563–1578. [CrossRef]
23.
De Caigny, A.; Coussement, K.; Verbeke, W.; Idbenjra, K.; Phan, M. Uplift modeling and its implications for B2B customer churn
prediction: A segmentation-based modeling approach. Ind. Mark. Manag. 2021,99, 28–39. [CrossRef]
24. Seymen, O.F.; Dogan, O.; Hiziroglu, A. Customer Churn Prediction Using Deep Learning. In Proceedings of the 12th International
Conference on Soft Computing and Pattern Recognition (SoCPaR 2020). SoCPaR 2020; Advances in Intelligent Systems and Computing;
Springer: Cham, Switzerland, 2021; Volume 1383. [CrossRef]
25.
Guo, W.W.; Xue, H. Crop yield forecasting using artificial neural networks: A comparison between spatial and temporal models.
Math. Probl. Eng. 2014,2014, 857865. [CrossRef]
26.
Shahi, T.B.; Shrestha, A.; Neupane, A.; Guo, W. Stock price forecasting with deep learning: A comparative study. Mathematics
2020,8, 1441. [CrossRef]
27.
Aldunate, Á.; Maldonado, S.; Vairetti, C.; Armelini, G. Understanding customer satisfaction via deep learning and natural
language processing. Expert Syst. Appl. 2022,209, 118309. [CrossRef]
28.
Raza, K. Improving the prediction accuracy of heart disease with ensemble learning and majority voting rule. In U-Healthcare
Monitoring Systems; Academic Press: Devon, UK, 2019; pp. 179–196.
29.
Ahmad, I.; Yousaf, M.; Yousaf, S.; Ahmad, M.O. Fake news detection using machine learning ensemble methods. Complexity 2020,
2020, 8885861. [CrossRef]

Algorithms 2024,17, 231 21 of 21
30.
Sahoo, A.K.; Pradhan, C.; Das, H. Performance evaluation of different machine learning methods and deep-learning based
convolutional neural network for health decision making. Nat. Inspired Comput. Data Sci. 2020,871, 201–212.
31.
Wei, C.P.; Chiu, I.T. Turning telecommunications call details to churn prediction: A data mining approach. Expert Syst. Appl. 2002,
23, 103–112. [CrossRef]
32.
Anees, R.T.; Nordin, N.A.; Anjum, T.; Cavaliere, L.P.L.; Heidler, P. Evaluating the Impact of Customer Relationship Management
(CRM) Strategies on Customer Retention. Bus. Manag. Strateg. 2020,11, 117–133. [CrossRef]
33.
Shrestha, Y.R.; Ben-Menahem, S.M.; Von Krogh, G. Organizational decision-making structures in the age of artificial intelligence.
Calif. Manag. Rev. 2019,61, 66–83. [CrossRef]
34.
Buchanan, B.G.; Wright, D. The impact of machine learning on UK financial services. Oxf. Rev. Econ. Pol. 2021,37, 537–563.
[CrossRef] [PubMed]
35.
Fernández-Rovira, C.; Valdés, J.Á.; Molleví, G.; Nicolas-Sans, R. The digital transformation of business. Towards the datafication
of the relationship with customers. Technol. Forecast. Soc. Chang. 2021,162, 120339. [CrossRef]
36.
Bauer, K.; von Zahn, M.; Hinz, O. Expl(AI)ned: The impact of explainable artificial intelligence on users’ information processing.
Inf. Syst. Res. 2023,34, 1582–1602. [CrossRef]
37.
Teng, Y.; Zhang, J.; Sun, T. Data-driven decision-making model based on artificial intelligence in higher education system of
colleges and universities. Expert Syst. 2023,40, e12820. [CrossRef]
38.
Djeffal, C. The Normative Potential of the European Rule on Automated Decisions: A New Reading for Art. 22 GDPR. Z.
Ausländisches Öffentl. Recht Völkerr. 2020,81, 847–879.
39.
Nkolele, R.; Wang, H. Explainable Machine Learning: A Manuscript on the Customer Churn in the Telecommunications Industry.
In Proceedings of the 2021 Ethics and Explainability for Responsible Data Science (EE-RDS), Johannesburg, South Africa, 27–28
October 2021; pp. 1–7.
40.
Shapley, L.S. A value for n-person games. In Contributions to the Theory of Games; Princeton University Press: Princeton, NJ, USA,
1953; Volum 2, pp. 307–317.
41.
Štrumbelj, E.; Kononenko, I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst.
2014,41, 647–665. [CrossRef]
42.
Gramegna, A.; Giudici, P. SHAP and LIME: An evaluation of discriminative power in credit risk. Front. Artif. Intell. 2021,
4, 752558. [CrossRef] [PubMed]
43.
Kwon, O.; Lee, N.; Shin, B. Data quality management, data usage experience and acquisition intention of big data analytics. Int. J.
Inf. Manag. 2014,34, 387–394. [CrossRef]
44.
Provost, F.; Fawcett, T. Data science and its relationship to big data and data-driven decision making. Big Data 2013,1, 51–59.
[CrossRef] [PubMed]
45.
Martínez-Plumed, F.; Contreras-Ochando, L.; Ferri, C.; Hernández-Orallo, J.; Kull, M.; Lachiche, N.; Ramírez-Quintana, M.J.;
Flach, P. CRISP-DM twenty years later: From data mining processes to data science trajectories. IEEE Trans. Knowl. Data Eng.
2019,33, 3048–3061. [CrossRef]
46.
Hakkoum, H.; Idri, A.; Abnane, I. Artificial neural networks interpretation using LIME for breast cancer diagnosis. In Trends and
Innovations in Information Systems and Technologies; Springer: Cham, Switzerland, 2020; pp. 15–24.
47.
Hutter, F.; Xu, L.; Hoos, H.H.; Leyton-Brown, K. Algorithm runtime prediction: Methods & evaluation. Artif. Intell. 2014,206,
79–111.
48.
Antwarg, L.; Miller, R.M.; Shapira, B.; Rokach, L. Explaining anomalies detected by autoencoders using Shapley Additive
Explanations. Expert Syst. Appl. 2021,186, 115736. [CrossRef]
49.
Lessmann, S.; Baesens, B.; Seow, H.V.; Thomas, L.C. Benchmarking state-of-the-art classification algorithms for credit scoring: An
update of research. Eur. J. Oper. Res. 2015,247, 124–136. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.