Application of irregular and unbalanced data to predict diabetic nephropathy using visualization and feature selection methods.

Abstract

Diabetic nephropathy is damage to the kidney caused by diabetes mellitus. It is a common complication and a leading cause of death in people with diabetes. However, the decline in kidney function varies considerably between patients and the determinants of diabetic nephropathy have not been clearly identified. Therefore, it is very difficult to predict the onset of diabetic nephropathy accurately with simple statistical approaches such as t-test or chi(2)-test. To accurately predict the onset of diabetic nephropathy, we applied various machine learning techniques to irregular and unbalanced diabetes dataset, such as support vector machine (SVM) classification and feature selection methods. Visualization of the risk factors was another important objective to give physicians intuitive information on each patient's clinical pattern.
We collected medical data from 292 patients with diabetes and performed preprocessing to extract 184 features from the irregular data. To predict the onset of diabetic nephropathy, we compared several classification methods such as logistic regression, SVM, and SVM with a cost sensitive learning method. We also applied several feature selection methods to remove redundant features and improve the classification performance. For risk factor analysis with SVM classifiers, we have developed a new visualization system which uses a nomogram approach.
Linear SVM classifiers combined with wrapper or embedded feature selection methods showed the best results. Among the 184 features, the classifiers selected the same 39 features and gave 0.969 of the area under the curve by receiver operating characteristics analysis. The visualization tool was able to present the effect of each feature on the decision via graphical output.
Our proposed method can predict the onset of diabetic nephropathy about 2-3 months before the actual diagnosis with high prediction performance from an irregular and unbalanced dataset, which statistical methods such as t-test and logistic regression could not achieve. Additionally, the visualization system provides physicians with intuitive information for risk factor analysis. Therefore, physicians can benefit from the automatic early warning of each patient and visualize risk factors, which facilitate planning of effective and proper treatment strategies.

Full text

Page 1

Application of irregular and unbalanced data to
predict diabetic nephropathy using visualization
and feature selection methods
Baek Hwan Choa, Hwanjo Yub, Kwang-Won Kimc,
Tae Hyun Kimc, In Young Kima,*, Sun I. Kima
aDepartment of Biomedical Engineering, Hanyang University, Seoul, Republic of Korea
bDepartment of Computer Science, University of Iowa, Iowa City, IA, USA
cDivision of Endocrinology and Metabolism, Department of Medicine,
Samsung Medical Center, Sungkyunkwan University School of Medicine,
Seoul, Republic of Korea
Received 6 April 2007; received in revised form 21 September 2007; accepted 21 September 2007
Artificial Intelligence in Medicine (2008) 42, 37—53
http://www.intl.elsevierhealth.com/journals/aiim
KEYWORDS
Decision support
systems;
Diabetic nephropathy;
Support vector
machines;
Visualization;
Risk factor analysis;
Feature selection
Summary
Objective: Diabeticnephropathy isdamage tothe kidney caused bydiabetesmellitus.
It is a common complication and a leading cause of death in people with diabetes.
However, the decline in kidney function varies considerably between patients and the
determinantsofdiabeticnephropathy havenot beenclearly identified. Therefore,it is
very difficult to predict the onset of diabetic nephropathy accurately with simple
statistical approaches such as t-test or x2-test. To accurately predict the onset of
diabetic nephropathy, weapplied various machine learning techniques toirregular and
unbalanced diabetes dataset, such as support vector machine (SVM) classification and
feature selection methods. Visualization of the risk factors was another important
objective to give physicians intuitive information on each patient’s clinical pattern.
Methods and materials: We collected medical data from 292 patients with diabetes
and performed preprocessing to extract 184 features from the irregular data. To
predicttheonsetofdiabeticnephropathy, wecomparedseveral classificationmethods
suchaslogisticregression,SVM,andSVMwithacostsensitivelearningmethod.Wealso
applied several feature selection methods to remove redundant features and improve
the classification performance. For risk factor analysis with SVM classifiers, we have
developed a new visualization system which uses a nomogram approach.
Results: Linear SVM classifiers combined with wrapper or embedded feature selection
methods showed the best results. Among the 184 features, the classifiers selected the
* Corresponding author at: Sungdong P.O. Box 55, Seoul 133-605, Republic of Korea. Tel.: +82 2 2291 1713; fax: +82 2 2296 5943.
E-mail address: iykim@hanyang.ac.kr (I.Y. Kim).
0933-3657/$ — see front matter # 2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.artmed.2007.09.005

Page 2

1. Introduction
Diabetes mellitus is a metabolic disorder character-
ized by chronic hyperglycemia (high blood sugar
level) resulting from defects in insulin secretion,
insulin action, or both [1]. A medical insurance
cohort study that included 1.2 million subjects indi-
cated that diabetes mellitus, in Korea, is the first
leading cause of burden of disease, and 8.4% of the
population suffers from this disease [2]. Diabetes
can cause devastating complications including car-
diovascular diseases, kidney failure, leg and foot
amputations, and blindness, which often result in
disability and death. Diabetic nephropathy is
damage to the kidney because of diabetes; it is a
common diabetic complication and a leading cause
of death in people with diabetes [3].
Many researchers have been trying to determine
risk factors of mortality in patients with diabetes via
statistical approaches. There are also many studies
onpredictorsofdiabeticnephropathy.Thelong-term
occurrenceover theyearsofhigh bloodglucoselevel
and high blood pressure is highly indicative of the
development of renal disease, other microvascular
lesions, and macrovascular disease [4,5]. Clinical
trials have demonstrated consistently that suppres-
sion of the glycosylated hemoglobin (HbA1C) level is
associated with decreased risk for clinical and struc-
tural manifestations of diabetic nephropathy in
patients with types 1 and 2 diabetes [6,7]. Torffvit
and Agardh showed that poor metabolic control and
high blood pressure is associated with development
of diabetic nephropathy in patients with type 2 dia-
betes [8]. A logistic regression analysis was adopted
to generate prediction rules for identifying patients
with diabetes at high risk of complications and for
analyzing risk factors [9]. However, most of those
prognostic studies compared mean values of inde-
pendent predictors between patients with diabetic
nephropathy and control groups, using simple statis-
tical methods such as Student’s t-test and the non-
parametric Mann—Whitney U-test. The decline
in kidney function varies considerably between
patients, and determinants of the diabetic nephro-
pathy have not been identified clearly. Therefore, it
is difficult to predict diabetic nephropathy accu-
rately using simple statistical approaches.
A number of studies have taken advantage of data
mining techniques in the diabetes domain. A 1998
review provided evidence of the use of decision
support systems to guide physicians through the clin-
ical consultation [10]. The paper used time series
methods and a causal probabilistic network, and
proposedtelemedicineandtelecareforpatientswith
diabetes.Othersintroducedvarioustypesofartificial
neural networks (ANNs) or decision trees to predict
the onset of diabetes and to identify risk factors
among the data [11—14]. However, most of those
papers attempted to predict the onset of diabetes
itself, even though its complications are much more
important for the quality of life and mortality.
Although many have tried to approach health
problems through data mining techniques, there
are many constraints and difficulties in this [15].
The main difficulty arises in data collection.
Because hospitals have not used electronic medical
record systems for long, it is very hard to collect a
dataset large enough for such research. Moreover,
in the university hospital setting, physicians and
medical practitioners have occasionally trans-
ferred to other medical institutions, which might
cause different patient care protocols, including
physical examinations and interviews formats.
Furthermore, because of over confidence in their
health condition or for other personal reasons,
some outpatients visit the hospital irregularly.
These circumstances may lead to very irregular,
incomplete, or missing data in the clinical setting,
and thus make it difficult to extract meaningful
information from the data.
Other difficult problems in medical data mining
lie in the artificial intelligence algorithm itself. To
diagnose whether patients have a disease, or to
predict if they would have a disease in the future,
several studies have attempted to apply learning
algorithmssuchas decision trees, ANNs, and support
vector machines (SVMs). Although they give high
generalization performances, the users can barely
38B.H. Cho et al.
same 39 features and gave 0.969 of the area under the curve by receiver operating
characteristics analysis. The visualization tool was able to present the effect of each
feature on the decision via graphical output.
Conclusions: Our proposed method can predict the onset of diabetic nephropathy
about 2—3 months before the actual diagnosis with high prediction performance from
an irregular and unbalanced dataset, which statistical methods such as t-test and
logistic regression could not achieve. Additionally, the visualization system provides
physicians with intuitive information for risk factor analysis. Therefore, physicians can
benefit from the automatic early warning of each patient and visualize risk factors,
which facilitate planning of effective and proper treatment strategies.
# 2007 Elsevier B.V. All rights reserved.

Page 3

understand the results by means of input variables,
especially with ANNs and SVMs; that is why they are
termed ‘‘black box’’ algorithms. Consequently,
despite its relatively poor generalization perfor-
mance, physicians and medical practitioners still
make use of the logistic regression (LR) as a gold
standard because it gives more information about
the results: it provides not only the percentage of
variance in the output variable (i.e., probabilistic
output), but also the odds ratio of each input vari-
able.
In information retrieval, there are various fea-
ture (i.e., variable) selection methods to rank the
importance of input features and thereby enhance
generalization performance by eliminating the dis-
turbing features. Most of those methods are mainly
concerned with the ranking of input features, rather
than about the insights of each feature. However, in
a practical clinical situation, physicians may wish to
understand the actual effect of each feature on the
results: an interpretation about how the prediction
result would change if a feature’s value were to
change. Jakulin et al. introduced a nomogram
approach for visualizing SVMs that can graphically
expose its internal structure and visualize the effect
of each feature by means of the log odds ratio, as
with LR [16].
In this study, we aimed to predict the onset of
diabetic nephropathy by learning SVM predictive
model from an irregular and unbalanced diabetic
dataset. We also attempted to identify some risk
factorsfromclinicalparametersusingfeatureselec-
tion and nomogram visualization.
2. Classification methods
2.1. Logistic regression and the ridge
estimator
LR is a popular method to generate a predictive
model for dichotomous target variable. The rela-
tionship between the input variables and the
response is not a linear function, but the logistic
regression function:
1
1 þ e?ðaþb1x1þb2x2þ???þbdxdÞ
where P is the probability that an event happens, a
is the coefficient of the equation and bi is the
coefficient of the input variable. The log-likelihood
function of the data is
X
The conventional LR optimizer finds the LR coeffi-
cients by maximizing L(b) using the maximum like-
PðxÞ ¼
(1)
LðbÞ ¼
i
ðyilogPðxiÞ þ ð1 ? yiÞlogð1 ? PðxiÞÞÞ (2)
lihood
parameter estimates may arise when the number
of input variables is large or the input variables are
highly correlated. Cessie and Houwelingen intro-
duced ridge estimators to resolve this case [17].
They included a restriction term in the log-likeli-
hood function thus
LlðbÞ ¼ LðbÞ ? ljjbjj2
where L(b) is the unrestricted log-likelihood func-
tion and jjbjj is the norm of the parameter vector b.
Theridgeparameter lshrinksthenormofb,andthe
restriction term stabilizes the system to provide
estimates with smaller variance.
estimate method.However,unstable
(3)
2.2. Cost sensitive learning in SVMs
SVMs, an emerging classification technique, have
been intensively benchmarked against a variety of
techniques; it is one of the best-known classifica-
tion techniques with computational advantages
and good generalization performance. The main
idea of SVMs is to maximize the margin, which is
defined as the distance from the separating hyper-
plane to the closest training samples (support vec-
tors) [18].
However, for an unbalanced dataset that has far
more positives than negatives or vice versa, the
general classifiers may produce poor generalization
performance as the hyperplane may be moved far
away to the minority training samples. For this
reason,Veropoulosetal.usedacost-sensitivelearn-
ing approach for SVM; their key point is to give
different cost (penalty) to the errors of each class
[19]. Accordingly, the optimization strategy of SVM
is as follows:
1
2jjwjj2þ Cþ
fijyi¼þ1g
minimize
X
n
jiþ C?
X
n
fijyi¼?1g
ji
(4)
subjecttoyiðw ? xiþ bÞ?1 ? ji;
ji?0;
i ¼ 1;...;n
i ¼ 1;...;n
(5)
where w is a weight vector of the separating
hyperplane, jiis a slack variable that allow the
margin constraints to be violated and C is a user
parameter to be tuned. The first term of the
objective function is about the margin maximiza-
tion, and the second and third part is for control-
ling the penalties for positive and negative
training samples. Geometrically, when positive
data are in the minority, it gives more weight to
the positive support vectors (C+is larger than C?),
and pushes the hyperplane towards where the
negative (majority) data exist.
Prediction of diabetic nephropathy39

Page 4

3. Feature selection methods
Feature selection is a machine learning process that
selects a feature subset from the whole feature set
and removes redundant features that do not con-
tribute to the performance. Feature selection
methods have been introduced to avoid the ‘‘curse
of dimensionality’’, which means the required num-
ber of calculations becomes huge as the number of
dimensions increases, while retaining or even
enhancing the performance.
There are three main approaches in feature
selection: filter, wrapper, and embedded methods
[20]. Filter methods select high ranked features
based on a statistical score as a preprocessing step;
ReliefF is a popular filter algorithm in microarray
classification problems because of its simplicity.
Wrapper performs selection taking into account
the classifier as a black box and ranking the subset
of features by their predictive power. Because a full
search requires 2ndifferent evaluations, forward
selection or backward elimination methods are
used. Sensitivity analysis could be adopted to cal-
culate the importance of each feature with any
classifier. Embedded methods, in contrast to wrap-
per approaches, select features considering the
classifier design at the same time.
3.1. ReliefF algorithm
A key idea in ReliefF is to evaluate the contribution
of each feature to inter-class difference and intra-
class similarity [21]. With a randomly selected data,
the algorithm looks for the knearest hits (those with
the same class label) and misses (those with a
different class label). After that, it updates the
quality of the contribution of features with respect
to the difference between the feature values of the
selected data and nearest ones. The pseudocode is
shown in Table 1. Function diff(f, Ii, Ij) calculates
the difference between the feature values of two
instances. Thus, the weight vector W[f] increases
when the feature value of the selected instance is
different from that of the nearest miss Mj(C). On the
other hand, it decreases when there is a difference
between the feature values of the selected instance
and the nearest hit Hj. Finally, according to the
weight values of W[f], one can identify the ranking
of each feature and perform feature selection by
eliminating features with smallest weight values.
3.2. Sensitivity analysis with SVM
Sensitivity analysis is another method that has been
widely used to rank input features in terms of their
contribution to the deviation of the outputs [22]. It
involves varying every input feature over a reason-
able range with the others fixed, and observing the
relative changes in the outputs. As a result, features
that produce larger deviation in the output are con-
sidered more important and one can select features
by the same method that is adopted in the ReliefF
method above. Table 2 shows the pseudocode of the
sensitivityanalysismethod.Thealgorithmcalculates
thedifferencebetweenmaximumandminimumout-
put of the predictive model when a feature value
varies from its possible minimum to maximum with
the other features fixed to their mean values.
40B.H. Cho et al.
Table 1
Algorithm of the ReliefF method

Page 5

3.3. Recursive feature elimination with
SVM (SVM—RFE)
SVM—RFE is an example of the embedded method
and a similar approach that removes less important
features recursively, except for using the weight
magnitude as a ranking criterion [23]. The outline
of the algorithm is presented in Table 3. In the
iterative training process in SVM—RFE, one can find
the best subset of features that provides the highest
performance. However, this algorithm is limited
theoretically to the linear kernel in SVM, because
it is difficult to calculate the weight vector for a
non-linear kernel because of the kernel character-
istics of the implicit mapping. The authors of the
paper[23] mentioned
version of SVM—RFE that is computationally more
expensive.
the non-linearkernel
4. Risk factor analysis with nomogram
visualization of SVM
4.1. General concept of the nomogram in
predictive models
The nomogram gives the insight of a model by
visualizing the effect of each feature on the pre-
diction. Fig. 1 shows an imaginary example of the
nomogram visualizing a predictive model. Suppose
that one is trying to predict whether a patient will
have diabetic nephropathy by considering two fea-
tures. One is whether the patient smokes or not; the
other is systolic blood pressure level. In the figure,
log odds ratio (Log OR) scores of the individual
features are summed up using the topmost axis of
the nomogram and used to estimate the probability
of having diabetic nephropathy (the bottommost
Prediction of diabetic nephropathy41
Table 2
Algorithm of the sensitivity analysis method
Table 3
Algorithm of the SVM—RFE method