Page 1

VOL. 10 NO. 6 JUNE 2008thE JOUrNaL Of CLiNiCaL hypErtENsiON

485

t

the divergent views have been most clearly laid

out in the case of C-reactive protein (Crp) and its

incremental value over conventional cardiovascular

risk factors. advocates of Crp argue that its addi-

tion improves global measures of fit,1 while others

point out that it does not improve measures of dis-

crimination.2,3 Discussions surrounding this debate

have largely been technical, focusing on unique

data sources and specific analytic approaches.

the underlying principles, in contrast, have not

been sufficiently highlighted. this paper presents a

clinical perspective on the potential value of adding

new cardiovascular risk factors to current methods

for risk stratification. it proposes that novel risk

factors should be evaluated by their effects on the

population risk distribution curve, which presents

the probability of occurrence of different levels of

risk in a population as determined by a predic-

tive model. Broader population risk distribution

curves represent superior risk stratification. Last,

it should be recognized that there are many correct

ways to risk-stratify a population and that the best

approach ultimately depends on the clinical goals.

here is an active debate on the best approach

to evaluate new cardiovascular risk factors.

THE USE OF RISk MODELS IS TO STRATIFY RISk

the goal of risk stratification is to identify sub-

populations of differing cardiovascular risk within

a larger general population to allow interventions

to be employed selectively. Mathematic risk models

must be applied to populations to determine their

clinical utility. New cardiovascular risk factors

that do not improve risk stratification within these

models are not of clinical value, although they may

increase scientific understanding or reveal new

opportunities to develop preventive measures.

applying a predictive model to an entire popula-

tion generates a population risk distribution curve.

these are rarely presented in the cardiovascular lit-

erature but have important implications for assess-

ing the value of risk factors. We have been unable to

locate a graph depicting the population distribution

of cardiovascular risk determined by a framingham

risk score, despite the widespread use of this risk

score in the literature. an example of this is depicted

in figure 1. this population risk distribution curve

shows how frequently patients with different lev-

els of cardiovascular risk occur in the population.

as expected, the higher their cardiovascular risk,

the less often these patients are encountered. it is

uncommon to find healthy patients with a 10-year

risk >20%, the range expected for patients with cor-

onary artery disease. Graphs of this type represent

a valuable way to evaluate the clinical importance

of risk factors. Neither global measures of fit nor

the receiver operating characteristic (rOC) curve so

directly communicate the results of the risk stratifi-

cation process.

BROADER RISk DISTRIBUTION

CURVES REPRESENT SUPERIOR RISk

STRATIFICATION

absent any information about risk factors, all

members of the population would be assigned the

same risk. the risk distribution “curve” would be

a spike at the mean population risk. as risk fac-

tors are added to a mathematic risk model, the

C o m m e n t a r y

Evaluating New Cardiovascular Risk

Factors for Risk Stratification

Ralph H. Stern, PhD, MD

From the Department of Internal Medicine, Division of

Cardiovascular Medicine, University of Michigan,

Ann Arbor, MI

Address for correspondence:

Ralph H . Stern, PhD, MD, CVC Cardiovascular

Medicine, 1500 E Medical Center Drive SPC5853,

Ann Arbor, MI 48109-5853

E-mail: stern@umich .edu

www.lejacq.com ID: 7814

Page 2

thE JOUrNaL Of CLiNiCaL hypErtENsiONVOL. 10 NO. 6 JUNE 2008

486

risk distribution curve should progressively widen,

so long as the risk factors contribute additional

information. figure 2 provides a hypothetical

example of risk distribution curves of increasing

width, demonstrating improved risk stratification.

the increasing width shows an increasing ability to

separate the population into subgroups of higher

and lower risk.

if two mathematic risk models produce identical

risk distribution curves when applied to the same

population, then the risk stratification of the two

models is equivalent. real-world data are unlikely

to generate mathematically identical risk distri-

bution curves; however, similar risk distribution

curves should be considered clinically equivalent,

even if there are mathematic differences.

RISk DISTRIBUTION CURVES AND

ROC CURVES CONTAIN IDENTICAL

INFORMATION

the rOC curve, which depicts the relationship

between sensitivity and specificity, has been an

important tool for assessing diagnostic methods

and predictive models.4 Diamond5 showed that

the rOC curve for a risk model can be calculated

from the population risk distribution. since the

rOC curve can be derived from the risk distri-

bution curve (and vice versa), these curves con-

tain identical information. this means that the

mathematic analyses of the rOC curve in the risk

factor research literature can be better understood

simply by evaluating the corresponding risk dis-

tribution curve.

Because the rOC curve and the risk distribution

curve contain identical information, properties of

one curve must correspond to properties of the

other. the area under the rOC curve (the rOC

curve aUC), used to compare both diagnostic

methods and predictive models, is an example.

although the rOC curve aUC is commonly under-

stood to measure the overlap of the risk distribu-

tion curves of patients with and without events,

also known as discrimination, it is equally valid to

consider the rOC curve aUC as a measure of the

width of the population risk distribution curve. for

example, the 3 population risk distribution curves

with increasing width in figure 2 have rOC curve

aUCs of 0.55, 0.65, and 0.75. Cook6 provides a

similar example using the b distribution.

THERE ARE MANY “CORRECT” WAYS TO

RISk-STRATIFY A POPULATION

for diagnostic methods, there is only one way to

be correct: perfectly discriminate between patients

with and without a disease. this is the definition

of a gold standard or perfect test. for a predictive

model to match this performance, it would need

to identify patients who will or will not have an

event in the future. Unfortunately, this is not pos-

sible because the occurrence of events is random

or stochastic.

an alternative and more realistic criterion

is that a predictive model is correct if it accu-

rately assigns risk to different subpopulations.

this property is referred to as calibration. it is

commonly evaluated by comparing the observed

to predicted risk for each decile of risk.7 for a

given population, there will be a multitude of

risk stratification methods, all of which are cor-

rect or calibrated. however, because they utilize

different risk factors, these methods will assign

different risks to the same individual. Consider

risk-stratifying the same population twice, once

by systolic pressure and once by low-density

lipoprotein cholesterol. assume the correspond-

ing cardiovascular risk for each decile was the

same for the 2 risk factors. then the risk strati-

fication of the population by the 2 risk factors is

identical. But patients with low blood pressure

may have high low-density lipoprotein cholester-

ol, and vice versa, so individuals may be assigned

very different risks depending on which risk fac-

tor is used to risk stratify the population.

Figure 1 . Approximation to the frequency distribution

of 10-year cardiovascular risk in US adults without

cardiovascular disease or cardiovascular disease risk

equivalents . This distribution (an exponential distribu-

tion with a lambda of 17) assigns patients to low, inter-

mediate, and high risk in the same proportions reported

in the literature .12

Page 3

VOL. 10 NO. 6 JUNE 2008thE JOUrNaL Of CLiNiCaL hypErtENsiON

487

DISCUSSION

Because the goal of risk stratification is to identify

subpopulations of differing cardiovascular risk, the

population risk distribution curve is the most useful

way to assess a predictive method’s risk stratifica-

tion in a given population. Neither global measures

of fit nor rOC curves show this directly.

framingham risk models are currently able

to identify adult patients without cardiovascular

disease with 10-year risks ranging from <1% to

>30%. figure 1 (an approximation) illustrates the

corresponding population risk distribution.

it is unknown whether this risk stratification

can be improved. Newer risk factors8 or even

measures of carotid atherosclerosis combined with

Crp9 have not improved risk stratification using

conventional risk factors. Diamond5 showed that

risk models have limited discriminatory ability.

as discussed above, perfect discrimination would

mean that a predictive method could identify

patients who will or will not have an event in the

future. But making accurate predictions over a

10-year period is only possible in deterministic sys-

tems (eg, prediction of eclipses in the solar system).

the occurrence of clinical events in low-risk indi-

viduals is often thought to indicate that improved

predictive methods are possible, but as long as the

predicted number of events match the observed

number of events in a low-risk subpopulation, this

is not the case.

in comparing risk models, one should recognize

that the predicted risk for individual patients can

vary between calibrated models. an example of

this for univariate risk models using systolic blood

pressure and low-density lipoprotein cholesterol

was given previously. if additional risk factors were

added to such univariate models, patients with

high levels of the additional risk factor would be

assigned higher risks, while those with low levels of

the additional risk factor would be assigned lower

risks. this would occur throughout the risk dis-

tribution curve, ensuring extensive scrambling or

reassortment of individual patients with the addi-

tion of each new risk factor. this would also occur

even if the population risk distribution curve is not

changed by the additional risk factors. ridker and

colleagues10 described 2 calibrated multivariate

models that differ in the number of risk factors.

in spite of having the same discrimination, these

models classified many patients into different risk

strata. in this situation, the risk distribution curves

should be the same, and thus the number of patients

moved out of a risk stratum will be balanced by the

number of patients moved into that risk stratum

when additional risk factors are added.

reynolds and associates11 compared 3 cardio-

vascular risk models that predict similar numbers

of patients above the same level of risk with a

simulated population and showed limited concor-

dance in the individuals identified. Lack of concor-

dance when multiple cardiovascular risk models

are used to risk stratify the same population has

not been extensively studied. however, it should

be expected that there will be a large number of

cardiovascular risk models based on different risk

factors that are calibrated and produce similar

population risk distribution curves, but these mod-

els may classify individuals differently.

this should be contrasted with the situation

with diagnostic testing. in that case, there is only

one correct answer for an individual, “intermedi-

ate” probability indicates a poor assessment, and

further testing is indicated. But once a population

has been risk-stratified by a calibrated model,

further testing may be counterproductive, even

for “intermediate-risk” patients. further testing

may simply replace one reasonable population risk

stratification with another equivalent one, while

the reassignment of individuals leads to confusion

and wasted effort. it is tempting to use multiple

risk estimation methods in the same patient to

avoid missing something. this will lead to errone-

ous risk stratification of the population and over-

treatment, as the number of individuals identified

as high-risk by any one of a number of risk models

Figure 2 . Sinusoidal population risk distribution curves

with the same mean risk but increasing width, indicat-

ing improved risk stratification . The broader the risk

distribution curve, the better the separation of patients

of differing risks .

Page 4

thE JOUrNaL Of CLiNiCaL hypErtENsiONVOL. 10 NO. 6 JUNE 2008

488

will exceed the number of high-risk individuals in

the population.

Whether a risk factor contributes to risk strati-

fication is a separate issue from whether it is

causative or should be treated even if a therapy

is available. Cook6 has shown that systolic blood

pressure, total cholesterol, high-density lipoprotein

cholesterol, or smoking status can be removed

from models including all the framingham risk

factors with minimal change in discrimination. it

appears that only a few cardiovascular risk factors

are required to adequately risk-stratify the popula-

tion. treatment decisions need to be based on clini-

cal trials, not on whether addition of a risk factor

improves risk stratification.

how well a model risk-stratifies a population

should be the primary criterion for choosing a

method for clinical use, but there are additional

criteria that could be considered. these would

include cost, precision, reproducibility, availability,

safety of the test, and whether modification of the

risk factor reduced risk.

there is a bewildering array of new risk factors

under investigation. Examples include Crp, coro-

nary artery calcium score, and measures of arterial

stiffness. from a clinical perspective, if risk factors

do not improve risk stratification as assessed by

the population risk distribution curve, they are

not improving patient care. risk factor evaluation

by regression methods only measures association

and provides no information on the population

risk distribution curve. in contrast, the standard

measure of discrimination, the rOC curve aUC,

does. few cardiovascular risk factors have been

appropriately evaluated for clinical utility. Crp

has been appropriately evaluated and has failed to

demonstrate clinical utility.1–3 therefore, at pres-

ent it does not appear that Crp needs to be added

to the framingham risk factors for cardiovascular

risk calculation. although diagnostic methods

arrive at unique assessments for an individual

patient, predictive methods do not, even if they

are calibrated and have similar population risk

distribution curves. in this case, restratification of

a population may be unhelpful.

RefeRences

1 Cook Nr, Buring JE, ridker pM. the effect of including

C-reactive protein cardiovascular risk prediction models

for women. Ann Intern Med . 2006;145:21–29.

2 Lloyd-Jones DM, Liu K, tian L, et al. Narrative review:

assessment of C-reactive protein in risk prediction for car-

diovascular disease. Ann Intern Med . 2006;145:35–42.

3 Wilson pWf, Nam B-h, pencina M, et al. C-reactive

protein and risk of cardiovascular disease in men and

women from the framingham study. Arch Intern Med .

2005;165:2473–2478.

4 Zou Kh, O’Malley aJ, Mauri L. receiver-operating char-

acteristic analysis for evaluating diagnostic tests and predic-

tive models. Circulation . 2007;115:654–657.

5 Diamond Ga. What price perfection? Calibration and dis-

crimination of clinical prediction models. J Clin Epidemiol .

1992;45:85–89.

6 Cook Nr. Use and misuse of the receiver operating

characteristic curve in risk prediction. Circulation .

2007;115:928–935.

7 hosmer DW, Lemeshow s. Applied Logistic Regression.

New york, Ny: John Wiley & sons; 1989.

8 Wang tJ, Gona p, Larson MG. Multiple biomarkers for the

prediction of first major cardiovascular events and death.

N Engl J Med . 2006;355:2631–2639.

9 Cao JJ, arnold aM, Manolio ta. association of carotid

artery intima-media thickness, plaques, and C-reactive

protein with future cardiovascular disease and all-cause

mortality: the Cardiovascular health study. Circulation .

2007;116:32–38.

10 ridker pM, Buring JE, rifai N, et al. Development and

validation of improved algorithms for the assessment of

global cardiovascular risk in women: the reynolds risk

score. JAMA . 2007;297:611–619.

11 reynolds tM, twomey pJ, Wierzbicki as. Concordance

evaluation of coronary risk scores: implications for

cardiovascular risk screening. Curr Med Res Opin .

2004;20:811–818.

12 ford Es, Giles Wh, Mokdad ah. the distribution of 10-year

risk for coronary heart disease among Us adults: findings

from the National health and Nutrition Examination

survey iii. J Am Coll Cardiol . 2004;43:1791–1796.