Content uploaded by Roland Thomas
Author content
All content in this area was uploaded by Roland Thomas on Aug 08, 2018
Content may be subject to copyright.
Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=hmbr20
Download by: [99.246.0.212] Date: 08 January 2016, At: 12:07
Multivariate Behavioral Research
ISSN: 0027-3171 (Print) 1532-7906 (Online) Journal homepage: http://www.tandfonline.com/loi/hmbr20
On Johnson's (2000) Relative Weights Method for
Assessing Variable Importance: A Reanalysis
D. Roland Thomas, Bruno D. Zumbo, Ernest Kwan & Linda Schweitzer
To cite this article: D. Roland Thomas, Bruno D. Zumbo, Ernest Kwan & Linda
Schweitzer (2014) On Johnson's (2000) Relative Weights Method for Assessing Variable
Importance: A Reanalysis, Multivariate Behavioral Research, 49:4, 329-338, DOI:
10.1080/00273171.2014.905766
To link to this article: http://dx.doi.org/10.1080/00273171.2014.905766
Published online: 18 Jul 2014.
Submit your article to this journal
Article views: 199
View related articles
View Crossmark data
Citing articles: 1 View citing articles
Multivariate Behavioral Research, 49:329–338, 2014
Copyright C
Taylor & Francis Group, LLC
ISSN: 0027-3171 print / 1532-7906 online
DOI: 10.1080/00273171.2014.905766
On Johnson’s (2000) Relative Weights Method
for Assessing Variable Importance: A Reanalysis
D. Roland Thomas
Sprott School of Business, Carleton University
Bruno D. Zumbo
University of British Columbia
Ernest Kwan and Linda Schweitzer
Sprott School of Business, Carleton University
This article provides a reanalysis of J. W. Johnson’s (2000) “relative weights” method for
assessing variable importance in multiple regression. The primary conclusion of the reanalysis
is that the derivation of the method is theoretically flawed and has no more validity than the
discredited method of Green, Carroll, and DeSarbo (1978) on which it is based. By means
of 2 examples, supplemented by other results from the literature, it is also shown that the
method can result in materially distorted inferences when it is compared with another widely
used importance metric, namely, general dominance (Azen & Budescu, 2003; Budescu, 1993).
Our primary recommendation is that J. W. Johnson’s (2000) relative weights method should
no longer be used as a variable importance metric for multiple linear regression. In the final
section of the article, 2 additional recommendations are made based on our analysis, examples,
and discussion.
This article provides a reanalysis of J. W. Johnson’s (2000)
relative weights method for assessing variable importance
in linear regression, a method that has been elaborated
upon and recommended by numerous authors, including
J. W. Johnson and LeBreton (2004); LeBreton, Hargis,
Griepentrog, Oswald, and Ployhart (2007); and Tonidandel
and LeBreton (2011). The method has been extended re-
cently to cover variable importance for multivariate mul-
tiple regression (LeBreton & Tonidandel, 2008), logis-
tic regression (Tonidandel & LeBreton, 2010), and the
importance analysis of higher order regression models
(LeBreton, Tonidandel, & Krasikova, 2013). It has been
compared with a variety of other variable importance meth-
ods by means of examples and Monte Carlo (MC) stud-
ies (J. W. Johnson, 2000; LeBreton, Binning, Adorno, &
Melcher, 2004; LeBreton, Ployhart, & Ladd, 2004; Shear,
Olvera, & Zumbo, 2012; Tonidandel & LeBreton, 2011).
Correspondence concerning this article should be addressed to
D. Roland Thomas, Sprott School of Business, Carleton University,
1125 Colonel By Drive, Ottawa, Ontario K1S 5B6, Canada. E-mail:
roland.thomas@carleton.ca
J. W. Johnson’s (2000) relative weights measure is one of
many measures of variable importance proposed for multiple
regression since the early work of Hoffman (1960), and de-
tailed reviews of the various proposals have been provided
by Bring (1996), Budescu (1993), Azen and Budescu (2003),
J. W. Johnson and LeBreton (2004), and Gromping (2007),
among others. The recent focus on J. W. Johnson’s method
is indicative of the continuing interest in the topic of vari-
able importance both by methodologists and practitioners.
Other methods that continue under active study include dom-
inance analysis (Azen & Budescu, 2003; Budescu, 1993;
Budescu & Azen, 2004), Hoffman’s (1960) measure now
generally referred to as Pratt’s measure in view of its the-
oretical justification by Pratt (1987), and various methods
based on machine learning (see Gromping, 2009). Some of
these methods also have been extended beyond linear regres-
sion: dominance analysis has been applied to multivariate
multiple regression (Azen & Budescu, 2006), logistic regres-
sion (Azen & Traxel, 2009), canonical correlation analysis
(Huo & Budescu, 2009), and hierarchical linear modeling
(Luo & Azen, 2013); Pratt’s method has been applied to
MANOVA (Thomas, 1992), discriminant functions (Thomas
Downloaded by [99.246.0.212] at 12:07 08 January 2016
330 THOMAS, ZUMBO, KWAN, SCHWEITZER
& Zumbo,1996), logistic regression (Thomas, Zhu, Zumbo,
& Dutta, 2008), multilevel modeling (Liu, Zumbo, & Wu, in
press), latent variable regression (Zumbo, 2007), and factor
analysis (Wu, Zumbo, & Marshall, 2014).
From the foregoing it is evident that variable importance
has been and continues to be an extremely active area of re-
search and J. W. Johnson’s (2000) relative weights method-
ology has been one of the foci of recent activity. However, it
is shown in this article that J. W. Johnson’s theoretical justi-
fication of his method is flawed and not readily amenable to
correction.
The article begins with a description of the general ap-
proach used by J. W. Johnson (2000), which builds on an
earlier attempt by Green, Carroll, and DeSarbo (1978; hence-
forth GCD) to develop an importance metric. In the interest
of completeness both the GCD approach and J. W. Johnson’s
attempt to circumvent the difficulties encountered by GCD
are described in detail. The flaws in J. W. Johnson’s strat-
egy are then described using both his algebraic formulation
and a geometric representation of the GCD/Johnson setup.
One property of the relative weights method that has been
considered beneficial both by J. W. Johnson (2000) and oth-
ers is that the importance measures (the relative weights) are
usually numerically close to a metric that summarizes the
pairwise relationships revealed by the full dominance analy-
sis of a multiple regression equation (Budescu, 1993). Now
referred to as general dominance (Azen & Budescu, 2003),
this metric is numerically identical to one proposed earlier
by Lindeman, Merenda, and Gold (1980), henceforth LMG.
Although this numerical similarity has been noted by many
authors, no theoretical explanation of it has been proposed.
A partial explanation is provided in this article based on
the geometrical representation referred to earlier. The article
concludes with a discussion and recommendations. The dis-
cussion includes other points bearing on the utility of the J.
W. Johnson method, along with two examples demonstrat-
ing that despite the frequently observed numerical similarity
between relative weights and general dominance referred
to earlier, the use of relative weights as approximations to
general dominance can still lead to materially distorted in-
ferences in practice.
The examination of J. W. Johnson’s method presented in
this article is quite detailed and is aimed at readers who are fa-
miliar with the technical theory behind variable importance
methods, and J. W. Johnson’s method in particular. Those
readers unfamiliar with this technical theory may neverthe-
less follow the gist of our argument, together with its impli-
cations, by focusing on the introduction, the short section on
the similarity of relative weights to dominance weights, and
the final discussion and recommendation section.
J. W. JOHNSON’S (2000) METHOD
Basic Setup
J. W. Johnson (2000) sought a method for assessing vari-
able importance that could be used when the explanatory
variables in a regression model possessed no inherent or-
dering and where interest focused on the relative contribu-
tion to prediction made by each variable “considering both
its unique contribution and its contribution when combined
with other variables” (p. 2). His method used as its basic
measure the proportion of R2for the full regression model
that could be allocated to each individual explanatory vari-
able. In the uncorrelated case the allocated contribution of
each variable equals its squared simple correlation with the
dependent variable. In the correlated case, however, this al-
location is not appropriate, hence the challenging “variable
importance” problem.
In the following we discuss J. W. Johnson’s (2000) method
using his notation except for the superscript (∗), which is dis-
carded (but for the normalized regression coefficients γ∗
jk). It
should also be noted that we follow his sample-based analy-
sis with no specific reference to populations. This not a major
limitation as the sample quantities referred to in the analy-
sis comprise consistent estimates of their population analog.
Following Gibson (1962) and R. M. Johnson (1966), the first
step in the evolution of the J. W. Johnson (2000) method was
to replace the explanatory variables by a set of orthogonal
variables that best approximate them in some appropriate
sense and by so doing to reduce the relative weights/relative
importance problem to the simple case described earlier. The
construction of the orthogonal set starts from the singular
value decomposition (SVD) of the nxpmatrix Xof explana-
tory variable scores, where it is assumed that Xis of full
column rank, with columns defined in standard score form.
Thus
X=PQ,(1)
where Pis the nxpmatrix of eigenvectors of XXcorre-
sponding to the nonzero eigenvalues; Qis the pxpmatrix
of eigenvectors of XX;and is a diagonal matrix contain-
ing the singular values of X, namely, the square roots of the
nonzero eigenvalues of the products XXand XX.R.M.
Johnson (1966) showed that the best fitting approximation
to X(in the sense of minimizing the sum of squared column
deviations) is given by
Z=PQ
.(2)
In what follows it is necessary to express Xin terms of
Zand vice versa, and the necessary expressions can easily
be derived from Equations 1 and 2 given that Qand are
nonsingular and Qsatisfies QQ
=I=QQ(properties
of the singular value decomposition of X). The resulting
expressions are, in J. W. Johnson’s (2000) notation,
Z=XQ−1Q=X(3)
and
X=ZQQ=Z−1=Z.(4)
Because Equations 3 and 4 are nonsingular linear trans-
formations, they can be thought of as the regressions of the
Downloaded by [99.246.0.212] at 12:07 08 January 2016
JOHNSON’S (2000) RELATIVE WEIGHTS METHOD 331
columns of Zon Xand the columns of Xon Z, respectively,
both sets of regressions having R2=1. This will allow for
the use of standard terminology later.
As noted earlier, a straightforward assessment of relative
variable importance (or relative weight) can be based on Zin
place of X. However, although Zis the “best” approximation
to X(in the sense defined earlier), that does not mean that it is
a “good” approximation. A set of highly correlated variables,
that is, highly correlated columns of X, will necessarily lead
to a poor approximation (see Table 1 of J. W. Johnson, 2000).
The Green, Carroll, and DeSarbo (GCD) Method
The attempt by Green et al. (1978) to remove this approxi-
mation error involves three steps:
Step A. Calculate the portion of the variance in the de-
pendent variable yexplained by each column of Z, a straight-
forward task given that these columns are uncorrelated by
design. The matrix of regression coefficient for regressing
yon Zis given by β=Zyand the portion of variance ex-
plained by the kth column of Z, namely, zk,is then given by
β2
k.
Step B. Regress Zon X, as in Equation 3, where the
coefficients pertaining to the kth column of Zand the vari-
ables xj,j=1,...,p, are denoted γjk,thejkth element of
given by Equation 3.
Step C. Use the aforementioned regression coefficients
to allocate the contribution of the yvariance explained by
each of the uncorrelated zk’s (the columns of Z) to each
of the pvariables xj(the columns of X). GCD represented
the contribution of each explanatory variable x
jto each zk
by γ∗2
jk , namely, each γ2
jk normalized to sum to unity over
all pvariables. The estimated contribution of each xjto the
dependent variable ywas then constructed by summing, for
each explanatory variable in turn, the pportions of explained
yvariance allocated to it. Thus the relative weight assigned
to the jth explanatory variable is
δ2
j=
p
k=1
γ∗2
jk β2
k.(5)
However, the set of “regression weights” specified in Equa-
tion 5 were criticized by Jackson (1980) because the un-
scaled regression coefficients γjk are derived from regres-
sions on correlated variables and cannot therefore be used
to assign importance, i.e., to determine the amount of the
zkvariance explained by the explanatory variables xj.Asa
result, GCD’s proposed importances/weights are now rarely
used. It is worthwhile describing in detail an algebraic pro-
cess for deriving Equation 5 as it will help clarify the dif-
ference between the GCD and J. W. Johnson approaches.
Consider the equation relating the yvariance explained (in
standardized terms) by the orthogonal Zvariables alone,
namely,
R2=βZZβ=βdiag(z
kzk)β,(6)
where Z=(z1,...,zk,...,zp). This value of R2is identical
to that produced by a regression of yon X, given that Zis
a complete basis for X. It will be convenient to maintain the
z
kzknotation in this derivation, even though the Zvariable
is standardized, that is, each z
kzk=1. The same applies for
the x
jxjterms here. From Equation 3, zkcan be expressed
as
zk=γ1kx1+...+γjkxj+... +γpkxp,(7)
so that
z
kzk=γ2
1kx
1x1+...+γ2
jkx
jxj+...+γ2
pkx
pxp
+⎛
⎝
p
j=1
j=j
γjkγjkx
jxj⎞
⎠.(8)
The second term in parentheses in Equation 8 contains the
sample correlations (x
jxj=rjj) between the pvariables and
is the term ignored by GCD. Substituting the remainder of
Equation 8 into Equation 6 and gathering those terms that
are multiples of x
jxjresults in GCD’s Equation 5, except
for the normalization of the γ2
jk to γ∗2
jk . This normalization is
required because omitting the correlation terms from Equa-
tion 8 means that the sum of the remaining terms will differ
from one. Without this normalization, GCD’s relative impor-
tances/weights would not sum to R2.
J. W. Johnson’s (2000) Modification of GCD’s
Measure
There are again three steps to J. W. Johnson’s (2000) method,
the first being identical to that of Green et al. (1978).
Step B. Instead of regressing Zon Xas in Step B earlier,
J. W. Johnson “regressed” the columns of Xon Z. That is, he
invoked the nonsingular linear transformation of Equation 4,
namely, X=Z,where the matrix of regression coefficients
is given by =ZX, with elements that are simple correla-
tions between the zk’s and the xj’s. He also noted that the
squares of the column correlations sum to one as shown here.
Denote the elements of by λjk and consider the expansion
of Equation 4 for the jth column of X, namely,
xj=λ1jz1+···+λkj zk+···λpj zp, (9)
.
which, given the orthogonality of the zk’s, leads to the fol-
lowing expression for the variance of xj, namely,
x
jxj=λ2
1jz
1z1+···+λ2
kj z
kzk+···+λ2
pj z
pzp.(10)
Downloaded by [99.246.0.212] at 12:07 08 January 2016
332 THOMAS, ZUMBO, KWAN, SCHWEITZER
Because the variances of the xj’s and the zk’s are stan-
dardized to one, it follows that
k
λ2
kj =1,j =1, ...,p. (11)
That is, the column sum of the squared correlations equals
one. Alternatively, the result follows because Equation 9 can
be regarded as a regression of a standardized xjonto an
orthogonal set of standardized zk’s having R2=1. J. W.
Johnson went on to state (see Quote A) the following:
Quote A. Because the λ2
jk’s [λjk’s—authors’ correction] are
correlations any particular λ2
jk represents the proportion of
variance in zkaccounted for by xj,just as it represents [em-
phasis added] the proportion of variance in xjaccounted for
by zk.Inotherwords,ZX=XZ....(2000, p. 8)
Step C. J. W. Johnson then described how the contribu-
tion of a particular xjto ycould be calculated by multiplying
the β2
kof GCD’s Step A by the aforementioned “proportion
of variance in each zkaccounted for by xj(λ2
jk)” (2000,
p. 8) and summing the products. This resulted in replacing
the quantities γ∗2
jk of Equation 5 with the squared simple
correlations λ2
jk, yielding
εj=
p
k=1
λ2
jkβ2
k,(12)
where εjrepresents the relative weight of the jth explanatory
variable. He summed up his argument by stating (see Quote
B) the following:
Quote B. Each zkis a linear combination of the xj’s, s o the
variance in yaccounted for by zkcan be easily partitioned
among the xj’s according to the proportion of variance in zk
accounted for by each xj. (2000, p. 8)
The Flaw in J. W. Johnson’s Argument
Quotes A and B together reveal a serious flaw in J. W.
Johnson’s argument. With respect to the last part of the first
sentence in Quote A, it is true that the squared simple corre-
lation coefficient λ2
jk (equal to λ2
kj by virtue of the symmetry
of )represents the proportion of variance in xjaccounted
for by zk, and this is true both for zkacting alone and for
zkacting together with the other z’s because of the orthog-
onality of the zk’s (see Equations 9 and 10). However, with
respect to the first part of the first sentence of Quote A, it is
NOT true that “any particular λ2
jk represents the proportion
of variance in zkaccounted for by xj”acting together with
the other xj’s, as is implied by the phrase in Quote A ital-
icized by the authors. Because the xj’s are correlated, it is
true only that “any particular λ2
jk represents the proportion of
variance in zkaccounted for by xj”acting alone. However,
this far weaker condition does not provide a theoretical basis
for J. W. Johnson’s Step C, namely, replacing the scaled re-
gression coefficients of GCD’s Equation 5 with the squared
simple correlations λ2
jk. Justification of this step would re-
quire the existence of an equation similar to Equation 10
with the role of the x’s and the z’s interchanged. But there
is no such equation. The second sentence in Quote A notes
that the matrix of correlations is symmetric, and because
the column sums of the squared correlations sum to one (see
Equation 11), so also must the row sums, that is,
k
λ2
jk =1,j =1, ... , p. (13)
The “column sum” shown in Equation 11 follows from
Equations 9 and 10 and yields a partition of x
jxjin terms
of the coefficients associated with the z
kzkterms, namely,
the λ2
kj ’s (equal to the λ2
jk’s by symmetry). But one cannot
argue backward from the “row sum” shown in Equation 13
to get a “mirror image” version of Equation 10 that has the
roles of the x’s and the z’s interchanged and that will allow
for an unambiguous partitioning of z
kzkin terms of the λ2
jk
coefficients associated with x
jxjterms.
J. W. Johnson’s Quote B underlines the fact that he really
did proceed as if it was valid to apportion the variance in
zkaccounted for by the correlated xj’s using squared simple
correlations. With respect to this second quote, it is true that
each zkis a linear combination of the xj’s. However, the
coefficients of the linear combination are not the λkj ’s of
Equation 9 (equal to the λkj ’s by symmetry) but the γjk’s
of the linear combination displayed in Equation 7, which
denote the elements of the coefficient matrix defined in
Equation 3. As noted earlier, there is no “mirror image” of
Equation 10 that he can draw upon; the best that can be
done is to proceed from Equations 3 and 7 to the variance
expression given by Equation 8 on which the GCD approach
is based. They proceeded by dropping the correlation terms.
J. W. Johnson did likewise but instead of normalizing the γ’s
as in the GCD, he replaced them with simple correlations
based upon a faulty argument. Thus his method has no more
validity than the GCD method.
A Geometric Representation
Consider a simple example featuring two explanatory vari-
ables x1and x2and hence two orthogonal variables z1and
z2. These four variables are displayed in Figure 1 as vectors
in the two-dimensional space spanned by the explanatory
variables, which is a subset of the N-dimensional space of
the observations. For a detailed explanation of this geometric
interpretation, and its application to least squares regression,
see Thomas, Hughes, and Zumbo (1998). The angle between
the explanatory variables x1and x2is defined to be 2θ, so that
the correlation between x1and x2is given by ρ12 =cos2θ,
another property of the geometrical interpretation of the vari-
ables. Because all variables are standardized, that is, in this
geometric representation they all have length one, the singu-
lar value decomposition results in a symmetric placement of
Downloaded by [99.246.0.212] at 12:07 08 January 2016
JOHNSON’S (2000) RELATIVE WEIGHTS METHOD 333
z1
z2
x1
x2
4
4
2
2
2
C
B
A
D
O
11x1
12x1
22x2
21x2
FIGURE 1 A geometric view of the singular value decomposition.
the orthogonal vectors as shown in Figure 1. This can be ver-
ified by solving the determinantal equation that defines the
matrix of orthogonal variables Z(see the earlier discussion
of the basic setup). In Figure 1 we illustrate the geometric
relationships between z1and the explanatory variables x1
and x2and between z2and the variables x1and x2. These
relationships require vector addition, which is geometrically
implemented using the triangle rule. In Figure 1 the vector
triangle OAB displays the linear relationship between the
vector variables z1,x1,and x2, which corresponds to the
algebraic form of the relationship, namely,
z1=γ11x1+γ21 x2.(14a)
The parameter γ11 defining the path −→
OB is positive, but the
parameter γ21 defining the path −→
BA must be negative because
the direction of the vector x2has to be reversed to complete
the triangle. Similarly, the vector triangle ODC provides a
geometric representation of the algebraic relationship
z2=γ12x1+γ22 x2.(14b)
Equations 14a and 14b are special cases of Equation 3.
To explore the difficulty with J. W. Johnson’s proposal
we consider the triangle OAB shown in Figure 1. The angle
subtended by z1equals 2θ, so that using a standard geometric
result the length of the z1vector satisfies
|z1|2=γ2
11|x1|2+γ2
21|x2|2−2|γ11 |γ21|x1||x2|cos 2θ,
(15)
where |w|=√wwdenotes the length of an arbitrary vec-
tor w. It is evident that a neat partitioning of |z1|2(i.e., the
variance of z1) into separate variance contributions due to
x1and x2exists only when cos 2θ=0, in other words when
x1and x2are orthogonal (2θ=π/2), that is, uncorrelated.
This is the difficulty encountered by GCD, and this diffi-
culty cannot be circumvented. The triangle OAB with base
z1shown in Figure 1 is unique because the linear relation-
ships implied by Equation 3 are exact. In geometric terms
one cannot find another triangle with sides composed of a
positive or negative multiple of x1(and parallel to x1) and a
positive or negative multiple of x2(and parallel to x2) that
has resultant z1. An equivalent argument can be applied to
triangle ODC, which represents Equation 14b. Construction
of a right-angled triangle that will yield the “desired” par-
titioning is thus impossible, and we therefore arrive at the
same conclusion as before, namely, J. W. Johnson’s attempt
to circumvent the difficulties encountered by GCD fails.
Apparent Plausibility
The failure of J. W. Johnson’s method is perhaps not im-
mediately obvious because it does result in “importance”
weights that sum to the value of R2for the regression of y
on X. However, it can easily be shown that allocating the
variance of zkto the pvariables xjusing any set of pcon-
stants that sum to one will result in a set of ultimate weights
that sum to R2. Let the constants αj1,α
j2, ...,α
jp satisfy
p
j=1αjk =1 for all k. Then the analog of Equation 12 be-
comes ε(α)
j=p
k=1αjkβ2
k, and summing over jresults in
p
k=1⎧
⎨
⎩
p
j=1
αjk⎫
⎬
⎭
β2
k=
p
k=1
β2
k=R2.(16)
In addition, individual values of (α)
jwill be nonnegative
provided the constants αjk are all nonnegative as is the case
with the squared simple correlation coefficients selected by
J. W. Johnson.
THE SIMILARITY OF RELATIVE WEIGHTS TO
THE GENERAL DOMINANCE (LMG) METRIC
J. W. Johnson (2000) noted that his proposed metric is not
invariant to the choice of orthogonal basis selected, the SVD
basis being only one of an infinity of possible choices. He
justified his use of the SVD on the grounds that it provides
a better approximation to the general dominance metric than
any of the “many different orthogonalization procedures”
that he had tried. Based on a collection of 31 data sets fea-
turing from 3 to 10 explanatory variables he showed that the
maximum mean absolute deviation between his SVD-based
relative weights and those provided by general dominance
(as a percentage of explainable variance) was only 2%. Sim-
ilar results have been reported by a number of authors (see,
e.g., LeBreton, Ployhart, et al., 2004), although an exception
has been reported by Shear et al. (2012), which is described
in more detail later.
The reason for this frequent similarity is not easy to see in
general given the very different structure of the two metrics.
Relative weights involve a single regression on porthog-
onalized variables, whereas the general dominance metric
Downloaded by [99.246.0.212] at 12:07 08 January 2016
334 THOMAS, ZUMBO, KWAN, SCHWEITZER
requires averaging the squared semipartial correlation across
all subsets of the explanatory variables. However, with the
aid of the geometric view of Johnson’s setup, it is shown in
Appendix A that for the case p=2, J. W. Johnson’s weights
and the importances provided by general dominance are al-
gebraically identical. Although equality for two explanatory
variables does not explain the frequently observed similarity
of the two metrics for three or more explanatory variables, it
does render the numerical similarity a little less surprising,
particularly for relatively small numbers of variables.
It has been shown by LeBreton, Ployhart, et al. (2004) that
the average difference between the two sets of weights does
increase as the number of explanatory variables increases.
Evidence regarding the equality of relative weights and
dominance metrics for two variables has previously appeared
in the literature. The result was observed and reported by
Braun and Oswald (2011) based on their reanalysis of an
example due to Budescu (1993), but no proof was provided.
Shear et al. (2012) included one two-variable scenario in their
MC study of variable importance metrics and reported a mean
absolute deviation of zero between the relative weights and
the general dominance weights (hence equality).
DISCUSSION, EXAMPLES, AND SOME
RECOMMENDATIONS
Discussion
In his original paper, J. W. Johnson (2000) noted another
drawback of his method, namely, that if a nonsingular linear
transformation is applied to a subset of the variables, then
the relative importance of the untransformed variables can
change. Although he considered this a minor issue, it can be
important. As noted by Pratt (1987), it would be natural to
assume that “the importance of a married couple’s income
relative to their ages is the same as it is relative to the average
and difference of their ages” (p. 253). However, in light of
the flaw in the derivation of J. W. Johnson’s method, such
considerations become moot.
J. W. Johnson (2000) presented his relative weights
method as a “more computationally efficient measure than
that proposed by Budescu (1993)” (p. 12). In view of the
numerical similarity between relative weights and the gen-
eral dominance or LMG metric, relative weights have been
recommended as an alternative to general dominance when-
ever computation of the latter becomes infeasible (LeBreton,
Ployhart, et al., 2004).
This recommendation was based on the results of a de-
tailed MC study and was therefore dependent only on the
form of J. W. Johnson’s relative weights, not on their deriva-
tion. However, the validation of statistical methods should
always be based on two considerations: first, the theory be-
hind the method must be sound; second, the method must
perform well in practice on the kind of data that could be en-
countered by practitioners. J. W. Johnson’s relative weights
method does reasonably well on the second requirement
(when closeness of individual weights to corresponding gen-
eral dominance weights is used as the standard for adop-
tion), but it completely fails on the first. Nevertheless, it
was suggested by a reviewer that some practitioners might
not be convinced that a theoretical flaw is sufficient to dis-
qualify the relative weights method. We therefore describe
two examples, one empirical and one synthetic, which to-
gether demonstrate that using relative weights in place of
general dominance can result in misleading inferences in
practice.
Two Examples Comparing Relative Weights with
General Dominance
The first example is taken from an article by de Vries, Bakker-
Pieper, and Oostenveld (2010), who examined the relation-
ship between leadership style, communication style, and
leadership outcomes. They collected data from 279 employ-
ees who were asked to report on their supervisor’s leadership
behavior, communication behavior, and performance as well
as their own satisfaction and commitment to their supervisor.
Although de Vries et al. included several leadership styles
in their study, for this example we have selected charismatic
leadership style as the dependent variable. The explanatory
variables included six communication styles: verbal aggres-
siveness,expressiveness,preciseness,assuredness,support-
iveness, and argumentativeness. It will be convenient later
to denote the dependent variable as yand the explanatory
variables as x1,x
2,x
3,x
4,x
5,and x6,respectively. De Vries
et al. reported an R2of 0.69 for the regression. The corre-
lation matrix for the dependent and independent variables,
taken from de Vries et al., is displayed in Table 1. The results
of the importance analysis are displayed in Table 2, the sec-
ond column of which provides standardized regression coef-
ficients for reference purposes. The third and fourth columns
of Table 2 provide individual relative weights and general
dominance weights (both standardized by R2to sum to one),
and the rightmost column gives their percentage difference,
measured relative to general dominance. It can be seen that
the relative differences between the two sets of weights range
from –9.2% to 12.0% across the six explanatory variables,
differences that are far from negligible and that do not rep-
resent a particularly good approximation of relative weights
to general dominance. In substantive terms, for example, the
estimate of the absolute difference in importance between
assuredness (x4) and supportiveness (x5) provided by the
relative weights method is 50% greater than that provided
by general dominance namely, 0.113 versus 0.075 (obtained
from Table 2). Nevertheless, the absolute and/or the rela-
tive differences in importance provided by the two methods
might still not be considered material by some practitioners
because they do not change the variable orderings. The issue
of variable orderings is dealt with in the following example.
Downloaded by [99.246.0.212] at 12:07 08 January 2016
JOHNSON’S (2000) RELATIVE WEIGHTS METHOD 335
TABLE 1
The Correlation Matrix for the Empirical Examplea
YX
1X2X3X4X5X6
Charismatic Leadership Y 1 –0.37 0.45 0.52 0.64 0.66 0.25
Verbal aggressiveness X1–0.37 1 –0.18 –0.40 –0.01 –0.57 –0.04
Expressiveness X20.45 –0.18 1 0.18 0.33 0.54 0.30
Preciseness X30.52 –0.40 0.18 1 0.43 0.44 –0.09
Assuredness X40.64 –0.01 0.33 0.43 1 0.35 0.00
Supportiveness X50.66 –0.57 0.54 0.44 0.35 1 0.18
Argumentativeness X60.25 –0.04 0.30 –0.09 0.00 0.18 1
aSubset of data provided by de Vries et al., 2010.
This second example is based on a synthetic correlation
matrix obtained from the de Vries et al. (2010) matrix by
modifying ρy,3, the correlation between yand x3.Thevalue
of this correlation was increased from 0.52 in increments of
0.01 until we obtained the results of Table 3, which corre-
spond to a correlation of 0.61. It can be seen from Table 3 that
whereas the general dominance weights yield the ordering
x5>x
3(unchanged from the results of Table 2 by the mod-
ification of ρy,3), J. W. Johnson’s relative weights now show
an order reversal, that is, x3>x
5. The difference between
the empirical correlation matrix and the synthetic matrix is
quite minor, consisting simply of a change from 0.52 to 0.61
in a single element, resulting in a valid synthetic correla-
tion matrix (all eigenvalues positive) that is well within the
range that could be observed in practice. This demonstrates
that by using relative weights as an approximation to gen-
eral dominance, practitioners could be led astray by distorted
inferences.
Further Discussion and Some
Recommendations
It should be noted that the examples do not account for sam-
pling error, which would further complicate any comparison
between relative and general dominance weights. This is il-
lustrated by some results due to Shear et al. (2012), who
simulated the effect of sampling variability by taking 1,000
TABLE 2
Empirical Example: The Differences Between
Relative (RW) and General Dominance (GD) Weights
Explanatory
Variables βaRWbGDb
(RW – GD)/
GD%
X1–0.100 0.084 0.075 12.0
X20.018 0.089 0.092 –3.3
X30.149∗∗ 0.154 0.153 0.7
X40.447∗∗ 0.360 0.347 3.7
X50.343∗∗ 0.247 0.272 –9.2
X60.189∗∗ 0.065 0.061 6.6
aRefer to Table 2 of de Vries et al. (2010); more accurate values supplied
by R. de Vries. bWeights normalized by R2to sum to 1. ∗∗pvalue <.01.
MC samples of size 301 for each of the seven scenarios
they studied. For one of their scenarios, based on a popu-
lation correlation matrix featuring suppressor variable rela-
tionships, they found different variable orderings between
relative and general dominance weights in 47% of the MC
sample cases despite the fact that the population matrix itself
generated identical orderings. These authors also examined a
correlation matrix taken from the well-known Holzinger and
Swineford (1939) data (with sample size 301) that exhibited
no serious multicollinearity or suppressor behavior. Even for
this benign case, close to 10% of the MC samples exhib-
ited different relative weights/general dominance orderings,
providing no grounds for complacency.
It is clear that using relative weights as an approxima-
tion strategy exposes researchers to an appreciable risk of
misleading inferences. But it is also an unnecessary strat-
egy given modern computing power. Even 10 years ago,
LeBreton, Ployhart, et al. (2004) noted that dominance anal-
yses were then feasible for up to 15 explanatory variables. In
practice, therefore, computational efficiency has long ceased
to be a practical reason for using relative weights instead of
general dominance. Current users of the risky approximation
strategy would be well advised to use general dominance
weights directly. Such a change in research procedure would
require nothing more than a change of software; some suit-
able software routines are listed in Appendix B.
TABLE 3
Synthetic Example: Different Variable Orderings
Between Relative (RW) and General Dominance
(GD) Weights
Explanatory
Variables RWa
Rank
(RW) GDa
Rank
(GD)
X10.073 5 0.066 5
X20.083 4 0.086 4
X30.239 20.236 3
X40.316 1 0.305 1
X50.223 30.245 2
X60.066 6 0.062 6
aWeights normalized by R2to sum to 1.
Downloaded by [99.246.0.212] at 12:07 08 January 2016
336 THOMAS, ZUMBO, KWAN, SCHWEITZER
Notwithstanding the practical drawbacks discussed ear-
lier, the principal argument against treating Johnson’s rel-
ative weights method as a variable importance metric re-
mains the fundamental flaw in its theoretical derivation, a
flaw that is very similar to that exhibited by the now dis-
credited GCD proposal. This argument is particularly salient
for those who use the relative weights method as an impor-
tance metric in its own right. Given the flaw in its derivation
it is not possible to define precisely what kind of variable
importance it represents. As noted earlier in this article (see
Equation 16), any set of nonnegative coefficients that sum to
one will yield a set of ultimate weights that sum to R2.J.W.
Johnson’s (2000) relative weights method simply provides
one such set, and it is not at all clear what importance prop-
erty these weights possess other than the risky and unneces-
sary approximation property discussed earlier. In conclusion
we make three recommendations:
1. J. W. Johnson’s (2000) relative weights method should
no longer be used as a variable importance metric for
multiple linear regression.
2. Researchers who use relative weights as approxima-
tions should instead use general dominance weights
directly.
3. Those researchers involved in developing extensions
of the relative weights method should reconsider its
validity and use in contexts other than multiple linear
regression.
REFERENCES
Azen, R., & Budescu, D. V. (2003). The dominance analysis approach for
comparing predictors in multiple regression. Psychological Methods,8,
129–148.
Azen, R., & Budescu, D. (2006). Comparing predictors in multivariate re-
gression models: An extension of dominance analysis. Journal of Educa-
tional and Behavioral Statistics,31, 157–180.
Azen, R., & Traxel, N. (2009). Using dominance analysis to determine
predictor importance in logistic regression. Journal of Educational and
Behavioral Statistics,34, 319–347.
Braun, M. T., & Oswald, F. L. (2011). Exploratory regression analysis: A
tool for selecting models and determining predictor importance. Behavior
Research Methods,43, 331–339.
Bring, J. (1996). A geometric approach to compare variables in a regression
model. American Statistician,50, 57–62.
Budescu, D. V. (1993). Dominance analysis: A new approach to the problem
of relative importance of predictors in multiple regression. Psychological
Bulletin,114, 542–551.
Budescu, D. V., & Azen, R. (2004). Beyond global measures of relative
importance: Some insights from dominance analysis. Organizational Re-
search Methods,7, 341–350.
de Vries, R. E., Bakker-Pieper, A., & Oostenveld, W. (2010). Leadership =
communication? The relations of leaders’ communication styles with
leadership styles, knowledge sharing and leadership outcomes. Journal
of Business and Psychology,25, 367–380.
Gibson, W. A. (1962). Orthogonal predictors: A possible resolution of the
Hoffman-Ward controversy. Psychological Reports,11, 32–34.
Green, P. E., Carroll, J. D., & DeSarbo, W. S. (1978). A new measure of
predictor variable importance in multiple regression. Journalof Marketing
Research,15, 356–360.
Gromping, U. (2007). Estimators of relative importance in linear regres-
sion based on variance decomposition. The American Statistician,61,
139–147.
Gromping, U. (2009). Variable importance in regression: Linear regression
versus random forest. The American Statistician,63, 308–319.
Hoffman, P. J. (1960). The paramorphic representation of clinical judgment.
Psychological Bulletin,57, 116–131.
Holzinger, K., & Swineford, F. (1939). A study in factor analysis: The
stability of a bifactor solution. Supplementary Educational Monograph,
48. Chicago, IL: University of Chicago Press.
Huo, Y., & Budescu, D. (2009). An extension of dominance analysis to
canonical correlation analysis. Multivariate Behavioral Research,44,
688–709.
Jackson, B. B. (1980). Comment on “A new measure of predictor variable
importance in multiple regression.” Journal of Marketing Research,17,
113–115.
Johnson, J. W. (2000). A heuristic method for estimating the relative weight
of predictor variables in multiple regression. Multivariate Behavioral
Research,35, 1–19.
Johnson, J. W., & LeBreton, J. M. (2004). History and use of relative
importance indices in organizational research. Organizational Research
Methods,7, 238–257.
Johnson, R. M. (1966). The minimal transformation to orthonormality. Psy-
chometrika,31, 61–66.
LeBreton, J. M., Binning, J. F., Adorno, A. J., & Melcher, K. M. (2004). Im-
portance of personality and job-specific affect for predicting job attitudes
and withdrawal behavior. Organizational Research Methods,7, 300–325.
LeBreton, J. M., Hargis, M. B., Griepentrog, B., Oswald, F. L, & Ployhart,
R. E. (2007). A multidimensional approach for evaluating variables in
organizational research and practice. Personnel Psychology,60, 475–498.
LeBreton, J. M., Ployhart, R. E., & Ladd, R. T. (2004). A Monte Carlo com-
parison of relative importance methodologies. Organizational Research
Methods,7, 258–282.
LeBreton, J. M., & Tonidandel, S. (2008). Multivariate relative importance:
Extending relative weight analysis to multivariate criterion spaces. Jour-
nal of Applied Psychology,93, 329–345.
LeBreton, J. M., Tonidandel, S., & Krasikova, D. V. (2013). Residualized
relative importance analysis: A technique for the comprehensive decom-
position of variance in higher order regression models. Organizational
Research Methods,16, 449–473.
Lindeman, R. H., Merenda, P. F., & Gold, R. Z. (1980). Introduction to
bivariate and multivariate analysis. Glenview, IL: Scott Foresman.
Liu, Y., Zumbo, B. D., & Wu, A. D. (in press). Relativeimportance of predic-
tors in multilevel model analysis. Journal of Modern Applied Statistical
Methods.
Luo, W., & Azen, R. (2013). Determining predictor importance in hierar-
chical linear models using dominance analysis. Journal of Educational
and Behavioral Statistics,38, 3–31.
Pratt, J. W. (1987). Dividing the indivisible: Using simple symmetry to
partition variance explained. In T. Pukkila & S. Putanen (Eds.), Proceed-
ings of the Second International Conference in Statistics (pp. 245–260).
Tampere, Finland: University of Tampere.
Shear, B. R., Olvera, O., & Zumbo, B. D. (2012). Relative variable im-
portance for multiple regression in suppressor situations: A Monte Carlo
study. Paper presented at the American Educational Research Association
annual meeting, Vancouver, British Columbia, Canada.
Thomas, D. R. (1992). Interpreting discriminant functions: A data analytic
approach. Multivariate Behavioral Research,27, 335–362.
Thomas, D. R., Hughes, E., & Zumbo, B. D. (1998). On variable importance
in linear regression. Social Indicators Research,45, 253–275.
Thomas, D. R., Zhu, P., Zumbo, B. D., & Dutta, S. (2008). On measuring
the relative importance of explanatory variables in a logistic regression.
Journal of Modern Applied Statistical Methods,7, 21–38.
Downloaded by [99.246.0.212] at 12:07 08 January 2016
JOHNSON’S (2000) RELATIVE WEIGHTS METHOD 337
Thomas, D. R., & Zumbo, B. D. (1996). Using a measure of variable im-
portance to investigate the standardization of discriminant coefficients.
Journal of Educational and Behavioral Statistics,21, 110–130.
Tonidandel, S., & LeBreton, J. M. (2010). Determining the relative impor-
tance of predictors in logistic regression: An extension of relative weights
analysis. Organizational Research Methods,13, 767–781.
Tonidandel, S., & LeBreton, J. M. (2011). Relative importance analysis:
A useful supplement to regression analysis. Journal of Business and
Psychology,26(1), 1–9.
Wu, A. D., Zumbo, B. D., & Marshall, S. K. (2014). A method to aid
in the interpretation of EFA results: An application of Pratt’s measures.
International Journal of Behavioral Development,38, 98–110.
Zumbo, B. D. (2007). Validity: Foundational issues and statistical method-
ology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol.
26. Psychometrics (pp. 45–79). Amsterdam, The Netherlands: Elsevier
Science.
APPENDIX A
The Equality of the Relative Weights and
General Dominance (LMG) Metrics for Two
Explanatory Variables
The variance allocated by the general dominance (LMG)
metric to explanatory variable x1can be written as
δ2
1+δ1δ2ρ12 +δ2
2−δ2
1ρ2
12/2(A1)
(see, e.g., Gromping, 2007), where δidenotes the standard-
ized regression coefficient in the regression of yon X, and
ρ12 denotes the correlation between variables x1and x2.In
what follows we show that J. W. Johnson’s (2000) relative
weight for x1is identical to Equation A1.
It follows from Equation 12 that the J. W. Johnson weight
for variable x1is given by
ε1=λ2
11β2
1+λ2
12β2
2.(A2)
With reference to Figure 1, it follows from symmetry that
the angle between x1and z1and between x2and z2is π
4−θ
and that the angle between x1and z2and between x2and z1
is π
4+θ. The corresponding correlations equal the cosines
of these angles, so that
λ11 =λ22 =cos π
4−θ=√2
2(cos θ+sin θ),
(A3a)
and
λ12 =λ21 =cos π
4+θ=√2
2(cos θ−sin θ).
(A3b)
We next need to express the β’s (which represent the re-
gression of yon Z) in terms of the δ’s (which represent the
regression of yon X). Because Zand Xspan the same sub-
space, Zβ=Xδand from Equations 3 and 4 it then follows
that
β=−1δ.(A4)
The elements of can be obtained with reference to tri-
angle OCD in Figure 1. First, equating the lengths of the
components of −→
CD and −→
OD orthogonal to −→
OC and rearrang-
ing terms results in the expression
γ12 =−|γ12|=√2
2(sin θ−cos θ).(A5a)
Then, summing the lengths of the components of −→
CD and
−→
OD parallel to −→
OC and equating this sum to the length of −→
OC
and again rearranging leads to
γ22 =cos π
4−θ+sin π
4−θcos 2θ/sin 2θ,
(A5b)
and from symmetry it follows that γ11 =γ22 and γ12 =γ21.
Equations A5 and their symmetric counterparts provide the
elements of the 2 ×2matrixfrom which the elements of
its inverse −1can be easily evaluated as
˘γ11 =˘γ22 =√2
2(cos θ+sin θ);
˘γ12 =˘γ21 =√2
2(cos θ−sin θ).(A6)
Equations A2 through A5 comprise the results needed
for a comparison with the LMG/Budescu metric for x1.As-
sembling these pieces and tidying up the expressions using
standard trigonometrical identities result in
1
2δ2
11+sin22θ+δ1δ2cos 2θ+1
2δ2
2cos22θ. (A7)
It remains to convert the trigonometric expressions into cor-
relations. It was noted earlier that ρ12 =cos θ, from which it
follows via trigonometric identities that
1+sin22θ=2−cos22θ=2−ρ2
12.(A8)
Substitution into Equation A7 confirms that ε1and the
general dominance (LMG) metric are identical.
APPENDIX B
Software for Implementing Dominance Analysis
and/or Evaluating the General Dominance
Importance Metric
Tonidandel and LeBreton (2011) provide a list of suitable
software macros in the appendix to their paper. An updated
selection is provided here.
1. SAS
The dominance analysis software referred to in Azen and
Budescu (2003) can be obtained from the following webp-
age: https://pantherfile.uwm.edu/azen/www/damacro.html
Downloaded by [99.246.0.212] at 12:07 08 January 2016
338 THOMAS, ZUMBO, KWAN, SCHWEITZER
It implements a full dominance analysis for up to 10
explanatory variables, including the general dominance
weights.
2. Excel
a. General dominance weights can be obtained using
Excel software written by James LeBreton, which is
available from the following webpage: http://www1.
psych.purdue.edu/∼jlebreto/dominance%20analysis%
204.4.xls
It aggregates the results of multiple regression runs and
produces general dominance weights for up to six
explanatory variables.
b. Excel software that implements a full dominance anal-
ysis, including general dominance weights, is described
by Braun and Oswald (2011). It can handle up to nine
explanatory variables and is available from the following
webpage: http://dl.dropbox.com/u/2480715/ERA.xlsm?dl
=1
Downloaded by [99.246.0.212] at 12:07 08 January 2016
