Content uploaded by Dave Bartram
Author content
All content in this area was uploaded by Dave Bartram on Jan 16, 2014
Content may be subject to copyright.
The Great Eight Competencies: A Criterion-Centric Approach to Validation
Dave Bartram
SHL Group
The author presents results of a meta-analysis of 29 validation studies (N ⫽ 4,861) that uses the Great
Eight competency factors (Kurz & Bartram, 2002) as the criterion measurement framework. Predictors
of the Great Eight competencies based only on personality scales show moderate to good correlations
with line-manager ratings for all 8 of the competencies. On their own, ability tests correlate with 4 of the
8 competencies, and together ability and personality data yield operational validities ranging from 0.20
to 0.44 for the 8 competencies. Operational validities for aggregated predictors with aggregated criteria
were estimated to be 0.53. The value of differentiating the criterion space and of relating predictor
variables to criterion variables in a one-to-one fashion is discussed.
Keywords: validation, personality, competency models, meta-analysis
This study presents a model of performance in the workplace
that defines eight broad competency factors, which we refer to as
the Great Eight (Bartram, Robertson, & Callinan, 2002; Kurz &
Bartram, 2002). The Great Eight have emerged from factor anal-
yses and multidimensional scaling analyses of self- and manager
ratings of workplace performance, not from the predictor domain
(i.e., ability tests, motivation or personality questionnaires). Thus,
they provide a criterion-centric model from which to explore the
validity of various potential predictors of workplace performance.
This model and its associated predictor– outcome relationships are
explored through a meta-analysis of 29 validity studies. The in-
tention behind this research is not just to add to the body of data
relating to the validity of personality and ability tests as predictors
of workplace behavior but also to demonstrate the value of an
approach that uses a model of the criterion domain as the orga-
nizing framework for meta-analysis rather than the more usual
predictor domain models (e.g., the Big Five personality factors
model).
Ability and Personality as Predictors of Job Performance
Ability measures have been acknowledged as good predictors of
job performance and even better predictors of training perfor-
mance. The early meta-analyses (e.g., Hunter & Hunter, 1984)
showed the generalizability of this finding. More recently, the
academic literature has begun to support the view that personality
measures also predict performance at work. Studies concentrating
on the Big Five personality factors have shown that Conscientious-
ness and Emotional Stability have broadly generalizable relation-
ships with overall job performance (OJP; Barrick & Mount, 1991;
Hough, 1992; Salgado, 1997, 1998). Barrick and Mount (1991)
report corrected mean validities of r ⫽ .22 for Conscientiousness
and r ⫽ .13 for Extraversion. Tett, Jackson, and Rothstein (1991)
quote corrected mean validities ranging from r ⫽ .16 for Extra-
version to r ⫽ .33 for Agreeableness. Salgado (1997, 1998) rep-
licated Barrick and Mount’s results with European data sets and
also found evidence for the validity of Emotional Stability (cor-
rected r ⫽ .19). More recently, Hurtz and Donovan (2000) have
reported corrected correlations of r ⫽ .22 for Conscientiousness,
r ⫽ .14 for Emotional Stability, and r ⫽ .09 for Extraversion.
Thus, the current evidence is generally supportive of some of the
Big Five in providing moderate predictions of relatively gross job
performance measures.
Predictor- Versus Criterion-Centric Approaches
The traditional approach to validation has been predictor centric.
Researchers have asked questions like “What does instrument X
predict?,” “How well does personality predict job performance?,”
and “What do ability tests predict?” As a consequence, we have
seen separate literatures develop regarding the validity of person-
ality scales and ability scales even though both types of instru-
ments are used to predict aspects of behavior in the workplace.
Through meta-analyses, there has been some pulling together of
the findings within each literature. However, this has tended to be
at the expense of a loss of detail, because there has been a relative
lack of focus on the nature or appropriateness of the criterion
measures (which are typically supervisor ratings of OJP) when
studies have been drawn together for meta-analysis.
The argument presented here is that we should refocus our
questions. We should be asking “How can we best predict Y?”
where Y is some meaningful and important aspect of workplace
behavior. Competency frameworks, when defined in terms of
observable workplace behaviors, provide the basis for a differen-
This research was funded by the SHL Group. Dave Bartram is research
director with the SHL Group.
I thank Rainer Kurz and Helen Baron, who contributed to the work
reported here while they were employees of SHL Group. Rainer Kurz made
a major contribution to the development of the Great Eight model, and
Helen Baron and my research team collected and collated all the validity
studies, which were subsequently analyzed for this study. I also thank Peter
Warr for his valuable comments and suggestions on a draft of this article
and Jerard Kehoe for his thoroughness and helpful comments and
suggestions.
Correspondence concerning this article should be addressed to Dave
Bartram, SHL Group plc, The Pavilion, 1 Atwell Place, Thames Ditton,
Surrey KT7 0NE, England. E-mail: dave.bartram@shlgroup.com
Journal of Applied Psychology Copyright 2005 by the American Psychological Association
2005, Vol. 90, No. 6, 1185–1203 0021-9010/05/$12.00 DOI: 10.1037/0021-9010.90.6.1185
1185
tiated criterion measurement. They support investigation of differ-
ent aspects of performance, promoting a more sophisticated un-
derstanding of the factors underlying OJP. Inconsistencies in
predictors of overall performance between jobs might be explained
by differences between jobs in the relative importance of different
aspects of performance. Differentiation of the criterion space
would allow better prediction of job performance for a particular
role once the competency requirements for the role were
understood.
Models of the Criterion Measurement Domain
If we are to show a more complete and consistent pattern of
relationships between predictors and workplace performance, we
need to differentiate the criterion measures in a meaningful way.
When this more differentiated approach has been adopted, a richer
picture of the relationship between predictors and performance in
the workplace emerges. Robertson and Kinder (1993) examined
the average validity of a range of personality scales judged to be
relevant for each of 12 different performance areas in a meta-
analysis and found generalizable validity for 10 of them. Nyfield,
Gibbons, Baron, and Robertson (1995) reported consistent patterns
of correlations between each of the Big Five personality dimen-
sions and different aspects of job performance as measured using
manager’s ratings. Robertson, Baron, Gibbons, MacIver, and Ny-
field (2000) demonstrated that ratings of OJP that correlated highly
with planning and organizing competency did show a positive
correlation with Conscientiousness, as expected based on the meta-
analysis literature.
The Robertson et al. study illustrates the importance of consid-
ering moderator variables in which artifact corrections leave sub-
stantial amounts of variance in the validity coefficients. As a
further example, Judge, Bono, Ilies, and Gerhardt (2002) have
shown how the correlations between conscientiousness and ratings
of leadership qualities are moderated by sample effects: For stu-
dent samples Conscientiousness is correlated with ratings of lead-
ership qualities (r ⫽ .36), but this drops to r ⫽ .17 for people in
government and military organizations and to r ⫽ .05 for people in
business and commercial settings.
Studies like these illustrate the importance of doing more than
just looking at correlations with overall measures of job perfor-
mance. We need to have a well-articulated model of the domain of
workplace behaviors and then evaluate the utility of our predictor
instruments in terms of how well they enable us to account for
variance in this domain. Campbell (1990), for example, defined
performance as follows:
Performance is behavior. It is something that people do and is re-
flected in the actions that people take. . .Performance is not the con-
sequence(s) or result(s) of action; it is the action itself. . . For any job,
there are a number of major performance components, distinguishable
in terms of their determinants and covariation patterns with other
variables. The correlations among their true scores are less than one.
(p. 704)
These views are extended in a later publication:
Performance is. . .something that people actually do and can be ob-
served. By definition, it includes only those actions or behaviors that
are relevant to the organization’s goals and that can be scaled (mea-
sured) in terms of each person’s proficiency (e.g., level of contribu-
tion). Performance is what the organization hires one to do, and do
well. Performance is not the consequence or result of action, it is the
action itself. . .Performance consists of goal-relevant actions that are
under the control of the individual, regardless of whether they are
cognitive, motor, psychomotor, or interpersonal. (Campbell, McCloy,
Oppler, & Sager, 1993, pp. 40– 41)
Similarly, Rotundo and Sackett (2002, p. 66) suggest that “job
performance is conceptualized as those actions and behaviors that
are under the control of the individual and contribute to the goals
of the organization.”
There have been a number of attempts to develop models of the
criterion domain. Campbell, McHenry, and Wise (1990) described
the performance of entry-level U.S. Army soldiers in terms of five
dimensions: core proficiency, general soldier proficiency, effort
and leadership, personal discipline, and physical fitness and mili-
tary bearing. Campbell et al. (1993) also describe a general model
of work performance consisting of eight factors: job-specific task
proficiency, non-job-specific task proficiency, written and oral
communication, demonstrating effort, maintaining personal disci-
pline, facilitating team and peer performance, supervision and
leadership, and management and administration.
Higher level categorizations of models have been proposed by
Borman and Motowidlo (1993), who distinguish between contex-
tual and task performance, and by J. Hogan and Holland (2003),
who talk of “getting along” and “getting ahead” competencies.
J. Hogan and Holland’s (2003) work is particularly relevant to
the current research in that they emphasized the need to align
predictors and criteria and explored the hypothesis that validities
would increase as one moved from broad multiple-construct cri-
teria (like OJP ratings) to more narrow single-construct criteria. In
their research, this was done using predicted relations between
seven personality constructs measured using the Hogan Personal-
ity Inventory (HPI) and two criteria: getting along and getting
ahead. The HPI scales Adjustment, Prudence, and Ambition all
provided good prediction of both criteria. HPI Likability predicted
getting along (contextual performance), whereas HPI Intellectance
was more related to getting ahead (task performance).
Scullen, Mount, and Judge (2003) found that general-level fac-
tor models (like the task vs. contextual performance model) fit 360
ratings data less well than more differentiated models. In particu-
lar, they showed a better fit for a four-factor model (technical
skills, administrative skills, human skills, and citizenship behav-
iors). This supports the view that it may be necessary to use more
differentiated models to fully capture the variance in the criterion.
Clearly, a balance is needed between highly differentiated models
that may not be generalizable and overly broad constructs that fail
to capture relevant general dimensions of performance. Confirma-
tory factor analyses have supported the distinction between task
and contextual or citizenship behaviors (Conway, 1996; Johnson,
2001), and other work has shown that, although task performance
is better predicted by ability than personality, the reverse is the
case with citizenship behavior (Borman, Penner, Allen, & Moto-
widlo, 2001; Hurtz & Donovan, 2000). Work by Borman, Buck,
Hanson, Motowidlo, Stark, and Drasgow (2001) classified a wide
range of citizenship behaviors into three main subcategories: per-
sonal support (helping, motivating, cooperating with and showing
consideration for others), organizational support (showing loyalty,
following rules and regulations, representing the organization in a
positive light), and conscientious initiative (engaging in self-
1186
BARTRAM
development, taking initiatives, persistence, and making extra ef-
fort to complete tasks). The latter is clearly related to conscien-
tiousness and is likely to facilitate both task and contextual
performance (Motowidlo, Borman, & Schmit, 1997).
The key point from all this work is the demonstration that
differentiating the criterion, if only into two broad areas, provides
a considerable gain in the clarity of how personality-based predic-
tors relate to performance. By doing this, one may capture criterion
specificity effects that are lost when OJP is used as the sole
criterion and treated as if it represented the same construct from
study to study.
The current study proposes disaggregating the criterion further
than this. Clearly, the degree to which it is possible to articulate the
criterion space into distinct factors will be dependent on the range
of measures available. If the only datum available is an OJP rating,
then all we can do is ask about how reliable or relevant that is.
Competency assessment by line managers and others typically
involves a wide range of measures and, therefore, provides the
potential for maximizing differentiation of the criterion space.
A Generic Competency Framework
The process adopted in developing the generic competency
framework used in this study (Bartram et al., 2002; Kurz &
Bartram, 2002) was similar to that described in Tett, Guterman,
Bleier, and Murphy (2000), who identified 53 dimensions of job
performance in managerial jobs from 12 published and practitioner
models. The work done by myself and my colleagues was based on
an analysis of a wide range of published and practitioner models.
With a definition of competencies as “sets of behaviors that are
instrumental in the delivery of desired results or outcomes” (Bar-
tram et al., 2002, p. 7), the resulting framework distinguishes 112
component competencies at the finest level of detail. These com-
ponents are clusters of similar workplace behavior, which, in
practice, are not found to be further differentiated in competency
models (see Appendix). These components can be thought of as
building blocks that can be aggregated together to produce com-
petencies. Sets of competencies, in turn, form competency models.
Within the current framework, one general purpose model is
defined, with 20 competencies. However, the important level for
the current research is the more general model, which aggregates
the 112 components under eight general factors (Kurz, Bartram, &
Baron, 2004). It is this framework on which the current research is
based. These factors have been labeled the Great Eight because
they appear to occupy a position within the work performance
domain (Table 1) similar to the Big Five in the personality pre-
dictor domain.
The Great Eight structure provides an articulation of the work
performance domain that is consistent with a wide range of models
used by practitioners in competency practice and supported em-
pirically by the way in which competency ratings cluster when
subjected to factor analysis (e.g., Gotoh, 1999; Kurz, 1999; Kurz
et al., 2004). For example, Kurz et al. (2004) report an analysis of
Table 1
Titles and High-Level Definitions of the Great Eight Competencies
Factor
Competency
domain title Competency domain definition
Hypothesized Big Five, motivation,
and ability relationships
a
1 Leading and
Deciding
Takes control and exercises leadership. Initiates action, gives direction,
and takes responsibility.
Need for power and control, extraversion
2 Supporting and
Cooperating
Supports others and shows respect and positive regard for them in
social situations. Puts people first, working effectively with
individuals and teams, clients, and staff. Behaves consistently with
clear personal values that complement those of the organization.
Agreeableness
3 Interacting and
Presenting
Communicates and networks effectively. Successfully persuades and
influences others. Relates to others in a confident, relaxed manner.
Extraversion, general mental ability
4 Analyzing and
Interpreting
Shows evidence of clear analytical thinking. Gets to the heart of
complex problems and issues. Applies own expertise effectively.
Quickly takes on new technology. Communicates well in writing
General mental ability, openness to new
experience
5 Creating and
Conceptualizing
Works well in situations requiring openness to new ideas and
experiences. Seeks out learning opportunities. Handles situations and
problems with innovation and creativity. Thinks broadly and
strategically. Supports and drives organizational change.
Openness to new experience, general
mental ability
6 Organizing and
Executing
Plans ahead and works in a systematic and organized way. Follows
directions and procedures. Focuses on customer satisfaction and
delivers a quality service or product to the agreed standards.
Conscientiousness, general mental ability
7 Adapting and
Coping
Adapts and responds well to change. Manages pressure effectively and
copes well with setbacks.
Emotional stability
8 Enterprising and
Performing
Focuses on results and achieving personal work objectives. Works best
when work is related closely to results and the impact of personal
efforts is obvious. Shows an understanding of business, commerce,
and finance. Seeks opportunities for self-development and career
advancement.
Need for achievement, negative
agreeableness
Note. More detailed definitions of each of the Great Eight are provided by the competency component level of the SHL Universal Competency
Framework™ (see Appendix).
a
Where more than one predictor is shown, the second is expected to be of lesser importance than the first. The competency titles and definitions are taken
from the SHL Universal Competency Framework™ Profiler and Designer Cards (copyright © 2004 by SHL Group plc, reproduced with permission of the
copyright holder). These titles may be freely used for research purposes subject to due acknowledgment of the copyright holder.
1187
A CRITERION-CENTRIC APPROACH TO VALIDATION
data from 365 managers drawn from four U.K. organizations in
different industry sectors and from a wide range of functional
areas. Two aptitude tests from the SHL Management and Graduate
Item Bank were used: Verbal Critical Reasoning (VMG2) and
Numerical Critical Reasoning (NMG2). In addition, two question-
naires were administered: the Occupational Personality Question-
naire (Concept Model, OPQ CM4.2; SHL Group, 1993b), which
measures 30 scales in an ipsative (forced-choice) format, and the
Inventory of Management Competencies (IMC; SHL Group,
1993a), which measures 16 generic competency dimensions using
a combined Likert-type and forced-choice format. A composite
180° performance score for each IMC competency was calculated
by averaging self- and boss ratings. These 16 performance scores
were entered together with selected marker scales from the OPQ
and the ability scales into a principal-components factor analysis to
ascertain the underlying factor structure of the competency per-
formance and competency potential variables. Eight factors
reached an eigenvalue greater than 1, accounting for 68% of the
variance. The factor structure was largely in line with that pre-
dicted by the Great Eight model. Factor scores generated from
these data were relatively independent; the strongest correlations
were just over 0.3. Other analyses on different data sets have
produced similar degrees of fit to the model (e.g., Gotoh, 1999;
Kurz, 1999). Multidimensional Scaling analyses of criterion IMC
competency ratings produce a circumplex pattern, with pairs of
scales clustering together according to their proximity as defined
by their expected Great Eight content loadings.
Factor analyses of Great Eight predictor or criterion data sets
tend to produce higher level solutions. In the predictor domain,
these reflect a broad motivational factor, general ability, and two
broad personality factors (Digman, 1997). For the criterion do-
main, two factors tend to emerge corresponding generally to Hog-
an’s getting along and getting ahead distinction or Borman’s
contextual and task performance distinction.
The choice of the Great Eight as the level of analysis (rather
than either higher level constructs or more detailed competency
models) was driven by the need to provide a degree of differenti-
ation of the criterion space that reflected the range of attributes that
managers and practitioners distinguish in practice, while retaining
sufficient generality to enable the same model to be applied across
a wide range of studies involving diverse competency models and
predictor instruments.
Our approach to validation views predictor instruments as valu-
able insofar as they can provide valid measures of competency
potential. The current research concerns the relationship between
measures of competency potential (based on personality and abil-
ity tests) and competencies (as assessed through supervisor ratings
of performance on various aspects of a competency model), with
both sets of measures being mapped onto the generic Great Eight
level of description.
Table 1 gives the “headline” definitions of the Great Eight
competency factors. Further details can be obtained by studying
the competency components that underpin each high-level factor
(see Appendix). Table 1 also summarizes a range of hypotheses
concerning the expected relationships between high level factors in
the predictor domain and the Great Eight competencies. These
hypotheses are based on an analysis of the content of the compe-
tencies and judged relevance of the various underlying traits.
General mental ability is expected to relate most strongly to
Analyzing & Interpreting competencies, because the content of
these are heavily loaded on general mental ability (“g”). On the
basis of the content of the competencies, we would also expect
general mental ability measures to correlate with the adjacent
factors of Presenting & Interacting as well as Creating & Concep-
tualizing. It is these three areas of competency, especially Analyz-
ing & Interpreting, that contain competency components that are
underpinned by job knowledge and skills and hence are most likely
to be well predicted by measures of general mental ability (see
Appendix, especially the Applying Expertise & Technology and
Writing & Reporting components). Organizing & Executing is also
likely to be correlated with general ability, but more weakly than
the other three competency factors.
In relation to personality, the relationship between the Great
Eight and the Big Five is not exact. A clearer pattern of results can
be obtained if some aspects of the Big Five are differentiated.
Notably, Big Five Conscientiousness combines both dependability
and achievement; and Extraversion combines aspects of interact-
ing with others on the one hand and dominance or potency on the
other (Hough, 1992; Hough, Ones, & Viswesvaran, 1998). In the
Great Eight competency model, we would expect the trait of
dependability to relate to Organizing & Executing while achieve-
ment relates to Enterprising & Performing; the trait of sociability
should relate to Interacting & Presenting, while dominance and
potency aspects of extraversion should relate to Leading & Decid-
ing and Interacting & Presenting competencies.
Although there is clearly some overlap between the current
model and that described by Campbell et al. (1993), the current
model has the advantage of being elaborated in terms of the 112
component competencies, which, in turn, are linked both to com-
petency assessment measures and to personality, motivation, and
ability scales in the predictor domain. As such, the model provides
a single framework for making predictions from measures of
competency potential (ability, personality, and motivation) to rat-
ings of actual work performance.
Hypotheses
On the basis of the rationale underlying the mapping of com-
petency ratings and competency potential measures to the Great
Eight model and from the results of previous meta-analysis re-
search, summarized previously, the data were examined to explore
a number of hypotheses:
Hypothesis 1: Correlations between matched pairs of Great
Eight competency ratings and competency potential scores
will be higher than for unmatched pairs.
For example, the correlation between the prediction of Leading
& Deciding, based on personality measures, and managerial rat-
ings of Leading & Deciding competencies should be higher than
the correlations between this predictor and managerial ratings on
any of the other seven Great Eight competencies.
Hypothesis 2: Personality-based predictors will show nonzero
relationships with all eight of the areas of line manager’s
competency ratings, whereas ability-based predictors will
only relate to those areas of competency that are underpinned
by job knowledge and skill acquisition.
1188
BARTRAM
In terms of the Great Eight model, we expect the strongest
association with ability measures to be found for Analyzing &
Interpreting and Creating & Conceptualizing. Less strong relation-
ships would be expected for Interacting & Presenting and Orga-
nizing & Executing.
Method
A total of 29 studies were collected from various client organizations.
They came from the United Kingdom and a number of other European
countries, Turkey and the Middle East, South Africa, the Far East, and the
United States and cover a wide range of different industry sectors and jobs
(although supervisory and managerial positions predominate). The 29
studies, only 5 of which have been included in any previous published
validation reports (Nyfield et al., 1995), have a total sample size of 4,861
people (Mdn ⫽ 125/study). Criterion data consist of line-manager ratings
on either standardized work performance competency instruments or
client-specific measures.
Nineteen of the 29 studies used one of several standardized competency
rating instruments. The Inventory of Management Competencies (IMC;
SHL Group, 1993a) includes 16 competency scales and 160 items. The
items are presented in sets of four. Each item has to be rated on a scale
ranging from 1 to 5, and the rater also has to select which of each set of four
items is most true and least true of the target subject. IMC scoring involves
a weighted sum of the scores obtained from the rating and forced-choice
measures. The Customer Contact Competency Inventory (CCCI; SHL
Group, 1997) also has 16 competency scales and follows the same mixed-
item style format as IMC. It has a total of 128 items arranged in 32 quads.
The Work Styles Competency Inventory (WSCI; SHL Group, 1999b) has
16 competency scales based on 96 items, which are responded to on a
rating scale ranging from 1 to 5. Perspectives on Management Competen-
cies (PMC; SHL Group, 1994) has 36 competency scales based on 144
items, which are responded to on a rating scale ranging from 1 to 5. These
were all developed for use as self- and other-rating instruments of perfor-
mance at work. For the current study, only line-manager ratings were used
as data. Ten of the 29 studies used client-specific rating instruments.
The predictor measures include a number of different instruments from
the Occupational Personality Questionnaire (OPQ) family. OPQ Concept
Model (30 scales plus Consistency or Social Desirability scale; SHL
Group, 1993b) is the precursor to OPQ32 (32 scales plus Consistency or
Social Desirability scale; SHL Group, 1999a). OPQ CM4.2 and OPQ32i
are the forced-choice format versions of OPQ Concept Model and OPQ32,
whereas OPQ CM5.2. and OPQ32n are the Likert-type versions. The
Customer Contact Styles Questionnaire (CCSQ; SHL Group, 1997) is a
version of OPQ designed for use in customer service and sales settings and
is available in both a Likert-type rating scale (CCSQ5.2: 16 scales plus
Social Desirability scale) and a combined rating and forced-choice
(CCSQ7.2: 16 scales plus Consistency scale) form. Finally, the Work
Styles Questionnaire (SHL Group, 1999b) is a Likert-type questionnaire
for use in production and lower level service positions (WSQn: 17 scales
plus Social Desirability scale). In addition to these personality instruments,
a wide range of ability tests (both verbal and numerical reasoning) were
used. For the purposes of this study, these are treated as equivalent
measures of either verbal or numerical reasoning.
All the studies have personality data from one or another of the instru-
ments just discussed. Eighteen of the 29 studies have data relating to one
or more ability tests. Seven of the studies were predictive validity designs,
and 21 were concurrent; no details were available for 1. Total samples sizes
and data about the studies, job types, industry sectors, and other demo-
graphics are presented in Tables 2–6.
Mapping Scales to the Competencies
A standard set of mappings of personality and ability scales onto the
Great Eight competencies had been defined for each of the predictor
instruments (personality and ability tests) and for the standardized compe-
tency rating instruments (IMC, CCCI, PMC, and WSCI). These mappings
are in the form of linear weighted composites and were defined a priori on
the basis of ratings of construct overlap between traits (individual scales)
and each of the 112 competency components in the framework. As de-
scribed earlier, in this framework each of the Great Eight competency
factors is composed of a subset of the 112 competency components. The
relevance of each competency component for each personality scale had
been initially estimated using independent judgments of three subject
matter experts. Each possible scale– component relationship was rated on a
scale ranging from 0 (not relevant)to4(highly relevant). (The scale was
anchored to expectations of correlations between scales and items of
between 0 and 0.40 for the 0 to 4 ratings.) Judgments were reviewed and
a final set of component relevancies agreed on. The outcome of this process
is a large matrix of components by scales, populated by relevance ratings
in the range of ⫺4to⫹ 4. Scale weights for producing Great Eight scores
from, for example, OPQ32 scales were then defined by aggregation of the
relevant sets of component weights into a single linear equation.
The validation of this approach is described in Bartram (2001). Further
research has also demonstrated the robustness of this approach of aggre-
gating measures at the component level. For the current study, aggregation
of scale– component relations up to the Great Eight factor level was used
to provide the basis for identifying a small number of marker scales
(generally no more than three scales) for each of the eight factors. Typi-
cally, this aggregation process results in a small number of scales having
large weights (i.e., aggregated relevancies) and a larger number having
small weights. Discarding the lower weighted scales reduces the correla-
tions between composites without adversely affecting their validities. The
outcome of this process was that for the OPQ32, for example, each of the
Great Eight competencies is measured by three scales (see Table 7 for list
of marker scales used in predictor measures of each of the Great Eight
competencies), with different scales being used for each of the Great Eight
(i.e., 24 of the 32 available scales are used in all). For WSQ and CCSQ, it
was not possible to avoid some overlap of scale content, given the need to
ensure good coverage of the content of each competency domain. How-
ever, this had relatively little impact on correlations among the final overall
set of eight composites (see Table 8).
It is important to note that the personality instruments used in these
studies have a broader coverage than Big Five instruments. In particular,
the OPQ model covers aspects of motivation and cognitive style as well as
the more traditional areas of personality. For example, Table 7 lists scales
such as Controlling, Achieving, and Competitive. Others that load on
competencies like Leading & Deciding or Enterprising & Performing
include scales such as Persuasive and Energetic. In the area of cognitive
style, the OPQ model includes scales such as Evaluative and Data Rational.
Table 2
Number of Studies and Total Sample Sizes Having Criterion
Data and Predictor Data for Each of the Great Eight
Competencies
Competency
Criteria Predictors
No. studies n Ability Personality
Leading/Deciding 28 3,869 2,697 4,455
Supporting/Cooperating 26 3,778 2,697 4,459
Interacting/Presenting 28 3,874 2,697 4,458
Analyzing/Interpreting 27 3,771 2,697 4,457
Creating/Innovating 21 3,280 2,697 4,455
Organizing/Executing 29 3,971 2,697 4,458
Adapting/Coping 25 3,664 2,697 4,453
Enterprising/Performing 27 3,742 2,697 4,458
No. studies 18 29
1189
A CRITERION-CENTRIC APPROACH TO VALIDATION
For client competency models, mapping to the Great Eight was carried
out post hoc by two subject matter experts on the basis of the content and
the definitions given to the competencies in the client model. From these
definitions, it was possible to map these models to the 20 dimension levels
of the framework and then aggregate the results up to the Great Eight level.
There was generally insufficient detail available on the nature of the
competencies from client models to permit an analysis in terms of the 112
components. Most of the client models covered a subset of the Great Eight
competencies. As a consequence, the total number of studies having data
on each competency factor varies for each of the Great Eight competencies
(see Table 2).
All scale scores (both predictor and criterion) were transformed into z
scores before being weighted and combined into Great Eight scores. The
Table 3
Summary of Individual Study Details
Ref. no. Country N
Predictors
Criteria-manager
ratings
Type of
study
Personality
inventory
Verbal
ability
Numerical
ability
Competency
model OJP
1 UK 128 OPQ CM4.2 VMG2 NMG2 IMC Y Concurrent
2 UK 35 OPQ CM4.2 VMG2 NMG2 IMC Y Concurrent
3 UK 92 OPQ CM4.2 VMG2 NMG2 IMC Y Concurrent
4 UK 139 OPQ CM4.2 VMG2 NMG2 IMC Y Concurrent
5 UK 68 OPQ CM4.2 VMG2 NMG2 IMC Y Concurrent
6 Belgium 83 OPQ CM4.2 IMC Predictive
7 UK 88 OPQ CM4.2 VA3 NA4 Client Concurrent
8 UK 449 OPQ CM4.2 IMC Y Predictive
9 S Africa 72 OPQ CM4.2 VC1.1 NC2.1 IMC Predictive
10 USA 175 OPQ CM4.2 VMG1 IMC Y Concurrent
11 Turkey 503 OPQ CM4.2 VMG1 NMG1 IMC Y Concurrent
12 France 491 OPQ32i IMC Concurrent
13 Korea 366 OPQ CM4.2 VMG1 NMG1 IMC Y Concurrent
14 Germany 132 CCSQ 7.2 CCCI Y Concurrent
15 S Africa 165 CCSQ 5.2 VCC3 CCCI Concurrent
16 UK/USA 236 OPQ32n PMC Concurrent
17 UK 54 CCSQ 5.2 CCCI Concurrent
18 UK 53 CCSQ 5.2 CCCI Concurrent
19 UK 61 WSQn WSCI No details
20 UK 64 WSQn WSCI Concurrent
21 UK 101 OPQ CM5.2 Client Concurrent
22 UK 42 OPQ CM4.2 VC1.1 NMG1 Client Predictive
23 Belgium 166 OPQ CM4.2 Client Predictive
24 UK/Egypt 60 OPQ32i VMG1 NMG1 Client Y Concurrent
25 UK 133 OPQ CM5.2 VMG3 NMG3 Client Y Concurrent
26 Netherlands 543 OPQ CM4.2 VA1 NIT2 Client Y Predictive
27 UK 144 CCSQ 5.2 VCC3 NCC4 Client Predictive
28 UK 93 OPQ CM4.2 VMT1 NMG2 Client Concurrent
29 UK 125 OPQ CM4.2 VMG1 NMG1 Client Concurrent
Note. See text for further information on the competency models and predictor measures. Complete data were not available for all samples. OJP ⫽ overall
job performance ratings; N ⫽ maximum number in each sample; Y ⫽ yes; IMC ⫽ Inventory of Management Competencies; CCCI ⫽ Customer Contact
Competency Inventory; PMC ⫽ Perspectives on Management Competencies; WSCI ⫽ Work Styles Competency Inventory.
Table 4
Sample Breakdown by Country
Country Frequency Percentage
Belgium 249 5.12
Egypt 60 1.23
France 491 10.10
Germany 132 2.72
Holland 543 11.17
Korea 366 7.53
Saudi Arabia 237 4.87
Turkey 503 10.35
United Kingdom 1,869 38.45
United States 411 8.46
Total 4,861 100.00
Table 5
Sample Breakdown by Job Type
Job type Frequency Percentage
Account executives 144 2.96
Broker consultants 165 3.39
Collections staff 54 1.11
Executives 166 3.41
Managers 3,243 66.72
Sales reps 132 2.72
Sales staff 53 1.09
Shop floor staff 64 1.32
System developers 236 4.86
Trainees 543 11.17
Unknown 61 1.25
Total 4,861 100.00
1190
BARTRAM
Great Eight scores were then transformed into z scores. In each case,
standard score transformations were carried out using all those people in
the combined set of studies having data on the respective measures.
OJP Ratings
Some of the studies also had data on ratings of OJP (13 studies). In most
cases, these ratings used a six-item rating instrument used in the Interna-
tional Validation Study reported by Nyfield et al. (1995). The reliability of
this scale was 0.70.
Meta-Analysis Procedures
The procedures used in the meta-analysis were as described by Hunter
and Schmidt (1990); individual corrections were carried out on each study.
Meta-analysis adjustments included corrections for criterion reliability and
predictor range restriction. The criterion reliability of the Great Eight
competencies based on the standardized competency instruments was set at
0.75, whereas that for client competency models was set at 0.52. The latter
value is taken from Hermelin and Robertson’s (2001) artifact distribution
estimates, whereas the former takes account of published data on the
reliability of ratings using the standardized instruments IMC (SHL Group,
1993a), PMC (SHL Group, 1994), and CCCI (SHL Group, 1997). Range
restriction was calculated for each study on the basis of differences be-
tween the composite score predictor standard deviations for the selected
samples and the standard deviations for the relevant applicant groups when
these data were available. When applicant data were not available (about
50% of the cases), a relevant norm group was used (i.e., one based on
similar applicant samples rather than on the general population) to estimate
degree of range restriction for the underlying scales of the composites.
Because range restriction tended to be uniform across scales for personal-
ity, range restrictions for composites in this case were estimated on the
basis of the average range restrictions across the relevant scales. The
average ratio of restricted standard deviations to unrestricted standard
deviations was 0.80 (0.045 SD across studies) for the personality measures
and 0.80 (0.062 SD across studies) for ability tests.
Results
Relationships Between Variables Within the Personality-
Based Predictor Set and the Criterion Set
The average correlation between the eight predictor composites
was .07, whereas for the criteria it was 0.45 (see Table 8 for the
Table 6
Sample Breakdown by Industry Sector
Industry sector Frequency Percentage
Banking 186 3.83
Call center 107 2.20
Distribution 139 2.86
Engineering 366 7.53
Finance 92 1.89
Food manufacturing 491 10.10
Hospitality 328 6.75
Insurance 309 6.36
Information technology 543 11.17
Manufacturing 870 17.90
Pharmaceuticals 88 1.81
Public sector 165 3.39
Retail 265 5.45
Telecommunications 236 4.86
Unknown 61 1.25
Various 615 12.65
Total 4,861 100.00
Table 7
Personality Inventory Scales Used to Generate Competency Potential Scores
G8 factor Competency domain title OPQ32 OPQ CM CCSQ WSQ
1 Leading & Deciding Controlling, Persuasive, Decisive Controlling, Persuasive, Decisive Persuasive, Results Oriented Assertive, Achieving, Decisive
2 Supporting & Cooperating Caring, Democratic, Affiliative Caring, Democratic, Affiliative Empathic, Participative, Self-
Control
Considerate, Team-Oriented,
Dependable
3 Interacting & Presenting Socially Confident, Outgoing,
Modest (-ve)
Socially Confident, Outgoing,
Modest (-ve)
Sociable, Modest (-ve), Persuasive Socially Confident, Assertive,
Adaptable
4 Analyzing & Interpreting Evaluative, Data Rational,
Conceptual
Critical, Data Rational, Conceptual Analytical, Structured, Innovative Practical, Organized, Innovative
5 Creating & Conceptualizing Innovative, Independent,
Conventional (-ve)
Innovative, Independent,
Traditional (-ve)
Innovative, Flexible, Analytical Innovative, Adaptable, Achieving
6 Organizing & Executing Conscientious, Detail Conscious,
Forward Planning
Conscientious, Detail Conscious,
Forward Planning
Conscientious, Detail Conscious,
Structured
Dependable, Detail Conscious,
Organized
7 Adapting & Coping Tough-minded, Relaxed, Optimistic Tough-minded, Relaxed,
Optimistic
Resilience, Self-Control, Flexible Resilient, Emotionally
Controlled, Optimistic
8 Enterprising & Performing Achieving, Competitive, Vigorous Achieving, Competitive, Active Results Oriented, Competitive,
Energetic
Achieving, Competitive, Active
Note. In each case, the first scale received a weight of two and the other scale(s) were unit weighted. In some instances for the WSQ and CCSQ, the same scales load on two Great Eight competencies.
Although this introduces a necessary degree of correlation between these Great Eight predictors (see Table 8), the overlap was included to ensure sufficient breadth of coverage of the relevant constructs.
OPQ ⫽ Occupational Personality Questionnaire; CM ⫽ Concept Model; CCSQ ⫽ Customer Contact Styles Questionnaire; WSQ ⫽ Work Styles Questionnaire.
1191
A CRITERION-CENTRIC APPROACH TO VALIDATION
complete matrix). The overall positive correlation for the criteria is
not surprising because these are line-manager rating data. How-
ever, this analysis does show that there is a good degree of
independence between the eight predictors. For the predictors, the
main correlations are between Analyzing & Presenting and Cre-
ating & Innovating on the one hand (r ⫽ .33) and between Leading
& Deciding, Interacting & Presenting, and Enterprising & Per-
forming on the other (average r ⫽ .33).
In line with previous work, principal-components analyses of
the predictor and criterion matrices produced three- and two-factor
solutions, respectively. For the predictors the three factors ac-
counted for 55.48% of the variance, and for the criteria the two
factors accounted for 65.03%. Varimax-rotated loadings are shown
in Table 9. For the criteria, these factors broadly represent task
(Factor 1) versus contextual behaviors (Factor 2), whereas for the
predictors the meaning of the factors is less clear.
The Validity of Personality Data for the Prediction of
Supervisor-Rated Competencies
Sample-weighted average predictor– criterion correlations for
each of the cells of the 8 ⫻ 8 matrix of predictors and competen-
cies across all the studies are shown in Table 10. The diagonal cells
are the hypothesized relationships (Hypothesis 1). It can be seen
that these are all nonzero and are the largest values in each row or
column in all cases but one. For the personality-based predictor of
Leading & Deciding, the correlation with Leading & Deciding
competency ratings is equal to that with Interacting & Presenting
and only slightly higher than that with Enterprising & Performing.
The results presented in Table 10 support Hypothesis 1, because
the average correlation of the hypothesized relationship is .16,
whereas the average correlation of the nonhypothesized relation-
ships (the off-diagonal cells in Table 10) is .02.
These results can be compared to use of the Big Five as a
framework for structuring the predictor scales. Standard Big Five
equations for OPQ32 and OPQ CM were applied to these instru-
ments to produce Big Five predictor scores. These equations are
based on correlations between OPQ and the NEO Personality
Inventory–Revised (NEO-PI-R; SHL Group, 1999a). Table 11
shows the correlations between Big Five and Great Eight predic-
tors of competency potential and provides an empirical underpin-
ning to the assignments noted in Table 1. There is close concor-
dance among Extraversion, Agreeableness, Conscientiousness,
and Neuroticism (negative) on the one hand and Interacting &
Presenting, Supporting & Cooperating, Organizing & Executing,
and Adapting & Coping on the other. Extraversion is also corre-
lated with Leading & Deciding, whereas Agreeableness correlates
negatively with Enterprising & Performing. Openness to New
Experience correlates with both Analyzing & Interpreting and
Creating & Conceptualizing. In general, these results show that the
Big Five relate to the Great Eight in the manner expected (as
described in Table 1).
Table 8
Average Intercorrelations (Uncorrected) for Great Eight Competency Manager Ratings (Above
Diagonal) and Great Eight Predictors Based on Personality Data Only (Below Diagonal)
Variable L/D S/C I/P A/I C/C O/E A/C E/P
L/D — 0.41 0.57 0.47 0.52 0.56 0.47 0.61
N 3,798 3,895 3,873 3,391 3,992 3,684 3,769
S/C ⫺0.08 — 0.43 0.28 0.26 0.37 0.51 0.36
N 4,846 4,016 3,814 3,522 4,012 3,848 3,783
I/P 0.44 0.13 — 0.50 0.47 0.37 0.40 0.48
N 4,845 4,849 3,912 3,606 4,193 3,987 3,882
A/I 0.09 ⫺0.16 ⫺0.04 — 0.58 0.56 0.31 0.50
N 4,844 4,848 4,847 3,403 4,008 3,700 3,897
C/C 0.27 ⫺0.11 0.20 0.33 — 0.46 0.35 0.55
N 4,842 4,846 4,845 4,844 3,600 3,516 3,372
O/E ⫺0.07 0.01 ⫺0.16 0.20 ⫺0.05 — 0.35 0.56
N 4,845 4,849 4,848 4,847 4,845 3,983 3,980
A/C 0.04 0.04 0.08 ⫺0.09 0.05 0.00 — 0.43
N 4,840 4,844 4,843 4,842 4,840 4,843 3,753
E/P 0.36 ⫺0.23 0.20 0.09 0.23 0.09 ⫺0.04 —
N 4,845 4,849 4,848 4,847 4,845 4,848 4,843
Note. L/D ⫽ Leading/Deciding; S/C ⫽ Supporting/Cooperating; I/P ⫽ Interacting/Presenting; A/I ⫽ Analyz-
ing/Interpreting; C/C ⫽ Creating/Conceptualizing; O/E ⫽ Organizing/Executing; A/C ⫽ Adapting/Coping;
E/P ⫽ Enterprising/Performing.
Table 9
Varimax Rotated Loadings From Principal-Components
Analyses of the Eight Predictors and Eight Criteria
Variable
Predictors: rotated
factor loadings
Criteria:
rotated
factor
loadings
12312
Leading/Deciding .79 ⫺.07 .02 .68 .44
Supporting/Cooperating ⫺.16 ⫺.10 .75 .15 .86
Interacting/Presenting .69 ⫺.27 .34 .57 .45
Analyzing/Interpreting .20 .74 ⫺.18 .82 .08
Creating/Conceptualizing .60 .36 ⫺.03 .80 .11
Organizing/Executing ⫺.17 .73 .15 .73 .26
Adapting/Coping .11 .08 .59 .23 .80
Enterprising/Performing .60 .12 ⫺.32 .73 .33
1192
BARTRAM
Table 12 indicates that Big Five predictors have the expected
pattern of correlations with the Great Eight competencies. As one
would expect, however, they do not fit as well as the predictors
designed specifically to measure the competencies on a one-to-one
basis. The average of the absolute values of the correlations
predicted for the five Great Eight criteria hypothesized to relate
most strongly to the Big Five is .11, which is somewhat lower than
the average correlation for the Great Eight (.16) for the pairwise
predictions shown in Table 10. The Big Five show much weaker
predictions of Leading & Deciding, Enterprising & Performing,
and Analyzing & Interpreting. As noted earlier, this is likely
because OPQ instruments provide coverage of aspects of motiva-
tion (specifically, need for achievement and need for power and
control) and cognitive style that lie outside the Big Five domain.
Meta-Analysis of the Great Eight Predictor–Criterion
Pairs for Personality-Based Predictors
Corrections for criterion reliability and range restriction were
carried out only on the hypothesized correlations (i.e., those shown
in the diagonal of Table 10). None of the off-diagonal (see Table
10) correlations were included in the meta-analyses. The resulting
corrected correlations (Table 13) for the personality-based predic-
tions of the Great Eight are moderate to high for all eight factors,
ranging from 0.16 to 0.28. The largest values obtained are, in fact,
for the motivation- and extraversion-driven factors (Leading &
Deciding, Interacting & Presenting, and Enterprising & Perform-
ing competencies).
The Validity of Ability Test Data as Predictors of
Supervisor-Rated Competencies
Ability tests were predicted to correlate most strongly with
Analyzing & Interpreting competencies and to show relationships
with Creating & Conceptualizing and with Interacting & Present-
ing. The results of the meta-analysis carried out on ability tests
only are presented in Table 13 and clearly show the expected
pattern. The strongest relationship (
⫽ .40) is, as expected, with
Analyzing & Interpreting competencies, which are strongly under-
pinned in the competency framework by components relating to
Table 11
Correlations Between Big Five and Great Eight Personality-Based Composite Predictors
Personality-based Great
Eight predictor Example OPQ32 marker scales
Big Five predictors
Extraversion Agreeableness Openness Conscientiousness Neuroticism
Leading/Deciding Controlling, Decisive 0.37 ⫺0.15 0.04 ⫺0.12 ⫺0.13
Supporting/Cooperating Caring, Affiliative 0.17 0.90 ⫺0.05 ⫺0.06 0.06
Interacting/Presenting Socially Confident, Outgoing 0.89 0.01 0.09 ⫺0.22 ⫺0.17
Analyzing/Interpreting Evaluative, Data Rational ⫺0.10 ⫺0.18 0.39 0.16 0.06
Creating/Conceptualizing Innovative, Independent 0.17 ⫺0.18 0.61 ⫺0.12 ⫺0.10
Organizing/Executing Conscientious, Forward Planning ⫺0.23 0.00 ⫺0.10 0.96 0.06
Adapting/Coping Tough-minded, Relaxed ⫺0.02 0.02 ⫺0.11 ⫺0.06 ⴚ0.86
Enterprising/Performing Achieving, Competitive 0.13 ⴚ0.41 0.01 0.06 0.01
Note. N varies by rows from 4,222 to 4,226. OPQ32 ⫽ Occupational Personality Questionnaire-32. Boldface numbers indicate hypothesized Great
Eight–Big Five relationships.
Table 10
Sample-Weighted Average Correlations (Uncorrected) Between Great Eight Competency Boss Ratings and Predictors Based on
Personality Data Only, With Hypothesized (Diagonal) and Nonhypothesized (Mean Off-Diagonal) Values
Predictor
Example OPQ32
marker scales
Competency ratings
L/D S/C I/P A/I C/C O/E A/C E/P Hypothesized Nonhypothesized
L/D Controlling, Decisive 0.18 ⫺0.02 0.17 0.04 0.11 0.02 0.00 0.17 0.18 0.07
S/C Caring, Affiliative ⫺0.02 0.11 ⫺0.03 ⫺0.10 ⫺0.10 ⫺0.04 0.03 ⫺0.07 0.10 ⫺0.04
I/P Socially Confident,
Outgoing 0.11 0.06 0.19 0.00 0.08 ⫺0.03 0.04 0.08 0.19 0.05
A/I Evaluative, Data
Rational 0.02 ⫺0.07 0.02 0.16 0.12 0.07 ⫺0.03 0.02 0.16 0.02
C/C Innovative, Independent 0.07 ⫺0.03 0.08 0.09 0.18 0.00 ⫺0.02 0.06 0.18 0.04
O/E Conscientious, Forward
Planning 0.00 ⫺0.03 ⫺0.07 0.05 ⫺0.04 0.15 ⫺0.02 0.03 0.14 ⫺0.01
A/C Tough-minded, Relaxed 0.00 0.02 ⫺0.02 ⫺0.01 ⫺0.01 ⫺0.01 0.12 ⫺0.01 0.12 ⫺0.01
E/P Achieving, Competitive 0.11 ⫺0.03 0.10 0.07 0.11 0.04 0.02 0.19 0.19 0.06
M 0.16 0.02
Note. Number of studies ⫽ 29; sample size range ⫽ 3,280–3,971. OPQ32 ⫽ Occupational Personality Questionnaire-32. L/D ⫽ Leading/Deciding; S/C ⫽
Supporting/Cooperating; I/P ⫽ Interacting/Presenting; A/I ⫽ Analyzing/Interpreting; C/C ⫽ Creating/Conceptualizing; O/E ⫽ Organizing/Executing;
A/C ⫽ Adapting/Coping; E/P ⫽ Enterprising/Performing.
1193
A CRITERION-CENTRIC APPROACH TO VALIDATION
job knowledge and job skills. Overall, there are relationships
between ability and the middle four competencies (from Interact-
ing & Presenting to Organizing & Executing). As expected (Hy-
pothesis 2), the first and last pairs of Great Eight competencies (see
Table 1) show no relationships with ability.
The Validity of Combined Personality and Ability Test
Data as Predictors of Supervisor-Rated Competencies
Personality and ability test predictions were combined using
regression weights (see Table 14). Regression analyses, with com-
posite personality and composite ability measures as the two
predictors and competency ratings as the criterion, were performed
for each of the Great Eight for those cases in which there were data
on both personality and ability tests. In cases in which there were
verbal and numerical reasoning tests (rather than just one or the other),
an equal-weighted composite of these was used as the estimate of
general ability. For ability tests there is, of course, only one predictor
for each of the Great Eight competencies, whereas for the personality
data each of the Great Eight has a distinct personality-based predictor.
It was decided not to use the standardized regression weights given in
Schmidt and Hunter (1998), because these relate to the effect of
adding measures of conscientiousness to general mental ability and
relate to validity studies in which the criteria were overall measures of
job performance. The ratio of ability to personality regression weights
from Schmidt and Hunter is 1.65. For the current study, it is 1.85 for
Analyzing & Interpreting (the most g-loaded competency) but less
than 1.0 for the rest of the Great Eight.
Table 12
Average Correlations Between Big Five Personality-Based Predictors and Great Eight Competency Criteria
Variable
Great Eight competency criterion ratings
Leading/
Deciding
Supporting/
Cooperating
Interacting/
Presenting
Analyzing/
Interpreting
Creating/
Conceptualizing
Organizing/
Executing
Adapting/
Coping
Enterprising/
Performing
N 3,236 3,142 3,309 3,221 2,757 2,845 3,103 3,200
Big Five predictors
Extraversion 0.09 0.06 0.18 0.00 0.07 ⫺0.05 0.00 0.09
Agreeableness ⫺0.01 0.09 ⫺0.05 ⫺0.06 ⫺0.08 ⫺0.01 0.01 ⴚ0.07
Openness 0.01 ⫺0.03 0.04 0.09 0.13 ⫺0.02 ⫺0.02 ⫺0.01
Conscientiousness 0.00 ⫺0.06 ⫺0.07 0.06 ⫺0.06 0.15 ⫺0.03 ⫺0.01
Neuroticism ⫺0.01 ⫺0.01 ⫺0.01 0.00 ⫺0.02 0.04 ⴚ0.09 ⫺0.02
Note. Boldface numbers indicate hypothesized Great eight–Big five relationships.
Table 13
Meta-Analysis Results for Great Eight Competencies Based on Personality-Only Predictors and
Ability-Only Predictors for All Those Cases Having Data on Either One or the Other or Both
Sets of Measures
Predictor No. studies Nr
SD
%var 10% CrI
Personality only
Leading/Deciding 28 3,595 0.181 0.267 0.054 84.98 0.197
Supporting/Cooperating 26 3,470 0.109 0.161 0.159 38.97 ⫺0.043
Interacting/Presenting 28 3,568 0.187 0.277 0.134 47.12 0.105
Analyzing/Interpreting 27 3,554 0.164 0.244 0.125 50.64 0.084
Creating/Conceptualizing 21 2,994 0.182 0.260 0.079 67.93 0.159
Organizing/Executing 29 3,670 0.150 0.218 0.177 35.06 ⫺0.009
Adapting/Coping 25 3,366 0.122 0.175 0.108 57.67 0.036
Enterprising/Performing 27 3,561 0.189 0.275 0.061 80.54 0.197
Ability only
Leading/Deciding 18 2,166 0.044 0.075 0.190 35.62 ⫺0.168
Supporting/Cooperating 16 2,128 0.008 0.024 0.102 64.11 ⫺0.107
Interacting/Presenting 18 2,170 0.147 0.216 0.159 42.75 0.012
Analyzing/Interpreting 17 2,131 0.276 0.404 0.198 26.70 0.150
Creating/Conceptualizing 12 1,730 0.172 0.241 0.096 59.88 0.118
Organizing/Executing 18 2,170 0.105 0.158 0.160 43.76 ⫺0.047
Adapting/Coping 15 1,986 0.050 0.075 0.044 90.31 0.018
Enterprising/Performing 16 2,057 0.029 0.053 0.163 40.63 ⫺0.156
Note. r ⫽ sample-weighted average of uncorrected correlations;
⫽ sample-weighted average of correlations
corrected for artifacts; SD
⫽ standard deviation of corrected correlations; 10%CrI ⫽ lower 10% boundary of
80% credibility interval; %var ⫽ percentage variance accounted for by artifact corrections for corrected
correlation distribution.
1194
BARTRAM
The results of meta-analysis of predictor–criterion relation-
ships conducted using only those studies that included both
personality and ability predictors are presented in Table 15.
This shows the outcome of the analysis of combining person-
ality and ability data and also the results for each alone for the
same subset of cases from the data set. The validity of predic-
tions of Interacting & Presenting, Analyzing & Interpreting,
Creating & Conceptualizing, and, to a lesser degree, Organizing
& Executing is increased by the addition of ability test infor-
mation. For the others, addition of ability test data results in
little or no change of validity. The final sample-weighted av-
erage corrected validities for the combined measures range
from 0.20 for Supporting & Cooperating to 0.44 for Analyzing
& Interpreting.
Table 14
Standardized Regression Weights Used to Combine Personality- and Ability-Based Predictors of
the Great Eight
Great Eight criterion
competency
Correlation:
personality
with ability
Personality Ability
Ratio
of

s R

p ⬍

p
Leading/Deciding 0.017 0.160 .001 0.053 .05 0.331 0.169
Supporting/Cooperating ⫺0.139 0.127 .001 0.028 ns 0.220 0.127
Interacting/Presenting ⫺0.008 0.221 .001 0.158 .001 0.715 0.271
Analyzing/Interpreting 0.259 0.122 .001 0.226 .001 1.852 0.283
Creating/Conceptualizing 0.186 0.186 .001 0.153 .001 0.823 0.262
Organizing/Executing ⫺0.109 0.183 .001 0.117 .001 0.639 0.206
Adapting/Coping ⫺0.056 0.127 .001 0.049 .05 0.386 0.133
Enterprising/Performing 0.380 0.166 .001 0.011 ns 0.066 0.166
Note. N ⫽ 1,727–2,157. Number of studies ⫽ 12–18.
Table 15
Meta-Analysis Results for Great Eight Competencies Based on Personality-Only Predictors,
Ability-Only Predictors, and Combined Personality and Ability Predictors for All Those Cases
Having Data on Both Personality and Ability
Variable No. studies Nr
SD
%var 10% CrI
Personality only
Leading/Deciding 18 2,152 0.164 0.245 0.072 78.77% 0.153
Supporting/Cooperating 16 2,114 0.130 0.197 0.159 40.00% ⫺0.007
Interacting/Presenting 18 2,157 0.221 0.329 0.106 60.36% 0.192
Analyzing/Interpreting 17 2,121 0.179 0.264 0.144 45.68% 0.079
Creating/Conceptualizing 12 1,727 0.213 0.305 0.062 77.41% 0.226
Organizing/Executing 18 2,156 0.163 0.238 0.139 49.65% 0.060
Adapting/Coping 15 1,977 0.115 0.164 0.114 57.06% 0.019
Enterprising/Performing 16 2,051 0.162 0.237 0.047 88.61% 0.177
Ability only
Leading/Deciding 18 2,152 0.043 0.074 0.188 36.30% ⫺0.167
Supporting/Cooperating 16 2,114 0.007 0.022 0.120 55.97% ⫺0.132
Interacting/Presenting 18 2,157 0.150 0.219 0.155 44.09% 0.021
Analyzing/Interpreting 17 2,121 0.276 0.404 0.198 26.67% 0.150
Creating/Conceptualizing 12 1,727 0.173 0.242 0.094 60.98% 0.122
Organizing/Executing 18 2,156 0.104 0.156 0.157 44.70% ⫺0.045
Adapting/Coping 15 1,977 0.051 0.076 0.054 86.41% 0.007
Enterprising/Performing 16 2,051 0.028 0.051 0.160 41.78% ⫺0.154
Personality and ability
Leading/Deciding 18 2,152 0.171 0.257 0.034 94.25% 0.213
Supporting/Cooperating 16 2,114 0.133 0.201 0.164 38.19% ⫺0.009
Interacting/Presenting 18 2,157 0.270 0.397 0.134 46.29% 0.225
Analyzing/Interpreting 17 2,121 0.299 0.438 0.177 30.31% 0.210
Creating/Conceptualizing 12 1,727 0.253 0.357 0.054 80.48% 0.287
Organizing/Executing 18 2,156 0.205 0.302 0.065 81.30% 0.219
Adapting/Coping 15 1,976 0.128 0.180 0.099 63.59% 0.053
Enterprising/Performing 16 2,051 0.163 0.240 0.028 95.48% 0.204
Note. r ⫽ sample-weighted average of uncorrected correlations;
⫽ sample-weighted average of correlations
corrected for artifacts; SD
⫽ standard deviation of corrected correlations; 10%CrI ⫽ lower 10% boundary of
80% credibility interval; %var ⫽ percentage variance accounted for by artifact corrections for corrected
correlation distribution.
1195
A CRITERION-CENTRIC APPROACH TO VALIDATION
Validity Generalizability and Situational Specificity
Examination of Table 15 shows considerable variance in the
corrected correlations for some competencies. For the combined
personality and ability predictors, the percentage of variance ac-
counted for passes the 75% rule in 50% of the cases, whereas for
personality alone this is only true for three of the eight competen-
cies (Leading & Deciding, Creating & Innovating, and Enterpris-
ing & Performing) and for ability alone just one (Adapting &
Coping) of the eight. However, in the case of ability and Adapting
& Coping competencies, the average validity is close to zero,
implying that ability is never a valid predictor for this aspect of
competency. In their study of meta-analyses, Hermelin and Rob-
ertson (2001) noted that the average percentage of variance ac-
counted for across studies tends to be nearer to 50% than 75%,
indicating the presence of genuine moderating effects. The average
variance accounted for across the eight competencies (for the
results presented in Table 15) is 66.24%, which is higher than the
Hermelin and Robertson estimate but well below the 75% cutoff.
Combining personality and ability as predictors has some un-
expected effects on generalizability. Although the three competen-
cies that are above the 75% cutoff for Personality alone remain so
for the combined predictors, Organizing & Executing passes the
75% cutoff for the combined predictors but not for either on its
own. Adapting & Coping, which was generalizable according to
the ability data, is not generalizable according to the combined
data. It is difficult to explain these anomalies from examination of
the data. One possibility is that they arise through instability of the
estimates of variance accounted for. This is likely to be especially
true when the average correlation is small and the between-study
variance is also small.
Relationships Between Aggregated Criteria and
Aggregated Predictors
All the analyses reported previously focus on the individual
pairwise relationships between each of the Great Eight predictor–
criterion pairs. In practice, selection decisions are made on the
basis of aggregation of information from multiple sources (such as
predictors of each of the Great Eight competencies) and are best
evaluated in terms of their overall relationship with aggregated
criteria. An estimate of the upper limit on the overall relationship
between optimally weighted aggregated sets of criterion and pre-
dictor measures can be obtained by examining the canonical cor-
relation between the predictor and criterion vectors. Canonical
components for personality only (eight predictors and eight crite-
rion measures) and personality and ability (nine predictors and
eight criterion measures) for the same set of cases accounted for
28.87% of the variance for personality only (R ⫽ .54) and 30.66%
of the variance for personality and ability combined (R ⫽ .55).
Although these figures provide an estimate of the validity obtain-
able with aggregated multiple criteria, the following should be
noted:
1. These results are based on the uncorrected correlation
matrices and as such are underestimates of the covariance
between predictor and criterion sets.
2. Canonical correlation, like multiple regression, capital-
izes on chance and provides us with a best fitting solution
for this data set. Thus, the covariance accounted for in
new data sets by the canonical equations developed from
this data set would show shrinkage.
The extent to which this level of correlation is obtainable in
practice can be estimated by assuming that ratings of OJP provide
an indication of the relative weights an organization places on the
eight criterion competencies. By regressing the OJP measures,
when these are available, onto the Great Eight criterion compe-
tency ratings, it will be possible to create aggregated criterion
competency scores using the regression beta values as weights.
Multiple regression can then be used to see how the predictor
competency potential scores relate to this aggregate for each study.
Predicting OJP
Data on OJP ratings were available for 13 studies. Of these, only
10 also had ratings on all eight of the criterion competencies. The
current analysis is restricted to those 10 studies (N ⫽ 1,864).
Regression analyses were carried out on the raw data (with no
corrections for artifacts) for each study. Individual study beta
weights and multiple correlations, and sample-weighted average
beta weights are shown in Table 16.
Of the eight competency ratings, those most strongly related to
OJP ratings, in order of importance (see Table 16), are Analyzing
& Interpreting, Organizing & Executing, Enterprising & Perform-
ing, Leading & Deciding, and Creating & Conceptualizing. The
more contextual competencies (Supporting & Cooperating, Adapt-
ing & Coping, Interacting & Presenting) are less strongly related.
This suggests that OJP ratings are primarily influenced by task
performance competencies. However, the data do show consider-
able differences in patterns between studies, with the standard
deviation of the beta weights across studies averaging about 0.15.
This suggests that the competency factors that influence judgments
of OJP do vary from situation to situation. Unfortunately, there is
insufficient information available about the individual studies to
establish whether these variations are meaningful in terms of
differences in job content or organizational culture.
The model underlying the Great Eight competencies (Kurz &
Bartram, 2002) is consistent with Spencer and Spencer’s (1993)
causal flow model, which postulates that personal characteristics
predict OJP achievement through competencies. The regression
weights shown in Table 16 were used to construct a single-
weighted aggregate competency criterion for each study. The
composite competency criterion variables were then regressed on
the eight predictor competencies (weighted composites of person-
ality and ability for eight of the studies and personality only for
two). The average multiple correlation across the 10 studies was
.35 (unweighted) or .27 (weighted by sample size; Table 17). As a
cross-validation, the sample-weighted average beta weights from
Table 16 were used to create composite criterion scores for 4
additional studies (N ⫽ 440), when there were data on all eight of
the criterion competencies. The same process was then used to
regress these composite criterion variables on the eight predictors.
For these 4 studies, mean R ⫽ .35 (both unweighted and weighted
by sample size). Across all 14 studies, the sample-weighted mean
R ⫽ .28, and the average unweighted R ⫽ .35. Given that the
impact of correcting for artifacts (range-restriction and criterion
reliability) for these data is to increase obtained coefficients by
1196
BARTRAM
between 47% and 60% (average increase ⫽ 51%: see Tables
13–15), one can estimate that operational validities for aggregated
predictor and aggregated criterion measures would be in the range
of 0.42 to 0.53.
Finally, the predictor composite scores were correlated with the
OJP ratings (Table 18). The average (uncorrected) correlation was
.27 (unweighted) or .22 (weighted by sample size).
The pattern of results for the prediction of the composite com-
petency criterion from personality and ability-based competency
predictors (see Table 17) is what one would expect from previous
research: The strongest predictor is Organizing & Executing
(which is related to Big Five Conscientiousness in the predictor
domain). The other main predictors are Leading & Deciding,
Creating & Conceptualizing, Analyzing & Reporting, and Inter-
acting & Presenting, which are most strongly related to general
mental ability and Extraversion in the predictor domain. Beta
weights for Supporting & Cooperating are negative or near zero.
Meta-Analysis of Relationships Between Combined
Personality and Ability Predictors and OJP
Meta-analysis of the eight competency potential scores as direct
predictors of OJP was carried out for the 10 studies examined
previously. For this analysis, the reliability of the OJP criterion
was set at 0.70 for the seven studies using the six-item OJP rating
scale and at 0.52 (as recommended by Hermelin & Robertson,
2001) for the remaining three studies, which used single-item
ratings of OJP. No correction for range restriction was carried out.
The results (Table 19) show a similar pattern to that reported
previously: Leading & Deciding and Organizing & Executing have
the strongest average relationships but have low generalizability.
Analyzing & Interpreting, in contrast, although having a lower
level of average correlation, shows high generalizability (with
artifact corrections actually overcorrecting for between-study
variance).
Table 16
Beta Weights and R Values for Regression of OJP on Line-Manager Competency Ratings
Line-manager
competency ratings
Study no.

a
1 2 3 4 5 8 10 11 13 14 MSD
Leading/Deciding 0.13 0.45 0.16 0.37 ⫺0.01 0.11 0.30 0.22 0.02 0.17 0.16 0.15
Supporting/Cooperating 0.09 0.11 0.00 ⫺0.01 0.28 0.10 0.33 0.11 ⫺0.02 ⫺0.32 0.06 0.18
Interacting/Presenting 0.02 ⫺0.16 0.05 0.00 0.11 0.20 ⫺0.11 ⫺0.07 0.01 0.42 0.04 0.17
Analyzing/Interpreting 0.12 0.15 0.24 0.09 0.17 0.18 0.32 0.29 0.50 0.00 0.26 0.14
Creating/Conceptualizing 0.20 ⫺0.05 0.09 0.19 0.12 0.11 ⫺0.06 0.19 0.12 ⫺0.08 0.12 0.11
Organizing/Executing 0.36 0.44 0.40 0.25 0.30 0.18 0.10 0.22 0.06 ⫺0.01 0.19 0.15
Adapting/Coping 0.08 ⫺0.40 0.23 ⫺0.01 ⫺0.21 0.15 0.01 ⫺0.09 0.10 0.03 0.03 0.18
Enterprising/Performing 0.03 0.11 0.00 ⫺0.01 0.19 0.11 0.25 0.15 0.43 0.29 0.18 0.14
R 0.63 0.85 0.72 0.63 0.67 0.77 0.78 0.84 0.93 0.68
N
b
114 35 91 132 65 378 86 503 362 98
Note. Mean beta values are sample weighted. All correlations are uncorrected for effects of artifacts. OJP ⫽ overall job performance.
a
Total mean

⫾ SD ⫽ 0.13 ⫾ 0.15.
b
Total sample size ⫽ 1,864.
Table 17
Beta Weights and R Values for Regression of Aggregated Great Eight Line-Manager Competency Ratings on Great Eight Competency
Potential Predictors
a
Competency potential
predictors
Study

b
1 2 3 4 5 8 10 11 13 14 MSD
Leading/Deciding ⫺0.02 0.01 0.04 0.11 ⫺0.25 0.08 0.01 0.05 0.13 0.27 0.07 0.13
Supporting/Cooperating 0.02 ⫺0.26 ⫺0.19 ⫺0.20 0.00 0.00 ⫺0.06 0.00 ⫺0.15 ⫺0.05 ⫺0.06 0.10
Interacting/Presenting 0.05 0.20 0.05 0.08 ⫺0.01 0.05 0.31 ⫺0.09 ⫺0.01 0.19 0.02 0.12
Analyzing/Interpreting ⫺0.02 0.33 0.39 ⫺0.28 0.04 0.01 0.25 0.07 0.05 ⫺0.17 0.04 0.21
Creating/Conceptualizing ⫺0.07 ⫺0.36 ⫺0.32 0.29 0.36 0.00 ⫺0.24 0.16 ⫺0.03 0.33 0.05 0.27
Organizing/Executing 0.11 0.41 ⫺0.21 0.19 0.19 0.04 ⫺0.02 0.06 0.18 ⫺0.01 0.08 0.16
Adapting/Coping ⫺0.03 ⫺0.32 ⫺0.08 ⫺0.03 ⫺0.29 0.06 ⫺0.08 0.02 ⫺0.01 ⫺0.14 ⫺0.02 0.13
Enterprising/Performing ⫺0.12 ⫺0.06 0.05 ⫺0.02 0.22 0.02 0.02 0.02 0.03 0.00 0.02 0.09
R 0.18 0.57 0.30 0.34 0.55 0.11 0.35 0.24 0.29 0.57
N 114 35 91 132 65 378 86 503 362 98
Note. All correlations are uncorrected for effects of artifacts. Mean beta values are sample weighted.
a
Personality and Ability composites, except for Study 8 and Study 14, which had personality data only.
b
Unweighted total mean ⫽ 0.35; sample-weighted total mean ⫽ 0.27.
1197
A CRITERION-CENTRIC APPROACH TO VALIDATION
As noted in the earlier analyses, there is a negative correlation
between OJP and Supporting & Cooperating (which is mainly Big
Five Agreeableness in the predictor domain), suggesting that peo-
ple who are high on Big Five Agreeableness are judged less
favorably on their OJP than those who are lower on this attribute.
Other studies have noted small negative correlations between
agreeableness and job performance (e.g., Hunthausen, Truxillo,
Bauer, & Hammer, 2003), but the reported effects are usually less
than ⫺0.1.
Summary
The results support the validity of point-to-point relationships
between Great Eight competencies and their predictors. The ob-
tained correlations were consistently higher than those between
unmatched pairs of predictors and criteria. Although personality-
based predictors showed moderate to high validities for all of the
Great Eight, ability tests only added to the prediction of criteria for
four of the eight competencies. As hypothesized, ability is most
strongly predictive of competencies in the areas of Analyzing &
Interpreting and Creating & Conceptualizing. The correlation be-
tween aggregated multiple predictors and aggregated multiple cri-
teria was, as one would expect, substantially higher than the
relationships between the predictors and OJP would suggest.
Discussion
The current results show that when there is a strong rationale
defining the predictors and the criterion, and when these can then
be matched on a one-to-one basis (rather than the traditional many
predictors to one or many to few criteria), a clear pattern of results
is found, which is consistent with the hypotheses presented early in
this article. The results confirm the hypothesized Great Eight
pairwise predictor–criterion competency relationships. Not only
are the relationships between matched predictor– competency pairs
substantially higher than those between unmatched pairs, but it
was also shown that personality and ability together and in isola-
tion predict competency performance ratings in a meaningful
manner. Specifically, ability tests predict four of the Great Eight;
the strongest relationship between ability and competencies oc-
curred for the Analyzing & Interpreting competencies. Personality
provides a far broader coverage of the competency domain than
ability, but ability data add to the level of prediction one obtains
from personality measures on their own in those areas where
ability is relevant.
The results show that when Big Five measures are used as
predictors, they also provide good coverage of the Great Eight
criterion competency model. However, the evidence suggests that
applying the Great Eight competency model to the predictor do-
main provides a clearer and stronger pattern of relationships than
using a mixed model with the Big Five as predictors and the Great
Eight as criteria. Mapping the predictor domain to the Great Eight
definitions rather than the Big Five accounts for more of the
criterion variance and also provides a stronger practitioner focus
by concentrating on what is being predicted rather than what is
doing the prediction.
The main advantage of the Great Eight model is that it provides
(a) a framework for integrating measures in the predictor domain,
such as ability, personality, and motivation scales, and (b) a clear
set of a priori hypotheses regarding the expected eight one-to-one
Table 19
Correlations (Operational Validities) Between Combined Personality- and Ability-Based
Predictions of Competencies and Independent Ratings of Overall Job Performance
Variable No. studies Nr
SD
%var 10% CrI
Leading/Deciding 10 1864 0.10 0.14 0.14 37.01% ⫺0.04
Supporting/Cooperating 10 1864 ⫺0.11 ⫺0.16 0.13 37.50% ⫺0.33
Interacting/Presenting 10 1864 0.05 0.07 0.12 43.16% ⫺0.09
Analyzing/Interpreting 10 1864 0.06 0.09 0.00 100.00% 0.09
Creating/Conceptualizing 10 1864 0.06 0.08 0.09 56.90% ⫺0.04
Organizing/Executing 10 1864 0.09 0.12 0.09 59.20% 0.01
Adapting/Coping 10 1864 0.00 ⫺0.01 0.00 100.00% ⫺0.01
Enterprising/Performing 10 1864 0.06 0.09 0.13 38.95% ⫺0.08
Note. r ⫽ sample-weighted average of uncorrected correlations;
⫽ sample-weighted average of correlations
corrected for artifacts; SD
⫽ standard deviation of corrected correlations; 10% CrI ⫽ lower 10% boundary of
80% Credibility interval; %var ⫽ percentage variance accounted for by artifact corrections for corrected
correlation distribution.
Table 18
Correlations Among Great Eight Aggregated Competency
Potential Predictors (AggPred), Great Eight Aggregated Line
Manager–Rated Competencies (AggCrit), and Overall Job
Performance (OJP)
Study
AggPred
w/AggCrit
AggCrit
w/OJP
AggPred
w/OJP N
1 0.18 0.63 0.14 114
2 0.57 0.85 0.35 35
3 0.30 0.72 0.29 91
4 0.34 0.63 0.18 132
5 0.55 0.67 0.36 65
8 0.11 0.77 0.16 378
11 0.35 0.78 0.21 86
12 0.24 0.84 0.16 503
14 0.29 0.93 0.26 362
15 0.57 0.68 0.59 98
Sample weighted mean 0.27 0.79 0.22
Unweighted mean 0.35 0.75 0.27
Note. All correlations are uncorrected for effects of artifacts. Total sam-
ple ⫽ 1,864.
1198
BARTRAM
predictor– criterion relationships. The contribution of the Great
Eight model for understanding of job performance is clear. Each of
the eight predictors was shown to predict a different area of job
performance consistently across jobs, measurement instruments,
and cultural contexts.
The correlation of these predictors with OJP was lower than one
would expect from the combination of the eight pairwise
predictor– criterion relationships: The average uncorrected corre-
lation across studies between aggregated predictor competencies
and OJP was between .22 (weighted) and .25 (unweighted),
whereas the average uncorrected correlation between aggregated
predictor and criterion competencies was between .27 (weighted)
and .35 (unweighted). The average uncorrected correlation be-
tween aggregated criterion competency ratings and OJP, however,
was between .75 (weighted) and .79 (unweighted). These results
are consistent with the causal flow model (Spencer & Spencer,
1993): Personality and ability act to predict the related behaviors
as rated by line managers as competencies, and these ratings of
competencies are, in turn, related to OJP ratings. Considering job
performance in an undifferentiated manner (as ratings of OJP do)
hides the pattern of relationships between predictors and more
specific competency factors. This suggests that the current meta-
analysis literature may be underestimating the capacity for person-
ality measures and ability test data to add value to assessment
procedures by enhancing their overall predictive power and pro-
viding more detailed diagnostic information on performance.
The levels of correlation obtained in the current meta-analysis
are quite high for personality-based predictors in comparison with
other meta-analysis studies (and clearly substantive in practical
terms). These results have been obtained using personality instru-
ments that adopt a clear work-related frame of reference (FOR). As
Hunthausen et al. (2003) have shown, instruments that have a
world-of-work FOR do yield higher validities than those that are
more general. Ones and Viswesvaran (2001) have reviewed the use
of “criterion-focused occupational personality scales (COPS)” in
selection and have also noted the higher validities associated with
scales that directly address issues of relevance in the workplace
compared with more general personality assessment instruments. It
should be noted that the OPQ instruments were developed as
work-related measures of personality, and the item content and
scales were developed through working with people in industry. It
was as a consequence of this that different instruments were
developed to cover general graduate and managerial use (OPQ),
customer service roles (CCSQ), and blue-collar jobs (WSQ). Be-
cause the development process was centered on covering all as-
pects of personality that are considered to be of relevance in the
workplace, the OPQ inventories cover a wider range of personal
attributes than instruments developed from a personality theory
focus, such as the five-factor model. In particular, aspects of
motivation are covered. This greater breadth becomes important
when one is attempting to cover the full range of personal at-
tributes assessed by line managers in their competency ratings.
The results for OJP show that this is not predicted equally by all
eight of the personality-based predictors, by ability, or by
personality–ability combinations. OJP is predicted mainly by Or-
ganizing & Executing, Leading & Deciding, and Analyzing &
Interpreting, with a negative association with Supporting & Co-
operating competencies. This may have more to say about what
factors drive managers’ general ratings of job performance than
anything else. It suggests a pattern whereby managers favor people
who are dependable, high achieving, and focused on the task rather
than those who display the prosocial behaviors of helping and
supporting others. Further work is needed to determine whether
actual productivity or other outcome measures are more strongly
related to aggregated multiple criteria or to single OJP ratings. For
the current data, at least, the relationship between the eight pre-
dictors and eight criteria shows how much stronger validities can
be obtained by aggregation of multiple criteria than by the use of
single overall rating measures.
Contextual and Task Performance
The analyses of the Great Eight predictors and criteria (see
Table 9) indicate a more general level of description that ties this
work in closely with the literature on contextual versus task-related
performance constructs, reviewed earlier here. For the criterion
measures, the first principal component is loaded by competencies
that are closely tied to task performance and that are best predicted
by motivation, general ability, conscientiousness, and openness to
new experience. The second principal component is related to
competencies associated with supporting and cooperating with
others and coping and adapting to change. The distinction between
these is very similar to that described by R. Hogan (1983) and J.
Hogan and Holland (2003) as dynamism, or “getting ahead,” and
social propriety, or “getting along.”
The personality-based Great Eight predictors provide some
more differentiation, in that the “getting ahead” competencies are
divided into two principal components in terms of potential. The
first reflects motivation, extraversion, and openness to new expe-
rience, whereas the second represents the configuration of vari-
ables that one sees emerging as consistent predictors of perfor-
mance across a wide range of studies: conscientiousness and
related aspects of thinking styles (e.g., Barrick & Mount, 1991;
Salgado, 1997, 1998; Schmidt & Hunter, 1998).
Approaches to the criterion domain reflect the same issues that
we have seen in the personality field in terms of how differentiated
the domain needs to be. Lexical analysis studies and research on
self-report personality measures have provided evidence for a
number of ways of looking at the domain space. At the most
general level, there is single personality factor, which differenti-
ates between desirable and undesirable attributes (Boies, Lee,
Ashton, Pascal, & Nicol, 2001; Goldberg & Somer, 2000; Saucier,
1997). This is analogous in the criterion domain to ratings of OJP,
which typically reflect broad evaluations of how good or bad a
person is. Also well established is the two-factor solution (Boies et
al., 2001; Digman, 1997; Paulhus & John, 1998; Saucier, 1997), in
which the first factor relates to positive dynamic attributes and
individual ascendancy and the second to social propriety and
community cohesion. Bakan (1966) described these factors as
agency and communion. They also clearly relate to task and
contextual performance factors in the criterion domain.
The Big Five level of analysis is the one focused on here, in that
the Great Eight competencies are designed to represent a level of
generality comparable to that represented by the Big Five in the
personality domain. Factor analyses of the OPQ produces five-
factor solutions that map onto the Big Five. Six- and seven-factor
solutions can also be found that provide a differentiation similar to
that of the HPI and that differentiate achievement from depend-
1199
A CRITERION-CENTRIC APPROACH TO VALIDATION
ability (these tend to be combined as Conscientiousness in most
Big Five models) and that differentiates sociability from impul-
sivity (often combined as Extraversion). However, the argument as
to whether there are five, six, or seven factors is primarily one
about the scope of the domain rather than the level of aggregation.
Whichever solution one adopts, it is then possible to disaggregate
these factors into facets or more specific subscales (just as the
OPQ32 resolves them into 32 scales, and the NEO-PI-R into
30-facet scales). However, the main thrust of the current argument
is that it is more useful to map predictor instruments onto criterion
models for the purposes of validation rather than to map the
criterion models onto predictor models. To do this, the level of
aggregation of the predictor scales should match that of the
criterion.
If sufficient data are available, it would be better to operate at a
more detailed level of description (e.g., in terms of the 20 com-
petencies presented in the Appendix) with comparable, more spe-
cific composites of personality and ability tests as predictors. This
was not possible for the current research, because insufficient
information was available on the criterion competencies to allow
mapping to a more detailed level than the Great Eight.
For the current research, the ability test data that were available
(verbal reasoning or numerical reasoning or both) were treated as
providing an estimate of general mental ability. At the Great Eight
level of description, it is probably not appropriate to consider the
differential impact of specific abilities. Examination of the com-
ponent level of Analyzing & Interpreting (see Appendix) shows,
however, that one would expect different patterns of validity for
more specific ability tests at more detailed levels of description
within the framework. For example, Component 4.2.7 relates to
“demonstrating spatial awareness” and Component 4.2.5 to “dem-
onstrating physical and manual skills.” One would expect to find
tests of spatial ability having higher validities than other ability
tests for jobs in which this component was relevant. We would also
expect tests of creative thinking to increase levels of prediction for
Creating & Conceptualizing, whereas tests of oral comprehension
and expression should relate to Interacting & Presenting. For the
current set of studies, however, the emphasis on service and
managerial positions and the focus on the general Great Eight level
of aggregation entails that general mental ability, as assessed by
verbal and numerical reasoning, is likely to account for most of the
ability-related variance. Future research needs to consider whether
the levels of prediction found here would be increased by the use
of more differentiated competency models and the use of a wider
range of more specific ability tests.
Limitations
The studies reported here were collected from corporate ar-
chives. It was not possible to exercise control, in retrospect, over
data collection, supervisor rating procedures, and other design
factors. In many cases, study design details were not available. The
total sample of studies, however, included five U.K. samples
(Studies 1–5 in Table 3) where there was good independent control
over the design of the studies and the data collection. These
supervised studies (previously reported in Nyfield et al., 1995)
provide a benchmark against which the quality of the data from the
other studies can be evaluated. Removal of these supervised stud-
ies from the data set generally resulted in a lowering of average
validities for the remaining studies. However, the overall pattern of
results was not affected.
On the issue of statistical power, Murphy (1997) suggested that
caution should be used in interpreting tests of situational specific-
ity when the average sample size is less than 100 to 200 or when
the number of studies (k) is less than 15 to 20. The current data set
has a median sample size of 125, with k generally in the 20⫹
range. On that basis, this collection of studies seems to have
sufficient power to detect situational specificities.
The samples examined here are predominantly from manage-
ment or are graduates in technical or sales positions or manage-
ment training. Although some of the data came from blue-collar
positions, their influence on the overall results will be quite small.
It is quite possible that the pattern of these results will vary for
different types and levels of job. In particular, we might expect to
find greater emphasis on the competency area most closely related
to job knowledge and skills (Analyzing & Interpreting) in lower
level jobs. The language used to define the competencies within
the current framework fits most easily with the way these behav-
iors are described for management positions. However, the frame-
work is intended to be generic. Future research needs to explore
the degree to which it is possible to produce better operational
definitions of the competencies in the framework for different job
levels.
Finally, all the data presented here have used predictor instru-
ments from one publisher (although they have included a variety of
instruments with a variety of response formats) with competency
measurement tools that were either from the same source or from
client-constructed measures of unknown psychometric quality.
Although all the personality instruments and much of the compe-
tency ratings data were collected using instruments from a single
publisher, there is no direct content or construct overlap between
the models underlying the personality instruments and the compe-
tency instruments. Indeed, one reason for developing a generic
competency framework was to be able to map between compe-
tency models like IMC, CCCI, and PMC, which are otherwise very
different in structure and content as well as in item type and
format.
Future work on the Great Eight would benefit from research that
mapped other predictor instruments and other criterion measure-
ment procedures onto this structure to test its generalizability and
robustness.
Conclusions
Perhaps we have been preoccupied for too long with the won-
derful personality questionnaires and ability tests we have con-
structed to measure all sorts of aspects of human potential. In so
doing, we may have lost sight of why it is important to be able to
measure these characteristics. As a consequence, practitioners
have often had difficulty explaining to their clients the value of
what we have to offer. We need to realize that this inability may
be due in no small part to our failure to address the issues that
actually concern clients: performance at work and the outcomes of
that performance. The Big Five and other classifications of per-
sonality factors, for example, are classifications focused on the
predictor domain. In this article, we have endeavored to show the
value of changing this focus to the criterion domain while still
providing the same level of differentiation within that domain.
1200
BARTRAM
Although the Great Eight provides an analogue in that domain to
“g,” motivation measures, and the Big Five of the predictor do-
main, it has the advantage of addressing directly the issues that are
of prime practical importance in selection testing: what it is that is
being predicted.
By differentiating performance in the criterion domain in this
way, we can better articulate the value of what we can provide as
predictors of work-related behaviors from a practice point of view
and better understand why particular patterns of predictor–
criterion relationships occur. To facilitate this, practitioners need
to encourage their clients to adopt more differentiated appraisal
tools. This would be not only of scientific value in improving the
quality of validity studies but also of value to the client in provid-
ing more reliable and more valid measures of people’s perfor-
mance. By comparing measures of actual performance on the
Great Eight or similar criterion classification models with mea-
sures of potential on the same constructs (using personality and
ability tests as predictors), clients would be better able to identify
those areas in which people would benefit most from learning
opportunities and developmental experiences.
References
Bakan, D. (1966). The duality of human existence: Isolation and commun-
ion in Western man. Boston: Beacon Books.
Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimen-
sions and job performance: A meta-analysis. Personnel Psychology, 44,
1–26.
Bartram, D. (2001). Predicting competency dimensions from components:
A validation of the two-step process. Thames Ditton, UK: SHL Group.
Bartram, D., Robertson, I. T., & Callinan, M. (2002). Introduction: A
framework for examining organizational effectiveness. In I. T. Robert-
son, M. Callinan, & D. Bartram (Eds.), Organizational effectiveness:
The role of psychology (pp. 1–10). Chichester, UK: Wiley.
Boies, K., Lee, K., Ashton, M. C., Pascal, S., & Nicol, A. A. M. (2001).
The structure of the French personality lexicon. European Journal of
Personality, 15, 277–295.
Borman, W. C., Buck, D. E., Hanson, M. A., Motowidlo, S. J., Stark, S.,
& Drasgow, F. (2001). An examination of the comparative reliability,
validity, and accuracy of performance ratings made using computerized
adaptive rating scales. Journal of Applied Psychology, 86, 965–973.
Borman, W. C., & Motowidlo, D. J. (1993). Expanding the criterion
domain to include elements of contextual performance. In N. Schmitt &
W. C. Borman (Eds.), Personnel selection in organizations (pp. 71–98).
San Francisco: Jossey-Bass.
Borman, W. C., Penner, L. A., Allen, T. D., & Motowidlo, S. J. (2001).
Personality predictors of citizenship performance. International Journal
of Selection and Assessment, 9, 52–69.
Campbell, J. P. (1990). Modeling the performance prediction problem in
industrial and organizational psychology. In M. D. Dunnette & L. M.
Hough (Eds.), Handbook of industrial and organizational psychology
(2nd ed., Vol. 1, pp. 687–732). Palo Alto: Consulting Psychologists
Press.
Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A
theory of performance. In N. Schmitt & W. C. Borman (Eds.), Personnel
selection in organizations (pp. 35–70). San Francisco: Jossey-Bass.
Campbell, J. P., McHenry, J. J., & Wise, L. L. (1990). Modeling job
performance in a population of jobs. Personnel Psychology, 43, 313–
333.
Conway, J. M. (1996). Additional construct validity evidence for the
task/contextual performance distinction. Human Performance, 9, 309–
329.
Digman, J. M. (1997). Higher order factors of the Big Five. Journal of
Personality and Social Psychology, 73, 1246–1256.
Goldberg, L. R., & Somer, O. (2000). The hierarchical structure of com-
mon Turkish person-descriptive adjectives. European Journal of Per-
sonality, 14, 497–531.
Gotoh, A. (1999). The evaluation of competencies predictors across nine
studies in five countries. Unpublished master’s thesis, Goldsmith Col-
lege, London.
Hermelin, E., & Robertson, I. T. (2001). A critique and standardization of
meta-analytic validity coefficients in personnel selection. Journal of
Occupational and Organizational Psychology, 74, 253–277.
Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and
job-performance relations: A socioanalytic perspective. Journal of Ap-
plied Psychology, 88, 100 –112.
Hogan, R. (1983). A socioanalytic theory of personality. In M. M. Page
(Ed.), Nebraska Symposium on Motivation (pp. 336–355). Lincoln:
University of Nebraska Press.
Hough, L. M. (1992). The Big Five personality variables—construct
confusion: Description versus prediction. Human Performance, 5, 139 –
155.
Hough, L. M., Ones, D. S., & Viswesvaran, C. (1998, April). Personality
correlates of managerial performance constructs. Paper presented at the
annual conference of the Society for Industrial and Organizational
Psychology, Dallas.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative
predictors of job performance. Psychological Bulletin, 96, 72–98.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis. Newbury
Park, CA: Sage.
Hunthausen, J. M., Truxillo, D. M., Bauer, T. N., & Hammer, L. B. (2003).
A field study of frame-of-reference effects in personality test validity.
Journal of Applied Psychology, 88, 545–551.
Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance:
The Big Five revisited. Journal of Applied Psychology, 85, 869– 879.
Johnson, J. W. (2001). The relative importance of task and contextual
performance dimensions to supervisor judgements of overall perfor-
mance. Journal of Applied Psychology, 86, 984 –996.
Judge, T. A., Bono, J. E., Ilies, R., & Gerhardt, M. W. (2002). Personality
and leadership: A qualitative and quantitative review. Journal of Applied
Psychology, 87, 765–780.
Kurz, R. (1999). Automated prediction of managerial competencies from
personality and ability variables. In Proceedings of the BPS Test User
Conference (pp. 96 –101). Leicester, UK: British Psychological Society.
Kurz, R., & Bartram, D. (2002). Competency and individual performance:
Modeling the world of work. In I. T. Robertson, M. Callinan, & D.
Bartram (Eds.), Organizational effectiveness: The role of psychology
(pp. 227–255). Chichester: Wiley.
Kurz, R., Bartram, D., & Baron, H. (2004). Assessing potential and
performance at work: The Great Eight competencies. In Proceedings of
the British Psychological Society Occupational Conference (pp. 91–95).
Leicester, UK: British Psychological Society.
Motowidlo, S. J., Borman, W. C., & Schmit, M. J. (1997). A theory of
individual differences in task and contextual performance. Human Per-
formance, 10, 71– 83.
Murphy, K. (1997). Meta-analysis and validity generalization. In N. Ander-
son & P. Herriot (Eds.), International handbook of selection and assess-
ment (pp. 323–342). Chichester, UK: Wiley.
Nyfield, G., Gibbons, P. J., Baron, H., & Robertson, I. (1995, May). The
cross cultural validity of management assessment methods. Paper pre-
sented at the 10th Annual Conference of the Society for Industrial and
Organizational Psychology, Orlando.
Ones, D. S., & Viswesveran, C. (2001). Integrity tests and other criterion-
focused occupational personality scales (COPS) used in personnel se-
lection. International Journal of Selection and Assessment, 9, 31–39.
Paulhus, D. L., & John, O. P. (1998). Egoistic and moralistic biases in
1201
A CRITERION-CENTRIC APPROACH TO VALIDATION
self-perception: The interplay of self-descriptive styles with basic traits
and motives. Journal of Personality, 66, 1025–1060.
Robertson, I. T., Baron, H., Gibbons, P. J., MacIver, R., & Nyfield, G.
(2000). Conscientiousness and managerial performance. Journal of Oc-
cupational and Organizational Psychology, 73, 171–180.
Robertson, I. T., & Kinder, A. (1993). Personality and job competencies:
The criterion-related validity of some personality variables. Journal of
Occupational and Organizational Psychology, 66, 225–244.
Rotundo, M., & Sackett, P. R. (2002). The relative importance of task,
citizenship, and counterproductive performance to global ratings of job
performance: A policy-capturing approach. Journal of Applied Psychol-
ogy, 87, 66 – 80.
Salgado, J. F. (1997). The five factor model of personality and job perfor-
mance in the European Community. Journal of Applied Psychology, 82,
30 – 43.
Salgado, J. F. (1998). Big Five personality dimensions and job perfor-
mance in army and civil occupations: A European perspective. Human
Performance, 11, 271–288.
Saucier, G. (1997). Effects of variable selection on the factor structure of
person descriptors. Journal of Personality and Social Psychology, 73,
1296 –1312.
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection
methods in personnel psychology: Practical and theoretical implications
of 85 years of research findings. Psychological Bulletin, 124, 262–274.
Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the
construct validity of developmental ratings of managerial performance.
Journal of Applied Psychology, 88, 50– 66.
SHL Group. (1993a). Inventory of Management Competencies: Manual
and user’s guide. Thames Ditton, United Kingdom: Author.
SHL Group. (1993b). OPQ concept model: Manual and user’s guide.
Thames Ditton, United Kingdom: Author.
SHL Group. (1994). Perspectives on managerial competencies: User’s
manual. Thames Ditton, United Kingdom: Author.
SHL Group. (1997). Customer contact: Manual and user’s guide. Thames
Ditton, United Kingdom: Author.
SHL Group. (1999a). OPQ32: Manual and user’s guide. Thames Ditton,
United Kingdom: Author.
SHL Group. (1999b). Work Styles Questionnaire, Version n: Manual and
user’s guide. Thames Ditton, United Kingdom: Author.
Spencer, L. M., & Spencer, S. M. (1993). Competence at work: Models for
superior performance. New York: Wiley.
Tett, R. P., Guterman, H. A., Bleier, A., & Murphy, P. J. (2000). Devel-
opment and content validation of a “hyperdimensional” taxonomy of
managerial competence. Human Performance, 13, 205–251.
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures
as predictors of job performance: A meta-analytic review. Personnel
Psychology, 44, 704 –742.
Appendix
Great Eight, 20 Competency Dimension and 112 Competency Component titles from the SHL
Universal Competency Framework
1 Leading and Deciding
1.1 Deciding & Initiating Action
1.1.1 Making Decisions
1.1.2 Taking Responsibility
1.1.3 Acting with Confidence
1.1.4 Acting on Own Initiative
1.1.5 Taking Action
1.1.6 Taking Calculated Risks
1.2 Leading and Supervising
1.2.1 Providing Direction and Coordinating Action
1.2.2 Supervising and Monitoring Behavior
1.2.3 Coaching
1.2.4 Delegating
1.2.5 Empowering Staff
1.2.6 Motivating Others
1.2.7 Developing Staff
1.2.8 Identifying and Recruiting Talent
2 Supporting and Cooperating
2.1 Working with People
2.1.1 Understanding Others
2.1.2 Adapting to the Team
2.1.3 Building Team Spirit
2.1.4 Recognizing and Rewarding Contributions
2.1.5 Listening
2.1.6 Consulting Others
2.1.7 Communicating Proactively
2.1.8 Showing Tolerance and Consideration
2.1.9 Showing Empathy
2.1.10 Supporting Others
2.1.11 Caring for Others
2.1.12 Developing and Communicating Self-knowledge and Insight
2.2 Adhering to Principles and Values
2.2.1 Upholding Ethics and Values
2.2.2 Acting with Integrity
2.2.3 Utilizing Diversity
2.2.4 Showing Social and Environmental Responsibility
3 Interacting and Presenting
3.1 Relating & Networking
3.1.1 Building Rapport
3.1.2 Networking
3.1.3 Relating Across Levels
3.1.4 Managing Conflict
3.1.5 Using Humor
3.2 Persuading and Influencing
3.2.1 Making an Impact
3.2.2 Shaping Conversations
3.2.3 Appealing to Emotions
3.2.4 Promoting Ideas
3.2.5 Negotiating
3.2.6 Gaining Agreement
3.2.7 Dealing with Political Issues
3.3 Presenting and Communicating Information
3.3.1 Speaking Fluently
3.3.2 Explaining Concepts and Opinions
3.3.3 Articulating Key Points of an Argument
3.3.4 Presenting and Public Speaking
3.3.5 Projecting Credibility
3.3.6 Responding to an Audience
4 Analyzing and Interpreting
4.1 Writing and Reporting
4.1.1 Writing Correctly
4.1.2 Writing Clearly and Fluently
1202
BARTRAM
4.1.3 Writing in an Expressive and Engaging Style
4.1.4 Targeting Communication
4.2 Applying Expertise and Technology
4.2.1 Applying Technical Expertise
4.2.2 Building Technical Expertise
4.2.3 Sharing Expertise
4.2.4 Using Technology Resources
4.2.5 Demonstrating Physical and Manual Skills
4.2.6 Demonstrating Cross Functional Awareness
4.2.7 Demonstrating Spatial Awareness
4.3 Analyzing
4.3.1 Analyzing and Evaluating Information
4.3.2 Testing Assumptions and Investigating
4.3.3 Producing Solutions
4.3.4 Making Judgments
4.3.5 Demonstrating Systems Thinking
5 Creating and Conceptualizing
5.1 Learning and Researching
5.1.1 Learning Quickly
5.1.2 Gathering Information
5.1.3 Thinking Quickly
5.1.4 Encouraging and Supporting Organizational Learning
5.1.5 Managing Knowledge
5.2 Creating and Innovating
5.2.1 Innovating
5.2.2 Seeking and Introducing Change
5.3 Formulating Strategies and Concepts
5.3.1 Thinking Broadly
5.3.2 Approaching Work Strategically
5.3.3 Setting and Developing Strategy
5.3.4 Visioning
6 Organizing and Executing
6.1 Planning and Organizing
6.1.1 Setting Objectives
6.1.2 Planning
6.1.3 Managing Time
6.1.4 Managing Resources
6.1.5 Monitoring Progress
6.2 Delivering Results and Meeting Customer Expectations
6.2.1 Focusing on Customer Needs and Satisfaction
6.2.2 Setting High Standards for Quality
6.2.3 Monitoring and Maintaining Quality
6.2.4 Working Systematically
6.2.5 Maintaining Quality Processes
6.2.6 Maintaining Productivity Levels
6.2.7 Driving Projects to Results
6.3 Following Instructions and Procedures
6.3.1 Following Directions
6.3.2 Following Procedures
6.3.3 Time Keeping and Attending
6.3.4 Demonstrating Commitment
6.3.5 Showing Awareness of Safety Issues
6.3.6 Complying with Legal Obligations
7 Adapting and Coping
7.1 Adapting and Responding to Change
7.1.1 Adapting
7.1.2 Accepting New Ideas
7.1.3 Adapting Interpersonal Style
7.1.4 Showing Cross-cultural Awareness
7.1.5 Dealing with Ambiguity
7.2 Coping with Pressure and Setbacks
7.2.1 Coping with Pressure
7.2.2 Showing Emotional Self-control
7.2.3 Balancing Work and Personal Life
7.2.4 Maintaining a Positive Outlook
7.2.5 Handling Criticism
8 Enterprising and Performing
8.1 Achieving Personal Work Goals and Objectives
8.1.1 Achieving Objectives
8.1.2 Working Energetically and Enthusiastically
8.1.3 Pursuing Self-development
8.1.4 Demonstrating Ambition
8.2 Entrepreneurial and Commercial Thinking
8.2.1 Monitoring Markets and Competitors
8.2.2 Identifying Business Opportunities
8.2.3 Demonstrating Financial Awareness
8.2.4 Controlling Costs
8.2.5 Keeping Aware of Organizational Issues
Note that each component is further defined within the framework in
terms of negative and positive behavioral indicators.
The competency titles in this Appendix are taken from the SHL Uni-
versal Competency Framework™ Profiler and Designer cards (copyright
© 2004 by SHL Group plc, reproduced with permission of the copyright
holder). These titles may be freely used for research purposes subject to
due acknowledgement of the copyright holder.
Received September 3, 2003
Revision received November 1, 2004
Accepted December 20, 2004 䡲
1203
A CRITERION-CENTRIC APPROACH TO VALIDATION
