The Utility and Need for Incorporating Noncognitive Skills Into Large-Scale Educational Assessments

Abstract

Most attention in large-scale assessments on educational progress and outcomes addresses cognitive measures of student proficiency. In part, this focus is due to the assumption that “skills” are cognitive in nature and have a high predictive value in terms of productivity. However, the predictive value of cognitive scores on worker productivity and earnings is more modest than commonly assumed. In fact, attempts to relate cognitive test scores from surveys to economic output, although meritorious, require substantial liberties in the interpretation of data. At the same time, there is considerable evidence that noncognitive attributes of individuals related to school experience are as important as—or even more important than—cognitive attributes in predicting both school outcomes and economic productivity. Noncognitive outcome measurement is more challenging to assess than cognitive because of its highly diverse dimensions and difficulties in sampling performance on these dimensions. This chapter addresses the highly incomplete knowledge base on the potential importance of noncognitive aspects of students and schools, issues of measurement and assessment, and their predictive value on adult outcomes.

Full text

UNCORRECTED PROOF
Dear Author/Editor,
Here are the proofs of your book/chapter as well as the metadata sheet.
Metadata
• Please carefully proof read the metadata, above all the name and address.
• In case there was/were no abstract(s) for this chapter/book submitted with the manuscript, the
rst 10-15 lines of the rst paragraph were taken. In case you want to replace this/these default
abstract(s), please submit new abstracts with your proof corrections.
Page proofs
• Please review the proofs and indicate any corrections either by:
− Opening the PDF le in Adobe Acrobat and using the annotations tool to add comments
or
− You may print the le and mark corrections on hardcopy. Please mark all corrections in dark
pen in the text and in the margin at least ¼” (6 mm) from the edge.
• Please note that any changes at this stage are limited to typographical errors and serious errors
of fact.
• If the gures were converted to black and white, please check that the quality of such gures is
sufcient and that all references to color in any text discussing the gures is changed accordingly.
• If the quality of some gures is judged to be insufcient, please send an improved grayscale
gure.

UNCORRECTED PROOF
Metadata of the chapter that will be visualized online
Please note: Color images will appear in color online. If they are printed
in black and white it is stated in the proofs.
Book title The Role of International Large-Scale Assessments: Perspectives
from Technology, Economy, and Educational Research
Chapter title The Utility and Need for Incorporating Noncognitive Skills Into
Large-Scale Educational Assessments
Copyright Springer Science+Business Media Dordrecht 2012
Author Family name Levin
Particle
Given name Henry M
Sufx
Division Teachers College
Organization Columbia University
Address 525 West 120 Street, New York, 10027 NY, USA
Email HL361@columbia.edu
Abstract Most attention in large-scale assessments on educational progress
and outcomes addresses cognitive measures of student prociency.
In part, this focus is due to the assumption that “skills” are cognitive
in nature and have a high predictive value in terms of productivity.
However, the predictive value of cognitive scores on worker
productivity and earnings is more modest than commonly assumed.
In fact, attempts to relate cognitive test scores from surveys to
economic output, although meritorious, require substantial liberties
in the interpretation of data. At the same time, there is considerable
evidence that noncognitive attributes of individuals related to
school experience are as important as—or even more important
than—cognitive attributes in predicting both school outcomes
and economic productivity. Noncognitive outcome measurement
is more challenging to assess than cognitive because of its highly
diverse dimensions and difculties in sampling performance on
these dimensions. This chapter addresses the highly incomplete
knowledge base on the potential importance of noncognitive aspects
of students and schools, issues of measurement and assessment, and
their predictive value on adult outcomes.

UNCORRECTED PROOF
67
Introduction
International comparisons of educational systems have become increasingly com-
mon as nations explore the potential of education for improving their citizenry and
economic productivity. It is not unusual to see headlines in the news for any particu-
lar country on how it ranks on the periodic surveys of the Program of International
Achievement (PISA), International Adult Literacy Survey (IALS), Trends in Inter-
national Mathematics and Science Study (TIMSS), and the Progress in International
Reading Literacy Study (PIRLS). Countries take their rankings very seriously, and
the media either praise their country’s performance or decry it, calling for major
educational reforms. At the same time, national and regional assessments compare
different regions and educational entities on the quality of their educational sys-
tems, primarily using the metrics of student achievement as the guide.
It is hardly surprising that the notion of a good school or good educational per-
formance is viewed through the prism of student achievement as represented by
standardized test scores. In the United States, real estate brokers use achievement
results to suggest the desirability of a particular residential neighborhood. School
districts feel pressed to raise their test scores as the primary indicator of their edu-
cational quality. Parents view the educational promise of their children in terms of
how well they do on such tests. And, of course, governments set out accountability
standards on the basis of test results as well as sanctions for poor test performance
such as those of the No Child Left Behind law. Correspondingly teachers and prin-
cipals seek ways to focus on raising achievement, even if it means narrowing the
curriculum to the subjects being tested and teaching primarily through strategies
that put instruction in the form of test formats and test practice. Clearly, there are
many advantages to the use of standardized testing, whether domestically or inter-
nationally. What students learn should be assessed, and few would question that
knowledge, and abilities to use that knowledge, are essential for human function.
M. von Davier et al. (eds.), The Role of International Large-Scale Assessments:
Perspectives from Technology, Economy, and Educational Research,
DOI 10.1007/978-94-007-4629-9_5, © Springer Science+Business Media Dordrecht 2012
Chapter 5
The Utility and Need for Incorporating
Noncognitive Skills Into Large-Scale
Educational Assessments
Henry M. Levin
H. M. Levin ()
Teachers College, Columbia University, 525 West 120 Street, New York, NY 10027, USA
e-mail: HL361@columbia.edu
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Author’s Proof !

UNCORRECTED PROOF
68
But, at least some of the attractiveness of cognitive test scores is due to the fact
that the assessment of cognitive skills has developed to the point where they are
relatively easy to measure. A relatively small sample of test performance can be ob-
tained at low cost and with what appears to have predictive validity for individuals,
at least for further academic performance and occupational placement and earnings.
Of course, this type of psychological testing has a long history of development. In
contrast, systematic assessment of other personality characteristics that may also
predict both academic and economic productivity is far less developed in educa-
tional assessments. Such social and behavioral aspects or measures of personality,
or what are commonly called noncognitive measures, are more complex in terms of
their underlying definitions, structure, and measurement, and there are many more
of these dimensions suggested in the literature. For these reasons, they are likely to
be more difficult to measure in the streamlined way—conventional testing—that is
used for cognitive outcomes. Unfortunately, even their terminologies differ among
disciplines and authors. In some cases they are called noncognitive, and in others,
affective, or social, behavioral, and emotional. For purposes of parsimony, I will use
these terms interchangeably, even though I recognize they may have very different
meanings in different contexts. My main concern will be to differentiate them from
the knowledge and skills that we normally measure with the use of cognitive test
scores.
This chapter argues that both domestic and international educational assessments
should expand their measures of educational outcomes to take account of the devel-
opment of noncognitive student attributes that are required for productive economic
and democratic participation and personal development. Some would assert that the
main ingredient for productive adulthood is the knowledge and abilities acquired,
and that these are best measured through cognitive testing. However, that view is
countered by the fact that microeconomic studies show that such tests explain only
a relatively small portion of the variance in earnings and supervisory ratings and
a minor portion of the statistical relation between schooling attainments and eco-
nomic outcomes. This is not to argue the irrelevance of what is measured by the test
scores to adult outcomes and economic results, but only that they account for much
less power in molding adult outcomes than is normally assumed and should not be
used exclusively as a statistical measure to evaluate the educational merit or quality
of educational systems. Cognitive achievement is important and should continue
to be assessed. But it is a highly incomplete category for measuring student and
adult success. This chapter sounds an appeal to consider the potential importance of
noncognitive skills and dimensions of human behavior as they comprise important
adult competencies and the role of schools in developing them. But first we must
acknowledge them, conceptualize their roles and identities, and measure them. The
latter is where large-scale assessment ultimately enters the picture. What follows is
designed to make the case.
Consider the following presentation by Alex Inkeles, one of the foremost social
psychologists of personality, in his study of individual and societal productivity.
Inkeles (1966) relied on a functionalist framework to identify the requirements of
competent adulthood and the “socialization of competence”:
H. M. Levin
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Author’s Proof !

UNCORRECTED PROOF
69
To perform effectively in contemporary society, one must acquire a series of qualities I
believe to be developed mainly in the socialization process. Effective participation in a
modern industrial and urban society requires certain levels of skill in the manipulation of
language and other symbol systems, such as arithmetic and time; the ability to comprehend
and complete forms; information as to when and where to go for what; skills in interpersonal
relations which permit negotiation, insure protection of one’s interests, and provide main-
tenance of stable and satisfying relations with intimates, peers, and authorities; motives to
achieve, to master, to persevere; defenses to control and channel acceptably the impulses to
aggression, to sexual expression, to extreme dependency, a cognitive style which permits
thinking in concrete terms while still permitting reasonable handling of abstractions and
general concepts; a mind which does not insist on excessively premature closure, is tolerant
of diversity, and has some components of flexibility; a conative style which facilitates rea-
sonably regular, steady, and persistent effort, relieved by rest and relaxation but not requir-
ing long periods of total withdrawal or depressive psychic slump; and a style of expressing
affect which encourages stable and enduring relationships without excessive narcissistic
dependence or explosive aggression in the face of petty frustration. This is already a long
list and surely much more could be added. (Inkeles 1966, pp. 280–281)
What is striking about this list is the complexity of an expert’s view on what needs
to be developed in the human personality for adult competence in modern life and
the relatively limited role of standardized tests for shedding light on these compe-
tencies.
In subsequent work, Inkeles and Smith (1974) developed an index of modernism
composed of many items, reflecting the following: informed citizenship; personal
efficacy; independence and autonomy relative to traditional sources of influence in
making personal decisions; and openness to new experience and ideas constructed
with 19 subscales. These scales were used to measure “modernity” among almost
6,000 men in six developing countries—Argentina, Bangladesh, Chile, India, Israel,
and Nigeria—using a stratified sample to obtain representation of distinct occupa-
tions and rural and urban populations. The researchers also formulated a range of
socialization variables that could influence modernity attitudes: education, work
experience, contact with mass media, consumer goods possessed, father’s educa-
tion, urbanism of residence, skill level, length of urban residence, modernity of
workplace, modernity of home, and school background. This combination of vari-
ables was able to explain statistically between 32–62 % of the variance in modernity
scores, considerably higher than most earnings equations among individual adults,
even today. In all six countries, education was the most powerful statistical influ-
ence, at least two to three times more powerful than any other influence in standard-
ized coefficients.
The sheer breadth of both the underlying theory and empirical findings of the In-
keles framework highlight the narrowness of the measures of educational outcome
on which our international surveys are focusing. That is, schools have far more
impact on important components of human formation that matter in the workplace,
community, and home than just what is measured by test scores. In this chapter I
will not attempt to develop new empirical information, largely because there al-
ready exists an impressive pattern of evidence that suggests: (1) schools influence
personality traits that are determinants of both achievement and work productivity;
and (2) by limiting attention only to the cognitive test scores dimension of educa-
5 The Utility and Need for Incorporating Noncognitive Skills …
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
Author’s Proof !

UNCORRECTED PROOF
70
tional outcomes, we are influencing the establishment of educational policies that
are likely to restrict social and economic productivity.
I will recommend that large-scale assessments, both international and domestic,
move beyond the focus on cognitive test scores to embrace a larger set of potential
educational outcomes including student attitudes, behaviors, and other noncogni-
tive measures that are important for explaining valuable individual and social out-
comes including economic productivity. I recognize that there is no simple dividing
line between so-called cognitive and noncognitive educational results or skills im-
parted by the educational system. Although we may refer to noncognitive attributes
or skills as social and behavioral attributes, it is clear that they can be heavily bound
up with cognitive knowledge. As a working distinction we can distinguish the cog-
nitive attributes that are measured by test scores, a category limited to knowledge
in particular test domains or subjects, and modes of measuring these domains or
subjects as the cognitive focus of schools. In contrast this chapter refers to noncog-
nitive skills essentially as those that are generally viewed as attitudes, behaviors,
and values that contribute to adult competencies.1 We should keep in mind that
some of these interact with cognitive skills such as problem-solving ability, where
modes of analytic and relational thinking must draw upon a knowledge base. While
the distinctions between cognitive and noncognitive will not be sharply delineated,
they will be sufficiently differentiated to understand the thrust of the arguments.
The Test Score Image and Reality
• Few college educated individuals will forget their college entrance scores (e.g.,
SAT) or test scores for graduate or professional school admissions, even after
many decades.
• Academics have fought bitterly over the origins of IQ (phenotype or genotype),
but few question the importance and social value of IQ as they take pride and
ownership in their own high IQs.
Cognitive testing has an impressive history. Its development and sophistication
have far outpaced assessment in noncognitive areas of performance in its precision,
statistical analysis, and widespread adoption. The test score illusion is that we tend
to overstate the importance of tests in accounting for human productivity. At both
individual and societal levels, they carry considerable influence. But, their impor-
tance is greater in the popular imagination than the evidence supports. The advent
of human capital theory in economics had important and deservingly profound ef-
fects on the thinking about the link between education and economic output. Edu-
cational investments became viewed as investments in human beings that increased
1 The most ambitious and encyclopedic review of personality characteristics as they relate to eco-
nomic outcomes is found in the comprehensive and magisterial treatment by Almlund et al. (2011).
Also see Borghans et al. (2008a).
H. M. Levin
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
Author’s Proof !

UNCORRECTED PROOF
71
productive skills, leading to greater productivity and economic output. Little was
said about the nature of such skills. In his pioneering work on human capital, Gary
Becker (1964) provides almost no analysis of the skills that are encompassed by
human capital. And the vacuum on precisely what skills were developed through
human capital investments—and the vacuum filler of educational attainment data—
combined to make the years of education attained as the standard measure of human
capital. The most comprehensive and widely used sources of data such as the U.S.
Census or household surveys on earnings of workers reported the amount of educa-
tion attained, but not test results.
Measures of educational attainment in terms of number of years of schooling
are highly errorful measures. These are self-reported and lack information on areas
of study, educational quality, rigor of courses, and student effort. As a result it was
logical to seek data sources that had more direct measures of academic attainment,
and test results were a more direct verification of skills than the amount of time
spent in schools. It seemed reasonable that most of what was learned in schools
could be measured by test scores.
This perspective was first questioned by Gintis (1971) and Bowles and Gintis
(1976) in the decade following the human capital revolution in their attempt to show
that school organizations reflect the practices of employers in student development
where many similar noncognitive demands are placed on both students and work-
ers. More recently, Bowles et al. (2001) summarized much of the ensuing research
that has addressed this phenomenon. One of their most salient findings is that only
a small portion of the overall statistical impact of schooling on earnings can be ex-
plained by test scores per se. A summary of 25 studies over a period of four decades
(late 1950s to early 1990s) provided 58 estimates of earnings functions where test
scores were available. Starting with the conventional human capital formulation in
which demographics, socioeconomic status, and schooling are used as explanatory
variables for predicting earnings, they estimate the coefficient for the schooling
contribution to earnings (usually measured by years of education). They then posit
that if the schooling variable is a just a rough proxy for achievement, it is highly
errorful relative to a direct measure of what is learned and contributes to produc-
tivity, a measure of test scores. By adding the test score to the equation, they can
test “how much” of the “naïve” schooling effect indicated by monetary returns to
years of schooling is reduced by a direct measure of cognitive skill created through
education. Across the 58 estimates they find that the schooling coefficient retains
about 82 % of its “naïve” value, suggesting that most of the effect of schooling on
earnings is due to factors other than those measured by standardized tests (Bowles
et al. 2001, pp. 1147–1150)
It is almost an article of faith among policymakers and the general public that
the impact of cognitive skills in labor markets is rising. Much of the support for
this view comes from the evidence of one well-constructed study that compares
test score impacts on earnings between 1978 and 1986 and finds that there was a
rise in hourly wage over those years based on returns to mathematic scores (Mur-
nane et al. 1995). But an analysis of a wider range of studies finds no such trend
among 65 estimates from 24 studies reflecting a 30-year period (Bowles et al. 2001,
5 The Utility and Need for Incorporating Noncognitive Skills …
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
Author’s Proof !
AQ1

UNCORRECTED PROOF
72
pp. 1154–1156). This study not only found no rising trend, but relatively small
estimated impacts of mathematics achievement on wages. A standard deviation in
test score was associated with a 10 % increase in wages, equal to about one year
of schooling. Of special pertinence is that no existing educational intervention has
shown effects even close to one standard deviation. Of the relatively few that seem
to improve mathematics achievement, it is rare to find results that exceed one-fifth
of a standard deviation. A study for the United finds no increase in the returns to
cognitive skills for the period 1995–2004, the most recent period found for these
studies (Vignoles et al. 2011). The overall support for the rising effect of cognitive
skills is absent or mixed in other research studies and is beset with methodological
issues (Cawley et al. 2001), which should at least raise a caution flag in asserting
rising returns.
The exaggeration of cognitive impacts of workers on worker productivity has
also been a feature of the literature on using test scores directly for worker selection.
The most important public use was that by the U.S. Employment Service, which
used the General Ability Test Battery (GATB) to rank workers for referral to em-
ployer requests for candidates. The GATB includes subtests of intelligence, verbal
aptitude and numerical aptitude as well as a range of other measures. State employ-
ment services informed prospective employers that they would refer the most pro-
ductive applicants for consideration on the basis of the GATB rankings. However,
there was considerable controversy over the practice of norming the rankings sepa-
rately within race so that two individuals of different races with different raw scores
might have the same percentile ranking. Because blacks had considerably lower
scores on the GATB, the normalized rankings for blacks had a much lower GATB
score than a white with the same ranking. The National Research Council of the Na-
tional Academy of Sciences and National Academy of Engineering formed a panel
that was asked to focus especially on the validity claims for GATB and other em-
ployee tests that were asserted to have predictive validities of .6–.7 on supervisory
ratings of worker productivity according to leading advocates (Hartigan and Wigdor
1989). The study panel found that the estimated predictive validities were vastly
inflated by questionable procedures, so the best estimate of validity was about .25,
a dramatic reduction from the claims. Thus, the tests used to refer workers to em-
ployers accounted for only about 6 % of the variance in performance, leaving 94 %
to be explained by other characteristics of workers. More recent summaries of the
empirical literature across many different studies and measures support this modest
finding (Sackett et al. 2001).
Even well-specified earnings functions that include more than one direct mea-
sure of cognitive skill and many other covariates show low total explained variance,
typically one third or less (Murnane et al. 2001). And the cognitive measures in
themselves show “modest” relations to earnings (Murnane et al. 2000). Clearly cog-
nitive abilities are important for many important dimensions of adult performance,
including economic, civic, and personal demands upon individuals. But they are
far from dominant in explaining economic and social outcomes and are probably
considerably less important than commonly believed. Yet the domestic and interna-
tional comparisons of educational achievement focus almost exclusively on these.
H. M. Levin
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
Author’s Proof !

UNCORRECTED PROOF
73
In the next section we address what is known about noncognitive aspects of school-
ing and work performance.
Multiple Sources of Support for Noncognitive Measures
When one reviews many different sources of information, the importance of social
and behavioral competencies beyond cognitive skills is apparent. In this section, I
will provide brief glimpses of a number of these sources.
Employer Needs
It is common for employers to explain that they seek workers both with good cog-
nitive skills and social/behavioral competencies to qualify for employment. This is
not a new phenomenon. Almost three decades ago, the National Research Council
convened a panel to set out the competencies that employers desired (National
Research Council 1984). The panel, composed almost entirely of employers from
a large range of business sectors and a few government agencies, was charged with
studying and formulating the set of core competencies that they would want among
the high school graduates they employ.2 The motivation of the NRC for forming
the panel was to recognize the knowledge needs of the changing workplace for
high school graduates. Panel members were asked to work closely with supervi-
sors in their human resources departments to get a ground-level view of worker
requirements.
The panel developed a comprehensive list that was heavy on cognitive require-
ments such as command of the English language, reasoning, reading, writing,
computation, and knowledge of basic science and technology. But the panel found
the same level of concern by human resource supervisors for a substantial list of
behavioral and social worker characteristics on “Interpersonal Relationships” and
“Personal Work Habits and Attitudes.” These included such attributes as interacting
in a socially appropriate manner; demonstrating respect for the opinions, customs,
and individual differences of others; handling conflict maturely; and participation
in reaching group decisions. They also included a realistic positive attitude toward
one’s self; self-discipline, including regular and punctual attendance and depend-
ability; ability to set goals and allocate time to achievement of them; and capacity to
accept responsibility (National Research Council 1984). To the degree that national
testing such as the National Assessment of Educational Progress (NAEP) and the
international comparisons of educational achievement are motivated by preparation
for the workplace and economic productivity, their results largely ignore these per-
spectives in providing information on educational preparation.
2 In the spirit of full disclosure, I was the “token academic” on this panel.
5 The Utility and Need for Incorporating Noncognitive Skills …
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
Author’s Proof !

UNCORRECTED PROOF
74
The Employer Employment Survey in the early 1990s, sponsored by the US De-
partment of Education, surveyed more than 4,000 employers “to identify employers’
practices and expectations in their search for a skilled and proficient work force.”
When asked to identify the recruitment characteristics that they used to make hiring
decisions on a scale of 1–5 (with 5 being the highest), applicant’s attitude was 4.6
and communication skills were 4.2, the two highest in the survey. Tests adminis-
tered by the firm, academic grades in school, and reputation of applicant’s school
were at 2.5 or 2.4, at the bottom of the list (Zemsky and Iannozzi 1995).
The latest National Employer Skills Survey for England 2009 (Shury et al. 2010)
is notable for its lack of discussion of academic skills. The survey finds that about
one fifth of the enterprises are affected by a skills gap, but for 71 % of these, the
“main cause” is lack of experience and recent recruitment. Thus, it is no surprise to
find that 64 % of employers were concerned with a lack of technical, practical, or
job-specific skills. A third of employers implicated a lack of motivation on the part
of workers. Employers also were concerned about such skills as customer-handling
(41 %), problem-solving (38 %), and team-working (37 %), with literacy and nu-
meracy further down the list. That is, social and behavioral skills were important
challenges for UK employers in this recent study.
It seems obvious that from the perspective of employer concerns, both in the past
and more recently, there is at least as much concern for the noncognitive attributes
of workers as for the cognitive ones. Indeed, the former may even be a stronger
source of concern.
Cognitive or Noncognitive Effects
The Perry Preschool is best known for its role as the earliest study showing substan-
tial long-term effects of preschool. The study followed the lives of 123 persons who
had been randomly assigned as 3–4-year-olds to experimental treatment and control
groups where the experimental group was enrolled in the preschool program. The
subjects were black inner-city children from poverty families. Study participants
were followed up to the age of 40 for their educational results and life experiences.
The experimental students showed initial intellectual and literacy gains over the
students in the control group, but the differences faded out in the early elementary
years. Yet when comparisons were made of life accomplishments, the Perry Pre-
school participants did substantially better than the control group in terms of edu-
cational attainments, reduction in crime, earnings, employment, and welfare costs
(Schweinhart 2010, p. 161). For example, 28 % of the Perry participants had been
convicted of a crime by age 40, relative to 52 % of the control group, and earnings
were about one third higher. High school graduation rates were higher for the Perry
group, and their attitudes toward school were more positive. Evaluations of the in-
vestments in Perry Preschool show a high return (Heckman et al. 2010). These types
of outcomes are important to both the individuals who benefited and society, even
though they do not seem to be attributable to the early test results. One interpreta-
H. M. Levin
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
Author’s Proof !

UNCORRECTED PROOF
75
tion is that Perry mainly had an influence on school readiness and other noncogni-
tive behaviors that contributed to the increase in school and life success.
A different challenge is the puzzle of the findings on the economic success and
social experience of students who acquire the General Education Development
(GED) credential in lieu of graduating from high school. The purpose of the GED
is to credential dropouts as equivalent to high school graduates if they succeed on
the GED examination. Heckman and Rubinstein (2001) found that they do about
as well on a cognitive test as high school graduates who do not enroll in college.
But their earnings patterns are considerably below high school graduates, and when
adjusted for their cognitive performance, are even lower than those of high school
dropouts who do not take the GED. In addition, their ultimate education attainment
also lags behind that of dropouts who did not take the GED. The authors conclude
that the GED recipients have lower noncognitive skills that count in employment,
and this interpretation is buttressed by a measure of illicit activity that is higher for
the GED students than for the non-GED dropouts or high school graduates.
A third potential example is that of the Tennessee Class Size or Star experiment
in which students in grades from kindergarten to grade three were assigned to large
classes (23–25 students) or small classes (13–17 students) at random in the schools
chosen for the experiment. Students could receive from one to four years of the
small-class treatment or none. In his review of the study, distinguished statistician
Fred Mosteller called the study “…one of the most important education investiga-
tions ever carried out” (Mosteller 1995). Test results showed moderate achievement
advantages in reading, word study, and mathematics that increased with the dura-
tion of the treatment. But perhaps what is most surprising is the substantial differ-
ence in graduation rates almost a decade later. This was particularly so for the dis-
advantaged students—those eligible for a free or reduced cost lunch. Disadvantaged
students with smaller classes for four years had graduation rates 18 % points higher
than similar students who had attended only regular size classes, 88–70 %. This was
found to be well beyond the predictive effect of the early academic achievement
that was experienced, suggesting that noncognitive effects accounted for at least
a portion, and perhaps a large portion, of the higher graduation performance (Finn
et al. 2005). Insights into a mechanism for explaining this noncognitive effect is
found in a recent study that linked class size reduction to improving student learning
behaviors (Dee and West 2011).
An intriguing study (Lindqvist and Vestman 2011) from Sweden evaluated cog-
nitive and noncognitive dimensions of military enlistees (enlistment is a mandatory
requirement for all Swedish males). All enlistees filled out an extensive question-
naire with 70–80 questions. A certified psychologist was provided with this infor-
mation as well as measures of cognitive ability and other attributes. Following a
specified set of procedures, the enlistee was interviewed by the psychologist and
evaluated according to the perceived ability of the conscript to cope with the psy-
chological requirements of military service. Each conscript was given a score ac-
cording to the same distribution used for the cognitive ability score. Using a random
sample of men born between 1965–1984, the authors evaluated the impact of cog-
nitive and noncognitive measures on wages, unemployment, and annual earnings.
5 The Utility and Need for Incorporating Noncognitive Skills …
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
Author’s Proof !

UNCORRECTED PROOF
76
They found that men who do poorly in the labor market lack noncognitive abilities.
In contrast, cognitive ability is a stronger predictor of wages and earnings for work-
ers with earnings above the median.
Schools and Noncognitive Outcomes
One question that might arise is whether schools can actually change noncognitive
outcomes. Relatively little attention has been devoted to systematic consideration of
this question and its measurement because there is not the body of rigorous research
available that exists for cognitive measures. However, considerable attention has
been devoted to this subject in early childhood education, where attempts have been
made to see if students are “school ready”.
Cognitive control, self-regulation, or executive function (EF) is the focus of a
study testing directly whether a noncognitive skill can be taught effectively. Dia-
mond et al. (2007) evaluated The Tools of the Mind curriculum, a framework that
contains 40 EF-promoting activities. Students and teachers were assigned randomly
to The Tools of the Mind curriculum and an alternative. The Tools of the Mind
curriculum not only had significant effects in promoting greater EF, but the higher
EF in itself was associated with higher standardized measures of reading. The im-
portance of this finding is magnified by the fact that EF has been more strongly
linked to school readiness than cognitive measures (Blair and Razza 2007). A more
extensive, recent randomization study confirms the findings on the educational ef-
fects of The Tools of the Mind curriculum, and particularly its impact on social
development of the child and improvement of classroom experience (Barnett et al.
2011). Distinguished psychologist Albert Bandura (1997) has also maintained that
there is an impressive knowledge base showing that self-efficacy (the belief that one
can influence a personal outcome) can be conditioned in the young in his extensive
lifelong study of self-efficacy.
Clearly, not all prekindergarten experiences contribute to children’s school readi-
ness, as evidenced by a more general study that focused on prekindergarten impacts
on school cognitive outcomes and behavior problems without examining the pro-
gram specifics (Magnuson et al. 2007). In contrast, The Tools of the Mind studies
highlight that the specific goals of the preschool program are central in determining
whether they improve noncognitive functioning in the school environment as ap-
plied to preschool experiences of any type. Program design matters in exploring the
impacts of educational programs.
Overall summaries of the literature also confirm the importance of early child-
hood interventions on behavioral or socioemotional change. Nores and Barnett
(2010) reviewed a total of 38 studies reviewing 30 interventions in 23 countries that
had applied quasiexperimental or random assignment designs. They took into con-
sideration the type of intervention, sample size, study design and duration, country,
target group, subpopulations, and dosage of interventions. They found both cogni-
tive benefits and behavioral benefits. Camilli et al. (2010) undertook a meta-anal-
H. M. Levin
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
Author’s Proof !

UNCORRECTED PROOF
77
ysis of 123 comparative studies of early childhood interventions. The evaluation
of all programs in the review had been designed using experimental principles. Al-
though the largest effects were found for cognitive outcomes, preschool experience
was also found to be associated with student’s social skills and school progress.
Duncan and associates (2007) used six longitudinal data sets to estimate the links
between academic, attention, and socioemotional skills at school entry and subse-
quent school reading and math achievement. Attention-related skills refer to task
persistence and self-regulation or EF. We do not know the content of the preschool
experience, so these measures are recorded at school entry. They found math skills
to show the greatest predictive power, followed by reading and attention skills. As
with the Magnuson et al. (2007) study, the focus was on participation in preschool,
but not on specific programs that focus on noncognitive skill development, as did
The Tools of the Mind curriculum. Duncan and Magnuson (2011) also find impor-
tant relations between both early childhood cognitive scores and social behavior on
later educational outcomes and criminal involvement.
The most extensive evaluation of the direct study of the teaching of social and
emotional skills and their impact is found in Durlak et al. (2011). This work is based
upon a meta-analysis of 213 school-based social and emotional learning (SEL) pro-
grams from kindergarten through high school, studies encompassing 270,000 chil-
dren overall from ages 5–18. Only intervention studies that had control groups were
included. Outcomes included six criteria:
• Social and emotional skills—includes evaluations of different types of cogni-
tive, affective, and social skills related to such areas as identifying emotions
from social cues, goal setting, perspective taking, interpersonal problem solving,
conflict resolution, and decision making.
• Attitudes toward self and others—includes positive attitudes about the self,
school, and social topics, including self-perceptions (e.g., self-esteem, self-
concept, and self-efficacy), school bonding (e.g., attitudes toward school and
teachers), and conventional (i.e., prosocial) beliefs about violence, helping oth-
ers, social justice, and drug use.
• Positive social behavior—includes outcomes such as getting along with others
derived from the student, teacher, parent, or an independent observer on the basis
of daily behavior as opposed to hypothetical situations.
• Conduct problems—includes measures of different types of behavior problems,
such as disruptive class behavior, noncompliance, aggression, bullying, school
suspensions, and delinquent acts.
• Emotional distress—includes internalized mental health issues. These included
reports of depression, anxiety, stress, or social withdrawal, which could be pro-
vided by students, teachers, or parents.
• Academic performance—includes standardized reading or math achievement
test scores from such measures as the Stanford Achievement Test or the Iowa
Test of Basic Skills, and school grades in the form of students’ overall grade
point average (GPA) or their grades in specific subjects (usually reading or
math). Only data drawn from school records were included.
5 The Utility and Need for Incorporating Noncognitive Skills …
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
Author’s Proof !

UNCORRECTED PROOF
78
Meaningful effect sizes were found for all six criteria: social and emotional skills,
0.57; attitudes, 0.23; positive social behavior, 0.24; student conduct problems,
0.22; emotional distress, 0.24; and academic performance, 0.27. Thirty-three of
the academic performance studies had follow-up evaluations of at least six months
after the intervention ended, with a median follow-up time of about one calendar
year. All effect sizes continued at statistically significant levels, with the effect
size for academic performance at 0.32 for the subgroup, suggesting that develop-
ment of social and emotional skills have particular salience for improving student
achievement.
A reasonable summary of this literature is that noncognitive skills can be taught
through purposive interventions and that they can make a difference for many valu-
able social/behavioral outcomes and for student achievement. The latter is an im-
portant conclusion because not only are these outcomes important in themselves,
but they also appear to have a positive impact on achievement. In the Durlak et al.
(2011) study, the average effect size among studies is adequate to raise standard-
ized student achievement scores by 11 percentiles. This is equivalent to an increase
of PISA scores by about 30 points—the difference between the United States and
higher-scoring Canada, and a rise in rankings from 17th to 5th place, or from 14th
to 3rd place if we exclude cities or city-states Shanghai, Hong Kong, and Singapore.
While this may not be a simple matter of policy, it does provide a framework for
considering the potential of noncognitive interventions.
Schooling and Labor Market Effects
Without question, the scholar who has done the most to develop an understanding
of the role of noncognitive skills in educational and economic outcomes is James
Heckman of the University of Chicago, aided by his colleagues.3 Heckman has
not only called attention to the importance of noncognitive skills, but has worked
with psychologists and neurologists to estimate optimal time patterns of invest-
ment between development of the different types of skills and their impact on labor
market returns (Knudsen et al. 2006). His masterful article with Flavio Cunha is
considered to be the most ambitious and sophisticated attempt to both formulate a
theory of optimal investment between cognitive and noncognitive skills from birth
to the labor force, but also to apply the model to a specific longitudinal data set to
measure the impact of cognitive and noncognitive skill development on earnings
(Cunha and Heckman 2008). The authors create a battery of noncognitive scores
3 Heckman has produced most of the important scholarship on this subject and has continued his
program to deepen understanding of the role of noncognitive skills. It would take pages to list all
of his contributions. However, it would be helpful to review the citations to Heckman and col-
leagues in the bibliography of the masterful article by Borghans et al. (2008a). Heckman’s role is
central to the content of the symposium on “The Noncognitive Determination of Labor Market and
Behavioral Outcomes,” XVII (4).
H. M. Levin
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
Author’s Proof !

UNCORRECTED PROOF
79
from their data set focused on an antisocial construct using student anxiety, head-
strongness, hyperactivity, and peer conflict to go along with cognitive test scores in
this analysis. Based upon the psychological, neurological, social, and other aspects
of child development, they model the developmental path and estimate the impact
of investments in cognitive skill and noncognitive skill on high school graduation
and earnings (at age 23) at three different periods during the span from age 6–13.
As the child ages, the impact of investment returns shifts markedly from cognitive
skills at the earlier ages (6–9) to noncognitive skills during the later period.
Clearly, this analysis, if it stands up to replication, has profound implications for
school policy and the construction of educational programs. The work of Heckman
and his students stands as a milestone in considering the optimal mix of interven-
tions and policy implications for enhancing human development through a combi-
nation of appropriate strategies of both cognitive and noncognitive skills. This work
also seems to correspond in many of its assumptions with the attempt to create a
unified theory of child development by Sameroff (2010), suggesting that the lead-
ing edge of this research is moving in similar directions. As with the program of
Heckman, Sameroff has developed a conceptual approach that interconnects the
individual and context in a dynamic manner.
Perhaps the best single source on the role of noncognitive skills and the economy
is the symposium on “The Noncognitive Determinants of Labor Market and Be-
havioral Outcomes” (2008).4 This unusually focused volume contains an article by
Borghans et al. (2008b) that analyzes tradeoffs in roles of caring and directness in
jobs that have different interpersonal requirements. Caring requires cooperation,
whereas directness requires clear communication. The returns to these attributes
depend upon relative supply and demand. The authors find that returns to these
roles, which are held in different combinations by different individuals, match their
assignment models. Articles by Fortin (2008), Krueger and Schkade (2008), Segal
(2008), and Urzua (2008) address other labor market consequences related to non-
cognitive skills and roles of workers as well as impacts of noncognitive skills of
students.
Noncognitive Variables
There exist so many concepts, constructs, and names for the personality and social
and behavioral characteristics that are referred to as noncognitive that I will not
allocate much space to attempting to list them or categorize them. The most com-
prehensive analysis of personality and its roles in labor markets, health, crime, and
civic behavior is that of Almlund et al. (2004).5 However, it is important to provide
4 Also see the papers presented at the recent IZA Workshop: Cognitive and Non-Cognitive Skills,
January 25–27, Bonn, Germany. Available at: http://www.iza.org/link/CoNoCoSk2011.
5 This is an overwhelmingly ambitious exercise to map personality traits into economic modelling.
5 The Utility and Need for Incorporating Noncognitive Skills …
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
Author’s Proof !
AQ2

UNCORRECTED PROOF
80
at least a glimpse of how they have been referred to and used in the psychological
literature.
The Five-Factor Model
For at least the last two decades, the five-factor model of personality has been used
to relate noncognitive skills to academic achievement, educational attainment, and
other outcomes. The history is one in which an accumulation of different hypoth-
eses and empirical studies were used to create statistical factor analytic dimensions
by independent researchers (Digman 1990). The consolidation of many different
dimensions of personality into the five-factor model was an attempt to find a basic
structure for what was a highly disorganized and idiosyncratic set of measures and
constructs. Accordingly, these have been considered to be the basic structure under-
lying all personality traits and have been used to integrate a variety of findings in
personality psychology.
The Big Five factors are:
1. Openness—inventive and curious as opposed to consistent and cautious
2. Conscientiousness—efficient and organized as opposed to easygoing and
careless
3. Extraversion—outgoing and energetic as opposed to solitary and reserved
4. Agreeableness—friendly and compassionate as opposed to cold and unkind
5. Neuroticism—sensitive and nervous as opposed to secure and confident
These categories have been used in many studies to predict behavior and are promi-
nent in the massive review by Almlund et al. (2011). An example of a study that
explores the relation between the Big Five and academic outcomes is Noftle and
Robins (2007). Four different university student samples were used in the study.
After controlling for high school GPA and SAT scores, the Big Five were tested,
but only the dimension of “conscientiousness” was found to predict college GPA.
SAT verbal score was predicted by “openness.” The researchers also found that
academic effort and perceived academic ability served to mediate the conscien-
tiousness-SAT relationship, independent of academic achievement.6 An example
of the use of the Big Five for a measure of workplace productivity is the study of
Neuman and Wright (1999). These authors studied the relation between personality
characteristics of 316 full-time human resource representatives at local stores of a
large wholesale department store enterprise. They found that “agreeableness” and
“conscientiousness” predicted peer ratings of team member performance beyond
controls for job-specific skills and general cognitive ability.
Promising work on the further development of noncognitive constructs and mea-
sures is being undertaken by the Research Division of Educational Testing Ser-
6 From an economist’s perspective, there would be concern for problems of endogeneity in use of
some of the explanatory variables.
H. M. Levin
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
Author’s Proof !

UNCORRECTED PROOF
81
vice (Kyllonen et al. 2008) in Princeton, NJ. This work focuses on both personality
characteristics and motivation, reviewing studies that link them to educational out-
comes. Their work considers various measurement approaches and also documents
particular interventions in developing certain personality facets that lead to higher
achievement. The report develops an approach to implement a comprehensive psy-
chosocial skills assessment at middle school and high school levels. At this time,
this report is protected as proprietary and its specific contents and findings cannot
be cited, although I expect that it might be released in modified form in the near
future.
Summary and Implications for Educational Assessments
Modern societies demand much of their members, and fostering competence in
meeting these demands must be a high social priority. Among all of the vehicles for
socializing the young, schools are a very powerful one because of the considerable
time spent there and the peculiar functions of schools to prepare the young in many
ways for adulthood. Clearly knowledge and cognitive functioning are an impor-
tant goal of schools and provide crucial skills for creating productive workers and
citizens. But noncognitive or behavioral/social skills and attitudes are also crucial
and of at least the same level of importance. Even with the same cognitive achieve-
ment, differences in effort, self-discipline, cooperation, self-presentation, tolerance,
respect, time management, and other noncognitive dimensions form both healthy
character and contribute to productive relations in workplaces, communities, fami-
lies, and politics.
To a large degree, the almost singular focus on test score performance in educa-
tional assessments at both domestic and international levels is not without founda-
tion. The cognitive domains tested are important determinants of both educational
outcomes and life chances, the measurement technologies are well developed, and
the process of assessment of cognitive skills is parsimonious in that a valid sample
of cognitive knowledge and behavior can be obtained and evaluated at low cost. But
I have emphasized that the assumptions that cognitive skills are all that counts, and
that they have singular influence on producing healthy and productive adult person-
alities, goes well beyond the evidence. Although they are important determinants
of productivity and income at both individual and societal levels, empirical studies
show that their measurable influence is far more modest than generally assumed.
Moreover, their impact does not seem to be rising despite the conventional wisdom.
Employers who indicate skill shortages place as much or more emphasis on getting
workers with proper attitudes and social behaviors as cognitive competencies. The
studies of Heckman and colleagues show that the connections between noncogni-
tive skills and workplace productivity are of comparable importance overall and of
even greater importance than cognitive skills in the productive development and
influence on wages and graduation of older children.
5 The Utility and Need for Incorporating Noncognitive Skills …
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
Author’s Proof !

UNCORRECTED PROOF
82
Cunha and Heckman (2010, p. 401) conclude that the noncognitive variables con-
tribute to the impact of cognitive variables on earnings, but there is weak evidence of
the reverse.7 Thus, there are at least three reasons that the singular use of academic
achievement measures to predict economic productivity and growth are overstated
when noncognitive measures are omitted. The first is that academic achievement is
correlated with noncognitive attributes and serves as a proxy for them when predict-
ing economic outcomes, overstating purely cognitive effects when noncognitive
variables are omitted. The second is that noncognitive attributes are not merely
correlated with cognitive attributes, but contribute to cognitive outcomes. The third
is that aggregated attempts to connect academic test scores with economic growth
at the country level suffer the same kind of upward bias that Hanushek et al. (1996)
stress when criticizing upward bias in aggregate estimates of educational production
functions. On this basis it appears that the dramatic and highly publicized extrapola-
tions by Hanushek and Woessman (2008) of contributions to economic growth of
international achievement results among countries overstate the impact of the tests
on economic output, possibly by a large magnitude.8 Unfortunately, the promise of
massive gains in economic output of even modest gains in test scores have been dis-
seminated widely and taken seriously; even though those administering policy are
not aware or knowledgeable about the degree to which upward bias is present in the
reported results and their policy extrapolations.
Far from being harmless, the obsessive focus on test scores and the omission
of the noncognitive impact of schools can provide far-reaching damage. In recent
years, in the United States and other countries, there is an attempt to marshal ev-
idence-based policies. But the evidence that is presented is limited to test score
comparisons with the explicit or tacit implication that test scores are the crucial
determinant of labor force quality. This message places pressure on schools by citi-
zens and government to focus exclusively on raising test scores. In particular, pres-
sures are placed on the schools through accountability sanctions to raise test scores
in the limited domains and measures used in the national and international assess-
ments, usually test scores in reading, mathematics, or sciences. Schools are pressed
to use their time and resources to improve scores on these subjects at the expense
of other activities and subjects including noncognitive goals. Yet other goals may
be as important or more important in the long run in terms of creating productive,
equitable, and socially cohesive societies and economic growth (Gradstein and Just-
man 2002).
The “evidence-based” arguments have led to a singular focus on a cognitive
achievement gap in the No Child Left Behind legislation, leading schools to nar-
7 As a more general proposition I would leave this as an open question. Some four decades ago I
used the Coleman data to estimate the determinants of multiple school outcomes in a model that
allowed for simultaneous equations estimation (Levin 1970). The results of that model estima-
tion suggested reciprocal relationships where motivation and sense of efficacy influence student
achievement and are also influenced by student achievement.
8 Hanushek has responded that even if this is true, the magnitude of the gains in income are so
large that even enormous biases still leave very large unrealized gains.
H. M. Levin
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
Author’s Proof !

UNCORRECTED PROOF
83
row their curriculum and focus on test preparation as a major instructional strategy
(Rothstein et al. 2008). It is difficult for an evidence-based policy to embrace non-
cognitive measures when the assessment practices exclude them from national and
international studies. Even the obsession with the test score gap among races ob-
scures the potential noncognitive impacts of schooling. For example, Fortin (2008)
found the effects of noncognitive ability to be stronger for blacks than whites on
labor market outcomes and a particularly strong predictor of the black-white gap for
males in their incarceration rates.
Singular focus on the cognitive test scores can also introduce teacher policies
that ignore the importance of noncognitive skills and fail to value roles of teach-
ers and schools in the noncognitive domain. For example, many states and local
school districts in the United States have adopted a value-added approach for teach-
er policy where student test score gains associated with individual teachers are the
basis for hiring, retaining, and remunerating teachers. With the recent cuts in public
funding, school districts are considering layoffs of teachers based upon the value-
added metric. But in addition to the serious methodological issues surrounding the
calculation of value-added for each teacher (Corcoran 2010; Harris 2009), there is
an even more fundamental question. Why has the purpose of schooling and teacher
productivity been reduced to the gains on narrowly construed math and verbal tests
if there are so many other results that we expect of schools, including noncognitive
outcomes? Even if there is a tradeoff between teacher effectiveness on cognitive
and noncognitive skill production, both must be taken account of in educational
policy. That is the case for incorporating noncognitive skill measurement in both
large-scale and small-scale assessments.9
Next Steps
To incorporate noncognitive skills into assessments is a major challenge. As Heck-
man and Rubinstein (2001) concluded in their study of the GED 10 years ago:
We have established the quantitative importance of noncognitive skills without identifying
any specific noncognitive skill. Research in the field is in its infancy. Too little is under-
stood about the formation of these skills or about the separate effects of all of these diverse
traits currently subsumed under the rubric of noncognitive skills (p. 149).
Fortunately, the research has exploded on this topic. Just seven years after the publi-
cation of this bleak statement, Cunha and Heckman (2008) were able to identify and
employ specific noncognitive measures in existing data sets that could be used for
analysis followed by an exceedingly productive exploration emerging from Alm-
lund et al. (2011) and Borghans et al. (2008). As mentioned above, Kyllonen et al.
(2008) have developed rich literature reviews of noncognitive skills, including their
9 This has been recognized increasingly on both sides of the Atlantic. See Brunello and Schlotter
(2010) for a report prepared for the European Commission.
5 The Utility and Need for Incorporating Noncognitive Skills …
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
Author’s Proof !

UNCORRECTED PROOF
84
measurement and predictive values, and linked these to specific school interven-
tions that might raise noncognitive performance in key areas.
My recommendation is to build on these efforts by selecting a few noncogni-
tive skill areas and measures that can be incorporated into research on academic
achievement, school graduation, postsecondary attainments, labor market out-
comes, health status, and reduced involvement in the criminal justice system in
conjunction with the standard academic performance measures. The Big Five are
certainly leading candidates, with guidelines already suggested in the review by
Almlund et al. (2011). Structural models and quasiexperimental designs might be
used to understand the interplay of cognitive and noncognitive skills in explaining
particular outcomes for specific demographic groups. At some point, we should
learn enough to incorporate specific noncognitive measures into both small-scale
and large-scale assessments that can lead to a deeper understanding of school ef-
fects and school policy.
References
Almlund, M., Duckworth, A. L., Heckman, J. and Kautz, T. 2011. Personality Psychology and
Economics, (January 17, 2001 version) IZA Workshop: Cognitive and Non-Cognitive Skills,
January 25–27,Bonn, Germany. http://iza.org/conference_files/CoNoCoSk2011/heckman_
j130.pdf. Accessed XXX.
Bandura, A. 1997. Self-efficacy: The exercise of control. New York: Macmillan.
Barnett, W. S. et al. 2008. Educational effects of the tools of the mind curriculum: A randomized
trial. Early childhood research quarterly 23(3):299–313.
Blair, C., and Razza R. P. 2007. Relating effortful control, executive function and false belief un-
derstanding to emerging math & literacy in kindergarten. Child Development 78(2):647–663.
Becker, G. 1964. Human capital: A theoretical and empirical analysis, with special reference to
education. Chicago: University of Chicago.
Borghans, L., Duckworth, A. L., Heckman, J. J., and ter Weel, B. 2008a. The economics and psy-
chology of personality traits. The Journal of Human Resources 43(4):972-1059.
Borghans, L., ter Weel, B. and Weinberg, B. A. 2008b. Interpersonal styles and labor market out-
comes. The Journal of Human Resources 43(4):815–858.
Bowles, S., Gintis, H., and Osborne, M. 2001. The determinants of earnings: A behavioral ap-
proach. Journal of Economic Literature 39(4):137–1176.
Brunello, G., and Schlotter, M. 2010. The effect of noncognitive skills and personality traits on la-
bour market outcomes, analytical report for the european commission prepared by the european
expert network on economics of education. http://www.epis.pt/downloads/dest_15_10_2010.
pdf. Accessed XXX.
Camilli, G., Vargas, S., Ryan, S., and Barnett, W. S. 2010. Meta-analysis of the effects of ear-
ly education interventions on cognitive and social development. Teachers College Record
112(3):579–620.
Cawley, J., Heckman, J., and Vytlacil, E. 2001. Three observations on wages and measured cogni-
tive ability. Labour Economics 8(4):419–442.
Corcoran, S. 2010. Can teachers be evaluated by their students’ test scores? should they be? the
use of value-added measures of teacher Effectiveness in policy and practice. Providence: The
Annenberg Institute for School Reform, Brown University.
Cunha, F., and Heckman, J. J. 2008. Formulating, identifying and estimating the technology of
cognitive and noncognitive skill formation. The Journal of Human Resources, 42(4):738–782.
H. M. Levin
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
Author’s Proof !
AQ3
AQ4

UNCORRECTED PROOF
85
Cunha F., and Heckman, J. J. 2010. Investing in our young people. In Childhood programs and
practices in the first decade of life, eds. Reynolds, A. J., Rolnick, A. J., Englund, M. M., and
Temple, J. A., 381–414. New York: Cambridge University Press.
Dee, T., and West, M. R. 2011. The non-cognitive returns to class size. Educational Evaluation and
Policy Analysis 33(1):23–46.
Diamond, A., Barnett, W. S., Thomas, J., and Munro, S. 2007. Preschool program improves cogni-
tive control. Science 318(5855):1387–1388.
Digman, J. 1990. Personality structure: Emergence of the five-factor model. Annual Review of
Psychology 41: 417–440.
Duncan, G. J. et al. 2007. School readiness and later achievement. Developmental Psychology.
43(6): 1428–1446.
Duncan, G. J., and Magnuson, K. (2011). The nature and impact of early achievement skills, at-
tention and behavior problems. In Whither opportunity: Rising inequality and the uncertain
life chance of low-income children, eds. Duncan, G and Murnane, R. New York: Russell Sage
Press.
Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., and Schellinger, K. B. 2011. The
impact of enhancing students’ social and emotional learning: A meta-analysis of school-based
universal interventions. Child Development 82(1):405–432.
Finn, J. D., Gerber, S. B., and Boyd-Zaharias, J. 2005. Small classes in the early grades, academic
achievement and graduating from high school. Journal of Educational Psychology 97(2):214–
223.
Fortin, N. M. 2008. The gender wage gap among young adults in the United States: The Impor-
tance of Money vs. People. The Journal of Human Resources 43(4):884–918.
Gintis, H. 1971. Education, technology, and the characteristics of worker productivity. American
Economic Review 61(2):266–279.
Gradstein, M., Justman, M. 2002. Education, social cohesion, & economic growth. American Eco-
nomic Review 92(4):1192–1204.
Hanushek, E., Rivkin, S., and Taylor, L. 1996. Aggregation and the estimated effects of school
resources. Review of Economics and Statistics 78(4):611–627.
Hanushek, E., Woessmann, L. 2008. The role of cognitive skills in economic development. Jour-
nal of Economic Literature 46(3):607–668.
Harris, D. 2009. Would accountability based on teacher value-added be smart policy? an exami-
nation of the statistical properties and policy alternatives. Educational Finance and Policy
4(4):319–350.
Hartigan J., and Wigdor A. 1989. Fairness in employment testing: Validity generalization, minority
issues, and the general aptitude test battery. Washington: National Academy Press.
Heckman, J., Moon, S. H., Pinto, R., Savelyev, P., and Yavitz, A. 2010. A new cost-benefit and rate
of return analysis for the perry preschool program: A summary. In Childhood programs and
practices in the first decade of life, eds. Reynolds, A. J., Rolnick, A. J., Englund, M. M., and
Temple, J. A., 366–380. New York: Cambridge University Press.
Heckman, J.,Rubenstein, Y. 2001. The importance of noncognitive skills: Lessons from the GED
testing program. American Economic Review 91(2): 145–149.
Inkeles, A. 1966. The socialization of competence. Harvard Educational Review 36(3):265–283.
Inkeles, A. 1975. Becoming modern: Individual change in six developing countries. Ethos
3(2):323–342.
Inkeles, A., and Smith, D. 1974. Becoming modern: Individual changes in six developing societies.
Cambridge: Harvard University Press.
Knudsen, E. I., Heckman, J. J., Cameron, J. L., and Shonkoff, J. P. 2006. Economic, neurobiologi-
cal, and behavioral perspectives on building America’s future workforce. PNAS (Proceedings
of the National Academy of Sciences) 103(27):10155–10162.
Krueger, A., and Schkade, D. 2008. Sorting in the labor market: Do gregarious workers flock to
interactive jobs? The Journal of Human Resources 43(4):859–883.
Kyllonen, P. C., Lipnevich, A. A., Burrus, J., and Roberts, R. D. 2008. Personality, motivation,
and college readiness: A prospectus for assessment and development. Princeton: Educational
Testing Service.
5 The Utility and Need for Incorporating Noncognitive Skills …
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
Author’s Proof !

UNCORRECTED PROOF
86
Levin, H. M. 1970. A new model of school effectiveness. In Do teachers make a differ-
ence? ed. Mood A. M. 55–78 Washington: Office of Education, U.S. Department of
Health, Education and Welfare. http://eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_
nfpb=true&_&ERICExtSearch_SearchValue_0=ED04025. Accessed XXX.
Lindqvist, E., and Vestman, R. 2011. The labor market returns to cognitive and noncognitive abil-
ity: Evidence from the Swedish Enlistment. American Economic Journal: Applied Economics
3:101–128.
Magnuson, K. A., Ruhm, C., and Waldfogel, J. 2007. Does prekindergarten improve school prepa-
ration and performance? Economics of Education Review 26:33–51.
Mosteller, F. 1995. The Tennessee study of class size in the early school grades. The Future of
Children 5(2):113–127.
Murnane, R., Willett, J., and Levy, F. 1995. The growing importance of cognitive skills in wage
determination. The Review of Economics and Statistics 77(2):251–266.
Murnane, R., Willett, J., Bratz, M., Duhaldeborde, Y. 2001. Do different dimensions of male high
school students’ skills predict labor market success a decade later? evidence from the NLSY.
Economics of Education Review 20:311–320.
Murnane, R., Willett, J., Duhaldeborde, Y., and Tyler, J. 2000. How important are the cognitive
skills of teenagers in predicting subsequent earnings? Journal of Policy Analysis and Manage-
ment 19(4):547–568.
National Research Council. 1984. High schools and the changing workplace: The employers’
view. Report of the Panel on Secondary School Education for the changing workplace Wash-
ington, DC: National Academy Press.
Neuman, G. A., and Wright, J. 1999. Team effectiveness: Beyond skills and cognitive ability.
Journal of Applied Psychology 84(3):376–389.
Noftle, E. E., and Robins, R. W. 2007. Personality predictors of academic outcomes: Big five cor-
relates of GPA and SAT scores. Journal of Personality and Social Psychology 93(1):116–130.
Nores, M., and Barnett, W. S. 2010. Benefits of early childhood interventions across the world:
(Under) Investing in the very young. Economics of Education Review 29:271–282.
Reynolds, A. J., Rolnick, A. J., Englund, M. M., and Temple, J. A. 2010. Childhood programs and
practices in the first decade of life. New York: Cambridge University Press.
Rothstein, R., Jacobson, R., Wilder, T. 2008. Grading education: Getting accountability right.
New York: Teachers College Press.
Sackett, P. R., Schmitt, N., Ellingson, J. E., and Kabin, M. B. 2001. High stakes testing in em-
ployment, credentialing, and higher education: Prospects in a post-affirmative action world.
American Psychologist 56(4):302–318.
Sameroff, A. 2010. A unified theory of development: A dialectic integration of nature and nurture.
Child development 81(1):6–22.
Schweinhart, L.J. 2010. The challenge of the highscope Perry preschool study. In Childhood pro-
grams and practices in the first decade of life., eds. Reynolds, A. J., Rolnick, A. J., Englund,
M. M., and Temple, J. A., 366–380. New York: Cambridge University Press.
Segal, C. 2008. Classroom behavior. The Journal of Human Resources 43(4):783–814.
Shury, J., Winterbotham, M., and Oldfield, K. 2010. National employer skills survey for England
2009: Key findings report. South Yorkshire: UK Commission for Employment & Skills.
ter Weel, B. 2008. Symposium on the noncognitive determinants of labor market and behavioral
outcomes. The Journal of Human Resources 43(4).
Urzua, S. 2008. Racial labor market gaps: The role of abilities and schooling choices. The Journal
of Human Resources 43(4):919–971.
Vignoles, A., De Coulon, A., and Marcenaro-Gutierrez, O 2011. The value of basic skills in the
British labour market. Oxford Economic Papers, 63(1):27–48.
Zemsky, R., and Iannozzi, M. 1995. A reality check: First finding from the EQW national employer
survey ( 10). Philadelphia: National Center on the Educational Quality of the Workforce, Uni-
versity of Pennsylvania. http://www.eric.ed.gov/PDFS/ED398385.pdf. Accessed XXX.
H. M. Levin
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
Author’s Proof !
AQ5

UNCORRECTED PROOF
Author Query
AQ1. “Bowles and Gintis (1976)” is cited in the text but is not given in the refe-
rence list. Please provide a full reference or delete the citation.
AQ2. Please specify whether the year “2008a” or “2008b” is valid for the citation
“Borghans et al. 2008”.
AQ3. “Inkeles 1975” is not cited in the text. Please provide the citation or delete
the entry from the reference list.
AQ4. Please provide accessed date for the following references: “Almlund et al.
2011”, “Brunello and Schlotter 2010”, “Zemsky and Lannozzi 1995”, “Levin 1970”.
AQ5. We have inserted author name and citation (ter Weel 2008) for reference
“Symposium on The Noncognitive Determinants of Labor Market and Behavioral
Outcomes”. Please check.