The Malleability of Workplace‐Relevant Noncognitive Constructs: Empirical Evidence From 39 Meta‐Analyses and Reviews

Abstract

We reviewed the current state of the literature on the intervention‐based development of interpersonal skills (e.g., teamwork, leadership) and intrapersonal skills (e.g., personality, motivation, etc.) relevant to success in workplace contexts. We adopted a multidisciplinary approach to our review, evaluating research from 39 reviews and meta‐analyses from several fields such as educational psychology, industrial/organizational (I/O) psychology, medicine, and personality psychology, among others, to examine the extent to which noncognitive constructs change as a result of intervention. We discuss key findings and trends and conclude by identifying gaps in the literature and directions for future research. Overall, findings suggest optimism regarding the malleability of noncognitive constructs.

Full text

R
ese
a
r
c
h R
e
p
o
r
t
The Malleability of
Workplace-Relevant
Noncognitive Constructs:
Empirical Evidence From 39
Meta-Analyses and Reviews
ETS RR–20-23
Michelle P. Martin-Raugh
Kevin M. Williams
Jennifer Lentini
September 2020

ETS Research Report Series
EIGNOR EXECUTIVE EDITOR
John Mazzeo
Distinguished Presidential Appointee
ASSOCIATE EDITORS
Beata Beigman Klebanov
Senior Research Scientist
Heather Buzick
Senior Research Scientist
Tim Davey
Research Director
John Davis
Research Scientist
Keelan Evanini
Research Director
Marna Golub-Smith
Principal Psychometrician
Priya Kannan
Managing Research Scientist
Irv Katz
Senior Research Director
Sooyeon Kim
Principal Psychometrician
Anastassia Loukina
Senior Research Scientist
Gautam Puhan
Psychometric Director
Jonathan Schmidgall
Research Scientist
Jesse Sparks
Research Scientist
Michael Walker
Distinguished Presidential Appointee
Rebecca Zwick
Distinguished Presidential Appointee
PRODUCTION EDITORS
Kim Fryer
Manager, Editing Services
Ayleen Gontz
Senior Editor
Since its 1947 founding, ETS has conducted and disseminated scientic research to support its products and services, and
to advance the measurement and education elds. In keeping with these goals, ETS is committed to making its research
freely available to the professional community and to the general public. Published accounts of ETS research, including
papers in the ETS Research Report series, undergo a formal peer-review process by ETS sta to ensure that they meet
established scientic and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that
outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions
expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and
not necessarily those of the Ocers and Trustees of Educational Testing Service.
e Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 ser ved the Research and
Development division as Editor for the ETS Research Report series. e Eignor Editorship has been created to recognize
the pivotal leadership role that Dr. Eignor played in the research publication process at ETS.

ETS Research Report SeriesISSN 2330-8516
RESEARCH REPORT
The Malleability of Workplace-Relevant Noncognitive
Constructs: Empirical Evidence From 39 Meta-Analyses
and Reviews
Michelle P. Martin-Raugh, Kevin M. Williams, & Jennifer Lentini
Educational Testing Service, Princeton, NJ
We reviewed the current state of the literature on the intervention-based development of interpersonal skills (e.g., teamwork, lead-
ership) and intrapersonal skills (e.g., personality, motivation, etc.) relevant to success in workplace contexts. We adopted a multi-
disciplinary approach to our review, evaluating research from 39 reviews and meta-analyses from several elds such as educational
psychology, industrial/organizational (I/O) psychology, medicine, and personality psychology, among others, to examine the extent
to which noncognitive constructs change as a result of intervention. We discuss key ndings and trends and conclude by identifying
gaps in the literature and directions for future research. Overall, ndings suggest optimism regarding the malleability of noncognitive
constructs.
Keywords Noncognitive skills; malleability; workplace
doi:10.1002/ets2.12306
Researchers, organizations, and educational institutions have become increasingly interested in the malleability and devel-
opment of constructs outside of traditional conceptualizations of intelligence (Heckman & Kautz, 2014; Kyllonen, 2012).
One approach to identifying these constructs involves the distinction between those typically considered “cognitive” from
those usually categorized as “noncognitive.” Whereas cognitive ability and cognitive skills are oen equated with tradi-
tional measures of intelligence and the ability to solve abstract problems (Gottfredson, 1998), noncognitive constructs
are conceptualized as those that are separate from and considered largely orthogonal to cognitive ability. Scholars have
dened noncognitive constructs in several ways. Klieger et al. (2015) described noncognitive constructs as “demonstra-
ble personality, motivational, attitudinal, self-regulatory, and learning approach constructs for which there are dierences
among people, which standardized tests of cognitive ability are not primarily designed to measure, and the behavioral
expression of which is considered useful” (p. 3). is denition overlaps with those proposed by Kyllonen (2012, pp. 7– 8),
Kautz et al. (2014, p. 2), and Duckworth and Yeager (2015, p. 239), suggesting a comprehensive scope of applicable con-
structs, such as personality traits, social skills, motivation, teamwork, and leadership skills, among others. However, it is
worth noting that the line between cognitive and noncognitiveconstructs may be blurry (e.g., Duckworth & Yeager, 2015;
Kell, 2018): Many constructs characterized as being largely noncognitive in nature actually contain cognitive elements, as
allrelyontheuseofmentalprocesses.
Noncognitive constructs are important predictors of a variety of workplace and economic outcomes (Barrick &
Mount, 1991; Lindqvist & Vestman, 2011) and explain meaningful variability even when cognitive ability is taken into
account (Schmidt & Hunter, 1998). In these contexts, noncognitive constructs are oen referred to as 21st century skills
or so skills (e.g., Heckman & Kautz, 2012). Despite the importance of these constructs, employers in today’s economic
climate perceive a widening gap between the noncognitive constructs they expect prospective employees to possess and
the level of skill actually exhibited by job candidates (Capelli, 2012; Society of Human Resource Management, 2019).
Although cognitive ability remains largely stable over time and tends to be somewhat xed (Jensen, 1998), noncognitive
constructs are thought to be more malleable (Aguinis & Kraiger, 2009; Heckman & Kautz, 2012). ese factors combine
to generate great interest in workplace domains (e.g., Heckman & Rubinstein, 2001) in exploring the extent to which
noncognitive constructs change through intervention. In 2018, organizations in the United States spent $87.6 billion on
Corresponding author: M. P. Martin-Raugh, E-mail: mmartin-raugh@ets.org
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 1

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 1 Noncognitive Construct Denitions
Construct Denition
Personality Includes the Big Five, a dominant typology of personality traits including agreeableness, openness to
experience, conscientiousness, extraversion, and emotional stability, along with personality models and
traits that typically predate the Big Five
Social skills Broad set of interpersonal skills that facilitate interaction with others
Communication e ability to use both oral and written messages perceived as appropriate and eective in workplace
settings
Leadership e ability to persuade followers to support and execute solutions
Teamwork Interrelated set of thoughts, behaviors, and feelings needed for a team to function as a unit
Attitudes Internal state that inuences an individual’s choice of personal action
Self-concept e manner in which individuals perceive and evaluate themselves
Emotion Encompasses emotional state, emotional regulation, positive and negative aect, and empathy
Motivation Intrinsic and extrinsic forces that impact work-related behavior in terms of form, direction, intensity, and
duration
interventions to foster employee learning and development (Freifeld, 2018), further underscoring the need for research
in this area.
In this review, using the denition of noncognitive constructs provided by Klieger et al. (2015), we systematically exam-
ine research focusing on the malleability of interpersonal skills and intrapersonal skills1relevant to workplace success and
the mechanisms through which these constructs change (see Table 1). We focus on social skills, communication skills,
leadership, and teamwork as part of our review of interpersonal skills. We consider intrapersonal skills to include person-
ality traits, attitudes, self-concept, emotion, and motivation. Although several comprehensive and informative reviews
of the workplace training literature exist (e.g., Aguinis & Kraiger, 2009; Baldwin & Ford, 1988; Bell et al., 2017; Blume
et al., 2010), the organization and foci of these prior reviews dier from our approach. Prior reviews have examined
training practices more generally, targeting aspects of approaches or methodologies rather than focusing on the develop-
ment of particular constructs (e.g., Aguinis & Kraiger, 2009; Baldwin & Ford, 1988; Bell et al., 2017; Blume et al., 2010).
Other reviews have concentrated on the enrichment of constructs that could be considered mainly cognitive (e.g., Scott
et al., 2004; Shipstead et al., 2012) orprimarily on job-specic skills (e.g., Salas et al., 2006; Weaver et al., 2010). We largely
draw on meta-analyses and reviews conducted in a diverse constellation of elds to provide a comprehensive review of
the malleability of noncognitive constructs.
We should a l s o c l a r if y ou r u se o f t he t e r m malleability. We use this term to refer to longitudinal change in con-
structs resulting from deliberate intervention. is denition aligns with our goal of summarizing research relevant
to employers’ explicit intervention eorts in service of improving workplace-relevant noncognitive constructs. Rather
than direct improvements in construct scores, employers may be more interested in facilitating various desirable work-
place outcomes such as increases in productivity, prots, employee retention, and employee satisfaction, among others.
Regardless, because this research is intended to inform overt training and development eorts, we argue that research
observing noncognitive constructs’ longitudinal change in the absence of intervention (e.g., Viswesvaran & Ones, 2000)
or in response to natural life events (e.g., Bleidorn et al., 2018) is not relevant. Moreover, these naturalistic studies should
not be viewed as support for these constructs’ amenability to eortful intervention. On the contrary, it is possible that
these results highlight innate maturational processes or environmental factors that deliberate interventions must strive to
overcome (see Briley & Tucker-Drob, 2014). Instead, the most signicant conclusion that may be drawn from longitudi-
nal studies that exclude eortful interventions is that they emphasize the value of non-intervention control groups in the
study of deliberate interventions.
Our review aims to answer the following research questions (RQs):
1. To what extent do workplace-relevant noncognitive constructs change as a result of intervention?
2. What are the mechanisms that eectively drive change?
We organize the existing literature on the malleability of various noncognitive constructs into two broad sections —
interpersonal and intrapersonal skills— within which we present evidence and evaluate research addressing each of the
two RQs. We conclude each section by providing a synthesis of the literature and recommendations for future research.
2ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
We include articles within a noncognitive construct section if the malleability of that construct was examined in the
article, regardless of whether the construct was a specic focus of the intervention. We conclude by noting the limitations
associated with our study and identifying viable directions for future research.
Methods
Literature Search
e goal of this study was to evaluate meta-analyses and narrative reviews relevant to our RQs. is type of evidence
was selected because of its well-documented advantages over primary studies, which has been argued elsewhere (e.g.,
Card, 2012). To be as inclusive as possible, we used Google Scholar as our primary search engine. A Boolean search
strategy combining three sets of search terms (Figure 2) was used. First, a set of noncognitive construct terms (e.g., Big Five,
leadership) was used to dene the construct of interest. Second, various dynamic process terms (e.g., development,training)
were entered to identify articles discussing the malleability of these noncognitive constructs that might be transportable
to the workplace. Finally, the terms meta-analysis and review were entered separately. For example, a full search string was
“Big Five” +development +“meta-analysis.” Each possible combination of the three sets of search terms was used. It is
worth noting that our list of search terms may not have been exhaustive. No publication date restrictions were imposed
on the search, which ended in February 2017.
Despite eorts to narrow the focus of the search using a Boolean strategy, an overwhelming amount of literature was
identied. For example, the string personality +training +meta-analysis returned 1,100,000 results. erefore, searches
within each string were terminated aer reaching (a) 30 consecutive results that were either irrelevant or already captured
in another search or (b) the rst 200 results, whichever occurred rst. Using the default method of sorting results by
relevanceprovidedabalanceofcapturingalargenumberofthemostpertinentarticleswhilelimitingthesearchtoa
manageable volume. Finally, this was augmented by a partial backward snowball search, in which the reference sections
of the articles identied through the initial search are reviewed for additional articles.
Inclusion Criteria
Basic inclusion criteria required that the article must be written in English; be published in a book, dissertation, or peer-
reviewed journal article; and consist of a meta-analysis or narrative review, or a relevant reply article to one of these
meta-analyses or reviews. Relevance criteria required that the noncognitive constructs, participants, interventions, or
outcomes included in the article could be reasonably expected to translate to the workplace. Methodological criteria
required that the article evaluate eortful changes in the noncognitive construct using a longitudinal or experimental
design. erefore, meta-analyses and reviews focusing on the predictive validity of noncognitive constructs but not the
eortful malleability of these constructs (e.g., Barrick & Mount, 1991) were excluded. Similarly, meta-analyses and reviews
examining longitudinal stability and change of noncognitive constructs in the absence of intervention (e.g., Viswesvaran
& Ones, 2000) or in response to natural life events (e.g., Bleidorn et al., 2018) were excluded. Furthermore, noncognitive
construct change must be measured at the level of the individual participant as opposed to individuals who were not the
target of the intervention.
ScreeningProcess
Articles were screened by the three authors in two phases. First, each article title and abstract were reviewed to ensure
they met our inclusion criteria. Next, the full text of the article was reviewed for the same criteria. Prior to each phase,
10 calibration articles were selected for each of the three authors to examine independently. Disagreements regarding
the eligibility of these articles were resolved by discussion. e remaining articles were divided among the three authors.
At the full-text screening stage, if a primary reviewer rejected an article, a second reviewer independently reviewed the
article to conrm or reject this decision. If the secondary reviewer disagreed with the primary reviewer, the third reviewer
determined the article’s eligibility. Of the rejected articles, exact agreement between the rst two reviewers was 88.5%.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 3

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Figure 1 PRISMA diagram of literature search results.
ScreeningResults
Aer removing duplicates, 10,006 articles were identied. Of these, 354 unique articles passed the initial title and abstract
screening phase, with 39 articles passing the nal full-text screening phase (Figure 1; Moher et al., 2009). Common ratio-
nales for rejections included low relevance to the workforce (e.g., clinical interventions; Roberts, Hill, & Davis, 2017),
failing to provide a meta-analysis or review, or a lack of discussion about the malleability of the construct(s). It should
be noted that in some cases a source article pertained to more than one noncognitive construct. Publication dates ranged
from 1941 to 2017, with the majority published aer 2000. Meta-analytic eect sizes indexing construct malleability as a
result of intervention are displayed in Table 2.
Results: QuantifyingNoncognitive Construct Change
Before describing specic noncognitive construct malleability research, it is necessary to clarify the means by which
changes are evaluated. Rank-order consistency and mean-level dierences are the two approaches used most oen. Rank-
order consistency (or rank-order stability) typically involves test– retest correlations between a group’s noncognitive scores
from an initial assessment with that same group’s scores at a subsequent assessment, following a signicant period of time
(e.g., 1 year later). ese results summarize the longitudinal stability of individuals’ rank ordering on the construct of
interest, such as whether the most extraverted individuals at Time 1 remain the most extraverted at Time 2, for instance.
Alternatively, mean-level dierences compare mean noncognitive assessment scores (a) either within the same group
prior to and following an intervention, (b) postintervention scores between two groups such as an intervention and con-
trol group, or (c) pre-postintervention dierences between two groups. Rank-order consistency and mean-level change
may be considered complementary methods: Rank-order consistency does not evaluate the degree to which the entire
sample’s level of the noncognitive construct in question has changed over time, whereas mean-level changes may obscure
4ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Figure 2 Literature Review Search Terms.
more nuanced results such as stability dierences across subgroups. In noncognitive construct research, rank-order meth-
odstendtobeusedinstudiesofnaturallyoccurringchange,whereasmean-leveldierencesaremorecommonineortful
change research. e majority of meta-analyses and reviews we describe used one of these two methods to evaluate eort-
ful longitudinal malleability (Table 2). ese eect sizes are typically interpreted using traditional eect size guidelines
(i.e., Cohen, 1988).
Results: Interpersonal Skills
Social Skills
RQ1: Malleability
Cheraghi-Sohi and Bower’s (2008) review of patient feedback on physicians’ social skills found limited evidence that
social skills can be improved through the interventions examined in their review. Conversely, separate meta-analyses have
reported medium to large eect size improvements in social skills following organizational training (Arthur Jr. et al., 2003)
or social skills training (Klein, 2009). Only six studies focused on social skills more broadly (see Table 2).
RQ2: Mechanisms of Change
Several sources explored the ecacy of social skills interventions (e.g., Klein, 2009; Robbins et al., 2009). Klein (2009)
suggested that social skills training can occur via traditional methods including motivating and goal setting, coaching
and mentoring, feedback, behavioral modeling training (BMT), multimedia and simulation-based training, and team
training. Pellegrino and Hilton’s (2012) review of social skills argued that deeper learning, where a person can take what
was learned in one situation and can apply it to new situations, allows trainees to eectively transfer what was learned
to new situations. ese authors argued that deep learning interventions should provide clear and discrete learning goals
andshouldshowhowlearningisexpectedtoprogress.PellegrinoandHiltonalsoadvocatedfortheuseofproblem-based
learning (PBL) approaches that present learners with extended problems that can engage learners while providing helpful
feedback and guidance. ey argued that PBL can encourage elaboration, questioning, and self-explanation.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 5

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Noncognitive Construct Malleability Meta-Analyses
Article k N
Ye a r s o f
coverage
Sample
description
Design
(experimental
vs. longitudinal) Intervention
Primary
outcome
measure Eect size Conclusion
Social skills
Arthur Jr. et al., 2003* 397 33,325 1960 –2000 Working adults Experimental Organizational training
focusing on
interpersonal skills
Group mean
dierences (d)
d=0.62 Interpersonal skills
demonstrate a medium
eect size change
through intervention
when assessed by the
Learning and
Behavioral criteria, and
a large eect size when
evaluated by the
Results criterion
Klein, 2009* 141 NR 1971– 2008 Adults Longitudinal Interpersonal skills
training (includes
mentoring, behavioral
modeling, feedback,
goal setting, team
training,
simulation-based
training)
Pre-post and/or
group mean
dierences (r)
r=.47 Interpersonal skills
training is moderately
eective in improving
general interpersonal
skills
Robbins et al., 2009* 107 11,183 1975 –2009 College students Experimental Self-management,
socialization,
First-Year-Experience
interventions
Corrected
correlation
coecient
ρ=.15 College interventions
yielded a small eect
on social control
Communication skills
Barth & Lannen, 2011* 13 1,137 1988– 2008 Healthcare
professionals
working with
cancer patients
Experimental Communication skills
training courses in
oncology
Group mean
dierences
d=0.50 Communication skills
demonstrated
moderate improvement
through intervention
Leadership skills
Avolio et al., 2009* 132 11,552 NR Adults Experimental Leadership training Group mean
dierences (d)
d=0.67 Leadership training
interventions showed
an overall medium
eect size
6ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
coverage
Sample
description
Design
(experimental
vs. longitudinal) Intervention
Primary
outcome
measure Eect size Conclusion
Lacerenza et al., 2017* 335 26,573 1951– 2014 Adults Experimental Leadership training Pre-post mean
dierences (d)
d=0.73 Leadership training is
more eective than
previously thought
Team w ork sk i l l s
Salas et al., 2007* 7 695 1975 –2000 Non-clinical adults Experimental Team training Pre-post and/or
group mean
dierences (r)
r=.29 Team training was shown
to have a small to
moderate, positive
eect on team
functioning
Salas, Diaz Granados,
Klein, et al., 2008*
45 2,650 1962–2008 Non-clinical adults Experimental Team training Pre-post and/or
group mean
dierences (ρ)
ρ=.34 Team training was shown
to have a moderate,
positive eect on team
functioning
Personality
Lipsey & Wilson, 1993 156 >1,000,000 1979– 1992 Clinical and
non-clinical youth
and adults
Experimental Various mental health,
I/O, and
educational
programs
Standardized
mean dierence
d=0.47 Collapsed across a wide
range of outcomes
(including personality),
interventions are
generally eective in
changing psychological
characteristics
Vanhove et al., 2016 42 16,348 1979 –2014 Workplace samples Both Workplace programs
targeting
psychosocial factors
Standardized
mean
dierences
(between- or
within-groups)
d=0.21 e eects of
resilience-building
programs are small and
do not endure
long-term
Klein et al., 2009* 20 1,562 1950–2007 Non-clinical adults Experimental Team-building Pre-post and/or
group mean
dierences (r)
r=.31 Team-building
interventions can have
amoderateeecton
team outcomes, being
more eective for
process and aective
outcomes, and least
eective for cognitive
outcomes
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 7

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
coverage
Sample
description
Design
(experimental
vs. longitudinal) Intervention
Primary
outcome
measure Eect size Conclusion
Attitudes
Bangert-Drowns, 1988 33 NR 1968 –1986 Traditional
elementary
through college
students
Experimental School-based
substance abuse
education
Group dierences
(d)
d=0.34aInterventions can reduce
pro-alcohol- and
drug-related attitudes
Bruvold & Rundall, 1988 NR NR 1972–1984 Students Experimental Alcohol and
tobacco
intervention
programs
Group dierences
in pre-post
changes (d)
NRbInterventions can reduce
pro-alcohol attitudes
Brecklin & Forde, 2001 45 NR 1861– 1999 College students Both Rape education
programs
Group dierences
or pre-post
changes (d)
d=0.35 Interventions can reduce
rape-supportive attitudes
Anderson &
Whiston, 2005
69 18,172 1978 –2002 College students Experimental Sexual assault
education
programs
Group dierences
(d)
NRbInterventions can reduce rape
attitudes, rape-related
attitudes, and behavioral
intent, but not rape
empathy
Barth & Lannen, 2011 13 1,137 1988–2008 Oncology health
professionals
Experimental Communication
skills training
Group dierences
(d)
d=0.35 Interventions can improve
attitudes toward terminally
ill patients, death, and
dying
Kalinoski et al., 2013 65 8,465 1977 –2011 High school
students,
college
students, and
employed adults
Both Diversity training Group dierences
(δ)
δ=0.33 Overall, diversity training had
positive small-to-moderate
eects on aective- (e.g.,
attitudes), cognitive-, and
skill-based outcomes
Beelmann &
Heinemann, 2014
81 NR 1958 –2010 Youth (<18 years) Exp erimental Programs
designed to
reduce
prejudice or
improve
intergroup
attitudes
Group dierences
(d)
d=0.30 Interventions can reduce
prejudice and improve
intergroup attitudes toward
otherethnicgroups,
disabled individuals, and
the elderly
8ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
coverage
Sample
description
Design
(experimental
vs. longitudinal) Intervention
Primary
outcome
measure Eect size Conclusion
Bezrukova et al., 2016 260 29,407 1972 –2013 Adults Both Diversity training Group dierences
(g)
g=0.38 With some exceptions,
diversity training
generally leads to
positive outcomes,
though less
pronounced for
attitudinal change
Jones, 2016 10 592 1993 –2012 Employed adults Both Workplace coaching Group dierences
or pre-post
changes (δ)
δ=0.51 Interventions can
improve work attitudes
Self-concept
Bowen & Neill, 2013 137 NR 1960–2012 At risk, clinical, or
adjudicated
Longitudinal Adventure therapy Pre-post
dierences
(Hedge’s g)
g=0.43 Adventure therapy can
improve self-concept
Emotion
Augusti ne &
Hemenover, 2009
34 2,958 1887 –2007 Non-clinical
adults
Longitudinal Aect repair strategies
(Behavioral or
cognitive; avoidance or
engagement)
Pre-post mean
dierences
d=0.45 Aect regulation
interventions can
increase positive aect
and decrease negative
aect
Delise et al., 2010 21 1,413 1985–2008 Military; civilian Both Team training Pre-post and/or
group mean
dierences
d=0.85 Aect - in the context of
team eectiveness —is
amenable to change
through intervention
Webb et al., 2012 190 >13,655 1977–2010 Nonclinical;
mostly adults
Both Attentional deployment,
cognitive change, or
response modulation
Pre-post and/or
group mean
dierences
NRbe eectiveness of
emotion regulation
interventions depends
on the specic strategy
used
Sheeran et al., 2014 208 52,976 1953– 2010 Adults, students,
youth
Experimental Practices that heighten
risk appraisal
Group mean
dierences (d)
d+=0.70 Both anticipatory (e.g.,
fear, worry) and
anticipated emotions
(e.g., regret, guilt) may
be increased by small
eect sizes by
heightening risk
appraisal
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 9

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
coverage
Sample
description
Design
(experimental
vs. longitudinal) Intervention
Primary
outcome
measure Eect size Conclusion
Tedi n g v a n B e rkho u t &
Malou, 2016
18 1,018 1973– 2014 University students;
health
professionals;
patients; other
adults; youth
Experimental Empathy training Group mean
dierences
g=0.53 Empathy, which includes
aective components, can
be improved through
intervention
Motivation
Rummel & Feinberg, 1988 45 NR 1971 –1985 Youth, students, and
adults
Experimental Extrinsic rewards Group mean
dierences (d)
d=0.33 Extrinsic rewards had a
detrimental eect on
intrinsic motivation
Cameron & Pierce, 1994 96 NR 1971 –1991 Youth and adults Experimental Rewards Group mean
dierences (d)
d=0.14 Results suggest that reward
does not decrease intrinsic
motivation
Deci et al., 1999 101 NR 1971 –1997 Youth and college
students
Experimental Reward Free-choice
intrinsic
motivation;
Group mean
dierences (d)
d=−0.24 Extrinsic rewards
signicantly undermined
free-choice intrinsic
motivation. Positive
feedback improved
free-choice behavior
Sitzmann & Ely, 2011 430 90,380 1989 –2011 Adults Experimental Self-regulatory
processes (Goal
level, persistence,
eort, self-ecacy)
Learning NR Self-regulation enables
individuals to monitor
their goal-directed
activities over time; goal
level, persistence, and eort
have small to medium
eects on learning
Jones, 2016 17 2,267 1993– 2012 Adults Experimental Coaching Organizational
outcomes;
Pre-post and/or
group mean
dierences (δ)
δ=0.36 Workplace coaching yields a
small to moderate impact
on eectiveness overall
Note.*=meta-analyses; NA =not applicable; NR =not reported. Articles listed in italics do not address longitudinal change of noncognitive skills but are otherwise relevant to our research questions.
aEect size is reported for attitude outcomes; authors also reported eect sizes for knowledge (d=0.76) and behavior outcomes (d=0.12). bAuthors reported multiple eect sizes based on factors such as
intervention type, duration, and outcome, but no omnibus estimate.
10 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Communication Skills
RQ1: Malleability
e majority of articles on communication skills we reviewed focused on health-care professionals (see Table 1). Commu-
nication between physicians and patients is a critical part of any treatment plan. However, physicians’ poor communication
skills represent a common complaint for patients and undermine health-care ecacy (Hulsman et al., 1999). In a review
of 14 studies of communication training for physicians, Hulsman et al. (1999) reported mixed results. Positive eects of
training occurred in fewer than half of the studies, and the studies that did report positive outcomes generally used sub-
optimal research designs (e.g., no control group). However, this study may not be representative of the eld as a whole, as
other systematic reviews and meta-analyses have reported improvements in communication skills through interventions
such as team training (Gillespie et al., 2010), simulated patients and roleplays (as opposed to didactic training; Lane &
Rollnick, 2007), and communication skills courses (Barth & Lannen, 2011).
RQ2: Mechanisms of Change
Lane and Rollnick (2007) stated in their review of the literature that the use of simulated patients, typically actors play-
ing the role of a patient, is a widespread, eective practice in health-care communication training. Identical patients
allow for standardization, experimentation with dierent communication skills, repeated training, and ongoing feedback.
Drawbacks include expense, the need for actor selection and training, and the risk of simulated patients going o-script.
Role-play with fellow trainees is also an eective communication training intervention, as it has benets and drawbacks
similar to simulated patients. Furthermore, both of these training methods were found to be more eective than didactic
training methods, such as lectures. Gillespie et al.’ (2010) found that communication skills training in the context of team
training improved health outcomes for patients. Interventions studied included checklists, simulations, and debriengs.
However, as this study did not compare the ecacy of various team training methods for communication, the relative
merits of each are still unknown. Barth and Lannen (2011) examined the eects of communication skills training courses
on communication behaviors, and found that courses lasting over 36 hours outperformed courses lasting less than 24 h,
although this eect was small.
Leadership
RQ1: Malleability
e three meta-analyses reviewed here provide some evidence to suggest that leadership skills are indeed amenable to
change through intervention (see Table 2). Avolio et al. (2009) concluded in their meta-analysis of the development of
leadership skills that leadership training interventions yielded a medium eect size in producing positive change in lead-
ership skills. A follow-up study (Avolio et al., 2010) showed that leadership interventions yielded a wide range of return
on development investment, with some estimates as high as 200%. Moreover, data from a recent meta-analysis (Lac-
erenza et al., 2017) supported the notion that leadership training is largely eective, resulting in medium-to-large eect
size improvement gaged by reactions, learning, transfer, and results criteria (i.e., Kirkpatrick, 1996).
RQ2: Mechanisms of Change
Avolio et al. (2009) found in their meta-analysis of leadership development eorts that there was largely no dierence in
eectiveness between interventions using leader training and development methods and those using a dierent method,
such as a scenario, actor or role-play, or assigned leader. ey also reported that more eective interventions were based on
Pygmalion theory, which posits that when leaders hold positive, high expectations for those they are leading, performance
improves. ese interventions yielded larger eects than those based on traditional or newer leadership theories, such
as charismatic or transformational leadership theories, which produced medium eects. Lacerenza et al.’s (2017) meta-
analysis concluded that leadership interventions using practice-based methods were more eective than other delivery
methods and that programs incorporating multiple methods of delivery (e.g., information-based, demonstration-based)
are signicantly more eective. us, they recommended that training programs use multiple delivery methods when
possible, and only practice-based delivery when only one approach may be used.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 11

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tea mwork
RQ1: Malleability
e eight teamwork skills studies we reviewed concluded that these skills are amenable to change through organiza-
tional interventions (e.g., Gordon et al., 2012; Weaver et al., 2014; see Table 2. ese studies typically incorporated out-
comes closely related to or demonstrations of teamwork rather than direct changes in an assessment that conceptualized
teamwork as a distinct noncognitive construct. For instance, Klein et al.’s (2009) meta-analysis found that team-building
interventions had a more pronounced eect for process (e.g., communication, coordination) and aective outcomes (e.g.,
trust) than for cognitive outcomes (e.g., knowledge). McCulloch et al. (2011) reviewed teamwork interventions specic
to health care and showed that team training programs can lead to improved sta attitudes and teamwork. However, the
authors concluded the eectiveness of these programs on safety culture and patient outcomes is questionable, with few
studies reporting outcomes. Studies that did explore outcomes found small eects.
RQ2: Mechanisms of Change
One eld that has taken particular interest in the ecacy of teamwork skills interventions is health care (e.g., Buljac-
Samardzic et al., 2010). Gordon et al. (2012) conducted a literature review of nontechnical skills interventions, including
teamwork, leadership, and communication, in health-care settings. ey noted that interventions primarily included sim-
ulations or role-plays, with an emphasis on debrieng, feedback, and simulation delity. McCulloch et al. (2011) reviewed
teamwork interventions for health-care sta and noted that most interventions were based on crew resource management
training adapted from the aviation eld. ey reported that literature in this area is lacking, as little detail is provided on
the specic components of training and reporting of outcome data is poor, with little statistical data provided.
In Weaver et al.’s (2014) review examining health-care team training evaluations, the authors suggested team train-
ing primarily targets communication, situational awareness, leadership, and role clarity. Training methods included
information-based methods, demonstration-based methods, and practice-based methods. Salas et al. (2008) qualitatively
examined team training studies and concluded that training is eective in both aviation and health care. ey stressed
that hands-on learning via simulationsis important and suggested that when high delity simulators are used, it is critical
that they be realistic. ey also state that behavior-based feedback is important, as it helps trainees determine where and
how trainees can improve.
Several other studies expanded in scope to focus on the development of teamwork skills in other domains. Klein
et al.’s (2009) meta-analysis showed that team building is intended to improve interpersonal relations and social interac-
tions, achieve results, and accomplish tasks. Team-building interventions were moderately eective regardless of strategy,
but the role-clarication component of interventions was slightly more eective than goal setting, interpersonal relations,
or problem solving components. Salas et al. (2007) conducted a meta-analysis of the eects of three dierent components
of team training on the eectiveness of team-training interventions. Results demonstrated that focusing on coordination
and adaptation in team training resulted in greater improvements in team eectiveness as opposed to cross-training and
team self-correction training.
Summary and Recommendations: Interpersonal Skills
To summarize, evidence has suggested interpersonal skills may be improved via intervention. Programs targeting these
skills are generally as eective as those targeting cognitive skills (Aguinis & Kraiger, 2009). However, not all studies
concluded that training interpersonal skills results in substantial improvement. Possible strategies for honing adult inter-
personal skills include motivating and goal setting, coaching and mentoring, role-plays, feedback, BMT, multimedia and
simulation-based training, team training, deep learning, and PBL.
Within the social skills domain, many studies reviewed have gathered support for the eectiveness of using interven-
tions to improve specic constructs. Communication skills interventions include debriengs, checklists, simulations,
instruction, modeling, skill practice, feedback, cognitive and experiential learning, simulated patients, and role-play
exercises. Although based on only three sources examining leadership interventions, the consensus across all of the
meta-analytic evidence reviewed is that leaders are likely made, not born. However, Avolio et al. (2009) reported there
was largely no dierence in the eectiveness of dierent intervention methods, with the exception that interventions
12 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
based on Pygmalion theory were more eective than those based on traditional or newer leadership theories. Overall,
teamwork skills appear to be amenable to moderate change through intervention. Training methods for developing
teamwork skills may take the form of simulations or role-plays, information-based methods, demonstration-based
methods, practice-based methods, and small-group learning.
Results: Intrapersonal Skills
Personality
RQ1: Malleability
e Big Five is the most widely adopted theoretical model of personality traits (e.g., McCrae & Costa Jr., 2008). How-
ever, longitudinal Big Five research appears to be limited to those investigating naturalistic developmental changes
(e.g., McCrae & John, 1992; Roberts et al., 2006) or clinical interventions (e.g., Roberts, Luo, et al., 2017; Lipsey &
Wilson, 1993). An additional meta-analysis provided relevant outcomes for resilience, a personality construct that
overlaps with the Big Five (e.g., Friborg et al., 2005; John & Srivastava, 1999), although they did not explicitly use Big
Five terminology. Specically, Vanhove et al. (2016) noted only small intervention eects for resilience in response to
workplace programs targeting psychosocial factors. e authors also observed diminishing eects of these programs
over time.
RQ2: Mechanisms of Change
Vanhove et al.’s (2016) meta-analysis of workplace resilience-building programs included a diverse array of primary pre-
vention methods that focus on developing psychosocial traits such as self-ecacy and optimism. Some programs also
incorporated secondary or tertiary methods such as stress management, physical tness, or meditation. Of note, these
interventions are oen informed by resilience theory. A central tenet of this theory is that occupational groups exposed
to greater levels of work-related stress and trauma will be most informative in identifying factors and mechanisms that
facilitate resilience (see review by Vanhove et al., 2016). Importantly, workplace stress is not isolated to that caused by
acute traumatic experiences, as repeated milder stressors may also result in cumulative psychological damage. However,
the diversity of programs precluded examination of ecacy dierences across intervention types.
Attitudes
RQ1: Malleability
We located 11 meta-analyses and narrative reviews examining the malleability of attitudes through deliberate intervention
(see Table 2. ese studies consistently demonstrated that interventions eectively reduce attitudes related to several types
of counterproductive work behavior (CWB; e.g., Bangert-Drowns, 1988; Bezrukova et al., 2016; Brecklin & Forde, 2001;
Kalinoski et al., 2013) and improve attitudes toward various positive job performance areas (Guskey, 1986; Jones, 2016;
Lipsey & Wilson, 1993). When these changes are quantied, they are typically of small-to-moderateeect sizes (e.g., Barth
& Lannen, 2011).
RQ2: Mechanisms of Change
Attitude interventions may be dierentiated by their content or approach. In some instances, a wide variety of approaches
exists for altering attitudes even within a specic context. Given that the attitude interventions we reviewed tended to be
relevant to specic forms of CWB, they were delivered in various workplace settings. For example, Anderson and Whis-
ton (2005) described four main categories of sexual assault education programs: informative programs discuss factual
information, statistics, myths, facts, and consequences; empathy-focused interventions emphasize developing empathy
for victims; socialization-focused programs examine gender-role stereotyping and societal inuences; and risk-reducing
interventions teach specic strategies to reduce victimization risk. For reducing rape attitudes and rape-related attitudes,
empathy-focused interventions tended to be less eective than the other strategies, whereas program content was unas-
sociated with reductions in behavioral intent. In improving intergroup attitudes (e.g., diversity and inclusion training),
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 13

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Beelmann and Heinemann (2014) reported that social-cognitive training designed to promote empathy and perspective
taking were most eective, whereas programs involving classication/social categorization (i.e., the cognitive process of
classifying individuals based on demographics) or problem-solving skills were least eective. Otherwise, theoretical orien-
tation (e.g., socialization/knowledge acquisition; social-cognitive development) did not impact eectiveness. Undesirable
attitudes toward controlled or illegal substances have been demonstrated to be more eectively reduced when interven-
tions employ relatively contemporary methods involving reinforcement, social norms, and developmental interventions
than when simply focusing on more traditional information such as adverse health eects.
In other areas, attitude change may be observed when interventions are keenly designed for an intended context, or
conversely when the attitudes in question were not the primary focus. For example, Jones (2016) concentrated on work-
place coaching strategies in identifying eective methods for improving workplace attitudes. On the other hand, Barth and
Lannen (2011) reported that communication skills training in health-care settings somewhat unintentionally improved
attitudes toward terminally ill patients, death, and dying. A slightly dierent perspective was described by Guskey (1986),
who used a process model to contend that improvements in teachers’ attitudes toward classroom practices (e.g., curricu-
lumchanges)shouldrstpromotechangesintheirownclassroombehaviorsandinstudentlearningoutcomesratherthan
vice-versa. In other words, attitude change is most eective when viewed as a secondary, indirect target of intervention
rather than a primary or direct one.
Self-Concept
RQ1: Malleability
We located three meta-analyses and reviews describing the eortful malleability of self-concept (see Table 2). ese studies
support the malleability of self-ecacy in the context of online learning (Hodges, 2008), general self-ecacy (Buljac-
Samardzic et al., 2010), and general self-concept (e.g., self-control, self-ecacy; Bowen & Neill, 2013). Meta-analytic
estimates of pre– post improvements in self-concept were in the small-to-moderate range (Bowen & Neill, 2013). How-
ever, these authors also noted that these changes were not maintained at long-term follow-up assessment. Unfortunately,
Bowen and Neill (2013) did not describe the specic time period representing long-term eects.
RQ2: Mechanisms of Change
Strategies employed to promote eortful improvements in self-esteem and self-ecacy have varied. Successful approaches
have included motivational messages (Hodges, 2008), goal-setting (Buljac-Samardzic et al., 2010), and adventure therapy
(Bowen & Neill, 2013), which is an intervention focused on outdoor experiential learning that is not necessarily clinical
in nature. Bowen and Neill (2013) also reported that alternative interventions to adventure therapy were not eective but
did not elaborate on what types of programs were evaluated. Overall, these results suggested a diverse assortment of self-
concept improvement tactics applicable to a variety of individuals,evenwhenself-conceptisnotnecessarilytheprimary
target of interventions.
Emotion
RQ1: Malleability
Five articles we identied examined deliberate eorts to alter emotions or aect (see Table 2. e consensus from these
reviews was that emotions are indeed malleable. ese ndings are consistent regardless of whether aect is dened
broadly (Augustine & Hemenover, 2009; Webb et al., 2012) or using more specic examples such as empathy (Teding
van Berkhout & Malou, 2016), or in contexts such as team eectiveness (Delise et al., 2010) or risk appraisal (Sheeran
et al., 2014).
RQ2: Mechanisms of Change
Regarding the specic programs used to engender deliberate aective change, our review revealed a diverse array of strate-
gies. Augustine and Hemenover (2009) identied over 300 aect regulation or repair strategies in the literature. For their
14 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
analyses, they adopted Parkinson and Totterdell’s (1999) four superordinate categories. In this model, the rst distinc-
tion separates behavioral distractions— which involve some type of overt physical action such as walking away from a
distressing situation—from cognitive actions, with examples including thinking about something other than the distress-
ing situation. e second distinction categorizes strategies as engagement—in which one actively attends to the aective
experience through actions such as reappraisal— versus avoidance, in which the individual behaviorally or cognitively
removes themselves from the distressing situation. Augustine and Hemenover (2009) compared the ecacy of these four
categories of interventions, 10 subordinate strategies that fall under these superordinate categories, and a separate tax-
onomy of specic strategies. e four superordinate categories demonstrated generally similar results (small-to-medium
eect sizes), with behavioral strategies being slightly more eective.
Webb et al. (2012) categorized interventions using a process model of emotion regulation, in which strategies are
dened based on whether they occur prior to the emotional experience (antecedent-focused) or aerward (response-
focused; Gross & ompson, 2007). One antecedent-focused example is attentional deployment, which includes distrac-
tions (i.e., either active or passive and either positive or neutral) and concentration (e.g., concentrating on feelings or on
causes and implications). Cognitive change is another antecedent-focused example, with specic strategies varying based
on their tendency to reappraise emotional responses, emotional stimuli, or through perspective taking. Finally, response-
focused strategies involve response modulation through the suppression of the emotion-related expression, experience,
or event. e authors reported that attentional deployment had no eect on emotional outcomes, response modulation
approached a small eect, and cognitive change had a small-to-medium eect.
One unique example of emotion manipulation was examined by Sheeran et al. (2014). ese authors studied the
impact of interventions designed to increase negative aect (NA) associated with perceptions of risk or threat as strate-
gies for decreasing various problematic behaviors. A common example includes health warnings designed to deter various
self-destructive behaviors (e.g., smoking, poor diet, etc.). In this context, emotions are typically categorized based on
whether they precede the target behavior (i.e., anticipatory emotions such as fear or worry) or follow it (i.e., anticipated
emotions such as regret and guilt). Accordingly, the strategies designed to increase these emotions are described as
heightening risk appraisal. e authors did not assess specic interventions separately, but noted that heightening risk
appraisal increased anticipatory and anticipated emotions by moderate-to-large and small-to-moderate eect sizes,
respectively.
e remaining reviews generally included more specic types of interventions. Perhaps the most pertinent class of
interventions— empathy training—was examined by Teding van Berkhout and Malou (2016). Empathy training tends
to include elements of behavioral skills training (i.e., modeling, instructions, rehearsal, and feedback) designed to target
the aective, cognitive, or behavioral components of empathy. Specic components include lectures, demonstrations,
practice, games, and role-play. Interestingly, despite the fact that emotions were not necessarily the intended target of
these interventions, each was eective in improving aect to varying degrees.
Motivation
RQ1: Malleability
e malleability of intrinsic motivation has been the source of intense debate. Cameron and Pierce (1994) argued in
their controversial meta-analysis of the eects of extrinsic rewards on intrinsic motivation that extrinsic rewards do not
decrease intrinsic motivation. ey posited that verbal praise increases intrinsic motivation when motivation is assessed
using free time and attitude measures. However, when individuals are given expected tangible rewards for completing a
task, intrinsic motivation decreases when measured as free time performance in which a participant has the opportunity to
work on a task when no rewards are being provided and when the presumption is that the experimenter is not monitoring
the participant’s activity. It is worth noting that both the number of eect sizes and size of the observed eects are small.
As noted by Ryan and Deci (1996), the authors’ aggregation of all reward categories into one global variable may mask
signicant eects at ner-grained categories. Deci et al. (1999, 2001) reiterated in later research that extrinsic rewards do
indeed undermine intrinsic motivation and supported this claim with meta-analytic evidence (Deci et al., 1999; see also
Rummel & Feinberg, 1988). us, Cameron and Pierce’s ndings warrant extreme caution.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 15

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
RQ2: Mechanisms of Change
Jones (2016) presented an overview of the history and eectiveness of workplace coaching. Workplace coaching is
described as learner-centered, collaborative, reective, goal-focused instruction that can provide coachees with a
tailored approach to understanding and applying work-based learning (Jones, 2016). According to Jones, coaching oen
focuses on challenges within the individual, between individuals, or a combination. Jones presented a meta-analysis of
coaching eectiveness, showing that coaching has a moderately positive eect on motivation. Additionally, Sitzmann
and Ely (2011) conducted a meta-analysis that suggested that goal setting results in greater learning when individuals
are committed to reaching a specic goal, possess the requisite task knowledge, and are provided with feedback on
their progress toward their goal or goals. ey argued that specic and dicult goals that are attainable motivate
performance.
Summary and Recommendations: Intrapersonal Skills
Our review suggests there is much to be learned about personality constructs’ amenability to deliberate intervention. As
mentioned, studies examining naturalistic, developmental changes (McCrae & John, 1992; Roberts et al., 2006) should
not be interpreted as support for the Big Five’s amenability to eortful intervention. However, this research may describe
common life events that catalyze personality change, and may in turn inform eortful interventions. Similarly, research
describing personality change in response to clinical intervention (e.g., Lipsey & Wilson, 1993) will likely possess debat-
able transportability to workplace settings. Further research is needed to determine if components of clinical interventions
(e.g., practice exercises; self-reection) may be relevant to workplace settings. Indeed, relevant intervention programs have
already been proposed: Roberts, Luo, et al.’s (2017) Sociogenomic Trait Intervention Model (STIM) incorporates elements
of behavioral activation theory, motivational theories, and developmental research as a potential strategy for stimulating
changes in Big Five conscientiousness, for example. Furthermore, most workplace interventions tend to be delivered by
trainers or coaches who do not hold a clinical certication. Lessons learned from successful workplace interventions
targeting constructs beyond the Big Five may be useful. However, given that resilience interventions appear to possess
relatively low ecacy, other noncognitive construct interventions may be more informative. Finally, even though the
Big Five is considered the predominant model of personality constructs, other workplace-relevant personality constructs
(e.g., those comprising the Dark Triad; Paulhus & Williams, 2002) would also benet from further eortful malleability
research.
Attitudes may be associated with virtually any topic, providing ample opportunities for empirical study. More speci-
cally, the fact that we identied a large number of studies examining attitudes in the context of eortful change suggests
that they are ideal targets for deliberate intervention. In particular, reviews involving workplace samples such as health-
care professionals (Barth & Lannen, 2011), teachers (Guskey, 1986), or general workplace settings (Jones, 2016) reveal
that attitudes are relevant for a variety of professions. Furthermore, the content of these interventions suggest that they
are pertinent to both positive and negative job performance outcomes. e consensus among these reviews is that atti-
tudes are amenable to change, albeit to a relatively small degree. At the same time, given that many attitudes appear to
have an aective component (e.g., Beelmann & Heinemann, 2014), it is possible that emotion-focused interventions may
have some value in promoting attitude change. It will also be benecial to examine whether attitude interventions evoke
behavioral change, which has been a challenge in some settings.
e fundamental importance of self-concept suggests that it is a highly salient target for workplace interventions, and
there is indeed empirical evidence for its eectiveness in these settings (i.e., health-care professionals; Buljac-Samardzic
et al., 2010). Nonetheless, given the variety of successful interventions, care must be taken to identify components that
are particularly relevant to the workplace. For instance, at rst glance, programs such as adventure therapy may appear
irrelevant to workplace settings. However, several components of adventure therapy may be conducted indoors and are
already employed in workplace interventions, including trust activities, initiative experiences, problem-solving scenarios,
and team-based tasks (Bowen & Neill, 2013), suggesting a high degree of transportability. Additionally, the notion that
program scope should match the intended content area of expertise—also referred to as the “bandwidth” issue—appears
particularly relevant to self-ecacy (Hodges, 2008): interventions promoting engineering self-ecacy would be of little
usetonursingprofessionals,forinstance.
16 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Emotion’somnipresenceindailylifeisreectedintherelativeabundanceofmeta-analysesandreviewsexaminingits
malleability. is research benets from a relatively high proportion of meta-analyses as opposed to narrative reviews,
allowing for a more systematic and quantitative summary. Overall, aect appears to represent a category of noncognitive
constructs susceptible to change, even when the intervention in question is not necessarily designed to target emotion per
se. However, the sheer volume of interventions (see Augustine & Hemenover, 2009) may preclude the investigation of any
one specic program. Nonetheless, another advantage of the interventions discussed in this section is that many of them
appear transportable to the workplace, such as the distraction and engagement strategies examined by Augustine and
Hemenover (2009). In fact, the emotion reviews we identied explicitly included occupations ranging from the military
(Delise et al., 2010) to health care (Teding van Berkhout & Malou, 2016). In turn, these interventions may be predom-
inantly relevant to occupations involving frequent interpersonal interaction, with signicant and frequent stressors, or
where prosocial behavior is particularly valued. e results reported by Sheeran et al. (2014) suggested that increasing NA
by heightening risk appraisal can be eective in reducing a variety of workplace-relevant undesirable behaviors. Although
this practice is common, the ethics of this strategy may be debatable, as the experience for the individual may be negative.
Furthermore, conicting results within emotion research (Webb et al., 2012) suggest that increasing positive aect and
decreasing NA are not necessarily opposite sides of the same coin and that future investigations clarifying this distinction
would be welcomed.
A great deal of evidence suggests motivation in the context of discrete tasks can be improved or diminished using
interventions. Interventions to improve motivation may include workplace coaching (Jones, 2016), goal setting (Sitzmann
& Ely, 2011), praise, and rewards. Taken together, the ndings of the studies reviewed suggest that extrinsic rewards tend
to decrease intrinsic motivation (c.f. Cameron & Pierce, 1994).
Discussion
is review aimed to summarize and evaluate the current state of the literature on the intervention-based development of
two broad categories of noncognitive constructs germane to workplace success: interpersonal and intrapersonal skills. Our
multidisciplinary approach drew from meta-analyses and reviews spanning a diverse set of domains to provide a thorough
review of the malleability of noncognitive constructs. More broadly, this review contributes to a growing interest in applied
arenas in the promotion and development of noncognitive constructs that have been shown to be critical in supporting
workplace success. Overall, recognizing a few exceptions and gaps in the literature, ndings suggest optimism regarding
the malleability of noncognitive constructs, and provide a preliminary blueprint for the optimal design, implementation,
and evaluation of intervention programs.
RQ1: Noncognitive Construct Malleability
Among the appealing features of noncognitive constructs are that they predict workplace success to a degree similar to
traditional factors such as cognitive skills yet are potentially more malleable (e.g., Roberts, Hill, & Davis, 2017; Roberts,
Luo, et al., 2017). Our review supported the malleability of several distinct noncognitive constructs. e majority of meta-
analytic estimates are of small to moderate eect sizes (e.g., d≈0.20–0.50), though occasionally interventions produce
smaller or larger eects. For instance, communication skills, leadership skills, and emotion tend to consistently produce
the strongest results supporting malleability, typically higher than Cohen’s (1988) guideline for a moderate eect size (e.g.,
d≈±0.50). Conversely, eect sizes for personality, interpersonal skills, teamwork, attitudes, self-concept, and motivation
tend to be smaller or more inconsistent. At the same time, the amount of meta-analytic research varies widely across
constructs: We were only able to locate one relevant meta-analysis each for personality, communication skills, and self-
concept, compared to 11 attitudes meta-analyses. Eortful malleability meta-analyses were absent altogether for several
workplace-relevant constructs including the Big Five, Dark Triad, and emotional intelligence.
is trend suggests that research investigating the longitudinal dynamics of noncognitive constructs lags behind that
of predictive validity research. Aer identifying noncognitive constructs based on their established links to job per-
formance, malleability research serves a critical supplementary role in that it claries the subset of constructs with an
evidence-based rationale for intervention. Moreover, this distinction provides essential practical guidance for employers:
Constructs—noncognitive or otherwise—that are relatively xed may be more relevant to personnel selection decisions,
whereas training decisions may be informed by identifying constructs that are more malleable.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 17

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
RQ2: Measurement and Mechanisms of Noncognitive Construct Change
e vast majority of malleability research quantied changes using rank-order stability or mean-level changes. As previ-
ously mentioned, these strategies are oen considered complementary, recognizing their respective strengths and weak-
nesses. It is likely that the eld would benet from additional research involving more sophisticated statistical approaches,
including latent variable modeling, growth curve analyses, structural continuity, ipsative continuity, and coherence (e.g.,
Caspi & Roberts, 2001; Curran et al., 2010; McArdle, 2009). Importantly, these methods permit the assessmentof changes
across more than two time points, which is a prerequisite for evaluating nonlinear longitudinal trends. Similarly, tech-
niques such as mediation analyses, repeated measures ANOVA, and structural equation modeling provide opportunities
to examine the causal impact of interventions (e.g., Cole & Maxwell, 2003; Pearl, 2009). ese advances may coincide
with increased usage of nontraditional noncognitive construct assessments, including forced-choice, game-based, and
performance-based measures.
e diversity of disciplines included in our review gives way to an even more abundant array of strategies for eortful
noncognitive construct change. e list of intervention approaches that have been used frequently enough to merit meta-
analytic study or narrative review is extremely diverse. Examples include modeling, goal-setting, character education,
sports participation, adventure therapy, role-playing, and reward-based programs. e specic exercises embedded within
these programs both aids in explaining their ecacy and may inform the development of new programs. Additionally, it
would be prudent to consider change catalysts observed in clinical interventions or naturalistic developmental, as a subset
of these features may be transportable to workplace settings. However, appropriate parameters must be applied to ensure
that interventions are appropriate for organizational trainers and coaches who do not have a clinical background and for
employeeswhomayengagewithprogramsoeredinaself-directedcapacity.
Limitationsand Future Directions
Our review is not without limitations, many of which surround our search strategy. First, we chose to review only source
articles that could be characterized as meta-analyses or reviews. Consequently, very recent research and some primary
studies may not have been captured by the selection of source material reviewed and would thus have been excluded
from this investigation. It should also be noted that, although meta-analyses have been widely accepted as a vital tool
for aggregating primary research, some authors remain skeptical of their value (e.g., Costa Jr. & McCrae, 2006). Second,
the use of Google Scholar as opposed to more traditional databases such as PsycINFO as our primary search engine
could be a potential limitation, based on the criticisms of some researchers (e.g., Boeker et al., 2013; Giustini & Bou-
los, 2013). Conversely, other authors have praised Google Scholar as “an invaluable tool for conducting literature research”
(de Winter et al., 2014, p. 1562), and have supported it as the rst and potentially sole search engine for reviews and meta-
analyses (e.g., Gehanno et al., 2013). Nonetheless, we supplemented our Google Scholar search with various additional
strategies.
Other limitations concern the nature of the meta-analyses and reviews that arose from our search. Most notably,
there is a paucity of studies examining the malleability of some noncognitive constructs (e.g., leadership skills). Meta-
analyses or systematic reviews for other constructs such as grit, emotional intelligence, interests, the Dark Triad of
personality, and integrity were either too few to be included in our review or were not located in our search. is issue
is exacerbated by the fact that there does not appear to be a universal, broadly accepted, exhaustive list of noncognitive
constructs, given that this research is constantly expanding and that some variability exists regarding the denition
of noncognitive (see Duckworth & Yeager, 2015; Heckman & Kautz, 2014; Kell, 2018). In other instances, outcomes
were oen measured at the group level rather than the individual participant level (e.g., teamwork). e expansion
of research to specic occupational elds beyond the somewhat narrow subset located in our review (e.g., health
care) would also be benecial. Although we restricted our review to studies that focused on adult participants, studies
for some constructs (e.g., emotion) included both youth and adults. Methodologically, another caveat worth noting
concerns the inclusion of research examining rank-order consistency as opposed to mean-level change or vice-versa.
Studies employed dierent approaches for examining change and both methods have drawbacks, especially compared to
more contemporary analytical techniques and their associated methodological designs (e.g., Biesanz et al., 2003; McAr-
dle, 2009). Moreover, even in studies that examine within-person change pre- and postintervention, natural regression
18ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
to the mean may be confounded with change as a result of intervention. Additionally, traditional eect size interpre-
tation guidelines (e.g., Cohen, 1988) could be supplemented with data-driven benchmarks specically derived from
noncognitive construct malleability research, a strategy that has been examined in other research elds (e.g., e.g., Bosco
et al., 2015; Gignac & Szodorai, 2016; Paterson et al., 2015). Each of these issues represents appropriate targets for future
research.
ConcludingComments
Empirical support for the eortful malleability of workplace-relevant noncognitive constructs abounds, though many
areas of research remain open. Meta-analyses and reviews provide data-driven guidance to stakeholders regarding eective
selection and training of employees in terms of skills whose value is both recognized by employers and supported by
research. At the same time, additional research regarding understudied noncognitive constructs, specic interventions,
and underused methodologies and statistical approaches will undoubtedly advance the eld. Ideally, this research will
facilitate the development of a workforce whose technical expertise and cognitive skills are complemented by important
noncognitive assets.
Note
1 Some components of personality may be considered interpersonal (e.g., Big Five Agreeableness), though we include them with
their intrapersonal counterparts to maintain discussion of the Big Five within one self-contained section.
References
References for articles included in the systematic review are marked with an asterisk
Aguinis, H., & Kraiger, K. (2009). Benets of training and development for individuals and teams, organizations, and society. Annual
Review of Psychology,60, 451 –474. https://doi.org/10.1146/annurev.psych.60.110707.163505
*Anderson, L. A., & Whiston, S. C. (2005). Sexual assault education programs: A meta-analytic examination of their eectiveness.
Psychology of Women Quarterly,29, 374– 388. https://doi.org/10.1111/j.1471-6402.2005.00237.x
*Arthur, W., Jr., Bennett, W., Jr., Edens, P. S., & Bell, S. T. (2003). Eectiveness of training in organizations: A meta-analysis of design
and evaluation features. Journal of Applied Psychology,88, 234–245. https://doi.org/10.1037/0021-9010.88.2.234
*Augustine, A. A., & Hemenover, S. H. (2009). On the relative eectiveness of aect regulation strategies: A meta-analysis. Cognition
and Emotion,23, 1181 –1220. https://doi.org/10.1080/02699930802396556
*Avolio, B. J., Avey, J. B., & Quisenberry, D. (2010). Estimating return on leadership development investment. Leadership Quarterly,
21, 633–644. https://doi.org/10.1016/j.leaqua.2010.06.006
*Avolio, B. J., Reichard, R. J., Hannah, S. T., Walumbwa, F. O., & Chan, A. (2009). A meta-analytic review of leadership impact research:
Experimental and quasi-experimental studies. Leadership Quarterly,20, 764– 784. https://doi.org/10.1016/j.leaqua.2009.06.006
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology,41, 63 –105.
https://doi.org/10.1111/j.1744-6570.1988.tb00632.x
*Bangert-Drowns, R. L. (1988). e eects of school-based substance abuse education—A meta-analysis. Journal of Drug Education,
18, 243–264. https://doi.org/10.2190/8U40-WP3D- FFWC-YF1U
Barrick, M. R., & Mount, M. K. (1991). e big ve personality dimensions and job performance: A meta-analysis. Personnel Psychology,
44, 1–26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x
*Barth, J., & Lannen, P. (2011). Ecacy of communication skills training courses in oncology: A systematic review and meta-analysis.
Annals of Oncology,22, 1030 – 1040. https://doi.org/10.1093/annonc/mdq441
*Beelmann, A., & Heinemann, K. S. (2014). Preventing prejudice and improving intergroup attitudes: A meta-analysis of child and
adolescent training programs. Journal of Applied Developmental Psychology,35, 10–24. https://doi.org/10.1016/j.appdev.2013.11.002
Bell, B. S., Tannenbaum, S. I., Ford, J. K., Noe, R. A., & Kraiger, K. (2017). 100 years of training and development research: What we
know and where we should go. Journal of Applied Psychology,102, 305–323. https://doi.org/10.1037/apl0000142
Bezrukova, K., Spell, C. S., Perry, J. L., & Jehn, K. A. (2016). A meta-analytical integration of over 40 years of research on diversity
training evaluation. Psychological Bulletin,142, 1227– 1274.
Biesanz, J. C., West, S. G., & Kwok, O.-M. (2003). Personality over time: Methodological approaches to the study of short-term and
long-term development and change. Journal of Personality,71, 905–941. https://doi.org/10.1111/1467-6494.7106002
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 19

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Bleidorn, W., Hopwood, C. J., & Lucas, R. E. (2018). Life events and personality trait change. Journal of Personality,86, 83 –96. https://
doi.org/10.1111/jopy.12286
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta-analytic review. Journal of Management,36,
1065–1105. https://doi.org/10.1177/0149206309352880
Boeker, M., Vach, W., & Motschall, E. (2013). Google scholar as replacement for systematic literature searches: Good relative recall and
precision are not enough. BMC Medical Research Methodology,13, 1 – 12. https://doi.org/10.1186/1471-2288-13- 131
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational eect size benchmarks. Journal of Applied Psychol-
ogy,100, 431–449. https://doi.org/10.1037/a0038047
*Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and moderators. Open Psychology Journal,6, 28 –53.
https://doi.org/10.2174/1874350120130802001
Brecklin, L. R., & Forde, D. R. (2001). A meta-analysis of rape education programs. Violence and Victims,16, 303-321.
Briley, D. A., & Tucker-Drob, E. M. (2014). Genetic and environmental continuity in personality development: A meta-analysis. Psy-
chological Bulletin,140, 1303–1331. https://doi.org/10.1037/a0037091
Bruvold, W. H., & Rundall, T. G. (1988). A meta-analysis and theoretical review of school based tobacco and alcohol intervention
programs. Psychology and Health,2, 53-78.
*Buljac-Samardzic, M., Dekker-van Doorn, C. M., van Wijngaarden, J. D., & van Wijk, K. P. (2010). Interventions to improve team
eectiveness: A systematic review. Health Policy,94, 183– 195. https://doi.org/10.1016/j.healthpol.2009.09.015
*Cameron, J., & Pierce, W. D. (1994). Reinforcement, reward, and intrinsic motivation: A meta-analysis. Review of Educational Research,
64, 363–423. https://doi.org/10.3102/00346543064003363
Capelli, P. (2012). Why good people can’t get jobs: e skills gap and what companies can do about it.WhartonDigitalPress.
Card, N.A. (2012). Applied meta-analysis for social science research. Guilford.
Caspi, A., & Roberts, B. W. (2001). Personality development across the life course: e argument for change and continuity. Psychological
Inquiry,12, 49–66. https://doi.org/10.1207/S15327965PLI1202_01
*Cheraghi-Sohi, S., & Bower, P. (2008). Can the feedback of patient assessments, brief training, or their combination, improve the
interpersonal skills of primary care physicians? A systematic review. BMC Health Services Research,8, 1 –10. https://doi.org/10.1186/
1472-6963-8-179
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Cole, D. A., & Maxwell, S. E. (2003). Testing mediation models with longitudinal data: Questions and tips in the use of structural
equation modeling. JournalofAbnormalPsychology,112, 558 – 577. https://doi.org/10.1037/0021-843X.112.4.558
Costa, P. T., Jr., & McCrae, R. R. (2006). Age changes in personality and their origins: Comment on Roberts, Walton, and Viechtbauer
(2006). Psychological Bulletin,132, 26– 28. https://doi.org/10.1037/0033-2909.132.1.26
Curran, P. J., Obeidat, K., & Losardo, D. (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognitive
Development,11, 121 –136. https://doi.org/10.1080/15248371003699969
de Winter, J. C. F., Zadpoor, A. A., & Dodou, D. (2014). e expansion of Google Scholar versus Web of Science: A longitudinal study.
Scientometrics,98, 1547– 1565. https://doi.org/10.1007/s11192-013-1089-2
*Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the eects of extrinsic rewards on
intrinsic motivation. Psychological Bulletin,25, 627 – 668. https://doi.org/10.1037/0033-2909.125.6.627
*Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education: Reconsidered once again.
Review of Educational Research,71, 1– 27. https://doi.org/10.3102/00346543071001001
*Delise, L. A., Gorman, C. A., Brooks, A. M., Rentsch, J. R., & Steele-Johnson, D. (2010). e eects of team training on team outcomes:
A meta-analysis. Performance Improvement Quarterly,22, 53–80. https://doi.org/10.1002/piq.20068
Duckworth, A. L., & Yeager, D. S. (2015). Assessing personal qualities other than cognitive ability for educational purposes. Educational
Researcher,44, 237– 251. https://doi.org/10.3102/0013189X15584327
Freifeld, L. (2018). TRAINING Industry report. https://trainingmag.com/trgmag-article/2018-training-industry-report/
Friborg, O., Barlaug, D., Martinussen, M., Rosenvinge, J. H., & Hjemdal, O. (2005). Resilience in relation to personality and intelligence.
International Journal of Methods in Psychiatric Research,14, 29– 42. https://doi.org/10.1002/mpr.15
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013). Is the coverage of Google scholar enough to be used alone for systematic reviews?
BMC Medical Informatics and Decision Making,13, 1– 5. https://doi.org/10.1186/1472-6947-13-7
Gignac, G. E., & Szodorai, E. T. (2016). Eect size guidelines for individual dierences researchers.Per sonality and Individual Dierences,
102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069
*Gillespie, B. M., Chaboyer, W., & Murray, P. (2010). Enhancing communication in surgery through team training interventions: A
systematic literature review. AORN Journal,92, 642–657. https://doi.org/10.1016/j.aorn.2010.02.015
Giustini, D., & Boulos, M. N. (2013). Google scholar is not enough to be used alone for systematic reviews. Online Journal of Public
Health Informatics,5, 214. https://doi.org/10.5210/ojphi.v5i2.4623
20 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
*Gordon, M., Darbyshire, D., & Baker, P. (2012). Non-technical skills training to enhance patient safety: A systematic review. Medical
Education,46, 1042–1054. https://doi.org/10.1111/j.1365-2923.2012.04343.x
Gottfredson, L. S. (1998). e general intelligence factor. Scientic American,9, 24– 29.
Gross, J. J., & ompson, R. A. (2007). Emotion regulation: Conceptual foundations. In J.J. Gross (Ed.), Handbook of emotion regulation
(pp. 3– 24). Guilford Press.
*Guskey, T. R. (1986). Sta development and the process of teacher change. Educational Researcher,15, 5 –12. https://doi.org/10.3102/
0013189X015005005
Heckman, J. J., & Kautz, T. D. (2012). Hard evidence on so skills. Labour Economics,19, 451– 464. https://doi.org/10.1016/j.labeco
.2012.05.014
Heckman, J. J., & Kautz, T. D. (2014). Achievement tests and the role of character in American life. In J. J. Heckman, J. E. Humphries, &
T. Kau t z ( E d s . ) , e myth of achievement tests: e GED and the role of character in American life (pp. 1– 71). University of Chicago
Press. https://doi.org/10.7208/chicago/9780226100128.001.0001
Heckman, J. J., & Rubinstein, Y. (2001). e importance of noncognitive skills: Lessons from the GED testing program. American
Economic Review,91, 145– 149. https://doi.org/10.1257/aer.91.2.145
*Hodges, C. B. (2008). Self-ecacy in the context of online learning environments: A review of the literature and directions for research.
Performance Improvement Quarterly,20, 7–25. https://doi.org/10.1002/piq.20001
*Hulsman, R. L., Ros, W. J. G., Winnubst, J. A. M., & Bensing, J. M. (1999). Teaching clinically experienced physicians communication
skills. A review of evaluation studies. Medical Education,33, 655– 668. https://doi.org/10.1046/j.1365-2923.1999.00519.x
Jensen, A. R. (1998). e g factor: e science of mental ability. Praeger.
John, O. P., & Srivastava, S. (1999). e big ve trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin &
O. P. John (Eds.), Handbook of personality (2nd ed.; pp. 102– 138). Guilford.
*Jones, R. (2016). e eectiveness of workplace coaching: A meta-analysis of learning and performance outcomes,scale development,
theoretical model of individual dierences and longitudinal study (Unpublished doctoral dissertation). Aston University, Birmingham,
AL.
*Kalinoski, Z. T., Steele-Johnson, D.,Peyton, E. J., Leas, K. A., Steinke, J., & Bowling, N. A. (2013). A meta-analytic evaluation of diversity
training outcomes. Journal of Organizational Behavior,34, 1076 –1104. https://doi.org/10.1002/job.1839
Kautz, T., Heckman, J. J., Diris, R., Ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and noncog-
nitive skills to promote lifetime success (NBER Working Paper No. 20749). National Bureau of Economic Research. https://doi.org/10
.3386/w20749
Kell, H. J. (2018). Noncognitive proponents’ conation of “cognitive skills” and “cognition” and its implications. Personality and Indi-
vidual Dierences,134, 25–32. https://doi.org/10.1016/j.paid.2018.05.025
Kirkpatrick, D. (1996). Great ideas revisited: Revisiting Kirkpatrick’s four-level model. Tra ini ng & De vel opm e nt ,50, 54– 57.
*Klein, C., DiazGranados, D., Salas, E., Le, H., Burke, C. S., Lyons, R., & Goodwin, G. F. (2009). Does team building work? Small Group
Research,40, 181– 222. https://doi.org/10.1177/1046496408328821
*Klein, C. R. (2009). What do we know about interpersonal skills? A meta-analytic examination of antecedents, outcomes, and the ecacy
of training (Unpublished doctoral dissertation). University of Central Florida, Orlando, FL.
Klieger, D., Ezzo, C., Bochenek, J., & Cline, F. (2015, April 16– 20). e predictive validity of non-cognitive skills for graduate and pro-
fessional student success: Some initial ndings. Paper presented at the Validity and Fairness Issues in Assessing Higher Education
Students Symposium, Annual Meeting of the American Educational Research Association, Chicago, IL.
Kyllonen, P. C. (2012, May 7–8). Measurement of 21st century skills within the Common Core State Standards. Paper presented at the
Invitational Research Symposium on Technology Enhanced Assessments, Princeton, NJ.
*Lacerenza, C. N., Reyes, D. L., Marlow, S. L., Joseph, D. L., & Salas, E. (2017). Leadership training design, delivery and implementation.
Journal of Applied Psychology,102, 1686–1718 https://doi.org/10.1037/apl0000241
*Lane, C., & Rollnick, S. (2007). e use of simulated patients and role-play in communication skills training: A review of the literature
to August 2005. Patient Education and Counseling,67, 13 –20. https://doi.org/10.1016/j.pec.2007.02.011
Lindqvist, E., & Vestman, R. (2011). e labor market returns to cognitive and noncognitive ability: Evidence from the Swedish enlist-
ment. American Economical Journal: Applied Economics,3, 101– 128. https://doi.org/10.1257/app.3.1.101
*Lipsey, M. W., & Wilson, D. B. (1993). e ecacy of psychological, educational, and behavioral treatment: Conrmation from meta-
analysis. American Psychologist ,48, 1181 –1209. https://doi.org/10.1037/0003-066X.48.12.1181
McArdle, J. J. (2009). Latent variable modeling of dierences and changes with longitudinal data. Annual Review of Psychology,60,
577–605. https://doi.org/10.1146/annurev.psych.60.110707.163612
McCrae, R. R., & Costa, P. T., Jr. (2008). Empirical and theoretical status of the Five-Factor Model of personality traits. In G. Boyle,
G. Matthews, & D. Saklofske (Eds.), Sage handbook of personality theory and assessment (Vol. 1, pp. 273–294). Sage Publications.
https://doi.org/10.4135/9781849200462.n13
McCrae, R. R., & John, O. P. (1992). An introduction to the ve-factor model and its applications. Journal of Personality,60, 175-215.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 21

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
*McCulloch, P., Rathbone, J., & Catchpole, K. (2011). Interventions to improve teamwork and communications among healthcare sta.
British Journal of Surgery,98, 469 – 479. https://doi.org/10.1002/bjs.7434
Moher, D., Liberati, A., Tetzla, J., Altman, D. G., & e PRISMA Group. (2009). Preferred reporting items for systematic reviews and
meta-analyses: e PRISMA Statement. PLoS Medicine,6, 1–6. https://doi.org/10.1371/journal.pmed.1000097
Parkinson, B., & Totterdell, P. (1999). Classifying aect regulation strategies. Cognition and Emotion,13, 277– 303. https://doi.org/10
.1080/026999399379285
Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2015). An assessment of the magnitude of eect sizes: Evidence from
30 years of meta-analysis in management. Journal of Leadership & Organizational Studies,23, 66 –81. https://doi.org/10.1177/
1548051815614321
Paulhus, D. L., & Williams, K. M. (2002). e dark triad of personality: Narcissism, Machiavellianism, and psychopathy. Journal of
Research in Personality,8, 556– 563.
Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys,3, 96 –146. https://doi.org/10.1214/09-SS057
*Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century.
e National Academies Press.
*Robbins, S. B., Oh, I.-S., Le, H., & Button, C. (2009). Intervention eects on college performance and retention as mediated by moti-
vational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology,94, 1163 –1184.
https://doi.org/10.1037/a0015738
Roberts, B. W., Hill, P. L., & Davis, J. P. (2017). How to change conscientiousness: e sociogenomic trait intervention model. Personality
Disorders: eory, Research, and Treatment,8, 199–205. https://doi.org/10.1037/per0000242
Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through
intervention. Psychological Bulletin,143, 117– 141. https://doi.org/10.1037/bul0000088
Roberts, B. W., Walton, K. E., & Viechtbauer, W. (2006). Patterns of mean-level change in personality traits across the life course: A
meta-analysis of longitudinal studies. Psychological Bulletin,132, 1 –25. .
Rummel, A., & Feinberg, R. (1988). Cognitive evaluation theory: A meta-analytic review of the literature. Social Behavior and Person-
ality: An International Journal,16, 147– 164.
*Ryan, R. M., & Deci, E. L. (1996). When paradigms clash: Comments on Cameron and Pierce’s claim that rewards do not undermine
intrinsic motivation. Review of Educational Research,66, 33– 38. https://doi.org/10.3102/00346543066001033
*Salas, E., DiazGanados, D., Weaver, S. J., & King, H. (2008). Does team training work? Principles for health care. Academic Emergency
Medicine,15, 1002 –1009. https://doi.org/10.1111/j.1553-2712.2008.00254.x
*Salas, E., Nichols, D. R., & Driskell, J. E. (2007). Testing three team training strategies in intact teams a meta-analysis. Small Group
Research,38, 471– 488. https://doi.org/10.1177/1046496407304332
Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does crew resource management training work? An update, an
extension, and some critical needs. Human Factors,48, 392 –412. https://doi.org/10.1518/001872006777724444
Schmidt, F. L., & Hunter, J. E. (1998). e validity and utility of selection methods in personnel psychology: Practical and theoretical
implications of 85 years of research ndings. Psychological Bulletin,124, 262 – 274. https://doi.org/10.1037/0033-2909.124.2.262
Scott, G., Leritz, L. E., & Mumford, M. D. (2004). e eectiveness of creativity training: A quantitative review. Creativity Research
Journal,16, 361 –388. https://doi.org/10.1080/10400410409534549
Sheeran, P., Harris, P. R., & Epton, T. (2014). Does heightening risk appraisals change People’s intentions and behavior? A meta-analysis
of experimental studies. Psychological Bulletin,140, 511– 543. https://doi.org/10.1037/a0033065
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training eective? Psychological Bulletin,138, 628–654. https://
doi.org/10.1037/a0027473
*Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What
we know and where we need to go. Psychological Bulletin,137, 421 –442. https://doi.org/10.1037/a0022777
Society of Human Resource Management. (2019, February 5). e global skills shortage: Bridging the talent gap with education, training
and sourcing. https://www.shrm.org/hr-today/trends-and-forecasting/research-and- surveys/Documents/SHRM%20Skills%20Gap
%202019.pdf
*Teding van Berkhout, E., & Malou, J. M. (2016). e ecacy of empathy training: A meta-analysis of randomized controlled trials.
Journal of Counseling Psychology,63, 32–41. https://doi.org/10.1037/cou0000093
*Vanhove, A. J., Herian, M. N., Perez, A. L. U., Harms, P. D., & Lester, P. B. (2016). Can resilience be developed at work? A meta-analytic
review of resilience-building programme eectiveness. Journal of Occupational and Organizational Psychology,89, 278–307. https://
doi.org/10.1111/joop.12123
Viswesvaran, C., & Ones, D. S. (2000). Measurement error in “Big Five factors” personality assessment: Reliability generalization across
studies and measures. Educational and Psychological Measurement,60, 224–235. https://doi.org/10.1177/00131640021970475
*Weaver, S. J., Dy, S. M., & Rosen, M. A. (2014). Team-training in healthcare: A narrative synthesis of the literature. BMJ Quality &
Safety,23, 359– 372. https://doi.org/10.1136/bmjqs-2013-001848
22 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service

M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Weaver, S. J., Lyons, R., DiazGranados, D., Rosen, M. A., Salas, E., Oglesby, J., …King, H. B. (2010). e anatomy of health
care team training and the state of practice: A critical review. Academic Medicine,85, 1746–1760. https://doi.org/10.1097/ACM
.0b013e3181f2e907
*Webb, T. L., Miles, E., & Sheeran, P. (2012). Dealing with feeling: A meta-analysis of the eectiveness of strategies derived from the
process model of emotion regulation. Psychological Bulletin,138, 775 – 808. https://doi.org/10.1037/a0027600
Suggested citation:
Martin Raugh, M. P., Williams, K. M., & Lentini, J. (2020). e malleability of workplace-relevant noncognitive constructs: Empirical
evidence from 39 meta-analyses and reviews (Research Report No.RR-20-23). Educational Testing Service. https://doi.org/10.1002/
ets2.12306
Action Editor: Jessie Sparks
Reviewers:Sam Rikoon and Katrina Roohr
ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS).All other trademarks are property of their
respective owners.
Find other ETS-published reports by searching the ETS ReSEARCHER database at http://search.ets.org/researcher/
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 23