ArticlePDF Available

The Malleability of Workplace‐Relevant Noncognitive Constructs: Empirical Evidence From 39 Meta‐Analyses and Reviews


Abstract and Figures

We reviewed the current state of the literature on the intervention‐based development of interpersonal skills (e.g., teamwork, leadership) and intrapersonal skills (e.g., personality, motivation, etc.) relevant to success in workplace contexts. We adopted a multidisciplinary approach to our review, evaluating research from 39 reviews and meta‐analyses from several fields such as educational psychology, industrial/organizational (I/O) psychology, medicine, and personality psychology, among others, to examine the extent to which noncognitive constructs change as a result of intervention. We discuss key findings and trends and conclude by identifying gaps in the literature and directions for future research. Overall, findings suggest optimism regarding the malleability of noncognitive constructs.
Content may be subject to copyright.
h R
The Malleability of
Noncognitive Constructs:
Empirical Evidence From 39
Meta-Analyses and Reviews
ETS RR20-23
Michelle P. Martin-Raugh
Kevin M. Williams
Jennifer Lentini
September 2020
ETS Research Report Series
John Mazzeo
Distinguished Presidential Appointee
Beata Beigman Klebanov
Senior Research Scientist
Heather Buzick
Senior Research Scientist
Tim Davey
Research Director
John Davis
Research Scientist
Keelan Evanini
Research Director
Marna Golub-Smith
Principal Psychometrician
Priya Kannan
Managing Research Scientist
Irv Katz
Senior Research Director
Sooyeon Kim
Principal Psychometrician
Anastassia Loukina
Senior Research Scientist
Gautam Puhan
Psychometric Director
Jonathan Schmidgall
Research Scientist
Jesse Sparks
Research Scientist
Michael Walker
Distinguished Presidential Appointee
Rebecca Zwick
Distinguished Presidential Appointee
Kim Fryer
Manager, Editing Services
Ayleen Gontz
Senior Editor
Since its 1947 founding, ETS has conducted and disseminated scientic research to support its products and services, and
to advance the measurement and education elds. In keeping with these goals, ETS is committed to making its research
freely available to the professional community and to the general public. Published accounts of ETS research, including
papers in the ETS Research Report series, undergo a formal peer-review process by ETS sta to ensure that they meet
established scientic and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that
outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions
expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and
not necessarily those of the Ocers and Trustees of Educational Testing Service.
e Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 ser ved the Research and
Development division as Editor for the ETS Research Report series. e Eignor Editorship has been created to recognize
the pivotal leadership role that Dr. Eignor played in the research publication process at ETS.
ETS Research Report SeriesISSN 2330-8516
The Malleability of Workplace-Relevant Noncognitive
Constructs: Empirical Evidence From 39 Meta-Analyses
and Reviews
Michelle P. Martin-Raugh, Kevin M. Williams, & Jennifer Lentini
Educational Testing Service, Princeton, NJ
We reviewed the current state of the literature on the intervention-based development of interpersonal skills (e.g., teamwork, lead-
ership) and intrapersonal skills (e.g., personality, motivation, etc.) relevant to success in workplace contexts. We adopted a multi-
disciplinary approach to our review, evaluating research from 39 reviews and meta-analyses from several elds such as educational
psychology, industrial/organizational (I/O) psychology, medicine, and personality psychology, among others, to examine the extent
to which noncognitive constructs change as a result of intervention. We discuss key ndings and trends and conclude by identifying
gaps in the literature and directions for future research. Overall, ndings suggest optimism regarding the malleability of noncognitive
Keywords Noncognitive skills; malleability; workplace
Researchers, organizations, and educational institutions have become increasingly interested in the malleability and devel-
opment of constructs outside of traditional conceptualizations of intelligence (Heckman & Kautz, 2014; Kyllonen, 2012).
One approach to identifying these constructs involves the distinction between those typically considered “cognitive” from
those usually categorized as “noncognitive.” Whereas cognitive ability and cognitive skills are oen equated with tradi-
tional measures of intelligence and the ability to solve abstract problems (Gottfredson, 1998), noncognitive constructs
are conceptualized as those that are separate from and considered largely orthogonal to cognitive ability. Scholars have
dened noncognitive constructs in several ways. Klieger et al. (2015) described noncognitive constructs as “demonstra-
ble personality, motivational, attitudinal, self-regulatory, and learning approach constructs for which there are dierences
among people, which standardized tests of cognitive ability are not primarily designed to measure, and the behavioral
expression of which is considered useful” (p. 3). is denition overlaps with those proposed by Kyllonen (2012, pp. 7– 8),
Kautz et al. (2014, p. 2), and Duckworth and Yeager (2015, p. 239), suggesting a comprehensive scope of applicable con-
structs, such as personality traits, social skills, motivation, teamwork, and leadership skills, among others. However, it is
worth noting that the line between cognitive and noncognitiveconstructs may be blurry (e.g., Duckworth & Yeager, 2015;
Kell, 2018): Many constructs characterized as being largely noncognitive in nature actually contain cognitive elements, as
Noncognitive constructs are important predictors of a variety of workplace and economic outcomes (Barrick &
Mount, 1991; Lindqvist & Vestman, 2011) and explain meaningful variability even when cognitive ability is taken into
account (Schmidt & Hunter, 1998). In these contexts, noncognitive constructs are oen referred to as 21st century skills
or so skills (e.g., Heckman & Kautz, 2012). Despite the importance of these constructs, employers in today’s economic
climate perceive a widening gap between the noncognitive constructs they expect prospective employees to possess and
the level of skill actually exhibited by job candidates (Capelli, 2012; Society of Human Resource Management, 2019).
Although cognitive ability remains largely stable over time and tends to be somewhat xed (Jensen, 1998), noncognitive
constructs are thought to be more malleable (Aguinis & Kraiger, 2009; Heckman & Kautz, 2012). ese factors combine
to generate great interest in workplace domains (e.g., Heckman & Rubinstein, 2001) in exploring the extent to which
noncognitive constructs change through intervention. In 2018, organizations in the United States spent $87.6 billion on
Corresponding author: M. P. Martin-Raugh, E-mail:
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 1
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 1 Noncognitive Construct Denitions
Construct Denition
Personality Includes the Big Five, a dominant typology of personality traits including agreeableness, openness to
experience, conscientiousness, extraversion, and emotional stability, along with personality models and
traits that typically predate the Big Five
Social skills Broad set of interpersonal skills that facilitate interaction with others
Communication e ability to use both oral and written messages perceived as appropriate and eective in workplace
Leadership e ability to persuade followers to support and execute solutions
Teamwork Interrelated set of thoughts, behaviors, and feelings needed for a team to function as a unit
Attitudes Internal state that inuences an individual’s choice of personal action
Self-concept e manner in which individuals perceive and evaluate themselves
Emotion Encompasses emotional state, emotional regulation, positive and negative aect, and empathy
Motivation Intrinsic and extrinsic forces that impact work-related behavior in terms of form, direction, intensity, and
interventions to foster employee learning and development (Freifeld, 2018), further underscoring the need for research
in this area.
In this review, using the denition of noncognitive constructs provided by Klieger et al. (2015), we systematically exam-
ine research focusing on the malleability of interpersonal skills and intrapersonal skills1relevant to workplace success and
the mechanisms through which these constructs change (see Table 1). We focus on social skills, communication skills,
leadership, and teamwork as part of our review of interpersonal skills. We consider intrapersonal skills to include person-
ality traits, attitudes, self-concept, emotion, and motivation. Although several comprehensive and informative reviews
of the workplace training literature exist (e.g., Aguinis & Kraiger, 2009; Baldwin & Ford, 1988; Bell et al., 2017; Blume
et al., 2010), the organization and foci of these prior reviews dier from our approach. Prior reviews have examined
training practices more generally, targeting aspects of approaches or methodologies rather than focusing on the develop-
ment of particular constructs (e.g., Aguinis & Kraiger, 2009; Baldwin & Ford, 1988; Bell et al., 2017; Blume et al., 2010).
Other reviews have concentrated on the enrichment of constructs that could be considered mainly cognitive (e.g., Scott
et al., 2004; Shipstead et al., 2012) orprimarily on job-specic skills (e.g., Salas et al., 2006; Weaver et al., 2010). We largely
draw on meta-analyses and reviews conducted in a diverse constellation of elds to provide a comprehensive review of
the malleability of noncognitive constructs.
We should a l s o c l a r if y ou r u se o f t he t e r m malleability. We use this term to refer to longitudinal change in con-
structs resulting from deliberate intervention. is denition aligns with our goal of summarizing research relevant
to employers’ explicit intervention eorts in service of improving workplace-relevant noncognitive constructs. Rather
than direct improvements in construct scores, employers may be more interested in facilitating various desirable work-
place outcomes such as increases in productivity, prots, employee retention, and employee satisfaction, among others.
Regardless, because this research is intended to inform overt training and development eorts, we argue that research
observing noncognitive constructs’ longitudinal change in the absence of intervention (e.g., Viswesvaran & Ones, 2000)
or in response to natural life events (e.g., Bleidorn et al., 2018) is not relevant. Moreover, these naturalistic studies should
not be viewed as support for these constructs’ amenability to eortful intervention. On the contrary, it is possible that
these results highlight innate maturational processes or environmental factors that deliberate interventions must strive to
overcome (see Briley & Tucker-Drob, 2014). Instead, the most signicant conclusion that may be drawn from longitudi-
nal studies that exclude eortful interventions is that they emphasize the value of non-intervention control groups in the
study of deliberate interventions.
Our review aims to answer the following research questions (RQs):
1. To what extent do workplace-relevant noncognitive constructs change as a result of intervention?
2. What are the mechanisms that eectively drive change?
We organize the existing literature on the malleability of various noncognitive constructs into two broad sections
interpersonal and intrapersonal skills— within which we present evidence and evaluate research addressing each of the
two RQs. We conclude each section by providing a synthesis of the literature and recommendations for future research.
2ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
We include articles within a noncognitive construct section if the malleability of that construct was examined in the
article, regardless of whether the construct was a specic focus of the intervention. We conclude by noting the limitations
associated with our study and identifying viable directions for future research.
Literature Search
e goal of this study was to evaluate meta-analyses and narrative reviews relevant to our RQs. is type of evidence
was selected because of its well-documented advantages over primary studies, which has been argued elsewhere (e.g.,
Card, 2012). To be as inclusive as possible, we used Google Scholar as our primary search engine. A Boolean search
strategy combining three sets of search terms (Figure 2) was used. First, a set of noncognitive construct terms (e.g., Big Five,
leadership) was used to dene the construct of interest. Second, various dynamic process terms (e.g., development,training)
were entered to identify articles discussing the malleability of these noncognitive constructs that might be transportable
to the workplace. Finally, the terms meta-analysis and review were entered separately. For example, a full search string was
Big Five” +development +“meta-analysis.” Each possible combination of the three sets of search terms was used. It is
worth noting that our list of search terms may not have been exhaustive. No publication date restrictions were imposed
on the search, which ended in February 2017.
Despite eorts to narrow the focus of the search using a Boolean strategy, an overwhelming amount of literature was
identied. For example, the string personality +training +meta-analysis returned 1,100,000 results. erefore, searches
within each string were terminated aer reaching (a) 30 consecutive results that were either irrelevant or already captured
in another search or (b) the rst 200 results, whichever occurred rst. Using the default method of sorting results by
manageable volume. Finally, this was augmented by a partial backward snowball search, in which the reference sections
of the articles identied through the initial search are reviewed for additional articles.
Inclusion Criteria
Basic inclusion criteria required that the article must be written in English; be published in a book, dissertation, or peer-
reviewed journal article; and consist of a meta-analysis or narrative review, or a relevant reply article to one of these
meta-analyses or reviews. Relevance criteria required that the noncognitive constructs, participants, interventions, or
outcomes included in the article could be reasonably expected to translate to the workplace. Methodological criteria
required that the article evaluate eortful changes in the noncognitive construct using a longitudinal or experimental
design. erefore, meta-analyses and reviews focusing on the predictive validity of noncognitive constructs but not the
eortful malleability of these constructs (e.g., Barrick & Mount, 1991) were excluded. Similarly, meta-analyses and reviews
examining longitudinal stability and change of noncognitive constructs in the absence of intervention (e.g., Viswesvaran
& Ones, 2000) or in response to natural life events (e.g., Bleidorn et al., 2018) were excluded. Furthermore, noncognitive
construct change must be measured at the level of the individual participant as opposed to individuals who were not the
target of the intervention.
Articles were screened by the three authors in two phases. First, each article title and abstract were reviewed to ensure
they met our inclusion criteria. Next, the full text of the article was reviewed for the same criteria. Prior to each phase,
10 calibration articles were selected for each of the three authors to examine independently. Disagreements regarding
the eligibility of these articles were resolved by discussion. e remaining articles were divided among the three authors.
At the full-text screening stage, if a primary reviewer rejected an article, a second reviewer independently reviewed the
article to conrm or reject this decision. If the secondary reviewer disagreed with the primary reviewer, the third reviewer
determined the article’s eligibility. Of the rejected articles, exact agreement between the rst two reviewers was 88.5%.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 3
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Figure 1 PRISMA diagram of literature search results.
Aer removing duplicates, 10,006 articles were identied. Of these, 354 unique articles passed the initial title and abstract
screening phase, with 39 articles passing the nal full-text screening phase (Figure 1; Moher et al., 2009). Common ratio-
nales for rejections included low relevance to the workforce (e.g., clinical interventions; Roberts, Hill, & Davis, 2017),
failing to provide a meta-analysis or review, or a lack of discussion about the malleability of the construct(s). It should
be noted that in some cases a source article pertained to more than one noncognitive construct. Publication dates ranged
from 1941 to 2017, with the majority published aer 2000. Meta-analytic eect sizes indexing construct malleability as a
result of intervention are displayed in Table 2.
Results: QuantifyingNoncognitive Construct Change
Before describing specic noncognitive construct malleability research, it is necessary to clarify the means by which
changes are evaluated. Rank-order consistency and mean-level dierences are the two approaches used most oen. Rank-
order consistency (or rank-order stability) typically involves test– retest correlations between a group’s noncognitive scores
from an initial assessment with that same group’s scores at a subsequent assessment, following a signicant period of time
(e.g., 1 year later). ese results summarize the longitudinal stability of individuals’ rank ordering on the construct of
interest, such as whether the most extraverted individuals at Time 1 remain the most extraverted at Time 2, for instance.
Alternatively, mean-level dierences compare mean noncognitive assessment scores (a) either within the same group
prior to and following an intervention, (b) postintervention scores between two groups such as an intervention and con-
trol group, or (c) pre-postintervention dierences between two groups. Rank-order consistency and mean-level change
may be considered complementary methods: Rank-order consistency does not evaluate the degree to which the entire
sample’s level of the noncognitive construct in question has changed over time, whereas mean-level changes may obscure
4ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Figure 2 Literature Review Search Terms.
more nuanced results such as stability dierences across subgroups. In noncognitive construct research, rank-order meth-
change research. e majority of meta-analyses and reviews we describe used one of these two methods to evaluate eort-
ful longitudinal malleability (Table 2). ese eect sizes are typically interpreted using traditional eect size guidelines
(i.e., Cohen, 1988).
Results: Interpersonal Skills
Social Skills
RQ1: Malleability
Cheraghi-Sohi and Bower’s (2008) review of patient feedback on physicians’ social skills found limited evidence that
social skills can be improved through the interventions examined in their review. Conversely, separate meta-analyses have
reported medium to large eect size improvements in social skills following organizational training (Arthur Jr. et al., 2003)
or social skills training (Klein, 2009). Only six studies focused on social skills more broadly (see Table 2).
RQ2: Mechanisms of Change
Several sources explored the ecacy of social skills interventions (e.g., Klein, 2009; Robbins et al., 2009). Klein (2009)
suggested that social skills training can occur via traditional methods including motivating and goal setting, coaching
and mentoring, feedback, behavioral modeling training (BMT), multimedia and simulation-based training, and team
training. Pellegrino and Hilton’s (2012) review of social skills argued that deeper learning, where a person can take what
was learned in one situation and can apply it to new situations, allows trainees to eectively transfer what was learned
to new situations. ese authors argued that deep learning interventions should provide clear and discrete learning goals
learning (PBL) approaches that present learners with extended problems that can engage learners while providing helpful
feedback and guidance. ey argued that PBL can encourage elaboration, questioning, and self-explanation.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 5
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Noncognitive Construct Malleability Meta-Analyses
Article k N
Ye a r s o f
vs. longitudinal) Intervention
measure Eect size Conclusion
Social skills
Arthur Jr. et al., 2003* 397 33,325 1960 –2000 Working adults Experimental Organizational training
focusing on
interpersonal skills
Group mean
dierences (d)
d=0.62 Interpersonal skills
demonstrate a medium
eect size change
through intervention
when assessed by the
Learning and
Behavioral criteria, and
a large eect size when
evaluated by the
Results criterion
Klein, 2009* 141 NR 1971– 2008 Adults Longitudinal Interpersonal skills
training (includes
mentoring, behavioral
modeling, feedback,
goal setting, team
Pre-post and/or
group mean
dierences (r)
r=.47 Interpersonal skills
training is moderately
eective in improving
general interpersonal
Robbins et al., 2009* 107 11,183 1975 –2009 College students Experimental Self-management,
ρ=.15 College interventions
yielded a small eect
on social control
Communication skills
Barth & Lannen, 2011* 13 1,137 1988– 2008 Healthcare
working with
cancer patients
Experimental Communication skills
training courses in
Group mean
d=0.50 Communication skills
moderate improvement
through intervention
Leadership skills
Avolio et al., 2009* 132 11,552 NR Adults Experimental Leadership training Group mean
dierences (d)
d=0.67 Leadership training
interventions showed
an overall medium
eect size
6ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
vs. longitudinal) Intervention
measure Eect size Conclusion
Lacerenza et al., 2017* 335 26,573 1951– 2014 Adults Experimental Leadership training Pre-post mean
dierences (d)
d=0.73 Leadership training is
more eective than
previously thought
Team w ork sk i l l s
Salas et al., 2007* 7 695 1975 –2000 Non-clinical adults Experimental Team training Pre-post and/or
group mean
dierences (r)
r=.29 Team training was shown
to have a small to
moderate, positive
eect on team
Salas, Diaz Granados,
Klein, et al., 2008*
45 2,650 1962–2008 Non-clinical adults Experimental Team training Pre-post and/or
group mean
dierences (ρ)
ρ=.34 Team training was shown
to have a moderate,
positive eect on team
Lipsey & Wilson, 1993 156 >1,000,000 1979– 1992 Clinical and
non-clinical youth
and adults
Experimental Various mental health,
I/O, and
mean dierence
d=0.47 Collapsed across a wide
range of outcomes
(including personality),
interventions are
generally eective in
changing psychological
Vanhove et al., 2016 42 16,348 1979 –2014 Workplace samples Both Workplace programs
psychosocial factors
(between- or
d=0.21 e eects of
programs are small and
do not endure
Klein et al., 2009* 20 1,562 1950–2007 Non-clinical adults Experimental Team-building Pre-post and/or
group mean
dierences (r)
r=.31 Team-building
interventions can have
team outcomes, being
more eective for
process and aective
outcomes, and least
eective for cognitive
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 7
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
vs. longitudinal) Intervention
measure Eect size Conclusion
Bangert-Drowns, 1988 33 NR 1968 1986 Traditional
through college
Experimental School-based
substance abuse
Group dierences
d=0.34aInterventions can reduce
pro-alcohol- and
drug-related attitudes
Bruvold & Rundall, 1988 NR NR 1972–1984 Students Experimental Alcohol and
Group dierences
in pre-post
changes (d)
NRbInterventions can reduce
pro-alcohol attitudes
Brecklin & Forde, 2001 45 NR 1861– 1999 College students Both Rape education
Group dierences
or pre-post
changes (d)
d=0.35 Interventions can reduce
rape-supportive attitudes
Anderson &
Whiston, 2005
69 18,172 1978 –2002 College students Experimental Sexual assault
Group dierences
NRbInterventions can reduce rape
attitudes, rape-related
attitudes, and behavioral
intent, but not rape
Barth & Lannen, 2011 13 1,137 1988–2008 Oncology health
Experimental Communication
skills training
Group dierences
d=0.35 Interventions can improve
attitudes toward terminally
ill patients, death, and
Kalinoski et al., 2013 65 8,465 1977 2011 High school
students, and
employed adults
Both Diversity training Group dierences
δ=0.33 Overall, diversity training had
positive small-to-moderate
eects on aective- (e.g.,
attitudes), cognitive-, and
skill-based outcomes
Beelmann &
Heinemann, 2014
81 NR 1958 2010 Youth (<18 years) Exp erimental Programs
designed to
prejudice or
Group dierences
d=0.30 Interventions can reduce
prejudice and improve
intergroup attitudes toward
disabled individuals, and
the elderly
8ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
vs. longitudinal) Intervention
measure Eect size Conclusion
Bezrukova et al., 2016 260 29,407 1972 –2013 Adults Both Diversity training Group dierences
g=0.38 With some exceptions,
diversity training
generally leads to
positive outcomes,
though less
pronounced for
attitudinal change
Jones, 2016 10 592 1993 –2012 Employed adults Both Workplace coaching Group dierences
or pre-post
changes (δ)
δ=0.51 Interventions can
improve work attitudes
Bowen & Neill, 2013 137 NR 1960–2012 At risk, clinical, or
Longitudinal Adventure therapy Pre-post
(Hedge’s g)
g=0.43 Adventure therapy can
improve self-concept
Augusti ne &
Hemenover, 2009
34 2,958 1887 2007 Non-clinical
Longitudinal Aect repair strategies
(Behavioral or
cognitive; avoidance or
Pre-post mean
d=0.45 Aect regulation
interventions can
increase positive aect
and decrease negative
Delise et al., 2010 21 1,413 1985–2008 Military; civilian Both Team training Pre-post and/or
group mean
d=0.85 Aect - in the context of
team eectiveness —is
amenable to change
through intervention
Webb et al., 2012 190 >13,655 19772010 Nonclinical;
mostly adults
Both Attentional deployment,
cognitive change, or
response modulation
Pre-post and/or
group mean
NRbe eectiveness of
emotion regulation
interventions depends
on the specic strategy
Sheeran et al., 2014 208 52,976 1953– 2010 Adults, students,
Experimental Practices that heighten
risk appraisal
Group mean
dierences (d)
d+=0.70 Both anticipatory (e.g.,
fear, worry) and
anticipated emotions
(e.g., regret, guilt) may
be increased by small
eect sizes by
heightening risk
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 9
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tabl e 2 Continued
Article k N
Ye a r s o f
vs. longitudinal) Intervention
measure Eect size Conclusion
Tedi n g v a n B e rkho u t &
Malou, 2016
18 1,018 1973– 2014 University students;
patients; other
adults; youth
Experimental Empathy training Group mean
g=0.53 Empathy, which includes
aective components, can
be improved through
Rummel & Feinberg, 1988 45 NR 1971 –1985 Youth, students, and
Experimental Extrinsic rewards Group mean
dierences (d)
d=0.33 Extrinsic rewards had a
detrimental eect on
intrinsic motivation
Cameron & Pierce, 1994 96 NR 1971 1991 Youth and adults Experimental Rewards Group mean
dierences (d)
d=0.14 Results suggest that reward
does not decrease intrinsic
Deci et al., 1999 101 NR 1971 –1997 Youth and college
Experimental Reward Free-choice
Group mean
dierences (d)
d=−0.24 Extrinsic rewards
signicantly undermined
free-choice intrinsic
motivation. Positive
feedback improved
free-choice behavior
Sitzmann & Ely, 2011 430 90,380 1989 2011 Adults Experimental Self-regulatory
processes (Goal
level, persistence,
eort, self-ecacy)
Learning NR Self-regulation enables
individuals to monitor
their goal-directed
activities over time; goal
level, persistence, and eort
have small to medium
eects on learning
Jones, 2016 17 2,267 1993– 2012 Adults Experimental Coaching Organizational
Pre-post and/or
group mean
dierences (δ)
δ=0.36 Workplace coaching yields a
small to moderate impact
on eectiveness overall
Note.*=meta-analyses; NA =not applicable; NR =not reported. Articles listed in italics do not address longitudinal change of noncognitive skills but are otherwise relevant to our research questions.
aEect size is reported for attitude outcomes; authors also reported eect sizes for knowledge (d=0.76) and behavior outcomes (d=0.12). bAuthors reported multiple eect sizes based on factors such as
intervention type, duration, and outcome, but no omnibus estimate.
10 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Communication Skills
RQ1: Malleability
e majority of articles on communication skills we reviewed focused on health-care professionals (see Table 1). Commu-
nication between physicians and patients is a critical part of any treatment plan. However, physicians’ poor communication
skills represent a common complaint for patients and undermine health-care ecacy (Hulsman et al., 1999). In a review
of 14 studies of communication training for physicians, Hulsman et al. (1999) reported mixed results. Positive eects of
training occurred in fewer than half of the studies, and the studies that did report positive outcomes generally used sub-
optimal research designs (e.g., no control group). However, this study may not be representative of the eld as a whole, as
other systematic reviews and meta-analyses have reported improvements in communication skills through interventions
such as team training (Gillespie et al., 2010), simulated patients and roleplays (as opposed to didactic training; Lane &
Rollnick, 2007), and communication skills courses (Barth & Lannen, 2011).
RQ2: Mechanisms of Change
Lane and Rollnick (2007) stated in their review of the literature that the use of simulated patients, typically actors play-
ing the role of a patient, is a widespread, eective practice in health-care communication training. Identical patients
allow for standardization, experimentation with dierent communication skills, repeated training, and ongoing feedback.
Drawbacks include expense, the need for actor selection and training, and the risk of simulated patients going o-script.
Role-play with fellow trainees is also an eective communication training intervention, as it has benets and drawbacks
similar to simulated patients. Furthermore, both of these training methods were found to be more eective than didactic
training methods, such as lectures. Gillespie et al.’ (2010) found that communication skills training in the context of team
training improved health outcomes for patients. Interventions studied included checklists, simulations, and debriengs.
However, as this study did not compare the ecacy of various team training methods for communication, the relative
merits of each are still unknown. Barth and Lannen (2011) examined the eects of communication skills training courses
on communication behaviors, and found that courses lasting over 36 hours outperformed courses lasting less than 24 h,
although this eect was small.
RQ1: Malleability
e three meta-analyses reviewed here provide some evidence to suggest that leadership skills are indeed amenable to
change through intervention (see Table 2). Avolio et al. (2009) concluded in their meta-analysis of the development of
leadership skills that leadership training interventions yielded a medium eect size in producing positive change in lead-
ership skills. A follow-up study (Avolio et al., 2010) showed that leadership interventions yielded a wide range of return
on development investment, with some estimates as high as 200%. Moreover, data from a recent meta-analysis (Lac-
erenza et al., 2017) supported the notion that leadership training is largely eective, resulting in medium-to-large eect
size improvement gaged by reactions, learning, transfer, and results criteria (i.e., Kirkpatrick, 1996).
RQ2: Mechanisms of Change
Avolio et al. (2009) found in their meta-analysis of leadership development eorts that there was largely no dierence in
eectiveness between interventions using leader training and development methods and those using a dierent method,
such as a scenario, actor or role-play, or assigned leader. ey also reported that more eective interventions were based on
Pygmalion theory, which posits that when leaders hold positive, high expectations for those they are leading, performance
improves. ese interventions yielded larger eects than those based on traditional or newer leadership theories, such
as charismatic or transformational leadership theories, which produced medium eects. Lacerenza et al.’s (2017) meta-
analysis concluded that leadership interventions using practice-based methods were more eective than other delivery
methods and that programs incorporating multiple methods of delivery (e.g., information-based, demonstration-based)
are signicantly more eective. us, they recommended that training programs use multiple delivery methods when
possible, and only practice-based delivery when only one approach may be used.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 11
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Tea mwork
RQ1: Malleability
e eight teamwork skills studies we reviewed concluded that these skills are amenable to change through organiza-
tional interventions (e.g., Gordon et al., 2012; Weaver et al., 2014; see Table 2. ese studies typically incorporated out-
comes closely related to or demonstrations of teamwork rather than direct changes in an assessment that conceptualized
teamwork as a distinct noncognitive construct. For instance, Klein et al.’s (2009) meta-analysis found that team-building
interventions had a more pronounced eect for process (e.g., communication, coordination) and aective outcomes (e.g.,
trust) than for cognitive outcomes (e.g., knowledge). McCulloch et al. (2011) reviewed teamwork interventions specic
to health care and showed that team training programs can lead to improved sta attitudes and teamwork. However, the
authors concluded the eectiveness of these programs on safety culture and patient outcomes is questionable, with few
studies reporting outcomes. Studies that did explore outcomes found small eects.
RQ2: Mechanisms of Change
One eld that has taken particular interest in the ecacy of teamwork skills interventions is health care (e.g., Buljac-
Samardzic et al., 2010). Gordon et al. (2012) conducted a literature review of nontechnical skills interventions, including
teamwork, leadership, and communication, in health-care settings. ey noted that interventions primarily included sim-
ulations or role-plays, with an emphasis on debrieng, feedback, and simulation delity. McCulloch et al. (2011) reviewed
teamwork interventions for health-care sta and noted that most interventions were based on crew resource management
training adapted from the aviation eld. ey reported that literature in this area is lacking, as little detail is provided on
the specic components of training and reporting of outcome data is poor, with little statistical data provided.
In Weaver et al.’s (2014) review examining health-care team training evaluations, the authors suggested team train-
ing primarily targets communication, situational awareness, leadership, and role clarity. Training methods included
information-based methods, demonstration-based methods, and practice-based methods. Salas et al. (2008) qualitatively
examined team training studies and concluded that training is eective in both aviation and health care. ey stressed
that hands-on learning via simulationsis important and suggested that when high delity simulators are used, it is critical
that they be realistic. ey also state that behavior-based feedback is important, as it helps trainees determine where and
how trainees can improve.
Several other studies expanded in scope to focus on the development of teamwork skills in other domains. Klein
et al.’s (2009) meta-analysis showed that team building is intended to improve interpersonal relations and social interac-
tions, achieve results, and accomplish tasks. Team-building interventions were moderately eective regardless of strategy,
but the role-clarication component of interventions was slightly more eective than goal setting, interpersonal relations,
or problem solving components. Salas et al. (2007) conducted a meta-analysis of the eects of three dierent components
of team training on the eectiveness of team-training interventions. Results demonstrated that focusing on coordination
and adaptation in team training resulted in greater improvements in team eectiveness as opposed to cross-training and
team self-correction training.
Summary and Recommendations: Interpersonal Skills
To summarize, evidence has suggested interpersonal skills may be improved via intervention. Programs targeting these
skills are generally as eective as those targeting cognitive skills (Aguinis & Kraiger, 2009). However, not all studies
concluded that training interpersonal skills results in substantial improvement. Possible strategies for honing adult inter-
personal skills include motivating and goal setting, coaching and mentoring, role-plays, feedback, BMT, multimedia and
simulation-based training, team training, deep learning, and PBL.
Within the social skills domain, many studies reviewed have gathered support for the eectiveness of using interven-
tions to improve specic constructs. Communication skills interventions include debriengs, checklists, simulations,
instruction, modeling, skill practice, feedback, cognitive and experiential learning, simulated patients, and role-play
exercises. Although based on only three sources examining leadership interventions, the consensus across all of the
meta-analytic evidence reviewed is that leaders are likely made, not born. However, Avolio et al. (2009) reported there
was largely no dierence in the eectiveness of dierent intervention methods, with the exception that interventions
12 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
based on Pygmalion theory were more eective than those based on traditional or newer leadership theories. Overall,
teamwork skills appear to be amenable to moderate change through intervention. Training methods for developing
teamwork skills may take the form of simulations or role-plays, information-based methods, demonstration-based
methods, practice-based methods, and small-group learning.
Results: Intrapersonal Skills
RQ1: Malleability
e Big Five is the most widely adopted theoretical model of personality traits (e.g., McCrae & Costa Jr., 2008). How-
ever, longitudinal Big Five research appears to be limited to those investigating naturalistic developmental changes
(e.g., McCrae & John, 1992; Roberts et al., 2006) or clinical interventions (e.g., Roberts, Luo, et al., 2017; Lipsey &
Wilson, 1993). An additional meta-analysis provided relevant outcomes for resilience, a personality construct that
overlaps with the Big Five (e.g., Friborg et al., 2005; John & Srivastava, 1999), although they did not explicitly use Big
Five terminology. Specically, Vanhove et al. (2016) noted only small intervention eects for resilience in response to
workplace programs targeting psychosocial factors. e authors also observed diminishing eects of these programs
over time.
RQ2: Mechanisms of Change
Vanhove et al.’s (2016) meta-analysis of workplace resilience-building programs included a diverse array of primary pre-
vention methods that focus on developing psychosocial traits such as self-ecacy and optimism. Some programs also
incorporated secondary or tertiary methods such as stress management, physical tness, or meditation. Of note, these
interventions are oen informed by resilience theory. A central tenet of this theory is that occupational groups exposed
to greater levels of work-related stress and trauma will be most informative in identifying factors and mechanisms that
facilitate resilience (see review by Vanhove et al., 2016). Importantly, workplace stress is not isolated to that caused by
acute traumatic experiences, as repeated milder stressors may also result in cumulative psychological damage. However,
the diversity of programs precluded examination of ecacy dierences across intervention types.
RQ1: Malleability
We located 11 meta-analyses and narrative reviews examining the malleability of attitudes through deliberate intervention
(see Table 2. ese studies consistently demonstrated that interventions eectively reduce attitudes related to several types
of counterproductive work behavior (CWB; e.g., Bangert-Drowns, 1988; Bezrukova et al., 2016; Brecklin & Forde, 2001;
Kalinoski et al., 2013) and improve attitudes toward various positive job performance areas (Guskey, 1986; Jones, 2016;
Lipsey & Wilson, 1993). When these changes are quantied, they are typically of small-to-moderateeect sizes (e.g., Barth
& Lannen, 2011).
RQ2: Mechanisms of Change
Attitude interventions may be dierentiated by their content or approach. In some instances, a wide variety of approaches
exists for altering attitudes even within a specic context. Given that the attitude interventions we reviewed tended to be
relevant to specic forms of CWB, they were delivered in various workplace settings. For example, Anderson and Whis-
ton (2005) described four main categories of sexual assault education programs: informative programs discuss factual
information, statistics, myths, facts, and consequences; empathy-focused interventions emphasize developing empathy
for victims; socialization-focused programs examine gender-role stereotyping and societal inuences; and risk-reducing
interventions teach specic strategies to reduce victimization risk. For reducing rape attitudes and rape-related attitudes,
empathy-focused interventions tended to be less eective than the other strategies, whereas program content was unas-
sociated with reductions in behavioral intent. In improving intergroup attitudes (e.g., diversity and inclusion training),
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 13
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Beelmann and Heinemann (2014) reported that social-cognitive training designed to promote empathy and perspective
taking were most eective, whereas programs involving classication/social categorization (i.e., the cognitive process of
classifying individuals based on demographics) or problem-solving skills were least eective. Otherwise, theoretical orien-
tation (e.g., socialization/knowledge acquisition; social-cognitive development) did not impact eectiveness. Undesirable
attitudes toward controlled or illegal substances have been demonstrated to be more eectively reduced when interven-
tions employ relatively contemporary methods involving reinforcement, social norms, and developmental interventions
than when simply focusing on more traditional information such as adverse health eects.
In other areas, attitude change may be observed when interventions are keenly designed for an intended context, or
conversely when the attitudes in question were not the primary focus. For example, Jones (2016) concentrated on work-
place coaching strategies in identifying eective methods for improving workplace attitudes. On the other hand, Barth and
Lannen (2011) reported that communication skills training in health-care settings somewhat unintentionally improved
attitudes toward terminally ill patients, death, and dying. A slightly dierent perspective was described by Guskey (1986),
who used a process model to contend that improvements in teachers’ attitudes toward classroom practices (e.g., curricu-
vice-versa. In other words, attitude change is most eective when viewed as a secondary, indirect target of intervention
rather than a primary or direct one.
RQ1: Malleability
We located three meta-analyses and reviews describing the eortful malleability of self-concept (see Table 2). ese studies
support the malleability of self-ecacy in the context of online learning (Hodges, 2008), general self-ecacy (Buljac-
Samardzic et al., 2010), and general self-concept (e.g., self-control, self-ecacy; Bowen & Neill, 2013). Meta-analytic
estimates of pre– post improvements in self-concept were in the small-to-moderate range (Bowen & Neill, 2013). How-
ever, these authors also noted that these changes were not maintained at long-term follow-up assessment. Unfortunately,
Bowen and Neill (2013) did not describe the specic time period representing long-term eects.
RQ2: Mechanisms of Change
Strategies employed to promote eortful improvements in self-esteem and self-ecacy have varied. Successful approaches
have included motivational messages (Hodges, 2008), goal-setting (Buljac-Samardzic et al., 2010), and adventure therapy
(Bowen & Neill, 2013), which is an intervention focused on outdoor experiential learning that is not necessarily clinical
in nature. Bowen and Neill (2013) also reported that alternative interventions to adventure therapy were not eective but
did not elaborate on what types of programs were evaluated. Overall, these results suggested a diverse assortment of self-
concept improvement tactics applicable to a variety of individuals,evenwhenself-conceptisnotnecessarilytheprimary
target of interventions.
RQ1: Malleability
Five articles we identied examined deliberate eorts to alter emotions or aect (see Table 2. e consensus from these
reviews was that emotions are indeed malleable. ese ndings are consistent regardless of whether aect is dened
broadly (Augustine & Hemenover, 2009; Webb et al., 2012) or using more specic examples such as empathy (Teding
van Berkhout & Malou, 2016), or in contexts such as team eectiveness (Delise et al., 2010) or risk appraisal (Sheeran
et al., 2014).
RQ2: Mechanisms of Change
Regarding the specic programs used to engender deliberate aective change, our review revealed a diverse array of strate-
gies. Augustine and Hemenover (2009) identied over 300 aect regulation or repair strategies in the literature. For their
14 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
analyses, they adopted Parkinson and Totterdell’s (1999) four superordinate categories. In this model, the rst distinc-
tion separates behavioral distractions— which involve some type of overt physical action such as walking away from a
distressing situation—from cognitive actions, with examples including thinking about something other than the distress-
ing situation. e second distinction categorizes strategies as engagementin which one actively attends to the aective
experience through actions such as reappraisal— versus avoidance, in which the individual behaviorally or cognitively
removes themselves from the distressing situation. Augustine and Hemenover (2009) compared the ecacy of these four
categories of interventions, 10 subordinate strategies that fall under these superordinate categories, and a separate tax-
onomy of specic strategies. e four superordinate categories demonstrated generally similar results (small-to-medium
eect sizes), with behavioral strategies being slightly more eective.
Webb et al. (2012) categorized interventions using a process model of emotion regulation, in which strategies are
dened based on whether they occur prior to the emotional experience (antecedent-focused) or aerward (response-
focused; Gross & ompson, 2007). One antecedent-focused example is attentional deployment, which includes distrac-
tions (i.e., either active or passive and either positive or neutral) and concentration (e.g., concentrating on feelings or on
causes and implications). Cognitive change is another antecedent-focused example, with specic strategies varying based
on their tendency to reappraise emotional responses, emotional stimuli, or through perspective taking. Finally, response-
focused strategies involve response modulation through the suppression of the emotion-related expression, experience,
or event. e authors reported that attentional deployment had no eect on emotional outcomes, response modulation
approached a small eect, and cognitive change had a small-to-medium eect.
One unique example of emotion manipulation was examined by Sheeran et al. (2014). ese authors studied the
impact of interventions designed to increase negative aect (NA) associated with perceptions of risk or threat as strate-
gies for decreasing various problematic behaviors. A common example includes health warnings designed to deter various
self-destructive behaviors (e.g., smoking, poor diet, etc.). In this context, emotions are typically categorized based on
whether they precede the target behavior (i.e., anticipatory emotions such as fear or worry) or follow it (i.e., anticipated
emotions such as regret and guilt). Accordingly, the strategies designed to increase these emotions are described as
heightening risk appraisal. e authors did not assess specic interventions separately, but noted that heightening risk
appraisal increased anticipatory and anticipated emotions by moderate-to-large and small-to-moderate eect sizes,
e remaining reviews generally included more specic types of interventions. Perhaps the most pertinent class of
interventions— empathy trainingwas examined by Teding van Berkhout and Malou (2016). Empathy training tends
to include elements of behavioral skills training (i.e., modeling, instructions, rehearsal, and feedback) designed to target
the aective, cognitive, or behavioral components of empathy. Specic components include lectures, demonstrations,
practice, games, and role-play. Interestingly, despite the fact that emotions were not necessarily the intended target of
these interventions, each was eective in improving aect to varying degrees.
RQ1: Malleability
e malleability of intrinsic motivation has been the source of intense debate. Cameron and Pierce (1994) argued in
their controversial meta-analysis of the eects of extrinsic rewards on intrinsic motivation that extrinsic rewards do not
decrease intrinsic motivation. ey posited that verbal praise increases intrinsic motivation when motivation is assessed
using free time and attitude measures. However, when individuals are given expected tangible rewards for completing a
task, intrinsic motivation decreases when measured as free time performance in which a participant has the opportunity to
work on a task when no rewards are being provided and when the presumption is that the experimenter is not monitoring
the participant’s activity. It is worth noting that both the number of eect sizes and size of the observed eects are small.
As noted by Ryan and Deci (1996), the authors’ aggregation of all reward categories into one global variable may mask
signicant eects at ner-grained categories. Deci et al. (1999, 2001) reiterated in later research that extrinsic rewards do
indeed undermine intrinsic motivation and supported this claim with meta-analytic evidence (Deci et al., 1999; see also
Rummel & Feinberg, 1988). us, Cameron and Pierce’s ndings warrant extreme caution.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 15
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
RQ2: Mechanisms of Change
Jones (2016) presented an overview of the history and eectiveness of workplace coaching. Workplace coaching is
described as learner-centered, collaborative, reective, goal-focused instruction that can provide coachees with a
tailored approach to understanding and applying work-based learning (Jones, 2016). According to Jones, coaching oen
focuses on challenges within the individual, between individuals, or a combination. Jones presented a meta-analysis of
coaching eectiveness, showing that coaching has a moderately positive eect on motivation. Additionally, Sitzmann
and Ely (2011) conducted a meta-analysis that suggested that goal setting results in greater learning when individuals
are committed to reaching a specic goal, possess the requisite task knowledge, and are provided with feedback on
their progress toward their goal or goals. ey argued that specic and dicult goals that are attainable motivate
Summary and Recommendations: Intrapersonal Skills
Our review suggests there is much to be learned about personality constructs’ amenability to deliberate intervention. As
mentioned, studies examining naturalistic, developmental changes (McCrae & John, 1992; Roberts et al., 2006) should
not be interpreted as support for the Big Five’s amenability to eortful intervention. However, this research may describe
common life events that catalyze personality change, and may in turn inform eortful interventions. Similarly, research
describing personality change in response to clinical intervention (e.g., Lipsey & Wilson, 1993) will likely possess debat-
able transportability to workplace settings. Further research is needed to determine if components of clinical interventions
(e.g., practice exercises; self-reection) may be relevant to workplace settings. Indeed, relevant intervention programs have
already been proposed: Roberts, Luo, et al.’s (2017) Sociogenomic Trait Intervention Model (STIM) incorporates elements
of behavioral activation theory, motivational theories, and developmental research as a potential strategy for stimulating
changes in Big Five conscientiousness, for example. Furthermore, most workplace interventions tend to be delivered by
trainers or coaches who do not hold a clinical certication. Lessons learned from successful workplace interventions
targeting constructs beyond the Big Five may be useful. However, given that resilience interventions appear to possess
relatively low ecacy, other noncognitive construct interventions may be more informative. Finally, even though the
Big Five is considered the predominant model of personality constructs, other workplace-relevant personality constructs
(e.g., those comprising the Dark Triad; Paulhus & Williams, 2002) would also benet from further eortful malleability
Attitudes may be associated with virtually any topic, providing ample opportunities for empirical study. More speci-
cally, the fact that we identied a large number of studies examining attitudes in the context of eortful change suggests
that they are ideal targets for deliberate intervention. In particular, reviews involving workplace samples such as health-
care professionals (Barth & Lannen, 2011), teachers (Guskey, 1986), or general workplace settings (Jones, 2016) reveal
that attitudes are relevant for a variety of professions. Furthermore, the content of these interventions suggest that they
are pertinent to both positive and negative job performance outcomes. e consensus among these reviews is that atti-
tudes are amenable to change, albeit to a relatively small degree. At the same time, given that many attitudes appear to
have an aective component (e.g., Beelmann & Heinemann, 2014), it is possible that emotion-focused interventions may
have some value in promoting attitude change. It will also be benecial to examine whether attitude interventions evoke
behavioral change, which has been a challenge in some settings.
e fundamental importance of self-concept suggests that it is a highly salient target for workplace interventions, and
there is indeed empirical evidence for its eectiveness in these settings (i.e., health-care professionals; Buljac-Samardzic
et al., 2010). Nonetheless, given the variety of successful interventions, care must be taken to identify components that
are particularly relevant to the workplace. For instance, at rst glance, programs such as adventure therapy may appear
irrelevant to workplace settings. However, several components of adventure therapy may be conducted indoors and are
already employed in workplace interventions, including trust activities, initiative experiences, problem-solving scenarios,
and team-based tasks (Bowen & Neill, 2013), suggesting a high degree of transportability. Additionally, the notion that
program scope should match the intended content area of expertisealso referred to as the “bandwidth” issueappears
particularly relevant to self-ecacy (Hodges, 2008): interventions promoting engineering self-ecacy would be of little
16 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
malleability. is research benets from a relatively high proportion of meta-analyses as opposed to narrative reviews,
allowing for a more systematic and quantitative summary. Overall, aect appears to represent a category of noncognitive
constructs susceptible to change, even when the intervention in question is not necessarily designed to target emotion per
se. However, the sheer volume of interventions (see Augustine & Hemenover, 2009) may preclude the investigation of any
one specic program. Nonetheless, another advantage of the interventions discussed in this section is that many of them
appear transportable to the workplace, such as the distraction and engagement strategies examined by Augustine and
Hemenover (2009). In fact, the emotion reviews we identied explicitly included occupations ranging from the military
(Delise et al., 2010) to health care (Teding van Berkhout & Malou, 2016). In turn, these interventions may be predom-
inantly relevant to occupations involving frequent interpersonal interaction, with signicant and frequent stressors, or
where prosocial behavior is particularly valued. e results reported by Sheeran et al. (2014) suggested that increasing NA
by heightening risk appraisal can be eective in reducing a variety of workplace-relevant undesirable behaviors. Although
this practice is common, the ethics of this strategy may be debatable, as the experience for the individual may be negative.
Furthermore, conicting results within emotion research (Webb et al., 2012) suggest that increasing positive aect and
decreasing NA are not necessarily opposite sides of the same coin and that future investigations clarifying this distinction
would be welcomed.
A great deal of evidence suggests motivation in the context of discrete tasks can be improved or diminished using
interventions. Interventions to improve motivation may include workplace coaching (Jones, 2016), goal setting (Sitzmann
& Ely, 2011), praise, and rewards. Taken together, the ndings of the studies reviewed suggest that extrinsic rewards tend
to decrease intrinsic motivation (c.f. Cameron & Pierce, 1994).
is review aimed to summarize and evaluate the current state of the literature on the intervention-based development of
two broad categories of noncognitive constructs germane to workplace success: interpersonal and intrapersonal skills. Our
multidisciplinary approach drew from meta-analyses and reviews spanning a diverse set of domains to provide a thorough
review of the malleability of noncognitive constructs. More broadly, this review contributes to a growing interest in applied
arenas in the promotion and development of noncognitive constructs that have been shown to be critical in supporting
workplace success. Overall, recognizing a few exceptions and gaps in the literature, ndings suggest optimism regarding
the malleability of noncognitive constructs, and provide a preliminary blueprint for the optimal design, implementation,
and evaluation of intervention programs.
RQ1: Noncognitive Construct Malleability
Among the appealing features of noncognitive constructs are that they predict workplace success to a degree similar to
traditional factors such as cognitive skills yet are potentially more malleable (e.g., Roberts, Hill, & Davis, 2017; Roberts,
Luo, et al., 2017). Our review supported the malleability of several distinct noncognitive constructs. e majority of meta-
analytic estimates are of small to moderate eect sizes (e.g., d0.20–0.50), though occasionally interventions produce
smaller or larger eects. For instance, communication skills, leadership skills, and emotion tend to consistently produce
the strongest results supporting malleability, typically higher than Cohen’s (1988) guideline for a moderate eect size (e.g.,
d≈±0.50). Conversely, eect sizes for personality, interpersonal skills, teamwork, attitudes, self-concept, and motivation
tend to be smaller or more inconsistent. At the same time, the amount of meta-analytic research varies widely across
constructs: We were only able to locate one relevant meta-analysis each for personality, communication skills, and self-
concept, compared to 11 attitudes meta-analyses. Eortful malleability meta-analyses were absent altogether for several
workplace-relevant constructs including the Big Five, Dark Triad, and emotional intelligence.
is trend suggests that research investigating the longitudinal dynamics of noncognitive constructs lags behind that
of predictive validity research. Aer identifying noncognitive constructs based on their established links to job per-
formance, malleability research serves a critical supplementary role in that it claries the subset of constructs with an
evidence-based rationale for intervention. Moreover, this distinction provides essential practical guidance for employers:
Constructs—noncognitive or otherwise—that are relatively xed may be more relevant to personnel selection decisions,
whereas training decisions may be informed by identifying constructs that are more malleable.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 17
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
RQ2: Measurement and Mechanisms of Noncognitive Construct Change
e vast majority of malleability research quantied changes using rank-order stability or mean-level changes. As previ-
ously mentioned, these strategies are oen considered complementary, recognizing their respective strengths and weak-
nesses. It is likely that the eld would benet from additional research involving more sophisticated statistical approaches,
including latent variable modeling, growth curve analyses, structural continuity, ipsative continuity, and coherence (e.g.,
Caspi & Roberts, 2001; Curran et al., 2010; McArdle, 2009). Importantly, these methods permit the assessmentof changes
across more than two time points, which is a prerequisite for evaluating nonlinear longitudinal trends. Similarly, tech-
niques such as mediation analyses, repeated measures ANOVA, and structural equation modeling provide opportunities
to examine the causal impact of interventions (e.g., Cole & Maxwell, 2003; Pearl, 2009). ese advances may coincide
with increased usage of nontraditional noncognitive construct assessments, including forced-choice, game-based, and
performance-based measures.
e diversity of disciplines included in our review gives way to an even more abundant array of strategies for eortful
noncognitive construct change. e list of intervention approaches that have been used frequently enough to merit meta-
analytic study or narrative review is extremely diverse. Examples include modeling, goal-setting, character education,
sports participation, adventure therapy, role-playing, and reward-based programs. e specic exercises embedded within
these programs both aids in explaining their ecacy and may inform the development of new programs. Additionally, it
would be prudent to consider change catalysts observed in clinical interventions or naturalistic developmental, as a subset
of these features may be transportable to workplace settings. However, appropriate parameters must be applied to ensure
that interventions are appropriate for organizational trainers and coaches who do not have a clinical background and for
Limitationsand Future Directions
Our review is not without limitations, many of which surround our search strategy. First, we chose to review only source
articles that could be characterized as meta-analyses or reviews. Consequently, very recent research and some primary
studies may not have been captured by the selection of source material reviewed and would thus have been excluded
from this investigation. It should also be noted that, although meta-analyses have been widely accepted as a vital tool
for aggregating primary research, some authors remain skeptical of their value (e.g., Costa Jr. & McCrae, 2006). Second,
the use of Google Scholar as opposed to more traditional databases such as PsycINFO as our primary search engine
could be a potential limitation, based on the criticisms of some researchers (e.g., Boeker et al., 2013; Giustini & Bou-
los, 2013). Conversely, other authors have praised Google Scholar as “an invaluable tool for conducting literature research”
(de Winter et al., 2014, p. 1562), and have supported it as the rst and potentially sole search engine for reviews and meta-
analyses (e.g., Gehanno et al., 2013). Nonetheless, we supplemented our Google Scholar search with various additional
Other limitations concern the nature of the meta-analyses and reviews that arose from our search. Most notably,
there is a paucity of studies examining the malleability of some noncognitive constructs (e.g., leadership skills). Meta-
analyses or systematic reviews for other constructs such as grit, emotional intelligence, interests, the Dark Triad of
personality, and integrity were either too few to be included in our review or were not located in our search. is issue
is exacerbated by the fact that there does not appear to be a universal, broadly accepted, exhaustive list of noncognitive
constructs, given that this research is constantly expanding and that some variability exists regarding the denition
of noncognitive (see Duckworth & Yeager, 2015; Heckman & Kautz, 2014; Kell, 2018). In other instances, outcomes
were oen measured at the group level rather than the individual participant level (e.g., teamwork). e expansion
of research to specic occupational elds beyond the somewhat narrow subset located in our review (e.g., health
care) would also be benecial. Although we restricted our review to studies that focused on adult participants, studies
for some constructs (e.g., emotion) included both youth and adults. Methodologically, another caveat worth noting
concerns the inclusion of research examining rank-order consistency as opposed to mean-level change or vice-versa.
Studies employed dierent approaches for examining change and both methods have drawbacks, especially compared to
more contemporary analytical techniques and their associated methodological designs (e.g., Biesanz et al., 2003; McAr-
dle, 2009). Moreover, even in studies that examine within-person change pre- and postintervention, natural regression
18ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
to the mean may be confounded with change as a result of intervention. Additionally, traditional eect size interpre-
tation guidelines (e.g., Cohen, 1988) could be supplemented with data-driven benchmarks specically derived from
noncognitive construct malleability research, a strategy that has been examined in other research elds (e.g., e.g., Bosco
et al., 2015; Gignac & Szodorai, 2016; Paterson et al., 2015). Each of these issues represents appropriate targets for future
Empirical support for the eortful malleability of workplace-relevant noncognitive constructs abounds, though many
areas of research remain open. Meta-analyses and reviews provide data-driven guidance to stakeholders regarding eective
selection and training of employees in terms of skills whose value is both recognized by employers and supported by
research. At the same time, additional research regarding understudied noncognitive constructs, specic interventions,
and underused methodologies and statistical approaches will undoubtedly advance the eld. Ideally, this research will
facilitate the development of a workforce whose technical expertise and cognitive skills are complemented by important
noncognitive assets.
1 Some components of personality may be considered interpersonal (e.g., Big Five Agreeableness), though we include them with
their intrapersonal counterparts to maintain discussion of the Big Five within one self-contained section.
References for articles included in the systematic review are marked with an asterisk
Aguinis, H., & Kraiger, K. (2009). Benets of training and development for individuals and teams, organizations, and society. Annual
Review of Psychology,60, 451 474.
*Anderson, L. A., & Whiston, S. C. (2005). Sexual assault education programs: A meta-analytic examination of their eectiveness.
Psychology of Women Quarterly,29, 374– 388.
*Arthur, W., Jr., Bennett, W., Jr., Edens, P. S., & Bell, S. T. (2003). Eectiveness of training in organizations: A meta-analysis of design
and evaluation features. Journal of Applied Psychology,88, 234245.
*Augustine, A. A., & Hemenover, S. H. (2009). On the relative eectiveness of aect regulation strategies: A meta-analysis. Cognition
and Emotion,23, 1181 –1220.
*Avolio, B. J., Avey, J. B., & Quisenberry, D. (2010). Estimating return on leadership development investment. Leadership Quarterly,
21, 633–644.
*Avolio, B. J., Reichard, R. J., Hannah, S. T., Walumbwa, F. O., & Chan, A. (2009). A meta-analytic review of leadership impact research:
Experimental and quasi-experimental studies. Leadership Quarterly,20, 764– 784.
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology,41, 63 –105.
*Bangert-Drowns, R. L. (1988). e eects of school-based substance abuse education—A meta-analysis. Journal of Drug Education,
18, 243–264. FFWC-YF1U
Barrick, M. R., & Mount, M. K. (1991). e big ve personality dimensions and job performance: A meta-analysis. Personnel Psychology,
44, 1–26.
*Barth, J., & Lannen, P. (2011). Ecacy of communication skills training courses in oncology: A systematic review and meta-analysis.
Annals of Oncology,22, 1030 1040.
*Beelmann, A., & Heinemann, K. S. (2014). Preventing prejudice and improving intergroup attitudes: A meta-analysis of child and
adolescent training programs. Journal of Applied Developmental Psychology,35, 10–24.
Bell, B. S., Tannenbaum, S. I., Ford, J. K., Noe, R. A., & Kraiger, K. (2017). 100 years of training and development research: What we
know and where we should go. Journal of Applied Psychology,102, 305323.
Bezrukova, K., Spell, C. S., Perry, J. L., & Jehn, K. A. (2016). A meta-analytical integration of over 40 years of research on diversity
training evaluation. Psychological Bulletin,142, 1227– 1274.
Biesanz, J. C., West, S. G., & Kwok, O.-M. (2003). Personality over time: Methodological approaches to the study of short-term and
long-term development and change. Journal of Personality,71, 905–941.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 19
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Bleidorn, W., Hopwood, C. J., & Lucas, R. E. (2018). Life events and personality trait change. Journal of Personality,86, 83 –96. https://
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta-analytic review. Journal of Management,36,
Boeker, M., Vach, W., & Motschall, E. (2013). Google scholar as replacement for systematic literature searches: Good relative recall and
precision are not enough. BMC Medical Research Methodology,13, 1 12. 131
Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational eect size benchmarks. Journal of Applied Psychol-
ogy,100, 431449.
*Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and moderators. Open Psychology Journal,6, 28 –53.
Brecklin, L. R., & Forde, D. R. (2001). A meta-analysis of rape education programs. Violence and Victims,16, 303-321.
Briley, D. A., & Tucker-Drob, E. M. (2014). Genetic and environmental continuity in personality development: A meta-analysis. Psy-
chological Bulletin,140, 1303–1331.
Bruvold, W. H., & Rundall, T. G. (1988). A meta-analysis and theoretical review of school based tobacco and alcohol intervention
programs. Psychology and Health,2, 53-78.
*Buljac-Samardzic, M., Dekker-van Doorn, C. M., van Wijngaarden, J. D., & van Wijk, K. P. (2010). Interventions to improve team
eectiveness: A systematic review. Health Policy,94, 183– 195.
*Cameron, J., & Pierce, W. D. (1994). Reinforcement, reward, and intrinsic motivation: A meta-analysis. Review of Educational Research,
64, 363–423.
Capelli, P. (2012). Why good people can’t get jobs: e skills gap and what companies can do about it.WhartonDigitalPress.
Card, N.A. (2012). Applied meta-analysis for social science research. Guilford.
Caspi, A., & Roberts, B. W. (2001). Personality development across the life course: e argument for change and continuity. Psychological
Inquiry,12, 4966.
*Cheraghi-Sohi, S., & Bower, P. (2008). Can the feedback of patient assessments, brief training, or their combination, improve the
interpersonal skills of primary care physicians? A systematic review. BMC Health Services Research,8, 1 –10.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Cole, D. A., & Maxwell, S. E. (2003). Testing mediation models with longitudinal data: Questions and tips in the use of structural
equation modeling. JournalofAbnormalPsychology,112, 558 577.
Costa, P. T., Jr., & McCrae, R. R. (2006). Age changes in personality and their origins: Comment on Roberts, Walton, and Viechtbauer
(2006). Psychological Bulletin,132, 26– 28.
Curran, P. J., Obeidat, K., & Losardo, D. (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognitive
Development,11, 121 136.
de Winter, J. C. F., Zadpoor, A. A., & Dodou, D. (2014). e expansion of Google Scholar versus Web of Science: A longitudinal study.
Scientometrics,98, 1547– 1565.
*Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the eects of extrinsic rewards on
intrinsic motivation. Psychological Bulletin,25, 627 668.
*Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education: Reconsidered once again.
Review of Educational Research,71, 1– 27.
*Delise, L. A., Gorman, C. A., Brooks, A. M., Rentsch, J. R., & Steele-Johnson, D. (2010). e eects of team training on team outcomes:
A meta-analysis. Performance Improvement Quarterly,22, 5380.
Duckworth, A. L., & Yeager, D. S. (2015). Assessing personal qualities other than cognitive ability for educational purposes. Educational
Researcher,44, 237– 251.
Freifeld, L. (2018). TRAINING Industry report.
Friborg, O., Barlaug, D., Martinussen, M., Rosenvinge, J. H., & Hjemdal, O. (2005). Resilience in relation to personality and intelligence.
International Journal of Methods in Psychiatric Research,14, 29– 42.
Gehanno, J.-F., Rollin, L., & Darmoni, S. (2013). Is the coverage of Google scholar enough to be used alone for systematic reviews?
BMC Medical Informatics and Decision Making,13, 1– 5.
Gignac, G. E., & Szodorai, E. T. (2016). Eect size guidelines for individual dierences researchers.Per sonality and Individual Dierences,
102, 74–78.
*Gillespie, B. M., Chaboyer, W., & Murray, P. (2010). Enhancing communication in surgery through team training interventions: A
systematic literature review. AORN Journal,92, 642657.
Giustini, D., & Boulos, M. N. (2013). Google scholar is not enough to be used alone for systematic reviews. Online Journal of Public
Health Informatics,5, 214.
20 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
*Gordon, M., Darbyshire, D., & Baker, P. (2012). Non-technical skills training to enhance patient safety: A systematic review. Medical
Education,46, 1042–1054.
Gottfredson, L. S. (1998). e general intelligence factor. Scientic American,9, 24– 29.
Gross, J. J., & ompson, R. A. (2007). Emotion regulation: Conceptual foundations. In J.J. Gross (Ed.), Handbook of emotion regulation
(pp. 3– 24). Guilford Press.
*Guskey, T. R. (1986). Sta development and the process of teacher change. Educational Researcher,15, 5 –12.
Heckman, J. J., & Kautz, T. D. (2012). Hard evidence on so skills. Labour Economics,19, 451– 464.
Heckman, J. J., & Kautz, T. D. (2014). Achievement tests and the role of character in American life. In J. J. Heckman, J. E. Humphries, &
T. Kau t z ( E d s . ) , e myth of achievement tests: e GED and the role of character in American life (pp. 1– 71). University of Chicago
Heckman, J. J., & Rubinstein, Y. (2001). e importance of noncognitive skills: Lessons from the GED testing program. American
Economic Review,91, 145– 149.
*Hodges, C. B. (2008). Self-ecacy in the context of online learning environments: A review of the literature and directions for research.
Performance Improvement Quarterly,20, 725.
*Hulsman, R. L., Ros, W. J. G., Winnubst, J. A. M., & Bensing, J. M. (1999). Teaching clinically experienced physicians communication
skills. A review of evaluation studies. Medical Education,33, 655– 668.
Jensen, A. R. (1998). e g factor: e science of mental ability. Praeger.
John, O. P., & Srivastava, S. (1999). e big ve trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin &
O. P. John (Eds.), Handbook of personality (2nd ed.; pp. 102– 138). Guilford.
*Jones, R. (2016). e eectiveness of workplace coaching: A meta-analysis of learning and performance outcomes,scale development,
theoretical model of individual dierences and longitudinal study (Unpublished doctoral dissertation). Aston University, Birmingham,
*Kalinoski, Z. T., Steele-Johnson, D.,Peyton, E. J., Leas, K. A., Steinke, J., & Bowling, N. A. (2013). A meta-analytic evaluation of diversity
training outcomes. Journal of Organizational Behavior,34, 1076 –1104.
Kautz, T., Heckman, J. J., Diris, R., Ter Weel, B., & Borghans, L. (2014). Fostering and measuring skills: Improving cognitive and noncog-
nitive skills to promote lifetime success (NBER Working Paper No. 20749). National Bureau of Economic Research.
Kell, H. J. (2018). Noncognitive proponents’ conation of “cognitive skills” and “cognition” and its implications. Personality and Indi-
vidual Dierences,134, 25–32.
Kirkpatrick, D. (1996). Great ideas revisited: Revisiting Kirkpatrick’s four-level model. Tra ini ng & De vel opm e nt ,50, 54– 57.
*Klein, C., DiazGranados, D., Salas, E., Le, H., Burke, C. S., Lyons, R., & Goodwin, G. F. (2009). Does team building work? Small Group
Research,40, 181– 222.
*Klein, C. R. (2009). What do we know about interpersonal skills? A meta-analytic examination of antecedents, outcomes, and the ecacy
of training (Unpublished doctoral dissertation). University of Central Florida, Orlando, FL.
Klieger, D., Ezzo, C., Bochenek, J., & Cline, F. (2015, April 16– 20). e predictive validity of non-cognitive skills for graduate and pro-
fessional student success: Some initial ndings. Paper presented at the Validity and Fairness Issues in Assessing Higher Education
Students Symposium, Annual Meeting of the American Educational Research Association, Chicago, IL.
Kyllonen, P. C. (2012, May 78). Measurement of 21st century skills within the Common Core State Standards. Paper presented at the
Invitational Research Symposium on Technology Enhanced Assessments, Princeton, NJ.
*Lacerenza, C. N., Reyes, D. L., Marlow, S. L., Joseph, D. L., & Salas, E. (2017). Leadership training design, delivery and implementation.
Journal of Applied Psychology,102, 1686–1718
*Lane, C., & Rollnick, S. (2007). e use of simulated patients and role-play in communication skills training: A review of the literature
to August 2005. Patient Education and Counseling,67, 13 –20.
Lindqvist, E., & Vestman, R. (2011). e labor market returns to cognitive and noncognitive ability: Evidence from the Swedish enlist-
ment. American Economical Journal: Applied Economics,3, 101– 128.
*Lipsey, M. W., & Wilson, D. B. (1993). e ecacy of psychological, educational, and behavioral treatment: Conrmation from meta-
analysis. American Psychologist ,48, 1181 1209.
McArdle, J. J. (2009). Latent variable modeling of dierences and changes with longitudinal data. Annual Review of Psychology,60,
McCrae, R. R., & Costa, P. T., Jr. (2008). Empirical and theoretical status of the Five-Factor Model of personality traits. In G. Boyle,
G. Matthews, & D. Saklofske (Eds.), Sage handbook of personality theory and assessment (Vol. 1, pp. 273–294). Sage Publications.
McCrae, R. R., & John, O. P. (1992). An introduction to the ve-factor model and its applications. Journal of Personality,60, 175-215.
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 21
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
*McCulloch, P., Rathbone, J., & Catchpole, K. (2011). Interventions to improve teamwork and communications among healthcare sta.
British Journal of Surgery,98, 469 479.
Moher, D., Liberati, A., Tetzla, J., Altman, D. G., & e PRISMA Group. (2009). Preferred reporting items for systematic reviews and
meta-analyses: e PRISMA Statement. PLoS Medicine,6, 1–6.
Parkinson, B., & Totterdell, P. (1999). Classifying aect regulation strategies. Cognition and Emotion,13, 277– 303.
Paterson, T. A., Harms, P. D., Steel, P., & Credé, M. (2015). An assessment of the magnitude of eect sizes: Evidence from
30 years of meta-analysis in management. Journal of Leadership & Organizational Studies,23, 66 81.
Paulhus, D. L., & Williams, K. M. (2002). e dark triad of personality: Narcissism, Machiavellianism, and psychopathy. Journal of
Research in Personality,8, 556– 563.
Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys,3, 96 –146.
*Pellegrino, J. W., & Hilton, M. L. (2012). Education for life and work: Developing transferable knowledge and skills in the 21st century.
e National Academies Press.
*Robbins, S. B., Oh, I.-S., Le, H., & Button, C. (2009). Intervention eects on college performance and retention as mediated by moti-
vational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology,94, 1163 –1184.
Roberts, B. W., Hill, P. L., & Davis, J. P. (2017). How to change conscientiousness: e sociogenomic trait intervention model. Personality
Disorders: eory, Research, and Treatment,8, 199205.
Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through
intervention. Psychological Bulletin,143, 117– 141.
Roberts, B. W., Walton, K. E., & Viechtbauer, W. (2006). Patterns of mean-level change in personality traits across the life course: A
meta-analysis of longitudinal studies. Psychological Bulletin,132, 1 –25. .
Rummel, A., & Feinberg, R. (1988). Cognitive evaluation theory: A meta-analytic review of the literature. Social Behavior and Person-
ality: An International Journal,16, 147– 164.
*Ryan, R. M., & Deci, E. L. (1996). When paradigms clash: Comments on Cameron and Pierce’s claim that rewards do not undermine
intrinsic motivation. Review of Educational Research,66, 33– 38.
*Salas, E., DiazGanados, D., Weaver, S. J., & King, H. (2008). Does team training work? Principles for health care. Academic Emergency
Medicine,15, 1002 –1009.
*Salas, E., Nichols, D. R., & Driskell, J. E. (2007). Testing three team training strategies in intact teams a meta-analysis. Small Group
Research,38, 471– 488.
Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does crew resource management training work? An update, an
extension, and some critical needs. Human Factors,48, 392 412.
Schmidt, F. L., & Hunter, J. E. (1998). e validity and utility of selection methods in personnel psychology: Practical and theoretical
implications of 85 years of research ndings. Psychological Bulletin,124, 262 274.
Scott, G., Leritz, L. E., & Mumford, M. D. (2004). e eectiveness of creativity training: A quantitative review. Creativity Research
Journal,16, 361 –388.
Sheeran, P., Harris, P. R., & Epton, T. (2014). Does heightening risk appraisals change People’s intentions and behavior? A meta-analysis
of experimental studies. Psychological Bulletin,140, 511– 543.
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training eective? Psychological Bulletin,138, 628–654. https://
*Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What
we know and where we need to go. Psychological Bulletin,137, 421 –442.
Society of Human Resource Management. (2019, February 5). e global skills shortage: Bridging the talent gap with education, training
and sourcing. surveys/Documents/SHRM%20Skills%20Gap
*Teding van Berkhout, E., & Malou, J. M. (2016). e ecacy of empathy training: A meta-analysis of randomized controlled trials.
Journal of Counseling Psychology,63, 32–41.
*Vanhove, A. J., Herian, M. N., Perez, A. L. U., Harms, P. D., & Lester, P. B. (2016). Can resilience be developed at work? A meta-analytic
review of resilience-building programme eectiveness. Journal of Occupational and Organizational Psychology,89, 278307. https://
Viswesvaran, C., & Ones, D. S. (2000). Measurement error in “Big Five factors” personality assessment: Reliability generalization across
studies and measures. Educational and Psychological Measurement,60, 224235.
*Weaver, S. J., Dy, S. M., & Rosen, M. A. (2014). Team-training in healthcare: A narrative synthesis of the literature. BMJ Quality &
Safety,23, 359– 372.
22 ETS Research Report No. RR-20-23. © 2020 Educational Testing Service
M. P. Martin-Raugh et al.The Malleability of Workplace-Relevant Noncognitive Constructs
Weaver, S. J., Lyons, R., DiazGranados, D., Rosen, M. A., Salas, E., Oglesby, J., King, H. B. (2010). e anatomy of health
care team training and the state of practice: A critical review. Academic Medicine,85, 17461760.
*Webb, T. L., Miles, E., & Sheeran, P. (2012). Dealing with feeling: A meta-analysis of the eectiveness of strategies derived from the
process model of emotion regulation. Psychological Bulletin,138, 775 808.
Suggested citation:
Martin Raugh, M. P., Williams, K. M., & Lentini, J. (2020). e malleability of workplace-relevant noncognitive constructs: Empirical
evidence from 39 meta-analyses and reviews (Research Report No.RR-20-23). Educational Testing Service.
Action Editor: Jessie Sparks
Reviewers:Sam Rikoon and Katrina Roohr
ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS).All other trademarks are property of their
respective owners.
Find other ETS-published reports by searching the ETS ReSEARCHER database at
ETS Research Report No. RR-20-23. © 2020 Educational Testing Service 23
... Indeed, several meta-analyses and reviews have demonstrated that various noncognitive constructs may be altered to varying degrees through intervention (e.g., Arthur Jr. et al., 2003;Barth & Lannen, 2011;Bezrukova et al., 2016;Bowen & Neill, 2013;Delise et al., 2010;Lacerenza et al., 2017;Lipsey & Wilson, 1993;Salas et al., 2008). A recent review by Martin-Raugh et al. (2020) synthesized 39 meta-analyses and systematic reviews on the malleability of three broad categories of noncognitive skills, drawing upon a diverse set of disciplines including industrial/organizational (I/O) psychology, educational psychology, and medicine, among others. We will be using the findings from Martin-Raugh et al. (2020) to develop a theory of change (ToC) to inform empirically supported, actionable recommendations for delivering and evaluating noncognitive construct interventions (NCIs) in higher education, occupational, and research settings. ...
... A recent review by Martin-Raugh et al. (2020) synthesized 39 meta-analyses and systematic reviews on the malleability of three broad categories of noncognitive skills, drawing upon a diverse set of disciplines including industrial/organizational (I/O) psychology, educational psychology, and medicine, among others. We will be using the findings from Martin-Raugh et al. (2020) to develop a theory of change (ToC) to inform empirically supported, actionable recommendations for delivering and evaluating noncognitive construct interventions (NCIs) in higher education, occupational, and research settings. ...
... Our ToC-NCI ToC (Figure 1)-illustrates the inputs, mechanisms, and intended outcomes of NCIs as informed by Martin-Raugh et al. (2020). Noncognitive constructs relevant to academic and workplace success may be altered through specific interventions (Path 1) or more general mechanisms of change (Path 2). ...
Full-text available
Researchers and practitioners in postsecondary and workplace settings recognize the value of noncognitive constructs in predicting academic and vocational success but also perceive that many students or employees are lacking in these areas. In turn, there is increased interest in interventions designed to enhance these constructs. We provide an empirically informed theory of change (ToC) that describes the inputs, mechanisms, and outputs of noncognitive construct interventions (NCIs). The components that inform this ToC include specific relevant constructs that are amenable to intervention, intervention content and mechanisms of change, methodological considerations, moderators of program efficacy, recommendations for evaluating NCIs, and suggested outcomes. In turn, NCIs should provide benefits to individuals, institutions, and society at large and also advance our scientific understanding of this important phenomenon.
... As their name implies, noncognitive constructs are largely unrelated to other academic-or workplace-relevant cognitive or technical skills such as traditional definitions of cognitive ability, mathematical skills, or problem-solving ability that standardized cognitive tests are designed to measure (e.g., Klieger et al., 2015). Furthermore, another appealing feature of noncognitive constructs is that they may be improved through training, possibly more easily than cognitive skills (e.g., Aguinis & Kraiger, 2009;Melby-Lervåg, Redick, & Hulme, 2016; see review by Martin-Raugh, Williams, & Lentini, 2018). An impressive body of reviews and metaanalyses demonstrates that noncognitive constructs predict academic and workplace success to a similar degree as cognitive or technical skills, and may do so independently (e.g., Roberts, Lejuez, Krueger, Richards, & Hill, 2014;Schwager et al., 2014;Schmidt & Hunter, 1998). ...
Full-text available
Overview of the open admissions model in the U.S. postsecondary academia system.
... Evaluations of competency change have generally provided encouraging evidence that many soft-skill competencies can be developed (e.g., Gibbons et al., 2006;Martin-Raugh et al., 2019;Mueller-Hanson et al., 2015;Straus et al., 2018). However, questions as to which soft-skill competencies are more or less amenable to development than others remain largely unanswered. ...
Full-text available
Researchers, theorists, and practitioners have expressed a renewed interest in the longitudinal dynamics of personality characteristics in adulthood, including organic life span trajectories and their amenability to volitional change. However, this research has apparently not yet expanded to include the Dark Triad (psychopathy, narcissism, Machiavellianism), despite approximately 2 decades of research that has thoroughly examined other important issues related to construct validity and interpersonal behavior. We argue that researchers in postsecondary, occupational, and community‐based settings are in a unique position to study the important phenomenon of Dark Triad malleability, as they are less hindered by obstacles in clinical and forensic contexts that have generated largely inconclusive results. In this article, we discuss several examples of methods for evaluating, quantifying, and interpreting Dark Triad malleability, examples of relevant extant training programs, possibilities for developing new programs, and factors that may moderate training efficacy, including Dark Triad levels themselves. Beyond addressing a fundamental question regarding the nature of these traits, the Dark Triad's destructive tendencies suggest that efforts to reduce them would provide myriad societal benefits and could propel Dark Triad research in an important new direction.
Full-text available
Recent estimates suggest that although a majority of funds in organizational training budgets tend to be allocated to leadership training (Ho, 2016; O'Leonard, 2014), only a small minority of organizations believe their leadership training programs are highly effective (Schwartz, Bersin, & Pelster, 2014), calling into question the effectiveness of current leadership development initiatives. To help address this issue, this meta-analysis estimates the extent to which leadership training is effective and identifies the conditions under which these programs are most effective. In doing so, we estimate the effectiveness of leadership training across four criteria (reactions, learning, transfer, and results; Kirkpatrick, 1959) using only employee data and we examine 15 moderators of training design and delivery to determine which elements are associated with the most effective leadership training interventions. Data from 335 independent samples suggest that leadership training is substantially more effective than previously thought, leading to improvements in reactions (δ = .63), learning (δ = .73), transfer (δ = .82), and results (δ = .72), the strength of these effects differs based on various design, delivery, and implementation characteristics. Moderator analyses support the use of needs analysis, feedback, multiple delivery methods (especially practice), spaced training sessions, a location that is on-site, and face-to-face delivery that is not self-administered. Results also suggest that the content of training, attendance policy, and duration influence the effectiveness of the training program. Practical implications for training development and theoretical implications for leadership and training literatures are discussed. (PsycINFO Database Record
Full-text available
Conscientiousness, the propensity to be organized, responsible, self-controlled, industrious, and rule-following, is related to numerous important outcomes including many forms of psychopathology. Given the increasing awareness of the importance of conscientiousness, it is becoming common to want to understand how to foster it. In this paper we first describe and update a recent model that was put forward as a theoretically informed intervention to change conscientiousness. We then consider recent life span theories focused on conscientiousness that might inform how best to use existing interventions as well as identify potential moderators of the effectiveness of intervention. Finally, we integrate these perspectives into a framework for how to foster conscientiousness that we label the Sociogenomic Trait Intervention Model (STIM).
Full-text available
Training and development research has a long tradition within applied psychology dating back to the early 1900s. Over the years, not only has interest in the topic grown but there have been dramatic changes in both the science and practice of training and development. In the current article, we examine the evolution of training and development research using articles published in the Journal of Applied Psychology (JAP) as a primary lens to analyze what we have learned and to identify where future research is needed. We begin by reviewing the timeline of training and development research in JAP from 1918 to the present in order to elucidate the critical trends and advances that define each decade. These trends include the emergence of more theory-driven training research, greater consideration of the role of the trainee and training context, examination of learning that occurs outside the classroom, and understanding training’s impact across different levels of analysis. We then examine in greater detail the evolution of 4 key research themes: training criteria, trainee characteristics, training design and delivery, and the training context. In each area, we describe how the focus of research has shifted over time and highlight important developments. We conclude by offering several ideas for future training and development research.
Noncognitive skills have drawn the interest of psychologists, educators, economists and policymakers over the past 30 years. Despite dissatisfaction with the label “noncognitive skills” that term is still commonly used to describe the construct domain. One reason cited for objecting to the label “noncognitive” is that it implies that the constructs and measures do not entail cognition, a virtual impossibility. I argue this seemingly innocent conflation of “cognition” and “cognitive skills” unknowingly glosses over a fundamental divide in how the subject of psychology is conceptualized and psychological research conducted: The differential and experimental traditions. Where “cognitive skills” originate in the differential tradition and are usually treated as synonymous with psychometric intelligence, “cognition” originates in the experimental tradition and encompasses seemingly all human mental activity. While cognitive skills constitute a variety of cognition not all cognition entails the higher-order, complex mental activity that defines cognitive skills. This seemingly minor conflation suggests that many working in the noncognitive domain do not possess a strong understanding of what cognitive skills are. I provide additional evidence for this assertion and discuss the potentially serious practical consequences of creating and using noncognitive skills assessments without possessing a thorough understanding of cognitive skills.
The current meta-analysis investigated the extent to which personality traits changed as a result of intervention, with the primary focus on clinical interventions. We identified 207 studies that had tracked changes in measures of personality traits during interventions, including true experiments and prepost change designs. Interventions were associated with marked changes in personality trait measures over an average time of 24 weeks (e.g., d = .37). Additional analyses showed that the increases replicated across experimental and nonexperimental designs, for nonclinical interventions, and persisted in longitudinal follow-ups of samples beyond the course of intervention. Emotional stability was the primary trait domain showing changes as a result of therapy, followed by extraversion. The type of therapy employed was not strongly associated with the amount of change in personality traits. Patients presenting with anxiety disorders changed the most, and patients being treated for substance use changed the least. The relevance of the results for theory and social policy are discussed.
Americans have long recognized that investments in public education contribute to the common good, enhancing national prosperity and supporting stable families, neighborhoods, and communities. Education is even more critical today, in the face of economic, environmental, and social challenges. Today's children can meet future challenges if their schooling and informal learning activities prepare them for adult roles as citizens, employees, managers, parents, volunteers, and entrepreneurs. To achieve their full potential as adults, young people need to develop a range of skills and knowledge that facilitate mastery and application of English, mathematics, and other school subjects. At the same time, business and political leaders are increasingly asking schools to develop skills such as problem solving, critical thinking, communication, collaboration, and self-management - often referred to as "21st century skills." Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century describes this important set of key skills that increase deeper learning, college and career readiness, student-centered learning, and higher order thinking. These labels include both cognitive and non-cognitive skills- such as critical thinking, problem solving, collaboration, effective communication, motivation, persistence, and learning to learn. 21st century skills also include creativity, innovation, and ethics that are important to later success and may be developed in formal or informal learning environments. This report also describes how these skills relate to each other and to more traditional academic skills and content in the key disciplines of reading, mathematics, and science. Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century summarizes the findings of the research that investigates the importance of such skills to success in education, work, and other areas of adult responsibility and that demonstrates the importance of developing these skills in K-16 education. In this report, features related to learning these skills are identified, which include teacher professional development, curriculum, assessment, after-school and out-of-school programs, and informal learning centers such as exhibits and museums. © 2012 by the National Academy of Sciences. All rights reserved.
Objective: Theory and research have emphasized the impact of life events on personality trait change. In this article, we review prospective research on personality trait change in response to nine major life events in the broader domains of love and work. Method: We expected to find that life events lead to personality trait change to the extent that they have a lasting influence on individuals' thoughts, feelings, and behavior. Moreover, we predicted that love-related life events such as marriage or parenthood would be more strongly related to changes in traits that emphasize affective content, whereas work-related life events would be more likely to lead to change in traits that reflect behavioral or cognitive content. Results: The current state of research provided some evidence that life events can lead to changes in personality traits and that different life events may be differently related to specific trait domains. Conclusions: A more general conclusion emerging from this review is that the evidence for the nature, shape, and timing of personality trait change in response to life events is still preliminary. We discuss the implications of the results for theory and research and provide directions for future studies on life events and personality trait change.