ArticlePDF Available

Abstract and Figures

Technological advances in assessment have radically changed the landscape of employee selection. This paper focuses on three areas where the promise of those technological changes remains undelivered. First, while new ways of measuring constructs are being implemented, new constructs are not being assessed, nor is it always clear what constructs the new ways are measuring. Second, while technology in assessment leads to much greater efficiency, there are also untested assumptions about effectiveness and fairness. There is little consideration of potential negative byproducts of contextual enhancement, removing human judges, and collecting more data. Third, there has been insufficient consideration of the changed nature of work due to technology when assessing candidates. Virtuality, contingent work arrangements, automation, transparency, and globalization should all be having greater impact on selection assessment design. A critique of the current state of affairs is offered and illustrations of future directions with regard to each aspect is provided.
Content may be subject to copyright.
The Unrealized Potential of Technology in Selection Assessment
Ann Marie Ryan
Michigan State University
Eva Derous
Ghent University
Author Note
Correspondence may be directed to Ann Marie Ryan, Department of Psychology,
Michigan State University, 316 Physics Drive, East Lansing, MI 48824.
Keywords: selection, assessment, technology
Technological advances in assessment have radically changed the landscape of employee
selection. This paper focuses on three areas where the promise of those technological changes
remains undelivered. First, while new ways of measuring constructs are being implemented,
new constructs are not being assessed, nor is it always clear what constructs the new ways are
measuring. Second, while technology in assessment leads to much greater efficiency, there are
also untested assumptions about effectiveness and fairness. There is little consideration of
potential negative byproducts of contextual enhancement, removing human judges, and
collecting more data. Third, there has been insufficient consideration of the changed nature of
work due to technology when assessing candidates. Virtuality, contingent work arrangements,
automation, transparency, and globalization should all be having greater impact on selection
assessment design. A critique of the current state of affairs is offered and illustrations of future
directions with regard to each aspect is provided.
Keywords: selection, assessment, technology
The Unrealized Potential of Technology in Selection Assessment
Over the past two decades, assessment for employee selection has undergone radical
changes due to technological advances (see Tippins, 2015, for a review). Examples include
changes in assessment delivery (computerized, online, mobile, use of adaptive testing), changes
in assessment content (greater use of video and audio, graphics, gamification), changes in
interactivity (recording of applicants, video interviewing), changes in scoring and reporting
(quicker, internet delivered), and changes in many other aspects (e.g., use of algorithms and data
mining). The purpose of this paper is not to provide another review of these advances as quality
treatments exist (Scott, Bartram, & Reynolds, 2017; Tippins, 2015) but to discuss 3 ways
incorporation and use of technological advances in assessment practice and research could be
First, we discuss the need to think more creatively regarding what is measured. That is,
despite the promise of technology as a means of measuring new KSAOs (i.e., knowledge, skills,
abilities, and other constructs like motivation) and richer content, many of our assessments are
just “transported” traditional tools. As an example, gamified assessments (Bhatia & Ryan, 2018)
most often are measuring traditional constructs in environments with game elements layered on.
Second, we discuss the need to move our focus from efficiency to effectiveness and
fairness in thinking about the use of technology (a point raised by Ryan & Derous, 2016). As
examples, the use of adaptive testing, mobile testing, and video interviewing have all been
undertaken because they make the hiring process more efficient (for both employer and
applicant), but the investigation of factors that impact their effectiveness is still quite nascent.
Similarly, the use of algorithms and social network information in hiring (e.g., Stoughton,
Thompson, & Meade, 2015) has wide appeal as an efficient means of screening, but the evidence
suggests these methods are not always more effective than more traditional approaches to
assessment of the same constructs.
Finally, the need to adapt assessments for how technology has changed the nature of
work should be a consideration in our exploration of new approaches. For example, how are we
thinking about assessment differently due to the changing nature of work arrangements to shorter
engagements and contract work (Spreitzer, Cameron, & Garrett, 2017)? Are we assessing the
skills needed for virtual workplaces (e.g., Schulze, Schultze, West, & Krumm, 2017)? How has
the shift to online talent pools and talent badges changed the “business model” underlying the
assessment industry (Chamorro-Premuzic, Winsborough, Sherman, & Hogan, 2016)? What are
validation strategies that work with these new models? These changes are fueled by changes in
the nature of work and employment relationships because of technology; our assessment research
and practice also needs to consider how technology has changed work, not just how it has
changed assessment.
New Construct Measurement
Much has been written about how technology has changed the way we assess individuals
in hiring contexts (Reynolds & Dickter, 2017; Scott & Lezotte, 2012; Tippins, 2015). Some of
the advances and advantages noted have been the efficiencies and cost savings that accompany
web-based delivery of assessments and adaptive testing technologies (see Scott et al., 2017, for a
book-length discussion). Technological advances are also credited with enhancing the candidate
experience through providing more convenient and engaging assessments (e.g., use of video,
However, of all the promises of technology promoted in the past decade, we would argue
the one which has been under-delivered is that of harnessing technology to assess new and
different attributes. The anticipation was that these technological changes in delivery and the use
of greater context would open the door for measuring things that could not be easily measured
before. We would argue that while new ways of measuring constructs are being implemented,
new constructs are not being measured. Further in some cases, new ways of measuring are
implemented but what constructs are being measured is unclear.
New ways of more easily assessing established content include using drag-and-drop
matching items, image matching, and hotspot items (e.g., pointing to location of photo), using
mobile functionality (e.g., swiping), recording video answers to interview or SJT questions, and
scraping social media or other data sources (Dickter, Jockin, & Delany, 2017; Kantrowitz &
Gutierrez, 2018). These new ways, however, are typically employed in service of assessing
traditional constructs (e.g., verbal skills, quantitative reasoning). Adler, Boyce, and Caputo,
(2018) note the majority of internet-delivered cognitive ability tests ask the traditional multiple
choice questions on traditional content (e.g., chart and paragraph interpretations). Indeed,
Chamorro-Premuzic et al. (2016) discuss that “new talent signals” such as social media and big
data are often still looking at the same essential attributes of inter- and intra-personal
competencies, abilities, and willingness to work hard. As another example, serious games or
gamified assessments are often described as novel, but many are not assessing new constructs,
just multiple traditional constructs simultaneously (with the concomitant concerns then about the
ability to assess many things well in limited time periods). As Bhatia and Ryan (2018) noted,
there is very little unpublished or published research on the validity evidence in support of games
and gamified assessments in selection, particularly on construct validity, something that needs to
be addressed.
Behaviors such as mouse-over hover times, response latencies, eye tracking,
measurement of facial micro-expressions, and biometric sensors assessing emotions are all being
implemented (Reynolds & Dickter, 2017). The question of the construct validity of these
measures seems to receive a more superficial treatment. Have we thought about what they
signify or why they might relate to valued criteria? That is, the constructs being assessed need to
be more clearly specified.
What are some examples of a new “what” that could be assessed? Adler et al. (2018)
discuss leveraging the interactive capabilities of technology to assess the rate of speed in
acquiring new knowledge (i.e., learning agility), but note that this has not been done yet to
efficient levels that would fit a hiring context. As a second example they note that natural
language processing technologies and capabilities to interpret visual input could lead to new
assessments of personality of a more projective nature (i.e., assessments that involve responses to
ambiguous or unstructured stimuli), tapping into less conscious motivations and tendencies, but
as yet these have not been adopted for wide use. An example of where steps are being taken to
measure new constructs is in assessing emotions, such as through applications of databases of
micro-expressions (Yan, Wang, Liu, Wu, & Fu, 2014) and speech patterns into digital
interviewing. Yet, even here, the detection of micro-expression and speech mining are discussed
as an advance, but that is a focus on the “how” (i.e., tools or methods to assess) and not the
“what” (i.e., constructs to be assessed). Tying these tools or methods more clearly to job-
relevant constructs (e.g., specific personality traits, emotional regulation concepts) is important
for the purpose of validation, explainability, and acceptability.
Realizing the potential for selection assessment to add something new to what we already
assess in hiring should start with the development of predictive hypotheses based on job analytic
information (Guion, 2011). That is, what KSAOs are important to work outcomes that we have
not historically assessed because it was deemed too difficult and too time-consuming? This is
where artificial intelligence (AI) might be leveraged in a less atheoretical fashion for assessing
prior behavior from existing records. As an example, the field of learning analytics has led those
in higher education to track all kinds of student behaviors (e.g., class attendance, course-taking
patterns, performance on varied types of assessments). Could such data be used to go beyond
GPA as an indicator of past learning ability and allow for assessment of more specific learning
capabilities and motivational constructs not typically assessed in selection contexts? Another
example would be to drill down into our job analytic results to specify what dynamic capabilities
are required for a given job (e.g., adaptation to certain types of changes or events) and leverage
the interactive nature of technology-enabled assessments to measure these more contextualized
types of adaptive behavior.
As our next section discusses in more detail, are our technology-enhanced assessments
better than what we had before or just different? Perhaps we are not assessing new things, but
that the same old things are now assessed in more efficient (and effective) ways.
Enhanced Efficiency, Effectiveness, and Fairness
There is certainly ample evidence that technology has increased efficiency through
increasing ease of access, shortening times of assessment and scoring, allowing greater use of
multiple assessment types, reducing costs in delivery and scoring, and other process
improvements (Adler et al., 2018). One early promise of technology was the ability to increase
reliability and validity in assessment. A closer look as to whether this is being fully realized is
warranted. We illustrate this with discussions of several assumptions that are unwarranted: 1) as
context and richness increases, validity increases, 2) as efficiency in delivery increases,
effectiveness is unharmed, 3) as people have less of a role in the process of assessing, validity
and fairness increases, and 4) as the quantity of data and information considered increases,
validity and fairness increases.
Effects of Contextual Enhancement
One assumption associated with technologically-enhanced assessment is that creating
more realistic items will more closely mimic what actually occurs at work and will therefore
improve the accuracy of our measurement. However, when we transform our items from written
descriptions to video items or other media rich depictions, we may actually be adding more noise
and even systematic error (Hawkes, Cek, & Handler, 2018).
As an example, video-SJTs are viewed as an improvement in effectiveness over written
SJTs because of the decreased reliance on reading comprehension as well as increased
engagement of test takers (Chan & Schmitt, 1997; Jones & DeCotiis, 1986; Lievens & Sackett,
2006). However, video SJTs can have an additional source of non-construct relevant variance in
scores, as actors in videos vary in gender, ethnicity, age and other factors. Individuals process
demographic cues quickly and automatically (Ito & Urland, 2003), and this information can
influence behavioral responses (e.g., Avery, McKay, & Wilson, 2008; Eagly & Crowley, 1986;
Kunstman & Plant, 2008; Perkins, Thomas, & Taylor, 2000; Russell & Owens, 2001). Research
on video SJTs has found that Black respondents perform better on an SJT when the videos
include Black actors (Golubovich & Ryan, 2012) and White respondents have been found to
react less favorably to a hypothetical organization after viewing its SJT videos featuring Black
actors (Golubovich & Ryan, 2013).
These studies provide an example of how adding context or “situation” to our measures
of “person” have effects. Besides how actors and avatars look, one can imagine effects for all
the nonverbal information that is conveyed in these scenarios. The question is whether the
technology advance over a written text for a question increases accuracy by assessing behavior in
context or adds information that affects measurement accuracy in negative ways. For example,
virtual reality (VR) can certainly transform assessments to feel more “real” for participants
(Reynolds & Dickter, 2017), but is a VR simulation a better measure of the targeted constructs?
Adler et al. (2018) suggest that a VR simulation of public speaking with a virtual audience might
be a better assessment than a more low-fidelity simulation of these skills; perhaps, but we do not
know. We would need to draw on existing research regarding audience presence to develop
clear hypotheses and test them.
As another example, video-resumes can provide more personalized, jobrelevant
information for use in screening, which should in theory result in less bias in information
processing. However, these methods also provide more social category information than
traditional resumes and more non-job related information, which can creates a cognitive
challenge for a rater (Apers & Derous, 2017). The category cues dominate our perceptual
systems and attract our attention and hence have a greater probability of being processed more
deeply (Kulik, Roberson, & Perry, 2007).
As a final example, Arthur, Doverspike, Kinney, and O’Connell (2017) provide a strong
note of caution regarding game-thinking in selection contexts, pointing out that job candidates
are likely already highly motivated and so the enhanced engagement which game elements are
meant to deliver may not have appreciable effects. Further, game mechanics related to the value
of feedback may have negative consequences such as increased anxiety (Arthur et al., 2017).
Hawkes et al. (2018) go so far as to note that the reliability of game assessments may be affected
negatively by practice effects such as those seen in the gaming world or by adding noisy variance
associated with hand-eye coordination and mouse control. One direction for the future will be to
show that all of this effort focused on enhancing fidelity and engagement adds something in
terms of validity and that something added is not more error in measurement.
Effects of Efficiency
We should not assume that moving to a more “advanced” technology necessarily
increases effectiveness even if it increases efficiency. This is clearly evident in the trend toward
shorter assessments (Hardy, Gibson, Sloan, & Carr, 2017; Kruyen, Emons, & Sijtsma, 2012,
2013) where efficiency gains may result in reliability and validity decrements.
One stream of research focused on how more efficient technology advancements may
distort what is measured is that on assessment mode equivalence. There is ample research
available to indicate that some measures are equivalent across modes (non-cognitive tests; see
review by Tippins, 2015) and others are not (e.g., speeded cognitive tests; King, Ryan,
Kantrowitz, Grelle, & Dainis, 2015; Mead & Drasgow, 1993). For example, non-cognitive
measures may be equivalent when moving from PC to mobile assessment but cognitive measures
and SJTs might not, depending on features like scrolling etc… (see Arthur et al., 2017; King et
al., 2015 for comparisons). Morelli, Potosky, Arthur, and Tippins (2017) note that “reactive
equivalence” studies comparing assessment modes are not theoretically (or even practically)
informative as they do not address why mode differences occur or the reasons for the construct
non-equivalence. Frameworks such as Potosky (2008) and Arthur, Keiser, and Doverspike
(2018) indicate ways in which different assessment delivery devices might differ; these need to
be expanded and tested to more systematically understand whether and why more efficient
delivery of assessments might be a more rather than less accurate way to measure (see Apers &
Derous, 2017, for an example in the context of resume screening). Morelli et al. (2017) make a
strong case for better theory-based predictions that go beyond considering variance associated
with technology use as “construct irrelevant.We echo their general call for elevating efforts
regarding construct specification and evaluating technological feature influence in a more
considered fashion.
As another example of this, while the potential for greater accuracy in measurement lies
in use of adaptive testing, many computerized tests are not adaptive (Adler et al., 2018). Wider
adoption will occur when we can scale up item banks more quickly and refresh them without
onerous effort those in educational testing are very much experimenting with ways to create
item clones and variations more efficiently (e.g., using automatic item generation; see Drasgow
& Olson-Buchanan, 2018; Kantrowitz & Gutierrez, 2018). Effectiveness of adaptive methods
can be severely hampered by small item pools that lack adequate numbers of items at certain trait
or ability levels, a not uncommon occurrence. As a final example, the use of an open badges
system in Belgium (Derous, 2019) significantly shortened recruitment and selection procedures
by allowing candidates with badges (for having passed assessments with another organization) to
bypass retesting. Time of both the employer and applicant are saved, and in this case there is no
observable negative impact on quality. Taken together, efficiency of assessment can mean more
valid assessment, but it can also mean faster and cheaper but not better: it is important to
recognize and address which is occurring.
Effects of Removing Judgment
Another broad assumption is that technology increases the accuracy of measurement as
we pull humans out of the process. That is, by removing human administration, scoring, and
judgment, a reduction in error in measurement is expected (e.g., structuring interviews can
improve reliability and validity). However, this is not always the case. For example, recently
Facebook was questioned regarding bias against women as job ads were targeted toward certain
demographic groups and not others (Scheiber, 2018; see also Datta, Tschantz, & Datta, 2015,
and Sweeney, 2013, for similar examples). Similarly, news stories regarding Amazon’s
abandonment of AI screening of resumes because of biases (Meyer, 2018) as well as questions
regarding video interview technology and bias continue (see Buolamwini, 2018; McIllvaine,
2018). Finally, a number of researchers have noted the potential biases that may emerge in using
social media screening of job candidates, leading to both perceived unfairness (Stoughton et al.,
2015) and actual discrimination (Van Iddekinge, Lanivich, Roth, & Junco, 2016). Because
existing databases can capture historical biases, the need to consider how technological advances
may tap into those biases needs to be at the forefront of assessment design considerations, rather
than assuming that the computer is a better judge (e.g., by training the algorithm to make non-
discriminatory decisions; Ajunwa, 2016).
As another example, Cascio and Montealegre (2016) noted that we might want to
consider the changes to the role of the recruiter due to constant connectivity. Research focused
on greater efficiency in reaching candidates has not always considered concurrent positive and
negative effects on the internal organizational members such as recruiters, interviewers and
selection system administrators. Rather than assuming that technology has made organizational
members’ lives easier, we might consider how their efficiency and effectiveness is both
enhanced and burdened by technological innovations in assessment (e.g., candidates are
processed more efficiently but the greater volume negates any gains in work time for recruiters).
The transfer of assessment tasks and roles to technology, as with any other automation in
the workplace, theoretically removes the low skills, tedious parts of the job and frees people up
to focus on the more creative tasks and the interpersonal elements of the work (Brynjolfsson &
McAfee, 2014). We need to attend to ways this is true for those involved in assessment as well as
counterinfluences that decrease effectiveness in other aspects of assessment professionals’ work.
Effects of More Data
The research on text analysis, natural language processing and social media scraping
similarly raises questions regarding the effectiveness of efficient methods. While unobtrusive
measurement has many advantages in terms of ease and efficiency (and many potential concerns
regarding privacy and information control), a key for assessment experts is always validity of
inferences. As Dickter et al. (2017) note, data science experts do not approach modeling
considering the concept of construct validity.
Research on validity of social media data has been mixed. Connections to job
performance have been found in several studies, e.g., based on Facebook ratings (e.g., Kluemper,
Rosen, & Mossholder, 2012; Kluemper & Rosen, 2009) and more recently- also based on
LinkedIn ratings (e.g., Roulin & Levashina, 2018). Van Iddekinge et al. (2016), however,
sounded a strong cautionary note, as they found that recruiter ratings of Facebook pages were
unrelated to job performance or turnover and also had adverse impact (i.e., lower scores or
selection rates for underrepresented groups; Guion, 2011). This line of research on validity and
social media scraping is a good illustration of the work that needs to be done to build our
understanding of when, where, why and how technology-enabled assessment tools are effective
and when they do not fulfill their potential.
Indeed, as we alluded to earlier, the notion that bias is reduced and fairness perceptions
increased when one relies on larger quantities of data has been shown to be a fallacy (see for
example, Caliskan, Bryson, & Narayanan, 2017). Algorithms can and do build in existing biases
when the data they are trained on may have been derived from a biased judgment process and/or
biased labor market. The lack of an ability to articulate to job candidates exactly what is being
assessed and why makes big data use in selection decision-making a challenge from a fairness
perceptions perspective as well.
Our examples in these last few sections should not be construed as indicating that
technology-enabled assessment is bound to be less effective than traditional methods; indeed, we
believe that validity and fairness can be enhanced through technology. Rather, we seek to
emphasize the importance of not conflating efficiency and effectiveness when discussing the
value of technology in assessment, and the need for more thoughtful, theory-driven examinations
of when effectiveness is likely to be enhanced and when it is not.
Changing Nature of Work Requires Changing Assessments
Besides focusing on how technology changes assessments, we should focus on how
technological changes to work itself should lead us to change what is assessed. To illustrate, we
discuss how the virtuality, contingency, automation, transparency, and globalization of work (in
some sense all byproducts of technological change) should be impacting selection assessment.
Discussions on the changing nature of work often focus on how individuals are much
more likely to be working remotely and collaborating virtually (Brawley, 2017; Spreitzer et al.,
2017). The literature on team virtuality has suggested that competencies for virtual collaborative
work and using computer-mediated communication may differ in depth and complexity from
those associated with face-to-face teamwork (see Schulze & Krumm, 2017, for a review; also
Hassell & Cotton, 2017; Schulze et al., 2017). Beyond offering assessments in digital formats,
are assessment designers considering more carefully what is being measured for virtual work?
KSAOs such as awareness of media capabilities, communication style adaptability, and other
“virtual skills” should be a greater focus of assessment developers.
As another example, in reviewing the effects of telecommuting, Allen, Golden, and
Shockley (2015) point out individual differences that relate to successful telecommuting (i.e.,
moderators of productivity as well as social isolation and satisfaction effects such as self-
management skills, personality characteristics, boundary management styles). Assessments
related to capability and satisfaction with greater levels of remote work may be valuable for
hiring contexts with greater remoteness.
Today’s workers are said to be more often employed in short-term engagements (gigs) or
contract work rather than long-term, traditional employee/employer arrangements. The
implications of this increased contingency of work (George & Chattopadhyay, 2017; Spreitzer et
al., 2017) should be given greater consideration. For example, does this mean administering
assessments more often as people move from contract to contract? Or does it imply less
assessing as organizations do not invest in evaluating KSAOs, relying on crowdsourced ratings
and rankings to evaluate talent (Aguinis & Lawal, 2013)? The use of online talent exchanges as
a means of securing employment should be forcing a consideration of what is assessed, as well
as when and how assessment for gigs and short term contracts may look different than that for
longer-term employment relationships. While there is a need for validation work for
determining what works in predicting performance and other outcomes in these arrangements, as
Brawley (2017) notes, we first need better theory and empirical evidence regarding what relates
to the attitudes and behavior of “serious” gig workers.
In one effort at expanding theory regarding contingent workers, Petriglieri, Ashford, and
Wrzesniewski, (2018) have provided a framework for understanding how some individuals are
successful at managing the uncertainty of a freelance career. For example, they discuss how
those who were able to cultivate established routines for their workdays are more effective than
others in managing the greater ambiguity and lack of structure of these careers. Assessments to
help individuals understand the predispositions and competencies required for success as a gig
worker may be useful tools for platforms that seek to connect workers to work, even if they are
not used as selection tools by employers but as self-selection tools used by gig workers.
As automation of jobs increases (Metz, 2018; Scheiber, 2018a; Wingfield, 2017), we
need to consider what else should enter into the selection process. Automation has led to a
decrease in lower skill jobs (Nedelkoska & Quintini, 2018), so perhaps some of the high volume
selection assessments that are a big share of the assessment market (e.g., basic math skills) will
have reduced demand, and the need for assessment of higher level skills (e.g., advanced math
skills) will emerge as a larger focus. In general, we tend to focus our assessment development
efforts on high volume entry level jobs that may be automated in the near future; as work
changes, what is assessed may need to change.
As technology changes jobs, some have also asked what new skills should be assessed.
As an example of one way change can impact assessment, research on the social acceptance of
technology (Gaudiello, Zibetti, Lefort, Chetouani, & Ivaldi, 2016; Seo et al., 2018; Syrdal,
Dautenhahn, Koay, & Walters, 2009) examines how people can collaborate with robots as
coworkers (cobots). The questions as to how the form and level of teamwork skills such as
communication, collaboration, and conflict management might differ when coworkers are
cobots, and what that means for assessment has not been explored.
One might also consider how robotic interfaces within the assessment process might
affect tool validity. For example, in many online assessments, individuals interact with avatars.
One can envision future assessment center role-players that are humanoid robots, freeing up
assessor time and enhancing consistency across candidates. Rather than assuming that interaction
with a “simulated other” is akin to an actual interpersonal interaction, understanding the
acceptance of and skills for interacting with automation can aid us adopt technology
enhancements to assessment processes that consider these issues of social acceptance.
Another example of a workplace trend affecting assessments is the move of organizations
toward greater transparency toward both external and internal stakeholders (Parris, Dapko,
Arnold, & Arnold, 2016). In the selection context, this involves greater transparency regarding
what is being assessed and why. Most of the research related to transparency in selection has
suggested positive effects on validity and candidate experience (e.g., Klehe, König, Richter,
Kleinmann, & Melchers, 2008; Kleinmann, Kuptsch, & Köller, 1996; Kolk, Born, & der Flier,
2003). However, Jacksch and Klehe (2016) demonstrated that transparency’s positive effects are
limited to nonthreatening performance dimensions; that is, transparency can benefit some
candidates and harm others if the attribute being assessed is associated with a negative stereotype
related to the social identity of those being assessed. Langer, König, and Fitili (2018)
demonstrated that providing greater transparency regarding what is assessed in an online
interview by an avatar (e.g., facial expression, gestures, voice pitch) had equivocal effects on
organizational attractiveness, with individuals appreciating the organization’s candor but
simultaneously decreasing their views of the organization overall. Langer et al. (2018) note that
what information and how much information to provide regarding technology-enhanced
assessments is deserving of greater research focus in this time of greater pressures for
The globalization of business also has implications for assessment (Ryan & Ployhart,
2014). Not only does it mean that assessments must be delivered in multiple languages, and that
all the ensuing efforts to ensure psychometric equivalence occur (see International Test
Commission, 2005, for standards), but it means all the associated implementation issues related
to cross-cultural implementation be considered (see Ryan & Tippins, 2012, for a comprehensive
discussion; see Fell, König, & Kammerhoff, 2016, for a specific example of a cross-cultural
differences in faking in interviews; see Ryan & Delany, 2017, for a discussion of recruiting
globally). Also, the need to assess cultural competencies is amplified, like cultural intelligence
(Ang et al., 2007), cultural values (Hofstede, 2001), cultural adjustment (Salgado & Bastida,
2017), and leadership and teamwork in multinational teams (Han & Beyerlein, 2016).
We also note that prior research has indicated there can be cross-cultural differences in
acceptance of and skill in interacting with technology (Dinev, Goo, Hu, & Nam, 2009; Nistor,
Lerche, Weinberger, Ceobanu, & Heymann, 2014). In the selection assessment space, we need
to be more mindful of how technological change in the workplace more broadly is varied across
cultures in both adoption rates and in user acceptance and how that might impact selection
tool effectiveness.
The premise of this paper is that there is a lot of unrealized potential in the incorporation
of technology into selection assessments. Figure 1 summarizes in a schematic where we believe
that the (unrealized) potential of technology in selection assessment is currently situated and
themes from this paper that both researchers and practitioners might focus on. First, as regards
the predictor side of selection assessment, we have noted that advances in delivery, scoring,
interactivity, and reporting have been implemented with a focus on greater efficiency as well as
job applicant engagement and other stakeholder reactions. However, we see the potential for
leveraging technology to assess new constructs as well as to increase validity as not yet fulfilled
and we urge greater focus and energy in these directions. We also have admonished those who
ignore the potential downsides of technology-enhanced assessments such as greater introduction
of construct-irrelevant variance or instances of unfairness to disadvantaged groups. Of course,
technology can and should continue to serve as a means of enhancing assessment efficiency.
Second, we urge selection researchers, assessment developers, and other practitioners to
take a closer look at the criterion side, i.e., how work and the workplace is changing, and seek
ways to better align assessment content (i.e., KSAs measured) as well as practice (e.g.,
transparency) with those changes (see Figure 1). While we have highlighted some of the more
prevalent trends in the changing nature of work (virtuality, automation, contingency,
globalization and transparency), there are likely other emerging changes that also can serve as
inspiration for developing new assessments, leveraging the advantages offered by technology.
The future of technology-enabled assessments is only limited by imagination; we anticipate
continued changes in the way in which assessments are developed, delivered, and scored. More
importantly, we hope that the next decade will have a greater focus on the areas we have outlined
in this paper as needing attention, as this would ensure the realization of the full potential of
technology-enabled assessments.
Adler, S., Boyce, A. S., & Caputo, P. M. (2018). Employment testing. In J. C. Scott, D. Bartram,
& D. H. Reynolds (Eds.), Next generation technology-enhanced assessment: Global
perspectives on occupational and workplace testing (pp. 335). Cambridge, UK:
Cambridge University Press.
Aguinis, H., & Lawal, S. O. (2013). eLancing: A review and research agenda for bridging the
sciencepractice gap. Human Resource Management Review, 23(1), 617.
Ajunwa, I. (2016). Hiring by algorithm. SSRN Electronic Journal. doi:10.2139/ssrn.2746078
Allen, T. D., Golden, T. D., & Shockley, K. M. (2015). How effective is telecommuting?
Assessing the status of our scientific findings. Psychological Science in the Public Interest,
16(2), 4068. doi:10.1177/1529100615593273
Ang, S., Van Dyne, L., Koh, C., Ng, K., Templer, K., Tay, C., & Chandrasekar, N. (2007).
Cultural intelligence: Its measurement and effects on cultural judgment and decision
making, cultural adaptation, and task performance. Management and Organization Review,
3, 335371. doi:10.1111/j.1740-8784.2007.00082.x
Apers, C., & Derous, E. (2017). Are they accurate? Recruiters’ personality judgments in paper
versus video resumes. Computers in Human Behavior, 73, 919.
Arthur, W. J., Doverspike, D., Kinney, T. B., & O’Connell, M. (2017). The impact of emerging
technologies on selection models and research: Mobile devices and gamification as
exemplars. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (2nd ed.,
pp. 967986). New York: Routledge.
Arthur, W., Keiser, N. L., & Doverspike, D. (2018). An information-processing-based
conceptual framework of the effects of unproctored internet-based testing devices on scores
on employment-related assessments and tests. Human Performance, 31(1), 132.
Avery, D. R., McKay, P. F., & Wilson, D. C. (2008). What are the odds? How demographic
similarity affects the prevalence of perceived employment discrimination. Journal of
Applied Psychology, 93(2), 235249. doi:10.1037/0021-9010.93.2.235
Bhatia, S., & Ryan, A. M. (2018). Hiring for the win: Game-based assessment in employee
selection. In J. H. Dulebohn & D. L. Stone (Eds.), The brave new world of eHRM 2.0 (pp.
81110). Charlotte, NC: Information Age Publishing.
Brawley, A. M. (2017). The Big, Gig Picture: We Can’t Assume the Same Constructs Matter.
Industrial and Organizational Psychology, 10(04), 687696. doi:10.1017/iop.2017.77
Brynjolfsson, E., & McAfee, A. (2014). The second machine age: Work, progress, and
prosperity in a time of brilliant technologies. New York: W.W. Norton.
Buolamwini, J. (2018). When the robot doesn’t see dark skin. NY Times. Retrieved from
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from
language corpora contain human-like biases. Science, 356(6334), 183186.
Cascio, W. F., & Montealegre, R. (2016). How technology is changing work and organizations.
Annual Review of Organizational Psychology and Organizational Behavior, 3(1), 349375.
Chamorro-Premuzic, T., Winsborough, D., Sherman, R. A., & Hogan, R. (2016). New Talent
Signals: Shiny New Objects or a Brave New World? Industrial and Organizational
Psychology, 9(03), 621640. doi:10.1017/iop.2016.6
Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in
situational judgment tests: Subgroup differences in test performance and face validity
perceptions. Journal of Applied Psychology, 82(1), 143159. doi:10.1037/0021-
Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated experiments on ad privacy settings.
Proceedings on Privacy Enhancing Technologies, 2015(1). doi:10.1515/popets-2015-0007
Derous, E. (2019). Van boe-boe machine tot sociale media: Evidence-based werven en selecteren
[From reaction time tests to social media: Evidence-based recruitment and selection]. In J.
Valk & L. Lopes de Leao Laguna (Eds.), HRM heden en morgen: Evidence-based practice
& Practice-based evidence (pp. 125). Amsterdam: Vakmedianet Management.
Dickter, D. N., Jockin, V., & Delany, T. (2017). The evolution of E-selection. In The Wiley
Blackwell Handbook of the Psychology of the Internet at Work (pp. 257283). Chichester,
UK: John Wiley & Sons, Ltd. doi:10.1002/9781119256151.ch13
Dinev, T., Goo, J., Hu, Q., & Nam, K. (2009). User behaviour towards protective information
technologies: The role of national cultural differences. Information Systems Journal, 19(4),
391412. doi:10.1111/j.1365-2575.2007.00289.x
Drasgow, F., & Olson-Buchanan, J. B. (2018). Technology-driven developments in
psychometrics. In J. C. Scott, D. Bartram, & D. H. Reynolds (Eds.), Next generation
technology-enhanced assessment: Global perspectives on occupational and workplace
testing. (pp. 239264). Cambridge, UK: Cambridge University Press.
Eagly, A. H., & Crowley, M. (1986). Gender and helping behavior: A meta-analytic review of
the social psychological literature. Psychological Bulletin, 100(3), 283308.
Fell, C. B., König, C. J., & Kammerhoff, J. (2016). Cross-cultural differences in the attitude
toward applicants’ faking in job interviews. Journal of Business and Psychology, 31(1), 65
85. doi:10.1007/s10869-015-9407-8
Gaudiello, I., Zibetti, E., Lefort, S., Chetouani, M., & Ivaldi, S. (2016). Trust as indicator of
robot functional and social acceptance. An experimental study on user conformation to iCub
answers. Computers in Human Behavior, 61, 633655. doi:10.1016/j.chb.2016.03.057
George, E., & Chattopadhyay, P. (2017). Understanding nonstandard work arrangements: Using
research to inform practice. SHRM-SIOP Science of HR Series. Retrieved from
Golubovich, J., & Ryan, A. M. (2012). Demographic cues in video-based situational judgment
items. Symposium presented at the Annual Meeting of the Society for Industrial and
Organizational Psychology. San Diego, CA.
Golubovich, J., & Ryan, A. M. (2013). Demographic cues in video-based situational judgment
items: An extension. Poster presented at the annual meeting of the Society for Industrial
and Organizational Psychology. Houston, TX.
Guion, R. M. (2011). Assessment, measurement, and prediction for personnel decisions. New
York: Routledge.
Han, S. J., & Beyerlein, M. (2016). Framing the effects of nultinational cultural diversity on
virtual team processes. Small Group Research, 47(4), 351383.
Hardy, J. H., Gibson, C., Sloan, M., & Carr, A. (2017). Are applicants more likely to quit longer
assessments? Examining the effect of assessment length on applicant attrition behavior.
Journal of Applied Psychology, 102(7), 11481158. doi:10.1037/apl0000213
Hassell, M. D., & Cotton, J. L. (2017). Some things are better left unseen: Toward more effective
communication and team performance in video-mediated interactions. Computers in Human
Behavior, 73, 200208. doi:10.1016/j.chb.2017.03.039
Hawkes, B., Cek, I., & Handler, C. (2018). The gamification of employee selection tools: An
exploration of viability, utility, and future directions. In J. C. Scott, D. Bartram, & D. H.
Reynolds (Eds.), Next generation technology-enhanced assessment: Global perspectives on
occupational and workplace testing. (pp. 288316). Cambridge, UK: Cambridge University
Hofstede, G. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and
organizations across nations. Thousand Oaks, CA: Sage Publications.
International Test Commission. (2005). Guidelines on computer-based and internet delivered
testing. Retrieved from
Ito, T. A., & Urland, G. R. (2003). Race and gender on the brain: Electrocortical measures of
attention to the race and gender of multiply categorizable individuals. Journal of
Personality and Social Psychology, 85(4), 616626. doi:10.1037/0022-3514.85.4.616
Jacksch, V., & Klehe, U.-C. (2016). Unintended consequences of transparency during personnel
selection: Benefitting some candidates, but harming others? International Journal of
Selection and Assessment, 24(1), 413. doi:10.1111/ijsa.12124
Jones, C., & DeCotiis, T. A. (1986). Video-Assisted Selection of Hospitality Employees. Cornell
Hotel and Restaurant Administration Quarterly, 27(2), 6773.
Kantrowitz, T. M., & Gutierrez, S. L. (2018). The changing landscape of technology-enhanced
test administration. In J. C. Scott, D. Bartram, & D. H. Reynolds (Eds.), Next generation
technology-enhanced assessment: Global perspectives on occupational and workplace
testing (pp. 1932015). Cambridge, UK: Cambridge University Press.
King, D. D., Ryan, A. M., Kantrowitz, T., Grelle, D., & Dainis, A. (2015). Mobile internet
testing: An analysis of equivalence, individual differences, and reactions. International
Journal of Selection and Assessment, 23(4), 382394. doi:10.1111/ijsa.12122
Klehe, U.-C., König, C. J., Richter, G. M., Kleinmann, M., & Melchers, K. G. (2008).
Transparency in structured interviews: Consequences for construct and criterion-related
validity. Human Performance, 21(2), 107137. doi:10.1080/08959280801917636
Kleinmann, M., Kuptsch, C., & Köller, O. (1996). Transparency: A necessary requirement for
the construct validity of assessment centres. Applied Psychology, 45(1), 6784.
Kluemper, D. H., & Rosen, P. A. (2009). Future employment selection methods: evaluating
social networking web sites. Journal of Managerial Psychology, 24(6), 567580.
Kluemper, D. H., Rosen, P. A., & Mossholder, K. W. (2012). Social networking websites,
personality ratings, and the organizational context: More than meets the eye? Journal of
Applied Social Psychology, 42(5), 11431172. doi:10.1111/j.1559-1816.2011.00881.x
Kolk, N. J., Born, M. P., & der Flier, H. van. (2003). The transparent assessment centre: The
effects of revealing dimensions to candidates. Applied Psychology, 52(4), 648668.
Kruyen, P. M., Emons, W. H. M., & Sijtsma, K. (2012). Test length and decision quality in
personnel selection: When is short too short? International Journal of Testing, 12(4), 321
344. doi:10.1080/15305058.2011.643517
Kruyen, P. M., Emons, W. H. M., & Sijtsma, K. (2013). On the shortcomings of shortened tests:
A literature review. International Journal of Testing, 13(3), 223248.
Kulik, C. T., Roberson, L., & Perry, E. L. (2007). The multiple-category problem: Category
activation and inhibition in the hiring process. Academy of Management Review, 32(2),
529548. doi:10.5465/AMR.2007.24351855
Kunstman, J. W., & Plant, E. A. (2008). Racing to help: Racial bias in high emergency helping
situations. Journal of Personality and Social Psychology, 95(6), 14991510.
Langer, M., König, C. J., & Fitili, A. (2018). Information as a double-edged sword: The role of
computer experience and information on applicant reactions towards novel technologies for
personnel selection. Computers in Human Behavior, 81, 1930.
Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: A
comparison in terms of predictive validity. Journal of Applied Psychology, 91(5), 1181
1188. doi:10.1037/0021-9010.91.5.1181
McIllvaine, A. R. (2018). In the fight against bias, AI faces backlash. Retrieved from
Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil
cognitive ability tests: A meta-analysis. Psychological Bulletin, 114(3), 449458.
Metz, C. (2018). FedEx follows Amazon into the robotic future. New York Times. Retrieved
Meyer, D. (2018). Amazon reportedly killed an AI recruitment system because it couldn’t stop
the tool from discriminating against women. Retrieved from
Morelli, N., Potosky, D., Arthur, W., & Tippins, N. (2017). A call for conceptual models of
technology in I-O Psychology: An example from technology-based talent assessment.
Industrial and Organizational Psychology, 10(04), 634653. doi:10.1017/iop.2017.70
Nedelkoska, L., & Quintini, G. (2018). Automation, skills use and training. (No. No. 202). Paris.
Nistor, N., Lerche, T., Weinberger, A., Ceobanu, C., & Heymann, O. (2014). Towards the
integration of culture into the Unified Theory of Acceptance and Use of Technology. British
Journal of Educational Technology, 45(1), 3655. doi:10.1111/j.1467-8535.2012.01383.x
Parris, D. L., Dapko, J. L., Arnold, R. W., & Arnold, D. (2016). Exploring transparency: A new
framework for responsible business management. Management Decision, 54(1), 222247.
Perkins, L. A., Thomas, K. M., & Taylor, G. A. (2000). Advertising and recruitment: Marketing
to minorities. Psychology and Marketing, 17(3), 235255.
Petriglieri, G., Ashford, S. J., & Wrzesniewski, A. (2018). Thriving in the gig economy. Harvard
Business Review, 140143. Retrieved from
Potosky, D. (2008). A conceptual framework for the role of the administration medium in the
personnel assessment process. Academy of Management Review, 33(3), 629648.
Reynolds, D., & Dickter, D. N. (2017). Technology and employee selection: An overview. In J.
L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (2nd ed., pp. 855873).
New York, NY: Routledge.
Roulin, N., & Levashina, J. (2018). LinkedIn as a new selection method: Psychometric
properties and assessment approach. Personnel Psychology. doi:10.1111/peps.12296
Russell, A., & Owens, L. (2001). Peer estimates of school-aged boys’ and girls' aggression to
same- and cross-sex targets. Social Development, 8(3), 364379. doi:10.1111/1467-
Ryan, A. M., & Delany, T. (2017). Attracting job candidates to organizations. In J. Farr & N.
Tippins (Eds.), Handbook of employee selection (2nd ed., pp. 165181). Routledge.
Ryan, A. M., & Derous, E. (2016). Highlighting tensions in recruitment and selection research
and practice. International Journal of Selection and Assessment, 24(1), 5462.
Ryan, A. M., & Ployhart, R. E. (2014). A Century of selection. Annual Review of Psychology,
65(1), 693717. doi:10.1146/annurev-psych-010213-115134
Ryan, A. M., & Tippins, N. (2012). Designing and implementing global selection systems.
Oxford, UK: Wiley-Blackwell.
Salgado, J. F., & Bastida, M. (2017). Predicting expatriate effectiveness: The role of personality,
cross-cultural adjustment, and organizational support. International Journal of Selection
and Assessment, 25(3), 267275. doi:10.1111/ijsa.12178
Scheiber, N. (2018a). High-skilled white-collar work? Machines can do that, too. New York
Times. Retrieved from
Scheiber, N. (2018b, September 18). Facebook accused of allowing bias against women in job
ads. NY Times. Retrieved from
Schulze, J., & Krumm, S. (2017). The “virtual team player.” Organizational Psychology Review,
7(1), 6695. doi:10.1177/2041386616675522
Schulze, J., Schultze, M., West, S. G., & Krumm, S. (2017). The knowledge, skills, abilities, and
other characteristics required for face-to-face versus computer-mediated communication:
Similar or distinct constructs? Journal of Business and Psychology, 32(3), 283300.
Scott, J. C., Bartram, D., & Reynolds, D. H. (2018). Next Generation Technology-Enhanced
Assessment. (J. C. Scott, D. Bartram, & D. H. Reynolds, Eds.). Cambridge, UK: Cambridge
University Press.
Scott, J. C., & Lezotte, D. V. (2012). Web-Based Assessments. Oxford University Press.
Seo, S. H., Griffin, K., Young, J. E., Bunt, A., Prentice, S., & Loureiro-Rodríguez, V. (2018).
Investigating people’s rapport Building and hindering behaviors when working with a
collaborative robot. International Journal of Social Robotics, 10(1), 147161.
Spreitzer, G. M., Cameron, L., & Garrett, L. (2017). Alternative work arrangements: Two
images of the new world of work. Annual Review of Organizational Psychology and
Organizational Behavior, 4(1), 473499. doi:10.1146/annurev-orgpsych-032516-113332
Stoughton, J. W., Thompson, L. F., & Meade, A. W. (2015). Examining Applicant Reactions to
the Use of Social Networking Websites in Pre-Employment Screening. Journal of Business
and Psychology, 30(1), 7388. doi:10.1007/s10869-013-9333-6
Sweeney, L. (2013). Discrimination in online ad delivery. Queue, 11(3), 10.
Syrdal, D. S., Dautenhahn, K., Koay, K. L., & Walters, M. L. (2009). The Negative Attitudes
towards Robots Scale and reactions to robot behaviour in a live human-robot interaction
study. In N. Taylor (Ed.), Adaptive and Emergent Behaviour and Complex Systems:
Proceedings of the 23rd Convention of the Society for the Study of Artificial Intelligence
and Simulation of Behaviour (pp. 109115). Edinburgh, UK: Society for the Study of
Artificial Intelligence and Simulation for Behavior (AISB). Retrieved from
Tippins, N. T. (2015). Technology and Assessment in Selection. Annual Review of
Organizational Psychology and Organizational Behavior, 2(1), 551582.
Van Iddekinge, C. H., Lanivich, S. E., Roth, P. L., & Junco, E. (2016). Social media for
selection? Validity and adverse impact potential of a facebook-based assessment. Journal of
Management, 42(7), 18111835. doi:10.1177/0149206313515524
Wingfield, N. (2017, September 10). As Amazon pushes forward with robots, workers find new
roles. New York Times. Retrieved from
Yan, W.-J., Wang, S.-J., Liu, Y.-J., Wu, Q., & Fu, X. (2014). For micro-expression recognition:
Database and suggestions. Neurocomputing, 136, 8287 doi:10.1016/j.neucom.2014.01.029
Figure 1. The Unrealized Potential of Technology in Selection Assessment
... The quality of recruitment and selection processes depends on attracting a sufficient number of suitable candidates. Technology now plays a fundamental role in this area Ryan & Derous, 2019). Organizations are adopting e-recruitment and online assessment as key practices in adapting selection processes to an increasingly global and demanding context (Derous & De Fruyt, 2016). ...
... In the context of the war for talent (Frasca & Edwards, 2017), technology has emerged as a key element in the field of human resource management (HRM) (Ryan & Derous, 2019). Social network websites (SNW) have become one of the main technology tools used by professionals in recruitment and selection processes (Nikolaou, 2014;Woods et al, 2020). ...
Full-text available
Background: The emergence of digital technology in the field of psychological and educational measurement and assessment broadens the traditional concept of pencil and paper tests. New assessment models built on the proliferation of smartphones, social networks and software developments are opening up new horizons in the field. Method: This study is divided into four sections, each discussing the benefits and limitations of a specific type of technology-based assessment: ambulatory assessment, social networks, gamification and forced-choice testing. Results: The latest developments are clearly relevant in the field of psychological and educational measurement and assessment. Among other benefits, they bring greater ecological validity to the assessment process and eliminate the bias associated with retrospective assessment. Conclusions: Some of these new approaches point to a multidisciplinary scenario with a tradition which has yet to be created. Psychometrics must secure a place in this new world by contributing sound expertise in the measurement of psychological variables. The challenges and debates facing the field of psychology as it incorporates these new approaches are also discussed.
... Researchers have also argued that there are number of limitations associated with them (Osoba & Welser, 2017). Following Ryan and Derous (2019), despite the fact that technology in assessment leads to much greater efficiency, there are also untested assumptions about effectiveness and fairness, and there is little consideration of potential negative byproducts of contextual enhancement, removing human judges, and collecting more data. Consequently, we identify some of these limitations below. ...
... Researchers have also argued that there are number of limitations associated with them (Osoba & Welser, 2017). Following Ryan and Derous (2019), despite the fact that technology in assessment leads to much greater efficiency, there are also untested assumptions about effectiveness and fairness, and there is little consideration of potential negative byproducts of contextual enhancement, removing human judges, and collecting more data. Consequently, we identify some of these limitations below. ...
Full-text available
Human Resources Analytics (HRA) is drawing more attention every year, and will be crucial to human resource development. However, the literature around the topic would appear to be more promotional than descriptive. With this in mind, we conducted a systematic literature review and content analysis with the following objectives: first, to address the current state of HRA and second, to propose a framework for the development of HRA as a sustainable practice. We analyzed 79 articles from research databases and found 34 empirical studies for subsequent content analysis. While the main results reflect the relative newness of the field of HRA, with the majority of the empirical articles focusing on financial aspects, they also reveal the growing importance given to ethics. Finally, we propose a framework for the development of sustainable HRA based on the triple bottom line and discuss the implications of our findings for researchers and practitioners.
Traditionelle analoge Testverfahren und Fragebögen kommen seit Jahrzehnten im Rahmen der Personalauswahl und Eignungsdiagnostik zum Einsatz. Entsprechend umfangreich sind empi- rische Befunde zu diesen klassischerweise papierbasierten Verfahren. Mit der zunehmenden Digitalisierung der Eignungsdiagnostik haben aber auch computer- und mobile-basierte Leis- tungstests und Fragbögen längst Einzug in der Personalauswahlprozess von Unternehmen ge- halten. Mehr noch, in vielen Organisation ersetzen sie zunehmend ihre altbewährten analogen Pendants. Doch wie sieht es um die diagnostische Qualität digitaler Testverfahren und deren Akzeptanz unter Bewerbenden aus und birgt die unbeaufsichtigte Testdurchführung über das Internet die Gefahr, dass Bewerbende Schummeln, um ihre Chancen auf ein Jobangebot zu verbessern? Diese Fragen und viele mehr sollen in diesem Kapitel beantwortet werden. Dazu werfen wir zunächst einen Blick auf die Grundlagen der Testdiagnostik, ehe wir die Eigenheiten technologiegestützter Testverfahren näher betrachten sowie abschließend deren Chancen und Risiken evaluieren.
Full-text available
We developed a game-based assessment (GBA) measuring cognitive ability for use in personnel selection and examined its construct-related validity. Moreover, applicant reactions towards this GBA were compared to a paper-pencil based ability test. Both assessment tools were designed to measure verbal, numerical, and figural ability. N = 183 participants completed the GBA, the paper-pencil test, and questions capturing applicant reactions and personality. We found a strong positive correlation of .51 between the overall GBA and paper-pencil test scores, showing evidence for its construct-related validity. Applicant reactions towards the GBA were consistently worse compared to the paper-pencil test. Furthermore, males and individuals with more video game experience held more positive perceptions than females and individuals with less video game experience.
AI has proven useful in sports, healthcare, transportation, and industry. Recruitment practises including locating applicants, vetting them, holding interviews, and hiring them have gained in popularity. This can affect a company's policies, candidate attitudes, and HR duties. In some circumstances, recruiters may not know about this technology, or companies may still be using it. HR's duty is to provide a safe and supportive workplace while giving employees the independence, intelligence, and empathy they need to accomplish their best work. AI has greatly improved the HR department. AI has taken over low-value HR tasks to free up time for strategy.
Introduction to Industrial/Organizational Psychology provides a complete overview of the psychological study of the world of work. Written with the student in mind, the book presents classic theory and research in the field alongside examples from real-world work situations to provide deeper insight. This edition has been thoroughly updated to include the latest research on each key topic, and now features: A spotlight on diversity, equity, and inclusion throughout, including coverage of LGBTQIA+ inclusion and racial justice Expanded coverage of ethics in I/O psychology practice Increased emphasis on cross-cultural and international issues Coverage of the changing nature of work, post-pandemic, including remote working, worker stress, and burnout A new focus on technologies related to I/O such as virtual reality and computer adaptive testing New figures, illustrations, and charts to grab the reader's attention and facilitate learning Accompanied by extensive student and instructor resources, it is a must read for all students on I/O psychology courses and courses in work psychology and organizational behavior, and for practicing managers who want a comprehensive overview of the psychology of work. © 2022 Ronald E. Riggio & Stefanie K. Johnson. All rights reserved.
Full-text available
Various surveys suggest LinkedIn is used as a screening and selection tool by many hiring managers. Despite this widespread use, fairly little is known about whether LinkedIn meets established selection criteria, such as reliability, validity, and legality (i.e., no adverse impact). We examine the properties of LinkedIn‐based assessments in two studies. Study 1 shows that raters reach acceptable levels of consistency in their assessments of applicant skills, personality, and cognitive ability. Initial ratings also correlate with subsequent ratings done one‐year later (i.e., demonstrating temporal stability), with slightly higher correlations when profile updates are taken into account. Initial LinkedIn‐based ratings correlate with self‐reports for more visible skills (leadership, communication, and planning) and personality traits (Extraversion), and for cognitive ability. LinkedIn‐based hiring recommendations are positively associated with indicators of career success. Potential adverse impact is also limited. Profiles that are longer, include a picture, and have more connections are rated more positively. Some of those features are valid cues to applicants’ characteristics (e.g., applicants high on Conscientiousness have longer profiles). In Study 2, we show that an itemized LinkedIn assessment is more effective than a global assessment. Implications of these findings for selection and future research are discussed. This article is protected by copyright. All rights reserved
Full-text available
Technologically advanced selection procedures are entering the market at exponential rates. The current study tested two previously held assumptions: (a) providing applicants with procedural information (i.e., making the procedure more transparent and justifying the use of this procedure) on novel technologies for personnel selection would positively impact applicant reactions, and (b) technologically advanced procedures might differentially affect applicants with different levels of computer experience. In a 2 (computer science students, other students) × 2 (low information, high information) design, 120 participants watched a video showing a technologically advanced selection procedure (i.e., an interview with a virtual character responding and adapting to applicants’ nonverbal behavior). Results showed that computer experience did not affect applicant reactions. Information had a positive indirect effect on overall organizational attractiveness via open treatment and information known. This positive indirect effect was counterbalanced by a direct negative effect of information on overall organizational attractiveness. This study suggests that computer experience does not affect applicant reactions to novel technologies for personnel selection, and that organizations should be cautious about providing applicants with information when using technologically advanced procedures as information can be a double-edged sword. Update: While not specifically mentioned in the paper it has implications for explainability and XAI research: providing people with more transparency can have simultaneous positive and negative effects on acceptance.
Het selecteren van goede werknemers is van groot belang voor het garanderen van goed functionerende organisaties. Allereerst bekijken we de huidige wetenschappelijk stand van zaken met betrekking tot het selecteren van werknemers. We bespreken de verschillende fases van een selectieprocedure, perspectieven om naar selectie te kijken en het selectiemodel. We eindigen het eerste deel met een kritische bespreking van de voornaamste selectie-instrumenten op basis van de best beschikbare wetenschappelijke evidentie en een samenvattende testwijzer. Vervolgens illustreren we een aantal recente uitdagingen voor werving en selectie aan de hand van casussen; we eindigen met een kritische noot over de zogenoemde scientist-practitioner gap (de kloof tussen wetenschap en praktijk) en wat we hieraan kunnen doen.
The use of technology for workplace and occupational testing blossomed in the early years of this century. This book offers a demonstration that the first generation of these technologies have now been implemented long enough to observe the patterns and issues that emerge when these approaches evolve through technical advancement and successive application. A new set of issues and opportunities has emerged and the next generation of these applications is now coming of age. This book reflects on the last few decades of this evolutionary process from a vantage point of global experience across a wide range of workplace applications, including employment selection, development, and occupational certification. The themes and issues that arise as this broad treatment unfolds provide an essential foundation for students, researchers, and professionals who are involved with the assessment of human capability and potential in organizational and workplace contexts. Covers the next generation of technical advancements in workplace testing. Draws from experts around the world and across practice domains. Offers a practical perspective for real-world application.
The widespread use of unproctored Internet-based testing (UIT) in employment-related assessments has resulted in a burgeoning use of mobile devices to complete these assessments. Coupled with this is a concomitant interest in UIT-device-type effects, specifically, differences or lack thereof between assessments completed on “mobile” versus “nonmobile” devices. However, there is limited theoretical or conceptual work that seeks to explain the observed effects of UIT device type on test scores. Consequently, this article first presents a review of the extant empirical literature and then, on the basis of that, presents a framework—the structural characteristics/information processing framework—for psychologically conceptualizing the effect of UIT device types on test and assessment scores. The framework is used to explain previous findings and advance testable propositions for future research.
I am concerned about industrial and organizational (I-O) psychology's relevance to the gig economy, defined here as the broad trends toward technology-based platform work. This sort of work happens on apps like Uber (where the app connects drivers and riders) and sites like MTurk (where human intelligence tasks, or HITs, are advertised to workers on behalf of requesters). We carry on with I-O research and practice as if technology comprises only things (e.g., phones, websites, platforms) that we use to assess applicants and complete work. However, technology has much more radically restructured work as we know it, to happen in a much more piecemeal, on-demand fashion, reviving debates about worker classification and changing the reality of work for many workers (Sundararajan, 2016). Instead of studying technology as a thing we use, it's critical that we “zoom out” to see and adapt our field to this bigger picture of trends towards a gig economy. Rather than a phone being used to check work email or complete pre-hire assessments, technology and work are inseparable. For example, working on MTurk requires constant Internet access (Brawley, Pury, Switzer, & Saylors, 2017; Ma, Khansa, & Hou, 2016). Alarmingly, some researchers describe these workers as precarious (Spretizer, Cameron, & Garrett, 2017), dependent on an extremely flexible (a label that is perhaps euphemistic for unreliable) source of work. Although it's unlikely that all workers consider their “gig” a full time job or otherwise necessary income, at least some workers do: An estimated 10–40% of MTurk workers consider themselves serious gig workers (Brawley & Pury, 2016). Total numbers for the broader gig economy are only growing, with recent tax-based estimates including 34% of the US workforce now and up to 43% within 3 years (Gillespie, 2017). It appears we're seeing some trends in work reverse and return to piece work (e.g., a ride on Uber, a HIT on MTurk) as if we've simply digitized the assembly line (Davis, 2016). Over time, these trends could accelerate, and we could potentially see total elimination of work (Morrison, 2017).