ArticlePDF Available

The Incremental Validity of Interview Scores Over and Above Cognitive Ability and Conscientiousness Scores

DOI:10.1111/j.1744-6570.2000.tb00204.x
Authors:
Jose Cortina at Virginia Commonwealth University
Jose Cortina
NANCY B. GOLDSTEIN
NANCY B. GOLDSTEIN
NANCY B. GOLDSTEIN
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Stephanie C Payne at Texas A&M University
Stephanie C Payne
H.KRISTL DAVISON
H.KRISTL DAVISON
H.KRISTL DAVISON
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Show all 5 authorsHide
Download full-text PDF
Read full-text
Download citation

Abstract

Recent research has suggested that scores on measures of cognitive ability, measures of Conscientiousness, and interview scores are positively correlated with job performance. There remains, however, a question of incremental validity: To what extent do interviews predict above and beyond cognitive ability and Conscientiousness? This question was addressed in this paper by (a) conducting meta-analyses of the relationships among cognitive ability, Conscientiousness, and interviews, (b) combining these results with predictive validity results from previous meta-analyses to form a “meta-correlation matrix” representing the relationships among cognitive ability, Conscientiousness, interviews, and job performance, and (c) performing 9 hierarchical regressions to examine the incremental validity of 3 levels of structured interviews in best, actual, and worst case scenarios for prediction. Results suggested that interview scores contribute to the prediction of job performance over and above cognitive ability and Conscientiousness to the extent that they are structured, with scores from highly structured interviews contributing substantially to prediction. Directions for future research are discussed.
Content uploaded by Stephanie C Payne
Author content
All content in this area was uploaded by Stephanie C Payne on Jun 11, 2019
Content may be subject to copyright.
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
... In the holistic adjustment condition, participants were presented with an applicant's predictor scores and an algorithm's prediction for each prediction. To construct the algorithm, the standardized predictor scores were multiplied by regression weights that we derived from the meta-analytic correlation matrix presented in Cortina et al. (2000). Since our archival applicant data contained scores from an unstructured interview, we used the correlations presented in the top panel of Table 3 in Cortina et al. (2000), in which a low level of interview structure is assumed (Huffcutt & Arthur, 1994). ...
... To construct the algorithm, the standardized predictor scores were multiplied by regression weights that we derived from the meta-analytic correlation matrix presented in Cortina et al. (2000). Since our archival applicant data contained scores from an unstructured interview, we used the correlations presented in the top panel of Table 3 in Cortina et al. (2000), in which a low level of interview structure is assumed (Huffcutt & Arthur, 1994). We derived standardized regression weights from this meta-correlation matrix using the setCor function from the psych package in R (see the R script "Applicant selection.R" on OSF). ...
When and Why Decision Makers Use Algorithms in Personnel Selection: Stakeholder Perceptions, Use Intentions, and Predictive Validity
Preprint
Full-text available
  • Aug 2022
Decision makers such as industrial and organizational psychologists often combine multiple pieces of information to make performance predictions and hiring decisions. More valid performance predictions are made when information is combined algorithmically (mechanical prediction) rather than in the decision-maker’s mind (holistic prediction). Yet, decision makers rarely use algorithms to make performance predictions in practice, which is also known as algorithm aversion. Decision makers underutilize algorithms partly because they are worried about negative evaluations from stakeholders such as colleagues when using algorithms. Based on attribution theory, we hypothesized that stakeholders evaluate decision makers more positively when they use autonomy-enhancing algorithmic procedures (AEAPs, holistically adjust predictions from a prescribed algorithm or self-design an algorithm), than when they use a prescribed algorithm. Relatedly, based on the theory of planned behavior, we hypothesized that decision makers who use AEAPs are less worried about negative stakeholder evaluations, and more likely to use algorithms in performance predictions. In Study 1 (N = 582), the results showed that stakeholders evaluated decision makers more positively when they used AEAPs rather than a prescribed algorithm. In Study 2 (N = 269), the results showed that decision makers were less worried about negative stakeholder evaluations and more likely to use AEAPs compared to a prescribed algorithm. Importantly, AEAPs also resulted in substantially higher predictive validity than holistic prediction. Based on our results, we suggest that decision makers should combine predictors with a self-designed algorithm to achieve acceptable stakeholder evaluations and predictive validity.
View
... Because these methods are likely to generate misleading results when effect size heterogeneity is high (e.g., Yu et al., 2016), it is vital for the authors of such studies to incorporate heterogeneity into their design, analysis, etc. An example for the latter can be found in Cortina et al. (2000), which examined the incremental validity of structured interviews on job performance over and above cognitive ability and conscientiousness. As such, Cortina et al. considered the degree of interview structure as an important moderator in the relation between interview scores and job performance. ...
Heterogeneity in meta-analytic effect sizes: An assessment of the current state of the literature
Article
Full-text available
Heterogeneity refers to the variability in effect sizes across different samples and is one of the major criteria to judge the importance and advancement of a scientific area. To determine how studies in the organizational sciences address heterogeneity, we conduct two studies. In study 1, we examine how meta-analytic studies conduct heterogeneity assessments and report and interpret the obtained results. To do so, we coded heterogeneity-related information from meta-analytic studies published in five leading journals. We found that most meta-analytic studies report several hetero-geneity statistics. At the same time, however, there tends to be a lack of detail and thoroughness in the interpretation of these statistics. In study 2, we review how primary studies report heteroge-neity-related results and conclusions from meta-analyses. We found that the quality of the reporting of heterogeneity-related information in primary studies tends to be poor and unrelated to the detail and thoroughness with which meta-analytic studies report and interpret the statistics. Based on our findings, we discuss implications for practice and provide recommendations for how heterogeneity assessments should be conducted and communicated in future research. Keywords heterogeneity, between-study variability, meta-analysis Meta-analytic techniques have been used to answer various research questions, primarily ones related to the magnitude of a mean effect size and its variability across studies, the latter having been termed heterogeneity. Schmidt and Hunter's (1977, 1998; Schmidt et al., 1993) development and use of psy-chometric meta-analytic techniques was aimed at assessing the validity and generalizability of
View
... Further, when personality measures are contextualized to the work context, they have been found to be stronger predictors of job performance (Shaffer & Pos-tlethwaite, 2012). Additionally, when used in combination with other assessment tools (e.g., interviews), cognitive ability tests and personality inventories can add incremental validity to the prediction of job performance (Cortina et al., 2000). A possible explanation for the research-practice gap in assessment use is that academics and practitioners have different priorities (Risavy et al., 2021). ...
Personnel Selection in Australia: Identifying Research-Practice Gaps and Understanding the Importance of Culture Fit
Article
Full-text available
  • May 2023
This study examined current personnel selection practices in Australia including (a) the types of assessments used, (b) the factors considered when choosing assessments, and (c) the characteristics targeted in successful applicants. Participants from 68 organizations responded to a questionnaire that asked about current selection practices. Several areas where current practice deviated from research-supported best practice were identified. First, psychometric tests were used rarely: Cognitive ability tests were used by 26% of organizations and self-report questionnaires (e.g., personality inventories) by 18% of organizations. Second, when choosing assessments, the three most important considerations (in order) were the candidate experience, reducing bias, and that the assessment provides consistent scores; validity of the assessment was fourth. Finally, the most common characteristic organizations considered when selecting applicants was “culture fit.” Supplementary analyses to determine how culture fit was defined and assessed suggested there is little consistency in what it means and how it is measured.
View
... Participants received advice from an algorithm that was constructed based on meta-analytic correlations rather than primary data to avoid overfitting, and because large enough primary datasets are rarely available in practice. Cortina, Goldstein, Payne, Davison, and Gilliland (2000) reported a meta-correlation matrix with correlations between cognitive ability, conscientiousness, an unstructured interview, and job performance (p. 339, Table 3). ...
View
... Developments in the use of this strategy include (a) the key role of sampling error, and hence the need for larger samples (e.g., Schmidt et al., 1976); (b) attenuation due to measurement error in the criterion (e.g., Viswesvaran et al., 1996) and differences in predictor variability between the research sample and applicant pool (e.g., Sackett & Yang, 2000); and (c) a move from relying on local validity evidence to cumulative evidence from other settings (e.g., Schmidt & Hunter, 1981). Meta-analytic estimates of the mean and variance of criterion-related validity of predictors have been obtained (e.g., Sackett et al., 2021;Schmidt & Hunter, 1998), as have meta-analytic estimates of the incremental validity of different predictors (e.g., Bobko et al., 1999;Cortina et al., 2000). ...
Personnel selection: A review of ways to maximize validity, diversity, and the applicant experience
Article
Full-text available
Personnel Psychology has a long tradition of publishing important research on personnel selection. In this article, we review some of the key questions and findings from studies published in the journal and in the selection literature more broadly. In doing so, we focus on the various decisions organizations face regarding selection procedure development (e.g., use multiple selection procedures, contextualize procedure content), administration (e.g., provide pre‐test explanations, reveal target KSAOs), and scoring (e.g., weight predictors and criteria, use artificial intelligence). Further, we focus on how these decisions affect the validity of inferences drawn from the procedures, how use of the procedures may affect organizational diversity, and how applicants experience the procedures. We also consider factors such as cost and time. Based on our review, we highlight practical implications and key directions for future research. This article is protected by copyright. All rights reserved
View
... The utility of novel constructs, however, requires demonstration of incremental prediction over existing predictors (Harms & Credé, 2010). Evidence of incremental prediction of dispositional traits above the Big Five is also critical to establish the uniqueness and utility of new predictors in work settings (e.g., Cortina et al., 2000). Furthermore, organizational scholars have recently begun to focus on personality traits outside of the Big Five (Hough et al., 2015). ...
The Bright, Dark, and Gray Sides of Risk Takers at Work: Criterion Validity of Risk Propensity for Contextual Work Performance
Article
Full-text available
Although risk takers are traditionally seen as liabilities, a growing body of research suggests that risk takers may be critical for organizational achievements because of their courage and willingness to take risks for the benefit of others. Despite the prevalence of risk taking in studies of strategic management and organizational behavior, we know very little about the implication of risk propensity on employee work performance. In this paper, we conceptualize contextual performance—behaviors that fall outside of normal job expectations—as a form of workplace risk taking. Results from a three-wave study of working adults revealed that risk propensity positively predicted both counterproductive work behaviors (CWBs) and pro-social rule-breaking behaviors (PSRBs) above the Big Five personality traits. We also found an indirect effect of risk propensity on organizational citizenship behaviors (OCBs) and PSRBs via an increased willingness to take risks at work. Taking an item-level approach, we found that the predictive validity of risk propensity was greater for specific performance items that independent judges rated as riskier. Our findings suggest that risk takers are more likely to engage in pro-social and counterproductive behaviors—particularly those that are rule-breaking or have elevated risks—in part due to an increased willingness to take risks at work.
View
... Participants received advice from an algorithm that was constructed based on metaanalytic correlations rather than primary data to avoid overfitting, and because large enough primary datasets are rarely available in practice. Cortina et al. (2000) reported a metacorrelation matrix with correlations between cognitive ability, conscientiousness, an unstructured interview, and job performance (p. 339, Table 3). ...
Predicting Decision-makers' Algorithm Use
Preprint
Full-text available
  • Nov 2022
Decision makers typically integrate multiple pieces of information to make predictions and decisions. They also sometimes receive algorithmic advice, but often discount such advice. This usually results in less consistent and less accurate predictions than strictly using the advice. Therefore, we investigated if conscientiousness, decision-making styles, predictor validity beliefs, and cognitive ability are related to algorithm use, judgment consistency, and predictive validity. Participants with hiring experience (N = 308) predicted the performance of job applicants based on test scores and interview ratings, and they also received algorithmic advice. We found that more dutiful participants and participants with more accurate predictor validity beliefs used the algorithm more, and made more consistent and more accurate predictions. We did not find evidence that an intuitive decision-making style was related to these outcomes. Exploratory analyses showed that cognitive ability was related to algorithm use and judgment consistency, but not to predictive validity. Organizations may want to hire more conscientious and knowledgeable decision makers, and provide training on predictor validities.
View
Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research
Article
  • Nov 2022
Some assert that noise (i.e., unwanted variance) is the most neglected yet most important source of error in judgment. We suggest that this problem was discovered nearly 100 years ago in the area of personnel selection and that a century of selection research has shown that noise can be demonstrably reduced by structuring the process (i.e., decomposing the component parts, agreeing on standards, and applying those standards consistently) and by aggregating judgments independently. Algorithms can aid significantly in this process but are often confused with methods that, in their current form, can substantially increase noise in judgment (e.g., artificial intelligence and machine learning). Expected final online publication date for the Annual Review of Organizational Psychology and Organizational Behavior, Volume 10 is January 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
View
Gatekeepers influencing careers of Canadian public sector employees: views from managers and union employees
Article
Purpose The purpose of this paper is to develop a better understanding of the types of career models that different managerial and union employees view influencing their career development. Design/methodology/approach In this study, the authors gathered interview data from 74 public sector employees in management and union positions illustrating examples of the career models they experienced. Findings The study explains how unique career models are often defined by gatekeepers who act in setting out the procedures, incentives, and norms for the careers of others. The results, based on interviews in a Canadian government organization, suggest that employees perceive that different gate keepers are central in shaping careers of management and union employees in a more traditional career hierarchy, even though other perspectives or orientations of career progression exist in encouraging people to take more responsibility for career development. Management participants were more likely to be guided by upper level managers who acted as gatekeepers within a ‘perception of organizational politics’ model where “the system has a procedural merit rather than real merit….and systemic biases are not even realized by the people doing the hiring.” Key gatekeepers for union employees were those in human resource departments and the union who defined the fairness of the procedures within a human capital model which generally managed career development as a reward for higher levels of experience, education, and training. The findings illustrate unique ways that each set of gatekeepers shape the way that selection and promotion processes are carried out. Research limitations/implications The authors are mindful that our results are, at best, exploratory. The qualitative interviews were from a sample of 74 government workers in the Canadian public service and should be verified with further research. Although the authors felt that interviews illustrated saturation and might only be a reliable reflection of a specific sample, other research should examine these findings in other contexts. Further examination of these findings might help us understand the challenges of developing systems and procedures which illustrate a distributive rather than merely a procedural fairness. Practical implications The process of socialization for a career in management and union positions is demonstrated by the boundaries through which a person moves from being an outsider to an insider to the organization. Unique career models are often defined by gatekeepers who act in setting out the procedures, incentives, and norms for the careers of others. The research and practical implications point to: (i) designing supplemental behavioral interview questions and tools in selection and promotion as a way to respond to systemic biases, (ii) building awareness of how to respond to biases of the powerful role of referrals and networks in shaping careers of managerial employees, and (iii) taking steps to develop a climate which might be supportive of merit processes. Social implications The process of socialization for a career in management and union positions is demonstrated by the boundaries through which a person moves from being an outsider to an insider to the organization. Unique career models are often defined by gatekeepers who act in setting out the procedures, incentives, and norms for the careers of others. Originality/value The study suggests that unique career models are often defined by gatekeepers who act in setting out the procedures, incentives, and norms for the careers of others.
View
Matching Efficiency of Admission Procedures in Online-Education
Chapter
  • Jan 2022
This paper empirically evaluates the relationship between the university admission process and the study success of online students. We investigate data from fully accredited higher education courses over multiple cohorts, yielding a sample of 502 students from both Bachelor- and Master-studies. We link rankings from 3 test procedures, an interview, a resume analysis, and an aptitude test, with grade achievements and successful course completion. We find that all procedures predict grades, but only resume analysis has a robust relationship with successful course completion. Crucially, the predictive power of the resume analysis depends on controlling for a student’s age. Our results thus call for nuance in evaluating and interpreting admission scores when allocating seats in online education.KeywordsAdmission procedureOnline studentsPredictive powerAdmission interviewStudy success
View
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings
Article
Full-text available
  • Sep 1998
This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.
View
The Validity of Employment Interviews: A Comprehensive Review and Meta-Analysis
Article
Full-text available
  • Aug 1994
This meta-analytic review presents the findings of a project investigating the validity of the employment interview. Analyses are based on 245 coefficients derived from 86,311 individuals. Results show that interview validity depends on the content of the interview (situational, job related, or psychological), how the interview is conducted (structured vs. unstructured; board vs. individual), and the nature of the criterion (job performance, training performance, and tenure; research or administrative ratings). Situational interviews had higher validity than did job-related interviews, which, in turn, had higher validity than did psychologically based interviews. Structured interviews were found to have higher validity than unstructured interviews. Interviews showed similar validity for job performance and training performance criteria, but validity for the tenure criteria was lower.
View
The Proactive Personality Scale and Objective Job Performance Among Real Estate Agents
Article
Full-text available
  • Aug 1995
A sample of 131 real estate agents was used to examine the criterion validity of the Proactive Personality Scale (T. S. Bateman & J. M. Crant, 1993). A job performance index was computed for each agent from archival records of the number of houses sold, number of listings obtained, and commission income over a 9-month period. Experience, social desirability, general mental ability, and 2 of the Big Five factors-Conscientiousness and Extraversion-were controlled for, and the Proactive Personality Scale explained an additional 8% of the variance in the objective measure of agents' job performance. These results provide additional evidence for the criterion validity of the Proactive Personality Scale and suggest that specific personality measures can have incremental validity over the Big Five factors.
View
View
View
Conscientiousness and Performance of Sales Representatives: Test of the Mediating Effects of Goal Setting
Article
  • Oct 1993
The authors used 91 sales representatives to test a process model that assessed the relationship of conscientiousness to job performance through mediating motivational (goal-setting) variables. Linear structural equation modeling showed that sales representatives high in conscientiousness are more likely to set goals and are more likely to be committed to goals, which in turn is associated with greater sales volume and higher supervisory ratings of job performance. Results also showed that conscientiousness is directly related to supervisory ratings. Consistent with previous research, results showed that ability was also related to supervisory ratings of job performance and, to a lesser extent, sales volume. Contrary to expectations, 1 other personality construct, extraversion, was not related to sales volume or to supervisory ratings of job performance. Implications and future research needs are discussed.
View
Criterion-Related Validities of Personality Constructs and the Effect of Response Distortion on Those Validities
Article
  • Oct 1990
A review of criterion-related validities of personality constructs indicated that six constructs are useful predictors of important job-related criteria. An inventory was developed to measure the 6 constructs. In addition, 4 response validity scales were developed to measure accuracy of self-description. These scales were administered in three contexts: a concurrent criterion-related validity study, a faking experiment, and an applicant setting. Sample sizes were 9,188,245, and 125, respectively. Results showed that (a) validities were in the.20s (uncorrected for unreliability or restriction in range) against targeted criterion constructs, (b) respondents successfully distorted their self-descriptions when instructed to do so, (c) response validity scales were responsive to different types of distortion, (d) applicants' responses did not reflect evidence of distortion, and (e) validities remained stable regardless of possible distortion by respondents in either unusually positive or negative directions.
View
MODELS OF SUPERVISORY JOB-PERFORMANCE RATINGS
Article
  • Dec 1991
Proposed and evaluated in this research were causal models that included measures of cognitive ability, job knowledge, task proficiency, two temperament constructs (achievement and dependability), awards, problem behavior, and supervisor ratings. The models were tested on a sample of 4362 US Army enlisted personel in nine different jobs.
View
View
The Arrow-Dot Test as a Predictor of Police Officer Performance
Article
  • Dec 1977
In the course of validating a test battery for the selection of police officers (Baehr, Furcon, & Froemel, 1969), project personnel were searching for objectively scored, pencil-and-paper measures of behavior variables. One product of this search was an adaptation of the Arrow-Dot subtest of the four-part IES test battery (Dombrose & Slobin, 1958). This adaptation was designed for group administration and for improved reliability as a possible predictor of performance. Intercorrelational analysis and factor analysis of intercorrelations with other potential predictor variables and performance criteria provided some information on the nature of the Arrow-Dot scores. The test's ability to predict police officers' performance was assessed through multiple regression analyses of its scores both as independent predictors and as contributors of unique variance in a wider predictor battery. Its usefulness for diagnosis of specific problems in police officers' performance was also investigated. Application in five additional studies permitted an exploration of the “characteristic” Arrow-Dot dimension profile for police officers.
View