ArticlePDF Available

The Incremental Validity of Interview Scores Over and Above Cognitive Ability and Conscientiousness Scores

Authors:
Jose Cortina at Virginia Commonwealth University
Jose Cortina
NANCY B. GOLDSTEIN
NANCY B. GOLDSTEIN
NANCY B. GOLDSTEIN
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Stephanie C Payne at Texas A&M University
Stephanie C Payne
H.KRISTL DAVISON
H.KRISTL DAVISON
H.KRISTL DAVISON
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Show all 5 authorsHide
Download full-text PDF
Read full-text
Download citation

Abstract

Recent research has suggested that scores on measures of cognitive ability, measures of Conscientiousness, and interview scores are positively correlated with job performance. There remains, however, a question of incremental validity: To what extent do interviews predict above and beyond cognitive ability and Conscientiousness? This question was addressed in this paper by (a) conducting meta-analyses of the relationships among cognitive ability, Conscientiousness, and interviews, (b) combining these results with predictive validity results from previous meta-analyses to form a “meta-correlation matrix” representing the relationships among cognitive ability, Conscientiousness, interviews, and job performance, and (c) performing 9 hierarchical regressions to examine the incremental validity of 3 levels of structured interviews in best, actual, and worst case scenarios for prediction. Results suggested that interview scores contribute to the prediction of job performance over and above cognitive ability and Conscientiousness to the extent that they are structured, with scores from highly structured interviews contributing substantially to prediction. Directions for future research are discussed.
Content uploaded by Stephanie C Payne
Author content
All content in this area was uploaded by Stephanie C Payne on Jun 11, 2019
Content may be subject to copyright.
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
... One of the techniques concerning the selection of personnel to fill new positions is to have interviews with related personnel. Robertson and Smith [18] and Cortina et al. [9] present notable ability and availability of interviews to predict the performance of the personnel in the job. The usages of different methods in some European countries are given in Dany and Torchy [10]. ...
... According to the total goal requirement given in Fig. 1, the decision maker took a part in the evaluation, marked the result of pair-by-pair comparison of first degree of hierarchy. Triangular fuzzy numbers (1,3,5,7,9) are used to indicate the relative strength of each pair of elements in the same hierarchy. By using triangular fuzzy numbers, via pair-wise comparison, fuzzy judgment matrixÃða i j Þ is constructed. ...
... For each criterion or alternative, the weights can be calculated using Eq. (9). Fuzzy weights are defuzzified by using Eq. ...
View
... personality traits (r = −.03 to .16), and communication skills (r = .08, Huffcutt, 2011; see also Cortina et al., 2000;Huffcutt et al., 2001;Roth & Huffcutt, 2013;Roth et al., 2005;Salgado & Moscoso, 2002) each partially account for interviewers' evaluations. Notably, however, job-irrelevant factors such as applicants' physical appearance (r = .37) ...
Evaluating interview criterion‐related validity for distinct constructs: A meta‐analysis
Article
Full-text available
  • Jul 2024
The employment interview is used to assess myriad constructs to inform personnel selection decisions. This article describes the first meta‐analytic review of the criterion‐related validity of interview‐based assessments of specific constructs (i.e., related to task and contextual performance). As such, this study explores the suitability of the interview for predicting specific dimensions of performance, and furthermore, if and how interviews should be designed to inform the assessment of distinct constructs. A comprehensive search process identified k = 37 studies comprising N = 30,646 participants ( N = 4449 with the removal of one study). Results suggest that constructs related to task ( ρ = .30) and contextual ( ρ = .28) performance are assessed with similar levels of criterion‐related validity. Although interview evaluations of task and contextual performance constructs did not show discriminant validity within the interview itself, interview evaluations were more predictive of the targeted criterion construct than of alternative constructs. We further found evidence that evaluations of contextual performance constructs might particularly benefit from the adoption of more structured interview scoring procedures. However, we expect that new research on interview design factors may find additional moderating effects and we point to critical gaps in our current body of literature on employment interviews. These results illustrate how a construct‐specific approach to interview validity can spur new developments in the modeling, assessment, and selection of specific work performance constructs.
View
... It is estimated that the future performance of the recruited personnel will be successful with the objective selection (Güngör, Serhadlıoğlu & Kesen, 2009). In order for the selection to be objective, basic methods such as personality factors (Salgado, 1997), written and verbal communication requirements for the job description (Jessop, 2004), interviews, and talent interviews (Cortina et al., 2000;Robertson & Smith, 2001) were adopted. Studies show that besides these basic methods, sectoral-based additions that deal with different criteria are also made with the teams formed by human resources managers and experts, and group decision-making techniques are also applied (Chen, 2000;Chen & Cheng, 2005;Shih, Shyur & Lee, 2007;Canós & Liern, 2008;Saremi et al., 2009;Zhang & Liu, 2011;Baležentis et al., 2012;Wan, Wang & Dong, 2013). ...
Determining the Importance Level of Effective Criteria in the Employees in the Defense Acquisition Process via Fuzzy DEMATEL Method
Article
Full-text available
  • Jun 2023
In today's globally competitive environment, sector managers are faced with various problems day by day. One of these problems is the selection of personnel, which is the most important element of an organization, in accordance with the job and workplace in terms of quality and quantity. The recruitment process consists of the stages of determining the expert group that will make the selection, determining the criteria to be used in the evaluation, and making a decision by evaluating the criteria. The most important step in this process is to determine the criteria in line with the needs of the sector. This situation becomes more special for sectors with long-term and strategic features such as the defense industry. In this study, it is aimed to determine the selection criteria with the group decision making method for the defense acquisition personnel, which is one of the keystones of the defense sector, and to examine the interaction of the criteria with each other. Comparisons of the experts on the criteria were made with a Multi-Criteria Decision Making (MCDM) method without discrimination of the working class (white, blue, gray, etc). The fuzzy DEMATEL (Decision Making Trial and Evaluation Laboratory) method was used in the study since the expert opinions on the criteria determined for the defense acquisition process are made with subjective expressions and it is used in the criteria analysis in various fields in the literature.
View
... In the holistic adjustment condition, participants saw each applicant's predictor scores and the algorithm's prediction. To construct the algorithm, the standardized predictor scores were multiplied by regression weights that we derived from the meta-analytic correlation matrix presented in the top panel of Table 3 in Cortina et al. (2000), by using the setCor function from the psych package in R (see the R script "Applicant selection.R" on OSF). To aid participants' interpretation of the predictions, we rescaled the algorithm's predictions to a five-point scale (1 = very bad job performance, 5 = very good job performance) using min-max scaling (see p. 114, formula 3.8 in Han et al., 2011). ...
View
... Because these methods are likely to generate misleading results when effect size heterogeneity is high (e.g., Yu et al., 2016), it is vital for the authors of such studies to incorporate heterogeneity into their design, analysis, etc. An example for the latter can be found in Cortina et al. (2000), which examined the incremental validity of structured interviews on job performance over and above cognitive ability and conscientiousness. As such, Cortina et al. considered the degree of interview structure as an important moderator in the relation between interview scores and job performance. ...
Heterogeneity in meta-analytic effect sizes: An assessment of the current state of the literature
Article
Full-text available
  • May 2023
Heterogeneity refers to the variability in effect sizes across different samples and is one of the major criteria to judge the importance and advancement of a scientific area. To determine how studies in the organizational sciences address heterogeneity, we conduct two studies. In study 1, we examine how meta-analytic studies conduct heterogeneity assessments and report and interpret the obtained results. To do so, we coded heterogeneity-related information from meta-analytic studies published in five leading journals. We found that most meta-analytic studies report several hetero-geneity statistics. At the same time, however, there tends to be a lack of detail and thoroughness in the interpretation of these statistics. In study 2, we review how primary studies report heteroge-neity-related results and conclusions from meta-analyses. We found that the quality of the reporting of heterogeneity-related information in primary studies tends to be poor and unrelated to the detail and thoroughness with which meta-analytic studies report and interpret the statistics. Based on our findings, we discuss implications for practice and provide recommendations for how heterogeneity assessments should be conducted and communicated in future research. Keywords heterogeneity, between-study variability, meta-analysis Meta-analytic techniques have been used to answer various research questions, primarily ones related to the magnitude of a mean effect size and its variability across studies, the latter having been termed heterogeneity. Schmidt and Hunter's (1977, 1998; Schmidt et al., 1993) development and use of psy-chometric meta-analytic techniques was aimed at assessing the validity and generalizability of
View
... Further, when personality measures are contextualized to the work context, they have been found to be stronger predictors of job performance (Shaffer & Pos-tlethwaite, 2012). Additionally, when used in combination with other assessment tools (e.g., interviews), cognitive ability tests and personality inventories can add incremental validity to the prediction of job performance (Cortina et al., 2000). A possible explanation for the research-practice gap in assessment use is that academics and practitioners have different priorities (Risavy et al., 2021). ...
Personnel Selection in Australia: Identifying Research-Practice Gaps and Understanding the Importance of Culture Fit
Article
Full-text available
  • May 2023
This study examined current personnel selection practices in Australia including (a) the types of assessments used, (b) the factors considered when choosing assessments, and (c) the characteristics targeted in successful applicants. Participants from 68 organizations responded to a questionnaire that asked about current selection practices. Several areas where current practice deviated from research-supported best practice were identified. First, psychometric tests were used rarely: Cognitive ability tests were used by 26% of organizations and self-report questionnaires (e.g., personality inventories) by 18% of organizations. Second, when choosing assessments, the three most important considerations (in order) were the candidate experience, reducing bias, and that the assessment provides consistent scores; validity of the assessment was fourth. Finally, the most common characteristic organizations considered when selecting applicants was “culture fit.” Supplementary analyses to determine how culture fit was defined and assessed suggested there is little consistency in what it means and how it is measured.
View
... Participants received advice from an algorithm that was constructed based on meta-analytic correlations rather than primary data to avoid overfitting, and because large enough primary datasets are rarely available in practice. Cortina, Goldstein, Payne, Davison, and Gilliland (2000) reported a meta-correlation matrix with correlations between cognitive ability, conscientiousness, an unstructured interview, and job performance (p. 339, Table 3). ...
Predicting decision-makers’ algorithm use
Article
Full-text available
  • Mar 2023
  • COMPUT HUM BEHAV
Decision makers typically integrate multiple pieces of information to make predictions and decisions. They also sometimes receive algorithmic advice, but often discount such advice. This usually results in less consistent and less accurate predictions than consistently using the advice. We hypothesized that individual differences on psychological traits such as dutifulness (a facet of conscientiousness), decision-making styles, and predictor validity beliefs are related to the consistent use of algorithmic advice, judgment consistency, and predictive validity. We sampled participants with hiring experience (N = 308) who predicted the performance of job applicants based on test scores and interview ratings, and they also received algorithmic advice. The results showed that more dutiful participants and participants with more accurate predictor validity beliefs used the algorithm more, and made more consistent and more accurate predictions. We did not find evidence that an intuitive decision-making style was related to these outcomes. Exploratory analyses showed that cognitive ability was positively related to the consistent use of algorithmic advice and judgment consistency, but not significantly related to predictive validity. Furthermore, the other conscientiousness facets and the general factor were similarly related to the outcome variables as dutifulness. Organizations may want to hire conscientious decision makers, and decision makers with accurate predictor validity beliefs. In addition, organizations could provide training on predictor validities.
View
... Developments in the use of this strategy include (a) the key role of sampling error, and hence the need for larger samples (e.g., Schmidt et al., 1976); (b) attenuation due to measurement error in the criterion (e.g., Viswesvaran et al., 1996) and differences in predictor variability between the research sample and applicant pool (e.g., Sackett & Yang, 2000); and (c) a move from relying on local validity evidence to cumulative evidence from other settings (e.g., Schmidt & Hunter, 1981). Meta-analytic estimates of the mean and variance of criterion-related validity of predictors have been obtained (e.g., Sackett et al., 2021;Schmidt & Hunter, 1998), as have meta-analytic estimates of the incremental validity of different predictors (e.g., Bobko et al., 1999;Cortina et al., 2000). ...
Personnel selection: A review of ways to maximize validity, diversity, and the applicant experience
Article
Full-text available
  • Mar 2023
  • PERS PSYCHOL
Personnel Psychology has a long tradition of publishing important research on personnel selection. In this article, we review some of the key questions and findings from studies published in the journal and in the selection literature more broadly. In doing so, we focus on the various decisions organizations face regarding selection procedure development (e.g., use multiple selection procedures, contextualize procedure content), administration (e.g., provide pre‐test explanations, reveal target KSAOs), and scoring (e.g., weight predictors and criteria, use artificial intelligence). Further, we focus on how these decisions affect the validity of inferences drawn from the procedures, how use of the procedures may affect organizational diversity, and how applicants experience the procedures. We also consider factors such as cost and time. Based on our review, we highlight practical implications and key directions for future research. This article is protected by copyright. All rights reserved
View
... The utility of novel constructs, however, requires demonstration of incremental prediction over existing predictors (Harms & Credé, 2010). Evidence of incremental prediction of dispositional traits above the Big Five is also critical to establish the uniqueness and utility of new predictors in work settings (e.g., Cortina et al., 2000). Furthermore, organizational scholars have recently begun to focus on personality traits outside of the Big Five (Hough et al., 2015). ...
The Bright, Dark, and Gray Sides of Risk Takers at Work: Criterion Validity of Risk Propensity for Contextual Work Performance
Article
Full-text available
Although risk takers are traditionally seen as liabilities, a growing body of research suggests that risk takers may be critical for organizational achievements because of their courage and willingness to take risks for the benefit of others. Despite the prevalence of risk taking in studies of strategic management and organizational behavior, we know very little about the implication of risk propensity on employee work performance. In this paper, we conceptualize contextual performance—behaviors that fall outside of normal job expectations—as a form of workplace risk taking. Results from a three-wave study of working adults revealed that risk propensity positively predicted both counterproductive work behaviors (CWBs) and pro-social rule-breaking behaviors (PSRBs) above the Big Five personality traits. We also found an indirect effect of risk propensity on organizational citizenship behaviors (OCBs) and PSRBs via an increased willingness to take risks at work. Taking an item-level approach, we found that the predictive validity of risk propensity was greater for specific performance items that independent judges rated as riskier. Our findings suggest that risk takers are more likely to engage in pro-social and counterproductive behaviors—particularly those that are rule-breaking or have elevated risks—in part due to an increased willingness to take risks at work.
View
View
A Causal Model of the Relationship Among Accidents, Biodata, Personality, and Cognitive Factors
Article
Full-text available
  • Feb 1989
The purpose of this research was to construct and test a causal model of the accident process. Data were gathered on 362 chemical industry workers. The causal model was analyzed and cross-validated using LISREL VI. It was proposed that social maladjustment traits, some characteristics of neurosis, cognitive ability, employee age, and job experience would have independent causal effects on the accident criterion, even when the effects of accident risk and involvement in counseling were controlled. Two rationally derived, content-validated scales based on MMPI items were created to measure social maladjustment and the aspects of neurosis that result in a state of distractibility. The results showed the causal model as a whole to be viable in the initial and cross-validation analyses, and the social maladjustment and distractibility variables were found to be significant causal parameters of accidents. This study developed a new direction for future accident research by its use of causal modeling and by the creation of two new scales for the assessment of employee accident potential.
View
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings
Article
Full-text available
  • Sep 1998
This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.
View
The Validity of Employment Interviews: A Comprehensive Review and Meta-Analysis
Article
Full-text available
  • Aug 1994
This meta-analytic review presents the findings of a project investigating the validity of the employment interview. Analyses are based on 245 coefficients derived from 86,311 individuals. Results show that interview validity depends on the content of the interview (situational, job related, or psychological), how the interview is conducted (structured vs. unstructured; board vs. individual), and the nature of the criterion (job performance, training performance, and tenure; research or administrative ratings). Situational interviews had higher validity than did job-related interviews, which, in turn, had higher validity than did psychologically based interviews. Structured interviews were found to have higher validity than unstructured interviews. Interviews showed similar validity for job performance and training performance criteria, but validity for the tenure criteria was lower.
View
The Proactive Personality Scale and Objective Job Performance Among Real Estate Agents
Article
Full-text available
  • Aug 1995
A sample of 131 real estate agents was used to examine the criterion validity of the Proactive Personality Scale (T. S. Bateman & J. M. Crant, 1993). A job performance index was computed for each agent from archival records of the number of houses sold, number of listings obtained, and commission income over a 9-month period. Experience, social desirability, general mental ability, and 2 of the Big Five factors-Conscientiousness and Extraversion-were controlled for, and the Proactive Personality Scale explained an additional 8% of the variance in the objective measure of agents' job performance. These results provide additional evidence for the criterion validity of the Proactive Personality Scale and suggest that specific personality measures can have incremental validity over the Big Five factors.
View
View
View
Conscientiousness and Performance of Sales Representatives: Test of the Mediating Effects of Goal Setting
Article
  • Oct 1993
The authors used 91 sales representatives to test a process model that assessed the relationship of conscientiousness to job performance through mediating motivational (goal-setting) variables. Linear structural equation modeling showed that sales representatives high in conscientiousness are more likely to set goals and are more likely to be committed to goals, which in turn is associated with greater sales volume and higher supervisory ratings of job performance. Results also showed that conscientiousness is directly related to supervisory ratings. Consistent with previous research, results showed that ability was also related to supervisory ratings of job performance and, to a lesser extent, sales volume. Contrary to expectations, 1 other personality construct, extraversion, was not related to sales volume or to supervisory ratings of job performance. Implications and future research needs are discussed.
View
Criterion-Related Validities of Personality Constructs and the Effect of Response Distortion on Those Validities
Article
  • Oct 1990
A review of criterion-related validities of personality constructs indicated that six constructs are useful predictors of important job-related criteria. An inventory was developed to measure the 6 constructs. In addition, 4 response validity scales were developed to measure accuracy of self-description. These scales were administered in three contexts: a concurrent criterion-related validity study, a faking experiment, and an applicant setting. Sample sizes were 9,188,245, and 125, respectively. Results showed that (a) validities were in the.20s (uncorrected for unreliability or restriction in range) against targeted criterion constructs, (b) respondents successfully distorted their self-descriptions when instructed to do so, (c) response validity scales were responsive to different types of distortion, (d) applicants' responses did not reflect evidence of distortion, and (e) validities remained stable regardless of possible distortion by respondents in either unusually positive or negative directions.
View
MODELS OF SUPERVISORY JOB-PERFORMANCE RATINGS
Article
  • Dec 1991
Proposed and evaluated in this research were causal models that included measures of cognitive ability, job knowledge, task proficiency, two temperament constructs (achievement and dependability), awards, problem behavior, and supervisor ratings. The models were tested on a sample of 4362 US Army enlisted personel in nine different jobs.
View
View