ArticlePDF Available

The Incremental Validity of Interview Scores Over and Above Cognitive Ability and Conscientiousness Scores

Authors:

Abstract

Recent research has suggested that scores on measures of cognitive ability, measures of Conscientiousness, and interview scores are positively correlated with job performance. There remains, however, a question of incremental validity: To what extent do interviews predict above and beyond cognitive ability and Conscientiousness? This question was addressed in this paper by (a) conducting meta-analyses of the relationships among cognitive ability, Conscientiousness, and interviews, (b) combining these results with predictive validity results from previous meta-analyses to form a “meta-correlation matrix” representing the relationships among cognitive ability, Conscientiousness, interviews, and job performance, and (c) performing 9 hierarchical regressions to examine the incremental validity of 3 levels of structured interviews in best, actual, and worst case scenarios for prediction. Results suggested that interview scores contribute to the prediction of job performance over and above cognitive ability and Conscientiousness to the extent that they are structured, with scores from highly structured interviews contributing substantially to prediction. Directions for future research are discussed.
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
Copyright © 2000 All Rights Reserved
... From this literature, a clear picture emerges-increasing structure in interviews (e.g., through standardization in the questions asked and/or the scoring protocols used to evaluate interviewees' answers) increases the validity and reliability of interviews (Barrick et al., 2009;Campion et al., 1997;Chapman & Zweig, 2005;Conway et al., 1995;Cortina et al., 2000;Huffcutt & Arthur, 1994;Macan, 2009;Melchers et al., 2011;Schmidt & Hunter, 1998). Structured interviews also increase fairness because unstructured interviews may increase the likelihood of sociocognitive biases that negatively affect certain groups (Buckley et al., 2007;Roth et al., 2002). ...
Article
As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate-school admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement “bias” and “fairness” to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used to help inform graduate-school admissions decisions: grade point average, personal statements, resumes/curriculum vitae, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions and highlighting areas in which future research would be beneficial.
... Early meta-analytical studies showed that job interviews often contain personality-related questions and information (e.g, Cortina et al., 2000;Huffcutt et al., 2001). Subsequent studies showed that job interviews can also be designed to specifically assess personality traits (e.g., Barrick et al., 2000;Blackman, 2002;Van Iddekinge et al., 2005) and that interviewers' (other-reported) personality ratings are less susceptible to faking than interviewees' (self-reported) personality ratings (Van Iddekinge et al., 2005). ...
... Due to its popularity, a great deal of research has been conducted to determine the best method for conducting interviews and how to improve the reliability and validity of interview evaluations. Much of this research has demonstrated that structured interviews, which use structured rules for eliciting, observing, and evaluating responses, are superior to unstructured interviews [40][41][42], that they provide incremental validity over personality tests and cognitive ability tests [43,44] and that they can be used effectively to measure different constructs and predict different criteria [45,46]. Structured interviews are effective because the predetermined rules and formal structure reduce the number of extraneous factors that can affect an interviewer's decisions about what questions to ask and how to evaluate performance. ...
Article
Full-text available
The popularity of forensic science television programs has created a hiring challenge for forensic science laboratories. Laboratories receive unprecedented numbers of applicants, yet struggle to identify highly-qualified candidates. Forensic examiners must possess a unique set of knowledge, skills, and abilities (KSAs) plus other characteristics. They must be critical thinkers, detail-oriented, decisive, and self-regulated; be able to communicate clearly and effectively within the laboratory, with customers (typically investigators or attorneys), and in the courtroom; and demonstrate the required core competencies. Currently, no validated instruments, standardized job descriptions, or lists of KSAs exist to aid in forensic science personnel recruitment and assessment, often resulting in high turnover leading to costly new recruitment and training cycles. This article describes how industrial/organizational psychology develops methods and tools to improve workforce selection; describes some tools currently in use in forensic science; and advocates for research and development of better tools for use in forensic science.
... However, in a criterion-related validity study involving managers, where the criterion was an assessment of teamwork skills, the correlation between the SI and GMA was not significant (Sue-Chan and Latham, 2004). This finding is consistent with a series of metaanalyses conducted by Cortina et al. (2000). They found that highly structured interviews have incremental validity beyond cognitive ability. ...
Article
Full-text available
Four experiments were conducted to determine whether participants’ awareness of the performance criterion on which they were being evaluated results in higher scores on a criterion valid situational interview (SI) where each question either contains or does not contain a dilemma. In the first experiment there was no significant difference between those who were or were not informed of the performance criterion that the SI questions predicted. Experiment 2 replicated this finding. In each instance the SI questions in these two experiments contained a dilemma. In a third experiment, participants were randomly assigned to a 2 (knowledge/no knowledge provided of the criterion) X 2 (SI dilemma/no dilemma) design. Knowledge of the criterion increased interview scores only when the questions did not contain a dilemma. The fourth experiment revealed that including a dilemma in a SI question attenuates the ATIC-SI relationship when participants must identify rather than be informed of the performance criterion that the SI has been developed to assess.
... Structured interviews also offer excellent incremental validity, or additional predictive power, to selection procedures that use cognitive ability as a lone predictor of job performance (Campion et al., 1994Cortina et al., 2000;Dipboye, 1994;Schmidt et al., 2016). In a recent meta-analytic review by Schmidt et al. (2016) where they examined 100 years of research in personnel psychology, they found that behind tests of general mental ability and integrity tests (mean validity of .78), ...
Thesis
Full-text available
Despite the numerous benefits structured interviews offer, prior research and literature has shown that hiring professionals are inclined to use unstructured interviews over structured interviews. While unstructured interviews are convenient, they pose several severe limitations when compared to structured interviews. Namely, unstructured interviews can result in adverse legal outcomes, significantly worse predictive validity, and difficulty in comparing applicants. Our research examines if three variables: Interviewer conventional personality, interviewer training, and recording improve the acceptance and use of structured interviews. Our study included 171 hiring managers from the SIOP user directory, SIUE alumni from the I/O psychology program, Human Resource managers from the Society for Human Resources Management (SHRM), and managers recruited through Amazon’s Mechanical Turk (MTurk). Our survey found that interviewer training helped improve acceptance and content standardization of the interview. Neither interviewer conventional personality nor recording of the interview improved structured interview acceptance or use. However, our results did find that structured interviews are used more often than previously thought and that recording was used by a third of respondents to compare applicants or for legal defense. We found marginal support that interviewers who recorded their interviews reported using lower content standardization. Given our findings, we recommend that interviewer training be used to improve structured interview acceptance and use above and beyond recording.
... Further, interviewees with high self-reported agreeableness display more eye contact, open postures, and friendly facial expressions (i.e., nonverbal behavior; Kristof-Brown et al., 2002). These relationships are important, as interviewee performance (i.e., verbal, paraverbal, and nonverbal behavior) influences interviewer judgments of interviewee personality, judgments that are commonly made in personnel selection (Cortina et al., 2000;Huffcutt et al., 2001). For instance, certain paraverbal behaviors (i.e., pitch variability, higher speech rate, amplitude variability; DeGroot & Gooty, 2009) influence interviewer judgments of extraversion, emotional stability, and openness. ...
Preprint
Full-text available
Organizations are increasingly adopting automated video interviews (AVIs) to screen job applicants despite a paucity of research on their reliability, validity, and generalizability. In this study, we address this gap by developing AVIs that use verbal, paraverbal, and nonverbal behaviors extracted from video interviews to assess Big Five personality traits. We developed and validated machine learning models within (using nested cross-validation) and across three separate samples of mock video interviews (total N = 1,073). Also, we examined their test–retest reliability in a fourth sample (N = 99). In general, we found that the AVI personality assessments exhibited stronger evidence of validity when they were trained on interviewer-reports rather than self-reports. When cross-validated in the other samples, AVI personality assessments trained on interviewer-reports had mixed evidence of reliability, exhibited consistent convergent and discriminant relations, used predictors that appear to be conceptually relevant to the focal traits, and predicted academic outcomes. On the other hand, there was little evidence of reliability or validity for the AVIs trained on self-reports. We discuss the implications for future work on AVIs and personality theory, and provide practical recommendations for the vendors marketing such approaches and organizations considering adopting them.
Article
Personal characteristics (e.g., cognitive ability (CA), conscientiousness, self-perceived abilities (SPA)) predict differences in school performance. While genetic influences on CA and conscientiousness are unequivocal, origins of SPA have long been assumed to be environmental, however substantial genetic contributions have been detected in behavioral genetic analyses. In this study, we investigate the common etiology of these three predictors and their association with school grades in math and German as well as how genetic and environmental contributions differ between age groups and school domains. The sample consists of 2101 twin pairs (aged 11 and 17) and their siblings participating in the German TwinLife study. Using a multivariate twin-sibling design, we analyze common genetic and environmental effects on CA, conscientiousness, SPA and school grades in math and German in two age groups. Results confirm genetic effects for all three predictors (42–51% for CA, 31–42% for conscientiousness, 26–48% for SPA) as well as (non-) shared environmental effects. Multivariate analyses demonstrate that the interrelation between predictors and their association with grades is largely due to common genetic effects. Additional non-shared environmental effects and differences across age groups and school domains are small. We discuss possible underlying mechanisms and implications on individual differences in school performance leading to educational inequality.
Article
Hospital and regulatory organizations recognize the downstream impact of interpersonal relationships on patient safety. There are many contributors to interpersonal dynamics, one of which includes disruptive physicians. This can stem from overt or passive behaviors and has been categorized in the literature by terms that include disruptive physicians, professionalism and burnout. The contributing factor factors to such behavior are varied and may include burnout, stress, skill deficiencies, and impairment. These behaviors can be impacted through personal and systemic influences that are difficult for administrators to elucidate due to physician and organizational confidentiality concerns. However, there are opportunities and tools where radiology practice leaders and organizations can intervene to assist the disruptive radiologist. These include during residency/fellowship training, clarifying employee expectations during the time of radiologist hiring, offering interventions during practice, providing resources such as employee assistance programs, and addressing systemic challenges. Utilizing these tool sets can improve healthcare team dynamics and increase both staff and physician retention.
Article
It is well‐established that selection decisions can be improved using multiple non‐redundant assessments. Two such assessments are cognitive ability and conscientiousness. Though meta‐analytic findings demonstrate little or no relationship between cognitive ability and conscientiousness, faking research suggests the two variables are related when test‐takers are motivated to fake. We extend this logic by showing that incremental validity for conscientiousness declines when respondents fake, due to enhanced collinearity between conscientiousness and cognitive ability. Three studies, employing within‐subjects designs and utilizing three different faking conditions, reveal a consistent increase in collinearity between conscientiousness and cognitive ability when respondents are motivated to fake, leading to reduced incremental validity of conscientiousness beyond cognitive ability in predicting the performance. Implications for selection systems are discussed.
Article
Full-text available
This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.
Article
Full-text available
This meta-analytic review presents the findings of a project investigating the validity of the employment interview. Analyses are based on 245 coefficients derived from 86,311 individuals. Results show that interview validity depends on the content of the interview (situational, job related, or psychological), how the interview is conducted (structured vs. unstructured; board vs. individual), and the nature of the criterion (job performance, training performance, and tenure; research or administrative ratings). Situational interviews had higher validity than did job-related interviews, which, in turn, had higher validity than did psychologically based interviews. Structured interviews were found to have higher validity than unstructured interviews. Interviews showed similar validity for job performance and training performance criteria, but validity for the tenure criteria was lower.
Article
Full-text available
A sample of 131 real estate agents was used to examine the criterion validity of the Proactive Personality Scale (T. S. Bateman & J. M. Crant, 1993). A job performance index was computed for each agent from archival records of the number of houses sold, number of listings obtained, and commission income over a 9-month period. Experience, social desirability, general mental ability, and 2 of the Big Five factors-Conscientiousness and Extraversion-were controlled for, and the Proactive Personality Scale explained an additional 8% of the variance in the objective measure of agents' job performance. These results provide additional evidence for the criterion validity of the Proactive Personality Scale and suggest that specific personality measures can have incremental validity over the Big Five factors.
Article
The authors used 91 sales representatives to test a process model that assessed the relationship of conscientiousness to job performance through mediating motivational (goal-setting) variables. Linear structural equation modeling showed that sales representatives high in conscientiousness are more likely to set goals and are more likely to be committed to goals, which in turn is associated with greater sales volume and higher supervisory ratings of job performance. Results also showed that conscientiousness is directly related to supervisory ratings. Consistent with previous research, results showed that ability was also related to supervisory ratings of job performance and, to a lesser extent, sales volume. Contrary to expectations, 1 other personality construct, extraversion, was not related to sales volume or to supervisory ratings of job performance. Implications and future research needs are discussed.
Article
A review of criterion-related validities of personality constructs indicated that six constructs are useful predictors of important job-related criteria. An inventory was developed to measure the 6 constructs. In addition, 4 response validity scales were developed to measure accuracy of self-description. These scales were administered in three contexts: a concurrent criterion-related validity study, a faking experiment, and an applicant setting. Sample sizes were 9,188,245, and 125, respectively. Results showed that (a) validities were in the.20s (uncorrected for unreliability or restriction in range) against targeted criterion constructs, (b) respondents successfully distorted their self-descriptions when instructed to do so, (c) response validity scales were responsive to different types of distortion, (d) applicants' responses did not reflect evidence of distortion, and (e) validities remained stable regardless of possible distortion by respondents in either unusually positive or negative directions.
Article
Proposed and evaluated in this research were causal models that included measures of cognitive ability, job knowledge, task proficiency, two temperament constructs (achievement and dependability), awards, problem behavior, and supervisor ratings. The models were tested on a sample of 4362 US Army enlisted personel in nine different jobs.
Article
In the course of validating a test battery for the selection of police officers (Baehr, Furcon, & Froemel, 1969), project personnel were searching for objectively scored, pencil-and-paper measures of behavior variables. One product of this search was an adaptation of the Arrow-Dot subtest of the four-part IES test battery (Dombrose & Slobin, 1958). This adaptation was designed for group administration and for improved reliability as a possible predictor of performance. Intercorrelational analysis and factor analysis of intercorrelations with other potential predictor variables and performance criteria provided some information on the nature of the Arrow-Dot scores. The test's ability to predict police officers' performance was assessed through multiple regression analyses of its scores both as independent predictors and as contributors of unique variance in a wider predictor battery. Its usefulness for diagnosis of specific problems in police officers' performance was also investigated. Application in five additional studies permitted an exploration of the “characteristic” Arrow-Dot dimension profile for police officers.