No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
The Comprehensive Infrequency/Frequency Item Repository (CIFR): An online database of items for detecting careless/insufficient-effort responders in survey data
Abstract
A promising method for detecting careless or insufficient effort (C/IE) responders is the infrequency/frequency-item method. Infrequency items are items that should be endorsed by almost no one (e.g., "I like being frustrated") and frequency items are items that should be endorsed by almost everyone (e.g., "I keep some of my thoughts to myself"). Participants are flagged as C/IE responders if they tend to agree with the infrequency items and tend to disagree with the frequency items. Here, we introduce the Comprehensive Infrequency/Frequency Item Repository (CIFR)--an online database of 660 infrequency/frequency items. The purpose of CIFR is to provide researchers with a tool for selecting infrequency/frequency items that are not only valid but also match the content of the other items in their surveys. An initial validation effort based on 1166 participants indicated that most of the infrequency items in CIFR are, in fact, infrequent, and most of the frequency items in CIFR are, in fact, frequent. Although additional validation work is necessary, the present results indicate that CIFR can be a valuable tool for researchers who wish to ensure the quality of their data.
... This general approach is used in some clinical instruments, such as the PAI's Infrequency and MMPI-3's F scales, and typically demonstrates relatively high-though far from perfect-convergence with inconsistency scales that are based on highly correlated item pairs. Recently, researchers have attempted to create a set of such items, the Comprehensive Infrequency/ Frequency Item Repository (CIFR; Kay & Saucier, 2023), which provides a wealth of items that can be embedded within survey data to detect careless responding. The CIFR pool contains 660 frequent/infrequent items covering a range of content domains that allows researchers to match item content to the types of data they plan to collect. ...
... The CIFR pool contains 660 frequent/infrequent items covering a range of content domains that allows researchers to match item content to the types of data they plan to collect. Using a large (n > 1,100 participants) initial validation sample, the repository authors found that most of these items had response rates that are consistent with expectations (Kay & Saucier, 2023). ...
... We then tested whether scale scores could differentiate between genuine and entirely (100%) randomized data, as well as partially randomized data (66%, 33%). Next, we examined the incremental validity of the DRIP relative to a short inconsistency scale embedded in the BFI-2 drawing on items from the CIFR (Kay & Saucier, 2023). ...
... The purpose of the present set of six studies is to validate two scales for detecting C/IE responders: the 14-item Invalid Responding Inventory for Statements (IDRIS; Kay, 2021) and the 6-item Invalid Responding Inventory for Adjectives (IDRIA; Kay, 2023). Despite recently being included in the Comprehensive Infrequency/Frequency Item Repository (CIFR; Kay & Saucier, 2023) -an online database of 660 infrequency/frequency items -these two scales have not undergone any form of formal validation. Here, we provide this validation by examining the IDRIS and IDRIA in relation to several well-established indices of C/IE responding. ...
... Accordingly, the IDRIS includes seven infrequency statements (e.g., "I am older than my parents") and seven frequency statements (e.g., "I can remember the names of most of my close family members") (Appendix 1). The IDRIA, on the other hand, was developed to be used with scales composed of adjectives, such as the Big Five 1 See Kay and Saucier (2023) for a discussion of the use of the phrase "frequency item" versus "negatively-keyed infrequency item". 2 Of course, there may be some participants who genuinely agree with the statement "I will be punished for meeting the requirements of my job" and some participants who genuinely disagree with the statement "It feels good to be appreciated". People invariably differ, and it is nearly impossible to find items that all participants will agree or disagree with. ...
... This appears to be the case for the IDRIS and IDRIA. Specifically, Kay and Saucier (2023) demonstrated that, on a five-point Likert scale, participants provided an average response of 1.08 to 1.42 to the infrequency items from the IDRIS and an average response of 4.44 to 4.72 to the frequency items from the IDRIS. Likewise, participants provided an average response of 1.47 to 1.91 to the infrequency items from the IDRIA and an average response of 4.60 to 4.81 to the frequency items from the IDRIA. ...
To detect careless and insufficient effort (C/IE) survey responders, researchers can use infrequency items – items that almost no one agrees with (e.g., “When a friend greets me, I generally try to say nothing back”) – and frequency items – items that almost everyone agrees with (e.g., “I try to listen when someone I care about is telling me something”). Here, we provide initial validation for two sets of these items: the 14-item Invalid Responding Inventory for Statements (IDRIS) and the 6-item Invalid Responding Inventory for Adjectives (IDRIA). Across six studies (N1 = 536; N2 = 701; N3 = 500; N4 = 499; N5 = 629, N6 = 562), we found consistent evidence that the IDRIS is capable of detecting C/IE responding among statement-based scales (e.g., the HEXACO-PI-R) and the IDRIA is capable of detecting C/IE responding among both adjective-based scales (e.g., the Lex-20) and adjective-derived scales (e.g., the BFI-2). These findings were robust across different analytic approaches (e.g., Pearson correlations; Spearman rank-order correlations), different indices of C/IE responding (e.g., person- total correlations; semantic synonyms; horizontal cursor variability), and different sample types (e.g., US undergraduate students; Nigerian survey panel participants). Taken together, these results provide promising evidence for the utility of the IDRIS and IDRIA in detecting C/IE responding.
... After excluding participants who responded incorrectly to two or more of four instructed response items embedded in the survey (n = 2; Curran, 2016), completed the survey in less than one-third of the median response time (n = 4; Bedford-Petersen & Saucier, 2021), provided the same response to over half of the survey items in a row (n = 10;Johnson, 2005), exhibited a response standard deviation of less than .50 (n = 0; Thalmayer & Saucier, 2014; see alsoDunn et al., 2018), and provided an average response of 0 or greater to four infrequency/frequency items embedded in the survey after reverse scoring the frequency items (n = 11;Kay & Saucier, 2023), ...
Hikikomori is a form of extreme social withdrawal that is characterized by self-isolating in one's living quarters. While past research had found a correlation between Hikikomori and internet addiction, the research was limited to a sample of college students in Japan. With cases of both Hikikomori and internet addiction being recognized across the globe, more research is required to investigate this correlation in non-Japanese countries. In the current study, we administered the Hikikomori Questionnaire and Young's Internet Addiction Test to 437 US college students recruited through Prolific. We found a sizeable positive correlation between Hikikomori and internet addiction (r = .30). We also found that the correlation among the US college students in our sample did not significantly differ from the correlation among the Japanese college students in the prior study (r = .39). Although more research is required to further understand the relationship, the present results indicate that extreme social withdrawal could be a risk factor for internet addiction in the US (and vice versa).
... After excluding participants who failed two instructed response items included in the survey (n = 4; Curran, 2016), demonstrated a response standard deviation of less than .50 (n = 1; Thalmayer & Saucier, 2014; see also Dunn et al., 2018), or provided an average response greater than zero to the six infrequency/frequency items embedded in the survey (n = 6; Kay & Saucier, 2023), the sample included 299 participants. A summary of the demographics for the sample can be found in Table 1. ...
Researchers have long argued that conspiracist ideation and prejudice are intimately linked, but the empirical research supporting this notion is relatively limited. Past studies have shown that conspiracist ideation is associated with Islamophobia, antisemitism, and xenophobia but few other forms of prejudice. The present project expands the scope of this research by examining the associations of three measures of conspiracist ideation with twenty measures of prejudice. Across two studies-one using a sample of undergraduate students (N = 299) and one using a sample of CloudResearch Connect participants demographically matched to the US population (N = 538)-we found that conspiracist ideation is associated with a diverse array of prejudicial beliefs, including hostile sexism, modern homonegativity, transnegativity, modern racism towards Black people, antisemitism, and classism, among many other forms of prejudice. We also found that conspiracist ideation is highly associated with generalized prejudice, represented by a latent factor capturing the shared variance among the twenty prejudice measures. Taken together, these findings indicate that there is a close relationship between conspiracist ideation and prejudice.
... Over five hundred (N = 504) participants responded to the first survey. After excluding participants who failed two or more of the six instructed response items included in the survey (n = 4;Curran, 2016;Kay & Saucier, 2023), responded faster than one-third of the median response time (n = 3; Bedford-Petersen & Saucier, 2021), provided the same response to over half of the items in the survey in a row (n = 3;Johnson, 2005), demonstrated a response standard deviation of less than .50 (n = 0; Thalmayer & Saucier, 2014; see alsoDunn et al., 2018), or provided an average response greater than zero to the six infrequency/frequency items embedded in the survey (n = 2;, the sample included 492 participants (MAge = 43.79, ...
Choosing a short-form measure of conspiracist ideation is fraught. Despite there being numerous scales to choose from, little work has been done to compare their psychometric properties. To address this shortcoming, we compared the internal consistency, two-week test-retest reliability, criterion validity, and construct validity of five short-form conspiracist ideation measures: the Generic Conspiracist Beliefs Scale – 5 (GCB-5), Conspiracy Mentality Questionnaire (CMQ), General Measure of Conspiracism (GMC), American Conspiracy Thinking Scale (ACTS), and One-Item Conspiracy Measure (1CM). The results of our investigation indicated that all five of the scales are reliable and valid measures of conspiracist ideation. That said, the GCB-5 tended to perform the best while the 1CM tended to perform the worst. We conclude our investigation by discussing trade-offs among the five scales, as well as providing recommendations for future research.
Surveys are one of the most popular ways to collect employee information. Because of their widespread use, data quality is an increasingly important concern. The purpose of this paper is to (1) introduce the intra-individual response variability (IRV) index as an easily calculated and flexible way to detect insufficient effort responding (IER); (2) examine the extent to which various IER indices detect the same or different respondents engaging in IER behavior; and (3) investigate relationships between individual differences and commonly used IER indices to better understand systematic and theoretically relevant IER behavior. In a two-part study, 199 undergraduates responded to questionnaires online, and various IER indices were calculated. The IRV index identifies different respondents than other IER indices. Values on the IRV index (as well as other IER indices) are related to scores on theoretically meaningful individual differences in conscientiousness, agreeableness, and boredom proneness. This study provides researchers with a robust, easily calculated, and flexible means for screening questionnaire data for IER behavior. Practical recommendations for finding and making decisions about IER behavior patterns are provided. This study introduces the IRV index, an extension of the long string, used to identify survey research participants who likely engaged in one type of IER behavior. It is also one of the first studies to evaluate the extent to which IER indices identify different respondents as having engaged in IER and provides additional evidence that values on these indices are related to individual differences.
Insufficient effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, 2012) to surveys has
largely been assumed to be a source of random measurement error that attenuates associations between
substantive measures. The current article, however, illustrates how and when the presence of IER can
produce a systematic bias that inflates observed correlations between substantive measures. Noting that
inattentive responses as a whole generally congregate around the midpoint of a Likert scale, we propose
that Mattentive, defined as the mean score of attentive respondents on a substantive measure, will be
negatively related to IER’s confounding effect on substantive measures (i.e., correlations between IER
and a given substantive measure will become less positive [or more negative] as Mattentive increases).
Results from a personality questionnaire (Study 1) and a simulation (Study 2) consistently support the
hypothesized confounding influence of IER. Using an employee sample (Study 3), we demonstrated how
IER can confound bivariate relationships between substantive measures. Together, these studies indicate
that IER can inflate the strength of observed relationships when scale means depart from the scale
midpoints, resulting in an inflated Type I error rate. This challenges the traditional view that IER
attenuates observed bivariate correlations. These findings highlight situations where IER may be a
methodological nuisance, while underscoring the need for survey administrators and researchers to deter
and detect IER in surveys. The current article serves as a wake-up call for researchers and practitioners
to more closely examine IER in their data.
Purpose
Insufficient effort responding (IER), which occurs due to a lack of motivation to comply with survey instructions and to correctly interpret item content, represents a serious problem for researchers and practitioners who employ survey methodology (Huang et al. 2012). Extending prior research, we examine the validity of the infrequency approach to detecting IER and assess participant reactions to such an approach.
Design/Methodology/Approach
Two online surveys (Studies 1 and 2) completed by employed undergraduates were utilized to assess the validity of the infrequency approach. An on-line survey of paid participants (Study 3) and a paper-and-pencil survey in an organization (Study 4) were conducted to evaluate participant reactions, using random assignment into survey conditions that either did or did not contain infrequency items.
Findings
Studies 1 and 2 provided evidence for the reliability, unidimensionality, and criterion-related validity of the infrequency scales. Study 3 and Study 4 showed that surveys that contained infrequency items did not lead to more negative reactions than did surveys that did not contain such items.
Implications
The current findings provide evidence of the effectiveness and feasibility of the infrequency approach for detecting IER, supporting its application in low-stakes organizational survey contexts.
Originality/Value
The current studies provide a more in-depth examination of the infrequency approach to IER detection than had been done in prior research. In particular, the evaluation of participant reactions to infrequency scales represents a novel contribution to the IER literature.
Data collected through self-report questionnaires are particularly susceptible to inappropriate or random responding, and such invalid data increase noise and attenuate true statistical relationships. While many researchers studying schizotypy have employed infrequency measures to exclude participants, such measures are not universally employed. Moreover, some researchers have even outright questioned whether validity scales are warranted. Here, we show the effect of invalid responses on the relationship between schizotypy and hedonic reaction. For valid responders, negative schizotypal traits were inversely related to both anticipatory and consummatory pleasure (p < .01). Invalid responses were found for 23% of respondents, and within these subjects, no relationship was found between any of the measures. When the valid and invalid respondents were pooled, the relationship was dampened. Furthermore, linear multiple regression modeling showed that validity trended toward moderating the relationship between the variables of interest. These data highlight the importance of screening for, and excluding, invalid responses in schizotypy research. Our results also affirm that screening for random responding is effective and warranted. Implications for future studies employing questionnaire-based methods are discussed. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
Random responding to psychological inventories is a long-standing concern among clinical practitioners and researchers interested in interpreting idiographic data, but it is typically viewed as having only a minor impact on the statistical inferences drawn from nomothetic data. This article explores the impact of random responding on the size and direction of correlations observed between multi-item inventory scores. Random responses to individual items result in nonrandomly distributed inventory-level scores. Therefore, even low base rates of random responding can significantly affect the statistical inferences made from inventory-level data. Study 1 uses simulations to show that even low base rates of random responding can significantly affect observed correlations, especially when the inventories in question assess low or high base rate phenomena. Study 2 uses archival data to illustrate the moderating effect of random responding on observed correlations in two samples.
A review of criterion-related validities of personality constructs indicated that 6 constructs are useful predictors of important job-related criteria. An inventory was developed to measure the 6 constructs. In addition, 4 response validity scales were developed to measure accuracy of self-description. These scales were administered in 3 contexts: a concurrent criterion-related validity study, a faking experiment, and an applicant setting. Sample sizes were 9,188, 245, and 125, respectively. Results showed that (a) validities were in the .20s (uncorrected for unreliability or restriction in range) against targeted criterion constructs, (b) respondents successfully distorted their self-descriptions when instructed to do so, (c) responsive validity scales were responsive to different types of distortion, (d) applicants' responses did not reflect evidence of distortion, and (e) validities remained stable regardless of possible distortion by respondents in either unusually positive or negative directions. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
When data are collected via anonymous Internet surveys, particularly under conditions of obligatory participation (such as with student samples), data quality can be a concern. However, little guidance exists in the published literature regarding techniques for detecting careless responses. Previously several potential approaches have been suggested for identifying careless respondents via indices computed from the data, yet almost no prior work has examined the relationships among these indicators or the types of data patterns identified by each. In 2 studies, we examined several methods for identifying careless responses, including (a) special items designed to detect careless response, (b) response consistency indices formed from responses to typical survey items, (c) multivariate outlier analysis, (d) response time, and (e) self-reported diligence. Results indicated that there are two distinct patterns of careless response (random and nonrandom) and that different indices are needed to identify these different response patterns. We also found that approximately 10%-12% of undergraduates completing a lengthy survey for course credit were identified as careless responders. In Study 2, we simulated data with known random response patterns to determine the efficacy of several indicators of careless response. We found that the nature of the data strongly influenced the efficacy of the indices to identify careless responses. Recommendations include using identified rather than anonymous responses, incorporating instructed response items before data collection, as well as computing consistency indices and multivariate outlier analysis to ensure high-quality data. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
Seven experts on personality measurement here discuss the viability of public-domain personality measures, focusing on the International Personality Item Pool (IPIP) as a prototype. Since its inception in 1996, the use of items and scales from the IPIP has increased dramatically. Items from the IPIP have been translated from English into more than 25 other languages. Currently over 80 publications using IPIP scales are listed at the IPIP Web site (http://ipip.ori.org), and the rate of IPIP-related publications has been increasing rapidly. The growing popularity of the IPIP can be attributed to five factors: (1) It is cost free; (2) its items can be obtained instantaneously via the Internet; (3) it includes over 2000 items, all easily available for inspection; (4) scoring keys for IPIP scales are provided; and (5) its items can be presented in any order, interspersed with other items, reworded, translated into other languages, and administered on the World Wide Web without asking permission of anyone. The unrestricted availability of the IPIP raises concerns about possible misuse by unqualified persons, and the freedom of researchers to use the IPIP in idiosyncratic ways raises the possibility of fragmentation rather than scientific unification in personality research.
Despite evidence of widespread belief in conspiracy theories, there remains a dearth of research on the individual difference correlates of conspiracist ideation. In two studies, we sought to overcome this limitation by examining correlations between conspiracist ideation and a range of individual psychological factors. In Study 1, 817 Britons indicated their agreement with conspiracist ideation concerning the July 7, 2005 (7/7), London bombings, and completed a battery of individual difference scales. Results showed that stronger belief in 7/7 conspiracy theories was predicted by stronger belief in other real-world conspiracy theories, greater exposure to conspiracist ideation, higher political cynicism, greater support for democratic principles, more negative attitudes to authority, lower self-esteem, and lower Agreeableness. In Study 2, 281 Austrians indicated their agreement with an entirely fictitious conspiracy theory and completed a battery of individual difference measures not examined in Study 1. Results showed that belief in the entirely fictitious conspiracy theory was significantly associated with stronger belief in other real-world conspiracy theories, stronger paranormal beliefs, and lower crystallized intelligence. These results are discussed in terms of the potential of identifying individual difference constellations among conspiracy theorists.
A new self-report assessment of the basic traits of psychopathy was developed with a general trait model of personality (five-factor model [FFM]) as a framework. Scales were written to assess maladaptive variants of the 18 FFM traits that are robustly related to psychopathy across a variety of perspectives including empirical correlations, expert ratings, and translations of extant assessments. Across 3 independent undergraduate samples (N = 210-354), the Elemental Psychopathy Assessment (EPA) scales proved to be internally consistent and unidimensional, and were strongly related to the original FFM scales from which they were derived (mean convergent r = .66). The EPA scales also demonstrated substantial incremental validity in the prediction of existing psychopathy measures over their FFM counterparts. When summed to form a psychopathy total score, the EPA was substantially correlated with 3 commonly used psychopathy measures (mean r = .81). Finally, in a small male forensic sample (N = 70), the EPA was significantly correlated with scores on a widely used self-report psychopathy measure, disciplinary infractions, alcohol use, and antisocial behavior. The EPA provides an opportunity to examine psychopathy and its nomological network through smaller, more basic units of personality rather than by scales or factors that blend these elements.
The present cross-sectional study (NParticipants = 397; NInformants = 460) examined the association of both grandiose narcissism and vulnerable narcissism with conspiracy beliefs in the context of four theoretically-relevant mediators. Participants who were higher in grandiose narcissism and vulnerable narcissism were more likely to believe in conspiracy theories, seemingly because they were more likely to hold unusual beliefs. There was, likewise, some evidence to suggest that vulnerable narcissists believe in conspiracy theories because they suffer from paranoia, whereas grandiose narcissists believe in conspiracy theories because of a desire to be unique. Together, these results suggest that the conspiracist ideation seen among grandiose and vulnerable narcissists is a consequence of features that are shared between and unique to each of the traits.
Participant carelessness is a source of invalidity in psychological data (Huang, Liu, & Bowling, 2015), and many methods have been created to screen for this carelessness (Curran, 2016; Johnson, 2005). These include items that researchers presume thoughtful individuals will answer in a given way (e.g., disagreement with “I am paid biweekly by leprechauns”, Meade & Craig, 2012). This paper reports on two samples in which individuals spoke aloud a series of these questions, and found that (a) individuals do occasionally report valid justifications for presumed invalid responses, (b) there is relatively high variance in this behavior over different items, and (c) items developed for this specific purpose tend to work better than those drawn from other sources or created ad-hoc.
Self-report data collections, particularly through online measures, are ubiquitous in both experimental and non-experimental psychology. Invalid data can be present in such data collections for a number of reasons. One reason is careless or insufficient effort (C/IE) responding. The past decade has seen a rise in research on techniques to detect and remove these data before normal analysis (Huang, Curran, Keeney, Poposki, & DeShon, 2012; Johnson, 2005; Meade & Craig, 2012). The rigorous use of these techniques is valuable tool for the removal of error that can impact survey results (Huang, Liu, & Bowling, 2015). This research has encompassed a number of sub-fields of psychology, and this paper aims to integrate different perspectives into a review and assessment of current techniques, an introduction of new techniques, and a generation of recommendations for practical use. Concerns about C/IE responding are a factor any time self-report data are collected, and all such researchers should be well-versed on methods to detect this pattern of response.
A review of criterion-related validities of personality constructs indicated that six constructs are useful predictors of important job-related criteria. An inventory was developed to measure the 6 constructs. In addition, 4 response validity scales were developed to measure accuracy of self-description. These scales were administered in three contexts: a concurrent criterion-related validity study, a faking experiment, and an applicant setting. Sample sizes were 9,188,245, and 125, respectively. Results showed that (a) validities were in the.20s (uncorrected for unreliability or restriction in range) against targeted criterion constructs, (b) respondents successfully distorted their self-descriptions when instructed to do so, (c) response validity scales were responsive to different types of distortion, (d) applicants' responses did not reflect evidence of distortion, and (e) validities remained stable regardless of possible distortion by respondents in either unusually positive or negative directions.
The current studies examined the adverse effects of inattentive responding on compliance with study tasks, data quality, correlational analyses, experimental manipulations, and statistical power. Results suggested that 3–9% of respondents engaged in highly inattentive responding, forming latent classes consistent with prior work that converged across existing indices (e.g., long-string index, multivariate outliers, even–odd consistency, psychometric synonyms and antonyms) and new measures of inattention (the Attentive Responding Scale and the Directed Questions Scale). Inattentive respondents provided self-report data of markedly poorer quality, sufficient to obscure meaningful regression results as well as the effects of experimental manipulations. Screening out inattentive respondents improved statistical power, helping to mitigate the notable drops in power and estimated effect sizes caused by inattention.
A frequently occurring phenomenon in factor and cluster analysis of personality or attitude scale items is that all or nearly all questionnaire items that are nega tively keyed will define a single factor. Although sub stantive interpretations of these negative factors are usually attempted, this study demonstrates that the negative factor could be produced by a relatively small portion of the respondents who fail to attend to the negative-positive wording of the items. Data were generated using three different correlation matrices, which demonstrated that regardless of data source, when only 10% of the respondents are careless in this fashion, a clearly definable negative factor is gener ated. Recommendations for instrument development and data editing are presented.
This article presents a test of a recently developed method for studying people's digital literacy. Self-reported instruments pose a risk of misreporting by respondents. Participants in the study are presented with items that are used to construct a previously established digital literacy measure, interspersed with strategically developed bogus items, to test how the latter measure up against the former. The bogus terms score the least in comparison with other items sug- gesting that the majority of people do not make up their responses to these questions, thereby adding support to the utility of the formerly proposed instrument. Results also show a consid- erable discrepancy between familiarity of older Internet-related terms and newer web-based concepts offering a more nuanced instrument for studies of users' digital literacy.
63 female undergraduates were given an identical general horoscope interpretation but were assigned randomly to 1 of 3 specificity conditions in which they were told the interpretation was (a) generally true of people, (b) derived for them on the basis of the year and month of their birth, or (c) derived for them on the basis of the year, month, and day of their birth. Results show that the more specific birth time referent the S ascribed to the astrological interpretation, the more the interpretation was accepted as an accurate description of the S's personality (p < .0001). A discussion is made of how horoscopes may achieve "verification" or acceptance because of situation factors alone, rather than any actual relationship between astrological interpretations and an individual's observed personality. (22 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Many self-report measures include some items worded in the direction opposite to that of other items. These so-called reverse-worded (RW) items can reduce the reliability and validity of a scale, and frequently form a separate method factor that does not appear to be substantively meaningful. One possible explanation for factors defined by RW items is respondent carelessness. The purpose of the present study was to evaluate whether relatively few careless responders to RW items can influence confirmatory-factor-analysis model fit enough that researchers would likely reject a one-factor model for a unidimensional scale. Results based on simulations indicated that if at least about 10% of participants respond to RW items carelessly, researchers are likely to reject a one-factor model for a unidimensional scale.
The research described in this article estimated the relative incidence of protocols invalidated by linguistic incompetence, inattentiveness, and intentional misrepresentation in Web-based versus paper-and-pencil personality measures. Estimates of protocol invalidity were derived from a sample of 23,994 protocols produced by individuals who completed an on-line version of the 300-item IPIP representation of the NEO-PI-R (Goldberg, 1999). Approximately 3.8% of the protocols were judged to be products of repeat participants, many of whom apparently resubmitted after changing some of their answers. Among non-duplicate protocols, about 3.5% came from individuals who apparently selected a response option repeatedly without reading the item, compared to .9% in a sample of paper-and-pencil protocols. The missing response rate was 1.2%, which is 2–10 times higher than the rate found in several samples of paper-and-pencil inventories of comparable length. Two measures of response consistency indicated that perhaps 1% of the protocols were invalid due to linguistic incompetence or inattentive responding, but that Web participants were as consistent as individuals responding to a paper-and-pencil inventory. Inconsistency did not affect factorial structure and was found to be related positively to neuroticism and negatively to openness to experience. Intentional misrepresentation was not studied directly, but arguments for a low incidence of misrepresentation are presented. Methods for preventing, detecting, and handling invalid response patterns are discussed. Suggested for future research are studies that assess the moderating effects of linguistic incompetence, inattentiveness, and intentional misrepresentation on agreement between self-report and acquaintance judgments about personality.
Participants are not always as diligent in reading and following instructions as experimenters would like them to be. When participants fail to follow instructions, this increases noise and decreases the validity of their data. This paper presents and validates a new tool for detecting participants who are not following instructions – the Instructional manipulation check (IMC). We demonstrate how the inclusion of an IMC can increase statistical power and reliability of a dataset.
Behavioral scientists routinely publish broad claims about human psychology and behavior in the world's top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers - often implicitly - assume that either there is little variation across human populations, or that these "standard subjects" are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species - frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior - hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.
Development of the meanness in psychopathy-self report
- S D Benning
- K A Barchard
- R S Westfall
- V Brouwers
- S M Molina
Benning, S. D., Barchard, K. A., Westfall, R. S., Brouwers, V., & Molina, S. M. (2018).
Development of the meanness in psychopathy-self report. Manuscript in Preparation.
The anatomy of antagonism: Exploring the relations of 20 lexical factors of personality with Machiavellianism, grandiose narcissism, and psychopathy
- C S Kay
Kay, C. S. (2023). The anatomy of antagonism: Exploring the relations of 20 lexical factors of
personality with Machiavellianism, grandiose narcissism, and psychopathy. Unpublished
Manuscript.
Assessing self-deception and impression management in self-reports: The Balanced Inventory of Desirable Responding. Vancouver, Canada: University of British Columbia
- D L Paulhus
Paulhus, D. L. (1988). Assessing self-deception and impression management in self-reports:
The Balanced Inventory of Desirable Responding. Vancouver, Canada: University of
British Columbia. Unpublished Manual.
Improving construct validity: Cronbach, Meehl, and Neurath's ship
- Westen
Westen, D., & Rosenthal, R. (2005). Improving construct validity: Cronbach, Meehl, and
Neurath's ship. Psychological Assessment, 17(4), 409-412.