Francisco J Abad

• Autonomous University of Madrid

Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which aim to...

Describing and understanding personality structure is fundamental to predict and explain human behavior. Recent research calls for large personality item pools to be analyzed from the bottom-up, as item-level analysis may reveal meaningful differences often obscured by aggregation. This study introduces and applies Taxonomic Graph Analysis (TGA), a...

Describing and understanding personality structure is fundamental to predict and explainhuman behavior. Recent research calls for large personality item pools to be analyzed fromthe bottom-up, as item-level analysis may reveal meaningful differences often obscured byaggregation. This study introduces and applies Taxonomic Graph Analysis (TGA), a co...

Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which aim to...

La evaluación de variables no cognitivas, como los factores de personalidad, es clave en los procesos de selección de personal en las administraciones públicas, si bien su implementación presenta ciertos desafíos. Este artículo aborda la importancia de validar estos procesos integrando los principios de publicidad y transparencia con las buenas prá...

Forced-choice (FC) questionnaires have gained scientific interest over the last decades. However, the inclusion of unequally keyed item pairs in FC questionnaires remains a subject of debate, as there is evidence supporting both their usage and avoidance. Designing unequally keyed pairs may be more difficult when considering social desirability, as...

Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement quality. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which...

Psychometric assessment is the foundation of psychological research, where the accuracy of outcomes and their interpretations depend on measurement. Due to the widespread application of factor models, factor loadings are fundamental to modern psychometric assessment. Recent advances in network psychometrics introduced network loadings which aim to...

This study proposes a procedure for substantive dimensionality estimation in the presence of wording effects, the inconsistent response to regular and reversed self-report items. The procedure developed consists of subtracting an approximate estimate of the wording effects variance from the sample correlation matrix and then estimating the substant...

Describing and understanding personality structure is fundamental to predict and explainhuman behavior. Recent research calls for large personality item pools to be analyzed fromthe bottom-up, as item-level analysis may reveal meaningful differences often obscured byaggregation. This study introduces and applies Taxonomic Graph Analysis (TGA), a co...

The accuracy of factor retention methods for structures with one or more general factors, like the ones typically encountered in fields like intelligence, personality, and psychopathology, has often been overlooked in dimensionality research. To address this issue, we compared the performance of several factor retention methods in this context, inc...

The number of available factor analytic techniques has been increasing in the last decades. However, the lack of clear guidelines and exhaustive comparison studies between the techniques might hinder that these valuable methodological advances make their way to applied research. The present paper evaluates the performance of confirmatory factor ana...

Exploratory bi-factor analysis (EBFA) is a very popular approach to estimate models where specific factors are concomitant to a single, general dimension. However, the models typically encountered in fields like personality, intelligence, and psychopathology involve more than one general factor. To address this circumstance, we developed an algorit...

The nonparametric classification (NPC) method has been proven to be a suitable procedure for cognitive diagnostic assessments at a classroom level. However, its nonparametric nature impedes the obtention of a model likelihood, hindering the exploration of crucial psychometric aspects, such as model fit or reliability. Reporting the reliability and...

Background: The emergence of digital technology in the field of psychological and educational measurement and assessment broadens the traditional concept of pencil and paper tests. New assessment models built on the proliferation of smartphones, social networks and software developments are opening up new horizons in the field. Method: This stud...

Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretatio...

Response biases related to wording effects, the inconsistent responding to regular and reversed self-report items, are pervasive in the behavioral and health sciences. Although several factor modeling strategies have been proposed to mitigate their adverse effects, there is limited simulation research assessing their performance with exploratory st...

This study proposes a procedure for substantive dimensionality estimation in the presence of wording effects, the inconsistent response to regular and reversed self-report items. Although extensive empirical research had shown that wording effects negatively impact latent dimensionality estimates, there was scarce systematic research assessing the...

The accuracy of factor retention methods for structures with one or more general factors, typically encountered in fields like intelligence, personality, and psychopathology, has often been overlooked in dimensionality research. To address this issue, we compared the performance of several factor retention methods in this context, including a new,...

Multidimensional forced-choice (FC) questionnaires have been consistently found to reduce the effects of socially desirable responding and faking in non-cognitive assessments. Although FC has been considered problematic for providing ipsative scores under the classical test theory, IRT models enable the estimation of non-ipsative scores from FC res...

Previous research on health and life satisfaction in older adults has suggested a bidirectional relationship. However, most evidence either is based on cross-sectional data, being unsuitable for inferring any directionality on the results, or disregards the within-person stability of both variables over time, thus providing potentially biased resul...

The new methodological and technological developments of the last decade make it possible to resolve or, at least, attenuate the psychometric problems of forced-choice (FC) tests for the measurement of personality. In these tests, the person being tested is shown blocks of two or more sentences of similar social desirability, from which he or she m...

Cognitive diagnosis models (CDMs) are confirmatory latent class models that provide fine-grained information about skills and cognitive processes. These models have gained attention in the last few years because of their usefulness in educational and psychological settings. Recently, numerous developments have been made to allow for the implementat...

The validity of cross-cultural comparisons of test scores requires that scores have the same meaning across cultures, which is usually tested by checking the invariance of the measurement model across groups. In the last decade, a large number of studies were conducted to verify the equivalence across cultures of the dimensional Alternative Model o...

The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for...

The item wording (or keying) effect consists of logically inconsistent answers to positively and negatively worded items that tap into similar (but polarly opposite) content. Previous research has shown that this effect can be successfully modeled through the random intercept item factor analysis (RIIFA) model, as evidenced by the improvements in t...

The item wording (or keying) effect consists of logically inconsistent answers to positively and negatively worded items that tap into similar (but polarly opposite) content. Previous research has shown that this effect can be successfully modeled through the random intercept item factor analysis (RIIFA) model, as evidenced by the improvements in m...

Cognitive diagnosis models (CDMs) allow classifying respondents into a set of discrete attribute profiles. The internal structure of the test is determined in a Q-matrix, whose correct specification is necessary to achieve an accurate attribute profile classification. Several empirical Q-matrix estimation and validation methods have been proposed w...

Decisions on how to calibrate an item bank might have major implications in the subsequent performance of the adaptive algorithms. One of these decisions is model selection, which can become problematic in the context of cognitive diagnosis computerized adaptive testing, given the wide range of models available. This article aims to determine wheth...

The Q‐matrix identifies the subset of attributes measured by each item in the cognitive diagnosis modelling framework. Usually constructed by domain experts, the Q‐matrix might contain some misspecifications, disrupting classification accuracy. Empirical Q‐matrix validation methods such as the general discrimination index (GDI) and Wald have shown...

Background: Unproctored Internet Tests (UIT) are vulnerable to cheating attempts by candidates to obtain higher scores. To prevent this, subsequent procedures such as a verification test (VT) is carried out. This study compares five statistics used to detect cheating in Computerized Adaptive Tests (CATs): Guo and Drasgow's Z-test, the Adaptive Mea...

Background: The inclusion of direct and reversed items in scales is a commonly-used strategy to control acquiescence bias. However, this is not enough to avoid the distortions produced by this response style in the structure of covariances and means of the scale in question. This simulation study provides evidence on the performance of two differe...

Background: Due to its flexibility and statistical properties, bi-factor Exploratory Structural Equation Modeling (bi-factor ESEM) has become an often-recommended tool in psychometrics. Unfortunately, most recent methods for approximating these structures, such as the SLiD algorithm, are not available in the leading software for performing ESEM (i...

As general factor modeling continues to grow in popularity, researchers have become interested in assessing how reliable general factor scores are. Even though omega hierarchical estimation has been suggested as a useful tool in this context, little is known about how to approximate it using modern bi-factor exploratory factor analysis methods. Thi...

In the context of cognitive diagnosis models, a Q-matrix reflects the correspondence between attributes and items. The Q-matrix construction process is typically subjective in nature, which may lead to misspecifications. All this can negatively affect the attribute classification accuracy. In response, several methods of empirical Q-matrix validati...

Currently, there are two predominant approaches in adaptive testing. One, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT), is based on cognitive diagnosis models, and the other, the traditional CAT, is based on item response theory. The present study evaluates the performance of two item selection rules (ISRs) originally d...

One important problem in the measurement of non-cognitive characteristics such as personality traits and attitudes is that it has traditionally been made through Likert scales, which are susceptible to response biases such as social desirability (SDR) and acquiescent (ACQ) responding. Given the variability of these response styles in the population...

Multidimensional forced-choice questionnaires are widely regarded in the personnel selection literature for their ability to control response biases. Recently developed IRT models usually rely on the assumption that item parameters remain invariant when they are paired in forced-choice blocks, without giving it much consideration. This study aims t...

There has been increased interest in assessing the quality and usefulness of short versions of the Raven’s Progressive Matrices. A recent proposal, composed of the last twelve matrices of the Standard Progressive Matrices (SPM-LS), has been depicted as a valid measure of g. Nonetheless, the results provided in the initial validation questioned the...

Here we test the developmental theory of sex differences in intelligence using latent modeling. The theory has been recently updated (Lynn, 2017). Based on men's greater average brain volumes and premature maturation in women, the theory predicts an intelligence advantage for women (or null sex differences) in early adolescence, but an advantage fo...

Cognitive diagnosis models (CDMs) are latent class multidimensional statistical models that help classify people accurately by using a set of discrete latent variables, commonly referred to as attributes. These models require a Q-matrix that indicates the attributes involved in each item. A potential problem is that the Q-matrix construction proces...

Bi-factor exploratory modeling has recently emerged as a promising approach to multidimensional psychological measurement. However, state-of-the-art methods relying on target rotation require researchers to select an arbitrary cut-off for defining the target matrix. Unfortunately, the consequences of such choice on factor recovery remain uninvestig...

This paper presents a new two-dimensional Multiple-Choice Model accounting for Omissions (MCMO). Based on Thissen and Steinberg multiple-choice models, the MCMO defines omitted responses as the result of the respondent not knowing the correct answer and deciding to omit rather than to guess given a latent propensity to omit. Firstly, using a Monte...

This study analyses the extent to which cheating occurs in a real selection setting. A two-stage, unproctored and proctored, test administration was considered. Test score inconsistencies were concluded by applying a verification test (Guo and Drasgow Z-test). An initial simulation study showed that the Z-test has adequate Type I error and power ra...

Contemporary models of personality assume a hierarchical structure in which broader traits contain narrower traits. Individual differences in response styles also constitute a source of score variance. In this study, the bifactor model is applied to separate these sources of variance for personality subscores. The procedure is illustrated using dat...

Different models might explain one the most replicated findings in Psychology, namely, the positive correlation among diverse tests of cognitive ability (positive manifold). The g factor and the mutualism models are examples. Mutualism suggests that the manifold emerges progressively during development, as a consequence of interactions among origin...

Multidimensional computerized adaptive testing based on the bifactor model (MCAT-B) can provide efficient assessments of multifaceted constructs. In this study, MCAT-B was compared with a short fixed-length scale and computerized adaptive testing based on unidimensional (UCAT) and multidimensional (correlated-factors) models (MCAT) to measure the B...

Bi-factor exploratory modelling has recently emerged as a promising approach to multidimensional psychological measurement. However, state-of-the-art methods relying on target rotation require researchers to select an arbitrary cut-off for defining the target matrix. Unfortunately, the consequences of such choice on factor recovery remain uninvesti...

Purpose: Developing valid and reliable instruments that can be used across countries is necessary. The present study aimed to test the comparability of quality of life scores across three European countries (Finland, Poland, and Spain). Method: Data from 9987 participants interviewed between 2011 and 2012 were employed, using nationally represen...

Background: Even though the Five Factor Model (FFM) has been the dominant paradigm in personality research for the past two decades, very few studies have measured the FFM adaptively. Thus, the purpose of this research was the building of a new item pool to develop a computerized adaptive test (CAT) for personality assessment. Method: A pool of...

Background: The development of an effective instrument to assess the risk of partner violence is a topic of great social relevance. This study evaluates the scale of “Predicción del Riesgo de Violencia Grave Contra la Pareja” –Revisada– (EPV-R - Severe Intimate Partner Violence Risk Prediction Scale-Revised), a tool developed in Spain, which is fa...

Research related to the fit evaluation at the item level involving cognitive diagnosis models (CDMs) has been scarce. According to the parsimony principle, balancing goodness of fit against model complexity is necessary. General CDMs require a larger sample size to be estimated reliably, and can lead to worse attribute classification accuracy than...

The current study proposes a new bi-factor rotation method, Schmid-Leiman with iterative target rotation (SLi), based on the iteration of partially specified target matrices and an initial target constructed from a Schmid-Leiman (SL) orthogonalization. SLi was expected to ameliorate some of the limitations of the previously presented SL bi-factor r...

There has been an increase of interest in psychometric models referred to as cognitive diagnosis models. A critical concern is selecting the most appropriate model. Several tests for model comparison have been employed, which include the likelihood ratio (LR) and the Wald (W) tests. Although the LR test is relatively more robust than the W test, th...

The bifactor model (BM) and the testlet response model (TRM) are the most common multidimensional models applied to testletbased tests. The common procedure is to estimate these models using different estimation methods (see, e.g., DeMars, 2006). A possible consequence of this is that previous findings about the implications of fitting a wrong mode...

Neuroimaging research involves analyses of huge amounts of biological data that might or might not be related with cognition. This relationship is usually approached using univariate methods, and, therefore, correction methods are mandatory for reducing false positives. Nevertheless, the probability of false negatives is also increased. Multivariat...

Forced-choice questionnaires have been proposed as a way to control some response biases associated with traditional questionnaire formats (e.g., Likert-type scales). Whereas classical scoring methods have issues of ipsativity, item response theory (IRT) methods have been claimed to accurately account for the latent trait structure of these instrum...

Conventional methods for assessing the validity and reliability of situational judgment test (SJT) scores have proven to be inadequate. For example, factor analysis techniques typically lead to nonsensical solutions, and assumptions underlying Cronbach’s alpha coefficient are violated due to the multidimensional nature of SJTs. In the current artic...

Background: Forced-choice tests (FCTs) were proposed to minimize response biases associated with Likert format items. It remains unclear whether scores based on traditional methods for scoring FCTs are appropriate for between-subjects comparisons. Recently, Hontangas et al. (2015) explored the extent to which traditional scoring of FCTs relates to...

An early step in the process of construct validation consists of establishing the fit of an unrestricted "exploratory" factorial model for a prespecified number of common factors. For this initial unrestricted model, researchers have often recommended and used fit indices to estimate the number of factors to retain. Despite the logical appeal of th...

Despite the globalization of HRM, there is a dearth of research on the potential use of contextualized selection instruments such as situational judgment tests (SJTs) in other countries than those where the selection instruments were originally developed. Therefore, two studies are conducted to examine the transportability of an integrity SJT that...

IQ summary scores may not involve equivalent psychological meaning for different educational levels. Ultimately, this relates to the distinction between constructs and measurements. Here, we explore this issue studying the standardization of the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) for Spain. A representative sample of 743 ind...

Given the growing interest in the study of subjective well-being as a measure of social progress, instruments that produce valid and reliable scores and that can be used within and across countries are needed. The aim of the present study was to analyze the measurement equivalence of the Day Reconstruction Method in its brief version, using nationa...

This article explores how traditional scores obtained from different forced-choice (FC) formats relate to their true scores and item response theory (IRT) estimates. Three FC formats are considered from a block of items, and respondents are asked to (a) pick the item that describes them most (PICK), (b) choose the two items that describe them the m...

As a general rule, the repeated administration of tests measuring a given cognitive ability in the same participants reveals increased scores. This brings to life the well-known practice effect and it must be taken into account in research aimed at the proper assessment of changes after the completion of cognitive training programs. Here we focus i...

Intelligence is composed of a set of cognitive abilities hierarchically organized. General and specific abilities capture distinguishable, but related, facets of the intelligence construct. Here, we analyze gray matter with three morphometric indices (volume, cortical surface area, and cortical thickness) at three levels of the intelligence hierarc...

Background: The Exploratory Factor Analysis (EFA) procedure is one of the most commonly used in social and behavioral sciences. However, it is also one of the most criticized due to the poor management researchers usually display. The main goal is to examine the relationship between practices usually considered more appropriate and actual decision...

Test security can be a major problem in computerized adaptive testing, as examinees can share information about the items they receive. Of the different item selection rules proposed to alleviate this risk, stratified methods are among those that have received most attention. In these methods, only low discriminative items can be presented at the b...

Previous research has suggested multiple factor structures for the 12-item General Health Questionnaire (GHQ-12), with contradictory evidence arising across different studies on the validity of these models. In the present research, it was hypothesized that these inconsistent findings were due to the interaction of 3 main methodological factors: am...

In computerized adaptive testing pretest items are presented in conjunction with operational items to renew the item bank. Pretest items are calibrated, and possible differential item functioning (DIF) is analyzed. Some difficulties arise due to the large amount of missing responses, which can be avoided by the use of fixed item parameter calibrati...

Background: Criterion-referenced interpretations of tests are highly necessary, which usually involves the difficult task of establishing cut scores. Contrasting with other Item Response Theory (IRT)-based standard setting methods, a non-judgmental approach is proposed in this study, in which Item Characteristic Curve (ICC) transformations lead to...

The analysis of sex differences in cognitive abilities is largely confusing because these differences are masked by the pervasive influence of the general factor of intelligence (g). In this study a battery of five reasoning tests (abstract [AR], numerical [NR], verbal [VR], mechanical [MR], and spatial [SR]) was completed by a sample of 3233 young...

Previous research evaluating the performance of Horn's parallel analysis (PA) factor retention method with ordinal variables has produced unexpected findings. Specifically, PA with Pearson correlations has performed as well as or better than PA with the more theoretically appropriate polychoric correlations. Seeking to clarify these findings, the c...

This paper describes several simulation studies that examine the effects of capitalization on chance in the selection of items and the ability estimation in CAT, employing the 3-parameter logistic model. In order to generate different estimation errors for the item parameters, the calibration sample size was manipulated (N = 500, 1000 and 2000 subj...

The current research compares the effects of several strategies to establish the anchor subtest when detecting for differential item functioning (DIF) using the IRT likelihood ratio test in one- and two-stage procedures. Two one-stage strategies were examined: (1) “One item” and (2) “All other items” used as anchor. Additionally, two two-stage stra...

In this study, eCAT-Listening, a new computerized adaptive test for the evaluation of English Listening, is described. Item bank development, anchor design for data collection, and the study of the psychometric properties of the item bank and the adaptive test are described. The calibration sample comprised 1.576 participants. Good psychometric gua...

Despite strong evidence supporting the use of Velicer’s minimum average partial (MAP) method to establish the dimensionality of continuous variables, little is known about its performance with categorical data. Seeking to fill this void, the current study takes an in-depth look at the performance of the MAP procedure in the presence of ordinal-leve...

In computerized adaptive testing, the most commonly used valuating function is the Fisher information function. When the goal is to keep item bank security at a maximum, the valuating function that seems most convenient is the matching criterion, valuating the distance between the estimated trait level and the point where the maximum of the informa...

The distance between nonparametric and parametric item characteristic curves has been proposed as an index of goodness of fit in item response theory in the form of a root integrated squared error index. This article proposes to use the posterior distribution of the latent trait as the nonparametric model and compares the performance of an index ba...

RESUMEN Se describe la construcción del test eCat y se informa de las consecuencias que su funcionamiento ha tenido en sus propiedades psicométricas. Los resultados obtenidos en una muestra de 3224 estudiantes mostraron que el banco tiene una adecuada fiabilidad y validez convergente en relación a los autoinformes de dominio del inglés. Las simulac...

In a typical study of the relative efficiency of two competing item selection rules in computerized adaptive testing, the common result is that they simultaneously differ in accuracy and security, making,it difficult to reach a conclusion on which is the most appropriate. This study proposes a strategy to conduct a global comparison of two or more...

The present research evaluates the dimension of Neuroticism in school children using a multi-source design that includes self-report, parents' and teachers' reports, semi-structured interview and behavior observation. Specifically, the study intended to verify the agreement level in evaluation of Neuroticism from different sources of information. T...

The aim of this study was to investigate the relationship between intelligence, personality and the extent of general and current information of students in the state of Minas Gerais. Two sample groups participated. The first was composed of students from three socially vulnerable levels from the city of Belo Horizonte (n=600), and the second group...

The Human Figure Drawing Test (HFD) constitutes an instrument often used in Brazilian psychological assessmentpractices. Few studies of its validity as a cognitive assessment instrument can be found. The present paperanalyses the performance of several samples of children from Belo Horizonte on the HFD (Wechsler scoringsystem) and on others tests s...

This study describes the parameter drift analysis conducted on eCAT (a Computerized Adaptive Test to assess the written English level of Spanish speakers). The original calibration of the item bank (N = 3224) was compared to a new calibration obtained from the data provided by most eCAT operative administrations (N = 7254). A Differential Item Func...

Recientemente se ha producido un considerable desarrollo de los tests adaptativos informatizados, en los que el test se adapta progresivamente al rendimiento del evaluando, y de otros tipos de tests: a) los test basados en modelos (se dispone de un modelo o teoría de cómo se responde a cada ítem, lo que permite predecir su dificultad), b) los tests...