Chia-Yi Chiu’s research while affiliated with Columbia University and other places

Cognitive Diagnosis Models in educational measurement are restricted latent class models that describe ability in a knowledge domain as a composite of latent skills an examinee may have mastered or failed. Different combinations of skills define distinct latent proficiency classes to which examinees are assigned based on test performance. Items of cognitively diagnostic assessments are characterized by skill profiles specifying which skills are required for a correct item response. The item-skill profiles of a test form its Q-matrix. The validity of cognitive diagnosis depends crucially on the correct specification of the Q-matrix. Typically, Q-matrices are determined by curricular experts. However, expert judgment is fallible. Data-driven estimation methods have been developed with the promise of greater accuracy in identifying the Q-matrix of a test. Yet, many of the extant methods encounter computational feasibility issues either in the form of excessive amounts of CPU times or inadmissible estimates. In this article, a two-step algorithm for estimating the Q-matrix is proposed that can be used with any cognitive diagnosis model. Simulations showed that the new method outperformed extant estimation algorithms and was computationally more efficient. It was also applied to Tatsuoka’s famous fraction-subtraction data. The paper concludes with a discussion of theoretical and practical implications of the findings.

Computerized adaptive testing for cognitive diagnosis (CD‐CAT) achieves remarkable estimation efficiency and accuracy by adaptively selecting and then administering items tailored to each examinee. The process of item selection stands as a pivotal component of a CD‐CAT algorithm, with various methods having been developed for binary responses. However, multiple‐choice (MC) items, an important item type that allows for the extraction of richer diagnostic information from incorrect answers, have been underemphasized. Currently, the Jensen–Shannon divergence (JSD) index introduced by Yigit et al. ( Applied Psychological Measurement , 2019, 43, 388) is the only item selection method exclusively designed for MC items. However, the JSD index requires a large sample to calibrate item parameters, which may be infeasible when there is only a small or no calibration sample. To bridge this gap, the study first proposes a nonparametric item selection method for MC items (MC‐NPS) by implementing novel discrimination power that measures an item's ability to effectively distinguish among different attribute profiles. A Q‐optimal procedure for MC items is also developed to improve the classification during the initial phase of a CD‐CAT algorithm. The effectiveness and efficiency of the two proposed algorithms were confirmed by simulation studies.

The nonparametric classification (NPC) method has been proven to be a suitable procedure for cognitive diagnostic assessments at a classroom level. However, its nonparametric nature impedes the obtention of a model likelihood, hindering the exploration of crucial psychometric aspects, such as model fit or reliability. Reporting the reliability and validity of scores is imperative in any applied context. The present study proposes the restricted deterministic input, noisy “and” gate (R-DINA) model, a parametric cognitive diagnosis model based on the NPC method that provides the same attribute profile classifications as the nonparametric method while allowing to derive a model likelihood and, subsequently, to compute fit and reliability indices. The suitability of the new proposal is examined by means of an exhaustive simulation study and a real data illustration. The results show that the R-DINA model properly recovers the posterior probabilities of attribute mastery, thus becoming a suitable alternative for comprehensive small-scale diagnostic assessments.

The multiple-choice (MC) item format has been implemented in educational assessments that are used across diverse content domains. MC items comprise two components: the stem that provides the context with a motivating narrative, and the collection of response options consisting of the correct answer, called the “key,” and several incorrect alternatives, the “distractors.” The MC-DINA model was the first diagnostic classification model for MC items that used distractors explicitly as potential sources of diagnostic information. However, the MC-DINA model requires that the q-vectors of the distractors are nested within each other and that of the key, which poses a serious constraint on item development. Consequently, later adaptations of the MC item format to cognitive diagnosis dropped the nestedness condition. The relaxation of the nestedness-condition, however, comes at a price: distractors may become redundant (i.e., they do not contribute to any further diagnostic differentiation between examinees), and they may induce undesirable diagnostic ambiguity (i.e., they are equally likely to be chosen by an examinee, but their q-vectors point at different diagnostic classifications). In this article, two criteria, useful and proper, are proposed to identify redundant and diagnostically ambiguous distractors.KeywordsCognitive diagnosisNonparametric cognitive diagnosisPolytomous itemsMC-DINAMC-NPC

The multiple-choice (MC) item format has been widely used in educational assessments across diverse content domains. MC items purportedly allow for collecting richer diagnostic information. The effectiveness and economy of administering MC items may have further contributed to their popularity not just in educational assessment. The MC item format has also been adapted to the cognitive diagnosis (CD) framework. Early approaches simply dichotomized the responses and analyzed them with a CD model for binary responses. Obviously, this strategy cannot exploit the additional diagnostic information provided by MC items. De la Torre’s MC Deterministic Inputs, Noisy “And” Gate (MC-DINA) model was the first for the explicit analysis of items having MC response format. However, as a drawback, the attribute vectors of the distractors are restricted to be nested within the key and each other. The method presented in this article for the CD of DINA items having MC response format does not require such constraints. Another contribution of the proposed method concerns its implementation using a nonparametric classification algorithm, which predestines it for use especially in small-sample settings like classrooms, where CD is most needed for monitoring instruction and student learning. In contrast, default parametric CD estimation routines that rely on EM- or MCMC-based algorithms cannot guarantee stable and reliable estimates—despite their effectiveness and efficiency when samples are large—due to computational feasibility issues caused by insufficient sample sizes. Results of simulation studies and a real-world application are also reported.

The Polytomous Local Independence Model (PoLIM) by Stefanutti, de Chiusole, Anselmi, and Spoto, is an extension of the Basic Local Independence Model (BLIM) to accommodate polytomous items. BLIM, a model for analyzing responses to binary items, is based on Knowledge Space Theory, a framework developed by cognitive scientists and mathematical psychologists for modeling human knowledge acquisition and representation. The purpose of this commentary is to show that PoLIM is simply a paraphrase of a DINA model in cognitive diagnosis for polytomous items. Specifically, BLIM is shown to be equivalent to the DINA model when the BLIM-items are conceived as binary single-attribute items, each with a distinct attribute; thus, PoLIM is equivalent to the DINA for polytomous single-attribute items, each with a distinct attribute.

Computerized adaptive testing (CAT) is characterized by its high estimation efficiency and accuracy, in contrast to the traditional paper-and-pencil format. CAT specifically for cognitive diagnosis (CD-CAT) carries the same advantages and has been seen as a tool for advancing the use of cognitive diagnosis (CD) assessment for educational practice. A powerful item selection method is the key to the success of a CD-CAT program, and to date, various parametric item selection methods have been proposed and well-researched. However, these parametric methods all require large samples, to secure high-precision calibration of the items in the item bank. Thus, at present, implementation of parametric methods in small-scale educational settings, such as classroom, remains challenging. In response to this issue, Chang, Chiu, and Tsai (Appl Psychol Meas 43:543–561, 2019) proposed the nonparametric item selection (NPS) method that does not require parameter calibration and outperforms the parametric methods for settings with only small or no calibration samples. Nevertheless, the NPS method is not without limitations; extra assumptions are required to guarantee a consistent estimator of the attribute profiles when data conform to complex models. To remedy this shortcoming, the general nonparametric item selection (GNPS) method that incorporates the newly developed general NPC (GNPC) method (Chiu et al. in Psychometrika 83:355–375, 2018) as the classification vehicle is proposed in this study. The inclusion of the GNPC method in the GNPS method relaxes the assumptions imposed on the NPS method. As a result, the GNPS method can be used with any model or multiple models without abandoning the advantage of being a small-sample technique. The legitimacy of using the GNPS method in the CD-CAT system is supported by Theorem 1 proposed in the study. The efficiency and effectiveness of the GNPS method are confirmed by the simulation study that shows the outperformance of the GNPS method over the compared parametric methods when the calibration samples are small.

Diagnostic classification models in educational measurement describe ability in a knowledge domain as a composite of specific binary skills called “cognitive attributes,” each of which an examinee may or may not have mastered. Attribute Hierarchy Models (AHMs) account for the possibility that attributes are dependent by imposing a hierarchical structure such that mastery of one or more attributes is a prerequisite of mastering one or more other attributes. Thus, the number of meaningfully defined attribute combinations is reduced, so that constructing a complete Q-matrix may be challenging. (The Q-matrix of a cognitively diagnostic test documents which attributes are required for solving which item; the Q-matrix is said to be complete if it guarantees the identifiability of all realizable proficiency classes among examinees.) For structured Q-matrices (i.e., the item attribute profiles are restricted to reflect the hierarchy postulated to underlie the attributes), the conditions of completeness have been established. However, sometimes, a structured Q-matrix cannot be assembled because the items of the test in question have attribute profiles that do not conform to the prerequisite structure imposed by the postulated attribute hierarchy. A Q-matrix composed of such items is called “unstructured.” In this article, the completeness conditions of unstructured Q-matrices for the DINA model are presented. Specifically, there exists an entire range of Q-matrices that are all complete for DINA-AHMs. Thus, the theoretical results presented here can be combined with extant insights about Q-completeness for models without attribute hierarchies into a unified framework on the completeness of Q-matrices for the DINA model.