Item Analysis - Science topic

Missingness is always interesting, as well as being bothersome. When there are large numbers of missing responses, my immediate response is to look for missingness patterns. These can range from simple (stopped answering part-way through) to revealing and important.

As an example of the latter, we used a stigma questionnaire that had a whole clump of missing data. The missing data were concentrated in a small number of items which all related to work-related stigma. It was easy to cross-check : they were coming from people who were not in work. We omitted those items as not relevant to the whole study population.

Other missing items may just be vague or badly-worded so that even the data you have may be suspect – the people who did answer them were not really sure what they were doing. I recommend an item analysis using Mokken scaling to ensure that the items you are using form a definable scale, but even looking at item-rest correlations can help to spot these items. Actually – now I think of it – critically reading the items is the first thing you should do!

Questionnaire adaption to suit research context is common.

Hello Partha,

No matter what kind of measure you are building, you no doubt have in mind some intended use(s) of the scores from that measure.

Gathering data and demonstrating that the scores have technical adequacy for your intended uses (both score reliability and score validity) is always a good idea. This may well involve at least some aspects of item analysis.

If your items/questions/stimuli are nominal, as in the example you give, you likely won't be computing things such as item-total correlations in the traditional sense. But, for determining temporal stability, you would like to see that the answers given by respondents on one occasion agree to a very high extent with those they might give, say, three weeks later. That could be quantified as a correlation (e.g., Cramer's V) or as a percent agreement, or via Cohen's kappa, computed for each item.

Good luck with your work.

By categorical, do you mean the responses are not a scale?

Maria Dolores Tolsá Thank you very much for your kind answer. I will love to collaborate for my ongoing project in the research publication stage.

If the "ipsative" scales are all dichotomous (such as introverted-extraverted) you should not have any major problems. The issue arises when all the scales are dependent on one another. My advice is to treat each dichotomy as a single scale rather than as two. Then follow the usual procedures: nothing extra-fancy. For each item you might calculate:

(1) The percentage of respondents answering in each direction

(2) the correlation of the score on the item (say, +1 for the extraverted option) with total score on the scale.

Ideally, (2) would be done after removing the item in question. Doing this one by one is very tedious, though, so some folks don't bother. If you are working in SPSS or a similar program, however, these statistics are readily available when running a reliability analysis (such as Cronbach's alpha).

Again: the main point is that you don't really have two separate scales for extraversion and introversion, even if you are going to use the results to classify people into one crude category or another. You have one overall measure of how extraverted (vs. introverted) they are.

Lertap5 is an item analysis program packaged as an Excel "app". The free version allows for an unlimited number of items, and up to 250 respondents. The "pro" version is perpetual and costs USD78. For both Windows and Mac. See:

You may want to check the following paper out:

Using XBRL to conduct a large-scale study of discrepancies between the accounting numbers in Compustat and SEC 10-K filings‏

R Chychyla, A Kogan - Journal of Information Systems, 2015‏ - meridian.allenpress.com‏

I agree that items 5 and 6 are reverse coded, but #4 essentially refers to "not worrying," so I would expect it to correlate positively with the first 3 items.

Dear Lili,

I can only second what David and Georg already said. Other than that I would not recommend using a median split for the Andersen LRT. Simulation studies show that the Andersen LRT performs rather poorly with a random split (https://dx.doi.org/10.22237/jmasm/1555594442).

All the best for your research,

Georg

Maria Acevedo Velazquez, just quickly in case it helps, I think that obtaining item-total correlations (through the reliability section of SPSS) is not going to give you the kind of answers that you want.

I think you also need to be clear about what is mean by discrimination. In some contexts, it means that scores on one construct are different from scores on another construct so that researchers are assured they're not just measuring the same construct in two different ways. In your context (from what I can pick up from your question), you want to be able to discriminate participants one from the other. That's a different kettle of fish.

I hope my pointing out those different things (both of which are dealt with in the Boateng et al. article that you mention) helps you set your sights a bit more clearly.

Hi Daniel, oh my....alone hearing terms such as "test banks" gives me goosebumps. In my very personal view, the whole personality psych suffers til today from its historical approach to "measurement"--that is creating masses of items and then aggregating these items with the help of principal component analysis. Still, 10 years ago I had weekly battles with colleagues to convince them about the fundamental differences between PCA and the factor model. While this may be understood by most of the folks, most "item banks" are still a result --either--of the procedure (PCA) or even a more fundamental positivist/operationalist perspective of constructs being constituted by measures.

This reminds of a sentence, Denny Borsboom wrote in this "attacks of the psychometricians" paper (2006) , about the failure of CFA to support the model structure proposed for the big five:

"Now it turns out that, with respect to the Big Five, CFA gives Big Problems. For instance, McCrae, Zonderman, Costa, Bond, & Paunonen (1996) found that a five factor model is not supported by the data, even though the tests involved in the analysis were specifically designed on the basis of the PCA solution. What does one conclude from this? Well, obviously, because the Big Five exist, but CFA cannot find them, CFA is wrong. “In actual analyses of personality data [. . .] structures that are known to be reliable [from principal components analyses] showed poor fits when evaluated by CFA techniques. We believe this points to serious problems with CFA itself when used to examine personality structure” (McCrae et al., 1996, p. 563)."

You will find more deeper ontological problems in related personality constructs, for instance this nonsensical core self-evaluations construct, which proponents sometimes describe as an aggregate of primary traits (self-esteem, self-efficacy, neuroticism, locus of control) and at other times as second-order factor. You can not have it both ways, but the ontological flexibility is breath taking--in the same way as the non-falsifiability of their models (by completely ignoring model test results). As aforementioned this goes along with very crude perceptions what a factor means, for instance, this undying convictions of a factor having "facets", "parts", or "breadth" which is utter nonsense (as a factor is ONE dimension). Look alone that the labels of factors you will see sometimes! How can ONE dimensions (=one varying attribute) be labeled in a pluralistic way (e.g., core self evaluationS)?

Okay, I stop my rant here. Here's more ;))

Best

--Holger

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425-440. doi:10.1007/s11336-006-1447-6

No problem at all. In fact the more creative you get with explanations the better (for your advisors and you- dont despair)

Thank you very much Yusuf F Zakariya !

Thanks, Cristian Ramos-Vera, and David Morse. I appreciate your suggestions and guidance. Is 't' useful for any item inclusion or exclusion making decision?

Hello Aleksa,

Yes, it's possible; yes, it's done quite often. Since you're building a scale, you might wish to consider evaluating the technical properties of the items and scale. IRT modeling is a useful approach here.

Good luck with your work.

I'm not sure it needs to be, but I'd recommend Rembretsin and Reise (2000) IRT text.

Matt

Good discussion and analysis.

attached file may help....

Sounds like you've created an instrument which is sound as it is based on theory - but this version should only be the pilot version, rather than the final instrument. As you see - there are too many items, which will affect the quality of the data collected. Have you conducted any sort of pilot study - factor analysis - and tested to see how those items relate? You'll probably find some items are redundant, and possibly even some scales ... use EFA to explore how the items load - then as you delete items - the number of items in each scale will decrease - you can then delete scales on the basis of insufficient items.

I find David De Vaus' work on survey design and validation very useful:

https://www.amazon.co.uk/Surveys-Social-Research-Today/dp/0415530180/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=ER3KY0K00XZC84NB7GHW

Please also read following essay:

Hi...I think you used wrong IRT model....your data contiuous....

I would say to make sure there is a definite explination with a varied analysis and just offer it both ways and see which one is most accepted in yor region or in that research. As long as it can be duplicated again and the hypoyhesis is shown to be supported or refuted by your researh then you should be fine.

Develop feasible indicators using IRT is subject to a series of steps that need to be performed. The IRT assumptions are essential in this regard, as the uni- dimensionality, local independence as well as speediness should be examined. The issue of fitting of your data to Rasch Rating Scale Model (may be it is the appropriate model in the Likert Scale) need to be investigated in a series of steps to identify the outfit persons as well as the outfit items. The criteria used is that : the outfit and In-fit index(MNSQ) should be between (0.7- 1.3) whereas, (ZSTD) should be between(-2, 2). The process should be replicated until you reach a set of items that are persons-free. In addition to that , as Rasch Scale model with one parameter may be used; so as, in such situations you may need to examine that the discriminations index is equal for all items, and that the guessing index is equal to zero for all items. Furthermore, person separation index and item separation index should be used to estimate scale reliability.

Hi Salim

Yes, you can get SPSS to add up the scores from the items that constitute a subscale. Alternatively you could ask SPSS to produce factor scores for each factor, these are probably better than summed scores as the factor scores will be weighted whereby the 'best' items contribute more to the factor scores.

However it may be that your research question could be better approached through confirmatory factor analysis. It appears that you know which items are the best indicators for each factor, so why not set up a CFA model and this will allow you to estimate the correlations among the factors/latent variables. You can then see which are correlated or uncorrelated. EFA rotation is a blunt tool, where the factors have to be all correlated or all uncorrelated. It may be that some of your factors are correlated and some are uncorrelated - CFA will tell you this.

Mark

You can also use Psychometrıc Tool Box which is very similar to the Excel.http://psico.fcep.urv.cat/utilitats/PsychometricTools/index.html

You also check free version of the CITAS at http://www.assess.com/citas/

This problem is something i try to solve in my study about developing spiritual serenity scale... In first study, i found the desirability of the items.. Rasch model analyze found that the items below of respondents ability...

In the next study, i will try change the descriptors of items to be more difficult... For example, never, seldom, sometimes, often, very often, and always..

Options "often", "very often" , and always will be chosed by persons only with high ability...

It sounds like you are using Allport's scale to measure Spranger's types of men. While they are somewhat useful, researchers usually rely more often on Schwartz's value model. The measures developed by Schwartz (1992 and Schwartz et al., 2001) are more reliable and better validated than Allport's scale. The "Schwartz Value Survey" is a relable measure of Schwartz's 10 value types, as is the Portrait Value Questionnaire (PVQ) or the PVQ-RR, which measures 19 value types. Gouveia's functional theory would also be an option.

So the best way to improve your survey is to use another value model. If you want to use Spranger's types of men, however, then you would keep each of the six types of men separately. Combining them would mean that you would loose a lot of information and the interpretation of the data would be difficult.

Hello Jari,

The point-biserial formula was derived as a short cut to the usual definitional or computational version of the Pearson product-moment correlation formula in the days before statistical computing software or cheap/free calculators. Both the short-cut and PPMC formulae yield the same numeric result. In the case of polytomous items, so long as one is willing to assume interval scale strength to the polytomous scores, the usual PPMC formula will also apply. If interval scale isn't a realistic claim for the data set, then a rank-order correlation would need to be used.

That doesn't address your question directly, but I strongly suspect that the corresponding "short-cut" polytomous generalization version would be more trouble to code than the basic PPMC formula. However, as an intellectual exercise, it may be of interest to some folks.

Good luck with your work!

Classical test theory can indeed answer some of your research questions, and there are a number of software programs that can calculate what you need since CTT is so incredibly simple. It wouldn't even take long to analyze it in Excel. However, you might want software like Iteman that specializes in it. CFA on the classical results might be useful.

Nevertheless, as Roberto recommended, I also highly recommend IRT. It is not unreasonably complex, and there are a number of models that are specifically designed for Likert type data. The two important ones you should examine are the Rasch Rating Scale model and the Graded Response model. You can download a free version of Xcalibre to try out on a small sample (http://www.assess.com/xcalibre/). Winsteps is also worth looking at, as it is the gold standard for Rasch analysis.

Hi

Item reliability index is equal to the product of the discrimination and standard deviation of the item. So item realibility index=r_jx.S_x

This index can be calculated for dichotomous or polytomous item type.

Allen & Yen (1979) explained this index in their book. Take a look for more information.

Allen, M. J. ve Yen, W. M. (1979). Introduction to measurement theory. Monterey: Brooks/Cole.

Good luck.

Hi Ale,

there are recent developments in IRT that allows extracting normative scores from forced-choice questionnaires. The Thurstonian IRT (TIRT) model by Brown and Maydeu-Olivares and the MUPP model by Stark and colleagues are good examples.

From my own experience, the TIRT model works best in practice (i.e., in terms of reliability and validity).

The main condition for calculating the Pearson linear correlation coefficient is that the variables involved are measured on the interval / ratio scale, along with the existence a form of distribution that does not deviate severely from the normal curve.

If the conditions for using the Pearson test are not met, the following non-parametric tests may be used: Chi-square test (for nominal data) or Spearman or Kendall (for ordinal data) correlation coefficients.

The SPSS procedure for calculating Pearson (r) linear correlation coefficient is: Statistics-Corelate-Bivariate.

Have a nice day,

It is perhaps go to bear in indm that the psychometric criteria (i) agreement, (ii) reliability and (iii) are hierarchically ordered. If there is no agreement among judges, a test cannot be reliable(judgment) , if a test is not reliable (i.e. has reproducinle results)), it cannot be valid. So, validity is the (does the test truly measure what it is supposed to easure) mtest in the final analysis, ?only criterion that matters. mvalidity

Thanks again. One remark: I do not use Cronbach's alpha for estimating the cluster's reliability but for increasing its homogeneity within proportions. If I would merely use the mean inter-item correlation (cohesion) of the cluster for that purpose, then the cluster would become very small and would only consist of very similar items. Alpha is a function of both cohesion and the size of the cluster. A larger size compensates for a smaller cohesion. So using alpha guards against reducing the cluster too much.

Why don't you try my software, X-Pro Milestone.

Ertuğrul, thank you!

Elaborating on Ivanouw's answer, for dichotomous models like the 3PL, the standard error of measurement (SEM) is equal to 1/sqrt(TIF(theta)). So, if your TIF has a height of 10, then SEM = 1/sqrt(10) = 0.316. So if that were at the point theta=1, that means that (assuming normally distributed errors) the 95% confidence interval for the true theta with an observed theta = 1 is [0.38,1.62].

So, that's your rule-of-thumb... You pick how narrow you want the 95% CI and work back to the height of the TIF required. I think SEM << 0.5  would be ideal.

However, that said, for most tests most of the time, we don't have the luxury of a rule of thumb, because the best items have some peak IIF and we're limited in how many items we can put on a form.  In other words, there are usually strong limits on how good we can make an exam. The huge advantage of using IRT and examining the TIF is that we avoid making really bad errors, like having holes in our scales where SEM is practically infinite.

In other words, the reason you haven't come across a rule of thumb is because that's not important. But spreading information across the range of scores that we care about is extremely important.

This is also the point where people start wishing for adaptive tests... If I can make the best 15-item test out of a pool of 100 good items, I'll be giving each person a much better exam than if I have to pick 15 items that everyone sees.

I would like to know, please, the SPSS commands  to make Item Response Theory combined with multilevel models. Can someone help me? 

Hello Rodrigo,It seems to be used with some frequency:example:

Validation of the French version of the alcohol, smoking and ...

University of Geneva

by RA Khan - ‎2011 - ‎Cited by 18 - ‎Related articles

Home Titles list Validation of the French version of the alcohol, smoking and substance ... Screening Test (ASSIST) was developed to detect substance use disorders. ... significant correlations between ASSIST scores and scores from ASI, AUDIT and RTQ, ... Substance-Related Disorders/diagnosis/epidemiology/psychology.

Also see:

PDF]Psychological Testing on the Internet - American Psychological ...

https://www.apa.org/.../testing/testing-on-internet....

American Psychological Association

by JA Naglieri - ‎Cited by 209 - ‎Related articles

there is a corresponding need for the ethical and professional use of test results. We ... changed, or translated without appropriate permission or validation. ...... prior computer use, who has reported difficulty in using the ASI-MV, including some.

The Use of Psychological Testing for Treatment Planning and Outcomes ...

Mark E. Maruish - 2004 - ‎Psychology

... had little prior experience (e.g., the costs and worth of psychological testing). ... validation of a computer-administered addiction severity index: The ASIBMV ...

ASI-6 - SciELO

by F Kessler - ‎2012 - ‎Cited by 31 - ‎Related articles

OBJECTIVE: To test the psychometric properties of ASI in its sixth version (ASI-6). ... Transcultural adaptation and validation of an instrument demands careful methodological .... They are psychometrically derived using nonparametric item response theory ... The 25 interviewers were either psychologists or psychiatrists.

Regards

Mehraneh,

Define difficulty as the mean observed score for an item (in your example, between 1-4).  Higher mean = easier task.

Define discrimination as Pearson correlation between item score and total score.  If you want to be a bit more precise, subtract item score from the total score first (so-called corrected item-total correlation).  Classical discrimination indices are all some variation on how well item scores relate to some larger criterion (e.g., total score).

You can, of course, use Qasim's approach, but that requires an extra step of first computing total scores for all respondents, then disaggregating cases into 'high' and 'low' scoring groups.

The questionnaire will have been "standardized" (this word can mean different things in different fields) for a particular culture or cultures. Thus, if you are working in a different culture than it was "standardized" it may not have the same psychometric properties. In short, often you will want to adapt it. Sometimes wording changes are necessary to have it make sense (e.g., from American English to British English) and for some populations some items do not make sense. This is one of the issues that makes cross-cultural research difficult.

There are copyright issues with some questionnaires. Also, if you make changes, stress this so that people do not falsely assume that you have not. This includes how you code responses.

I think you should formulate two groups one with high scores on the scale (top 25% of the group) and other with low scores(i.e. bottom 25% of the group) then apply t-test between these two groups  for each item . The items with significant t values will be accepted while those with insignificant t-values will be rejected

In addition, such high correlation (>.85) is classified as multicolinearity in Regression analysis and the researcher should drop out one of those highly correlated variables.

Hallo Irshad,

You may find my paper useful: https://www.researchgate.net/publication/226320418_Constructing_an_Item_Bank_Using_Item_Response_Theory_The_AMC_Linear_Disability_Score_Project 

In this paper, we describe setting up the foundations for an item bank that could be used as a basis for CAT. It focuses on the 2PL model, but the approach could be used for the 1PL model too.

Yours,

If you have one fewer explanatory variables than data points, the model fit will be perfect by definition. Imagine you're trying to model Y as a linear function of X, but have only two points.  Any line you draw will fit perfectly.

More generally, model overfitting is when your function fits the training data a lot better than it fits new data.  It tends to happen when you have too many explanatory variables and/or too small a sample.

Hi

I agree with Joan, and the following reference is very helpfull

Nunnally J.C. & Bernstein I.H. (1994) Psychometric Theory.

McGraw-Hill, New York.

Good luck

Hi Nathan, thank you for your answer. My aim is to evaluate which item format is the best/most informative as this is a crucial decision in the process of psychometric measurement development. In detail, we want to prove evidence, that so-called ability-items ("Are you able to wash your car?" - 5 ordinal response options) are more informative than performance-items ("Did you wash your car last week"  - yes/No) for measuring physical impaiment. I`ve already analyzed the data by grafical/descriptive illustrations. What I am interested in is, how other authors use to present comparable findings in their publications.

I guess you have to remember back in the old days before desk top computer processing became available and calculations were done by hand or by punch-cards, you would want a rule of thumb that was easy to derive. So rpb=.20 is just a convention for ensuring that the item has a non-chance positive association with the total. Of course arbitrary cut-offs are just that--arbitrary. If you rounded the example of .199 and .201 to .20 you would make the same rule of thumb decision. However, with modern computing you can determine rpb very quickly and accurately. Indeed, if the correlation is >0.00 you could probably keep the item even though it might not help discriminate candidates and it might never get used in any kind of selection or adaptive testing. Rules of thumb are how our industry started over 100 years ago when statistical and computing technology was primitive....keep it in mind when approaching conventions.

Dear Mahmoud, there are several available options for estimating item bias (or more adequately termed, Differential Item Functioning) using SPSS if the items are dichotomous. Two of the most well known non IRT methods are available using the Mantel Haenszel Statistic in crosstabs (you'll need to cross item response by group by total test score, but it will yield only uniform DIF results). Alternatively you may use Logistic regression methods using item response as dependent variable, and group, total score and Total score by group interacion for detecting non-uniform DIF. You'll need to interpret significance and pseudo-R2 values.

If items are ordinal, you cannot use the Mantel Haenszel test "as is", but you may still use Ordinal Logistic regression for detecting DIF in this case.

Interpretation depends of what kind of model you are using: Rasch or non-Rasch models... http://www.rasch.org/rmt/rmt191a.htm