Science topic

# Computerized Adaptive Testing - Science topic

Computerized adaptive testing (CAT) selects those items from the bank according to estimated abilities of a student, seek to maximize the test's accuracy, based on individual or multiple simultaneous examination abilities.
Questions related to Computerized Adaptive Testing
Question
I am beginning an experiment assessing timing-related behavior in adults with ADHD and the perceptual measures I plan to use are adaptive, and determine perceptual thresholds using standard adaptive algorithm procedures (e.g. staircase method). However, I'm concerned about the inevitable impact of attentional lapses on thresholds. I am interested in suggestions for how best to tune the staircase parameters and/or suggestions for other adaptive algorithms that may be more resilient to lapses of attention. Any thoughts?
2-the adaptive procedure: I think, indeed the most suitable are the staircase
Question
I have an IRT question regarding calibration of the items. I am also aware of the item parameters are invariant to the sample using which they are estimated.
Suppose there are three schools, namely:A,B,and C. Students in school A are having below the average intelligence (IQ) and the school doesn't have exposure to computerized adaptive test (CAT) while students in school B have moderate IQ and have partial information about technology of the CAT, whereas students from School C are the brightest ones and also they are experienced with CAT.
Suppose, We have estimated the item parameters of an item bank (of say Mathematics subject) using response data from the school C (students with not so good intelligence and not aware of CAT)
Now suppose if I assume the item invariant assumption of IRT to be true and ask school C(one with bright students) students to take the test on same item bank, and turns out they all perform well.
So adjustments should I do with item bank so that I should be able to compare the results students from School A , B, and C with the numerical score I have got from the test.
Is this test a good way to compare these students?
Would the results be different had I estimated the item parameters using the response data from school C? (First thought comes to my mind is yes).
Am I missing something here?
I am open to the discussion.
Yes, the Rasch model is sample-free and test-free provided that (a) the data fit the Rasch model (as indicated by item and model fit) and (b) there is no DIF present in the items. So assessing DIF is important to determining if the measure has the property of being sample free.
Question
I have a dataset with a lot of missing values due to the fact that this data was generated by a computer adaptive test.
I want to show unidimensionality of the test, but factor analysis is a problem due to the missings. The paper below seems to suggest that we can use ones theta estimate (I'm using IRTPRO to estimate it based on the EAP method) to calculate expected scores and impute them - which seems intuitive. Is this a correct way?
Can I simply proceed with factor analysis after this, or would I need to round the probabilities to 0 and 1's? Are there other methods?
Regards, Dirk
You are taking on something that is very challenging because of the amount of missing data and the pattern of missingness.  Also, the number of observations per test item will differ.  I imagine that some will have very low frequency.  I would recommend first creating a large person by item matrix and then sort the item dimension by the frequency of observations.  Then, plan on doing your analysis on the items that have a frequency greater then some number like 100.  I would prefer 200, but I don't know how large your data-set is.  Then, check the dimensionality on the smaller set of items.  Using a program that will handle the missing data is a necessity, but it is also possible to impute the missing values, but I would not impute the missing item responses with a unidimensional IRT model.  That will cause underestimation of the dimensionality.  When I have done this, I have computed the correlation matrix for the items and then imputed the correlations for missing item pairs.  The resulting correlation matrix may need to be smoothed to avoid negative eigenvalues.  Bock has a procedure for doing this.  Then, I prefer doing parallel analysis on the correlation matrix.  This all needs to be done cautiously because of all of the issues that are involved.
Question
Ideally such a tool would use a computerized adaptive testing approach?
I am looking for a method which can be used to more precisely screen for a spectrum of behavioral health issues without using an extensive battery of questions?
Such a tool would ideally be used to anonymously screen large numbers of individuals.  The validity and reliability would need to have been established so that the tool could stand up to some scrutiny.
Thank you all for your conscientious answers.  Do you know of similar IRT based tools for adults?
Question
Hello everyone,
I need to know what all properties should an item bank have to carry out a precise CAT. I am particularly interested in the 1PL model.
Properties I am looking for
1. What should be the optimum size of the item bank?
2. What should be the distribution of item difficulty?
Any references are also welcome.
In this paper, we describe setting up the foundations for an item bank that could be used as a basis for CAT. It focuses on the 2PL model, but the approach could be used for the 1PL model too.
Yours,
Question
Hello everyone,
I was wondering what kind of data analysis on the results do these companies which are into  computerized adaptive testing provide?
What kind of charts or plots on the test results of the individual they provide?
Does anyone know such companies?
The PROMIS Initiative of the NIH has extensive work on CATs for a range of domains. You could do a demonstration from their website and see what information they report for an individual. They are not a company but should provide a good example of best practice. (You can also use their CAT systems free of charge)
Question
I'm looking into the development of an on-line IRT adaptive test for a specific test instrument. Any pointers to help me start out such has published research or case studies would be grateful. I've come across the Concerto platform but would be interested to know what else is out there.
Hello Edmund,
In case you are looking for IRT based adaptive testing R packages , please go through catR , mirt and mirtCAT packages. Especially mirtCAT package , it provides tools to generate an HTML interface for creating adaptive testing.
Hope it helps you.