Categorical Algebra - Science topic

Cramer's V will range from 0 to 1. You can play with some toy data to see how it reacts in different cases. The interpretation for a "large" effect changes depending on how many categories there are (in the dimension with the smaller number of categories). These interpretations are addressed in Cohen, 1988, Statistical Power Analysis for the Behavioral Sciences, 2nd Edition. Some other effect size statistics to consider are Tschuprow's T and Goodman Kruskal lambda.

Of course it is well known that the category of R-modules is suitable for this setting. Let me add one more reference

"Continuous Geometry" by J von Neumann, Oxford University Press, 1960.

Linda,

You can do it by using Stata's winsor2 command.  I recommend that you check the code. In excel, it would be a tougher way than in Stata.

Hi Daniel, 

I agree with both Jochen and Luke that are losing quite a lot of information using your approach. Now, I can think of two paths you could take to analyze your model, depending on the effort you want to put in: 

1) The easiest is just to fit a generalized linear model on your full data; the Accuracy would be the DV, the log reaction time (and any experimental condition you may have) the IV, the subject would be a random effect. 

In R, it would look somewhat like this: 

fit <- glmer(Accuracy ~ I(log(RT)) + ExpCondition + (1 | ID), family="binomial")

I suggest using the log RT to make the RT distribution somewhat more normally distributed. Mind you, it'll stiill probably be non-normal. 

2) The trickier path would be fitting a hierachical drift-diffusion model. It's a model that jointly analyzes RTs and Accuracy, taking in account their inter-dependence and their non-normal distribution, and it's specifically tailored for two-alternative forced choice response tasks. You would expect, I think, a difference between experimental conditions for parameter alpha. 

It's much trickier to implement, but I think it's the perfect fit for your experimental design. So, if you have time to dedicate to the project, check it out. The article I'm linking would be a good starting point, I think. 

KR-20 doesn't offer useful insight about score validity, though it is true that the maximum criterion-related validity coefficient for a given measure is equal to the square root of its estimated internal consistency reliability (given by KR-20). 

While the Rasch model suggests that item difficulty is the only specific aspect of items that is important to consider (which is why it is characterized as a one-parameter item response theory model), KR-20 and classical measurement models presume that items are fully interchangeable, and are therefore analogous to a zero-parameter model!  IRT models offer many advantages over classical models, including: (1) ability to check how well the model functions with a given data set (model-data fit); (2) estimates of item parameters that are theoretically independent of the sample of examinees used; (3) estimates of examinee proficiency that are theoretically independent of the set of items used; (4) simplifying many technical issues, like equating; and (5) calibrating both stimuli (items) and objects (examinees) onto a common scale. Moreover, Rasch model parameter estimates tend to converge quickly and accurately even with modest sample sizes.  For these reasons, IRT models have rightfully gained popularity since their introduction over 60 years ago. 

But, score validity is a broad arena, and neither classical reliability estimates nor IRT analysis alone will address the concerns associated with validity.