Adaptive design optimization: a mutual information-based approach to model discrimination in cognitive science.

Department of Psychology, Ohio State University, Columbus, OH 43201, USA.
Neural Computation (Impact Factor: 1.69). 12/2009; 22(4):887-905. DOI: 10.1162/neco.2009.02-09-959
Source: PubMed

ABSTRACT Discriminating among competing statistical models is a pressing issue for many experimentalists in the field of cognitive science. Resolving this issue begins with designing maximally informative experiments. To this end, the problem to be solved in adaptive design optimization is identifying experimental designs under which one can infer the underlying model in the fewest possible steps. When the models under consideration are nonlinear, as is often the case in cognitive science, this problem can be impossible to solve analytically without simplifying assumptions. However, as we show in this letter, a full solution can be found numerically with the help of a Bayesian computational trick derived from the statistics literature, which recasts the problem as a probability density simulation in which the optimal design is the mode of the density. We use a utility function based on mutual information and give three intuitive interpretations of the utility function in terms of Bayesian posterior estimates. As a proof of concept, we offer a simple example application to an experiment on memory retention.

1 Follower
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Knowledge representations are central to many cognitive processes, and how these representations change is a central issue in learning and cognitive development. Here we developed and implemented a Bayesian inferential procedure to detect and elucidate representational change in numerical estimation. The proposed procedure of an adaptive numerical experiment both infers a learner's representation and predicts the feedback that is likely to induce representational change. We provide an application of this procedure using simulated subjects and demonstrate its effectiveness in inferring representational state and inducing change.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Experimentation is at the core of research in the behavioral and neural sciences, yet observations can be expensive and time-consuming to acquire (e.g., MRI scans, responses from infant participants). A major interest of researchers is designing experiments that lead to maximal accumulation of information about the phenomenon under study with the fewest possible number of observations. In addressing this challenge, statisticians have developed adaptive design optimization methods. This letter introduces a hierarchical Bayes extension of adaptive design optimization that provides a judicious way to exploit two complementary schemes of inference (with past and future data) to achieve even greater accuracy and efficiency in information gain. We demonstrate the method in a simulation experiment in the field of visual perception.
    Neural Computation 08/2014; 26(11):1-28. DOI:10.1162/NECO_a_00654 · 1.69 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Computer games can be motivating and engaging experiences that facilitate learning, leading to their increasing use in education and behavioural experiments. For these applications, it is often important to make inferences about the knowledge and cognitive processes of players based on their behaviour. However, designing games that provide useful behavioural data are a difficult task that typically requires significant trial and error. We address this issue by creating a new formal framework that extends optimal experiment design, used in statistics, to apply to game design. In this framework, we use Markov decision processes to model players' actions within a game, and then make inferences about the parameters of a cognitive model from these actions. Using a variety of concept learning games, we show that in practice, this method can predict which games will result in better estimates of the parameters of interest. The best games require only half as many players to attain the same level of precision.
    Proceedings of The Royal Society A Mathematical Physical and Engineering Sciences 07/2014; 470(2167):20130828. DOI:10.1098/rspa.2013.0828 · 2.00 Impact Factor

Full-text (2 Sources)

Available from
Jul 29, 2014