Adaptive design optimization: a mutual information-based approach to model discrimination in cognitive science.

Department of Psychology, Ohio State University, Columbus, OH 43201, USA.
Neural Computation (Impact Factor: 1.76). 12/2009; 22(4):887-905. DOI: 10.1162/neco.2009.02-09-959
Source: PubMed

ABSTRACT Discriminating among competing statistical models is a pressing issue for many experimentalists in the field of cognitive science. Resolving this issue begins with designing maximally informative experiments. To this end, the problem to be solved in adaptive design optimization is identifying experimental designs under which one can infer the underlying model in the fewest possible steps. When the models under consideration are nonlinear, as is often the case in cognitive science, this problem can be impossible to solve analytically without simplifying assumptions. However, as we show in this letter, a full solution can be found numerically with the help of a Bayesian computational trick derived from the statistics literature, which recasts the problem as a probability density simulation in which the optimal design is the mode of the density. We use a utility function based on mutual information and give three intuitive interpretations of the utility function in terms of Bayesian posterior estimates. As a proof of concept, we offer a simple example application to an experiment on memory retention.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Experimentation is at the core of research in the behavioral and neural sciences, yet observations can be expensive and time-consuming to acquire (e.g., MRI scans, responses from infant participants). A major interest of researchers is designing experiments that lead to maximal accumulation of information about the phenomenon under study with the fewest possible number of observations. In addressing this challenge, statisticians have developed adaptive design optimization methods. This letter introduces a hierarchical Bayes extension of adaptive design optimization that provides a judicious way to exploit two complementary schemes of inference (with past and future data) to achieve even greater accuracy and efficiency in information gain. We demonstrate the method in a simulation experiment in the field of visual perception.
    Neural computation. 08/2014;
  • [Show abstract] [Hide abstract]
    ABSTRACT: Experimentation is ubiquitous in the field of psychology and fundamental to the advancement of its science, and one of the biggest challenges for researchers is designing experiments that can conclusively discriminate the theoretical hypotheses or models under investigation. The recognition of this challenge has led to the development of sophisticated statistical methods that aid in the design of experiments and that are within the reach of everyday experimental scientists. This tutorial paper introduces the reader to an implementable experimentation methodology, dubbed Adaptive Design Optimization, that can help scientists to conduct "smart" experiments that are maximally informative and highly efficient, which in turn should accelerate scientific discovery in psychology and beyond.
    Journal of Mathematical Psychology 01/2013; 57(3-4):53-67. · 1.62 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Two experiments used a novel, open-ended, and adaptive test procedure to examine visual cognition in animals. Using a genetic algorithm, a pigeon was tested repeatedly from a variety of different initial conditions for its solution to an intermediate brightness search task. On each trial, the animal had to accurately locate and peck a target element of intermediate brightness from among a variable number of surrounding darker and lighter distractor elements. Displays were generated from 6 parametric variables, or genes (distractor number, element size, shape, spacing, target brightness, and distractor brightness). Display composition changed over time, or evolved, as a function of the bird's differential accuracy within the population of values for each gene. Testing 3 randomized initial conditions and 1 set of controlled initial conditions, element size and number of distractors were identified as the most important factors controlling search accuracy, with distractor brightness, element shape, and spacing making secondary contributions. The resulting changes in this multidimensional stimulus space suggested the existence of a set of conditions that the bird repeatedly converged upon regardless of initial conditions. This psychological "attractor" represents the cumulative action of the cognitive operations used by the pigeon in solving and performing this search task. The results are discussed regarding their implications for visual cognition in pigeons and the usefulness of adaptive, subject-driven experimentation for investigating human and animal cognition more generally. (PsycINFO Database Record (c) 2013 APA, all rights reserved).
    Journal of Experimental Psychology Animal Behavior Processes 09/2013; · 2.38 Impact Factor

Full-text (2 Sources)

Available from
Jul 29, 2014