Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science

Department of Psychology, Ohio State University, Columbus, OH 43201, USA.
Neural Computation (Impact Factor: 2.21). 12/2009; 22(4):887-905. DOI: 10.1162/neco.2009.02-09-959
Source: PubMed

ABSTRACT Discriminating among competing statistical models is a pressing issue for many experimentalists in the field of cognitive science. Resolving this issue begins with designing maximally informative experiments. To this end, the problem to be solved in adaptive design optimization is identifying experimental designs under which one can infer the underlying model in the fewest possible steps. When the models under consideration are nonlinear, as is often the case in cognitive science, this problem can be impossible to solve analytically without simplifying assumptions. However, as we show in this letter, a full solution can be found numerically with the help of a Bayesian computational trick derived from the statistics literature, which recasts the problem as a probability density simulation in which the optimal design is the mode of the density. We use a utility function based on mutual information and give three intuitive interpretations of the utility function in terms of Bayesian posterior estimates. As a proof of concept, we offer a simple example application to an experiment on memory retention.

Download full-text


Available from: Daniel R Cavagnaro, Jul 29, 2014
1 Follower
14 Reads
  • Source
    • "In this paper, we investigate the extent to which functional forms of the probability weighting function are discriminable in practice, and attempt to identify which functional form best describes human choice behavior. We do this by conducting experiments in which the choice-stimuli are optimized for discriminating between functional forms, using Adaptive Design Optimization (ADO; Cavagnaro et al., 2010). ADO is a computer-based experimentation methodology in which choice-stimuli (e.g., pairs of monetary gambles) are adapted in real-time in response to choices made by participants. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Probability weighting functions relate objective probabilities and their subjective weights, and play a central role in modeling choices under risk within cumulative prospect theory. While several different parametric forms have been proposed, their qualitative similarities make it challenging to discriminate among them empirically. In this paper, we use both simulation and choice experiments to investigate the extent to which different parametric forms of the probability weighting function can be discriminated using adaptive design optimization, a computer-based methodology that identifies and exploits model differences for the purpose of model discrimination. The simulation experiments show that the correct (data-generating) form can be conclusively discriminated from its competitors. The results of an empirical experiment reveal heterogeneity between participants in terms of the functional form, with two models (Prelec-2, Linear in Log Odds) emerging as the most common best-fitting models. The findings shed light on assumptions underlying these models.
    Journal of Risk and Uncertainty 12/2013; 47(3):255-289. DOI:10.1007/s11166-013-9179-3 · 1.53 Impact Factor
  • Source
    • "An information-theoretic method for model comparison was recently derived by Cavagnaro et al. (2010). Given a set of models with the i-th model having prior probability P0(i), stimuli are chosen to maximize the mutual information between the stimulus and the model index i by minimizing the expected model space entropy in a manner directly analogous to information-theoretic model estimation (Paninski, 2005), except that in this case the unknown variable is a discrete model index i rather than a continuous parameter value θ. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this paper, we review several lines of recent work aimed at developing practical methods for adaptive on-line stimulus generation for sensory neurophysiology. We consider various experimental paradigms where on-line stimulus optimization is utilized, including the classical optimal stimulus paradigm where the goal of experiments is to identify a stimulus which maximizes neural responses, the iso-response paradigm which finds sets of stimuli giving rise to constant responses, and the system identification paradigm where the experimental goal is to estimate and possibly compare sensory processing models. We discuss various theoretical and practical aspects of adaptive firing rate optimization, including optimization with stimulus space constraints, firing rate adaptation, and possible network constraints on the optimal stimulus. We consider the problem of system identification, and show how accurate estimation of non-linear models can be highly dependent on the stimulus set used to probe the network. We suggest that optimizing stimuli for accurate model estimation may make it possible to successfully identify non-linear models which are otherwise intractable, and summarize several recent studies of this type. Finally, we present a two-stage stimulus design procedure which combines the dual goals of model estimation and model comparison and may be especially useful for system identification experiments where the appropriate model is unknown beforehand. We propose that fast, on-line stimulus optimization enabled by increasing computer power can make it practical to move sensory neuroscience away from a descriptive paradigm and toward a new paradigm of real-time model estimation and comparison.
    Frontiers in Neural Circuits 06/2013; 7:101. DOI:10.3389/fncir.2013.00101 · 3.60 Impact Factor
  • Source
    • "In the present study, we adopt the ADO framework of Cavagnaro et al. (2010) to adaptive experimentation for discriminating generalized expected utility models of risky choice. Before describing the details of the ADO framework, we discuss two prerequisite issues in its application: (1) model specification (what is the proper probabilistic specification of a " qualitative " model of risky choice that captures the effect of stochastic variation in choice behavior?) "
    [Show abstract] [Hide abstract]
    ABSTRACT: Authors are encouraged to submit new papers to INFORMS journals by means of a style file template, which includes the journal title. However, use of a template does not certify that the paper has been accepted for publication in the named jour-nal. INFORMS journal templates are for the exclusive purpose of submitting to an INFORMS journal and should not be used to distribute the papers in print or online or to submit the papers to another publication. Collecting data to discriminate between models of risky choice requires careful selection of decision stimuli. Models of decision making aim to predict decisions across a wide range of possible stimuli, but practical limitations force experimenters to select only a handful of them for actual testing. Some stimuli tend to be more diagnostic between models than others, so the choice of stimuli is critical. This paper provides the theoretical background and a methodological framework for adaptive selection of optimal stimuli for discriminating among models of risky choice. The approach, called Adaptive Design Optimization (ADO), adapts the stimulus in each experimental trial based on the results of the preceding trials. We demonstrate the validity of the approach with simulation studies aiming to discriminate Expected Utility and Weighted Expected Utility models.
    Management Science 02/2013; 59(2). DOI:10.1287/mnsc.1120.1558 · 2.48 Impact Factor
Show more