[show abstract][hide abstract] ABSTRACT: We derive a family of risk-sensitive reinforcement learning methods for
agents, who face sequential decision-making tasks in uncertain environments. By
applying a utility function to the temporal difference (TD) error, nonlinear
transformations are effectively applied not only to the received rewards but
also to the true transition probabilities of the underlying Markov decision
process. When appropriate utility functions are chosen, the agents' behaviors
express key features of human behavior as predicted by prospect theory
(Kahneman and Tversky, 1979), for example different risk-preferences for gains
and losses as well as the shape of subjective probability curves. We derive a
risk-sensitive Q-learning algorithm, which is necessary for modeling human
behavior when transition probabilities are unknown, and prove its convergence.
As a proof of principle for the applicability of the new framework we apply it
to quantify human behavior in a sequential investment task. We find, that the
risk-sensitive variant provides a significantly better fit to the behavioral
data and that it leads to an interpretation of the subject's responses which is
indeed consistent with prospect theory. The analysis of simultaneously measured
fMRI signals show a significant correlation of the risk-sensitive TD error with
BOLD signal change in the ventral striatum. In addition we find a significant
correlation of the risk-sensitive Q-values with neural activity in the
striatum, cingulate cortex and insula, which is not present if standard
Q-values are used.
[show abstract][hide abstract] ABSTRACT: Without non-linear basis functions many problems can not be solved by linear algorithms. This article proposes a method to automatically construct such basis functions with slow feature analysis (SFA). Non-linear optimization of this unsupervised learning method generates an orthogonal basis on the unknown latent space for a given time series. In contrast to methods like PCA, SFA is thus well suited for techniques that make direct use of the latent space. Real-world time series can be complex, and current SFA algorithms are either not powerful enough or tend to over-fit. We make use of the kernel trick in combination with sparsification to develop a kernelized SFA algorithm which provides a powerful function class for large data sets. Sparsity is achieved by a novel matching pursuit approach that can be applied to other tasks as well. For small data sets, however, the kernel SFA approach leads to over-fitting and numerical instabilities. To enforce a stable solution, we introduce regularization to the SFA objective. We hypothesize that our algorithm generates a feature space that resembles a Fourier basis in the unknown space of latent variables underlying a given real-world time series. We evaluate this hypothesis at the example of a vowel classification task in comparison to sparse kernel PCA. Our results show excellent classification accuracy and demonstrate the superiority of kernel SFA over kernel PCA in encoding latent variables.
[show abstract][hide abstract] ABSTRACT: Brain stimulation is having remarkable impact on clinical neurology. Brain stimulation can modulate neuronal activity in functionally segregated circumscribed regions of the human brain. Polarity, frequency, and noise specific stimulation can induce specific manipulations on neural activity. In contrast to neocortical stimulation, deep-brain stimulation has become a tool that can dramatically improve the impact clinicians can possibly have on movement disorders. In contrast, neocortical brain stimulation is proving to be remarkably susceptible to intrinsic brain-states. Although evidence is accumulating that brain stimulation can facilitate recovery processes in patients with cerebral stroke, the high variability of results impedes successful clinical implementation. Interestingly, recent data in healthy subjects suggests that brain-state dependent patterned stimulation might help resolve some of the intrinsic variability found in previous studies. In parallel, other studies suggest that noisy "stochastic resonance" (SR)-like processes are a non-negligible component in non-invasive brain stimulation studies. The hypothesis developed in this manuscript is that stimulation patterning with noisy and oscillatory components will help patients recover from stroke related deficits more reliably. To address this hypothesis we focus on two factors common to both neural computation (intrinsic variables) as well as brain stimulation (extrinsic variables): noise and oscillation. We review diverse theoretical and experimental evidence that demonstrates that subject-function specific brain-states are associated with specific oscillatory activity patterns. These states are transient and can be maintained by noisy processes. The resulting control procedures can resemble homeostatic or SR processes. In this context we try to extend awareness for inter-individual differences and the use of individualized stimulation in the recovery maximization of stroke patients.
Frontiers in Human Neuroscience 01/2013; 7:325. · 2.91 Impact Factor
[show abstract][hide abstract] ABSTRACT: Evaluating the importance of higher-order correlations of neural spike counts has been notoriously hard. A large number of samples are typically required in order to estimate higher-order correlations and resulting information theoretic quantities. In typical electrophysiology data sets with many experimental conditions, however, the number of samples in each condition is rather small. Here we describe a method that allows to quantify evidence for higher-order correlations in exactly these cases. We construct a family of reference distributions: maximum entropy distributions, which are constrained only by marginals and by linear correlations as quantified by the Pearson correlation coefficient. We devise a Monte Carlo goodness-of-fit test, which tests--for a given divergence measure of interest--whether the experimental data lead to the rejection of the null hypothesis that it was generated by one of the reference distributions. Applying our test to artificial data shows that the effects of higher-order correlations on these divergence measures can be detected even when the number of samples is small. Subsequently, we apply our method to spike count data which were recorded with multielectrode arrays from the primary visual cortex of anesthetized cat during an adaptation experiment. Using mutual information as a divergence measure we find that there are spike count bin sizes at which the maximum entropy hypothesis can be rejected for a substantial number of neuronal pairs. These results demonstrate that higher-order correlations can matter when estimating information theoretic quantities in V1. They also show that our test is able to detect their presence in typical in-vivo data sets, where the number of samples is too small to estimate higher-order correlations directly.
[show abstract][hide abstract] ABSTRACT: We introduce a unified framework to incorporate risk in Markov decision
processes (MDPs), via prospect maps, which generalize the idea of
coherent/convex risk measures in mathematical finance. Most of the existing
risk-sensitive approaches in various literature concerning with decision-making
problems are contained in the framework as special instances. Within the
framework, we solve the optimal control problems according to two criteria, the
newly invented temporal discounted criterion, which generalizes the
conventional discount scheme, and the average criterion, by value iteration
algorithms under different assumptions. Two online algorithms are proposed to
solve the optimal controls problem when the exact MDP is unknown and has to be
estimated during optimization.
[show abstract][hide abstract] ABSTRACT: Visual stimulation often leads to elevated fluctuations of the membrane potential in the γ-frequency range (25-70 Hz) in visual cortex neurons. Recently, we have found that the strength of γ-band fluctuations is coupled to the oscillation of the membrane potential at the temporal frequency of the stimulus, so that the γ-band fluctuations are stronger at depolarization peaks, but weaker at troughs of the stimulus frequency oscillation of the membrane potential. We hypothesized that this coupling may improve stimulus encoding. Here, we tested this hypothesis by using a single-compartment conductance-based neuron model, with parameters of the input adjusted to reproduce typical features of membrane potential and spike responses, recorded in cat visual cortical neurons in vivo during the presentation of moving gratings. We show that modulation of the γ-range membrane potential fluctuations by the amplitude of the slow membrane depolarization greatly improves stimulus encoding. Moreover, changing the degree of modulation of the γ-activity by the low-frequency signal within the range typically observed in visual cortex cells had a stronger effect on both the firing rates and information rates than changing the amplitude of the low-frequency stimulus itself. Thus, modulation of the γ-activity represents an efficient mechanism for regulation of neuronal firing and encoding of the temporal characteristics of visual stimuli.
European Journal of Neuroscience 03/2011; 33(7):1223-39. · 3.75 Impact Factor
[show abstract][hide abstract] ABSTRACT: This contribution extends the Bron Kerbosch algorithm for solving the maximum
weight clique problem, where continuous-valued weights are assigned to both,
vertices and edges. We applied the proposed algorithm to graph matching
[show abstract][hide abstract] ABSTRACT: The chromosome aberration test is frequently used for the assessment of the potential of chemicals and drugs to elicit genetic damage in mammalian cells in vitro. Due to the limitations of experimental genotoxicity testing in early drug discovery phases, a model to predict the chromosome aberration test yielding high accuracy and providing guidance for structure optimization is urgently needed. In this paper, we describe a machine learning approach for predicting the outcome of this assay based on the structure of the investigated compound. The novelty of the proposed method consists in combining a maximum common subgraph kernel for measuring the similarity of two chemical graphs with the potential support vector machine for classification. In contrast to standard support vector machine classifiers, the proposed approach does not provide a black box model but rather allows to visualize structural elements with high positive or negative contribution to the class decision. In order to compare the performance of different methods for predicting the outcome of the chromosome aberration test, we compiled a large data set exhibiting high quality, reliability, and consistency from public sources and configured a fixed cross-validation protocol, which we make publicly available. In a comparison to standard methods currently used in pharmaceutical industry as well as to other graph kernel approaches, the proposed method achieved significantly better performance.
Journal of Chemical Information and Modeling 09/2010; 50(10):1821-38. · 4.30 Impact Factor
[show abstract][hide abstract] ABSTRACT: Extracellular recordings are a key tool to record the activity of neurons in vivo. Especially in the case of experiments with behaving animals, however, the tedious procedure of electrode placement can take a considerable amount of expensive and restricted experimental time. Furthermore, due to tissue drifts and other sources of variability in the recording setup, the position of the electrodes with respect to the recorded neurons can change causing low recording quality. The contributions of this work are threefold. We introduce a quality measure for the recording position of the electrode which should be maximized during recordings and is especially suitable for the use of multi-electrodes. An automated positioning system based on this quality measure is proposed. The system is able to find favorable recording positions and adapts the electrode position smoothly to changes of the neuron positions. Finally, we evaluate the system using a new simulator for extracellular recordings based on realistically reconstructed 3D neurons.
Conference proceedings: ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference 01/2010; 2010:593-7.
[show abstract][hide abstract] ABSTRACT: In the case of extracellular recordings, spike detection algorithms are necessary in order to retrieve information about neuronal activity form the data. We present a new spike detection algorithm which is based on methods from the field of blind equalization and beamforming. In contrast to existing approaches, our method estimates several waveforms directly from the data and corresponding linear filters are constructed. The estimation is done in an unsupervised manner, and the few parameters of the algorithm are intuitive to set. The algorithm allows for superior detection performance, even when multiple neurons with various waveforms are present in the data. We compare our method with current state-of-the-art spike detection algorithms, and show that the proposed method achieves favorable results.
Conference proceedings: ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference 01/2010; 2010:4636-41.
[show abstract][hide abstract] ABSTRACT: In this paper, we consider a class of network models of Hodgkin-Huxley type neurons arranged according to a biologically plausible two-dimensional topographic orientation preference map, as found in primary visual cortex (V1). We systematically vary the strength of the recurrent excitation and inhibition relative to the strength of the afferent input in order to characterize different operating regimes of the network. We then compare the map-location dependence of the tuning in the networks with different parametrizations with the neuronal tuning measured in cat V1 in vivo . By considering the tuning of neuronal dynamic and state variables, conductances and membrane potential respectively, our quantitative analysis is able to constrain the operating regime of V1: The data provide strong evidence for a network, in which the afferent input is dominated by strong, balanced contributions of recurrent excitation and inhibition, operating in vivo . Interestingly, this recurrent regime is close to a regime of "instability", characterized by strong, self-sustained activity. The firing rate of neurons in the best-fitting model network is therefore particularly sensitive to small modulations of model parameters, possibly one of the functional benefits of this particular operating regime.
Journal of Physics: Conference Series. 01/2010; 233(1):012020.
[show abstract][hide abstract] ABSTRACT: This paper formulates a necessary and sufficient condition for a generic graph matching problem to be equivalent to the maximum vertex and edge weight clique problem in a derived association graph. The consequences of this results are threefold: first, the condition is general enough to cover a broad range of practical graph matching problems; second, a proof to establish equivalence between graph matching and clique search reduces to showing that a given graph matching problem satisfies the proposed condition; and third, the result sets the scene for generic continuous solutions for a broad range of graph matching problems. To illustrate the mathematical framework, we apply it to a number of graph matching problems, including the problem of determining the graph edit distance. Comment: 19 pages
[show abstract][hide abstract] ABSTRACT: Finite structures such as point patterns, strings, trees, and graphs occur as "natural" representations of structured data in different application areas of machine learning. We develop the theory of structure spaces and derive geometrical and analytical concepts such as the angle between structures and the derivative of functions on structures. In particular, we show that the gradient of a differentiable structural function is a well-defined structure pointing in the direction of steepest ascent. Exploiting the properties of structure spaces, it will turn out that a number of problems in structural pattern recognition such as central clustering or learning in structured output spaces can be formulated as optimization problems with cost functions that are locally Lipschitz. Hence, methods from nonsmooth analysis are applicable to optimize those cost functions.
Journal of Machine Learning Research 12/2009; 10:2667-2714. · 3.42 Impact Factor
[show abstract][hide abstract] ABSTRACT: Simultaneous spike-counts of neural populations are typically modeled by a Gaussian distribution. On short time scales, however, this distribution is too restrictive to describe and analyze multivariate distributions of discrete spike-counts. We present an alternative that is based on copulas and can account for arbitrary marginal distributions, including Poisson and negative binomial distributions as well as second and higher-order interactions. We describe maximum likelihood-based procedures for fitting copula-based models to spike-count data, and we derive a so-called flashlight transformation which makes it possible to move the tail dependence of an arbitrary copula into an arbitrary orthant of the multivariate probability distribution. Mixtures of copulas that combine different dependence structures and thereby model different driving processes simultaneously are also introduced. First, we apply copula-based models to populations of integrate-and-fire neurons receiving partially correlated input and show that the best fitting copulas provide information about the functional connectivity of coupled neurons which can be extracted using the flashlight transformation. We then apply the new method to data which were recorded from macaque prefrontal cortex using a multi-tetrode array. We find that copula-based distributions with negative binomial marginals provide an appropriate stochastic model for the multivariate spike-count distributions rather than the multivariate Poisson latent variables distribution and the often used multivariate normal distribution. The dependence structure of these distributions provides evidence for common inhibitory input to all recorded stimulus encoding neurons. Finally, we show that copula-based models can be successfully used to evaluate neural codes, e.g., to characterize stimulus-dependent spike-count distributions with information measures. This demonstrates that copula-based models are not only a versatile class of models for multivariate distributions of spike-counts, but that those models can be exploited to understand functional dependencies.
[show abstract][hide abstract] ABSTRACT: In this analytical study we derive the optimal unbiased value estimator (MVU) and compare its statistical risk to three well known value estimators: Temporal Difference learning (TD), Monte Carlo estimation (MC) and Least-Squares Temporal Difference Learning (LSTD). We demonstrate that LSTD is equivalent to the MVU if the Markov Reward Process (MRP) is acyclic and show that both differ for most cyclic MRPs as LSTD is then typically biased. More generally, we show that estimators that fulfill the Bellman equation can only be unbiased for special cyclic MRPs. The main reason being the probability measures with which the expectations are taken. These measure vary from state to state and due to the strong coupling by the Bellman equation it is typically not possible for a set of value estimators to be unbiased with respect to each of these measures. Furthermore, we derive relations of the MVU to MC and TD. The most important one being the equivalence of MC to the MVU and to LSTD for undiscounted MRPs in which MC has the same amount of information. In the discounted case this equivalence does not hold anymore. For TD we show that it is essentially unbiased for acyclic MRPs and biased for cyclic MRPs. We also order estimators according to their risk and present counter-examples to show that no general ordering exists between the MVU and LSTD, between MC and LSTD and between TD and MC. Theoretical results are supported by examples and an empirical evaluation. Comment: Final version is under review. 38 pages, 8 figures
[show abstract][hide abstract] ABSTRACT: For the analysis of neuronal cooperativity, simultaneously recorded extracellular signals from neighboring neurons need to be sorted reliably by a spike sorting method. Many algorithms have been developed to this end, however, to date, none of them manages to fulfill a set of demanding requirements. In particular, it is desirable to have an algorithm that operates online, detects and classifies overlapping spikes in real time, and that adapts to non-stationary data. Here, we present a combined spike detection and classification algorithm, which explicitly addresses these issues. Our approach makes use of linear filters to find a new representation of the data and to optimally enhance the signal-to-noise ratio. We introduce a method called "Deconfusion" which de-correlates the filter outputs and provides source separation. Finally, a set of well-defined thresholds is applied and leads to simultaneous spike detection and spike classification. By incorporating a direct feedback, the algorithm adapts to non-stationary data and is, therefore, well suited for acute recordings. We evaluate our method on simulated and experimental data, including simultaneous intra/extra-cellular recordings made in slices of a rat cortex and recordings from the prefrontal cortex of awake behaving macaques. We compare the results to existing spike detection as well as spike sorting methods. We conclude that our algorithm meets all of the mentioned requirements and outperforms other methods under realistic signal-to-noise ratios and in the presence of overlapping spikes.
Journal of Computational Neuroscience 07/2009; 29(1-2):127-48. · 2.44 Impact Factor
[show abstract][hide abstract] ABSTRACT: The number of studies on imaging genetics has risen considerably over the last few years, and haplotypes are being increasingly applied as a model to increase the explained variance in functional brain activation. Haplotypes, however, are not always the preferable approach. While such highly complex models have a greater capacity for fitting data, they might also lead to over-fitting. This study compares individual single nucleotide polymorphisms (SNPs) with haplotypes by applying both models to effects of catechol-O-methyltransferase (COMT), one of the most extensively studied genes in psychiatric research and imaging genetics, on the central processing of affective cues. To our knowledge, this is the first study to compare haplotypes and SNPs of the COMT gene in an imaging genetics study. The model comparison in this study is based on the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC), introducing the novel concepts of posterior evidence ratio maps and best model maps. Findings reveal the simplest model, comprising only the well studied COMT Val(158)Met polymorphism, to be the most informative one. These results do not necessarily mean that haplotype models are in general inferior to individual SNP analysis. They do underline, however, that techniques for model comparison such as the ones used in this study need to be employed to establish whether the increase in likelihood provided by a more complex haplotype-based model is large enough to warrant the increase in model complexity.
[show abstract][hide abstract] ABSTRACT: In V1, local circuitry depends on the position in the orientation map: close to pinwheel centers, recurrent inputs show variable orientation preferences; within iso-orientation domains, inputs are relatively uniformly tuned. Physiological properties such as cell's membrane potentials, spike outputs, and temporal characteristics change systematically with map location. We investigate in a firing rate and a Hodgkin-Huxley network model what constraints these tuning characteristics of V1 neurons impose on the cortical operating regime. Systematically varying the strength of both recurrent excitation and inhibition, we test a wide range of model classes and find the likely models to account for the experimental observations. We show that recent intracellular and extracellular recordings from cat V1 provide the strongest evidence for a regime where excitatory and inhibitory recurrent inputs are balanced and dominate the feed-forward input. Our results are robust against changes in model assumptions such as spatial extent and strength of lateral inhibition. Intriguingly, the most likely recurrent regime is in a region of parameter space where small changes have large effects on the network dynamics, and it is close to a regime of "runaway excitation," where the network shows strong self-sustained activity. This could make the cortical response particularly sensitive to modulation.