Figure 5 - uploaded by Laureline Logiaco
Content may be subject to copyright.
Using a thalamic transition module rescues transitioning for nonlinear RNNs trained with SGD. a,b,c Additive model. a. Eigenspectrum of g ad J + U prep V prep after training of U prep and V prep (orange crosses) and when replacing J with a random matrix not used during training (blue dots). Black circle has radius g ad . b. |r| versus time before (dotted lines) and after (solid line) training of U prep and V prep . c. Left: network outputs for each motif when starting from 9 different random x values. Saturations (light to dark) indicate different trials. Right: network outputs for each motif when starting from the final x values of the other 9 motifs. Colors indicate the prior motif. The grey bars indicate the time during which the transition module was active. d,e,f. As in a,b,c for the multiplicative model.

Using a thalamic transition module rescues transitioning for nonlinear RNNs trained with SGD. a,b,c Additive model. a. Eigenspectrum of g ad J + U prep V prep after training of U prep and V prep (orange crosses) and when replacing J with a random matrix not used during training (blue dots). Black circle has radius g ad . b. |r| versus time before (dotted lines) and after (solid line) training of U prep and V prep . c. Left: network outputs for each motif when starting from 9 different random x values. Saturations (light to dark) indicate different trials. Right: network outputs for each motif when starting from the final x values of the other 9 motifs. Colors indicate the prior motif. The grey bars indicate the time during which the transition module was active. d,e,f. As in a,b,c for the multiplicative model.

Source publication
Preprint
Full-text available
We study learning of recurrent neural networks that produce temporal sequences consisting of the concatenation of re-usable "motifs". In the context of neuroscience or robotics, these motifs would be the motor primitives from which complex behavior is generated. Given a known set of motifs, can a new motif be learned without affecting the performan...

Contexts in source publication

Context 1
... more complex motif dynamics. However, asymptotic convergence guarantees are lost in this setting. Indeed, during a particular transition the network's dynamics (characterized by the connectivity g x J + U prep V prep and input b µ specific to the next motif) could have several fixed-points asides from the one which is reached in our simulations (Fig. 5). Therefore, in a more general context where the activity of the network at the end of a motif would settle in an 'exotic' part of the state-space far away from this fixed point, then the transition dynamics may not be able to bring the Here, we would like to mention another training strategy that would retain asymptotic convergence ...
Context 2
... more complex motif dynamics. However, asymptotic convergence guarantees are lost in this setting. Indeed, during a particular transition the network's dynamics (characterized by the connectivity g x J + U prep V prep and input b µ specific to the next motif) could have several fixed-points asides from the one which is reached in our simulations (Fig. 5). Therefore, in a more general context where the activity of the network at the end of a motif would settle in an 'exotic' part of the state-space far away from this fixed point, then the transition dynamics may not be able to bring the Here, we would like to mention another training strategy that would retain asymptotic convergence ...

Similar publications

Preprint
Full-text available
In 2006, during a meeting of a working group of scientists in La Jolla, California at The Neurosciences Institute (NSI), Gerald Edelman described a roadmap towards the creation of a Conscious Artifact. As far as I know, this roadmap was not published. However, it did shape my thinking and that of many others in the years since that meeting. This sh...
Article
Full-text available
INTRODUCCIÓN. El trabajo en el ámbito educativo se caracteriza por la constante preocupación de sus profesionales acerca de las mejores formas de enseñar y de aprender. En los últimos años, esta inquietud se ha venido reflejando en un interés creciente del profesorado por fundamentar neurocientíficamente sus prácticas educativas. El presente estudi...
Article
Full-text available
A fundamental assumption of learning theories is that the credit assigned to predictive cues is not simply determined by their probability of reinforcement, but by their ability to compete with other cues present during learning. This assumption has guided behavioral and neural science research for decades, and tremendous empirical and theoretical...