Content uploaded by Aaron Russell Voelker

Author content

All content in this area was uploaded by Aaron Russell Voelker on Oct 02, 2017

Content may be subject to copyright.

Analysis of oscillatory weight changes from online

learning with ﬁltered spiking feedback

Aaron R. Voelker and Chris Eliasmith

Centre for Theoretical Neuroscience technical report.

October 1, 2017

Abstract

Prescribed Error Sensitivity (PES) is a biologically plausible super-

vised learning rule that is frequently used with the Neural Engineering

Framework (NEF). PES modiﬁes the connection weights between popula-

tions of spiking neurons to minimize an error signal. Continuing the work

of Voelker (2015), we solve for the dynamics of PES, while ﬁltering the

error with an arbitrary linear synapse model. For the most common case

of a lowpass ﬁlter, the continuous-time weight changes are characterized

by a second-order bandpass ﬁlter with frequency ω=pτ−1κkak2and

bandwidth Q=pτ κkak2, where τis the exponential time constant, κis

the learning rate, and ais the activity vector. Therefore, the error con-

verges to zero, yet oscillates if and only if τκkak2>1

4. This provides a

heuristic for setting κbased on the synaptic τ, and a method for engineer-

ing remarkably accurate decaying oscillators using only a single spiking

leaky integrate-and-ﬁre neuron.

1 Introduction

The Neural Engineering Framework (NEF; Eliasmith and Anderson, 2003) is a

method for constructing biologically plausible spiking networks. To build and

simulate such models, the Centre for Theoretical Neuroscience makes extensive

use of the open-source software, Nengo (Bekolay et al., 2014). Nengo typically

learns its connection weights oﬄine, but also supports a number of biologically

plausible supervised and unsupervised learning rules to learn its weights on-

line. By far, the most commonly used learning rule in Nengo is the Prescribed

Error Sensitivity (PES; Bekolay et al., 2013) rule, which learns a function by

minimizing a supervised error signal from external and recurrent feedback.

Previously, Voelker (2015) fully characterized the discrete-time dynamics of

PES under the restricted setting of a constant input signal, constant reference

signal, and no noise. Due to the absence of noise, no ﬁlter was required for the

error signal. However, for spiking networks considered in practice, a lowpass is

applied to the error to ﬁlter out spike-noise (e.g., DeWolf et al., 2016; Rasmussen

et al., 2017),

In this report, we relax the assumption of a constant reference signal, and

apply an arbitrary linear ﬁlter to the error signal. For simplicity, we do so for

the case of a continuous-time simulation, but our analysis can also be applied

1

r(t)

y(t)

x

–

h(t)

d(t)

e(t)

a

+

Figure 1: Network diagram used to analyze the PES rule. A constant in-

put xis represented by a population of nspiking neurons with the activ-

ity vector a∈Rn. A dynamic reference signal r(t) determines the error

e(t) = ((y−r)∗h) (t) (equation 3), which in turn drives y(t) towards r(t) by

modulating the connection weights via PES (equation 2). These learned con-

nection weights decode y(t) via the decoders d(t)∈Rn(equation 1). A linear

ﬁlter h(t) models the postsynaptic current induced by each spike.

to the discrete-time setting via the Z-transform. To keep our analysis tractable,

we still assume a constant input signal, and brieﬂy discuss implications for the

general dynamic setting.

We begin by formulating a mathematical description of the network in sec-

tion 2. We present our theoretical results in section 3, and prove them in

section 4. In section 5, we validate our results with numerical simulations, and

demonstrate the utility of this analysis by engineering oscillators with prede-

termined frequencies and decay rates. Finally, we conclude in section 6 by dis-

cussing some implications of this report for learning spiking dynamical networks

online.

2 Prescribed Error Sensitivity

Consider a network in Nengo, containing a population of nspiking neurons,

encoding the constant scalar input x. Let a∈Rnbe the average (i.e., rate)

activity of each neuron in response to this encoding.1This vector is determined

by the ﬁrst principle of the NEF, and remains ﬁxed for constant x. The decoders

d(t)∈Rndetermine the scalar output y(t) via the dot-product:

y(t) = aTd(t). (1)

The PES rule learns these decoders, online, according to the following dynamics:

˙

d(t) = −κe(t)a, (2)

where κ > 0 is the learning rate,2and e(t) is the chosen error signal:3

e(t) = ((y−r)∗h) (t), (3)

1Here on, we assume that a6=0, otherwise PES will have no eﬀect.

2κis automatically scaled by n−1in Nengo, to balance the linear scaling of kak2.

3Signs are ﬂipped in Voelker (2015); equations 2 and 3 are consistent with Nengo.

2

where r(t) is the reference (i.e., ideal) output, and h(t) is some arbitrary lin-

ear ﬁlter modeling the postsynaptic current (PSC) induced by a spike arriving

at the synaptic cleft. Typically, h(t) is a ﬁrst-order lowpass ﬁlter with time

constant τ > 0 (i.e., modeling an exponentially decaying PSC):

h(t) = 1

τe−t

τ⇐⇒ H(s) = 1

τs + 1.4(4)

The ﬁnal network is summarized in Figure 1. This also naturally extends to the

case where xand yare vectors (using a population code), but we consider the

scalar case for simplicity.

Now, we aim to characterize the dynamics of e(t) in response to the control

signal r(t). Alternatively, we could characterize the dynamics of y(t) or d(t),

but the former is easier to work with, while describing the latter via equations 1

and 2 (i.e., by integrating e(t)).

3 Results

Let φ=τκkak2. For the network described in Figure 1 and section 2, we have:

e(t) = (r∗f)(t), (5)

where:

F(s) = −s

sH(s)−1+κkak2, (6)

hence F(s) is the transfer function from R(s) to E(s). For the case of a ﬁrst-

order lowpass ﬁlter (equation 4),

=⇒F(s) = −s

τs2+s+κkak2=−κkak2−1s

1

ω2s2+1

ωQ s+ 1

(7)

ω=pτ−1κkak2=τ−1pφ(8)

Q=pτκkak2=pφ. (9)

Thus, F(s) is a second-order Q-bandpass ﬁlter with frequency ωin radians per

second ( ω

2πis the frequency in hertz) (Zumbahlen et al., 2011, pp. 8.9–8.10).

The poles of F(s) are:

s=−1±√1−4φ

2τ. (10)

Since φ > 0, this system is exponentially stable, and, moreover, the impulse

response, f(t), is a decaying oscillator if and only if φ > 1

4.

4 Proof

We begin by transforming equations 1–3 into the Laplace domain:

Y(s) = aTD(s)

sD(s) = −κE(s)a

E(s) = (Y(s)−R(s)) H(s).5

4Capital-case variables denote the Laplace transforms of their corresponding lower-case

(time-domain) variables.

3

Substituting the ﬁrst two equations into the last, yields:

E(s) = aTD(s)−R(s)H(s)

=−aTs−1κE(s)a−R(s)H(s)

=−κkak2s−1H(s)E(s)−R(s)H(s)

⇐⇒ 1 + κkak2s−1H(s)E(s) = −R(s)H(s)

⇐⇒ E(s) = R(s)−H(s)

1 + κkak2s−1H(s)

=R(s)−s

sH(s)−1+κkak2

=R(s)F(s).

Equations 5 and 6 follow from the convolution theorem. Equations 7–9 are

veriﬁed by substituting H(s)−1=τs + 1 into equation 6.

The poles of the system (equation 10) are obtained by applying the quadratic

formula to the denominator polynomial from equation 7 (τs2+s+κkak2).

Exponential stability is implied by both poles being strictly in the left half-

plane. Lastly, f(t) oscillates if and only if the poles are complex, if and only if

the discriminant (1 −4φ) is negative, if and only if φ > 1

4.

5 Validation

We construct the network from Figure 1 using Nengo 2.5.0 (Bekolay et al., 2013),

n= 1 spiking leaky integrate-and-ﬁre neurons (mean ﬁring rate of 262 Hz),6

τ= 0.1 s (equation 4), x= 0, and κsuch that φ > 1

4. We construct the transfer

function from equation 7 using nengolib 0.4.0 (Voelker, 2017):

im po rt nengolib

from ne ng ol ib. s ig na l import s

H = nengo li b.L ow pass( tau )

F = −s / (s /H + ka pp a∗a. do t( a ))

where tau ←τis the time constant of the synapse, kappa ←κis the learning-

rate supplied to Nengo (divided by n), and a←ais the NumPy array for

the population’s activity. We evaluate (r∗f)(t) using F.filt(r, dt=dt), and

compare this to the e(t) obtained numerically in simulation. The filt method

automatically discretizes F(s) according to the simulation time-step (dt = 1 ms)

using zero-order hold (ZOH).7

In Figure 2, we conﬁrm that (r∗f)(t) approximates the numerical e(t) given

white noise r(t). In Figure 3-Top, we exploit our knowledge of the impulse

response, f(t), to engineer a number of decaying oscillators by controlling r(t).

In Figure 3-Bottom, we evaluate np.abs(F.evaluate(freqs)) at a variety of

frequencies (freqs) to visualize the bandpass behaviour of each ﬁlter.

5This has nice form since ais a constant – otherwise multiplication of two time-varying

signals becomes a complex integral in the Laplace domain.

6Spikes are used in place of ain equations 1 and 2.

7Technically, for a discrete-time simulation, the problem and results should have been

formulated in the discrete-time domain using the Z-transform to begin with, as opposed to

discretizing at the end, but the diﬀerence is quite subtle.

4

Figure 2: Comparison of the analytical error (equation 7) to the numerical e(t)

obtained by simulating the network from Figure 1 (κ= 10−3). The control

signal r(t) is randomly sampled white noise with a cutoﬀ frequency of 10 Hz.

The normalized root-mean-square error is approximately 3.7%.

Figure 3: Harnessing the dynamics of PES with various κto engineer decaying

oscillators with predetermined frequencies (ω) and bandwidths (Q). The time

constant of the ﬁrst-order lowpass ﬁlter is ﬁxed at τ= 0.1 s, while κis set

to achieve the desired ωvia equation 8. (Top) Once every second, r(t) is set

to a unit-area impulse. Consequently, e(t) oscillates according to the impulse

response, f(t). (Bottom) Visualizing the ideal frequency response of F(s) (equa-

tion 7). Dashed lines at ω(equation 8) align with the peak of each bandpass

ﬁlter, or equivalently the frequency of each oscillation. The width of each ﬁlter

is proportional to the decay rate Q−1(equation 9).

5

6 Discussion

Since φ=τκkak2>1

4if and only if the weights oscillate, this motivates a simple

heuristic for setting the learning rate to prevent oscillatory weight changes: set

κ≤1

4τkak2, where kak2is maximal over all possible activity vectors. In this

case, equation 7 factors into a (diﬀerentiated) double-exponential:

F(s) = −τ1τ2s

τ(τ1s+ 1)(τ2s+ 1) =−τ1τ2τ−1s1

τ1s+ 1 1

τ2s+ 1,

that is, two ﬁrst-order lowpass ﬁlters chained together, where:

(τ1, τ2) = 2τ

1∓√1−4φ,

by equation 10. In other words, the non-oscillatory regime (0 < φ ≤1

4) of

PES is characterized by the dynamics of a double-exponential. We remark that

τ1=τ2= 2τ(i.e., an alpha ﬁlter) directly on the point of bifurcation from

double-exponential to oscillatory behaviour (φ=1

4; see Figure 4).

In all applications involving online learning (that we are aware of) oscillatory

weight changes are viewed as problematic, and so the relevant constants (τ,κ,

and kak2) are tweaked until the issue disappears. In contrast, we have shown

that not only can the relationship between these constants and the oscillations

be fully understood, but they can be harnessed to engineer bandpass ﬁlters

(with respect to the transformation r(t)7→ e(t)) with speciﬁc frequencies (ω)

and bandwidths (Q). More generally, the PES learning rule can be used to

construct dynamical systems whose transfer function (equation 6) depends on

H(s), κ, and kak2. As we used only a single spiking neuron, the accuracy of

these systems rely solely on the accuracy of the PES implementation, the model

of H(s), and the constancy of (a∗h)(t) in practice (i.e., given spiking activity).

Although we have analyzed the continuous-time setting, the same proof tech-

nique can be applied to the discrete-time domain by use of the Z-transform.

Likewise, although we have assumed xis a constant, we can apply a “separa-

tion of timescales” argument (i.e., assuming x(t) changes on a slower timescale

than f(t)) to carry this same analysis over to dynamic x(t). By equation 10, this

analysis holds approximately for x(t) with frequencies − 1

4πτ Hz, by applying

a time-varying ﬁlter to d(t) that depends on the current activity vector.

In conclusion, we have extended our previous analysis of PES to include

linearly ﬁltered feedback and a dynamic reference signal. This fully characterizes

the rule in the context of NEF networks representing a constant value, as a

transfer function from the reference signal to the error signal. This transfer

function may then be readily analyzed and exploited using linear systems theory.

This demonstrates a more general principle of recurrently coupling available

dynamical primitives in biological models (here, a PSC that is integrated by an

online learning rule) to improve network-level computations.

Acknowledgements

We thank Terrence C. Stewart for inspiring this work, in part by providing his

perspective on the PES rule applied to adaptive control in Nengo (DeWolf et al.,

2016), at the 2017 Telluride Neuromorphic Cognition Engineering Workshop.

6

Figure 4: Visualizing the poles of F(s) (equation 10) by sweeping κ > 0 (while

kak2and τ= 0.1 s remain ﬁxed). Arrows follow the direction of increasing κ.

When φ≤1

4, the dynamics of PES are a double-exponential. The learning rule

becomes an alpha ﬁlter when the two poles collide: φ=1

4⇐⇒ s=−1

2τ

(marked by a solid circle). When φ > 1

4, the weight changes become oscillatory

(due to complex poles). As κincreases, the oscillatory frequency, ω, scales as

O(√κ). As κdecreases, the ﬁrst pole converges to s=−1

τ(marked by a solid

x) while the second pole cancels the zero at s= 0.

7

References

Trevor Bekolay, Carter Kolbeck, and Chris Eliasmith. Simultaneous unsuper-

vised and supervised learning of cognitive functions in biologically plausible

spiking neural networks. In 35th Annual Conference of the Cognitive Science

Society, pages 169–174. Cognitive Science Society, 2013.

Trevor Bekolay, James Bergstra, Eric Hunsberger, Travis DeWolf, Terrence C

Stewart, Daniel Rasmussen, Xuan Choo, Aaron Russell Voelker, and Chris

Eliasmith. Nengo: A python tool for building large-scale functional brain

models. Frontiers in Neuroinformatics, 7(48), 2014. ISSN 1662-5196. doi:

10.3389/fninf.2013.00048.

Travis DeWolf, Terrence C Stewart, Jean-Jacques Slotine, and Chris Elia-

smith. A spiking neural model of adaptive arm control. Proceedings of

the Royal Society B, 283(48), 2016. doi: 10.1098/rspb.2016.2134. URL

http://dx.doi.org/10.1098/rspb.2016.2134.

Chris Eliasmith and Charles H. Anderson. Neural engineering: Computation,

representation, and dynamics in neurobiological systems. MIT Press, Cam-

bridge, MA, 2003.

Daniel Rasmussen, Aaron R. Voelker, and Chris Eliasmith. A neu-

ral model of hierarchical reinforcement learning. PLoS ONE,

12(7):1–39, 2017. doi: 10.1371/journal.pone.0180234. URL

https://doi.org/10.1371/journal.pone.0180234.

Aaron R. Voelker. A solution to the dynamics of the Prescribed Error Sensi-

tivity learning rule. Technical report, Centre for Theoretical Neuroscience,

Waterloo, ON, 10 2015.

Aaron R. Voelker. Nengolib – Additional extensions and tools for modelling

dynamical systems in Nengo. https://github.com/arvoelke/nengolib/,

2017. Accessed: 2017-08-12.

Hank Zumbahlen et al. Linear circuit design handbook. Newnes, 2011.

8