Content uploaded by Jonas Vibell
Author content
All content in this area was uploaded by Jonas Vibell on Oct 18, 2021
Content may be subject to copyright.
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/342560120
Fast Variational Alignment of non-flat 1D Displacements for Applications in
Neuroimaging
Preprint · June 2020
DOI: 10.1101/2020.06.27.151522
CITATIONS
0
READS
39
6 authors, including:
Some of the authors of this publication are also working on these related projects:
An innovative multimodal approach for the study of the barrel cortex through VSD imaging and high-resolution LFP recordings View project
GRACE: hiGh-Resolution imAging of the barrel CortEx through VSD and LFP recordings View project
Philipp Flotho
Universität des Saarlandes
17 PUBLICATIONS10 CITATIONS
SEE PROFILE
David Thinnes
Universität des Saarlandes
4 PUBLICATIONS0 CITATIONS
SEE PROFILE
Bernd Kuhn
Okinawa Institute of Science and Technology
54 PUBLICATIONS772 CITATIONS
SEE PROFILE
Christopher Joel Roome
Okinawa Institute of Science and Technology
23 PUBLICATIONS152 CITATIONS
SEE PROFILE
All content following this page was uploaded by Philipp Flotho on 17 July 2020.
The user has requested enhancement of the downloaded file.
Fast Variational Alignment of non-flat
1D Displacements for Applications in Neuroimaging
Philipp Flotho1,2, David Thinnes1,3, Bernd Kuhn4,
Christopher J. Roome4, Jonas F. Vibell3and Daniel J. Strauss1
1Systems Neuroscience and Neurotechnology Unit, Neurocenter, Faculty of Medicine, Saarland
University, Homburg/Saar, Germany
2Summer Program, Japan Society for the Promotion of Science (JSPS), Tokyo, Japan
3Department of Psychology, University of Hawai’i at M¯anoa.
4Optical Neuroimaging Unit, Okinawa Institute of Science and Technology Graduate University,
Tancha, Onna-son, Kunigami, Okinawa, Japan
Abstract
Background In the context of signal analysis and pattern matching, align-
ment of 1D signals for the comparison of signal morphologies is an important pro-
blem. For image processing and computer vision, 2D optical flow (OF) methods
find wide application for motion analysis and image registration and variational
OF methods have been continuously improved over the past decades.
New Method We propose a variational method for the alignment and displace-
ment estimation of 1D signals. We pose the estimation of non-flat displacements
as an optimization problem with a similarity and smoothness term similar to va-
riational OF estimation. To this end, we can make use of efficient optimization
strategies that allow real-time applications on consumer grade hardware.
Results We apply our method to two applications from functional neuroimaging:
The alignment of 2-photon imaging line scan recordings and the denoising of evo-
ked and event-related potentials in single trial matrices. We can report state of
the art results in terms of alignment quality and computing speeds.
Existing Methods Existing methods for 1D alignment target mostly constant
displacements, do not allow native subsample precision or precise control over
regularization or are slower than the proposed method.
Conclusions Our method is implemented as a MATLAB toolbox and is online
available. It is suitable for 1D alignment problems, where high accuracy and high
speed is needed and non-constant displacements occur.
Keywords— 1D alignment, 1D displacement, Variational methods, Evoked potentials,
Event related potentials, Line scan, Two-photon microscopy, Confocal microscopy
1 Introduction
Optical flow (OF) estimation methods form an important family of image processing
algorithms with applications in many areas where motion analysis, tracking or regis-
tration is needed. In the context of OF estimation of 2D natural images and image
1
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
registration, variational methods form a powerful group of well-studied methods that
perform with very high accuracy in the presence of small displacements [1]. Up until
today, they find applications in state of the art OF algorithms, where they can be used
for the refinement of small displacements [2] or for the generation of ground truth dis-
placement fields [3], [4]. They are able to estimate non-parametric, dense and smooth
displacement fields with subpixel accuracy [1].
The displacement field is calculated by minimizing an energy functional with a data
term as the error measure and a regularization term that enforces smooth solutions.
The data term and regularizer are modeled with respect to the data characteristics and
the expected type of motion.
Most commonly, variational OF methods are solved within an Euler-Lagrange frame-
work, where a linearized approximation of the problem is solved with a coarse-to-fine
strategy. This yields a sequence of convex programming problems, that can be optimi-
zed efficiently with parallel implementations on graphics processors [5], [6], [7].
Imaging areas, that require very high frame rates or spectral rather than spatial reso-
lutions are suitable for the application of line scan sensors. Line scans are sequences
of 1D recordings typically of a scene under static conditions or of a scene with camera
or object movement orthogonal to the spatial dimension. Among others, they find
application in industrial quality inspection [8] and biomedical imaging [9]. There are
many sources for movement artifacts in line scan recordings: In the context of moving
objects, poor synchronization, or jitter in the object movement results in non-uniform
sampling. Brosch et al. [8] have proposed a variational formulation to estimate the
warping function in the context of movement jitter for light field line scan data to
reconstruct a recording that is sampled at even intervals.
Movements parallel to the spatial dimension result in shifts of the signal, that can
often be approximated by constant displacements. Depending on the camera and lens
/ illumination setup, movements towards the camera can have additional effects such
as linear displacements due to zooming or changes in signal morphology.
Within the scope of single trial analysis of evoked and event-related potentials (EP and
ERP) in electroencephalography (EEG) recordings, event locked responses are analyzed
[10]. The average response may have a high signal to noise ratio (SNR) but also
a lower resolution of the signal. From the average signal it is also not possible to
analyze temporal variations across trials. Popular approaches to increase SNR in ERP
recordings for single trial analysis are often motivated by image processing methods
[11]. This is because the single trial matrix representation of ERP recordings resembles
the 2D representation of line scan data: Each line is a measurement with high temporal
resolution of a stimulus locked event.
To remove constant displacements in 1D data, methods that maximize the cross-
correlation with respect to a reference profile find wide application. Methods to compare
two arbitrary 1D signals are often based on dynamic time warping (DTW) [12]. The
idea of DTW is to repeat individual samples of both signals to minimize a distance me-
tric. This way, distinct features of the two signals can be directly compared. Nielsen et
al. [13] introduced a method where they extend DTW with piecewise cross-correlation.
Their correlation optimized warping approach is applied for the alignment of chroma-
tographic data [14], [15].
2
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
In their preliminary work, Deriso et al. [16] have presented a variational formulation of
the DTW problem. In their method they optimize the complete non-parametric warping
function. They solve the non-convex optimization problem with dynamic programming
in a multigrid framework.
In this work, we investigate non-parametric displacement estimation for 1D data and
describe a variational method for non-parametric, dense displacement estimation of 1D
signals, which is motivated by variational OF estimation.
By exploiting the linear motion assumption within a coarse-to-fine strategy, our ap-
proach solves a sequence of convex problems. We demonstrate the performance of our
algorithm on two problems from neuroimaging with alignment of optical imaging line
scan recordings and the compensation of latency jitter in ERP recordings. Our imple-
mentation is fast enough to allow real time alignments on consumer grade hardware.
2 Materials and Methods
In this section we describe our method for the alignment and displacement analysis of
1D signals. To this end, we propose a variational model and pre-processing strategy
and describe our implementation.
We work with 2D representations of multiple 1D signals which can be temporal (EP
and ERP alignment) or spatial (line scan alignment). Let mbe the length (spatial or
temporal) of the 1D signal and nbe the number of measurements. Then an individual
measurement is given by ˆ
fc(x) : Ω1→Rwith x∈Ω1= (0, m)⊂R. The 2D matrix
representation of multiple measurements is given by the function fc(x, y) : Ω2×N→R,
where (x, y)∈Ω2= (0, m)×(0, n)⊂R2are the coordinates. For multichannel recor-
dings c= 1, . . . , k ⊂Ndenotes the discrete number of the channel. The first dimension
is either spatial or temporal and goes along the 1D signals and the second dimension is
orthogonal to the signal and usually temporal and represents individual measurements.
We depict the reference profile with gc(x) : (0, m)→R. The discrete version of gis
denoted by gc
iand of fby fc
i,j where the dimension along the first dimension is sampled
uniformely and along the second can be arbitrarily sampled and explicitly defined by
the input data. By means of simplicity, we depict partial derivatives with lower case
letters xand yand use the lower case letters i, j exclusively for discretized access. Con-
tinuous access to the discrete data is realized via bicubic interpolation as recommended
by [1].
2.1 Variational Model for 1D Alignment
We formulate the problem of aligning 1D signals as a variational problem, where we
want to find displacements u(x, y) : Ω2→Rof 1D signals ˆ
fwith respect to a reference
ˆg. We assume gradient constancy between ˆ
fand ˆgas well as small gradients in u. We
penalize both assumptions with the generalized Charbonnier penalty function Ψa. Let
a > 0, then the generalized Charbonnier penalty function [1] is given by
Ψa(x) = x2+2a(1)
Following the authors in [1], we choose the penalization of the data and smoothness
terms with as, ad∈[0.45,1] ⊂R, where Ψ0.5gives the Chabonnier penalty function and
Ψ1is quadratic penalization. Ψ0.45 is slightly non-convex but has been shown to perform
3
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
well in the context of OF estimation [1]. The energy functional for 1D displacements
of single channel signals is then given by
E(u) = ZΩ
Ψadg1
x(x)−f1
x(x+u(x), y)+αΨas(ux)dxdy (2)
where the relation of the data and smoothness term is weighted by the smoothness
weight parameter α∈R>0. The displacement field uis then calculated as argmin E(u).
To achieve that, we linearize the constancy assumption into u·fc
xx +fc
x−gc
xon a Gaussian
pyramid and calculate the increments on each level.
The linearized assumption is a good approximation for small displacements and it has
been shown, that the final solution of such a scheme approximates the unlinearized con-
stancy assumption [6]. Gradient constancy with subquadratic penalization is invariant
under global additive changes of the signal and robust with respect to outliers. We nor-
malize the input data, which can additionally compensate for multiplicative changes. In
the context of variational multimodal image registration, there exist intensity similarity
measures [17], that allow affine (cross correlation), functional (correlation ratio), and
arbitrary (mutual information) dependence of the two signals. Cross correlation is a
commonly used metric for 1D signal analysis. Additionally to the gradient constancy
assumption, we implement a cross correlation data term as presented in [17] with isotro-
pic smoothness term, however the gradient descent scheme from [17] has much slower
convergence than our gradient constancy model.
The variational formulation allows a straightforward extension to data with multiple
channels. Given data fkwith k channels, we can extend our data term with separate
penalization on each channel as in
k
X
c=1
wc(x)·Ψac
data (gc
x(x)−fc
x(x+u, t)) (3)
The weight term w(x) : Ω →Ris multiplied pointwise to each channel and can be used
to integrate knowledge on the information content / SNR per channel and to change
the weight of each channel or datapoint.
To our knowledge, there exist no methods for the alignment of 1D signals that natively
support the integration of multiple channels to calculate displacements.
Pre-Processing
In the following, we apply 1D Gaussian convolution with a fixed standard deviation
ˆσalong the first dimension before calculating the displacements. This reduces high
frequency noise and makes the signals differentiable. We normalize the input data to
compensate for different scaling.
On top of that, due to potentially low SNR of the input data, we need an additional
pre-processing strategy. We propose a novel, anisotropic kernel for the processing of 2D
single trial matrices (see figure 1).
4
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
We choose a filter that smooths orthogonally to the discrete time sequences and line
scans, while allowing displacements of the individual measurements. Our kernel is a 2D
Gaussian function Gσwith zero mean and standard deviation σ= (σ1, σ2)>
Kσ(x, y) = w(x, y)·Gσ(x, y)
|w(x, y)·Gσ(x, y)|(4)
that is weighted by the angular distance to the 1D signals
w(x, y) =
1 if x=y= 0
y
√x2+y2
else. (5)
For the rest of the paper we use equal standard deviation σ1=σ2.
A
B
C
D
Figure 1: The anisotropic kernel (A) defined by equation 4 which we use for pre–
smoothing: A normalized Gaussian kernel Kσwith zero mean and standard deviation
σthat is weighted by the spatial angle. Applied on an ERP map (B), we see that traces
across trials are accentuated (D) when comparing with Gaussian convolution (C). For
(C) and (D) we used a standard deviation of σ= 10.
In the context of real time alignment of data with low SNR, at each point in time we
have only access to the past measurements. In this case, we can use half the kernel and
weight Gσwith
˜w(x, y) =
0 if y > 0
1 if x=y= 0
y
√x2+y2
else.
(6)
2.2 Implementation
We discretize the Euler-Lagrange equations of the linearized functional following Brox
et al. [6] and solve it with an iterative solver with lagged diffusivity on a Gaussian
scale space. It has been empirically shown, that median filtering of 2D flow increments
improves the estimation accuracy [1]. Accordingly, at each pyramid level, we apply a
1D median filter to the estimated displacement increments. For the results presented in
this paper, we choose a filtersize of 5. We use cubic interpolation for the compensation
5
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
of displacement increments as well as for the final alignment. For multimodal data
terms, we follow Hermosillo et al. [17] and implement a gradient descent algorithm on
a Gaussian scale space with cross-correlation in the data term.
Our model requires a common reference and computes displacements and alignment
for all measurements with respect to this reference. Analysis of the displacements of
subsequent measurements can be realized from the displacement field without explicitly
comparing two signals. However, for certain data, registration to a fixed reference can
result in large displacements. Large displacements can violate the linearized constancy
assumption on the highest level of the pyramid and hinder the estimation of the joint
pdf for the multimodal data terms.
For this reason, we assume that we can find a row order with gradually increasing
displacement magnitudes, meaning for all i∈1, . . . , m we want to find the bijective
function π(x) : N→Nsuch that the displacement magnitudes are gradually increasing
ui,π(1)
≤
ui,π(j)
≤
ui,π(n)
with respect to the reference profile. This mapping π
integrates experiment specific a-priori knowledge into the alignment calculation or can
be estimated heuristically. For example, we expect line scan data with high temporal
resolution to show small inter line movements making the identity a reasonable choice
for π.
In terms of implementation, for all i∈1, . . . , m we initialize ui,π(1) with a constant
displacement field and then calculate the displacements row by row, where we initialize
the highest pyramid level of row πi,j with the previous result ui,π(j−1). This strategy
results in the smallest possible displacement increments, as long as we can provide some
good π.
The parameters of our method are on the one hand the solver specific parameters such
as warping depth, number of iterations, update frequency of the non-linear term and the
warping stepsize ηwhich can be increased for better accuracy. On the other hand, we
have model specific parameters which modify the objective function: The smoothness
parameter αdefines the weight of the smoothness term with respect to the data term.
The penalizer exponents of the generalized Charbonnier function 0.45 ≤ad, as≤1
control how deviations from the model assumptions are penalized: Large values of as
enforce smooth displacements and low values allow discontinuities. Smaller values of
admake the data term more robust with respect to outliers and noise.
We have implemened our method in MATLAB with the MATLAB Image Processing
Toolbox and C++. The warping loop is written in MATLAB and on grounds of com-
puting speed, the iterative solver is written in C++ and compiled into a mex function.
3 Results
In this section, we report the performance of our alignment approach on EEG single trial
matrices and functional 2-photon imaging line scan recordings. We use real physiological
as well as synthetic data. For the sake of visualization, we report the results for line scans
(section 3.2) always transposed, meaning the displacements have been estimated along
the y-axis. For quantitative analysis, we compare variational 1D alignment with the
removal of constant shifts by maximizing cross correlation (no subsample refinement)
and correlation optimized warping as implemented in [15].
6
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
3.1 Synthetic Experiments
We use two different synthetic datasets: The first one simulates a line scan recording
with a non-constant motion artifact, that is difficult to remove. We use it to demonstrate
the compensation by our method of motion artifacts that cannot be parametrized by
constant shifts. To investigate overfitting and clustering of signal features as well as to
benchmark computing speed, we use random 2D matrices.
Line Scan with Diverging Structure
For our first synthetic data experiment, we simulate a line scan recording that has been
contaminated by non-constant motion noise. The data consists of 800 line measure-
ments with 150 samples each. The average of the ground truth displacement field ˆuis
zero, which means the motion cannot be parametrized by constant shifts. The signal
features converge and diverge in the center as seen by changes in ˆux(see figure 2).
For the evaluation, we use different values of asand choose parameters with less iterati-
ons for a fast estimation (see table 1). We choose the average profile as reference, which
in the case of the variational method is refined one time. As expected, the dataset
cannot be denoised with constant displacements (see figure 3 and table 1). Constant
alignment with maximization of the cross correlation does not improve the signal. Quan-
titatively, correlation optimized warping performs even worse than constant alignment,
which might be due to the high amount of noise in our data set. Because of the discon-
tinuities in the displacement fields, a low choice of asperforms better than quadratic
penalization of the smoothness.
Alignment of Random Noise with structured Reference
We observe that our method tends to overfit noisy data if we do not apply pre-
smoothing. This results in clustering of vertical traces (see figure 4 and 5), that is,
if the SNR is low in a single line, features in the noise are aligned with respect to featu-
res in the reference. When we provide a structured pattern such as a sine as reference,
overfitting can reproduce the pattern in the signal average after alignment (see figure
4). Regarding the average of an aligned matrix with the average profile as reference,
overfitting seems to accentuate features in the average signal after alignment.
Method STD PSNR Time (ms)
Correlation optimized warping 0.048 26.09 58.5
Raw signal 0.046 26.51 –
Constant displacement cross correlation 0.046 26.52 0.1
Variational fast as= 0.45 0.034 29.25 1.2
Variational as= 1 0.033 29.43 7.8
Variational as= 0.45 0.033 29.55 9.3
Table 1: Performance of our method in comparison with cross correlation based con-
stant alignment (no subsample refinement) and correlation optimized warping. Smaller
values for the average standard deviation (STD) and larger values for the average peak
signal to noise ratio (PSNR) are better. Variational fast uses less iterations, smaller η
and no refinement of the reference.
7
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
Synthetic data Registered data
Ground truth displacement
-10
-5
0
5
10
Ground truth divergence
0
20
40
60
80
Displacement, as = 1
-5
0
5
Divergence, as = 1
-2
0
2
4
Displacement, as = 0.45
100 200 300 400 500 600 700 800
Number of samples
-10
-5
0
5
Divergence, as = 0.45
100 200 300 400 500 600 700 800
Number of samples
0
10
20
30
Figure 2: Synthetic line scan recording with converging and diverging features (top left)
which can be almost completely compensated by our method (top right). The bottom
left plots depict the ground truth displacement map and the estimated displacement
map with as= 1 and as= 0.45. On the right is the first derivative in along the spatial
dimension of the displacement map. The ground truth displacement map contains
discontinuities which is why as= 0.45 yields a better estimation than as= 1.
In our method, overfitting is regularized with the size of our filter from equation 4 across
trials and the size of 1D filter along trials. Additionally, the subquadratic data term
reduces the influence of outliers. The underlying assumption for the filtering across trials
is a self-similarity in terms of a row coherence with respect to the provided permutation
mapping π(j). Our anisotropic filter enhances alignment relevant regularities, while
suppressing fluctuations that can be attributed to noise. Therefore, depending on the
application, other denoising methods can be applied as well (e.g. compare [18], [19]).
It is beyond the scope of this work to present a universal solution to avoid overfitting
with this method. However, a general good practice is to adjust parameters, based on
SNR. Problems where we generally deal with higher SNRs, or where temporal filtering
can increase the SNR produce good results without much inter-trial filtering (see figure
8). Problems with inherently low signal to noise ratio are prone to overfitting and
require pre-filtering, e.g., compare [18] and figure 10.
We recommend to filter spatio-temporally such that the average SNR of signal and
reference / average is positive. This way, the alignment only compensates visible dis-
placements in the data. Also traces that appear after alignment without any visible
correlates in the input data should be interpreted with caution and according to the
respective application.
8
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
Raw signal Constant CC DTW Variational, as = 1 Variational, as = 0.45
Figure 3: Qualitative registration performance on the first synthetic line scan recording.
Left to right is the raw signal, removal of constant shifts with cross correlation (Constant
CC), correlation optimized warping (DTW) and variational alignment with different
parameter choices as. Due to absence of constant shifts, constant CC cannot improve
the input data. DTW contains many outliers and misalignments while variational
alignment with as= 0.45 can almost perfectly align the sequence.
Real-Time Applications
Our method is fast enough to run in real-time on consumer grade hardware for diffe-
rent use cases. Given a varying sample size n, we benchmark the processing time of
our MATLAB implementation on 512 ×nmatrices as well as on single lines with n
samples and with a fixed number of 1000 samples for varying warping stepsizes ηwith
20 iterations (see figure 6). The alignment of the matrices took less than a few seconds
with the average time per line in the order of milliseconds and no more than 120 ms
for an isolated registration of a single line. The machine used for the estimation was a
desktop computer with an AMD Ryzen 3900X processor (3.8 GHz, 12 cores, 24 logical
processors) and 64 GB of memory. The alignment of an isolated line can be applied
in the context of EPs and ERPs (see section 3.3), where we experiment with inherent
delays of more than 100 ms in-between events. The registration of a full matrix might
be applied to the online alignment of line scan data with block-wise acquisition and
processing / display of line scan data.
For the alignment of short segments with η= 0.9 or for 1000 samples with small warping
stepsize, our method performs with up to 1000 Hz. When aligning ERPs (see section
3.3), η= 0.9 with 1000 samples allows the compensation of a 2 s segment in 4.5 ms
given enough room for other processing steps. In their preliminary paper, Deriso et
al. [16] report an average speed of 0.25 s for a 1000 sample segment. Even though
we cannot directly compare the results due to different hardware architectures, the
sequential version of our method can perform around two magnitudes faster on average
for multiple lines and depending on the choice of ηcan be one magnitude faster for the
alignment of an isolated line.
9
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
Figure 4: Alignment of random signals: We have aligned a matrix containing random,
temporally lowpass filtered values with a sine wave as reference. It demonstrates the
importance of integrating temporal coherence. When pre-filtering with σ= 4, the
displacement field is relatively smooth across trials. With σ= 0.5 our method overfits
and can reconstruct a reference that is not present in the data.
Figure 5: Alignment of random signals: When using the average line profile as refe-
rence, we observe a clustering effect when using a small filter size.
10
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
Figure 6: Alignment speed of our method per line with respect to varying number of
samples and warping stepsize. We used 20 iterations and η= 0.9 for the varying sample
size and 1000 samples for varying η. Left is the average time per line when using a
multitrial matrix as input and right is the performance for the alignment of an isolated
line.
When comparing our method with popular DTW approaches, on average our system
needs 4.6 ms for 1000 samples and default parameters in MATLAB. On our system,
the python package fastdtw takes around 229.6 ms with default parameters and 6 s
with radius of 50. We constitute the slower alignment of isolated lines mainly to the
characteristics of our MATLAB implementation: On the one hand, for multitrial input,
filtering and resizing methods benefit from parallelized implementations in MATLAB.
On the other hand, MATLAB overhead for calling external functions amortizes for lar-
ger matrices and is much more dominant for the alignment of individual lines. Also
given the fact that the type of solver we use can be implemented on graphics cards and
can easily be parallelized [7], we expect the performance of an optimized implementa-
tion of our method for alignment of a single line to be close to the reported average
performance.
3.2 Line Scan Data
Method STD PSNR Time (ms)
Raw signal 821.48 -57.58 –
Cross correlation 536.90 -54.81 0.1
Correlation optimized warping 494.46 -54.04 24.8
Variational alignment 20 iterations 470.15 -53.77 0.6
Variational alignment 100 iterations 466.97 -53.75 2.2
Table 2: Performance of our method on the 2-photon imaging line scan data (unnor-
malized) in comparison with cross correlation based constant alignment (no subsample
refinement) and correlation optimized warping. Lower values for the average standard
deviation (STD) and higher values for the average peak signal to noise ratio (PSNR)
are better.
We apply our method to a 2-photon line scan recording from Roome et al. [9] of
Purkinje neuron dendrites in the cerebellum of awake mice.
11
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
A
C
B
Figure 7: 2-photon imaging line scan recording without alignment (A), with cross
correlation alignment (B) and with variational alignment (C). The cross correlation
approach fails to remove high frequency motion. During the strong motion artifact
(5.5 s), removal of constant displacements cannot account for the converging structures.
This results in an average temporal standard deviation of 241.2 for (A), 190.6 for (B)
and 179.7 for (C). The recording is taken from Roome et al. [9] and used with the
authors permission.
12
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
The recording contains two simultaneously recorded line scan channels, the first repor-
ting calcium changes, the second voltage changes, and was recorded at 2 kHz. Imaging
artifacts occurred because of animal movement and are composed of different compo-
nents, including a slight convergence of the structures after movement onset, see figure
8 and 7 around 5.5 s. In the same way as for section 3.1, the performance of our met-
hod is compared with cross correlation based alignment of constant displacements and
correlation optimized warping.
Optical imaging recordings can contain multiple channels [9] and sometimes contain a
structural as well as a functional channel. We register on both channels simultaneously
and weigh each channel by structure content.
Figure 8: Section from the 2-photon imaging line scan recording from left to right
without alignment, with cross correlation alignment, with correlation optimized warping
alignment and with variational alignment.
For the alignment of the data in figure 7 and 8 our method took on average 2.2 ms per
line with 100 iterations and with 20 iterations only 0.6 ms. From table 2 we see that
there is not a large increase in performance for 100 compared to 20 iterations. Thus,
state of the art results for a line scan of width 270 can be achieved at a little more than
1.6 kHz, which conforms with our results from the previous section.
However, difficult motion artifacts in two-photon imaging data as seen from 5.5 s on-
wards can coincide with shifts orthogonally to the imaging plane. This way, structures
can completely disappear and concentration gradients of dyes can be misinterpreted as
functional activity. Therefore, such points in time are usually detected and discarded
systematically in evaluations. With an improvement in registration accuracy, it might
be possible to use the previously discarded data for analysis. Importantly, the impro-
ved registration accuracy does not come at the cost of much higher computation time.
Therefore, we see our method as a superior alternative to alignment approaches that
estimate constant displacements.
3.3 ERP Single Trial Analysis
Analysis of 2D matrix representations of ERP in EEG recordings has been suggested
for the analysis of trial to trial variances in critical wave morphologies, (e.g. see [11]).
Inter-trial and inter-subject variances not only distort signal and grand averages, but
also cannot be analyzed after averaging – even though they can contain different cues on
13
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
dynamics during experiments, in particular, when analyzing endogenous processes and
exogenous parameter variations. Because of the low SNR in individual trials, single trial
analysis requires denoising strategies, which, due to the possibility of 2D representation
of the data, are often motivated by image denoising techniques [11], [18].
Figure 9: Alignment of an ERP recording sorted by subject’s response time, see [11].
Using only past trials results in a similar performance as the full kernel. However, a low
level of temporal integration results in strong clustering of the signal. All ERP maps
where filtered with σ= 4 for better visualization. The estimated trace is marked in
black in the displacement maps on the right.
Knowledge of the displacement or latency jitter in ERP matrices on a per trial basis
has the potential to greatly improve ERP single trial analysis: In the context of denoi-
sing, aligning signals with respect to their morphology allows smoothing vertically to
the time dimension with linear filters. Compared to anisotropic filtering approaches,
the direction of inter-trial changes does not need to be estimated. Filtering methods
based on non-local means estimation such as [11] produce state of the art results for two
dimensional ERP denoising. However, they do not aim to recover the implicitly estima-
ted spatial relation of the self-similarity. Also, the displacement maps in our method
allow the analysis of the estimated traces (see section 3.1 and figure 9), while boosting
important features in the signal average after alignment1. The same holds true for EP
alignment: We apply our method to auditory brainstem responses (ABR) collected in
Corona-Strauss et al. [20]. We observe, that the diagnostically most relevant wave V
[21] is easier to recognize and its amplitude is enhanced after alignment as compared to
classical averaging without alignment, see figure 10. This has the potential to reduce
the required number of trials for analysis.
1For an in depth analysis of the application of our method to ERP data, we refer the reader to
Thinnes et al. [18].
14
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
A B
CD
Figure 10: Application of our method to auditory brainstem responses (ABRs) data. In
the average response, the diagnostically most relevant wave V is easier to recognize and
its amplitude is enhanced after alignment as compared to classical averaging without
alignment. However, we can also observe the clustering effects described in section 3.1
for lower stimulation amplitudes. Stimulation is done with 40 dB dB (A), 30 dB (B),
20 dB (C) and (D) is spontaneous activity. The refinement step is registration with the
average of the matrix after initial alignment as reference.
4 Discussion
In this work, we have presented a novel variational method for the alignment and
estimation of non-flat and non-parametric displacements of 1D data with applications
in neuroimaging. Our method produces state of the art results for the alignment of 1D
signals and is fast enough to enable real-time applications on consumer grade hardware.
The method is implemented as an accessible MATLAB toolbox and publicly available
on GitHub2. Our continuous formulation is motivated by 2D OF estimation and thereby
builds on a rich framework of different regularizers and constancy assumptions as well
as efficient optimization strategies. Our model uses a total variation data term and
quadratic or linear smoothness term. It natively supports weighted multichannel data.
We have demonstrated the performance of our approach on 2-photon line scan recor-
dings as well as 2D single trial matrix representations of ERPs and EPs. The alignment
of ERP matrices opens the way for improved analysis with reduced number of trials.
Due to its high accuracy and fast implementation forms an alternative to established
2https://github.com/phflot/variational_aligner
15
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
methods such as DTW and constant template matching. In future work it can to be
additionally evaluated against methods currently under development such as Deriso et
al. [16] (unpublished work).
For line scan recordings, our method is an alternative to constant displacement esti-
mation with higher accuracy, only slightly less computation time and the ability to
compensate non-flat displacements. In addition, we believe that this method could
find many further applications for the alignment of line scan data beyond the field of
neuroimaging.
5 Acknowledgements
This work was partially funded by the Japan Society for the Promotion of Science
(JSPS) with the Summer Program 2019 and partially conducted at the Optical Neu-
roimaging Unit under Prof. Bernd Kuhn at the Okinawa Institute of Science and
Technology Graduate University.
References
[1] Sun, D., Roth, S. & Black, M. J. Secrets of optical flow estimation and their
principles. In 2010 IEEE computer society conference on computer vision and
pattern recognition, 2432–2439 (IEEE, 2010).
[2] Chen, Z., Jin, H., Lin, Z., Cohen, S. & Wu, Y. Large displacement optical flow
from nearest neighbor fields. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2443–2450 (2013).
[3] Maurer, D., Stoll, M. & Bruhn, A. Order-adaptive and illumination-aware varia-
tional optical flow refinement. In BMVC (2017).
[4] Maurer, D. & Bruhn, A. Proflow: Learning to predict optical flow. arXiv preprint
arXiv:1806.00800 (2018).
[5] Sun, D., Roth, S., Lewis, J. P. & Black, M. J. Learning optical flow. In European
Conference on Computer Vision, 83–97 (Springer, 2008).
[6] Brox, T., Bruhn, A., Papenberg, N. & Weickert, J. High accuracy optical flow
estimation based on a theory for warping. In Computer Vision-ECCV 2004, 25–36
(Springer, 2004).
[7] Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A. & Weickert, J. A highly efficient
gpu implementation for variational optic flow based on the euler-lagrange frame-
work. In European Conference on Computer Vision, 372–383 (Springer, 2010).
[8] Brosch, N., ˇ
Stolc, S. & Antensteiner, D. Warping-based motion artefact com-
pensation for multi-line scan light field imaging. Electronic Imaging 2018, 273–1
(2018).
[9] Roome, C. J. & Kuhn, B. Simultaneous dendritic voltage and calcium imaging and
somatic recording from purkinje neurons in awake mice. Nature communications
9, 3388 (2018).
16
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
[10] Handy, T. C. Event-related potentials: A methods handbook (MIT press, 2005).
[11] Strauss, D. J., Teuber, T., Steidl, G. & Corona-Strauss, F. I. Exploiting the self-
similarity in erp images by nonlocal means for single-trial denoising. IEEE Trans
Neural Syst Rehabil Eng 21, 576–583 (2013).
[12] Sakoe, H. & Chiba, S. Dynamic programming algorithm optimization for spoken
word recognition. IEEE transactions on acoustics, speech, and signal processing
26, 43–49 (1978).
[13] Nielsen, N.-P. V., Carstensen, J. M. & Smedsgaard, J. Aligning of single and
multiple wavelength chromatographic profiles for chemometric data analysis using
correlation optimised warping. Journal of chromatography A 805, 17–35 (1998).
[14] Tomasi, G., Van Den Berg, F. & Andersson, C. Correlation optimized warping
and dynamic time warping as preprocessing methods for chromatographic data.
Journal of Chemometrics: A Journal of the Chemometrics Society 18, 231–241
(2004).
[15] Skov, T., van den Berg, F., Tomasi, G. & Bro, R. Automated alignment of chroma-
tographic data. Journal of Chemometrics: A Journal of the Chemometrics Society
20, 484–497 (2006).
[16] Deriso, D. & Boyd, S. A general optimization framework for dynamic time warping.
arXiv preprint arXiv:1905.12893 (2019).
[17] Hermosillo, G., Chefd’Hotel, C. & Faugeras, O. Variational methods for multi-
modal image matching. International Journal of Computer Vision 50, 329–343
(2002).
[18] Thinnes, D., Flotho, P., Corona-Strauss, F. I., Strauss, D. J. & Vibell, J. F. Com-
pensation of p300 latency jitter using fast variational 1d displacement estimation.
(in preparation) (2020).
[19] Kohl, M. C., Schebsdat, E., Schneider, E. N. & Strauss, D. J. Denoising of single-
trial event-related potentials by shrinkage and phase regularization of analytic
wavelet filterbank coefficients (accepted). In Neural Engineering (NER), 9th In-
ternational IEEE EMBS Conference on (2019).
[20] Corona-Strauss, F., Delb, W., Schick, B. & J Strauss, D. Phase stability analysis
of chirp evoked auditory brainstem responses by gabor frame operators. IEEE
transactions on neural systems and rehabilitation engineering : a publication of
the IEEE Engineering in Medicine and Biology Society 17, 530–6 (2009).
[21] Picton, T. W. Human auditory evoked potentials (Plural Publishing, 2010).
17
(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.
The copyright holder for this preprintthis version posted June 29, 2020. . https://doi.org/10.1101/2020.06.27.151522doi: bioRxiv preprint
View publication statsView publication stats
