Page 1
Proceedings of IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2629, 2002
CONTINUOUS NORMALIZED CONVOLUTION
Kenneth Andersson and Hans Knutsson
Medical Informatics, Dept. of Biomedical Engineering
Link¨ oping University,
http://www.ami.imt.liu.se
ABSTRACT
The problem of signal estimation for sparsely and irregu
larly sampled signals is dealt with using continuous normal
ized convolution. Image values on realvalued positions are
estimated using integration of signals and certainties over a
neighbourhood employing a local model of both the signal
and the used discrete filters. The result of the approach is
that an output sample close to signals with high certainty is
interpolated using a small neighbourhood. An output sam
ple close to signals with low certainty is spatially predicted
from signals in a large neighbourhood.
1. INTRODUCTION
Continuous normalized convolution, CNC, is an extension
of normalized convolution [1] for representation of irregu
larly sampled signals at arbitrary positions, see Figure 1. An
example of such a signal is a motion compensated predic
tion with subpixel accuracy, see the example in section 4.1.
Related material to this approach is part of a technical report
[2].
1
3
5
x
x
x
x
x
x
xx
xx
xx
x
xx
o
o
o
o
o
o
o
oo
o
o
o
o
o
x
1
2
435
y
Figure 1: Sampling points of the signals and the certain
ties (o). Desired output sampling pattern (x), here at the
sampling of the top field for an interlaced frame, but can be
arbitrary sampled. An example of signal is a motion com
pensated prediction.
This work was supported by the Swedish Foundation for Strategic Re
search (SSF) and the program Visual Information Technology (VISIT).
2. NORMALIZED CONVOLUTION
Normalized convolution is a general framework for estima
tion of a local model for representation of a signal. The
method takes uncertainties in the signal values into account
and at the same time allows spatial localization of possi
bly unlimited analysis functions. The method was first pre
sented by Knutsson and Westin [1] using tensor algebra.
The paper also points out that the method can be seen as
the solution to a shiftvariant weighted least square prob
lem. Normalized convolution has also been presented in
Westin [3] and in Farneb¨ ack [4]. Normalized convolution is
also related to an approach for surface interpolation [5].
Here is a short description of normalized convolution
seen as a weighted least square problem. Let s denote the
neighbourhood of a given signal. A local model of the sig
nal can be found using a weighted sum of basis functions
bi. The basis functions can be stored in a matrix B
B =
?



b1

b2
..
bm
?
.
(1)
To localize the basis functions to the center of the neigh
bourhood an applicability is used. The applicability a is
represented by nonnegative real numbers. To use signal
values in accordance to how good or certain they are each
signal value has a corresponding certainty value given by a
nonnegative real number. Let c denote the certainty of the
signal values in s. Missing data can be handled by using a
zero certainty at such positions.
How to combine the basis functions to represent the sig
nal is determined from a weighted least square problem
arg min
r
?????WaWc(Br − s)
?????,
(2)
where Wa= diag(a) and Wc= diag(c). The solution to
Eq. 2 can be written as
r = (B∗WaWcB)−1B∗WaWcs.
(3)
where B∗is the conjugate transpose of B. Our signal rep
resented in the local basis can now be found as
ˆ s = Br.
(4)
1
Page 2
Proceedings of IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2629, 2002
A neighbourhood with many values missing might result in
anoninvertiblematrix. Suchoccasions canbehandled with
use of a constant signal model, i.e. normalized averaging,
and/or usage of a larger neighbourhood.
3. CONTINUOUS NORMALIZED CONVOLUTION
To use irregularly sampled input signals for the estimation
of the signal at arbitrary realvalued positions, we need a
continuous representation of the basis functions and the ap
plicability function.
Ifmathematicalexpressionstogeneratetheneededfunc
tions exists they can be used directly. However it is impor
tant to note that CNC can be used even if no such expression
exists. CNC is based on a computational efficient way to
compute the basis functions and the applicability function
in the sampling points. The sampling is specific for each
neighbourhood which means that forestimationof each out
put sample we have to compute the function values at the
specific input sampling points of that neighbourhood.
The algorithm is described for the 1D and 2D case us
ing only one constant base function, i.e. interpolation by
normalized averaging. The applicability function is here de
noted f. The method can however be extended to arbitrary
signal dimensions and several basis functions.
3.1. Generation of a continuous filter
The irregularly sampled signal has to be aligned on a ref
erence regular output sampling, denoted as reference grid.
The convolution of the signal with a discrete filter should
be performed on this reference grid. The filter’s value on an
irregularly sampled point is approximated using a first or
der spline for each filter coefficient of a discrete filter as in
Eq. 5.
The discrete filter is sampled inbetween the reference
grid, denoted as filter grid. This is because the discrete filter
should have even length and it then has its center inbetween
the two central filter coefficients. A filter of even length
is required to be able to control the filter coefficient in the
center of a symmetric filter with the derivative. A discrete
symmetric filter of odd length would have zero derivative at
the center which means that the center coefficient can not
be controlled.
˜fjqm(k) = f(k) +∂f(k)
∂x
dxjqm(m − k),
(5)
where dxjqm(m − k) is the distance between the position
of filter coefficient f(k) and the j:th input sample point at
reference output sample m − k combined with the devia
tion between the q:th output sample point and the reference
output sampling point m. This is illustrated in Figure 2.
For continuity it is required that the discrete filter and
the derivative of the filter are estimated from the same con
tinuous filter positions. Assume that we want to approxi
mate a continuous Gaussian filter
g(x) = exp(−x2
2σ2),
(6)
jqm
f (k)
f(k)
dxjqm(m−k)
~
m−k
Figure 2: Illustration of how the sampling effects the filter
value˜fjqm(k). The filter is centered on the desired output
sample m.
where x ∈ R and σ is the standard deviation, in irregularly
sampled points. The discrete filters can then be estimated as
f(k)=
f1(k) + f2(k)
2
f1(k) − f2(k)
T
(7)
∂f(k)
∂x
=
,
(8)
where, for N divisible by 2,
T = 1,k = 0..N − 1
f1(k) = g(kT + α −T
f2(k) = g(kT + α +T
2)
2)
α = −N
2+T
2
A continuous approximation of a discrete filter is illustrated
in Figure 3.
3
0
0.5
1
−5 −4 −3−2−101245
3
0
0.5
1
−5 −4−3 −2 −101245
Figure 3: Top: Discrete filter values on the reference grid,
using a Gaussian with σ = 1.5. Bottom: Filter values at
realvalued positions using the approximation (solid). Dis
crete filter values on the filter grid (square).
Using knowledge of a signal vector s and a certainty
vectorc, bothatthesamebutarbitrarycontinuouspositions,
estimates of the signal value on an output sample is calcu
2
Page 3
Proceedings of IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2629, 2002
lated as
ˆ sq(m) =
?
k
?
?
j˜fjqm(k)cj(m − k)sj(m − k)
?
kj˜fjqm(k)cj(m − k)
,
(9)
where the approximated filter coefficient˜fjqm(k) is for the
part of the convolution that involves the j:th input sample at
the reference output sample m − k for the estimation of the
q:th output sample at the reference output sample m as in
Eq. 5.
3.2. CNC for a 2D signal
For the 2D case can the filter’s value on an irregularly sam
pled point be approximated as
˜fjmn(k,l) = f(k,l) +∂f(k,l)
∂x
dxjmn(m − k,n − l)+
∂f(k,l)
∂y
∂f(k,l)
∂x∂y
dyjmn(m − k,n − l),
dyjmn(m − k,n − l)+
dxjmn(m − k,n − l)·
(10)
where dxjmn(m−k,n−l) and dyjmn(m−k,n−l) are the
distances between the position of filter coefficient f(k,l)
and the j:th input sample point at the reference grid point
(m − k,n − l).
An illustration of the continuous filter approximation is
shown in Figure 4.
x .
.
.
.
.
x .
.
.
.
.
x .
.
.
.
.
x .
.
.
.
.
.
.
.
.
.
20
0
20
−20
0
20
0
0.5
1
Figure 4: Approximation of a Gaussian filter using discrete
filters. Sampling points of the signals in the neighbourhood
of the filter (?). Filter centered at the desired output position
(x). Top: f. 2:nd:
Approximated filter.
∂f
∂x. 3:rd:
∂f
∂y. 4:th:
∂f
∂x∂y. 5:th and 6th:
4. EXPERIMENTAL RESULTS
The approximation of a continuous filter gives very similar
results as using a ’continuous’ filter, however the compu
tational time is about twice as fast as for using a Gaussian
filterwhen bothapproaches arecompiled inC.Further com
putational savings can be achieved if the CNC is made sep
arable.
2D examples of using CNC on irregular and sparse sam
pling is found in Figure 5. The irregular sampling is gener
ated as a resampling of the image at +/ 0.5 random devi
ation from the reference sampling grid using bicubic inter
polation. The sparse sampling is due to randomly assigning
the samples the certainty 1 and 0, respectively. The spa
tial resolution of the original image is almost kept using
CNC on the irregularly and sparsely sampled image using
a sharp filter as in Eq. 11. This can be explained with the
CNC working as a linear interpolator, omitting the neigh
bourhood, when the certainty is 1 in an image point and us
ing the neighbourhood of an image point when the certainty
of the image point is 0. A wider filter always smoothes an
image point if there exist neighbouring points with nonzero
certainty. This gives a too smooth result in this case but can
be advantageous to use for noisy images.
g(r) =
1
(r + 0.25)3
(11)
where r is the distance in pixel from the center pixel.
Figure 5: Top left: Part of Lenna, both irregularly and
sparsely sampled (60 %). Top right: CNC using a Gaus
sian filter (applicability) with σ = 1. Bottom: CNC using
applicability function as in Eq. 11.
4.1. Backward prediction
An application of the CNC is video coding using a dense
motion field for backward prediction [6]. In backward video
coding the motion is estimated at the decoder. Based on the
estimated motion vt−1and decoded image It−1we want
to predict the current image It. Assuming that the motion is
predictable in time, this can be solved in two different ways:
3
Page 4
Proceedings of IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland, August 2629, 2002
• Direct prediction of Itusing vt−1and It−1.
ˆIt(x) = It−1(x + vt−1)
(12)
• Prediction in two steps, first prediction of vtusing
vt−1
ˆ vt(x) = vt−1(x + vt−1)
and then prediction of Itusing ˆ vtand It−1.
ˆIt(x) = It−1(x − ˆ vt)
(13)
(14)
Some potential problems with backward prediction in this
case are:
1. Irregular sampling. The positions of the predicted
signals and the sampling pattern of the desired out
put are not aligned. There can also be ’holes’ at the
desired output positions, because no prediction end
up there. An example is uncovered regions due to
motion.
2. Multiple motions. Many predictions can end up in the
same output neighbourhood, i.e. occlusion. A sim
ple example is a moving object on a stationary back
ground.
The problems can be solved with use of CNC. The effects
of using CNC on interlaced data, based on Eq. 12, with two
different certainty fields are shown in Figure 6. It should be
noted that the images at t − 1 and t have different vertical
positions due to the interlacing. The prediction for the case
of multiple motions is a certainty weighted average of the
involved gray values. Positions with ’holes’ are spatially
predicted using the neighbourhood.
Stateofthe art methods for interpolation of irregularly
andsparselysampleddatagenerallylacktheuseofcertainty
of the data. Therefore, following comparison is with equal
certainty for all the data. We use the motion field and the
image of frame 9 of the well known Yosemite sequence to
predict frame 10, according to Eq. 12. We make the com
parison with trianglebased interpolation (griddata.m, Mat
lab version 6.0). The results both regarding motion com
pensated prediction error and computational time are shown
in Table 1. The optimal interpolator minimizing the mean
squared prediction error is kriging [7]. Kriging is a very
computational intensive approach, based on a model of the
covariance of the data, and is not considered here.
Table 1: Motion compensated prediction of frame 10 of the
Yosemite sequence, using Eq. 12.
methodtime [s]prediction error [RMS]
nearest
4.7
linear
4.7
cubic
21
CNC
0.33
8.87
6.75
6.43
6.54
Figure 6: CNC for motion compensated prediction using a
synthetic interlaced test sequence with known motion field.
Top left: Image at time t − 1. Top right: Image at time t.
Middle left: Ideal motion certainty of the ideal motion field
at time t−1. Certainty is one everywhere but at the position
of the moving objects at time t. Middle right: Prediction
with ideal motion certainty. Bottom left: Constant motion
certaintyoftheidealmotionfieldattimet−1. Bottomright:
Prediction with constant certainty. Moving objects will end
up at a stationary background, both with high certainty. The
result is a combination of the background and the moving
objects. The convolutions are performed with the filter in
Eq. 11.
5. CONCLUSIONS
We have presented a method for interpolation of irregularly
and sparsely sampled data considering data of possibly dif
ferent reliability. The method is fast and accurate, and can
be used for motion compensated prediction in video coding.
6. REFERENCES
[1] H. Knutsson and CF. Westin, “Normalized and differential convolu
tion: Methods for interpolation and filtering of incomplete and uncer
tain data,” in Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, June 1993, pp. 515–523.
[2] K. Andersson, “Quality and Motion Estimation for Image Sequence
Coding,”Lic. Thesis LiUTekLic2002:01, Link¨ oping University,
Sweden, SE581 85 Link¨ oping, Sweden, February 2002, Thesis No.
928, ISBN 9173732648.
[3] CF. Westin, A Tensor Framework for Multidimensional Signal Pro
cessing,Ph.D. thesis, Link¨ oping University, Sweden, SE581 83
Link¨ oping, Sweden, 1994, Dissertation No 348, ISBN9178714214.
[4] G. Farneb¨ ack, “Spatial domain methods for orientation and velocity
estimation,” Lic. Thesis LiUTekLic1999:13, Dept. EE, Link¨ oping
University, SE581 83 Link¨ oping, Sweden, March 1999, Thesis No.
755, ISBN 9172194413.
[5] P. J. Burt, “Moment images, polynomial fit filters and the problem
of surface interpolation,” in Proc. of Computer Vision and Pattern
Recognition, Ann Arbor. 1988, Computer Society Press.
[6] K. Andersson, P. Johansson, R. Forcheimer, and H. Knutsson,
“Backwardforward motion compensated prediction,” in Advanced
Concepts for Intelligent Vision Systems (ACIVS’02), Ghent, Belgium,
September 2002, To appear.
[7] G. Matheron, “Principles of geostatistics,” Econom. Geology, vol. 58,
pp. 1246–1266, 1963.
4
Download fulltext