Simultaneous Higher-Order Optical Flow Estimation and Decomposition.

Full text

Page 1

Simultaneous Higher-Order Optical Flow
Estimation and Decomposition
Jing Yuan∗
Christoph Schn¨ orr†
Gabriele Steidl‡
January 8, 2007
Abstract
We study the estimation and decomposition of optical flows from
highly non-rigid motions. To this end, recent methods from image de-
composition into structural and textural parts are combined with vari-
ational optical flow estimation. The approaches we suggest amount to
minimizing discrete convex functionals using second order cone pro-
gramming. Higher order regularization is necessary in order to ac-
curately recover important flow structure like vortices, and to incor-
porate key physically properties such as vanishing divergence, for in-
stance. For proper discretization, we apply the finite mimetic difference
method that preserves the identities fulfilled by the continuous differ-
ential operators. Numerical examples demonstrate the feasibility of
the complex approaches.
AMS Subject Classification: 68Y10, 65K10, 90C25, 76-04
Running Title: Flow estimation and decomposition
Key words: Optical flow estimation, optical flow decomposition, divergence
free flows, higher order regularization, mimetic finite difference method, con-
vex optimization, dual optimization techniques, second order cone program-
ming
1Introduction
The representation, estimation, and analysis of non-rigid motions is rele-
vant to many scenarios in computer vision, medical imaging, remote sensing
∗yuanjing@uni-mannheim.de,
Mannheim, Germany
†schnoerr@uni-mannheim.de,
Mannheim, Germany
‡steidl@math.uni-mannheim.de, University of Mannheim, Institute of Mathematics,
68131 Mannheim
Universityof Mannheim,CVGPR-Group,68131
Universityof Mannheim,CVGPR-Group, 68131
1

Page 2

and experimental fluid dynamics. Especially in fluid dynamics, sophisti-
cated measurement techniques including pulsed laser light sheets and mod-
ern CCD cameras, as well as dedicated hardware enable the recording of
high-resolution image sequences that reveal the evolution of spatio-temporal
structures of unsteady flows [25]. In this connection, an important research
problem is to develop variational approaches that render flow estimation
from image sequences into a well-posed and numerically stable problem,
while preserving small-scale flow structures that are important for empirical
investigations of turbulent phenomena.
To this end, we investigate a novel class of variational flow estimation
schemes by combining higher-order flow regularization with recent tech-
niques developed for non-smooth image decomposition – see below for a
more detailed exposition. As a result, we obtain variational approaches that
allow not only for estimating fluid flow from image sequences but simulta-
neously yield a decomposition of the flow into coherent spatio-temporal flow
patterns and small-scale structures.
In the following, we briefly describe the respective basic ideas in a con-
tinuous setting. In the remainder of the paper, we will derive and investigate
discrete approaches using the so–called mimetic finite difference method de-
veloped by Hyman and Shashkov [18] that preserves the integral identities
fulfilled by the corresponding continuous integral operators.
Image decomposition.
removing noise without destroying important structures such as edges. This
goal cannot be achieved with linear filters, e.g. by minimizing
In image denoising one is typically interested in
J(f) := ?g − f?2
L2(Ω)+ λ
?
Ω
|∇f|2dxdy(1)
for a given noisy image g(x,y) in Ω ⊂ R2. The regularizer incorporates the
quadratic function Φ(s) = s2with s = |∇f|. Via the Euler–Lagrange equa-
tion this variational approach can be related to a linear diffusion equation.
As a consequence, the solution f smoothes g in a completely homogeneous
way and blurs therefore semantically important signal structures. To over-
come this drawback a variety of nonlinear methods have been proposed. One
of the frequently applied approaches replaces the function in the regulariza-
tion term by Φ(s) = s and thus penalizes larger deviations of |∇f| not as
hard as the quadratic function:
J(f) :=1
2?g − f?2
L2(Ω)+ λ
?
Ω
|∇f|dxdy.(2)
This functional was first considered by Rudin, Osher and Fatemi [26]. For
f having partial derivatives in L1 the regularizer coincides with the TV
2

Page 3

semi-norm of f
?f?TV:=
?
|∇f|dxdy.
In contrast to the linear approach (1) we will refer to this method as TV
approach. Based on the dual TV norm, the so–called G norm, this denoising
model was enlarged for the decomposition of given images g into a structural
(cartoon) part fsand a textural part ftas
J(fs,ft)=
?g − (fs+ ft)?2
L2(Ω)+ λ
?
Ω
|∇fs|dxdy,(3)
s.t.
?ft?G≤ δ.
For a more sophisticated treatment of the TV and G norms we refer to
[13, 21]. Meanwhile there exist various numerical realizations of (3), e.g.
[3, 24, 32, 34] . Although not considered in this paper, we note that there
exist other models including other than Gaussian white noise by using the
L1 norm in the data fitting term [5, 23], noise as a third decomposition
component by applying Besov norms [4] or frame and TV/frame approaches
[12, 28] from Harmonic Analysis. Moreover, second order derivatives were
incorporated into the regularizer to avoid for example staircasing effects,
see, e.g. [8, 29].
In this paper, we are interested in the decomposition of vector fields
rather than scalar images. Specifically, we want to deal with optical flow
fields arising, e.g., in experimental fluid dynamics.
Optical flow estimation.
image sequence {g(x,y,t) : t ∈ [0,T]} with a time parameter t. A common
assumption is that image intensities are preserved over time:
Instead of a single image let us consider an
g?x(t1),y(t1),t1
?= g?x(t2),y(t2),t2
?,t2> t1.(4)
Generalizations to other constraints exist but are not relevant for our present
investigation. The linearized version of the gray value constancy assumption
(4) yields the optical flow constraint
gxu1+ gyu2+ gt= ∇g · u + gt= 0 ,(5)
where the optical flow field u = (u1,u2)⊤:= (˙ x, ˙ y)⊤denotes the instanta-
neous velocities of image elements.
Obviously, equation (5) cannot be solved pointwise because at each lo-
cation and time point it consists in solving a single scalar equation for two
scalar unknowns. To overcome this so–called aperture problem additional re-
quirements have to be imposed. Instead of (5) we consider the least squares
approximation
F(u) := ?∇g · u + gt?2
L2(Ω).(6)
3

Page 4

In 1981 Horn and Schunck [17] pioneered the field of regularization methods
for optical flow computation by suggesting to minimize the functional
J(u) = F(u) + λ
?
Ω
|∇u1|2+ |∇u2|2dxdy. (7)
We will refer to this widely used approach as Horn–Schunck model. Ob-
viously, the Horn–Schunck model suffers from the same drawbacks as the
linear image filtering model (1). The solution creates blurry optic flow fields
where the blur appears also across important flow discontinuities. One way
to overcome this limitation consists again in using another function in the
regularizer, e.g.
J(u) =1
2F(u) + λ
?
Ω
(|∇u1|2+ |∇u2|2)
1
2dxdy(8)
as counterpart to the TV model (2), cf. [2, 14, 16].
In this paper we focus on non-rigid motion analysis. In particular, we
are interested in the representation of motions by components that capture
different physical aspects, e.g. solenoidal (divergence free) flows. Referring
again to experimental fluid dynamics, for example, the extraction of coher-
ent flow structures which are immersed into additional motion components
at different spatial scales [19] poses a challenge for image sequence analy-
sis. However, this goal cannot be achieved using first order derivatives in
the regularizer. Rather, based on early work on second–order regularizers
constraining the gradients of the flows divergence and curl [1, 9, 15, 30], we
deal instead of (7) with the functional
J(u) = F(u) + λd
?
Ω
|∇divu|2dxdy + λc
?
Ω
|∇curlu|2dxdy + γ
?
∂Ω
(∂nu)2ds,
(9)
where the boundary is incorporated for stability reasons, cf. [36, 38]. A
key property of this approach is that due to the second order of the flow
derivatives involved, the regularizing terms do not penalize the magnitude
of the basic flow components involving first-order derivatives, i.e. divergence
and curl. To further motivate this approach consider Fig. 1 which shows the
estimation of a solenoidal flow field by first and second order approaches.
The figure clearly demonstrates the superiosity of the div–curl model.
Finally, related to the TV model (8), we deal with
J(u) =1
2F(u)+λd
?
Ω
|∇divu|dxdy+λc
?
Ω
|∇curlu|dxdy+γ
2
?
∂Ω
(∂nu)2ds ,
(10)
involving regularizing terms that are of second order as in (9), and that are
additionally suited to preserve jumps of the divergence and the curl of a flow
4

Page 5

field, respectively, by utilizing the TV-norm. As in image decomposition
we will extend this estimation approach to the decomposition of motion
vector fields into physically relevant components at different scales by using
a discrete equivalent of the G norm. Moreover, we will study the feasibility
of an extension to the simultaneous estimation and decomposition of optical
flows.
Figure 1: Estimation of a typical solenoidal flow field u. Top: Restored flow
based on the Horn–Schunck model (7). Bottom: Restored flow based on
the second order model (9) Vortex structures are better recovered by the
div–curl approach (bottom), cf. [36].
Organization of the paper.
ting based on the mimetic finite difference method in Section 2. We define
scalar and vector fields on primal and dual grids together with appropriate
norms. Then we introduce the first-order operators, where we focus on their
matrix representation using tensor product notation. We show how integral
identities as the Gaussian integral identity and the Helmholtz decomposition
carry over to the discrete situation. Finally, we define discrete TV and G
norms with respect to the primal and dual grid. Based on these definitions
and results we consider the flow estimation task in Section 3. Here we are in-
We start by introducing our discrete set-
5