To appear in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Madison, Wisconsin, June 2003.
A Variational Framework for Image Segmentation
Combining Motion Estimation and Shape Regularization
Department of Computer Science
University of California, Los Angeles – CA 90095
Based on a geometric interpretation of the optic flow con-
straint equation, we propose a conditional probability on
the spatio-temporal image gradient. We consistently derive
a variational approach for the segmentation of the image
domain into regions of homogeneous motion.
The proposed energy functional extends the Mumford-
Shah functional from gray value segmentation to motion
segmentation. It depends on the spatio-temporal image gra-
dient calculated from only two consecutive images of an im-
age sequence. Moreover, it depends on motion vectors for
a set of regions and a boundary separating these regions.
In contrast to most alternative approaches, the problems
of motion estimation and motion segmentation are jointly
solved by minimizing a single functional.
Numerical evaluation with both explicit and implicit
(level set based) representations of the boundary shows the
strengths and limitations of our approach.
1. Introduction and Related Work
The estimation of motion from image sequences has a long
tradition in computer vision. In recent years, many ap-
proaches have been proposed to segment the image plane
on the basis of this motion information. The fields of im-
age sequence analysis and video compression provide nu-
merous applications. In some approaches, motion discon-
tinuities are modeled implicitly [1, 10, 9, 15]. Other ap-
proaches treat the problems of motion estimation in disjoint
sets and optimization of the motion boundaries separately
[14, 2, 12, 13, 7].
In , we presented a variational approach to motion
segmentation with an explicit contour where both the mo-
tion estimation and the boundary optimization are derived
from minimizing a single energy functional. Yet, this ap-
proach had an important drawback: Satisfactory results
were only obtained upon applying two posterior normaliza-
tions to the terms driving the evolution of the motion bound-
In the present paper, we derive a novel variational formu-
lation for segmenting the image plane into regions of homo-
geneous motion. It is based on a simple probabilistic model
forthespatio-temporalimage gradient determined fromtwo
consecutive images of a sequence. We show that local min-
imization of an appropriate energy functional leads to an
eigenvalue problem for the motion parameters and to a gra-
dient descent evolution for the motion boundaries. In con-
trast to our previous approach, all normalizations comprised
in the evolution equation are derived in a consistent manner
by minimizing the proposed functional.
We present numerical results for two implementations of
the functional: one with an explicit spline based representa-
tion of the contour, and one with an implicit level set based
representation. In particular, these results cover the cases of
a moving object on a moving background and of multiple
2. From the Optic Flow Constraint ...
Let f : Ω × R+→ R+be a gray value image sequence.
We assume the intensity of a moving point to be constant
throughout time. This induces the optic flow constraint
where (u, w)tis the local velocity vector. Geometrically,
must either vanish or be orthogonal to the homogeneous
velocity vector v = (u, w, 1)t:
∇3ftv = 0.
This constraint has been employed in many motion estima-
tion approaches. Commonly — for example in the seminal
work of Horn and Schunck  and many subsequent works
Figure 6: Segmenting multiple moving objects on a moving background. Top row: Evolving motion boundary and estimated
motion field superimposed on one of the two input images. Bottom row: Corresponding evolution of the embedding level set
function φ. The motion segmentation functional permits to segment differently moving regions. Due to the level set implementation,
the contour topology is not fixed such that multiple regions can be segmented on the basis of their motion. Note that both the
location of the motion boundary and the motion estimates for cars and background are gradually improved during minimization of
the proposed energy. Minor discrepancies between the final segmentation and the car boundaries are probably due to the fact that
the gray value of the street is not sufficiently structured to permit a reliable motion estimation.
We illustrated the implicit scheme by showing how the
embedding surface generates topological changes of the
motion boundary. In particular, we demonstrated that our
method is capable of detecting interior motion boundaries.
Present work focuses on generalizations of the proposed
approach to more than two motion phases  and to the si-
multaneous segmentation of multiple frames in a sequence.
We showed that the method is robust to non-
The author thanks C. Schn¨ orr, J. Weickert, S. Weber, S.
Soatto, P. Favaro, A. Yuille and S.-C. Zhu for stimulating
discussions. This research was supported by ONR N00014-
02-1-0720 and AFOSR F49620-03-1-0095.
 M. J. Black and P. Anandan. The robust estimation of multiple mo-
tions: Parametric and piecewise–smooth flow fields.
Graph. Image Proc.: IU, 63(1):75–104, 1996.
 V. Caselles and B. Coll. Snakes in movement. SIAM J. Numer. Anal.,
 T. Chan and L. Vese. Active contours without edges. IEEE Trans.
Image Processing, 10(2):266–277, 2001.
 D. Cremers. A multiphase level set framework for variational motion
segmentation. In L. Griffith, editor, Int. Conf. on Scale Space Theo-
ries in Computer Vision, Isle of Skye, 2003. Springer. To appear.
 D. Cremers and C. Schn¨ orr. Statistical shape knowledge in varia-
tional motion segmentation. Im. and Vis. Comp., 21(1):77–86, 2003.
 D. Cremers, F. Tischh¨ auser, J. Weickert, and C. Schn¨ orr. Diffusion
Snakes: Introducing statistical shape knowledge into the Mumford–
Shah functional. Int. J. of Comp. Vis., 50(3):295–313, 2002.
 G. Farneb¨ ack. Very high accuracy velocity estimation using orien-
tation tensors, parametric motion, and segmentation of the motion
field. In Proc. 8th ICCV, volume 1, pages 171–177, 2001.
 B.K.P. Horn and B.G. Schunck. Determining optical flow. Artif.
Intell., 17:185–203, 1981.
 P. Kornprobst, R. Deriche, and G. Aubert. Image sequence analysis
via partial differential equations. J. Math. Im. Vis., 11(1):5–26, 1999.
 E. Memin and P. Perez. Dense estimation and object-based segmen-
tation of the optical flow with robust techniques. IEEE Trans. on Im.
Proc., 7(5):703–719, 1998.
 D. Mumford and J. Shah. Optimal approximations by piecewise
smooth functions and associated variational problems. Comm. Pure
Appl. Math., 42:577–685, 1989.
 J.-M. Odobez and P. Bouthemy. Direct incremental model-based im-
age motion segmentation for video analysis. Signal Proc., 66:143–
 N. Paragios and R. Deriche. Geodesic active contours and level sets
for the detection and tracking of moving objects. IEEE Trans. on
Patt. Anal. and Mach. Intell., 22(3):266–280, 2000.
 C. Schn¨ orr.
main decomposition and shape optimization. Int. J. of Comp. Vis.,
Computation of discontinuous optical flow by do-
 J. Weickert and C. Schn¨ orr.
vex regularizers in PDE–based computation of image motion.
Int. J. of Comp. Vis., 45(3):245–264, 2001.
A theoretical framework for con-
 S. C. Zhu and A. Yuille. Region competition: Unifying snakes, re-
gion growing, and Bayes/MDL for multiband image segmentation.
IEEE Trans. on Patt. Anal. and Mach. Intell., 18(9):884–900, 1996.