Surface completion by minimizing energy based on similarity of shape

Abstract

3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.

Full text

Page 1

SURFACE COMPLETION BY MINIMIZING ENERGY BASED ON SIMILARITY OF SHAPE
Norihiko Kawai, Tomokazu Sato and Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 630-0192, Japan
?norihi-k,tomoka-s,yokoya?@is.naist.jp
ABSTRACT
3D mesh models generated with range scanner or video im-
ages oftenhave holes dueto manyocclusions by other objects
and the object itself. This paper proposes a novel method to
fill the missing parts in the incomplete models. The missing
parts are filled by minimizing the energy function, which is
defined based on similarity of local shape between the miss-
ing region and the rest of the object. The proposed method
can generate complex and consistent shapes in the missing
region. In the experiment, the effectiveness of the method is
successfully demonstrated by applying it to complex shape
objects with missing parts.
Index Terms— Surface Completion, 3D Inpainting, En-
ergy Minimization, Pattern Similarity
1. INTRODUCTION
3D models of real environments are widely used for enter-
tainment and digital archives. Thus, automatic generation of
3D models by using range scanner and video images has been
investigated. Such methods can obtain whole 3D models by
measuringthetarget objectsfrommultipleviewpoints andin-
tegrate the partial shapes. However, when the target is a large
and complex environment such as an outdoor scene, it is dif-
ficult to generate the complete 3D models without holes due
to many occlusions. For this problem, lots of surface comple-
tion methods which fill the missing regions and obtain com-
plete 3D mesh models have been proposed. These methods
are classified into two categories: One is based on consider-
ing the smoothness of surface shape and the other is based on
using example shapes.
As methods considering the smoothness of surface shape,
partial differential equation, moving least squares and Will-
more flow have been used for filling holes [1, 2, 3]. These
methods can fill the missing regions with smooth surface
patches. Although they are effective for small holes in a 3D
model, unnatural shapes may be generated when the missing
parts are large and surrounding shape is complex because the
methods cannot generate a complex surface. As another ap-
proach considering the smoothness, volumetric-based meth-
ods have been proposed [4, 5]. Davis et al. [4] have filled
holes by diffusing the signed distance function. This method
largely depends on the location and the number of measuring
points. Thus, it is difficult to obtain good results when the lo-
cation is grossly one-sided and the number is small. Podolak
et al. [5] have used the graph cut to interpolate a surface.
Though this method can generate a continuous surface, to
generate complex shapes in the missing region, user’s manual
inputs are required.
On the other hand, in order to generate complex shapes in
missing regions, the methods using example shapes on other
parts of the object (data region) have been proposed [6, 7, 8,
9, 10]. These methods calculate the similarity between the
local surface shape around the missing regions and that in the
data region and fill the missing regions by copying the most
similar surface patches successively. The methods can gen-
erate complex surfaces. However, a discontinuous surface is
easily generated on the seam in the completed model.
In this paper, to solve these problems, the energy function
which represents the non-plausibility of surface shape is de-
fined using the similarity between the local surface shapes of
the missing and data regions. By minimizing the energy for
whole the missing region, the model without discontinuous
surface shapes can be generated as an optimal solution.
2. SURFACE COMPLETION BY MINIMIZING
AN ENERGY FUNCTION
Fig. 1 illustrates the flow diagram of the proposed method.
First, a user manually selects missing regions to be repaired
in a model (a). Next, initial points and faces are generated
to the missing regions (b). Finally, by repeating addition and
deletion of points considering density of points (c) and updat-
ing positions of points by minimizing energy (d), whole sur-
face is optimally completed. In the following sections, first,
the energy function is defined based on local shape similarity
SSD in Section2.1. Next, Section 2.2describesthe procedure
that minimizes the energy function.
2.1. Energy function based on similarity of local shape
As illustratedinFig. 2, first, a 3Dmodelis dividedintoregion
?
data region
in the missing region
?including the missing region
? selected by a user and the
? which is the rest of the object. The plausibility
? is defined by using surface shape in

Page 2

(a) Specify target region(a) Specify target region
(d) Update positions of points by minimizing energy
(d) Update positions of points by minimizing energy
Is energy converged?Is energy converged?
YesYes
NoNo
StartStart
EndEnd
(b) Generate initial points and faces (b) Generate initial points and faces
(c) Add and delete points(c) Add and delete points
Fig. 1. Flow diagram of the proposed method.
Ω
pp
Ω′
Ω
Φ
fA
(p
(p
Similar shape pair Similar shape pair
Missing
region region
Data regionData region
AreaArea
AreaArea
))(p
))
p A
ff
Ω′
Φ
fA
Missing
(p
p A
Fig. 2. Missing and data regions in a 3D model.
the data region
?. Here,
?
?is the expanded area of region
?.
radius in theregion
region
of SSD between the point set around the point p in the region
?
?is determined so that a spherical area
? with constant
?
?includesat least oneofthepointsin the
?. The energy function is defined as the weighed sum
?
follows:
?and surface shape around the point f?p? in the region
? as
?
?
?
p
??
?
?p
???
?p?f?p??
?
p??
?
?p
?
(1)
Here, the weight
?pis set as
? if p is inside of the region
?
erwise
of
around the boundary have higher confidence than those in the
center of the missing region. In Eq. (1), the weight is nor-
malized because the weight of each point change due to the
movement of points. f?p? in the data region
central point of the spherical area whose local shape is the
most similar to that of point p. The point f?p? for minimizing
?
?? becausepositionsofpoints in thisregionarefixed: oth-
(? is the distance from the boundary
?pis set as
?, and
?
??
?
?
?
?
?and
?
?are constants) because positions of points
? denotes the
? is determined as follows:
f?p????????
p
?
??
????
?p?p
?
???
(2)
In this paper,
local shape between the missing and data regions is defined
as the sum of distances between the points in the area
whose central point is p and the aligned surface around f
as follows:
???
?p?f
?p
?? that represents the similarity of
?p
?p
?
???
?p?f
?p???
?
?
??p
?
?
p
?
??p
? p
?
? Mf
?p?pg?p
?
??
?
? (3)
))(p
(pff
ppppff
MM
))((
ip
))((
))((
ppffMMpp
ppppff
==
Normal
vectorvector
Normal vector
)(
g
p
)(
g
p
Surface
in data regionin data region
Points in missing regionPoints in missing region
p x
)
)
(p
(p
fz
))(p
(pfx
))(p
(pfy
pp
yy
p z
))((
i pgg
))((
i pMM
ppff
Matrix for surface alignmentMatrix for surface alignment
ip
Normal
Normal vector
Surface
p x
fz
fx
fy
p z
i p
i p
Fig. 3. Alignment of point clouds and surface for similarity
measure of local shape.
where Mf?p?pdenotes the matrix for surface alignment as
shown in Fig. 3. Mf?p?pg
in the data region which exists in the normal direction of
the point p
spherical area
Matrix for surface alignment Mf
?p
?
? is a point on the aligned surface
?(?
?p).
?
??p
? is the number of points in the
?p.
?p?pis defined as follows:
Mf
?p?p
?
?
?
?
?
xp
yp
zp
??p
??p
??p
????
?
?
?
?
??
?
?
?
?
x
y
z
?
?p
?
??f
?p?
?
?p
?
??f
?p?
?
?p?
??f
?p?
????
?
?
?
?
?
(4)
where
p and f?p), and
sitions ofp andf?p? in global coordinate system, respectively.
In this study, zpis determined as the unit normal vector of the
point p. xpand ypis calculated as Eqs. (5) and (6).
?xp
?yp
?zp
? and
?xf
?p?
?yf
?p?
?zf?p?
? are basis vectors for
??p
??p
??p
? and
??f
?p?
??f?p?
??f?p?
? are 3D po-
xp
?
???
??
??
?
? zp
????
??
??
?
? zp
?
?
(5)
yp
? zp
? xp
?
(6)
xf
?p?, yf?p
?and zf?p
?are also determined in the same way.
2.2. Energy minimization: Update of positions of points
After generating initial points and faces in missing regions,
positions of points are updated so as to minimize the energy
function
algorithm. In our definition of the energy
each point can be treated independently if similar shape pairs
(p, f?p?) determined by Eq. (2) can be fixed. Thus, the fol-
lowing two processes are repeateduntil the energy converges:
(i) update of similar shape pairsfixing positions of points, and
(ii) parallel update of all the positions of points fixing similar
shape pairs.
? defined in Eq. (1) using a framework of greedy
?, the energy for

Page 3

Ω′
i p
pp
Φ
i p
(pgi
Similar shape pair
Ω′
i p
))(pgi
))((
ff
Area Area
p A
Φ
i p
Similar shape pair
p A
Fig. 4. Relationship between points in energy calculation.
In the process (i) above, the update of the similar shape
pair is performed by calculating f
all the points in the missing region with Eq. (2). Concretely,
?p
? fixing the positions of
?
gion
termined as the most similar pointf?p?. In the process (ii), the
positions of all the points p in the missing region are updated
in parallel so as to minimize the energy defined by Eq. (1).
In the following, we describe the method for calculating the
positions of points p for fixed similar shape pairs. First, the
energy
point in the missing region. As shown in Fig. 4, the target
point to be updated is p, and the position of the point inside a
spherical area
to f
the point p is g
defined in terms of p, g
alignment as follows:
?? is calculated by Eq. (3) for all the points in the data re-
? andthe pointwhichgivesa minimum SSD value is de-
? is resolved into the element energy
?
?p? for each
?pcan be expressed as p
? by Eq. (2). In this case, the point corresponding to
?and is corresponded
?p
?
?
?p?. Now, the element energy
?p
?
?p? can be
?
? and matrix Mf
?p
?
?p
?for surface
?
?p??
?
p
?
??p
?p
?
?
??p
?
? p
? Mf
?p
?
?p
?g
?
?p
??
?
?
(7)
The relationship between the total energy
energy
? and the element
?
?p? for each point can be written as follows:
?
?
?
p
?
??
?
?p
?
??
??
(8)
? is the energy for the points in the region
treated as a constant because positions of points and all the
similar shape pairs are fixed in this region in the process (ii).
Here, note that all the corresponding points Mf?p
(?p
as shown in Fig. 5, points p and Mf
sented by using the normal vector npof p, an arbitrary point
p
as follows:
???
?, and is
?
?p
?g
?
?p
?
?
?
?p) exist in the normal direction of point p. Thus,
?p
?
?p
?g
?
?p
? can be repre-
?in the normal direction of p and parameters
?pand
?
?p
?
?p?
p
Mf
? p
?
?
?pnp
?
(9)
(10)
?p
?
?p
?g
?
?p?? p
?
?
?
?p
?
?p?np
?
By substituting them into Eq. (7), the element energy is rep-
resented as follows:
?
?p
??
?
p
?
??p
?p
?
?
??p
?
??p
?
?
?p
?
?p?
?
?
?
(11)
ip
pp
Normal vector of ：
Normal vector of ：
Aligned surface in
the data region the data region
Points in
the missing region the missing region
))((
))((
ppggMM
ppppff
ii
iiii
))((
))((
ppggMM
ppppff
ss
ssss
s p
00
pp
pp
n
n
pt
)),,((
pppi
pi
tt
)),,((
ppps
ps
tt
pp
ip
Aligned surface in
Points in
s p
pt
Fig. 5. Conversion of parameters.
Here, by the assumption that the normal vector npdoes not
change after updating the position of p, all the parameters in
?
dose not affect the element energy of other points. Thus, by
minimizing the element energy
energy
?p? except
?pare constants. In addition, the change of
?p
?
?p
? independently, whole
? can be minimized. The parameter
?pthat minimizes
?
?p? is calculated as follows:
?p
?
?
p
?
??p
?p
?
?
?p
?
?p?
?
p
?
??p
?p
?
?
(12)
From Eqs. (9), (10) and (12), the position of p that minimizes
?
?p? is calculated as follows:
p
?
?
p
?
??p
?p
?Mf?p
?
?p
?g
?
?p
?
?
p
?
??p
?p
?
?
(13)
Actually, the value in Eq. (13) is an approximate solution be-
cause the normal vector npchanges due to the update of the
positions of p. However, a good solution can be obtained as
the energy converges because the change of the normal vector
becomes smaller as the energy converges.
3. EXPERIMENT
To demonstrate the effectiveness of the proposed method, we
have applied the method to two surface models. They have
different characteristics as shown in Figs. 6(a) and 7(a). The
missing region of each model were given manually. In the ex-
periment, as the initial points in the missing region, the grav-
ity point of the boundary points of the missing region and the
median points between the gravity point and each boundary
point were given. Faces were generated so as to connect these
points.
ModelA is a bowlmodel that has asmoothcurvedsurface
and raised edges along the curve around the missing region.
Fig. 6(b) shows the model with initial points and faces. The
initialmodel is notnatural duetorectilinearfaces. Byrefining
this initial model with our method, as illustrated in Fig. 6(c),
the missing region is completed with a smooth curved surface
similar to the surface in the data region. Raised edges are

Page 4

(a) Model with a hole(b) Initial model
(c) Completed model
Fig. 6. Completion for a bowl with a hole (Model A).
alsoreconstructed andtotallythe completed model looks very
natural.
Model B is Stanford Bunny that has a rugged surface and
a dent edge between the body and the leg of the bunnyaround
the missing region. By refining the surface from the initial
model shown in Fig. 7(b), a natural rugged surface is gener-
ated and a dent edge is connected in the missing region as
shown in Fig. 7(c). The completed solid and mesh model
from a different angle is shown in Fig. 7(d). Mesh arrange-
ment looks also natural on the rugged face and the dent edge.
4. CONCLUSION
In this paper, we have presented a novel method for surface
completion based on energy minimization. In the experiment
fortwo kinds of shapeobjects, ourmethodhave generated the
complex and natural surface shapes in the missing regions. In
future work, our method will be applied to 3D models for
large environments such as outdoor scenes. A surface com-
pletion method for shape and color should also be investi-
gated.
Acknowledgement
This research was partially supported by the Ministry of Ed-
ucation, Science, Sports and Culture, Grant-in-Aid for Sci-
entific Research (A), 19200016, and Young Scientists (B),
20700162.
5. REFERENCES
[1] J. Verdera, V. Caselles, M. Bertalmio, and G. Sapiro, “Inpaint-
ing Surface Holes,” in Proc. Int, Conf. on Image Processing,
2003, vol. 2, pp. 903–906.
(a) Model with a hole(b) Initial model
(c) Completed model(d) Completed solid and mesh
model from a different angle
Fig. 7. Completion for Bunny with a hole (Model B).
[2] J. Wang and M.M. Oliveira, “A Hole-Filling Strategy for Re-
construction of Smooth Surfaces in Range Images,” in Proc.
SIBGRAPI, 2003, pp. 11–18.
[3] H. Xie, K.T. McDonnell, and H. Qin,
Method for Surface Restoration with Smooth Boundary Con-
ditions,” Computer Aided Geometric Design, vol. 21, No. 5,
pp. 427–445, 2004.
[4] J. Davis, S.R. Marschner, M. Garr, and M. Levoy, “Filling
Holes in Complex Surfaces Using Volumetric Diffusion,” in
Proc. Int. Symp. on 3D Data Processing, Visualization and
Transmission, 2002, pp. 428–438.
[5] J. Podolak and S. Rusinkiewicz, “Atomic Volumes for Mesh
Completion,” in Proc. Eurographics Symp. on Geometry Pro-
cessing, 2005, pp. 33–41.
[6] A. Sharf, M. Alexa, and D. Cohen-Or, “Context-based Surface
Completion,” in Proc. ACM SIGGRAPH, 2004, pp. 878–887.
[7] M. Pauly, “Example-Based 3D Scan Completion,” in Proc.
Eurographics Symp. on Geometry Processing, 2005, pp. 23–
32.
[8] V. Kraevoy and A. Sheffer, “Template-Based Mesh Comple-
tion,” in Proc. Eurographics Symp. on Geometry Processing,
2005, pp. 13–22.
[9] T.P Breckon and R. B. Fisher, “Plausible 3D Colour Surface
Completion Using Non-parametric Techniques,” in Proc. Int.
Conf. on the Mathematics of Surfaces, 2005, pp. 102 – 120.
[10] S. Park, X. Guo, H. Shin, and H. Qin, “Surface Completion for
Shape and Appearance,” Int. J. of Computer Graphics, vol. 22,
No. 3, pp. 168 – 180, 2006.
“A Finite Element