# Surface completion by minimizing energy based on similarity of shape

**ABSTRACT** 3D mesh models generated with range scanner or video images often have holes due to many occlusions by other objects and the object itself. This paper proposes a novel method to fill the missing parts in the incomplete models. The missing parts are filled by minimizing the energy function, which is defined based on similarity of local shape between the missing region and the rest of the object. The proposed method can generate complex and consistent shapes in the missing region. In the experiment, the effectiveness of the method is successfully demonstrated by applying it to complex shape objects with missing parts.

**0**Bookmarks

**·**

**60**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**Two denoising techniques using reflectivity for noisy range images are proposed: range image smoothing by trilateral filter and range image inpainting by belief propagation. The trilateral filter makes use of reflectivity as well as spatial and intensity information so that geometric features, such as jump and roof edges, are preserved while smoothing. The range image inpainting technique based on belief propagation recovers a deteriorated range image using not only the adjacent range values but also the continuity of the reflectance image. We conduct simulations and experiments using synthesized images and actual range images taken by a laser scanner and verify that the proposed techniques suppress noise while preserving jump and roof edges and repair deteriorated range images.2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2011, San Francisco, CA, USA, September 25-30, 2011; 01/2011

Page 1

SURFACE COMPLETION BY MINIMIZING ENERGY BASED ON SIMILARITY OF SHAPE

Norihiko Kawai, Tomokazu Sato and Naokazu Yokoya

Graduate School of Information Science, Nara Institute of Science and Technology

8916-5 Takayama, Ikoma, Nara 630-0192, Japan

?norihi-k,tomoka-s,yokoya?@is.naist.jp

ABSTRACT

3D mesh models generated with range scanner or video im-

ages oftenhave holes dueto manyocclusions by other objects

and the object itself. This paper proposes a novel method to

fill the missing parts in the incomplete models. The missing

parts are filled by minimizing the energy function, which is

defined based on similarity of local shape between the miss-

ing region and the rest of the object. The proposed method

can generate complex and consistent shapes in the missing

region. In the experiment, the effectiveness of the method is

successfully demonstrated by applying it to complex shape

objects with missing parts.

Index Terms— Surface Completion, 3D Inpainting, En-

ergy Minimization, Pattern Similarity

1. INTRODUCTION

3D models of real environments are widely used for enter-

tainment and digital archives. Thus, automatic generation of

3D models by using range scanner and video images has been

investigated. Such methods can obtain whole 3D models by

measuringthetarget objectsfrommultipleviewpoints andin-

tegrate the partial shapes. However, when the target is a large

and complex environment such as an outdoor scene, it is dif-

ficult to generate the complete 3D models without holes due

to many occlusions. For this problem, lots of surface comple-

tion methods which fill the missing regions and obtain com-

plete 3D mesh models have been proposed. These methods

are classified into two categories: One is based on consider-

ing the smoothness of surface shape and the other is based on

using example shapes.

As methods considering the smoothness of surface shape,

partial differential equation, moving least squares and Will-

more flow have been used for filling holes [1, 2, 3]. These

methods can fill the missing regions with smooth surface

patches. Although they are effective for small holes in a 3D

model, unnatural shapes may be generated when the missing

parts are large and surrounding shape is complex because the

methods cannot generate a complex surface. As another ap-

proach considering the smoothness, volumetric-based meth-

ods have been proposed [4, 5]. Davis et al. [4] have filled

holes by diffusing the signed distance function. This method

largely depends on the location and the number of measuring

points. Thus, it is difficult to obtain good results when the lo-

cation is grossly one-sided and the number is small. Podolak

et al. [5] have used the graph cut to interpolate a surface.

Though this method can generate a continuous surface, to

generate complex shapes in the missing region, user’s manual

inputs are required.

On the other hand, in order to generate complex shapes in

missing regions, the methods using example shapes on other

parts of the object (data region) have been proposed [6, 7, 8,

9, 10]. These methods calculate the similarity between the

local surface shape around the missing regions and that in the

data region and fill the missing regions by copying the most

similar surface patches successively. The methods can gen-

erate complex surfaces. However, a discontinuous surface is

easily generated on the seam in the completed model.

In this paper, to solve these problems, the energy function

which represents the non-plausibility of surface shape is de-

fined using the similarity between the local surface shapes of

the missing and data regions. By minimizing the energy for

whole the missing region, the model without discontinuous

surface shapes can be generated as an optimal solution.

2. SURFACE COMPLETION BY MINIMIZING

AN ENERGY FUNCTION

Fig. 1 illustrates the flow diagram of the proposed method.

First, a user manually selects missing regions to be repaired

in a model (a). Next, initial points and faces are generated

to the missing regions (b). Finally, by repeating addition and

deletion of points considering density of points (c) and updat-

ing positions of points by minimizing energy (d), whole sur-

face is optimally completed. In the following sections, first,

the energy function is defined based on local shape similarity

SSD in Section2.1. Next, Section 2.2describesthe procedure

that minimizes the energy function.

2.1. Energy function based on similarity of local shape

As illustratedinFig. 2, first, a 3Dmodelis dividedintoregion

?

data region

in the missing region

?including the missing region

? selected by a user and the

? which is the rest of the object. The plausibility

? is defined by using surface shape in

Page 2

(a) Specify target region(a) Specify target region

(d) Update positions of points by minimizing energy

(d) Update positions of points by minimizing energy

Is energy converged?Is energy converged?

YesYes

NoNo

StartStart

EndEnd

(b) Generate initial points and faces (b) Generate initial points and faces

(c) Add and delete points(c) Add and delete points

Fig. 1. Flow diagram of the proposed method.

Ω

pp

Ω′

Ω

Φ

fA

(p

(p

Similar shape pair Similar shape pair

Missing

region region

Data regionData region

AreaArea

AreaArea

))(p

))

p A

ff

Ω′

Φ

fA

Missing

(p

p A

Fig. 2. Missing and data regions in a 3D model.

the data region

?. Here,

?

?is the expanded area of region

?.

radius in theregion

region

of SSD between the point set around the point p in the region

?

?is determined so that a spherical area

? with constant

?

?includesat least oneofthepointsin the

?. The energy function is defined as the weighed sum

?

follows:

?and surface shape around the point f?p? in the region

? as

?

?

?

p

??

?

?p

???

?p?f?p??

?

p??

?

?p

?

(1)

Here, the weight

?pis set as

? if p is inside of the region

?

erwise

of

around the boundary have higher confidence than those in the

center of the missing region. In Eq. (1), the weight is nor-

malized because the weight of each point change due to the

movement of points. f?p? in the data region

central point of the spherical area whose local shape is the

most similar to that of point p. The point f?p? for minimizing

?

?? becausepositionsofpoints in thisregionarefixed: oth-

(? is the distance from the boundary

?pis set as

?, and

?

??

?

?

?

?

?and

?

?are constants) because positions of points

? denotes the

? is determined as follows:

f?p????????

p

?

??

????

?p?p

?

???

(2)

In this paper,

local shape between the missing and data regions is defined

as the sum of distances between the points in the area

whose central point is p and the aligned surface around f

as follows:

???

?p?f

?p

?? that represents the similarity of

?p

?p

?

???

?p?f

?p???

?

?

??p

?

?

p

?

??p

? p

?

? Mf

?p?pg?p

?

??

?

? (3)

))(p

(pff

ppppff

MM

))((

ip

))((

))((

ppffMMpp

ppppff

==

Normal

vectorvector

Normal vector

)(

g

p

)(

g

p

Surface

in data regionin data region

Points in missing regionPoints in missing region

p x

)

)

(p

(p

fz

))(p

(pfx

))(p

(pfy

pp

yy

p z

))((

i pgg

))((

i pMM

ppff

Matrix for surface alignmentMatrix for surface alignment

ip

Normal

Normal vector

Surface

p x

fz

fx

fy

p z

i p

i p

Fig. 3. Alignment of point clouds and surface for similarity

measure of local shape.

where Mf?p?pdenotes the matrix for surface alignment as

shown in Fig. 3. Mf?p?pg

in the data region which exists in the normal direction of

the point p

spherical area

Matrix for surface alignment Mf

?p

?

? is a point on the aligned surface

?(?

?p).

?

??p

? is the number of points in the

?p.

?p?pis defined as follows:

Mf

?p?p

?

?

?

?

?

xp

yp

zp

??p

??p

??p

????

?

?

?

?

??

?

?

?

?

x

y

z

?

?p

?

??f

?p?

?

?p

?

??f

?p?

?

?p?

??f

?p?

????

?

?

?

?

?

(4)

where

p and f?p), and

sitions ofp andf?p? in global coordinate system, respectively.

In this study, zpis determined as the unit normal vector of the

point p. xpand ypis calculated as Eqs. (5) and (6).

?xp

?yp

?zp

? and

?xf

?p?

?yf

?p?

?zf?p?

? are basis vectors for

??p

??p

??p

? and

??f

?p?

??f?p?

??f?p?

? are 3D po-

xp

?

???

??

??

?

? zp

????

??

??

?

? zp

?

?

(5)

yp

? zp

? xp

?

(6)

xf

?p?, yf?p

?and zf?p

?are also determined in the same way.

2.2. Energy minimization: Update of positions of points

After generating initial points and faces in missing regions,

positions of points are updated so as to minimize the energy

function

algorithm. In our definition of the energy

each point can be treated independently if similar shape pairs

(p, f?p?) determined by Eq. (2) can be fixed. Thus, the fol-

lowing two processes are repeateduntil the energy converges:

(i) update of similar shape pairsfixing positions of points, and

(ii) parallel update of all the positions of points fixing similar

shape pairs.

? defined in Eq. (1) using a framework of greedy

?, the energy for

Page 3

Ω′

i p

pp

Φ

i p

(pgi

Similar shape pair

Ω′

i p

))(pgi

))((

ff

Area Area

p A

Φ

i p

Similar shape pair

p A

Fig. 4. Relationship between points in energy calculation.

In the process (i) above, the update of the similar shape

pair is performed by calculating f

all the points in the missing region with Eq. (2). Concretely,

?p

? fixing the positions of

?

gion

termined as the most similar pointf?p?. In the process (ii), the

positions of all the points p in the missing region are updated

in parallel so as to minimize the energy defined by Eq. (1).

In the following, we describe the method for calculating the

positions of points p for fixed similar shape pairs. First, the

energy

point in the missing region. As shown in Fig. 4, the target

point to be updated is p, and the position of the point inside a

spherical area

to f

the point p is g

defined in terms of p, g

alignment as follows:

?? is calculated by Eq. (3) for all the points in the data re-

? andthe pointwhichgivesa minimum SSD value is de-

? is resolved into the element energy

?

?p? for each

?pcan be expressed as p

? by Eq. (2). In this case, the point corresponding to

?and is corresponded

?p

?

?

?p?. Now, the element energy

?p

?

?p? can be

?

? and matrix Mf

?p

?

?p

?for surface

?

?p??

?

p

?

??p

?p

?

?

??p

?

? p

? Mf

?p

?

?p

?g

?

?p

??

?

?

(7)

The relationship between the total energy

energy

? and the element

?

?p? for each point can be written as follows:

?

?

?

p

?

??

?

?p

?

??

??

(8)

? is the energy for the points in the region

treated as a constant because positions of points and all the

similar shape pairs are fixed in this region in the process (ii).

Here, note that all the corresponding points Mf?p

(?p

as shown in Fig. 5, points p and Mf

sented by using the normal vector npof p, an arbitrary point

p

as follows:

???

?, and is

?

?p

?g

?

?p

?

?

?

?p) exist in the normal direction of point p. Thus,

?p

?

?p

?g

?

?p

? can be repre-

?in the normal direction of p and parameters

?pand

?

?p

?

?p?

p

Mf

? p

?

?

?pnp

?

(9)

(10)

?p

?

?p

?g

?

?p?? p

?

?

?

?p

?

?p?np

?

By substituting them into Eq. (7), the element energy is rep-

resented as follows:

?

?p

??

?

p

?

??p

?p

?

?

??p

?

??p

?

?

?p

?

?p?

?

?

?

(11)

ip

pp

Normal vector of ：

Normal vector of ：

Aligned surface in

the data region the data region

Points in

the missing region the missing region

))((

))((

ppggMM

ppppff

ii

iiii

))((

))((

ppggMM

ppppff

ss

ssss

s p

00

pp

pp

n

n

pt

)),,((

pppi

pi

tt

)),,((

ppps

ps

tt

pp

ip

Aligned surface in

Points in

s p

pt

Fig. 5. Conversion of parameters.

Here, by the assumption that the normal vector npdoes not

change after updating the position of p, all the parameters in

?

dose not affect the element energy of other points. Thus, by

minimizing the element energy

energy

?p? except

?pare constants. In addition, the change of

?p

?

?p

? independently, whole

? can be minimized. The parameter

?pthat minimizes

?

?p? is calculated as follows:

?p

?

?

p

?

??p

?p

?

?

?p

?

?p?

?

p

?

??p

?p

?

?

(12)

From Eqs. (9), (10) and (12), the position of p that minimizes

?

?p? is calculated as follows:

p

?

?

p

?

??p

?p

?Mf?p

?

?p

?g

?

?p

?

?

p

?

??p

?p

?

?

(13)

Actually, the value in Eq. (13) is an approximate solution be-

cause the normal vector npchanges due to the update of the

positions of p. However, a good solution can be obtained as

the energy converges because the change of the normal vector

becomes smaller as the energy converges.

3. EXPERIMENT

To demonstrate the effectiveness of the proposed method, we

have applied the method to two surface models. They have

different characteristics as shown in Figs. 6(a) and 7(a). The

missing region of each model were given manually. In the ex-

periment, as the initial points in the missing region, the grav-

ity point of the boundary points of the missing region and the

median points between the gravity point and each boundary

point were given. Faces were generated so as to connect these

points.

ModelA is a bowlmodel that has asmoothcurvedsurface

and raised edges along the curve around the missing region.

Fig. 6(b) shows the model with initial points and faces. The

initialmodel is notnatural duetorectilinearfaces. Byrefining

this initial model with our method, as illustrated in Fig. 6(c),

the missing region is completed with a smooth curved surface

similar to the surface in the data region. Raised edges are

Page 4

(a) Model with a hole(b) Initial model

(c) Completed model

Fig. 6. Completion for a bowl with a hole (Model A).

alsoreconstructed andtotallythe completed model looks very

natural.

Model B is Stanford Bunny that has a rugged surface and

a dent edge between the body and the leg of the bunnyaround

the missing region. By refining the surface from the initial

model shown in Fig. 7(b), a natural rugged surface is gener-

ated and a dent edge is connected in the missing region as

shown in Fig. 7(c). The completed solid and mesh model

from a different angle is shown in Fig. 7(d). Mesh arrange-

ment looks also natural on the rugged face and the dent edge.

4. CONCLUSION

In this paper, we have presented a novel method for surface

completion based on energy minimization. In the experiment

fortwo kinds of shapeobjects, ourmethodhave generated the

complex and natural surface shapes in the missing regions. In

future work, our method will be applied to 3D models for

large environments such as outdoor scenes. A surface com-

pletion method for shape and color should also be investi-

gated.

Acknowledgement

This research was partially supported by the Ministry of Ed-

ucation, Science, Sports and Culture, Grant-in-Aid for Sci-

entific Research (A), 19200016, and Young Scientists (B),

20700162.

5. REFERENCES

[1] J. Verdera, V. Caselles, M. Bertalmio, and G. Sapiro, “Inpaint-

ing Surface Holes,” in Proc. Int, Conf. on Image Processing,

2003, vol. 2, pp. 903–906.

(a) Model with a hole(b) Initial model

(c) Completed model(d) Completed solid and mesh

model from a different angle

Fig. 7. Completion for Bunny with a hole (Model B).

[2] J. Wang and M.M. Oliveira, “A Hole-Filling Strategy for Re-

construction of Smooth Surfaces in Range Images,” in Proc.

SIBGRAPI, 2003, pp. 11–18.

[3] H. Xie, K.T. McDonnell, and H. Qin,

Method for Surface Restoration with Smooth Boundary Con-

ditions,” Computer Aided Geometric Design, vol. 21, No. 5,

pp. 427–445, 2004.

[4] J. Davis, S.R. Marschner, M. Garr, and M. Levoy, “Filling

Holes in Complex Surfaces Using Volumetric Diffusion,” in

Proc. Int. Symp. on 3D Data Processing, Visualization and

Transmission, 2002, pp. 428–438.

[5] J. Podolak and S. Rusinkiewicz, “Atomic Volumes for Mesh

Completion,” in Proc. Eurographics Symp. on Geometry Pro-

cessing, 2005, pp. 33–41.

[6] A. Sharf, M. Alexa, and D. Cohen-Or, “Context-based Surface

Completion,” in Proc. ACM SIGGRAPH, 2004, pp. 878–887.

[7] M. Pauly, “Example-Based 3D Scan Completion,” in Proc.

Eurographics Symp. on Geometry Processing, 2005, pp. 23–

32.

[8] V. Kraevoy and A. Sheffer, “Template-Based Mesh Comple-

tion,” in Proc. Eurographics Symp. on Geometry Processing,

2005, pp. 13–22.

[9] T.P Breckon and R. B. Fisher, “Plausible 3D Colour Surface

Completion Using Non-parametric Techniques,” in Proc. Int.

Conf. on the Mathematics of Surfaces, 2005, pp. 102 – 120.

[10] S. Park, X. Guo, H. Shin, and H. Qin, “Surface Completion for

Shape and Appearance,” Int. J. of Computer Graphics, vol. 22,

No. 3, pp. 168 – 180, 2006.

“A Finite Element