IMAGE INPAINTING CONSIDERING BRIGHTNESS CHANGE
AND SPATIAL LOCALITY OF TEXTURES
Norihiko Kawai, Tomokazu Sato, Naokazu Yokoya
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 630-0192, Japan
firstname.lastname@example.org, email@example.com, firstname.lastname@example.org
image inpainting, image completion, energy minimization
Image inpainting is a tequnique for removing undesired visual objects in images and filling the missing re-
gions with plausible textures. Conventionally, the missing parts of an image are completed by optimizing the
objective function, which is defined based on pattern similarity between the missing region and the rest of
the image (data region). However, unnatural textures are easily generated due to two factors: (1) available
samples in the data region are quite limited, and (2) pattern similarity is one of the required conditions but is
not sufficient for reproducing natural textures. In this paper, in order to improve the image quality of com-
pleted texture, the objective function is extended by allowing brightness changes of sample textures (for (1))
and introducing spatial locality as an additional constraint (for (2)). The effectiveness of these extensions is
successfully demonstrated by applying the proposed method to one hundred images and comparing the results
with those obtained by the conventional methods.
Image inpainting is a tequnique for removing unde-
sired visual objects in images and filling the missing
regions with plausible textures. This research can be
classified into two categories. One is a non-exemplar-
based method and the other is an exemplar-based
method. The non-exemplar-based methods(A. Levin
et al., 2003; C. Ballester et al., 2001a; C. Ballester
et al., 2001b; D. Tschumperl´ e, 2006; E. Vill´ eger
et al., 2004; M. Bertalmio et al., 2001; M. Bertalmio
et al., 2000; S. Esedoglu and J. Shen, 2003; S. Mas-
nou and J.M. Morel, 1998; T. Chan and J. Shen,
2001; T. Chan et al., 2002) are based on pixel inter-
polation considering the continuity of pixel intensity.
These methods are effective for small image gaps like
scratches in a photograph. However, the resultant im-
age easily becomes unclear when the missing region
is large. Therefore, recently many exemplar-based
inpainting methods have been intensively developed
because they can synthesize complex textures in the
Exemplar-basedmethods basicallysynthesize tex-
tures for the missing region based on pattern simi-
larity that is defined between the missing region and
the rest of the image. Some of the exemplar-based
methods use the distance in the feature space as a
similarity measure. As the feature space, Fourier
space, wavelet domain and eigenspace have been
used (A.N. Hirani and T. Totsuka, 1996; S.D. Rane
et al., 1996; T. Amano, 2004). Most of the other
exemplar-based methods simply employ SSD (sum
of squared differences)-based pattern similarity mea-
sures (A. Criminisi et al., 2004; A.A. Efros and T.K.
Leung, 1999; B. Li et al., 2005; C. All` ene and N.
Paragios, 2006; I. Drori et al., 2003; J. Jia and C.
Tang, 2003; J. Sun et al., 2005; N. Komodakis and
G. Tziritas, 2006; R. Bornard et al., 2002; Y. Wexler
et al., 2007). Efros et al. (A.A. Efros and T.K. Le-
ung, 1999) have proposed a method that successively
copies the most similar patternfrom the data region to
the missing region. Although this method can gener-
ate complex textures, the quality of resultant images
is severely affected by the order of texture copy. To
obtain good results with the successive texture copy,
confidence maps such as the number of fixed pixels
in a window, strength of isophotes around the missing
regions and pattern similarity have been used to de-
pletion with a single weight coefficient does not al-
ways work well, and thus it is necessary to determine
the parameter adaptively considering the characteris-
tics of the image in order to obtain good results for
many images containing complex textures.
In this paper, the objective function for image inpaint-
ing is extended to acquire natural images. To obtain
good results, two factors were considered: (1) bright-
ness change of sample textures was allowed, (2) spa-
tial locality was introduced as a new constraint. By
considering these two factors, the missing region was
completed successfully for many images. In experi-
ments, we have demonstrated the effectiveness of our
methodby comparing the resultant images of the con-
ventional and proposed methods. In addition, by a
questionnaire evaluation using 37 subjects, we have
verified that the proposed method could obtain good
In experiments, parameters such as the size of win-
dow and the weight in the energy function were de-
cided empirically. In future work, we shouldestablish
a method to decide optimum parameters.
A. Criminisi, P. P´ erez, and K. Toyama (2004). Region Fill-
ing and Object Removal by Exemplar-Based Image
Inpainting. In Trans. on Image Processing, volume
13, No. 9, pages 1200–1212.
A. Levin, A. Zomet, and Y. Weiss (2003). Learning How to
Inpaint from Global Image Statistics. In Proc. ICCV,
volume 1, pages 305–312.
A.A. Efros and T.K. Leung (1999).
by Non-parametric Sampling. In Proc. ICCV, pages
A.N. Hirani and T. Totsuka (1996). Combining Frequency
and Spatial Domain Information for Fast Interactive
Image Noise Removal.
In Proc. SIGGRAPH1996,
B. Li, Y. Qi, and X. Shen (2005). An Image Inpainting
Method. In Proc. IEEE Int. Conf. on Computer Aided
Design and Computer Graphics, pages 531–536.
C. All` ene and N. Paragios (2006). Image Renaissance Us-
ing Discrete Optimization. In Proc. ICPR, pages 631–
C. Ballester, M. Bertalmio, V. Sapiro, and J. Verdera
(2001a). Filling-In by Joint Interpolation of Vector
Fields and Gray Levels. In Trans. on Image Process-
ing, volume 10, No. 8, pages 1200–1211.
C. Ballester, V. Caselles, J. Verdera, M. Bertalmio, and G.
Sapiro (2001b). A Variational Model for Filling-In
Gray Level and Color Images. In Proc. ICCV, pages
D. Tschumperl´ e (2006). Curvature-Preserving Regulariza-
tion of Multi-valued Images Using PDE’s. In Proc.
ECCV, volume 2, pages 295–307.
E. Vill´ eger, G. Aubert, and L. Blanc-F´ eraud (2004). Image
Disocclusion Using a Probabilistic Gradient Orienta-
tion. In Proc. ICPR, volume 2, pages 52–55.
I. Drori, D. Cohen-Or, and H. Yeshurun (2003). Fragment-
Based Image Completion. In Proc. SIGGRAPH2003,
J. Jia and C. Tang (2003). Image Repairing: Robust Image
Synthesis by Adaptive ND Tensor Voting. In Proc.
CVPR, pages 643–650.
J. Sun, L. Yuan, J. Jia, and H. Shum (2005). Image Com-
pletion with Structure Propagation.
GRAPH2005, pages 861–868.
M. Bertalmio, A.L. Bertozzi, and G. Sapiro (2001). Navier-
Stokes, Fluid Dynamics, and Image and Video In-
painting. In Proc. CVPR, pages 355–362.
M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester
(2000). Image Inpainting. In Proc. SIGGRAPH2000,
N. Komodakis and G. Tziritas (2006). Image Completion
Using Global Optimization. In Proc. CVPR, pages
R. Bornard, E. Lecan, L. Laborelli, and J. Chenot (2002).
Missing Data Correction in Still Images and Image
Sequences. In Proc. ACM Int. Conf. on Multimedia,
S. Esedoglu and J. Shen (2003). Digital Inpainting Based
on the Mumford-shah-euler Image Model. In Euro-
pean J. of Applied Mathematics, volume 13, pages
S. Masnou and J.M. Morel (1998). Level Lines Based Dis-
occlusion. In Proc. ICIP, volume 3, pages 259–263.
S.D. Rane, J. Remus, and G. Sapiro (1996).
Domain Reconstruction of Lost Blocks in Wireless
Image Transmission and Packet-Switched. In Proc.
ICIP, volume 1, pages 309–312.
T. Amano (2004). Image Interpolation by High Dimen-
sional Projection Based on Subspace Method. In Proc.
ICPR, volume 4, pages 665–668.
T. Chan and J. Shen (2001).
Curvature-Driven Diffusions (CDD). In J. of Visual
Communication and Image Representation, volume
12, No. 4, pages 436–449.
T. Chan, S. Kang, J. Shen, and S. Osher (2002). Euler’s
Elastica and Curvature Based Inpaintings. In SIAM
J. of Applied Mathematics, volume 63, No. 2, pages
Y. Wexler, E. Shechtman, and M. Irani (2007). Space-Time
Completion of Video. In Trans. on Pattern Analysis
and Machine Intelligence, volume 29, No. 3, pages
In Proc. SIG-
Non-texture Inpainting by