Overview of Digital Image Restoration

Full text

Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
JNM. doi:10.32604/jnm.2019.05803 www.techscience.com/jnm
Overview of Digital Image Restoration
Wei Chen
1
, 2, Tingzhu Sun1, 2, Fangming Bi1, 2, *, Tongfeng Sun1, 2, Chaogang Tang1, 2
and Biruk Assefa1, 3
Abstract: Image restoration is an image processing technology with great practical value
in the field of computer vision. It is a computer technology that estimates the image
information of the damaged area according to the residual image information of the
damaged image and carries out automatic repair. This article firstly classify and
summarize image restoration algorithms, and describe recent advances in the research
respectively from three aspects including image restoration based on partial differential
equation, based on the texture of image restoration and based on deep learning, then
make the brief analysis of digital image restoration of subjective and objective evaluation
method, and briefly summarize application of digital image restoration technique in the
future and prospects, provide direction for the research on image after repair.
Keywords: Image inpainting, variational PDE, texture, evaluation method.
1 Introduction
Bertalmio et al. first proposed the concept of image restoration technology in 2000 [Bertalmio,
Sapiro, Caselles, et al. (2000)]. Their research is based on the inpainting algorithm of partial
differential equation, which spreads the edge information of the damaged area of the image to
the small scale damage repair of the region to be repaired, so as to achieve the repair effect
invisible to the human eye. With the development of technology, image restoration
technology has greater practical value, which makes the research and application of image
restoration reach a leap forward. Therefore, image restoration technology has become a
research hotspot in computer graphics and computer vision.
Generally speaking, there are many factors that can cause local information defects in
digital images [Shugen (2004)].The repair algorithm based on partial differential equation
(PDE) was first proposed to apply to small-scale damage repair of images. For large-scale
damage repair of images, text-based image repair algorithm appeared. The most
commonly used algorithm was Criminisi algorithm [Criminisi, Perez and Toyama
(2004)]. With the rise of deep learning, in order to improve the repair effect and
1
College of Computer Science and Technology, China University of Mining and Technology, Xuzhou,
221116, China.
2 Mine Digitization Engineering Research Center of the Ministry of Education, China University of Mining
and Technology, Xuzhou, 221116, China.
3 Infromation Communication Technology Department, Wollo University, Dessie Ethiopia, P.o Box 1145,
Ethiopia.
 Corresponding Author: Fangming Bi. Email: bfm@cumt.edu.cn.

36 Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
strengthen the limitation of traditional algorithm in content repair, researchers are
committed to the research of image repair algorithm based on convolutional neural
network and the generate antagonism network.
In addition, the repair model in image restoration technology only USES the undamaged
region information to estimate and predict the region to be repaired according to the law
of human visual psychology. Therefore, the predicted solution is not the only one, and the
repair effect can be invisible to the human eye.
2 Related work
2.1. Image restoration based on variational PDE
Variational PDE based image restoration is a non-texture image restoration method
suitable for small area damage, while for large incomplete images, due to the loss of too
much information, the repair effect of this method is not good. The main idea is to use the
edge information to be repaired to spread some distance to the damaged area along the
normal direction of the isolux line, as shown in the figure below.
Damaged area Isolux line
Normal direction
Figure 1: Variational PDE algorithm for image restoration
Aiming at the defects of texture restoration, Yong et al. proposed a new PDE framework
for the restoration of fringe texture images [Zhu, Wang and Han (2009)]. The main idea is
to use the direction field as the constraint of diffusion direction. Once the direction field can
be estimated correctly, the gray information can be propagated to the damaged area along
the local positioning direction. However, this model is not enough to repair the detailed
features. In addition, Li et al. proposed a diffusion-based digital image restoration region
localization method, and constructed a feature set based on local variance within and
between channels to identify the restoration region. The purpose of this method is to judge
whether the image is forged or not. The stronger the designed classifier is, the greater
contribution it will make to image restoration [Li, Luo and Huang (2017)].
In the following research, many image repair algorithms based on variational PDE are
developed. It mainly includes total variation (TV) model [Chan and Shen (2001)], Euler’s

Overview of Digital Image Restoration 37
elastica model [Chan, Tony, Kang et al. (2002)], Mumford-shah model [Tsai, Yezzi and
Willsky (2001)] and Mumford-Shah-Euler model [Esedoglu and Shen (2002)]. These
models are introduced into image restoration by TV (total variation) denoising or MS
(Mumford-Shah) segmentation model and other variational models, which are not ideal
for image boundary restoration.
2.2 Image restoration based on texture
Image restoration method based on texture, can repair any damaged scale, its basic idea is
the damaged area on the edge of the texture block as a template, choose from known
image and template match most texture image block, copy to templates area, to ensure the
texture structure similarity and continuity at the same time, the basic texture repair as
shown in the figure below.
Template
Best match image block
Copy
Damaged area
Figure 2: The basic texture repair
Commonly used method is to Criminisi algorithm, aiming at the best matching block in
Criminisi algorithm search and populate the shortcomings, Hu put forward a kind of
Criminisi algorithm combined with sparse representation, sparse representation method is
used to replace Criminisi algorithm search the best matching patch, optimized the
drawing marked area, increase the priority of credibility. The algorithm has strong anti-
interference ability, and the coloring effect is obviously better than other algorithms, but
the complexity of the algorithm still needs to be improved [Hu, Xiong and Iee (2017)].
To reduce the repair time, Ruzic et al. use texture descriptors to guide and utilize context
information to accelerate the search for well-matched (candidate) patches, which could be
used to improve the speed and performance of almost any (patch based) repair method
[Ruzic and Pizurica (2015)]. Wei et al. proposed a new image restoration algorithm by
combining PDE with texture restoration. This method can also reduce the repair time by
classifying the known image texture and reducing the texture search area. However, like
most traditional repair methods, it ignores the color information of the image even though
the repair of image structure and texture information is taken into account [Yao, Sun, Zou,
et al. (2010)].

38 Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
To sum up, text-based image restoration algorithm can repair large damaged areas and has
a good effect on some images. However, it is difficult to repair complex images, and it may
even be impossible to find similar texture blocks. In addition, it takes a long time to search
similar texture blocks around the damaged area, and the repair efficiency is not high.
2.3 Image restoration based on deep learning
Since CNN was proposed, there has been some important progress:
1) CNN can effectively extract abstract information of images in the convolution process.
2) Perceptual Loss enables a trained CNN network’s feature extraction part to be an
auxiliary tool of Perceptual Loss function in image generation.
3) GAN can use supervised learning to enhance the effect of network generation.
The emergence of generated antagonism network makes the research of image restoration
technology reach a peak. At present, the loss function commonly used in image repair is
the combination of confrontation loss and L2 loss. Calculation variance of L2 loss can
stimulate the generation of network output, but the output result cannot capture high-
frequency details and repair clear texture structure. Therefore, the introduction of
confrontation loss can effectively solve the above problems. Counter losses are as follows:
)))]((1[log()]([log),(maxmin )()(
zGDExDEGDV zPzxPx
DG datadata
−+= −−
(1)
The basic model of generating antagonism network applied to image restoration is shown
in the following figure. The latest and effective image restoration model based on deep
learning is developed on this basis. In 2016, Yang et al. used two CNN convolutional
networks to train and repair images from two scales of texture and content. This method
can repair high-resolution images through iteration, but it has defects in performance and
memory [Yang, Lu, Lin et al. (2017)]. Deepak et al. attempted to use the generated
confrontation network to achieve face image repair, and the combination of L2 loss and
confrontation loss could achieve better repair effect. However, the region shape repaired
by this method is fixed, which has strong limitations in practical application [Pathak,
Krahenbuhl, Donahue, et al. (2016)]. In view of this problem, Liu et al. introduced local
convolution, which can repair arbitrary non-central and irregular regions. However, this
method still needs to create a mask based on the deep neural network and carry out pre-
training for random lines [Liu, Reda, Shih, et al. (2018)]. Iizuka et al. used dilated
convolutional layers to increase feelings of wild, trying to get a broader range of image
information of no loss of additional information at the same time, also can fix any of the
center and the effect of irregular region, but its repair effect is poorer for large structure
of objects using the method. In order to improve the distorted structure or fuzzy texture of
the reconstruction region boundary based on deep learning method [Iizuka, Simoserra
and Ishikawa (2017)]. Yu et al. improved the generation network of image restoration on
the basis of Iizuka's research, and decomposed the generation network into two networks:
coarse network with reconstruction loss refers to training loss; refined network with
reconstruction loss and GAN losses refers to training loss [Yu, Lin and Yang (2018)].
The expanded network structure extended the training time. In the same year, Yu
proposed a new GAN loss, known as SN-PatchGAN, which applies spectral

Overview of Digital Image Restoration 39
normalization discriminator to dense image patches to make the training fast and stable
[Yu, Lin and Yang (2018)].
Generate against network based image restoration, however, sometimes the result of the
repair still cannot get a fine texture, Yan and others in the U-Net framework introduced a
special kind of shift-connection layer, namely the Shift-Net, it can with sharp structure
and fine texture to fill the lack of any shape area, this method can effectively improve the
effect of repair [Yan, Li, Li et al. (2018)]. In recent years, great breakthroughs have been
made in the application of GAN to image restoration. In the future, there will still be
more research progress in image restoration based on deep learning.
Encoder Decoder
Generator
Giscriminator True/False
Figure 3: Image restoration model based on generation of antagonism network
3 Evaluation index
The evaluation methods of digital image restoration algorithm mainly include subjective
evaluation and objective evaluation:
Subjective evaluation is generally judged by the observer based on the image evaluated.
The quality of the repair effect depends on the visual judgment of the observer, that is,
the repaired image is graded according to the predetermined evaluation scale or the
observation experience of the observer. The results of the evaluation were obtained using
the average score of a certain number of observers. Subjective evaluation mainly has two
measurement scales: absolute scale and relative scale. As shown in the following table:
Table 1: Subjective evaluation criteria of image restoration
Level
Absolute scale
Relative scale
1
very poor
the worst in the picture
2
poor
worse than the average
3
general
average in the picture
4
good
better than the average
5
very good
the best in the picture
The objective evaluation method of image quality can be classified according to the
required information level of the original image for reference. At present, there are three

40 Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
kinds of objective evaluation methods: the method based on mean variance, the method
based on SNR and the method based on peak SNR.
The evaluation method based on mean square error evaluates the quality of image
restoration by calculating the difference between the mean square value of the restored
image and the original image. The calculation formula is as follows:
2
0 0
')(
1 
 
−

=
Mi Nj
ijij ii
NM
MSE
(2)
The image in the above formula is the size of M*N pixels. Symbols
ij
i
and
'
ij
i
represent
the gray value of the original image and the restored image pixel, respectively.
The image quality evaluation method based on SNR is obtained by calculating the ratio
of signal intensity variance to noise intensity variance. The mathematical formula is as
follows:
 














−
=
 
 
= =
= =
M
i
N
j
M
i
N
j
jiujiu
jiu
SNR
1 1
2
0
1 1
2
10
),(),(
),(
log10
(3)
Image quality evaluation method based on peak signal-to-noise ratio between original
image and image pixels after repair by calculation relative to the mean square error (2^n -
1)^2 for numerical evaluation of the quality of image restoration. The mathematical
formula is as follows:
 














−


=
 
= =
M
1 1
2
0
10
),(),(
1
255255
log10
i
N
j
jiujiu
NM
PSNR
(4)
In addition to judging the quality of restoration by the image restoration effect, the
algorithm running time is another evaluation index. In the case of similar restoration
effects, the less time it takes to repair, the better the image restoration quality.
4 Application
With the improvement of image and video visual requirements, digital image restoration
plays an increasingly irreplaceable role in digital image processing.
Firstly, the development of digital image restoration technology can drive the
development of other areas of image processing. Image restoration has a strong
correlation with the basic problems involved in image restoration, image compression
and image enhancement. The research on image restoration can promote the progress of
the basic problems of image processing.

Overview of Digital Image Restoration 41
Image restoration is different from common image processing problems, such as image
restoration, compression and enhancement. The idea of image restoration is to restore or
reconstruct degraded images by some prior knowledge. Image enhancement is the
processing of images for specific applications to make the visual effects better and more
useful. Image compression is to reduce the amount of data needed to express the image
information and reduce the redundant information of the original image. In other words,
it is to restore the image with the least bit and the least distortion. These image processing
techniques refer to the real information of the original image, while the pixel of the defect
area is almost completely unknown in the image restoration technique, and the goal of
restoration is usually to obtain the complete image based on the prior knowledge of
human beings. In other words, image restoration is to analyze images according to the
rules of human vision. The improvement of this technology mainly relies on the research
of image model and human visual cognitive rules.
At the same time, digital image restoration technology is widely used in many other
applications. Mainly includes:
(1) Image super-resolution analysis. Image super-resolution analysis (SR) refers to the
algorithm used to reconstruct high-resolution images with low-resolution images as the
template. Common methods are the methods of image interpolation, such as spline
interpolation, zero order retention, bilinear interpolation, etc. However, due to the
complexity of the image signal, the high frequency components of the original image
cannot be restored by interpolating only one image. At this point, image restoration
technology can be used to take the points under low resolution as the initial value under
high resolution, and then combine with human vision rules to repair the remaining areas.
(2) Image coding and compression. At present, the structure of most image compression
algorithms is transformation plus entropy coding. The transformation mainly includes
discrete cosine transform, fractal transform, wavelet transform, etc. The restoration
technique can only encode part of the image information when the image is compressed,
while the rest of the image can be reconstructed by the restoration method. This method
makes use of human visual redundancy and can improve the coding efficiency and image
quality [Wu, Zhang, Sun et al. (2009)].
(3) Error hiding in image and video transmission. Video is prone to packet loss during
transmission [Shibata, Iiyama, Hashimoto et al. (2017)]. Image repair technology is used
to process blocks with errors received. Video quality can be improved without changing
transmission bandwidth and communication protocol, and secret messages can be
embedded in video [Nie, Xu, Feng et al. (2018)].
(4) Expand the view and virtual scene construction. Panoramic splicing with image
fusion technology can expand the view of image browsing, extend the image from the
boundary by image restoration technology, carry out image roaming, and edit online
references of the Internet and large image database through scene restoration.
(5) Image steganography. The secret image first generates normal and independent
images with different meanings from the secret image. The generated image is then sent
to the receiver and fed to the generated model database to generate another image that is
visually the same as the secret image. This method has high capacity, security and
reliability [Duan, Song, Qin et al. (2018)].

42 Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
The maturity of image restoration technology also brings another problem: realistic
computer-generated graphics can be forged into photographic images, which leads to
serious security problems. The use of deep neural network can effectively detect
photographic images and computer-generated graphics [Cui, McIntosh and Sun (2018)].
5 Conclusion
Since 2000, the concept of image restoration has been proposed, from the traditional non-
text-based image restoration technology and text-based image restoration technology to
the current deep learning-based image restoration technology. At present, the digital
image restoration technology has made some achievements in the theory and practical
application, but it still has some deficiencies and needs to be further improved. Even
though the application of deep learning makes the image restoration without PS traces, it
is still a big difficulty in the acquisition of the image restoration area. The algorithm itself
cannot automatically acquire the area that needs to be repaired.
In addition, although deep learning can solve some problems in traditional image
restoration techniques, it is urgent to improve training speed. Finally, the digital image
technology to repair the success will not only greatly broaden the application field of
image repair technology, and because in these application fields of research and
development at the same time, will give feedback from these applications in the field of
new problems, which will further enrich the content of the digital image restoration
technology and promote the development of it.
Acknowledgement: The research is supported by National Natural Science Foundation
of China (Grant No. 51874300), the National Natural Science Foundation of China and
Shanxi Provincial People’s Government Jointly Funded Project of China for Coal Base
and Low Carbon (Grant No. U1510115), National Natural Science Foundation of China
(51104157), the Qing Lan Project, the China Postdoctoral Science Foundation (Grant No.
2013T60574), the Scientific Instrument Developing Project of the Chinese Academy of
Sciences (Grant No. YJKYYQ20170074).
References
Bertalmio, M.; Sapiro, G.; Caselles, V.; Ballester, C. (2000): Image inpainting.
International Conference on Computer Graphics and Interactive Techniques.
Chan; Tony, F.; Kang, S. H.; Shen, J. (2002): Euler's elastica and curvature based
inpainting. Siam Journal on Applied Mathematics, vol. 63, pp. 564-592.
Chan, T. F.; Shen, J. (2001): Mathematical models for local nontexture inpaintings.
SIAM Journal on Applied Mathematics, vol. 62, pp. 1019-1043.
Criminisi, A.; Perez, P.; Toyama, K. (2004): Region filling and object removal by
exemplar-based image inpainting. IEEE Transactions on Image Processing, vol. 13, pp.
1200-1212.
Cui, Q.; McIntosh, S.; Sun, H. (2018): Identifying materials of photographic images
and photorealistic computer generated graphics based on deep CNNs. Computers,
Materials & Continua, vol. 55, pp. 229-241.

Overview of Digital Image Restoration 43
Duan, X.; Song, H.; Qin, C.; Khan, M. K. (2018): Coverless Steganography for
digital images based on a generative model. Computers, Materials & Continua, vol. 55,
pp. 483-493.
Esedoglu, S.; Shen, J. (2002): Digital inpainting based on the mumford-shah-euler
image model. European Journal of Applied Mathematics, vol. 13, pp. 353--370.
Hu, G.; Xiong, L. (2017): Criminisi-based sparse representation for image inpainting.
IEEE Third International Conference on Multimedia Big Data.
Iizuka, S.; Simoserra, E.; Ishikawa, H. (2017): Globally and locally consistent image
completion. ACM Transactions on Graphics, vol. 36, pp. 107.
Li, H.; Luo, W.; Huang, J. (2017): Localization of diffusion-based inpainting in digital
images. IEEE Transactions on Information Forensics and Security, vol. 12, pp. 3050-3064.
Liu, G.; Reda, F. A.; Shih, K. J.; Wang, T.; Tao, A. J. et al. (2018): Image inpainting
for irregular holes using partial convolutions. European Conference on Computer Vision,
pp. 89-105.
Nie, Q.; Xu, X.; Feng, B.; Zhang, L. Y. (2018): Defining embedding distortion for intra
prediction mode-based video steganography. Computers, Materials & Continua, vol. 55,
pp. 59-70.
Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A. A. (2016): Context
Encoders: Feature Learning by Inpainting. Computer Vision and Pattern Recognition, pp.
2536-2544.
Ruzic, T.; Pizurica, A. (2015): Context-aware patch-based image inpainting using
markov random field modeling. IEEE Transactions on Image Processing, vol. 24, pp.
444-456.
Shibata, S.; Iiyama, M.; Hashimoto, A.; Minoh, M. (2017): Restoration of sea surface
temperature images by learning-based and optical-flow-based inpainting. IEEE
International Conference on Multimedia and Expo, pp. 193-198.
Shugen, W. (2004): Image inpainting for information lost area based on the texture
matching approach. Bulletin of Surveying and Mapping.
Tsai, A.; Yezzi, A. R.; Willsky, A. S. (2001): Curve evolution implementation of the
Mumford-Shah functional for image segmentation, denoising, interpolation, and
magnification. IEEE Transactions on Image Processing A Publication of the IEEE Signal
Processing Society, vol. 10, pp. 1169-1186.
Wu, Y.; Zhang, H.; Sun, Y.; Guo, H. (2009): Two image compression schemes based
on image inpainting. Proceedings of the Second International Joint Conference on
Computational Sciences and Optimization, vol. 1.
Yan, Z.; Li, X.; Li, M.; Zuo, W.; Shan, S. (2018): Shift-net: image inpainting via deep
feature rearrangement. European Conference on Computer Vision, pp. 3-19.
Yang, C.; Lu, X.; Lin, Z.; Shechtman, E.; Wang, O. et al. (2017): High-Resolution
Image Inpainting Using Multi-scale Neural Patch Synthesis. Computer Vision and
Pattern Recognition, pp. 4076-4084.

44 Copyright © 2019 Tech Science Press JNM, vol.1, no.1, pp.35-44, 2019
Yao, W.; Sun, J. X.; Zou, G.; Teng, S.; Wen, G. J. (2010): PDE image inpainting with
texture synthesis based on damaged region classification. 2nd International Conference
on Advanced Computer Control, vol. 5, pp. 447-450).
Yu, J.; Lin, Z.; Yang, J.; Shen, X.; Lu, X. et al. (2018): Generative image inpainting
with contextual attention. Computer Vision and Pattern Recognition, pp. 5505-5514.
Yu, J.; Lin, Z.; Yang, J.; Shen, X.; Lu, X.;Huang, T. S. (2018): Free-form image
inpainting with gated convolution. Computer Vision and Pattern Recognition.
Zhu, Y.; Wang, G.; Han, Z. (2009): A new oriented-diffusion image inpainting
framework for striped texture images. International Forum on Information Technology
and Applications.