Content uploaded by Odej Kao
Author content
All content in this area was uploaded by Odej Kao on Feb 26, 2021
Content may be subject to copyright.
Content uploaded by Timo Bretschneider
Author content
All content in this area was uploaded by Timo Bretschneider
Content may be subject to copyright.
Detection and Removal of Scratches
in Digitised Film Sequences
T. Bretschneider, O. Kao
Department of Computer Science, Technical University of Clausthal,
Julius-Albert-Straße 4, 38678 Clausthal, Germany
{bretschneider, okao}@informatik.tu-clausthal.de
Abstract: In this paper a method for the detection and
removal of scratches in digitised film sequences is dis-
cussed. The proposed technique highlights the
physical structure of the scratch to distinguish
between an actual scratch and an image feature. The
advantage of the technique is the reliability even in the
presence of dominant features parallel to the scratch.
The method for the removal of the scratch utilises an
iterative interpolation to reconstruct the affected
pixels. The convergence of the algorithm is guaran-
teed for the noise free case but proved to be stable
even in the presence of noise.
Keywords: scratch removal, unsharp masking,
iterative interpolation
1 Introduction
Movie films are often damaged through ageing,
chemical changes and abrasion by contacts with
mechanical parts of the film projector. The recon-
struction of already damaged material and preser-
vation of the movie heritage is an important task,
but manual restoration is expensive in time and
money due to the huge data volume. Therefore
unsupervised processing methods for removal of
frequently occurring defects are highly desirable.
In this paper a method for the detection and
removal of vertical scratches is discussed. The
specific type of a vertical scratch is caused by
contact of the film material with a mechanical
part of the film projector. A part of the film
surface, i.e. the emulsion, is lost and the result is
a scratch visible over a number of frames. The
bright or dark characteristic of the scratch is
related to the type of film material, i.e. whether it
is a positive print or a negative. The scratches
start at the top of the image and run vertically
over the entire image and can be found at the
same horizontal location on at least several sub-
sequent frames. The analysis of film material
sampled to the PAL norm (768×576 pixels) has
shown that the typical width of a scratch is
approximately five pixels wide. Although the
horizontal extent of the scratch is small with
respect to the entire frame, the attention of the
observer is attracted by the strong discontinuity.
Figure 1 shows a section of a frame with a
vertical scratch.
Figure 1: Section of degraded film frame with
dominant vertical features and scratch.
The problem of scratch removal has been ad-
dressed in numerous papers. Standard techniques
[1], [2] are based on variations of the spatio-
temporal mean and median filters restricted to
local regions of interest. These methods are
straightforwardly implemented and involve only
a modest computational load. Although the result
is satisfying for a still image, in a film sequence
the lack of texture due to the employed
reconstruction method is noticeable as a blurring.
In the case of vertical scratches due to abrasion
by contact with mechanical parts the effect is em-
phasised by the fact that the distorted region does
not move. Other algorithms using nonlinear
operations [3], adaptive multidimensional
prediction [4], min-max functions [5], and
Fourier series with parameter estimation in a
local neighbourhood [6] have been suggested, but
none have been found to be totally satisfactory.
Unfortunately the majority of archived film
material is relatively noisy. Thus standard
interpolation techniques for the removal of the
scratches fail since their interpolation kernels act
as low pass filters. Also the incorporation of
adjacent frames does not lead to adequate results
since the averaging process suppress the higher
frequency components and requires a motion,
zoom, and pan compensation. Therefore the
proposed technique for the reconstruction is
limited to single frames.
The example images in this paper are mono-
chromatic. While the majority of old movie films
are black-and-white, it is straightforward to apply
the technique to colour films. For the multi-
spectral case the Karhunen-Loeve transformation
[7] may be used to minimise the correlation
between the spectral bands. Thereby each band is
processed individually. The corresponding
inverse transformation produces the reconstructed
multispectral film sequence. Note that the
individual processing of the spectral bands and
final transfer back into the colour space might
result in colour artefacts [8] in comparison to a
multispectral processing technique. However, this
could not be observed in the actual experiments.
2 Scratch characteristic
A suitable model for a scratch caused by a
contact with a mechanical part is a clean groove.
According to the laws of optics the projected
profile of the scratch can be described by the sinc
function. In practice the diffraction is not
observable over an infinite extent and therefore a
weighted version of the sinc, e.g. by a Kaiser
window, is more appropriate to approximate the
profile.
This model is only of limited use for the
detection process since thin vertical features
exhibit almost the same characteristic [9]. Thus a
distinction using the intensity profile of the pro-
jection along the scratch, as suggested by
Kokaram [10], is almost impossible, particularly
if no information about the number, type of copy,
and developing processes is available.
Scratch detection is difficult to perform in the
presence of dominant features in the image. Un-
supervised techniques often result in a variety of
falsely detected scratches, i.e. features, although
an observer can clearly distinguish between such
a feature and a scratch. An example is the image
shown in Figure 1 where apparently the scratch
almost vanishes above the doctor’s head and
relativises the prior made assumption that a
scratch extends over the entire image plane. This
difficulty in scratch detection might be the reason
why recent publications [1], [3], [4], [10], [11],
[12] assume either prior knowledge about the
scratch position or illustrate methods on imagery
with hardly any vertical features.
3 Method
The proposed method embraces the three steps of
pre-processing, detection, and removal which are
schematically depicted in Figure 2.
Image
g
Image
g
↓
↓↓
↓
↓
↓↓
↓
↓
↓↓
↓
↓
↓↓
↓
↓
↓↓
↓
↓
↓↓
↓
L
HL
row-
wise
column-wise
Pre-processing
Removal
f
f^Constraints
Constraints
gA
gA gV
gV
Detection
Scratch position
Projection
Projection
Preselection
Preselection
Unsharp
masking
Unsharp
masking
Projection
or Hough
transformation
Projection
or Hough
transformation
Maxima
Image
f
Image
f
^
row-
wise
Figure 2: Model for the detection and
removal of vertical scratches.
The individual processing levels are explained
in the following subsections.
3.1 Pre-processing
The main emphasis in the pre-processing step is
on the reduction of the data volume and the
amplification of the scratch for an easier
detection. First a 1-dimensional low pass filter
(LPF) is applied column-wise using the Nyquist
frequency of the image g as the cut-off
frequency. Afterwards the image can be
downsampled without loss by the factor of two
since no higher frequency components exist.
Note that the vertically aligned scratch is not
affected by this procedure. In the following step
the same filter and its inverse high pass filter
(HPF) are applied together with another
downsampling step along the rows of the image
to produce the images gA and gV, respectively.
The data of gA represents the approximation of
the input image g, while gV highlights the vertical
features and scratches. The main advantage is the
reduction of data volume which is of positive
significance for the processing time in the
following step of scratch detection.
3.2 Scratch detection
The algorithm extracts possible scratch positions
using the vertical detail coefficients in gV by
computing the means down the columns. A pre-
selection is obtained by selecting the local
maxima of the 1-dimensional plot. The threshold
can be chosen in a conservative way since both
features and scratches exhibit a strong
characteristic in gV. Although they are not easily
distinguished in the gV component, the purpose of
this step is to decrease the computational load for
the second detection step by excluding large
areas of the frame or the complete frame (very
often) if there are no dominant vertical
components.
Subsequently the algorithm uses unsharp
masking [7] and thresholds the unsharp masked
version gU of the approximation gA, i.e. it
generates a binary image gB with gB(x,y)=1 if
gU(x,y) is positive. The thresholding is with
respect to dark scratches and has to be inverted
for bright scratches. As an example Figure 3(a)
shows the corresponding binary image gB. For the
convenience the entire frame is displayed instead
of the sections which were pre-selected in the
first step. The scratch above the doctor’s head is
clearly emphasised due to the utilisation of a
locally adaptive threshold by comparing a filtered
version of the image with itself. Figure 3(b)
depicts the projection of the binary image gB in
Figure 3(a) and exhibits a global maximum at the
x-position of the scratch. The extent of the
scratch is determined by the minima to the left
and right of the scratch centre.
(a)
50 100 150 200 250 300 350
0
50
100
150
200
250
300
350
400
Column
Accumulated pixels
(b)
Figure 3: Unsharp masking for scratch detection:
(a) binary image gB of Figure 1, (b) projection of
gB with maximum at the x-position of the scratch.
In a comparison with other suggested
algorithms the better performance of the
proposed technique becomes evident since it does
not rely on images with negligible vertical
features. Figure 4(a) and (b) show the plots of the
projection and the variance of the image gA
respectively. Note the vague indication of the
scratch in comparison with Figure 3(b).
050 100 150 200 250 300 350
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5x104
(a)
050 100 150 200 250 300 350
0
10
20
30
40
50
60
70
(b)
Figure 4: Comparison with other detection
techniques: (a) projection of the image gA,
(b) variance in the y-direction of gA.
Although the method was developed to detect
vertical scratches it can be utilised for the detec-
tion of scratches with arbitrary orientation. Note
that in case of non-vertical scratches the used fil-
ters in Section 3.1 have to be chosen with greatest
care to preserve the scratch for the actual scratch
detection step. A useful extension of the method
is given by applying the LPF and HPF
successively along the rows and columns, i.e.
computing the approximation and the vertical,
horizontal, and diagonal detail coefficients. The
detection has to use all components and their
combinations to detect any scratch orientation.
The final step of computing the projection is
replaced by a Hough transformation [7].
3.3 Scratch removal
The main problem for the scratch removal is the
reconstruction of higher frequency components.
This section presents a technique based on an
iterative reconstruction method. The idea is
similar to the Gerchberg-Papoulis algorithm [13],
[14] which was proposed for the extrapolation of
band-limited signals with a known support. The
method is based on the successive imposition of
the compact support constraint upon the solution
in the signal domain and the imposition of the
known samples in the Fourier domain. In this
paper the concept of reversing the two constraints
is utilised, i.e. the Fourier transform of the data is
band-limited and known samples are super-
imposed in the signal domain. For convenience
the problem is herein after treated as an 1-dimen-
sional problem of signals taken perpendicular to
the main extent of the scratch. Therefore the
number of missing samples in a sequence is
minimised and the process stabilised.
The model for sampling a movie m is depicted
in Figure 5. The pairs (x,y) and (n,k) represent
continuous and discrete co-ordinates respectively.
B(x,y)
Band-limiting
B(x,y)
Band-limiting
m(x,y)
Movie f (n,k)
Sample
δ
(x,y)
Figure 5: Model for the sampling of a movie.
The band-limiting B is necessary to avoid
aliasing in f and the sampling process
δ
has to be
chosen with respect to B. For convenience it is
assumed that the entire model can be expressed
digitally. The corresponding model for the
scratching and reconstruction is shown in Figure
6. In the following all co-ordinates indices are
dropped.
B
Band-limiting
B
Band-limiting S
Scratching
S
Scratching
f
Signal g
Sample
Reconstruction
u
Figure 6: Degradation and reconstruction
model.
According to Figure 5 f has already been
band-limited due to the avoidance of aliasing in
the sampling process and therefore B has no
influence on f. However, B is needed as a
constraint on the solution space in the
reconstruction process since the scratching S
introduces high frequency components in g.
Using algebraic notation the process in Figure
6 can be described by
,nSBfg += (1)
where n accounts for noise. The reconstruction
process estimates u using the available sample
sequence g with components g0,g1,…,gN-1, the
knowledge about the position of affected samples
in g, i.e. S, and the band-limiting operation B.
Thus with respect to the Landweber iteration [15]
the reconstruction process using algebraic nota-
tion can be described by
(
)
.
ˆˆˆ 1lll fSBgff −+=
+
µ
(2)
This iterative solution is used to avoid the ill-
posed nature of the direct solution. Multiplication
of Equation (2) with B on both sides and using
ll fBu ˆ
ˆ= leads to
()
lll uSgBuu ˆˆˆ 1−+=
+
µ
(3)
where
µ
is the relaxation parameter. The matrix B
is given by B=F-1
Ω
F with the Fourier matrix F as
{}
.1,,1,0,
2exp
1
−∈∀
=
Nkm
N
mk
j
N
Fmk
π
(4)
Note that F is unitary and thus T
F
F
=
−1. The
matrix
Ω
consists of ones along the main
diagonal for frequencies which are passed other-
wise of zeros. The sampling matrix S is
constructed similar to
Ω
and contains only zeros
and ones on the diagonal according to the
positions of missing and available values,
respectively.
Characteristically the process in Equation (3)
converges rapidly at first and then more and more
slowly as the limit is approached. An extensive
understanding of the convergence and choice of
the relaxation parameter
µ
can be gained by an
eigenvalue analysis of the composition of the ma-
trices B and S [17].
4 Results
In a first investigation the effect of additive white
Gaussian noise on the reconstruction of four
missing data points in a sequence of 64 values
was studied. Figure 7 shows the reconstruction
error e with respect to the signal-to-noise ratio
(SNR) and is given by
()
2
ˆl
uule −= (5)
where u is the band-limited signal as shown in
Figure 7 and l
u
ˆ the estimate of u after l iterations.
Using the method proposed by Rank et al.
[16] the SNR for the sampled frame displayed in
Figure 1 was estimated to be approximately 22dB
which leads – accordingly to Figure 7 – to a
reasonable reconstruction result.
10 15 20 25 30 35
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
SNR in dB
||e||
2
Figure 7: Estimation of four missing values
of the true signal u with respect to the SNR
(additive white Gaussian noise).
In a second investigation the influence of the
number of missing values was analysed to decide
about the usability of the proposed method for
interpolation of scratches in motion picture films.
Figure 8 displays the results for the number of
missing pixels in the range from two to sixteen
values. Note that the gap was always centred in
the sequence of 64 values and no noise was as-
sumed. Clearly the iterative interpolation is suit-
able for the scratch type under investigation with
a typical width of approximately five pixels.
2 4 6 8 10 12 14 16
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Miss ing pixels
||e||
2
Figure 8: Reconstruction error with respect to
the number of missing values/pixels.
Finally Figure 9 gives an example for the
reconstruction of real imagery using the proposed
algorithm. The reconstructed image corresponds
to the image shown in Figure 1.
Figure 9: Reconstructed section of film
frame.
The proposed scratch removal method
interpolates reliable missing pixels in historical
motion picture films. Even for homogenous areas
[17] which exhibit a strong influence of noise to
the observer, an almost perfect reconstruction is
obtained without any blurring effect which was
reported repeatedly for other methods [2], [3],
[5].
5 Conclusions
In this paper unsupervised techniques for detec-
tion and removal of scratches in movies sampled
to standard television broadcasting resolution are
presented. After obtaining filtered versions of the
input frame the detection phase starts with using
the vertical detail components to pre-select pos-
sible scratch positions. In a succeeding step real
scratches and thin features are distinguished
using unsharp masking and vertical projection.
The two-phase approach reduces the
computational load by excluding most of the
frames of a movie from further examination.
Moreover only those parts of an image which
exhibit a characteristic similar to a scratch are
passed to the second stage. The detection process
shows a good performance in the presence of thin
vertical features.
The proposed iterative reconstruction method
shows a good performance in the reconstruction
of the image parts lost by the scratching. The re-
sult does not exhibit any blurring which fre-
quently occurs for other interpolation techniques.
Especially homogenous areas in the image which
are mainly dominated by noise exhibit less arte-
facts which are typical for kernel-based interpola-
tion methods.
6 Acknowledgement
This work has been partly supported by the Ger-
man National Merit Foundation. The authors
wish to express their thanks to Jan Engehausen
for providing the motion picture frames used in
this paper.
References
[1] E. Abreu, S.K. Mitra: A simple
algorithm for restoration of images
corrupted by streaks. IEEE International
Conference on Circuits and Systems
(1996) 2:730-733.
[2] O. Kao, J. Engehausen: Scratch removal
in digitised film sequences. Conference
on Imaging Science, Systems, and
Technology (2000) 171-179.
[3] N.-D. Kim, S. Udpa: Nonlinear
operations for edge detection and line
scratch removal. IEEE International
Conference on Systems, Man, and
Cybernetics (1998) 5:4401-4404.
[4] M. Maindl, S. Šimberová: A scratch
removal method. Kybernetika (1998)
34(4):423-428.
[5] S. Armstrong, A. Kokaram, and P.J.W.
Rayner: Non-linear interpolation of
missing data using min-max functions.
IEEE International Conference on
Nonlinear Signal and Image Processing,
July, 1997.
[6] L. Joyeux, O. Buisson, B. Besserer, and
S. Boukir: Detection and removal of line
scratches in motion picture films. IEEE
Computer Society Conference on
Computer Vision and Pattern
Recognition (1999) 1:548-553.
[7] W.K. Pratt: Digital Image Processing.
John Wiley & Sons, Inc., 2nd Edition,
1991
[8] N.P. Galatsanos, R.T. Chin: Digital
restoration of multichannel images.
IEEE Transactions on Acoustics,
Speech, and Signal Processing (1989)
37(3):415-421.
[9] T. Bretschneider, O. Kao, and P.J.
Bones: Removal of vertical scratches in
digitised historical film sequences using
wavelet decomposition. Proceedings of
the Image and Vision Computing New
Zealand, 38-43, 2000
[10] A. Kokaram: Detection and removal of
line scratches in degraded motion picture
sequences. Signal Processing (1996)
8(1):5-8.
[11] R.D. Morris, W.J. Fitzgerald, and A.C.
Kokaram: A sampling based approach to
line scratch removal from motion picture
frames. IEEE International Conference
on Image Processing (1996) 1:801-804.
[12] A.C. Kokaram: Removal of line artefacts
for digital dissemination of archived film
and video. IEEE International
Conference on Multimedia Computing
and Systems (1999) 2:245-249.
[13] R.W. Gerchberg: Super-resolution
through error energy reduction. Optica
Acta (1974) 21:709-720.
[14] A. Papoulis: A new algorithm in spectral
analysis and band-limited extrapolation.
IEEE Transactions on Circuits and
Systems (1975) 22:735-742.
[15] L. Landweber: An iteration formula for
Fredhold integral equations of the first
kind. American Journal of Mathematics
(1951) 73:615-624.
[16] K. Rank, M. Lendl, and R. Unbehauen:
Estimation of image noise variance. IEE
Proceedings, Visual Image Signal
Processing, (1999) 146(2):80-84.
[17] T. Bretschneider, C. Miller, and O. Kao:
Interpolation of scratches in motion
picture films. Proceedings of the
International IEEE Conference on
Acoustics, Speech, and Signal
Processing 2001, Salt Lake City, to be
published.
