Content uploaded by Timo Bretschneider
Author content
All content in this area was uploaded by Timo Bretschneider on Sep 25, 2013
Content may be subject to copyright.
Removal of Vertical Scratches
in Digitised Historical Film Sequences
Using Wavelet Decomposition
T. Bretschneider
1
, O. Kao
2
, P.J. Bones
1
1
Department of Electrical and Electronic Engineering,
University of Canterbury, New Zealand
2
Department of Computer Science,
Technical University of Clausthal, Germany
timo@elec.canterbury.ac.nz
Abstract
Movie films are often damaged through ageing, chemical changes and contact with mechanical
parts of the film projector. In this paper methods for the detection and removal of vertical
scratches in digitised film sequences are discussed. This specific type of scratch arises by the
contact of the film material running against mechanical part of the camera or film projector. The
proposed technique is based on the discrete wavelet decomposition. This transformation splits an
image into approximation and detail coefficients, where the latter separate into horizontal,
vertical, and diagonal representations. The algorithm reconstructs the missing data in the region
of the scratch and finally the synthesis of the wavelet coefficients generates a restored version of
the scratched image frame. The results show that the combination of scratch detection and
removal in the wavelet domain is superior to other techniques based in the image domain which
use single frames.
Keywords: scratch detection, scratch removal, wavelet transformation, digitised film sequences
1 Introduction
Movie films are often damaged through ageing, chemical changes and abrasion by contacts with
mechanical parts of the film projector. The reconstruction of already damaged material and
preservation of the movie heritage is an important task, but manual restoration is expensive in time
and money due to the huge data volume. Therefore unsupervised processing methods for removal
of frequently occurring defects are highly desirable.
In this paper methods for the detection and removal of vertical scratches are discussed. This
specific type of scratch is caused by contact of the film material with a mechanical part of the film
projector. A part of the film surface, i.e. the emulsion, is lost and the result is a scratch visible over
a number of frames. The bright or dark characteristic of the scratch is related to the type of film
material, i.e. whether it is a positive print or a negative. The scratches start at the top of the image
and run vertically over the entire image and can be found at the same horizontal location on at
least several subsequent frames. The analysis of film material sampled to the European standard
television broadcasting norm (PAL: 768×576 pixels) has shown that the typical width of a scratch
is approximately five pixels wide. Although the horizontal extent of the scratch is small with
respect to the entire frame, the attention of the observer is attracted by the strong discontinuity.
Figure 1 shows single frames out of different sequences with vertical scratches due to the abrasion
of the emulsion.
The problem of scratch detection and removal has been addressed in numerous papers. Standard
techniques [1], [2], [3] are based on variations of the spatio-temporal mean and median filters
restricted to local regions of interest. These methods are straightforwardly implemented and
involve only a modest computational load. Although the result is satisfying for a still image, in a
film sequence the lack of texture due to the employed reconstruction method is noticeable as a
blurring. The effect is emphasised by the fact that the distorted region is does not move. A method
which overcomes this problem was proposed by Kokaram [4]. The introduction of a 2-dimensional
autoregressive model for the image allows an interpolation of missing information that is
consistent with the local neighbourhood. A major disadvantage, however, is the expense of
computing the coefficients for the proposed model. Other algorithms using nonlinear operations
[5], adaptive multidimensional prediction [6], and min-max functions [7] have been suggested, but
none have been found to be totally satisfactory.
(a) (b)
Figure 1: Sections of degraded film frames: (a) frame without dominant vertical features,
(b) frame with dominant vertical features.
Unfortunately the majority of archived film material is relatively noisy. Thus standard
interpolation techniques for the removal of the scratches fail since they do not incorporate the
statistics of the noise. The incorporation of adjacent frames partly helps to overcome this difficulty
by dropping the noise level. However, the need for a motion, zoom, and pan compensation almost
nullifies this advantage. Therefore the following investigation is limited to single frames. A
wavelet decomposition is proposed. The transformation is realised by a combination of filters so
that the majority of the noise gets separated from the actual image content. Parts of the removal
process can therefore be performed using standard interpolation methods [8]. Finally the synthesis
from the wavelet coefficients creates a restored version of the scratched image frame while
preserving the typical noisy characteristic and preventing blurred image regions. The results show
that the combination of scratch detection and removal in the wavelet domain is comparable and
sometimes superior to the other techniques mentioned. A significant advantage of the applied
method is the possibility of an efficient implementation in hardware [9].
The example images in this paper are monochrome. While the majority of old movie films are
black-and-white, it is straightforward to apply the technique to colour films. For the multispectral
case the Karhunen-Loeve transformation [8] may be used to minimise the correlation between the
spectral bands. Thereby each band may be individually processed using the proposed technique.
The corresponding inverse transformation produces the reconstructed multispectral film sequence.
2 Scratch characteristic
Horizontal profiles through areas of a frame that are affected by a scratch exhibit a similar
characteristic although a scratch is more distinctive for bright regions while dark regions appear
less affected like shown in Figure 1(b). The reason is the different thickness of the original
emulsion. The damage for a thin layer is less significant than for a thicker layer of the emulsion
since the depth of the scratch is smaller.
A suitable model for a scratch caused by a contact with a mechanical part is a clean groove.
According to the laws of optics the projected profile of the scratch can be described by the sinc
function. In practice the diffraction is not observable over an infinite extent and therefore a
weighted version of the sinc, e.g. by a Kaiser window, is more appropriate to approximate the
profile. Figure 2(a) shows an example for both functions. Unfortunately this model is only of
limited use for the detection process since thin vertical features can exhibit almost the same
characteristic. These features are best modelled by two step functions with opposite orientations a
small distance apart. Figure 2(b) depicts the density profile of the emulsion after the exposure for a
single step function. Placing a second step function with the opposite orientation left of the first
step to describe a thin feature results in a density profile similar to the groove of a real scratch and
therefore in a similar diffraction pattern. Thus a distinction between a scratch and a thin vertical
feature using the intensity profile of the projection along the scratch, like suggested by Kokaram
[10], is almost impossible, particularly if no information about the number and type of copy and
develop processes is available.
sinc
weighted sinc
Position
Intensity
Distance
Density
(a) (b)
Figure 2: Characteristic of scratch and edge feature: (a) Theoretical (
…
) and
weighted (-) diffraction profile of a scratch, (b) density profile for an edge.
Scratch detection is difficult to perform in the presence of dominant vertical features in the image.
Unsupervised techniques often result in a variety of falsely detected scratches, i.e. vertical features,
although an observer can clearly distinguish between such a feature and a scratch. An example is
the image shown in Figure 1(b). This difficulty might be the reason why recent publications [2],
[4], [5], [6], [10], [11] assume either prior knowledge about the scratch position or illustrate
methods on imagery with hardly any vertical features.
3 Method
The first step is the transformation of the intensity image I into the approximation A, and the detail
coefficients V, H, and D of the discrete wavelet transform. The approximation A is a low-pass
filtered version of I, while the detail coefficients are generated by a combination of low- and high-
pass filters extracting vertical, horizontal, and diagonal details, respectively [12]. The Haar
wavelet was utilised for the imagery under investigation due to its robust characteristic and
efficient computation. The scratch detection is based on the A and V coefficients as described in
Section 3.1. The position of detected scratches is passed together with A and V to the
reconstruction phase. A and V are transformed to A’ and V’ in the scratch removal process (Section
3.2). A restoration of H and D is only necessary if the components are detectably affected by the
scratch. The final reconstructed image I’ is synthesised using the components (A’, V’, H, D) or (A’,
V’, H’, D’), respectively. Figure 3 shows a schematic of the process.
Image I
Image I
A
A
V
V
H
H
D
D
A’
A’
V’
V’
Image I’
Image I’
Wavelet decomposition
Wavelet decomposition
Synthesis
Synthesis
Reconstruction
Reconstruction
Reconstruction
Reconstruction
Detection
Detection
Reconstruction
Reconstruction
Reconstruction
Reconstruction
H’
H’
D’
D’
Figure 3: Schematic of the restoration process.
Note that both the detection and reconstruction operate in the wavelet domain. The advantage is
that much of the noise is separated together with the detail information from the approximation of
the image. Therefore standard reconstruction methods can be applied to the approximation A.
Moreover, vertical structures (features and scratches) are emphasised in the V component and are
hence easier to detect. The whole process is now described in more detail.
3.1 Scratch detection
This paper suggests a simple multistage process limited to single frames with the immense volume
of data to be processed in mind, e.g. approximately 180,000 frames for a two-hour movie. To
decrease the number of false detections the method considers the special characteristic of a scratch
which differs in contrast, thickness, and often vertical extent from a vertically aligned feature.
The first step in the detection algorithm extracts possible scratch positions using the vertical detail
coefficients in V by computing the means down the columns. A preselection is obtained by
selecting the local maxima of the 1-dimensional plot. The threshold can be chosen in a
conservative way since both features and scratches exhibit a strong characteristic in V. Although
they are not easily distinguished in the V component, the purpose of this step is to decrease the
computational load for the second step by excluding large areas of the frame or the complete frame
(very often) if there are no dominant vertical components.
The second step of the scratch detection algorithm uses unsharp masking [8] and thresholds the
unsharp masked version A
u
of the approximation A, i.e. it generates a binary image B with
B(x,y)=1 if A
u
(x,y) is positive. The thresholding is with respect to dark scratches and has to be
inverted for bright scratches. It is sufficient to restrict the filter for the unsharp masking process to
one dimension since the scratch is aligned with the y-axis. Note that the length of the filter window
has to exceed the extent of a scratch by at least one pixel. As an example Figure 4(a) shows the
binary image B corresponding to Figure 1(b). The advantage of using unsharp masking is the
higher sensitivity achieved with respect to the intensity of the pixel neighbourhood. Thus even
dark regions like those in Figure 1(b) between the upper image border and the doctor’s head
contribute towards the detection. Finally the projection of the binary image B along the y-axis is
thresholded. Figure 4(b) depicts the projection of the binary image B in Figure 4(a) and exhibits a
global maximum at the x-position of the scratch. The extent of the scratch is determined by the
minima to the left and right of the scratch centre.
50 100 150 200 250 300 350
0
50
100
150
200
250
300
350
400
Column
Accumulated pixels
(a) (b)
Figure 4: Unsharp masking for scratch detection: (a) binary image B of Figure 1(b),
(b) projection of B with a maximum at the x-position of the scratch.
3.2 Scratch removal
The removal of the detected scratches is done separately for A and V. Three different methods for
the reconstruction of A have been tried and are presented here. (1) A median filter restores a
distorted pixel, i.e. a pixel which is part of the detected scratch, by replacing it with the median
intensity value of the nearest pixels outside the scratch. (2) A cubic spline interpolation is
employed to restore the region of the scratch. (3) Polynomials of the third degree are fitted in a
least squares sense along the rows over the neighbourhood of the scratch to generate an estimate
for the missing pixels. For the reconstruction of the vertical coefficients V the implemented version
uses the first method, i.e. the median filter, since it has been found that this restoration method
gives the best results. The different characteristic of the V coefficients is due to the vertical high
pass filter which emphasises all vertical structures. The median filter preserves this emphasis
better than the others.
4 Results
Different measures for the qualitative validation of the proposed methods using images with
simulated scratches were employed. The root mean squared (RMS) error between an unscratched
image and the reconstructed image was calculated. However, this measure fails in a noisy
environment and therefore the two ratio-based measures
µ
u
/
µ
r
and
σ
u
/
σ
r
were introduced. The
parameters
µ
and
σ
describe the mean and the standard deviation, respectively, while the sub-
scripts u and r represent the unscratched and the restored version of the image under investigation.
A ratio close to unity denotes that the new image has a similar statistical characteristic to the
original image. This does not necessarily indicate that the reconstruction is perceptibly similar,
however together with the RMS it gives a good indication of the restoration quality achieved.
Table 1 lists the results for the different reconstruction approaches for two levels of added
Gaussian noise. The noise level is given as a signal-to-noise ratio (SNR). Note that visual in-
spections indicate that 20dB is a reasonable value with respect to the sampled frames of the movie.
40dB 20dB
RMS
µ
u
/
µ
r
σ
u
/
σ
r
RMS
µ
u
/
µ
r
σ
u
/
σ
r
Median filter 12.045 1.0008 1.6066 16.158 1.0023 1.4212
Cubic spline Interpolation 12.575 1.0001 1.3248 15.003 1.0032 1.3781
Polynomial least squares fit 12.846 1.0013 1.6182 14.703 1.0018 1.3262
Table 1: RMS results for an artificial scratch in an image with different noise levels
The cubic spline interpolation performs better than the least squares fit for a high SNR due to its
more accurate approximation of the sinc interpolator. But the result degrades performance-wise for
an increasing noise level. This indicates that the approximation A is still affected by some noise.
This influence is decreased using a polynomial least squares fit of the third degree. Although the
median filter achieves good results with respect to the RMS and the mean ratio for a SNR of 40dB,
the final images exhibit a significant blurring in the scratch region. In case of a realistic noise level
the quality decreases further. However, comparisons with other reconstruction methods (nearest
neighbour, bilinear, and cubic spline interpolation) have indicated that for the vertical component
V the usage of the median filter gives the best results.
Finally Figure 5(a) and (b) give examples for the reconstruction of real imagery using the poly-
nomial least squares approach to interpolate the missing values for the approximation and the
median filter for the vertical details.
(a) (b)
Figure 5: Scratch removal in real imagery: (a) reconstruction result for Figure 1(a),
(b) enlarged section of Figure 5(a).
5 Conclusions
In this paper a wavelet-based unsupervised technique for detection and removal of scratches in
movies sampled to standard television broadcasting resolution was presented. After obtaining the
wavelet coefficients the detection phase starts with using the vertical detail components to
preselect possible scratch positions. In a succeeding step real scratches and thin features are
distinguished using unsharp masking and vertical projection. The advantage of the wavelet
transformation is the emphasis of vertical image structures for the detection while the two-phase
approach reduces the computational load by excluding most of the frames of a movie from further
examination. Moreover only those parts of an image which exhibit a characteristic similar to a
scratch are passed to the second stage. The proposed detection process shows a good performance
in the presence of thin vertical features.
The reconstruction of the image content in areas of detected scratches is carried out separately for
the image approximation and the vertical details using interpolation and median filtering,
respectively. The benefit of using the wavelet transformed version of the frame is the preservation
of horizontal and diagonal features and a separation of the image noise from the image content.
Thus simple reconstruction techniques can be applied.
The proposed process of scratch detection and removal is fast since it operates on single frames
and is implemented as a multistage process. Moreover recent publications have demonstrated that
the used transform can be efficiently implemented in hardware. Future work includes extensions of
the method incorporating the noise model for the interpolation process and the reduction of falsely
detected scratches. A study to investigate the success of scratch detection over different scenes is
also suggested.
6 Acknowledgment
This work has been partly supported by the German National Merit Foundation. The authors wish
to express their thanks to J. Engehausen for providing the motion picture frames used in this paper.
References
[1]
P.L. Venetianer, F. Werblin, T. Roska, and L.O. Chua: Analogic CNN algorithm for some image
compression and restoration tasks. IEEE Transactions on Circuits and Systems 42:5 (1995) 278-
284.
[2]
E. Abreu, S.K. Mitra: A simple algorithm for restoration of images corrupted by streaks. IEEE
International Conference on Circuits and Systems 2 (1996) 730-733.
[3]
O. Kao, J. Engehausen: Scratch removal in digitised film sequences. Proceedings of the
International Conference on Imaging Science, Systems, and Technology (2000) 171-179.
[4]
A.C. Kokaram: Removal of line artefacts for digital dissemination of archived film and video. IEEE
International Conference on Multimedia Computing and Systems 2 (1999) 245-249.
[5]
N.-D. Kim, S. Udpa: Nonlinear operations for edge detection and line scratch removal. IEEE
International Conference on Systems, Man, and Cybernetics 5 (1998) 4401-4404.
[6] M. Maindl, S. Šimberová: A scratch removal method. Kybernetika 34:4 (1998) 423-428.
[7]
S. Armstrong, A. Kokaram, and P.J.W. Rayner: Non-linear interpolation of missing data using min-
max functions. IEEE International Conference on Nonlinear Signal and Image Processing, July,
1997.
[8] W.K. Pratt: Digital Image Processing. John Wiley & Sons, Inc., 2
nd
Edition (1991).
[9]
K. Bong-hoon, L. Ho-joon, and K. Hyung-Hwa: ASIC design of wavelet transform filter for
moving picture. Journal of the Institute of Electronics Engineers of Korea S 36-S:12 (1999) 67-75.
[10]
A. Kokaram: Detection and removal of line scratches in degraded motion picture sequences. Signal
Processing VIII 1 (1996) 5-8.
[11]
R.D. Morris, W.J. Fitzgerald, and A.C. Kokaram: A sampling based approach to line scratch
removal from motion picture frames. IEEE International Conference on Image Processing 1 (1996)
801-804.
[12]
S. Mallat: A Wavelet Tour of Signal Processing. Academic Press, 2
nd
Edition (1999).
