Content uploaded by V. Ralph Algazi
Author content
All content in this area was uploaded by V. Ralph Algazi
Content may be subject to copyright.
M12.25
DIRECTIONAL INTERPOLATION
OF
IMAGES BASED ON VISUAL PROPERTIES
AND RANK ORDER FILTERING
V.
Ralph Algazi, Gary
E.
Ford,
and Ravindra Potharlanka
CIPIC, Center for Image Processing and Integrated Computing
University of Califomia, Davis
The goal of our research is to develop interpolation
techniques which preserve or enhance the local structure
critical to image quality. In this paper, we present
preliminary results which exploit either the properties of
vision or the properties of the image
in
order to achieve our
goals. Few algorithms systematically take advantage of
both. The median filter, based solely
on
properties of data,
removes statistical outliers and thus isolated errors in
images, and preserves approximately the sharpness of
isolated image transitions. The visual quality of the resulting
images is problematic. Further, the edge preserving
property of the median filter does not extend to comers or
other
two
dimensional structures.
In
this paper, we consider directional image
interpolation, based
on
a local analysis of the spatial image
structure. We also consider the extension of techniques for
the design of linear filters based
on
properties of human
perception reported previously to enhance the perceived
quality of interpolated images.
I.
A BRIEF DISCUSSION AND OVERVIEW
OF INTERPOLATION TECHNIQUES
Interpolation is one of the fundamental signal processing
operations. For digital images, interpolation is necessary
when the display density of images is changed, except in the
case of the subsampling by an integer. Interpolation is also
required in any geometric transformation or warping of
images, even for the same spatial sampling density.
Interpolation is finally one of the intermediate operations in
the multirate processing of images.
Although interpolation may
be
considered with reference
to
the design of a low pass filter based
on
the frequency
content or bandwidth of images, such an approach is seldom
fruitful in image processing applications. First, the extent of
images is generally small,
so
that the use of large support
filters creates large artifacts at the boundary of images.
Second, the design philosophy for low pass filters which is
based
on
the approximation of an ideal low pass frequency
characteristic is inappmpriate for images
[l].
Finally,
if
one
is interested
in
preserving the whole range of detail available
in the original sample, the sampling is then exactly at the
Nyquist rate and formal fiter design specifications cannot be
formulated.
Our interest is in preserving, or even extending the
detailed information content in the image. In that context,
the classical or common interpolation schemes are pixel
replication, bilinear interpolation and bicubic interpolations
[4].
Bilinear and bicubic interpolations are small support
operations which attempt to preserve the detail by providing
a very high bandwidth. Of course, they result
in
significant
aliasing errors, observed most commonly as staircasing for
high contrast edges, or moire patterns for high detail parallel
lines or streaks.
11.
DIRECTIONAL FILTERING AND
INTERPOLATION
Directional interpolation recognizes that high detail areas
in images most often have a definite geometric structure or
pattern, such as
in
the case of edges. In such cases,
interpolation in the low frequency
direction,
along on edge,
is much better than interpolation
in
the high frequency
direction, across the edge. Thus, a directional interpolation
scheme has to perform a local andysis of the image structure
first, and then base the interpolation on that local structure if
a low frequency direction does exist. A number of
techniques have been developed through the years, which
perform image filtering by either analyzing the local image
structure or by performing operations which preserve some
types of local structure. The best known method which
preserves a specific image structure
is
the median filter
[5].
The median filter is best suited to remove outliers in a local
distribution of pixels within a data window. Because
of
the
use of exactly the mid value or median of the distribution, it
will also preserve a high contrast edge. For such an edge,
the distribution is bimodal and the median will transition,
with a single pixel shift of the data window, from one mode
of the distribution to the other. The median filter will not
preserve other local structures and perform quite poorly for
random noise.
Among methods for directional filtering based
on
image
analysis, directional smoothing does a local analysis
of
the
image and generates a direction dependent set of estimates
for each central pixel, from which the final estimate
is
chosen optimally
[6].
Other methods for edge-preserving
smoothing filters
are
presented and discussed in
[7,8].
In a
recent publication,
an
edge preserving interpolation method
has
been
reported
[9].
The method first detects the presence
of a high contrast edge, then estimates its location and
orientation, and finally bases the interpolation
on
that edge
estimate. The method assumes a two level edge with
no
transition width. As
an
altemative to this ideal edge model,
the algorithm reverts to a bilinear interpolation. We frrst
consider a simple generalization
of
the model of an image
transition which works well for isolated edges.
A. Interpolation based on
a
planar transition
model
In
this work, we have restricted our attention
to
the
doubling of sample densities both in the horizontal and
vertical directions. We detect the local areas
of
the image
which can be modeled as a planar model for an isolated
transition. For such a simple model, we can perform a
detection test by evaluation of the local gradient and
Laplacian.
Directional Interpolation usine grad ients: Consider the
image I(j,k). At pixel location
j&,
evaluate the gradient
vector
=
VI using apdient
3x3
Operator such
as
the
Sobel Operator. Let
G
have components G,,G,. The
-
3005
-
CH2977-719110000-3005
$1.00
0
1991
IEEE
direction perpendicular to the gradient corresponds to
isointensity contours on the tangent plane to the surface.
kt
vlc,
has for components
V,=-G,,
Vy=Gx
and has
a direction
8
in
the
xy
plane
I1
I2
0
0
14"
I3
Values I1,I2,I3 and
b
are known. Weestimate IO by
directional interpolation of the four known values for
-x
x.
--<e<-
2 2
We compute
I,,
I,, and take
f,
=
Idl+Id2
If we have
an
exact isointensity line, then
IA,,I,2
should
Again, to compute I,,,I,, we perform a linear
2
be
the same, otherwise, averaging them is reasonable.
interpolation between the adjacent values on the grid.
with
a
=
-
tan
8,
and tan 0
=
p,
from which we obtain
-1
4
I,,
=
aI,
+
(1
-a&;
I,,
=
aI,
+
(1
-
a)I,
I,
=-[I, +I,
+I,+
I,
+P(I,
+
I, -I,
-I,)]
(1)
For
-
>
181
>
5
the figure is no longer valid. The intersected
values are on different axes. By symmetry,
it
is clear that
we have then
n
x
2
1
1
I,
=
-
I,
+
1,
+
I,
+
I,
+
-(Iz
+
I,
-
I,
-
I,)
-
4
'[
P
Thus, if we perform the test
tan
8=p,<l. For
p>l
we use
equapon
(1)
and for
p<l
we shall use equation
(2).
The test
ps1 is equivalent to testing
lGxlS
IGyI.
We have examined only case of interpolation, when
the unknown sample x is at the center of the square formed
by the four known pixels. The only other case for a
2:l
interpolation is shown below
0
0
I1
I2
/-
The approach is completely similar with two exceptions.
a) The nearest pixel changes for
tan
8
=
k
2
;
18126.56"
1
..
b) as
8
>
the nearest neighbors change again from
I,,
I,
to
$,
I,.
This second case is also applicable with
obvious symmetric transpositions, for horizontal
interpolation between known pixels.
Estimation of the gradient: We have assumed that the
gradient is determined by a
3x3
gradient operator.
However, in the case of interpolation it is not clear which
3x3
array is used to estimate the gradient. As before, there
are
2
cases
Case
1:
Averaging Pixels:
0
0
0 0
0 0
0
0
The averages #1 etc form
3x3
arrays which can
be
used to
estimate the gradient.
Case
2:
Averaging Pixels: We now have the situation
shown below
0
0
0
0
0
0
use nine averages as shown in
#1,
#2.
Because the figure is
not symmetric, we
use
either
4
pixel or
2
pixel averages. An
alternate method is to evaluate
two
gradients on the
two
3x3
arrays which straddle the unknown pixel and average the
results. The Laplacian
vz
is estimated by
using
the common
operator
The planar model is accepted
if
11c112
T, and
Vz
2
T,
(3)
where Ti and
T2
are determined empirically. When the test
fails, bilinear interpolation is applied. Results of
this
method
based on a planar model are shown in Figure
1.
The method tuns out to
be
fairly similar to what was
reported in
[9]
in
many of its details, except that a specific
test for the magnitude of the Laplacian is made here, while
in
[9]
only the edge magnitude is used. The results for our
method seem to be somewhat better visually.
B.
Interpolation
for
other local image
structures. Use
of
Quartiles.
Other local image structures commonly encountered, for
which image analysis may provide an improvement in the
interpolation scheme, are streaks and comers. Streaks or
lines are local linear structures which are narrower
than
the
analysis window width. Wedges or comers also have a
definite local structure characterized by two intersecting
directions within an analysis window. Thus, three types of
local structures, edges, streaks and comers, are worth
detecting to improve interpolation. They
are
all characterized
by a bimodal distribution within the analysis window.
These two modes
are
not necessary symmemc with respect
to the median as in the case of an isolated edge. This
observation suggest the use of rank order statistics, within,
the analysis window, to the occurrence of one of the
three structures of interest. Once the presence of of
-
3006
-
these three structures is detected, a 'spatial analysis has to
be
performed to determine the type of structure and its
orientation. Because of the small analysis window, we limit
our study to the use of quartiles. Within an analysis
window, say 3x3, we group the eight exterior pixel values
into quartiles, Qi,@,Q3,Q4. In each quartile there are two
ordered values
Qj,
i=1.
...,
4, j=1,2. A preliminary test for
the detection of a local structure is
where T3 is a threshold determined experimentally. Thus,
the statistic of (4) is an indicator of a bimodal distribution,
which allows for asymmetric modes. Preliminary
experiments show that such a detection scheme has promise
for the detection of the high contrast local structures
including streaks and comers.
Suatial Analvsis: When the test of (4) indicate that a local
skcture may be present, we analyze the distribution of pixel
values in 8 possible directions. Instead of pixels, we also
use a statistic in each of the
8
directions, such as the mean of
a 2
or
4 pixel cluster. As indicated earlier, we expect that the
spatial structure of interest is characterized by a bimodal
distribution, but the modes are not always distributed about
the median. Thus, we classify the pixels into two classes,
High
(H)
and Low
(L)
by using a threshold between the
adjacent quartiles with the largest separation, i.e. the largest
of Qi+l,l-Qi,~
;
i=1,2,3. Thus the number of high and
lows within an analysis window is no longer the same. We
have, for example, the patterns shown below
HLL HHL
LXL HxL
LHH LLL
Case 1 Case 2
Case 1 corresponds to possible streaks and case 2 to a
possible corner. To confirm the presence of a streak we
analyze further
the
immediate neighbors of x
so
as
to
classify
x as H or
L
and perform an interpolation in the appropriate
direction. For a comer, a local analysis about x is used to
resolve whether x should be estimated only from the high
values within the window. Experimental evaluation is not
complete, but preliminary tests indicate that such a method
complements the interpolation schemes discussed earlier and
provides useful results for streaks, which are often
encountered in images.
C . Directional enhancement
of
interpolated
images based on visual properties.
In
[
13 we have developed an approach to the design of
FIR filters based on properties of visual perception. This
formulation results in an optimization
in
both the spatial and
frequency domain,
so
as
to achieve some desired frequency
behavior, while maintaining the quality of the image in the
vicinity of the edge. This second condition requires a spatial
domain constraint so as to avoid excessive rippling, common
in filters with sharp frequency domain transitions. In [2,3],
we have extended this approach to image enhancement in the
horizontal and vertical directions while controlling the noise
variance increase due to the enhancement process. Here we
use the same basic formalism to provide for selective
enhancement in one of four possible directions.
Because we wish to maintain directionality, we now
design one dimensional filter of horizontal, vertical or
diagonal orientations. Thus, we process the image with four
distinct directional FIR filters to enhance the quality of the
image. Since the image will be modified by the enhancement
filters, this step in the processing will be performed as a last
step on an image interpolated to the final display resolution.
Q31-QI2
><T3 (4)
D.
Some Experimental Results
We show in Figure 1 some of the results obtained by the
methods reported in this paper. The original, Figure la, is a
512x512 image. The other images are originally 256x256
interpolated to 512x512. Figure lb shows bilinear
interpolation with significant visual artifacts due to aliasing
errors. Figure IC shows the result of directional
interpolation using the planar model of Section EA. Figure
Id illustrates the results of applying a directional
enhancement as discussed in Section 1I.C. The results
obtained are quite
good
in
the
removal of all remaining
artifacts due to aliasing errors along horizontal, vertical or
diagonal high contrast edges. The enhancement filter cannot
improve portions of the images where the directional image
structure has not been preserved by directional interpolation.
In this paper we have presented some new results on
image interpolation which are based on an analysis
of
the
structure of images in a small window. Because of the
sparsity of data which is available for such analysis, we have
focussed on simple high contrast directional structures, such
as edges, streaks and corners. For edges, we have
examined a simple planar transition model which perform
fairly well on
our
test images.
In
order to detect streaks and
comers, we propose a local analysis of images which group
pixel values into quartiles. Finally, we have applied some of
our previous work on the design of filters based on
properties of human perception to the design of directional
filters which enhance structure. Our results are encouraging
and indicate that this is a promising approach to an area of
research with a number of applications in high quality
imaging.
References:
111.
DISCUSSION [AND CONCLUSIONS
1.
2.
3.
4.
5.
6.
7.
8.
9.
T.A. Hentea and V.R. Algazi. "Perceptual Models and
the Filtering of High-Contrast Achromatic Images".
IEEE
Transactions
on
Systems, Man, and Cybernetics,
V.R. Algazi. "Fir Anisotropic Filters for Image
Enhancement".
Proceedings ICASSP
Vol.
3,
1986
V.R. Algazi, G.E. Ford, and E. Hildum. "Digital
Representation and Storage of High Quality Color
Images by Anisotropic Enhancement and
Subsampling".
Proc. ICASSP
'89,
pp. 1846-1 849.
1989
A.K.
Jain.
Fundamentals
of
Digital Image Processing.
SMC- 14(2)230-246. 1984
.-
Prentice
Hall.
1988.
T.S. Huang and G.Y. Tang. "A Fast Two Dimensional
Median Filtering Algorithm".
IEEE
Trans. ASSP.,
27:13-18. 1979
M. Nagao et T. Matsuyama. "Edge Preserving
Smoothing".
Computer Graphics and Image
Processing,
9:394-407, Academic
Press,
April 1979.
D. Harwood, M. Subbarao, H. Hakalatiti and
L.S.
Davis. "A New Class of Edge-Preserving
Smoothing
Filters".
Pattern Recognition Letters
6: 155-162,
(1987).
T.
Pavlidis. "Algorithms for Image Processing and
Computer Graphics,"
Computer Scfence Press.
K.
Jensen and D. Anastassiou. Spatial Resolution
Enhancement of Images Using Nonlinear Interpolation".
Proc. ICASSP
1990, pp. 2045-2048.
This
research
supported
in
pan
by the
Research
Program
(MICRO)
ofthc
University
of
Cd-
ifomia
and
by
Pacific
Bell
and
HCW~II
Packard
-
3007
-
a)
Original
b)
Linear
Interpolation
c) Directional
Interpolation
d)
Directional Enhancement
Figure
1:
Some
Experimental
Results
-
3008
-