Reconstruction of Image Structure in Presence of Specular Reflections.

Full text

Page 1

Reconstruction of Image Structure
in Presence of Specular Reflections
Martin Gr¨ oger, Wolfgang Sepp, Tobias Ortmaier, and Gerd Hirzinger
German Aerospace Center (DLR)
Institute of Robotics and Mechatronics
D-82234 Wessling, Germany
http://www.robotic.dlr.de
martin.groeger@dlr.de
Abstract. This paper deals with the reconstruction of original image
structure in the presence of local disturbances such as specular reflec-
tions. It presents two novel schemes for their elimination with respect
to the local image structure: an efficient linear interpolation scheme and
an iterative filling-in approach employing anisotropic diffusion. The al-
gorithms are evaluated on images of the heart surface and are suited to
support tracking of natural landmarks on the beating heart.
1Introduction
Glossy surfaces give rise to specular reflection from light sources. Without proper
identification specularities are often mistaken for genuine surface markings by
computer vision applications such as matching models to objects, deriving mo-
tion fields from optical flow or estimating depth from binocular stereo [BB88].
This paper presents two approaches to reconstruct the original image struc-
ture in the presence of local disturbances. The algorithms have been developed
to enable robust tracking of natural landmarks on the heart surface [GOSH01]
as part of the visual servoing component in a minimally invasive robotic surgery
scenario [ORS+01]. There specular reflections of the point light source arise on
the curved and deforming surface of the beating heart. Due to sudden and ir-
regular occurrence these highlights disturb tracking of natural landmarks on
the beating heart considerably [Gr¨ o00]. Reconstruction schemes are sufficiently
general for application in other fields, where disturbances in images should be
eliminated ensuring continuity of local structures.
Previous work mainly investigates specular, together with diffuse, reflection
[BB88, Wol94], which aim to suppress the specular component while enhancing
the diffuse. This work considers local specular reflections with no detectable dif-
fuse components, which cause total loss of information. Therefore reconstruction
can only be guided by surrounding image structures.
The following section introduces robust extraction of image structure by the
structure tensor which two schemes for reconstruction are based on: linear in-
terpolation between boundary pixels and anisotropic diffusion within a filling-in
scheme. The algorithms are evaluated on video images of the heart with specu-
larities (Sect. 3), before concluding with a summary of results and perspectives.
Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer53

Page 2

2Reconstruction
Specularities occur as highlights on the glossy heart surface (Fig. 1). Since their
grey values are distinctively high and independent of neighbourhood intensities,
simple thresholding can be applied for segmentation. Structure inside specular
areas is restored from local structure information determined by the well-known
structure tensor. This yields reconstruction which is most likely to correspond
to the original area on condition that surface structures possess some continuity.
Therefore intensity information mainly from boundary points along the current
orientation is used. Results are presented for the mechanically stabilized area of
interest of the beating heart (Fig. 1).
Fig.1. Original image with specularities (detail)
2.1Structure Detection
The structure tensor provides a reliable measure of the coherence of structures
and their orientation, derived from surrounding gradient information. For more
details see [Wei98].
Definition 1 (Structure tensor). For an image f with Gaussian smoothed
gradient ∇fσ
def
= ∇(gσ∗ f) the structure tensor is defined as
Jρ(∇fσ)def
= gρ∗ (∇fσ⊗ ∇fσ) = gρ∗
?
(∂fσ
∂fσ
∂x
∂x)2
∂fσ
∂y(∂fσ
∂fσ
∂x
∂fσ
∂y
∂y)2
?
(1)
where gρis a Gaussian kernel of standard deviation ρ ≥ 0, separately convolved
with the components of the matrix resulting from the tensor product ⊗.
The noise scale σ reduces image noise before the gradient operator is applied,
while the integration scale ρ adjusts J to the size of structures to be detected.
The eigenvalues λ1/2of the structure tensor Jρ measure the average contrast
in the direction of the eigenvectors (over some area specified by the integration
54Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer

Page 3

scale ρ). Since Jρis symmetric, its eigenvectors v1,v2are orthonormal [Wei98].
The eigenvector v1corresponding to the eigenvalue with greatest absolute value
(λ1) gives the orientation of highest grey value fluctuation. The other eigenvector
v2, which is orthogonal to v1, gives the preferred local orientation, the coherence
direction. Moreover, the term (λ1−λ2)2is a measure of the local coherence and
becomes large for anisotropic structures [Wei98].
The structure tensor Jρ is used to extract structure orientation for recon-
struction. Anisotropic confidence-based filling-in also makes use of the coherence
measure (λ1− λ2)2for structure dependent diffusion (Sect. 2.3).
2.2Structure Tensor Linear Interpolation
Specular areas are reconstructed by interpolation between boundary points along
the main orientation of the local structure, extracted by the structure tensor.
Extraction of Local Structure. Local orientation is given by the eigenvector v2
belonging to the minor eigenvalue λ2 of the structure tensor Jρ (Sect. 2.1),
which is required for every specular pixel. To detect structure in the given heart
images appropriate values for noise and integration scale are σ = 1 and ρ = 2.8,
corresponding to a catchment area of about 9 × 9 pixels.
Reconstruction. For each point inside the specular area, search for the two
boundary points along the structure orientation associated to it. Linear interpo-
lation between the intensities of the boundary points, weighted according to their
relative distances, yields the new value at the current position. Final low-pass
filtering ensures sufficient smoothness in the reconstructed area.
2.3Anisotropic Confidence-Based Filling-In
This reconstruction scheme fills specular areas from the boundary based on local
structure information. It employs coherence enhancing anisotropic diffusion.
Coherence Enhancing Anisotropic Diffusion. Diffusion is generally con-
ceived as a physical process that equilibrates concentration without creating or
destroying mass. Applied to images, intensity at a certain location is identified
with concentration. Thus the diffusion process implies smoothing of peaks and
sharp changes of intensity, where the image gradient is strong.
The discussed type of diffusion, designed to enhance the coherence of struc-
tures [Wei98], is anisotropic and inhomogeneous. Inhomogeneous means that
the strength of diffusion depends on the current image position, e.g. the ab-
solute value of the gradient ∇f. Further, anisotropic diffusion corresponds to
non-uniform smoothing, e.g. directed along structures. Thus edges can not only
be preserved but even enhanced.
The diffusion tensor D = D(Jρ(∇fσ)), needed to specify anisotropy, has the
same set of eigenvectors v1,v2 as the structure tensor Jρ (Sect. 2.1), reflect-
ing local image structure. Its eigenvalues λ?
1,λ?
2are chosen to enhance coherent
Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer55

Page 4

structures, which implies a smoothing preference along the coherence direction
v2with diffusivity λ?
?α
2increasing with respect to the coherence (λ1− λ2)2of Jρ:
λ?
1
def
= α;
λ?
2
def
=
if λ1= λ2,
otherwise
α + (1 − α)exp
?
−C
(λ1−λ2)2m
?
(2)
where C > 0,m ∈ N,α ∈]0,1[, and the exponential function ensures the smooth-
ness of D. For homogeneous regions, α specifies the strength of diffusion. We
follow [Wei98] with C = 1, m = 1, and α = 0.001.
Anisotropic Confidence-Based Filling-In. A filling-in scheme for structure
driven reconstruction is developed, employing coherence enhancing anisotropic
diffusion. The algorithm is an extension of the confidence-based filling-in model
of Neumann and Pessoa [NP98], and is described by the following equation:
Definition 2 (Anisotropic confidence-based filling-in). For an image f,
evolving over time as ft, anisotropic confidence-based filling-in is given by
∂ft
∂t
= (1 − c)div(D(Jρ(∇f0))∇ft) + c(f0− ft)(3)
where f0is the initial image and c : dom(f) → [0,1] is a confidence measure.
In the present work, c maps to {0,1}, where c = 0 refers to unreliable image
information, i.e. specularities, and non-specular image points provide reliable
information (c = 1). Therefore non-specular regions are not modified, while
specular points are processed according to
∂ft
∂t
= div(D(Jρ(∇f0))∇ft) .
(4)
Algorithm. Linear diffusion equations like (4) are commonly solved by relaxation
methods. Intensity from boundary pixels is propagated into the specular area,
or vice versa, specular intensity is drained from it (source–sink model). In the
current implementation reconstruction calculates the dynamic changes of diffu-
sion over time in an iterative approach. In each step diffusion is represented by
convolution with time-invariant kernels, shaped according to the diffusion tensor
as in [Wei98]. The algorithm runs until the relative change of specular intensity
is sufficiently small.
As intensities at the boundary are constant, diffusion corresponds to inter-
polation between boundary points. Filling-In is related to regularization theory
[TA77] in employing a smoothness and a data-related term. The filling-in model
discussed here incorporates a linear diffusion scheme. Local structure informa-
tion for the diffusion process is extracted only from the initial image, which
allows efficient implementation of the filling-in process.
56 Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer

Page 5

3 Evaluation
Reconstruction is required to meet the following criteria:
(R0) Restriction to specular areas
(R1) Smooth reconstruction inside specular areas
(R2) Smooth transition to boundaries
(R3) Continuity of local structures
(A1) Realtime operability
As a basic requirement only specular areas shall be altered (R0). To avoid new
artefacts, e.g. edges, smoothness is assessed both inside specular areas (R1)
and at the boundary (R2). Structures next to specularities should be continued
within reconstructed areas (R3), because uniform filling-in from the boundary
may cause new artefacts by interrupted structures. Criterion (A1) is kept in mind
to enable realtime application, as required for tracking in a stream of images.
3.1Reconstruction at Specularities on the Heart Surface
First, quality of reconstruction within structured areas is measured by introduc-
ing an artificial disturbance on a horizontal edge, i.e. a 5 × 5 pixel square with
intensity similar to specular reflection (Fig. 2, left). The other two images show
reconstruction by structure tensor linear interpolation and anisotropic filling-in:
The artificial specularity vanishes and original structure is restored.
Since the underlying structure is known, reconstructed areas of both methods
can be compared with the original area by the sum of squared differences (SSD),
a similarity measure also commonly used for tracking: With intensities between 0
and 255, SSD is 872 for structure tensor interpolation versus 553 for anisotropic
filling-in, which is slightly superior as intensities are closer to the original image.
Fig.2. Elimination of artificial specularity (left to right: original, dto. with artificial
specularity (bright square), reconstruction by linear interpolation, and by anisotropic
filling-in)
Secondly, reconstruction is considered with real specularities on the heart
surface for the area of interest in the original image (Fig. 1). Figures 3 and 4
show reconstruction by linear interpolation and anisotropic filling-in.
Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer57