Page 1

Reconstruction of Image Structure

in Presence of Specular Reflections

Martin Gr¨ oger, Wolfgang Sepp, Tobias Ortmaier, and Gerd Hirzinger

German Aerospace Center (DLR)

Institute of Robotics and Mechatronics

D-82234 Wessling, Germany

http://www.robotic.dlr.de

martin.groeger@dlr.de

Abstract. This paper deals with the reconstruction of original image

structure in the presence of local disturbances such as specular reflec-

tions. It presents two novel schemes for their elimination with respect

to the local image structure: an efficient linear interpolation scheme and

an iterative filling-in approach employing anisotropic diffusion. The al-

gorithms are evaluated on images of the heart surface and are suited to

support tracking of natural landmarks on the beating heart.

1Introduction

Glossy surfaces give rise to specular reflection from light sources. Without proper

identification specularities are often mistaken for genuine surface markings by

computer vision applications such as matching models to objects, deriving mo-

tion fields from optical flow or estimating depth from binocular stereo [BB88].

This paper presents two approaches to reconstruct the original image struc-

ture in the presence of local disturbances. The algorithms have been developed

to enable robust tracking of natural landmarks on the heart surface [GOSH01]

as part of the visual servoing component in a minimally invasive robotic surgery

scenario [ORS+01]. There specular reflections of the point light source arise on

the curved and deforming surface of the beating heart. Due to sudden and ir-

regular occurrence these highlights disturb tracking of natural landmarks on

the beating heart considerably [Gr¨ o00]. Reconstruction schemes are sufficiently

general for application in other fields, where disturbances in images should be

eliminated ensuring continuity of local structures.

Previous work mainly investigates specular, together with diffuse, reflection

[BB88, Wol94], which aim to suppress the specular component while enhancing

the diffuse. This work considers local specular reflections with no detectable dif-

fuse components, which cause total loss of information. Therefore reconstruction

can only be guided by surrounding image structures.

The following section introduces robust extraction of image structure by the

structure tensor which two schemes for reconstruction are based on: linear in-

terpolation between boundary pixels and anisotropic diffusion within a filling-in

scheme. The algorithms are evaluated on video images of the heart with specu-

larities (Sect. 3), before concluding with a summary of results and perspectives.

Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer53

Page 2

2Reconstruction

Specularities occur as highlights on the glossy heart surface (Fig. 1). Since their

grey values are distinctively high and independent of neighbourhood intensities,

simple thresholding can be applied for segmentation. Structure inside specular

areas is restored from local structure information determined by the well-known

structure tensor. This yields reconstruction which is most likely to correspond

to the original area on condition that surface structures possess some continuity.

Therefore intensity information mainly from boundary points along the current

orientation is used. Results are presented for the mechanically stabilized area of

interest of the beating heart (Fig. 1).

Fig.1. Original image with specularities (detail)

2.1Structure Detection

The structure tensor provides a reliable measure of the coherence of structures

and their orientation, derived from surrounding gradient information. For more

details see [Wei98].

Definition 1 (Structure tensor). For an image f with Gaussian smoothed

gradient ∇fσ

def

= ∇(gσ∗ f) the structure tensor is defined as

Jρ(∇fσ)def

= gρ∗ (∇fσ⊗ ∇fσ) = gρ∗

?

(∂fσ

∂fσ

∂x

∂x)2

∂fσ

∂y(∂fσ

∂fσ

∂x

∂fσ

∂y

∂y)2

?

(1)

where gρis a Gaussian kernel of standard deviation ρ ≥ 0, separately convolved

with the components of the matrix resulting from the tensor product ⊗.

The noise scale σ reduces image noise before the gradient operator is applied,

while the integration scale ρ adjusts J to the size of structures to be detected.

The eigenvalues λ1/2of the structure tensor Jρ measure the average contrast

in the direction of the eigenvectors (over some area specified by the integration

54Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer

Page 3

scale ρ). Since Jρis symmetric, its eigenvectors v1,v2are orthonormal [Wei98].

The eigenvector v1corresponding to the eigenvalue with greatest absolute value

(λ1) gives the orientation of highest grey value fluctuation. The other eigenvector

v2, which is orthogonal to v1, gives the preferred local orientation, the coherence

direction. Moreover, the term (λ1−λ2)2is a measure of the local coherence and

becomes large for anisotropic structures [Wei98].

The structure tensor Jρ is used to extract structure orientation for recon-

struction. Anisotropic confidence-based filling-in also makes use of the coherence

measure (λ1− λ2)2for structure dependent diffusion (Sect. 2.3).

2.2Structure Tensor Linear Interpolation

Specular areas are reconstructed by interpolation between boundary points along

the main orientation of the local structure, extracted by the structure tensor.

Extraction of Local Structure. Local orientation is given by the eigenvector v2

belonging to the minor eigenvalue λ2 of the structure tensor Jρ (Sect. 2.1),

which is required for every specular pixel. To detect structure in the given heart

images appropriate values for noise and integration scale are σ = 1 and ρ = 2.8,

corresponding to a catchment area of about 9 × 9 pixels.

Reconstruction. For each point inside the specular area, search for the two

boundary points along the structure orientation associated to it. Linear interpo-

lation between the intensities of the boundary points, weighted according to their

relative distances, yields the new value at the current position. Final low-pass

filtering ensures sufficient smoothness in the reconstructed area.

2.3 Anisotropic Confidence-Based Filling-In

This reconstruction scheme fills specular areas from the boundary based on local

structure information. It employs coherence enhancing anisotropic diffusion.

Coherence Enhancing Anisotropic Diffusion. Diffusion is generally con-

ceived as a physical process that equilibrates concentration without creating or

destroying mass. Applied to images, intensity at a certain location is identified

with concentration. Thus the diffusion process implies smoothing of peaks and

sharp changes of intensity, where the image gradient is strong.

The discussed type of diffusion, designed to enhance the coherence of struc-

tures [Wei98], is anisotropic and inhomogeneous. Inhomogeneous means that

the strength of diffusion depends on the current image position, e.g. the ab-

solute value of the gradient ∇f. Further, anisotropic diffusion corresponds to

non-uniform smoothing, e.g. directed along structures. Thus edges can not only

be preserved but even enhanced.

The diffusion tensor D = D(Jρ(∇fσ)), needed to specify anisotropy, has the

same set of eigenvectors v1,v2 as the structure tensor Jρ (Sect. 2.1), reflect-

ing local image structure. Its eigenvalues λ?

1,λ?

2are chosen to enhance coherent

Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer55

Page 4

structures, which implies a smoothing preference along the coherence direction

v2with diffusivity λ?

?α

2increasing with respect to the coherence (λ1− λ2)2of Jρ:

λ?

1

def

= α;

λ?

2

def

=

if λ1= λ2,

otherwise

α + (1 − α)exp

?

−C

(λ1−λ2)2m

?

(2)

where C > 0,m ∈ N,α ∈]0,1[, and the exponential function ensures the smooth-

ness of D. For homogeneous regions, α specifies the strength of diffusion. We

follow [Wei98] with C = 1, m = 1, and α = 0.001.

Anisotropic Confidence-Based Filling-In. A filling-in scheme for structure

driven reconstruction is developed, employing coherence enhancing anisotropic

diffusion. The algorithm is an extension of the confidence-based filling-in model

of Neumann and Pessoa [NP98], and is described by the following equation:

Definition 2 (Anisotropic confidence-based filling-in). For an image f,

evolving over time as ft, anisotropic confidence-based filling-in is given by

∂ft

∂t

= (1 − c)div(D(Jρ(∇f0))∇ft) + c(f0− ft)(3)

where f0is the initial image and c : dom(f) → [0,1] is a confidence measure.

In the present work, c maps to {0,1}, where c = 0 refers to unreliable image

information, i.e. specularities, and non-specular image points provide reliable

information (c = 1). Therefore non-specular regions are not modified, while

specular points are processed according to

∂ft

∂t

= div(D(Jρ(∇f0))∇ft) .

(4)

Algorithm. Linear diffusion equations like (4) are commonly solved by relaxation

methods. Intensity from boundary pixels is propagated into the specular area,

or vice versa, specular intensity is drained from it (source–sink model). In the

current implementation reconstruction calculates the dynamic changes of diffu-

sion over time in an iterative approach. In each step diffusion is represented by

convolution with time-invariant kernels, shaped according to the diffusion tensor

as in [Wei98]. The algorithm runs until the relative change of specular intensity

is sufficiently small.

As intensities at the boundary are constant, diffusion corresponds to inter-

polation between boundary points. Filling-In is related to regularization theory

[TA77] in employing a smoothness and a data-related term. The filling-in model

discussed here incorporates a linear diffusion scheme. Local structure informa-

tion for the diffusion process is extracted only from the initial image, which

allows efficient implementation of the filling-in process.

56Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer

Page 5

3Evaluation

Reconstruction is required to meet the following criteria:

(R0) Restriction to specular areas

(R1) Smooth reconstruction inside specular areas

(R2) Smooth transition to boundaries

(R3) Continuity of local structures

(A1) Realtime operability

As a basic requirement only specular areas shall be altered (R0). To avoid new

artefacts, e.g. edges, smoothness is assessed both inside specular areas (R1)

and at the boundary (R2). Structures next to specularities should be continued

within reconstructed areas (R3), because uniform filling-in from the boundary

may cause new artefacts by interrupted structures. Criterion (A1) is kept in mind

to enable realtime application, as required for tracking in a stream of images.

3.1Reconstruction at Specularities on the Heart Surface

First, quality of reconstruction within structured areas is measured by introduc-

ing an artificial disturbance on a horizontal edge, i.e. a 5 × 5 pixel square with

intensity similar to specular reflection (Fig. 2, left). The other two images show

reconstruction by structure tensor linear interpolation and anisotropic filling-in:

The artificial specularity vanishes and original structure is restored.

Since the underlying structure is known, reconstructed areas of both methods

can be compared with the original area by the sum of squared differences (SSD),

a similarity measure also commonly used for tracking: With intensities between 0

and 255, SSD is 872 for structure tensor interpolation versus 553 for anisotropic

filling-in, which is slightly superior as intensities are closer to the original image.

Fig.2. Elimination of artificial specularity (left to right: original, dto. with artificial

specularity (bright square), reconstruction by linear interpolation, and by anisotropic

filling-in)

Secondly, reconstruction is considered with real specularities on the heart

surface for the area of interest in the original image (Fig. 1). Figures 3 and 4

show reconstruction by linear interpolation and anisotropic filling-in.

Proc. DAGM 2001, LNCS 2191 c ? 2001 Springer57