Content uploaded by Aaron Mavrinac

Author content

All content in this area was uploaded by Aaron Mavrinac on Sep 20, 2017

Content may be subject to copyright.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEE/ASME TRANSACTIONS ON MECHATRONICS 1

Semiautomatic Model-Based View Planning for

Active Triangulation 3-D Inspection Systems

Aaron Mavrinac, Xiang Chen, and Jose Luis Alarcon-Herrera

Abstract—A semiautomatic model-based approach to the view

planning problem for high-resolution active triangulation 3-D in-

spection systems is presented. First, a comprehensive, general,

high-ﬁdelity model of such systems is developed for the evaluation

of conﬁgurations with respect to a model of task requirements, with

a bounded scalar performance metric. The design process is ana-

lyzed, and the automated view planning problem is formulated only

for the critically difﬁcult aspects of design. A particle swarm opti-

mization algorithm is applied to the latter portion, including proba-

bilistic modeling of positioning error, using the performance metric

as an objective function. The process leverages human strengths

for the high-level design, reﬁnes low-level details mechanically, and

provides an absolute measure of task-speciﬁc performance of the

resulting design speciﬁcation. The system model is validated, allow-

ing for a reliable rapid design cycle entirely in simulation. Param-

eterization of the optimization algorithm is analyzed and explored

empirically for performance.

Index Terms—Inspection, laser scanner, range camera, view

planning.

I. INTRODUCTION

OPTICAL range sensors based on active triangulation [1]

are popular in industrial applications of noncontact 3-D

metrology and inspection, owing to high ﬁdelity and robustness

in comparison to other noncontact methods, and high data ac-

quisition rates in comparison to mechanical methods. A typical

inspection task involves measuring the surface of an object at

resolutions on the order of tens of micrometers. The speciﬁed

tolerances often run within an order of magnitude of the physi-

cal capabilities of modern devices, when factors such as ﬁeld of

view and occlusion are taken into account. Hence, planning the

conﬁguration of the sensor system is not trivial, requiring care-

ful balancing of a number of constraints to achieve an acceptable

solution.

In industry, typically the end user of the inspection system

delegates this difﬁcult task to a third-party specialist integrator,

who will assemble a custom solution starting from modern off-

the-shelf range cameras [2] and software. The main expertise

is in designing a physical conﬁguration capable of meeting the

client’s speciﬁcations, usually assisted by planning guidelines

provided by the range camera manufacturer. Considering even

Manuscript received May 10, 2012; revised January 23, 2013, September

1, 2013, and February 2, 2014; accepted March 13, 2014. Recommended by

TechnicalEditor Z. Zhu. This work was supported in part by the Natural Sciences

and Engineering Research Council of Canada.

A. Mavrinac is with the Robert Bosch LLC Chassis Systems Control Division,

Plymouth, MI 48170 USA (e-mail: aaron.mavrinac@us.bosch.com).

X. Chen and J. L. Alarcon-Herrera are with the Department of Electrical

and Computer Engineering, University of Windsor, Windsor, ON N9B 3P4,

Canada (e-mail: xchen@uwindsor.ca; alarconj@uwindsor.ca).

Color versions of one or more of the ﬁgures in this paper are available online

at http://ieeexplore.ieee.org

Digital Object Identiﬁer 10.1109/TMECH.2014.2318729

a small subset of factors is very labor intensive. Additionally,

the remaining factors often have subtle and poorly understood

effects that can render the result unsuitable. To compensate, the

integrator often adds a ﬁxed error factor to the client speciﬁca-

tions. However, this can greatly increase the cost of the solution

(or preclude one altogether).

Automated view planning holds promise of efﬁciency and

reliability, yet such methods are not currently in use by integra-

tors. Scott et al. [3] present a survey of view planning, including

a detailed analysis of the constraints and requirements. While

some are of interest mainly in reconstruction and do not apply

to inspection, there is an evident gap between the remaining

points and the state of the art. The authors conclude that over-

simpliﬁcation of the problem leads to unrealistic solutions and

is the main barrier to adoption, and suggest that semiautomatic

methods may yield tangible advances in the near term.

We propose such a semiautomatic design process, leveraging

the strengths of both human expertise and computational efﬁ-

ciency. Our approach addresses two key issues. First, we derive

a high-ﬁdelity model of multicamera bistatic coverage of active

triangulation systems, based on low-level criteria in the vein of

classic sensor planning models [4]. The task and system models

(Sections II and III, respectively) are very general, and directly

reﬂect requirements and conﬁguration parameters. We provide

a bounded scalar performance metric which predicts online per-

formance (see Section IV), allowing costly and time-consuming

hardware implementation to be replaced with simulation in the

design loop. Second, we reduce the scope of the automatic plan-

ning problem to only the critical portion that presents difﬁculty

to human designers or expert-based heuristics (see Section V).

The reduced problem is then addressed using a particle swarm

optimization (PSO) technique (see Section VI). Experimental

results demonstrate the validity of the model, the optimality and

convergence properties of the optimization algorithm, and the

robustness of solutions to positioning error (see Section VII).

A. Prior Work

For a comprehensive account of work on view planning for

3-D reconstruction and inspection with active triangulation

range sensors up to 2003, the reader is referred to the survey

by Scott et al. [3]. The authors note that most approaches suffer

from oversimpliﬁcations of the problem, including modeling

the bistatic system as a monostatic sensor, placing unrealistic

constraints on viewpoints, and ignoring several signiﬁcant low-

level criteria. They conclude that no effective general-purpose

solution exists.

While several new methods targeted at reconstruction have

been put forward since (e.g., by Larsson and Kjellander [5] and

by Scott himself [6]) these attempt to solve the next best view

1083-4435 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.

See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2IEEE/ASME TRANSACTIONS ON MECHATRONICS

problem for unknown object surface reconstruction. Even recent

methods aimed at inspection (e.g., Yang and Ciarallo [7], Shi

et al. [8], Rodrigues Martins et al. [9], Englot and Hover [10]),

assuming aprioriknowledge of object geometry, still tend to fo-

cus on path planning for total surface inspection. Our scope does

not include registration-based inspection, so these concerns are

irrelevant. We are unaware of any work exclusively addressing

the planning of static multicamera active triangulation systems.

Some previous model-based approaches have been put for-

ward. Of note is the work of Tarbox and Gottschlich [11], who

employ a volumetric coverage model of a bistatic active trian-

gulation sensor. Scott [6] proposes an improved model making

fewer assumptions about the object; his concept of veriﬁed mea-

surability is similar to our bounded performance metric. Alarcon

Herrera et al. [12] present an initial analysis of the viewpoint

evaluation component using an early formulation of the cover-

age model of Section IV.

Scott et al. [3] observe that the sensor planning literature lacks

clear, explicit characterization of task requirements. Cowan and

Kovesi [13] and Tarabanis et al. [14] touch on quantifying the

coverage of speciﬁc subsets (features) of the target surface with

respect to performance requirements. Scott [6] explicitly models

the task with quantitative requirements for measurement preci-

sion and sampling density, ending with quantitative performance

metrics, but stops at specifying a uniform requirement over the

entire target surface, though subset regions and heterogeneous

criteria are mentioned.

In approaching the view planning problem, researchers have

repeatedly attempted to model visual coverage in forms conve-

nient for various optimization techniques. These models have

invariably suffered from a loss of ﬁdelity, resulting in poor solu-

tions in general or restricting the scope of applicability [3], [15].

Rather than forcing the model to ﬁt the optimization algorithm,

we opt instead to develop a relatively high-ﬁdelity model with

only a general consideration for optimization, then choose and

parameterize an optimization algorithm based on the nature of

the objective function.

B. Application Scope

Most industrial inspection tasks can be addressed by a basic

single-pass linear or rotary range scan conﬁguration, or can be

decomposed into several subtasks of this form. This type of so-

lution entirely avoids online registration, which is prohibitively

expensive for online scanning rates in most applications [16].

Such a conﬁguration includes exactly one line-projection light

source (usually a laser) and one or more range cameras. Conﬁg-

urations with single cameras are well-studied; multiple-camera

coﬁgurations are less common, but are beginning to ﬁnd use

in industry. We assume that multiple-camera conﬁgurations are

mutually calibrated: see treatments by Saint-Marc et al. [17],

Vilac¸a et al. [18], and Mavrinac et al. [19].

The objective is to design a system conﬁguration which

achieves the requirements of the task. This conﬁguration is static

in the sense that laser and camera parameters are ﬁxed, and

the target object undergoes deterministic linear or rotary mo-

tion through the system. The design parameters are the poses

(3-D position and orientation) and respective properties of the

laser and cameras, as well as the number of cameras. Some

of these parameters will be ﬁxed or constrained during the

manual portion of design, and the remainder will be optimized

automatically.

Certain sensible restrictions are placed on the geometry of

the system, based on the operating characteristics of the range

cameras. We deﬁne a right-handed Cartesian coordinate system

of R3with axes x,y, and z; the origin is arbitrary, and the basis

depends on the motion of the object being scanned. In the case of

linear motion, the y-axis is the transport direction, and in the case

of rotary motion, the axis of rotation is parallel to the x-axis. The

laser position is unconstrained, but the laser plane—the plane

in which all projected laser lines lie—must be parallel to the

x-zplane.1Similarly, camera positions are unconstrained, but

they may only rotate about the x-axis, as rotations about yand z

generally violate the internal assumptions of the sensor design.

This rotation is subject to the requirement that the (positive)

optical axis intersect the laser plane. The initial pose of the

target object is arbitrary.

As is standard in inspection applications, we assume that

the geometry of the object to be inspected is known; gener-

ally, clients will provide a CAD model of the part along with

the speciﬁcation. Issues relating to the object material are not

considered. We assume that the speciﬁcation indicates which

portions of the surface are to be inspected, with tolerance values

for each.

C. Constraints and Requirements

Scott et al. [3] identify a set of 16 qualitative constraints

and requirements a complete view planning system should con-

sider. We refer the reader to the survey for details on previous

approaches.

1) Model Quality Speciﬁcation: We provide a parameteriz-

able task model (see Section II) which allows the task require-

ments to be speciﬁed precisely in the language of functional

requirements [20].

2) Generalizable Algorithm: Our system model (see

Section III) is general over the class of active triangulation

vision systems used in inspection applications. The model is

readily extensible to a broader range of sensor classes [21].

3) Generalized Viewpoints: All system parameters are in-

corporated into the viewpoint construct of the system model in

Section III.

4) View Overlap: As our scope is limited to single-scan

inspection applications, view overlap for registration is not

considered.

5) Robust: The view planning algorithm presented in Sec-

tion VI is robust to sensor and positioning system noise (see

Section VI-C).

6) Efﬁcient: Our experimental results in Section VII indicate

that complex view planning tasks are completed in a matter

of seconds or minutes, quite competitive with skilled human

engineers.

1Many systems allow rotation of the laser about the x-axis. The resulting

oblique projection adds complexity to the process of range mapping. Since it

is possible to solve most industrial inspection problems with the laser plane

perpendicular to the transport direction, we restrict our analysis to this case.

However, it is certainly possible to generalize the approach to handle oblique

laser conﬁgurations.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 3

7) Self-Terminating: The algorithm presented in Section VI

incorporates three separate termination conditions. Addition-

ally, since the performance function developed in Section IV is

bounded, termination can be precisely controlled with respect

to task requirements.

8) Limited a Priori Knowledge: The effectiveness of the al-

gorithm in view planning for a task is dependent on the quality

of the apriorimodel of the target object provided; since our

scope is not a next best view problem, no feedback may be used.

However, it can provide a useful general solution given only ap-

proximate bounding dimensions (e.g., as in Section VII-E).

9) Shape Constraints: The algorithm is effective for any rea-

sonable object geometry and topology.

10) Material Constraints: It is assumed that issues relating

to the target material (e.g., choice of laser and camera, incidence

angle constraints, etc.) have been accounted for by the human

designer.

11) Frustum: Our camera model (see Section IV-B) is a

high-ﬁdelity representation of lens and sensor effects [21].

12) Shadow Effect: The bistatic nature of the laser

and camera is explicitly considered by the functions in

Section IV.

13) Measurement Performance: Measurement performance

is evaluated over the entire high-ﬁdelity model against the task

model, and summarized in a single bounded performance metric.

14) 6-D Pose: While our system model is capable of model-

ing unconstrained 3-D pose, the sensors are operationally lim-

ited to four degrees of freedom, as described in Section V-B.

15) Pose Constraints: Constraints on the range of the posi-

tioning system are modeled and respected by the algorithm.

16) Positioning System Performance: Pose error is explicitly

modeled by the algorithm, as described in Section VI-C. As

target conﬁgurations are static, no online reconﬁguration time

is considered.

II. TASK MODEL

A. Object Geometry

The target object geometry is represented in the task model as

a triangulated polygonal mesh of (opaque) surfaces; the triangles

form a set TT. It is assumed that some means exists of obtaining

such a mesh apriori, at some degree of ﬁdelity, e.g., from a

CAD model.

We deﬁne a function intersect(Δ,p,q), which returns the

point of intersection between line segment pq and triangle Δ∈

TT,or∅if none exists. M¨

oller and Trumbore [22] present an

efﬁcient means of computing this intersection.

B. Task Points

An inspection task speciﬁes a set of points TP⊂R3on the

surface of the object; formally, ∃Δ∈T

T|t∈Δ∀t∈T

P. Each

task point therefore has an associated surface normal, which is

the normal of the triangle in TTon which it lies. If continuous

areas are speciﬁed (e.g., the entire surface), these may be dis-

cretized according to the scanning density requirement without

loss of information.

TAB L E I

TASK PARAMETERS

Parameter Descriptiona

Ri,Raminimum ideal/acceptable scan density (l/px)

Hi,Haminimum ideal/acceptable height resolution (l/px)

ci,camaximum ideal/acceptable blur circle (px)

ζi,ζamaximum ideal/acceptable view angle (a)

ωcincidence angle cutoff (a)

γimage boundary padding (px)

aNumerical parameters are deﬁned in units of length (l), pixel units

(px), and/or angle units (a).

C. Task Requirements

The task requirements are captured in a set of parameters,

listed in Table I. These directly describe criteria commonly

speciﬁed by sensor components, software algorithms, and/or the

application as operating requirements or tolerances, and should

be familiar to the experienced designer. The ﬁrst four are divided

into “ideal” and “acceptable” values: the latter should be set to

the limit at which the task produces no useful information, and

the former should be set to the point beyond which the task

performance does not increase.

The scan density is the horizontal resolution of the range

image, and can be thought of as a requirement on the “instan-

taneous” resolution at task point t. The height resolution is the

precision to which range values are measurable at t; this is gen-

erally the most important requirement in an inspection task (see

the discussion of quantization error by Yang et al. [23]). The

blur circle requirement is derived from the dependence of the

proﬁle interpolation algorithm on focus, usually discussed in

the supplier’s documentation. The view angle requirement de-

rives from the sensor’s ability to image a proﬁle, which varies

inversely with the cosine of view angle relative to the laser pro-

jection, and depends on the laser, sensor, and material properties.

The incidence angle cutoff is the angle beyond which the light

received by the camera drops below a threshold [6]. Finally,

the image boundary padding requirement degrades coverage at

the edges of the image, a simple way of accounting for effects

such as residual lens distortion, proﬁle interpolation bias, and

positioning error.

III. SYSTEM MODEL

A. Laser

Within the laser’s local coordinate system, we model the ef-

fective sheet projection as an isosceles triangle in the x–zplane,

with top angle λ(the so-called fan angle), and base parallel to

the x-axis at distance zP(the maximum effective depth) from

the top vertex (see Figs. 1 and 2).

Given the laser’s local coordinate system with origin at the

projection source and negative z-axis along the axis of projec-

tion, the laser has pose PLwhich, as discussed in Section I-B,

is restricted to translation by a vector TL(Table II).

B. Camera

We model each camera geometrically according to the pinhole

model [24], with additional parameters related to the physical

sensor and optics necessary for the full coverage model de-

4IEEE/ASME TRANSACTIONS ON MECHATRONICS

Fig. 1. Scanning by active triangulation—A 3-D model of the surface of the

object is obtained by measuring along a series of cross sections of the object,

yielding a series of 2-D proﬁles which are combined into a 3-D point cloud

according to the transport pitch.

Fig. 2. World coordinate system—The laser coordinate system is parallel to

(and usually coincides with) the x–zplane, while the coordinate systems of the

cameras are determined by their respective poses.

TAB L E I I

LASER PARAMETERS

Parameter Descriptiona

λfan angle (a)

zPprojection depth (l)

TL=TL,x,T

L,y,T

L,zposition (l)

scribed in Section IV-B. According to this model, the projected

image qof a 3-D point pis

q=f

su·px

pz

+ru,f

sv·py

pz

+rvT

(1)

where the horizontal and vertical pixel dimensions suand sv,

respectively, the effective focal length f, in units of length, and

the image of the principal point r=(ru,r

v), in pixel coordi-

nates, are determined by calibration. The actual image extends

from the origin (top-left corner of the image) to (w, h), where

wand hare the width and height (resolution) of the sensor,

respectively (Fig. 3).

Fig. 3. Pixel coordinates—The origin of the image in pixel coordinates is at

the top left corner of the image. Pixel distances are related to units of length by

the physical pixel dimensions suand sv.

TABLE III

CAMERA PARAMETERS

Parameter Descriptiona

w,hwidth and height of the sensor (px)

su,svphysical pixel dimensions (l/px)

feffective focal length (l)

r=ru,r

vcoordinates of the principal point (px)

Aeffective aperture diameter (l)

zSsubject distance (l)

TC=TC,x,T

C,y,T

C,zposition (l)

θrotation about x-axis (a)

ψrotation about z-axis (a)

We assume that lens distortion is corrected, e.g., according

to Brown’s model [25], and that residual error is handled as

described in Section IV-B1. The effective aperture size Ais

signiﬁcant, however, and can be determined from the f-number

setting on the lens.

For a given lens, the intrinsic parameters f,r, and Avary

depending on lens parameters, which include focus, zoom, and

physical aperture settings. The relationship is nonlinear and

complex, but is possible to estimate for a given lens by perform-

ing a series of calibrations with varying settings, and interpolat-

ing [26]. We assume that zoom and physical aperture are ﬁxed,

and that only the focus setting varies; we construct the lookup

table for f,r, and Aby varying the subject distance zSdirectly.

Given the camera’s local coordinate system with origin at the

principal point and positive z-axis coinciding with the optical

axis, a camera has pose PCi, comprising translation by a vector

TCi, rotation about the x-axis by an angle θCi∈[π

2,π), and

rotation about the z-axis by an angle ψCi∈{0,π}such that the

optical axis intersects the laser plane (Table III).

C. Static Environment Geometry

For purposes of occlusion, deterministic objects in the en-

vironment may be modeled similarly to the target object in

Section II-A, with a set of triangles ST. Again, it is assumed

that the mesh is available a priori. The intersect function applies

here as well.

IV. VIEW EVALUATION

A. Laser Coverage Function

The laser coverage function is a mapping L:R3→{0,1},

with

L(p)=LP(p)·LI(p)·LO(p)(2)

comprising three components.

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 5

Fig. 4. Field of view—The quadrilateral pyramid extending along the optical

axis to inﬁnity (shown truncated), deﬁned by the apex angles αhand αv.

1) Projection: The task point is covered only if it is within

the triangle of projection

LP(p)=⎧

⎨

⎩

1,if arctan |px|

−pz≤λand −pz≤zP

0,otherwise. (3)

2) Incidence Angle: The task point is covered only if the

angle of incidence within the laser plane is less than the cutoff

angle

LI(p)=1,if arccos (ˆ

u·ˆ

n)≤ωc

0,otherwise (4)

where ˆ

uis a unit vector in the direction of u=p−TL, and ˆ

n

is the surface normal at p.

3) Occlusion: The task point is covered only if the ray from

the laser source to the point is unobstructed

LO(p)=

1,if intersect (Δ,TL,p)∈{∅,p}∀Δ∈T

T∪S

T

0,otherwise. (5)

B. Camera Coverage Function

The camera coverage function is a mapping C:R3→[0,1],

with

C(p)=CV(p)·CD(p)·CH(p)·CF(p)·CA(p)·CO(p).

(6)

In deﬁning the components of the camera coverage function,

we use a bounding function B[0,1](x) = min(max(x, 0),1) to

limit xto the range [0,1], simplifying the formulation.

1) Field of View: The ﬁeld of view is a pyramid with apex at

the camera’s optical center and rectangular base at inﬁnity along

the optical axis, induced by the projective geometry and the

limits of the image sensor (see Fig. 4). The task point is visible

(barring occlusion) if it is within this pyramid. The horizontal

and vertical ﬁeld of view criteria are given, respectively, by

CVh

(p)=B[0,1] ⎛

⎝min px

pz+ tan αl,tan αr−px

pz

γh⎞

⎠(7)

CVv

(p)=B[0,1] ⎛

⎝min py

pz+ tan αt,tan αb−py

pz

γv⎞

⎠(8)

where αland αrare the horizontal ﬁeld of view angles, and αt

and αbare the vertical ﬁeld-of-view angles, as given by

αl= arctan rusu

f(9)

αr= arctan (w−ru)su

f(10)

αt= arctan rvsv

f(11)

αb= arctan (h−rv)sv

f(12)

and, for γ>0,γhand γvare given by

γh=γ

w(tan αl+ tan αr)(13)

γv=γ

h(tan αt+ tan αb).(14)

The full ﬁeld of view criterion is then given by

CV(p)=min(CVh

(p),CVv

(p)),if pz>0

0,otherwise. (15)

2) Scanning Density (Resolution): The scanning density

along the ydirection is entirely determined by the transport

pitch, and we assume that this is designed appropriately for the

requirements. The scanning density along the xdirection is a

function of the resolution in units of length per pixel, and the

criterion is given by

CD(p)=B[0,1] zD(Ra)−pz

zD(Ra)−zD(Ri)(16)

for Ri<R

a, where

zD(R)= Rw

tan αl+ tan αr

. (17)

Without the multiplier R, the RHS of (17) expresses the num-

ber of pixels subtended by a bundle of rays one unit of distance

wide along x,atazdepth of one unit of distance. Multiply-

ing by Ryields the maximum zdepth for which the speciﬁed

resolution is met.

3) Height Resolution: Height resolution is computed simi-

larly to the scanning density criterion, but is additionally depen-

dent on the view angle due to distortion of the proﬁle [6], [27].

The height resolution criterion is given by

CH(p)=B[0,1] zH(Ha,p)−pz

zH(Ha,p)−zH(Hi,p)(18)

with

zH(H, p)= Hhsin ζ(p)

tan αl+ tan αr

(19)

where ζ(p)is the view angle to p, as given by (23). (19) is

derived similarly to (17), with the additional term accounting

for the effect of the oblique angle of incidence. Our application

scope does not allow for rotation about z, so this term is always

unity in (17).

4) Focus: The quality of focus is a function of the blur circle

diameter of the projected task point (Fig. 5), obtained from

standard depth of ﬁeld equations from Gaussian optics, and the

criterion is given by

CF(p)=B[0,1] min pz−zn

z−zn

,zf−pz

zf−z (20)

6IEEE/ASME TRANSACTIONS ON MECHATRONICS

Fig. 5. Blur circle—A point at zSmaps to a point in the image; points closer

or farther away map to blur circles, with diameters proportional to depth.

for ca>c

i, where (z,z

)and (zn,z

f)are the near and far

limits of depth of ﬁeld as given by (21), substituting blur circle

diameters cimin(su,s

v)and camin(su,s

v), respectively, for c,

z=AfzS

Af ±c(zS−f).(21)

5) View Angle: The view angle with respect to the laser

projection axis also impacts camera coverage [6]. The criterion

is given by

CA(p)=B[0,1] cos ζ(p)−cos ζa

cos ζi−cos ζa(22)

with ζ(p), the view angle to p, computed as

ζ(p) = arccos P−1

C(ˆ

z)·ˆ

p(23)

where ˆ

zis a unit vector in the positive zdirection, and ˆ

pis a

unit vector in the direction of p.

This is a special case of a more complex criterion [21] for

generic view angle requirements from surface normals. In this

case, the normals are along the projection axis, so that the view

angle criterion expresses the obliqueness of the laser to the

camera.

6) Occlusion: The task point is covered only if the ray from

the point to the camera is unobstructed

CO(p)=

1,if intersect (Δ,p,TC)∈{∅,p}∀Δ∈T

T∪S

T

0,otherwise. (24)

C. Task Performance Metric

We require a coverage metric M(t)∈[0,1] for a task point

t∈T. According to the task speciﬁcation, a task point is an

arbitrary point in R3, but scanning takes place exclusively within

the laser plane. The transport motion of the target object carries

each task point to a new point t∈R3such that t

y=TL,y,from

which laser and camera coverage may then be computed.

For linear transport, this motion consists simply of translating

the target object by T=0,T

L,y −ty,0.

For rotary motion, the target object is rotated about an axis

parallel to the x-axis passing through 0,T

L,y,z

r.Fort∈T

P,

the angle of rotation is φr= arccos(ˆ

v·ˆ

z)(or 2π−φr, depend-

ing on the direction of rotation), where ˆ

vis a unit vector in the

direction of v=t−tx,T

L,y,z

r.

Armed with an appropriate means of obtaining the transported

task point t, the overall system coverage metric for a single

point is

M(t)=LP−1

L(t)·max

iCiP−1

Ci(t). (25)

Finally, given a task and system context, the performance

metric over all task points is

F=TPM(t)

|TP|. (26)

V. VIEW PLANNING

A. Manual Initialization

Based on the observation of the design process, the critical

difﬁculty is encountered in planning the position and orienta-

tion of the sensor: the six criteria in Section IV-B are competing

objectives over this space, and since focus settings must be ad-

justed as well, it can be very challenging to ﬁnd a conﬁguration

which meets them all simultaneously. Accordingly, the manual

initialization stage must determine many of the system param-

eters discussed in Section III:

1) initial pose of the target object;

2) transport style (linear or rotary);

3) axis of rotation, if applicable;

4) position and properties of the laser;

5) speciﬁc sensors and lenses to use; and

6) number (n) and ψ-orientations of cameras.

A complete discussion of the heuristic for designing these

aspects is beyond our scope, but we will comment further on

the justiﬁcation for this division. Items 1 to 3 are typically dic-

tated by ﬁxturing, physical constraints, and the locations of the

task points on the object surface; there is usually little to be

gained by attempting to automate their design. The calcula-

tions for an appropriate laser conﬁguration in item 4 are quite

straightforward: the x–zprojections of TPshould lie within

the triangle deﬁned by λ,zP, and TL, which is not normally

difﬁcult to achieve. The sensor and lens of item 5 are typically

chosen from a limited pool of available products. The zoom

and physical aperture settings of the lens require only coarse

tuning in this stage, since other codependent parameters—the

focus and standoff distance—will subsequently be optimized.

Finally, there is a small, discrete set of feasible possibilities for

the number and ψ-orientations of cameras in item 6, and even

if the best among them is not obvious to the designer, several

options may be tested.

B. Automatic Reﬁnement

In the automatic reﬁnement stage, all parameters of the system

are ﬁxed, except the cameras’ positions, their rotation about the

x-axis, and their focus setting (thus varying zS, and by extension

f,r, and A). Further, the focus setting is constrained such that

zSalong the optical axis (in camera coordinates) is always in

the laser plane.

Without loss of generality, we discuss the single-camera prob-

lem. For each camera, we vary four independent values equiv-

alent to the parametrization of a rigid transformation in the

y–zplane plus translation in x, but more representative of the

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 7

TAB L E I V

DESIGN IMPACT MAT RIX FO R AUTOMATIC REFINEMENT

xhdβ

Field of View •••◦

Scanning Density ◦•◦

Height Resolution ◦••

Focus ◦•◦

View Angle ◦◦◦•

Occlusion ◦•◦•

◦– minor impact, •– major impact

functional design of the system. These design variables are as

follows:

1) xthe horizontal position of the camera;

2) hheight (z) at which the optical axis intersects the laser

plane;

3) ddistance from TCto the point x, 0,h;

4) βangle between the z-axis and the optical axis.

From these variables, the position of the camera is computed

as

TC,x =x(27)

TC,y =TL,y ±dsin β(28)

TC,z =h+dcos β(29)

where the sign in (28) depends on whether the camera is down-

stream (positive) or upstream (negative) from the laser plane

along the transport axis. The angle of rotation is β+πabout

the negative or positive x-axis, respectively. The intrinsic pa-

rameters f,r, and Aare obtained via interpolation on the lens

lookup table, with zS=d.

A chart showing the impact of each design parameter on each

coverage criterion, based on the concept of the design matrix

[20], is shown in Table IV. The “minor impact” designation

indicates that modifying the parameter impacts the criterion

indirectly due to the ray from the point to the camera not coin-

ciding with the optical axis; these effects are normally relatively

minor, though still signiﬁcant, over the visible volume. The

criteria are strongly coupled with respect to the four design pa-

rameters (or any other possible set), and the nature of the bistatic

range sensor design—speciﬁcally, the 3-D coverage of points

within the 2-D laser plane—makes it impossible to decouple

them within the scope of available parameters. Furthermore, the

solution space is highly nonlinear and discontinuous in terms

of the performance metric F, which depends directly on the six

criteria. This is clearly an ill-posed problem for human design

and for direct numerical optimization.

For a n-camera conﬁguration, the solution space is

4n-dimensional: x,h,d, and βare varied independently for

each camera. F(s)denotes the objective function (26) with re-

spect to solution point s∈R4n, where system parameters are

calculated from sas described above. Given an initial manual

conﬁguration as prescribed in Section V-A, the problem is to

ﬁnd swhich maximizes F(s), subject to any design constraints

on s.

VI. AUTOMATIC REFINEMENT ALGORITHM

A. Particle Swarm Optimization

The view planning problem described in Section V-B con-

sists essentially of maximizing a complex, irregular, nondiffer-

entiable objective function deﬁned over a bounded, continuous

solution space. Among metaheuristics, PSO [28], [29] lends it-

self particularly well to this form of problem. The algorithm

itself has been published in various forms in numerous sources,

but in order to expose the particular structure and features of our

application, we describe it in full as used in Algorithm 1.

Each particle’s state at a given iteration is described by a

position si∈R4nand a velocity vi∈R4n. For simplicity of

exposure, where any position variable sis uninitialized in Al-

gorithm 1, F(s)=−∞. The lower and upper solution space

bounds, respectively, are given as vectors bland bu(values are

derived in Section VI-B). U(l,u)denotes a random uniform

vector bounded by land u.

8IEEE/ASME TRANSACTIONS ON MECHATRONICS

TAB L E V

BOUNDS ON DESIGN VARIABLES

Variable Lower Bound Upper Bound

xx

min xmax

hz

min zmax

dd

min du

β0ζa

The parameters S,ω,φp,φn, and κare endemic to the algo-

rithm; they denote the size (number of particles) of the swarm,

the particle momentum, the disposition toward the present best

local and neighborhood solutions, and the velocity clamping

factor, respectively. Appropriate values for these parameters are

discussed in Section VI-E.

The particle state also stores pi, the best solution found by

particle i, and ni, the best solution found by any particle in par-

ticle i’s neighborhood set Ni⊆{1,...,S}. The best solution

found by any particle i∈{1,...,S}is stored globally in g.

The function topology(S, i)returns a subset of {1,...,S},

describing a topological relationship between the particles. In

the original PSO algorithm, all particles are considered adjacent

(i.e., {1,...,S}is returned), but using other topologies can im-

prove performance in terms of convergence and optimality [30],

[31]. The function constrain(si,bl,bu)modiﬁes sidepending

on the solution space bounds. In the original PSO algorithm,

no constraints are applied (i.e., siis returned), but constraint

handling mechanisms can also improve performance [32]. Both

are discussed in Section VI-E, and our selections are deﬁned in

(31) and (32), respectively.

The positions and velocities of the particles are initialized

randomly (lines 2 to 6). Each iteration then consists of updating

the local, neighborhood, and global best solutions based on the

value of Fat each particle’s current position (lines 9 to 29), then

updating each particle’s position and velocity (lines 30 to 41).

The lines involving Fe, comprising the block in lines 16 through

23 and line 25, are a nonstandard feature related to positioning

error compensation, explained in Section VI-C.

The algorithm proceeds until one of three stop conditions

is met:the global best solution exceeds Faccept, a proportion

Cmax ∈(0,1] of particles cluster within of g(by Euclidean

distance in a solution space normalized on the bounds), or the

algorithm has run for Imax iterations.

B. Solution Space Bounds

We derive a set of conservative bounds based on an assump-

tion which appears to hold in practical, nonpathological cases.

If xmin,xmax,zmin, and zmax represent the x-zbounds of

the task point set, the proposed bounds on x,h,d, and βwill

not exclude any solution with M(t)>0for all t∈T

Pwhere

xmin ≤x≤xmax and zmin ≤h≤zmax (the assumption be-

ing that these inequalities hold for some optimal solution). The

bounds are listed in Table V.

The values for the bounds on x,h, and βhave already been

deﬁned; those for drequire explanation. Let dmin and dmax

represent, respectively, the minimum and maximum values of

zSin the lens lookup table. The upper bound is deﬁned as

du= min[zD(Ra),z

H(Ha,P−1

C(x, 0,h))] (30)

where the values of fand r—upon which zDand zHdepend,

per (17) and (19)—are obtained from the lens lookup table with

zS=dmax. The values of xand hare irrelevant as long as they

are in accord with TCfor the purposes of the calculation.

Design constraints on the conﬁguration of a camera, including

limits on lens settings and physical position, are modeled by

overriding the value of Fto −∞ for all solutions violating

the constraints, as proposed by Mendes [31]. These may take

arbitrary form, and are not necessarily restricted to rectangular

bounds in the x-h-d-βdesign space.

C. Compensating for Positioning Error

Due to limitations on the physical accuracy of positioning

the cameras and the target object ﬁxture, the true poses of these

entities will inevitably differ, to some extent, from the opti-

mized design [3], [23]. If the algorithm converges on a global

optimum s, and the actual position achieved in practice is se,

then F(se)≤F(s). The true practical goal is to maximize the

expected performance E[F(se)]. In cases where the global max-

imum of Fis nonunique, which is almost certainly the case if

F(s)=1.0, the algorithm may reach a solution near the “edge”

of a nonsingular optimal solution space region, and terminate.

By considering positioning error within the algorithm, it is pos-

sible to encourage convergence to a solution with better sur-

rounding solutions.

The relationship of pose error to the ultimate value of Fis

complex, and deriving a closed-form probabilistic model of cov-

erage is infeasible. Instead, we compensate by directly testing

the robustness of a promising candidate solution using a prob-

abilistic model of positioning error and Monte Carlo sampling.

Let Fe(s)represent the mean value of F(s), with the error model

applied to the cameras and target object, over nesamples.2The

essence of lines 16 through 23 and line 25 of Algorithm 1 is that

if F(si)for one or more iis at the global best for a given iter-

ation, the algorithm begins evaluating and ranking by Fe(si)as

well. The motivation for this conditional evaluation, as opposed

to simply incorporating it into the ﬁtness function, is that Feis

netimes as expensive to evaluate as F, and there is no reason to

evaluate it for points known to be nominally suboptimal. An em-

pirical analysis of the tradeoff between additional performance

cost and improvement in optimality with positioning error is

reported in Section VII-D.

Random error is introduced to the poses of the cameras and

target object according to the following model. For the cameras,

a position offset is applied to each coordinate sampled from a

zero-mean Gaussian distribution with standard deviation σCT,

and an orientation offset is applied as a rotation about the x-axis

by an angle sampled from a zero-mean Gaussian distribution

with standard deviation σCR. For the target object, a similar

position offset is applied, with the Gaussian distribution having

standard deviation σTT, and the orientation offset is applied as

a rotation about a random axis uniformly sampled over the unit

sphere, and with the Gaussian distribution for the angle having

standard deviation σTR.

2For completeness, if ne=0,Fe(s)=−∞ for all s∈R4n.

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 9

D. Computational Complexity of the Fitness Function

Since F(26) is evaluated repeatedly in Algorithm 1—at

least Stimes per iteration and, in the worst case, (ne+1)S

times per iteration—it is necessary to consider its computational

complexity.

The fact that all of the laser parameters are static allows for

caching of the laser plane points and the results of the laser cover-

age function L(2). The camera coverage function C(6) must be

evaluated for each point t∈T

P, for each camera i∈{0,...,n}.

The critical component is CO(24): each triangle in TT∪S

Tin-

duces an evaluation of intersect. Some of STmaybeculled

for each new camera conﬁguration, but in general the entire set

are candidates. Therefore, the computational complexity is in

O(n×|T

P|×|T

T∪S

T|).

In practice, nis quite low (usually 1 or 2), but it is not

unreasonable for |TP|and |TT∪S

T|each to be on the order

of thousands for a high-ﬁdelity model of an inspection task. In

our implementation, a single evaluation of Fin a very large

model can take on the order of several seconds on a modern

workstation platform.3

E. Optimality and Convergence

Trelea [33] analyzes optimality and convergence of the PSO

algorithm using results from dynamic system theory, and de-

rives guidelines for parameter selection. We seek parameters

for which the algorithm will reliably converge to a global opti-

mum within a ﬁnite number of evaluations of F.

It is ﬁrst necessary to consider the behavior of the ﬁtness

function over the solution space. For a given camera position,

the subset of R3for which CO=1 is, in general, bounded

by an arbitrarily complex polyhedron, depending on TT. Since

intersect is symmetric over the endpoints of the line segment,

the same can be said of the subset of possible TCfor which

CO(p)=1, for some given p∈R3(Tarabanis et al. [14] com-

pute this subset as part of their sensor planning approach, and the

complexity is evident). At such boundaries, due to the bivalent

nature of CO,F(s)is, in general, discontinuous, and this effect

produces numerous local maxima dispersed in a nonuniform

fashion throughout the solution space.

Avoiding global convergence to one of these local maxima

suggests the use of a loosely connected topology [30]. We

choose a ring topology, in which each particle has exactly two

neighbors in addition to itself, deﬁned by

topology(S, i)=⎧

⎪

⎨

⎪

⎩

{S, 1,2},if i=1

{S−1,S,1},if i=S

{i−1,i,i+1},otherwise.

(31)

A large Simproves optimality, but requires more evalua-

tions of Fper iteration, and reduces the global convergence rate

[33]; we have found values near S=25 to perform acceptably

well while converging to the global optimum for various tasks.

Clamping velocity with κ=0.5 and setting φn>φ

pincrease

3Our implementation is written primarily in Python, which incurs signiﬁcant

interpreter overhead, even with the critical loop for intersectimplemented as an

efﬁcient C extension. Additionally, the implementation of intersect and related

code is highly tractable for SIMD parallelization.

Fig. 6. Simulated validation conﬁguration—Dual SICK-IVP Ranger D cam-

eras with x=0 mm, h=142 mm, d=206 mm, and β=π/4image a struc-

tured target of toy bricks.

the convergence rate without overly damping exploration. Val-

ues of ω=0.7298, φp=1.1959, and φn=1.7959 yield a good

tradeoff [34].

In selecting a boundary constraint handling mechanism, we

consider that the solution space is relatively low-dimensional,

and that there is no particular expected large-scale structure of F.

The nearest method [32] improves convergence by keeping the

particles within the subspace assumed to contain the optimum,

and is deﬁned by

constrain (s,bl,bu)=

4n

i=1

min(max(si,b

l,i),b

u,i)ˆıi(32)

where ˆıiis the ith unit vector of the solution space basis.

VII. EXPERIMENTAL RESULTS

In both simulation and real-world experimentation, we em-

ploy the Adolphus4simulation environment, which implements

the model and coverage functions described in Sections II–IV.

Thirty black toy bricks are used to rapidly construct physical

test targets for linear scans, in a diversity of shapes with known

structures. Individual bricks are modeled in Adolphus with 10

triangles per brick for occlusion, for a maximum total of 300

triangles in each target model. Various sets of between 12 and

72 task points are deﬁned on the surface of these models. Task

requirements are varied, for both single- and dual-camera con-

ﬁgurations, to cover several cases of existence and abundance

of F=1.0 global optima for each task.

A. Validation of the Performance Metric

Comprehensive validation experiments on coverage models

including that presented in Section IV have produced exten-

sive results [21] which exceed the scope of this correspondence.

Nevertheless, as our approach fundamentally relies on the accu-

racy of these models, we summarize the more important results,

viz. those relating to occlusion and scan density. System model

parameters are obtained by calibration, so positioning error is

not considered.

The combined laser and camera occlusion, derived from the

formulation of (25) as

MO(t)=LOP−1

L(t)·max

iCO

iP−1

Ci(t)(33)

4Adolphus is free software licensed under the GNU General Public License.

Full Python source code, documentation, and sample models are available at

http://github.com/ezod/adolphus.

10 IEEE/ASME TRANSACTIONS ON MECHATRONICS

Fig. 7. Structured target scan—Surface scan of the target obtained using

the real implementation of the system in Fig. 6. Visualized using PCL [35].

(a) Simulated target. (b) Scanned target.

Fig. 8. Validation of CD—The black line indicates the nominal Rgiven by

the relation (17), and red crosses indicate actual mean scan density in the point

cloud in the neighborhoods of the nominal task points.

may be validated against real data simply by verifying that

MO(t)is an indicator function for the existence of data at t.

Experiments on structured targets demonstrate perfect correla-

tion throughout the imaged laser plane. For the object shown in

Fig. 7, of 48 task points located on the studs of the toy bricks,

six are occluded to the laser, and a further six are occluded to

the cameras. The 36 remaining points yielding MO(t)=1are

fully imaged in the scan, while those yielding MO(t)=0are

absent from the scan.

The characterization of scanning density is validated by solv-

ing for Rin (17) for pzof a task point in camera coordinates

pand comparing to the actual horizontal density of data in

the immediate neighborhood of pin the scan. The visible sub-

set of task points on the object shown in Fig. 7 is tested in

this way at four calibrated positions of a single range camera.

Fig. 8 shows the strong correlation; overall mean squared error

is 6.344 ×10−5mm/px.

B. Comparison With Existing Methods

We compare our coverage metric and PSO algorithm with

Scott’s metric and 3M algorithm, the latter being implemented

according to the description provided in [6]. Before introducing

the experimental setup and presenting the results, we provide

some discussion on the compatibility of the coverage and task

models of both algorithms.

1) Coverage Model: Scott’s visibility criterion is equivalent

to ours, with γ=0. The focus criterion is also present in both

models; as Scott provides no details about its implementation,

we use our own implementation. Scott’s resolution criterion is

equivalent to our sampling density criterion. There is no equiv-

alent to our height resolution criterion: Scott assumes that the

TAB L E V I

COMPARISON TASK PARAMETERS,SOLUTIONS,AND PERFORMANCE

Task 3M-1 3M-2 PSO-1 PSO-2

Ri1.0 1.0 1.0 1.0

Ra1.0 1.0 1.0 1.0

Hi0.2 3.0 0.2 3.0

Ha0.2 3.0 0.2 3.0

ci1.0 1.0 1.0 1.0

ca6.0 6.0 6.0 6.0

x0.00 0.00 0.00 0.00

y353.52 353.52 38.00 330.00

z353.52 353.52 460.00 54.00

β45.00◦45.00◦85.27◦9.29◦

R0.31 0.31 0.2 0.25

H0.43 0.43 0.2 2.89

emean 0.1418 0.1418 0.0901 0.1000

%occlusion 18.7 18.7 31.17 13.7

camera and laser are ﬁxed relative to one another. Scott’s mea-

surement precision criterion is equivalent to our view angle cri-

terion. Both models provide a performance metric in the [0,1]

range; however, 3M employs a binary metric, so we limit our

PSO algorithm to a binary metric as well.

2) Task Model: Given the aforementioned equivalences, we

adapt the 3M algorithm to our own nomenclature for task param-

eters. Since Scott only deﬁnes two task parameters, the equiv-

alents are as follows: Ri=Ra=ρs, where ρsis the sampling

density task parameter, and ζi=ζa=σs, where σsis the mea-

surement precision task parameter.

Using toy bricks, we construct a test object for inspection

(see Fig. 10). The object is made of a pair of “staircases” of

four 9.6 mm steps each. The second set is positioned directly

behind the ﬁrst, introducing occlusion. We design two tasks

for this comparison. The ﬁrst imposes a strict height resolution

requirement, while the second relaxes this height resolution

requirement (thereby shifting the emphasis to other factors such

as occlusion).

Camera conﬁguration solutions are produced by both algo-

rithms for each task (see Table VI). Each conﬁguration is im-

plemented and a 3-D model of the test object is obtained using

a SICK Ranger E50 camera (see Fig. 9). Using the scanned 3-D

model, we measure the height between the steps to obtain the

mean error in mm, the resolution in the x−yplane in mm/px,

the height resolution in the x–zplane in mm/px, and the pro-

portion of occlusion in each scan.

The results in Table VI demonstrate that although both algo-

rithms are able to meet the resolution requirement for all tasks,

only the PSO algorithm was able to meet the height resolu-

tion requirement. Furthermore, even though this requirement

was relaxed in tasks 2 and 4, only the PSO algorithm was able

to substantially reduce occlusion such that all steps of the test

object are visible (see Fig. 9). A key advantage of the PSO algo-

rithm is the ability to optimize the angle βbetween the camera

and the laser plane.

C. Optimality and Convergence of Automated Planning

Experiments on the PSO algorithm itself are conducted in

closed-loop simulation. Three targets of realistic complexity,

shown in Fig. 11, are designed along with several sets of task

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 11

Fig. 9. Range images and 3-D visualization of the test object using the 3M and PSO algorithms. (a) 3M. (b) PSO high resolution. (c) PSO low resolution. (d)

3M. (e) PSO high resolution. (f) PSO low resolution.

Fig. 10. Physical apparatus—The camera and laser source are precisely po-

sitioned, and the conveyor belt encoder provides accurate and repeatable linear

transport.

Fig. 11. Simulated task targets—Targets and task models are designed to be

difﬁcult, to rarify F=1.0solutions.

Fig. 12. Task Performance—The PSO algorithm converges to the nominal

and expected value optima within a small number of iterations. (a) Task 1. (b)

Task 2. (c) Task 3. (d) Task 4.

TAB L E V I I

EXPERIMENTAL TASKS AND RESULTS

Task 1234

Tar get A A B C

n1222

Ri0.15 0.20 0.25 0.50

Ra0.19 0.60 0.30 2.00

Hi0.19 0.20 0.25 0.10

Ha0.23 0.60 0.30 1.00

ci1.00 1.00 1.00 1.00

ca2.00 2.00 2.00 2.00

ζi1.00 1.00 0.90 1.00

ζa1.00 1.00 0.90 π/2

ωc1.20 1.20 1.20 1.20

γ10 10 10 20

F1.0000 1.0000 1.0000 0.9999

E[F]1.0000 1.0000 1.0000 0.9161

Iterations 32 53 47 500

Evaluations 4301 4336 3166 15060

points and requirements each. Table VII lists some details for

four representative linear scan tasks, along with the statisti-

cal results of the experiments. The tasks are designed to be

difﬁcult, such that only a small subset of the solution space

yields F=1.0, and that numerous local maxima exist. This

is achieved by positioning task points among occluding sur-

faces, and by tuning the requirements to be demanding while

admitting at least one optimal solution. In all cases, the particle

swarm parameter values and functions discussed in Section VI-

E are used, and error compensation parameters are set to ne=

20, σCT =σTT =2.0, and σCR =σTR =0.01. Termination

conditions are Faccept =1.0 and Imax =500. No clustering

terminator is used.

For each task, 48 optimizations are conducted. The bottom

portion of Table VII lists the mean values of Fand E[F]for

the best solution, the mean number of iterations to convergence

or halting, and the mean number of evaluations of the objective

function. Fig. 12 shows statistics for Fand E[F]per iteration.

A number of relevant observations can be made about these

results. The initial F(g)is virtually always nonzero and often

relatively high, indicating that, for most tasks, the bounds de-

12 IEEE/ASME TRANSACTIONS ON MECHATRONICS

Fig. 13. Robustness to positioning error—Increasing neimproves the solution

but increases computation time. (a) Expected value of Fwith pose error (300

Trials). (b) Mean total evaluations of F.

ﬁned in Table V are a tight initial guess about the location of

the optimum. Although occasional temporary stagnation is ob-

served, for tasks 1, 2, and 3, the algorithm converges quickly to

the nominal optimum, and the expected value optimum follows

shortly thereafter.

Task 4 exempliﬁes what occurs when no nominal optimum

exists; as the algorithm incrementally improves its estimate of

the optimum, the stochastic E[F]ﬂuctuates slightly, and eventu-

ally settles. This behavior can be mitigated using the clustering

termination condition.

D. Robustness to Positioning Error

A series of 48 optimizations are conducted on a similar model

for each ne∈{0, 5, 10, 15, 20, 25, 30}, with positioning error

modeled with σCT =σTT =2.0 and σCR =σTR =0.01. All

solutions converged to a nominal F=1.0.

Fig. 13(a) shows the mean value of F, obtained over 300

trials with the same error parameters, for each of the 48 solu-

tions for each value of ne. Fig. 13(b) shows the mean number of

evaluations of Fconducted in the original optimizations (i.e.,

those required to produce the solutions). As expected, the re-

lationship between the performance cost and the accuracy in

ﬁnding robust solutions is roughly linear, for ﬁxed optimization

parameters and a ﬁxed number of iterations.

E. Industrial 3-D Gage Device Implementation

This method has been successfully applied to the design of an

industrial noncontact 3-D measurement device currently mar-

keted by Vista Solutions, Inc. This hybrid-modality device per-

forms some conventional machine vision tasks using an inten-

sity image as well as range sensing tasks using triangulation

with three line laser sources. The PSO algorithm was actually

leveraged to optimize over both the coverage model presented

in Section IV (for each of the three laser sources) as well as a

related model for conventional vision [21] from which it was

derived. The ﬁnal design is shown in Fig. 14.

Fig. 14. Final design of 3-D gage device—The design is optimized over both

triangulation and conventional vision models using the PSO algorithm.

The 3-D gage device serves as an example of real-world spec-

iﬁcation of the task model described in Section II. Requirements

were speciﬁed by the application engineers, directly based on ac-

tual tolerances speciﬁed by the customer. For the range imaging

modality, a scan density of Ri=Ra=0.05 mm and a height

resolution of Hi=Ha=0.05 mm were speciﬁed. Experimen-

tation with the range imaging software suggested ci=1.0 and

ca=10.0 as reasonable constraints for the blur circle. Based

on the software documentation, ζi=ζa=1.22 and ωc=1.22

were chosen for the maximum viewing angle and cutoff angles.

The γparameter was unused.

As the device is, in this case, a generic design, no target

geometry or speciﬁc set of task points were speciﬁed. Instead, a

regular “grid” of task points 50 mm wide (along the xaxis) and

30 mm high (along the zaxis), corresponding to the speciﬁed

ﬁeld of view, was used.

VIII. CONCLUSION

In this paper, we have presented a novel view planning ap-

proach for high-accuracy 3-D inspection systems based on active

triangulation. Rather than attempting to optimize over a large

set of design variables, with the assistance of expert integration

engineers, we have analyzed the design process, and reduced

the critically difﬁcult portion to only four design parameters per

camera. The result is a semiautomatic process which leverages

human expertise to create a basic system design, for which the

camera parameters are then reﬁned through automatic optimiza-

tion.

The ﬁrst contribution is a high-ﬁdelity task-oriented model

of coverage performance, which evaluates the overall design—

including manual components—with a bounded scalar metric,

and also serves as the objective function for optimization. This

model is based on research into general high-ﬁdelity representa-

tions of multiple camera coverage for other related optimization

problems. Our validation experiments demonstrate its effective-

ness in predicting a posteriori task performance, so that op-

timized designs can be reliably tested in simulation, moving

costly hardware testing further outside the cycle.

The second contribution is the identiﬁcation of the critical

part of the design process, and the successful application of a

particle swarm optimization algorithm to this reduced problem.

Although the high-ﬁdelity objective function is expensive to

compute, experimental evaluation demonstrates that, with ap-

propriate parameters, the algorithm reliably converges to opti-

mal solutions within a small number of iterations. The algorithm

MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 13

also directly compensates for positioning system uncertainty for

the cameras and target.

Actual convergence times vary widely, depending on the task

and number of cameras; typical “difﬁcult” problems with local-

ized optima tend to complete on the order of 10 min, using our

implementation on modern (single-core) hardware. In general,

at these rates, the algorithm is already expected to far outperform

human experts, supported by its successful application presented

in Section VII-E. With further optimization and parallelization,

it should be possible to substantially reduce convergence times.

ACKNOWLEDGMENT

The authors thank Vista Solutions Inc. for generously provid-

ing expertise, equipment, and facilities for experimentation.

REFERENCES

[1] P. J. Besl, “Active, optical range imaging sensors,” Mach. Vis. Appl.,vol.1,

no. 2, pp. 127–152, 1988.

[2] F. Blais, “Review of 20 years of range sensor development,” J. Electron.

Imag., vol. 13, no. 1, pp. 231–243, 2004.

[3] W. R. Scott, G. Roth, and J.-F. Rivest, “View planning for automated

three-dimensional object reconstruction and inspection,” ACM Comput.

Surveys, vol. 35, no. 1, pp. 64–96, 2003.

[4] K. A. Tarabanis, P. K. Allen, and R. Y. Tsai, “A survey of sensor planning

in computer vision,”IEEE Trans. Robot. Autom., vol. 11, no. 1, pp. 86–104,

Feb. 1995.

[5] S. Larsson and J. A. P. Kjellander, “Path planning for laser scanning with

an industrial robot,” Robot. Auton. Syst., vol. 56, no. 7, pp. 615–624, 2008.

[6] W. R. Scott, “Model-based view planning,” Mach. Vis. Appl., vol. 20,

no. 1, pp. 47–69, 2009.

[7] C. C. Yang and F. W. Ciarallo, “Optimized sensor placement for active

visual inspection,” J. Robot. Syst., vol. 18, no. 1, pp. 1–15, 2001.

[8] Q. Shi, N. Xi, and Y. Chen, “Development of an automatic optical measure-

ment system for automotive part surface inspection,” in Proc. IEEE/ASME

Int. Conf. Adv. Intell. Mechatronics, 2005, pp. 1557–1562.

[9] F. A. Rodrigues Martins, J. G. Garc´

ıa-Bermejo, E. Zalama Casanova, and

J. R. Per´

an Gonz´

alez, “Automated 3D surface scanning based on CAD

model,” Mechatronics, vol. 15, pp. 837–857, 2005.

[10] B. Englot and F. Hover, “Inspection planning for sensor coverage of 3D

marine structures,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2010,

pp. 4412–4417.

[11] G. H. Tarbox and S. N. Gottschlich, “Planning for complete sensor cov-

erage in inspection,” Comput. Vis. Image Understanding, vol. 61, no. 1,

pp. 84–111, 1995.

[12] J. L. Alarcon Herrera, A. Mavrinac, and X. Chen, “Sensor planning for

range cameras via a coverage strength model,” in Proc. IEEE/ASME Int.

Conf. Adv. Intell. Mechatronics, Jul. 2011, pp. 838–843.

[13] C. K. Cowan and P. D. Kovesi, “Automatic sensor placement from vision

task requirements,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 10, no. 3,

pp. 407–416, May1988.

[14] K. A. Tarabanis, R. Y. Tsai, and A. Kaul, “Computing occlusion-free

viewpoints,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 3, pp.

279–292, Mar. 1996.

[15] A. Mavrinac and X. Chen, “Modeling coverage in camera networks: A

survey,” Int. J. Comput. Vis., vol. 101, no. 1, pp. 205–226, 2013.

[16] J. Salvi, C. Matabosch, D. Foﬁ, and J. Forest, “A reviewof recent range im-

age registration methods with accuracy evaluation,” Image Vis. Comput.,

vol. 25, pp. 578–596, 2007.

[17] P. Saint-Marc, J.-L. Jezouin, and G. G. Medioni, “A versatile PC-based

range ﬁnding system,” IEEE Trans. Robot. Autom., vol. 7, no. 2, pp. 250–

256, Apr. 1991.

[18] J. L. Vilac¸a, J. C. Fonseca, and A. M. Pinho, “Calibration procedure for

3D measurement systems using two cameras and a laser line,” Opt. Laser

Technol., vol. 41, no. 2, pp. 112–119, 2009.

[19] A. Mavrinac, X. Chen, P. Denzinger, and M. Sirizzotti, “Calibration of

dual laser-based range cameras for reduced occlusion in 3D imaging,” in

Proc. IEEE/ASME Int. Conf. Adv. Intell. Mechatronics, 2010, pp. 79–83.

[20] N. P. Suh, The Principles of Design. London, U.K.: Oxford Univ. Press,

1990.

[21] A. Mavrinac, “Modeling and optimizing the coverage of multicamera

systems,” Ph.D. dissertation, Univ. Windsor, Windsor, ON, Canada, 2012.

[22] T. M¨

oller and B. Trumbore, “Fast, minimum storage ray/triangle intersec-

tion,” J. Graph. Tools, vol. 2, no. 1, pp. 21–28, 1997.

[23] C. C. Yang, M. M. Marefat, and F. W. Ciarallo, “Error analysis and plan-

ning accuracy for dimensional measurement in active vision inspection,”

IEEE Trans. Robot. Autom., vol. 14, no. 3, pp. 476–487, Jun. 1998.

[24] Y. Ma, S. Soatto, J. Koˇ

seck´

a, and S. S. Sastry, An Invitation to 3-D

Computer Vision. New York, NY, USA: Springer-Verlag, 2004.

[25] D. C. Brown, “Decentering distortion of lenses,” Photogrammetric Eng.,

vol. 32, no. 3, pp. 444–462, 1966.

[26] Y.-S. Chen, S.-W. Shih, Y.-P. Hung, and C.-S. Fuh, “Simple and efﬁcient

method of calibrating a motorized zoom lens,” Image Vis. Comput., vol. 19,

no. 14, pp. 1099–1110, 2001.

[27] G. Fu, P. Corradi, A. Menciassi, and P. Dario, “An integrated triangulation

laser scanner for obstacle detection of miniature mobile robots in indoor

environment,” IEEE/ASME Trans. Mechatronics, vol. 16, no. 4, pp. 778–

783, Aug. 2011.

[28] J. Kennedy and R. C. Eberhart, “Particle swarm optimization,” in Proc.

IEEE Int. Conf. Neural Netw., 1995, pp. 1942–1948.

[29] J. Kennedy and R. C. Eberhart, Swarm Intelligence. San Mateo, CA, USA:

Morgan Kaufmann, 2001.

[30] J. Kennedy and R. Mendes, “Population structure and particle swarm

performance,” in Proc. IEEE Congr. Evol. Comput., 2002, pp. 1671–1676.

[31] R. Mendes, “Population topologies and their inﬂuence in particle swarm

performance,” Ph.D. dissertation, , Dept. Informatica, Escola Engenharia,

Univ. Minho, Portugal, 2004.

[32] S. Helwig and R. Wanka, “Particle swarm optimization in high-

dimensional bounded search spaces,” in Proc. IEEE Swarm Intell. Symp.,

2007, pp. 198–205.

[33] I. C. Trelea, “The particle swarm optimization algorithm: Convergence

analysis and parameter selection,” Inf. Process. Lett., vol. 85, pp. 317–

325, 2003.

[34] M. Clerc and J. Kennedy, “The particle swarm—Explosion, stability, and

convergence in a multidimensional complex space,” IEEE Trans. Evol.

Comput., vol. 6, no. 1, pp. 58–73, Feb. 2002.

[35] R. B. Rusu and S. Cousins, “3D is here: point cloud library (PCL),” in

Proc. IEEE Int. Conf. Robot. Autom., 2011, pp. 1–4.

Aaron Mavrinac received the B.A.Sc., M.A.Sc., and

Ph.D. degrees in electrical and computer engineer-

ing from the University of Windsor, Windsor, ON,

Canada, in 2005, 2008, and 2012, respectively.

Since 2012, he has been working on research and

development on driver assistance and highly auto-

mated driving technologies at Robert Bosch LLC,

Plymouth, MI, USA. His research interests include

coverage analysis, calibration, and data fusion in het-

erogeneous multisensor systems.

Xiang Chen received the M.Sc. and Ph.D. degrees

from Louisiana State University, Baton Rouge, LA,

USA, in 1996 and 1998, respectively, both in systems

and control.

Since 2000, he has been with the Department of

Electrical and Computer Engineering, University of

Windsor,Windsor, ON, Canada, where he is currently

a Professor. His research interests include robust con-

trol, vision sensor networks, vision-based control sys-

tems, networked control systems, and industrial ap-

plications of control theory.

Jose Luis Alarcon-Herrera receivedthe B.A.Sc. de-

gree in electrical engineering from the Durango Insti-

tute of Technology, Durango, Mexico, in 2009. Since

2010, he has been working toward the Ph.D. degree

at the University of Windsor, Windsor, ON, Canada.

His research interests include multicamera net-

works, computer vision, and vision-guided robotics.