Content uploaded by Aaron Mavrinac
Author content
All content in this area was uploaded by Aaron Mavrinac on Sep 20, 2017
Content may be subject to copyright.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE/ASME TRANSACTIONS ON MECHATRONICS 1
Semiautomatic Model-Based View Planning for
Active Triangulation 3-D Inspection Systems
Aaron Mavrinac, Xiang Chen, and Jose Luis Alarcon-Herrera
Abstract—A semiautomatic model-based approach to the view
planning problem for high-resolution active triangulation 3-D in-
spection systems is presented. First, a comprehensive, general,
high-fidelity model of such systems is developed for the evaluation
of configurations with respect to a model of task requirements, with
a bounded scalar performance metric. The design process is ana-
lyzed, and the automated view planning problem is formulated only
for the critically difficult aspects of design. A particle swarm opti-
mization algorithm is applied to the latter portion, including proba-
bilistic modeling of positioning error, using the performance metric
as an objective function. The process leverages human strengths
for the high-level design, refines low-level details mechanically, and
provides an absolute measure of task-specific performance of the
resulting design specification. The system model is validated, allow-
ing for a reliable rapid design cycle entirely in simulation. Param-
eterization of the optimization algorithm is analyzed and explored
empirically for performance.
Index Terms—Inspection, laser scanner, range camera, view
planning.
I. INTRODUCTION
OPTICAL range sensors based on active triangulation [1]
are popular in industrial applications of noncontact 3-D
metrology and inspection, owing to high fidelity and robustness
in comparison to other noncontact methods, and high data ac-
quisition rates in comparison to mechanical methods. A typical
inspection task involves measuring the surface of an object at
resolutions on the order of tens of micrometers. The specified
tolerances often run within an order of magnitude of the physi-
cal capabilities of modern devices, when factors such as field of
view and occlusion are taken into account. Hence, planning the
configuration of the sensor system is not trivial, requiring care-
ful balancing of a number of constraints to achieve an acceptable
solution.
In industry, typically the end user of the inspection system
delegates this difficult task to a third-party specialist integrator,
who will assemble a custom solution starting from modern off-
the-shelf range cameras [2] and software. The main expertise
is in designing a physical configuration capable of meeting the
client’s specifications, usually assisted by planning guidelines
provided by the range camera manufacturer. Considering even
Manuscript received May 10, 2012; revised January 23, 2013, September
1, 2013, and February 2, 2014; accepted March 13, 2014. Recommended by
TechnicalEditor Z. Zhu. This work was supported in part by the Natural Sciences
and Engineering Research Council of Canada.
A. Mavrinac is with the Robert Bosch LLC Chassis Systems Control Division,
Plymouth, MI 48170 USA (e-mail: aaron.mavrinac@us.bosch.com).
X. Chen and J. L. Alarcon-Herrera are with the Department of Electrical
and Computer Engineering, University of Windsor, Windsor, ON N9B 3P4,
Canada (e-mail: xchen@uwindsor.ca; alarconj@uwindsor.ca).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org
Digital Object Identifier 10.1109/TMECH.2014.2318729
a small subset of factors is very labor intensive. Additionally,
the remaining factors often have subtle and poorly understood
effects that can render the result unsuitable. To compensate, the
integrator often adds a fixed error factor to the client specifica-
tions. However, this can greatly increase the cost of the solution
(or preclude one altogether).
Automated view planning holds promise of efficiency and
reliability, yet such methods are not currently in use by integra-
tors. Scott et al. [3] present a survey of view planning, including
a detailed analysis of the constraints and requirements. While
some are of interest mainly in reconstruction and do not apply
to inspection, there is an evident gap between the remaining
points and the state of the art. The authors conclude that over-
simplification of the problem leads to unrealistic solutions and
is the main barrier to adoption, and suggest that semiautomatic
methods may yield tangible advances in the near term.
We propose such a semiautomatic design process, leveraging
the strengths of both human expertise and computational effi-
ciency. Our approach addresses two key issues. First, we derive
a high-fidelity model of multicamera bistatic coverage of active
triangulation systems, based on low-level criteria in the vein of
classic sensor planning models [4]. The task and system models
(Sections II and III, respectively) are very general, and directly
reflect requirements and configuration parameters. We provide
a bounded scalar performance metric which predicts online per-
formance (see Section IV), allowing costly and time-consuming
hardware implementation to be replaced with simulation in the
design loop. Second, we reduce the scope of the automatic plan-
ning problem to only the critical portion that presents difficulty
to human designers or expert-based heuristics (see Section V).
The reduced problem is then addressed using a particle swarm
optimization (PSO) technique (see Section VI). Experimental
results demonstrate the validity of the model, the optimality and
convergence properties of the optimization algorithm, and the
robustness of solutions to positioning error (see Section VII).
A. Prior Work
For a comprehensive account of work on view planning for
3-D reconstruction and inspection with active triangulation
range sensors up to 2003, the reader is referred to the survey
by Scott et al. [3]. The authors note that most approaches suffer
from oversimplifications of the problem, including modeling
the bistatic system as a monostatic sensor, placing unrealistic
constraints on viewpoints, and ignoring several significant low-
level criteria. They conclude that no effective general-purpose
solution exists.
While several new methods targeted at reconstruction have
been put forward since (e.g., by Larsson and Kjellander [5] and
by Scott himself [6]) these attempt to solve the next best view
1083-4435 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2IEEE/ASME TRANSACTIONS ON MECHATRONICS
problem for unknown object surface reconstruction. Even recent
methods aimed at inspection (e.g., Yang and Ciarallo [7], Shi
et al. [8], Rodrigues Martins et al. [9], Englot and Hover [10]),
assuming aprioriknowledge of object geometry, still tend to fo-
cus on path planning for total surface inspection. Our scope does
not include registration-based inspection, so these concerns are
irrelevant. We are unaware of any work exclusively addressing
the planning of static multicamera active triangulation systems.
Some previous model-based approaches have been put for-
ward. Of note is the work of Tarbox and Gottschlich [11], who
employ a volumetric coverage model of a bistatic active trian-
gulation sensor. Scott [6] proposes an improved model making
fewer assumptions about the object; his concept of verified mea-
surability is similar to our bounded performance metric. Alarcon
Herrera et al. [12] present an initial analysis of the viewpoint
evaluation component using an early formulation of the cover-
age model of Section IV.
Scott et al. [3] observe that the sensor planning literature lacks
clear, explicit characterization of task requirements. Cowan and
Kovesi [13] and Tarabanis et al. [14] touch on quantifying the
coverage of specific subsets (features) of the target surface with
respect to performance requirements. Scott [6] explicitly models
the task with quantitative requirements for measurement preci-
sion and sampling density, ending with quantitative performance
metrics, but stops at specifying a uniform requirement over the
entire target surface, though subset regions and heterogeneous
criteria are mentioned.
In approaching the view planning problem, researchers have
repeatedly attempted to model visual coverage in forms conve-
nient for various optimization techniques. These models have
invariably suffered from a loss of fidelity, resulting in poor solu-
tions in general or restricting the scope of applicability [3], [15].
Rather than forcing the model to fit the optimization algorithm,
we opt instead to develop a relatively high-fidelity model with
only a general consideration for optimization, then choose and
parameterize an optimization algorithm based on the nature of
the objective function.
B. Application Scope
Most industrial inspection tasks can be addressed by a basic
single-pass linear or rotary range scan configuration, or can be
decomposed into several subtasks of this form. This type of so-
lution entirely avoids online registration, which is prohibitively
expensive for online scanning rates in most applications [16].
Such a configuration includes exactly one line-projection light
source (usually a laser) and one or more range cameras. Config-
urations with single cameras are well-studied; multiple-camera
cofigurations are less common, but are beginning to find use
in industry. We assume that multiple-camera configurations are
mutually calibrated: see treatments by Saint-Marc et al. [17],
Vilac¸a et al. [18], and Mavrinac et al. [19].
The objective is to design a system configuration which
achieves the requirements of the task. This configuration is static
in the sense that laser and camera parameters are fixed, and
the target object undergoes deterministic linear or rotary mo-
tion through the system. The design parameters are the poses
(3-D position and orientation) and respective properties of the
laser and cameras, as well as the number of cameras. Some
of these parameters will be fixed or constrained during the
manual portion of design, and the remainder will be optimized
automatically.
Certain sensible restrictions are placed on the geometry of
the system, based on the operating characteristics of the range
cameras. We define a right-handed Cartesian coordinate system
of R3with axes x,y, and z; the origin is arbitrary, and the basis
depends on the motion of the object being scanned. In the case of
linear motion, the y-axis is the transport direction, and in the case
of rotary motion, the axis of rotation is parallel to the x-axis. The
laser position is unconstrained, but the laser plane—the plane
in which all projected laser lines lie—must be parallel to the
x-zplane.1Similarly, camera positions are unconstrained, but
they may only rotate about the x-axis, as rotations about yand z
generally violate the internal assumptions of the sensor design.
This rotation is subject to the requirement that the (positive)
optical axis intersect the laser plane. The initial pose of the
target object is arbitrary.
As is standard in inspection applications, we assume that
the geometry of the object to be inspected is known; gener-
ally, clients will provide a CAD model of the part along with
the specification. Issues relating to the object material are not
considered. We assume that the specification indicates which
portions of the surface are to be inspected, with tolerance values
for each.
C. Constraints and Requirements
Scott et al. [3] identify a set of 16 qualitative constraints
and requirements a complete view planning system should con-
sider. We refer the reader to the survey for details on previous
approaches.
1) Model Quality Specification: We provide a parameteriz-
able task model (see Section II) which allows the task require-
ments to be specified precisely in the language of functional
requirements [20].
2) Generalizable Algorithm: Our system model (see
Section III) is general over the class of active triangulation
vision systems used in inspection applications. The model is
readily extensible to a broader range of sensor classes [21].
3) Generalized Viewpoints: All system parameters are in-
corporated into the viewpoint construct of the system model in
Section III.
4) View Overlap: As our scope is limited to single-scan
inspection applications, view overlap for registration is not
considered.
5) Robust: The view planning algorithm presented in Sec-
tion VI is robust to sensor and positioning system noise (see
Section VI-C).
6) Efficient: Our experimental results in Section VII indicate
that complex view planning tasks are completed in a matter
of seconds or minutes, quite competitive with skilled human
engineers.
1Many systems allow rotation of the laser about the x-axis. The resulting
oblique projection adds complexity to the process of range mapping. Since it
is possible to solve most industrial inspection problems with the laser plane
perpendicular to the transport direction, we restrict our analysis to this case.
However, it is certainly possible to generalize the approach to handle oblique
laser configurations.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 3
7) Self-Terminating: The algorithm presented in Section VI
incorporates three separate termination conditions. Addition-
ally, since the performance function developed in Section IV is
bounded, termination can be precisely controlled with respect
to task requirements.
8) Limited a Priori Knowledge: The effectiveness of the al-
gorithm in view planning for a task is dependent on the quality
of the apriorimodel of the target object provided; since our
scope is not a next best view problem, no feedback may be used.
However, it can provide a useful general solution given only ap-
proximate bounding dimensions (e.g., as in Section VII-E).
9) Shape Constraints: The algorithm is effective for any rea-
sonable object geometry and topology.
10) Material Constraints: It is assumed that issues relating
to the target material (e.g., choice of laser and camera, incidence
angle constraints, etc.) have been accounted for by the human
designer.
11) Frustum: Our camera model (see Section IV-B) is a
high-fidelity representation of lens and sensor effects [21].
12) Shadow Effect: The bistatic nature of the laser
and camera is explicitly considered by the functions in
Section IV.
13) Measurement Performance: Measurement performance
is evaluated over the entire high-fidelity model against the task
model, and summarized in a single bounded performance metric.
14) 6-D Pose: While our system model is capable of model-
ing unconstrained 3-D pose, the sensors are operationally lim-
ited to four degrees of freedom, as described in Section V-B.
15) Pose Constraints: Constraints on the range of the posi-
tioning system are modeled and respected by the algorithm.
16) Positioning System Performance: Pose error is explicitly
modeled by the algorithm, as described in Section VI-C. As
target configurations are static, no online reconfiguration time
is considered.
II. TASK MODEL
A. Object Geometry
The target object geometry is represented in the task model as
a triangulated polygonal mesh of (opaque) surfaces; the triangles
form a set TT. It is assumed that some means exists of obtaining
such a mesh apriori, at some degree of fidelity, e.g., from a
CAD model.
We define a function intersect(Δ,p,q), which returns the
point of intersection between line segment pq and triangle Δ∈
TT,or∅if none exists. M¨
oller and Trumbore [22] present an
efficient means of computing this intersection.
B. Task Points
An inspection task specifies a set of points TP⊂R3on the
surface of the object; formally, ∃Δ∈T
T|t∈Δ∀t∈T
P. Each
task point therefore has an associated surface normal, which is
the normal of the triangle in TTon which it lies. If continuous
areas are specified (e.g., the entire surface), these may be dis-
cretized according to the scanning density requirement without
loss of information.
TAB L E I
TASK PARAMETERS
Parameter Descriptiona
Ri,Raminimum ideal/acceptable scan density (l/px)
Hi,Haminimum ideal/acceptable height resolution (l/px)
ci,camaximum ideal/acceptable blur circle (px)
ζi,ζamaximum ideal/acceptable view angle (a)
ωcincidence angle cutoff (a)
γimage boundary padding (px)
aNumerical parameters are defined in units of length (l), pixel units
(px), and/or angle units (a).
C. Task Requirements
The task requirements are captured in a set of parameters,
listed in Table I. These directly describe criteria commonly
specified by sensor components, software algorithms, and/or the
application as operating requirements or tolerances, and should
be familiar to the experienced designer. The first four are divided
into “ideal” and “acceptable” values: the latter should be set to
the limit at which the task produces no useful information, and
the former should be set to the point beyond which the task
performance does not increase.
The scan density is the horizontal resolution of the range
image, and can be thought of as a requirement on the “instan-
taneous” resolution at task point t. The height resolution is the
precision to which range values are measurable at t; this is gen-
erally the most important requirement in an inspection task (see
the discussion of quantization error by Yang et al. [23]). The
blur circle requirement is derived from the dependence of the
profile interpolation algorithm on focus, usually discussed in
the supplier’s documentation. The view angle requirement de-
rives from the sensor’s ability to image a profile, which varies
inversely with the cosine of view angle relative to the laser pro-
jection, and depends on the laser, sensor, and material properties.
The incidence angle cutoff is the angle beyond which the light
received by the camera drops below a threshold [6]. Finally,
the image boundary padding requirement degrades coverage at
the edges of the image, a simple way of accounting for effects
such as residual lens distortion, profile interpolation bias, and
positioning error.
III. SYSTEM MODEL
A. Laser
Within the laser’s local coordinate system, we model the ef-
fective sheet projection as an isosceles triangle in the x–zplane,
with top angle λ(the so-called fan angle), and base parallel to
the x-axis at distance zP(the maximum effective depth) from
the top vertex (see Figs. 1 and 2).
Given the laser’s local coordinate system with origin at the
projection source and negative z-axis along the axis of projec-
tion, the laser has pose PLwhich, as discussed in Section I-B,
is restricted to translation by a vector TL(Table II).
B. Camera
We model each camera geometrically according to the pinhole
model [24], with additional parameters related to the physical
sensor and optics necessary for the full coverage model de-
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
4IEEE/ASME TRANSACTIONS ON MECHATRONICS
Fig. 1. Scanning by active triangulation—A 3-D model of the surface of the
object is obtained by measuring along a series of cross sections of the object,
yielding a series of 2-D profiles which are combined into a 3-D point cloud
according to the transport pitch.
Fig. 2. World coordinate system—The laser coordinate system is parallel to
(and usually coincides with) the x–zplane, while the coordinate systems of the
cameras are determined by their respective poses.
TAB L E I I
LASER PARAMETERS
Parameter Descriptiona
λfan angle (a)
zPprojection depth (l)
TL=TL,x,T
L,y,T
L,zposition (l)
scribed in Section IV-B. According to this model, the projected
image qof a 3-D point pis
q=f
su·px
pz
+ru,f
sv·py
pz
+rvT
(1)
where the horizontal and vertical pixel dimensions suand sv,
respectively, the effective focal length f, in units of length, and
the image of the principal point r=(ru,r
v), in pixel coordi-
nates, are determined by calibration. The actual image extends
from the origin (top-left corner of the image) to (w, h), where
wand hare the width and height (resolution) of the sensor,
respectively (Fig. 3).
Fig. 3. Pixel coordinates—The origin of the image in pixel coordinates is at
the top left corner of the image. Pixel distances are related to units of length by
the physical pixel dimensions suand sv.
TABLE III
CAMERA PARAMETERS
Parameter Descriptiona
w,hwidth and height of the sensor (px)
su,svphysical pixel dimensions (l/px)
feffective focal length (l)
r=ru,r
vcoordinates of the principal point (px)
Aeffective aperture diameter (l)
zSsubject distance (l)
TC=TC,x,T
C,y,T
C,zposition (l)
θrotation about x-axis (a)
ψrotation about z-axis (a)
We assume that lens distortion is corrected, e.g., according
to Brown’s model [25], and that residual error is handled as
described in Section IV-B1. The effective aperture size Ais
significant, however, and can be determined from the f-number
setting on the lens.
For a given lens, the intrinsic parameters f,r, and Avary
depending on lens parameters, which include focus, zoom, and
physical aperture settings. The relationship is nonlinear and
complex, but is possible to estimate for a given lens by perform-
ing a series of calibrations with varying settings, and interpolat-
ing [26]. We assume that zoom and physical aperture are fixed,
and that only the focus setting varies; we construct the lookup
table for f,r, and Aby varying the subject distance zSdirectly.
Given the camera’s local coordinate system with origin at the
principal point and positive z-axis coinciding with the optical
axis, a camera has pose PCi, comprising translation by a vector
TCi, rotation about the x-axis by an angle θCi∈[π
2,π), and
rotation about the z-axis by an angle ψCi∈{0,π}such that the
optical axis intersects the laser plane (Table III).
C. Static Environment Geometry
For purposes of occlusion, deterministic objects in the en-
vironment may be modeled similarly to the target object in
Section II-A, with a set of triangles ST. Again, it is assumed
that the mesh is available a priori. The intersect function applies
here as well.
IV. VIEW EVALUATION
A. Laser Coverage Function
The laser coverage function is a mapping L:R3→{0,1},
with
L(p)=LP(p)·LI(p)·LO(p)(2)
comprising three components.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 5
Fig. 4. Field of view—The quadrilateral pyramid extending along the optical
axis to infinity (shown truncated), defined by the apex angles αhand αv.
1) Projection: The task point is covered only if it is within
the triangle of projection
LP(p)=⎧
⎨
⎩
1,if arctan |px|
−pz≤λand −pz≤zP
0,otherwise. (3)
2) Incidence Angle: The task point is covered only if the
angle of incidence within the laser plane is less than the cutoff
angle
LI(p)=1,if arccos (ˆ
u·ˆ
n)≤ωc
0,otherwise (4)
where ˆ
uis a unit vector in the direction of u=p−TL, and ˆ
n
is the surface normal at p.
3) Occlusion: The task point is covered only if the ray from
the laser source to the point is unobstructed
LO(p)=
1,if intersect (Δ,TL,p)∈{∅,p}∀Δ∈T
T∪S
T
0,otherwise. (5)
B. Camera Coverage Function
The camera coverage function is a mapping C:R3→[0,1],
with
C(p)=CV(p)·CD(p)·CH(p)·CF(p)·CA(p)·CO(p).
(6)
In defining the components of the camera coverage function,
we use a bounding function B[0,1](x) = min(max(x, 0),1) to
limit xto the range [0,1], simplifying the formulation.
1) Field of View: The field of view is a pyramid with apex at
the camera’s optical center and rectangular base at infinity along
the optical axis, induced by the projective geometry and the
limits of the image sensor (see Fig. 4). The task point is visible
(barring occlusion) if it is within this pyramid. The horizontal
and vertical field of view criteria are given, respectively, by
CVh
(p)=B[0,1] ⎛
⎝min px
pz+ tan αl,tan αr−px
pz
γh⎞
⎠(7)
CVv
(p)=B[0,1] ⎛
⎝min py
pz+ tan αt,tan αb−py
pz
γv⎞
⎠(8)
where αland αrare the horizontal field of view angles, and αt
and αbare the vertical field-of-view angles, as given by
αl= arctan rusu
f(9)
αr= arctan (w−ru)su
f(10)
αt= arctan rvsv
f(11)
αb= arctan (h−rv)sv
f(12)
and, for γ>0,γhand γvare given by
γh=γ
w(tan αl+ tan αr)(13)
γv=γ
h(tan αt+ tan αb).(14)
The full field of view criterion is then given by
CV(p)=min(CVh
(p),CVv
(p)),if pz>0
0,otherwise. (15)
2) Scanning Density (Resolution): The scanning density
along the ydirection is entirely determined by the transport
pitch, and we assume that this is designed appropriately for the
requirements. The scanning density along the xdirection is a
function of the resolution in units of length per pixel, and the
criterion is given by
CD(p)=B[0,1] zD(Ra)−pz
zD(Ra)−zD(Ri)(16)
for Ri<R
a, where
zD(R)= Rw
tan αl+ tan αr
. (17)
Without the multiplier R, the RHS of (17) expresses the num-
ber of pixels subtended by a bundle of rays one unit of distance
wide along x,atazdepth of one unit of distance. Multiply-
ing by Ryields the maximum zdepth for which the specified
resolution is met.
3) Height Resolution: Height resolution is computed simi-
larly to the scanning density criterion, but is additionally depen-
dent on the view angle due to distortion of the profile [6], [27].
The height resolution criterion is given by
CH(p)=B[0,1] zH(Ha,p)−pz
zH(Ha,p)−zH(Hi,p)(18)
with
zH(H, p)= Hhsin ζ(p)
tan αl+ tan αr
(19)
where ζ(p)is the view angle to p, as given by (23). (19) is
derived similarly to (17), with the additional term accounting
for the effect of the oblique angle of incidence. Our application
scope does not allow for rotation about z, so this term is always
unity in (17).
4) Focus: The quality of focus is a function of the blur circle
diameter of the projected task point (Fig. 5), obtained from
standard depth of field equations from Gaussian optics, and the
criterion is given by
CF(p)=B[0,1] min pz−zn
z−zn
,zf−pz
zf−z (20)
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6IEEE/ASME TRANSACTIONS ON MECHATRONICS
Fig. 5. Blur circle—A point at zSmaps to a point in the image; points closer
or farther away map to blur circles, with diameters proportional to depth.
for ca>c
i, where (z,z
)and (zn,z
f)are the near and far
limits of depth of field as given by (21), substituting blur circle
diameters cimin(su,s
v)and camin(su,s
v), respectively, for c,
z=AfzS
Af ±c(zS−f).(21)
5) View Angle: The view angle with respect to the laser
projection axis also impacts camera coverage [6]. The criterion
is given by
CA(p)=B[0,1] cos ζ(p)−cos ζa
cos ζi−cos ζa(22)
with ζ(p), the view angle to p, computed as
ζ(p) = arccos P−1
C(ˆ
z)·ˆ
p(23)
where ˆ
zis a unit vector in the positive zdirection, and ˆ
pis a
unit vector in the direction of p.
This is a special case of a more complex criterion [21] for
generic view angle requirements from surface normals. In this
case, the normals are along the projection axis, so that the view
angle criterion expresses the obliqueness of the laser to the
camera.
6) Occlusion: The task point is covered only if the ray from
the point to the camera is unobstructed
CO(p)=
1,if intersect (Δ,p,TC)∈{∅,p}∀Δ∈T
T∪S
T
0,otherwise. (24)
C. Task Performance Metric
We require a coverage metric M(t)∈[0,1] for a task point
t∈T. According to the task specification, a task point is an
arbitrary point in R3, but scanning takes place exclusively within
the laser plane. The transport motion of the target object carries
each task point to a new point t∈R3such that t
y=TL,y,from
which laser and camera coverage may then be computed.
For linear transport, this motion consists simply of translating
the target object by T=0,T
L,y −ty,0.
For rotary motion, the target object is rotated about an axis
parallel to the x-axis passing through 0,T
L,y,z
r.Fort∈T
P,
the angle of rotation is φr= arccos(ˆ
v·ˆ
z)(or 2π−φr, depend-
ing on the direction of rotation), where ˆ
vis a unit vector in the
direction of v=t−tx,T
L,y,z
r.
Armed with an appropriate means of obtaining the transported
task point t, the overall system coverage metric for a single
point is
M(t)=LP−1
L(t)·max
iCiP−1
Ci(t). (25)
Finally, given a task and system context, the performance
metric over all task points is
F=TPM(t)
|TP|. (26)
V. VIEW PLANNING
A. Manual Initialization
Based on the observation of the design process, the critical
difficulty is encountered in planning the position and orienta-
tion of the sensor: the six criteria in Section IV-B are competing
objectives over this space, and since focus settings must be ad-
justed as well, it can be very challenging to find a configuration
which meets them all simultaneously. Accordingly, the manual
initialization stage must determine many of the system param-
eters discussed in Section III:
1) initial pose of the target object;
2) transport style (linear or rotary);
3) axis of rotation, if applicable;
4) position and properties of the laser;
5) specific sensors and lenses to use; and
6) number (n) and ψ-orientations of cameras.
A complete discussion of the heuristic for designing these
aspects is beyond our scope, but we will comment further on
the justification for this division. Items 1 to 3 are typically dic-
tated by fixturing, physical constraints, and the locations of the
task points on the object surface; there is usually little to be
gained by attempting to automate their design. The calcula-
tions for an appropriate laser configuration in item 4 are quite
straightforward: the x–zprojections of TPshould lie within
the triangle defined by λ,zP, and TL, which is not normally
difficult to achieve. The sensor and lens of item 5 are typically
chosen from a limited pool of available products. The zoom
and physical aperture settings of the lens require only coarse
tuning in this stage, since other codependent parameters—the
focus and standoff distance—will subsequently be optimized.
Finally, there is a small, discrete set of feasible possibilities for
the number and ψ-orientations of cameras in item 6, and even
if the best among them is not obvious to the designer, several
options may be tested.
B. Automatic Refinement
In the automatic refinement stage, all parameters of the system
are fixed, except the cameras’ positions, their rotation about the
x-axis, and their focus setting (thus varying zS, and by extension
f,r, and A). Further, the focus setting is constrained such that
zSalong the optical axis (in camera coordinates) is always in
the laser plane.
Without loss of generality, we discuss the single-camera prob-
lem. For each camera, we vary four independent values equiv-
alent to the parametrization of a rigid transformation in the
y–zplane plus translation in x, but more representative of the
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 7
TAB L E I V
DESIGN IMPACT MAT RIX FO R AUTOMATIC REFINEMENT
xhdβ
Field of View •••◦
Scanning Density ◦•◦
Height Resolution ◦••
Focus ◦•◦
View Angle ◦◦◦•
Occlusion ◦•◦•
◦– minor impact, •– major impact
functional design of the system. These design variables are as
follows:
1) xthe horizontal position of the camera;
2) hheight (z) at which the optical axis intersects the laser
plane;
3) ddistance from TCto the point x, 0,h;
4) βangle between the z-axis and the optical axis.
From these variables, the position of the camera is computed
as
TC,x =x(27)
TC,y =TL,y ±dsin β(28)
TC,z =h+dcos β(29)
where the sign in (28) depends on whether the camera is down-
stream (positive) or upstream (negative) from the laser plane
along the transport axis. The angle of rotation is β+πabout
the negative or positive x-axis, respectively. The intrinsic pa-
rameters f,r, and Aare obtained via interpolation on the lens
lookup table, with zS=d.
A chart showing the impact of each design parameter on each
coverage criterion, based on the concept of the design matrix
[20], is shown in Table IV. The “minor impact” designation
indicates that modifying the parameter impacts the criterion
indirectly due to the ray from the point to the camera not coin-
ciding with the optical axis; these effects are normally relatively
minor, though still significant, over the visible volume. The
criteria are strongly coupled with respect to the four design pa-
rameters (or any other possible set), and the nature of the bistatic
range sensor design—specifically, the 3-D coverage of points
within the 2-D laser plane—makes it impossible to decouple
them within the scope of available parameters. Furthermore, the
solution space is highly nonlinear and discontinuous in terms
of the performance metric F, which depends directly on the six
criteria. This is clearly an ill-posed problem for human design
and for direct numerical optimization.
For a n-camera configuration, the solution space is
4n-dimensional: x,h,d, and βare varied independently for
each camera. F(s)denotes the objective function (26) with re-
spect to solution point s∈R4n, where system parameters are
calculated from sas described above. Given an initial manual
configuration as prescribed in Section V-A, the problem is to
find swhich maximizes F(s), subject to any design constraints
on s.
VI. AUTOMATIC REFINEMENT ALGORITHM
A. Particle Swarm Optimization
The view planning problem described in Section V-B con-
sists essentially of maximizing a complex, irregular, nondiffer-
entiable objective function defined over a bounded, continuous
solution space. Among metaheuristics, PSO [28], [29] lends it-
self particularly well to this form of problem. The algorithm
itself has been published in various forms in numerous sources,
but in order to expose the particular structure and features of our
application, we describe it in full as used in Algorithm 1.
Each particle’s state at a given iteration is described by a
position si∈R4nand a velocity vi∈R4n. For simplicity of
exposure, where any position variable sis uninitialized in Al-
gorithm 1, F(s)=−∞. The lower and upper solution space
bounds, respectively, are given as vectors bland bu(values are
derived in Section VI-B). U(l,u)denotes a random uniform
vector bounded by land u.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8IEEE/ASME TRANSACTIONS ON MECHATRONICS
TAB L E V
BOUNDS ON DESIGN VARIABLES
Variable Lower Bound Upper Bound
xx
min xmax
hz
min zmax
dd
min du
β0ζa
The parameters S,ω,φp,φn, and κare endemic to the algo-
rithm; they denote the size (number of particles) of the swarm,
the particle momentum, the disposition toward the present best
local and neighborhood solutions, and the velocity clamping
factor, respectively. Appropriate values for these parameters are
discussed in Section VI-E.
The particle state also stores pi, the best solution found by
particle i, and ni, the best solution found by any particle in par-
ticle i’s neighborhood set Ni⊆{1,...,S}. The best solution
found by any particle i∈{1,...,S}is stored globally in g.
The function topology(S, i)returns a subset of {1,...,S},
describing a topological relationship between the particles. In
the original PSO algorithm, all particles are considered adjacent
(i.e., {1,...,S}is returned), but using other topologies can im-
prove performance in terms of convergence and optimality [30],
[31]. The function constrain(si,bl,bu)modifies sidepending
on the solution space bounds. In the original PSO algorithm,
no constraints are applied (i.e., siis returned), but constraint
handling mechanisms can also improve performance [32]. Both
are discussed in Section VI-E, and our selections are defined in
(31) and (32), respectively.
The positions and velocities of the particles are initialized
randomly (lines 2 to 6). Each iteration then consists of updating
the local, neighborhood, and global best solutions based on the
value of Fat each particle’s current position (lines 9 to 29), then
updating each particle’s position and velocity (lines 30 to 41).
The lines involving Fe, comprising the block in lines 16 through
23 and line 25, are a nonstandard feature related to positioning
error compensation, explained in Section VI-C.
The algorithm proceeds until one of three stop conditions
is met:the global best solution exceeds Faccept, a proportion
Cmax ∈(0,1] of particles cluster within of g(by Euclidean
distance in a solution space normalized on the bounds), or the
algorithm has run for Imax iterations.
B. Solution Space Bounds
We derive a set of conservative bounds based on an assump-
tion which appears to hold in practical, nonpathological cases.
If xmin,xmax,zmin, and zmax represent the x-zbounds of
the task point set, the proposed bounds on x,h,d, and βwill
not exclude any solution with M(t)>0for all t∈T
Pwhere
xmin ≤x≤xmax and zmin ≤h≤zmax (the assumption be-
ing that these inequalities hold for some optimal solution). The
bounds are listed in Table V.
The values for the bounds on x,h, and βhave already been
defined; those for drequire explanation. Let dmin and dmax
represent, respectively, the minimum and maximum values of
zSin the lens lookup table. The upper bound is defined as
du= min[zD(Ra),z
H(Ha,P−1
C(x, 0,h))] (30)
where the values of fand r—upon which zDand zHdepend,
per (17) and (19)—are obtained from the lens lookup table with
zS=dmax. The values of xand hare irrelevant as long as they
are in accord with TCfor the purposes of the calculation.
Design constraints on the configuration of a camera, including
limits on lens settings and physical position, are modeled by
overriding the value of Fto −∞ for all solutions violating
the constraints, as proposed by Mendes [31]. These may take
arbitrary form, and are not necessarily restricted to rectangular
bounds in the x-h-d-βdesign space.
C. Compensating for Positioning Error
Due to limitations on the physical accuracy of positioning
the cameras and the target object fixture, the true poses of these
entities will inevitably differ, to some extent, from the opti-
mized design [3], [23]. If the algorithm converges on a global
optimum s, and the actual position achieved in practice is se,
then F(se)≤F(s). The true practical goal is to maximize the
expected performance E[F(se)]. In cases where the global max-
imum of Fis nonunique, which is almost certainly the case if
F(s)=1.0, the algorithm may reach a solution near the “edge”
of a nonsingular optimal solution space region, and terminate.
By considering positioning error within the algorithm, it is pos-
sible to encourage convergence to a solution with better sur-
rounding solutions.
The relationship of pose error to the ultimate value of Fis
complex, and deriving a closed-form probabilistic model of cov-
erage is infeasible. Instead, we compensate by directly testing
the robustness of a promising candidate solution using a prob-
abilistic model of positioning error and Monte Carlo sampling.
Let Fe(s)represent the mean value of F(s), with the error model
applied to the cameras and target object, over nesamples.2The
essence of lines 16 through 23 and line 25 of Algorithm 1 is that
if F(si)for one or more iis at the global best for a given iter-
ation, the algorithm begins evaluating and ranking by Fe(si)as
well. The motivation for this conditional evaluation, as opposed
to simply incorporating it into the fitness function, is that Feis
netimes as expensive to evaluate as F, and there is no reason to
evaluate it for points known to be nominally suboptimal. An em-
pirical analysis of the tradeoff between additional performance
cost and improvement in optimality with positioning error is
reported in Section VII-D.
Random error is introduced to the poses of the cameras and
target object according to the following model. For the cameras,
a position offset is applied to each coordinate sampled from a
zero-mean Gaussian distribution with standard deviation σCT,
and an orientation offset is applied as a rotation about the x-axis
by an angle sampled from a zero-mean Gaussian distribution
with standard deviation σCR. For the target object, a similar
position offset is applied, with the Gaussian distribution having
standard deviation σTT, and the orientation offset is applied as
a rotation about a random axis uniformly sampled over the unit
sphere, and with the Gaussian distribution for the angle having
standard deviation σTR.
2For completeness, if ne=0,Fe(s)=−∞ for all s∈R4n.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 9
D. Computational Complexity of the Fitness Function
Since F(26) is evaluated repeatedly in Algorithm 1—at
least Stimes per iteration and, in the worst case, (ne+1)S
times per iteration—it is necessary to consider its computational
complexity.
The fact that all of the laser parameters are static allows for
caching of the laser plane points and the results of the laser cover-
age function L(2). The camera coverage function C(6) must be
evaluated for each point t∈T
P, for each camera i∈{0,...,n}.
The critical component is CO(24): each triangle in TT∪S
Tin-
duces an evaluation of intersect. Some of STmaybeculled
for each new camera configuration, but in general the entire set
are candidates. Therefore, the computational complexity is in
O(n×|T
P|×|T
T∪S
T|).
In practice, nis quite low (usually 1 or 2), but it is not
unreasonable for |TP|and |TT∪S
T|each to be on the order
of thousands for a high-fidelity model of an inspection task. In
our implementation, a single evaluation of Fin a very large
model can take on the order of several seconds on a modern
workstation platform.3
E. Optimality and Convergence
Trelea [33] analyzes optimality and convergence of the PSO
algorithm using results from dynamic system theory, and de-
rives guidelines for parameter selection. We seek parameters
for which the algorithm will reliably converge to a global opti-
mum within a finite number of evaluations of F.
It is first necessary to consider the behavior of the fitness
function over the solution space. For a given camera position,
the subset of R3for which CO=1 is, in general, bounded
by an arbitrarily complex polyhedron, depending on TT. Since
intersect is symmetric over the endpoints of the line segment,
the same can be said of the subset of possible TCfor which
CO(p)=1, for some given p∈R3(Tarabanis et al. [14] com-
pute this subset as part of their sensor planning approach, and the
complexity is evident). At such boundaries, due to the bivalent
nature of CO,F(s)is, in general, discontinuous, and this effect
produces numerous local maxima dispersed in a nonuniform
fashion throughout the solution space.
Avoiding global convergence to one of these local maxima
suggests the use of a loosely connected topology [30]. We
choose a ring topology, in which each particle has exactly two
neighbors in addition to itself, defined by
topology(S, i)=⎧
⎪
⎨
⎪
⎩
{S, 1,2},if i=1
{S−1,S,1},if i=S
{i−1,i,i+1},otherwise.
(31)
A large Simproves optimality, but requires more evalua-
tions of Fper iteration, and reduces the global convergence rate
[33]; we have found values near S=25 to perform acceptably
well while converging to the global optimum for various tasks.
Clamping velocity with κ=0.5 and setting φn>φ
pincrease
3Our implementation is written primarily in Python, which incurs significant
interpreter overhead, even with the critical loop for intersectimplemented as an
efficient C extension. Additionally, the implementation of intersect and related
code is highly tractable for SIMD parallelization.
Fig. 6. Simulated validation configuration—Dual SICK-IVP Ranger D cam-
eras with x=0 mm, h=142 mm, d=206 mm, and β=π/4image a struc-
tured target of toy bricks.
the convergence rate without overly damping exploration. Val-
ues of ω=0.7298, φp=1.1959, and φn=1.7959 yield a good
tradeoff [34].
In selecting a boundary constraint handling mechanism, we
consider that the solution space is relatively low-dimensional,
and that there is no particular expected large-scale structure of F.
The nearest method [32] improves convergence by keeping the
particles within the subspace assumed to contain the optimum,
and is defined by
constrain (s,bl,bu)=
4n
i=1
min(max(si,b
l,i),b
u,i)ˆıi(32)
where ˆıiis the ith unit vector of the solution space basis.
VII. EXPERIMENTAL RESULTS
In both simulation and real-world experimentation, we em-
ploy the Adolphus4simulation environment, which implements
the model and coverage functions described in Sections II–IV.
Thirty black toy bricks are used to rapidly construct physical
test targets for linear scans, in a diversity of shapes with known
structures. Individual bricks are modeled in Adolphus with 10
triangles per brick for occlusion, for a maximum total of 300
triangles in each target model. Various sets of between 12 and
72 task points are defined on the surface of these models. Task
requirements are varied, for both single- and dual-camera con-
figurations, to cover several cases of existence and abundance
of F=1.0 global optima for each task.
A. Validation of the Performance Metric
Comprehensive validation experiments on coverage models
including that presented in Section IV have produced exten-
sive results [21] which exceed the scope of this correspondence.
Nevertheless, as our approach fundamentally relies on the accu-
racy of these models, we summarize the more important results,
viz. those relating to occlusion and scan density. System model
parameters are obtained by calibration, so positioning error is
not considered.
The combined laser and camera occlusion, derived from the
formulation of (25) as
MO(t)=LOP−1
L(t)·max
iCO
iP−1
Ci(t)(33)
4Adolphus is free software licensed under the GNU General Public License.
Full Python source code, documentation, and sample models are available at
http://github.com/ezod/adolphus.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
10 IEEE/ASME TRANSACTIONS ON MECHATRONICS
Fig. 7. Structured target scan—Surface scan of the target obtained using
the real implementation of the system in Fig. 6. Visualized using PCL [35].
(a) Simulated target. (b) Scanned target.
Fig. 8. Validation of CD—The black line indicates the nominal Rgiven by
the relation (17), and red crosses indicate actual mean scan density in the point
cloud in the neighborhoods of the nominal task points.
may be validated against real data simply by verifying that
MO(t)is an indicator function for the existence of data at t.
Experiments on structured targets demonstrate perfect correla-
tion throughout the imaged laser plane. For the object shown in
Fig. 7, of 48 task points located on the studs of the toy bricks,
six are occluded to the laser, and a further six are occluded to
the cameras. The 36 remaining points yielding MO(t)=1are
fully imaged in the scan, while those yielding MO(t)=0are
absent from the scan.
The characterization of scanning density is validated by solv-
ing for Rin (17) for pzof a task point in camera coordinates
pand comparing to the actual horizontal density of data in
the immediate neighborhood of pin the scan. The visible sub-
set of task points on the object shown in Fig. 7 is tested in
this way at four calibrated positions of a single range camera.
Fig. 8 shows the strong correlation; overall mean squared error
is 6.344 ×10−5mm/px.
B. Comparison With Existing Methods
We compare our coverage metric and PSO algorithm with
Scott’s metric and 3M algorithm, the latter being implemented
according to the description provided in [6]. Before introducing
the experimental setup and presenting the results, we provide
some discussion on the compatibility of the coverage and task
models of both algorithms.
1) Coverage Model: Scott’s visibility criterion is equivalent
to ours, with γ=0. The focus criterion is also present in both
models; as Scott provides no details about its implementation,
we use our own implementation. Scott’s resolution criterion is
equivalent to our sampling density criterion. There is no equiv-
alent to our height resolution criterion: Scott assumes that the
TAB L E V I
COMPARISON TASK PARAMETERS,SOLUTIONS,AND PERFORMANCE
Task 3M-1 3M-2 PSO-1 PSO-2
Ri1.0 1.0 1.0 1.0
Ra1.0 1.0 1.0 1.0
Hi0.2 3.0 0.2 3.0
Ha0.2 3.0 0.2 3.0
ci1.0 1.0 1.0 1.0
ca6.0 6.0 6.0 6.0
x0.00 0.00 0.00 0.00
y353.52 353.52 38.00 330.00
z353.52 353.52 460.00 54.00
β45.00◦45.00◦85.27◦9.29◦
R0.31 0.31 0.2 0.25
H0.43 0.43 0.2 2.89
emean 0.1418 0.1418 0.0901 0.1000
%occlusion 18.7 18.7 31.17 13.7
camera and laser are fixed relative to one another. Scott’s mea-
surement precision criterion is equivalent to our view angle cri-
terion. Both models provide a performance metric in the [0,1]
range; however, 3M employs a binary metric, so we limit our
PSO algorithm to a binary metric as well.
2) Task Model: Given the aforementioned equivalences, we
adapt the 3M algorithm to our own nomenclature for task param-
eters. Since Scott only defines two task parameters, the equiv-
alents are as follows: Ri=Ra=ρs, where ρsis the sampling
density task parameter, and ζi=ζa=σs, where σsis the mea-
surement precision task parameter.
Using toy bricks, we construct a test object for inspection
(see Fig. 10). The object is made of a pair of “staircases” of
four 9.6 mm steps each. The second set is positioned directly
behind the first, introducing occlusion. We design two tasks
for this comparison. The first imposes a strict height resolution
requirement, while the second relaxes this height resolution
requirement (thereby shifting the emphasis to other factors such
as occlusion).
Camera configuration solutions are produced by both algo-
rithms for each task (see Table VI). Each configuration is im-
plemented and a 3-D model of the test object is obtained using
a SICK Ranger E50 camera (see Fig. 9). Using the scanned 3-D
model, we measure the height between the steps to obtain the
mean error in mm, the resolution in the x−yplane in mm/px,
the height resolution in the x–zplane in mm/px, and the pro-
portion of occlusion in each scan.
The results in Table VI demonstrate that although both algo-
rithms are able to meet the resolution requirement for all tasks,
only the PSO algorithm was able to meet the height resolu-
tion requirement. Furthermore, even though this requirement
was relaxed in tasks 2 and 4, only the PSO algorithm was able
to substantially reduce occlusion such that all steps of the test
object are visible (see Fig. 9). A key advantage of the PSO algo-
rithm is the ability to optimize the angle βbetween the camera
and the laser plane.
C. Optimality and Convergence of Automated Planning
Experiments on the PSO algorithm itself are conducted in
closed-loop simulation. Three targets of realistic complexity,
shown in Fig. 11, are designed along with several sets of task
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 11
Fig. 9. Range images and 3-D visualization of the test object using the 3M and PSO algorithms. (a) 3M. (b) PSO high resolution. (c) PSO low resolution. (d)
3M. (e) PSO high resolution. (f) PSO low resolution.
Fig. 10. Physical apparatus—The camera and laser source are precisely po-
sitioned, and the conveyor belt encoder provides accurate and repeatable linear
transport.
Fig. 11. Simulated task targets—Targets and task models are designed to be
difficult, to rarify F=1.0solutions.
Fig. 12. Task Performance—The PSO algorithm converges to the nominal
and expected value optima within a small number of iterations. (a) Task 1. (b)
Task 2. (c) Task 3. (d) Task 4.
TAB L E V I I
EXPERIMENTAL TASKS AND RESULTS
Task 1234
Tar get A A B C
n1222
Ri0.15 0.20 0.25 0.50
Ra0.19 0.60 0.30 2.00
Hi0.19 0.20 0.25 0.10
Ha0.23 0.60 0.30 1.00
ci1.00 1.00 1.00 1.00
ca2.00 2.00 2.00 2.00
ζi1.00 1.00 0.90 1.00
ζa1.00 1.00 0.90 π/2
ωc1.20 1.20 1.20 1.20
γ10 10 10 20
F1.0000 1.0000 1.0000 0.9999
E[F]1.0000 1.0000 1.0000 0.9161
Iterations 32 53 47 500
Evaluations 4301 4336 3166 15060
points and requirements each. Table VII lists some details for
four representative linear scan tasks, along with the statisti-
cal results of the experiments. The tasks are designed to be
difficult, such that only a small subset of the solution space
yields F=1.0, and that numerous local maxima exist. This
is achieved by positioning task points among occluding sur-
faces, and by tuning the requirements to be demanding while
admitting at least one optimal solution. In all cases, the particle
swarm parameter values and functions discussed in Section VI-
E are used, and error compensation parameters are set to ne=
20, σCT =σTT =2.0, and σCR =σTR =0.01. Termination
conditions are Faccept =1.0 and Imax =500. No clustering
terminator is used.
For each task, 48 optimizations are conducted. The bottom
portion of Table VII lists the mean values of Fand E[F]for
the best solution, the mean number of iterations to convergence
or halting, and the mean number of evaluations of the objective
function. Fig. 12 shows statistics for Fand E[F]per iteration.
A number of relevant observations can be made about these
results. The initial F(g)is virtually always nonzero and often
relatively high, indicating that, for most tasks, the bounds de-
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12 IEEE/ASME TRANSACTIONS ON MECHATRONICS
Fig. 13. Robustness to positioning error—Increasing neimproves the solution
but increases computation time. (a) Expected value of Fwith pose error (300
Trials). (b) Mean total evaluations of F.
fined in Table V are a tight initial guess about the location of
the optimum. Although occasional temporary stagnation is ob-
served, for tasks 1, 2, and 3, the algorithm converges quickly to
the nominal optimum, and the expected value optimum follows
shortly thereafter.
Task 4 exemplifies what occurs when no nominal optimum
exists; as the algorithm incrementally improves its estimate of
the optimum, the stochastic E[F]fluctuates slightly, and eventu-
ally settles. This behavior can be mitigated using the clustering
termination condition.
D. Robustness to Positioning Error
A series of 48 optimizations are conducted on a similar model
for each ne∈{0, 5, 10, 15, 20, 25, 30}, with positioning error
modeled with σCT =σTT =2.0 and σCR =σTR =0.01. All
solutions converged to a nominal F=1.0.
Fig. 13(a) shows the mean value of F, obtained over 300
trials with the same error parameters, for each of the 48 solu-
tions for each value of ne. Fig. 13(b) shows the mean number of
evaluations of Fconducted in the original optimizations (i.e.,
those required to produce the solutions). As expected, the re-
lationship between the performance cost and the accuracy in
finding robust solutions is roughly linear, for fixed optimization
parameters and a fixed number of iterations.
E. Industrial 3-D Gage Device Implementation
This method has been successfully applied to the design of an
industrial noncontact 3-D measurement device currently mar-
keted by Vista Solutions, Inc. This hybrid-modality device per-
forms some conventional machine vision tasks using an inten-
sity image as well as range sensing tasks using triangulation
with three line laser sources. The PSO algorithm was actually
leveraged to optimize over both the coverage model presented
in Section IV (for each of the three laser sources) as well as a
related model for conventional vision [21] from which it was
derived. The final design is shown in Fig. 14.
Fig. 14. Final design of 3-D gage device—The design is optimized over both
triangulation and conventional vision models using the PSO algorithm.
The 3-D gage device serves as an example of real-world spec-
ification of the task model described in Section II. Requirements
were specified by the application engineers, directly based on ac-
tual tolerances specified by the customer. For the range imaging
modality, a scan density of Ri=Ra=0.05 mm and a height
resolution of Hi=Ha=0.05 mm were specified. Experimen-
tation with the range imaging software suggested ci=1.0 and
ca=10.0 as reasonable constraints for the blur circle. Based
on the software documentation, ζi=ζa=1.22 and ωc=1.22
were chosen for the maximum viewing angle and cutoff angles.
The γparameter was unused.
As the device is, in this case, a generic design, no target
geometry or specific set of task points were specified. Instead, a
regular “grid” of task points 50 mm wide (along the xaxis) and
30 mm high (along the zaxis), corresponding to the specified
field of view, was used.
VIII. CONCLUSION
In this paper, we have presented a novel view planning ap-
proach for high-accuracy 3-D inspection systems based on active
triangulation. Rather than attempting to optimize over a large
set of design variables, with the assistance of expert integration
engineers, we have analyzed the design process, and reduced
the critically difficult portion to only four design parameters per
camera. The result is a semiautomatic process which leverages
human expertise to create a basic system design, for which the
camera parameters are then refined through automatic optimiza-
tion.
The first contribution is a high-fidelity task-oriented model
of coverage performance, which evaluates the overall design—
including manual components—with a bounded scalar metric,
and also serves as the objective function for optimization. This
model is based on research into general high-fidelity representa-
tions of multiple camera coverage for other related optimization
problems. Our validation experiments demonstrate its effective-
ness in predicting a posteriori task performance, so that op-
timized designs can be reliably tested in simulation, moving
costly hardware testing further outside the cycle.
The second contribution is the identification of the critical
part of the design process, and the successful application of a
particle swarm optimization algorithm to this reduced problem.
Although the high-fidelity objective function is expensive to
compute, experimental evaluation demonstrates that, with ap-
propriate parameters, the algorithm reliably converges to opti-
mal solutions within a small number of iterations. The algorithm
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
MAVRINAC et al.: SEMIAUTOMATIC MODEL-BASED VIEW PLANNING FOR ACTIVE TRIANGULATION 3-D INSPECTION SYSTEMS 13
also directly compensates for positioning system uncertainty for
the cameras and target.
Actual convergence times vary widely, depending on the task
and number of cameras; typical “difficult” problems with local-
ized optima tend to complete on the order of 10 min, using our
implementation on modern (single-core) hardware. In general,
at these rates, the algorithm is already expected to far outperform
human experts, supported by its successful application presented
in Section VII-E. With further optimization and parallelization,
it should be possible to substantially reduce convergence times.
ACKNOWLEDGMENT
The authors thank Vista Solutions Inc. for generously provid-
ing expertise, equipment, and facilities for experimentation.
REFERENCES
[1] P. J. Besl, “Active, optical range imaging sensors,” Mach. Vis. Appl.,vol.1,
no. 2, pp. 127–152, 1988.
[2] F. Blais, “Review of 20 years of range sensor development,” J. Electron.
Imag., vol. 13, no. 1, pp. 231–243, 2004.
[3] W. R. Scott, G. Roth, and J.-F. Rivest, “View planning for automated
three-dimensional object reconstruction and inspection,” ACM Comput.
Surveys, vol. 35, no. 1, pp. 64–96, 2003.
[4] K. A. Tarabanis, P. K. Allen, and R. Y. Tsai, “A survey of sensor planning
in computer vision,”IEEE Trans. Robot. Autom., vol. 11, no. 1, pp. 86–104,
Feb. 1995.
[5] S. Larsson and J. A. P. Kjellander, “Path planning for laser scanning with
an industrial robot,” Robot. Auton. Syst., vol. 56, no. 7, pp. 615–624, 2008.
[6] W. R. Scott, “Model-based view planning,” Mach. Vis. Appl., vol. 20,
no. 1, pp. 47–69, 2009.
[7] C. C. Yang and F. W. Ciarallo, “Optimized sensor placement for active
visual inspection,” J. Robot. Syst., vol. 18, no. 1, pp. 1–15, 2001.
[8] Q. Shi, N. Xi, and Y. Chen, “Development of an automatic optical measure-
ment system for automotive part surface inspection,” in Proc. IEEE/ASME
Int. Conf. Adv. Intell. Mechatronics, 2005, pp. 1557–1562.
[9] F. A. Rodrigues Martins, J. G. Garc´
ıa-Bermejo, E. Zalama Casanova, and
J. R. Per´
an Gonz´
alez, “Automated 3D surface scanning based on CAD
model,” Mechatronics, vol. 15, pp. 837–857, 2005.
[10] B. Englot and F. Hover, “Inspection planning for sensor coverage of 3D
marine structures,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2010,
pp. 4412–4417.
[11] G. H. Tarbox and S. N. Gottschlich, “Planning for complete sensor cov-
erage in inspection,” Comput. Vis. Image Understanding, vol. 61, no. 1,
pp. 84–111, 1995.
[12] J. L. Alarcon Herrera, A. Mavrinac, and X. Chen, “Sensor planning for
range cameras via a coverage strength model,” in Proc. IEEE/ASME Int.
Conf. Adv. Intell. Mechatronics, Jul. 2011, pp. 838–843.
[13] C. K. Cowan and P. D. Kovesi, “Automatic sensor placement from vision
task requirements,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 10, no. 3,
pp. 407–416, May1988.
[14] K. A. Tarabanis, R. Y. Tsai, and A. Kaul, “Computing occlusion-free
viewpoints,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 3, pp.
279–292, Mar. 1996.
[15] A. Mavrinac and X. Chen, “Modeling coverage in camera networks: A
survey,” Int. J. Comput. Vis., vol. 101, no. 1, pp. 205–226, 2013.
[16] J. Salvi, C. Matabosch, D. Fofi, and J. Forest, “A reviewof recent range im-
age registration methods with accuracy evaluation,” Image Vis. Comput.,
vol. 25, pp. 578–596, 2007.
[17] P. Saint-Marc, J.-L. Jezouin, and G. G. Medioni, “A versatile PC-based
range finding system,” IEEE Trans. Robot. Autom., vol. 7, no. 2, pp. 250–
256, Apr. 1991.
[18] J. L. Vilac¸a, J. C. Fonseca, and A. M. Pinho, “Calibration procedure for
3D measurement systems using two cameras and a laser line,” Opt. Laser
Technol., vol. 41, no. 2, pp. 112–119, 2009.
[19] A. Mavrinac, X. Chen, P. Denzinger, and M. Sirizzotti, “Calibration of
dual laser-based range cameras for reduced occlusion in 3D imaging,” in
Proc. IEEE/ASME Int. Conf. Adv. Intell. Mechatronics, 2010, pp. 79–83.
[20] N. P. Suh, The Principles of Design. London, U.K.: Oxford Univ. Press,
1990.
[21] A. Mavrinac, “Modeling and optimizing the coverage of multicamera
systems,” Ph.D. dissertation, Univ. Windsor, Windsor, ON, Canada, 2012.
[22] T. M¨
oller and B. Trumbore, “Fast, minimum storage ray/triangle intersec-
tion,” J. Graph. Tools, vol. 2, no. 1, pp. 21–28, 1997.
[23] C. C. Yang, M. M. Marefat, and F. W. Ciarallo, “Error analysis and plan-
ning accuracy for dimensional measurement in active vision inspection,”
IEEE Trans. Robot. Autom., vol. 14, no. 3, pp. 476–487, Jun. 1998.
[24] Y. Ma, S. Soatto, J. Koˇ
seck´
a, and S. S. Sastry, An Invitation to 3-D
Computer Vision. New York, NY, USA: Springer-Verlag, 2004.
[25] D. C. Brown, “Decentering distortion of lenses,” Photogrammetric Eng.,
vol. 32, no. 3, pp. 444–462, 1966.
[26] Y.-S. Chen, S.-W. Shih, Y.-P. Hung, and C.-S. Fuh, “Simple and efficient
method of calibrating a motorized zoom lens,” Image Vis. Comput., vol. 19,
no. 14, pp. 1099–1110, 2001.
[27] G. Fu, P. Corradi, A. Menciassi, and P. Dario, “An integrated triangulation
laser scanner for obstacle detection of miniature mobile robots in indoor
environment,” IEEE/ASME Trans. Mechatronics, vol. 16, no. 4, pp. 778–
783, Aug. 2011.
[28] J. Kennedy and R. C. Eberhart, “Particle swarm optimization,” in Proc.
IEEE Int. Conf. Neural Netw., 1995, pp. 1942–1948.
[29] J. Kennedy and R. C. Eberhart, Swarm Intelligence. San Mateo, CA, USA:
Morgan Kaufmann, 2001.
[30] J. Kennedy and R. Mendes, “Population structure and particle swarm
performance,” in Proc. IEEE Congr. Evol. Comput., 2002, pp. 1671–1676.
[31] R. Mendes, “Population topologies and their influence in particle swarm
performance,” Ph.D. dissertation, , Dept. Informatica, Escola Engenharia,
Univ. Minho, Portugal, 2004.
[32] S. Helwig and R. Wanka, “Particle swarm optimization in high-
dimensional bounded search spaces,” in Proc. IEEE Swarm Intell. Symp.,
2007, pp. 198–205.
[33] I. C. Trelea, “The particle swarm optimization algorithm: Convergence
analysis and parameter selection,” Inf. Process. Lett., vol. 85, pp. 317–
325, 2003.
[34] M. Clerc and J. Kennedy, “The particle swarm—Explosion, stability, and
convergence in a multidimensional complex space,” IEEE Trans. Evol.
Comput., vol. 6, no. 1, pp. 58–73, Feb. 2002.
[35] R. B. Rusu and S. Cousins, “3D is here: point cloud library (PCL),” in
Proc. IEEE Int. Conf. Robot. Autom., 2011, pp. 1–4.
Aaron Mavrinac received the B.A.Sc., M.A.Sc., and
Ph.D. degrees in electrical and computer engineer-
ing from the University of Windsor, Windsor, ON,
Canada, in 2005, 2008, and 2012, respectively.
Since 2012, he has been working on research and
development on driver assistance and highly auto-
mated driving technologies at Robert Bosch LLC,
Plymouth, MI, USA. His research interests include
coverage analysis, calibration, and data fusion in het-
erogeneous multisensor systems.
Xiang Chen received the M.Sc. and Ph.D. degrees
from Louisiana State University, Baton Rouge, LA,
USA, in 1996 and 1998, respectively, both in systems
and control.
Since 2000, he has been with the Department of
Electrical and Computer Engineering, University of
Windsor,Windsor, ON, Canada, where he is currently
a Professor. His research interests include robust con-
trol, vision sensor networks, vision-based control sys-
tems, networked control systems, and industrial ap-
plications of control theory.
Jose Luis Alarcon-Herrera receivedthe B.A.Sc. de-
gree in electrical engineering from the Durango Insti-
tute of Technology, Durango, Mexico, in 2009. Since
2010, he has been working toward the Ph.D. degree
at the University of Windsor, Windsor, ON, Canada.
His research interests include multicamera net-
works, computer vision, and vision-guided robotics.
