Management of tracking for industrial AR setups

Abstract

The accuracy of a real time tracking system for industrial AR (IAR) applications often needs to comply with production tolerances. Such a system typically incorporates different off-/online devices so that the overall precision and accuracy cannot be trivially stated. Additionally, tracking needs to be flexible to not interfere with existing working processes and it needs to be operated and maintained free of error by on-site personnel who typically have a quality management (QM) background. For the final validation of such a complex tracking setup, empiric testing alone is either too expensive or lacks generality. This paper demonstrates a new approach to define and verify, deploy and validate, as well as to operate and maintain an IAR tracking infrastructure. We develop our concepts on the basis of an IAR application in the field of QM in the aircraft production process. It integrates a qualitative visual comparison with accurate quantitative measurements of 3D coordinates using a metrological probe. The focus is on the verification, validation, and error free operation. Monte Carlo simulation predicts the error for arbitrary system states. Using a limited set of empiric measurements in the target environment allows us to validate the simulation and thereby validate the application. This combination assures compliance of the IAR application with the required production tolerances. We show that our simulation model yields realistic results, using an in-depth analysis of an optical IR tracking system and a high-precision coordinate measurement machine capable of densely sampling the entire tracking volume. Additionally, it allows for a straightforward derivation of run-time consistency checks for the automatic identification of possible system failures. Also, estimation of the system performance during the planning and definition phases becomes possible, using the elementary accuracy specifications of the involved sensor systems.

Full text

Page 1

Management of Tracking for Industrial AR Setups
Peter Keitler∗
Technische Universit¨ at
M¨ unchen
Benjamin Becker†
EADS Innovation Works
Gudrun Klinker‡
Technische Universit¨ at
M¨ unchen
ABSTRACT
The accuracy of a real time tracking system for industrial AR
(IAR) applications often needs to comply with production toler-
ances. Such a system typically incorporates different off-/online
devices so that the overall precision and accuracy cannot be triv-
ially stated. Additionally, tracking needs to be flexible to not inter-
fere with existing working processes and it needs to be operated and
maintained free of error by on-site personnel who typically have a
quality management (QM) background. For the final validation of
such a complex tracking setup, empiric testing alone is either too
expensive or lacks generality.
This paper demonstrates a new approach to define and verify, de-
ploy and validate, as well as to operate and maintain an IAR track-
ing infrastructure. We develop our concepts on the basis of an IAR
application in the field of QM in the aircraft production process. It
integrates a qualitative visual comparison with accurate quantitative
measurements of 3D coordinates using a metrological probe. The
focus is on the verification, validation, and error free operation.
Monte Carlo simulation predicts the error for arbitrary system
states. Using a limited set of empiric measurements in the target
environment allows us to validate the simulation and thereby val-
idate the application. This combination assures compliance of the
IAR application with the required production tolerances.
We show that our simulation model yields realistic results, using
an in-depth analysis of an optical IR tracking system and a high-
precision coordinate measurement machine capable of densely
sampling the entire tracking volume. Additionally, it allows for
a straightforward derivation of run-time consistency checks for the
automatic identification of possible system failures. Also, estima-
tion of the system performance during the planning and definition
phases becomes possible, using the elementary accuracy specifica-
tions of the involved sensor systems.
1
A tracking infrastructure is required in all industrial AR (IAR) ap-
plications to perceive and interact simultaneously with real and vir-
tual objects. Especially in the naval and avionic industry, products
are very complex and highly customized. The high rate of manual
work is anopportunity to create new toolsusing AR technology that
have the potential to decrease production time and cost. Thereby,
the reliability, robustness, and accuracy of the tracking infrastruc-
ture is an important factor for the success of such applications. Be-
sides the tracking system, there are various other application con-
straints towards usability and work safety that have to be dealt with,
but which are out of the scope of this paper. A detailed description
of the challenge to bring AR out of the laboratory into an industrial
context is provided in [6].
An exemplary IAR application for the support of quality man-
agement (QM) processes in the avionic industry is depicted in Fig-
MOTIVATION
∗e-mail: keitler@in.tum.de
†e-mail:benjamin.becker@eads.net
‡e-mail:klinker@in.tum.de
Figure 1: Industrial Augmented Reality
Figure 2: Metrological Measurement & Tracking Device
ure 1, where a worker sees an x-ray view on the electric wiring. Be-
sides qualitative comparison by video overlay or display of purely
virtual content, there is a need for quantitative measurements in real
time, e.g. by using a probing device as depicted in Figure 2. There-
fore, acertainlevelofaccuracyhastobeguaranteedforallspecified
modesofoperation. ThepurposeistouseARasanormaltool, fully
integrated into the working process. However, many factors might
complicate the design of a suitable tracking infrastructure. Tools
within the existing work procedures in the target environment need
to be robustly tracked using added markers. Also for registration of
the virtual and the real model, markers have to be attached to the
product itself. Besides typical issues like occlusion and reliability,
such modifications are strictly limited in order not to compromise
the existing process and product.
Especially in the avionics industry, another problem is the large
potential area the tracking system should be used in. This requires
solutions to increase the flexibility of the tracking system. Man-
ufacturing procedures might require removal and reinstall of the
tracking system, invalidating the registration and resulting in re-
calibration and maintenance effort. Ideally, such procedures can
be performed through on-site personnel without requiring detailed

Page 2

technological insight by providing an appropriate user interface.
In most common IAR scenarios, these issues exist at least par-
tially. Often, they can only be addressed by strongly limiting the
use case scenario or by the integration of various sensor devices.
Especially for those complex IAR setups it is challenging to assure
compliance with requirements like production tolerance. This has
to cover the entire error chain, consisting of the individual sensor
errors accumulated and propagated throughout the entire system
setup. This therefore covers not only the real-time error but also
registration procedures of rigidly mounted sensors, objects, and
markers, based on error-prone measurements [10]. Typical generic
vendor specifications are often insufficient to provide a reliable as-
sessment of the overall tracking system performance in the explicit
use case scenario.
The goal of this paper is not to find solutions to all these IAR
problems1. Many solutions exist in the literature to provide prag-
matic solutions to special problems, e.g. [9][14]. Rather, general
guidelines shall be provided to handle the complexity of the track-
ing setup, with a focus on the need in IAR for a well-defined track-
ing accuracy.
Following classic approaches from the metrological domain in
production engineering, the validation of the tracking infrastructure
would have to be accomplished through exhaustive empiric mea-
surements, see e.g. [22][23]. This is affordable only for single
systems such as a mechanical measurement arm or a theodolite that
can easily be analyzed in a controlled setup. A complex tracking
infrastructure incorporates several sensor systems that potentially
have to be rigidly installed in the target environment, making the
acquisition of empiric measurements for comparison very difficult.
In this paper we propose to combine a small set of representative
empiric measurements with a comprehensive simulation to cover
all possible constellations and system configurations in the use case
scenario. We rely on a tracking framework to model the simula-
tion using standardized and integrated components for all neces-
sary computations. This approach requires identifying the degree
to which the simulation model is an accurate representation of the
real world [2].
After introducing the relevant literature in this field, Section 3
describes our approach for the various design and implementation
stages of an IAR application. Section 4 details our exemplary IAR
scenario depicted in Figure 1&2. Section 5 describes in detail the
application of our approach which is then evaluated in Section 6 on
thebasisofourexemplaryscenario. Itwillbeshownthatsimulation
methods scale well to such complex setups and provide reasonable
and valid results.
2
Prior work to analyze the accuracy of tracking setups and sys-
tems can mainly be categorized in online and offline error estima-
tion/propagation techniques and the analysis of individual systems.
Online error propagation approaches such as [3][5][9] provide
an estimate of the error in real time. It is based on the current sys-
tem state, e.g. distances between camera and marker as well as
the visibility of fiducials. Due to performance reasons, the predic-
tion throughout the entire error propagation chain usually requires
the linearization of each computational step. This simplification is
fully sufficient to give the user an real-time insight into the current
system performance. In IAR however, the validation of a track-
ing system requires an assessment of the estimated overall tracking
accuracy before deployment.
Offline techniques are split into two domains, simulation and
system analysis using ground truth measurements. Simulation tech-
niques have been applied successfully to assess the accuracy of of-
fline photogrammetric methods or metrological measurement de-
RELATED WORK
1althoughwehadtotacklesomeofthedescribedissuesinourexemplary
scenario, of course
vices such as theodolites or laser trackers as well as real time track-
ing systems. The goal is to validate a system or to detect the main
sources of error and to point out optimization potentials. Penten-
rieder et al. use a Monte Carlo simulation approach to predict the
accuracy of an optical square marker tracker by automatic gener-
ation of artificial camera images [17]. Hastedt also uses a Monte
Carlo approach to simulate the behavior of a photogrammetric sys-
tem [8]. Both validate the simulation approach using high-precision
measurement data. Simulation has also been used to assess algo-
rithm performance [4][7].
’Ground truth’ measurements have also been used many times to
benchmark various types of tracking systems. Schmidt et al. use a
high precision linear stage to evaluate various IR tracking systems
manufactured by NDI2along the three spatial axes [21]. Satoh et
al. use an industrial robot to move an HMD through the tracking
volume to evaluate the VICON3and LaserBird4optical tracking
systems [20]. They provide a framework which is applicable to
various systems but requires dense ground truth data. However, the
practical use of such a framework is limited since it requires a robot
to provide dense empirical measurement data, which is not feasi-
ble in a typical IAR target environment, as described in Section 1.
Also, no general statement for design criteria such as the world reg-
istration procedure or the layout of a marker, which have a strong
impact on the overall accuracy, is possible. Although an elemen-
tary uncertainty specification is determined, it can hardly be reused
because typically too many factors will differ, even in similar sce-
narios. This is also covered by our own experience with optical IR
tracking [14]. Other works restrict themselves to the assessment of
an individual system. Lieberknecht et al. use a FARO measure-
ment arm to generate ground truth image data which is then used to
assess the quality of different markerless tracking algorithms [15].
Similar work has been done in the medical domain where Rohling
benchmarked the tracking of medical instruments using a FARO
measurement system as a ground truth [19]. Even vendors provide
insight into their systems to propose means to detect invalid cali-
bration [23].
Allen et al. presented a general method for the evaluation and
comparison of the expected performance of tracking systems [1].
Their approach even incorporates scene dynamics and provides ad-
vanced visualization concepts. However, their simulation considers
only a single tracking system using a moving target and a pose esti-
mation algorithm. The required registration and calibration proce-
dures within IAR scenarios are not covered.
3
It is important to provide means to setup and implement robust and
maintainable tracking infrastructures to allow AR applications to
spread in the industrial domain. We already proposed the graph-
ical tool trackman for this purpose [13]. It is based on the Ubi-
track5library. The spatial relationship graph (SRG) and spatial re-
lationship pattern concept allows for the creation of efficient and
semantically correct runtime data flow descriptions for arbitrary
calibration/registration and runtime tracking problems [18]. This
generic approach shall now be strengthened, to obtain an integrated
approach for the management of tracking in industrial setups.
Three important development phases can be distinguished in an
industrial context. During the definition phase, the tracking system
is planned, algorithms are implemented or combined, and buying
decisions are taken. Then, the planned system is installed in the tar-
get environment. This deployment phase also incorporates the ini-
tial calibration of devices and registration of spatial transformations
APPROACH
2www.ndigital.com
3www.vicon.com
4www.ascension-tech.com
5campar.in.tum.de/UbiTrack/WebHome

Page 3

between sensors, markers, and objects. Subsequently, the system is
used productively in the operation and maintenance phase.
AsdisplayedinFigure3, weextendthedefinitionphasebysimu-
lation based verification, the deployment with a validation step and
the final tracking system operation phase by run-time checks that
detect system malfunction and propose mitigation steps to the user.
Definition
Verification
based on simulation of the tracking setup
Deployment
Validation
based on simulation and empiric
measurements in target environment
Operation and
Maintenance
Runtime Error Mitigation
using runtime error estimation and system
failure detection
Figure 3: Proposed development process for an industrial tracking
infrastructure
To verify the design we propose to not only rely on analyzing the
stand-alone tracking system in detail but to expand the evaluation
to the entire error chain of the IAR application. Using the elemen-
tary uncertainty specifications of the involved sensor systems in a
Monte Carlo based simulation, it is possible to derive the behav-
ior of the complete setup. Since the simulation approach requires
known uncertainties of the used tracking devices, the analytic eval-
uation as presented in Section 2 is of great importance. Using our
simulation it is possible to benchmark various hypothetical scenar-
ios without needing real hardware. Thereby, it can help to state the
general feasibility of a setup, to justify buying decisions, and to de-
cide on which particular algorithms to use. The final goal of the
simulation is to verify the overall concept and to decide on a certain
variant before deployment.
Generally, the correctness of a simulation system for a certain
purpose first needs to be proven [2]. This assures that the assump-
tions made are specific enough to obtain realistic results. Based on
our use case, we demonstrate that existing simulation approaches
used to evaluate individual tracking systems and algorithms (cf.
Section 2) can be extended to a more complex IAR tracking setup.
Following industrial standards, classical validation would re-
quire exhaustive empirical measurements to validate the entire sys-
tem operation [2][22]. As shown in Figure 3, we reduce the effort
using the simulation to validating that the simulation represents a
realistic system behavior. If the assumptions for the simulations are
incorrect for the productive environment, a redefinition of the sys-
tem might be required, resulting in an iteration with the definition
phase (red arrow).
As proposed in Figure 3, such detailed system knowledge allows
to implement runtime error mitigation using consistency checks for
the automatic identification of possible system malfunction. These
checks are based on the detection of violations of systematic and
random error limits that were developed during the definition and
deployment stages. Those can be interpreted and traced back to a
certain system component and propose mitigation steps to the user.
To allow for Monte Carlo simulation of a planned tracking setup,
an elementary uncertainty specification is needed for all involved
sensors:
Abstraction Level:
As input, our simulation framework uses
sensor uncertainty specifications at different abstraction levels. De-
pending on the used system, it can be given for the individual 2D
measurements of a camera, for the 3D positions of fiducials, or for
the 6DoF pose of a marker. A simple isotropic error model or a
more general covariance matrix are supported. The decision on
simulation system granularity highly depends on the used system
setup. If the layout and the amount of fiducials allowed for marker
design are predefined, it could make sense to specify a 6DoF pose
error. Similarly, in an optical multi-camera tracking setup with ar-
bitrary camera arrangement, rather a 2D value is specified since the
uncertainty of a 3D position of a fiducial depends on the amount,
distances, and distribution of cameras. Sensor uncertainties can and
should be specified at a level that allows for maximum generality.
This concerns specifications provided by system vendors as well as
third-party accuracy assessments (cf. Section 2).
Error Distribution: According to the Guide to the Expression
of Uncertainty in Measurement (GUM) [12], sensor noise may fol-
low different error distributions. By default, one might assume a
Gaussian error distribution. However, there are tracking systems
where global systematic distortions of the tracking volume domi-
nate the overall error behavior [14]. In such cases, a uniform distri-
bution could be more realistic in order not to underestimate extreme
values. Also a combination of different error distribution might be
used since systems often suffer from sensor noise as well as sys-
tematic errors.
Error Magnitude: Finally the magnitude of the error has to be
provided. Again, this can be either taken from the detailed system
analysis or from the system vendor specification. Also the degree of
the noise depends on the setup and on environmental influences. If
theactualamountoferrorisnotknownapriori, simulationwithdif-
ferent assumed error levels can still help to define the maximum al-
lowed sensor noise to stay in-line with the overall application spec-
ifications.
4
AtypicalQMuserneedstoqualitativelycomparethedigitalandthe
physical product using a visualization device (see Figure 1). Due to
the size and complexity of the aircraft, there is a strong benefit in
tracking this device to offer a real-time view on the digital model.
Additionally, it is of great importance to assess production qual-
ity by quantitatively validating the location of new parts through
length or coordinate measurements. This is usually done using a
metrological device (see Figure 2). Both are integrated into an IAR
application which is used integrated in the production processes.
Already experts use metrology systems to provide measurements
within the production environment. For these precise offline mea-
surements within a production environment as large as our aircraft,
photogrammetric or laser based metrology systems are used. Such
systems are less easily integrated in the standard production pro-
cesses and often require a fixed registration sensitive to vibrations
and therefore interrupt other production process running in parallel.
To cover the large area of the aircraft where the QM procedures
are performed, it is not feasible to deploy countless tracking sys-
tems. Not only would this result in high cost but also would the
permanent installation of tracking hardware collide with other pro-
duction processes. Rather a quick setup and dismantling procedure
is required. We therefore use a mobile tracking system that is able
to reference itself within the aircraft and to perform the real time
tracking of the visualization and probing devices. This is done by
adding reference targets to the aircraft. Their static transformation
within the aircraft’s coordinate frame is determined in a registration
routine using an offline metrological system. Therefore, this could
be called an indirect tracking setup, see also [14].
For application in the QM process, it is fundamental to assess
whethertheaccuracyofthetrackingsetupisinlinewiththeproduc-
tion tolerances. This requires assessing the accuracy of the tracking
system throughout the entire error chain from registration to real
time tracking. Evaluation using ground truth measurements in the
target environment is unfeasible due to the huge operating volume
paired with varying situations and conditions. Therefore, we apply
our approach using simulation throughout the development phases.
USE CASE SCENARIO

Page 4

Reference
Target 1
CAD
Reference
Target 1
Realtime
Tracker
Probe
Probe
Tip
World
Figure 4: Use case spatial relationship graph (SRG)
Figure 4 shows the spatial relations between the involved enti-
ties. Nodes represent coordinate frames, edges depict spatial trans-
formations. Initially, the reference targets are added to the aircraft.
They are precisely manufactured and can be detected by the offline
metrological system. Measuring multiple points on the CAD model
of the reference target allows computing its 6DoF rigid transforma-
tion in the aircraft (World→Cad-Reference-Target) using an abso-
lute orientation algorithm, e.g. [11]. This measurement is prone to
the error of the offline metrological system.
Second, we add a real time IR tracking system which can iden-
tify and estimate the 6DoF pose of LED targets such as shown
in Figure 2. The system computes the pose of the targets inter-
nally using an unknown absolute orientation algorithm (Real Time-
Tracker→Probe, Real Time-Tracker→Reference-Target).
pose transformations are subject to error due to the positional error
of each LED. Additionally, to use the probing device, a tip calibra-
tion of the 3D offset between the LED target and the tip is required.
Using many samples leaves a negligibly small error on this calibra-
tion routine.
The final missing registration is between the CAD model and
the LED marker of the reference marker. This is done by measur-
ing 25 known reference points6on the CAD model with the probe
and computing the transformation using an absolute orientation al-
gorithm (Reference-Target→Cad-Reference-Target, see Figure 5).
Completing the required registrations allows to compute the po-
sition of the probe tip in the world (dashed edge) - the basic for our
IAR application. This is derived by applying trivial inversions and
concatenations to the transformations in the depicted SRG [18][13].
The tablet PC for qualitative visualization has been omitted for sim-
plicity, it can be tracked analogously to the probe.
These
5
This section explains the application of our simulation-based con-
cept for verification, validation, and runtime error mitigation de-
picted in Figure 3. The results, as well as the correctness of our
approach, will be addressed in Section 6.
CONCEPT
5.1
During the definition phase, the proposed system setup shall be
checked for compliance with the application requirements, by
means of a suitable verification process. First of all, the applica-
tion data flow has to be modeled on the basis of the SRG depicted
in Figure 4. The application SRG serves as a basis for the data flow
descriptions to be instantiated using the tracking framework [13].
Since the simulation has to cover the entire error chain, also the
registration data flows are required. They are obtained by a refine-
ment of the application SRG (cf. Figure 4). In our case the level of
abstraction requires to explicitly formulate the algorithms that are
Verification
6A large number is used to mitigate the influence of probing errors.
actually performed inside the black-box tracking system to use the
sensor accuracy description that is given for a single LEDs only.
CAD
Reference
Target
Realtime
Tracker
Probe
Tip
25 Tactile
Feature Points
4 LED Probe
Feature Points
6 LED
Feature Points
Reference
Target
Probe
=
Figure 5: Refined SRG
Figure 5 depicts this situation. There are two absolute orienta-
tions that are performed internally in the real time tracking device
locating the LED features in its coordinate frame (dashed edge).
The calibration routine to derive the transformation between the
reference target used by the real time tracking and its CAD model
uses the CAD features measured by the tip of the probe (Tactile-
Feature-Points=Probe-Tip). The cycle in the SRG allows comput-
ing the dotted transformation using an absolute orientation.
Simulation Preconditions
For the simulation of the entire er-
ror chain, all registration procedures as well as the final application
data flow have to be simulated using the elementary uncertainty
specifications, i.e. the abstraction level as well as the distribution
and magnitude of the errors (cf. Section 3). In our case this is less
complex since the system is pre-calibrated and uses three rigidly
installed cameras (see Figure 2).
The system provides 6DoF poses of the probe and reference tar-
gets. Nevertheless it is not feasible to specify the error at that ab-
straction level since we use custom marker layouts. The error of
6DoF pose tracking highly depends on layout and dimension and
therefore cannot be stated in a general way by the vendor. Applying
the error to the 3D positions of the individual LEDs is appropriate
since this is the provided elementary uncertainty specification7for
the position of a tracked LED. An uncertainty specification on the
sensor’s image plane using 2D noise would also be possible, how-
ever internal details about the intrinsic and extrinsic camera param-
eters would be required for this. It is apparent that the used system
and setup define the used granularity.
Forourusecasewedecidedtouseanon-isotropicGaussianerror
distribution to approximate the real error distribution, following the
vendor’s elementary uncertainty specification with an overweight
error in the depth direction. This might not be a perfect assumption
for the complex systematic error we have found in our in-depth
analysis but it is a sufficient bound.
Simulation Setup
The simulation was divided into two parts:
in the first step, static spatial transformations are estimated (regis-
tration), then the overall application accuracy is estimated (opera-
tion) in a second step. During operation, all pre-calibrated trans-
formations (colored in Figure 4) from prior registration are used,
together with their estimated covariances.
Ground Truth Data: The simulation system shall imitate the
expected constellations of the planned real system. For this, hypo-
thetical data has to be provided in terms of assumed spatial relation-
ships. This also incorporates static transformations that might be
known from preceding simulation steps. This synthetic data repre-
sents a ground truth; it does not contain any measurement errors. In
7www.ndigital.com/industrial/certushd.php, RMS error [mm] for hori-
zontal/vertical/depthdirections(dependingondepth): 0.1/0.1/0.15[mm](at
2m), 0.15/0.15/0.25[mm] (at 4m), 0.25/0.25/0.45[mm] (at 6m)

Page 5

our scenario this includes exemplary poses for the reference target
in the airplane, possible poses of the real time tracker with respect
to the reference target, as well as an exemplary grid of probe poses
in the volume of the real time tracker for which the uncertainties
will be simulated. Also the layouts of tracking targets are specified,
in our case defining the LED constellations. They can typically
be obtained directly from the proprietary target calibration routines
provided by the system vendor.
Synthetic Measurements: Based on the ground truth data, syn-
thetic measurements are generated. To apply the sensor noise, the
ground truth data needs to be transformed into the coordinate sys-
tem(s) in which the elementary sensor uncertainty specification is
provided. By sampling from the given probability density function
the data is perturbed accordingly. Using the estimated covariance,
noise is also applied to the static transformations from precedent
simulation steps. In our case, the LED positions in the coordinate
system of the real time tracker have to be derived from the assumed
6DoF poses of the targets and the relative 3D offsets of the LEDs
belonging to these targets. Another example would be to transform
3D points to the image plane of a camera and apply 2D pixel noise
there, if the elementary uncertainty specification were given at that
level.
Registration/Tracking Algorithm:
tial transformations are propagated through the SRG to the coor-
dinate frame in which they are required for the intended calibra-
tion/registration or application. This represents the normal opera-
tion of a Ubitrack data flow. Various kinds of spatial relationship
patterns are at our disposal, besides the trivial inversion and multi-
plication, this comprises many common calibration and registration
methods. The absolute orientation pattern using 3D-3D point corre-
spondences is just one example, see also [18]. In the last simulation
step of our evaluation, the tip of the probing device is estimated in
world coordinates. Similarly, one could also estimate the 2D over-
lay error on an HMD by propagating the error to its image plane.
Covariance Estimation:
Now, the previous two steps are re-
peated iteratively. Samples are produced by perturbation and then
propagated to the coordinate frame of interest by running the corre-
sponding algorithms. Accumulating those, using descriptive statis-
tics, the covariance associated with the registration or tracking re-
sult can be estimated [16][12].
To better understand this approach, we give an exemplary de-
scription of estimating the covariance of the registration of the ref-
erence target using 25 tactile points, based on our refined SRG
depicted in Figure 5. Our synthetic data describes that the tactile
points will be probed consecutively. Also, the real time tracking
system is positioned in the setup so that the reference target is at an
optimal distance with minimal error.
To apply the noise in our setup, we use the ground truth of the
probe keeping the tip fixed on the one of the tactile points. We de-
rive the location of the LEDs of the probe marker and the reference
target in the coordinate frame of the real time tracking. There, the
Gaussian noise is applied to each LED position. Using an absolute
orientation algorithm, an error-prone estimate for the probe and the
reference target and consequently for the tactile point and the probe
tip is computed. Iterating over the tactile points, we derive 25 cor-
responding 3D point pairs (inner loop). The erroneous registration
of the LEDs with the tactile points is obtained by another absolute
orientation. Sampling this transformation multiple times results in
the desired registration and the associated covariance (outer loop).
Thecomputedregistrationandcovarianceisthenusedinthesub-
sequent simulation step for the entire system. It follows the same
principle. We specify several poses of the real time tracker with
respect to the reference target, as well as grid of assumed probe po-
sitions in the tracking volume. Again the simulation iterates over
these poses, perturbing the pre-calibrated transformations as well as
of each of the currently visible LED positions to sample the covari-
Now, the perturbed spa-
ances of the probe at the different positions in the tracking volume.
5.2
Following the ASME standard [2], validation is ’the process of de-
termining the degree to which a model is an accurate representation
of the real world. The intention is to validate the simulation and its
preconditions by performing some selected experiments in the tar-
get environment under varying conditions. This has the character
of a mandatory final inspection of the system after deployment, to
show that all relevant influences have actually been considered and
the defined system behaves as expected in the target environment.
This validation is not suited for the derivation of a suitable elemen-
tary sensor noise specification and corresponding simulation model
based on it. It might however help to estimate the error magnitude,
assuming that abstraction level and error distribution are known
(cf. Section 3). For now we assume that the chosen preconditions
and simulation model are basically correct; a proof will be given in
6.1, based on extensive empiric measurements in a controlled setup.
In our case the simulation predicts the position of the tip at a
certain position in the world. For comparison, a corresponding
set of real reference points in world coordinates is needed, known
with high accuracy. This explains why validation is expensive and
should be reduced to the necessary minimum. In our case, this
ground truth is obtained from the multiple reference targets whose
tactile points have already been registered with the world using a
metrological device. While one target is being used for indirect
tracking, points on this and also other targets can be probed.
Depending on the complexity of the setup, a step-by-step valida-
tion might be more efficient. First, the assumed elementary uncer-
tainty specification is validated in the real environment, based on
a few reference measurements. This might reveal potential system
malfunctions due to so far unconsidered environmental influences
or invalidated system calibration. Next, the individual simulation
steps for registration (cf. 5.1) are validated. The procedure is car-
ried out several times under varying conditions (distance, viewing
angle, ...) and a covariance is estimated from the individual results.
In our case, this applies to the registration procedure for the refer-
ence target described above (cf. Figure 5, dotted edge). Differences
between the simulated and empiric covariances could point to user
errors such as point mismatches or imprecise probing. Finally, the
complete error propagation chain is validated (see above). When
these steps are completed, one can assume the simulation repre-
sents the real world sufficiently. This allows using the simulation
as the basis for verification & validation of the entire system.
Statistical methods can be applied to formally reject the hypothe-
sis of contradictory measurements and covariances. It can be tested
whether all individual empiric measurements reside inside the pre-
dicted confidence intervals. A T-test decides whether the means of
two data sets are identical, e.g. when a series of measurements of
a certain point or pose shall be compared with the predicted mean
and covariance. It can reveal systematic errors. Furthermore, the
identity of a predicted with an empiric covariance matrix can be
shown using the Box’ M-test. In a static setup this indicates that the
elementary uncertainty specification used for simulation may be in-
correct. In a dynamic setup, as for example a probe rotating on its
tip, it indicates an error in the calibration of the probe or its tip.
Validation showed that the simulation describes the real tracking
infrastructure sufficiently. As described in Section 4, this is funda-
mental for the qualification of industrial processes. Furthermore,
the findings can also help to maintain the proper condition of the
infrastructure during runtime, as described next.
Validation
5.3
During operation & maintenance, means are needed to reliably de-
tect, trace back, and eliminate potential system failures. The ex-
pected accuracy and precision have been analyzed through the ver-
Runtime Error Mitigation