Content uploaded by Vinicius Rofatto

Author content

All content in this area was uploaded by Vinicius Rofatto on Jul 08, 2019

Content may be subject to copyright.

Revista Brasileira de Computação Aplicada, July, 2019

DOI: 10.5335/rbca.v11i2.8906

Vol. 11, No2, pp. 74–85

Homepage: seer.upf.br/index.php/rbca/index

ORIGINAL ARTICLE

Application of Articial Random Numbers and Monte Carlo

Method in the Reliability Analysis of Geodetic Networks

Maria L. S. Bonimani1, Vinicius F. Rofatto1,2, Marcelo T. Matsuoka1,2 and

Ivandro Klein3,4

1Universidade Federal de Uberlândia, Instituto de Geograa, Engenharia de Agrimensura e Cartográca,

Monte Carmelo - MG - Brasil and 2Universidade Federal do Rio Grande do Sul, Programa de Pós-Graduação

em Sensoriamento Remoto, Porto Alegre - RS - Brasil and 3Universidade Federal do Paraná, Programa de

Pós-Graduação em Ciências Geodésicas, Paraná - PR - Brasil and

4

Instituto Federal de Santa Catarina, Curso

de Agrimensura, Florianópolis - SC - Brasil

*malubonimani@hotmail.com; vfrofatto@gmail.com; tomiomatsuoka@gmail.com; ivandroklein@gmail.com

Received: 2018-11-28. Revised: 2018-11-28. Accepted: 2018-11-28.

Abstract

A Geodetic Network is a network of point interconnected by direction and/or distance measurements or

by using Global Navigation Satellite System receivers. Such networks are essential for the most geodetic

engineering projects, such as monitoring the position and deformation of man-made structures (bridges,

dams, power plants, tunnels, ports, etc.), to monitor the crustal deformation of the Earth, to implement an

urban and rural cadastre, and others. One of the most important criteria that a geodetic network must meet is

reliability. In this context, the reliability concerns the network’s ability to detect and identify outliers. Here,

we apply the Monte Carlo Method (MMC) to investigate the reliability of a geodetic network. The key of the

MMC is the random number generator. Results for simulated closed levelling network reveal that identifying

an outlier is more dicult than detecting it. In general, considering the simulated network, the relationship

between the outlier detection and identication depends on the level of signicance of the outlier statistical

test.

Key words

: Computational Simulation; Geodetic Network; Hypothesis Testing; Monte Carlo Method; Outlier

Detection; Quality Control.

Resumo

Uma rede geodésica consiste de pontos devidamente materializados no terreno, cujas coordenadas são

estimadas por meio de medidas angulares e de distâncias entre os vértices, e/ou por meio de técnicas

de posicionamento por Sistema Global de Navegação por Satélite. Estas redes são essenciais para os diversos

ramos da Ciências e Engenharia, como por exemplo, no monitoramento de estruturas (barragens, pontes,

usinas hidrelétricas, portos, túneis, portos, etc), no monitoramento da deformação da crosta terrestre, na

implantação de um cadastro urbano e/ou rural georreferenciado, entre outros. Um dos critérios que uma

rede geodésicas deve atender é a conabilidade. Neste contexto, a conabilidade pode ser entendida como a

capacidade da rede em detectar e identicar outliers à um certo nível de probabilidade. Aqui, usamos o Método

Monte Carlo (MMC) para investigar a conabilidade de uma rede geodésica. O elemento chave do MMC é o

gerador de números aleatórios. Os resultados de uma rede de nivelamento simulada revelam que identicar

um outlier é mais difícil que detectá-lo. De modo geral, a relação entre a detecção e a identicação de um

outlier depende do nível de signicância do teste estatístico empregado para tratar os outliers.

Palavras-Chave

: Método Monte Carlo; Outliers; Redes Geodésicas; Simulação Computacional; Teste de

Hipóteses; Controle de Qualidade.

74

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |75

1 Introduction

The foundation of the Monte Carlo Method (MMC)

was Buon’s needle problem by Georges Louis Leclerc

in the eighteenth century. Later, in the nineteenth

century, William Sealy Gosset, otherwise known as

‘Student’, Fisher’s disciple, discovered the form of the

‘t-distribution’ by a combination of mathematical and

empirical work with random numbers, which is now

known as an early application of the MMC. However,

the MMC became well known in the 1940s, when

Stanisław Ulam, Nicholas Metropolis, and John von

Neumann worked on the atomic bomb project. That

method was used to solve the problem of diusion

and absorption of neutrons, which was dicult to

consider in any analytical approaches (Stigler;2002).

Despite advances in science and technology to

solve highly complex systems, one of the major

obstacles to run a MMC up until the 1980s

was the analysis time and computing resources

(run time and memory). However, the advent

of personal computers with powerful processors

has rendered MMC a particularly attractive and

cost-eective approach to performance analysis of

complex systems. Therefore, the MMC emerged as

a solution to help analysts understand how well a

system performs under a given regime or a set of

parameters.

The key of the MMC is the random number

generator. A random number generator is an

algorithm that generates a deterministic sequence of

numbers, which simulates a sequence of independent

and identically distributed (i.i.d.) numbers chosen

uniformly between 0 and 1. It is random in the sense

that the sequence of numbers generated passes the

statistical tests for randomness. For this reason,

random number generators are typically referred

to as pseudo-random number generators (PRNGs).

PRNGs are part of many machine learning and

data mining techniques. In simulation, a PRNG

is implemented as a computer algorithm in some

programming language, and is made available to the

user via procedure calls or icons (Altiok and Melamed;

2007). A good generator produces numbers that are

not distinguishable from truly random numbers in a

limited computation time. This is, in particular, true

for Mersenne Twister (Matsumoto and Nishimura;

1998), a popular generator with a long period length

of 219937 – 1.

In essence, the MMC replaces random variables

by computer PRNGs, probabilities by relative

frequencies, and expectations by arithmetic means

over large sets of such numbers. A computation

with one set of PRNG is a Monte Carlo experiment

(Lehmann and Scheer;2011), also referred to as

the number of Monte Carlo simulations (Altiok and

Melamed;2007;Gamerman and Lopes;2006).

It is evident that in the last decades, the use

of MMC for quality control proposals in geodesy

has been increasing. Hekimoglu and Koch (1999)

pioneered the idea of using MMC to geodesy for

evaluating some probabilities as simple ratios from

simulated experiments. Aydin (2012) used 5,000

MMC simulations to investigate the global test

procedure in structure deformation analysis. Yang

et al. (2013) used MMC to analyze the probability

levels of data snooping. Koch (2015) investigated the

non-centrality parameter of the F-distribution by

using 100,000 simulated random variables. Klein

et al. (2017) ran 1000 experiments to verify the

performance of sequential likelihood ratio tests for

multiple outliers. Rofatto et al. (2018a) used MMC

for designing a geodetic network.

In this work, we seek to investigate the reliability

of a geodetic network. One of the frequently used

reliability measures is the Minimal Detectable Bias

-MDB, see e.g. Teunissen (2006) and Teunissen

(1998). The MDB is a diagnostic tool which allows

analyzing the network’s ability to detect outliers.

However, not the MDB, but the Minimal Identiable

Bias (MIB) should be used as the proper diagnostic

tool for outlier identication purposes (Imparato

et al.;2018). Unlike the MDB, the MIB is too complex

and even practically impossible to obtain in a closed

form. On the other hand, today we have fast and

powerful computers, large data storage systems and

modern software, which paves the way for the use

of numerical simulation. In this sense, therefore, we

propose the use of the MMC in order to analyze the

reliability of a geodetic network in terms of the MIB.

The rest of the article is organized as follows: rst,

we provide a brief explanation on what an outlier is

and explain the dierence between outlier detection

and outlier identication. Second, we present a MMC

approach as a computational analysis tool of the

reliability of a geodetic network. Third, a numerical

example of the proposed method is given for a

leveling network. Finally, the concluding remarks

are summarized at the end of this article.

2 Outlier Detection and Identication

The most often quoted denition of outliers is that

of Hawkins (1980): "An outlier is an observation

that deviates so much from other observations as to

arouse suspicions that it was generated by a dierent

mechanism". In geodesy, the term outlier is dened

based on a statistical hypothesis test for the presence

of gross measurement errors in the observations

(Baarda;1968). Observations that are rejected by

such an outlier test are called outliers. Therefore,

an observation that is not grossly erroneous but is

rejected by an outlier test can also be called outlier. In

this context, outliers are most often caused by gross

errors and gross errors most often cause outliers. But

on the one hand outliers may rarely be the result

of fully correct measurements, and on the other

hand, mistakes or malfunctioning instruments may

not always lead to large deviations, e.g., a small

correction wrongly applied (Lehmann;2013).

Since Hawkin’s and most of the other denitions

of outliers restrict themselves to samples (repeated

observations), we follow the Lehmann (2013)

denition: "An outlier is an observation that is so

probably caused by a gross error that it is better not

used or not used as it is".

In this section we provide the elements related to

hypothesis testing for the detection and identication

of a single outlier in linear(ised) models.

76 |Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85

2.1 Outlier Detection and Minimal Detectable

Bias - MDB

Baarda (1968) proposed a procedure based on

hypothesis testing for the detection of a single

outlier in linear(ized) models, which he called data

snooping. Although data snooping was introduced as

a testing procedure for use in geodetic networks,

it is a generally applicable method (Lehmann;

2012). Baarda’s data snooping consists of screening

each individual observation for a possible outlier

(Teunissen;2006). Baarda’s w-test statistic for his

data snooping is given by a normalised least-squares

residual. This test, which is based on a linear mean-

shift model, can also be derived as a particular case

of the generalised likelihood ratio test.

In principle, Baarda’s w-test only makes a

decision between the null H

0

and a single alternative

hypothesis H

i

. The null hypothesis, which is also

called the working hypothesis, corresponds to a

supposedly valid model describing the physical reality

of the observations without the presence of an outlier.

When it is assumed to be ‘true’, this model is used

to estimate the unknown parameters, typically in a

least-squares approach. Thus, the null hypothesis

of the standard Gauss–Markov model in linear or

linearised form is given by equation (1) (Koch;1999).

H0:E(y) = Ax,D(y) = Σyy (1)

Where:

•E(.) is the expectation operator;

•y∈Rnis the vector of measurements;

•A∈Rn×u

is the Jacobian matrix (also called design

matrix) of full rank u;

•x∈Ruis the unknown parameter vector;

•D(.) is the dispersion operator; and

•Σyy ∈Rn×n

is the known positive denite co-

variance matrix of the measurements.

The redundancy (or freedom degrees) of the

model in (1) is r=n-u, where nis the number of

measurements and uthe number of parameters.

Instead of H

0

,Baarda (1968) proposed a mean shift

alternative hypothesis H

i

, also referred to as model

misspecication by Teunissen (2006), as follows:

Hi:E(y) = Ax +ci∇i,D(y) = Σyy (2)

In the equation (2),

ci

is a canonical unit vector,

which consists exclusively of elements with values of

0 and 1, where 1 means that an outlier of magnitude

∇i

aects an i-th measurement and 0 otherwise, e.g.

ci

= [0 0

. . .

1

i

0 0

. . .

0]. Therefore, the purpose of the

data snooping procedure is to screen each individual

observation for an outlier.

To verify if there are sucient evidences to reject

or not the null hypothesis, the test for binary case

should be performed as (3):

Accept H0if |wi|≤qχ2

α0(r= 1, 0) = √k(3)

Where:

|wi| = c>

iΣ–1

yy b

e0

c>

iΣ–1

yy Σb

e0Σ–1

yy

(4)

In the equations 3and 4, |

wi

| is the Baarda’s w-

test statistic for the data snooping, which represents

the normalised least-squares residual for each

measurement;

Σb

e0

is the co-variance matrix of the

best linear unbiased estimator of

b

e0

under H

0

; and

b

e0

is the least-squares residuals vector of H

0

which

has this distribution under H

0

. The critical value

√k

=

qχ2

α0(r= 1, 0)

is computed from the central chi-

squared distribution with r= 1 degree of freedom

and type I error, also known as false alarm or level

of signicance,

α0

(note: the index ‘0’ represents

the case of a single alternative hypothesis testing).

The second argument of

qχ2

α0(r= 1, 0)

is the non-

centrality parameter

λr=1

, that in this case is

λr=1

= 0.

In the case of accepting in favour of H

i

, there is an

outlier that causes the expectation of |

wi

| to become

λr=1

. The non-centrality parameter (

λr=1

) describes

the discrepancy between H

0

of equation (1) and H

i

of

equation (4), and it is given by (5):

λr=1 =c>

iΣ–1

yy Σb

e0Σ–1

yy ∇2

i(5)

Because Baarda’s w-test in its essence is based

on binary hypothesis testing, in which one decides

between the null hypothesis H

0

of equation (1) and

a unique alternative hypothesis H

i

of equation (2),

it may lead to type I error

α0

and type II error

β0

.

The probability of type I error

α0

is the probability of

rejecting the null hypothesis when it is true, whereas

the type II error

β0

is the probability of failing to

reject the null hypothesis when it is false.

Instead of

α0

and

β0

, there is the condence level

(CL = 1 –

α0

) and power of the test

γ0

= 1 –

β0

,

respectively. The rst deals with the probability of

accepting a true null hypothesis; the second, with

the probability of correctly accepting the alternative

hypothesis. The Fig. 1shows an example of the

relationship between these variables.

Note in (5) that the non-centrality parameter

λr=1

requires knowledge of the outlier size

∇i

, which in

practice is unknown. On the other hand,

λr=1

can be

computed as a function of

α0

,

γ0

, and for r= 1. In

such case, the term

c>

iΣ–1

yy Σb

e0Σ–1

yy ∇2

i

becomes a scalar

and the solution of the quadratic equation (5) is given

by (6) (Teunissen;2006):

|∇i| = MDBi=sλr=1(α0,γ0)

c>

iΣ–1

yy Σb

e0Σ–1

yy ci

(6)

In the equation 6, |

∇i

| is the Minimal Detectable

Bias (MDB

i

), which is computed for each of the n

alternative hypotheses according to equation (2). For

more details about MDB see e.g. (Rofatto et al.;

2018b).

Although Baarda’s w-test belongs to the class of

generalised likelihood ratio tests and has the property

of being a uniformly most powerful invariant (UMPI)

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |77

Figure 1: A non centrality parameter of λr=1 = 3.147 with α0= 0.01(√k= 2.576) lead to γ0= 0.8 (or β0= 0.2)

(Adapted from Rofatto et al. (2018b)).

test when the null hypotheses is tested against a

single alternative (Arnold;1981;Teunissen;2006),

this test may not necessarily be a UMPI when more

than one alternative hypothesis are considered, as

is the case of the data snooping procedure (Kargoll;

2007). In the next section, we will briey review the

multiple alternative hypotheses case and the Minimal

Identiable Bias (MIB).

2.2 Outlier identication and Minimal

Identiable Bias

The sizes of type I and II errors are given for a single

alternative hypothesis H

i

of equation (2). Under this

assumption, the MDB can be obtained as a lower

bound of the outlier that can be successfully detected

(Yang et al.;2013). In practice, however, we do not

have a single alternative hypothesis during the data

snooping procedure, but we have multiple alternative

hypotheses. Therefore, the data snooping procedure

has an eect when it returns the largest absolute

value among the wi, i.e. (Teunissen;2006):

w=max |wi|, i∈{1, . . . ,n} (7)

The concept of multiple testing says that if H

0

is

rejected, among all H

i

’s the one should be accepted,

which would have rejected H

0

with the least

α

. In the

case that all critical values are identical, it is most

simple: H

i

with the maximum test statistic should

be accepted. In order to check its signicance, the

maximum value wshould be compared with a critical

value (

√k

) (Rofatto et al.;2018b). In that case, the

data snooping procedure is therefore given as:

Accept H0if w ≤√k(8)

Otherwise,

Accept Hiif w >√k(9)

According to the inequalities (8) and (9), If none

of the nw-tests gets rejected, then we accept the

null hypothesis H0.

For the test with multiples alternative hypotheses,

apart from type I and type II errors, there is a

third type of wrong decision when Baarda’s data

snooping is performed. Baarda’s data snooping

can also ag a non-outlying observation while

the ‘true’ outlier remains in the dataset. We are

referring to the type III error (Hawkins;1980). The

determination of the type III error (here denoted

by

κij

) involves a separability analysis between the

alternative hypotheses (Förstner;1983). Therefore,

we are now interested in the identication of

the correct alternative hypothesis. In this case,

rejection of H

0

does not necessarily imply the correct

identication of a particular alternative hypothesis.

Under multiple alternative hypotheses, the

probabilities of type I errors in the data snooping

procedure for outlier identication, when there are

no outliers, are given by (10):

α0i=Z|wi|>|wj|∀, |wi|>√k

f0

0dw1. . . dwn(10)

In the equation (10), f

0

0

is the probability density

function when the expectation of the multivariate

Baarda’s w-test statistics is zero (i.e. µn=0).

Based on the assumption that one outlier is in the

ith position of the dataset, the probability of a correct

identication is given by (11):

1 – βii =Z|wi|>|wj|∀, |wi|>√k

f0

idw1. . . dwn(11)

Where f

0

i

is the probability density function when

the expectation of the multivariate Baarda’s w-test

statistics is not equal to zero (µn6= 0).

The probability of type II error for multiple testing

is given by (12):

78 |Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85

βi0=P"\n

i=1|wi|≤√kHi:true#(12)

In that case, the probability of type III error is

given by (13):

n

X

i=1

P[|wj| > |wi|∀i, |wj| > √k(i6=j) | Hi:true]

=

n

X

i=1

κij (i6=j)

(13)

Testing H

0

against H

1

,H

2

,H

3

,

. . .

,H

n

is not a trivial

task for identication purposes, because the higher

the dimensionality of the alternative hypotheses, the

more complicated the level probabilities associated

with the data snooping procedure.

Teunissen (2018) recently introduced the concept

of Minimal Identiable Bias (MIB) as the smallest

outlier that leads to its identication for a given

correct identication rate. The detection and

identication are equal in the case where we only

have the one alternative hypothesis. However, under

nalternative hypotheses (multiple testing), we have

from equations (11), (12) and (13):

βii =βi0+

n

X

i=1

n

X

i=1

κij (i6=j)(14)

or

1 – βii =γ0–

n

X

i=1

κij (i6=j)∴γ0= 1 – βii +

n

X

i=1

κij (i6=j)(15)

The probability of correct detection

γ0

(power

of the test for a single alternative hypothesis) is

the sum of the probability of correct identication

1 –

βii

(selecting a correct alternative hypothesis)

and the probability of misidentication

Pn

i=1 κij (i6=j)

(selecting one of the n-1 other hypotheses). Thus,

we have the follow inequality (Imparato et al.;2018):

1 – βii ≤γ0(16)

As a consequence of that inequality (16), the MIB

will be larger than MDB, i.e. MIB ≥MDB.

Because the acceptance region (as well as

the critical region) for the multiple alternative

hypotheses case is analytically intractable, the

computation of MIB should be based on Monte

Carlo integration method (MMC). In this respect,

Imparato et al. (2018); Teunissen (2018) showed

how to compute the MIB. They found that the

larger the size of the outlier and/or more precisely,

the estimated outlier, the higher the probability of

being correctly identied. In addition, increasing

the type I error (i.e. reducing the acceptance region)

leads to higher probabilities of correct identication.

Furthermore, increasing the number of alternative

hypotheses leads to a lower probability of correct

identication.

There is no dierence between MDB and MIB in

the case of a single alternative hypothesis. As the

number of alternative hypotheses increases, however,

MDB’s become smaller, whereas MIB’s become larger.

The theory presented so far is for a single round

of data snooping. In practice, however, the data

snooping is applied iteratively in the process of

estimation, identication, and adaptation. First,

the least-squares residual vector is estimated and

Baarda’s w-test statistics are computed by (4). Then,

the detector given by (7) is applied to identify

the most likely outlier. The identied outlier

is then excluded from the dataset and the least-

squares estimation adjustment is restarted without

the rejected observation. Then, Baarda’s w-test

(4) as well as the detector (7) are again computed.

Obviously, if redundancy permits, this procedure is

repeated until no more (possible) outliers can be

identied. This procedure is called iterative data

snooping procedure - IDS (Teunissen;2006).

In the case of IDS, a reliability measure cannot

be easily computed for quality control purposes.

Consequently, MIB is valid only for the case where

data snooping is run once, and they cannot be used

as a diagnostic tool for IDS. Because an analytical

formula is not easy to compute, a MMC should be

run to obtain the MIB for IDS. The MMC allows

insights into these cases where analytical solutions

are extremely complex to fully understand, are

doubted for one reason or another, or are not available

(Rofatto et al.;2018b).

Recent studies by Rofatto et al. (2017) showed

how to extract the probability levels associated with

Baarda’s IDS procedure by MMC. Furthermore, they

introduced two new classes of wrong decisions for

IDS, which they called over-identication. One is the

probability of IDS agging simultaneously the outlier

and good observations. Second is the probability of

IDS agging only the good observations as outliers

(more than one) while the outlier remains in the

dataset. Obviously, these two new false decisions

could occur during the iterative process of estimation,

identication, and exclusion, as is the case of IDS.

3 MIB based on Monte Carlo Method

The probability levels associated with IDS are not

easy to study using analytical models owing to the

paucity or lack of practically computable solutions

(closed form or numerical). Therefore, identifying an

outlier is still a bottleneck in geodesy. On the other

hand, a MMC method can almost always be run to

generate system histories that yield useful statistical

information on system operation and performance

measures as pointed out by Altiok and Melamed

(2007).

A geodetic network are typically composed by

distances and angles measurements. Generally, the

random errors of good measurements are normally

distributed with expectation zero. In order to have

normal random errors, uniformly distributed random

number sequences (produced by the Mersenne

Twister algorithm, for example) are transformed

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |79

into a normal distribution by using the Box–Muller

transformation (Box and Muller;1958). Box–Muller

has been used in geodesy for MMC (Lehmann;2012).

A procedure based on the MMC is applied to

compute the probability levels of IDS as follows

(summarised as a owchart in Fig. 2).

In the rst step, the design matrix

A∈Rn×u

and

the co-variance matrix of the measurements

Σyy ∈

Rn×n

are entered; then, the signicance level

α

and

the magnitude intervals of simulated outliers are

dened.

The magnitude intervals of outliers are based on a

standard deviation of measurements (e.g. |3

σ

to 9

σ

|,

where

σ

is the standard deviation of measurement.

The random error vectors are articially generated

based on a multivariate normal distribution, because

the assumed stochastic model for random errors is

based on a matrix co-variance of the measurements.

In this work, we use the Mersenne Twister algorithm

to generate a sequence of PRNG and Box–Muller to

transform it into a normal distribution. On the other

hand, the magnitude of the outlier (one outlier at a

time, r= 1) is selected based on magnitude intervals

of the outliers for each Monte Carlo experiment. We

use the continuous uniform distribution to select

the outlier magnitude. The uniform distribution is

a rectangular distribution with constant probability

and implies the fact that each range of values that

has the same length on the distributions support

has equal probability of occurrence. Thus, the total

error

is a combination of the random errors and its

corresponding outlier, which is given as as follows:

=e+ci∇i(17)

Where:

e∈Rn

is the PRNG from normal

distribution, i.e.

e∼ N

(0,

Σyy

),

ci

consists exclusively

of elements with values of 0 and 1, where 1 means

that an outlier of magnitude

∇i

aects an i-th

measurement, and 0 otherwise.

After the total error has been generated, the

least-squares residuals vector

b

e0

is computed using

equation (18):

b

e0=R,with R=I–A(A>WA)–1A>W(18)

In the equation (18), we have

R∈Rn×n

as

the redundancy matrix,

W

=

σ02Σ–1

yy ∈Rn×n

the

weight matrix of the measurements, where

σ02

is

the variance scalar factor, and

I∈Rn×n

the identity

matrix (Koch;1999).

For IDS, the hypothesis of (2) for one outlier

is assumed and the corresponding test statistic is

computed according to (4). Then, the maximum test

statistic value is computed according to (7). After

identifying the observation suspected as the most

likely outlier, it is typically excluded from the model,

and least-squares estimation and data snooping are

applied iteratively until there are no further outliers

identied in the dataset. The procedure should be

performed for mexperiments of random error vectors

with each experiment contaminated by an outlier.

If mis the total number of MMC experiments, we

count the number of times that the outlier is correctly

identied (denoted as n

CI

), i.e.

max |wi

ν|

>

√k

for

ν

= {1,

. . .

,

m

}. Then, the probability of correct

identication (P

CI

) can be approximated as follows

(Rofatto et al.;2018b):

PCI ≈nCI

m(19)

The error probabilities are also approximated as

follow:

PMD ≈nMD

m(20)

PWE ≈nWE

m(21)

Pover+≈nover+

m(22)

Pover–≈nover–

m(23)

Where:

•

n

MD

is the number of experiments in which the

IDS does not detect the outlier;

•

P

MD

represents the type II error, also referred to

as missed detection probability;

•

n

WE

is the number of experiments in which the IDS

procedure ags a non-outlying observation while

the ‘true’ outlier remains in the dataset;

•

P

WE

represents the type III error, also referred to

as wrong exclusion probability;

•

n

over+

is the number of experiments where the IDS

identies correctly the outlying observation and

others;

•Pover+corresponds to the probability of over+;

•

n

over–

represents the number of experiments

where the IDS identies more than one non-

outlying observation, whereas the ‘true outlier’

remains in the dataset;

•

P

over–

corresponds to the probability of over– class;

In practice, as the magnitudes of outliers are

unknown, one can dene the probability of the

correct identication in order to nd the MIB for a

given application. In the next section, the procedure

based on MMC for the computation of MIB is applied

in a geodetic network. The relationship between

detection by MDB and identication by MIB is also

studied.

4 An example of the Monte Carlo Method

applied to the reliability analysis of

geodetic networks

As an example, the procedure based on MMC

experiments for the computation of probability

levels of IDS is applied to the simulated closed-

levelling network given by Rofatto et al. (2018b),

with one control (xed) point (A) and three points

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |81

Table 1: MDB and MIB for each signicance level α(%) and for a power of γ= 0.8(80.0%)

Measurement α0.1% α1% α5% α10%

1MDB 5.3σ4.4σ3.6σ3.2σ

MIB 5.5σ4.8σ4.7σ6.5σ

2MDB 6.6σ5.4σ4.4σ4.0σ

MIB 6.8σ6.0σ5.8σ> 9σ

3MDB 6.6σ5.4σ4.4σ4.0σ

MIB 6.8σ6.0σ5.8σ6.5σ

4MDB 5.3σ4.4σ3.6σ3.2σ

MIB 5.5σ4.8σ4.7σ> 9.0σ

5MDB 6.6σ5.4σ4.4σ4.0σ

MIB 6.8σ6.0σ5.8σ7.0σ

6MDB 5.3σ4.4σ3.6σ3.2σ

MIB 5.5σ4.8σ4.7σ6.0σ

Figure 3: Simulated geodetic levelling network

with unknown heights (B, C, and D), totalling four

minimally constrained points (Fig. 3). The simulated

geodetic network has a minimal number of redundant

measurements that lead the identication of a single

outlier.

It is important to mention that geodetic network

presents a minimum conguration to identify at

least one single outlier. As mentioned by Xu (2005)

that: "in order to identify outliers, one also has to

further assume that for each model parameter, there

must, at least, exist two good data that contain the

information on such a parameter". For example,

consider the one unknown height into a leveling

network (one-dimensional - 1D). Two observations

would lead to dierent solutions and allow the

detection of an inconsistency between them. Three

observations would lead to dierent solutions and

the identication of one outlying observation, and so

on. Thus, in a general case, the number of possible

identiable outliers should be equal to the minimal

number of redundant measurements across each and

every point, minus one.

There are

n

= 6 measurements,

u

= 3 unknowns,

and

n

–

u

= 3 redundant measurements in this

network. Therefore, the geodetic network would be

able to identify one outlier. The measurements 1,

2, 3, 4, 5, and 6 are assumed normally distributed,

uncorrelated, and with nominal precision (a prior

standard deviation

σ

) of

±

8mm,

±

5.6mm,

±

5.6mm,

±

8mm,

±

5.6mm, and

±

8mm, respectively. The

magnitude interval of outlier is from the minimum 3

σ

to maximum 9

σ

, with an interval rate of 0.1

σ

. Here,

positive and negative outliers are considered for each

measurement. Four values were considered for the

signicance level:

α

= 0.001(0.1%),

α

= 0.01(1%),

α

= 0.05(5%) and

α

= 0.1(10%). We ran 10,000 MMC

experiments for each measure and for each outlier

magnitude interval, totalling 12,960,000 numerical

experiments.

Figure 4) shows the power of the test, type II and

III errors of IDS, and (Figure 5) the over-identication

probabilities, for the case where there is a single

outlier contaminating the measurements. In general,

the larger magnitude of the outlier, the higher the

success rate (i.e. power of the test). It can be noted

that the type III error is the smallest for

α

= 0.001

and largest for the type II error. Furthermore, it is

rare for an outlier of small magnitude, say 3

σ

to 4

σ

,

to be identied on that network.

In general, for the simulated network, the smaller

α

, the larger is the

β

. On the other hand, the smaller

α

,

the smaller type III error (

κ

). For two classes of over-

identication probabilities, in general, the inuence

of committing the over-identication+ and over-

identication– is directly related to probability level

α

: the larger

α

, the larger the over-identications

case. Note that for

α

= 0.001, the over-identication

cases are practically absent.

Besides that, the MDB were computed for each

measurement and for the four signicance level

described above. The relationship between MDB

and MIB is showed in the Tab. 1. The higher the

level of signicance

α

, the higher is the probability

of detecting it, i. e. the smaller the MDB. This

relationship, however, does not work for MIB. The

MIB is slightly larger than the MDB for that geodetic

network, except for the signicance level of 10%,

for which the MIB is approximately two times

larger than the MDB. Therefore, due to the low

redundancy of measurements in the network, it is

not recommended to use a signicance level of 10%

for outlier identication proposals.

This example shows how to compute for the

IDS case based on the MMC. Obviously, should

be computed for a given probability of correct

identication (γ) and signicance level (α).

82 |Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85

Figure 4: Power of the test, type II and type III error for each signicance level α.

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |83

Figure 5: Over-identication probabilities for each signicance level α

84 |Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85

5 Final Remarks

In this study, we highlighted that Monte Carlo

method (MMC) is a primary tool for deriving

solutions to complex problems. We used the

Monte Carlo method as a key tool for studying

the IDS procedure. We emphasized that, the

method discards the use of real measurements.

Actually, it is assumed that the random errors of

the good measurements are normally distributed,

and therefore can be articially generated by means

of a PRNG. Thus, in fact, the only needs are the

geometrical network conguration (given by design

matrix); the uncertainty of the observations (which

can be given by nominal standard deviation of the

equipment); and the magnitude intervals of the

outliers.

We also highlighted that in contrast to the well-

dened theories of reliability, the IDS procedure

is a heuristic method, and therefore, there is no

theoretical reliability measure for it. Hence, an

analytical model with tractable solution is unknown,

and therefore, one needs to resort to MMC. Based on

the work by Rofatto et al. (2018b), we showed how

to nd the probability levels associated with IDS and

how to obtain its for each observation by means of

the MMC for a given correct identication probability

and signicance level.

Acknowledgements

The authors would like to acknowledge the support

from FAPEMIG (Fundação de Amparo à Pesquisa do

Estado de Minas Gerais, research project 2018/7285).

The authors also like to extend our gratitude to the

anonymous referee for some valuable comments on

a previous version of this text.

References

Altiok, T. and Melamed, B. (2007). Simulation

Modeling and Analysis with Arena, 1 edn, Academic

Press.

Arnold, S. (1981). The theory of linear models and

multivariate analysis, 1 edn, Wiley.

Aydin, C. (2012). Power of global test in deformation

analysis, Journal of Surveying Engineering

138

(2): 51–

56.

https://doi.org/10.1061/(ASCE)SU.1943-5428.

0000064.

Baarda, W. (1968). A testing procedure for use in

geodetic networks., Publ. on geodesy, New Series

2

(5).

Box, G. E. P. and Muller, M. E. (1958). A note on the

generation of random normal deviates, The Annals of

Mathematical Statistics

29

(2): 610–611.

https://doi:

10.1214/aoms/1177706645.

Förstner, W. (1983). Reliability and discernability

of extended gauss-markov models, Seminar on

mathematical models of Geodetic/Photogrammetric

Point Determination with Regard to Outliers and

Systematic Errors, Vol. Series A, Deutsche

Geodätische Kommision, Munich, Germany,

pp. 79–103.

Gamerman, D. and Lopes, H. F. (2006). Markov

Chain Monte Carlo: Stochastic Simulation for Bayesian

Inference, 2 edn, Chapman and Hall/CRC.

Hawkins, D. M. (1980). Identication of Outliers, 1 edn,

Springer Netherlands.

https://doi.org/10.1007/

978-94-015-3994-4.

Hekimoglu, S. and Koch, K. R. (1999). How can

reliability of the robust methods be measured?, in

M. O. Altan and L. Gründig (eds), Third Turkish-

German joint geodetic days, Vol. 1, Istanbul Technical

University, Istanbul, Turkey, pp. 179–196.

Imparato, D., Teunissen, P. J. D. and Tiberius, C. C.

J. M. (2018). Minimal detectable and identiable

biases for quality control, Survey Review .

https:

//doi.org/10.1080/00396265.2018.1437947.

Kargoll, B. (2007). On the theory and application

of model misspecication tests in geodesy, Doctoral

thesis, University of Bonn, Landwirtschaftliche

Fakultät, German, Bonn.

Klein, I., Matsuoka, M. T., Guzatto, M. P. and

Nievinski, F. G. (2017). An approach to identify

multiple outliers based on sequential likelihood

ratio tests, Survey Review

49

(357): 449–457.

https:

//doi.org/10.1080/00396265.2016.1212970.

Koch, K. R. (1999). Parameter estimation and hypothesis

testing in linear models., 2 edn, Springer.

Koch, K. R. (2015). Minimal detectable outliers

as measures of reliability, Journal of Geodesy

89

(5): 483–490.

https://doi.org/10.1007/

s00190-015-0793-5.

Lehmann, R. (2012). Improved critical values for

extreme normalized and studentized residuals

in gauss-markov models, Journal of Geodesy

86

(12): 1137–1146.

https://doi.org/10.1007/

s00190-012-0569-0.

Lehmann, R. (2013). On the formulation of

the alternative hypothesis for geodetic outlier

detection, Journal of Geodesy 87(4): 373–386.

Lehmann, R. and Scheer, T. (2011). Monte carlo

based data snooping with application to a geodetic

network, Journal of Applied Geodesy

5

(3-4): 123–134.

https://doi.org/10.1515/JAG.2011.014.

Matsumoto, M. and Nishimura, T. (1998). Mersenne

twister: A 623-dimensionally equidistributed

uniform pseudo-random number generator, ACM

Transactions on Modeling and Computer Simulation

8

(1): 3–30.

https://dl.acm.org/citation.cfm?id=

272991.

Rofatto, V. F., Matsuoka, M. T. and Klein, I. (2017). An

attempt to analyse baarda’s iterative data snooping

procedure based on monte carlo simulation, South

African Journal of Geomatics

6

(6): 416–435.

http:

//dx.doi.org/10.4314/sajg.v6i3.11.

Rofatto, V. F., Matsuoka, M. T. and Klein, I.

(2018a). Design of geodetic networks based

on outlier identication criteria: An example

applied to the leveling network, Bulletin of Geodetic

Sciences

24

(2): 152–170.

http://dx.doi.org/10.

1590/s1982-21702018000200011.

Bonimani et al./ Revista Brasileira de Computação Aplicada (2019), v.11, n.2, pp.74–85 |85

Rofatto, V. F., Matsuoka, M. T., Klein, I., Veronez, M.,

Bonimani, M. L. and Lehmann, R. (2018b). A half-

century of baarda’s concept of reliability: A review,

new perspectives, and applications, Survey Review .

https://doi.org/10.1080/00396265.2018.1548118.

Stigler, S. M. (2002). Statistics on the Table: The History

of Statistical Concepts and Methods, 1 edn, Harvard

University Press.

Teunissen, P. J. G. (1998). Minimal detectable biases

of gps data, Journal of Geodesy

72

(4): 236–244.

https://doi.org/10.1007/s001900050163.

Teunissen, P. J. G. (2006). Testing Theory: an

introduction, 2 edn, Delft University Press.

Teunissen, P. J. G. (2018). Distributional theory for

the dia method, Journal of Geodesy

91

(1): 59–80.

https://doi.org/10.1007/s00190-017-1045-7.

Xu, P. (2005). Sign-constrained robust least squares,

subjective breakdown point and the eect of

weights of observations on robustness, Journal of

Geodesy 79(1): 146–159.

Yang, L., Wang, J., Knight, N. and Shen, Y.

(2013). Outlier separability analysis with a

multiple alternative hypotheses test, Journal of

Geodesy

87

(6): 591–604.

https://doi.org/10.1007/

s00190-013-0629-0.