Content uploaded by Ademir Marques Junior

Author content

All content in this area was uploaded by Ademir Marques Junior on Nov 27, 2018

Content may be subject to copyright.

Available via license: CC BY 4.0

Content may be subject to copyright.

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=tejr20

European Journal of Remote Sensing

ISSN: (Print) 2279-7254 (Online) Journal homepage: http://www.tandfonline.com/loi/tejr20

Geometry accuracy of DSM in water body margin

obtained from an RGB camera with NIR band and

a multispectral sensor embedded in UAV

Dalva Maria de Castro Vitti, Ademir Marques Junior, Taina Thomassin

Guimarães, Emilie Caroline Koste, Leonardo Campos Inocencio, Maurício

Roberto Veronez & Frederico Fábio Mauad

To cite this article: Dalva Maria de Castro Vitti, Ademir Marques Junior, Taina Thomassin

Guimarães, Emilie Caroline Koste, Leonardo Campos Inocencio, Maurício Roberto Veronez &

Frederico Fábio Mauad (2018): Geometry accuracy of DSM in water body margin obtained from

an RGB camera with NIR band and a multispectral sensor embedded in UAV, European Journal of

Remote Sensing, DOI: 10.1080/22797254.2018.1547989

To link to this article: https://doi.org/10.1080/22797254.2018.1547989

© 2018 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis

Group.

Published online: 26 Nov 2018.

Submit your article to this journal

View Crossmark data

Geometry accuracy of DSM in water body margin obtained from an RGB

camera with NIR band and a multispectral sensor embedded in UAV

Dalva Maria de Castro Vitti

a,b

, Ademir Marques Junior

c

, Taina Thomassin Guimarães

b

,

Emilie Caroline Koste

c

, Leonardo Campos Inocencio

c

, Maurício Roberto Veronez

c

and Frederico Fábio Mauad

b

a

Faculty of Technology of Jahu, Environment and Water Resources, Jau-SP, Brazil;

b

EESC, University of Sao Paulo, Sao Carlos-SP, Brazil;

c

Polytechnic School, Unisinos University, São Leopoldo-RS. Brazil

ABSTRACT

The photogrammetry techniques are known to be accessible due to its low cost, while

the geometric accuracy is a key point to ensure that models obtained from photogram-

metry are a feasible solution. This work evaluated the discrepancies in 3D (DSM) and 2D

(orthomosaic) models obtained from photogrammetry using control points (GCPs) near

a reflective/refractive area (water body), where the objective was to evaluate these

points, analysing the independence, normality and randomness and other basic statistic.

Theimageswereobtainedwitha16MPCanonPowerShotELPH110Swithamodified

NiR band and a multispectral sensor Parrot Sequoia, both embedded in a hex-rotor UAV

in flight over the Unisinos University’s artificial lake in the city of São Leopoldo, Rio

Grande do Sul, Brazil. Due the distribution of the data found to be not normal, we

applied non-parametric tests Chebyshev’s Theorem and the Mann–Whitney’sUtest,

where it showed that the values obtained from Sequoia DSM presented significant

similarities with the values obtained from the GCP’s considering the confidence level

of 95%; however, this was not confirmed for the model generated from a Canon camera,

showing that we found better results using the multispectral Parrot Sequoia.

ARTICLE HISTORY

Received 30 December 2017

Revised 5 November 2018

Accepted 10 November

2018

KEYWORDS

Accuracy; SfM; UAV;

mapping; geo-statistics;

DSM; photogrammetry

Introduction

The 3D modelling generated from images acquired

from high-resolution cameras embedded in

unmanned aerial vehicles (UAV) is consolidating as

an alternative technique with low cost in large-scale

mappings (Smith, Carrivick, & Quincey, 2016).

The Structure from Motion (SfM) technique with

the Multi-View Stereo algorithm aims to reconstruct

a surface or object from the matching of common

points collected from several images, where each

point consists of a position and a colour extracted

from an image (Debevec, Taylor, & MaliK, 1990;

Fonstad, Dietrich, Couville, Jensen, & Carbonneau,

2013; Oliveira, 2002; Remondino, Scaioni, & Sarazzi,

2011; Snavely, Seitz, & Szeliski, 2008). This method

does not require metric cameras, giving the SfM

a status of a more feasible approach, due to the

cheaper price and availability of this type of the non-

metric cameras (Smith et al., 2016).

For an efficient construction of the 3D surface is

important that the angular separation between the

images (considering the position of the camera dur-

ing the capture) does not exceed 25–30 degrees,

achieved with a minimum of 60–80% of images over-

lapping in an individual location. Also requiring good

photo quality for a better performance of the detect-

ing algorithms (Smith et al., 2016; Verhoeven, 2011).

The SfM algorithm generates a sparse point cloud

using a local reference system. To georeferencing the

point cloud we need distributed points referenced

with a global navigation satellite system (GNSS) to

generate correct metrics (scale and global position)

(Lobnig, Tscharf, & Mayer, 2015). We need at least

four control points in a global system to obtain an

equivalent metric to the real-world scene. The model

in an absolute reference system is achieved with the

7-parameter Helmert’s similarity transformation

(Lerma, 2002; Lobnig et al., 2015).

The next step is to create a dense point cloud that

is generated with the Multi-View Stereo (MVS) algo-

rithm that calculates the geometry pixel-by-pixel,

reproducing higher level of detail in the scene

(Furukawa & Curless, 2010; Kraus, 2004).

After the dense point cloud creation, we are able to

build the proper 3D model applying triangulation

algorithms to build a mesh, also with the camera

positions and interior orientation parameters we

have the orthophotos, where all objects with

a certain height are accurately positioned in a 2D

plane (Verhoeven, 2011).

CONTACT Ademir Marques Junior adejunior@edu.unisinos.br Polytechnic School, Unisinos University, São Leopoldo-RS. Brazil

EUROPEAN JOURNAL OF REMOTE SENSING

https://doi.org/10.1080/22797254.2018.1547989

© 2018 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits

unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The first generated 3D model is the digital elevation

model (DEM) or the digital surface model (DSM) that

considers the elevation of all elements above the

ground. The proper georeference of this model give

us the correct global position and scale of the model.

The digital terrain model (DTM) is a 3D model

that does not consider elements above the ground,

e.g. trees and buildings. To achieve this, filters are

applied to remove these elements, as this process can

be repeated by varying the reclassification parameters

to obtain a cleaner DTM. As final products of SfM,

we have the DSM or DTM (3D models), and the

orthomosaic (2D image).

To evaluate the positional accuracy of products from

SfM + MVS most works have in common the compar-

ison with differential GNSS control points, while others

compare with laser scanner data. In addition, the root

mean square error is the most used statistical value.

Another important aspect of the statistical evaluation is

the predominance of studies only considering parametric

statistics (normal distribution) and the analysis centred

in the horizontal plane (2D). (Harwin & Lucieer, 2012;

Joaquim Höhle, 2009;Laliberte,Herrick,Rango,&

Winters, 2010;Netoetal.,2017; Turner, Lucieer, &

Christopher Watson, 2012).

Each country regulates the mapping process differ-

entially. Most of them recommend at least 20 control

points and assume that the sample has a normal

distribution (Atkinson, 2005; Ariza & Atkinson,

2008). In addition, the standards do not detail the

steps of assessment of accuracy and whether the

assessment of horizontal (positional) accuracy is per-

formed independently Xand Y(linear error) or joint

XY (circular error). The Zcomponent (height) is

always evaluated independently (Ariza & Atkinson,

2008). Our work uses “E”for “X”,“N”for “Y”and “h”

for “Z”in UTM coordinates in meters (m).

An additional consideration is, that we did not

find works that consider large reflective areas as

lakes and others types of water bodies, this research

aimed to evaluate the positional accuracy of products

generated by the SfM and MVS for mapping pur-

poses using an RGB (with modified near infrared

band) and a multispectral sensor.

This work is presented as follows: the methods and

material, workflow and the statistical basis; the study

over the artificial lake at the Unisinos University,

where the products of the SfM + MVS process were

evaluated; the discussion of results of this work and

how they compare with similar works; and lastly the

conclusion of this work were we establish some

remarks and considerations for this and future works.

Materials and methods

This section describes the methods and materials

used in this work. First with the equipment used to

acquire the images and second describing the statis-

tical techniques to evaluate the products obtained

from the sensors.

Studied area

Following the premise of evaluating the model accu-

racy around a water body, the area chosen for this

work was the artificial lake presented in the campus

of the Unisinos University in the south of Brazil in

the state of “Rio Grande do Sul”in the city of “São

Leopoldo”. The selected area of evaluation is marked

in Figure 1.

The studied area was chosen in order to evaluate

the influence of a reflective/refractive area near the

points used for assessment localized in the water

body margin and the use of two different types of

cameras/sensors and consequently to indicate the

better camera in this situation.

Workflow

This work followed the steps presented in Figure 2,as

some of the processes are detailed in the next

subsections.

Image acquisition

83 images were collected for both the Canon

PowerShot camera ELPH 110HS 16.1 MP (Toyo,

Japan) (15 cm GSD and focal distance of 4.4 mm)

with modified NIR band and the multispectral sensor

Parrot Sequoia 16 MP (Paris, France) (14.52 cm

(11 cm) GSD and focal distance of 4.88 mm) in flight

over the studied area with height of 120 m above the

ground. Both camera/sensors were shipped in the

hex-rotor UAV ST800, as shown in Figure 3.

Structure from motion processing

In this phase, the images collected are combined with the

flight log into specific software that uses computational

vision and the algorithm Structure from Motion (SfM) to

match common points in the images using points of same

colour to recreate the scenes in a 3D model (the digital

terrain model) or in a flat surface model (orthomosaic).

The software Pix4Dmapper (Lausanne, Switzerland) was

used to perform this task.

Statistical analysis

The verification of the positional accuracy should be

performed independently for the horizontal compo-

nents (Eand N) and the vertical component (h),

where Eand Nare the positions in UTM coordinates,

and his the soil elevation (Atkinson, 2005; Ariza &

Atkinson, 2008).

2D. M. D. C. VITTI ET AL.

To assess the accuracy, the coordinates of the

points identified in the orthomosaic (EN) and in the

3D model (h) need to be compared with their homo-

logues extracted from a source at least three times

more accurately (Preciado, 2000), (the most accurate

technique is the GNSS differential survey), and the

minimum number of checkpoints recommended is

20 (ASPRS, 2015; Brazil, 1984; Federal Geographic

Data Committee, 1998) to 167 as in STANAG:2215.

The first step is the calculation of the discrepancies

(horizontal and vertical) “e

c

”between the coordinates

“C

o

”from the orthomosaic (2D positional or EN) and

Figure 1. Unisinos University satellite view.

Figure 2. Flowchart of research activities for positional accuracy analysis of photogrammetry from UAVs and SfM processing.

Figure 3. Aircraft ST800 with the cameras Canon ELPH 110 HS and the Sequoia sensor.

EUROPEAN JOURNAL OF REMOTE SENSING 3

the 3D model (combining the three axes E, N and h),

against the coordinates of control “C

c

”obtained by

a more precise method (Atkinson, 2005; ASPRS,

2015; Federal Geographic Data Committee, 1998;

Preciado, 2000) as in Equation (1):

ec¼CoCc(1)

This work uses the positions as EN coordinates, as “e

c

”values obtained are the distances in meters between

the control coordinates obtained with differential

GNSS and the control points identified in the geor-

eferenced orthomosaic. The discrepancies in height

(h) or soil elevation are measured using the elevation

obtained with the differential GNSS in the control

points against the elevation found in the control

points identified in the georeferenced 3D model.

Following, the inspection of the atypical points or

outliers is performed (Preciado, 2000). These points

are not necessarily coarse errors but must be investi-

gated. We need to estimate the expected error (σ

priori

)

basing on the steps of data acquisition and image

processing.

Estimating the expected error

The expected error value must consider all possible

errors during the phases of data acquisition and pro-

cessing (Brazil, 1984). In 3D Modelling with UAV the

expected error, σ

priori

, for the image-processing phase

can be estimated by Equation (2), where its elements

are described in Table 1.

Then, the σ

priori

error is multiplied by 3 and the

result obtained is compared with the absolute discre-

pancies at each point, where values that exceeded this

value are excluded from the analysis. The points

excluded should be between 10% and less than 20%

of the sample (Bustos, 1981). After the inspection for

outliers, the study of previous hypotheses of indepen-

dence, normality and randomness is carried out.

Independence test of sample discrepancies

This test is done after the exclusion of outliers and

before the normality test, to verify the correlation

between the discrepancies and the distances between

the checkpoints, and properly select the most ade-

quate statistical tests for each case.

The independence test of the data can be made by

geostatistical analysis (Santos, 2015; Santos, Medeiros,

Santos, & Lisboa Filho, 2017), using the experimental

semivariogram. The experimental semivariogram for

each positional discrepancy (EN and h) are obtained

from the calculation of the semivariances σGNSS UAV

given by Equation (3), where N(h) is the number of

positional discrepancy values pairs of dp(x

i

) and

dp(x

i

+h) separated by a distance of h.

^

γhðÞ¼1

=

2:NhðÞ

:X

n

i¼1

dp xi

ðÞdp xiþhðÞ½

2(3)

The experimental semivariogram is represented by

a chart, where the ordinate is the semivariance, and

the abscissa is the distance between the elements of the

sample. The correlation occurs until the point where

the semivariance of the discrepancies stabilizes in the

semivariogram, making the “RANGE”distance, and

after that point, the discrepancies are independent

(Santos, 2015; Yamamoto & Landim, 2013). In Figure

4, in (a) we observe an adjusted theoretical model, with

all elements of a semivariogram, “RANGE”,“NUGGET

EFFECT”and “SILL”; In (b) there was no correlation

between the discrepancies (only the “NUGGET

EFFECT”is present) indicating independence.

If the independence is verified, the analysis of

normality and randomness, followed by the analysis

of accuracy is performed. Otherwise, the theoretical

model can be adjusted to exponential, spherical or

Gaussian for semivariograms with “SILL”(Santos,

2015; Soares, 2000; Yamamoto & Landim, 2013).

After the semivariogram modelling, the adjust-

ment is done by techniques such as ordinary least

squares, weighted least squares, maximum likelihood

and or restricted maximum likelihood. The data are

cross-validated generating the standardized residue

(independent, normal, non-tendentious and homoge-

neous), and the correlation coefficient R

2

is deter-

mined. Then, the correlation of the standardized

residuals with the positional discrepancies is verified.

If the correlation is greater than 0.6 (strong correla-

tion), it is followed by the verification of the accuracy

based on the standard residue, if it is less than 0.6

(weak correlation), the semivariogram should be

revised (Santos, 2015; Santos et al., 2017).

To verify the independence of data using the semi-

variogram, we need to analyse the pairs of points in

multiple directions (directional semivariogram). This

is done to verify the independence between the loca-

tion of the points to the distance and the geographical

direction (in relation to north). Considering this, this

work used the directions 0°, −45°, 45°, 60° and 90° to

analyse the correlation in multiple directions, and

a length of 30 m for lag increment, that is, the

distance interval to be considered as we approach

the cut (maximum) distance in each direction

σpriori ¼σ2

GNSS UAV þσ2

sensor þσ2

support þσ2

image support þσ2

processing þσ2

DEM þσ2

checking

0:5ð2Þ

4D. M. D. C. VITTI ET AL.

(1000 m in this case), and an Rvalue of 10 (tabulated

value) for both EN and hdiscrepancies.

Normality tests

The null hypothesis in the normality test is that the

discrepancies are distributed in a Gaussian function

(ASPRS, 2015; Brazil, 1984; Federal Geographic Data

Committee, 1998); however, most accuracy standards

for cartographic products do not consider free dis-

tribution. ASPRS (2015) standards recommend per-

forming accuracy tests for data that are not in

a normal distribution.

The EN position data usually have a normal dis-

tribution from 20 sample points, so parametric tests

and the estimators, mean, standard deviation and

RMSE satisfy them. As this distribution occurs in

less frequency for the elevation discrepancies, which

usually has a free distribution, so the previous

hypothesis must be based on non-parametric meth-

ods based on the median (Atkinson, 2005).

The Shapiro–Wilk test was selected for the nor-

mality test for having a better performance compared

with Kolmogorov–Smirnov, Chi-

square and Student’st-test, when evaluating free dis-

tribution data (Neto et al., 2017; Santos, 2015;

Torman, Coster, & Riboldil, 2012).

The previously established hypothesis considering

a significance of 0.05 is described below:

●H

0

: the distribution of the discrepancies EN or

hfulfil the normal function.

●H

1

: the distribution of the discrepancies EN or

hdo not fulfil the normal function.

If the previous hypothesis (H

0

) is confirmed, the

mean and the standard deviation of the sample can be

attributed to the population and the horizontal and

altimetric accuracy can be verified by the mean, the

standard deviation or the RMSE. On the other hand, if

the data are independent and the distribution function

is unknown, or if it is known as non-normal, the central

limit theorem (CLT) can be applied (Santos, 2015).

Otherwise, if we have free distribution, the

Chebyshev’s theorem could be applicable as it can

accept any shape of distribution and is valid for

a“k”factor (multiplier of standard deviation) greater

than 1, where at least (1−1/k

2

) of the data values are

positioned within of the limits of “k”times the stan-

dard deviation in relation to the arithmetic mean

(Mann, 2008). This theorem allows estimating if the

points are within a range for a free distribution. For

example, for a threshold of twice (kvalue) the stan-

dard deviation is assumed that at least 75% of the

data positioned within this range, while in the normal

distribution, it is assumed that 95% of the data would

be in that range.

Randomness tests

The most used test to verify the randomness is the

“Runs Test”, representing a series with one or more

consecutive occurrences of the same result in which,

there are only two results (Atkinson, 2005; Mann,

2008; Torman et al., 2012). For this analysis, the

previous hypothesis is declared as follows:

●H

0

∶the discrepancies are randomly distributed.

●H

1

∶the discrepancies are not randomly

distributed.

To perform this test, the median is used as

a parameter as below:

●H

0

∶discrepancy ≥median.

●H

1

∶discrepancy < median.

The discrepancies are separated into two groups:

n

1

to the positives to H

0

and n

2

to the answers that

follow H

1

. After that, the number of arrangements

designated by Ris counted as each event of a chain of

“yes”or “no”(e.g. YYNNNYNYY, R= 5). Therefore,

Rrepresents a statistic with its own sample distribu-

tion, where its critical values are determined in two

ways:

Table 1. Error elements considered in the calculated global expected error.

Error element Description

σ

GNSS_UAV

Mean positioning error of the images in function of the embedded GNSS receptor technology, which will be

considered only for the models directly georeferenced.

σ

sensor

Error in the camera calibration, given by the differences between the initial coordinates of the centre of the image

and the coordinates optimized by the camera.

σ

support

Positioning error of the control points collected in the field; extracted from the post-processing report.

σ2D support ¼σ2

xþσ2

y

0:5For the analysis of the horizontal accuracy.

σ3D support ¼σ2

xþσ2

yþσ2

z

0:5For the analysis accuracy of the DTM.

σ

processing

Identification error of key points, usually caused by flight characteristics, weak photo overlap and the software

performance. It can be reduced by selecting better quality images or using filters, in this case, one must

consider the efficiency of filtering anomalous or undesirable points.

σ

image_support

Identification error of the points of support in the image, function of the resolution of the targets. This study

adopted a value of 3 times the image spatial resolution.

σ

DEM

Interpolation error due to filter failure in discretization of dense cloud layers. Has a stronger influence on the

Zcomponent, since not separating points from the terrain of other features, such as trees and buildings, the

filtered model of the DSM could be a little different from the DTM.

σ

checking

Error of identification of the control points in the cartographic product, function of the final spatial resolution of

the format of the check elements, whether artificial targets or existing structures. Similarly, to the σ2

image support,

in this study was adopted the resolution of 3 times the spatial resolution of the orthomosaic.

EUROPEAN JOURNAL OF REMOTE SENSING 5

For values of n

1

and n

2

lesser than 15, we use the

table of critical values for a two-tailed test equal to

0.05. This table presents two critical values (c

1

and c

2

)

in function of n

1

and n

2

. These values represent the

limit values to accept the initial hypothesis H

0

.

For values of n

1

and n

2

greater than 15, we can use

the normal approximation, where the Zvalue is cal-

culated in function of the mean of R, μRand from the

standard deviation of RσR, given in Equations (4),

(5) and (6).

Z¼RμR

=σR(4)

μR¼ð2:n1:n2=n1þn2Þþ1 (5)

σR¼½2:n1:n22:n1:n2n1n2

ðÞ=

n1þn2

ðÞ

2n1þn21ðÞ

0;5(6)

With the Zvalue and establishing a confidence level

(0.01 < a< 0.05), the standardized normal distribu-

tion table is searched for the critical Zvalue, and then

a decision is taken accepting or not accepting the null

hypothesis.

The median test shows as a result, that at least

50% of the points tested fit into the null hypothesis

with a confidence level of 95–99%, according to the

established value, attesting the sample randomness.

Descriptive statistical measures

After the tests of the previous hypotheses of indepen-

dence, normality and randomness of the sample dis-

crepancies of EN and h, the standard deviation,

σ

c

(Equation 7) and RMSE

c

(Equation 8) were

calculated.

σc¼ð1=n1ðÞ

X

i¼n

i¼1

ec

ec

ðÞ

2

"#

0:5(7)

RMSEc¼ð1=nÞ:X

i¼n

i¼1

e2

c

"#

0;5(8)

where e

c

= discrepancy in the position EN or in the

height h;

ec= mean discrepancy in the position EN or

in the height h; n = number of tested points.

Other useful statistics are the mean referring to the

average value of a sum of numbers divided by the

quantity of values (for the arithmetic mean), the

median indicating the most central value in the sam-

ple (not easily distorted by outliers), and the CV or

the Coefficient of variation that is the standard devia-

tion divided by the mean resulting in a percentage

indicating the sample variability in relation to the

mean.

Mann–Whitney’sU test

The Mann–Whitney’sUtest is a non-parametric test

correspondent to the Student’sT-test applicable to

verify how similar are two sample groups based on

the median. The null hypothesis premise is that the

median of the populations is equal, and the alterna-

tive hypothesis is that the medians are different.

In this test, the values from the two samples are

joined and sorted in crescent order alongside the

ranked position for each value, then the ranked posi-

tion values are summed for each sample, then in the

equation below we calculate the Uvalue for each

sample using Equations (9) and (10):

U1¼n1n2þn1n1þ1ðÞ

2R1(9)

U2¼n1n2þn2n2þ1ðÞ

2R2

¼n1n2U1(10)

where U

1

and U

2

are the calculated Uvalues; n

1

and

n

2

are the number of entries for each sample; R

1

and

R

2

are the sum of the ranked values for each sample.

The lesser Uvalue found is compared against the

Figure 4. Elements of a semivariogram for data with spatial dependence (a) and for spatial independent data (b).

6D. M. D. C. VITTI ET AL.

table of critical Uvalues given according to the con-

fidence level. In this work we used the values found

in the models generated (E, N and hvalues) com-

pared with its pair obtained with the ground control

points (GCPs) values obtained with the differential

RTK-GNSS, using the table for Ucritical values for

a confidence level of 95%.

If the value found in the table of Ucritical values is

smaller than the calculated Uvalue the null hypoth-

esis is accepted and we can affirm that the two sam-

ples are similar (Mann, 2008).

Study case: Unisinos University’s artificial lake

Following the workflow presented in Figure 2,83

images were collected in both sensors in a unique

flight. Alongside the image acquisition, we performed

the collection of 31 GCPs to be used in the statistical

evaluation, where six of this points were also used for

georeferencing the products obtained from the

SfM + MVS method. Figure 5 shows the points

extracted as control points used for georeferencing,

that were collected using a GNSS RTK marked in the

vertical signalling presented in the perimeter of the

lake.

SfM image processing

The image processing was performed in the

Laboratory of Advanced Visualization and

Geoinformatics in the Unisinos University. They

were imported to the software Pix4Dmapper Pro

version 3.1.23 alongside the flight log, position and

RGB of the central pixel in the images and the height

of the flight (yaw, pitch, roll).

The products obtained in the processing stage are

the 2D models (orthomosaic) for the Canon camera

(Figure 6) and the Sequoia multispectral sensor

(Figure 7), and the 3D models obtained from the

Canon camera (Figure 8) and the Sequoia sensor

(Figure 9) image processing.

Figure 5. Control points used for georeferencing and for the statistical analysis.

EUROPEAN JOURNAL OF REMOTE SENSING 7

Figure 6. Orthomosaic obtained from the Canon camera.

Figure 7. Orthomosaic from the Sequoia sensor.

8D. M. D. C. VITTI ET AL.

Figure 8. DSM obtained from the Cannon camera.

Figure 9. DSM obtained from images from the Sequoia sensor.

EUROPEAN JOURNAL OF REMOTE SENSING 9

Positional accuracy verification

The positional accuracy was performed for EN (cir-

cular position) and hfor the 31 homologous points

identified in the orthomosaic and collected in the

field with GNSS Differential RTK. The Eand

Ncoordinates were extracted from the orthomosaic

using the sample point tool in QGIS 2.8.3 Wien.

Likewise, the altitude hwas read from the DEM/

DSM. Then a table with the position of all points

was exported with the “CSV”extension and tabulated

along with their counterparts.

Estimating the “σ

priori

”error

The perceived error in each stage of the data acquisi-

tion, calculated by the Equation (1), in terms of the

resultant GSD (~6 cm) and the propagation of the

a priori error during the processing of the carto-

graphic products are shown in Table 2.

For the outlier inspection, the value of 2.5 times

the σ

priori

was adopted instead of three times, with the

objective to reduce the error range in the final sam-

ple, but the same quantity of anomalous points was

excluded, resulting in three anomalous points both in

EN (circular error) and in h(altimetry), falling in the

expected rate of outliers according to Santos (2015).

We suspect that these outliers were generated due to

the fact that these GCPs were positioned near a tall

building, creating a more shadowed area with lower

visibility compared to the rest of the scanned area.

Analysing the positional data in Figure 10 for both

Sequoia (a) and Canon (b), the worse result are

clearly visible for the points 25, 26 and 27 (our out-

liers) for the data for both sensors, while the points 7

and 28 are almost in the error limit for the Canon 2D

model.

With the discrepancies in EN and hinspected, the

analysis of previous hypotheses of independence,

normality and randomness of the sample is done.

Verifying the previous hypothesis

The independence of the EN and hdiscrepancies was

verified by geostatistical analysis, as reported in the

previous section. The point spacing and the semivar-

iograms of the EN discrepancies are shown in Figure

11 for the data obtained from the Canon camera,

where we observed no correlation between the dis-

crepancies and the distances (no dependence), with

the discrepancies for hpresenting a similar result also

for the Sequoia sensor.

The normality of the EN and hdiscrepancies for the

Canon and Sequoia sensor samples was verified by

analysing the discrepancy histogram compared to the

Figure 10. Positional data of the GCPs and the points extracted in the orthomosaic from Sequoia (a) and Canon (b).

Table 2. The resultants GSD values for the errors considered.

Sensor σ

GNSS_UAV

(m) σ

sensor

(m) σ

control

(m) σ

process

(m) σ

support image

(m) σ

DEM

(m) σ

checking

(m) σ

priori

orto/DTM (m)

CANON 0.000 0.000 0.049 0.553 0.235 1.050 0.450 0.73/1.28

Sequoia 0.000 0.028 0.030 0.230 0.235 1.050 0.450 0.56/0.42

10 D. M. D. C. VITTI ET AL.

Shapiro–Wilk probability distribution function. The

p-value of the normality test by the Shapiro-Wilk func-

tion was below the threshold of 0.05 confirming that

the null hypothesis was rejected in all cases; hence, the

data used do not have a normal distribution.

The randomness test of the EN discrepancies was

performed applying the runs test considering the

median as base to the hypothesis of randomness.

This hypothesis was accepted for all samples.

Facing the results of the previous analysis we con-

cluded that the discrepancies for EN and hfor both

sensors (Canon and Sequoia) were independent and

random but with the distribution not normal indicat-

ing that we must proceed with non-parametric tests.

Descriptive statistics measures comparison and

inference tests

The descriptive statistic measures observed for the

orthomosaic and the DSM for both Canon and

Sequoia sensors are presented in Table 4.

We analysed three variables considering; the cir-

cular error or the 2D positional accuracy for

orthomosaic, which indicates torsion in the model;

the error in the altimetry for the 3D model; and the

3D positional error in the 3D model regarding the

three axes. This table shows better results in general

(lower values) for the Sequoia sensor in comparison

against the modified Canon cameraT (Table 3).

As the analysed data does not have a normal distribu-

tion, the mean and the standard deviation cannot repre-

sent the population, hence, the parametrical test based on

the mean are not applicable, instead, we have to use

a non-parametric test based on the median like the

Chebyshev Theorem and the Mann–Whitney’sUtest.

Analysing the data, and how it varies around the

mean, we can affirm according to the Chebyshev’stheo-

rem that the data from the Canon camera and the multi-

spectral sensor Sequoia fit in a free distribution of 95%

and a kvalue of 3.16, limiting the distribution ktimes the

standard deviation above and below the mean.

Considering the similarity test, or the Mann–

Whitney’sUtest, we calculated the Uvalues for the

pairs of values obtained from the GCP and values

obtained in the 3D model and the orthomosaic (E,

Nand hindividually) from both camera/sensors, as

Figure 11. Spatial distribution and semivariograms of the discrepancies EN (0°, −45°, 45°, 60° and 90°) where is possible to verify

that there is no correlation between the points and the distance.

EUROPEAN JOURNAL OF REMOTE SENSING 11

presented in Tables 4 and 5for a confidence level of

95% and critical Zof 1.96.

For the Mann–Whitney’sUtest as explained in the

prior section the null hypothesis (or H

0

) attest that

the two sets of data are similar, and the alternative

hypothesis is that the two set of data are different,

while the test accepts or rejects H

0

.

As we can observe in Table 4, we cannot reject the null

hypothesis only for the Eposition, as the null hypothesis

is rejected for the positions Nand h,wecaninferthatthe

positional data presents significant difference with the

data collected with the differential GNSS RTK.

In the opposite, the results for the Sequoia sensor

(Table 5) do not reject the null hypothesis implying

that the data do not have a significant difference to the

data collected in the field also indicating a better result

fortheSequoiasensorincomparisontotheCanon

camera.

Discussion

According to the statistical tests applied to verify the

EN and hpositional accuracy, the tests showed that

the sample satisfied the requirements of the basic

hypothesis of independence and randomness but

not for normality forcing us to apply non-

parametric tests like the Mann–Whitney’sUtest

and the Chebyshev’s theorem.

Comparing the two sensors/cameras, we found

betters results for the Sequoia sensor in comparison

against the Canon Powershot used in this work, as

the 2D (orthomosaic) and 3D (DEM/DSM) products

had lower standard deviation indicating more

precision. Although we found certain individual

points with lower discrepancy for the Canon camera

(more accuracy), the data obtained from all points

varied significantly showing no consistency, rein-

forced by the Mann–Whitney’sUtest.

Also for data generated from both cameras/sen-

sors, we had similar outliers, where we suspect were

caused by a shadowed area given by a taller building

near these specific control points. In addition, we

believe we had valid data, considering the objective

of analysing the accuracy of the 2D and 3D models

around a water body, due to its reflective and refrac-

tive nature making the use of the algorithms during

the photogrammetry process difficult.

Comparing similar works (Fonstad et al., 2013;

Harwin & Lucieer, 2012; Jaud et al., 2016; Laliberte

et al., 2010; Lobnig et al., 2015; Neto et al., 2017;

Verhoeven, 2011), we had lower accuracy and preci-

sion, however, is not possible to affirm we had worse

or better results due differences in methodology and

the particular presence of a water body in our work.

Moreover, in order to compare similar works, we

reinforce the importance of a common protocol for

image capture and UAV flight plan, allowing the

direct comparison between.

Conclusion

The SfM + MVS technique is gaining importance due

to its accessibility and feasibility to mapping areas

where geometric accuracy is required, although

water bodies are excluded for assessment in the

majority of works that apply photogrammetry.

Table 3. Descriptive statistical measures of the discrepancies for the altimetry, 2D positional and 3D

positional for Sequoia and Canon sensors.

Discrepancy Mean Median Standard deviation RMSE CV

EN (m)

2D positional

Sequoia 0.330 0.258 0.182 0.375 0.552

Canon 0.386 0.256 0.345 0.514 0.894

h(m)

altimetry

Sequoia 0.432 0.518 0.246 0.494 0.569

Canon −0.481 −0.282 0.970 1.060 2.016

DSM (m)

3D positional

Sequoia 0.544 0.579 0.306 0.620 0.793

Canon 0.617 0.381 1.030 1.178 1.669

Table 4. Mann–Whitney’sUtest results applied to the orthomosaic and the DSM from the Canon camera.

Coord. n

1

n

2

R

1

R

2

UZ

cal

Decision

E28 28 796 797 393 0.124 Don’t rejects H

0

N28 28 793 848 342 6.186 Rejects H

0

H28 28 833 688 357 4.330 Rejects H

0

Table 5. Mann–Whitney’sUtest results applied to the orthomosaic and the DSM from the Sequoia sensor.

Coord. n

1

n

2

R

1

R

2

UZ

cal

Decision

E28 28 795 801 389 −0.371 Don’t rejects H

0

N28 28 800 796 390 −0.247 Don’t rejects H

0

h28 28 755 802 393 −0.495 Don’t rejects H

0

12 D. M. D. C. VITTI ET AL.

Two types of camera/sensors were used in this

work: the 16 MP Canon Powershot camera; and the

16 MP multispectral sensor Sequoia embedded in

a UAV hex-rotor capturing 83 images each in one

flight.

This work aimed to evaluate the accuracy of 2D

(orthomosaic) and 3D models using GCPs obtained

in the margin of an artificial lake. The photogram-

metry community less explores the presence of

a water body, and we found considerable results in

accuracy and precision for both cameras/sensors,

with the multispectral Sequoia sensor showing better

results than our modified Canon camera.

Due the nature of the overall data studied, along-

side the descriptive statistics we applied non-

parametric tests that consider free distribution, as

the Mann–Whitney’sUtest to compare the simila-

rities between the GCP data and the data obtained

from the models, and Chebyshev’s theorem to affirm

that 95% of our data fit between 3.16 times of the

standard deviation.

With the results found in this work and its statis-

tical validation, we consider that we had relevant

results analysing the data obtained from GCP’s

around a water body, allowing the community to

make similar comparisons in the future.

Disclosure statement

No potential conflict of interest was reported by the

authors.

ORCID

Ademir Marques Junior http://orcid.org/0000-0003-

4739-7394

References

Ariza, F.J., & Atkinson, A.D. (2008). Analysis of some

positional accuracy assessment methodologies. Journal

of Surveying Engineering.,45–54. doi:10.1061/(ASCE)

0733-9453(2008)134:2(45)

ASPRS. (2015). Positional accuracy standards for digital

geospatial data. Photogrammetric Engineering & Remote

Sensing,81,3.

Atkinson, A.D. (2005). Control de Calidad Posicional en

Cartografía: Analisis dos Principales Estandares

y propuesta de mejora. (Doctoral Thesis). Universidad

de Jaen.

Brazil. (1984). Normas Técnicas da Cartografia Nacional.

Decreto Nº 89.817 de 20 de junho de 1984. Rio de Janeiro:

Fundação Instituto Brasileiro de Geográfica e Estatística.

Bustos, O. (1981). Estimação Robusta no Modelo de

Posição. Rio de Janeiro: Instituto de Matemática Pura

e Aplicada.

Debevec, P.C., Taylor, C., & MaliK, J. (1990). Modeling and

rendering architecture from department of defense.

(M. S. 60001., Ed.) Mapping, charting and geodesy accu-

racy. 136 327–330 doi: 10.1099/00221287-136-2-327

Federal Geographic Data Committee. (1998). Geospatial

positioning accuracy standards part 3: National standard

for spatial data accuracy. Reston, VA: Subcommittee for

Base Cartographic Data.

Fonstad, M.A., Dietrich, J.T., Couville, B.C., Jensen, J.L., &

Carbonneau, P.E. (2013). Topographic structure from

motion: A new development in photogrammetric

measurement. Earth Surface Processes and Landforms,

38(4), 421–430. doi:10.1002/esp.3366

Furukawa, Y., & Curless, B.S. (2010). Towards Internet-

scale multi view stereo. In Computer Vision and Pattern

Recognition (CVPR) (pp. 1434–1441). San Francisco,

CA: IEEE.

Harwin, S., & Lucieer, A. (2012). Assessing the accuraccy of

georeferenced point clouds produced via multi-view

stereopsis from Unmanned Aerial Vehicle (UAV)

imagery. Remote Sensing,4(6), 1573–1599. doi:10.3390/

rs4061573

Jaud, M., Passot, S., Le Bivic, R., Delacourt, C.,

Grandjean, P., & Le Dantec, N. (2016). Assessing the

accuracy of high resolution digital surface models com-

puted by photoscan® and MicMac® in sub-optimal survey

conditions. Remote Sensing,8(6), 465. doi:10.3390/

rs8060465

Joaquim Höhle, M.H. (2009). Accuracy assessment of digi-

tal elevation models by means of robust statistical

methods. ISPRS Journal of Photogrammetry and Remote

Sensing,64, 398–406. doi:10.1016/j.isprsjprs.2009.02.003

Kraus, K. (2004). Photogrammetrie (Vol. 1). Berlin: Walter

de Gruyter.

Laliberte, A.S., Herrick, J.E., Rango, E., & Winters, C. (2010).

Acquisition, orthorectification, and object-based classfica-

tion of Unmanned Aerial Vehicle (UAV) imagery for range-

land monitoring. Photogrammetric Engineering & Remote

Sensing,76(6), 661–672. doi:10.14358/PERS.76.6.661

Lerma, J.L. (2002). Fotogrametría Moderna: Analítica

y Digital. Valencia: Universitat Politécnica de Valência.

Lobnig, K., Tscharf, A., & Mayer, G. (2015). Einfluss der

Georeferenzierung auf die absolute

Rekonstruktionsgenauigkeit der photogrammetrischen

UAV-gestützten Tagebauvermessung. (influence of geo-

referencing on the absolute accuracy of UAV-based

photogrammetric reconstructions in open pit surveying).

BHM Berg-und Hüttenmännische Monatshefte,160(8),

373–378. doi:10.1007/s00501-015-0388-8

Mann, P.S. (2008). Introdução a Estatística (5 ed.).

Translated Rio de Janeiro: LTC.

Neto, F., Gripp, F.D., Jr, Botelho, J., Santos, M.F.,

Nascimento, A.P., & Fonseca, A.L. (2017). Assessment

of positional quality in spatial data generated by VANT

using point and linear feature for cadastre applications.

Boletim de Ciências Geodésicas,23(1), 134–149.

Oliveira, M. (2002). Aage-based modeling and rendering

techniques: a survey. Rita,IX(2), 37–66.

Preciado, A.D. (2000). Investigation sobre los Metodos de

Estimacion Robusta aplicados a problemas fundamentals

de la Fotogrametria. (Doctoral Thesis). Universidad de

Cantabria.

Remondino, F.B., Scaioni, M., & Sarazzi, D. (2011). UAV

photogrammetry for mapping and 3D modeling –cur-

rent status and future perspectives. ISPRS ICWG I/V

UAV-g (unmanned aerial vehicle in geomatics) confer-

ence. Zurich, Switzerland.

EUROPEAN JOURNAL OF REMOTE SENSING 13

Santos, A.D., Medeiros, N.D., Santos, G.R., & Lisboa

Filho, J. (2017). Use of geostatistics on absolute posi-

tional accuracy assessment of geospatial data. Boletim de

Ciências Geodésicas,23(3), 405–418. doi:10.1590/s1982-

21702017000300027

Santos, P.A. (2015). Controle de qualidade cartográfica:

metodologias para avaliação da acurácia posicional em

dados espaciais. Doctoral Thesis, Viçosa, Minas Gerais,

Brazil: Universidade Federal de Viçosa.

Smith, M., Carrivick, J., & Quincey, D. (2016). Structure

from motion in physical geography. Progress in Physical

Geography,40, 247–275. doi:10.1177/0309133315615805

Snavely, N., Seitz, S., & Szeliski, R. (2008). Modelling the

world from Internet photo collections. International

Journal of Computer Vision,80, 189–210. doi:10.1007/

s11263-007-0107-3

Soares, A. (2000). Geoestatística para as Ciências da Terra

e do Ambiente. Lisboa: Instituto Superior Técnico.

Torman, V.B., Coster, R., & Riboldil, J. (2012).

Normalidade de variáveis: Métodos de verificação

e comparação de alguns testes não-paramétricos por

simulação. Revista HCPA,32(2), 227–234.

Turner, D., Lucieer, A., & Christopher Watson, C.

(2012). An automated technique for generating geor-

ectified mosaics from ultra-high resolution

Unmanned Aerial Vehicle (UAV) Imagery, based

on Structure from Motion (SfM) point clouds.

Remote Sensing,4(5), 1392–1410. doi:10.3390/

rs4051392

Verhoeven, G. (2011). Taking computer vision aloft –

archaeological three dimensional reconstructions

from aerial photographs with PhotoScan.

Archaeological Prospection,18(1), 67–73.

doi:10.1002/arp.v18.1

Yamamoto, J.K., & Landim, P.M. (2013). Geoestatística:

Conceitos e Aplicações. São Paulo: Oficina de Textos.

14 D. M. D. C. VITTI ET AL.