Full Terms & Conditions of access and use can be found at
European Journal of Remote Sensing
ISSN: (Print) 2279-7254 (Online) Journal homepage: http://www.tandfonline.com/loi/tejr20
Geometry accuracy of DSM in water body margin
obtained from an RGB camera with NIR band and
a multispectral sensor embedded in UAV
Dalva Maria de Castro Vitti, Ademir Marques Junior, Taina Thomassin
Guimarães, Emilie Caroline Koste, Leonardo Campos Inocencio, Maurício
Roberto Veronez & Frederico Fábio Mauad
To cite this article: Dalva Maria de Castro Vitti, Ademir Marques Junior, Taina Thomassin
Guimarães, Emilie Caroline Koste, Leonardo Campos Inocencio, Maurício Roberto Veronez &
Frederico Fábio Mauad (2018): Geometry accuracy of DSM in water body margin obtained from
an RGB camera with NIR band and a multispectral sensor embedded in UAV, European Journal of
Remote Sensing, DOI: 10.1080/22797254.2018.1547989
To link to this article: https://doi.org/10.1080/22797254.2018.1547989
© 2018 The Author(s). Published by Informa
UK Limited, trading as Taylor & Francis
Published online: 26 Nov 2018.
Submit your article to this journal
View Crossmark data
Geometry accuracy of DSM in water body margin obtained from an RGB
camera with NIR band and a multispectral sensor embedded in UAV
Dalva Maria de Castro Vitti
, Ademir Marques Junior
, Taina Thomassin Guimarães
Emilie Caroline Koste
, Leonardo Campos Inocencio
, Maurício Roberto Veronez
and Frederico Fábio Mauad
Faculty of Technology of Jahu, Environment and Water Resources, Jau-SP, Brazil;
EESC, University of Sao Paulo, Sao Carlos-SP, Brazil;
Polytechnic School, Unisinos University, São Leopoldo-RS. Brazil
The photogrammetry techniques are known to be accessible due to its low cost, while
the geometric accuracy is a key point to ensure that models obtained from photogram-
metry are a feasible solution. This work evaluated the discrepancies in 3D (DSM) and 2D
(orthomosaic) models obtained from photogrammetry using control points (GCPs) near
a reflective/refractive area (water body), where the objective was to evaluate these
points, analysing the independence, normality and randomness and other basic statistic.
NiR band and a multispectral sensor Parrot Sequoia, both embedded in a hex-rotor UAV
in flight over the Unisinos University’s artificial lake in the city of São Leopoldo, Rio
Grande do Sul, Brazil. Due the distribution of the data found to be not normal, we
applied non-parametric tests Chebyshev’s Theorem and the Mann–Whitney’sUtest,
where it showed that the values obtained from Sequoia DSM presented significant
similarities with the values obtained from the GCP’s considering the confidence level
of 95%; however, this was not confirmed for the model generated from a Canon camera,
showing that we found better results using the multispectral Parrot Sequoia.
Received 30 December 2017
Revised 5 November 2018
Accepted 10 November
Accuracy; SfM; UAV;
The 3D modelling generated from images acquired
from high-resolution cameras embedded in
unmanned aerial vehicles (UAV) is consolidating as
an alternative technique with low cost in large-scale
mappings (Smith, Carrivick, & Quincey, 2016).
The Structure from Motion (SfM) technique with
the Multi-View Stereo algorithm aims to reconstruct
a surface or object from the matching of common
points collected from several images, where each
point consists of a position and a colour extracted
from an image (Debevec, Taylor, & MaliK, 1990;
Fonstad, Dietrich, Couville, Jensen, & Carbonneau,
2013; Oliveira, 2002; Remondino, Scaioni, & Sarazzi,
2011; Snavely, Seitz, & Szeliski, 2008). This method
does not require metric cameras, giving the SfM
a status of a more feasible approach, due to the
cheaper price and availability of this type of the non-
metric cameras (Smith et al., 2016).
For an efficient construction of the 3D surface is
important that the angular separation between the
images (considering the position of the camera dur-
ing the capture) does not exceed 25–30 degrees,
achieved with a minimum of 60–80% of images over-
lapping in an individual location. Also requiring good
photo quality for a better performance of the detect-
ing algorithms (Smith et al., 2016; Verhoeven, 2011).
The SfM algorithm generates a sparse point cloud
using a local reference system. To georeferencing the
point cloud we need distributed points referenced
with a global navigation satellite system (GNSS) to
generate correct metrics (scale and global position)
(Lobnig, Tscharf, & Mayer, 2015). We need at least
four control points in a global system to obtain an
equivalent metric to the real-world scene. The model
in an absolute reference system is achieved with the
7-parameter Helmert’s similarity transformation
(Lerma, 2002; Lobnig et al., 2015).
The next step is to create a dense point cloud that
is generated with the Multi-View Stereo (MVS) algo-
rithm that calculates the geometry pixel-by-pixel,
reproducing higher level of detail in the scene
(Furukawa & Curless, 2010; Kraus, 2004).
After the dense point cloud creation, we are able to
build the proper 3D model applying triangulation
algorithms to build a mesh, also with the camera
positions and interior orientation parameters we
have the orthophotos, where all objects with
a certain height are accurately positioned in a 2D
plane (Verhoeven, 2011).
CONTACT Ademir Marques Junior email@example.com Polytechnic School, Unisinos University, São Leopoldo-RS. Brazil
EUROPEAN JOURNAL OF REMOTE SENSING
© 2018 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
The first generated 3D model is the digital elevation
model (DEM) or the digital surface model (DSM) that
considers the elevation of all elements above the
ground. The proper georeference of this model give
us the correct global position and scale of the model.
The digital terrain model (DTM) is a 3D model
that does not consider elements above the ground,
e.g. trees and buildings. To achieve this, filters are
applied to remove these elements, as this process can
be repeated by varying the reclassification parameters
to obtain a cleaner DTM. As final products of SfM,
we have the DSM or DTM (3D models), and the
orthomosaic (2D image).
To evaluate the positional accuracy of products from
SfM + MVS most works have in common the compar-
ison with differential GNSS control points, while others
compare with laser scanner data. In addition, the root
mean square error is the most used statistical value.
Another important aspect of the statistical evaluation is
the predominance of studies only considering parametric
statistics (normal distribution) and the analysis centred
in the horizontal plane (2D). (Harwin & Lucieer, 2012;
Joaquim Höhle, 2009;Laliberte,Herrick,Rango,&
Winters, 2010;Netoetal.,2017; Turner, Lucieer, &
Christopher Watson, 2012).
Each country regulates the mapping process differ-
entially. Most of them recommend at least 20 control
points and assume that the sample has a normal
distribution (Atkinson, 2005; Ariza & Atkinson,
2008). In addition, the standards do not detail the
steps of assessment of accuracy and whether the
assessment of horizontal (positional) accuracy is per-
formed independently Xand Y(linear error) or joint
XY (circular error). The Zcomponent (height) is
always evaluated independently (Ariza & Atkinson,
2008). Our work uses “E”for “X”,“N”for “Y”and “h”
for “Z”in UTM coordinates in meters (m).
An additional consideration is, that we did not
find works that consider large reflective areas as
lakes and others types of water bodies, this research
aimed to evaluate the positional accuracy of products
generated by the SfM and MVS for mapping pur-
poses using an RGB (with modified near infrared
band) and a multispectral sensor.
This work is presented as follows: the methods and
material, workflow and the statistical basis; the study
over the artificial lake at the Unisinos University,
where the products of the SfM + MVS process were
evaluated; the discussion of results of this work and
how they compare with similar works; and lastly the
conclusion of this work were we establish some
remarks and considerations for this and future works.
Materials and methods
This section describes the methods and materials
used in this work. First with the equipment used to
acquire the images and second describing the statis-
tical techniques to evaluate the products obtained
from the sensors.
Following the premise of evaluating the model accu-
racy around a water body, the area chosen for this
work was the artificial lake presented in the campus
of the Unisinos University in the south of Brazil in
the state of “Rio Grande do Sul”in the city of “São
Leopoldo”. The selected area of evaluation is marked
in Figure 1.
The studied area was chosen in order to evaluate
the influence of a reflective/refractive area near the
points used for assessment localized in the water
body margin and the use of two different types of
cameras/sensors and consequently to indicate the
better camera in this situation.
This work followed the steps presented in Figure 2,as
some of the processes are detailed in the next
83 images were collected for both the Canon
PowerShot camera ELPH 110HS 16.1 MP (Toyo,
Japan) (15 cm GSD and focal distance of 4.4 mm)
with modified NIR band and the multispectral sensor
Parrot Sequoia 16 MP (Paris, France) (14.52 cm
(11 cm) GSD and focal distance of 4.88 mm) in flight
over the studied area with height of 120 m above the
ground. Both camera/sensors were shipped in the
hex-rotor UAV ST800, as shown in Figure 3.
Structure from motion processing
In this phase, the images collected are combined with the
flight log into specific software that uses computational
vision and the algorithm Structure from Motion (SfM) to
match common points in the images using points of same
colour to recreate the scenes in a 3D model (the digital
terrain model) or in a flat surface model (orthomosaic).
The software Pix4Dmapper (Lausanne, Switzerland) was
used to perform this task.
The verification of the positional accuracy should be
performed independently for the horizontal compo-
nents (Eand N) and the vertical component (h),
where Eand Nare the positions in UTM coordinates,
and his the soil elevation (Atkinson, 2005; Ariza &
2D. M. D. C. VITTI ET AL.
To assess the accuracy, the coordinates of the
points identified in the orthomosaic (EN) and in the
3D model (h) need to be compared with their homo-
logues extracted from a source at least three times
more accurately (Preciado, 2000), (the most accurate
technique is the GNSS differential survey), and the
minimum number of checkpoints recommended is
20 (ASPRS, 2015; Brazil, 1984; Federal Geographic
Data Committee, 1998) to 167 as in STANAG:2215.
The first step is the calculation of the discrepancies
(horizontal and vertical) “e
”between the coordinates
”from the orthomosaic (2D positional or EN) and
Figure 1. Unisinos University satellite view.
Figure 2. Flowchart of research activities for positional accuracy analysis of photogrammetry from UAVs and SfM processing.
Figure 3. Aircraft ST800 with the cameras Canon ELPH 110 HS and the Sequoia sensor.
EUROPEAN JOURNAL OF REMOTE SENSING 3
the 3D model (combining the three axes E, N and h),
against the coordinates of control “C
a more precise method (Atkinson, 2005; ASPRS,
2015; Federal Geographic Data Committee, 1998;
Preciado, 2000) as in Equation (1):
This work uses the positions as EN coordinates, as “e
”values obtained are the distances in meters between
the control coordinates obtained with differential
GNSS and the control points identified in the geor-
eferenced orthomosaic. The discrepancies in height
(h) or soil elevation are measured using the elevation
obtained with the differential GNSS in the control
points against the elevation found in the control
points identified in the georeferenced 3D model.
Following, the inspection of the atypical points or
outliers is performed (Preciado, 2000). These points
are not necessarily coarse errors but must be investi-
gated. We need to estimate the expected error (σ
basing on the steps of data acquisition and image
Estimating the expected error
The expected error value must consider all possible
errors during the phases of data acquisition and pro-
cessing (Brazil, 1984). In 3D Modelling with UAV the
expected error, σ
, for the image-processing phase
can be estimated by Equation (2), where its elements
are described in Table 1.
Then, the σ
error is multiplied by 3 and the
result obtained is compared with the absolute discre-
pancies at each point, where values that exceeded this
value are excluded from the analysis. The points
excluded should be between 10% and less than 20%
of the sample (Bustos, 1981). After the inspection for
outliers, the study of previous hypotheses of indepen-
dence, normality and randomness is carried out.
Independence test of sample discrepancies
This test is done after the exclusion of outliers and
before the normality test, to verify the correlation
between the discrepancies and the distances between
the checkpoints, and properly select the most ade-
quate statistical tests for each case.
The independence test of the data can be made by
geostatistical analysis (Santos, 2015; Santos, Medeiros,
Santos, & Lisboa Filho, 2017), using the experimental
semivariogram. The experimental semivariogram for
each positional discrepancy (EN and h) are obtained
from the calculation of the semivariances σGNSS UAV
given by Equation (3), where N(h) is the number of
positional discrepancy values pairs of dp(x
+h) separated by a distance of h.
The experimental semivariogram is represented by
a chart, where the ordinate is the semivariance, and
the abscissa is the distance between the elements of the
sample. The correlation occurs until the point where
the semivariance of the discrepancies stabilizes in the
semivariogram, making the “RANGE”distance, and
after that point, the discrepancies are independent
(Santos, 2015; Yamamoto & Landim, 2013). In Figure
4, in (a) we observe an adjusted theoretical model, with
all elements of a semivariogram, “RANGE”,“NUGGET
EFFECT”and “SILL”; In (b) there was no correlation
between the discrepancies (only the “NUGGET
EFFECT”is present) indicating independence.
If the independence is verified, the analysis of
normality and randomness, followed by the analysis
of accuracy is performed. Otherwise, the theoretical
model can be adjusted to exponential, spherical or
Gaussian for semivariograms with “SILL”(Santos,
2015; Soares, 2000; Yamamoto & Landim, 2013).
After the semivariogram modelling, the adjust-
ment is done by techniques such as ordinary least
squares, weighted least squares, maximum likelihood
and or restricted maximum likelihood. The data are
cross-validated generating the standardized residue
(independent, normal, non-tendentious and homoge-
neous), and the correlation coefficient R
mined. Then, the correlation of the standardized
residuals with the positional discrepancies is verified.
If the correlation is greater than 0.6 (strong correla-
tion), it is followed by the verification of the accuracy
based on the standard residue, if it is less than 0.6
(weak correlation), the semivariogram should be
revised (Santos, 2015; Santos et al., 2017).
To verify the independence of data using the semi-
variogram, we need to analyse the pairs of points in
multiple directions (directional semivariogram). This
is done to verify the independence between the loca-
tion of the points to the distance and the geographical
direction (in relation to north). Considering this, this
work used the directions 0°, −45°, 45°, 60° and 90° to
analyse the correlation in multiple directions, and
a length of 30 m for lag increment, that is, the
distance interval to be considered as we approach
the cut (maximum) distance in each direction
GNSS UAV þσ2
image support þσ2
4D. M. D. C. VITTI ET AL.
(1000 m in this case), and an Rvalue of 10 (tabulated
value) for both EN and hdiscrepancies.
The null hypothesis in the normality test is that the
discrepancies are distributed in a Gaussian function
(ASPRS, 2015; Brazil, 1984; Federal Geographic Data
Committee, 1998); however, most accuracy standards
for cartographic products do not consider free dis-
tribution. ASPRS (2015) standards recommend per-
forming accuracy tests for data that are not in
a normal distribution.
The EN position data usually have a normal dis-
tribution from 20 sample points, so parametric tests
and the estimators, mean, standard deviation and
RMSE satisfy them. As this distribution occurs in
less frequency for the elevation discrepancies, which
usually has a free distribution, so the previous
hypothesis must be based on non-parametric meth-
ods based on the median (Atkinson, 2005).
The Shapiro–Wilk test was selected for the nor-
mality test for having a better performance compared
with Kolmogorov–Smirnov, Chi-
square and Student’st-test, when evaluating free dis-
tribution data (Neto et al., 2017; Santos, 2015;
Torman, Coster, & Riboldil, 2012).
The previously established hypothesis considering
a significance of 0.05 is described below:
: the distribution of the discrepancies EN or
hfulfil the normal function.
: the distribution of the discrepancies EN or
hdo not fulfil the normal function.
If the previous hypothesis (H
) is confirmed, the
mean and the standard deviation of the sample can be
attributed to the population and the horizontal and
altimetric accuracy can be verified by the mean, the
standard deviation or the RMSE. On the other hand, if
the data are independent and the distribution function
is unknown, or if it is known as non-normal, the central
limit theorem (CLT) can be applied (Santos, 2015).
Otherwise, if we have free distribution, the
Chebyshev’s theorem could be applicable as it can
accept any shape of distribution and is valid for
a“k”factor (multiplier of standard deviation) greater
than 1, where at least (1−1/k
) of the data values are
positioned within of the limits of “k”times the stan-
dard deviation in relation to the arithmetic mean
(Mann, 2008). This theorem allows estimating if the
points are within a range for a free distribution. For
example, for a threshold of twice (kvalue) the stan-
dard deviation is assumed that at least 75% of the
data positioned within this range, while in the normal
distribution, it is assumed that 95% of the data would
be in that range.
The most used test to verify the randomness is the
“Runs Test”, representing a series with one or more
consecutive occurrences of the same result in which,
there are only two results (Atkinson, 2005; Mann,
2008; Torman et al., 2012). For this analysis, the
previous hypothesis is declared as follows:
∶the discrepancies are randomly distributed.
∶the discrepancies are not randomly
To perform this test, the median is used as
a parameter as below:
∶discrepancy < median.
The discrepancies are separated into two groups:
to the positives to H
to the answers that
. After that, the number of arrangements
designated by Ris counted as each event of a chain of
“yes”or “no”(e.g. YYNNNYNYY, R= 5). Therefore,
Rrepresents a statistic with its own sample distribu-
tion, where its critical values are determined in two
Table 1. Error elements considered in the calculated global expected error.
Error element Description
Mean positioning error of the images in function of the embedded GNSS receptor technology, which will be
considered only for the models directly georeferenced.
Error in the camera calibration, given by the differences between the initial coordinates of the centre of the image
and the coordinates optimized by the camera.
Positioning error of the control points collected in the field; extracted from the post-processing report.
σ2D support ¼σ2
0:5For the analysis of the horizontal accuracy.
σ3D support ¼σ2
0:5For the analysis accuracy of the DTM.
Identification error of key points, usually caused by flight characteristics, weak photo overlap and the software
performance. It can be reduced by selecting better quality images or using filters, in this case, one must
consider the efficiency of filtering anomalous or undesirable points.
Identification error of the points of support in the image, function of the resolution of the targets. This study
adopted a value of 3 times the image spatial resolution.
Interpolation error due to filter failure in discretization of dense cloud layers. Has a stronger influence on the
Zcomponent, since not separating points from the terrain of other features, such as trees and buildings, the
filtered model of the DSM could be a little different from the DTM.
Error of identification of the control points in the cartographic product, function of the final spatial resolution of
the format of the check elements, whether artificial targets or existing structures. Similarly, to the σ2
in this study was adopted the resolution of 3 times the spatial resolution of the orthomosaic.
EUROPEAN JOURNAL OF REMOTE SENSING 5
For values of n
lesser than 15, we use the
table of critical values for a two-tailed test equal to
0.05. This table presents two critical values (c
in function of n
. These values represent the
limit values to accept the initial hypothesis H
For values of n
greater than 15, we can use
the normal approximation, where the Zvalue is cal-
culated in function of the mean of R, μRand from the
standard deviation of RσR, given in Equations (4),
(5) and (6).
With the Zvalue and establishing a confidence level
(0.01 < a< 0.05), the standardized normal distribu-
tion table is searched for the critical Zvalue, and then
a decision is taken accepting or not accepting the null
The median test shows as a result, that at least
50% of the points tested fit into the null hypothesis
with a confidence level of 95–99%, according to the
established value, attesting the sample randomness.
Descriptive statistical measures
After the tests of the previous hypotheses of indepen-
dence, normality and randomness of the sample dis-
crepancies of EN and h, the standard deviation,
(Equation 7) and RMSE
(Equation 8) were
= discrepancy in the position EN or in the
ec= mean discrepancy in the position EN or
in the height h; n = number of tested points.
Other useful statistics are the mean referring to the
average value of a sum of numbers divided by the
quantity of values (for the arithmetic mean), the
median indicating the most central value in the sam-
ple (not easily distorted by outliers), and the CV or
the Coefficient of variation that is the standard devia-
tion divided by the mean resulting in a percentage
indicating the sample variability in relation to the
The Mann–Whitney’sUtest is a non-parametric test
correspondent to the Student’sT-test applicable to
verify how similar are two sample groups based on
the median. The null hypothesis premise is that the
median of the populations is equal, and the alterna-
tive hypothesis is that the medians are different.
In this test, the values from the two samples are
joined and sorted in crescent order alongside the
ranked position for each value, then the ranked posi-
tion values are summed for each sample, then in the
equation below we calculate the Uvalue for each
sample using Equations (9) and (10):
are the calculated Uvalues; n
are the number of entries for each sample; R
are the sum of the ranked values for each sample.
The lesser Uvalue found is compared against the
Figure 4. Elements of a semivariogram for data with spatial dependence (a) and for spatial independent data (b).
6D. M. D. C. VITTI ET AL.
table of critical Uvalues given according to the con-
fidence level. In this work we used the values found
in the models generated (E, N and hvalues) com-
pared with its pair obtained with the ground control
points (GCPs) values obtained with the differential
RTK-GNSS, using the table for Ucritical values for
a confidence level of 95%.
If the value found in the table of Ucritical values is
smaller than the calculated Uvalue the null hypoth-
esis is accepted and we can affirm that the two sam-
ples are similar (Mann, 2008).
Study case: Unisinos University’s artificial lake
Following the workflow presented in Figure 2,83
images were collected in both sensors in a unique
flight. Alongside the image acquisition, we performed
the collection of 31 GCPs to be used in the statistical
evaluation, where six of this points were also used for
georeferencing the products obtained from the
SfM + MVS method. Figure 5 shows the points
extracted as control points used for georeferencing,
that were collected using a GNSS RTK marked in the
vertical signalling presented in the perimeter of the
SfM image processing
The image processing was performed in the
Laboratory of Advanced Visualization and
Geoinformatics in the Unisinos University. They
were imported to the software Pix4Dmapper Pro
version 3.1.23 alongside the flight log, position and
RGB of the central pixel in the images and the height
of the flight (yaw, pitch, roll).
The products obtained in the processing stage are
the 2D models (orthomosaic) for the Canon camera
(Figure 6) and the Sequoia multispectral sensor
(Figure 7), and the 3D models obtained from the
Canon camera (Figure 8) and the Sequoia sensor
(Figure 9) image processing.
Figure 5. Control points used for georeferencing and for the statistical analysis.
EUROPEAN JOURNAL OF REMOTE SENSING 7
Figure 6. Orthomosaic obtained from the Canon camera.
Figure 7. Orthomosaic from the Sequoia sensor.
8D. M. D. C. VITTI ET AL.
Figure 8. DSM obtained from the Cannon camera.
Figure 9. DSM obtained from images from the Sequoia sensor.
EUROPEAN JOURNAL OF REMOTE SENSING 9
Positional accuracy verification
The positional accuracy was performed for EN (cir-
cular position) and hfor the 31 homologous points
identified in the orthomosaic and collected in the
field with GNSS Differential RTK. The Eand
Ncoordinates were extracted from the orthomosaic
using the sample point tool in QGIS 2.8.3 Wien.
Likewise, the altitude hwas read from the DEM/
DSM. Then a table with the position of all points
was exported with the “CSV”extension and tabulated
along with their counterparts.
Estimating the “σ
The perceived error in each stage of the data acquisi-
tion, calculated by the Equation (1), in terms of the
resultant GSD (~6 cm) and the propagation of the
a priori error during the processing of the carto-
graphic products are shown in Table 2.
For the outlier inspection, the value of 2.5 times
was adopted instead of three times, with the
objective to reduce the error range in the final sam-
ple, but the same quantity of anomalous points was
excluded, resulting in three anomalous points both in
EN (circular error) and in h(altimetry), falling in the
expected rate of outliers according to Santos (2015).
We suspect that these outliers were generated due to
the fact that these GCPs were positioned near a tall
building, creating a more shadowed area with lower
visibility compared to the rest of the scanned area.
Analysing the positional data in Figure 10 for both
Sequoia (a) and Canon (b), the worse result are
clearly visible for the points 25, 26 and 27 (our out-
liers) for the data for both sensors, while the points 7
and 28 are almost in the error limit for the Canon 2D
With the discrepancies in EN and hinspected, the
analysis of previous hypotheses of independence,
normality and randomness of the sample is done.
Verifying the previous hypothesis
The independence of the EN and hdiscrepancies was
verified by geostatistical analysis, as reported in the
previous section. The point spacing and the semivar-
iograms of the EN discrepancies are shown in Figure
11 for the data obtained from the Canon camera,
where we observed no correlation between the dis-
crepancies and the distances (no dependence), with
the discrepancies for hpresenting a similar result also
for the Sequoia sensor.
The normality of the EN and hdiscrepancies for the
Canon and Sequoia sensor samples was verified by
analysing the discrepancy histogram compared to the
Figure 10. Positional data of the GCPs and the points extracted in the orthomosaic from Sequoia (a) and Canon (b).
Table 2. The resultants GSD values for the errors considered.
CANON 0.000 0.000 0.049 0.553 0.235 1.050 0.450 0.73/1.28
Sequoia 0.000 0.028 0.030 0.230 0.235 1.050 0.450 0.56/0.42
10 D. M. D. C. VITTI ET AL.
Shapiro–Wilk probability distribution function. The
p-value of the normality test by the Shapiro-Wilk func-
tion was below the threshold of 0.05 confirming that
the null hypothesis was rejected in all cases; hence, the
data used do not have a normal distribution.
The randomness test of the EN discrepancies was
performed applying the runs test considering the
median as base to the hypothesis of randomness.
This hypothesis was accepted for all samples.
Facing the results of the previous analysis we con-
cluded that the discrepancies for EN and hfor both
sensors (Canon and Sequoia) were independent and
random but with the distribution not normal indicat-
ing that we must proceed with non-parametric tests.
Descriptive statistics measures comparison and
The descriptive statistic measures observed for the
orthomosaic and the DSM for both Canon and
Sequoia sensors are presented in Table 4.
We analysed three variables considering; the cir-
cular error or the 2D positional accuracy for
orthomosaic, which indicates torsion in the model;
the error in the altimetry for the 3D model; and the
3D positional error in the 3D model regarding the
three axes. This table shows better results in general
(lower values) for the Sequoia sensor in comparison
against the modified Canon cameraT (Table 3).
As the analysed data does not have a normal distribu-
tion, the mean and the standard deviation cannot repre-
sent the population, hence, the parametrical test based on
the mean are not applicable, instead, we have to use
a non-parametric test based on the median like the
Chebyshev Theorem and the Mann–Whitney’sUtest.
Analysing the data, and how it varies around the
mean, we can affirm according to the Chebyshev’stheo-
rem that the data from the Canon camera and the multi-
spectral sensor Sequoia fit in a free distribution of 95%
and a kvalue of 3.16, limiting the distribution ktimes the
standard deviation above and below the mean.
Considering the similarity test, or the Mann–
Whitney’sUtest, we calculated the Uvalues for the
pairs of values obtained from the GCP and values
obtained in the 3D model and the orthomosaic (E,
Nand hindividually) from both camera/sensors, as
Figure 11. Spatial distribution and semivariograms of the discrepancies EN (0°, −45°, 45°, 60° and 90°) where is possible to verify
that there is no correlation between the points and the distance.
EUROPEAN JOURNAL OF REMOTE SENSING 11
presented in Tables 4 and 5for a confidence level of
95% and critical Zof 1.96.
For the Mann–Whitney’sUtest as explained in the
prior section the null hypothesis (or H
) attest that
the two sets of data are similar, and the alternative
hypothesis is that the two set of data are different,
while the test accepts or rejects H
As we can observe in Table 4, we cannot reject the null
hypothesis only for the Eposition, as the null hypothesis
is rejected for the positions Nand h,wecaninferthatthe
positional data presents significant difference with the
data collected with the differential GNSS RTK.
In the opposite, the results for the Sequoia sensor
(Table 5) do not reject the null hypothesis implying
that the data do not have a significant difference to the
data collected in the field also indicating a better result
According to the statistical tests applied to verify the
EN and hpositional accuracy, the tests showed that
the sample satisfied the requirements of the basic
hypothesis of independence and randomness but
not for normality forcing us to apply non-
parametric tests like the Mann–Whitney’sUtest
and the Chebyshev’s theorem.
Comparing the two sensors/cameras, we found
betters results for the Sequoia sensor in comparison
against the Canon Powershot used in this work, as
the 2D (orthomosaic) and 3D (DEM/DSM) products
had lower standard deviation indicating more
precision. Although we found certain individual
points with lower discrepancy for the Canon camera
(more accuracy), the data obtained from all points
varied significantly showing no consistency, rein-
forced by the Mann–Whitney’sUtest.
Also for data generated from both cameras/sen-
sors, we had similar outliers, where we suspect were
caused by a shadowed area given by a taller building
near these specific control points. In addition, we
believe we had valid data, considering the objective
of analysing the accuracy of the 2D and 3D models
around a water body, due to its reflective and refrac-
tive nature making the use of the algorithms during
the photogrammetry process difficult.
Comparing similar works (Fonstad et al., 2013;
Harwin & Lucieer, 2012; Jaud et al., 2016; Laliberte
et al., 2010; Lobnig et al., 2015; Neto et al., 2017;
Verhoeven, 2011), we had lower accuracy and preci-
sion, however, is not possible to affirm we had worse
or better results due differences in methodology and
the particular presence of a water body in our work.
Moreover, in order to compare similar works, we
reinforce the importance of a common protocol for
image capture and UAV flight plan, allowing the
direct comparison between.
The SfM + MVS technique is gaining importance due
to its accessibility and feasibility to mapping areas
where geometric accuracy is required, although
water bodies are excluded for assessment in the
majority of works that apply photogrammetry.
Table 3. Descriptive statistical measures of the discrepancies for the altimetry, 2D positional and 3D
positional for Sequoia and Canon sensors.
Discrepancy Mean Median Standard deviation RMSE CV
Sequoia 0.330 0.258 0.182 0.375 0.552
Canon 0.386 0.256 0.345 0.514 0.894
Sequoia 0.432 0.518 0.246 0.494 0.569
Canon −0.481 −0.282 0.970 1.060 2.016
Sequoia 0.544 0.579 0.306 0.620 0.793
Canon 0.617 0.381 1.030 1.178 1.669
Table 4. Mann–Whitney’sUtest results applied to the orthomosaic and the DSM from the Canon camera.
E28 28 796 797 393 0.124 Don’t rejects H
N28 28 793 848 342 6.186 Rejects H
H28 28 833 688 357 4.330 Rejects H
Table 5. Mann–Whitney’sUtest results applied to the orthomosaic and the DSM from the Sequoia sensor.
E28 28 795 801 389 −0.371 Don’t rejects H
N28 28 800 796 390 −0.247 Don’t rejects H
h28 28 755 802 393 −0.495 Don’t rejects H
12 D. M. D. C. VITTI ET AL.
Two types of camera/sensors were used in this
work: the 16 MP Canon Powershot camera; and the
16 MP multispectral sensor Sequoia embedded in
a UAV hex-rotor capturing 83 images each in one
This work aimed to evaluate the accuracy of 2D
(orthomosaic) and 3D models using GCPs obtained
in the margin of an artificial lake. The photogram-
metry community less explores the presence of
a water body, and we found considerable results in
accuracy and precision for both cameras/sensors,
with the multispectral Sequoia sensor showing better
results than our modified Canon camera.
Due the nature of the overall data studied, along-
side the descriptive statistics we applied non-
parametric tests that consider free distribution, as
the Mann–Whitney’sUtest to compare the simila-
rities between the GCP data and the data obtained
from the models, and Chebyshev’s theorem to affirm
that 95% of our data fit between 3.16 times of the
With the results found in this work and its statis-
tical validation, we consider that we had relevant
results analysing the data obtained from GCP’s
around a water body, allowing the community to
make similar comparisons in the future.
No potential conflict of interest was reported by the
Ademir Marques Junior http://orcid.org/0000-0003-
Ariza, F.J., & Atkinson, A.D. (2008). Analysis of some
positional accuracy assessment methodologies. Journal
of Surveying Engineering.,45–54. doi:10.1061/(ASCE)
ASPRS. (2015). Positional accuracy standards for digital
geospatial data. Photogrammetric Engineering & Remote
Atkinson, A.D. (2005). Control de Calidad Posicional en
Cartografía: Analisis dos Principales Estandares
y propuesta de mejora. (Doctoral Thesis). Universidad
Brazil. (1984). Normas Técnicas da Cartografia Nacional.
Decreto Nº 89.817 de 20 de junho de 1984. Rio de Janeiro:
Fundação Instituto Brasileiro de Geográfica e Estatística.
Bustos, O. (1981). Estimação Robusta no Modelo de
Posição. Rio de Janeiro: Instituto de Matemática Pura
Debevec, P.C., Taylor, C., & MaliK, J. (1990). Modeling and
rendering architecture from department of defense.
(M. S. 60001., Ed.) Mapping, charting and geodesy accu-
racy. 136 327–330 doi: 10.1099/00221287-136-2-327
Federal Geographic Data Committee. (1998). Geospatial
positioning accuracy standards part 3: National standard
for spatial data accuracy. Reston, VA: Subcommittee for
Base Cartographic Data.
Fonstad, M.A., Dietrich, J.T., Couville, B.C., Jensen, J.L., &
Carbonneau, P.E. (2013). Topographic structure from
motion: A new development in photogrammetric
measurement. Earth Surface Processes and Landforms,
38(4), 421–430. doi:10.1002/esp.3366
Furukawa, Y., & Curless, B.S. (2010). Towards Internet-
scale multi view stereo. In Computer Vision and Pattern
Recognition (CVPR) (pp. 1434–1441). San Francisco,
Harwin, S., & Lucieer, A. (2012). Assessing the accuraccy of
georeferenced point clouds produced via multi-view
stereopsis from Unmanned Aerial Vehicle (UAV)
imagery. Remote Sensing,4(6), 1573–1599. doi:10.3390/
Jaud, M., Passot, S., Le Bivic, R., Delacourt, C.,
Grandjean, P., & Le Dantec, N. (2016). Assessing the
accuracy of high resolution digital surface models com-
puted by photoscan® and MicMac® in sub-optimal survey
conditions. Remote Sensing,8(6), 465. doi:10.3390/
Joaquim Höhle, M.H. (2009). Accuracy assessment of digi-
tal elevation models by means of robust statistical
methods. ISPRS Journal of Photogrammetry and Remote
Sensing,64, 398–406. doi:10.1016/j.isprsjprs.2009.02.003
Kraus, K. (2004). Photogrammetrie (Vol. 1). Berlin: Walter
Laliberte, A.S., Herrick, J.E., Rango, E., & Winters, C. (2010).
Acquisition, orthorectification, and object-based classfica-
tion of Unmanned Aerial Vehicle (UAV) imagery for range-
land monitoring. Photogrammetric Engineering & Remote
Sensing,76(6), 661–672. doi:10.14358/PERS.76.6.661
Lerma, J.L. (2002). Fotogrametría Moderna: Analítica
y Digital. Valencia: Universitat Politécnica de Valência.
Lobnig, K., Tscharf, A., & Mayer, G. (2015). Einfluss der
Georeferenzierung auf die absolute
Rekonstruktionsgenauigkeit der photogrammetrischen
UAV-gestützten Tagebauvermessung. (influence of geo-
referencing on the absolute accuracy of UAV-based
photogrammetric reconstructions in open pit surveying).
BHM Berg-und Hüttenmännische Monatshefte,160(8),
Mann, P.S. (2008). Introdução a Estatística (5 ed.).
Translated Rio de Janeiro: LTC.
Neto, F., Gripp, F.D., Jr, Botelho, J., Santos, M.F.,
Nascimento, A.P., & Fonseca, A.L. (2017). Assessment
of positional quality in spatial data generated by VANT
using point and linear feature for cadastre applications.
Boletim de Ciências Geodésicas,23(1), 134–149.
Oliveira, M. (2002). Aage-based modeling and rendering
techniques: a survey. Rita,IX(2), 37–66.
Preciado, A.D. (2000). Investigation sobre los Metodos de
Estimacion Robusta aplicados a problemas fundamentals
de la Fotogrametria. (Doctoral Thesis). Universidad de
Remondino, F.B., Scaioni, M., & Sarazzi, D. (2011). UAV
photogrammetry for mapping and 3D modeling –cur-
rent status and future perspectives. ISPRS ICWG I/V
UAV-g (unmanned aerial vehicle in geomatics) confer-
ence. Zurich, Switzerland.
EUROPEAN JOURNAL OF REMOTE SENSING 13
Santos, A.D., Medeiros, N.D., Santos, G.R., & Lisboa
Filho, J. (2017). Use of geostatistics on absolute posi-
tional accuracy assessment of geospatial data. Boletim de
Ciências Geodésicas,23(3), 405–418. doi:10.1590/s1982-
Santos, P.A. (2015). Controle de qualidade cartográfica:
metodologias para avaliação da acurácia posicional em
dados espaciais. Doctoral Thesis, Viçosa, Minas Gerais,
Brazil: Universidade Federal de Viçosa.
Smith, M., Carrivick, J., & Quincey, D. (2016). Structure
from motion in physical geography. Progress in Physical
Geography,40, 247–275. doi:10.1177/0309133315615805
Snavely, N., Seitz, S., & Szeliski, R. (2008). Modelling the
world from Internet photo collections. International
Journal of Computer Vision,80, 189–210. doi:10.1007/
Soares, A. (2000). Geoestatística para as Ciências da Terra
e do Ambiente. Lisboa: Instituto Superior Técnico.
Torman, V.B., Coster, R., & Riboldil, J. (2012).
Normalidade de variáveis: Métodos de verificação
e comparação de alguns testes não-paramétricos por
simulação. Revista HCPA,32(2), 227–234.
Turner, D., Lucieer, A., & Christopher Watson, C.
(2012). An automated technique for generating geor-
ectified mosaics from ultra-high resolution
Unmanned Aerial Vehicle (UAV) Imagery, based
on Structure from Motion (SfM) point clouds.
Remote Sensing,4(5), 1392–1410. doi:10.3390/
Verhoeven, G. (2011). Taking computer vision aloft –
archaeological three dimensional reconstructions
from aerial photographs with PhotoScan.
Archaeological Prospection,18(1), 67–73.
Yamamoto, J.K., & Landim, P.M. (2013). Geoestatística:
Conceitos e Aplicações. São Paulo: Oficina de Textos.
14 D. M. D. C. VITTI ET AL.