Available via license: CC BY 4.0
Content may be subject to copyright.
INTERACTIVE VIRTUAL OBJECTS (IVOs) FOR NEXT GENERATION OF VIRTUAL
MUSEUMS: FROM STATIC TEXTURED PHOTOGRAMMETRIC AND HBIM MODELS
TO XR OBJECTS FOR VR-AR ENABLED GAMING EXPERIENCES
F. Banfi1*, A. M andelli1
1 Architecture, Built environment and Construction engineering (A.B.C.) Department, Politecnico di Milano, Milano, Italy -
(fabrizio.banfi; alessandro.mandelli)@polimi.it
KEYWORDS: Interactive Virtual Objects (IVOs), Virtual Museum,eXtended Reality (XR), heritage documentation, 3D modelling,
Virtual Reality (VR), photogrammetry, Unmanned Aerial System (UAS),
ABSTRACT:
Virtual museums should not live on the internet through traditional applications sharing collections through simple panoramas or static
images and descriptions but reach novel innovative and interactive forms of Virtual Reality (VR) and Augmented Reality (AR),
providing a more creative, intimate, personal learning experience. For this reason, the authors propose a method based on advanced
Information Technologies (IT) to fully return the tangible and intangible values of different type of works of art. The primary purpose
of this study is to create a system of virtual environments through which the digital user, mainly the visitor of an unconventional
museum, will be able to physically interact, through visual and tactile methods, with 3D digital models of sculptures, information, and
art objects. Thanks to the integration of the latest 3D modelling and digital survey techniques with the Visual Programming Language
(VPL) and eXteded Reality (XR) development platforms, authors propose new levels of interactivity between users and Interactiv e
Virtual Objects (IVOs) capable of coming to life, sharing new forms of real-time human-computer interaction for VR-AR enabled
gaming experiences and virtual museums, using multiple devices such VR headset, w eb-based AR platforms, mobile phones, tablets,
and PC workstation.
1. INTRODUCTION
1.1 General context
The International Council of Museums (ICOM), an international
organisation of museums and museum professionals, was created
to raise awareness and protect the world’s Cultural Heritage.
Founded in 1948 on the initiative of Chauncey J. Hamlin,
President of the American Association of Museums, this
organisation considers the museum a permanent, non-profit
institution serving society and its development. It is open to the
public and carries out research concerning the material and
immaterial testimonies of humanity and its environment; it
acquires them, preserves them, communicates them and above
all, exhibits them for study, education, and pleasure.
The museum is considered as the element of grouping, continuity
(Sylaiou et al., 2010). It plays its role by preserving for the
community and communicating with it through the objects (an
ordered collection of works of art or scientific instruments,
ancient finds, precious objects, testimonies of the history of one’s
country or customs, natural curiosities, etc.), which must be
collected and treated with the utmost respect and scientific rigour.
With the transformation of information and cultural systems, the
advent of the latest generation technologies, 3D survey, eXtended
Reality (XR), the selection criteria for the objects considered
relevant, the purposes and forms for their organisation, exposure
and dissemination have changed. Just think of the Building
Information Modelling (BIM) process and the reengineering and
digitisation process that has reformed the construction sector in
recent years. Notably, the scan-to-BIM process applied to
heritage buildings has been applied to archaeological sites,
infrastructures, and monuments, not just heritage buildings. The
added value of integrating innovative 3D survey methods such as
terrestrial and aerial photogrammetry, laser scanning with
advanced modelling techniques has allowed passing from simple
* Corresponding author
2D vector representations to digital models capable of faithfully
representing reality and each type of survey artefact. In addition,
thanks to the scan-to-BIM process applied to artefacts of high
historical and cultural value (HBIM), it has also been possible to
expand the information value of each architectural and structural
element, reaching different types of experts such as architects
engineers, archaeologists, and restorers (Bosché et al., 2015). F or
this reason, this article outlines a method capable of increasing
the paradigm of the “utility” of digital models and digitised
objects even for non-expert users such as virtual tourists and
students through the latest generation XR techniques, digitising
archaeological and ancient finds and trying to find new forms of
interactivity and sharing of the tangible and intangible values of
our built heritage.
1.2 State-of-the-art: from photogrammetric models to
Digital Cu ltu ral Heritage (DHC)
Museums contribute to preserving the objects and memories that
converge in the community’s Cultural Heritage (CH), through
which society needs to be founded on history, on its past, to have
its own characteristic identity. A museum, therefore, deals with
scientific research, communication, and conservation.
Consequently, the museum aims to: collect, safeguard,
document, research, publish, introduce, explain, and finally
disseminate knowledge to different levels of users.
In this specific field of application, digital photogrammetry has
made it possible to lay the appropriate foundations for collecting
metric and geometric information of any artefact in recent
decades. In particular, innovative studies have made it possible
to define processes capable of improving the level of detail and
accuracy of digitised objects more and more (Volk et al., 2014).
As is well known, photogrammetry is a technique that is mainly
used in cartography, topography, and architecture. It allows you
to metrically determine the shape and position of objects,
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
47
identifying the spatial position of all the points of interest of the
object considered. Although initially created to be used in the
architectural survey, this technique is currently used for the
topographical survey of the territory, mainly developing in the
form of aerial photogrammetry.
The latest generation software able to manage many data and
computer graphics have also allowed a simpler and faster use and
lower costs. The advent of methods, software and technologies
has made old optical equipment obsolete, favouring its
commitment even in areas where it was rarely used in the past.
In this context, the first result of the photogrammetric survey is a
point cloud, which can be integrated with other 3D point clouds
generated by different survey technologies such as laser s canners,
total stations, and GPS.
This integration of tools and data allows professionals to interact
with a single 3D environment characterised by different inputs
and formats while maintaining the same georeferencing and
project measurement unit. This technique, also known as
Structure from Motion (SfM), is based on a calculation that
allows the user to reconstruct the shape of objects through the
automatic collimation of points from a set of photos. The S fM
extracts the notable points from the individual images, deduces
the photographic parameters, crosses the recognisable points on
several images, and finds the coordinates in the space of the
points themselves based on computer vision algorithms.
The extraction of the “key points” is helpful in the processing of
the point cloud and the next phase, the creation of a textured mesh
model. In the field of Digital Cultural Heritage (DCH) over the
years, this specific methodology allowed to lay the appropriate
foundations for correct digitisation of architectural and structural
elements, pushing more and more professionals to define
methods capable of replicating even the smallest artefacts.
Some studies have made it possible to distinguish and optimise
digitisation and modelling techniques, identifying the pros and
cons of mesh models. In particular, from a digital model point of
view, digital photogrammetry only allows the creation of a
textured mesh model.
On the other hand, it was found that a mesh is not manageabl e
and automatically recognised as a BIM object by major
applications such as Autodesk Revit and Graphisoft ArchiCAD.
The leading cause is to be attributed to its composition. A mesh
model is composed of a dense network of polygons and
elementary surfaces of triangular geometry. Its size is too heavy
for BIM platforms and cannot be transformed into an object
capable of receiving information.
Consequently, recent studies have made it possible to define
HBIM digitisation techniques, scan-to-BIM and Grades Of
Generation (GOG) (Banfi, 2017) capable of transforming simple
points from the 3D survey into BIM parameter objects. The
primary data sources used in these studies are geometric entities
such as points, surfaces, and solids that can be automatically
transformed into informative models capable of communicating
each material, phys ical, mechanical, historical, and cultural
characteris tics of the elements. Furthermore, recent studies have
defined methods that could benefit from digital photogrammetry
and obtain digital models that can also communicate different
types of information using different devices (laptops, PC
workstations, mobile phones) and software application (Chen et
al., 2020). However, the step not yet thoroughly investigated is
the one related to how to transform a textured mesh into an
intelligent object for the latest generation XR and BIM projects,
interacting and responding to user input.
For this reason, this study outlines a process trying to improve
the interoperability between the outputs produced by the
digitisation process and enhancing the usefulness of
photogrammetric models for other forms of model sharing, such
as virtual museums and XR.
1.3 State-of-the-art: from virtual museum to XR
We all know how a museum is made, it is a structure which can
be divided into rooms and each room contains works of different
types. Usually, the works can be accompanied by a description,
caption, audio headphones with a built-in guide or a guide that he
can tell us about the work we see at that precise moment.
On the other hand, the work is decontextualised, adapted, and
forced into a new environment, consequently losing its real
context’s historical and cultural charm. In addition, visit times
must respect the needs and trends of the flow, safety distances
imposed by the global health emergency, avoiding staying too
long in front of a work and promoting sustainable use of the
rooms (Atsiz, 2021). For these cons, recently, the creation of
virtual museums alongside an actual museum has turned out to
be a solution capable of reaching many users even at a distance,
exponentially increasing the knowledge and dissemination of
historical and cultural information, renewing the interest in the
works of art placed inside (Kwok, Koh, 2020).
The virtual museum had its origins in the academic field by the
student Nicolas Pioch, who in 1995 conceived and created the
WebMuseum (Ibiblio, 2021), a network for the dissemination of
works of art. After this project, sites related to art galleries began
to spread with images and written captions. By now, almost all
museums in the world (artistic, archaeological, anthropological,
or scientific-technical) have created their own website, often
conceived as a digital representation of the real museum, with
very similar structure and contents (Huhtamo, 2013). There is no
shortage of sites released from real museum institutions and
multimedia databases, organised in consortia and non-profit
organisations, which offer the digitised cultural heritage of
several museums.
In this context, thanks to the integration of advanced digitisation
techniques such as the s can-to-BIM process, VR and AR have
made it possible to reconstruct the objects and their historical
context. Interesting studies have shown the potential of these
techniques, recreating objects, buildings, small, medium, and
large artefacts such as entire ancient cities and complex urban
scenarios. Over the years, different types of virtual museums
have been seen, from the completion or replica of the real
museum to the one exclusively on-line, where applications based
on 360 panoramas simulate a virtual path. On the other hand, the
recent form of the virtual museum does not allow high levels of
interactivity between the user and the museum itself. In fact, a
virtual museum can be the collection and sharing of a large
number of objects: from paintings, photographs to large and
small sculptures accompanied by an information apparatus.
Forms of VR-AR are not yet fully integrated into the museum
reality as they require high skills in multiple disciplinary and
application sectors such as computer graphics, computer
programming, digital representation using advanced modelling,
restoration, and archaeology.
1.4 Research objectives : the need to improve the
information sharing for virtual museums based on high
resolution 3D textured models and IVOs
Choosing to create a virtual museum undoubtedly has different
advantages, but it requires you to have in-depth knowledge in
many disciplines and master different digital tools, which go
beyond the simple creation of a website based on panoramas.
The benefits can be the most varied: giving the possibility to
every user in the world, to visit the museum and to personalise
the visit, the itinerary, not being tied to entrance times, not having
specialised personnel to supervise the rooms or that deals with
security, collect works scattered all over the world, reconstruct
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
48
objects or works or sites that have been lost through animations
and 3D models.
As already briefly mentioned in the previous paragraph, through
XR, it is now possible to increase the level of inter-
competitiveness of digital worlds between user and machine,
enhancing the levels of communication through digital forms
capable of sharing different types of data and formats.
Consequently, the understanding and IT management of digital
models and the leading XR development platforms open new
doors for architects, engineers, restorers, archaeologists,
historians, students, virtual tourists, and museum curators, who
for the most part do not have computer skills oriented to the
development of specific IT applications (Hammady et al., 2021).
For those reasons, this research intends to consolidate a method
capable of orienting itself to any artefact and showing how
different 3D objects coming from photogrammetry and 3D
modelling can come to life in different XR ways. In this specific
field of application:
• 3D modelling;
• Visual Programming Language (VPL);
• model interoperability;
were found to be the key elements to be able to make this
transition. In fact, to be as credible as possible, a virtual museum
or XR environment must poss ess specific characteristics such as
(i) models with a high level of detail capable of corresponding to
the objects exhibited in real museums, (ii) high-resolution
textures able to show chromatic and authentic material values and
(iii) the correct sizing in the virtual environment in relation to the
user. These and many other requirements are defined by proper
3D modelling. The latter one articulated in its most advanced
forms, must be ass ociated w ith the VPL, which allows it to switch
from a static mesh to an IVO. Finally, the appropriate use of
exchange formats (from modelling software to XR development
platforms) is crucial for correct visualisation, navigation and
interaction with the digital environment created.
Accordingly, this study proposes a digitisation process as
sustainable as possible concerning a current technological reality,
integrating several forms of photogrammetric digitisation
(terrestrial and aerial) with advanced modelling techniques and
development of VR-AR environments where the end-user can
interact with new forms of IVOs.
In the following paragraphs, the methods applied to two research
case studies describe how simple textured mesh models can be
transformed into smart XR objects capable of communicating
different types of information, coming to life, and interactin g
with user input.
Furthermore, in order to show how the process is applicable to
different 3D realities and objects, this article summarises and
describes the method applied for two case studies of different
nature, both in geometric, historical and cultural terms: the Arco
della Pace in Milan and the archaeological site of Bajardo, Italy.
Both case studies required a cognitive investigation of their
historical heritage and an in-depth analysis of different historica l
sources and manuscripts.
In particular, for the Arco della Pace, in-depth study of historical
treatises allows authors to understand the history of the city of
Milan described and reported appropriately in the decorative
apparatus of the monument (Giani, 1988; Gaberden 1850; Reina,
1839), while for the archaeological site in Bajardo, interviews
and investigations for archives, and websites have allowed the
identification and collection of information that were in danger
of being lost from generation to generation.
2. THE RESEARCH CASE STUDIES: THE ARCO
DELLA PACE OF MILAN AND THE
ARCHAEOLOGICAL SITE OF BAJARDO
2.1 The growing demand for new forms of digitisation for
built heritage and virtual museums: the Covid-19 year and
the new global awareness
In the year of Covid-19, museums lose 75% of visitors and 78%
of revenues. With about four months behind closed doors in
2020, the Italian state museums, monuments and archaeological
areas present the bill, and it could only be negative: -75.67%
visitors and -78.98% net revenues for a total of 41,991,929.03
euros for the 268 institutions monitored by the Sistan (National
Statistical System of the M inis try of Culture). It would help if
you went back to 1996 to find similar collections. At the time,
there were 166 institutions surveyed, and in any case, 25 years
ago, the number of visitors was double. Fortunately, this figure
must be enriched with all the digital activity -from virtual tours
to do-it-yourself exhibitions, from talks with curators and
directors to social activity- which animated the long months of
lockdown. Technologies have helped keeping the contact
between cultural institutions and communities alive but opening
to digital has meant “going out” of one’s habits to enter an
unknown space. Not everyone knows the grammars and not
everyone already structurally has the professional skills prepared
and dedicated. The pandemic, through digital, has shown new
fragility of state museums, which must quickly be overcome. But
returning to the physicality of the visits, for the Colosseum and
Pompei, the data are the worst of all, they lose between 85% and
84% for visitors and revenues, yet the visits were outdoors.
Table 1 shows the data referred to single or cumulative tickets
issued respectively for each Institute or Museum Circuit.
Cumulative tickets are not included in those of the individual
museum institutes making up the Circuit. The object of the
investigation is the ticket sold and not the visitor’s acces s, given
the impossibility of detecting the entrance. From the ticket count
n. 464,639 for a gross amount of 6,723,768.73 euros and a net
amount of 5,781,605.37 euros. These are tickets transformed into
vouchers and not used during the year 2020, which will
eventually be counted at the time of their use.
Museum
City
Total
Viso tors
2019/2020
Delta
Intr oitus
2019/2020
Ga llerie degli Uff izi
Florence
1.206.175
-73
-76
Parco archeologico del
Colosse o
Rome
1.085.907
-86
-85
Parco a rcheologico di
Pompei
Pompei
597.280
-85
-85
Galleria dell’Accademia
di Firenze
Florence
319.451
-81
-85
Reggia di Caserta
Caserta
293.891
-60
-63
Museo delle Antichità
Egizie (gestito dalla
Fondazio ne)
Torin
241.139
-72
-76
Museo Nazionale di
Cast el Sant’Angel o
Rome
219.844
-82
-81
Villa Adria na e Villa
D’Este
Tivoli
218.998
-70
-69
Galleria Borghese
Rome
153.068
-73
-75
Musei Reali
Torin
141.709
-71
-68
Table 1. The top 10 Italian state mus eums , monuments, and
paid state archaeological areas (Il Sole 24 Ore, 2021).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
49
2.2 The monument Arco della Pace in Milan, Italy, and its
decorative complexity
In the centre of the vast Piazza Sempione, which serves as the
head of Corso Sempione, stands the Arco della Pace, begun in
1807 by Luigi Cagnola, with on the sides the toll booths built
both for the collection of the duty and for the delimitation of the
urban territory compared to the campaign. The monument
consists of three arched doors, flanked by colossal, fluted
Corinthian columns and surmounted by the Sestiga della Pace,
bronze by Abbondio Sangiorgio, a chariot pulled by six horses
that welcomes Minerva in Pace, a statue over 4 meters high and
heavier more than 10 tons, accompanied by four “Victories on
horseback” by Giovanni Putti. In Baveno granite, with Crevola
d’Ossola marble cladding, 25 meters high and 24 meters wide,
the imposing work is inspired by Roman models dear to the
neoclassical culture of the 18th and early 19th century and
represents a high example of neoclassical art. Above the
entablature, the four main rivers of the Lombardy-Veneto region
(Po, Ticino, Adige and Tagliamento) are depicted. A rich plastic
decoration, in the academic style of the early 1800s, takes place
on the fronts and sides with exuberant ornaments and bass r eliefs,
mostly on the themes of the restoration. With the return to the
Habsburgs, its commemorative purpose also changed: from Arco
della Vittoria, in memory of the battle of Jena in Thuringia
(Germany) won by the French in October 1806, to Arco della
Pace. On the two fronts of the attic, there are the inscriptions of
the entry of Napoleon III and Vittorio Emanuele II of 1859,
which replaced the previous inscriptions made for Francesco I
and Ferdinando I. Synthetically, the case study of the Arco della
Pace turned out to be a useful application field for the
development of an XR project capable of telling, showing,
cataloguing, and archiving the story told in its unique decorative
apparatus. As described in the following paragraphs, the method
made use of a UAS survey, which made possible to survey unique
works of art as accurately and truthfully as possible (Figure 1).
Figure 1. The Arco della Pace in Milan and its decorative
apparatus. Photos coming from UAV survey.
2.3 The arch aeological sites of Bajardo and its historical-
cultural background
The first certain dating of the building dates to 1245, but it is
thought that it may have been built on a pre-Christian temple
dedicated to Celtic cults. On February 23, 1887, and a large part
of the faithful of the country (212) died due to an earthquake that
struck a large part of Italy. Today the place presents itself with an
austere and w elcoming aspect at the same time, where the living
stone characterises the architectural and structural elements
handed down over the centuries. On one side, the altar on which
a simple wooden crucifix still stands, on the other the chapel of
St. Antonio, miraculously unharmed during the earthquake, the
only testimony of the original Baroque character of the building
before the collapse.
The complex, with its particular aspect, is in turn immersed in a
magical context. To the west, a ruin of a portico with columns
and strange capitals with an ambiguous oriental flavour, evidence
of the passage of ancient stone masters of Cenova, who made
these figures with peculiar features the distinctive feature of their
highly appreciated art. And again, always to the west, a terrace
that overlooks the Ligurian Sea directly, which the eye reaches
accompanied by an expanse of mountains, gorges and hills that
almost seem to imitate the waves.
From here, on clear days, you can see as far as the French coast.
To the north, on the other hand, the Balcony on the mountains, a
large grassy square that offers a breath-taking view of the
Maritime Alps. All around the “caruggi” of the medieval villag e
make their w ay which, between arches and secret passages, draw
a topography to be discovered step by step. The Old Church is a
place full of magnetism and vibrations, with a mysterious
atmosphere, probably the legacy of an ancient history that seems
to hover over the site. A historic place, with a mythical past and
a vivid present, in fact, in the summer, it becomes a place for
cultural shows and meetings, open-air cinema and suggestive
Celtic rites (Alia, Cuomo, 2017) (Figure 2).
Figure 2. The archaeological site of Bajardo and its unique
archaeological artifacts.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
50
3. VIRTUAL OBJECTS AND HIGH-RESOLUTION 3D
MAPPING FOR VR-AR ENABLED GAMING
EXPERIENCES
3.1 3D s urvey o f the research cas e st udies , from primary and
secondary data sources to high resolution textured models
The survey of the Arco della P ace in Milan considered the
integration of different sensors and instruments. Namely, the 3D
survey was performed using a consumer camera, a consumer
Unmanned Aerial Vehicle (UAV) and a total station to check the
accuracy of the final results. These instruments let the operator to
collect the so-called “primary data sources,” i.e. the information
coming directly from the object, its dimensions, colour, position
and peculiar characteristics . The “secondary data sources” refer
to the collection of information coming from the bibliography
and literature about the monument. The integration of primary
and secondary data sources is always fundamental in approaching
a real object’s description, w hether it is a small artefact or a whole
building.
In the case of the Arco della Pace, the first step was to collect the
images from the ground for the photogrammetric elaboration,
supported by a topographic net around the monument. These data
and three different monographs about the monument were
enough to describe it through an affordable, accurate and textured
3D digital model. The upper parts that were not visible from the
ground due to the overhangs of the friezes were reconstructed
thanks to the precise drawings and measurements of the design
project.
However, the authors decided to plan an aerial survey to
complete the missing elements of the arch, i.e. the statues and the
bass reliefs that repres ent a key characteris tic of the monument.
Planning a flight in the city centre of a city as Milan is not trivial :
there are many variables that must be considered before
approaching such a task.
The current regulation about UAV in Italy does not allow the
flight of drones in crowded and sens itive places (ENAC, 2021;
EASA, 2021). The city centre of Milan (D-flight, 2021) falls back
in five different restriction areas: i) LI-R9 Milano-Città, ii)
Milano/Bresso 18/36, iii) Milano/Linate 18/36, iv) Linate
Aerodrome Traffic Zone (ATZ), and v) Linate Control Traffic
Region (CTR). Each of these areas has its own rules for flight and
the most restrictive one is LI-R9 Milano-Città w here the flight is
completely forbidden for security reasons. Actually, temporary
permission can be obtained by the city’s authorities by s ubmitting
all the necessary documents and a high detailed relation that
describes the activity, the timetable of the flights, the risk
assessment and the precautions taken to decrease the level of the
risk.
Moreover, the person in charge for the flight must obtain a
certificate for driving drones after passing an on-line test and a
practical exam. The authors, both holding a piloting license,
provided the material above-mentioned to the prefecture of Milan
that has the faculty to issue or not the permission, then the
permission mus t also be approved by ENAC. Even if the
prefecture issues the permission, the Authority can revoke it.
After a month from the submission of the request, the survey
activities described in the relation received a positive judgment
by the two authorities. The positive judgment was achieved
because it was considered a sufficient buffer area around the
monument, and the date chosen for the survey activity fell into
the lockdown period linked with the Covid-19 pandemic.
Therefore, all the shops restoration activities w ere clos ed and
there were no crowds around the monument since the prohibition
to stay in public spaces without a proven reason. Nevertheless ,
since the Italian regulation considers special rules for UAVs
weighing less than 250 grams, it was decided to use a UAV with
this peculiar characteristic, namely the DJI M avic Mini.
The data processed were composed by 229 images acquired from
the ground with a Canon EOS 1100D coupled with an 18 mm
lens, 945 photos taken during four different flights around the
arch, and some architectural points measured on the four façades
from a simple topographic network around the monument,
measured with the total station Leica TS12.
The distances of the acquisition were calculated with the formula
c: D : px : GSD, where c is the focal length, D is the distance
from the object, px is the pixel size of the sensor, and GSD is the
Ground Sample Distance, to achieve a 1:50 drawing scale. The
GSD for 1:50 scale was assumed equal to 1 cm, multiplying the
plotting error (0.2 mm) by the s cale’s denominator.
As a result, the dis tances calculated for each sens or to reach a
GSD equal to 1 cm are respectively 27 m for the DJI M avic Mini
and 35 m for the terrestrial survey.
The results are theoretical values considering ideal conditions
with a camera placed on a tripod without external interferences.
In the practical activity, it is suggested, even in good conditions,
to halve the results obtained to assure the reaching of the GSD
designed.
Regarding the UA V survey, once the distance was calculated, the
path of the flight followed as much as possible parallel vertical
strips. The vertical (longitudinal) overlap of the images was
controlled, setting the auto interval acquisition of the images
equal to 2 seconds, the side (transversal) overlap was valued
directly on the screen by the video operator of the drone.
DJI Mavic Mini
Canon D 1100
Sensor size (pixel)
4000 * 3000
4272 * 2848
Sensor size (mm)
6.48 * 4.86
22,2 * 14,7
Pixel s ize (mm )
0.00162
0.00520
Focal length (mm)
4.49
18
Flight time (min)
28
x
Proximity sensors
Downwards
x
Table 2. Specs of the sensors used for the photogrammetric
survey.
The photogrammetric data were elaborated using Agisoft
Metashape Pro (version 1.7.2 build 10270), the architectonical
points measured with the total station were used to optimise the
project and to check the accuracy of the result. The terrestrial and
UAV survey were divided in two different chunks and merged
before the generation of the mesh model.
Once the photogrammetric elaboration was completed, the
primary source data, namely the dense point cloud and the high
resolution meshes of the statues, were processed to obtain a
NURBS model using different software and exchange file
formats (Piegl, Tiller, 1996). The dense point cloud was imported
as .e57 file into Mc Neel Rhinoceros 7, here it was possible to
make some slices through the point cloud and reconstruct the
primitive geometries. This phase was also supported by the
bibliography and technical drawings of the arch. A different
pipeline was followed for the generation of the NURBS statues.
The models were exported in .obj file format and imported into
Geomagic Des ign X . The auto-surfacing command lets to adapt
NURBS geometries on the mesh objects, capturing at best the
original shape. Once the NURBS were fixed from a topological
point of view, they were exported in .igs format and imported in
Rhinoceros 7. This workflow preserves the dimensions and the
georeferencing of the models, so it was not necessary to move or
scale the objects in the 3D modelling environment. The models
were finally textured using the images and the orthophotos
coming from the photogrammetric survey (Figure 3).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
51
Figure 3. The survey and post-processing steps of the Arco
della Pace: from terrestrial and aerial photogrammetry to
NURBS textured models.
For the case study of the archaeological site of Bajardo, terrestrial
photogrammetry and the creation of many textured mesh models
made it possible to lay the foundations for creating an orientable
3D environment for the development of the VR-AR projects.
Digital 3D survey techniques quickly highlighted the shape,
dimensions from a morphological and typological point of view
of the altar, apses, external and internal walls, antiquities, and the
entrance characterised by a sail vault. 3D survey obtained an
accurate non-invasive digital data collection, achieving the best
point clouds for each architectural and structural element. The
importance of constructive techniques has played a key role in
the reverse engineering of each architectural component. The
features of the building have been identified directly on the field
thanks to historical documents and 2D drawings, which
completed the interpretive analysis of the building and materials
used for the construction. In this context, as is known, at the
beginning of the transformation process, point cloud processing
applications lead to an automatic generation of mesh models.
Software as Agisoft Metahsape and ContextCapture enabled the
automatic creation of mesh from dense point cloud thanks to
specific algorithms that recognise the 3D points like the data
source to generate the mesh models. Mesh interpreted the
complexity of the shapes through the union of points through
polygons based on different algorithms. It found that mesh
polygons represented a huge constraint for the model generation
and the subsequent management of mesh models in
HBIM/VR/AR objects. For this reason, thanks to the application
of GOG 9 and 10 it has been possible to transform simple meshes
into geometrical models capable of accommodating high-
resolution orthophotos. Furthermore, this approach also made it
possible to better orient digital models for the next phase of VR-
AR development: thanks to specific exchange formats it has been
possible to outline a sustainable transformation process without
remodelling or remapping objects (Figure 4). Starting from the
photogrammetric outputs, the models were transformed into the
following formats: Wavepoint OBJ, FBX, 3DM, RVT, updated
FBX for VR and AR.
Figure 4. The 3D survey and post-processing steps of the St.
Nicolò Archeological Site, Bajardo: from terrestrial
photogrammetry to NURBS textured models.
3.2 From high resolution textured models to VPL
The process of transforming textured mesh model into an object
capable of interacting with user input was determined through the
development of specific visual scripting, also known as
Blueprints. Blueprints is Unreal Engine visual scripting system
accessible to different levels of developers, from professionals to
advanced users to beginners.
As mentioned in the previous paragraphs, programming in C++
can be a daunting task for beginners and Architecture,
Engineering and Construction (AEC) professionals
(Alizadehsalehi, Hadavi, Huang, 2020). Therefore, for
developing the code in C++, the authors implemented several
Blueprints using nodes, events, actions , and conditions linked
together in visual form. This IT development allowed the authors
to transform simple textured mesh models into IVOs composed
of nodes that have input and output values. Furthermore, the
Blueprint-based visual workflow allows the development of
different codes linked together, giving life to the static 3D objects
from photogrammetry, controlling them through a digital
compiler.
For the research case studies, specific Blueprints have been
developed, integrating different functions, actions, and
movements for different 3D objects.
Creating a virtual museum based on new forms of human-
computer interaction had to consider the accessibility and use of
additional information associated with various objects (Helander,
2014). For instance, the possibility of leafing through books able
to describe the decorative apparatus of the monument has led the
authors to develop functions associated with a specific 3D object
to make the book itself browsable, which in turn has been mapped
with the recovered historical texts, drawings, animation, and
audios. Another crucial Blueprint for managing several museum
rooms has been developed to teleport the user (in first or third
person) between one level and another without carrying out long
distances in the virtual environment (Figure 5).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
52
Consequently, other Blueprints have been developed to improve
the interactivity level of the virtual museum and the related IVOs
placed in each level room. The following figure shows one of the
main Blueprints created in relation to the designed XR
environment.
Figure 5. The virtual museum of the Arco della Pace and the
main Blueprints developed for different type of IVOs and
devices.
3.3 IVOs and human-computer interaction based on scan-
to-BIM-to-XR objects
The development stage of the virtual museum had to consider
other technological aspects. With the development of the
Blueprints, different tests were conducted regarding the level of
immersion of the two projects. It was found that the levels of
interactivity highly depended on the device used. In this context,
the test and the use of VR headsets, workstations, mobile phones,
and laptops were decisive for obtaining a complete virtual
experience both from a technical and an informative point of
view. In fact, one of the objectives of the two virtual museums
was to tell the historical and cultural background found during
the study and deepening of the case studies. An open approach
was chosen for the Arco della P ace, capable of developing new
levels, rooms, Blueprints, and information associated with them
over time. This development required Unreal Engine platform
capable of allowing continuous development over time, without
physical limits related to the size of the model. Integrating the
VR headset with a gaming-oriented workstation was crucial in
developing a large XR project. More portable forms were also
investigated for the case study of the archaeological site. Figure
6 shows the tools used to navigate both virtual museums, passing
from different VR-AR interaction and devices forms according
to the IV Os created. For mobile devices , two solutions have been
developed to test the development limits of the geometri c
environments in terms of the number of polygons and textured
model’s size. AR libraries have been oriented for iOS and
Android devices using OBJ and FBX formats (first solution).
Clicking the AR button, the experience shows an auto generated
QR code that viewers can s can to automatically open the model
in AR. The second solution considered mobile game
development. To create and deploy an Android project, authors
needed to install several Android development prerequisites and
ensure that the device was ready for testing.
Figure 6. Human-computer interaction and virtual museums of
both research case studies, from IVOs to VR headsets, mobile
phones, and web-based AR libraries.
Following the guide in Setting Up Android SDK and NDK for
Unreal to install Android Studio and set up the required SDK
components for Android development in Unreal, has been
possible to go beyond a simple web platform and reach a proper
serious game. In this way, developing a training tool where the
serious and playful aspects are in balance has been possible.
The desire to create an effective and pleasant training experienc e
was the main goal of this phase. On the other hand, it w as difficult
to distinguish between entertainment games becaus e it is often
the player himself that determines the formative aspect.
Interactive virtual simulation is also often considered a serious
game. Both have been found to have the fundamental purpose of
developing skills and competencies to be applied in the real world
through exercise in a simulated environment.
4. RESULTS AND CONCLUSION
The proposed process is structured by levels of interoperability
that allow the user to s witch from a static textured model coming
from aerial or terrestrial photogrammetry into IVOs capable of
increasing the human-computer interaction between virtual
museums, XR users and different types of devices, from VR
headset to mobile phone. To demonstrate the reliability ,
sustainability and applicability of the method, different types of
photogrammetric textured mesh models were transformed into
objects capable of being associated with different types of visual
scripting. In particular, both the project of the A rco della Pace
monument and the Bajardo archaeological site project have made
it possible to identify exchange formats capable of transmitting
high Levels Of Detail (LOD) and Level Of Information (LOI) for
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
53
different types of devices, from workstations to mobile phones
(VR-AR). In this context, great flexibility of NURBS textured
models has been found, unlike mesh models that still require long
cleaning, decimation, and simplification processes, especially
those oriented towards web platforms. For these reasons, the
workflow proposes digital transformations of the models with the
aim of increasing the level of modelling automation as much as
possible, passing from simple mesh formats to exchange formats
capable of interoperating with XR development platforms such
as Unreal Engine, Unity and Twinmotion. In addition, the great
advantage of VR over AR was found.
VR allows the implementation of rich interactive projects in
terms of content, size of the objects and textures used. On the
other hand, AR still requires web platforms and apps through
which it is not possible to overcome certain limits of polygons
and size of the models (50, 100, 200 Mb), thus limiting the use
of scan-to-BIM models aimed at managing high LODs.
Furthermore, LOI is still very low, where at most, the end-user
can interact with very concise descriptive panels and
development functions.
Future developments will improve the scan-to-BIM-to-XR
process, and the level of interoperability oriented to different
types of analysis such as VR and AR. In this context, the latest
generation devices such as the VR headset and mobile phones
will require specific XR and visual scripting projects to make
human-computer interaction as fluid as possible. Consequently,
XR development platforms, Blueprint and IVOs will play a
fundamental role in the transmissibility of content and improving
the world of virtual museums and DCH in general.
ACKNOWLEDGEMENTS
The authors thank the Municipality of Milan and Bajardo (Italy),
ENAC and Dr. Michela Sartori Prefettura of Milan Area I –
Ordine e Sicurezza Pubblica for the authorizations to fly.
The authors thank Arch. Jacopo Alberto Bonini and the students
Mohamed Jaabar and Luca Melotto for the content
implementation process of the virtual museum of Arco della
Pace, Milan, Italy.
REFERENCES
Alia, A., & Cuomo, L., 2017. Bajardo 360: strategie di
rigenerazione per un borgo dell’entroterra ligure.
Alizadehsalehi, S., Hadavi, A., & Huang, J. C., 2020. From BIM
to extended reality in AEC industry. Automation in
Construction, 116, 103254.
Atsiz, O., 2021. Virtual reality technology and physical
distancing: A review on limiting human interaction in
tourism. Journal of multidisciplinary academic tourism, 6(1),
27-35.
Banfi, F., 2017. BIM orientation: Grades of generation and
information for different type of analysis and management
process, in: International Archives of the Photogrammetry,
Remote Sensing and Spatial Information Sciences - ISPRS
Archives . International Society for Photogrammetry and Remote
Sensing, pp. 57–64. https://doi.org/10.5194/isprs-archives-XLII -
2-W5-57-2017.
Bosché, F., Ahmed, M., Turkan, Y., Haas, C. T., & Haas, R.
(2015). The value of integrating Scan-to-BIM and Scan-vs-BIM
techniques for construction monitoring using laser scanning and
BIM: The case of cylindrical MEP components. Automation in
Construction, 49, 201-213.
Chen, M., Feng, A., McAlinden, R., & Soibelman, L. (2020).
Photogrammetric point cloud segmentation and object
information extraction for creating virtual environments and
simulations. Journal of Management in Engineering, 36(2),
04019046.
D-flight, 2021, https://www .d-flight.it/new_portal/ (7 J une 2021)
ENAC, 2021, http s://ww w.enac.g ov.it/sites /default/fil es/al leg a ti
(2 June 2021)
EASA, 2021, https://www.easa.europa.eu/document-library/
(2 June 2021)
Gaberden, F. J., 1850. Der Friedensbogen in Mailand: Arco
della pace, Joh M einers.
Giani, G., 1988. L’Arco della pace di Milano. Di Baio Editore.
Hammady, R., Ma, M., Ziad, A. K., & Strathearn, C. , 2021. A
framework for constructing and evaluating the role of MR as a
holographic virtual guide in museums. Virtual Reality, 1-24.
Helander, M. G. (Ed.), 2014. Handbook of human-computer
interaction. Elsevier.
Huhtamo, E., 2013. On the origins of the virtual museum (pp.
134-148). Routledge.
Ibiblio, 2021, http://www.ibiblio.org/w m/ (14 June 2021)
Il Sole 24 Ore, 2021, https://w ww .ils ole24ore.com/art/nell-anno-
covid-musei-perdono-75percento-visitatori-e-78percento-
introiti-AEkEcN G?refres h_ce= 1 (14 June 2021)
Kwok, A. O., & Koh, S. G., 2020. COVID-19 and extended
reality (XR). Current Issues in Tourism, 1-6.
Piegl, L., & Tiller, W., 1996. The NURBS book. Springer
Science & Business Media.
Reina, G., 1839. Descrizione dell’Arco della pace in Milano.
Tipografia Ronchetti.
Sylaiou, S., Mania, K., Karoulis, A., & White, M., 2010.
Exploring the relationship between presence and enjoyment in a
virtual museum. International journal of human-computer
studies, 68(5), 243-253.
Volk, R., Stengel, J., & Schultmann, F. (2014). Building
Information Modeling (BIM) for existing buildings – Literatur e
review and future needs. Automation in construction, 38, 109-
127.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021
28th CIPA Symposium “Great Learning & Digital Emotion”, 28 August–1 September 2021, Beijing, China
This contribution has been peer-reviewed.
https://doi.org/10.5194/isprs-archives-XLVI-M-1-2021-47-2021 | © Author(s) 2021. CC BY 4.0 License.
54
