Content uploaded by Tim Collins
Author content
All content in this area was uploaded by Tim Collins on Mar 31, 2021
Content may be subject to copyright.
Virtual Museum ‘Takeouts’ and DIY Exhibitions –
Augmented Reality Apps for Scholarship, Citizen Science
and Public Engagement
Sandra Woolley1[0000-0002-7623-2866], James Mitchell2[0000-0001-6051-2567],
Tim Collins3[0000-0003-2841-1947], Richard Rhodes4[0000-0002-6955-7216],
Tendai Rukasha5[0000-0003-3378-0843], Erlend Gehlken6[0000-0003-0332-4759],
Eugene Ch’ng7[0000-0003-3992-8335] and Ashley Cooke8
1,2,4,5 Keele University, Staffordshire, UK
3 Manchester Metropolitan University, Manchester, UK
6 Goethe-Universität, Frankfurt am Main, Germany
7 Nottingham Ningbo University, Ningbo, China
8 National Museums Liverpool (World Museum), Liverpool, UK
s.i.woolley@keele.ac.uk
Abstract. This paper presents an Augmented Reality (AR) project for the cura-
tion of virtual museum ‘takeouts’ and DIY exhibitions. The project’s outputs in-
clude novel AR app technology demonstrators to support co-design with museum
users and stakeholders - the goal being to create useful and easy-to-use AR apps
for scholars, citizen scientists and the interested public.
The apps were designed for users to create, display, animate and interact with
exhibitions of selected 3D artefacts that could, for example, reflect academic spe-
cialisms for sharing with fellow researchers, support curators in exhibition plan-
ning or enable friends and students to share eclectic favourites from museum vis-
its. The overarching project ambition was to create AR apps to support research,
engagement and education, and to enable interactive and personalized visualiza-
tions of individual artefacts as well as reconstructed forms. As presented in the
paper, these forms are exemplified in the AR apps with 3D models of a cuneiform
envelope and its tablet contents, viewable either as i) separate artefacts or ii) in
their reconstructed enveloped form, with the AR apps enabling animated opening
and ‘X-ray views’ of the contents within. In this way, the apps can enable users
to visualize individual objects and reconstructions that could, for example, incor-
porate artefacts held in different museums.
Keywords: Augmented reality, Digital heritage, Virtual reconstruction.
1 Introduction
Augmented Reality (AR) offers new and exciting opportunities to enrich museum and
art gallery visits and provide additional interactions and experiences for visitors [1-4].
AR can also provide digital heritage visualization opportunities, for example, for ar-
chaeological finds [5] and reconstructed artefacts such as glass plates [6], as well as
Preprint
The final publication is available at Springer via https://www.springer.com/us/book/9783030730420
DOI: 10.1007/978-3-030-73043-7
2
affording the potential for engaging educators and student learners in cultural heritage
experiences [7].
Notable museum AR examples include Saltzburg’s Museum of Celtic Heritage
‘Speaking Celt’ historical AR avatar [8], Cleveland’s Museum of Art ArtLens AR app
that provides enriched tour experiences of displayed artworks [9] and The Franklin In-
stitute’s AR app that enables visitors to interact with virtual 3D models of Terracotta
Warriors [10]. These AR apps enrich the in-museum experiences of displayed artefacts
[11, 12] but, like other heritage AR apps, they do not support interaction with artefacts
that are not on display or enable the creation of virtual exhibitions. More broadly,
amongst art and digital heritage AR apps, there is scope for improved personalization
and better ways of revealing non-visible components and communicating artefact in-
formation [13].
The Virtual Museum ‘Takeouts’ and DIY Exhibitions project [14] evolved from The
Virtual Cuneiform Tablet Reconstruction Project [15] whose original ambitions in-
cluded support for virtual access to museum artefacts and the creation of tools to sup-
port artefact reconstruction [16] as exemplified by the virtual reconstruction of the At-
rahasis cuneiform tablet [17-19]. The aims of the Virtual Museum ‘Takeouts’ and DIY
Exhibitions project were (i) to create technology demonstrators that could supplement
the co-design of useful and easy-to-use AR apps aimed at benefiting diverse user groups
that include scholars, citizen scientists and the interested public, and (ii) to provide in-
teractive, informative and personalized views of individual artefacts and virtual recon-
structions. The creation of technology demonstrators was important because co-design
improves technology outcomes [20] and functional technology demonstrators can sup-
port the process by helping to inspire ideas [21]. In addition, technology demonstrators
are useful when some user groups may be unfamiliar with a new technology.
2 AR Apps for Museum ‘Takeouts’ and DIY Exhibitions
The requirements for the AR apps were to incorporate functionality to demonstrate:
•virtual presentations of museum artefacts that may or may not be on physical display,
•the collection and arrangement of virtual life-sized artefacts in augmented reality
exhibitions,
•alternative point cloud and wire mesh views of artefacts for interest and for educa-
tional insights into the structure of 3D models,
•the optional display of artefact information,
•artefact rotation to provide all-around views,
•the option to display individual artefacts or to view them in their assembled or re-
constructed forms.
2.1 3D-Model Acquisitions
The models for the AR apps were acquired in 2018 and 2019 from National Museums
Liverpool (World Museum), UK using the Virtual Cuneiform Tablet Reconstruction
3
Project photogrammetric turntable system [22] and an Einscan-SP 3D scanner with
Discovery Pack. The models included a ‘shabti’ figurine and cuneiform tablets.
Shabtis are small funerary figurines that were placed in Egyptian tombs to act as
servants for the deceased. The shabti scanned for use in the app is over 2,500 years old
and was once owned by Florence Nightingale who spent several months in Egypt in
1849-50.
Cuneiform tablets are early written records from Mesopotamia, created by making
impressions in clay tablets. The tablets scanned for the app included one with an ac-
companying envelope bearing cylinder seal impressions. This envelope originally se-
cured a silver purchase recorded on a tablet within (which had been removed and the
envelope resealed). The scanned cuneiform tablets are included in the collection of
World Museum Liverpool tablets published in [23] and in the Cuneiform Digital Li-
brary Initiative (CDLI) database [24].
2.2 AR App Development
The Android AR app was developed using the Processing for Android AR Java library,
which itself uses the Google ARCore library. The iPhone AR app was coded in Swift
with Xcode using ARKit. The apps were tested in development by researchers and test-
ers accessing pre-release versions in Google Play test track and the Apple App Store
Test Pilot. Both apps require minimum device specifications; Android’s ARCore, at the
time of writing, requires Android 8.1 or later [25] and the iPhone app requires iOS11
and an A9 or later processor [26]. Example screenshots of the Android and iPhone apps
are shown in Fig. 1.
ARKit and Swift for iOS.
ARKit is Apple’s library for iOS device augmented reality development, supporting
motion tracking, surface detection and light estimation. It uses the smartphone’s camera
to identify and track features in the environment and estimate ambient light so that it
can position and illuminate selected objects in the camera view of the environment.
When running, ARKit extracts features from the camera images and builds a topo-
graphic map of the real world.
Xcode is Apple’s iOS Operating System Integrated Development Environment
(IDE) [27]. Xcode (IDE) version 11.7 (11E801a) was used to create the prototype ap-
plication with Swift being the predominant coding language. The app was developed
using SwiftUI (iOS 13). This allows for the building of user interfaces for any Apple
device using just one set of tools and APIs
4
(i)
(ii)
(iii)
(iv)
(v)
(vi)
Fig. 1. Android and iPhone AR app screen examples. For Android: (i) initial Android splash
screen, (ii) a positioned shabti and (iii) help menu explaining interface buttons. Below for iPh-
one: (iv) the splash screen, (vi) positioned shabti artefact with information display ON and (v) a
screenshot from the opening/closing animation of the cuneiform envelope that reveals the cu-
neiform tablet within.
5
ARCore and Processing for Android.
ARCore [28] is Google’s library for Android smartphone augmented reality devel-
opment, supporting motion tracking, surface detection and light estimation. It uses the
smartphone’s camera to identify and track features in the environment and estimate
ambient light so that it can position and illuminate selected objects in the camera view
of the environment. When running, ARCore extracts features from the camera images
and builds a 3D map of the real world. A collection of features located on a horizontal
or vertical plane is known as a ‘trackable’ and can have virtual objects attached to it
using an ‘anchor’. The Android AR app was created using ‘Processing for Android’
[29] which enables the creation of Processing ‘sketch’ programs for Android devices
and includes support for ‘AR in Processing’ [30] via the ARCore library.
Processing for Android has an advantage over development using Android Studio in
terms of ease-of-use and the simplicity of the code. A rudimentary demonstration AR
Processing app can be produced using less than 30 lines of code whereas an equivalent
Android Studio project would require the creation and navigation of a folder structure
containing 50 or more different files. However, the ease of getting started in Processing
for Android should not trivialize the coding required of full app development. At the
time of writing, the current version of the AR Museum app comprises six files that add
up to approximately 1200 lines of code (much of which is required for the loading and
interpretation of 3D object files).
There are also limitations to the Processing for Android approach; not all of the fea-
tures of the ARCore library are exposed to programmers by the Processing interface.
For example, depth mapping, augmented images and cloud anchors [28] cannot (at the
time of writing) be utilised in Processing. Additionally, the multiple ‘activities’ made
available by Android Studio are not available in Processing for Android. For example,
implementation of an initial 2D splash screen could be easily achieved in Android Stu-
dio by creating a new activity. In Processing, however, an overlay is needed to obscure
the main activity. The Processing splash screen screenshot is shown in Fig. 1(i).
To enhance the functionality of the app, user interface control buttons overlaying the
3D display were added. No library functions for this are provided. Normally, imple-
menting a 2D overlay on a 3D view in Processing would simply require the ‘camera’
to be placed in its default position meaning a 3D coordinate of (x, y, 0) would corre-
spond to a screen coordinate of (x, y); then, the controls can be drawn using standard
2D graphics functions. When using the AR library, however, camera positioning is dis-
abled so a lower level approach was required. Instead of utilizing Processing’s camera
placement function, the OpenGL projection matrix coefficients were directly reset to
achieve the same effect.
3 Design Decisions and View Options
Several design decisions were required during app development. For example, An-
droid’s ARCore library enables the detection of any flat surfaces, which would allow
objects to be impossibly attached to walls or suspended from ceilings. To simplify the
surface detection process, only horizontal surface detections were permitted. Additional
6
decisions were required to create a functional user interface. This was achieved using
intuitive icon designs based on past research [16]. The icons were placed away from
the centre of the camera view and artefact information was placed so that it remained
in view of its ON/OFF control. Improving the usability of user experience of AR apps
[31] is an important goal of further research and co-design. In addition to improving
the interface to the objects, it is also important to guide users in the basics of getting
started with the use of AR apps. For example, making recommendations on ambient
light levels, smartphone movement and surface selections because dim lighting, rapid
camera movement and smooth featureless surfaces all make surfaces difficult to detect.
As shown in Fig. 1, artefacts were placed on labelled plinths with clickable infor-
mation ‘i’ buttons in both the Android and iPhone apps to toggle the display of artefact
information. Fig. 1(ii) shows the Android interface and (iii) shows the help screen (ac-
tivated by the top right ‘?’ help button) indicating the function of each of the interface
buttons.
3.1 3D Model Views
One of the education goals of the apps included insights about the underpinning 3D
graphics of virtual objects. For this purpose, we implemented alternative artefact view
options to enable users to see low-resolution point cloud views of models and wire
mesh views, as well as photographically rendered views, as shown in Fig. 2.
(i)
(ii)
(iii)
Fig. 2. 3D model views. (i) low-fidelity point cloud view, (ii) wire mesh view and (iii) 3D pho-
tographically rendered view.
7
For the iPhone AR app, which did not implicitly support point cloud and mesh views
of objects, 3D models were created comprising small 3D sphere ‘points’ and meshes
made from narrow cylinder ‘wires’, respectively, to give the appearance of the views.
An additional supporting webpage multi-view interface [32, 33] was also created for
viewing from desktop computers and other devices, as shown in Fig. 3. The web page
also provides links to 3D file downloads for artefact models as both .ply and .stl formats
and in low- and medium-resolutions, for example, for printing artefacts at different
sizes as shown by the 3D printed shabti models in Fig. 4. Models were also made avail-
able for the cuneiform tablet [32].
(i) (ii)
Fig. 3. Interactive desktop browser interface with 3D multi-view options [33].
(i) Photographically rendered and (ii) wire mesh view.
Fig. 4. 3D prints of the shabti figurine. Transparent clear resin prints at full and reduced sizes.
8
3.2 Exhibition Views and Interactions
As shown in the example screenshots in Fig. 5, users can (i) arrange artefacts on de-
tected surfaces (with or without information display) and (ii) select an automated posi-
tioning format to arrange artefacts in line with each other.
The apps also support interaction with the artefacts, for example, with a toggle arte-
fact “rotate” function in the Android app.
(i)
(ii)
Fig. 5. Views of eclectic Android app exhibition arrangements (i) user-positioned artefacts with
information display ON and background grid OFF and (ii) Exhibits viewed with automatic line-
up ON, information display OFF and background grid ON.
3.3 Individual and Reconstructed Artefact Views
Fragmented artefacts can be separated within collections or between museum collec-
tions that may be many kilometres apart [17]. For example, joining pieces of cuneiform
9
tablets have been found separated by 1,000 kilometres, in the British Museum and the
Musée d’Art et d’Histoire in Geneva [18, 19]. AR apps can enable the viewing of indi-
vidual fragmented or component artefacts or their constructed forms. For example, as
shown in Fig. 6, cuneiform envelopes and their tablet contents can be shown separately
or assembled with the contents (as it would have originally been) inside the envelope.
In Fig. 6 (i) the iPhone app animation opens and closes the envelope to reveal the con-
tents, in (ii) the Android app, on close inspection, reveals the enveloped contents to the
user.
(i)
(ii)
Fig. 6. Reconstructed views showing animated and ‘X-ray’ inside views, respectively for the
cuneiform tablet envelope contents in (i) the iPhone AR app and (ii) the Android AR app.
4 Opportunities and Future Development
4.1 Opportunities
3D models of artefacts offer new opportunities for curators in museum exhibition plan-
ning and design. Models provide accurate data for the arrangement of artefacts within
the fixed space of a showcase, giving a greater visual impression before installation,
conceivably reducing the need for last-minute changes in the days before opening. This
is of particular advantage when creating exhibitions that bring together artefacts from
numerous institutions and private collections. There is a greater call on museums to
extend the legacy of short-lived temporary exhibitions beyond the usual publication of
a catalogue. 3D models could be used to enhance a virtual tour of the gallery with in-
scribed artefacts such as cuneiform and shabtis, offering greater intellectual stimulation
for virtual visitors.
10
4.2 Future Development
The apps and their 3D models make demands of smartphone storage space and, in the
current apps, the example models are downloaded as part of the app installation. Ideally,
the app and models would be separate downloads and new models would be down-
loadable individually when selected. This would speed up updates to the app and allow
for new models to be downloaded more efficiently. In museum contexts this could be
via a local Wi-Fi service to reduce the burden on mobile data transfer limits.
In physical exhibition spaces, QR codes could reveal the details and locations of
available AR ‘takeout’ artefacts, enable easy selection and download, and provide links
to more information and multimedia resources. To engage and incentivize visitors, the
app could support social media sharing of artefacts, and user achievements could be
recognised for artefact collection milestones.
Achieving the functionality of the AR app with a minimal and uncluttered interface
was one of the main challenges of the app development. Further work and co-design
with different users and stakeholder groups is needed to evolve the apps from technol-
ogy demonstrators to fully functioning apps. Ideally, the apps would be supported by
infrastructure and data repositories to enable the download of 3D artefacts at appropri-
ate scale and resolution, and in suitable formats. Ideally, the apps would also support
language options, for example, from Google Translate, so that exhibit information dis-
plays could be shown in the user's chosen language. A further enhancement to this
would include the option for text to speech delivery of the information.
Acknowledgements: The development of the app was supported by Keele University
Faculty of Science Awards. The authors thank Ash Leake, from the School of Compu-
ting and Mathematics at Keele University, for the 3D printing of artefacts. The authors
also wish to thank National Museums Liverpool for permissions and support in achiev-
ing the acquisitions and apps.
References
1. Ding, M.: Augmented reality in museums. Museums & augmented reality–A collection of
essays from the arts management and technology laboratory, 1-15. (2017)
2. Morozova, A., How to use augmented reality in museums: examples and use cases,
https://jasoren.com/how-to-use-augmented-reality-in-museums-examples-and-use-cases/,
last accessed 2020/09/13.
3. González Vargas, J.C., Fabregat, R., Carrillo-Ramos, A., Jové, T.: Survey: Using augmented
reality to improve learning motivation in cultural heritage studies. Applied Sciences, 10(3),
897. (2020)
4. Challenor, J., Ma, M.: A review of augmented reality applications for history education and
heritage visualisation. Multimodal Technologies and Interaction, 3(2), 39. (2019)
5. Fernández-Palacios, B.J., Rizzi, A. and Nex, F., 2012, October. Augmented reality for ar-
chaeological finds. In Euro-Mediterranean Conference (pp. 181-190). Springer, Berlin, Hei-
delberg.
11
6. Abate, A.F., Barra, S., Galeotafiore, G., Díaz, C., Aura, E., Sánchez, M., Mas, X. and Ven-
drell, E., 2018, October. An augmented reality mobile app for museums: virtual restoration
of a plate of glass. In Euro-Mediterranean Conference (pp. 539-547). Springer, Cham.
7. Tzima, S., Styliaras, G. and Bassounas, A., 2019. Augmented reality applications in educa-
tion: Teachers point of view. Education Sciences, 9(2), p.99.
8. Breuss-Schneeweis, P.: “The speaking celt”: augmented reality avatars guide through a mu-
seum - case study. In Proceedings of the ACM International Joint Conference on Pervasive
and Ubiquitous Computing: Adjunct, 1484-1491 (2016)
9. Alexander, J.: Gallery one at the Cleveland Museum of Art. Curator: The Museum Jour-
nal 57(3), 347-362 (2014)
10. Terracotta Warriors meet augmented reality at The Franklin Institute, https://www.wiki-
tude.com/showcase/terracotta-warriors-augmented-reality-at-the-franklin-institute/, last ac-
cessed 2020/09/13.
11. Serravalle, F., Ferraris, A., Vrontis, D., Thrassou, A. and Christofi, M., 2019. Augmented
reality in the tourism industry: A multi-stakeholder analysis of museums. Tourism Manage-
ment Perspectives, 32, p.100549.
12. Damala, A., Stojanovic, N., Schuchert, T., Moragues, J., Cabrera, A. and Gilleade, K., 2012,
October. Adaptive augmented reality for cultural heritage: ARtSENSE project. In Euro-
Mediterranean Conference (pp. 746-755). Springer, Berlin, Heidelberg.
13. Pucihar, K.Č. and Kljun, M., 2018. ART for art: augmented reality taxonomy for art and
cultural heritage. In Augmented Reality Art (pp. 73-94). Springer, Cham.
14. The augmented reality museum app homepage, https://virtualcuneiform.org/AR-
Museum.html, last accessed 2020/09/13.
15. The Virtual Cuneiform Tablet Reconstruction Project homepage, https://virtualcunei-
form.org/index.html, last accessed 2020/09/13.
16. Woolley, S.I., Ch’ng, E., Hernandez-Munoz, L., Gehlken, E., Collins, T., Nash, D., Lewis,
A., Hanes, L.: A collaborative artefact reconstruction environment. In: Proceedings of the
British HCI Conference. BCS. Sunderland, UK, 3rd-6th July, 2017.
17. Woolley, S.I., Gehlken, E., Ch’ng, E., Collins, T.: Virtual archaeology: how we achieved
the first long-distance reconstruction of a cultural artefact, The Conversation, UK (Art and
Culture), 28 Feb 2018. https://theconversation.com/virtual-archaeology-how-we-achieved-
the-first-long-distance-reconstruction-of-a-cultural-artefact-91725, last accessed
02/09/2020.
18. Gehlken, E., Collins, T., Woolley, S.I., Hanes, L., Lewis, A., Munoz L.H., Ch’ng, E.:
Searching the past in the future - joining cuneiform tablet fragments in virtual collections,
63rd Rencontre Assyriologique Internationale, 2017.
19. Collins, T., Woolley, S., Gehlken, E., Ch'ng, E.: Computational aspects of model acquisition
and join geometry for the virtual reconstruction of the Atrahasis cuneiform tablet. In 23rd
International Conference on Virtual System & Multimedia (VSMM), 1-6. (2017)
20. Steen, M., Manschot, M., De Koning, N.: Benefits of co-design in service design projects,
International Journal of Design, 5(2), (2011).
21. Ciolfi, L., Avram, G., Maye, L., Dulake, N., Marshall, M.T., van Dijk, D., McDermott, F.:
Articulating co-design in museums: Reflections on two participatory processes. In Proceed-
ings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social
Computing, 13-25 (2016).
22. Collins, T., Woolley, S.I., Gehlken, E., Ch’ng, E.: Automated low-cost photogrammetric
acquisition of 3D models from small form-factor artefacts. Electronics, 8(12)-1441, 2019.
23. Cripps, E.L., Sargonic and presargonic texts in the World Museum Liverpool. Archaeopress,
(2010).
12
24. Cripps, E.L. Short history of the cuneiform collection in the World Museum Liverpool,
https://cdli.ucla.edu/collections/liverpool/liverpool_intro.pdf, last accessed 2020/09/13.
25. ARCore supported devices, https://developers.google.com/ar/discover/supported-devices,
last accessed 2020/09/13.
26. Apple Augmented Reality, https://www.apple.com/uk/augmented-reality/, last accessed
2020/09/13.
27. Apple, 2020. Xcode - Apple Developer. [online] Apple Developer. Available at: https://de-
veloper.apple.com/xcode/ [Accessed 14 September 2020]
28. ARCore Overview, https://developers.google.com/ar/discover, last accessed 2020/09/13.
29. Processing for Android, https://android.processing.org/index.html, last accessed
2020/09/13.
30. https://android.processing.org/tutorials/ar_intro/index.html, last accessed 2020/09/13.
31. Dey, A., Billinghurst, M., Lindeman, R.W., Swan, J.: A systematic review of 10 years of
augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37 (2018).
32. 3D interactive cuneiform tablet, https://virtualcuneiform.org/tellohModels.html, last ac-
cessed 2020/09/13.
33. 3D interactive shabti model, https://virtualcuneiform.org/shabtiModels.htm, last accessed,
last accessed 2020/09/13.