Content uploaded by David Bouck-Standen
Author content
All content in this area was uploaded by David Bouck-Standen on Feb 26, 2018
Content may be subject to copyright.
A preview of the PDF is not available
NEMO Converter 3D: Reconstruction of 3D Objects from Photo and Video Footage for Ambient Learning Spaces
Abstract and Figures
In ambient and mobile learning contexts, 3D renderings create higher states of immersion compared to still images or video. To cope with the considerable effort to create 3D objects from images, with the NEMO Converter 3D (NOC3D) this paper presents a technical approach to automatically reconstruct 3D objects from semantically annotated media, such as photos and more importantly video footage, in a background process. By using the Mobile Learning Exploration System (MoLES) with a smartphone, the user creates and collects media in mobile context, which are automatically uploaded into the NEMO-Framework (Network Environment for Multimedia Objects) together with semantic annotations for contextualized access and retrieval. NEMO provides an extendable web-based framework to store media like photos, videos and 3D objects together with semantic annotations. The framework has been developed for Ambient Learning Spaces (ALS) in a research project. With InfoGrid, a mobile augmented reality application connected to NEMO, the user experiences the previously generated 3D object placed and aligned into real world scenes. 3D objects automatically reconstructed from photo and video footage by NOC3D are stored in NEMO and thus provided to all applications accessing the NEMO API. Related to the pedagogical background of our research project, this paper focuses on the technical realization and validation of NOC3D with reference to a realistic scenario for the usage of NOC3D in ambient and mobile contexts. In Section 2, we regard related work. In Section 3, we present a practical scenario for using NOC3D. In Section 4, we describe the technical environment for NOC3D and our research project. In Section 5, we outline the realization of NOC3D in the ambient context of our scenario. In Section 6, we present our findings and conclude with a summary and outlook in Section 7.
Figures - uploaded by David Bouck-Standen
Author content
All figure content in this area was uploaded by David Bouck-Standen
Content may be subject to copyright.
... Media Conversion: As digital devices with very different sizes and capabilities are used in ALS, an automatic media converter has been built, which transforms media to the best fitting size, resolution, or aspect ratio to be cached and delivered to the application devices. Additionally, the NEMO 3D Object Converter (NOC3D) can be used to automatically create 3D models of physical objects from a series of photographs taken [42,43]. The user can upload photos and videos into the object converter, which then automatically processes the sources using photogrammetric methods building a 3D object with an increasing quality with an increasing number of images taken (Section 5.4). ...
... For this, photos or videos of physical 3D objects will be taken. A special module of the NEMO media conversion layer as well as the 3DEdit authoring system will be used to create these 3D objects [42,43] (Section 5.4). The InfoGrid app contains an inline 3D placement editor for mobiles to place and align overlays into the real scene [49] (Fig. 15). ...
... VR engines, like in the Unity framework, can create similar effects. The VR modes additionally support the interaction with 3D objects that can be created by teachers, learners, and curators themselves through a photogrammetric pipeline from images and video footage [42,43] (Section 5.4). Figure 18. 360° productions can be brought into the ALS InteractiveDome system. ...
Abstract: Ambient Learning Spaces (ALS) is a platform to create, use, and manage digital teaching and learning environments. The concepts for ALS have been based on contemporary pedagogical theories in awareness that learning processes are going to be less dependent on teaching technologies and content and more on the active processes of learning itself. ALS has been developed over more than 15 years in a series of research projects. ALS enables individuals as well as groups of learners to create, share, distribute, and reuse semantically annotated media in social teaching and learning processes. The central component of ALS, the Network Environment for Multimedia Objects (NEMO), forms the media and logic backend for multimodal interactive frontend learning applications. It is a cloud-based repository and allows the creation of semantically annotated multimedia content to be used across all ALS learning applications. These ALS applications can be used on mobile, stationary, and immersive interactive systems such as domes, large screens, tangibles, mobiles, and wearables. Media content in form of text, image, video, or 3D can be reused and shared by different learning applications and will be adapted on the fly as needed for various interaction devices with different interaction modalities. Interaction methods depending on the available devices reach from direct manipulation interfaces on touch displays to mobile augmented reality and interactive 360° virtual reality applications. Plain media can be constructed into Knowledge Media by semantic markups. Through the ALS Portal, teachers and learners include, edit, annotate, and semantically model their own media. Special user interfaces like tangible media and large multi-touch screens as well as mobile media and wearables enable body-and space-related post-constructivist learning in real life contexts. ALS has been used in schools, museums, universities, biotopes, and urban spaces in pilot installations with different configurations, some of them being used in long-term studies over years.
Purpose: This is the final technical research report for giving a detailed overview about the vision, goals, methodology, technology, and studies performed over more than 15 years development and pilot installations of the ALS Project. Most of the content, especially the studies, can be found in more detail in the different publications referenced in the report.
... Media such as still images and videos from users stored in NEMO are used in automated background processes by NEMO to create 3D objects from these media without the need of user supervision [6]. In our approach, we use these 3D objects to enhance the learner's experience by placing them into context, e.g. with the application InfoGrid, with which web-based augmented reality (AR) tours for smartphones can be experienced [7]. ...
... All data processed are images and videos taken with different camera models. These are stored as semantic media in NEMO and selected by NEMO for the process of 3D reconstruction [6]. NEMO stores the reconstructed 3D object together with semantic annotations of the media used for reconstruction. ...
... Thus, when viewed, the reconstructed 3D objects may appear with an offset to the side or turned away from the viewer (cf. Figure 1). The intuitive solution to isolate the physical object prior to reconstruction is not an option, as the process of 3D reconstruction requires surroundings to be part of the original footage [6]. ...
In this contribution, we present an approach in enhancing 3D objects, which are automatically reconstructed from semantic media in a cloud-based ambient platform. Due to the automatic background process of 3D reconstruction, the objects contain artifacts from the reconstruction process and are not aligned and not position well for direct use in mobile augmented reality apps, such as our InfoGrid system. The goal is to automate the process of enhancing these 3D objects. In our approach we monitor users' interactions with a web-based 3D editor. From these interactions, we derive constraints and show, that for our scenario these parameters can be generalized and applied to other 3D objects, in order to process them automatically in the background. This continues previous work and extends the Network Environment for Multimedia Objects (NEMO), a web-based framework used as the technical platform for our research project Ambient Learning Spaces (ALS). NEMO is the basis for ALS and among other features provides contextualized access and retrieval of semantic media. In various contexts of ALS, compared to still images or video, 3D renderings create higher states of immersion. We conclude this article with a discussion of our findings and with a summary and outlook.
... Together with ambient and mobile technology, these various individual interconnections between physical and digital worlds play an important role. Thus, being and acting in the physical world is accompanied by the creation of technology-assisted environments, like through the creation and visualization of 3D objects [1]. ...
... Any query result of the internal or any external Figure 6. The NEMO-Framework [1]. The NEMO Logic computes, e.g., coherences, semantic models, and data mapping and a modularised interface for feature extendibility. ...
... Preparing the media for processing is performed with linear effort, including extracting usable still images from video as shown in Figure 7. All further steps require significantly more effort, depending on the number of images, the objects complexity and the image resolution [1]. ...
In ambient and mobile learning contexts, 3D renderings create higher states of immersion compared to still images or video. To cope with the considerable effort to create 3D objects from images, with the NEMO Converter 3D we present a technical approach to automatically reconstruct 3D objects from semantically annotated media, such as photos and more importantly video footage, in an automated background process.
Although the 3D objects are rendered in a quality acceptable for the scenario presented in this article, they still contain unwanted surroundings or artifacts and will not be positioned well for, e.g., augmented reality applications. To address this matter, with 3DEdit we present a web-based solution allowing users to enhance these 3D objects. We present a technical overview and reference pedagogical background of our research project Ambient Learning Spaces, in which both the NEMO Converter 3D and 3DEdit have been developed. We
also describe a real usage scenario, starting by creating and collecting media using the Mobile Learning Exploration System, a mobile application from the application family of Ambient Learning Spaces. With InfoGrid, a mobile augmented realityapplication, users can experience the previously generated 3D
objects placed and aligned into real world scenes. All systems and applications of Ambient Learning Spaces interconnect through the NEMO-Framework (Network Environment for Multimedia Objects). This technical platform features contextualized access and retrieval of media.
... We implemented a special functionality for the generation of 3D objects from 2D photographic footage [18]. This has especially been used to generate 3D models for augmented or virtual reality applications. ...
... The NEMO 3D Object Converter can be used to automatically create 3D models out of a series of photographs taken of a physical object [18]. The user can upload photos into the object converter, which then automatically processes all files using photometry methods. ...
The contribution presents a system concept for digital teaching and learning environments, which includes current media concepts and enables a pedagogy that is only possible through the use of new
digital interactive media. The concepts for this learning environment have been based on contemporary pedagogical theories in awareness that learning processes are going to be less dependent on teaching
technologies and content and more on active processes of learning itself. A platform and environment called Ambient Learning Spaces (ALS) has been developed in a series of research projects, which
enables individuals as well as groups of learners to create, share, distribute, and reuse semantically annotated media in a social teaching and learning process. The Network Environment for Multimedia Objects (NEMO) forms the technological backend for different ALS frontend learning applications. It contains a semantically modelled, cloud-based repository for rich media applications and allows the creation of semantically annotated multimedia content to be used across these applications. These applications can be used on mobile or stationary interactive systems such as domes, large screens, tangibles, mobiles, and wearables. The media like text, image, video, or 3D can be reused in the different learning applications and will be adapted automatically for various interaction devices with
different media capabilities. Interaction methods depending on the available devices reach from direct manipulation interfaces on touch displays to augmented reality or 360° interactive virtual reality
applications. In the ALS-Portal, learners include and edit their own media and annotate them semiautomatically. Teachers select and manage the learning applications and prepare the teaching content
and logic for their pedagogical purposes. The learning applications support both stationary and mobile learning, all based on NEMO and its semantic media repository. Special user interfaces like tangible
media and large multitouch screens as well as mobile media and wearables enable body- and spacerelated learning in real life contexts inside and outside of school. Based on pedagogical approaches and
requirements, the contribution describes the structure and function of the platform and gives an overview of the different learning applications together with evaluation results. Besides the school context, ALS has been used in museums.
... Fig. 2). Details about these 3D modules and results about their usage can be found in ( [25], [26], [27]). ...
There is a worldwide discussion about digitization in school education. This discussion is mostly technology-centered, leaving a lack of solutions for daily schooling or focuses on isolated educational applications. With the Ambient Learning Spaces (ALS) platform, we developed a didactic infrastructure as an integrated environment that supports self-directed learning inside and outside school. The platform interlinks mobile and stationary learning applications. The artificial division between the classroom and the world outside vanishes through the pervasive cloud-based backend system NEMO (Network Environment for Multimedia Objects) connecting students' mobile applications with central semantic media storage. This paper emphasizes on the two mobile applications for task-oriented (MoLES) and discovery-oriented learning (InfoGrid). The InfoGrid System is a mobile application that supports the creation of augmented reality tours enabling the learners to study and discover physical places like urban spaces, architecture, museums, industrial settings, or natural habitats. Physical objects will be overlaid by media like text, images, audio, video, or 3D objects augmenting the physical/natural world with a digital informational layer by discovering and selecting visual (photographic) targets. Teachers or students can create content and tours by themselves through a modeling and editing application. The mobile application Mobile Learning Exploration System (MoLES) allows setting up tasks (challenges) to be solved by students inside and preferably outside school. Tasks can be created by teachers or students themselves depending on the pedagogical goals. The students use their own smartphones or tablets to solve these tasks by collecting information in form of different media like texts, photos, audio recordings or videos. This information will then be tagged and transmitted to NEMO. When the tasks have been solved, the students return to school and use the information collected for presentations in the classrooms or the school foyer. Both mobile applications make use of large multi-touch screens (InteractiveWall) or multi-touch tables (InteractiveTable) located in their school to discuss, sort and structure the media into media collections by a tool called MediaGallery for further use. Using their own digital devices in the sense of "Bring your own Device" (BYOD) in a physical environment keeps the students connected to their physical (real) world. They will solve tasks, discover interactive content in a project-and topic-oriented manner outside school and bring the results back to the classroom to be discussed in a structured way.
... For museum curators, we provide the web-based ALS-Portal to add and edit content for InfoGrid and other ALS applications. We furthermore developed tools accessible through the ALS-Portal that enable museum curators to create and edit 3D objects [7]. ...
Augmented Reality (AR) is a technology that can be used to provide personalized and contextualized information regarding physical objects in form of digital overlays. We use this technology in our research project Ambient Learning Spaces (ALS) to provide museum visitors with specific additional digital 3D information regarding the exhibits presented. With this technology, we enable museum curators to use a new form of transporting contextualized information without the need for additional physical space. However, the use of AR brings up new challenges for the creation and placement of digital contents into the museum space. In this context, we ran an anonymous survey on the use of AR in museums throughout Germany and studied responses of (N = 133) museum professionals. The results indicate that, although many museum professionals are interested in using AR technology, currently the integration is very costly and complex. This paper proposes a system we developed in a user-centered design process with a museum. This system provides an interface that helps museum professionals to cope with the complexity when placing and aligning digital 3D objects in their exhibition using mobile devices. Through this solution, visitors have the chance to experience the virtual objects spatially embedded in the exhibition by the curators themselves. In multiple user studies during the development phase we measured the usability of the interface. The findings show that the system provides a high degree of usability and can be applied effectively by museum professionals.
... For museum curators we provide the web-based ALS-Portal to add and edit content for these ALS applications. We furthermore developed tools accessible through the ALS-Portal that enable museum curators to create and edit 3D objects (Bouck-Standen, Ohlei, Daibert, Winkler, & Herczeg, 2017). Providing these applications, our research focuses on self-directed learning with interactive media (Thomas Winkler, Bouck-standen, Ide, Ohlei, & Herczeg, 2016). ...
In this paper, we present the concept and prototype of the Augmented Reality (AR) app InfoGrid for mobile devices, which enables museum visitors to experience digital media as an overlay for physical exhibits. With InfoGrid, it is possible to view 3D objects, videos, and animations, or listen to audio recordings prepared by museum professionals. InfoGrid interfaces with our web-based framework, the Network Environment for Multimedia Objects (NEMO), which stores and handles all data in a cloud-based semantic database. In this contribution, we present the usability field evaluation of InfoGrid in a nature museum, using the System Usability Scale (SUS) questionnaire and additional observations.
... Das Bildmaterial stammt aus dem jeweils eigenen Bestand der Museen. Das 3D-Objekt der Skulptur "7 Vögel" (siehe Abbildung 1), welches in die GGH-App eingebunden ist, wurde mit dem NEMO 3D Object-Converter [21] erzeugt. N3DOC ist ein Modul von NEMO, in das mehrere Fotos eines physischen Objektes über das ALS-Portal, der zentralen Eingabemaske für das NEMO-Framework, hochgeladen werden können. ...
Museumskuratoren stehen vor der Herausforderung, neue Konzepte für die inhaltliche Präsentation ihrer Ausstellung unter Verwendung aktueller Informations-und Kommunikationstechnologien zu entwickeln. Im DFG-Forschungsprojekt "Ambient Learning Spaces (ALS)" wird diese Fragestellung aufgegriffen und das System InfoGrid 4Museum entwickelt und erprobt. InfoGrid 4Museum ermöglicht eine Darstellung von Texten, Bildern, Videos und 3D-Objekten und die Wiedergabe von Audioaufnahmen auf den Smartphones und Tablets von Museumsbesuchern. Zur Darstellung der Informationen wird dabei die Augmented-Reality-Technologie genutzt. Die Daten, die in InfoGrid 4Museum dargestellt werden, liefert das webbasierte Framework Network Environment for Multimedia Objects (NEMO). Durch die Anbindung an NEMO können in InfoGrid 4Museum orts-, kontext-und personenbezogene Daten dargestellt werden. Zwei Lübecker Museen, das Museum für Natur und Umwelt sowie das Günter Grass-Haus wirken als Partner im ALS-Projekt intensiv bei der Ausarbeitung und Bereitstellung von Inhalten und der Erprobung des Systems mit. Für InfoGrid 4Museum wurden zwei Szenarien entwickelt, welche in der Lübecker Museumsnacht 2017 erprobt wurden.
This paper focuses on the design, implementation and evaluation of the InteractiveWall 3.1 (IW), which supports formal and non-formal learning in primary and secondary schools. An installation of an IW usually consists of multiple large multi-touch screens. With the third generation we developed based on latest technology, new simplified ways of delivering and interacting with media have been achieved. This was made possible by advances in the NEMO framework, which powers the IW. We designed a web-based framework called NEMO, an information repository that stores data and most importantly media enriched by semantic annotations. Annotations are collected from the users or generated automatically by algorithms e.g. through photo and video recognition. Media and annotations are uploaded into NEMO via our so-called Ambient Learning Spaces Portal (ALS-Portal), but also by using our Mobile Learning Exploration System (MoLES) running on smartphones. Apart from the standard applications Announcements, MediaGallery, TimeLine, and HyperVid, the IW also features non-competitive learning games for playing together. These games can be created and extended by teachers in schools with the help of the ALS-Portal. For our research, we are able to track each interaction though NEMO. Therefore, this paper presents data gathered in during typical everyday school life. In this paper, we present a practical example related to learning English as a second language. The interpretation of the first recorded usage data and the first statements by learners in context of the IW 3.1 and the ALS-Portal give an initial impression of the many planned qualitative and quantitative evaluations in the near future.
This paper describes and discusses work in progress. With Tangicons and SpelLit we have developed tangible learning games using physical Sifteo Cubes. From this idea we evolved to a web-based platform running the games on mini-tablets. The new approach we introduce in this contribution focuses on Electronic Learning Blocks (ELBlocks), which integrate with our semantic repository, the Network Environment for Multimedia Objects (NEMO) hence interconnecting with all applications and devices of the Ambient Learning Spaces (ALS) research platform.
Semantic technologies have been studied and used in different areas of computer science. E-learning has been one of this, but the most frequent use of semantic technologies in this discipline has been in the extraction and indexing of contents, like forums, blogs, learning objects etc. The research presented in this paper aims at taking advantage of semantic technologies in a different way respect to the mainstream use that has been done in the past. This new approach refers to the use of semantic technologies in the management of the persistence layer of Learning Management Systems (LMS), i.e., where all the contents are stored. Our research follows the idea of using semantics technologies as a support, if not an entire replacement, of the backend and persistence mechanisms of LMSs. As a testbed, we will present the design and early results of the application of this approach to the persistence layer of a Virtual Communities System, where these technologies will enriched the platform to address two fundamental issues: a) entities disambiguation and identification inside the persisted objects b) adding new features to the platform without refactoring it.
Education provides a plethora of tools that can be used (alone or combined) for achieving better results. One of the most recent technological advances that can be used as an educational tool is Augmented Reality, a technology that can combine virtual and physical objects in order to enhance the real world. However, little is known about this technology and its possible applications in primary and secondary education. This paper consists a literature review focused on AR and its current and future incorporation in modern education via various context aware technologies (e.g. tablets, smartphones) which can provide opportunities for more interactive and joyful educational experiences. Also, it is described the possibility of implementing AR in Open Course Project situations, such as the one which is available at the Eastern Macedonia and Thrace Institute of Technology. Its purpose is to inform " creators " and stimulate " users " so that the benefits of this promising technology may be diffused throughout the educational process.
This paper presents a knowledge representation of the final project course (FPC) in the e-Learning domain using modified methontology. The built system is proposed to overcome several student issues related to the final research project (e.g., project idea generation, proposal writing,
report analysis, engineering activity, and schedule management) in Indonesia engineering education. Moreover, differentiating from the current available e-Learning systems, it can provide full data release of its semantics. As a result, the ontology of the created semantic e-Learning has sufficient
information content based on various measurements (i.e., relationship, attribute, and inheritance richnesses).
The Internet of Things is a paradigm where everyday objects can be equipped with identifying, sensing, networking and processing capabilities that will allow them to communicate with one another and with other devices and services over the Internet to accomplish some objective. Ultimately, IoT devices will be ubiquitous, context-aware and will enable ambient intelligence. This article reports on the current state of research on the Internet of Things by examining the literature, identifying current trends, describing challenges that threaten IoT diffusion, presenting open research questions and future directions and compiling a comprehensive reference list to assist researchers.
This chapter addresses body-and space-related learning of children and adolescents, supported by a wide range of media platforms. These media constitute ambient learning spaces, in which learners use a variety of annotatable multimedia objects (e.g. image, video, sound, 3D models). These multimedia objects can be used for e.g. mobile learning and for learning with tangible interfaces in different environments. Our research started more than a decade ago and relies on a pedagogical background known as critical constructivism. The development of our body-and space-related media is based on innovative scenarios and was done within a participative design process with future users. This media can be assigned to different layers of our Model of Media Layers. Technically, this is made possible by the Network Environment for Multimedia Objects (NEMO), which is currently under development. It offers contextualized, personalized, semantically rich, device-specific access to multimedia objects. The research on the innovative learning environments is founded on Design-Based Research. According to this approach theory and real-world usage are conjoined and of equal importance. Thus theory and applications evolve simultaneously during the design process.
The term Linked Data refers to a set of best practices for publishing and connecting structured data on the Web. These best practices have been adopted by an increasing number of data providers over the last three years, leading to the creation of a global data space containing billions of assertions-the Web of Data. In this article we present the concept and technical principles of Linked Data, and situate these within the broader context of related technological developments. We describe progress to date in publishing Linked Data on the Web, review applications that have been developed to exploit the Web of Data, and map out a research agenda for the Linked Data community as it moves forward.
In a collaborative e-learning content management environment, the heterogeneity of data in different learning management systems presents many difficulties for data sharing; some of these difficulties are how to integrate data, produce results for user queries, and find the correct data from heterogeneous learning management systems. Over the last few years, numerous e-learning system architectures have been proposed; however, issues related to sharing and integrating data from different e-learning systems have been given less attention. Considering this need, this paper presents solutions for semantic data interoperability, distributed metadata management, and an agent-based query processing approach for supporting the exchange of learning content from different e-learning systems. The paper presents an empirical evaluation of the user acceptability of the proposed solutions to find qualitative measures of the users' acceptability and satisfaction; our proposed solutions resulted in high user satisfaction.