No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Modular augmented reality platform for smart operator in production environment
... (Fig. 3) highlights the groups of AR-based object recognition motivations. workers Since the declaration of the 4th industrial revolution, various manufacturing businesses have been motivated to incorporate AR with their operation units [27] facilitate individuals' jobs [16,40,47] including assisting the worker in complicated assembly tasks [15,31], as well as in operations, such as maintenance, repair, and product inspection [6,8,11,14,29,32,34,35,48]. Decreasing costs and time consumed in the process of training workers is one of the motivations of the study [13]. ...
... Usability, computational power, and the weight of device issues were major concerns in devices used in AR. Various issues have been considered main concerns for smart glasses like HoloLens, such as limited processing power and short battery life [40]. With the fast growth in AR technology, the challenge to develop or expand AR support devices has also increased. ...
... Additional research is needed to validate the proposed method and investigate additional AR techniques for better use [33]. Additional research is also recommended to decrease the cost of implementing these systems and make them more affordable [40]. ...
Augmented reality (AR) and object recognition are two relatively new technologies that can be employed to improve the possibility of better human perception and
understanding of the surroundings in the real world. The potential benefits of the
integration of these two technologies can be found in many areas. This study
synthesizes selected papers from 2017 to 2021 to provide a thorough overview of
existing AR-based object recognition systems. Several selections and scanning processes were employed using the inclusion criteria on all 2020 papers acquired. However, only 48 papers met the criteria. The study discusses and highlights the challenges, motivations, and recommendations of using the combinations of these technologies. In addition, it provides a classification of the tools and hardware mentioned in the selected studies. Finally, a summarization of the general characteristics of systems and applications developed and implemented in the selected studies. This study aims to enrich the understanding of this type of AR and, hopefully, inspire researchers to assist in the development and growth of AR.
... Meier et al. [78] integrated laser and HoloLens during manual mounting tasks, supporting assembly planning, monitoring, and operator training. Um et al. [79] proposed a platform to combine wearable devices and edge equipment in the factory, aiming to improve the computational and interoperable ability of AR and other wearable devices to support tasks such as assembly. ...
... Data transfer between the smart glasses and the edge server [79] contributes to achieving bidirectional empathy [89] or mutual cognition between humans and cobots. Multi-robot system -Cai et al. [87] proposed a layout configuration method for a multi-robot AM system, using AR-marker, screen augmentation, and voice guidance to assist users in deploying the pre-determined layout of robots. ...
... A cognitive twin can tackle more complex and unpredictable situations dynamically with enhanced computational capabilities. AR can play a significant role in promoting a cognitive twin to some extent because it can serve as a wearable end computational unit in the edge-cloud architecture [79]. Moreover, HoloLens 2, one of the most commonly used AR devices, has a certain computing power compared to other wearable devices (such as sensors), which is 1T FLOP. ...
The combination of Augmented Reality (AR) and Digital Twin (DT) has begun to show its potential nowadays, leading to a growing research interest in both academia and industry. Especially under the currenthuman-centric trend, AR embraces the potential to integrate operators into the new generation of Human-Cyber–Physical System (HCPS), in which DT is a pillar component. Some review articles have focused on thistopic and discussed the benefits of combining AR and DT, but all of them are limited to a specific domain. To fill the gap, this research conducts a state-of-the-art survey (till 17-July-2022) from the AR-assisted DTperspective across different sectors of the industrial field, covering a total of 118 selected publications. Firstly, application scenarios and functions of AR-assisted DT are summarized by following the engineering lifecycle, among which service design, production process, and Human–Machine Interaction (HMI) are hot topics. Then,improvements specifically brought by AR are analyzed according to three dimensions, namely virtual twin,hybrid twin, and cognitive twin, respectively. Finally, challenges and future perspectives of AR-assisted DT areproposed, including promoting product design, robotic-related works, cyber–physical interaction, and human ergonomics.
... The concept of Industry 4.0 originated in Germany and focused on the digitalization of manufacturing processes (Çinar et al., 2021). The technologies involved enable the interconnection of different sectors, actors, systems, and artifacts, creating intelligent manufacturing systems, and changing the focus of mass production into mass customization (Um et al., 2018;Gualtieri et al., 2020;Ramadan et al., 2020;Agnusdei et al., 2021;El-Haouzi et al., 2021). This concept encompasses a series of technological components which is briefly synthesized and listed in Table 3. ...
... Smart wearable solutions Devices which monitor in real-time a range of Operator 4.0 vital signs and the surrounding workplace environment, through sensors and ambient intelligence, and can thus help to make sure operators are healthy and safe (Gazzaneo et al., 2020;Romero et al., 2020). They are also enhancers of the operator's sensorial and interaction capabilities (Um et al., 2018;Gazzaneo et al., 2020). ...
... al., 2016a;Gregor et al., 2017;Iordache, 2017;Lee et al., 2017;Santos et al., 2017;Badri et al., 2018;Um et al., 2018;Gray-Hawkins et al., 2019;Hamdi et al., 2019;Kumar and Kumar, 2019;Madonna et al., 2019;Ahmad et al., 2020;Blštáková et al., 2020;Brozzi et al., 2020;Cimini et al., 2020;Gajšek et al., 2020;Gazzaneo et al., 2020;Longo et al., 2020;Miśkiewicz and Wolniak, 2020;Ramadan et al., 2020;Berrah et al., 2021;Çinar et al., 2021; El-Haouzi et al., 2021;Rupp et al., 2021; Stawiarska et al.al., 2016b;Kolbeinsson et al., 2017;Langfinger et al., 2017;Thomay et al., 2018;Ivaschenko et al., 2019;Minnetti et al., 2019;Pavel et al., 2019;Baldissone et al., 2020;Digiesi et al., 2020;Fruggiero et al., 2020;Gualtieri et al., 2020;Kadir and Broberg, 2020;Santo et al., 2020;Shi et al., 2020;Van Acker et al., 2020; Agnusdei et al.al., 2017;Marrella and Mecella, 2017;Schloegl et al., 2017;Badri et al., 2018;Mark et al., 2019;Thun et al., 2019;Udayangani et al., 2019;Ahmad et al., 2020;Blštáková et al., 2020; Broday, 2020;Cimini et al., 2020;Gajšek et al., 2020;Gazzaneo et al., 2020;Gualtieri et al., 2020;Hoyer et al., 2020;Kaasinen et al., 2020; Broberg, 2020, 2021;Longo et al., 2020;Saabye et al., 2020;Sony and Naik, 2020;Chistyakova et al., 2021;Di Carlo et al., 2021;Golsch and Seegers, 2021;Ivaldi et al., 2021;Paliga and Pollak, 2021;Patriarca et al., 2021;Rangraz and Pareto, 2021;Reiman et al., 2021; Shi et al.al., 2018;Adriaensen et al., 2019;Gunasekaran, 2019;Blštáková et al., 2020;Fruggiero et al., 2020;Hoedt et al., 2020;Longo et al., 2020;Saabye et al., 2020;Serras et al., 2020;Rangraz and Pareto, 2021;Weiss et al., 2021. ...
Industry 4.0 (I4.0) promises to transform jobs and working conditions through the implementation of unprecedented human-machine interaction modes. As the operator working in these new settings, known as the Operator 4.0, is a relatively recent concept, and although technological developments are expected to support workers and require higher labor skills, the risks and health impacts resulting from these changes remain underexplored. This systematic review aims to (i) systematize literature findings on how workers are perceived and participate in I4.0 work systems; (ii) identify the main technological changes driven by I4.0; and (iii) instigate discussion regarding the impacts these changes may have for workers and the sustainability of work systems. Following a systematic review approach using the PRISMA protocol, the articles were organized into two main analysis axes: the technical changes brought about by I4.0, and the representation of the human worker within these new work settings. The findings reveal that a techno-centered approach still seems to be dominant in guiding the implementation of I4.0 models; secondly, as a consequence, the social dimensions of work tend to remain as residual issues, overshadowed by the promises related with technology (e.g., productivity, efficiency); finally, the representation of the Operator 4.0 remains blurry, as he/she is perceived as gender neutral, skillful, and perfectly fit for work, assuring the functioning (and compensating for the limits) of these systems. While I4.0 promises safer and more productive workplaces, issues related to employment conditions, emerging risks and health impacts become more prominent when analyzed from an activity-centered perspective. In terms of future research, a more heuristic analysis could be achieved through a participatory and work-centered approach and following a gender perspective. This way, visibility could be conferred to another side of I4.0, thus guaranteeing conditions for the sustainable development of these work situations.
... Um et al. [34] used resources from the equipment camera to collect environment data, process this information, and present it conforming to the operator's vision. ...
... Um et al. [34] developed a platform operating on an industrial modulated system. The platform had an MR headset (Hololens) and an edge computing platform. ...
... Finally, Segura et al. and Um et al. [30,34] discussed the need for performance tests to identify limitations and allows the search for better ways to execute high computational software as DTs. ...
Digital Twins (DTs) in Industry 4.0 are complex cyber-physical systems that can provide an interface that allows humans and machines to combine their better skills to improve industrial activity. However, in a limited and risky industrial scenario, DTs challenged working with high-demand computer power, transfer rate, and restricted devices. Thus, this work address DT challenges into asynchronous and synchronous requirements. First, this work reviewed the literature best approaches, hardware, and software while implementing a DT. Then, it proposes a high-level architecture over a Petri Net (PN) model to address the async issue. After that, to perform a sync test, it implements a DT prototype using the literature equipment and respecting the architecture proposed. The results show that the case study respects the requirements for safe operation regarding timing and modeling constraints.
... Fig. 3 shows the distribution of publication years of the found studies. The first relevant paper about AI-DT included in our analysis was published in 2018 [24], with the subsequent years showing an increase in popularity. In 2022, 88 relevant papers were published, more than double compared to 2021. ...
... Barricelli et al. [31] propose a fitness digital twin, where a fitness coach, acting as a rule editor, interacts with the DT environment to support the decision-making process. Um et al. [24] utilize smart glasses, integrating the user with a virtual reality environment, serving as the DT. ...
Artificial intelligence and digital twins have become more popular in recent years and have seen usage across different application domains for various scenarios. This study reviews the literature at
the intersection of the two fields, where digital twins integrate an artificial intelligence component. We follow a systematic literature review approach, analyzing a total of 149 related studies. In the
assessed literature, a variety of problems are approached with an artificial intelligence-integrated digital twin, demonstrating its applicability across different fields. Our findings indicate that there
is a lack of in-depth modeling approaches regarding the digital twin, while many articles focus on the implementation and testing of the artificial intelligence component. The majority of publications do not demonstrate a virtual-to-physical connection between the digital twin and the real-world system. Further, only a small portion of studies base their digital twin on real-time data from a physical system, implementing a physical-to-virtual connection.
Keywords: Artificial Intelligence, Digital Twin, Machine Learning, Literature Review, Business Intelligence, Data Mining
... Based on our own experience in AR projects and other studies (Rosales et al., 2021;Anshari and Almunawar, 2021;Chidambaram et al., 2021;Um, Popper and Ruskowski, 2018), authors consider following logical requirements as important while choosing an AR application development environment for SMEs: ...
... A search for an appropriate solution for a novel unit may thus be a task from the scratch; however, not too many researchers try to describe this problem in more details. A modular approach, however, may be helpful here (Um et al., 2018); BTU is also underway in developing its own modular solution within MoDeARA project, or Modular-Designed AR Application. ...
Albeit augmented reality (AR) technologies first have been discovered in the third quarter of the 20th century, their widespread use began just two decades ago. Existing paper trails show that AR has a wide range of industrial application: it simplifies human-machine communication, improves human-machine interfaces (HMI) for quick information exchange in training (including feedback to study the workflow), correction of errors, machine maintenance, assembly assistance etc. However, broader industrial acceptance of AR, prior to all by small and medium-sized enterprises (SMEs), recently faced considerable problems and the expansion of AR solutions does not match the high potential it has demonstrated. That results in a limited practical use, mainly for demonstration and advertising purposes. This short review is to present the state of the art of the industry, challenges that SMEs face in adopting AR technologies, and several practical examples of a (commercial) adoption of AR. Some prospects for further development of AR and its ongoing integration into industry are briefly discussed in the summary.
... Research and industry leverage Digital Twins to monitor and control (cyber-physical) systems in various domains, including autonomous driving , biology (Joordens and Jamshidi, 2018), medicine (Lauzeral et al., 2019), smart manufacturing (Um et al., 2018), and many more. They promise tremendous potential to reduce cost and time and improve our understanding of the represented systems. ...
... • Artificial Intelligence Software, such as Apache MXNet (Uzun et al., 2019), the IBM Watson software development kit (Dingli and Haddod, 2019), or TensorFlow (Um et al., 2018). ...
Digital Twins are currently investigated as the technological backbone for providing an enhanced understanding and management of existing systems as well as for designing new systems in various domains, e.g., ranging from single manufacturing components such as sensors to large-scale systems such as smart cities. Given the diverse application domains of Digital Twins, it is not surprising that the characterization of the term Digital Twin, as well as the needs for developing and operating Digital Twins are multi-faceted. Providing a better understanding what the commonalities and differences of Digital Twins in different contexts are, may allow to build reusable support for developing, running, and managing Digital Twins by providing dedicated concepts, techniques, and tool support. In this paper, we aim to uncover the nature of Digital Twins based on a systematic mapping study which is not limited to a particular application domain or technological space. We systematically retrieved a set of 1471 unique publications of which 356 were selected for further investigation. In particular, we analyzed the types of research and contributions made for Digital Twins, the expected properties Digital Twins have to fulfill, how Digital Twins are realized and operated, as well as how Digital Twins are finally evaluated. Based on this analysis, we also contribute a novel feature model for Digital Twins from a software engineering perspective as well as several observations to further guide future software engineering research in this area.
... Edge-based inference for ML at Facebook [281] Image classiication DNN Supporting vehicle-to-edge for vehicle AR [295] Object detection Deep CNN (YOLO) AR platform for operators in production environments [253] Object detection SSD Spatial AR with single IR camera [87] 3D pose estimation, image classiication Hough Forests, Random Ferns Federated learning for low-latency object detection and classiication [50] Object classiication modelling Federated learning ...
... Examples of server-based deployment include the usage of CNNs for object detection to support AR tracking [7], Mask R-CNN for object detection and instance segmentation to support smart task assistance for HoloLens-deployed AR [185], and CNN for object recognition in improving retail AR shopping experiences [53]. Additionally, some works explicitly state their usage of edge servers, citing the processing acceleration gained by the object detection and recognition tasks when a GPU is used to execute the functions, for example, using YOLO to accomplish object detection to support vehicle-to-edge AR [295], and SSD for supporting an edge-based AR platform for operators in production environments [253]. Comparatively, several research works deploy object detection and recognition in an enclosed client system, i.e., without needing to oload computation components to an external server or device. ...
Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and perform seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences using MAR devices to provide universal access to digital content. Over the past 20 years, several MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discuss the latest studies on MAR through a top-down approach: (1) MAR applications; (2) MAR visualisation techniques adaptive to user mobility and contexts; (3) systematic evaluation of MAR frameworks, including supported platforms and corresponding features such as tracking, feature extraction, and sensing capabilities; and (4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields and the current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.
... Edge-based inference for ML at Facebook [281] Image classiication DNN Supporting vehicle-to-edge for vehicle AR [295] Object detection Deep CNN (YOLO) AR platform for operators in production environments [253] Object detection SSD Spatial AR with single IR camera [87] 3D pose estimation, image classiication Hough Forests, Random Ferns Federated learning for low-latency object detection and classiication [50] Object classiication modelling Federated learning ...
... Examples of server-based deployment include the usage of CNNs for object detection to support AR tracking [7], Mask R-CNN for object detection and instance segmentation to support smart task assistance for HoloLens-deployed AR [185], and CNN for object recognition in improving retail AR shopping experiences [53]. Additionally, some works explicitly state their usage of edge servers, citing the processing acceleration gained by the object detection and recognition tasks when a GPU is used to execute the functions, for example, using YOLO to accomplish object detection to support vehicle-to-edge AR [295], and SSD for supporting an edge-based AR platform for operators in production environments [253]. Comparatively, several research works deploy object detection and recognition in an enclosed client system, i.e., without needing to oload computation components to an external server or device. ...
Mobile Augmented Reality (MAR) integrates computer-generated virtual objects with physical environments for mobile devices. MAR systems enable users to interact with MAR devices, such as smartphones and head-worn wearables, and performs seamless transitions from the physical world to a mixed world with digital entities. These MAR systems support user experiences by using MAR devices to provide universal accessibility to digital contents. Over the past 20 years, a number of MAR systems have been developed, however, the studies and design of MAR frameworks have not yet been systematically reviewed from the perspective of user-centric design. This article presents the first effort of surveying existing MAR frameworks (count: 37) and further discusses the latest studies on MAR through a top-down approach: 1) MAR applications; 2) MAR visualisation techniques adaptive to user mobility and contexts; 3) systematic evaluation of MAR frameworks including supported platforms and corresponding features such as tracking, feature extraction plus sensing capabilities; and 4) underlying machine learning approaches supporting intelligent operations within MAR systems. Finally, we summarise the development of emerging research fields, current state-of-the-art, and discuss the important open challenges and possible theoretical and technical directions. This survey aims to benefit both researchers and MAR system developers alike.
... Um et. al also tested a solution of transmitting captured images from smart glasses to a server for processing and re-transmitting the results, thus reducing the strain on the batteries [48]. The servers were placed in the architecture in the form of edge computing. ...
... This solution would also have the added advantage of being less dependent on specific smart glass interface designs. The results did not show an improvement in time with current wireless technology [48]. Szajna et al. also proposes a setup using edge computing but uses it to monitor the production line rather than visualization [49]. ...
This article aims to provide a better understanding of Augmented Reality Smart Glasses (ARSG) for assembly operators from two perspectives, namely, manufacturing engineering and technological maturity. A literature survey considers both these perspectives of ARSG. The article's contribution is an investigation of the current status as well as challenges for future development of ARSG regarding usage in the manufacturing industry in relation to the two perspectives. This survey thereby facilitate a better future integration of ARSG in manufacturing. Findings include that commercially available ARSG differ considerably in their hardware specifications. The Technological Readiness Level (TRL) of some of the components of ARSG is still low, with displays having a TRL of 7 and tracking a TRL of 5. A mapping of tracking technologies and their suitability for industrial ARSG was done and identified Bluetooth, micro-electro mechanical sensors (MEMS) and infrared sensors as potentially suitable technologies to improve tracking. Future work identified is to also explore the operator perspective of ARSG in manufacturing.
... Research and industry employ Digital Twins (DTs) [1], [2] to monitor and control Cyber-Physical Systems (CPSs) in various domains, including automotive [3], biology [4], medicine [5], manufacturing [6], and many more. They promise vast potential to reduce development costs and time, optimize operations, and improve our understanding of the represented systems. ...
... Today, there are cameras available for almost all tablet PCs and mobile phones. Head-worn devices having cameras and projectors are specially designed for the application such as an intelligent assistance system for the human [28]. In the research work called HMD-EgoPose [8], a single shot learning-based approach is implemented for hand and surgical instrument pose estimation using Microsoft HoloLens. ...
A great challenge in applying AI to specific problems in the industry is to select the proper method when multiple methods areavailable. In this paper, we intend to address this issue with various AI methods in the context of adaptive assembly assistancesystems. The paper is a synthesis that discusses and highlights advantages and disadvantages, applicability, and recommendationsfor several AI-based methods. For illustration, we present the methods applied to provide choices for the next assembly step in thecontext of a highly customizable and modular tablet used as a target product. To choose the algorithm that best suits the needs ofa specific working environment, we present an approach to consider several criteria weighted based on their importance based onthe needs and resources of the use case. The Hidden Markov Model fulfilled our criteria with the highest score and is the selectedprediction method for integration into the assembly assistance system.
... However, the player's translated words may be inconsistent with his/her facial expressions if heterogeneous types of applications are improperly coordinated and processed, consequently harming the player's QoE, especially considering real-time communications with the other players in the digital world. Considering DT-empowered intelligent manufacturing as another example, where step-by-step instructions with multimedia AR services are given for AR-assisted manual assembly [3]. Under this setting, the object recognition tasks may be asynchronous to the risk situation-aware tasks for multiple workers on the factory's pipeline if multiple heterogeneous AI services lack close collaboration. ...
Intelligent edge services should consider not only service provider’s resource constraints but also end user’s quality of experience (QoE) demands. In this regard, we study a digital twin service caching and request routing problem with consideration of both resource and fairness-awareness under the collaboration of multi-access edge computing (MEC) servers. We first formulate the problem as an integer nonlinear programming by maximizing the minimum task completion ratio when there are multiple heterogeneous digital twin services, subject to computation, storage, and heterogeneous task delay constraints. Then, we propose an online stochastic randomized rounding algorithm (ORA) and an efficient online greedy algorithm (OGA) to solve the formulated problem. Extensive simulation results demonstrate the effectiveness of the proposed ORA and OGA algorithms in terms of the minimum application completion ratio.
... However, these points only concern data glasses and mobile devices. Today, smart glasses do not have sufficient computing power (Chatzopoulos et al., 2017;Porcelli et al., 2013;Um et al., 2018). For decentralized computation of AR content, new forms of network technology such as 5G could solve the problem (Lv et al., 2022). ...
As part of the digital transformation towards Industry 4.0, the tasks of staff on the shop floor are changing. Despite increasing automation, complex assembly steps still have to be carried out by humans, especially when it comes to complex products rich in variants, whose assembly cannpt be fully automated for various reasons. Due to increasing individualization and the steadily growing complexity of products, providing the right information at the right time and in the right place is becoming more important. In this context, the visualization of information via novel technologies such as augmented reality plays a crucial role towards an efficient and error-free production process. This paper compiles existing challenges when using augmented reality as a visualization form for an assistance system. On the one hand, the challenges found originate from a systematic literature review and are organized according to predefined categories. On the other hand, these challenges are complemented and compared through findings gained from expert interviews, which are conducted with employees of two European commercial vehicle manufacturers in the field of production. The analysis of the two methods highlights the need for further research.
... These features can each be captured by a single numerical value and can be recorded, stored, and processed, for example, in the form of time series. Examples of features with a high degree of complexity are geometry, kinematics, or optical appearance (e.g., Jinfeng Um et al., 2018;Y. Zheng et al., 2018) which can be represented by 3D meshes, 3D-point clouds, or images. ...
For thousands of years, people have depicted the real world through geographic maps or portraits of people, landscapes, and machines for different reasons, like representing complex environments simply and understandably or depicting detailed aspects of reality. Due to the increasing digitalization and connectivity over the last years, it is possible to continuously collect more and more data describing the real world, including interactions between environments, objects, and people, and to gain more precise, comprehensive, and completely new insights. Activity trackers for the collection and analysis of health data, smart home systems for the monitoring and energy-efficient control of light, temperature, and household appliances, or the collection and analysis of production data in industrial manufacturing for the monitoring, prediction, and optimization of production processes are used. An emerging approach in this context for creating virtual data-based representations of real objects is the Digital Twin Concept, examined by the research presented in this thesis. Digital Twins are virtual models that mirror physical objects throughout their lifecycle, and real-time connections between the physical and virtual worlds allow Digital Twins to monitor and control physical objects from any location. Physical objects can be any living or non-living object, such as aircraft, manufacturing equipment, cars, buildings, wind turbines, food, or even people. Companies see great potential in Digital Twins, and more and more organizations are using Digital Twins or plan to do so. In addition to the interest of practitioners, Digital Twins are being intensively studied as research objects. However, the discussion on Digital Twins in the scientific literature shows that there is currently no consolidated understanding of Digital Twins or the Digital Twin Concept. Nevertheless, the interest in Digital Twins has grown strongly in business and research in recent years. Therefore, specific findings and solutions from research are becoming increasingly important to enable organizations to apply Digital Twins in meaningful and efficient ways. This thesis aims to contribute to the understanding and application of Digital Twins by addressing two research areas. In the context of the first research area, a rigorous multi-level literature review was conducted to contribute to the understanding of Digital Twins. Two studies were conducted in the second research area to contribute to the application of Digital Twins. The first research area contains a rigorous multi-level literature review consisting of three studies. First, a preliminary study was conducted to develop a search and selection strategy for scientific literature and, based on this, a review sample. In the second study, a review of reviews was conducted based on the identified literature reviews from the review sample. As a result, it could be shown that the number of literature reviews has increased significantly since 2018 and that the research is distributed across various outlets and disciplines, with a clear focus on the manufacturing domain. Furthermore, a content analysis was conducted that revealed four research areas containing different aspects of the research on Digital Twins, which are examined more closely. Since the literature reviews did not cover a large part of the publications in the review sample, a third study was conducted. In the third study of this research area, a scoping review, an analysis of the Digital Twin research community, and a citation analysis were conducted to analyze the primary studies of the review sample. The literature analysis showed that the research could be well structured according to seven different research topics, which are examined and characterized in detail. Two studies were conducted on applying Digital Twins in the second research area. In the first study, an analysis of Digital Twin application cases was conducted to provide a comprehensive and structured overview. Based on a set of application cases retrieved from previous scientific literature, the study identified six dimensions with a total of 24 characteristics to classify Digital Twin application cases. Furthermore, a comprehensive overview of selected Digital Twin application cases is presented. In the second study, an IT system based on the Digital Twin Concept was implemented to gain detailed insights into the design and implementation of Digital Twin systems in an enterprise setting. Therefore a Digital Twin introductory project of an industry partner was accompanied over two years according to scientific principles. Within this project, an IT system based on the Digital Twin Concept was implemented, and its validity and utility were demonstrated and evaluated. Five application cases were realized and applied to production settings for the demonstration and evaluation. The study presents a detailed description of the artifact's conceptual architecture and technical implementation. In summary, this thesis provides valuable insights into two specific research areas in the field of Digital Twins. It provides a comprehensive picture of the research landscape and presents detailed insights into the application of Digital Twins. Therefore this thesis provides a solid basis for further research on the possibilities and challenges of converging real and virtual worlds.
... First of all, user requirements analysis of AR applications is conducted in terms of usability [33], and implementation challenges [34]. Later, analysis is conducted based on the use cases of the BIM applications, and 5G-based BIM AR service analysis is performed. ...
Informed decision-making is crucial for construction site operators. Cyber-physical systems, including various technologies such as augmented reality and building automation tools, are gathering popularity within the infrastructure management sector. However, they are expensive and inaccessible to adopt for most small organizations. This paper describes a prototype of low-cost mobile augmented reality service for BIM and demonstrated its usability for pipe maintenance by supporting inspection, workflow management, data reduction, and augmented reality. User views about the decision support, integration and ease of use of the prototype were also collected from the organisations in the UK and South Korea. The results showed that integrating the augmented reality service with the building automation tool connected to the BIM server enhances decision-making for on-site operations by generating a closed loop. This paper also highlights the need for developing low-cost digital solutions to foster the digitalization of construction organisations with limited budgets.
... The creation of an application for a novel unit tends to become a task "from scratch" considerably fast. However, not many articles are dedicated to solving this problem now (Um et al. (2018)). ...
The Augmented reality (AR) technologies have been first discovered in the
third quarter of the twentieth century. However, the wider development of them has
taken place only in the last two decades. By now, the research has shown that AR
can be used in various areas of human activity. In industry, AR simplifies humanmachine
communication and improves human-machine interfaces (HMI) for fast and
feedback-provided retrieval of training/guidance information for operation pattern
study, error correction, machine maintenance, assembly assistance, etc. In spite of that, the broad practical implementation of AR in industry, including small
and medium-sized enterprises (SMEs), has faced considerable problems. As a result,
the following controversy emerged: the comprehensive study of AR is combined with
a rather narrow practical use primarily for advertising and demonstration tasks. This chapter attempts not only to overview the current state of AR in the industry, but
also demonstrate the current challenges the AR is facing, as well as to analyse their
respective causes and suggest solution ideas. It is also intended to assess the prospects
for further development of AR and its continued integration into the industry. For
this purpose, several examples of AR projects, their development, practical use and
upgrading (performed by the authors of this study as well) are presented.
... The creation of an application for a novel unit tends to become a task "from scratch" considerably fast. However, not many articles are dedicated to solving this problem now (Um et al., 2018). ...
Small and medium enterprises (SMEs) are said to struggle with several challenges when transforming parts of their value chain to smart production with the means of digitalization. [1] Since SMEs are the backbone of our economy it is important to understand those challenges to cope with them accordingly. [2] This book chapter provides an overview of SME-specific digitalization challenges including possibly challenging SME characteristics and necessary prerequisites from a literature review as well as deeper insights about singular companies from the region of Brandenburg in Germany. Those insights are derived from semi-structured interviews with SME managers. None of the identified obstacles is a showstopper for our interviewees but some of them are more severe than others. This chapter contributes to the existing literature by showing an overview of challenges for SMEs when transforming their value chain to smart production. Additionally, we provide insights with less abstract depictions about the process of SMEs coping with those challenges. This chapter is of interest for SME managers when defining the transformation process of their own SME under consideration of the digitalization challenges. The overview, as well as the insights, help researchers defining their approaches for finding solutions from their perspective on realizing smart factories in SMEs.
... The creation of an application for a novel unit tends to become a task "from scratch" considerably fast. However, not many articles are dedicated to solving this problem now (Um et al., 2018). ...
Digital Twin recently became one of the key enablers of Smart Manufacturing. Although several architectures are currently available to support digital twin design, implementation, use, and assessment, most of such procedures are not adapted to the reality of small and medium enterprises. In this chapter, we provide a solid foundation to support SMEs toward adopting digital twins while exemplifying the implementation procedure with two use cases in manufacturing industries.KeywordsDigital twinSMESmart manufacturingDigital manufacturingSimulation
... [Yao et al. 2018]), object recognition (see, e.g. [Um et al. 2018]), or supporting decision-making concerning system design (see, e.g. ). However, for these models to be of the highest level, external factors such as supply chains and internal factors should be considered -employee skills and device failures. ...
... Research and industry leverage digital twins (DTs) to monitor and control (cyber-physical) systems in various domains, including autonomous driving [9], biology [16], medicine [21], smart manufacturing [29], and many more. They promise the tremendous potential to reduce cost and time and to improve our understanding of the represented systems. ...
Research and industry leverage digital twins to monitor and control (cyber-physical) systems in various domains. For their efficient engineering, these twins need to become Systems-of-Systems (SoS), in which digital twins of smaller systems (e.g., a production machine) become parts of digital twins of larger systems (e.g., a factory). Yet, research on digital twins as SoS largely ignores reusing digital twins in SoS. Based on our experience in engineering digital twins with experts from various domains related to production systems engineering, we present insights on the challenges of composing and integrating that need to be addressed for efficient engineering of digital twins as SoS. These insights may guide future research on engineering digital twins as well as practitioners considering the challenges in building and composing digital twin systems-of-systems.
... Digital Twins provide means to monitor and control Cyber-Physical Systems (CPSs) in various domains, such as smart manufacturing [29], biology [19], or autonomous driving [8]. They serve different purposes, such as analysis [23], control [31], or behavior prediction [21]. ...
Digital Twins in smart manufacturing must be highly adaptable for different challenges, environments, and system states. In practice, there is a need for enabling the configuration of Digital Twins by domain experts. Low-code approaches seem to be a meaningful solution for configuration purposes but often lack extension options. We propose a model-driven low-code approach for the configuration and reconfiguration of Digital Twins using language plugins. This approach uses model-driven software engineering and software language engineering methods to derive a configurable digital twin implementation. Moreover, we discuss some remaining challenges such as interoperability, language modularity, evolution, integration of assistive services, collaborative development, and web-based debugging.
... In order to improve the yield of assembly operations, providing support to human workers is necessary. Augmented Reality (AR) could be used, reducing the number of engineering/production management resources needed to provide assembly operators with cognitive support to perform their tasks [45,46]; as well as cognitive/handling skills transfer systems [47], self-adapting automatic quality control [48] or cognitive automation strategies [49]. Automation needs to ensure human safety, which led to research on Human-Robot Collaboration (HRC) plan recognition and trajectory prediction [50], and the concept of "safety bubble" [51]. ...
In a demand context of mass customization, shifting towards the mass personalization of products, assembly operations face the trade-off between highly productive automated systems and flexible manual operators. Novel digital technologies—conceptualized as Industry 4.0—suggest the possibility of simultaneously achieving superior productivity and flexibility. This article aims to address how Industry 4.0 technologies could improve the productivity, flexibility and quality of assembly operations. A systematic literature review was carried out, including 234 peer-reviewed articles from 2010–2020. As a result, the analysis was structured addressing four sets of research questions regarding (1) assembly for mass customization; (2) Industry 4.0 and performance evaluation; (3) Lean production as a starting point for smart factories, and (4) the implications of Industry 4.0 for people in assembly operations. It was found that mass customization brings great complexity that needs to be addressed at different levels from a holistic point of view; that Industry 4.0 offers powerful tools to achieve superior productivity and flexibility in assembly; that Lean is a great starting point for implementing such changes; and that people need to be considered central to Assembly 4.0. Developing methodologies for implementing Industry 4.0 to achieve specific business goals remains an open research topic.
... Finally, Volkan et al. [6] proposed an edge computing system as a solution of implementing Cyber-Physical system. Um et al. [7] also shows the usage of auxiliary devices for manual operators supported by object detection service running on the edge device. These studies will be the technological basis to create an CPPS that can utilize AI services on the shop-floor. ...
Flexibility in mass-customized manufacturing can be supported significantly by the introduction of Cyber-Physical Production System and the connection of production modules to AI (artificial intelligence) Cloud services. Even though there exist standardized protocols from device to IT system, there are still challenges for the synchronization between cyber-model and physical object, and the application of decision making in the cyber-model. Although high performance machine learning services make the Cloud a preferred computation node, possible unstable connection with manufacturing resources enforce new service distribution approaches in the network. This paper proposes an Edge Computing architecture which is the mediator between machines, by providing local Cloud services with fast response time and preprocessing resources for a vast amount of data. As an illustrative example the selected Edge service pre-processes data form an augmented reality device in order to communicate with the cyber-model in real time. The Edge platform controls the computing resources and prioritizes all processes of Edge Services for a dynamic update of production lines and human-machine-interaction.
This survey paper comprehensively reviews Digital Twin (DT) technology, a virtual representation of a physical object or system, pivotal in Smart Cities for enhanced urban management. It explores DT's integration with Machine Learning for predictive analysis, IoT for real-time data, and its significant role in Smart City development. Addressing the gap in existing literature, this survey analyzes over 4,220 articles from the Web of Science, focusing on unique aspects like datasets, platforms, and performance metrics. Unlike other studies in the field, this research paper distinguishes itself through its comprehensive and bibliometric approach, analyzing over 4,220 articles and focusing on unique aspects like datasets, platforms, and performance metrics. This approach offers an unparalleled depth of analysis, enhancing the understanding of Digital Twin technology in Smart City development and setting a new benchmark in scholarly research in this domain. The study systematically identifies emerging trends and thematic topics, utilizing tools like VOSviewer for data visualization. Key findings include publication trends, prolific authors, and thematic clusters in research. The paper highlights the importance of DT in various urban applications, discusses challenges and limitations, and presents case studies showcasing successful implementations. Distinguishing from prior studies, it offers detailed insights into emerging trends, future research directions, and the evolving role of policy and governance in DT development, thereby making a substantial contribution to the field.
Modern technologies and recently developed digital solutions make their way into all aspects of lives of individuals and businesses, and manufacturing industry is no exception. In the era of digital revolution of industry, manufacturing processes can benefit from digitalization technologies immensely. Digital twin (DT) is a technology concept that aims to create a digital mirror of a physical system with a constant data flow between two components. This idea can be used for monitoring and optimization of the present system as well as forecasting and estimating future states of it. There have been theoretical and practical studies conducted on DT in manufacturing area. This systematic literature review (SLR) aims to summarize the current state of literature and shine a light on open areas for future research. Using a rigorous SLR method, 247 relevant studies from 2015 to 2020 are examined to answer a set of research questions. The current state of DT in manufacturing literature is analyzed and explained with an emphasis on where the future studies may go in this area.
As factories move towards Industry 4.0, the replacement of control communications based on Ethernet and other wired technologies with wireless becomes an imperative to fulfill the envisioned flexibility and easy reconfigurability of the production facilities. While wired connections are often unwanted for their limited flexibility and high maintenance costs, common wireless technologies in manufacturing such as Bluetooth, Wi-Fi, ZigBee, or 4G cannot fulfil the requirements of timeliness, reliability, data rates, scalability and availability of cyber-physical production systems. Today, the fifth-generation (5G) broadband cellular network technology holds the promise for an enhanced Quality of Service (QoS) and Quality of Experience (QoE) that unlock a huge amount of value and opportunity. However, at present, application of 5G wireless communication technology in the manufacturing industry is still at its infancy. In this paper, 5G is presented as a game changer for human-machine and human-robot interaction in three manufacturing use cases (i.e. uRLLC-based human-robot collaboration, eMBB-based AR-assisted operations and mMTC-based interaction with the digital twin). A QoS/QoE model is presented to drive the implementation of 5G-aided solutions for the Operator 4.0, thus maximizing the network quality and acceptance rate of digital technologies in the shop floor (that have never been widely accepted and used in practice by the industrial workforce). This paper explores how a 5G could finally provide the necessary network infrastructure for the human-machine symbiosis in the factory of the future.
The purpose of this work is to analyze trends in the use of augmented reality technologies in Russian organizations. The authors consider the problem of implementing AR technologies in the economic processes of enterprises in the conditions of Russian economic, legislative, social and technical barriers. The relevance of this problem is confirmed by the high demand of Russian enterprises for AR projects, while the Russian consumer market for augmented reality technologies is lagging behind the world market. This article analyzes the materials of analytical and consulting companies, as well as current data and indicators that characterize consumers of augmented reality projects and the market of AR technologies. As a result, opportunities to overcome barriers to the introduction of augmented reality technologies and prospects for the development of the Russian consumer market for these technologies were identified.
In the Industry 4.0 era, the Digital Twin (DT), virtual copies of the system that are able to interact with the physical counterparts in a bi-directional way, seem to be promising enablers to replicate production systems in real time and analyse them. A DT should be capable to guarantee well-defined services to support various activities such as monitoring, maintenance, management, optimization and safety. Through an analysis of the current picture of manufacturing and a literature review about the already existing DT environment, this paper identifies what is still missing in the implemented DT to be compliant to their description in literature. Particular focuses of this paper are the degree of integration of the proposed DT with the control of the physical system, in particular with the Manufacturing Execution Systems (MES) when the production system is based on the Automation Pyramid, and the services offered from these environments, comparing them to the reference ones.
This paper proposes also a practical implementation of a DT in a MES equipped assembly laboratory line of the School of Management of the Politecnico di Milano. The application has been created to pose the basis to overcome the missing implementation aspects found in literature. In such a way, the developed DT paves the way for future research to close the loop between the MES and the DT taking into consideration the number of services that a DT could offer in a single environment.
Maintenance processes are generally subdivided into tasks with specified goals for the concerned practitioners. Collaboration is achieved through coordination, cooperation and communication. Smart devices and the Internet of Things (IoT) improve communication between men and machine alike, so the potential gain on cooperation and coordination in an IoT-enabled environment is examined. In this approach a concept for a framework is proposed to create Augmented Reality based collaboration assistant systems. AR is not only a tool for the visualization of maintenance data, but also for team-wide communication and the display of warnings or other coordination-related indications. The framework is validated with the presented use case in a laboratory environment.
In this paper, we study the trade-off between accuracy and speed when building an object detection system based on convolutional neural networks. We consider three main families of detectors --- Faster R-CNN, R-FCN and SSD --- which we view as "meta-architectures". Each of these can be combined with different kinds of feature extractors, such as VGG, Inception or ResNet. In addition, we can vary other parameters, such as the image resolution, and the number of box proposals. We develop a unified framework (in Tensorflow) that enables us to perform a fair comparison between all of these variants. We analyze the performance of many different previously published model combinations, as well as some novel ones, and thus identify a set of models which achieve different points on the speed-accuracy tradeoff curve, ranging from fast models, suitable for use on a mobile phone, to a much slower model that achieves a new state of the art on the COCO detection challenge.
In today's business environment, the efficiency of warehouses can be critical for the efficiency of the overall supply chains they belong to. As a result, new technologies are being tested and adopted in industry to improve the performance of warehouse operations. An example technology that has recently gained interest by both academia and industry is augmented reality. In this paper, we investigate the opportunities arising from the usage of augmented reality in warehouses as well as the barriers for its industrial adoption. This is done via a series of practitioners interviews and via an experiment designed using Google Glass. Our results indicate that even though the technology is not mature enough at the moment, the potential benefits it can offer
make it promising for the near future.
One of the main problems encountered in manual assembly workstations is human error in performing the operations. Several approaches are currently used to face this problem, such as intensive training of personnel, poka-yoke devices or invasive sensing systems (e.g. sensing gloves) used for monitoring the process and detect wrong procedures or errors in joining the parts. This paper proposes an innovative system based on the interaction between a force sensor and an augmented reality (AR) equipment used to give to the worker the necessary information about the correct assembly sequence and to alert him in case of errors. The force sensor is placed under the workbench and it is used to monitor the assembly process by collecting force and torque data with respect to an XYZ reference system; a pattern recognition technique allows the error identification and the selection of the appropriate recovery procedure. Two AR devices have been tested in this application: a video-mixing spatial display and an optical see-through apparatus, comparing the pro and cons of these two solutions. The first device includes a CCD camera positioned over the workstation and an LCD display used by the worker as a support for the correct execution of assembly operations and receiving instructions about recovery procedures. The latter consists of a head mounted display (HMD) having the capability of reflecting projected images in front of the worker's eyes, allowing a real-world view with the superimposition of virtual objects. The CCD camera is also used for identifying errors that are not detectable by the force sensor. At the end, a case study concerning a typical assembly procedure is presented and discussed.
Manual spot welding loses the comparison with automated spot welding, not because of a higher execution time, but due to an inferior quality of welded points, mostly a low repeatability. It is not a human fault. Human welder is compelled to operate without having at disposal the knowledge of significant process features that are known by the robot: exact position of the welding spot, electric parameters to be adopted for every specific point, quality of the welded spot and, based on it, possible need for repetition of a defective weld.
We are on the verge of ubiquitously adopting Augmented Reality (AR) technologies to enhance our perception and help us see, hear, and feel our environments in new and enriched ways. AR will support us in fields such as education, maintenance, design and reconnaissance, to name but a few. This paper describes the field of AR, including a brief definition and development history, the enabling technologies and their characteristics. It surveys the state of the art by reviewing some recent applications of AR technology as well as some known limitations regarding human factors in the use of AR systems that developers will need to overcome.
Caching at mobile edge servers can smooth temporal traffic variability and reduce the service load of base stations in mobile video delivery. However, the assignment of multiple video representations to distributed servers is still a challenging question in the context of adaptive streaming, since any two representations from different videos or even from the same video will compete for the limited caching storage. Therefore it is important, yet challenging, to optimally select the cached representations for each edge server in order to effectively reduce the service load of base station while maintaining a high quality of experience (QoE) for users. To address this, we study a QoE-driven mobile edge caching placement optimization problem for dynamic adaptive video streaming that properly takes into account the different rate-distortion (R-D) characteristics of videos and the coordination among distributed edge servers. Then, by the optimal caching placement of representations for multiple videos, we maximize the aggregate average video distortion reduction of all users while minimizing the additional cost of representation downloading from the base station, subject not only to the storage capacity constraints of the edge servers, but also to the transmission and initial startup delay constraints of the users. We formulate the proposed optimization problem as an integer linear program (ILP) to provide the performance upper bound, and as a submodular maximization problem with a set of knapsack constraints to develop a practically feasible cost benefit greedy algorithm. The proposed algorithm has polynomial computational complexity and a theoretical lower bound on its performance. Simulation results further show that the proposed algorithm is able to achieve a near-optimal performance with very low time complexity. Therefore, the proposed optimization framework reveals the caching performance upper bound for general adaptive video streaming systems, while the proposed algorithm provides some design guidelines for the edge servers to select the cached representations in practice based on both the video popularity and content information.
Augmented Reality (AR) systems in last few years show great potentialities in the manufacturing context: recent pilot projects were developed for supporting quicker product and process design, as well as control and maintenance activities. The high technological complexity together with the wide variety of AR devices require a high technological skill; on the other hand, evaluating their actual impacts on the manufacturing process is still an open question. Few recent studies have analysed this topic by using qualitative approaches based on an “ex post” analysis - – i.e. after the design and/or the adoption of the AR system - for evaluating the effectiveness of a developed AR application. The paper proposes an expert based tool for supporting production managers and researchers in effectively evaluating a preliminary ex-ante feasibility analysis for assessing quantitatively most efficient single AR devices (or combinations) to be applied in specific manufacturing processes. A multi-criteria model based on Analytic Hierarchy Process (AHP) method has been proposed to provide decision makers with quantitative knowledge for more efficiently designing AR applications in manufacturing. The model allows to integrate, in the same decision support tool, technical knowledge regarding AR devices with critical process features characterizing manufacturing processes, thus allowing to assess the contribution of the AR device in a wider prospective compared to current technological analyses. A test case study about the evaluation of AR system in on-site maintenance service is also discussed aiming to validate the model, and to outline its global applicability and potentialities. Obtained results highlighted the full efficacy of the proposed model in supporting ex-ante feasibility studies.
In 1991, Mark Weiser described the vision of a future world under the name of Ubiquitous Computing. Since then, many details of the described vision have become reality: Our mobile phones are powerful multimedia systems, our cars computer systems on wheels, and our homes are turning into smart living environments. All these advances must be turned into products for very cost-sensitive world markets in shorter cycles than ever before.
While convolutional neural network (CNN) architectures have achieved great
success in various vision tasks, the critical scale problem is still much
under-explored, especially for pedestrian detection. Current approaches mainly
focus on using large numbers of training images with different scales to
improve the network capability or result fusions by multi-scale crops of images
during testing. Designing a CNN architecture that can intrinsically capture the
characteristics of large-scale and small-scale objects and also retain the
scale invariance property is still a very challenging problem. In this paper,
we propose a novel scale-aware Fast R-CNN to handle the detection of small
object instances which are very common in pedestrian detection. Our
architecture incorporates a large-scale sub-network and a small-scale
sub-network into a unified architecture by leveraging the scale-aware weighting
during training. The heights of object proposals are utilized to specify
different scale-aware weights for the two sub-networks. Extensive evaluations
on the challenging Caltech~\cite{dollar2012pedestrian} demonstrate the
superiority of the proposed architecture over the state-of-the-art
methods~\cite{compact,ta_cnn}. In particular, the miss rate on the Caltech
dataset is reduced to by our method, significantly smaller than
by CompACT-Deep~\cite{compact} and by TA-CNN~\cite{ta_cnn}.
We present an augmented reality system that supports human workers in a rapidly changing production environment. By providing spatially registered information on the task directly in the user's field of view the system can guide the user through unfamiliar tasks (e.g. assembly of new products) and visualize information directly in the spatial context were it is relevant. In the first version we present the user with picking and assembly instructions in an assembly application. In this paper we present the initial experience with this system, which has already been used successfully by several hundred users who had no previous experience in the assembly task.
In 1991, Mark Weiser described the vision of a future world under the name of Ubiquitous Computing. Since then, many details of the described vision have become reality: Our mobile phones are powerful multimedia systems, our cars computer systems on wheels, and our homes are turning into smart living environments. All these advances must be turned into products for very cost-sensitive world markets in shorter cycles than ever before. Today, the resulting requirements for design, setup, and operation of our factories become crucial for success. In the past, we often increased the complexity in structures and control systems, resulting in inflexible monolithic production systems. But the future must become leannot only in organization, but also in planning and technology! We must develop technologies which allow us to speed up planning and setup, to adapt to rapid product changes during operation, and to reduce the planning effort. To meet these challenges we should also make use of the smart technologies of our daily lives. But for industrial use, there are many open questions to be answered. The existing technologies may be acceptable for consumer use but not yet for industrial applications with high safety and security requirements. Therefore, the SmartFactoryKL initiative was founded by industrial and academic partners to create and operate a demonstration and research test bed for future factory technologies. Many projects develop, test, and evaluate new solutions. This presentation describes changes and challenges, and it summarizes the experience gained to date in the SmartFactoryKL.
We discuss about industrial augmented reality. Each industrial process imposes its own peculiar requirements. This creates the need for specialized technical solutions, which in turn poses new sets of challenges. Because most industries must concern themselves with at least some of these industrial procedures, we consider design, commissioning, manufacturing, quality control, training, monitoring and control, and service and maintenance. AR lets users reconstruct virtual models of their area of interest and visualize models within their static views of a real scene.
Summary form only given. The author is working on two research projects in Boeing Computer Services that have to do with virtual reality technology. The first involves importing aircraft CAD data into a VR environment. Applications include a side range of engineering and design activities, all of which involve being able to view and interact with the CAD geometry as if one were inside an actual physical mockup of the aircraft. He refers to the technology being explored in the second project as “Augmented Reality”. This entails the use of a see-through head-mounted display with an optical focal length of about 20 inches, along with VR-style position/orientation sensing system. The intended application area is in touch labor manufacturing: superimposing diagrams or text onto the surface of a workpiece and stabilizing it there on specific coordinates, so that the appropriate information needed by a factory worker for each step of a manufacturing or assembly operation appears on the surface of the workpiece as if it were painted there. The hardest technical problem for augmented reality is position tracking. Long-range head position/orientation sensing systems that can operate in factory environments are needed. This requirement and others give rise to some interesting computational problems, including wearer registration and position sensing using image processing