Shuo Feng’s research while affiliated with Northwestern Polytechnical University and other places

Compared to traditional techniques, augmented reality (AR) confers notable benefits in facilitating complex product assembly processes. The efficacy of AR systems in assembly contexts is notably influenced by the pivotal role of AR instructions. As such, meeting users’ demands for AR instructions is crucial during AR-guided assembly processes. In the present study, an investigation was conducted into the influence of complex assembly task types and user assembly experience on their demands for AR instructions. Firstly, complex assembly tasks were categorized into repetitive complex assembly tasks (RAT) and non-repetitive complex assembly tasks (NRAT) based on their complex characteristics. A user study was conducted using HMD-HoloLens 2 as the experimental device. User performances were recorded during iterative execution of AR experimental tasks under the aforementioned task conditions. The specific measures included users’ attention process, interface interaction behaviors, assembly errors, and users’ subjective experience. The results indicate significant differences in users’ demands for AR instructions across different task types. Moreover, users’ demands for AR instructions also changed with increasing assembly experience. Through comprehensive analysis, the rules of users’ demands for AR instructions were summarized. The present findings enhance the current comprehension of users’ demands regarding AR instructions and offer valuable insights for designing and developing efficient AR-guided assembly systems.

With the rapid development of augmented reality (AR) technology and devices, it is widely used in education, design, industry, game, medicine and other fields. It brings new development opportunities for computer-supported cooperative work. In recent years, there has been an increasing number of studies on AR collaboration. Many professional researchers have also summarized and commented on these local and remote applications. However, to the best of our knowledge, there is no comprehensive review specifically on AR-enabled local collaboration (AR-LoCol). Therefore, this paper presents a comprehensive survey of research between 2012 and 2022 in this domain. We surveyed 133 papers on AR-LoCol in Web of Science, 75% of which were published between 2018 and 2022. Next, we provide an in-depth review of papers in seven areas, including time (synchronous and asynchronous), device (hand-held display, desktop, spatial AR, head-mounted display), participants (double and multiple), place (standing, indoor and outdoor), content (virtual objects, annotations, awareness cues and multi-perspective views), and area (education, industry, medicine, architecture, exhibition, game, exterior design, visualization, interaction, basic tools). We discuss the characteristics and specific work in each category, especially the advantages and disadvantages of different devices and the necessity for shared contents. Following this, we summarize the current state of development of AR-LoCol and discuss possible future research directions. This work will be useful for current and future researchers interested in AR-LoCol systems.

In the development of significant products, assembly efficiency is essential to the improvement of the output and quality of the equipment, and there is still a large amount of work that needs to be done manually by the user during the assembly process. Due to the complex internal structure of the product, the user’s hands, assembly parts, and internal construction are often invisible. The lack of visual information leads to low assembly efficiency, high error rates and user fatigue. In this study, we developed a video-assisted assembly guidance system, called ARScope, in the augmented reality (AR) environment. Users can freely place the mini camera module on the back of their hands or inside the narrow blind area and obtain the video from the firstperson perspective (FPP) and the third-person perspective (TPP). In AR, users can get a comfortable guiding experience by transforming, rotating and scaling the video screen. In addition, we designed and conducted the user study to investigate the effect of different viewpoints on assembly efficiency and user’s experience during video-assisted blind assembly in AR space, and also explored other influencing factors of this process. The results showed that FPP had more synergy with the user than TPP, resulting in an efficient assembly. In addition, the stability of the screen, the tactile feedback and the portability of the system hardware were also essential factors in the blind assembly process.

The shortcomings of established input methods for Augmented Reality (AR) head-mounted displays (HMDs) motivate us to investigate the use of AR HMDs with smartphones and smartwatches to improve AR interaction in physical tasks. However, it is unclear whether the cross-device interaction interfaces are efficient for AR systems on physical tasks because the physical tasks can break the interaction flow. In this work, we conducted a user study to explore this. The user study consists of three subtasks, respectively requiring one of the three representative user interface (UI) controls (buttons, sliders, or text input). We implemented them with mid-air gestures, a smartphone, and a smartwatch. We compared the three interfaces and found that the smartphone and the smartwatch interaction interfaces have greater usability, provide a better user experience, and have lower workloads than the mid-air gesture interface. However, there is no significant difference in terms of efficiency for simple interactions. We discuss the limitations of this research and directions for future work.

Interaction with virtual objects is one of the essential features of Augmented Reality (AR) systems. One of its main issues is how to provide precise manipulation of distant virtual objects in AR. In this work, we explore bare-hand manipulation of distant objects in AR with DOF (degree-of-freedom) separation, motion scaling, and near-field metaphors. We developed two manipulation techniques: the distant widget-based metaphor (DWBM), and the near-field widget-based metaphor (NFWBM). We conducted a user study with 20 participants to compare the two techniques in terms of performance and user experience. We found that NFWBM has faster speed, lower mental effort, better ease-of-use, and a friendlier user experience. However, there is no significant difference in terms of precision; both techniques could manipulate distant objects precisely, with an average position error of less than 0.7 cm and an average orientational error of less than 1°. We also discussed the limitations of this research and directions for future work.

Most augmented reality (AR) assembly guidance systems only utilize visual information. Regarding the sound, the human binaural effect helps users quickly identify the general direction of sound sources. At the same time, pleasant sounds can give people a sense of pleasure and relaxation. However, the effect on workers is still unknown when stereo sound and visual information are used together for assembly guidance. To assess the combination of sound and vision in AR assembly guidance, we constructed a stereo sound-assisted guidance system (SAG) based on AR. In our SAG system, we used the tone of a soft instrument called the Chinese lute as the sound source. To determine if SAG has an impact on assembly efficiency and user experience, we conducted a usability test to compare SAG with visual information alone. Results showed that the SAG system significantly improves the efficiency of assembly guidance. Moreover, simultaneous visual and auditory information processing does not increase user workload or learning difficulty. Additionally, in a noisy environment, pleasant sounds help to reduce mental strain.

During component assembly, some operations must be completed by two or more workers due to the size or assembly mode. For instance, in manual riveting, two workers are positioned on either side of a steel plate, which blocks the view. Traditional collaborative approaches limit assembly efficiency and is difficult to ensure accurate and rapid interaction between workers. In this study, we developed an AR-Assisted Cooperative Assembly System (ARCoA) to address the issue. ARCoA allows users to view their partner’s key information that is occluded, including tools, gestures, orientations, and shared markers. Besides, we presented a user experiment that compared this method with the traditional approach. The results indicated that the new system could significantly improve assembly efficiency, system availability, and sense of social presence. Moreover, most users we surveyed preferred ARCoA. In the future, we will incorporate more functions and improve the accuracy of the system to solve complex multi-person collaborative problems.

The assembly stage is a vital phase in the production process and currently, there are still many manual tasks in the assembly operation. One of the challenges of manual assembly is the issue of blind area assembly since the visual obstruction of the hands or a part can lead to more errors and lower assembly efficiency. In this study, we developed an AR-assisted assembly system that solves the occlusion problem. Assembly workers can use the system to achieve comprehensive and precise hand–eye coordination (HEC). Additionally, we designed and conducted a user evaluation experiment to measure the learnability, usability, and mental effort required for the system for other HEC modes. Results indicate that hand position is the first visual information that should be considered in blind areas. Besides, the Intact HEC mode can effectively reduce the difficulty of learning and mental burden in operation, while at the same time improving efficiency.

Sketch design is generally a feasible concept within virtual reality (VR). However, VR controllers suffer from a fundamental problem that limits operability and immersion, which is the difficulty of imitating the texture of natural materials. To solve these challenges, we developed a tablet-based design system named Pressure-Sketch (PSK) around the tactile features of the digital tablet and stylus. PSK could provide an intuitive design workflow for VR. The digital tablet is covered with natural wood to give a more realistic feel. In addition, utilizing the pressure property, users can directly touch the material by the stylus to create lines with varying widths. An evaluated user study revealed that PSK could significantly improve designs’ accuracy and aesthetic quality. Moreover, the system has a better sense of immersion. We are willing to explore the differences between more types of materials on sketching in VR.