Weipintg He

• Northwestern Polytechnical University

The visualization of complex spatial data is typically achieved using the Windows, Icons, Menus, and Pointer (WIMP) method. However, this approach has limitations, including a steep learning curve, non-intuitive interfaces, and limited visualization capabilities. To overcome these challenges, this study proposes an augmented reality-based (AR-based...

Virtual content in Augmented Reality (AR) applications can be constructed according to the designer's requirements, but real environments, are difficult to be accurate control or completely reproduce. This makes it difficult to prototype AR applications for certain real environments. One way to address this issue is to use Virtual Reality (VR) to s...

Manual precise manipulation of objects is an essential skill in everyday life, and Augmented Reality (AR) is increasingly being used to support such operations. In this study, we investigate whether detailed visualizations of position and orientation deviations are helpful for AR-assisted manual precise manipulation of objects. We developed three A...

In this research, a blind area information perception and guidance approach for dynamic context is proposed as a solution to the issue of difficult and time-consuming assembly in blind areas. The proposed approach involves the utilization of real-time RGBD data to perceive both blind area context and operator hand information. The resulting data is...

AR assembly instruction is a method that helps users with manual assembly by overlaying virtual information on the real environment. However, users have to manually change the information display form, which fails to adapt to their varying information needs in different stages of the assembly task. This study proposes a user-centered adaptive visua...

With the rapid development of augmented reality (AR) technology and devices, it is widely used in education, design, industry, game, medicine and other fields. It brings new development opportunities for computer-supported cooperative work. In recent years, there has been an increasing number of studies on AR collaboration. Many professional resear...

In this research, a blind area information perception and guidance approach for dynamic context is proposed as a solution to the issue of difficult and time-consuming assembly in blind areas. The proposed approach involves the utilization of real-time RGBD data to perceive both blind area context and operator hand information. The resulting data is...

In the development of significant products, assembly efficiency is essential to the improvement of the output and quality of the equipment, and there is still a large amount of work that needs to be done manually by the user during the assembly process. Due to the complex internal structure of the product, the user’s hands, assembly parts, and inte...

In mixed reality (MR) remote collaborative assembly, remote experts can guide local users to complete the assembly of physical tasks by sharing user cues (eye gazes, gestures, etc.) and spatial visual cues (such as AR annotations, virtual replicas). At present, remote experts need to carry out complex operations to transfer information to local use...

Anti-counterfeiting tags with physical unclonable functions (PUFs) are critical for ensuring the security of products. In general, PUFs exhibit unique random physical patterns as a result of a stochastic minutiae that generates a large number of robust tags. In the current study, a novel flower-like dendritic micropattern security tag is created in...

The assembly in Virtual Reality (VR) enables users to fit virtual parts into existing 3D models immersively. However, users cannot physically feel the haptic feedback when connecting the parts with the virtual model. This work presents a robot-enabled tangible interface that dynamically moves a physical structure with a robotic arm to provide physi...

The shortcomings of established input methods for Augmented Reality (AR) head-mounted displays (HMDs) motivate us to investigate the use of AR HMDs with smartphones and smartwatches to improve AR interaction in physical tasks. However, it is unclear whether the cross-device interaction interfaces are efficient for AR systems on physical tasks becau...

Interaction with virtual objects is one of the essential features of Augmented Reality (AR) systems. One of its main issues is how to provide precise manipulation of distant virtual objects in AR. In this work, we explore bare-hand manipulation of distant objects in AR with DOF (degree-of-freedom) separation, motion scaling, and near-field metaphor...

Virtual reality (VR) controllers are widely used for 3D virtual object selection and manipulation in immersive virtual worlds, while touchscreen-based devices like smartphones or tablets provide precise 2D tangible input. However, VR controllers and touchscreens are used separately in most cases. This research physically integrates a VR controller...

In recent years, the multi-dimensional, multi-faceted, and multi-level applications of augmented reality/mixed reality have attracted more and more attention from researchers in the field of smart manufacturing. However, the disadvantage is that AR interaction cues supporting user cognition have not been fully analyzed and summarized in manual asse...

Related research has shown that collaborating with Intelligent Virtual Agents (IVAs) embodied in Augmented Reality (AR) or Virtual Reality (VR) can improve task performance and reduce task load. Human cognition and behaviors are controlled by brain activities, which can be captured and reflected by Electroencephalogram (EEG) signals. However, littl...

In the process of selective assembly for precision and complex mechanical products, it is necessary to perform the selective material kitting one by one and conduct check repeatedly according to the optimal matching result to ensure quality. The traditional manual picking and checking method by Bill of Material is inefficient and laborious. Current...

As an application paradigm of augmented reality in industry, the projected augmented reality (PAR) system is helpful in assembly, improving the ability of human–computer assembly cooperation. Precious research mainly focuses on visualization whereas the problem of how users naturally interact with the system has still not been discussed. In this pa...

The efficient management of aero-engine blades plays a vital role in improving material turnover and productivity, while the automatic recognition of blade code provides support for digital production. However, there is a lack of discussion about intelligent recognition, result correction, and effective solutions to material management methods for...

In the colloidal-salt system of sodium silicate and sodium chloride, three-fold and four-fold symmetric dendrites and diffusion-limited aggregation (DLA) of sodium chloride crystals have been synthesized by a simple one-step gelation process without changing the solution composition. The morphology variation is realized by changing the sol pretreat...

In AR operation guidance training, for assembly parts with similar geometric shapes, there are still two problems in the visual representation of AR instructions: (1) AR instructions cannot accurately represent the micro-geometric differences between similar parts. (2) The AR instruction design specification that reflects the micro-geometric differ...

With the rapid development of mixed reality (MR) technology, many compact, lightweight, and powerful devices suitable for remote collaboration, such as MR headsets, hand trackers, and 3D cameras, become readily available, providing hardware and software support for remote collaboration. Consequently, exploring MR technologies for remote collaborati...

Despite a broad range of mobile AR applications available to date, the majority still primarily use surface gestures, i.e., gesturing on the touch screen surface of the device and do not utilise the affordance of three-dimensional user interaction that AR interfaces support. In this research, we compared two methods of gesture interaction for mobil...

Most augmented reality (AR) assembly guidance systems only utilize visual information. Regarding the sound, the human binaural effect helps users quickly identify the general direction of sound sources. At the same time, pleasant sounds can give people a sense of pleasure and relaxation. However, the effect on workers is still unknown when stereo s...

During component assembly, some operations must be completed by two or more workers due to the size or assembly mode. For instance, in manual riveting, two workers are positioned on either side of a steel plate, which blocks the view. Traditional collaborative approaches limit assembly efficiency and is difficult to ensure accurate and rapid intera...

Consistent visual and haptic feedback is an important way to improve the user experience when interacting with virtual objects. However, the perception provided in Augmented Reality (AR) mainly comes from visual cues and amorphous tactile feedback. This work explores how to simulate positional vibrotactile feedback (PVF) with multiple vibration mot...

The assembly stage is a vital phase in the production process and currently, there are still many manual tasks in the assembly operation. One of the challenges of manual assembly is the issue of blind area assembly since the visual obstruction of the hands or a part can lead to more errors and lower assembly efficiency. In this study, we developed...

This research explores augmented perception by investigating the effects of spatial scale manipulation in Virtual Reality (VR) to simulate multiple levels of virtual eye height (EH) and virtual interpupillary distance (IPD) of the VR users in the design context. We have developed a multiscale VR system for design applications, which supports a dyna...

Sketch design is generally a feasible concept within virtual reality (VR). However, VR controllers suffer from a fundamental problem that limits operability and immersion, which is the difficulty of imitating the texture of natural materials. To solve these challenges, we developed a tablet-based design system named Pressure-Sketch (PSK) around the...

Augmented Reality (AR) and Virtual Reality (VR) techniques are widely used for people to experience and interact with 3D digital content. However, the user has to choose either AR or VR separately with different devices for different interaction modalities. This work presents ARinVR, a novel concept that seamlessly and synchronously integrates hand...

This paper provides a review of research into using Augmented Reality (AR) and Mixed Reality(MR) for remote collaboration on physical tasks. AR/MR-based remote collaboration on physical tasks has recently become more prominent in academic research and engineering applications. It has great potential in many fields, such as real-time remote medical...

The alignment of 3D point clouds consists of coarse alignment and precise alignment. The preliminary coarse alignment must be implemented for point clouds with a significant initial pose difference before time-consuming precise alignment. However, this procedure is normally finished on 2D interfaces manually, which leads to a partial perception of...

The manual assembly has a high proportion in industry. However, in many industrial scenarios, manual assembly in the Vision Blind Areas (VBAs) is time-consuming and challenging due to the lack of necessary visual information. This study presented a see-through Augmented Reality (AR) system to solve the problems during manual assembly in the vision...

With the continuous emergence and vigorous development of various new technologies in the information age, human-computer interaction is also changing constantly. People are increasingly connected with the virtual world, so it is necessary to study more natural ways of human-computer interaction. Projection interaction is one of the most important...

As Virtual Reality(VR), Augmented Reality(AR), Mixed Reality(MR) technology becomes more accessible, it is important to explore VR/AR/MR technologies that can be used for remote collaboration on physical tasks. Previous research has shown that gesture-based interaction is intuitive and expressive for remote collaboration, and using 3D CAD models ca...

With the development of augmented reality supporting manual assembly collaboration, it is particularly important to explore the transformation from traditional “human-machine” cooperation mode to smart “human-machine” cooperation mode. Early studies have shown that user cues (i.e., head, gesture, eye) and scene cues (i.e., objects, tools, space) ar...

The research of augmented reality physical tasks for human-computer collaboration is still an attractive dynamic research field. In recent years, AR Instruction supporting collaborative assembly has been developed, which can transfer graphic annotation between man and machine. However, there is little research on high precision manual assembly. Thi...

AR instruction is a kind of virtual information presented on a human–computer interface. It allows users to view the geometric state, spatial relationship, operation method, and other information involved in the physical task, to form the spatial cognition of the current interaction process. At present, AR instructions cannot support high-precision...

The prop-based 3D virtual object manipulation method is widely used for interaction in Augmented Reality (AR) due to its convenience and flexibility. However, when the represented virtual object is different from the physical prop, the look and feel of the object are not well aligned. To address this problem, we present a dynamic finger remapping a...

In the modern assembly, manual assembly is one of the essential works in the factory. By the help of the robots and any other automatic equipments, modern assembly is quite important to improve the efficiency of the necessary manual assembly. Projection is one of the most significant assisted method to solve the assemble problem. This paper present...

For more than 2 decades, Augmented Reality (AR)/Mixed Reality (MR) has received an increasing amount of attention by researchers and practitioners in the manufacturing community, because it has applications in many fields, such as product design, training, maintenance, assembly, and other manufacturing operations. However, to the best of our knowle...

With the rapid development of human-computer interaction, computer graphics and other technologies, Augmented Reality is widely used in the assembly of large equipment, but in practical applications there are a lot of problems such as unreasonable color design. Based on this, we propose a projection display color design system based on color theory...

This paper investigates the effect of using augmented reality (AR) annotations and two different gaze visualizations, head pointer (HP) and eye gaze (EG), in an AR system for remote collaboration on physical tasks. First, we developed a spatial AR remote collaboration platform that supports sharing the remote expert’s HP or EG cues. Then the protot...

Multistroke drawing occurs frequently in conceptual design sketches; however, it is almost unsupported by the current sketch-based user interfaces. We proposed a sketch recognition system based on the multistroke primitive grouping method. Based on grouping the strokes that lie within the mutual boundaries between adjacent regions, we create line d...

Multiple overtracing strokes are common drawing behaviors in freehand sketching; that is, additional strokes are often drawn repeatedly over the existing ones to add more details. This paper proposes a method based on stroke-tolerance zones to group multiple overtraced strokes which are drawn to express a 2D primitive, aiming to convert online free...

In AR assembly, geometric level visualization (GLV) embeds nongeometric attributes as guidance information into the real world and realizes the integration of human intelligence and assembly process at the geometric level. The guidance information generated by GLV is geometric level AR instruction, which contains the product manufacturing informati...

Recently, there are some problems in AR instruction research in assembly field, such as irregular instruction design form and redundant display of instruction content. The reason is that there is no AR instruction design rule for AR assembly. The goal of this rule should be to maximize users’ cognitive efficiency of guided content. In order to solv...

In AR/MR remote collaboration, a remote expert often has to demonstrate tasks for a local worker. To make this easier, we have developed a new adaptive MR remote collaborative architecture which enables a remote expert to guide a local user in physical assembly and training tasks. A remote user can activate the instructions through simple and intui...

Current remote collaborative systems in manufacturing are mainly based on video-conferencing technology. Their primary aim is to transmit manufacturing process knowledge between remote experts and local workers. However, it does not provide the experts with the same hands-on experience as when synergistically working on site in person. The mixed re...

Haptic feedback has shown its great value in HCI research and applications for enhancing user experiences. Simulating people's tactile sensations of virtual objects is currently a primary research target. Different from wearing motors on fingers or hands, we attached vibration motors on a physical object to simulate the augmented sense of collision...

In this study we present a Mixed-Reality based mobile remote collaboration system that enables an expert providing real-time assistance over a physical distance. By using the Google ARCore position tracking, we can integrate the keyframes captured with one external depth sensor attached to the mobile phone as one single 3D point-cloud data set to p...

To improve the applicability and robustness of the three-dimensional tracking method of an augmented reality-aided assembly guiding system for mechanical products, a tracking method based on the combination of point cloud and visual feature is proposed. First, the tracking benchmark coordinate system is defined using a reference model point cloud t...

Purpose This paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step. Design/methodology/approach The prototype system’s pr...

Markerless tracking is still a very challenging problem in augmented reality applications, especially the real elements are textureless. In this paper, we proposed a model-based method to tackle the markerless tracking problem. Motivated by LINE-MOD algorithm, one of the state-of-the-art object detection methods, and multiview-based 3D model retrie...

In order to improve the corrosion resistance of Data Matrix (DM) code laser marked on aluminum alloy, the salt spray corrosion resistance was determined. First, the corrosion resistance of DM code laser marked by 3 different processing modes was investigated. Meanwhile, the SEM images and EDS data for modules and substrate of DM code are analyzed,...

Precise miniaturization of 2D Data Matrix (DM) Codes on Aluminum alloy formed by raster mode laser direct part marking is demonstrated. The characteristic edge over-burn effects, which render vector mode laser direct part marking inadequate for producing precise and readable miniature codes, are minimized with raster mode laser marking. To obtain t...

Aiming at the serious homogenization problems between the products, particularly with a huge number of the same model and batches products, and the product identification could not be read due to wear, pollution and other factors which led small characteristic differences between products failure to identify, a homogenization product identification...

Aiming at the problem that the two-dimensional Data Matrix symbol was wear, pollution and corrosion to fail identify leaded by production environment and process complexity in discrete manufacturing enterprise, a multi-information fusion to remediation failure identification based on D-S evidence theory was put forward. The correlation and data str...

For a more scientifically evaluate of home appliance product design, on basis of determining the evaluation index, Fuzzy AHP (Fuzzy-Analytic Hierarchy Process) comprehensive evaluation method is used to converse qualitative index to quantitative index, establish a home appliance product design evaluation model, overcome subjectivity and arbitrarine...

Direct Part Marking (DPM) is used for part identification and tracking. In order to avoid discrete information caused by quality data incompleteness or disorder in each stage of a product life cycle, especially for a large and complex product information, DPM is an effective method to ensure the security and traceability of the process quality stat...

In order to solve the problem of bar code failure because of inadequate protection, pollution and other factors in the process of production, presented a new multi-information fusion technology used of residual bar code information, shape, texture and the weight information to identify the tool information in the database and recovery the tool iden...

For the problem of the data matrix (DM) code Positioning marked in cylinder glass, this paper presents an accurate positioning method of DM code image marked in cylinder glasses. Firstly, the method enhances the contrast ratio of the image based on the Retinex theory. Secondly, the paper uses projection and morphology for the early positioning of t...

A mosaic method of based on rotational scan sequence cylindrical barcode which is suitable for metal parts is put forward, which combined the hardware and the software. Firstly, extract the feature points of each serial image based on scale-interaction of Marr wavelets. Then, get the optimal match of feature points with improved image local entropy...

The reconfigurability of Manufacturing Execution System (MES) means that it can adapt to different manufacturing environment and application requirements rapidly and easily. A Reconfigurable MES Model (RMM) is discussed with a hierarchical architecture including Production Object Level, Business Activity Level, Business Function Level and Organizat...

For the special requirements of recognition the Data Matrix barcode which marked on the cutter cylinder, considering of traditional recognition methods can’t solve the problems such as highlight reflection, cylinder distortion, the information incomplete collection, etc., causing low recognition rate. This paper presents a cutter cylinder Data Matr...

Parting at accurate locating challenges of Data Matrix code that polluted by abrasion, in this paper we present an extracting of best edges combination approach based on code boundary multi-features fusion. Firstly, transform DataMatrix code from image into Hough domain in which candidate boundary combinations be extracted. Afterwards, we design an...

To enable rapid response to the needs of MES reconfiguration, we introduce a component -based MES reconfiguration approach based on PIM. In this approach we get feature values by extending PIM, and use them to query reusable components that are needed to perform MES reconfiguration. By this means we can ensure system flexibility, on the other hand...

The traditional detection algorithms fall into non-machine learning method, which utilize geometrical characteristic of 2D code, introduce digital image analysis method,marginal detection and geometry detection, etc. while these algorithms are no more than elementary methods, they are limited to printed paper, and not applicable to other material s...

The existing distortion correction methods are not fully suitable the multi-distortion image of Data Matrix symbol directly marked on cylinder surface, so an ellipse-arc grid method is proposed to correct this kind of distortion image. Firstly, two ellipse-arc boundaries and two line boundaries are detected based on the symbol contour point set, Se...