Conference Paper

Deep Learning-Based Instance Segmentation for Feature Extraction of Branched Deformable Linear Objects for Robotic Manipulation

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Wang et al. also presented a deep-learning solution for detecting automotive connectors in [12] using object detection networks. In [13] we proposed to semiautomate the annotation process using intermediate models and predicting the annotation, which significantly speed up the annotation process. ...
Chapter
Full-text available
Wire harness installation is one of the most challenging processing steps for automation in automotive production. Wire harnesses have an infinite number of degrees of freedom, such that they change their shape continuously during manipulation. As of today, the human’s ability to perceive the wire harness and deal with its shape changes, is unmatched by any technical solution. Therefore, wire harnesses are still installed manually. This paper proposes a concept for wire harness localization and applies it for robotic wire harness installation. The concept uses two stereo cameras, one to perceive the shape of the wire harness and obtain rough position estimates of wire harness components, and a second for accurate 6D pose estimation of the individual components. The concept is evaluated with a case study on localization in a car chassis, where the accuracy and limitations of the concept are investigated by practical experiments.
Article
Full-text available
The wiring harness manufacturing follows a high-variant manufacturing philosophy with a high degree of manual labor to produce customized products. Therefore, the process step of optical inspection is crucial to monitor, assess, and proactively safeguard the quality of produced wiring harnesses. To automate the optical inspection, which is currently manually conducted by operators, deep learning has become a powerful algorithm outperforming traditional computer vision approaches. Deep learning-based automated optical inspection systems can robustly and reliably detect rigid and deformable components, such as connectors, clips, single wires, and wire bundles. However, the bottleneck for scalable deep learning solutions in the industrial environment and high deep learning model performance is the database for model training and optimization. To address this research gap, we propose a deep learning-based data processing pipeline for automated optical inspection of wiring harnesses using real and synthetically generated point clouds. This paper outlines the process of real and synthetic data generation and evaluates the potential of synthetic data to enrich real data for model training. The data processing pipeline is implemented and experimental findings are generated to deduct important parameters for synthetic data generation and deep learning model training which impact model performance.
Article
Full-text available
In this article, an innovative algorithm for instance segmentation of wires called Ariadne+ is presented. Although vastly present in many manufacturing environments, the perception and manipulation of wires is still an open problem for robotic applications. Wires are deformable linear objects lacking of any specific shape, color, and feature. The proposed approach uses deep learning and standard computer vision techniques aiming at their reliable and time effective instance segmentation. A deep convolutional neural network is employed to generate a binary mask showing where wires are present in the input image, then the graph theory is applied to create the wire paths from the binary mask through an iterative approach that aims to maximize the graph coverage. In addition, the B-Spline model of each instance, useful in manipulation tasks, is provided. The approach has been validated quantitatively and qualitatively using a manually labeled test dataset and by comparing it against the original Ariadne algorithm. The timings performances of the approach have been also analyzed in depth.
Article
Full-text available
Effective information management is critical for the development of manufacturing processes. This paper aims to provide an overview of ontologies that can be utilized in building Industry 4.0 applications. The main contributions of the work are that it highlights ontologies that are suitable for manufacturing management and recommends the multilayer-network-based interpretation and analysis of ontology-based databases. This article not only serves as a reference for engineers and researchers on ontologies but also presents a reproducible industrial case study that describes the ontology-based model of a wire harness assembly manufacturing process.
Article
Full-text available
In the context of the robotisation of industrial operations related to manipulating deformable linear objects, there is a need for sophisticated machine vision systems, which could classify the wiring harness branches and provide information on where to put them in the assembly process. However, industrial applications require the interpretability of the machine learning system predictions, as the user wants to know the underlying reason for the decision made by the system. We propose several different neural network architectures that are tested on our novel dataset to address this issue. We conducted various experiments to assess the influence of modality, data fusion type, and the impact of data augmentation and pretraining. The outcome of the network is evaluated in terms of the performance and is also equipped with saliency maps, which allow the user to gain in-depth insight into the classifier's operation, including a way of explaining the responses of the deep neural network and making system predictions interpretable by humans.
Conference Paper
Full-text available
Within the paradigm shift of electrification and autonomous driving, the automotive wiring harness has evolved to a safety-critical system, as it is responsible for the energy and information provision for safety-critical functions like steering and braking. To fulfill higher requirements concerning quality and safety, manufacturing automation is the key-enabler because it supports transparent, reproducible, and comprehensible processes. However, the state-of-the-art wiring harness manufacturing process is characterized by a high degree of manual work reaching up to 90%. In this paper, we systematically identified and clustered research publications and patents for wiring harness manufacturing automation, and analyzed the automation concepts as well as underlying technologies. Based on the findings, we derive research gaps that require an in-depth investigation in the future to increase the degree of automation in the wiring harness manufacturing.
Article
Full-text available
Object detection or localization is an incremental step in progression from coarse to fine digital image inference. It not only provides the classes of the image objects, but also provides the location of the image objects which have been classified. The location is given in the form of bounding boxes or centroids. Semantic segmentation gives fine inference by predicting labels for every pixel in the input image. Each pixel is labelled according to the object class within which it is enclosed. Furthering this evolution, instance segmentation gives different labels for separate instances of objects belonging to the same class. Hence, instance segmentation may be defined as the technique of simultaneously solving the problem of object detection as well as that of semantic segmentation. In this survey paper on instance segmentation, its background, issues, techniques, evolution, popular datasets, related work up to the state of the art and future scope have been discussed. The paper provides valuable information for those who want to do research in the field of instance segmentation.
Article
Full-text available
The mismatched-pins inspection of complex aviation connector is a critical process to ensure the correct wiring harness assembly, of which the existing manual operation is error-prone and time-consuming. Aiming to fill this gap, this work proposes an augmented reality (AR)-assisted deep learning-based approach to tackle three major challenges in aviation connector inspection, including small pins detection, multi-pins sequencing, and mismatched pins visualization. Firstly, the proposed spatial-attention pyramid network approach extracts image features in multilayers and searches for their spatial relationships among images. Secondly, based on the cluster-generation sequencing algorithm, these detected pins are clustered into annuluses of expected layers and numbered according to their polar angles. Lastly, AR glass as the inspection visualization platform, highlights mismatched pins in the augmented interface to warn operators automatically. Compared with other existing methodologies, the experimental result shows that the proposed approach can achieve better performance accuracy, and support operator's inspection process efficiently.
Article
Full-text available
Today’s automotive wire harness manufacturing process involves a high volume of manual work. However, current and future application requirements like miniaturization of electronic components, monitoring of process parameters, increasing demand of process documentation and rising wages call for a higher degree of automation. This paper presents the state of the art in wire harness assembly, reviews currently available special purpose machinery and equipment as well as their limitations and highlights the challenges of automation. It discusses potentials for automation, current innovations and research and concludes with an overview of current trends in the automotive industry.
Article
Full-text available
Traditional automated systems, such as industrial robots, are applied in well-structured environments, and many automated systems have a limited adaptability to deal with complexity and uncertainty; therefore, the applications of industrial robots in small- and medium-sized enterprises (SMEs) are very limited. The majority of manual operations in SMEs are too complicated for automation. The rapidly developed information technologies (IT) has brought new opportunities for the automation of manufacturing and assembly processes in the ill-structured environments. Note that an automation solution should be designed to meet the given requirements of the specified application, and it differs from one application to another. In this paper, we look into the feasibility of automated testing for electric cable harnesses, and our focus is on some of the generic strategies for the improvement of the adaptability of automation solutions. Especially, the concept of modularization is adopted in developing hardware and software to maximize system adaptability in testing a wide scope of products. A proposed system has been implemented, and the system performances have been evaluated by executing tests on actual products. The testing experiments have shown that the automated system outperformed manual operations greatly in terms of cost-saving, productivity and reliability. Due to the potential of increasing system adaptability and cost reduction, the presented work has its theoretical and practical significance for an extension for other automation solutions in SMEs.
Article
Wire harnesses are essential connecting components in manufacturing industry but are challenging to be automated in industrial tasks such as bin picking. They are long, flexible and tend to get entangled when randomly placed in a bin. This makes it difficult for the robot to grasp a single one in dense clutter. Besides, training or collecting data in simulation is challenging due to the difficulties in modeling the combination of deformable and rigid components for wire harnesses. In this work, instead of directly lifting wire harnesses, we propose to grasp and extract the target following a circle-like trajectory until it is untangled. We learn a policy from real-world data that can infer grasps and separation actions from visual observation. Our policy enables the robot to efficiently pick and separate entangled wire harnesses by maximizing success rates and reducing execution time. To evaluate our policy, we present a set of real-world experiments on picking wire harnesses. Our policy achieves an overall 84.6% success rate compared with 49.2% in baseline. We also evaluate the effectiveness of our policy under different clutter scenarios using unseen types of wire harnesses. Results suggest that our approach is feasible for handling wire harnesses in industrial bin picking.
Conference Paper
Handling of deformable linear objects (DLOs) is one of the most challenging tasks in robotics. This is because DLOs change their shape continuously during manipulation, which makes perception difficult. Therefore, this paper proposesa method for localization and tracking of DLOs. The proposed approach uses self organizing maps (SOMs) to register a model of the DLO to a point cloud obtained from a stereo vision system observing the scene. The registration is performed in two steps. First, a rigid registration roughly aligns the model with the acquired sensor data without allowing deformation. Second, a non-rigid registration accurately tracks the configuration of the observed DLO by allowing deformation. The model is simulated using a physics engine which is iteratively updated with each step of the non-rigid registration. The approach is evaluated in a series of experiments on a single rope, as well as a wire harness. It is shown that the approach is able to successfully track several configurations of the single rope and the wire harness, without prior knowledge of a previous configuration. The method is evaluated and compared to state-of-the-art tracking algorithms with regards to accuracy and robustness towards an uncertain initial guess.
Article
In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow we show that practitioners face issues at each stage of the deployment process. The goal of this paper is to lay out a research agenda to explore approaches addressing these challenges.
Article
The wiring harness is responsible for the information and energy flow within automotive vehicles and has become a safety-critical component within the trends of autonomous driving and electrification. As a result, the functionality, correctness, and traceability of the wiring harness need to be ensured with holistic quality assurance in the manufacturing. Addressing this research gap, this paper proposes an image processing pipeline for automated optical inspection of rigid and deformable linear objects in wiring harnesses. The deep learning-based optical inspection is outlined and implemented, the findings are presented, and the effectiveness and the relevance for industrial implementation are discussed.
Conference Paper
We present a method that estimates graspability measures on a single depth map for grasping objects randomly placed in a bin. Our method represents a gripper model by using two mask images, one describing a contact region that should be filled by a target object for stable grasping, and the other describing a collision region that should not be filled by other objects to avoid collisions during grasping. The graspability measure is computed by convolving the mask images with binarized depth maps, which are thresholded differently in each region according to the minimum height of the 3D points in the region and the length of the gripper. Our method does not assume any 3-D model of objects, thus applicable to general objects. Our representation of the gripper model using the two mask images is also applicable to general grippers, such as multi-finger and vacuum grippers. We apply our method to bin picking of piled objects using a robot arm and demonstrate fast pick-and-place operations for various industrial objects.
Conference Paper
This paper describes a system for automatically routing cable harnesses in three-dimensional environments using a pair of genetic algorithms. The cable harness routing problem (CHRP) can be formulated as a graph search problem with a large, convex search space. A genetic approach is used to intelligently and adaptively search for routings which are close to the global optimum. The CHRP is decomposed into two problems: generating a harness configuration (topology) and routing the harness in the environment. This paper defines the various genetic operators used and suggest parameter settings which quickly find routings which match the geometry of the environment
Pytorch: An imperative style, high-performance deep learning library
  • Paszke
Ariadne+: Deep learning-based augmented framework for the instance segmentation of wires
  • A Caporali
  • R Zanella
  • D D Greogrio
  • G Palli
Robotized assembly of a wire harness in a car production line
  • X Jiang
  • K.-M Koo
  • K Kikuchi
  • A Konno
  • M Uchiyama
Pytorch: An imperative style, high-performance deep learning library
  • A Paszke
  • S Gross
  • F Massa
  • A Lerer
  • J Bradbury
  • G Chanan
Label Studio: Data labeling software
  • M Tkachenko
  • M Malyuk
  • A Holmanyuk
  • N Liubimov