No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
An intelligent and automated 3D surface defect detection system for quantitative 3D estimation and feature classification of material surface defects
Abstract
To evaluate defects on the surface of the materials at the 3D level accurately and quantitatively, a 3D surface defect detection system based on stereo vision is presented, which can extract the precise 3D defect features of the detected object. The proposed detection system consists of two image capture modules and a turntable to capture the complete 3D information and color texture information from the object surface. More precisely, each image capture module is a binocular stereo vision system containing two monochrome cameras, a color camera, and a speckle projector which is used to reconstruct the 3D point clouds of the object surface based on stereo digital image correlation (stereo-DIC). Furthermore, a point-image mapping relationship between the reconstructed 3D object points and the color images is established. Eventually, the 3D characteristic parameters of defects are calculated by the corresponding 3D point cloud of the defect area obtained by segmenting the defect area using the image segmentation and point cloud segmentation algorithms according to this point-image mapping relationship. A convolutional neural network named DenseNets is employed to identify defect types intelligently. A high-precision multi-camera calibration method based on close-range photogrammetry is applied to ensure system detection accuracy in the proposed system. The experimental results demonstrate that the system has higher accuracy and better performance in system calibration, 3D reconstruction, and defect feature calculation.
... Tujuan penggunaan dwg agar desain yang sudah dirancang dapat di baca oleh sistem monitoring mesin laser cutting secara 2D dan dapat memotong sesuai desain dari format dwg. Setelah melalui proses perancangan maka tahap berikutnya adalah dengan menentukan parameter potong dari mesin laser cutting , tetapi menjadi faktor kesalahan utama apabila parameter diinput tidak sesuai dengan kekuatan material maka akan menimbulkan hasil pemotongan pada material menjadi hangus [2,3,4]. Dengan permasalahan tersebut maka penelitian ini bertujuan untuk mengembangkan kamera ESP32 CAM untuk mendeteksi cacat hangus dan cacat tidak rata pada material yang dipotong dengan menggunakan pemrograman bahasa C dan Open CV. ...
... 1. Alur PenelitianSesuai dengan alur penelitian pada Gambar 1 menjelaskan bahwa semua tahapan agar menghasilkan proses pengamatan cacat material menggunakan kamera ESP32CAM. Metode yang dikebangkan dalam penelitian ini merujuk pada penelitian Yulong Zong dan teman-teman[3] membahas "kamera yang dapat mendeteksi cacat akibat pemotongan menggunakan kamera dalam pengolahan citra warna yaitu metode segmentasi warna, perubahan citra warna dan pendeteksi cacat akibat perubahan segemntasi warna menggunakan pemrograman Open CV".2.1. Persiapan Material dan Laser CuttingMaterial utama yang digunakan dalam penelitian ini adalah material Aluminium 6061 yang dapat dilihat spesifikasinya pada Tabel 1. ...
Using high-pressure and high-temperature laser light emission techniques, laser cutting works to cut materials in such a way that components at the laser cutting point produce cutting results. However, the current problem is that laser cutting often produces material defects including protruding parts, burnt surfaces, and pores on the laser-cut parts. Given this problem, the purpose of this study is to develop a camera capable of identifying material defects caused by laser cutting. The research method is the preparation of 6061 aluminum material, the preparation of the ESP32CAM camera, and the preparation of laser cutting parameter settings. In order to run the ESP32CAM program, C and OpenCV programming languages are needed to identify items with material defects, color images, histograms, and FPS are needed. The results of the study showed that detecting defects at the highest FPS reading of 15.57 and a histogram value of 250 at the x coordinate and 950000 at the y coordinate. Eight defects in the material were found using Open CV detection on the ESP32CAM camera sample 3. From this technique it can be concluded that ESP32CAM is capable of identifying material defects caused by laser cutting.
... Tujuan penggunaan dwg agar desain yang sudah dirancang dapat di baca oleh sistem monitoring mesin laser cutting secara 2D dan dapat memotong sesuai desain dari format dwg. Setelah melalui proses perancangan maka tahap berikutnya adalah dengan menentukan parameter potong dari mesin laser cutting , tetapi menjadi faktor kesalahan utama apabila parameter diinput tidak sesuai dengan kekuatan material maka akan menimbulkan hasil pemotongan pada material menjadi hangus [2,3,4]. Dengan permasalahan tersebut maka penelitian ini bertujuan untuk mengembangkan kamera ESP32 CAM untuk mendeteksi cacat hangus dan cacat tidak rata pada material yang dipotong dengan menggunakan pemrograman bahasa C dan Open CV. ...
... 1. Alur PenelitianSesuai dengan alur penelitian pada Gambar 1 menjelaskan bahwa semua tahapan agar menghasilkan proses pengamatan cacat material menggunakan kamera ESP32CAM. Metode yang dikebangkan dalam penelitian ini merujuk pada penelitian Yulong Zong dan teman-teman[3] membahas "kamera yang dapat mendeteksi cacat akibat pemotongan menggunakan kamera dalam pengolahan citra warna yaitu metode segmentasi warna, perubahan citra warna dan pendeteksi cacat akibat perubahan segemntasi warna menggunakan pemrograman Open CV".2.1. Persiapan Material dan Laser CuttingMaterial utama yang digunakan dalam penelitian ini adalah material Aluminium 6061 yang dapat dilihat spesifikasinya pada Tabel 1. ...
Using high-pressure and high-temperature laser light emission techniques, laser cutting works to cut materials in such a way that components at the laser cutting point produce cutting results. However, the current problem is that laser cutting often produces material defects including protruding parts, burnt surfaces, and pores on the laser-cut parts. Given this problem, the purpose of this study is to develop a camera capable of identifying material defects caused by laser cutting. The research method is the preparation of 6061 aluminum material, the preparation of the ESP32CAM camera, and the preparation of laser cutting parameter settings. In order to run the ESP32CAM program, C and OpenCV programming languages are needed to identify items with material defects, color images, histograms, and FPS are needed. The results of the study showed that detecting defects at the highest FPS reading of 15.57 and a histogram value of 250 at the x coordinate and 950000 at the y coordinate. Eight defects in the material were found using Open CV detection on the ESP32CAM camera sample 3. From this technique it can be concluded that ESP32CAM is capable of identifying material defects caused by laser cutting. Abstrak Menggunakan teknik cahaya laser bertekanan tinggi dan bersuhu tinggi, pemotongan laser bekerja untuk memotong material sedemikian rupa sehingga komponen pada titik pemotongan laser menghasilkan hasil pemotongan. Namun, masalah saat ini adalah pemotongan laser sering kali menghasilkan cacat material termasuk bagian yang menonjol, permukaan yang terbakar, dan pori-pori pada bagian yang dipotong laser. Mengingat masalah ini, tujuan penelitian ini adalah untuk mengembangkan kamera yang mampu mengidentifikasi cacat material yang disebabkan oleh pemotongan laser. Metode penelitian persiapan material aluminium 6061, persiapan kamera ESP32CAM, dan persiapan pengaturan parameter pemotongan laser. Supaya dapat menjalakan program ESP32CAM maka dibutuhkan pemrograman bahasa C dan OpenCV bertujuan mengidentifikasi item dengan cacat material, citra warna, histogram, dan FPS diperlukan. Hasil penelitian menunjukkan mendeteksi cacat pada pembacaan FPS tertinggi sebesar 15,57 dan nilai histogram sebesar 250 pada koordinat x dan 950000 pada koordinat y. Delapan cacat pada material ditemukan menggunakan deteksi Open CV pada 3 sampel kamera ESP32CAM. Dari teknik ini dapat disimpulkan bahwa ESP32CAM mampu mengidentifikasi cacat material yang disebabkan oleh pemotongan laser. Kata kunci: ESP32CAM, pemotongan laser, openCV, segmentasi warna.
... However, Retinex still shows unnatural images in uneven, low-illumination images. Therefore, constraining the illumination component and reflection component to avoid information loss while maintaining the naturalness of the enhanced image is one of the urgent problems to be solved [19,20]. leads to a very dim scene. ...
... However, Retinex still shows unnatural images in uneven, low-illumination images. Therefore, constraining the illumination component and reflection component to avoid information loss while maintaining the naturalness of the enhanced image is one of the urgent problems to be solved [19,20]. Low-light image enhancement techniques allow observation and localization of surface defects in the first wall of the vacuum chamber. ...
This paper addresses image enhancement and 3D reconstruction techniques for dim scenes inside the vacuum chamber of a nuclear fusion reactor. First, an improved multi-scale Retinex low-light image enhancement algorithm with adaptive weights is designed. It can recover image detail information that is not visible in low-light environments, maintaining image clarity and contrast for easy observation. Second, according to the actual needs of target plate defect detection and 3D reconstruction inside the vacuum chamber, a defect reconstruction algorithm based on photometric stereo vision is proposed. To optimize the position of the light source, a light source illumination profile simulation system is designed in this paper to provide an optimized light array for crack detection inside vacuum chambers without the need for extensive experimental testing. Finally, a robotic platform mounted with a binocular stereo-vision camera is constructed and image enhancement and defect reconstruction experiments are performed separately. The results show that the above method can broaden the gray level of low-illumination images and improve the brightness value and contrast. The maximum depth error is less than 24.0% and the maximum width error is less than 15.3%, which achieves the goal of detecting and reconstructing the defects inside the vacuum chamber.
... In fact, the current (2D) machine vision methods only perform the qualitative evaluation of defects and are not applicable when the 3D geometric parameters of the defect area need to be given accurately. Therefore, in order to capture full information about the geometric characteristics of the 3D defects, 3D capturing techniques attract wide attention and become the development trend of the planar materials inspection [26,27]. Enzberg and Al-Hamadi [28] proposed a model-based 3D surface quality inspection of car body panels with ordered 3D PCs reconstructed from 4-camera setup and pattern projection, which combines a machine learning approach with multiresolution paradigms. ...
... In this regard and according to the fact that object detection in 2D image is a well-developed area and more straightforward, some researchers tried to combine 2D and 3D vision-based methods such that detection is executed in 2D image and then obtained result is improved by the spatial information of 3D PC. For instance, Zong et al. [27] proposed to use combination of a stereo vision with (2D) image information; their system consists of one color camera, two monochrome cameras as stereo vision rig, and a speckle projector used to reconstruct the 3D PCs. They aimed to detect defects based on RGB (red, green and blue) change in the color image and define the defects characteristics based on the reconstructed 3D PCs. ...
Online inspection and surface characterization including 3D defect detection of surfaces using 3D scanning and
coordinate metrology data has crucial applications in today’s Industry 4.0. Although 3D vision-based metrology
methods are superior to 2D in providing spatial information, its processing remains challenge. A novel automated
Detailed Deviation Zone Evaluation method is proposed in this paper in which 3D unorganized PC is converted
into 2D grayscale image such that the intensity variation of image is proportional to the surface topography. This
developed image is addressed as “skin Image” of the scanned surface. The considered point clouds include only
XYZ-coordinates. No prior knowledge of defects, and no training set is required. The methodology is fully
implemented, and verified by inspecting point clouds of several workpieces with predefined defects. The
experimental results show high efficiency of the developed methodology in defect detection and 3D-feature
identification irrespective of shape and size.
... Point clouds, one type of common 3D data, can accurately depict the physical world and represent 3D shapes with a more concise data structure than image data. Point cloud-based defect detection techniques have been widely used in various industries, such as aircraft manufacturing [6], rail manufacturing [7], PCB [8], heritage restoration [9], and agriculture and forestry industries [10]. Object defect detection based on 3D point clouds usually includes methods based on standard model comparison [11][12][13][14] and methods based on point cloud geometric features [6,[15][16][17]. ...
... Cao et al. [7] completed the detection of rail defects by projecting the point cloud and fitting the model to the projected data. Zong et al. [9] use images to locate the defect, then maps the defect information to 3D space and marks it. Chu et al. [27] first used the images for defect detection, and the defective ROI(Region Of Interest) regions were reconstructed in 3D to evaluate the defects in detail. ...
Non‐destructive detection of wire bonding defects in integrated circuits (IC) is critical for ensuring product quality after packaging. Image‐processing‐based methods do not provide a detailed evaluation of the three‐dimensional defects of the bonding wire. Therefore, a method of 3D reconstruction and pattern recognition of wire defects based on stereo vision, which can achieve non‐destructive detection of bonding wire defects is proposed. The contour features of bonding wires and other electronic components in the depth image is analysed to complete the 3D reconstruction of the bonding wires. Especially to filter the noisy point cloud and obtain an accurate point cloud of the bonding wire surface, a point cloud segmentation method based on spatial surface feature detection (SFD) was proposed. SFD can extract more distinct features from the bonding wire surface during the point cloud segmentation process. Furthermore, in the defect detection process, a directional discretisation descriptor with multiple local normal vectors is designed for defect pattern recognition of bonding wires. The descriptor combines local and global features of wire and can describe the spatial variation trends and structural features of wires. The experimental results show that the method can complete the 3D reconstruction and defect pattern recognition of bonding wires, and the average accuracy of defect recognition is 96.47%, which meets the production requirements of bonding wire defect detection.
... For example, Lin [3] et al. utilized point cloud data of blade profiles to analyze and detect defects on the blade's surface, successfully identifying and locating the defects. Zong [4] et al. proposed a method that involves using a vision system to capture point cloud data of surface defects and quantifying this data. By establishing a point-toimage mapping relationship between points on the object's surface and those in color images, they achieved precise segmentation of the defect areas and calculated the 3D feature data of the defects. ...
Thin-walled parts are widely used in various fields of industrial production, but their inherent characteristics, such as low stiffness and susceptibility to deformation, make them prone to various defects during use. Thin-walled covering parts, which serve as thin shells or covers to protect other components, are extensively utilized in aerospace, automotive, and shipbuilding industries. The condition of these parts is directly related to the structural integrity, performance stability, and operational safety of the associated components and equipment. Due to their direct exposure to external environments, thin-walled covering parts are vulnerable to damage. Traditional defect detection methods, such as manual visual inspection and 2D imaging, have limitations in addressing these challenges. To improve the speed and accuracy of damage detection in thin-walled covering parts, this paper proposes a defect detection method based on point cloud data processing. This approach involves collecting point cloud data of the thin-walled parts, applying preprocessing techniques to remove noise, and utilizing feature extraction and machine learning algorithms to achieve automatic detection and classification of defects. The research results demonstrate that this method offers high accuracy and efficiency, effectively meeting the quality inspection needs for thin-walled parts in industrial production.
... In another approach, presented in [14], the detection of anomalies in images is based on the features extraction capabilities of convolutional neural networks. An interesting application to evaluate defects on the surface of the materials at the 3D level is presented in paper [15]. A DenseNet convolutional neural network is employed to identify defect types using a 3D point cloud of a designated defect area. ...
The algorithm and prototype presented in the article are part of a quality control system for plastic objects coming from injection-molding machines. Some objects contain a flaw called inclusion, which is usually observed as a local discoloration and disqualifies the object. The objects have complex, irregular geometry with many edges. This makes inclusion detection difficult, because local changes in the image at inclusions are much less significant than grayscale changes at the edges. In order to exclude edges from calculations, the presented method first classifies the object and then matches it with the corresponding mask of edges, which is prepared off-line and stored in the database. Inclusions are detected based on the analysis of local variations in the surface grayscale in the unmasked part of the image under inspection. Experiments were performed on real objects rejected from production by human quality controllers. The proposed approach allows tuning the algorithm to achieve very high sensitivity without false detections at edges. Based on input from the controllers, the algorithm was tuned to detect all the inclusions. At 100% recall, 87% precision was achieved, which is acceptable for industrial applications.
... The results indicated that the CNN model could be used in realtime detection, yet the recognition of edge defects is less satisfactory. Zong et al. [20] introduced an intelligent automatic three-dimensional defect detection system, which transformed the defect detection from simple image detection to 3D detection. The results show that the system can be used for quantitative three-dimensional estimation and feature classification of material surface defects. ...
Workpiece surface defect detection is an indispensable part of intelligent production. The surface information obtained by traditional 2D image detection has some limitations due to the influence of environmental light factors and part complexity. However, the digital twin model has the characteristics of high fidelity and scalability, and the digital twin surface can be obtained by a device with a scanning accuracy of 0.02mm to achieve the representation of the real surface of the workpiece. The surface defect detection system for digital twin models is proposed based on the improved YOLOv5 model in this paper. Firstly, the digital twin model of the workpiece is reconstructed by the point cloud data obtained by the scanning device, and the surface features with defects are captured. Subsequently, the training dataset is calibrated based on the defect surface, where the defect types include Inclusion, Perforation, pitting surface and Rolled-in scale. Finally, the improved YOLOv5 model with CBAM mechanism and BiFPN module was used to identify the surface defects of the digital twin model and compare it with the original YOLOv5 model and other common models. The results show that the improved YOLOv5 model can realize the identification and classification of surface defects. Compared with the original YOLOv5 model, the mAP value of the improved YOLOv5 model has increased by 0.2%, and the model has high precision. On the basis of the same data set, the improved YOLOv5 model has higher recognition accuracy than other models, improving 11.7%, 3.4%, 6.2%, 33.5%, respectively. As a result, this study provides a practical and systematic detection method for digital twin model surface during the intelligent production process, and realizes the rapid screening of the workpiece with defects.
... With the rapid development of computer hardware and vision algorithms, automatic defect detection by computer vision has been paid more attention [1][2][3]. Yun et al. [4] inspected cracks in the steel plate surface with images acquired by the visual system. Li et al. [5] used visual saliency to detect patterned and non-patterned fabric flaws by fabric images. ...
Occluded target location is a challenge in three dimensions defect detection of complicated devices, which easily leads to the wrong location. In this paper, we propose an occluded target location method based on the most salient geometric primitive and its invariant spatial topology relationship with the occluded target. First, several 3D primitives are extracted by using the Random Sample Consensus (RANSAC) method. Then these primitive’s saliency is calculated with the proposed primitive saliency calculation algorithm, and the most salient primitive which has the maximum saliency value is selected. At last, the occluded target location is determined by the invariant spatial topology relationship between the most salient primitive and the occluded target. The occluded targets location experiment was completed on two 3D datasets, the occluded targets were all accurately located with the proposed method. The template matching method was also carried out to compare with our proposed method, the results showed that our method exhibited better performance in serious occlusion and noises and efficiency.
... Concerning the fact that the boundary of an object in the 2D image is clearer and easier to extract, but the image lacks spatial information, recently, researchers have attempted to boost spatial defect detection by using a concurrent 2D image and 3D PC. They perform defect detection in the 2D space and then add the spatial information of the detected defects using the 3D information of the PC [46][47][48]. However, they were limited to specific kinds of PC, such as dense density points, points acquired in grid form, or requiring careful and tedious calibration of multiple devices. ...
Closed-loop manufacturing is crucial in Industry 4.0, since it provides an online detection–correction cycle to optimize the production line by using the live data provided from the product being manufactured. By integrating the inspection system and manufacturing processes, the production line achieves a new level of accuracy and savings on costs. This is far more crucial than only inspecting the finished product as an accepted or rejected part. Modeling the actual surface of the workpiece in production, including the manufacturing errors, enables the potential to process the provided live data and give feedback to production planning. Recently introduced “skin imaging” methodology can generate 2D images as a comprehensive digital twin for geometric deviations on any scanned 3D surface including analytical geometries and sculptured surfaces. Skin-Image has been addressed as a novel methodology for continuous representation of unorganized discrete 3D points, by which the geometric deviation on the surface is shown using image intensity. Skin-Image can be readily used in online surface inspection for automatic and precise 3D defect segmentation and characterization. It also facilitates search-guided sampling strategies. This paper presents the implementation of skin imaging for primary engineering surfaces. The results, supported by several industrial case studies, show high efficiency of skin imaging in providing models of the real manufactured surfaces.
... Stereo reconstruction is a widely used technique that can generate 3D reconstructions for quantitative assessment of defects [7][8][9][10][11]. Integrating multiple cameras makes it possible to directly obtain the physical dimensions of the defects from the reconstructions. ...
... The defected area of the image is then extracted using the Canny algorithm and the AND logical operation. The texture, edge, and HOG [8] features are combined to extract the features of the defected area in the image. Finally, a support vector machine optimized using particle swarm optimization is used to identify and categorize the defects in the images automatically. ...
Detecting the lithium battery surface defects is a difficult task due to the illumination reflection from the surface. To overcome the issue related to labeling and training big data by using 2D techniques, a 3D point cloud-based technique has been proposed in this paper. The 3D point cloud-based defect detection of lithium batteries used feature-based techniques to downscale the point clouds to reduce the computational cost, extracting the normals of the points and calculating their differences to detect the defects of the battery which assure the quality of the product. This paper offers a novel strategy using 3D point clouds to get beyond the labeling and training challenges involved with conventional 2D approaches. This 3D point cloud-based approach for lithium battery fault identification makes use of feature-based methods to improve the point cloud data and lessen the computing burden. In our work, the experiments show that the feature-based technique precisely detects the affected surface of the battery.
... Based on pseudo-random arrays of coded surface structured light, Lin et al [17] proposed a technique for coded surface structured light, which requires shape and color decoding. Also based on DLP, Zong et al [18] used four cameras and two DLP projectors to measure the geometric parameters of color patch defects, it demands calibrating the parameters of each camera in the sensor, and the calibration procedure is tedious. Besides, custom color full Stokes polarization cameras were applied to obtain color and polarization data in Liang's research [19]. ...
To obtain geometric information and color texture simultaneously, a surface structured light sensor consisting of a monochrome camera, a color camera, and a projector is proposed. The sensor uses a color camera to acquire surface color information while using it as a geometric measurement unit and matching with the monochrome camera to obtain geometric information. Due to the Bayer array and demosaicing algorithm of the color camera, pixel RGB components are always coupled with interference from other channels. However, existing color de-crosstalk in reconstruction is merely applied to the decoupling of color composite patterns, ignoring the intensity errors present in color fringe patterns under monochrome illumination. In our sensor, de-crosstalk of monochromatic patterns is considered to guarantee the reconstruction accuracy. The high-accuracy measurement of the sensor is validated by reconstructing standard steps, yielding a mean absolute error of 0.008 mm for distance measurements. In addition, the reconstruction experiment of a terracotta warrior verifies that the proposed sensor has potential application in the digital preservation of cultural relics.
... Automation and pipeline processing are constantly emerging in industrial manufacturing. As an important means of obtaining object surface information in industrial manufacturing, three-dimensional (3D) reconstruction is widely used in the fields of production line quality inspection [1], intelligent machine vision [2,3], and cultural relic restoration [4]. Precise measurement of 3D topography for micro-components is one of the most considered issues. ...
A line laser scanning microscopy system with a larger depth of field based on the Scheimpflug principle is proposed for high-resolution surface topography restoration and quantitative measurement on miniature non-transparent samples. An imaging model based on the Scheimpflug principle is established, and a calibration method without system parameters is derived, which is further extended to a microscopic system. The measuring range of the system is 5 m m × 4 m m × x m m , where x is the movement distance of the displacement stage. In the z-axis direction, the relative error of measurement is about 1% when z is of the millimeter level and less than 7% when z is of the micron level, and the spatial resolution is better than 3.8 µm. In the y-axis direction, the relative error of measurement is less than 5%. Finally, three-dimensional scanning of two samples with different surfaces is carried out to verify the feasibility of the system. The experimental results show that our system has the capability of high-resolution topography restoration and can be applied in industrial production scenarios such as automatic measurement and intelligent identification.
... Point cloud segmentation has become a key research topic in the field of industrial big data analysis, with the wide application of 3-D sensing technologies such as laser scanning and depth camera in the era of Industry 4.0. Its objective is to divide the point cloud into several regions with different properties [6], which has also been widely applied in many other fields such as autonomous driving [7], robotics [8], and product quality inspection [9]. ...
Point cloud is widely available in the manufacturing system with the continuous development of three-dimensional (3D) sensors. Accurate point cloud segmentation can automatically identify different components during the manufacturing process, which is essential to the quality assurance of the final product. However, the existing methods for point cloud segmentation fail to accurately identify the boundary of each region, which decays the quality of many manufacturing operations (i.e., welding). In this paper, a point cloud segmentation model, AR-PointNet, is proposed to improve the segmentation accuracy, especially on the regions’ boundaries. More specifically, the PointNet is used as the backbone, and the novel channel and spatial statistical feature attention modules are proposed and combined to enhance the features along the regions’ edges. In addition, context features among different regions in the point cloud are further extracted to consider the interactions among components. The experiments are conducted on the ShapeNet dataset and the real-scanned dataset of manufactured products, respectively. The results demonstrated that (1) on the ShapeNet dataset, the proposed method outperforms the state-of-the-art segmentation models. The mean Intersection-over-Union (mIoU) is selected as the evaluation metric to indicate the segmentation accuracy. The proposed method improves mIoU by 2.3% compared with its backbone (PointNet); (2) on the simulated and real-scanned sheet metal part, it successfully recognizes the boundaries of components in the sheet metal part and significantly improves the recognition accuracy. Boundary Mean Accuracy (BMA) is defined as the evaluation metric to indicate the boundary recognition accuracy. Results of the simulated data and the real-scanned data show that the proposed method improves the BMA by 4.4% and 3.6%, respectively, compared to its backbone (PointNet).
... The configurations already evaluated by this strategy are multiple; some of them are based on the use of a turntable to apply the SfM algorithm. Other fusion monovision, with stereo vision, different kinds of cameras (monochromatic, color, depth, thermal), and also could include a projector for structured lighting of the object's surface [163,164]. Classical configurations are shown in Fig. 9. In the configuration shown in Fig. 9(a) object and the cameras are at a fixed place, in this configuration the cameras are around the object. ...
The measurement of 3D spatial coordinates for model reconstruction through artificial machine vision systems based on optical sensors and the corresponding signal processing associated with algorithms is a powerful module for cyber systems. It provides an efficient, functional, and intelligent vision and data information of the objects and scenes under observation for decisions, as well as for remote environment interactivity and autonomous robot systems actuation. Over the past 20 years, the artificial machine vision has benefited from emerging technology and a promising huge potential is peeking out, but also technical difficulties achieving customized and true commercial applications. This paper reviews the research progress, trends, and future research directions; the state-of-the-art of topics related to the 3D spatial measurement for model reconstruction. It classifies the technology by its fundamental principles and applications, to construct an outlook about its advantages, disadvantages, and challenges.
... To solve this problem, both multi-sensor assisted positioning and machining error compensation are taken as effective solutions which involve in lidar ranging, structured light scanning, and digital optical micro-mirror processing, etc. As a consequence, the poor positioning error caused by unclearness due to camera imaging details can be compensated within existing dynamic ranges [13][14][15][16], which effectively improves the positioning accuracy of the vision-guided robot. The above methods, however, result in small imaging field-of-view and high requirement for scanning and measurement conditions, which restricts the robot positioning. ...
The machining quality of large and complex components highly depends on the positioning accuracy of the vision guided robot. In order to solve the problems of limited imaging field-of-view and dynamic range of existing parallel binocular cameras, a convergent binocular vision algorithm under extended imaging dynamic range is proposed to guide the machining robot for precise positioning. Specifically, synthesize high dynamic range images combined with sigmoid function model, and chromaticity space color correction algorithm introduced for adaptive correction of image chromaticity information. The pose between the convergent binocular camera and the robot is calibrated to correct the three-dimensional coordinates of the target workpiece CAD model. The experimental results indicate that the high dynamic images synthesized by the proposed algorithm have clear surface details and balanced colors. This work further enhances the positioning accuracy of the machining robot to the target workpiece which is placed in arbitrary pose within the field-of-view.
... In the field of defects detection, major efforts have been devoted to 2D image-based object detection or segmentation methods [8][9][10]. However, 2D image-based results cannot directly provide the depth information of defects, so subsequent quantitative filling processes cannot be carried out effectively. ...
Surface defects of fiber-reinforced resin matrix composites (FRRMCs) adversely affect their appearance and performance. To accurately and efficiently detect the three-dimensional (3D) surface defects of FRRMCs, a novel lightweight and two-stage semantic segmentation network, i.e., Mask-Point, is proposed. Stage 1 of Mask-Point is the multi-head 3D region proposal extractors (RPEs), generating several 3D regions of interest (ROIs). Stage 2 is the 3D aggregation stage composed of the shared classifier, shared filter, and non-maximum suppression (NMS). The two stages work together to detect the surface defects. To evaluate the performance of Mask-Point, a new 3D surface defects dataset of FRRMCs containing about 120 million points is produced. Training and test experiments show that the accuracy and the mean intersection of union (mIoU) increase as the number of different 3D RPEs increases in Stage 1, but the inference speed becomes slower when the number of different 3D RPEs increases. The best accuracy, mIoU, and inference speed of the Mask-Point model could reach 0.9997, 0.9402, and 320,000 points/s, respectively. Moreover, comparison experiments also show that Mask-Point offers relatively the best segmentation performance compared with several other typical 3D semantic segmentation networks. The mIoU of Mask-Point is about 30% ahead of the sub-optimal 3D semantic segmentation network PointNet. In addition, a distributed surface defects detection system based on Mask-Point is developed. The system is applied to scan real FRRMC products and detect their surface defects, and it achieves the relatively best detection performance in competition with skilled human workers. The above experiments demonstrate that the proposed Mask-Point could accurately and efficiently detect 3D surface defects of FRRMCs, and the Mask-Point also provides a new potential solution for the 3D surface defects detection of other similar materials
... On this basis, a 3D point cloud based on destructive testing technology provides a fast, convenient, and applicable solution for target detection and surface 3D reconstruction (Jiang et al., 2018;Wu et al., 2019;Das Choudhury et al., 2020). It has been successfully used to detail road surface defects, composite wrinkle defects, and seamless steel pipe wear defects (Zhang et al., 2018;Hu and Furukawa, 2020;Zong et al., 2021). The 3D point cloud technology has an excellent performance in object detection (Ying et al., 2013). ...
The dents and cracks of cabbage caused by mechanical damage during transportation have a direct impact on both commercial value and storage time. In this study, a method for surface defect detection of cabbage is proposed based on the curvature feature of the 3D point cloud. First, the red-green-blue (RGB) images and depth images are collected using a RealSense-D455 depth camera for 3D point cloud reconstruction. Then, the region of interest (ROI) is extracted by statistical filtering and Euclidean clustering segmentation algorithm, and the 3D point cloud of cabbage is segmented from background noise. Then, the curvature features of the 3D point cloud are calculated using the estimated normal vector based on the least square plane fitting method. Finally, the curvature threshold is determined according to the curvature characteristic parameters, and the surface defect type and area can be detected. The flat-headed cabbage and round-headed cabbage are selected to test the surface damage of dents and cracks. The test results show that the average detection accuracy of this proposed method is 96.25%, in which, the average detection accuracy of dents is 93.3% and the average detection accuracy of cracks is 96.67%, suggesting high detection accuracy and good adaptability for various cabbages. This study provides important technical support for automatic and non-destructive detection of cabbage surface defects.
... Many traditional measurement techniques have been applied to two-dimensional (2D) or three-dimensional (3D) surface strain estimation of rock, including strain gauges (Tsai et al. 1996), particle image velocimetry (PIV) (Sheng et al. 2021), and digital image correlation (DIC) (Pan et al. 2009a, b;Zong et al. 2021). However, surface analysis is not sufficient for understanding the internal failure mechanism of rocks because of their geometric complexities and heterogeneity. ...
The digital volume correlation (DVC) method combined with X-ray computed tomography has evolved into a powerful technique for the interior deformation measurement of sandstone. The accuracy and precision of DVC measurements are influenced by the subset size and initial displacement estimate. In routine implementation, the subset size, which generally is not exact, especially for the complex heterogeneous deformation in sandstone, has to be fixed and defined by users according to prior experience. Furthermore, nearly all the existing methods for estimating the initial displacement estimate the displacement of a single calculation point and seldom estimate the initial displacements of all calculation points at once. Finding the optimal subset size and a reliable displacement initial displacement for all calculation points is, therefore, highly desirable to achieve accurate measurement of the internal deformation field. This study propose an adaptive DVC by introducing a three-dimensional scale-invariant feature transform (3D SIFT) as an effective tool for determining the subset size and initial displacement estimate simultaneously. The superiority of the proposed adaptive DVC over the classic one using the fixed subset size with regard to measurement accuracy is verified using numerical experiments with different-sized granlues, which can simulate the microstructure of sandstone. The credibility of the initial displacement estimate is also analyzed through the true Brazilian test of red sandstone.
... Chu and Wang [15] used image processing to automatically extract weld joint contours and feature points, measure weld size, and detect defects. Zong et al. [16] designed a stereo vision system based on speckle projection, reconstructed the 3D point cloud on the surface of the object to be detected, and established a mapping relationship with the color image to segment the defect area to extract the defect feature parameters, and used the convolutional neural network Densenets to identify the type of defect. ...
With the rapid development of welding automation, manual visual detection and manual measurement of weld appearance contour are gradually transitioning to automated detection. In this paper, a compound 3D vision system of projection structured light and shape from shading is constructed, which solves the problem that the current laser scanning vision method is not applicable in the occasions with high beat requirements. The projection structured light vision system is encoded by sinusoidal fringe to realize the high-precision 3D recovery of highly reflective metal surface. The shape from shading method can obtain the high-definition gradient information of the weak fluctuation of the weld surface and the Lambert reflection theorem in this method is used to solve the problem that the reflectivity is difficult to obtain. Using the above compound vision system, taking the weld of aluminum alloy variable polarity plasma arc (VPPA) welding as an example, the relevant image processing algorithm has been developed to accurately realize the measurement of contour size and the extraction of contour defects on the face and back of the weld. In view of the features that the 3D boundary features of face of weld in VPPA welding are not obvious, the shadow recovery shape method is used to obtain the gradient image of the metal surface. Combined with the fish scale information on the weld surface, the reliable extraction of the weld boundary is realized.
... On this basis, an additional color camera is added to establish a matching relationship between the color information and the point cloud data. Zong et al. [18] applied texture mapping technology to quantitative evaluation of surface defects and proposed a method of quantitative measurement and detection of defects based on laser speckle 3D reconstruction and neural network defect extraction. However, the current related research mainly focuses on the reconstruction of relatively large scenes, and pays less attention to the detailed feature location. ...
This paper proposes a 3D reconstruction and positioning method based on a single detecting camera and geometric constraint to evaluate the defect quantitatively. We realize the camera orientation through datum point coordinates acquired by a photogrammetry system and then solve the equation of the defect detection camera model and surface constraints to reconstruct the color point cloud in 3D space. To realize the reconstruction and orientation of defects on objects without marker points, the system is operated by a repetitive moving of robotic arms. Finally, the experiment achieves the reconstruction of three types of products with marker points and one of the same types of objects without marks. The experiment shows that this method can limit the accuracy of surface feature positioning to 0.7 mm, and relative accuracy of the area calculation to 0.125%.
As a critical engine component, the appearance quality of piston rings directly affects engine performance and lifespan. Subtle defects like minor chipping, cracks, and scratches can significantly impact engine safety and durability. Traditional 2D image processing and AI-based methods struggle to detect these defects, being prone to interference and resulting in high false positive and negative rates. This paper introduces a 3D vision-based detection method, which captures detailed surface morphology to improve defect detection accuracy. Experimental results demonstrate that this approach effectively reduces false positives and negatives, significantly enhancing detection accuracy.
Wooden plank images in industrial measurements often contain numerous textureless areas. Furthermore, due to the thin plate structure, the three-dimensional (3D) disparity of these planks is predominantly confined to a narrow range. Consequently, achieving accurate 3D matching of wooden plank images has consistently presented a challenging task within the industry. In recent years, deep learning has progressively supplanted traditional stereo matching methods due to its inherent advantages, including rapid inference and end-to-end processing. Nonetheless, the acquisition of datasets for stereo matching networks poses an additional challenge, primarily attributable to the difficulty in obtaining accurate disparity data. Thus, this paper presents a novel stereo matching method incorporating three key innovations. Firstly, an enhanced gated recurrent unit network is introduced, accompanied by a redesigned structure to achieve higher matching accuracy. Secondly, an efficient preprocessing module is proposed, aimed at improving the algorithm’s efficiency. Lastly, in response to the challenges posed by datasets acquisition, we innovatively employed image simulation software to obtain a high-quality simulated dataset of wooden planks. To assess the feasibility of our approach, we conducted both simulated and real experiments. The experiments results clearly exhibit the superiority of our method when compared to existing approaches in terms of both stability and accuracy. In the simulation experiment, our method attained a bad1.0 score of 2.1% (compared to the baseline method’s 9.76%); In the real experiment, our method achieved an average error of 0.104 mm (compared to the baseline method’s 0.268 mm). It is worth noting that our study aims to address the challenge of acquiring datasets for deep learning and bridging the gap between simulated and real data, resulting in increased applicability of deep learning in more industrial measurement domains.
Magnetic particle testing (MT) has the advantages of high sensitivity and adaptability. However, during the formation of magnetic particle indication (MPI) corresponding to surface opening defects, the defects are filled with magnetic particles, resulting in the weakening of the leakage magnetic field (LMF), a phenomenon that has been ignored in existing studies. In order to enhance the visual feature of MPI, this paper proposes an isolation strip-based magnetic particle testing method (ISMT), which can avoid the attenuation of LMF. Meanwhile, ISMT can isolate the surface conditions of the specimen to avoid the formation of pseudo-magnetic particle indication (PMPI). The enhancement principle of ISMT is analyzed by simulation. Comparative experiments between ISMT and traditional MT were conducted to verify the visual enhancement effect of ISMT on defects of multiple depths. The results show that the enhancement effect of ISMT is 1.51 times, 1.36 times, and 1.08 times for the standard test shims of 60/100, 30/100, and 15/100, respectively. In addition, the advantage of ISMT in detecting larger defects is verified by experiments on hole defects of different sizes, and the isolation effect against surface oil stains is discussed.
We introduce a method for automatic corrosion detection based on the application of machine learning techniques to 3D point cloud data generated by a LIDAR sensor. In our approach a point is assigned one of the considered class labels (healthy, stain, weld or rust) by processing its feature vector with a cascade of three binary classifiers. The effectiveness of the proposed system is demonstrated through a case study on three different bulkheads in the hold of a merchant ship. The experimental results show that the corrosion detection rate is improved by combining colour and local geometry features.
In the aerospace sector, the identification and extraction of surface defects on composite parts is particularly important as they can have a very serious impact. This paper proposes a new extraction method for surface defects of composite materials, where 3D point clouds on the surface of composite workpieces are directly manipulated to finally extract a point cloud of defects containing both morphological and spatial location information. The specific operation explores the influencing factors in the formation and extraction process of composite surface defects and summarizes them into five categories: material type, fiber direction, curvature algorithm, neighborhood size, and filtering threshold. The Multilayer Perceptron‐Moth Flame algorithm two‐stage network model was constructed, which can reveal the relationship between these five types of influencing factors and the formation and extraction of surface defects in composites in the forward direction with an accuracy of 93.07%. It can also achieve optimal parameter recommendation in the reverse direction.
Highlights
Automatic identification and extraction of defects on the surface of composite workpieces is achieved based on 3D point cloud and curvature calculation.
The forward direction reveals the relationship between the surface defect formation process and the material type and fabric direction.
The reverse direction can automatically select the best curvature algorithm, neighborhood size, and filtering condition parameters.
Build a Multilayer Perceptron‐Moth Flame Algorithm network structure based on multilayer perceptron and moth‐flame algorithm for model training of defect extraction, with an accuracy of 93.07%.
In recent years, machine vision has played an important role in product surface quality detection. The promotion and use of this technology have largely avoided the subjectivity caused by human detection and improved detection efficiency and accuracy. Different from the image data commonly used in machine vision, point cloud can describe the spatial structure of an object, provide more information than image data, and have the ability to expand the data to build multi-dimensional data models. Due to the strong anti-interference ability of point cloud sensors and the high accuracy of three-dimensional positioning information point cloud, nondestructive testing technology based on point cloud has received more and more attention. This paper summarizes the research progress of product surface quality detection methods based on 3D point cloud in recent years. According to different data processing methods, the detection research is divided into five categories: based on point cloud contour, based on local geometric feature, based on template matching, based on multimodal point cloud, and based on deep learning. The five methods are reviewed and summarized respectively. Finally, the key problems of each detection method and the future trend of product surface quality detection technology based on 3D point cloud are discussed.
Galvanized steel sheets have been widely used due to their excellent corrosion resistance and sound formability, but their surface defects can severely affect their performance, so it is of immense significance to identify them effectively and accurately. This paper selected the images of surface defects of galvanized steel sheets as the research objects, investigated the segmentation of surface defects under complex texture backgrounds, and offered an optimized two-dimensional asymmetric Tsallis cross entropy image segmentation algorithm based on Chaotic Bee Colony Algorithm. On the basis of Tsallis cross entropy threshold segmentation algorithm, a simpler expression was adopted to define the asymmetric Tsallis cross entropy in order to reduce its calculation complexity; chaotic algorithm and Artificial Bee Colony Algorithm were combined to construct Chaotic Bee Colony Algorithm, so that the optimal threshold of Tsallis entropy could be searched quickly. The experimental results showed that compared with other commonly used threshold segmentation algorithms, the algorithm proposed by this paper could rapidly and effectively segment defect targets, a more suitable method of detecting surface defects for factories with a rapid production pace.
As a reliable and high-precision three-dimensional (3D) vision method, line laser structured light vision has been widely used in the engineering field. However, to obtain overall 3D shape of target to be measured, scanning mechanisms need to be added. Multi-line laser structured light vision can obtain multi-line array 3D shape of workpiece without scanning; however, the encoding of multi-line lasers is a difficult task. This paper proposes a composite vision system, which is composed of time-of-flight(TOF) camera, color camera and vertical cross multi-line laser structured light. Color camera and multi-line laser form a multi-line array 3D vision system, while the coded signal of multi-line laser stripes is provided by TOF camera and corresponding coding algorithm is also proposed in this paper. Based on this system, automatic positioning of welding seam is realized by registration of CAD data and multi-line laser 3D data. The registration is divided into two steps, which are coarse registration and precise registration. Coarse registration is performed by using CAD point cloud data and original point cloud data of multi-line laser; while precise registration is performed with CAD welding seam point cloud data and welding seam point cloud data of multi-line laser. This paper also analyzes accuracy of vision system based on constructing structured light vision model. The measurement accuracy of vision system in x, y and z directions is 0.25, 0.06 and 0.29 mm respectively. This composite vision system can realize precise positioning of target within a single exposure time, which provides possibility for positioning under movement.
To achieve high-precision and high-efficiency 3D scanning of industrial parts, this research proposes an automated 3D scanning system based on path planning algorithms, in which the 3D scanning sensor is realized based on the principle of raster projection. The proposed automated scanning system is composed of a robotic arm, a 3D scanner, and a turntable to achieve a complete 3D scanning of the object to be measured. More precisely, this paper proposes a virtual 3D scanning principle, which can accurately determine whether a certain point on the CAD model can be scanned by a viewpoint. Further, this paper proposes a path planning algorithm based on the virtual 3D scanning principle, which can calculate the optimal scanning path required to scan the object according to the CAD model of the scanned object. Furthermore, to ensure the accuracy of the system, a high-precision system calibration method is proposed to establish the accurate positional relationship between various parts of the system. Firstly, this research proposes a camera calibration algorithm based on close-range photogrammetry, which can calculate the high-precision interior and exterior orientation parameters of the camera, and apply the method of calculating camera external parameters to the hand-eye calibration, making the accuracy of hand-eye calibration is also improved accordingly. In addition, this research proposes a photogrammetry-based projector calibration algorithm to determine the positional relationship between the projector and the camera in the 3D scanner, and a turntable calibration method to determine the positional relationship between the rotation center of the turntable and the camera. Eventually, according to different usage requirements, two 3D scanning data splicing methods are proposed. One is a high-efficient automatic splicing method based on the results of system calibration without pasting mark points, and the other is a high-precision data splicing method based on photogrammetry. Experimental results demonstrate that this system has higher accuracy and better performance in system calibration, scanning path calculation, and multi-view scanning data splicing.
Accurate and quick detection has a significant bearing on overall productivity of remanufacture. 3D scanning technologies have been widely applied in defects detection by comparing the damaged model with the nominal model. In this process, a huge amount of point cloud data is required to ensure detection accuracy whereas resulting in large storage space and long processing time of detection. This paper proposed an efficient two-step method based on octree to detect defects accurately and quickly for remanufacturing. In this method, the damaged point cloud and the nominal point cloud are first registered. Then a two-step detection approach is developed to extract the surface defects, coarse detection and detailed extraction, where the octree method is applied to create an effective topology of discrete points and perform the Boolean operation for defects extraction. In coarse detection, rough location and size information of the defects are acquired from the whole point cloud data. Based on coarse detected boundary box containing defects, the detailed extraction step is applied to extract corresponding defects shape accurately. The feasibility of proposed method was validated by using a case to detect defects of a damaged turbine blade and the detection results can be used to generate restoration tool path. The results show that the proposed method outperforms state-of-art defects detection methods, which can reduce time by 74.03% and reduce error by 36.86%, respectively.
The need for improved quality control in industry makes object detection crucial. This work addresses the challenging problem of subsurface defect detections using a combination of active thermography and deep learning. The novel contribution of this work is to pose the problem as one of object detection rather than semantic segmentation or classification. The images used as input for the deep learning algorithms are three-channel color images obtained using principal component thermography (PCT). The use of these images improves the signal-to-noise ratio (SNR). A framework to label ground truths automatically is also created. The most widely used deep learning detector algorithms were evaluated, and the fifth version of you only look once (YOLO), i.e., YOLOv5, was selected because of its excellent average precision (AP) and its low inference time. The resulting combination of this algorithm and active thermography is effective and accurate in detecting subsurface defects.
Recent work has shown that convolutional networks can be substantially deeper, more accurate and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper we embrace this observation and introduce the Dense Convolutional Network (DenseNet), where each layer is directly connected to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections, one between each layer and its subsequent layer (treating the input as layer 0), our network has L(L+1)/2 direct connections. For each layer, the feature maps of all preceding layers are treated as separate inputs whereas its own feature maps are passed on as inputs to all subsequent layers. Our proposed connectivity pattern has several compelling advantages: it alleviates the vanishing gradient problem and strengthens feature propagation; despite the increase in connections, it encourages feature reuse and leads to a substantial reduction of parameters; its models tend to generalize surprisingly well. We evaluate our proposed architecture on five highly competitive object recognition benchmark tasks. The DenseNet obtains significant improvements over the state-of-the-art on all five of them (e.g., yielding 3.74% test error on CIFAR-10, 19.25% on CIFAR-100 and 1.59% on SVHN).
The ImageNet Large Scale Visual Recognition Challenge is a benchmark in
object category classification and detection on hundreds of object categories
and millions of images. The challenge has been run annually from 2010 to
present, attracting participation from more than fifty institutions.
This paper describes the creation of this benchmark dataset and the advances
in object recognition that have been possible as a result. We discuss the
challenges of collecting large-scale ground truth annotation, highlight key
breakthroughs in categorical object recognition, provide detailed a analysis of
the current state of the field of large-scale image classification and object
detection, and compare the state-of-the-art computer vision accuracy with human
accuracy. We conclude with lessons learned in the five years of the challenge,
and propose future directions and improvements.
In this study, a machine vision system is developed to achieve fabric inspection and defect classification processes automatically. The system consists of an image acquisition hardware and an image processing software. A simple and portable system was designed so that it can be adapted easily to all types of the fabric inspection machines. The software of the system consists of defect detection and classification algorithms. The defect detection algorithm is based on wavelet transform, double thresholding binarization, and morphological operations. It was applied real time via a user interface prepared by using MATLAB® program. The defect classification approach is based on gray level co-occurrence matrix and feed forward neural network. Five commonly occurring defect types, warp lacking, weft lacking, soiled yarn hole, and yarn flow, were detected and classified. The defective and defect-free regions of the fabric were detected with an accuracy of 93.4% and the defects are classified with 96.3% accuracy rate.
The state of the art in machine vision inspection and a critical overview of real-world applications are presented in this paper. Two independent ways to classify applications are proposed, one according to the inspected features of the industrial product or process and the other according to the inspection independent characteristics of the inspected product or process. The most contemporary software and hardware tools for developing industrial vision systems are reviewed. Finally, under the light of recent advances in image sensors, software and hardware technology, important issues and directions for designing and developing industrial vision systems are identified and discussed.
In recent years, there is an increased need for quality control in the manufacturing sectors. In the steel making, the rolling operation is often the last process that significantly affects the bulk microstructure of the steel. The cost of having defects on rolled steel is high because it takes more than 5000 KW-Hr to produce a ton of steel. Early detection of defects can reduce product damage and manufacturing cost. This paper describes a real-time visual inspection system that uses support vector machine to automatically learn complicated defect patterns. Based on the experimental results generated from over one thousand images, the proposed system is found to be effective in detecting steel surface detects. The speed of the system for feature extraction and defect detection is less than 6 msec per one-megabyte image.
In digital image correlation (DIC), to obtain the displacements of each point of interest, a correlation criterion must be predefined to evaluate the similarity between the reference subset and the target subset. The correlation criterion is of fundamental importance in DIC, and various correlation criteria have been designed and used in literature. However, little research has been carried out to investigate their relations. In this paper, we first provide a comprehensive overview of various correlation criteria used in DIC. Then we focus on three robust and most widely used correlation criteria, i.e., a zero-mean normalized cross-correlation (ZNCC) criterion, a zero-mean normalized sum of squared difference (ZNSSD) criterion, and a parametric sum of squared difference ( PSSD a b ) criterion with two additional unknown parameters, since they are insensitive to the scale and offset changes of the target subset intensity and have been highly recommended for practical use in literature. The three correlation criteria are analyzed to establish their transversal relationships, and the theoretical analyses clearly indicate that the three correlation criteria are actually equivalent, which elegantly unifies these correlation criteria for pattern matching. Finally, the equivalence of these correlation criteria is further validated by numerical simulation and actual experiment.
We present a fast and memory efficient algorithm that generates a manifold triangular mesh S with or without boundary passing through a set of unorganized points P⊂ℛ3 with no other additional information. Nothing is assumed about the geometry or topology of the sampled manifold model, except for its reasonable smoothness. The speed of our algorithm is derived from a projection-based approach we use to determine the incident faces on a point. Our algorithm has successfully reconstructed the surfaces of unorganized point clouds of sizes varying from 10,000 to 100,000 in about 3-30 seconds on a 250 MHz, R10000 SGI Onyx2. Our technique can be specialized for different kinds of input and applications. For example, our algorithm can be specialized to handle data from height fields like terrain and range scan, even in the presence of noise. We have successfully generated meshes for range scan data of size 900,000 points in less than 40 seconds.
This paper proposes a two-phase scheme for removing salt-and-pepper impulse noise. In the first phase, an adaptive median filter is used to identify pixels which are likely to be contaminated by noise (noise candidates). In the second phase, the image is restored using a specialized regularization method that applies only to those selected noise candidates. In terms of edge preservation and noise suppression, our restored images show a significant improvement compared to those restored by using just nonlinear filters or regularization methods only. Our scheme can remove salt-and-pepper-noise with a noise level as high as 90%.
Multilayer neural networks trained with the back-propagation
algorithm constitute the best example of a successful gradient based
learning technique. Given an appropriate network architecture,
gradient-based learning algorithms can be used to synthesize a complex
decision surface that can classify high-dimensional patterns, such as
handwritten characters, with minimal preprocessing. This paper reviews
various methods applied to handwritten character recognition and
compares them on a standard handwritten digit recognition task.
Convolutional neural networks, which are specifically designed to deal
with the variability of 2D shapes, are shown to outperform all other
techniques. Real-life document recognition systems are composed of
multiple modules including field extraction, segmentation recognition,
and language modeling. A new learning paradigm, called graph transformer
networks (GTN), allows such multimodule systems to be trained globally
using gradient-based methods so as to minimize an overall performance
measure. Two systems for online handwriting recognition are described.
Experiments demonstrate the advantage of global training, and the
flexibility of graph transformer networks. A graph transformer network
for reading a bank cheque is also described. It uses convolutional
neural network character recognizers combined with global training
techniques to provide record accuracy on business and personal cheques.
It is deployed commercially and reads several million cheques per day
Defect inspection of integrated circuits (ICs) solder joints is a long-standing task. This paper proposes a novel IC solder joint inspection method based on the idea of robust principle component analysis (RPCA), which is a celebrated optimization technique. To the best of our knowledge, this is the first time IC solder joint inspection is formulated as an optimization problem. In particular, we model an IC solder joint image as observed data corrupted by gross errors. According to the RPCA theory, the observed data can be approximately decomposed into a low-rank component and an error component. In this paper, an appearance model of qualified IC solder joints is defined for IC solder joint inspection based on RPCA. Then, a defect score is defined based on the appearance model and is used to evaluate the quality of IC solder joints. Meanwhile, location prior knowledge related to human perception is incorporated in the defect score to further evaluate the defects of IC solder joints. Finally, a simple discrimination scheme is proposed to inspect IC solder joints. Experimental results indicate that the proposed method is superior to the existing methods in inspection performance.
In the field of automatic optical inspection (AOI), defect recognition for an integrated circuit (IC) solder joint is a long-standing task. Inspired by a visual background extraction (ViBe) algorithm, an object detection method in computer vision, we propose a new inspection method for IC solder joints with an improved ViBe algorithm. To the best of our knowledge, we are the first to consider the defect inspection problem as an object detection problem. We build a solder joint model using the ViBe model updating scheme. Then, we compare the solder joint image with the well-trained model to detect potential defects. Finally, we introduce a frequency map method and define a metric named defect degree to evaluate the qualities of the solder joints. Experimental results show that our method is universal, accurate, and easily debugged compared with the other existing methods.
Deeper neural networks are more difficult to train. We present a residual
learning framework to ease the training of networks that are substantially
deeper than those used previously. We explicitly reformulate the layers as
learning residual functions with reference to the layer inputs, instead of
learning unreferenced functions. We provide comprehensive empirical evidence
showing that these residual networks are easier to optimize, and can gain
accuracy from considerably increased depth. On the ImageNet dataset we evaluate
residual nets with a depth of up to 152 layers---8x deeper than VGG nets but
still having lower complexity. An ensemble of these residual nets achieves
3.57% error on the ImageNet test set. This result won the 1st place on the
ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100
and 1000 layers.
The depth of representations is of central importance for many visual
recognition tasks. Solely due to our extremely deep representations, we obtain
a 28% relative improvement on the COCO object detection dataset. Deep residual
nets are foundations of our submissions to ILSVRC & COCO 2015 competitions,
where we also won the 1st places on the tasks of ImageNet detection, ImageNet
localization, COCO detection, and COCO segmentation.
An automated corrosion monitoring system using the electrical resistance technique was applied for assessment of the corrosivity towards carbon steel and zinc in different phases of a complex accelerated corrosion test recently introduced by VDA, an association of German car makers. It comprises salt spray, wet, dry, and freezing phases. The developed small and battery-driven atmospheric corrosion loggers provided high sensitivity allowing for sub-angstrom (<10−10 m) measurements of corrosion depth and good accuracy. The actual corrosion rate was affected by the exposure history due to a limited rate of wetting/drying and oxygen and ion transport to the reaction interface under a layer of corrosion products. The hysteresis was particularly strong for carbon steel. Except the freezing phase, the steel corrosion rate varied in a narrow range from 0.2 to 0.6 µm/h. For zinc, the corrosion rate varied from 0.001 to 0.1 µm/h in particular phases of the cycle with the maximum in the salt spray phase. Seventy-five percent of the metal corroded in the salt spray phase and in the following drying period representing only 13% of the total test time. The obtained data suggest that the proposed test cycle allowed for rather efficient drying of the zinc surface, which is believed to be crucial for the formation of corrosion products with certain protective ability observed also in field conditions.
To improve the performance of automatic optical inspection (AOI), a neural network combined with genetic algorithm for the diagnosis of solder joint defects on printed circuit boards (PCBs) assembled in surface mounting technology (SMT) is presented. Six types of solder joint have been classified in respect to the reality in the manufacture. The images of solder joint under test are acquired and 14 features are extracted as input features for the classification. The neural network is easily become over-fitting because these input features are not independent of each other, so the genetic algorithm is introduced to select and remove redundant input features. The experimental results have proved that the neural network combined with genetic algorithm reduced the number of input feature and had a satisfying recognition rate. (c) 2013 Elsevier GmbH. All rights reserved.
Continuous casting is a highly efficient process used to produce most of the world steel production tonnage, but can cause cracks in the semi-finished steel product output. These cracks may cause problems further down the production chain, and detecting them early in the process would avoid unnecessary and costly processing of the defective goods. In order for a crack detection system to be accepted in industry, however, false detection of cracks in non-defective goods must be avoided. This is further complicated by the presence of scales; a brittle, often cracked, top layer originating from the casting process. We present an approach for an automated on-line crack detection system, based on 3D profile data of steel slab surfaces, utilizing morphological image processing and statistical classification by logistic regression. The initial segmentation successfully extracts 80% of the crack length present in the data, while discarding most potential pseudo-defects (non-defect surface features similar to defects). The subsequent statistical classification individually has a crack detection accuracy of over 80% (with respect to total segmented crack length), while discarding all remaining manually identified pseudo-defects. Taking more ambiguous regions into account gives a worst-case false classification of 131 mm within the 30 600 mm long sequence of 150 mm wide regions used as validation data. The combined system successfully identifies over 70% of the manually identified (unambiguous) crack length, while missing only a few crack regions containing short crack segments. The results provide proof-of-concept for a fully automated crack detection system based on the presented method.
This is the second edition of the established guide to close-range photogrammetry which uses accurate imaging techniques to analyse the three-dimensional shape of a wide range of manufactured and natural objects.
After more than 20 years of use, close-range photogrammetry, now for the most part entirely digital, has become an accepted, powerful and readily available technique for engineers, scientists and others who wish to utilise images to make accurate 3D measurements of complex objects. Here they will find the photogrammetric fundamentals, details of system hardware and software, and broad range of real-world applications in order to achieve this.
Following the introduction, the book provides fundamental mathematics covering subjects such as image orientation, digital imaging processing and 3D reconstruction methods, as well as a discussion of imaging technology, including targeting and illumination, and its implementation in hardware and software. It concludes with an overview of photogrammetric solutions for typical applications in engineering, manufacturing, medical science, architecture, archaeology and other fields.
Appearance defects inspection plays a vital role in bearing quality control. Human inspection is a traditional way to remove defective bearings, which is instable and time consuming. In this paper, we develop a machine vision system for bearing defect inspection, which can inspect various types of defects on bearing covers, such as deformations, rusts, scratches and so on. The proposed system designs a novel image acquisition system to enhance the defects appearances and get controlled image acquisition environment. A series of image processing methods are proposed or utilized to inspect the defects. Especially, for the deformation defects on seal, we find a common rule on the distribution of projection, and design a simple but effective inspection algorithm based on the rule. The proposed system is evaluated and compared with skilled human by the recall, precision and F-measure. Experimental results show that the proposed vision system has high accuracy and efficiency.
A four-camera videogrammetric system with large field-of-view is proposed for 3-D motion measurement of deformable object. Four high-speed commercial-grade cameras are used for image acquisition. Based on close-range photogrammetry, an accurate calibration method is proposed and verified for calibrating the four cameras simultaneously, where a cross target as calibration patterns with feature points pasted on its two-sides is used. The key issues of the videogrammetric processes including feature point recognition and matching, 3-D coordinate and displacement reconstruction, and motion parameters calculation are discussed in detail. Camera calibration experiment indicates that the proposed calibration method, with a re-projection error less than 0.05 pixels, has a considerable accuracy. Accuracy evaluation experiments prove that the accuracy of the proposed system is up to 0.5 mm on length dynamic measurement within 5000 mm×5000 mm field-of-view. Motion measurement experiment on an automobile tire is conducted to validate performance of our system. The experimental results show that the proposed four-camera videogrammetric system is available and reliable for position, trajectory, displacement and speed measurement of deformable moving object.
Discrete surface defects impact the riding quality and safety of a railway system. However, it is a challenge to inspect such defects in a vision system because of illumination inequality and the variation of reflection property of rail surfaces. This paper puts forward a real-time visual inspection system (VIS) for discrete surface defects. VIS first acquires a rail image by the image acquisition system, and then, it cuts the subimage of rail track by the track extraction algorithm. Subsequently, VIS enhances the contrast of the rail image using the local normalization (LN) method, which is nonlinear and illumination independent. At last, VIS detects defects using the defect localization based on projection profile (DLBP), which is robust to noise and very fast. Our experimental results demonstrate that VIS detects the Type-II defects with a recall of 93.10% and Type-I defects with a recall of 80.41%, and the proposed LN method and DLBP algorithm are better than the related well-established approaches. Furthermore, VIS is very fast with a linear computational time complexity, and it can be in real time to run on a 216-km/h test train under our experimental setup.
A design of PCB automatic defects detection system based on AOI technology is presented. The hardware design is emphatically introduced including illumination module, image acquisition module, motion control unit, PC, graphic display device and operation unit. Simultaneously, the software design is briefly explained. This design is a non-contact PCB defects detection technology which can not only detect open circuit and short circuit defects, but also can detect wire gaps, voids, scratch defects etc. The highest resolution of the design is 15µm and the detection success rate is over 95%.
A machine vision system developed for inspecting metal ball surface defects is presented. The proposed system is capable of inspecting the entire surface of a ball by capturing multiple gray-scale images with two progressive CCD cameras as the ball rolls on an inclined rail. The specular reflectance of the metal surface is lessened by installing a shade around the ball. Defects are detected by simply comparing each captured image with its corresponding reference image. The system built for the experiment could sort two chrome balls per second with a spatial resolution better than 0.1 mm.
A method for photogrammetric data reduction without the necessity for neither fiducial marks nor initial approximations for inner and outer orientation parameters of the camera has been developed. This approach is particularly suitable for reduction of data from non-metric photography,
but has also distinct advantages in its application to metric photography. Preliminary fictitious data tests indicate that the approach is promising. Experiments with real data are underway.
This paper presents an accurate stereo vision system for industrial inspection, which uses a self-calibration method based on photogrammetry. A cross-shaped calibration pattern which is portable and easy to be manufactured is designed. The cross target can be used to calibrate stereo vision systems and obtain higher measurement precision conveniently. The mathematical model of the stereo vision system with 10 distortion parameters for each camera is proposed. The feature point detection method with sub-pixel accuracy is explored. The calibration initial values are computed using the relative orientation method and the direct linear transform (DLT) method of photogrammetry. The bundle adjustment algorithm is used to optimize the calibration parameters as well as the 3D coordinates of the feature points. Experiment results show that the RMS error of the reprojection in our method is less than 0.05 pixels and the distance measurement error is 0.031mm with a high precision scale bar which length is 221.001±0.003mm.
Inspection of solder joints has been a critical process in the electronic manufacturing industry to reduce manufacturing cost, improve yield, and ensure product quality and reliability. This paper proposes two inspection modules for an automatic solder joint classification system. The “front-end” inspection system includes illumination normalisation, localisation and segmentation. The “back-end” inspection involves the classification of solder joints using the Log-Gabor filter and classifier fusion. Five different levels of solder quality with respect to the amount of solder paste have been defined. The Log-Gabor filter has been demonstrated to achieve high recognition rates and is resistant to misalignment. This proposed system does not need any special illumination system, and the images are acquired by an ordinary digital camera. This system could contribute to the development of automated non-contact, non-destructive and low cost solder joint quality inspection systems.
A digital photogrammetry measurement system (XJTUDP) is developed in this work, based on close range industry. Studies are carried out on key technologies of a photogrammetry measurement system, such as the high accuracy measurement method of a marker point center based on a fitting subpixel edge, coded point design and coded point autodetection, calibration of a digital camera, and automatic image point matching algorithms. The 3-D coordinates of object points are reconstructed using colinear equations, image orientation based on coplanarity equations, direct linear transformation solution, outer polar-line constraints, 3-D reconstruction, and a bundle adjustment solution. Through the use of circular coded points, the newly developed measurement system first locates the positions of the camera automatically. Matching and reconstruction of the uncoded points are resolved using the outer polar-line geometry of multiple positions of the camera. The normal vector of the marker points is used to eliminate the error caused by the thickness of the marker points. XJTUDP and TRITOP systems are tested on the basis of VDI/VDE2634 guidelines, respectively. Results show that their precision is less than 0.1 mm/m. The measurement results of a large-scale waterwheel blade by XJTUDP show that this photogrammetry system can be applied to industrial measurements.
Fabric defect detection has been an active area of research since a long time and still a robust system is needed which can fulfill industrial requirements. A robust automatic fabric defect detection system (FDDS) would results in quality products and more revenues. Many different approaches and method have been tried to implement FDDS. Most of them are based on two approaches, one is statistical like gray level co-occurrence (GLCM) and other is transform based like Gabor filter. This paper presents a new scheme for automated FDDS implementation using GLCM and also compare it with Gabor filter approach. GLCM texture statistics are extracted and plotted against the inter-pixel distance of GLCM as signal graph. The non-defective fabric image information is compared with the test fabric image. In Gabor filter based approach, a bank of Gabor filter with different scales and orientations is generated and fabric images are filtered with convolution mask. The generated magnitude responses are compared for defect decision. In our implementation of both approaches in same environment, the GLCM approach produces higher defect detection accuracies than Gabor filter approach and more computationally efficient.
Learning techniques have been applied increasingly for food quality evaluation using computer vision in recent years. This paper reviews recent advances in learning techniques for food quality evaluation using computer vision, which include artificial neural network, statistical learning, fuzzy logic, genetic algorithm, and decision tree. Artificial neural network (ANN) and statistical learning (SL) remain the primary learning methods in the field of computer vision for food quality evaluation. Among the applications of learning algorithms in computer vision for food quality evaluation, most of them are for classification and prediction, however, there are also some for image segmentation and feature selection. In this paper, the promise of learning techniques for food quality evaluation using computer vision is demonstrated, and some issues which need to be resolved or investigated further to expedite the application of learning algorithms are also discussed.
In this paper, a novel method to recognize stem or calyx regions of ‘Jonagold’ apples by pattern recognition is proposed. The method starts with background removal and object segmentation by thresholding. Statistical, textural and shape features are extracted from each segmented object and these features are introduced to several supervised classification algorithms. Linear discriminant, nearest neighbor, fuzzy nearest neighbor, support vector machines classifiers and adaboost are the ones tested. Relevant features are selected by floating forward feature selection algorithm. Support vector machines, which is found to be the best among all classification algorithms tested, correctly recognized 99% of the stems and 100% of the calyxes using selected feature subset. These results exhibit considerable improvement relative to the ones introduced in the literature.
Ultrasonic monitoring of steel corrosion during accelerated corrosion testing and outdoor field exposures
- G Rannou
- D Thierry
- Bozec Nle
Rannou G, Thierry D, Bozec NLe. Ultrasonic monitoring of steel corrosion during
accelerated corrosion testing and outdoor field exposures. NACE International; 2010.
A New IC Solder Joint Inspection Method for an Automatic Optical Inspection System Based on an Improved Visual Background Extraction Algorithm
- N Cai
- J Lin
- Q Ye
- H Wang
- S Weng
- Bwk Ling
Cai N, Lin J, Ye Q, Wang H, Weng S, Ling BWK. A New IC Solder Joint Inspection Method for an Automatic Optical Inspection System Based on an Improved Visual Background Extraction Algorithm. IEEE Trans Compon Packag Manuf Technol
2016;6(1):161-72.
A fast and efficient projection-based approach for surface reconstruction. Brazilian symposium on computer graphics & image processing
- M Gopi
- Krishnan Shankar
Gopi M, Krishnan Shankar. A fast and efficient projection-based approach for surface reconstruction. Brazilian symposium on computer graphics & image processing.
IEEE; 2003.