Conference PaperPDF Available

ChangeChip: A Reference-Based Unsupervised Change Detection for PCB Defect Detection

Authors:
ChangeChip: A Reference-Based Unsupervised
Change Detection for PCB Defect Detection
Yehonatan Fridman
Ben-Gurion University of the Negev
Israel Atomic Energy Commission
fridyeh@post.bgu.ac.il
Matan Rusanovsky
Ben-Gurion University of the Negev
Israel Atomic Energy Commission
matanru@post.bgu.ac.il
Gal Oren
Technion Israel Institute of Technology
Nuclear Research Center Negev
galoren@cs.technion.ac.il
Abstract—The usage of electronic devices increases, and be-
comes predominant in most aspects of life. Surface Mount
Technology (SMT) is the most common industrial method for
manufacturing electric devices in which electrical components
are mounted directly onto the surface of a Printed Circuit Board
(PCB). Although the expansion of electronic devices affects our
lives in a productive way, failures or defects in the manufacturing
procedure of those devices might also be counterproductive
and even harmful in some cases. It is therefore desired and
sometimes crucial to ensure zero-defect quality in electronic
devices and their production. While traditional Image Processing
(IP) techniques are not sufficient to produce a complete solution,
other promising methods like Deep Learning (DL) might also be
challenging for PCB inspection, mainly because such methods
require big adequate datasets which are missing, not available
or not updated in the rapidly growing field of PCBs. Thus,
PCB inspection is conventionally performed manually by human
experts. Unsupervised Learning (UL) methods may potentially
be suitable for PCB inspection, having learning capabilities
on the one hand, while not relying on large datasets on the
other. In this paper, we introduce ChangeChip, an automated
and integrated change detection system for defect detection in
PCBs, from soldering defects to missing or misaligned electronic
elements, based on Computer Vision (CV) and UL. We achieve
good quality defect detection by applying an unsupervised change
detection between images of a golden PCB (reference) and the
inspected PCB under various setting. In this work, we also
present CD-PCB, a synthesized labeled dataset of 20 pairs of PCB
images for evaluation of defect detection algorithms. The sources
of ChangeChip, as well as CD-PCB, are available at: https:
//github.com/Scientific-Computing-Lab- NRCN/ChangeChip.
Index Terms—Change Detection, Unsupervised Machine
Learning, PCA-Kmeans, PCBs, SMT Quality Control
I. INTRODUCT IO N
PCB manufacturing and Surface Mount Technology (SMT)
are not zero-defect soldering processes. The ever-changing
technology in fabrication, placement, and soldering of SMT
electronic components have caused an increase in PCB defects
both in terms of numbers and types [1]. Defects in SMT
processes such as missing or misaligned components, and
other soldering defects like solder bridging and solder balling,
might all cause PCBs to fail. The detection of such defects is
an important part in the control over PCB manufacturing and
SMT processes [2].
Generally, there are several main ways to detect defects in
PCBs, each with a unique disadvantage:
Manual Visual Inspection (MVI) by experts [3]: As PCBs
have evolved to become more complicated, multi-layered,
and assembled with much tinier components, manual
visual inspection has become very tiring, costly, and
subjective task. Therefore, there is a growing need for
automation in the PCB inspection field.
Classic Image Processing (IP) [1], [4]–[7]: Seemingly
not satisfactory to produce a complete solution since it
cannot understand abstract features from the data.
Supervised (e.g. Deep) Learning [7]–[10]: Since PCB
architectures change rapidly it is sometimes very hard to
attain a proper and general enough dataset for supervised
learning tasks.
Unsupervised Learning [7], [11]: Usually unsupervised
learning models are weaker than supervised ones since
they lack the supervision of the field experts, and there-
fore it is harder for them to be optimized against the
knowledge of the experts on the training data.
Therefore, there is a need for algorithms that have learning
capabilities that successfully extract features of interest, and at
the same time do not require large datasets. In this work, we
present ChangeChip, an automated change detection system
for defect detection for PCBs based on an unsupervised learn-
ing approach. The objective of ChangeChip is to recommend
experts on the most salient changes between an inspected
image and a ’perfect’ reference image, without any prior
knowledge on the PCB design and types of components. This
objective is crucial since the technology is in a constant
state of development toward smaller components which allows
integration of more compact designs at the PCB level, from
small single-board microcontrollers to large motherboards
and GPUs. Fig.1presents the workflow of ChangeChip as
a comprehensive integrated system. ChangeChip performs a
preprocessing stage on the input images to make a correspon-
dence between the two PCBs in cases of imaging discrepancies
such as lighting, imaging view, etc. The preprocessing stage
is based on a deep learning tool named Deep Extreme Cut
(DEXTR) [12] that allows to extract the PCBs and disregard
the background, and on IP and CV methods to perform image
registration and histogram equalization. The preprocessing
stage can be interpreted as a pipeline: The PCBs are first
extracted in order to ignore the background if needed, then
2021 IEEE Physical Assurance and Inspection of Electronics (PAINE) | 978-1-6654-1010-6/21/$31.00 ©2021 IEEE | DOI: 10.1109/PAINE54418.2021.9707699
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
Input (Reference and Target PCBs)
Cropping (e.g. DEXTR)
Registration
Exact Histogram Matching
PCA-Kmeans Change Detection
Classes Analysis and Output
optional
optional
Fig. 1: The workflow of ChangeChip.
are registered to align with each other by applying affine
transformation on one of the images, and finally are matched
by their histograms via the Exact Histogram Specification
algorithm [13] to reduce illumination differences between the
images. Subsequently, at the heart of the system, ChangeChip
uses PCA-Kmeans on the features of both images to cluster
pixels into classes of change significance. A post-processing
stage is then applied to map the clustering result into a
meaningful clear detection of defects using a Mean Squared
Error (MSE) heuristic.
In Section II we mention previous work and discuss the
contribution of our suggested system for quality control of
PCBs. In Section III we elaborate on the algorithmic phases
of ChangeChip. Then, in Section IV we present CD-PCB, a
database of PCB images that were created for the change
detection problem to evaluate defect detection algorithms
for PCB inspection. We then evaluate the performance of
ChangeChip on that database. In section Vwe discuss the
conclusions of this work, limitations, and future work.
II. PREVIOUS WORK
In the last decades, many techniques and computer al-
gorithms were implemented in Automated Visual Inspection
(AVI) systems for quality assurance for PCB manufacturing
[4], [5], [14]–[20]. The main class of such algorithms is
change detection algorithms which use a reference comparison
approach by detecting changes between a reference image of
a zero-defects already-inspected PCB, with the inspected PCB
image. Previous work has been done on change detection for
PCBs, and various state-of-the-art algorithms were developed
using Image Processing (IP), Computer Vision (CV), and Ma-
chine Learning (ML) [1], [6]. The constantly evolving nature
and scope of the PCB assurance problem, pose challenges
when using traditional IP and CV approaches, which operate
best in closed systems [7]. Hence, intelligent algorithms capa-
ble of learning (i.e., machine learning and deep learning) are
necessary for achieving and maintaining high accuracy, while
avoiding becoming outdated [9], [19]. Yet, these methods
mostly require a large and regularly updated dataset of images
labeled by Subject Matter Experts (SMEs). Although there
are several existing datasets of PCB components, there are
few that are publicly available, such as [10]. Moreover, many
of the available datasets are intended and formatted for a
specific study, so it is difficult to combine and re-purpose them
effectively for a different study.
In this work, we present a change detection system for
PCB inspection based on IP and UL. The machine learning
algorithm that is used in our system is PCA-Kmeans, and is a
variation of an implementation that was originally suggested
for satellite images in [21]. The usage of UL methods allows
us to waive the need for a labeled updated dataset on the one
hand, and on the other hand to perform a learning procedure
that allows extraction of main features from the data.
III. THE ALGORITHMIC STAGES OF CHANGECHIP
A. Preprocessing Stage
One of the key elements of an AVI system is the imaging
setting. Naturally, if the images were captured in identical
conditions of lighting, positioning, and image resolution, then
it was easier to detect true changes between them while
avoiding false positives. By performing image registration
and histogram matching, ChangeChip, in effect, relaxes the
necessity of both input images to be captured under the
identical conditions. However, insufficient efforts to maintain
identical imaging conditions can still lead to challenges that
would cause the system to fail. For example, capturing a 3D
PCB from two different directions can reveal different sides of
the same elements, which can falsely be detected as changes.
Hence, we suggest capturing both input images in the same
imaging environment, maintaining the imaging conditions as
identical as possible. As described in Section IV-A, the CD-
PCB database was constructed under a fixed setting. We note
that illumination changes and different object orientations are
still present, despite our efforts to maintain the same settings.
1) Deep Extreme Cut (DEXTR): [12] is a deep learning
tool that extracts objects from an image using 4 extreme points
of each object (left-most, right-most, top, bottom pixels).
DEXTR is based on adding an extra channel to the image
in the input of a Convolutional Neural Network (CNN),
which contains a Gaussian centered in each of the extreme
points. The CNN learns to transform this information into a
segmentation of an object that matches those extreme points.
We use DEXTR in the preprocessing stage of ChangeChip for
PCBs extraction in order to ignore the background, or even
to extract a specific component if comparing only parts of the
PCBs is desired. We stress that ChangeChip is not limited to
DEXTR, and other cropping methods might be suitable too.
2) Image Registration: Image registration is the process of
estimating an optimal transformation between two images, in
order to bring them into one coordinate system, which then
makes it easier to compare them. We use a feature-based
registration, based on extraction of feature points along with
descriptors and matching them, then using the corresponding
pairs to calculate a transformation matrix that transforms
the target image into the same orientation and position as
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
the reference image. In order to extract and describe feature
points in PCBs we use the Scale-Invariant Feature Transform
(SIFT) [22] algorithm which is rotation, scale, and luminance
invariant. Pairs are then matched by finding for each feature
point in one image the best match among the feature points in
the other image (which minimizes the Euclidean distance). We
then reject matches in which the ratio of the nearest neighbor
distance to the second-nearest neighbor distance is greater then
0.8 (a value suggested by Lowe [23]). According to Lowe,
this rejection eliminates 90% of the false matches while
discarding less than 5% of the correct matches. Then, a
transformation matrix is determined to map one image to align
with the other, based on these distinct matches. Because the
number of matches probably exceeds the minimum required
to define the appropriate transformation, and they might not
follow a single linear model, the iterative method RANSAC
[24] is used for a robust estimation of the transformation
matrix.
3) Reducing Illumination Changes: Various technologies
are in use in the PCB inspection industry, ranging from optical
photography like still photography and micro-photography, to
radiography using X-rays, gamma rays, etc [4]. The use of
radiography enables the inspection of inner layers of PCBs,
which is not possible with optical imaging. The fact that
optical images are easily influenced by illumination makes
it harder to compare them. In order to reduce illumination
differences, we apply Exact Histogram Specification [13] to
match the two images’ histograms. This however does not
provide a complete solution, especially since optical images
taken from different points of view or with different lighting
settings, might defer more significantly due to reflections
on shiny objects, that cannot be disregarded by applying
histogram matching. Yet, histogram equalization might help
in the case of a fixed capture system, with a slowly fading
light source over time. We note that these issues of lighting do
not apply to radiography, and therefore this step of histogram
matching is optional.
B. PCA-Kmeans Change Detection
After performing image registration and histogram match-
ing, we use a variation of the PCA-Kmeans change detec-
tion algorithm that was presented in [21]. This algorithm
requires both images to have the same dimensions [1, width]×
[1, height]. In general, this algorithm uses a descriptor that
extracts for each pixel a set of features, that describe whether
that pixel is a ’change’ from one image to another. Each set
of features (i.e. descriptor) is reduced to a smaller dimension
via PCA, and then all these reduced features are clustered into
classes of change degree using Kmeans. In [21], a grayscale
difference window is used to describe the change of each
pixel, mainly because that work attends grayscale images. In
our adaptation for colored images, we use the concatenation
of all the RGB-channels difference windows, along with the
grayscale difference window, to preserve and learn meaningful
information about the color features of changes. The PCA-
Kmeans algorithm is performed on small windows around each
Reference and Target PCBs
Extracting the PCBs with DEXTR
Matching feature points for registration
The registration and histogram matching result
The clustering map and the final detection
Fig. 2: The workflow of ChangeChip on two optical images
of PCBs
pixel, and therefore the learning is window-based. This kind
of learning can eliminate further noise that exists in the input
images.
In more details, we define a descriptor for each pixel (i, j)
[1, width]×[1, height]as follows:
des[i, j]=(desRGB [i, j], desg ray[i, j ]) (1)
where
desRGB [i, j]=(|I1(S)RI2(S)R|,|I1(S)GI2(S)G|,
|I1(S)BI2(S)B|)
(2)
desgray [i, j] = (|I1(S)g ray I2(S)gray |)(3)
where S= (ih
2:i+h
2, j h
2:j+h
2), a subset of
indices that constitutes a window of h×hpixels around
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
Reference and Target PCBs
Matching feature points for registration
The result of registration
The clustering map and the final detection
Fig. 3: The workflow of ChangeChip on two radiography
images of PCBs
the pixel (i, j).I1and I2represent the two images, and
|I1(S)RI2(S)R|stands for the absolute values of subtraction
between the Red channels of I1and I2restricted to the window
S. The same notation is also applied to the Green (G) and
Blue (B) channels, and also to the grayscale value (gray)
which is calculated as the luminance of a pixel according to
the following relation: gray = 0.3R+ 0.59G+ 0.11B. For
windows that cross the border, we pad I1and I2with zeros.
Pixels are partitioned into h×hnon-overlapping windows
(e.g. 5×5). We denote this set of windows by W. For
each window Wwe calculate desRGB [i, j]for the pixel
(i, j)in the center of it. SRGB h2orthonormal eigen-
vectors are extracted through Principal Component Analy-
sis (PCA) of the set {desRGB [i, j]}w indowWto create an
eigenvector space EigenRGB . PCA is also applied on the
set {desgray [i, j]}w indowWto achieve an eigenvector space
Eigengray , consists of Sgray h2vectors.
Each pixel (i, j)[1, width]×[1, heig ht]can be repre-
sented by a (SRGB +Sgray )-dimensional feature vector that
is the projection of {desRGB [i, j]}onto the generated eigen-
vector space EigenRGB, concatenated with the projection of
{desgray [i, j]}onto E igengr ay. According to [21], each h×h
block of the difference image could contain one of the three
different types of data coming from the difference image: 1)
no-change data; 2) change data; and 3) mixture of change and
no-change data. The first two cases happen when the h×h
block is completely localized on the changed or unchanged
areas of the difference image. The last case happens when the
h×hblock is localized on the boundaries between changed
and unchanged regions of the difference image. Then, if Sgray
is set to 3, each type of data could be represented by a single
eigenvector. The authors of [21] report that according to an
extensive set of experiments (on a dataset of grayscale satellite
images), when 3< Sgray h2, there is no significant change
in the change detection performance with respect to that of
Sgray = 3. For our application, we also choose Sgr ay = 3,
and for the RGB features, we choose SRGB = 9. Based on
experiments that we have conducted on our domain, changing
these parameters leads to no significant change in the results.
According to [21], change detection is achieved by partitioning
the feature vector space into two clusters. We find that in the
case of PCBs images, higher resolution clustering is needed to
successfully separate the changes from the other pixels. The
reason for that is the high resolution usually PCBs images are
taken in, and the bigger intricacy PCBs images have compared
to satellite images. The number of classes, n, is a parameter
of our application, usually is set around 10 and might differ
from system to system to achieve optimized results.
C. Post-processing Stage: Classes analysis using an MSE
heuristic
The previous stage clusters all the pixels into classes
{C1, C2, ...Cn}. In the post-processing stage, we characterize
each class according to a certain heuristic, in order to deter-
mine which classes represent a ’change’ in a more meaningful
fashion. We use the Mean Squared Error (MSE) measure,
defined as follow for each class Ci:
MSE(Ci) = 1
3|Ci|X
(i,j)C i
X
c∈{R,G,B}
(I1(i, j)cI2(i, j )c)2
The classes are sorted according to the MSE scores, and each
class receives a color according to its index in that order, from
blue for the class with the minimum MSE score, to red for
the class with the maximum MSE score. The assignment of
colors to the classes yields a meaningful heat map, which
indicates the intensity of the changes. However, in order to
produce a map with only the suspected changes tagged, some
of the lower MSE score classes have to be discarded. For
this purpose, we suggest clustering the MSE scores using the
Density-Based Spatial Clustering of Applications with Noise
(DBSCAN) algorithm [25], followed by discarding the lowest
MSE score class. The advantage of DBSCAN over Kmeans,
for example, is that DBSCAN does not require to specify
the number of clusters. Instead, it requires setting the density
parameter (eps), which according to our tests produces better
results. Although eps may be tuned and fixed based on some
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
learning samples, it still might be modified from application
to application for optimized results.
IV. EVALUATION
A. Change-Detection-PCB (CD-PCB) Dataset
In order to evaluate the performance of ChangeChip, and
as a proposed dataset for future works, we present Change-
Detection-PCB (CD-PCB) dataset, which is composed of 20
pairs of PCB images with annotated changes between them.
CD-PCB includes images of PCBs with various types of
challenging defects. Two of the pairs were captured via radio-
graphic technology, and the other 18 pairs were synthesized
and captured under an established environment that maintains
the same imaging conditions (although a better setting can be
achieved [26]). To our knowledge, such a database does not
exist in this form. The imaging system that was used to form
CD-PCB, consists of a camera, tripod, 2 studio flashes, and a
fixed desk. A sketch of the system is presented in Fig. 4.
Fig. 4: The capture system model used to construct CD-PCB
B. Evaluation Method
We use Precision and Recall to evaluate ChangeChip.Recall
is the fraction of true ’change’ pixels that were classified as
such by the algorithm, out of all the true ’change’ pixels.
Precision is the fraction of true ’change’ pixels among all pix-
els that were classified (correctly and incorrectly) as changes
by the algorithm. High Recall is vital for an algorithm to be
considered as providing high assurance. Since every defect on
a PCB can be crucial to its functioning, high assurance of
defect detection is desired. On the other hand, high Precision
is also desired in order to focus on the true defects. High
rates of false positives detection point to the weakness of the
algorithm to characterize and extract defects correctly.
C. Results
In this section, we evaluate ChangeChip on CD-PCB. The
parameters that were used to conduct this experiment are
h= 5 (windows of size 5×5) and n= 16 (number of
classes to cluster the descriptors). For the optical images we
use, eps = 0.02 (DBSCAN’s density parameter for clustering
the nMSE values), and for the radiographic images we set
eps = 0.05, since the noises are less concentrated around the
lowest MSE score class. We note that this set of parameters
might change from application to application for optimized
results.
The recall of ChangeChip on CD-PCB is 0.87 and the pre-
cision is 0.8. Although it seems that the precision is low, we
note that the true changes are tiny and therefore, in effect,
the number of false positives is very small in relation to the
whole image. Fig. 5presents couples of images from CD-PCB
in the first two columns, with ground truth changes in the
third column, and the result of ChangeChip on those images
in the fourth column. The quantitative results represent the
performance of the algorithm on the pixel level, while the
qualitative results intuitively refer to the object level. It can
be seen that ChangeChip marks all the changes that were
present in the ground truth, without any significant noise
except for couple number 14, which we discuss in Chapter V.
We conclude that ChangeChip excels qualitatively to highlight
all changes to the human expert that operates the system.
V. CONCLUSIONS, LIMITATIONS AND FUTURE WO RK
In this work, we present ChangeChip, a comprehensive
system for detecting defects on PCBs by applying change
detection. The system suggests a comprehensive procedure
to compare a golden PCB and an inspected PCB. We note
that ChangeChip is not limited to PCBs only and may be
found suitable to other domains as well. As a limitation
of this work, we stress that the system is targeted to find
changes between the two images, while changes, in fact,
might include noise in addition to defects. To illustrate that
point, refer to row 14 in table 5. In this example, certain
changes are characterized as noise by our experts, and thus
are considered to be false positives. To cope with this issue,
one can specifically characterize the attributes of noise in his
system and discard pixels that appear to be noise in some
probability. Moreover, manual labeling of false positives over
time can be used to build a classified data-set of ’noise’ and
’defect’ pixels (or windows of pixels) in one’s system, then use
Supervised (e.g. Deep) Learning techniques to automatically
classify new pixels. Another direction for decreasing the effect
of noise might be by choosing more complicated descriptors
that describe the pixels, with higher tolerance to noise, and
more sensitivity to defects. These descriptors can include for
example color, edges, and texture features. As future work,
we also suggest increasing the support of the system in
radiography images, by analyzing multiple images of adjacent
layers, in order to strengthen the assurance in a significant
defect if it appears as a ’change’ in multiple results. This
future work will amplify the support of this work to 3D
scanning of multi-layered PCBs.
ACK NOW LE DGMEN TS
We would like to thank Amir Ellenbogen and Gabi Dadush
for a fruitful collaboration with our team in this project. We
also thank Ori Stern and Ami Moskowitz for helping in the
creation of CD-PCB. This work was supported by the Lynn
and William Frankel Center for Computer Science. Computa-
tional support was provided by the NegevHPC project [27].
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
Reference PCB Target PCB Ground Truth ChangeChip Results
1
2
3
4
5
6
7
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
8
9
10
11
12
13
14
Fig. 5: Example of pairs from CD-PCB, the ground truth changes and ChangeChip results according to the parameters described
in the Results section. The red circles are for easy identification by the reader.
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
REFERENCES
[1] Kenneth Sundaraj. Pcb inspection for missing or misaligned components
using background subtraction. WSEAS transactions on information
science and applications, 6(5):778–787, 2009.
[2] Hasit Desai. Defect free assembly of smt devices. In Proceedings:
Electrical Insulation Conference and Electrical Manufacturing and Coil
Winding Conference, pages 677–682. IEEE, 1997.
[3] Paul HP Yeow and Rabindra Nath Sen. Ergonomics improvements
of the visual inspection process in a printed circuit assembly factory.
International journal of occupational safety and ergonomics, 10(4):369–
385, 2004.
[4] Madhav Moganti, Fikret Ercal, Cihan H Dagli, and Shou Tsunekawa.
Automatic pcb inspection algorithms: a survey. Computer vision and
image understanding, 63(2):287–313, 1996.
[5] Jianjie Ma. Defect detection and recognition of bare pcb based on
computer vision. In 2017 36th Chinese Control Conference (CCC),
pages 11023–11028. IEEE, 2017.
[6] Abdel-Aziz IM Hassanin, Fathi E Abd El-Samie, and Ghada M
El Banby. A real-time approach for automatic defect detection from
pcbs based on surf features and morphological operations. Multimedia
Tools and Applications, 78(24):34437–34457, 2019.
[7] Ulbert J Botero, Ronald Wilson, Hangwei Lu, Mir Tanjidur Rahman,
Mukhil A Mallaiyan, Fatemeh Ganji, Navid Asadizanjani, Mark M
Tehranipoor, Damon L Woodard, and Domenic Forte. Hardware trust
and assurance through reverse engineering: A survey and outlook from
image analysis and machine learning perspectives. arXiv preprint
arXiv:2002.04210, 2020.
[8] Weibo Huang and Peng Wei. A pcb dataset for defects detection and
classification. arXiv preprint arXiv:1901.08204, 2019.
[9] Mukhil Azhagan Mallaiyan Sathiaseelan, Olivia P Paradis, Shayan
Taheri, and Navid Asadizanjani. Why is deep learning challenging
for printed circuit board (pcb) component recognition and how can we
address it? Cryptography, 5(1):9, 2021.
[10] Hangwei Lu, Dhwani Mehta, Olivia P Paradis, Navid Asadizanjani,
Mark Tehranipoor, and Damon L Woodard. Fics-pcb: A multi-modal
image dataset for automated printed circuit board visual inspection.
IACR Cryptol. ePrint Arch., 2020:366, 2020.
[11] Ihar Volkau, Abdul Mujeeb, Dai Wenting, Erdt Marius, and Sourin
Alexei. Detection defect in printed circuit boards using unsupervised
feature extraction upon transfer learning. In 2019 International Confer-
ence on Cyberworlds (CW), pages 101–108. IEEE, 2019.
[12] Kevis-Kokitsi Maninis, Sergi Caelles, Jordi Pont-Tuset, and Luc
Van Gool. Deep extreme cut: From extreme points to object segmenta-
tion. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 616–625, 2018.
[13] Philippe Bolon Coltuc, Dinu and J-M. Chassery. Exact histogram
specification. IEEE Transactions on Image Processing, 15.5:1143–1152,
2006.
[14] Huihui Wu, Guanglin Feng, Huiwen Li, and Xianrong Zeng. Automated
visual inspection of surface mounted chip components. In 2010 IEEE
International Conference on Mechatronics and Automation, pages 1789–
1794. IEEE, 2010.
[15] EK Teoh, DP Mital, BW Lee, and LK Wee. Automated visual inspection
of surface mount pcbs. In [Proceedings] IECON’90: 16th Annual
Conference of IEEE Industrial Electronics Society, pages 576–580.
IEEE, 1990.
[16] G Acciani, G Brunetti, E Chiarantoni, and G Fornarelli. An automatic
method to detect missing components in manufactured products. In
Proceedings. 2005 IEEE International Joint Conference on Neural
Networks, 2005., volume 4, pages 2324–2329. IEEE, 2005.
[17] Ulbert J Botero, Fatemeh Ganji, Navid Asadizanjani, Damon L
Woodard, and Domenic Forte. Semi-supervised automated layer iden-
tification of x-ray tomography imaged pcbs. In 2020 IEEE Physical
Assurance and Inspection of Electronics (PAINE), pages 1–6. IEEE,
2020.
[18] Nathan T Jessurun, Olivia P Paradis, Mark Tehranipoor, and Navid
Asadizanjani. Shade: Automated refinement of pcb component esti-
mates using detected shadows. In 2020 IEEE Physical Assurance and
Inspection of Electronics (PAINE), pages 1–6. IEEE, 2020.
[19] Eyal Weiss. Electronic component solderability assessment algorithm
by deep external visual inspection. In 2020 IEEE Physical Assurance
and Inspection of Electronics (PAINE), pages 1–6. IEEE, 2020.
[20] RD Taylor. Determination of hole location on printed circuit boards.
Technical report, 1972.
[21] Turgay Celik. Unsupervised change detection in satellite images using
principal component analysis and k-means clustering. IEEE Geoscience
and Remote Sensing Letters, 6(4):772–776, 2009.
[22] David G. Lowe. Object recognition from local scale-invariant features.
In Proceedings of the International Conference on Computer Vision,
volume 2, pages 1150–1157, 1999.
[23] David G. Lowe. Distinctive image features from scale-invariant key-
points. In International Journal of Computer Vision, volume 60 (2),
pages 91–110, 2004.
[24] Konstantinos G Derpanis. Overview of the ransac algorithm. Image
Rochester NY, 4(1):2–3, 2010.
[25] Kamran Khan, Saif Ur Rehman, Kamran Aziz, Simon Fong, and
Sababady Sarasvady. Dbscan: Past, present and future. In The fifth
international conference on the applications of digital information and
web technologies (ICADIWT 2014), pages 232–238. IEEE, 2014.
[26] Gal Oren, Lior Amar, David Levy-Hevroni, and Guy Malamud. The
looking-glass system: A unidirectional network for secure data transfer
using an optic medium. In International Conference on Future Network
Systems and Security, pages 81–97. Springer, 2016.
[27] NegevHPC Project. www.negevhpc.com. [Online].
Authorized licensed use limited to: Ben-Gurion University of the Negev. Downloaded on September 05,2022 at 05:41:31 UTC from IEEE Xplore. Restrictions apply.
... • K-means Clustering: Fridman et al. [64] implemented a k-means clustering in 2021 to identify defects and missing/incorrectly positioned joints. Using the image histogram and Principal Component Analysis, they achieved a precision of 80 % (14 tested PCBs). ...
... To summarize Conventional Unsupervised Learning, kmeans and (spatial-)fuzzy-C-means are predominantly used as unsupervised learning methods for inspecting solder joints. A k-means implementation achieved an 80 % precision on 14 PCBs [64], a k-means-based robot-assisted system reached a detection rate exceeding 95 % in tests involving 300 joints [63]. When a SFCM algorithm is applied, a 75.56 % error detection rate and a 1.11 % false alarm rate on 90 solder joints can be achieved [65]. ...
... Existing literature employs various approaches to achieve this. Fridman et al. [64] apply a conventional Principal Component Analysis alongside k-means clustering to isolate unusual patterns, achieving 80 % precision across 14 PCBs. Hao et al. [74] utilize a genetic algorithm to identify redun-VOLUME 1, 2025 dant features, eliminating some of the 14 input features and evaluating the fitness of each combination. ...
Article
Full-text available
As electrical devices take on more life-critical roles, such as in autonomous driving, ensuring the quality of solder joints during production becomes increasingly important. Recently, there has been a growing interest in using machine learning techniques for this purpose. However, current research lacks a comprehensive overview that categorizes and analyzes relevant studies based on their specific intervention points within the production process. This literature review aims to examine and evaluate research coverage along three dimensions: intervention points in the process, non-destructive testing methods, and machine learning techniques employed. For this review, 112 conference papers and journal articles published since 2010 were selected from three databases using the PRISMA methodology. These publications were classified into the three dimensions previously mentioned, summarized, and analyzed. Furthermore, the literature core is critically evaluated to identify research gaps and limitations. The analysis shows that most studies focus on solder joint control, with few addressing intervention points in solder paste and component placement. Visual imaging and neural networks are the dominant techniques for non-destructive testing and machine learning, respectively. Despite a variety of literature that uses high-performance neural networks, meeting industrial detection standards often requires tolerating high false alarm rates. The findings contribute to structuring existing research and identifying research needs, particularly in validating these systems and integrating data from various testing methods and intervention points.
... Модель глибокого автоенкодера навчено декодувати оригінальні зображення без дефектів з дефектних зображень [93]. Нова методологія виявлення дефектів на друкованих платах ChangeChip використовує порівняння зображень для пошуку змін, але стикається з проблемами шуму, що можна вирішити за допомогою вдосконалених дескрипторів [94]. У статті [95] автори показують високу пропускну здатність класифікації та виявлення компонентів друкованих плат. ...
... ML-based techniques are highly recommended for predictive, detective, and other analyses and also play a vital role in analytics in failure prediction [18][19][20]. The ML-based methods consist of two main purposes, firstly, the dependent variables are predicted using independent variables, and secondly, the effects of independent variables are estimated [21][22][23][24]. The traditional ML method has more shortcomings such things were dealt with DL techniques. ...
Article
Full-text available
A printed circuit board (PCB) is one of the important components in every single electronic device, which assists in connecting each component for many purposes. Somehow, the PCB can be affected due to spurs, short circuits, mouse bites, and so on. Therefore, the detection strategy for such defects is very important and also complicated. So, this research concentrates on developing a deep learning model, a multi-level attention-based printed circuit board with a generative adversarial network, and a YOLOv5 (MuAP-GAN-YOLOv5) model for defect detection in PCB. The contribution of this research is to enhance image quality using the proposed multi-level attention-based PCB-GAN (MuAP-GAN) method, which is embedded with a multi-level attention mechanism to enhance image quality. Therefore, the model can efficiently learn and train for accurate defection as well as localize the defected area in PCB. Here, the YOLOv5 model plays an important role in training based on enhanced features and, therefore provides accurate results. In addition, this model requires less computational expenses, is quite reliable, also provides a maximum accuracy of 95.24% compared to other traditional methods.
... It is widely applied in handwritten signature verification [44], street scene [45], and remote sensing change detection [42]. In ChangChip [46], surface defects in PCB are identified through manual image registration and comparison. However, it entails prolonged preprocessing times and necessitates hyperparameter fine-tuning for image subtraction. ...
Preprint
Full-text available
Despite the eye-catching breakthroughs achieved by deep visual networks in detecting region-level surface defects, the challenge of high-quality pixel-wise defect detection remains due to diverse defect appearances and data scarcity. To avoid over-reliance on defect appearance and achieve accurate defect segmentation, we proposed a change-aware Siamese network that solves the defect segmentation in a change detection framework. A novel multi-class balanced contrastive loss is introduced to guide the Transformer-based encoder, which enables encoding diverse categories of defects as the unified class-agnostic difference between defect and defect-free images. The difference presented by a distance map is then skip-connected to the change-aware decoder to assist in the location of both inter-class and out-of-class pixel-wise defects. In addition, we proposed a synthetic dataset with multi-class liquid crystal display (LCD) defects under a complex and disjointed background context, to demonstrate the advantages of change-based modeling over appearance-based modeling for defect segmentation. In our proposed dataset and two public datasets, our model achieves superior performances than the leading semantic segmentation methods, while maintaining a relatively small model size. Moreover, our model achieves a new state-of-the-art performance compared to the semi-supervised approaches in various supervision settings.
Article
Full-text available
Despite the significant advancements made by deep visual networks in detecting surface defects at a regional level, the challenge of achieving high-quality pixel-wise defect detection persists due to the varied appearances of defects and the limited availability of data. To address the over-reliance on defect appearance and enhance the accuracy of defect segmentation, we proposed a Transformer-based Siamese network with change awareness, which formulates the defect segmentation under a complex background as change detection to mimic the human inspection process. Specifically, we introduced a novel multi-class balanced contrastive loss to guide the Transformer-based encoder, enabling it to encode diverse categories of defects as a unified, class-agnostic difference between defective and defect-free images. This difference is represented through a distance map, which is then skip-connected to the change-aware decoder, assisting in localizing pixel-wise defects. Additionally, we developed a synthetic dataset featuring multi-class liquid crystal display (LCD) defects set within a complex and disjointed background context. In evaluations using our proposed and two public datasets, our model outperforms leading semantic segmentation methods while maintaining a relatively compact model size. Furthermore, our model achieves a new state-of-the-art performance compared to semi-supervised approaches across various supervision settings. Our code and dataset are available at https://github.com/HATFormer/CADNet.
Article
The 3D-printed mesh screen with dense penetrating pores is a new structure for massive manufacturing of molded pulp package products. However, some of the pores may be clogged by the printing material powder during the printing process. Such defects negatively affect the quality of the pulp packages produced using the mesh screen mold. To pinpoint the defects, we design a model-based robotic visual sensing system, called RoboCam, which uses a robotic arm to carry a high-resolution camera for full inspection of a mold consisting of joined mesh screens. To inspect the entire mold, RoboCam plans the camera poses to capture multiple images of the mold and render synthesized images as references for identifying the clogged pores. In particular, we propose novel designs to rectify the inherent run-time pose errors of the robotic system for ensuring the reference quality and to accelerate the reference rendering for reducing inspection latency. Extensive evaluation shows that RoboCam’s design outperforms various baselines, including three existing computer vision and convolution neural network-based inspection systems. RoboCam achieves a recall rate of 94.95% within 528 seconds latency for inspecting an entire mold with 13,000 designed pores.
Article
Full-text available
In this paper, we present the need for specialized artificial intelligence (AI) for counterfeit and defect detection of PCB components. Popular computer vision object detection techniques are not sufficient for such dense, low inter-class/high intra-class variation, and limited-data hardware assurance scenarios in which accuracy is paramount. Hence, we explored the limitations of existing object detection methodologies, such as region based convolutional neural networks (RCNNs) and single shot detectors (SSDs), and compared them with our proposed method, the electronic component localization and detection network (ECLAD-Net). The results indicate that, of the compared methods, ECLAD-Net demonstrated the highest performance, with a precision of 87.2% and a recall of 98.9%. Though ECLAD-Net demonstrated decent performance, there is still much progress and collaboration needed from the hardware assurance, computer vision, and deep learning communities for automated, accurate, and scalable PCB assurance.
Article
Full-text available
To cope with the difficulties in inspection and classification of defects in printed circuit board (PCB), many methods have been proposed in previous work. However, few of them publish their datasets before, which hinders the introduction and comparison of new methods. In this study, HRIPCB, a synthesised PCB dataset that contains 1386 images with 6 kinds of defects is proposed for the use of detection, classification and registration tasks. Besides, a reference‐based method is adopted to inspect and an end‐to‐end convolutional neural network is trained to classify the defects, which are collectively referred to as the RBCNN approach. Unlike conventional approaches that require pixel‐by‐pixel processing, the RBCNN method proposed in this study firstly locates the defects and then classifies them by deep neural networks, which shows superior performance on the dataset.
Article
Full-text available
This paper presents an automatic inspection approach for Printed Circuit Boards (PCBs) with accurate determination of the fault location and identification of the fault type. This approach depends on several digital image processing techniques including registration, filtering, foreground segmentation, mathematical morphological operations, subtraction, feature extraction, and component matching. The Speeded Up Robust Feature extraction (SURF) technique is used for two purposes: registration of the PCB to be checked to a reference PCB and detection of feature points of each missing component on the PCB that is localized from the subtraction process from the reference PCB. Operation is performed on the hue component of the color PCB images. A dictionary is first built for all possible components on the available PCBs with SURF feature descriptors, and hence if a missing item is detected on a PCB during the inspection process, the SURF feature descriptors for features extracted from the difference between the tested and reference PCBs at the position of the lost component are matched with those in the built dictionary or database. A distance metric is used in the matching process. The importance of the proposed approach lies in its ability to build a dictionary of feature descriptors for all possible components in a diversity of PCBs and its ability to localize and identify the missing components regardless of the PCB position, rotation, or type. All operations are formulated in a Graphical User Interface (GUI) using MATLAB environment.
Conference Paper
Reverse engineering (RE) for printed circuit boards(PCB) can be achieved through X-ray Computed Tomography(CT) in a non-destructively yet predominantly manual process. For practical RE to be possible in applications such as obsolescence replacement and hardware assurance, it is important that this process be as automated and fast as possible. This paper introduces a framework to identify which slices of an X-ray CT 3-D PCB stack belong to what layer on a physical PCB in an automated and generalizable fashion for any X-rayed PCB. To the best of our knowledge, this is the first method for identifying the correspondence between an X-ray CT slice and layers of a PCB using reference design information, demonstrated on a 6 layered PCB. First, a spatial pyramid bag-of-visual-words technique is leveraged to enable a semi-supervised approach where the user has either digital or X-ray layout information available for heterogeneous image comparison between X-ray CT image slices. Second, a weighting scheme based on entropy of an image region is proposed. Our results show near perfect quantitative and qualitative layer identification. The completion of this step facilitates improved performance in later RE stages, such as feature extraction or analysis of an X-rayed PCB.
Article
This paper explores the use of extreme points in an object (left-most, right-most, top, bottom pixels) as input to obtain precise object segmentation for images and videos. We do so by adding an extra channel to the image in the input of a convolutional neural network (CNN), which contains a Gaussian centered in each of the extreme points. The CNN learns to transform this information into a segmentation of an object that matches those extreme points. We demonstrate the usefulness of this approach for guided segmentation (grabcut-style), interactive segmentation, video object segmentation, and dense segmentation annotation. We show that we obtain the most precise results to date, also with less user input, in an extensive and varied selection of benchmarks and datasets. All our models and code will be made publicly available.