Content uploaded by Kensuke Harada

Author content

All content in this area was uploaded by Kensuke Harada on Dec 26, 2018

Content may be subject to copyright.

Industry 4.0, Intelligent Visual Assisted

Picking Approach

Mario Arbulu1(B

), Paola Mateus1(B

), Manuel Wagner1(B

),

Cristian Beltran2(B

), and Kensuke Harada2(B

)

1Universidad Nacional Abierta y a Distancia (UNAD), Bogota 111511, Colombia

{mario.arbulu,paola.mateus,manuel.wagner}@unad.edu.co

2Graduate School of Engineering Science, Department of Systems Innovation,

Osaka University, 1-3 Machikaneyama, Toyonaka 560-8531, Japan

{beltran,harada}@sys.es.osaka-u.ac.jp

http://www.unad.edu.co

http://www.hlab.sys.es.osaka-u.ac.jp/people/harada/

Abstract. This work deals with a novel intelligent visual assisted pick-

ing task approach, for industrial manipulator robot. Intelligent searching

object algorithm, around the working area, by RANSAC approach is pro-

posed. After that, the image analysis uses the Sobel operator, to detect

the objects conﬁgurations; and ﬁnally, the motion planning approach by

Screw theory on SO(3), allows to pick up the selected object to move it,

to a target place. Results and whole approach validation are discussed.

Keywords: Artiﬁcial intelligence ·Autonomous picking

Artiﬁcial vision ·Sobel ·RANSAC ·Screws modeling

1 Introduction

The Industry 4.0 challenges are directed around integrating automation process,

cloud and IoT. Furthermore, robotics manipulation, and autonomy are currently

improving, by artiﬁcial intelligence algorithms, [1,2]. For instance the proposed

World Robotics Summit (WRS), motivate researchers around the world, in order

to overcome some challenges at industrial robotics applications too, [3]. Cur-

rently, artiﬁcial vision is used to assist robotic systems, by extracting visual

features from given images. Some research have been done for obtaining the nec-

essary features, such as foreground extraction, noise removal and unnecessary

objects. A proposal of color segmentation method, is detailed in [4]. A back-

ground modeling through statistical edge is given by [5]. Additionally, in [6]

visual control systems had been used, which are based on images for trajectory

tracking. Also, for vision pre-processing, object detection algorithms are used

[7], visual tracking [8], and color intensity [9]. Some of them are conventional

Supported by UNAD, Convocatoria 007.

c

Springer Nature Switzerland AG 2018

A. Groza and R. Prasath (Eds.): MIKE 2018, LNAI 11308, pp. 1–10, 2018.

https://doi.org/10.1007/978-3-030-05918-7_18

2 M. Arbulu et al.

methods, which have limitations on objects detection; and others extract depth

features, which have complex processing [10–12].

So, the object detection and features extraction, for intelligent vision assisted

proposed in this work, is focused in select the interest working planes where the

objects are located. After that, the Sobel [13] operator is proposed to edge detec-

tion, with morphological operations and dilatation [14,15]. And ﬁnally, regions

are labeled for obtaining the interest object features as: area, position (x, y),

centroid and orientation angle.

2 Theoretical Background

In this section theoretical background will be detailed regarding: artiﬁcial vision,

artiﬁcial intelligence, and motion computation; which will describe the overall

approach proposed (see Fig. 1). Where the user sends a pieces set inquiry, in

order to develop the manipulator intelligent picking task. This proposal will be

applied in the Industrial Assembly Challenge, in the WRS, speciﬁcally in the

kitting task.

Fig. 1. Overall approach proposal.

2.1 Artiﬁcial Vision Approach

Through the Sobel algorithm [16], the horizontal and vertical edges detection is

realized, (see Fig. 2(b)).

The edges detection is obtained, with a central approximation of the ﬁrst

derivative, as following:

df (x)

dx =f(x+1)−f(x)

2(1)

with a mask [−1/2 0 1] for the vertical edges, and other mask −1/201

Tfor

the horizontal edges.

Industry 4.0, Intelligent Visual Assisted Picking Approach 3

Next, in order to remove the generated false edges, the Sobel operator is

evaluated by the gradient at the xand ycoordinates (Gx,andGy), such as:

f=[Gx,G

y] (2)

where, Gx=−1

201

2*⎡

⎣

2

4

2

⎤

⎦=⎡

⎣

−101

−202

−101

⎤

⎦

being ⎡

⎣

2

4

2

⎤

⎦the vertical smoothing, which is proposed by the Sobel operator

and Gy=⎡

⎣

−1

2

0

1

2

⎤

⎦*242

=⎡

⎣

−1−2−1

000

121

⎤

⎦

being 242

the horizontal smoothing, which is proposed by the Sobel

operator.

For obtaining better pixels information on each edges previously detected,

the square morphological dilatation is proposed, (see Eq. 3). It is by using the

logic operator OR and selecting a 9 pixels window I(m, n). With the I(m, n)

window, a whole image sweep is realized, thus a new image is generated which

corresponds to the square dilatation, (see Fig. 2(c)).

W[I(m, n)] = ⎡

⎣

I(m−1,n−1) I(m−1,n)I(m−1,n+1)

I(m, n −1) I(m, n)I(m, n +1)

I(m+1,n−1) I(m+1,n)I(m+1,n +1

)

⎤

⎦

being mthe coordinate in the xpixel and nthe coordinate in the ypixel.

Dil(m, n)=OR {W[I(m, n)]}=max {W[I(m, n)]}(3)

In order to obtain the features, and diﬀerentiate each one of the detected objects

in the image; the object labeled is developed by the mask B3X3(see Fig. 2(d)).

That mask sweeps vertically each pixel from the skeleton, it ﬁnd adjacent pixels

with the value 1, and it assign a label value.

B3x3=⎛

⎝

P(m−3,n−3) P(m−3,n)P(m−3,n+3)

P(m, n −3) P(m, n)P(m, n +3)

P(m+3,n−3) P(m+3,n)P(m+3,n +3)

⎞

⎠where P(m, n)is

the evaluated pixel value.

Being Okeach labeled k-th object, a rectangle which embed to each object

(rok), is obtained as follows: rok=mknkAKBkk∈N

Rectangle with higher left vertex in (mk,y

k), height Bkand width Ak.Thus,

each object centroid cokis obtained, by the following expressions: xk=mkAk

2,

yk=nkBk

2,cok=(xk,y

k)

4 M. Arbulu et al.

Fig. 2. (a) Working area image in RGB, (b) Obtained image by Sobel operator, (c)

Dilatation at square shape and ﬁlls holes, (d) Objects label: bounding box is the red

square, and the centroid is the red cross on each one.

2.2 Artiﬁcial Intelligence Approach

The method in this subsection deals with features detection; which is the work-

ﬂow of extraction and correspondence, and them are saved in a features vector.

This method is used to ﬁnd and object, inside of working area (i.e. Fig. 3),

and it is called “Random Sample Consensus” (RANSAC). Speciﬁcally, features

detection is developed with “Speeded Up Robust Features” algorithm (SURF),

which is based in Hessian Matrix (H(i, j)), where in a given point x= (i, j) in a

image I:

H(i, j)=Lxx (i, j)Lxy (i, j )

Lxy(i, j )Lyy(i, j)(4)

Where Lxx(i, j ) corresponds to convolution of second order derivative of g(j),

(Gauss function) d2g(j)/dx2with the Iimage in the xpoint, and in the similar

way for the elements Lxy (i, j)andLyy(i, j ), [17].

Industry 4.0, Intelligent Visual Assisted Picking Approach 5

Fig. 3. (a) Object SURF features detection (b) Work space SURF features detection

(c) Object location in the work space.

The RANSAC algorithm application removes outliers, which can produce

error detection. And at ﬁrst, it obtain a data set with inliers and outliers,

given by:

⎡

⎣n−o

s

⎤

⎦

⎡

⎣n

s

⎤

⎦

=(n−s)(n−s−1)...(n−s−o+1)

n(n−1)...(n−o+1)

Where sis the set size and nis the data number, [18]. These outliers set is shifted

by a probabilistic values set q(Desired probability for drawing an outlier free

subset.), which reduces computational cost. If the probability of an inlier is w,

so the probability of a outlier is: =1−w. It is necessary to make at least N

selections of sets, given by: N=log(1 −q)/log(1 −ws)

2.3 Screws Modeling

In order to compute suitable manipulator motion, the screw approach theory

embedded on Special Euclidean groups SO(3), [19], is detailed in this section.

Being the forward kinematics of 5 DOF manipulator robot of Fig. 4, as following:

gth(θ)=eζ1.

θ1.eζ2.

θ2.eζ3.

θ3.eζ4.

θ4.eζ5.

θ5.gth(0) (5)

Regarding the Eq. 5,gth (0) is the 4 ×4 matrix, which describes the initial end-

eﬀector conﬁguration (position and orientation); gth(θ) is the 4 ×4 matrix, which

6 M. Arbulu et al.

Fig. 4. Five DOF manipulator arm frames (T, H), joint axes (wi), rotation angles θi,

pand kaxes cross points for modeling.

describes the target end-eﬀector conﬁguration, where θis the 5 ×1 vector of

joints rotations. For the ith joint, the joint angle rotation is θi; the twist is ζi;

and ﬁnally, the exponential matrix is eζi.

θi. The product of exponential applied

to the initial end-eﬀector conﬁguration gth(0), allows to model the end-eﬀector

motion to a target conﬁguration, through successive rotations around the free

joints axes. In order to compute the inverse kinematics, the Paden-Kahan (P-K)

subproblems will be applied, [20]. Thus, for solving the θ3joint rotation, the

third P-K subproblem is used, because it solve what is the rotation, around any

free axis, which translates a point to a given distance:

||eζ1.

θ1.eζ2.

θ2.eζ3.

θ3.eζ4.

θ4.eζ5.

θ5.p −k|| =δ(6)

Applying the exponential matrices from axes 1 to 5, to the cross point of axes 4

and 5 (p), the axes rotations θ4and θ5do not aﬀect to that point (see Eq. 6).

Furthermore, the distance δ=||gsh(θ).gsh (0)−1.p −k|| from the resulting

rotated point pto the point k, is not aﬀected by exponential matrices 1 and 2.

So, the θ3joint angle rotation is solved with the third P-K subproblem, by the

following simpliﬁed expression:

||eζ3.

θ3.p −k|| =δ(7)

Next, the second P-K subproblem give us the solution, of two rotation joint

angles with crossed axes; so, the ﬁrst and second joint angles rotations θ1and

θ2are given by:

eζ1.

θ1.eζ2.

θ2.eζ3.

θ3.eζ4.

θ4.eζ5.

θ5.p =p (8)

Evaluating the exponential matrices 1 to 5, in ppoint (see Eq. 8), only the rota-

tions 1 to 3 aﬀect to ppoint; thus, that point achieves the p =gsh(θ).gsh (0)−1.p

Industry 4.0, Intelligent Visual Assisted Picking Approach 7

position. As, the joint rotation θ3has been already solved, and axes 1 and 2 are

crossed at kpoint, the joint angle rotations θ1and θ2could be solved with the

second P-K subproblem, as next, where p=eζ3.

θ3.p.:

eζ1.

θ1.eζ2.

θ2.p=p (9)

As the joints rotation angles θ1to θ3have been computed, and it is notice that,

the joints axes ω4and ω5are crossed at p, following expression is obtained,

through apply rotations θ4and θ5to point k:

eζ4.

θ4.eζ5.

θ5.k =k(10)

The above expression (Eq. 10) allows to solve the joints rotations θ4and θ5,by

the second P-K subproblem, being k=e−ζ3.

θ3.e−ζ2.

θ2.e−ζ1.

θ1.gsh(θ).gsh(0)−1.k

Furthermore, some via points have been selected in 3D space, in order to

deﬁne smooth Cartesian trajectories for approaching, picking and dispatching

the objects to deﬁned targets. Those via points are obtained, as orthogonal

projections from the objects (or pieces) locations, computed in the artiﬁcial

vision approach previously proposed.

3 Results

The proposal was validated with simulation and experimental tests. After, the

user inquiry, the robot can do successfully the kitting task autonomously. The

RANSAC approach identiﬁes where is the desired piece (see Fig. 5), next the

image analysis compute the pieces position, by border detection with Sobel

Fig. 5. (a) Screw detection (b) Packing ring detection.

8 M. Arbulu et al.

operator, and shape dilatation. Using a pixel to mm , scale adaptation, and

the reference translation to robot base; the pieces conﬁgurations (position and

orientation) are obtained, as targets in Cartesian space. Those target pieces

conﬁgurations are introduced to compute robot motion, with Screw theory, (see

Fig. 6). The correction factor of pieces conﬁgurations, due to the image analysis

precision, give us an small displacement in ydirection up to maximum value of

10mm.

Fig. 6. Snapshots of intelligent picking from user inquiry, for diﬀerent type of pieces,

in order to achieve the kitting task. (Robot: Scorbot ER 4u)

Industry 4.0, Intelligent Visual Assisted Picking Approach 9

4 Conclusions

The RANSAC algorithm removes outliers, which allows increase the detection

probability to ﬁnd an object inside the working area. The image analysis algo-

rithm detects accurately enough, each object conﬁguration, which allows to move

the robot end-eﬀector, for picking any object in the working area. Kinemat-

ics motion computation by screw theory, allows eﬃcient computation of joint

patterns, avoiding singularities, and with meaningful analytic description. The

whole intelligent picking algorithm have been successfully tested in order to

achieve the kitting tasks. Current research is focused on increase the robustness

of image analysis, intelligence and motion planning.

References

1. Indri, M., Grau, A., Ruderman, M.: Guest editorial special section on recent trends

and developments in industry 4.0 motivated robotic solutions. IEEE Trans. Ind.

Inform. 14(4), 1677–1680 (2018). https://doi.org/10.1109/TII.2018.2809000

2. Wan, J., Tang, S., Hua, Q., Li, D., Liu, C., Lloret, J.: Context-aware cloud robotics

for material handling in cognitive industrial Internet of Things. IEEE Internet

Things J. 5(4), 2272–2281 (2018). https://doi.org/10.1109/JIOT.2017.2728722

3. World Robot Summit 2018. http://worldrobotsummit.org/en/

4. Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive

multilabel segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1234–

1247 (2013). https://doi.org/10.1109/TPAMI.2012.183

5. Rivera, A.R., Murshed, M., Kim, J., Chae, O.: Background modeling through

statistical edge-segment distributions. IEEE Trans. Circuits Syst. Video Technol.

23(8), 1375–1387 (2013). https://doi.org/10.1109/TCSVT.2013.2242551

6. Mezouar, Y., Chaumette, F.: Optimal camera trajectory with imagebased control.

Int. J. Robot. Res. 22(10–11), 781–804 (2003)

7. Guo, M., Zhao, Y., Zhang, C., Chen, Z.: Fast object detection based on selective

visual attention. Neurocomputing 144, 184–197 (2014)

8. Stalder, S., Grabner, H., Van Gool, L.: Dynamic objectness for adaptive tracking.

In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol.

7726, pp. 43–56. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-

37431-9 4

9. Zhu, W., Liang, S., Wei, Y., Sun, J.: Saliency optimization from robust background

detection. In: Proceedings of CVPR, pp. 2814–2821, June 2014

10. Seo, H.J., Milanfar, P.: Static and space-time visual saliency detection by self-

resemblance. J. Vis. 9(12), 15 (2009). Art. no. 15

11. Bai, X., Sapiro, G.: A geodesic framework for fast interactive image and video

segmentation and matting. In: IEEE 11th International Conference on Computer

Vision, pp. 1–8 (2007)

12. Price, B., Morse, B., Cohen, S.: Geodesic graph cut for interactive image segmen-

tation. In: IEEE Computer Society Conference on Computer Vision and Pattern

Recognition, pp. 3161–3168 (2010)

13. Kanopoulos, N., Vasanthavada, N., Baker, R.L.: Design of an image edge detection

ﬁlter using the Sobel operator. IEEE J. Solid State Circuits 23(2), 358–367 (1988)

10 M. Arbulu et al.

14. Chen, J., Su, C., Grimson, W.E.L., Liu, J., Shiue, D.: Ob ject segmentation of

database images by dual multiscale morphological reconstructions and retrieval

applications. IEEE Trans. Image Process. 21(2), 828–843 (2012). https://doi.org/

10.1109/TIP.2011.2166558

15. Jin, X.C., Ong, S.H., Jayasooriah, A.: Domain operator for binary morphological

processing. IEEE Trans. Image Process. 4(7), 1042–1046 (1995). https://doi.org/

10.1109/83.392348

16. Lim, J.S.: Two-Dimensional Signal and Image Processing, pp. 478–488. Prentice

Hall, Englewood Cliﬀs (1990)

17. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In:

Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–

417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023 32

18. Yaniv, Z.: Random sample consensus (RANSAC) algorithm, a generic implementa-

tion. Insight J., 1–14 (2010). http://www.insight-journal.org/browse/publication/

769

19. Mustafa, S.K., Agrawal, S.K.: On the force-closure analysis of n-DOF cable-driven

open chains based on reciprocal screw theory. IEEE Trans. Robot. 28(1), 22–31

(2012). https://doi.org/10.1109/TRO.2011.2168170

20. Dimovski, I., Trompeska, M., Samak, S., Dukovski, V., Cvetkoska, D.: Algorith-

mic approach to geometric solution of generalized PadenKahan subproblem and

its extension. Int. J. Adv. Robot. Syst., 1–11 (2018). https://doi.org/10.1177/

1729881418755157