PresentationPDF Available

# Support Vector Machine (SVM)

Authors:
Presentaon Prepared by Hajra Zareen, February 03, 2022
Support Vector Machine (SVM)
Presentation Edited
By
Hajra Zareen


(Date: 03/2/2022)
Under the Supervision of

1
Outline
Preliminaries
Introduction
Hyper-plane
Optimal Hyper plane
How to choose Optimal Hyperplane?
How can we find the biggest margin?
Minimizing the norm of w
Vanilla (Plain) SVM & its Objective Function
Soft Margin SVM
The Kernel Trick
Kernel Functions
2
Presentaon Prepared by Hajra Zareen, February 03, 2022
2
Preliminaries (1/2)
3
Basic Linear Algebra
Vectors:  !"#\$   % &'''( )!
%'*%\$&' (
Length of Vectors: +& *!!!'(,!! \$* &(
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Fig.1: Vectors
Preliminaries (2/2)
4
Basic Linear Algebra
Direction of vectors: *(
Dot Product: '%\$\$*!"#(,!! \$*!'(
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Introduction
7%''\$ &/!&(
SVM8'!
SVR8&'&
Classification
9 &!7!& \$*'%#8!':(81'!;#<(8 *18
53(
9 )1!( ' '!#\$ !!' =!91\$  !!\$
 7!#!!#%!8'(
*!!#'1'' 7&!\$  !!\$
!8\$ &7(
Regression
,51!('*7&'!(,
!!'&(
+1 !78 '!!'  #' 1!1(
5
Presentaon Prepared by Hajra Zareen, February 03, 2022
5
Hyper-plane
6
,! !!#'*' '!'\$(,) #!
!3!(
,!!!'8n-dimensional line. Fig.28 \$%!! #!!!# '
\$(
, 8Fig.2, hyperplane!'*''\$!'>&\$
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Fig.2: Linearly Separable Fig.3: Non-Linearly Separable
Optimal Hyperplane
7
If#!1%!\$8Fig.48 % #!  '\$(8
So optimal hyperplane is one which divides the data points very well("\$ #''
 ! #!?
So, if you choose sub-optimal hyperplane, no doubt after number of training iteration , training error will
decrease but during testing when an unseen instance will come, it will result in high test error.
In that case it is must to choose an optimal plane to get good accuracy.
Presentaon Prepared by Hajra Zareen, February 03, 2022
Fig 4: Different Hyperplanes
How to choose Optimal Hyperplane? (1/2)
8
Margin and Support Vectors
+@ !'%!1!%*&(! #!'\$''!
#!\$  & & ' ! #!(
9 '%\$ #!'! #!1\$&' !'
1\$*(
7&\$  '#'(9 \$!!%\$ \$ *'
&%& \$\$1&'!'(
Presentaon Prepared by Hajra Zareen, February 03, 2022
Fig.5: Margin and Support Vectors
How to choose Optimal Hyperplane? (2/2)
9
Margin and Support Vectors
\$ \$ &! #!\$\$!! & #!\$   & 
' !'(
,! #!*#!' &\$!!%*#!!'\$!!&!A\$!!
&'%\$ '\$!!\$!!!&!A\$!!;!'%*(8
&!;A & !%!&!A\$!!(
So, In SVM our goal is to choose an optimal hyperplane which maximizes the margin.
Mathematical Interpretation of Optimal Hyperplane
Presentaon Prepared by Hajra Zareen, February 03, 2022
Fig .6: Optimal Hyperplane
How can we find the biggest margin? (1/14)
< *'
!\$ #!\$   '8\$ 8%\$ 
7;A ' &
Step 1: You have a dataset@D,@and you want to classify it
7 #'\$!!%'88*8;(
 8;\$!!!%'\$ *!8#8'& !%!& !B
(
: 8#88!# *\$%!*!B(
7* \$ #';!#*8;8'
*&!'(C# 8;88'!* 88'(
9 !'!' #/
DE;# ;  # EFGGD
10
Presentaon Prepared by Hajra Zareen, February 03, 2022
10
-). /00\$\$\$(*!(0)40)0***\$* 0
How can we find the biggest margin? (2/14)
Step 2: You@need to select two hyperplanes separating the data with no points between them
!H8 '88,!!#%!(
C\$\$'\$ #!\$ %\$ %\$'H *8\$#
*!A (
C '\$1\$% #! !' !?
Taking another look at the hyperplane equation
89 " #!%\$
\$9;D
Any hyperplane can be written as the set of points;satisfying\$ ;B%D.
\$&A  '' !8\$ ;8'8\$9;(
<& \$'(((8C ' 8B%8?8
,"(,' *8\$8'8;8 * '\$ !
 C1'' # *\$'/
11
Presentaon Prepared by Hajra Zareen, February 03, 2022
11
-). /00\$\$\$(*!(0)40)0***\$* 0
How can we find the biggest margin? (3/14)
Given two 3-dimensional vectors@w(b,−a,1) and@x(1,x,y)
\$ ;D%IBFI;BI#
\$ ;D#F;B%
Given two 2-dimensional@vectors@w′(−a,1) and@x′(x,y)
\$J ;JDFI;BI#
\$J ;JD#F;
Now if we add@b@on both side of the equation@(2)@we got :
\$J ;JB%D#F;B%
\$J ;JB%D\$ ; 
K* #!8L8& ''#&/
\$ ;B%D
12
Presentaon Prepared by Hajra Zareen, February 03, 2022
12
-). /00\$\$\$(*!(0)40)0***\$* 0
(1)
(2)
(3)
How can we find the biggest margin? (4/14)
C!\$  #!8L8'8L)8\$  ! '' * !!\$&
"/
\$ ;B%DM8
'
\$ ;B%DFM8
 8L8"'8L8'8L)(
L\$*  *%!8M8#(\$8MD!# %!(
\$ ;B%D8
'
\$ ;B%DF8
:\$\$\$8%  # *%\$ (
C\$H!8any #!N\$\$!!!#! \$  \$!!\$&8constraints/
 *8;8 /
\$ ;B%O8; *& !

\$ ;B%PF8; *& !F
13
Presentaon Prepared by Hajra Zareen, February 03, 2022
13
-). /00\$\$\$(*!(0)40)0***\$* 0
(4)
(5)
How can we find the biggest margin? (5/14)
Understanding the constraints
Q !!\$&&!!' * !88'!!%! * !8F(
!H!18Figure 4%!\$'' 8(,'  !88'\$'
*#'*! 8\$ ;B%O8
C 8;D8\$    #!8\$ ;B%D' 
'(9 !8R(
C 8;D\$  %*8  #!8\$ ;B%S8' 
'(9 !8888'8K(
C !&&# !''  '' 
!8F(
What does it mean when a constraint is not respected ?
, \$!8 \$ #!(8< *#8 
'Figure 6, 7 and 8)8 %\$ \$ #!(
R#'& \$'\$# !&!!&8\$
#!8without points between them.8
,\$1!#;!%!8'T
14
Presentaon Prepared by Hajra Zareen, February 03, 2022
14
How can we find the biggest margin? (6/14)
15
Presentaon Prepared by Hajra Zareen, February 03, 2022
15
&4/9\$ #!#&  &U/9\$ #!!#&  &/9 &  #!'# 
&/9 ! #!'# ' &/R '
How can we find the biggest margin? (7/14)
Combining both constraints
"8(4)8'8(5)8%%'&!/
C\$ "8(5)
for@xi having the class−1
w xi+b ≤−1
'!!#% '%#8#\$  !\$# "
yi(w xi+b) ≥ yi (−1)
C  "8(5)8!%8\$/
#\$ ;B%O8; *& !F
16
Presentaon Prepared by Hajra Zareen, February 03, 2022
16
(6)
How can we find the biggest margin? (8/14)
,"8(4)8#D8'H & & "(
#\$ ;B%O8; *& !
C%"@(6)8'8(7)@/
yi (w xi+b) ≥ 1for all 1 ≤ I ≤ n
C\$ *""88'\$"848'858% #
 !!#"*!(
  8 \$!!%%\$ \$ #!(
Step 3:@Maximize the distance between the two hyperplanes
a) What is the distance between our two hyperplanes ?
R#&;A '%\$ \$ #!\$\$!!81!*/ \$
'\$8?
17
Presentaon Prepared by Hajra Zareen, February 03, 2022
17
(7)
(8)
How can we find the biggest margin? (9/14)
+/
L%  #! *& "8\$ ;B%DF
L%  #! *& "8\$ ;B%D
V%  #!8L(
C\$!!!!88 '!'8;8  #!8L8(R#'88\$ 
\$'!!8the margin(
8;88L88 '%\$ #!8L8'8L8(
We will now try to find the value of8(
<& %' 1 \$''888;8\$\$!!& ' \$!!
%   #!T
R8it@does not work%888scalar'8;88vector8'''&!\$ *
%!(
L\$*\$1\$ ''&\$*%!\$88*8\$\$!!
%%!'''(
C' !!\$  '8888;(,%'8
!/
18
Presentaon Prepared by Hajra Zareen, February 03, 2022
18
How can we find the biggest margin? (10/14)
Step 3:@Maximize the distance between the two hyperplanes
+1& & #*%!(C 2 !& 88\$'H *
!/8the direction(* &'''(
CH''!*%\$1\$\$8!!#!\$ *\$\$!!
&8 *(
From our initial statement, we want @this vector:
 *&'8
%'!  #!8L
!#\$!'#1\$*'!L 8\$8%88LD\$ ;B%D
+H'8D\$0W\$W8 8*8\$(*8  D8'  ' 
8\$8!'!  #!(
,\$8!!#88%#88\$& *81D8'/
(1 D
)(18'!8L8%8  '8
 \$8 818 *\$\$!1&(
19
Presentaon Prepared by Hajra Zareen, February 03, 2022
19
How can we find the biggest margin? (11/14)
20
Presentaon Prepared by Hajra Zareen, February 03, 2022
20
&)/ '%\$ \$ #! &3!! ! '; &6/\$'!L
&/!'!L &4/AL
&5/1*!& '!L
How can we find the biggest margin? (12/14)
1DD\$0 \$ 
C''TC'!88*818\$  \$''
\$ 8 *8;(
,\$ 8;8'''818\$'  8AD;B188  #!8L8
 \$8Figure 14.
9  8A888L8 
\$ AB%D
C!8A8%#8;B18%  \$\$'(
\$ ;B1B%D
C\$!818&"8
\$ ;B\$0 \$ B%D  
C\$;'"8)
\$ ;B\$ \$ \$ B%D   
21
Presentaon Prepared by Hajra Zareen, February 03, 2022
21
(9)
(10)
(11)
(12)
(13)
How can we find the biggest margin? (13/14)
9 ''*\$ ! "/
8;888L8 8\$ ;B%DF
FDF \$  
 \$ D) 
D) \$  
22
Presentaon Prepared by Hajra Zareen, February 03, 2022
22
(14)
(15)
(16)
(17)
(18)
(19)
How can we find the biggest margin? (14/14)
b) How to maximize the distance between our two hyperplanes
C\$ *! &/
D) \$ 
9 !#*%!\$ & ! 8\$(
+H#&*'*!/
C 8 \$ D 8D) 
C 8 \$ D) 8D 
C 8 \$ D68 8D0) 
Q!#  %&&  !! &%(
23
Presentaon Prepared by Hajra Zareen, February 03, 2022
23
Minimizing the norm of w
24
C 8M8 &%\$ &(&%\$0X\$X(! #!
;A &  7%2*%!''\$;A& 1/|w|
Q%*  &! %''''& !8w# "8w.x+b=0T
L\$*   !  \$           !   ! \$    \$ 
#!T9 % !!#'/
Presentaon Prepared by Hajra Zareen, February 03, 2022
(5)
Vanilla (Plain) SVM & its Objective Function
25
So, what do these constraints mean?
,  %'  #! \$ %!\$&' 
!!'8Hard Margin SVM (Vanilla SVM).
9  %&'\$%17T9 \$!'%!!#%!(9 * !
\$!''T9 \$ 8Soft Margin SVMs8!#(
Do SVMs use Gradient Descent to find the minima?:\$#T7!!1\$ (9 
'!#'%#!*&'** 8Objective8function(
 \$@!!'1 8Langrangian of the Objective Function8!* 
(8
Presentaon Prepared by Hajra Zareen, February 03, 2022
-). /00\$''(0* '!!"'&&"!!A463U46''
&(U/L'7& &/7&
Soft Margin SVM (1/2)
26
'%*7&7 '!!\$ %!'(9 &%!\$
;! !!#T
 @ & '% '7&/
9!!\$ 711'#1 &\$'%!(
Presentaon Prepared by Hajra Zareen, February 03, 2022
&)/7&
Soft Margin SVM (2/2)
27
:\$ \$%!'%!!7?
C2''!!A& Q%2*!!' 𝜉(9  '
;'' * #!%#'\$'  !(
9  '\$!!\$  %'# #! !A& 8(!
 '  '  8"!'%\$ '' 
#!(L *! 8&*%(
9 %A'\$  %!\$"/
' !''!!# %YmistakesZ 71(9 \$%2*
\$!'%/
Presentaon Prepared by Hajra Zareen, February 03, 2022
(20)
(21)
The Kernel Trick (1/2)
28
L!!\$  8 F8% %1T9 '
 8!! '
[!17 %''!#%#!*&'**+&& 
'**(
The Kernel Trick
:\$!1&%1\$ \$@*'*'! \$!#&8w.x+b(9 !#!#!
"(9 7\$1%\$ #!# '!!#T
9  !!# &!TL\$*   *' 1 > @ 1!
1(,!/
Presentaon Prepared by Hajra Zareen, February 03, 2022
-). /00\$''(0* '!!"'&&"!!A463U46''
&)/=!91
The Kernel Trick (2/2)
29
9 =!!#* !'!'  7%\$
!(
9  \$ %!\$&(
L \$!"\$!'%
L1'* 8n-dimension88N-dimension8\$ 8N>n(
9 =!8 @%# !!\$&'/
Presentaon Prepared by Hajra Zareen, February 03, 2022
&))/:!+ &)3/:!+
(8)
Kernel Functions
30
9 *!1!'7( !/
\$  > 0 (
!8 = 1/ ² 2
Gaussian Kernel/
Polynomial Kernel/
Sigmoid kernel/
Presentaon Prepared by Hajra Zareen, February 03, 2022
(22)
(25)
(24)
(23)
&!A8%!/87 +)&!A( &'&!A
%!\$  **&(
L'!!'!#/87!# '!!'&=!1(
!*% !'&%!/87%'!*% !
'&%!(
7'!%!\$ !8SVR (Support Vector Regression)8'
&%!(
%!#/8!! & ''&!#  #!'  7(
 7'!%!(
 &88=!'!/8 &88=!
'! !'#1(
,!'%1#'!;(,& & '=!#& &
#*\$  ' &''!!#(8
31
Presentaon Prepared by Hajra Zareen, February 03, 2022
31
;*#"/8!& !;#'#"7
*# & (
<'!## *!! * #' 
%&\$%!#\$  &'A(
"!&/8Q'!&*%!%!#&7(
+&&/871!&&!&'(
!/87'!'!'''%# %&!1
9(
32
Presentaon Prepared by Hajra Zareen, February 03, 2022