Content uploaded by Nadeem Javaid
Author content
All content in this area was uploaded by Nadeem Javaid on Feb 22, 2022
Content may be subject to copyright.
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Support Vector Machine (SVM)
Presentation Edited
By
Hajra Zareen
(Date: 03/2/2022)
Under the Supervision of
Dr. Nadeem Javaid
COMSATS University Islamabad, Islamabad Pakistan
1
Outline
Preliminaries
Introduction
Hyper-plane
Optimal Hyper plane
How to choose Optimal Hyperplane?
How can we find the biggest margin?
Minimizing the norm of w
Vanilla (Plain) SVM & its Objective Function
Soft Margin SVM
The Kernel Trick
Kernel Functions
Advantages and Disadvantages
2
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
2
Preliminaries (1/2)
3
Basic Linear Algebra
Vectors: !"#$ % &'''( )!
%'*%$&' (
Length of Vectors: +& *!!!'(,!! $* &(
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Fig.1: Vectors
Preliminaries (2/2)
4
Basic Linear Algebra
Direction of vectors: *(
Dot Product: '%$$*!"#(,!! $*!'(
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Introduction
7%''$ &/!&(
SVM8'!
SVR8&'&
Classification
9 &!7!& $*'%#8!':(81'!;#<(8 *18
53(
9 )1!( ' '!#$ !!' =!91$ !!$
7!#!!#%!8'(
*!!#'1'' 7&!$ !!$
!8$ &7(
Regression
,51!('*7&'!(,
!!'&(
+1 !78 '!!' #' 1!1(
5
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
5
Hyper-plane
6
,! !!#'*' '!'$(,) #!
!3!(
,!!!'8n-dimensional line. Fig.28 $%!! #!!!# '
$(
, 8Fig.2, hyperplane!'*''$!'>&$
-. /00123('(0 % '* *4345'6'
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Fig.2: Linearly Separable Fig.3: Non-Linearly Separable
Optimal Hyperplane
7
If#!1%!$8Fig.48 % #! '$(8
So optimal hyperplane is one which divides the data points very well("$ #''
! #!?
So, if you choose sub-optimal hyperplane, no doubt after number of training iteration , training error will
decrease but during testing when an unseen instance will come, it will result in high test error.
In that case it is must to choose an optimal plane to get good accuracy.
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Fig 4: Different Hyperplanes
How to choose Optimal Hyperplane? (1/2)
8
Margin and Support Vectors
+@ !'%!1!%*&(! #!'$''!
#!$ & & ' ! #!(
9 '%$ #!'! #!1$&' !'
1$*(
7&$ '#'(9 $!!%$ $ *'
&%& $$1&'!'(
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Fig.5: Margin and Support Vectors
How to choose Optimal Hyperplane? (2/2)
9
Margin and Support Vectors
$ $ &! #!$$!! & #!$ &
' !'(
,! #!*#!' &$!!%*#!!'$!!&!A$!!
&'%$ '$!!$!!!&!A$!!;!'%*(8
&!;A & !%!&!A$!!(
So, In SVM our goal is to choose an optimal hyperplane which maximizes the margin.
Mathematical Interpretation of Optimal Hyperplane
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
Fig .6: Optimal Hyperplane
How can we find the biggest margin? (1/14)
< *'
!$ #!$ '8$ 8%$
7;A ' &
Step 1: You have a dataset@D,@and you want to classify it
7 #'$!!%'88*8;(
8;$!!!%'$ *!8#8'& !%!& !B
(
: 8#88!# *$%!*!B(
7* $ #';!#*8;8'
*&!'(C# 8;88'!* 88'(
9 !'!' #/
DE;# ; # EFGGD
10
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
10
-). /00$$$(*!(0)40)0***$* 0
How can we find the biggest margin? (2/14)
Step 2: You@need to select two hyperplanes separating the data with no points between them
!H8 '88,!!#%!(
C$$'$ #!$ %$ %$'H *8$#
*!A (
C '$1$% #! !' !?
Taking another look at the hyperplane equation
89 " #!%$
$9;D
Any hyperplane can be written as the set of points;satisfying$ ;B%D.
$&A '' !8$ ;8'8$9;(
<& $'(((8C ' 8B%8?8
,"(,' *8$8'8;8 * '$ !
C1'' # *$'/
11
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
11
-). /00$$$(*!(0)40)0***$* 0
How can we find the biggest margin? (3/14)
Given two 3-dimensional vectors@w(b,−a,1) and@x(1,x,y)
$ ;D%IBFI;BI#
$ ;D#F;B%
Given two 2-dimensional@vectors@w′(−a,1) and@x′(x,y)
$J ;JDFI;BI#
$J ;JD#F;
Now if we add@b@on both side of the equation@(2)@we got :
$J ;JB%D#F;B%
$J ;JB%D$ ;
K* #!8L8& ''#&/
$ ;B%D
12
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
12
-). /00$$$(*!(0)40)0***$* 0
(1)
(2)
(3)
How can we find the biggest margin? (4/14)
C!$ #!8L8'8L)8$ ! '' * !!$&
"/
$ ;B%DM8
'
$ ;B%DFM8
8L8"'8L8'8L)(
L$* *%!8M8#($8MD!# %!(
$ ;B%D8
'
$ ;B%DF8
:$$$8% # *%$ (
C$H!8any #!N$$!!!#! $ $!!$&8constraints/
*8;8 /
$ ;B%O8; *& !
$ ;B%PF8; *& !F
13
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
13
-). /00$$$(*!(0)40)0***$* 0
(4)
(5)
How can we find the biggest margin? (5/14)
Understanding the constraints
Q !!$&&!!' * !88'!!%! * !8F(
!H!18Figure 4%!$'' 8(,' !88'$'
*#'*! 8$ ;B%O8
C 8;D8$ #!8$ ;B%D'
'(9 !8R(
C 8;D$ %*8 #!8$ ;B%S8'
'(9 !8888'8K(
C !&&# !'' ''
!8F(
What does it mean when a constraint is not respected ?
, $!8 $ #!(8< *#8
'Figure 6, 7 and 8)8 %$ $ #!(
R#'& $'$# !&!!&8$
#!8without points between them.8
,$1!#;!%!8'T
14
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
14
How can we find the biggest margin? (6/14)
15
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
15
&4/9$ #!#& &U/9$ #!!#& &/9 & #!'#
&/9 ! #!'# ' &/R '
How can we find the biggest margin? (7/14)
Combining both constraints
"8(4)8'8(5)8%%'&!/
C$ "8(5)
for@xi having the class−1
w xi+b ≤−1
'!!#% '%#8#$ !$# "
yi(w xi+b) ≥ yi (−1)
C "8(5)8!%8$/
#$ ;B%O8; *& !F
16
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
16
(6)
How can we find the biggest margin? (8/14)
,"8(4)8#D8'H & & "(
#$ ;B%O8; *& !
C%"@(6)8'8(7)@/
yi (w xi+b) ≥ 1for all 1 ≤ I ≤ n
C$ *""88'$"848'858% #
!!#"*!(
8 $!!%%$ $ #!(
Step 3:@Maximize the distance between the two hyperplanes
a) What is the distance between our two hyperplanes ?
R#&;A '%$ $ #!$$!!81!*/ $
'$8?
17
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
17
(7)
(8)
How can we find the biggest margin? (9/14)
+/
L% #! *& "8$ ;B%DF
L% #! *& "8$ ;B%D
V% #!8L(
C$!!!!88 '!'8;8 #!8L8(R#'88$
$'!!8the margin(
8;88L88 '%$ #!8L8'8L8(
We will now try to find the value of8(
<& %' 1 $''888;8$$!!& ' $!!
% #!T
R8it@does not work%888scalar'8;88vector8'''&!$ *
%!(
L$*$1$ ''&$*%!$88*8$$!!
%%!'''(
C' !!$ '8888;(,%'8
!/
18
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
18
How can we find the biggest margin? (10/14)
Step 3:@Maximize the distance between the two hyperplanes
+1& & #*%!(C 2 !& 88$'H *
!/8the direction(* &'''(
CH''!*%$1$$8!!#!$ *$$!!
&8 *(
From our initial statement, we want @this vector:
*&'8
%'! #!8L
!#$!'#1$*'!L 8$8%88LD$ ;B%D
+H'8D$0W$W8 8*8$(*8 D8' '
8$8!'! #!(
,$8!!#88%#88$& *81D8'/
(1 D
)(18'!8L8%8 '8
$8 818 *$$!1&(
19
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
19
How can we find the biggest margin? (11/14)
20
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
20
&)/ '%$ $ #! &3!! ! '; &6/$'!L
&/!'!L &4/AL
&5/1*!& '!L
How can we find the biggest margin? (12/14)
1DD$0 $
C''TC'!88*818$ $''
$ 8 *8;(
,$ 8;8'''818$' 8AD;B188 #!8L8
$8Figure 14.
9 8A888L8
$ AB%D
C!8A8%#8;B18% $$'(
$ ;B1B%D
C$!818&"8
$ ;B$0 $ B%D
C$;'"8)
$ ;B$ $ $ B%D
21
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
21
(9)
(10)
(11)
(12)
(13)
How can we find the biggest margin? (13/14)
9 ''*$ ! "/
8;888L8 8$ ;B%DF
FDF $
$ D)
D) $
22
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
22
(14)
(15)
(16)
(17)
(18)
(19)
How can we find the biggest margin? (14/14)
b) How to maximize the distance between our two hyperplanes
C$ *! &/
D) $
9 !#*%!$ & ! 8$(
+H#&*'*!/
C 8 $ D 8D)
C 8 $ D) 8D
C 8 $ D68 8D0)
Q!# %&& !! &%(
23
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
23
Minimizing the norm of w
24
C 8M8 &%$ &(&%$0X$X(! #!
;A & 7%2*%!''$;A& 1/|w|
Q%* &! %''''& !8w# "8w.x+b=0T
L$* ! $ ! ! $ $
#!T9 % !!#'/
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
(5)
Vanilla (Plain) SVM & its Objective Function
25
So, what do these constraints mean?
, %' #! $ %!$&'
!!'8Hard Margin SVM (Vanilla SVM).
9 %&'$%17T9 $!'%!!#%!(9 * !
$!''T9 $ 8Soft Margin SVMs8!#(
Do SVMs use Gradient Descent to find the minima?:$#T7!!1$ (9
'!#'%#!*&'** 8Objective8function(
$@!!'1 8Langrangian of the Objective Function8!*
(8
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
-). /00$''(0* '!!"'&&"!!A463U46''
&(U/L'7& &/7&
Soft Margin SVM (1/2)
26
'%*7&7 '!!$ %!'(9 &%!$
;! !!#T
@ & '% '7&/
9!!$ 711'#1 &$'%!(
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
&)/7&
Soft Margin SVM (2/2)
27
:$ $%!'%!!7?
C2''!!A& Q%2*!!' 𝜉ᵢ(9 '
;'' * #!%#'$' !(
9 '$!!$ %'# #! !A& ᵢ8(!
' ' ᵢ8"!'%$ ''
#!(L *! ᵢ8&*%(
9 %A'$ %!$"/
' !''!!# %YmistakesZ 71(9 $%2*
$!'%/
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
(20)
(21)
The Kernel Trick (1/2)
28
L!!$ 8 F8% %1T9 '
ᵢ8!! '
[!17 %''!#%#!*&'**+&&
'**(
The Kernel Trick
:$!1&%1$ $@*'*'! $!#&8w.x+b(9 !#!#!
"(9 7$1%$ #!# '!!#T
9 !!# &!TL$* *' 1 > @ 1!
1(,!/
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
-). /00$''(0* '!!"'&&"!!A463U46''
&)/=!91
The Kernel Trick (2/2)
29
9 =!!#* !'!' 7%$
!(
9 $ %!$&(
L $!"$!'%
L1'* 8n-dimension88N-dimension8$ 8N>n(
9 =!8 @%# !!$&'/
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
&))/:!+ &)3/:!+
(8)
Kernel Functions
30
9 *!1!'7( !/
Gaussian Radial Basis Function (RBF)/
•$ > 0 (
•!8 = 1/ ² 2
•Gaussian Kernel/
Polynomial Kernel/
Sigmoid kernel/
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
(22)
(25)
(24)
(23)
Advantages and Disadvantages (1/2)
Advantages:
&!A8%!/87 +)&!A( &'&!A
%!$ **&(
L'!!'!#/87!# '!!'&=!1(
!*% !'&%!/87%'!*% !
'&%!(
7'!%!$ !8SVR (Support Vector Regression)8'
&%!(
%!#/8!! & ''&!# #!' 7(
7'!%!(
Disadvantages:
&88=!'!/8 &88=!
'! !'#1(
,!'%1#'!;(,& & '=!#& &
#*$ ' &''!!#(8
31
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
31
Advantages and Disadvantages (2/2)
Disadvantages:
;*#"/8!& !;#'#"7
*# & (
<'!## *!! * #'
%&$%!#$ &'A(
"!&/8Q'!&*%!%!#&7(
+&&/871!&&!&'(
!/87'!'!'''%# %&!1
9(
32
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
32
-). /00$''(0* '!!"'&&"!!A463U46''
Thank You !!!
33
Presentaon Prepared by Hajra Zareen, February 03, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan