Content uploaded by Nadeem Javaid
Author content
All content in this area was uploaded by Nadeem Javaid on Feb 23, 2022
Content may be subject to copyright.
K- Nearest Neighbor Algorithm (KNN)
Edited By:
Shakira Musa Baig
(Date: 1/1/2022)
Supervisor: Prof. Dr. Nadeem Javaid
COMSATS University Islamabad, Islamabad Pakistan
!! "#$%&'('()
Outline
!""# $!""%
&'!""
()*)!""# %
+*)!""*,%
+* -! !""# %
)! !""# %
!""#
!""#
!""#
!! "#$%&'('()
Preliminaries (1/3)
Parametric Algorithm: . # ( , )$) * *
,* $)) *$
) $) ) / Examples: 0
("-1()/
Non-Parametric Algorithm: # ),()(#
* $ )/ Examples: 2 3 )
!""/
Lazy Learner: 405(0( #))*- )/
605( ) $ ))) #
)/ 6 , # $ )/ 05( ,*
$)$ ( #)#$)
/Examples: !"")$)#/
Eager Learner: &# 0 )- ) $) # ) $
-#)/705(�,#)
)/ , $# )( - ) # (
$-))/Examples: ."8-1(""/
!! "#$%&'('()
Preliminaries (2/3)
Majority Rule: 39() -* -9(
-/
Curse of Dimensionality: . $ # ),* (/
Overfitting: )#)* )6- )
:) ) #- )/ 1 ) #
) - ) ) ) ( )
( )/
Distance: * $9 /. )
1));1;/
Right Angle Triangle: # ## #)#( ##
#* ## # <)##/
Hypotenuse: 6 # ) # #) # ) # #/ .
# ($)# ( # /
!! "#$%&'('()
*
Preliminaries (3/3)
Pythagorean theorem: 6# #)# : (): :
* )/ . ) # - $ ) )1 ) +(/.
( #) #<=/
Euclidean Distance: . &) ) $* * &) #- ) $* *
# )/ 6 $ ) ) #
( # ($#) ( #)/
!! "#$%&'('()
*
Fig 1. Euclidean distance between point P1 and P2
(1+
What is K-Nearest Neighbor (KNN)? (1/3)
!# $!""# (-)30# * $)$
*#)-$/+*-()/
!""4non-parametric algorithm* ),()(#
)/
6 *) ))) )/. *$-(
* *)))* /
6 )4lazy learner algorithm4$ ) # )(
) )) )/
. ,# )# *) /(# )
(/
6 *!"")))#)*:
(##)/
!""# ($* *>))-$) *
#( -$#/
!! "#$%&'('()
,
What is K-Nearest Neighbor (KNN)? (2/3)
!"" # -$ ) ) * ) $) (/
. * *) $()*#($(#
!""# /
!""# # 9 ))* #*)
)#( *)/
!!"" $ # $*)(*)/
. $# $ ))#/
!#())$ $/
? * $))/() ?)
$ #)?/
. ($(9(-,# $/
& $9- ) * -, )/)#
() )$*#) &)
)/
!! "#$%&'('()
-
What is K-Nearest Neighbor (KNN)? (3/3)
!"" *#$@
?/ )
/ )# $
/ 2$
!! "#$%&'('()
.
Fig 1. KNN steps
Example
* -# ,))#$**,*
)#/ )* !""# *,(
/7!"")*) *) ))##
)$) * )##(/
!! "#$%&'('()
/
Fig 2. Example of KNN classifier
Why do we need a KNN algorithm?
*#//#()#(1)* -*)'?
) * * #/ .- ( $* ) !""
# /
!""*()( #()/)
$*)#@
!! "#$%&'('()
0
Fig 3. Example of KNN
How does KNN work? (1/4)
. !""*,#$') $ $*# @
Step-1:4 $! # $
Step-2:4 &))4K number of neighbors
Step-3:4., !# $ )&))/
Step-4:4# ,# $ $ ) #(/
Step-5:4# *) #(* $ # $'/
Step-6:47))(/
!! "#$%&'('()
1
How does KNN work? (2/4)
* - * )) * ) :) #(/) $*
#
!! "#$%&'('()
Fig 4. Example of KNN
How does KNN work? (3/4)
(** $# $** ,AB/
"'** 4Euclidean distance4$* )/. &))
)$*** * -)())#(/6$)@
!! "#$%&'('()
Fig 5. Euclidean distance
How does KNN work? (4/4)
1( # &) ) * # # $ # $
#()*# $#(1/
) $*#@
* # $#( *)$#
#(/
!! "#$%&'('()
Fig 6. Example of KNN
What does K represents in the KNN Algorithm?
. $ # $!!"" ( () )
$)#/ ,!#-$ ))/
6 $ # $*)(*)/
& ) C*:/6 $# $ * -
# )#$# $,('-/
# $ '$ * * - * $ $ # - )
#$ # $ * - ) $)( * * - $
# $/
23,
!! "#$%&'('()
*
Fig 7. Example of KNN
How to select the value of K in the KNN Algorithm?
1*$* # -! !""# @
. *() $-D!D*)(-)
$ /. )-!B/
-(*-! !A?!A$())
)/
0#-!#)$())/
!! "#$%&'('()
,
Pros of KNN Algorithm
6)))/
6$ (#)/
. # !# $ )
# /
. ) ) #5 * ( !"" ,* )
$)## /
6-()$ $) # /
6-# **#/
6 -( # (/+*- $-)#) !""/
6$- #)#/
!! "#$%&'('()
-
Cons of KNN Algorithm
*()) -!* ($'/
. # ! # $ * ) )
(/
6:#(# #))/
!""$#))/
. # $ # ) $* )
#/3- #)/
+# (#:)) -)## /
)*$#"/
6-(- )*-/
!! "#$%&'('()
.
Applications of KNN Algorithm
. *# * !""$)(E
Banking System: !""$)$,#()* )-)
-% )-) - )%
Calculating Credit Ratings: !"" # $ ) ) )-)F ) # $(
#* -#/
Politics: !"" # *( --,
G2HG2HG2(I#FG2(I1JF/
7 * !"" # $ ) # +)*#
6##)2)#/
!! "#$%&'('()
/
References
456677789&8!6!)(
4566
7778(8!6!:(:7:6!:(:7:::(:;
::8
4566777886))6
45667!!8!6)()7)7)7)7),,-!.!*.<5=5>32??@@@
@7@@@@!(;8A>32??@!(;@@7@
@@@@
8
!! "#$%&'('()
0
Thank You !!!
!! "#$%&'('()
1