eXGBoost

Full text







 !"#$
$%&'#

( )*+(,- . #/*+
/*0
1
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

2
,!
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
•
•
• !"#$$%
•!"#&'(
•
•!"#&)*(&
•!"#&&*(&
•)
•

3
1
• !"#) & '+ & &*&   (+ ( &
&(,-
•&&&+(&.
•&&
•(&&
•($  ( &/ (   &  (0&  
&1&&&2-
•2 . & /*& (  3/   , &  , 
&&  (&& ,&4 , &  &   &*(
&&,&-
• &  & &&4 + / & &   &  
&4 , & &   & &//    &&  
-
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan


• &  &,  3 &   *   &,& + 
-
•%' *( ,&&(&&-
•#5*(&&&-

•6&4///&/,// /(/-/% / /&//&/,/+ 
/&/// /&/&// /&/-
•% &//&///&//// / //(/
/&/&//-
41
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

5
11

•/(/,&(4/ //&/&/&&7&/-
• //&//&(/ /&/&//&&/&,&-
•) & 8--  (& 8
&& +(-
Figure. 1
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

•639(( -
•%+&&&.
- :+ ( &
- ;(  < 4 +   & ,  ,  #((( 
#&( -
Figure. 2 shows the Bagging and Boosng Steps in XGBoost
3,, ./"
6
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

7
•&.
•!/"/#&(-
•!"#&&+*(( ,&(,&(-
•,& ( *&4+ &,5*&&,-
Fig. 3 shows the features of XGBoost
/!2
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

8
•%+&
- 8."4!"#&&&- &+  
 &(,&(-
- < . !"#& & &  , && 
&&(&&*(,&-
Fig. 4 shows the boosng in series and bagging in parallel
/!2
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

9
•#&&!"#&
•.4*5=, !"#&"#&(&&&
& ,&&&-
•25= &,&(&&4 &&,&-
•;   ,& &&   & ( &  (&& &  
&&&-
•% &&,&&&&&-
•."/=)+ 8&0 &0 
&'*( -
•* 5= &   6&  )#&4  & ( &
,& &&- 4 &,& 
&-
/!2
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

10
•)/ shrinkage & * &&, (-
•% &,& (  *(&*
 &-
•)*/&(0-&(,&&
 &*& &+ +,&*&-
•+  *(      ( *&   & +  
,&*&-
•#!"**/& ,,**&&-
•% ,,&&&,->*,&*4+  &&
, &-
•6  &,&3 &4   *& ,, &  ,&  
&&-
/!21
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

11
•$ + *        4 +      "- %
 "& +(.

•% &+  / ( &"/&  && ,&& -
•!"#&& &/" 4+  &-
•+&&  *"-( &+ /"
="?-(,+
•)&4  ( *:,++&
&&+"-
•;4&":,, &&*
    * ,&*& , ( & *&  ,(  
+  ,&*&-
/!2
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

12
(3)
Sum of Outputs of All the Trees
(2)
Set of base learners
•!"#&&&/2/&,&&-
•,(+'&&(&& +3-
•)&&&&&,( -
•@& &--&' -
306
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

13
306
•8(,&&(00*(4+  
(*&,-
•$,5*6.%(:&&A(0

•%  (0  &   8    4 +   & & 
**(-
•% &&&,,&4&&& +(, +(
-
•B&'//*&&(* & 
 (-
•  &(   '& ,&C



Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

14
Figure.5 Shows the balance between Loss and regularizaon terms.
3016
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

15
•%#&8"#&()( -
Figure. 6 Shows complete XGBoost working
306
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

16
•       !"#& 4 ;( &  &  ,
,  *"&,&&4<8% && 
, ( -
•1:( 2 <'&     ,& , & '(   +( & 
 &-
•$(.$(/&/(*(/4/-/ /&-
/(///&-& ,&&&&&,-
•8D .& 1 ( 2 *&&-
•E.% ,**( (-
>

F
3066
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

17
•% /γ/& + (*(-
•&&  &&3' 
 -
•--(+  &7   && ,4 +
+7-
•+/&/ /*//&&//*&//-
•Gradient(G) =(p-y) and Hessian(H) = p*(1-p).
• ,+& && ( -
3077
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

18
•6$(+  /&/(*(/-
•-/ /&//(///&-& ,&&&&&,-
•6++  /&/ /*//&&//*&//-
•%/&/ /,//*& G*&& , ( -
•6&&(&(&-
•6 & &4&&4(0
4(-
•$& &&&(06-
!
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

19
Draw Bootstrap
Samples
Train dataset
Build random
tree
Predict based
on each tree
Majority vote
Figure. 7 & 8 Shows steps required in XGBoost Figure. 8
8"
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

20
8"9
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

21
8"9
•First we take the base learner, by default the base model always take the average salary i.e (100k). Now, we calculate the residual values:
•6& + '   ,& 4,    ,&  +& '   *(
&-HF'-I+4+ &*&*(.
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

22
8"19
•I+4 7& &   & 4 + + , &(    ,& 
8?H +&-
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

23
8"9
•First we take the base learner, by default the base model always take the average salary i.e (100k). Now, we calculate the residual values:
•I+4 7&    & &    (  &- 4  &  
&&, &+,.
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

24
8"69
•First we take the base learner, by default the base model always take the average salary i.e (100k). Now, we calculate the residual values:
•I+4 ( &&&.
•4+&& (-+'
 &+   ( &(-
•:7&+' &(-6 4++& &
 &(-
•I ( &+-- ( 8&-
•& ( &-
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

25
8"79
•First we take the base learner, by default the base model always take the average salary i.e (100k). Now, we calculate the residual values:
•I && &,& "
,&(*-4+& ( &-
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

26
8"99
•First we take the base learner, by default the base model always take the average salary i.e (100k). Now, we calculate the residual values:
•% 4++ &+ 
(J 7&'->+  *&,&
 &-
•-.
G/HFA->KH>-
•4 ( & &, 
*(,&&-
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

27
#*
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan
•&;( 68,-
•&& +&&(-
•&& "#&(-
•&&( &&(+ &,&-
•'&+&&-
•&       &&  &*  ,  " 
,&(-
•8,&&&-
• -

28
•% 8 4 '++'-#&
&,&&-
•&&4&(+,3!"#&&
(&-
•!"#&&&+&-
•'&-
#$
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

29
% ( &&+(&.
•&&
•;( ( &&*&&
•,8&&
•&, *
•<<+&&
•(0
•;0&'
•<&&*&
""2$
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

30
:$2
LM &.GG&-G'G8(,&(,&(
LM &.GG+++-&& -G&&&F,N>G8(,&
LNM &.GG+++-&& -G (8O (G8(,&
LM!"#&P"'&"'&
L>M &.GG+++--G6D(- CHN>QQRHS
LFM &.GG+&&-G8(,&8(,&( +*((,&(>FFFS
LSM &.GG*--G8(,& &8>QF>NT
LQM &.GG+++-'((&-GQGQG*( &, 8(,&- 
LTM &.GG-G,(G8(,&* ('+U.V.8H&*(&4*W W&W W&,-
LM &.GG+++-&& -GX&+00'G8(,& (  +&*
LM &.GG+++-&& -G5G8(,&&&**+C3H,Q>,N,QNQNSTR*HR,HRD& HS
LM &.GG+++-&& -G O (NNG(8(,&SNFC3H,Q>,N,QNQNSTR*HR,HRD& 
LNM &.GG+++-&* -G,(GFGNG((8(,&+ & G
LM &.GG-5-(G+,G'(&G8(,&G*(&G8(,&&- 
L>M &.GG+++-&* -G,(GFGNG((8(,&+ & G
LFM &.GG+++-&* -G,(GQGTG(&  , 8(,&G
LSM &.GG+++--GG&G8(,& 
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan

31
.0;!<<<
Presentaon Prepared by Arshid Ali, February 14, 2022
Department of Computer Science, COMSATS University Islamabad, Islamabad-Pakistan