Figure - available via license: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Content may be subject to copyright.
The generalization performance of the fine-tuned transfer learning models for each model (InceptionV3 and VGG16). And generalization and in-train performance of a one-layer GBNN and NN-trained models on the output of pre-trained models on the CIFAR-10.
Source publication
This paper presents a novel technique based on gradient boosting to train the final layers of a neural network (NN). Gradient boosting is an additive expansion algorithm in which a series of models are trained sequentially to approximate a given function. A neural network can also be seen as an additive expansion where the scalar product of the res...
Context in source publication
Context 1
... the differences between method in this dataset both in-train and in test are negligible. Finally, the performances of the GBNN and NN models trained on the output of InceptionV3 and VGG16 for CIFAR-10, and the fine-tuned transfer learning models as well are shown in Table 5. The application of GBNN on top of the pretrained and fine-tuned models, gained 0.08 and 0.07 percent points in accuracy with respect to InceptionV3 and VGG16 respectively. ...Citations
... The final model combines all generated models in an additive manner. These ideas have also been applied to sequentially train Neural Networks or NNs [17]- [20]. In [19], a Gradient Boosting (GB) based approach that uses a weight estimation model to classify image labels is proposed. ...
... In addition, it also involves formulating linear classifiers and feature extraction, where the feature extraction produces input for the linear classifiers, and the resulting approximated values are stacked in a ResNet layer. In [17], a novel technique for training shallow NNs sequentially via GB is proposed. The method called Gradient Boosted Neural Network (GBNN) involves constructing one NN by utilizing the trained weights of multiple individual networks, each trained on the residual loss sequentially. ...
... In fact, the method is tested experimentally only for datasets with two features. In another line of work, a shallow NN is sequentially trained as an additive expansion using GB [17], [21]. The weights of the trained models are stored to form a final neural network. ...
Deep learning has revolutionized computer vision and image classification domains. In this context Convolutional Neural Networks (CNNs) based architectures and Deep Neural Networks (DNNs) are the most widely applied models. In this article, we introduced two procedures for training CNNs and DNNs based on Gradient Boosting (GB), namely GB-CNN and GB-DNN. These models are trained to fit the gradient of the loss function or pseudo-residuals of previous models. At each iteration, the proposed method adds one dense layer to an exact copy of the previous deep NN model. The weights of the dense layers trained on previous iterations are frozen to prevent over-fitting, permitting the model to fit the new dense as well as to fine-tune the convolutional layers (for GB-CNN) while still utilizing the information already learned. Through extensive experimentation on different 2D-image classification and tabular datasets, the presented models show superior performance in terms of classification accuracy with respect to standard CNN and DNN with the same architectures.
