No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Convolutional Neural Network Hyperparameter Tuning with Adam Optimizer for ECG Classification
... Then the error value is converted to classification accuracy by subtracting the corresponding error value from 1. The Xavier initialization [55], rectified linear unit (ReLU) [56], and Adam optimizer [57] are used in this VOLUME 11, 2023 7 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. ...
... First, each generated CNN model must be trained using the given training dataset D train . The Xavier initialization [55], rectified linear unit (ReLU) [56], and Adam optimizer [57] are used in this work to initialize weights, activation function, and optimizing parameters of the CNN model. ...
... Finally, the number of neurons (f n ) for FC layers ranges from 1 to 300. Furthermore, to train the generated CNN models, we utilized ReLu or rectified linear unit [56] as activation function, Xavier weight initialization [55], and Adam optimization [57] algorithm due to their popularity in deep learning, respectively. The popular categorical cross-entropy [54] is used as to compute the classification accuracy of generated CNN models. ...
Recently, convolutional neural networks (CNNs) have shown promising achievements in various computer vision tasks. However, designing a CNN model architecture necessitates a high-domain knowledge expert, which can be difficult for new researchers while solving real-world problems like medical image diagnosis. Neural architecture search (NAS) is an approach to reduce the human intervention by automatically designing CNN architecture. This study proposes a two-phase evolutionary framework to design a suitable CNN model for medical image classification named TPEvo-CNN. The proposed framework mainly focuses on architectural depth search and hyper-parameter settings of the layered architecture for the CNN model. In the first phase, differential evolution (DE) is applied to determine the optimal number of layers for a CNN architecture, which enhances faster convergence to achieve CNN model architectures. In the second phase, the genetic algorithm (GA) is used to fine-tune the hyper-parameter settings of the generated CNN layer architecture in the first phase. Crossover and mutation operations of GA are devised to explore the hyper-parameter search space. Also, an elitism selection strategy is introduced to select the potential hyper-parameters of the CNN architecture for the next generation. The suggested approach is experimented on six medical image datasets, including pneumonia, skin cancer, and four COVID-19 datasets, which are categorized based on image types and class numbers. The experimental findings demonstrate the superiority of the proposed TPEvo-CNN model compared to existing hand-crafted, pre-trained, and NAS-based CNN models in terms of classification metrics, confusion matrix, radar plots, and statistical analysis.
... The optimizer type can be selected from: Stochastic Gradient Descent (SGD), Adaptive Moment Estimation (Adam), and Root Mean Square Propagation (RMSprop). An example of using the Adam optimizer can be found in [93]. The learning rate is another important hyperparameter that determines the step size in each iteration, which allows the speed of convergence to be calibrated. ...
... Dropout is a standard regularization method for deep learning models and has been, proposed to reduce overfitting. At dropout, a proportion of neurons is randomly removed, and the percentage of neurons to be removed must be adjusted [93]. ...
Cardiac arrhythmias are one of the main causes of death worldwide; therefore, early detection is essential to save the lives of patients who suffer from them and to reduce the cost of medical treatment. The growth of electronic technology, combined with the great potential of Deep Learning (DL) techniques, has enabled the design of devices for the early and accurate detection of cardiac arrhythmias. This article carries out a Systematic Literature Review (SLR) through a Systematic Mapping study and Bibliometric Analysis, through a set of relevant research questions (RQs), in relation to DL techniques applied to the automatic detection and classification of cardiac arrhythmias using electrocardiogram (ECG) signals, during the period 2017-2023. To identify the most pertinent scholarly articles, the PRISMA 2020 methodology was employed by quering the following databases: Scopus, IEEE Xplore, and PhysioNet Challenges, resulting in a total of 494 publications being retrieved. This study also includes a bibliometric analysis aimed at tracing the evolution of the primary technologies utilized in the automatic detection and recognition of cardiac arrhythmias. Additionally, it evaluates the performance of each technology, offering insights crucial for guiding future research endeavors.
... This optimiser can adjust learning rates based on gradients, offering faster convergence for MAGE data. Sena et al. [38] utilised Adam for ECG classification using Convolutional Neural Networks (CNNs). Random Adaptive Moment Estimation (RanAdam) is an extension of the Adam optimisation algorithm with the addition of randomisation. ...
... Adam is one of the commonly used optimisation algorithms used in training. It is more effective in handling non-convex optimisation problems as mentioned by Sena et al. [38]. The key parameters used by Adam for tuning are learning rate, β 1 , β 2 , ∈, and decay rates. ...
Microarray gene expression analysis is a powerful technique used in cancer classification and research to identify and understand gene expression patterns that can differentiate between different cancer types, subtypes, and stages. However, microarray databases are highly redundant, inherently nonlinear, and noisy. Therefore, extracting meaningful information from such a huge database is a challenging one. The paper adopts the Fast Fourier Transform (FFT) and Mixture Model (MM) for dimensionality reduction and utilises the Dragonfly optimisation algorithm as the feature selection technique. The classifiers employed in this research are Nonlinear Regression, Naïve Bayes, Decision Tree, Random Forest and SVM (RBF). The classifiers’ performances are analysed with and without feature selection methods. Finally, Adaptive Moment Estimation (Adam) and Random Adaptive Moment Estimation (RanAdam) hyper-parameter tuning techniques are used as improvisation techniques for classifiers. The SVM (RBF) classifier with the Fast Fourier Transform Dimensionality Reduction method and Dragonfly feature selection achieved the highest accuracy of 98.343% with RanAdam hyper-parameter tuning compared to other classifiers.
... This optimizer can adjust learning rates based on gradients, offering faster convergence for MAGE data. Sena et al. [38] utilized Adam for ECG classification using Convolutional Neural Networks (CNNs). Random Adaptive Moment Estimation (RanAdam) is an extension of the Adam optimization algorithm with the addition of randomization. ...
... Adam is one of the commonly used optimization algorithms used in training. It is more effective in handling non-convex optimization problems as mentioned by Sena et al. [38]. The key parameters used by Adam for tuning are learning rate, β1, β2, ∈, and decay rates. ...
Microarray gene expression analysis is a powerful technique used in cancer classification and research to identify and understand gene expression patterns that can differentiate between different cancer types, subtypes, and stages. However, microarray databases are highly redundant, inherently nonlinear, and noisy. Therefore, extracting meaningful information from such a huge database is a challenging one. The paper adopts the Fast Fourier Transform (FFT) and Mixture Model (MM) for dimensionality reduction and utilizes the Dragonfly optimization algorithm as the feature selection technique. The classifiers employed in this research are Nonlinear Regression, Naïve Bayes, Decision Tree, Random Forest and SVM (RBF). The classifiers' performances are analyzed with and without feature selection methods. Finally, Adaptive Moment Estimation (Adam) and Random Adaptive Moment Estimation (RanAdam) hyper-parameter tuning techniques are used as improvisation techniques for classifiers. The SVM (RBF) classifier with the Fast Fourier Transform Dimensionality Reduction method and Dragonfly feature selection achieved the highest accuracy of 98.343% with RanAdam hyperparameter tuning compared to other classifiers.
... Adam, öğrenme oranını otomatik olarak ölçeklendirmekte ve geniş bir kullanım alanına sahip olmaktadır. Literatür incelendiğinde trafik işareti sınıflandırmasında [15], Rulman arızası teşhisi amacıyla [16]ve EKG sınıflandırması [17] çalışmalarında kullanıldığı görülmektedir. Matematiksel ifadesi Denklem 5 ile verilmiştir. ...
Bu çalışma, farklı konvolüsiyonel sinir ağlarında kullanılan optimizatörlerin eğitim doğruluğu (train accuracy), doğrulama doğruluğu (validation accuracy), eğitim kaybı (train loss) ve doğrulama kaybı (validation loss) metriklerinin üzerindeki etkilerini karşılaştırmayı amaçlamaktadır. Çalışma kapsamında yapılan deneysel işlemlerde SGD, Momentum SGD, RMSprop, AdaGrad, Adam, AdaDelta ve Nadam olmak üzere yedi farklı optimizatör kullanılmıştır. Kullanılan her optimizatör için, modelin eğitim sürecinde elde edilen doğruluk ve kayıp değerleri analiz edilmiştir. Çalışmanın bulguları, Momentum SGD, Adam ve Nadam gibi modern optimizatörlerin, diğer optimizatörlere göre daha yüksek eğitim doğruluğu ve düşük kayıp değerleri sağladığını göstermektedir. Özellikle, Adam optimizatörü, en düşük eğitim ve doğrulama kaybına sahip olup, en yüksek doğruluk oranlarını sunmaktadır. Öte yandan, AdaDelta gibi bazı optimizatörler, oldukça yüksek eğitim ve doğrulama kayıplarına yol açmış, doğruluk oranları ise düşük kaldığı görülmüştür. Aynı zamanda yapılan çalışmada her bir optimizatör için doğruluk ve kayıp değerlerinin görsel olarak karşılaştırılması sağlanmıştır. Çizgi grafikleri kullanılarak her optimizatör için eğitim ve doğrulama süreçleri detaylandırılmış ve her birinin zamanla nasıl bir performans gösterdiği net bir şekilde gözler önüne serilmiştir. Sonuç olarak, bu çalışma, optimizatörlerin model performansı üzerindeki önemli etkilerini vurgulamaktadır.
... Ultimately, the Adam optimizer [42,43] was chosen for its superior performance, with a learning rate of 0.0001. The Adam optimization algorithm enhances the computational efficiency of models and improves performance when working with large datasets and numerous parameters [44,45]. The mean square error was used as the basis of the loss function, and the root mean square error was selected as the metric for evaluating model accuracy. ...
Accurate soil moisture prediction is fundamental to precision agriculture, facilitating optimal irrigation scheduling, efficient water resource allocation, and enhanced crop productivity. This study employs a Long Short-Term Memory (LSTM) deep learning model, integrated with high-resolution ERA5 remote sensing data, to improve soil moisture estimation at the field scale. Soil moisture dynamics were analyzed across six commercial potato production sites in Quebec—Goulet, DBolduc, PBolduc, BNiquet, Lalancette, and Gou-new—over a five-year period. The model exhibited high predictive accuracy, with correlation coefficients (R) ranging from 0.991 to 0.998 and Nash–Sutcliffe efficiency (NSE) values reaching 0.996, indicating strong agreement between observed and predicted soil moisture variability. The Willmott index (WI) exceeded 0.995, reinforcing the model’s reliability. The integration of NDMI assessments further validated the predictions, demonstrating a strong correlation between NDMI values and LSTM-based soil moisture estimates. These findings confirm the effectiveness of deep learning in capturing spatiotemporal variations in soil moisture, underscoring the potential of AI-driven models for real-time soil moisture monitoring and irrigation optimization. This research study provides a scientifically robust framework for enhancing data-driven agricultural water management, promoting sustainable irrigation practices, and improving resilience to soil moisture variability in agricultural systems.
... Average and Variance are the first and second moments, respectively. In every iteration, the Adam method employs exponential moving averages to estimate moments in each batch [26]. Following are some mathematical formulae that may be understood in light of the update rule for the Adam optimizer: ...
Researchers have been more interested in automated insect recognition in recent years, and many different approaches have been taken to studying the practical implications of this field. When it comes to designing pest management tactics and safeguarding beneficial insects, accurate identification of the insects at play is crucial. Insect target detection has always relied heavily on artificial identification methods; however, deep learning can automatically extract characteristics for detection, solving the issue of poor detection accuracy due to subjective considerations. Here, we present our work on an improved method of insect identification using segmentation-based visual cues. The bug images were segmented using Faster RCNN with the ADAM optimizer, and the features were extracted using InceptionV3. The findings show that our suggested model outperformed competing methods in terms of accuracy and performance.
... ADAM (Adaptive Moment Estimation) is a viral optimization algorithm in deep learning [29]. It combines concepts from stochastic gradient descent (SGD) and momentum algorithms into an adaptive optimizer. ...
The palm oil industry faces significant challenges in accurately classifying fruit ripeness, which is crucial for optimizing yield, quality, and profitability. Manual methods are slow and prone to errors, leading to inefficiencies and increased costs. Deep Learning, particularly the AlexNet architecture, has succeeded in image classification tasks and offers a promising solution. This study explores the implementation of AlexNet to improve the efficiency and accuracy of palm oil fruit maturity classification, thereby reducing costs and production time. We employed a dataset of 1500 images of palm oil fruits, meticulously categorized into three classes: raw, ripe, and rotten. The experimental setup involved training AlexNet and comparing its performance with a conventional Convolutional Neural Network (CNN). The results demonstrated that AlexNet significantly outperforms the traditional CNN, achieving a validation loss of 0.0261 and an accuracy of 0.9962, compared to the CNN's validation loss of 0.0377 and accuracy of 0.9925. Furthermore, AlexNet achieved superior precision, recall, and F-1 scores, each reaching 0.99, while the CNN scores were 0.98. These findings suggest that adopting AlexNet can enhance the palm oil industry's operational efficiency and product quality. The improved classification accuracy ensures that fruits are harvested at optimal ripeness, leading to better oil yield and quality. Reducing classification errors and manual labor can also lead to substantial cost savings and increased profitability. This study underscores the potential of advanced deep learning models like AlexNet in revolutionizing agricultural practices and improving industrial outcomes.
... Random Search can explore hyperparameters more broadly and is generally more efficient than Grid Search in terms of computational time, particularly for large search spaces [54]. Approximately 25% of the studies reviewed adopted Model-free algorithms, including Grid Search [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69] and Random Search [70], [71], [72], [73], [74]. The details of these studies are summarized in Table 2. ...
Time series classification (TSC) is essential in various application domains to understand the system dynamics. The adoption of deep learning has advanced TSC, however its performance is sensitive to hyperparameters configuration. Manual tuning of high-dimensional hyperparameters can be labor intensive, leading to a preference for automatic hyperparameters optimization (HPO) methods. To the best of our knowledge, survey papers covering various studies on automatic hyperparameters optimization (HPO) of deep learning for TSC are scarce and even none. To address this gap, we present a systematic literature review to assist researchers in addressing the HPO problem for deep learning in TSC. We analyzed studies published between 2018 and June 2024. This review examines the HPO methods, hyperparameters, and tools utilized in this context based on 77 primary studies sourced from academic databases. The findings indicate that Metaheuristic algorithm and Bayesian Optimization are commonly employed approaches, with a focus on hyperparameters related to the deep learning architectures. This review provides insights that can inform the design and implementation of HPO strategies for deep learning models in time series analysis.
... The Adam optimizer is a well-known algorithm that dynamically adjusts learning rates by utilizing adaptive gradient-based momentum updates that consider past gradients. In essence, this algorithm merges aspects of RMSProp and Stochastic Gradient Descent with Momentum (SGDM), using squared gradients to adjust the learning rate akin to RMSProp [33]. Model optimization is a critical process aimed at enhancing the performance of AI models by meticulously identifying the optimal parameter set that minimizes the error or loss function, which is essential for improving both accuracy and the model's ability to generalize to new, unseen data. ...
The prediction of loan defaults is crucial for banks and financial institutions due to its impact on earnings, and it also plays a significant role in shaping credit scores. This task is a challenging one, and as the demand for loans increases, so does the number of applications. Traditional methods of checking eligibility are time-consuming and laborious, and they may not always accurately identify suitable loan recipients. As a result, some applicants may default on their loans, causing financial losses for banks. Artificial Intelligence, using Machine Learning and Deep Learning techniques, can provide a more efficient solution. These techniques can use various classification algorithms to predict which applicants will likely be eligible for loans. This study uses five Machine Learning classification algorithms (Gaussian Naive Bayes, AdaBoost, Gradient Boosting, K Neighbors Classifier, Decision Trees, Random Forest, and Logistic Regression) and eight Deep Learning algorithms (MLP, CNN, LSTM, Transformer, GRU, Autoencoder, ResNet, and DenseNet). The use of Ensemble Methods and SMOTE with SMOTE-TOMEK Techniques also has a positive impact on the results. Four metrics are used to evaluate the effectiveness of these algorithms: accuracy, precision, recall, and F1-measure. The study found that DenseNet and ResNet were the most accurate predictive models. These findings highlight the potential of predictive modeling in identifying credit disapproval among vulnerable consumers in a sea of loan applications.
... Further research is needed to enhance the understanding and improve the performance of CP decomposition. This includes reducing the training time of CP models through various optimization techniques, such as Adaptive Moment Estimation (Adam) [40], Adaptive Moment Estimation with a Modified Second Moment Term (AMSGrad) [41], or Adaptive Bound (AdaBound) [42,43]. Regularization methods, such as ridge regularization or, more generally, ℓ 2 regularization [44], can be implemented to introduce asymmetric treatment for two parameter vectors of the same axis but belonging to different ranks (i.e. ...
... The optimizers used here are, Adam Adam (Adaptive Moment Estimation) is an optimization algorithm that combines the concepts of momentum and RMSProp to achieve efficient and effective optimization in deep learning models. Adam uses momentum technique [20,21]. This technique accelerates gradient descent by include a fraction (β 1 ) of the prior update vector in the current update. ...
Images are the primary inputs to the computer vision applications. The quality of the captured images is depending on the characteristics of the environment. A poor visual background can result in dark images. The darker environment has less illumination of light. So, the captured image is of low quality, thus the images are termed as low light images. If this kind of image is the input to computer vision applications such as classification, clustering, and recognition, then the efficiency becomes very low. To improve the image quality this paper introduces deep learning-based architecture. This paper modifies the existing ResNet50 architecture by adding extra layers to make it compatible with the present objective. To improve the novelty of the model, it has been trained from the base layer with the low-light dataset. The proposed model is trained with different optimizers. Machine learning models are trained using a variety of optimizers, each with distinct properties that influence the model's convergence, learning speed and generalization ability to new data. The performance of the model is evaluated by using the performance parameters PSNR, Entropy, SSIM, MSE and Global Illumination.
... where w is the model weight, α is the learning rate, t is the number of iterations, and ε is the default value of 10−8 to avoid division by zero [55]. ...
Predetermining the risk of possible future droughts enables proactive measures to be taken in key areas such as agriculture, water management, and food security. Through these predictions, governments, non-governmental organizations, and farmers can develop water-saving strategies, encourage more efficient use of water, and minimize economic losses that may occur due to drought. Thus, future drought forecasts stand out as a strategic planning tool for the protection of natural resources. To achieve this aim, forecasted drought conditions for the next decade (2024–2034) at nine meteorological stations in the Sakarya basin, located northwest of Türkiye, are examined, using historical monthly precipitation data from 1991 to 2023. This study uses the Standardized Precipitation Index (SPI) and Long Short-Term Memory (LSTM) deep learning methods to investigate future meteorological droughts. The research confirms the compatibility and reliability of the LSTM method for forecasting meteorological droughts by comparing historical and forecasted SPI values’ correlograms and trends. In addition, drought maps are created to visually represent the spatial distribution of the most severe droughts expected in the coming years, and areas at risk of drought in the Sakarya Basin are determined. The study contributes to the limited literature on forward-looking drought forecasts in the Sakarya Basin and provides valuable information for long-term water resource planning and drought management in the region.
... The Adam optimizer [21], known for its effectiveness in handling sparse gradients and adaptable learning rate capabilities, was employed in the training process. The training process comprised multiple epochs, with the early termination of training based on validation loss, thus preventing overtraining. ...
This paper appraises convolutional neural network (CNN) models’ capabilities in emotion detection from facial expressions, seeking to aid the diagnosis of psychosomatic illnesses, typically made in clinical setups. Using the FER-2013 dataset, two CNN models were designed to detect six emotions with 64% accuracy—although not evenly distributed; they demonstrated higher effectiveness in identifying “happy” and “surprise.” The assessment was performed through several performance metrics—accuracy, precision, recall, and F1-scores—besides further validation with an additional simulated clinical environment for practicality checks. Despite showing promising levels for future use, this investigation highlights the need for extensive validation studies in clinical settings. This research underscores AI’s potential value as an adjunct to traditional diagnostic approaches while focusing on wider scope (broader datasets) plus focus (multimodal integration) areas to be considered among recommendations in forthcoming studies. This study underscores the importance of CNN models in developing psychosomatic diagnostics and promoting future development based on ethics and patient care.
... Given that the smoothness constant (L) is not available, practitioners have to tune the value of the different hyperparameters [37]. One may think that adaptive GD are less sensitive to these choices than GD and Momentum as the moving average (s n ) is used to scale the learning rate. ...
Both accelerated and adaptive gradient methods are among state of the art algorithms to train neural networks. The tuning of hyperparameters is needed to make them work efficiently. For classical gradient descent, a general and efficient way to adapt hyperparameters is the Armijo backtracking. The goal of this work is to generalize the Armijo linesearch to Momentum and RMSProp, two popular optimizers of this family, by means of stability theory of dynamical systems. We establish convergence results, under the Lojasiewicz assumption, for these strategies. As a direct result, we obtain the first guarantee on the convergence of the iterates for RMSProp, in the non-convex setting without the classical bounded assumptions.
... In this model, we have used 30 epochs, 1e-4 learn rate, and a 0.3 learn rate drop factor along with the Adam optimizer to get better performance. This optimizer gives the better performance compare all other optimizers [34]. ...
... These precise parameters were determined using best practices in the field of natural language processing, numerous tests, and prior research. Multiclass classification tasks use the "Adam" [73][74][75][76] optimizer and the "categorical_crossentropy" loss function [77,78]. A series of exploratory experiments were conducted to determine the best values for the input shape, batch size, and number of epochs in order to maximize model performance and minimize overfitting. ...
Recently, the widespread use of social media and easy access to the Internet have brought about a significant transformation in the type of textual data available on the Web. This change is particularly evident in Arabic language usage, as the growing number of users from diverse domains has led to a considerable influx of Arabic text in various dialects, each characterized by differences in morphology, syntax, vocabulary, and pronunciation. Consequently, researchers in language recognition and natural language processing have become increasingly interested in identifying Arabic dialects. Numerous methods have been proposed to recognize this informal data, owing to its crucial implications for several applications, such as sentiment analysis, topic modeling, text summarization, and machine translation. However, Arabic dialect identification is a significant challenge due to the vast diversity of the Arabic language in its dialects. This study introduces a novel hybrid machine and deep learning model, incorporating an attention mechanism for detecting and classifying Arabic dialects. Several experiments were conducted using a novel dataset that collected information from user-generated comments from Twitter of Arabic dialects, namely, Egyptian, Gulf, Jordanian, and Yemeni, to evaluate the effectiveness of the proposed model. The dataset comprises 34,905 rows extracted from Twitter, representing an unbalanced data distribution. The data annotation was performed by native speakers proficient in each dialect. The results demonstrate that the proposed model outperforms the performance of long short-term memory, bidirectional long short-term memory, and logistic regression models in dialect classification using different word representations as follows: term frequency-inverse document frequency, Word2Vec, and global vector for word representation.
... The ultimate goal of this work is to create a model capable of distinguishing various seed conditions, such as normal, abnormal, fresh germination, and dead seeds, while also determining their growth days. We evaluated the models performance using metrics such as accuracy, precision, recall, and F1 score, as discussed in references [12], [13], and [14]. ...
Rice Seeds (Oryza sativa) are currently certified by
BPSBTPH (Balai Pengawasan dan Sertifikasi Benih Tanaman
Pangan dan Hortikultura) based on their germination rate.
This process requires manual inspection by researchers and
takes a considerable amount of time. To address this issue, this
research proposes rice seed classification CNN (Convolutinal
Neural Network) models using our proposed models as STRNet (SINTANUR-Neural Network). The model is constructed
to classifying paddy rice seeds with multi-labels by growth
dataset withdays 3, 5, 7, and 14 and the rice seed quality
normal, abnormal, fresh, and dead categories. The result from
the growth dataset yields accuracy rates of 96% on day 5, 94%
on day 7, and 91% on day 14 when utilizing the STR-Net
architecture. On the other hand, the ResNet50v2 architecture
achieves accuracy rates of 98% on day 5, 97% on day 7, and
92% on day 14 with fixed hyperparameters. In addition, the
accuracy of STR-Net model generated from the Quality dataset.
The accuracy of the predictions obtained from the STR-Net
models is satisfactory, reaching 92.59%, while the ResNet50v2
model achieves a slightly higher accuracy of 92.84%.
... The batch sizes were tested with the same values as Case Study B1. The available values for Beta1 and Beta2 were selected based on [63], with Beta2 being modified to include a value of 0.0. The available values for Epsilon were selected from the Keras documentation. ...
Traditionally, condition monitoring of wind turbines has been performed manually by certified rope teams. This method of inspection can be dangerous for the personnel involved, and the resulting downtime can be expensive. Wind turbine inspection can be performed using autonomous drones to achieve lower downtime, cost, and health risks. To enable autonomy, the field of drone path planning can be assisted by this research, namely machine learning that detects wind turbines present in aerial RGB images taken by the drone before performing the maneuvering for turbine inspection. For this task, the effectiveness of two deep learning architectures is evaluated in this paper both without and with a proposed fuzzy contrast enhancement (FCE) image preprocessing algorithm. Efforts are focused on two convolutional neural network (CNN) variants: VGG19 and Xception. A more traditional approach involving support vector machines (SVM) is also included to contrast a machine learning approach with our deep learning networks. The authors created a novel dataset of 4500 RGB images of size 210×210 to train and evaluate the performance of these networks on wind turbine detection. The dataset is captured in an environment mimicking that of a wind turbine farm, and consists of two classes of images: with and without a small-scale wind turbine (12V Primus Air Max) assembled at Utah Valley University. The images were used to describe in detail the analysis and implementation of the VGG19, Xception, and SVM algorithms using different optimization, model training, and hyperparameter tuning technologies. The performances of these three algorithms are compared in depth alongside those augmented using the proposed FCE image preprocessing technique.
... The Adam approach involves employing squared gradients and exponential moving averages. The validation of hyperparameters is achieved based on the expressions provided below [56]: ...
Lung cancer is a prevalent malignancy that impacts individuals of all genders and is often diagnosed late due to delayed symptoms. To catch it early, researchers are developing algorithms to study lung cancer images. The primary objective of this work is to propose a novel approach for the detection of lung cancer using histopathological images. In this work, the histopathological images underwent preprocessing, followed by segmentation using a modified approach of KFCM-based segmentation and the segmented image intensity values were dimensionally reduced using Particle Swarm Optimization (PSO) and Grey Wolf Optimization (GWO). Algorithms such as KL Divergence and Invasive Weed Optimization (IWO) are used for feature selection. Seven different classifiers such as SVM, KNN, Random Forest, Decision Tree, Softmax Discriminant, Multilayer Perceptron, and BLDC were used to analyze and classify the images as benign or malignant. Results were compared using standard metrics, and kappa analysis assessed classifier agreement. The Decision Tree Classifier with GWO feature extraction achieved good accuracy of 85.01% without feature selection and hyperparameter tuning approaches. Furthermore, we present a methodology to enhance the accuracy of the classifiers by employing hyperparameter tuning algorithms based on Adam and RAdam. By combining features from GWO and IWO, and using the RAdam algorithm, the Decision Tree classifier achieves the commendable accuracy of 91.57%.
... then bring the previously obtained C ðtÞ and o ðtÞ into o ðtÞ Á tanhðC ðtÞ Þ to obtain h ðtÞ , filter the data based on the activation functions sigmoid and tanh, and output the hidden features of the area data (These are formulas and functions related to LSTM [27]). In this section, we apply the Adaptive Moment Estimation (Adam) [28] optimization algorithm to optimize the experimental network, improve the network training method, minimize the loss function, and adjust the weight and deviation value of the model. Adam is an optimization algorithm used to replace the random gradient descent in the deep learning model, combining AdaGrad and RMSProp algorithm to adjust the optimal performance parameters and relatively simple default parameters to deal with most problems. ...
With the rapid development of 5G mobile communication technology, the growing demand for media services in society is becoming a characteristic of smart cities. Technical study has exhaustively investigated the low-latency resource supply of edge computing systems for web hosting services. However, external variables (such as transmission and network delays) impact the transmission between edge servers and service requests, resulting in service request delays. In addition, resource service requests that are constantly updated and in dynamic distribution may overload some servers while others are idle, resulting in poor load balancing. Consequently, this research presents a Resource Pre-Allocation method (RPA) for cognitive analytics-based social media platforms, which aims to improve the load balancing of edge servers while serving requests with strict latency requirements, so as to obtain the optimal resource allocation strategy. First, the resource requirement prediction algorithm is developed based on temporal-spatial demand history. Then, we propose a multi-objective algorithm combined with the optimal solution selection techniques to obtain the ideal resource allocation decisions. Finally, the performance of RPA is tested and evaluated. The experimental results show that RPA can allocate resources better than other methods.
... Saleem et al. [44] compared the performances of six optimizers, including SGD and Adam, for CNN models applied in plant disease classification. Şen et al. [45] presented results of the grid search for the hyperparameters of the Adam optimizer in an electrocardiogram classification. With regard to fault diagnosis, Rezaeianjouybari and Shang [46] verified the effects of SGD and Momentum's cyclic learning rate and cyclic momentum on bearing fault detection. ...
Deep learning has recently resulted in remarkable performance improvements in machine fault diagnosis using only raw input vibration signals with no signal preprocessing. However, research on machine fault diagnosis using deep learning has primarily focused on model architectures, even though optimizers and their hyperparameters used for training can have a significant impact on model performance. This paper presents extensive benchmarking results on the tuning of optimizer hyperparameters using various combinations of datasets, convolutional neural network (CNN) models, and optimizers with varying batch sizes. First, we set the hyperparameter search space and then trained the models using hyperparameters sampled from a quasi-random distribution. Subsequently, we refined the search space based on the results of the first step and finally evaluated model performances using noise-free and noisy data. The results showed that the learning rate and momentum factor, which determine training speed, substantially affected the model’s accuracy. We also discovered that the impacts of batch size and model training speed on model performance were highly correlated; large batch sizes led to higher performances at higher learning rates or momentum factors. Conversely, model performances tended to be high for small batch sizes at lower learning rates or momentum factors. In addition, regarding the growing attention to on-device artificial intelligence (AI) solutions, we assessed the accuracy and computational efficiency of candidate models. A CNN with training interference (TICNN) was the most efficient model in terms of computational efficiency and robustness against noise among the benchmarked candidate models.
... While creating the CNN model, optimizers are implemented to minimize the error between the predicted output and the true output. Due to its capacity to manage the complexity of deep learning and its ability to adaptively adjust the learning rate for each parameter, the proposed network implements the Adaptive Moment Estimation (Adam) Optimizer with a learning rate of 0.001 30 . The selection of hyper-parameters such as number of convolution layers, activation functions, optimizer, dropout rate, learning rate and batch size used in this study were determined by grid search algorithm and the hyper-parameters that gave the best results were used in the study. ...
We intended to compare the doctors with a convolutional neural network (CNN) that we had trained using our own unique method for the Lateral Pillar Classification (LPC) of Legg-Calve-Perthes Disease (LCPD). Thousands of training data sets are frequently required for artificial intelligence (AI) applications in medicine. Since we did not have enough real patient radiographs to train a CNN, we devised a novel method to obtain them. We trained the CNN model with the data we created by modifying the normal hip radiographs. No real patient radiographs were ever used during the training phase. We tested the CNN model on 81 hips with LCPD. Firstly, we detected the interobserver reliability of the whole system and then the reliability of CNN alone. Second, the consensus list was used to compare the results of 11 doctors and the CNN model. Percentage agreement and interobserver analysis revealed that CNN had good reliability (ICC = 0.868). CNN has achieved a 76.54% classification performance and outperformed 9 out of 11 doctors. The CNN, which we trained with the aforementioned method, can now provide better results than doctors. In the future, as training data evolves and improves, we anticipate that AI will perform significantly better than physicians.
With the advancement of technology and the increasing use of wearable devices, information security have become a necessity. Although many biometrics authentication methods have been studied on these devices to ensure information security, an activity-aware deep learning (DL) model that is compatible with different device types and uses only electrocardiogram signals has not been studied. Our objective is to investigate DL models that exclusively use ECG signals during several physical activities, facilitating their implementation on various devices. Through this research, we aim to contribute to the advancement of wearable devices for the purpose of biometric verification. In this context, this study investigates the application of adaptive techniques that rely on prior activity classification to potentially improve biometric performance using DL models. In this study, we compare three time-frequency representations to generate images for activity classification using GoogleNet, ResNet50 and DenseNet201, and for biometric verification using ResNet50 and DenseNet201. Despite employing various convolutional neural network (CNN) models, we could not achieve high accuracy in activity classification. Consequently, manually classified samples were used for activity-aware biometric verification. We also provide a detailed comparison of various DL parameters. We use a public dataset simultaneously collected from both medical and wearable devices to offer a cross-device comparison. The results demonstrate that our method can be applied to both wearable and medical devices for activity classification and biometric verification. Besides, although it is known that DL requires a large amount of training data, our model, which was created using a small amount of training data and a real-life biometric verification scenario, achieved comparable results to studies using a large amount of data. The model was achieved 0.16% to 30.48% better results when classified according to their physical activities.
In recent years, Convolutional Neural Networks (CNN) have revolutionized the field of medical science by efficiently expediting the process of detection and classification of diseases such as cancer. Adam is one of the most popular techniques to fasten the optimization process of the gradient descent used by CNNs to converge toward the solution. However, manually tuning the hyperparameters of Adam and CNN is a tedious process as it requires continuous trial and error, and can lead to suboptimal performance. Such suboptimal training can lead to the wrong diagnosis of diseases, endangering the lives of patients. Hence, the presented work focuses on implementing the grey wolf optimizer, a nature-inspired metaheuristic optimization algorithm, for optimizing the hyperparameters of the Adam optimizer specific to each dataset. The classification methodology has been implemented using the MobileNet-V2model on three well-studied medical image datasets, viz. Warwick-QU, SIPaKMeD, and MHIST, where it has achieved a remarkable 2.5%, 6.83%, and 5.12% improvements in test accuracy over the base models without optimization, respectively. The proposed paradigm has been validated and its improved performance has been shown using the Mcnemar Test and SmoothGradCAM++ visualization. This clearly shows the efficacy of the proposed method in performing dataset-specific optimization to heighten the classification accuracy.
Alzheimer’s disease is a degenerative disorder of the brain that affects memory and cognitive function, and is becoming more prevalent as the population ages. Alzheimer causes brain cells of a person to die, and with time the brain works less. As a result of this there is a change in the behavior and personality of Alzheimer patients. It is observed that the patients very often suffer from fluctuating mood. Due to the change in mood of the patient, the caretaker or the attendant of the patient is unable to distinguish what triggered the changes, and provide proactive support or timely intervention. Hence the study is undertaken to detect the mood of the Alzheimer’s patient employing the voice analysis, computer vision and deep learning. This paper comprises of two phases. The first phase detects the emotions of the patients in real time with the help of computer vision (CV), voice analysis (VA) and Convolutional Neural Network (CNN). Two CNN models were trained, first with the attributes extracted from the image and second the features extracted from the voice dataset. Then the predictions are compared to get the result from the proposed model. The second phase of the paper examines the practicality of the proposed approach by applying it to detect the emotions in four Alzheimer’s patients. Finally, the results are compared and validated. Overall, the proposed model holds promise as a valuable tool in the real time detection of emotions in Alzheimer patient, enabling timely intervention and improved patient outcomes.
The extraction of roads from UAV images is challenged by lighting, noise, occlusions, and similar non-road objects, making high-quality road extraction difficult. To addressing these issues, this study proposes an enhanced U-Net network to automate the extraction and 3D modeling of real road scenes using UAV imagery. Initially, a cascaded atrous spatial pyramid module was integrated into the encoder to capitalize on global context information, thereby refining the fuzzy segmentation outcomes. Subsequently, a module for augmenting road feature extraction was added within the channel, and a spatial attention mechanism was introduced in the decoder to enhance edge clarity. Experimental results demonstrated that this model captures more road information compared to mainstream networks and effectively incorporates topological structure perception for road extraction in complex scenarios, thus improving road connectivity. The model achieved an F1 score and mean Intersection over Union (mIoU) of 85.6% and 81.2%, respectively, on UAV images of road scenes – marking improvements of 3.9% and 3.4% over the traditional U-Net model, thereby exhibiting superior automatic road extraction capabilities. Ultimately, the model facilitated refined modeling and visual analysis of road scenes, achieving high overall accuracy and detailed local restoration of the actual scene.
Spatial-exploitation convolutional neural networks (CNNs) have
a simplified architecture compared to other CNN models. However, devices with limited computational resources could struggle with processing spatial-exploitation CNNs. To address this, we investigate two methods to optimise spatial-exploitation CNN models for time efficiency and classification accuracy: hyperparameter-tuning, and human-in-the-loop (HITL). We apply grid-search to optimise the hyperparameter space, whilst HITL is used to identify whether the time-to-accuracy relationship of the
optimised model can be improved. To show the versatility of combining the two methods, CIFAR-10, MNIST, and Imagenette are used as model input. This paper contributes to spatial-exploitation CNN optimisation by combining hyperparameter-tuning and HITL. Results show that this combination improves classification accuracy by 1.47–2.34% and reduces the time taken to conduct this task by 27–28%, depending on dataset. We conclude that combining hyperparameter-tuning and HITL are a viable approach to optimise spatial-exploitation CNNs for devices with limited computational resources.
Acute lymphoblastic leukemia (ALL), one of the prevalent types of carcinogenic disease, has been seen a deadly illness exposing numerous patients across the world to potential threats of lives. It impacts both adults and children providing a narrow range of chances of being cured if diagnosed at a later stage. A definitive diagnosis often demands highly invasive diagnostic procedures thereby proving time-consuming and expensive. Peripheral Blood Smear (PBS) images have been playing a crucial role in the initial screening of ALL in suspected individuals. However, the nonspecific nature of ALL poses a substantial challenge in the analysis of these images thus leaving space for misdiagnosis. Aiming at contribute to the early diagnoses of this life-threatening disease, we put forward automated platform for screening the presence of ALL concerning its specific subtypes (benign, Early Pro-B, Pro-B and Pre-B) using PBS images. The proposed web based platform follows weighted ensemble learning technique using a Residual Convolutional Neural Network (ResNet-152) as the base learner to identify ALL from hematogone cases and then determine ALL subtypes. This is likely to save both diagnosis time and the efforts of clinicians and patients. Experimental results are obtained and comparative analysis among 7 well-known CNN Network architectures (AlexNet, VGGNet, Inception, ResNet-50, ResNet-18, Inception and DenseNet-121) is also performed that demonstrated that the proposed platform achieved comparatively high accuracy (99.95%), precision (99.92%), recall (99.92%), F1-Score (99.90%), sensitivity (99.92%) and specificity (99.97%). The promising results demonstrate that the proposed platform has the potential to be used as a reliable tool for early diagnosis of ALL and its sub-types. Furthermore, this provides references for pathologists and healthcare providers, aiding them in producing specific guidelines and more informed choices about patient and disease management.
BACKGROUND
Methylcellulose has been applied as a primary binding agent to control the quality attributes of plant‐based meat analogues. H owever, a great deal of effort has been made to search for hydrocolloids to replace methylcellulose because of increasing awareness of clean labels. In this study, a machine learning framework was proposed in order to describe and predict the flow behavior of six hydrocolloid solutions, and the predicted viscosities were correlated with the textural features of their corresponding plant‐based meat analogues.
RESULTS
Different shear‐thinning and Newtonian behaviors were observed depending on the type of hydrocolloid and the shear rate. Methylcellulose exhibited an increasing viscosity pattern with increasing temperature, compared to the other hydrocolloids. The machine learning algorithms (random forest and multilayer perceptron models) showed a better viscosity fitting performance than the constitutive equations (power law and Cross models). In addition, three hyperparameters of the multilayer perceptron model (optimizer, learning rate, and the number of hidden layers) were tuned using the Bayesian optimization algorithm.
CONCLUSION
The optimized multilayer perceptron model exhibited superior performance in viscosity prediction (R² = 0.9944–0.9961/RMSE = 0.0545–0.0708). Furthermore, the machine learning‐predicted viscosities overall showed similar patterns to the textural parameters of the meat analogues. © 2024 Society of Chemical Industry.
In coastal waters, phytoplankton, suspended particulate matter, and dissolved organic matter intricately and nonlinearly alter the reflectivity of seawater. Neural network technology, which has been rapidly advancing recently, offers the advantage of effectively representing complex nonlinear relationships. In previous studies, a three-stage neural network was constructed to extract the inherent optical properties of each component. However, this study proposes an algorithm that directly employs a deep neural network. The dataset used in this study consists of synthetic data provided by the International Ocean Color Coordination Group, with the input data comprising above-surface remote-sensing reflectance at nine different wavelengths. We derived inherent optical properties using this dataset based on a deep neural network. To evaluate performance, we compared it with a quasi-analytical algorithm and analyzed the impact of log transformation on the performance of the deep neural network algorithm in relation to data distribution. As a result, we found that the deep neural network algorithm accurately estimated the inherent optical properties except for the absorption coefficient of suspended particulate matter (R^2 greater than or equal to 0.9) and successfully separated the sum of the absorption coefficient of suspended particulate matter and dissolved organic matter into the absorption coefficient of suspended particulate matter and dissolved organic matter, respectively. We also observed that the algorithm, when directly applied without log transformation of the data, showed little difference in performance. To effectively apply the findings of this study to ocean color data processing, further research is needed to perform learning using field data and additional datasets from various marine regions, compare and analyze empirical and semi-analytical methods, and appropriately assess the strengths and weaknesses of each algorithm.
The primary source of water for irrigation and other agricultural activities is rainfall. It has an immediate effect on crop growth and productivity. Forecasting this rainfall in advance allows farmers to effectively plan their cropping pattern. In recent years, forecasting rainfall has become very popular due to the availability of the latest computation techniques. Artificial neural networks (ANNs) are one such technique widely used for rainfall prediction by a number of researchers. These models are more reliable as they make better predictions because of their nonlinear data learning method. In the present study, an ANN model was developed to predict the annual, monsoon and postmonsoon season rainfall. The model was developed using 34 years of data from 1985 to 2018 in the command area of the Loktak Lift Irrigation Project in Manipur, India. The ANN model was trained using the rectified linear unit (ReLU) activation function. The 3‐year input model excelled in all seasons, with the best model achieving a 0.36 coefficient of determination ( R ² ), 75.7 root mean square error, 0.60 correlation coefficient and 62.5 mean absolute error. These performance indicators were comparable with studies performed by other researchers. Thus, the model can be adopted for the study area.
Lung cancer is a prevalent malignancy that impacts individuals of all genders and is often diagnosed late due to delayed symptoms. To catch it early, researchers are developing algorithms to study lung cancer images. One such method uses computer aided detection to classify as benign or malignant from the histopathological images. Standard histopathological images were used from a Lung and Colon Cancer Histopathological Image Dataset (LC25000) which contains two classes of benign and malignant of 5000 each. Images were preprocessed and features extracted using Particle Swarm Optimization (PSO) and Grey Wolf Optimization (GWO). Feature selection methods used are KL Divergence and Invasive Weed Optimization (IWO). Seven different classifiers like SVM, KNN, Random Forest, Decision Tree, Softmax Discriminant, Multilayer Perceptron, and BLDC were used to analyze and classify the images as benign or malignant. Results were compared using standard metrics, and kappa analysis assessed classifier agreement. The Decision Tree Classifier with GWO feature extraction achieved good accuracy of 85.01% without Feature selection and Hyperparameter Tuning approaches. Furthermore, we present a methodology to enhance the accuracy of the classifiers by employing hyperparameter tuning algorithms based on Adam and RAdam. By combining features from GWO and IWO, and using the RAdam algorithm, the Decision Tree classifier achieves the commendable accuracy of 91.57%
In this paper, we study the multi-task allocation design for vehicular fog computing (VFC) systems, where multiple vehicles offload multiple tasks simultaneously to enhance performance. The data, often originating from multiple sources, must be transmitted at a high data rate in parallel. For catering the thriving demands for transmitting thriving data generated by vehicles, multiple input multiple output (MIMO) antennas are mounted on all the vehicles. However, with an ordinary communication strategy, the data rate in MIMO-empowered VFC systems may decrease due to the interference generated from MIMO antennas. On the other hand, more antennas may lead to higher energy consumption, which is antagonistic to vehicles powered by limited electricity. To address all these challenges, we aim to offload multiple tasks simultaneously with higher data rates and lower power consumption in MIMO-empowered VFC, while taking into account the mobility of vehicles and communication interference. This design is a mixed integer non-convex optimization problem that is challenging to solve. Therefore, a
Fountain
system is proposed, which is a novel multi-task allocation strategy adopting deep learning and convex optimization, that transforms the multi-task allocation optimization problem into a convex solvable one for achieving an effective sub-optimal solution. For the purpose to utilize
Fountain
in the real world, the effectiveness of
Fountain
is evaluated based on real-world vehicle trajectories. Compared with other schemes, simulation results illustrate that our proposed
Fountain
improves the average transmission data rate by up to 163% and reduces the average power consumption by up to 40%.
Parameter-specific adaptive learning rate methods are computationally
efficient ways to reduce the ill-conditioning problems encountered when
training large deep networks. Following recent work that strongly suggests that
most of the critical points encountered when training such networks are saddle
points, we find how considering the presence of negative eigenvalues of the
Hessian could help us design better suited adaptive learning rate schemes,
i.e., diagonal preconditioners. We show that the optimal preconditioner is
based on taking the absolute value of the Hessian's eigenvalues, which is not
what Newton and classical preconditioners like Jacobi's do. In this paper, we
propose a novel adaptive learning rate scheme based on the equilibration
preconditioner and show that RMSProp approximates it, which may explain some of
its success in the presence of saddle points. Whereas RMSProp is a biased
estimator of the equilibration preconditioner, the proposed stochastic
estimator, ESGD, is unbiased and only adds a small percentage to computing
time. We find that both schemes yield very similar step directions but that
ESGD sometimes surpasses RMSProp in terms of convergence speed, always clearly
improving over plain stochastic gradient descent.
In this paper, a multi-stage network including two multilayer perceptron (MLP) and one self organizing map (SOM) networks is presented. The input of the network is a combination of independent features and the compressed ECG data. The proposed network as a form of data fusion, performs better than using the raw data or individual features. We classified six common ECG waveforms using ten ECG records of the MIT/BIH arrhythmia database. An average recognition rate of 0.883 was achieved within a short training and testing time.
A simple algorithm using topological mapping has been developed
for a real-time detection of the QRS complexes of ECG signals. As a
measure of QRS complex energy, the authors used topological mapping from
one dimensional sampled ECG signals to two dimensional vectors. To
describe a change of curvature, the authors derive modified spatial
velocity (MSV), from MSV the authors can locate QRS complexes more
easily. The proposed algorithm consists of very small C-language
procedures which reliably recognize the QRS complexes. For evaluation
the authors used the MIT/BIH arrhythmia database. The proposed algorithm
provides a good performance, a 99.58% detection rate of QRS complexes, a
99.57% sensitivity and 99.87% positive predictivity, respectively
We introduce Adam, an algorithm for first-order gradient-based optimization
of stochastic objective functions. The method is straightforward to implement
and is based an adaptive estimates of lower-order moments of the gradients. The
method is computationally efficient, has little memory requirements and is well
suited for problems that are large in terms of data and/or parameters. The
method is also ap- propriate for non-stationary objectives and problems with
very noisy and/or sparse gradients. The method exhibits invariance to diagonal
rescaling of the gradients by adapting to the geometry of the objective
function. The hyper-parameters have intuitive interpretations and typically
require little tuning. Some connections to related algorithms, on which Adam
was inspired, are discussed. We also analyze the theoretical convergence
properties of the algorithm and provide a regret bound on the convergence rate
that is comparable to the best known results under the online convex
optimization framework. We demonstrate that Adam works well in practice when
experimentally compared to other stochastic optimization methods.
In this paper, we present a new system for the classification of electrocardiogram (ECG) beats by using a fast least square support vector machine (LSSVM). Five feature extraction methods are comparatively examined in the 15-dimensional feature space. The dimension of the each feature set is reduced by using dynamic programming based on divergence analysis. After the preprocessing of ECG data, six types of ECG beats obtained from the MIT-BIH database are classified with an accuracy of 95.2% by the proposed fast LSSVM algorithm together with discrete cosine transform. Experimental results show that not only the fast LSSVM is faster than the standard LSSVM algorithm, but also it gives better classification performance than the standard backpropagation multilayer perceptron network.
Adam- latest trends in deep learning optimization
- bushaev
AdaGrad stepsizes: sharp convergence over nonconvex landscapes, from any initialization
- R Ward
ADADELTA: An Adaptive Learning Rate Method
- R W There
- L Rate
- A There