ArticlePublisher preview available

Machine learning for KPIs prediction: a case study of the overall equipment effectiveness within the automotive industry

Authors:
Choumicha EL Mazgualdi
Choumicha EL Mazgualdi
Choumicha EL Mazgualdi
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Tawfik Masrour at Université Moulay Ismail de Meknes
Tawfik Masrour
Ibtissam Hassani at Université Moulay Ismail de Meknes
Ibtissam Hassani
Abdelmoula Khdoudi at Université Moulay Ismail de Meknes
Abdelmoula Khdoudi
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Key performance indicators are tools for management, decision support and forecasting; they reflect the strategy and vision of the company in terms of objectives and allow to always staying in step with the expectations of the stakeholders. Accurate forecasting of the indicators allows decisions to be reoriented to ensure performance optimization while reducing both cost and effort. This paper aims to apply different machine learning methods, namely support vector regression, optimized support vector regression (using genetic algorithm), random forest, extreme gradient boosting and deep learning to predict the overall equipment effectiveness as a case study. We will make use of several configurations of the listed models in order to provide a wide field of comparison. The data used to train our models were provided by an automotive cable production industry. The result shows that the configuration in which we used cross-validation technique, and we performed a duly splitting of data, provides predictor models with the better performances.
Figures - available from: Soft Computing
This content is subject to copyright. Terms and conditions apply.
A preview of this full-text is provided by Springer Nature.
Learn more
Content available from Soft Computing
This content is subject to copyright. Terms and conditions apply.
METHODOLOGIES AND APPLICATION
Machine learning for KPIs prediction: a case study of the overall
equipment effectiveness within the automotive industry
Choumicha EL Mazgualdi
1
•Tawfik Masrour
1
•Ibtissam El Hassani
1
•Abdelmoula Khdoudi
1
Published online: 5 October 2020
Springer-Verlag GmbH Germany, part of Springer Nature 2020
Abstract
Key performance indicators are tools for management, decision support and forecasting; they reflect the strategy and vision
of the company in terms of objectives and allow to always staying in step with the expectations of the stakeholders.
Accurate forecasting of the indicators allows decisions to be reoriented to ensure performance optimization while reducing
both cost and effort. This paper aims to apply different machine learning methods, namely support vector regression,
optimized support vector regression (using genetic algorithm), random forest, extreme gradient boosting and deep learning
to predict the overall equipment effectiveness as a case study. We will make use of several configurations of the listed
models in order to provide a wide field of comparison. The data used to train our models were provided by an automotive
cable production industry. The result shows that the configuration in which we used cross-validation technique, and we
performed a duly splitting of data, provides predictor models with the better performances.
Keywords Machine learning Key performance indicators Overall equipment effectiveness Prediction
Improvement
1 Introduction
1.1 Key performance indicators (KPI)
Today, dashboards have become an indispensable tool. It
must provide managers, from operational management to top
management, with the information they need in order to
make decisions. It consists of a set of indicators designed to
allow managers to see the progress of their systems, and it
reflects the company’s strategy and vision in terms of
objectives. So, it allows to follow both the targeted results
and the actions, both corrective and preventive ones that
achieve the objectives set. Key performance indicators are
most often the result of a long chain of information gathering
and aggregation, and they generally allow responsiveness
and decisions are made more and more quickly.
But that is not enough in the current context. Today, in order
to maintain or gain competitive advantage, organizations place
the search for ‘‘proactivity’’ at the forefront of their concerns:
the precise forecasts of the indicators make possible to redirect
decisions in order to guarantee an optimization of the perfor-
mances, and reducing costs and efforts. Nowadays, in the
digitalization age, information systems are managing more and
more data, making them useful for predicting KPI’s thanks to
advances in artificial intelligence algorithms, notably machine
learning. This new generation of intelligent systems of pre-
diction is precisely the engines of the proactivity, as computing
power has become cheaper and more available.
1.2 Overall equipment effectiveness (OEE)
The overall equipment effectiveness (OEE) is a perfor-
mance measurement metric widely used. Its calculation
Communicated by V. Loia.
&Tawfik Masrour
t.masrour@ensam.umi.ac.ma
Choumicha EL Mazgualdi
c.elmazgualdi@edu.umi.ac.ma
Ibtissam El Hassani
i.elhassani@ensam.umi.ac.ma
Abdelmoula Khdoudi
a.khdoudi@edu.umi.ac.ma
1
Laboratory of Mathematical Modeling, Simulation and Smart
Systems (L2M3S), Artificial Intelligence for Engineering
Sciences Team, ENSAM-Meknes, Moulay ISMAIL
University, B.P. 15290, Marjane 2, Al-Mansor,
50500 Meknes, Morocco
123
Soft Computing (2021) 25:2891–2909
https://doi.org/10.1007/s00500-020-05348-y(0123456789().,-volV)(0123456789().,-volV)
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... OEE is proposed as an operational improvement measure [30]. OEE is a vital tool for eliminating and identifying production losses which help in improving the efficiency of production systems [31,32]. The categories of these losses can be classified into six categories: minor stoppage loss, adjustment loss, breakdown loss, speed loss, and quality defects, start-up loss, and rework loss [33,34]. ...
... Different machine learning methods are applied to predict the OEE of an automotive cable production industry. These methods provide predictor models with improved and better performances [31]. An improved OEE method is developed for a full process cycle in the steel market. ...
Evaluation and Improvement of a Plastic Production System Using Integrated OEE Methodology: A Case Study
Article
Full-text available
  • Sep 2024
Overall equipment effectiveness (OEE) is a key indicator to measure the effectiveness of production systems. This paper aims to evaluate and improve a plastic production line based on OEE evaluation. An integrated framework is proposed to enhance the production system efficiency. This paper presents the data for a Plastic production line in Jordan under real working conditions. The data covers three months. A framework process to improve the OEE of the Plastic production system was proposed. Six major stoppage losses were inspected with the help of Pareto analysis. Furthermore, the actual availability, efficiency, and quality rate measures, together with the whole OEE for each working day, week, and month of the production line were shown. The methodology is based on determining the OEE of a Plastic production line after determining the causes of failures. The fishbone diagram tool is used to determine the root causes of failures. To improve the OEE measure, several losses are identified. The results reveal that the company should improve its policy to improve the production line’s performance and reduce losses. Top management should also pay attention to reducing the speed losses, which consist of 58.1%, and eliminate the planned and unscheduled disruptions covering 12.73% of all losses. This can be achieved by establishing a proper operation management procedure and strategy. This, in turn, optimized the equipment’s effectiveness. The quality procedure should include the changeover program that may be executed every day. Similarly, all preventive maintenance procedures for the six machines should be properly executed in predetermined intervals. There are several limitations in the research. Firstly, the research case study is only the plastic production system. Secondly, the research is related to the downtime or stoppage by analyzing it using fishbone diagram. Further, supported by other techniques such as the Pareto chart, six big losses analyses and CED. This research conducted on a Plastic industry. However, similar studies can be carried out in future in other manufacturing industries like electronic, pharmaceutical, textile industries, etc., and service industry. However, as future research work the contributions of this paper with other lean manufacturing concept like six sigma, quality function deployment, TQM, and just-in-time manu-facturing, can also be conducting to assess the overall production line efficiency. On the other hand, several statistical tests can be implemented based on data collected of TPM performance indicators. The proposed method supports policymakers in their decision-making process on the operations management line. Further-more, it improves the production systems’ productivity quality, and performance, reducing unplanned stop-pages and breakdowns, and reducing maintenance costs.
View
... A deeper understanding of the sensitivity and robustness of the forecasts may be obtained as well through the use of simulation approaches. System dynamics designing, discrete event simulation, agent-based modeling, and other similar approaches are just some of the common simulation methods that can be used for KPI forecasting [12]. ...
... Methods of expert judgment have the potential to also take into account the numerous viewpoints and preferences held by forecasters and other stakeholders. Methods such as brainstorming, the nominal group technique, and the analytic hierarchy process are examples of popular expert judgment methods that can be used for KPI forecasting [12]. Statistical approaches are based on the use of mathematical models and formulae to analyze historical data and identify patterns, trends, cycles, correlations, and effects among the KPIs and their related factors. ...
Enhancing KPI Forecasting Through Regression Algorithms Using Historical Data
Chapter
Full-text available
  • Jul 2024
Key performance indicators (KPIs) are essential tools for organizations across industries, providing a means to assess and enhance performance, efficiency, and quality. However, predicting KPIs presents challenges due to data nonlinearity, uncertainty, and variability. Traditional approaches, like trend analysis, may struggle to cope with these issues, leading to imperfect and unreliable predictions. In response, this research aims to enhance KPI prediction accuracy using machine learning algorithms. The study compares seven techniques—random forest (RF), XGBoost (XGB), decision tree (DT), linear regression (LR), support vector regression (SVR), neural network (NN), and multi-horizon quantile recurrent forecaster (MHQRF)—using three sales datasets. Evaluation includes four error measures: mean absolute error (MAE), mean squared error (MSE), root mean squared error (RMSE), and R-squared. Results show MHQRF’s superior performance across all datasets, with the lowest MAE (618.6), MSE (1,505,991.394), and RMSE (683.98). However, XGB achieves a higher average R-squared value (0.886) compared to MHQRF, introducing more nuanced considerations for model selection in diverse KPI forecasting scenarios.
View
... El Haddad et al. [25] investigated the application of ML algorithms for predicting Overall Equipment Effectiveness (OEE) in manufacturing environments. Their findings emphasize the adaptability of ML in improving industrial efficiency through accurate KPI measurement. ...
A Brief Discussion on KPI Development in Public Administration
Preprint
Full-text available
  • Dec 2024
Efficient and effective service delivery in Public Administration (PA) relies on the development and utilization of key performance indicators (KPIs) for evaluating and measuring performance. This paper presents an innovative framework for KPI construction within performance evaluation systems, leveraging Random Forest algorithms and variable importance analysis. The proposed approach identifies key variables that significantly influence PA performance, offering valuable insights into the critical factors driving organizational success. By integrating variable importance analysis with expert consultation, relevant KPIs can be systematically developed, ensuring that improvement strategies address performance-critical areas. The framework incorporates continuous monitoring mechanisms and adaptive phases to refine KPIs in response to evolving administrative needs. This study aims to enhance PA performance through the application of machine learning techniques, fostering a more agile and results-driven approach to public administration.
View
... Another research stream, present in the literature, includes the application of various machine learning methods for predicting the most used efficiency measures in industrial settings. An industrial case study from the automotive industry documents the application and comparison of different machine learning algorithms to determine OEE [15]. The application of deep learning algorithms has been documented to predict performance indicators of packaging equipment in [16]. ...
Exploring statistical and machine learning methods for modeling probability distribution parameters in downtime length analysis: a paper manufacturing machine case study
Article
Full-text available
  • Nov 2024
Manufacturing companies focus on improving productivity, reducing costs, and aligning performance metrics with strategic objectives. In industries like paper manufacturing, minimizing equipment downtime is essential for maintaining high throughput. Leveraging the extensive data generated by these facilities offers opportunities for gaining competitive advantages through data-driven insights, revealing trends, patterns, and predicting future performance indicators like unplanned downtime length, which is essential in optimizing maintenance and minimizing potential losses. This paper explores statistical and machine learning techniques for modeling downtime length probability distributions and correlation with machine vibration measurements. We proposed a novel framework, employing advanced data-driven techniques like artificial neural networks (ANNs) to estimate parameters of probability distributions governing downtime lengths. Our approach specifically focuses on modeling parameters of these distribution, rather than directly modeling probability density function (PDF) values, as is common in other approaches. Experimental results indicate a significant performance boost, with the proposed method achieving up to 30% superior performance in modeling the distribution of downtime lengths compared to alternative methods. Moreover, this method facilitates unsupervised training, making it suitable for big data repositories of unlabelled data. The framework allows for potential expansion by incorporating additional input variables. In this study, machine vibration velocity measurements are selected for further investigation. The study underscores the potential of advanced data-driven techniques to enables companies to make better-informed decisions regarding their current maintenance practices and to direct improvement programs in industrial settings.
View
... • using decision tree, neural networks, support vector machine at intraday forecasting of OEE [13] • simple moving average with Holt's double exponential smoothing method using Python [14] • recurrent neural networks to reveal hidden correlation that affecting to OEE [15] • Person correlation and one-sample T-test with Design Expert software [16] • using grid search algorithm by decision tree, K-nearest neighbor, neural networks and support vector machine [17] • support vector machine, random forest, gradient boost, deep learning [18,19] • OEE prediction by Bayes-based machine learning, decision tree, support vector machine and logistic regression [20,21]. ...
Prediction of Overall Equipment Effectiveness in Assembly Processes Using Machine Learning
Article
Full-text available
  • Oct 2024
Nowadays, a lot of data is generated in production and also in the domain of assembly, from which different patterns can be extracted using machine learning methods with the support of data mining. With the support of various modern technical and Information Technology (IT) tools, the recording, storage and processing of large amounts of data is now a routine activity. Based on machine learning, efficiency metrics including Overall Equipment Effectiveness (OEE), can be partially predicted, but industrial companies need more accurate and reliable methods. The analyzed algorithms can be used in general for all production units or machines where production data is recorded by Manufacturing Execution System (MES) or other Enterprise Resource Planning (ERP) systems are available. This paper presents and determinates which most used machine learning methods should be combined with each other in order to achieve a better prediction result.
View
... For example, Kuo & Lin [7] developed an integration of neural networks and decision trees to predict availability efficiency, a metric strongly correlated with OEE, specifically for a set of washing machines. Meanwhile, Mazgualdi et al. [8] [9] employed SVM, SVM with optimization, RF, XGBoost and ANNs to predict OEE in an automotive cable production industry. Among these models, ANNs and RF demonstrated superior accuracy and performance. ...
Artificial Intelligence Models for Overall Equipment Effectiveness Prediction: A Case Study in an Assembly Manufacturing Company
Conference Paper
  • Jul 2024
Overall Equipment Effectiveness (OEE) stands as a key performance metric widely adopted in the manufacturing industry, aiding in enhancing productivity. This metric offers a comprehensive overview to higher management, enabling them to identify equipment-related losses. With the advancements in Industry 4.0 technologies, the Manufacturing Execution System facilitates real-time data collection, enhancing production efficiency and reducing manufacturing costs. However, the real-time depiction of OEE often fails to provide decision-makers with timely insights to conduct their tasks effectively. This study formulated machine learning models to tackle this challenge by forecasting the OEE for the upcoming working shift. Firstly, the historical dataset encompasses 31 features collected and processed to estimate the OEE value. Then, prominent machine learning models were utilized as prediction models: Linear Regression, Support Vector Regression, Random Forest, Extreme Gradient Boosting, and Artificial Neural Networks. The results show that the Extreme Gradient Boosting performs well for the OEE prediction with accuracy in training higher than 99% and testing nearly 90%. Our study illustrates an actionable knowledge-discovery process using a real-world data mining approach for the manufacturing industry, potentially applicable to other sectors.
View
... For example, Kuo & Lin [7] developed an integration of neural networks and decision trees to predict availability efficiency, a metric strongly correlated with OEE, specifically for a set of washing machines. Meanwhile, Mazgualdi et al. [8] [9] employed SVM, SVM with optimization, RF, XGBoost and ANNs to predict OEE in an automotive cable production industry. Among these models, ANNs and RF demonstrated superior accuracy and performance. ...
Sustainable supplier selection in the apparel industry: an integrated AHP-TOPSIS model for multi-criteria decision analysis
Article
  • Aug 2024
Purpose In the apparel industry, suppliers play a significant role, directly affecting customer service levels and business profits. Integrating sustainable requirements into supplier selection not only aligns with global environmental goals but also enhances business performance, social responsibility and overall industry well-being. This study aims to design a multi-criteria model to evaluate and select the most sustainable suppliers in the fashion industry, trying to balance the conflicts in the set of sustainable development criteria. Design/methodology/approach The integration of analytic hierarchy process (AHP) and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) offers advantages in sustainable supplier selection within the apparel industry. The AHP plays a crucial role in engaging multiple decision-makers with conflicting criteria to reach a consensus during the decision-making process. Conversely, the TOPSIS is used to compute alternative ratings. By simultaneously determining criteria weights and incorporating stakeholder preferences, hybrid models enhance decision-making strength and overcome limitations observed in classical multi-criteria decision-making techniques. Findings This research identified and classified 16 critical criteria impacting the selection of apparel industry suppliers, focusing on sustainable development. The criteria were weighted, providing a robust statistical foundation for the selection model. The results indicated that the most influential criteria were staff training, production capability, flexibility and practice of recycling. The proposed sustainable supplier selection model explains to decision-makers how criteria influence supplier ranking results compared to traditional models, supporting managers in making informed and sustainable supply chain decisions through continuous updates and enhancements. Originality/value This research provides new insight into the weighted impact of factors related to sustainable supplier selection in the apparel industry. The combination of a precise process and scientific knowledge will improve the quality of sustainable supplier selection.
View
... In image data classification, the XGB algorithm can be employed to predict the class or label of each pixel or region within an image based on the features extracted from the image. An illustration of the XGB algorithm can be seen in Figure 4. [24] First, XGB initializes the model with a single decision tree that partitions the image data into several groups based on image features such as color, texture, shape, and more. This decision tree is referred to as the base tree or base learner. ...
View
View
View
Support Vector Regression
Chapter
Full-text available
  • Jan 2015
Rooted in statistical learning or Vapnik-Chervonenkis (VC) theory, support vector machines (SVMs) are well positioned to generalize on yet-to-be-seen data. The SVM concepts presented in Chapter 3 can be generalized to become applicable to regression problems. As in classification, support vector regression (SVR) is characterized by the use of kernels, sparse solution, and VC control of the margin and the number of support vectors. Although less popular than SVM, SVR has been proven to be an effective tool in real-value function estimation. As a supervised-learning approach, SVR trains using a symmetrical loss function, which equally penalizes high and low misestimates. Using Vapnik’s -insensitive approach, a flexible tube of minimal radius is formed symmetrically around the estimated function, such that the absolute values of errors less than a certain threshold are ignored both above and below the estimate. In this manner, points outside the tube are penalized, but those within the tube, either above or below the function, receive no penalty. One of the main advantages of SVR is that its computational complexity does not depend on the dimensionality of the input space. Additionally, it has excellent generalization capability, with high prediction accuracy.
View
Deep Learning
Article
Full-text available
  • May 2015
  • NATURE
Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.
View
Introduction to Machine Learning
Chapter
  • Aug 2019
This chapter talks about what machine learning is, how machine learning systems are classified, and examples of real‐world applications of machine learning. It discusses some of the tools commonly used by data scientists to build machine learning solutions. Most machine learning scientists use one of two programming languages: Python or R. There are several different types of machine learning systems today. The classification of a machine learning system is usually based on the manner in which the system is trained and the manner in which the system can make predictions. Machine learning systems are classified as follows: supervised learning; unsupervised learning; semi‐supervised learning; reinforcement learning; batch learning; incremental learning; instance‐based learning; and model‐based learning. These labels are not mutually exclusive; it is quite common to come across a machine learning system that falls into multiple categories. The chapter examines these classification labels in more detail.
View
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Book
  • Jan 2018
A comprehensive introduction to Support Vector Machines and related kernel methods. In the 1990s, a new type of learning algorithm was developed, based on results from statistical learning theory: the Support Vector Machine (SVM). This gave rise to a new class of theoretically elegant learning machines that use a central concept of SVMs—-kernels—for a number of learning tasks. Kernel machines provide a modular framework that can be adapted to different tasks and domains by the choice of the kernel function and the base algorithm. They are replacing neural networks in a variety of fields, including engineering, information retrieval, and bioinformatics. Learning with Kernels provides an introduction to SVMs and related kernel methods. Although the book begins with the basics, it also includes the latest research. It provides all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms and to understand and apply the powerful algorithms that have been developed over the last few years.
View
A KPI-based process monitoring and fault detection framework for large-scale processes
Article
  • Feb 2017
  • ISA T
Large-scale processes, consisting of multiple interconnected subprocesses, are commonly encountered in industrial systems, whose performance needs to be determined. A common approach to this problem is to use a key performance indicator (KPI)-based approach. However, the different KPI-based approaches are not developed with a coherent and consistent framework. Thus, this paper proposes a framework for KPI-based process monitoring and fault detection (PM-FD) for large-scale industrial processes, which considers the static and dynamic relationships between process and KPI variables. For the static case, a least squares-based approach is developed that provides an explicit link with least-squares regression, which gives better performance than partial least squares. For the dynamic case, using the kernel representation of each subprocess, an instrument variable is used to reduce the dynamic case to the static case. This framework is applied to the TE benchmark process and the hot strip mill rolling process. The results show that the proposed method can detect faults better than previous methods.
View
XGBoost: A Scalable Tree Boosting System
Conference Paper
  • Aug 2016
Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.
View
XGBoost: A Scalable Tree Boosting System
Article
  • Mar 2016
Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.
View
An Advanced PLS Approach for Key Performance Indicator Related Prediction and Diagnosis in Case of Outliers
Article
  • Jan 2015
In the process industry, the key performance indicator (KPI)-related prediction and fault diagnosis are important steps to guarantee the product quality and improve economic benefits. A popular monitoring method as it has been, the partial least squares (PLS) algorithm is sensitive to outliers in training datasets, and cannot efficiently distinguish faults related to KPI from those unrelated to KPI due to its oblique projection to the input space. In this paper, a novel robust data-driven approach, named advanced partial least squares (APLS), is presented to handle process outliers under an improved framework of PLS. By means of a weighting strategy, APLS can remove the impact of outliers on process measurements and establish a more accurate model than PLS for fault diagnosis in the monitoring scheme, whose effectiveness has been verified through the Tennessee Eastman (TE) benchmark process. Simulation results demonstrate that the proposed approach is suitable not only for the KPI-related process prediction but also for the diagnosis of KPI-related faults.
View
The Lower Bound of the Capacity for a Neural Network with Multiple Hidden Layers
Chapter
  • Jan 1993
We show the lower bound of the capacity of a hierarchical neural network, having multiple hidden layers whose node unit takes the value of a real number between zero and one as the output of a sigmoid function. It is shown that n.⌈h12⌉+⌊h12⌋⋅⌈h22−1⌉+⋯+⌊hN−12⌋⋅⌈hN2−1⌉n.\left \lceil \frac{h_1}{2} \right \rceil+\left \lfloor \frac{h_1}{2} \right \rfloor\cdot \left \lceil \frac{h_2}{2}-1 \right \rceil+\cdots +\left \lfloor \frac{h_{N-1}}{2} \right \rfloor\cdot\left \lceil \frac{h_N}{2}-1 \right \rceil examples in the general position (i.e. no subset of n or less input vectors degenerate) can be memorized by the network which has n input units in the input layer, h ℓ hidden units in the ℓ-th layer of N hidden layers, and a single output unit in the output layer.
View
Adam: A Method for Stochastic Optimization
Article
  • Dec 2014
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions. The method is straightforward to implement and is based an adaptive estimates of lower-order moments of the gradients. The method is computationally efficient, has little memory requirements and is well suited for problems that are large in terms of data and/or parameters. The method is also ap- propriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The method exhibits invariance to diagonal rescaling of the gradients by adapting to the geometry of the objective function. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. We demonstrate that Adam works well in practice when experimentally compared to other stochastic optimization methods.
View