Figure 6 - uploaded by Ana Maria Madureira
Content may be subject to copyright.
Source publication
In this paper, we present and discuss an innovative approach to solve Job Shop scheduling problems based on machine learning techniques. Traditionally, when choosing how to solve Job Shop scheduling problems, there are two main options: either use an efficient heuristic that provides a solution quickly, or use classic optimization approaches (e.g.,...
Contexts in source publication
Context 1
... example, Vilím et al. [65] present the optimal solution value for the instances TAI27 and TAI28 (among others), but the optimal solution value for the TAI29 problem was proposed by Siala et al. [66]. Figure 6 shows a graph containing the optimal (or best known upper bound) makespan (OPT) and the makespan obtained by each method. The OPT line (in blue) represents the known optimal value [49] (or best known upper bound) and, as such, no other method can go below it. ...
Context 2
... OPT line (in blue) represents the known optimal value [49] (or best known upper bound) and, as such, no other method can go below it. By analyzing Figure 6, it can be verified that the SPT method is the one that needs more time to complete all problems, while CUN obtained the better solution values. Frequently, the normalized makespan is used as a measure (instead using the raw makespan value) to compare the quality of the solutions [2]. ...
Similar publications
The job shop scheduling problem (JSSP) is a very popular NP-hard optimization problem that involves assigning jobs to resources. Recent advances in the field of memetic algorithms show that explicitly managing the diversity of the population by taking into account the stopping criterion with the aim of dynamically adapting the balance between explo...
The problem of minimizing total quadratic completion time in flow shops presents a significant challenge in industries such as chemical, metallurgical, and ceramic manufacturing. Initially investigated by Ren and coauthors (2016) [1], this problem addresses the need to balance intermediate inventory reduction with maximizing resource utilization, p...
Quality scheduling in industrial job shops is crucial. Although neural networks excel in solving these problems, their limited explainability hinders their widespread industrial adoption. In this research, we introduce an innovative framework for solving job shop scheduling problems (JSSP). Our methodology leverages Petri nets to model the job shop...
The most challenging aspect identified in this study revolves around effectively managing machine breakdowns to ensure uninterrupted production. This paper presents a real-time dynamic scheduling model that addresses the challenges of the Flexible Job Shop Scheduling Problem (FJSSP) while considering the occurrence of random machine breakdowns. An...
Nowadays many industry consider an interval time as a due date instead of precise points in time. In this study, the hybrid flow shop scheduling problem with basic blocking constraint is tackled. Where jobs, if done within a due window, are deemed on time. Therefore, the criterion is to minimize the sum of weighted earliness and tardiness. This var...
Citations
... An intelligent scheduling approach using reinforcement learning was developed and evaluated. Enhanced decision-making capabilities and improved scheduling performance were demonstrated by adapting to changing conditions in real time [41]. In another approach, to select the best dispatch rule for a machine at each decision point, a neural network was trained, using shop floor characteristics inherited from a simulation [42]. ...
Production scheduling is a critical task in the management of manufacturing systems. It is difficult to derive an optimal schedule due to the problem complexity. Computationally expensive and time-consuming solutions have created major issues for companies trying to respect their customers’ demands. Simple dispatching rules have typically been applied in manufacturing practice and serve as a good scheduling option, especially for small and midsize enterprises (SMEs). However, in recent years, the progress in smart systems enabled by artificial intelligence (AI) and machine learning (ML) solutions has revolutionized the scheduling approach. Under different production circumstances, one dispatch rule may perform better than others, and expert knowledge is required to determine which rule to choose. The objective of this work is to design and implement a framework for the modeling and deployment of a deep reinforcement learning (DRL) agent to support short-term production scheduling. The DRL agent selects a dispatching rule to assign jobs to manufacturing resources. The model is trained, tested and evaluated using a discrete event simulation (DES) model that simulates a pilot case from the bicycle production industry. The DRL agent can learn the best dispatching policy, resulting in schedules with the best possible production makespan.
... Scheduling optimization is another vital sector in which machine learning can bring change. Traditional scheduling techniques face bot-oul challenges involving rapid and unexpected changes in workload patterns, which are typical in modern advanced data centers [23]- [25]. In a nutshell, Fernández-Cerero et al. [26] proposed a model that uses gradient boosting regression to enhance the scheduling process. ...
The accelerating adoption of cloud models has increased the amount of complexity in cloud data centers with particular emphasis on the energy management load efficiency on resources in relation to the workload tris. This paper introduces a new fully-converged architectural framework enhanced by machine learning features that addresses several common issues: workload forecasting, adaptive scheduling and energy optimization for better performance at hyper scale cloud data centers. Using a Gated Recurrent Unit (GRU), the method described in the paper learns and remembers the complicated sequence of nonlinear workloads, enabling it to allocate resources appropriately in advance. Schedule optimization is achieved via a gradient boosting method in which resource managers are proactively chosen based on the expected workload in order to enhance scheduling and reduce task waiting times. Furthermore, virtual machine clustering models based on energy consumption patterns are incorporated into the design to enhance the framework's efficiency in energy usage optimization by controlling the number of virtual machines and their migration. The test case application based on realistic data center traces verified better energy effectiveness and resource utilization levels with service level agreement compliance for this entire integrated method. The study further underscores the opportunity for machine learning models to pinpoint and even combine distinct operational stresses prevalent in cloud data center environments.
... The advent of RL has introduced new possibilities in solving COPs, attracting significant attention for its potential in complex decision-making environments [24]. One of the widely used RL algorithms is Q-learning, in which the agent learns the optimal policy for scheduling jobs [5] [6]. ...
The job shop scheduling problem (JSSP) is a well-known NP-hard combinatorial optimization problem that focuses on assigning tasks to limited resources while adhering to certain constraints. Currently, deep reinforcement learning (DRL)-based solutions are being widely used to solve the JSSP by defining the problem structure on disjunctive graphs. Some of the proposed approaches attempt to leverage the structural information of the JSSP to capture the dynamics of the environment without considering the time dependency within the JSSP. However, learning graph representations only from the structural relationship of nodes results in a weak and incomplete representation of these graphs which does not provide an expressive representation of the dynamics in the environment. In this study, unlike existing frameworks, we defined the JSSP as a dynamic graph to explicitly consider the time-varying aspect of the JSSP environment. To this end, we propose a novel DRL framework that captures both the spatial and temporal attributes of the JSSP to construct rich and complete graph representations. Our DRL framework introduces a novel attentive graph isomorphism network (Attentive-GIN)-based spatial block to learn the structural relationship and a temporal block to capture the time dependency. Additionally, we designed a gated fusion block that selectively combines the learned representations from the two blocks. We trained the model using the proximal policy optimization algorithm of reinforcement learning. Experimental results show that our trained model exhibits significant performance enhancement compared to heuristic dispatching rules and learning-based solutions for both randomly generated datasets and public benchmarks.
... Recently, RL has been applied to a variety of construction-related scenarios, including maintenance strategy planning for components [26], [27], resource allocation [28], schedule optimization [29], and machinery operation on construction sites [30]. Therefore, to effectively use RL in short-term construction processes, which involve modeling actors and considering resource flows, it is crucial to transform agents' spatial actions and decision-making related to target selection into an MDP framework. ...
Fine-grained simulation of floor construction processes is essential for supporting lean management and the integration of information technology. However, existing research does not adequately address the on-site decision-making of constructors in selecting tasks and determining their sequence within the entire construction process. Moreover, decision-making frameworks from computer science and robotics are not directly applicable to construction scenarios. To facilitate intelligent simulation in construction, this study introduces the Construction Markov Decision Process (CMDP). The primary contribution of this CMDP framework lies in its construction knowledge in decision, observation modifications and policy design, enabling agents to perceive the construction state and follow policy guidance to evaluate and reach various range of targets for optimizing the planning of construction activities. The CMDP is developed on the Unity platform, utilizing a two-stage training approach with the multi-agent proximal policy optimization algorithm. A case study demonstrates the effectiveness of this framework: the low-level policy successfully simulates the construction process in continuous space, facilitating policy testing and training focused on reducing conflicts and blockages among crews; and the high-level policy improving the spatio-temporal planning of construction activities, generating construction patterns in distinct phases, leading to the discovery of new construction insights.
... Cunha vd. [65], makine öğrenimi tekniklerine dayalı atölye tipi çizelgeleme problemlerini çözmek için yenilikçi bir yaklaşım sunmuşlardır. Genel performanslarını iyileştirmek ve mevcut yaklaşımların sunduğu sınırlamaların üstesinden gelmek için pekiştirmeli öğrenmeyi, çizelgeleme sistemlerine dahil eden yeni bir mimari sunmuşlardır. ...
... 14 Cunha vd. [65] 2021 Atölye tipi çizelgeleme problemlerini çözmek için makine öğrenimi tekniklerine dayalı, pekiştirmeli öğrenme yaklaşımını sunmuşlardır. 15 Khuntiyaporn vd. ...
Pekiştirmeli öğrenme, günümüz dünyasında birçok gerçek hayat problemine çözüm bulmada aktif bir şekilde kullanılmakta ve endüstri içerisinde de umut verici yöntemler arasında gösterilmektedir. Bu çalışmada, makine öğrenmesinin bir alt dalı olan pekiştirmeli öğrenmenin iş çizelgeleme problemlerinin çözümündeki etkisi araştırılmıştır. Bu kapsamda, öncelikle pekiştirmeli öğrenmede durum tanımı, eylem seçimi ve öğrenme algoritmaları açıklanmıştır. Ardından, iş çizelgeleme probleminin sınıflandırmasına yer verilmiştir. Literatürde yer alan iş çizelgelemede, pekiştirmeli öğrenme yönteminin kullanıldığı, son yirmi yılda yayımlanan, 50 makale çalışmasına yer verilmiştir. Literatürde yer alan çalışmaların çizelgeleme problemlerinin çözümü üzerinde gösterdiği etki değerlendirilmiştir. Son bölümde pekiştirmeli öğrenmenin diğer çözüm yöntemlerine kıyasla güçlü ve zayıf yönlerine yer verilmiş ayrıca gelecekte yapılacak araştırmalara yönelik değerlendirmelerde bulunulmuştur.
... It distinguishes between the cloud and fog computing models, highlighting the unique characteristics and challenges presented by fog computing. The author proposed [26] delves into the nuances of resource allocation and scheduling in fog environments, emphasizing the need for adaptive and responsive strategies. It surveys existing scheduling algorithms. ...
... Cunha et al. [20] developed a complete scheduling system that incorporates reinforcement learning into scheduling solutions in order to improve their performance. The system is activated by an instance of the problem to be solved, and the decoder converts the information contained in the file into viable objects in the system, i.e. objects representing machines, operations, among others. ...
Scheduling is an important process that can have a significant impact on the productivity of a company. This literature review explores how Machine Learning algorithms have been used to solve scheduling problems. This article is composed of several stages: the two most significant areas – Scheduling and Machine Learning - are examined, a bibliometric analysis of the existing literature is performed, and case studies in the areas of Scheduling and Machine Learning are analyzed. The bibliometric analysis evidenced the recent growth of this research area. Several supervised learning algorithms are used to solve scheduling problems, although the reinforcement learning ones have seen considerable advances in recent years. They are applied to autonomously solve real-world problems and for enhancing the performance of traditional optimization techniques,
such as Metaheuristics. Improving characteristics of these techniques, such as their exploration capabilities, has shown significant developments, however, it is still limited to a certain number of Metaheuristics. Therefore, in future research,
it would be interesting to use these algorithms to enhance the performance of less-explored Metaheuristics and thereby overcome their main challenges. Machine Learning stands out as an emerging field with the potential to positively contribute to the development of effective strategies capable of solving scheduling problems.
... The target of the scheduling operation is to maximize or minimize the objective function, i.e., finding an optimal plan. Scheduling problems can be summarized within three categories [4]: i) flow shop: all operations have a similar order of execution through the machine, and all tasks are the same; ii) Job shop: unlike flow shop, in job shop tasks are different, and each task has its unique operations. Operation orders of tasks are predefined and can be different from task to task; iii) Open shop: operation execution order is not an issue, i.e., an operation can be executed at any time on any machine. ...
... In reinforcement learning, the agent has no insight into the best possible actions, so it must rely on an iterative process to determine the most gainful options. The main shortcomings to date are the lengthy time required to train an agent and the challenge of developing an agent capable of acting competently in a complex scenario [4]. The agent and the environment are the two main constituents of any reinforcement learning problem. ...
Nowadays, rule-based heuristic methods for scheduling planning in production environments are commonly used, but their effectiveness is heavily dependent on expert domain expertise. In this manner, decision-making performance cannot be assured, nor can the dynamic scheduling demand in the job-shop production environment be met. Therefore, Dynamic Job Shop Scheduling Problems (DJSSPs) have received increased interest from researchers in recent decades. However, the development of reinforcement learning (RL) approaches for solving DJSSPs has not been fully realized. In this paper, we used Deep Reinforcement Learning (DRL) approach on DJSSP to minimize the Makespan. A Deep Q Network (DQN) algorithm is designed with state features, actions, and rewards. Finally, the performance of the proposed solution is compared to other algorithms and benchmark research using two categories of benchmark instances. The empirical results show that the proposed DRL approach outperforms other DRL methods and dispatching rules (heuristics).
... In the actual world of production, the time necessary to produce solutions is quite important because receiving plans late might impair a company's productivity and resource use. As a result, the performance of the scheduling strategy is defined not only by the quality of the solution but also by the time required to produce the solution [41]. Although DRL-based approaches deliver acceptable solutions timely, training the models takes a long time, especially for larger problem instances. ...
Job shop scheduling problem (JSSP) is one of the well‐known NP‐hard combinatorial optimization problems (COPs) that aims to optimize the sequential assignment of finite machines to a set of jobs while adhering to specified problem constraints. Conventional solution approaches which include heuristic dispatching rules and evolutionary algorithms has been largely in use to solve JSSPs. Recently, the use of reinforcement learning (RL) has gained popularity for delivering better solution quality for JSSPs. In this research, we propose an end‐to‐end deep reinforcement learning (DRL) based scheduling model for solving the standard JSSP. Our DRL model uses attention‐based encoder of Transformer network to embed the JSSP environment represented as a disjunctive graph. We introduced Gate mechanism to modulate the flow of learnt features by preventing noise features from propagating across the network to enrich the representations of nodes of the disjunctive graph. In addition, we designed a novel Gate‐based graph pooling mechanism that preferentially constructs the graph embedding. A simple multi‐layer perceptron (MLP) based action selection network is used for sequentially generating optimal schedules. The model is trained using proximal policy optimization (PPO) algorithm which is built on actor critic (AC) framework. Experimental results show that our model outperforms existing heuristics and state of the art DRL based baselines on generated instances and well‐known public test benchmarks. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.
... Also, there is a well-documented body of literature for solving these classic optimization algorithms (Taha 2017). But, in a recent study, for example, Cunha et al. (2021) proposed a novel and general RL-based framework using DQN to solve one form of scheduling and resource allocation problem (i.e., job shop scheduling problem). Through a computational study, they showed that the trained agent could yield near-optimal solutions with negligible computation time. ...
The construction engineering and management (CEM) domain frequently meets complex tasks due to the unavoidable complicated operation environments and the involvement of numerous workers. Being able to simulate these tasks with promised designed goals, reinforcement learning (RL) can help CEM engineers reach enhanced strategies in multi-/single-objective sequential decision-making under various sources of uncertainties. To provide a better understanding of the status quo of the RL application in CEM and its potential benefits with their strengths and limitations, this study systematically reviewed 85 CEM-related RL-based studies as a result of queries from three main scientific databases, namely Scopus, Science Direct, and Web of Science. The results of this review reveal that researchers have been increasingly applying RL methods in CEM domains, such as building energy management, infrastructure management, construction machinery, and even safety in the last few years. Our analysis showed that the reviewed papers are associated with different limitations such as generalizability, justification of selecting the approaches, and validation. This review paper alongside the presented overview of the RL methodology can assist researchers and practitioners in CEM with (1) gaining a high-level and intuitive understanding of RL algorithms, (2) identifying previous and possible future opportunities for applying RL in complex decision-making, and (3) fine tuning, proper validation, and optimizing to-be-developed RL frameworks in their future studies and applications.
