ArticlePublisher preview available

Qsmix: Q-learning-based task scheduling approach for mixed-critical applications on heterogeneous multi-cores

Authors:
Fatemeh Afshari
Fatemeh Afshari
Fatemeh Afshari
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Athena Abdi at K.N.Toosi University of Technology
Athena Abdi
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

In this paper, a Q-learning-based task scheduling approach for mixed-critical application on heterogeneous multi-cores (QSMix) to optimize their main design challenges is proposed. This approach employs reinforcement learning capabilities to optimize execution time, power consumption, reliability and temperature of the heterogeneous multi-cores during task scheduling process. In QSMix, a reward function is defined to consider all target design parameters simultaneously and is tuned based on applying punishment for unwanted conditions during the learning. The learning process of QSMix is led by utilizing the defined reward function during constructing the Q-table for various execution scenarios. Afterward, the best solution is selected from the constructed Q-table based on the system’s policy to achieve a near-optimal solution that meets the existing trade-offs among objectives while considering its constraints properly. To evaluate our proposed QSMix, several experiments are performed to show its effectiveness in finding appropriate solutions and its gradual behavior during learning process. Moreover, the performance of QSMix in terms of optimizing the target design parameters is compared to various related research. The results confirm that QSMix has average improvement about 9% over related studies in joint optimization of execution time, power consumption, reliability and temperature.
Figures - available from: The Journal of Supercomputing
This content is subject to copyright. Terms and conditions apply.
A preview of this full-text is provided by Springer Nature.
Learn more
Content available from The Journal of Supercomputing
This content is subject to copyright. Terms and conditions apply.
Vol.:(0123456789)
The Journal of Supercomputing (2024) 80:17895–17922
https://doi.org/10.1007/s11227-024-06096-8
1 3
Qsmix: Q‑learning‑based task scheduling approach
formixed‑critical applications onheterogeneous
multi‑cores
FatemehAfshari1· AthenaAbdi1
Accepted: 20 March 2024 / Published online: 6 May 2024
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature
2024
Abstract
In this paper, a Q-learning-based task scheduling approach for mixed-critical appli-
cation on heterogeneous multi-cores (QSMix) to optimize their main design chal-
lenges is proposed. This approach employs reinforcement learning capabilities to
optimize execution time, power consumption, reliability and temperature of the het-
erogeneous multi-cores during task scheduling process. In QSMix, a reward function
is defined to consider all target design parameters simultaneously and is tuned based
on applying punishment for unwanted conditions during the learning. The learning
process of QSMix is led by utilizing the defined reward function during constructing
the Q-table for various execution scenarios. Afterward, the best solution is selected
from the constructed Q-table based on the system’s policy to achieve a near-opti-
mal solution that meets the existing trade-offs among objectives while considering
its constraints properly. To evaluate our proposed QSMix, several experiments are
performed to show its effectiveness in finding appropriate solutions and its gradual
behavior during learning process. Moreover, the performance of QSMix in terms
of optimizing the target design parameters is compared to various related research.
The results confirm that QSMix has average improvement about 9% over related
studies in joint optimization of execution time, power consumption, reliability and
temperature.
Keywords Task scheduling· Mixed-critical applications· Heterogeneous multi-
cores· Q-learning· Multi-objective optimization
* Athena Abdi
a_abdi@kntu.ac.ir
1 Faculty ofComputer Engineering, K. N. Toosi University ofTechnology, Tehran, Iran
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... Among them, Q-learning, as a reinforcement learning method, has attracted much attention because it can learn adaptive strategies through experience in dynamic environments. Q-learning can maximize resource utilization when computing resources are limited, thereby optimizing system performance and improving overall efficiency [9]. ...
Reinforcement Learning for Adaptive Resource Scheduling in Complex System Environments
Preprint
  • Nov 2024
This study presents a novel computer system performance optimization and adaptive workload management scheduling algorithm based on Q-learning. In modern computing environments, characterized by increasing data volumes, task complexity, and dynamic workloads, traditional static scheduling methods such as Round-Robin and Priority Scheduling fail to meet the demands of efficient resource allocation and real-time adaptability. By contrast, Q-learning, a reinforcement learning algorithm, continuously learns from system state changes, enabling dynamic scheduling and resource optimization. Through extensive experiments, the superiority of the proposed approach is demonstrated in both task completion time and resource utilization, outperforming traditional and dynamic resource allocation (DRA) algorithms. These findings are critical as they highlight the potential of intelligent scheduling algorithms based on reinforcement learning to address the growing complexity and unpredictability of computing environments. This research provides a foundation for the integration of AI-driven adaptive scheduling in future large-scale systems, offering a scalable, intelligent solution to enhance system performance, reduce operating costs, and support sustainable energy consumption. The broad applicability of this approach makes it a promising candidate for next-generation computing frameworks, such as edge computing, cloud computing, and the Internet of Things.
View
View
Machine learning in manufacturing and industry 4.0 applications
Article
Full-text available
  • Aug 2021
The machine learning (ML) field has deeply impacted the manufacturing industry in the context of the Industry 4.0 paradigm. The industry 4.0 paradigm encourages the usage of smart sensors, devices, and machines, to enable smart factories that continuously collect data pertaining to production. ML techniques enable the generation of actionable intelligence by processing the collected data to increase manufacturing efficiency without significantly changing the required resources. Additionally, the ability of ML techniques to provide predictive insights has enabled discerning complex manufacturing patterns and offers a pathway for an intelligent decision support system in a variety of manufacturing tasks such as intelligent and continuous inspection, predictive maintenance, quality improvement, process optimisation, supply chain management, and task scheduling. While different ML techniques have been used in a variety of manufacturing applications in the past, many open questions and challenges remain, from Big data curation, storage, and understanding, data reasoning to enable real-time actionable intelligence to topics such as edge computing and cybersecurity aspects of smart manufacturing. Hence, this special issue is focused on bringing together a wide range of researchers to report the latest efforts in the fundamental theoretical as well as experimental aspects of ML and their applications in manufacturing and productionsystems.
View
A heuristic-based task scheduling algorithm for scientific workflows in heterogeneous cloud computing platforms
Article
Full-text available
  • Jun 2021
Task scheduling problem in a cloud computing environment is one of the most important issues in this field. Because task schedulers should be aware of underlying platform heterogeneity, task interdependencies, and virtual machine (VM) variable configurations. An efficient task scheduling algorithm can potentially increase the efficiency of cloud computing, by choosing the appropriate virtual machine to do a specific task-scheduling problem, for dependent tasks on heterogeneous resources which is a well-known NP-Hard problem. To reduce minimum total execution time, makespan, in cloud computing many heuristics were presented in literature but lots of them suffer from low efficiency. In this article, a new task priority strategy and applying task duplication methods are proposed, for solving the task scheduling problem of the dependent tasks in heterogeneous cloud computing systems. The novelty of the current paper is to present a new list scheduling algorithm with a new task priority strategy and applying pertinent task duplication techniques. This paper uses optimistic cost table downward (OCTd) and optimistic cost table upward (OCTu) procedures to prioritize tasks in an efficient ordered list; then, it utilizes Heterogeneous Earliest Finish Time (HEFT)-duplication method for performing task duplication technique which significantly reduces makespan. To validate the proposal, we experimentally analyzed the proposed scheduling algorithm with different scientific workflows such as Molecular, LU-Like, FFT, and Montage datasets. The performance comparison of the novel heuristic scheduling algorithm against other existing approaches proved the superiority of proposed algorithm in terms of makespan, speedup, SLR, and efficiency which are prominent scheduling evaluation metrics.
View
Embedded System Design: Embedded Systems Foundations of Cyber-Physical Systems, and the Internet of Things
Book
Full-text available
  • Jan 2021
A unique feature of this open access textbook is to provide a comprehensive introduction to the fundamental knowledge in embedded systems, with applications in cyber-physical systems and the Internet of things. It starts with an introduction to the field and a survey of specification models and languages for embedded and cyber-physical systems. It provides a brief overview of hardware devices used for such systems and presents the essentials of system software for embedded systems, including real-time operating systems. The author also discusses evaluation and validation techniques for embedded systems and provides an overview of techniques for mapping applications to execution platforms, including multi-core platforms. Embedded systems have to operate under tight constraints and, hence, the book also contains a selected set of optimization techniques, including software optimization techniques. The book closes with a brief survey on testing. This fourth edition has been updated and revised to reflect new trends and technologies, such as the importance of cyber-physical systems (CPS) and the Internet of things (IoT), the evolution of single-core processors to multi-core processors, and the increased importance of energy efficiency and thermal issues. • Provides a semester-length textbook, with comprehensive coverage of the fundamental knowledge in embedded and cyber-physical systems; • Links modeling and hardware, in order to bridge the gap between hardware and software, allowing readers to put these into perspective; • Couples practical aspects of embedded system design with an introduction to more theoretical aspects; • Includes extensive exercises for each chapter; • Videos, software, and PowerPoint slides are separately available.
View
Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learning
Article
Full-text available
  • Jan 2021
We propose a mechanism for distributed resource management and interference mitigation in wireless networks using multi-agent deep reinforcement learning (RL). We equip each transmitter in the network with a deep RL agent that receives delayed observations from its associated users, while also exchanging observations with its neighboring agents, and decides on which user to serve and what transmit power to use at each scheduling interval. Our proposed framework enables agents to make decisions simultaneously and in a distributed manner, unaware of the concurrent decisions of other agents. Moreover, our design of the agents’ observation and action spaces is scalable, in the sense that an agent trained on a scenario with a specific number of transmitters and users can be applied to scenarios with different numbers of transmitters and/or users. Simulation results demonstrate the superiority of our proposed approach compared to decentralized baselines in terms of the tradeoff between average and 5th percentile user rates, while achieving performance close to, and even in certain cases outperforming, that of a centralized information-theoretic baseline. We also show that our trained agents are robust and maintain their performance gains when experiencing mismatches between train and test deployments.
View
A Heuristic Method for Data Allocation and Task Scheduling on Heterogeneous Multiprocessor Systems Under Memory Constraints
Chapter
  • Mar 2024
  • Lect Notes Comput Sci
Computing workflows in heterogeneous multiprocessor systems are often depicted as directed acyclic graphs (DAGs) comprising tasks and data blocks. These elements represent computational modules and their interdependencies, where the output data generated by one task serves as input for other tasks. However, in certain workflows, such as task schedules in digital signal processors, it is essential to account for the memory capacity limitations of data blocks when scheduling the tasks. The main objective of this paper is to address the challenge of data allocation and task scheduling under memory constraints, particularly on shared memory platforms. We propose an integer linear programming model to formulate the problem, which can be viewed as an extension of the flexible job shop scheduling problem. The goal is to minimize the length of the critical path. We propose a tabu search algorithm (TS) to tackle the data allocation and task scheduling problem under memory constraints. The TS algorithm incorporates several distinguishing features, such as a greedy initial solution construction method and a mixed neighborhood evaluation strategy that combines exact and approximate evaluation methods. Experimental results on randomly generated instances demonstrate that the TS algorithm can obtain high-quality solutions within a reasonable computational time. On average, the TS algorithm improves the makespan by 5–25% compared to the widely used classical load balancing algorithms in the literature. Furthermore, we analyze some key features of the TS algorithm to identify the contributing factors to its success.
View
ENF-S: An Evolutionary-Neuro-Fuzzy Multi-Objective Task Scheduler for Heterogeneous Multi-Core Processors
Article
  • Jul 2023
In this paper, an evolutionary-neuro-fuzzy-based task scheduling approach (ENF-S) to jointly optimize the main critical parameters of heterogeneous multi-core systems is proposed. This approach has two phases: first, the fuzzy neural network (FNN) is trained using a non-dominated sorting genetic algorithm (NSGA-II), considering the critical parameters of heterogeneous multi-core systems on a training data set consisting of different application graphs. These critical parameters are execution time, temperature, failure rate, and power consumption. The output of the trained FNN determines the criticality degree for various processing cores based on the system's current state. Next, the trained FNN is employed as an online scheduler to jointly optimize the critical objectives of multi-core systems at runtime. Due to the uncertainty in sensor measurements and the difference between computational models and reality, applying the fuzzy neural network is advantageous. The efficiency of ENF-S is investigated in various aspects including its joint optimization capability, appropriateness of generated fuzzy rules, comparison with related research, and its overhead analysis through several experiments on real-world and synthetic application graphs. Based on these experiments, our ENF-S outperforms the related studies in optimizing all design criteria. Its improvements over related methods are estimated 19.21%19.21\% in execution time, 13.07%13.07\% in temperature, 25.09%25.09\% in failure rate, and 13.16%13.16\% in power consumption, averagely.
View
ReDeSIGN: Re use of De bug S tructures for I mprovement in Performance G ain of N oC based MPSoCs
Article
  • Jan 2022
Network-on-Chip (NoC) is considered as a scalable interconnect medium for Multiprocessor System-on-Chip (MPSoC) due to its ability to provide high bandwidth and low latency communication. With the increasing intricacy of the modern-day systems, the state-of-the-art NoCs are becoming extremely complex. Design-for-Debug (DFD) structures are integrated to the system for the validation of such complex modules during post-silicon debug. However, after the system validation and mass production, the DFD hardware remains vestigial on the design. In this context, we propose ReDeSIGN, a framework to reuse the DFD infrastructure during the in-field operation for performance enhancement of the NoC-based MPSoCs. Major contributions of our work include reuse of (i) trace buffer as extended Virtual Channel (VC) for network throughput improvement, (ii) trace prioritization hardware for critical data prioritization, and (iii) packet monitor module for packet starvation control. Experimental evaluations with real benchmarks show an average of 11.46% increase in network throughput, 34.93% decrease in critical data latency, and 19.17% decrease in packet starvation for an 8x8 homogeneous system.
View
Deadline-Aware Task Scheduling for IoT Applications in Collaborative Edge Computing
Article
  • Jul 2021
Collaborative edge computing (CEC) is a promising technology for supporting latency-sensitive Internet of Things (IoT) applications by distributing computation tasks among edge nodes. However, the edge resources are limited compared to the traditional cloud. Therefore, it is important to decide when and where to execute tasks from IoT applications to meet the latency requirements of the applications. In this study, we propose a task scheduling algorithm in the CEC environment, which considers the priority (i.e., deadline) of tasks and network flows with the aim of reducing the deadline miss ratio (DMR) of IoT applications. The simulation results demonstrate that the proposed algorithm can significantly reduce the DMR of other benchmark methods.
View
Dynamic Frequency Scaling for Low-Power Operation of a Single-Core Processor: A Radial Basis Function Approach
Article
  • Mar 2021
Dynamic Frequency Scaling (DFS) is one of the most important approaches to saving power in modern day processors. With ever increasing complexity at system, circuit and device levels, the problem of achieving an efficient DFS boils down to multi-parametric non-linear optimization. Therefore, it is imperative to explore ingenuous approaches to DFS which could identify an optimal underclocking frequency on-the-fly using an adaptive mechanism. This paper proposes a neural network approach to DFS of a ubiquitous single core processor where several performance parameters of the processor were monitored under application of a number of clocking frequencies. The data set thus generated was used to train two classifiers viz. the Radial Basis Function Network (RBFN) and the Probabilistic Neural Network (PNN). Under changing parametric conditions of the proposed network, the model was fit to performance monitoring data while running 64-point & 1024-point FFT applications; and one benchmark application named basicmath. The performance of both the classifiers was found to be promising and good generalization was obtained with all three data sets. The results indicate towards suitability of trained Radial Basis family of networks for on-chip deployment for implementing on-the-fly DFS.
View
A Survey of Low-Energy Parallel Scheduling Algorithms
Article
  • Feb 2021
High energy consumption is one of the biggest obstacles to the rapid development of computing systems, and reducing energy consumption is quite urgent and necessary for sustainable computing. Low-energy scheduling based on dynamic voltage and frequency scaling (DVFS) is one of the most commonly used energy optimization techniques. Recent survey works have reviewed some low-energy scheduling algorithms, but there is currently no systematic review in low-energy parallel scheduling algorithms. With the increasing complexity of function requirements, many parallel applications have been executed in various sustainable computing systems. In this paper, we survey recent advances in low-energy parallel scheduling algorithms according to three scheduling styles, namely: 1) energy-efficient parallel scheduling algorithms; 2) energy-aware parallel scheduling algorithms; and 3) energy-conscious parallel scheduling algorithms. Low-energy parallel scheduling algorithms basically involve five categories of 1) heuristic algorithms; 2) meta-heuristic algorithms; 3) integer programming algorithms; 4) machine learning algorithms; and 5) game theory algorithms. Further, we introduce the future trends in low-energy parallel scheduling algorithms from the perspectives of new requirements and future developments. By surveying the recent advances and introducing the future trends, we expect to provide researchers with a systematic reference and development directions in low-energy parallel scheduling for sustainable computing systems.
View