Query Optimization - Science topic

Artificial intelligence and Database Management Systems Integration bring intelligence, adaptability, and independence in the world of databases. Relational database management systems structure the data and have been the foundations for implementing them, although they face several challenges that have arisen from modern-day environments of comput...

In the era of data-driven decision-making, dashboards have become indispensable tools for visualizing and interpreting vast datasets in real-time. However, as data complexity and volume grow, the performance of dashboards can deteriorate due to inefficient query execution. This paper explores an AI-driven approach to query optimization tailored spe...

Database query optimization plays a crucial role in the efficiency of database management systems (DBMS). As data complexity increases, optimizing query performance becomes even more critical. This paper investigates the application of combinatorial mathematics in query optimization, particularly focusing on the algorithms and techniques that aid i...

Recent deployments of learned query optimizers use expensive neural networks and ad-hoc search policies. To address these issues, we introduce \textsc{LimeQO}, a framework for offline query optimization leveraging low-rank learning to efficiently explore alternative query plans with minimal resource usage. By modeling the workload as a partially ob...

Disk-spill occurs when query execution exceeds available memory, forcing intermediate data to be written to disk, significantly impacting performance. This paper explores the mechanics of disk-spill, its triggers, and its implications for query optimization in database management systems. We examine how modern query engines handle disk-spill, the t...

The exponential growth of big data has necessitated the development of novel approaches to query processing that can yield results with high efficiency and reduced resource consumption. Traditional query processing techniques, while accurate, often fall short in delivering timely insights due to the scale and complexity of modern datasets. Approxim...

The exponential growth of data in modern computing environments has necessitated advanced techniques for data analytics and query optimization. Traditional database management systems struggle to efficiently process vast and complex datasets. Graph-based approaches have emerged as a powerful solution due to their ability to model intricate relation...

This technical report extends the SIGMOD 2025 paper "A Modular Graph-Native Query Optimization Framework" by providing a comprehensive exposition of GOpt's advanced technical mechanisms, implementation strategies, and extended evaluations. While the original paper introduced GOpt's unified intermediate representation (GIR) and demonstrated its perf...

The rapid deployment of NoSQL databases to manage large complex data, unstructured, and sample dataset has posed huge challenges for the efficient processing and analysis of such data samples. Current approaches, particularly those using rule-based schemes for schema detection and query optimization, fail to address the dynamic and heterogeneous na...

Query optimization in high-dimensional heterogeneous data spaces has become a crucial challenge in modern data management systems. The exponential growth of data in diverse domains, such as scientific computing, healthcare, finance, and social networks, has led to the proliferation of complex and high-dimensional datasets. These datasets often exhi...

Secure Multi-Party Computation (MPC) enables privacy-preserving collaborative data analysis but suffers from high computational overhead, communication latency, and inefficient query execution. Traditional caching mechanisms, such as SMPCache, attempt to optimize performance but face limitations due to high storage overhead, low cache hit rates, an...

In the era of digital transformation, the explosion of data from diverse sources has led to the development of heterogeneous big data environments comprising structured, semi-structured, and unstructured datasets distributed across various platforms and technologies. These environments pose significant challenges to efficient query processing due t...

Bloom filters have emerged as a fundamental data structure for approximate query optimization in database management and big data processing. The increasing scale of data necessitates efficient query execution techniques, as exact query processing can be computationally expensive. Bloom filters offer a probabilistic approach to approximate membersh...

Database query optimization is a critical aspect of ensuring the performance and efficiency of database systems, particularly in environments with large volumes of data. Query optimization aims to reduce the time and resources required to retrieve data, ultimately improving response times for end-users and ensuring scalability as database size grow...

Deep Learning for Data Management: Enhancing Data Structuring and Query Optimization The rapid expansion of big data has created significant challenges in data structuring, indexing, and query optimization. Traditional data management approaches often struggle with scalability, efficiency, and adaptability in handling large and complex datasets. De...

The integration of artificial intelligence into medical decision support systems has revolutionized healthcare by improving diagnostic accuracy and treatment planning. However, a major challenge in AI-driven decision-making is the lack of explainability, which creates barriers to trust and adoption among medical professionals. Explainable AI (XAI)...

Clinical knowledge graphs have emerged as powerful tools for representing and reasoning about medical data, enabling researchers and practitioners to extract meaningful insights. However, querying these graphs efficiently remains a significant challenge due to the complexity and size of medical ontologies. Ontology-based query optimization seeks to...

Bioinformatics has experienced a rapid increase in data generation, necessitating efficient query optimization for managing and retrieving biological information. Traditional query optimization techniques struggle to cope with the volume and complexity of bioinformatics data. Machine learning (ML) has emerged as a transformative approach to improvi...

The rapid expansion of digital medical records and literature has created a pressing need for efficient and precise query optimization techniques. Traditional search mechanisms often struggle with the complexity and ambiguity of medical terminology, leading to irrelevant or incomplete search results. This study explores the application of context-a...

The rapid digitization of healthcare systems has led to an explosion of patient health records, requiring efficient temporal query optimization to ensure accurate and timely retrieval of patient data. Temporal databases manage time-dependent information, making them essential for tracking patient histories, medical treatments, and disease progressi...

With the rapid digitization of healthcare services, electronic health records (EHRs) have become an essential tool for efficient patient care, research, and administrative purposes. However, the sensitive nature of healthcare data necessitates robust privacy measures while ensuring that database queries remain efficient and effective. Query optimiz...

Query optimization, which finds the optimized execution plan for a given query, is a complex planning and decision-making problem within the exponentially growing plan space in database management systems (DBMS). Traditional optimizers heavily rely on a certain cost model constructed by various heuristics and empirical tuning, probably leading to g...

In recent years, there has been a growing interest in using machine learning (ML) in query optimization to select more efficient plans. Existing learning-based query optimizers use certain model architectures to convert tree-structured query plans into representations suitable for downstream ML tasks. As the design of these architectures significan...

The available parallelism and heterogeneity of emerging computer systems must be exploited for being able to process the huge amounts of data produced every day. As a consequence, we observe an increasing research interest in accelerating database query processing on multi-cores and attached co-processors like Graphics Processing Units (GPUs) and F...

Many traditional query optimizers support setting query configuration hints to steer the optimization process. Using these hints efficiently to decrease analytical query runtimes has been successful for learned models. However, the most successful approaches use queries annotated with their best optimizer hint beforehand. Even though learned models...

We propose a novel model for learned query optimization which provides query hints leading to better execution plans. The model addresses the three key challenges in learned hint-based query optimization: reliable hint recommendation (ensuring non-degradation of query latency), efficient hint exploration, and fast inference. We provide an in-depth...

The exponential growth of data in modern computing environments has necessitated advanced techniques for data analytics and query optimization. Traditional database management systems struggle to efficiently process vast and complex datasets. Graph-based approaches have emerged as a powerful solution due to their ability to model intricate relation...

In the world of relational database management, the query optimizer is a critical component that significantly impacts query performance. To address the challenge of optimizing query performance due to the complexity of optimizers -- especially with join operations -- we introduce Jovis. This novel visualization tool provides a window into the ofte...

This chapter delves into the emerging field of neuro-symbolic query optimization for knowledge graphs (KGs), presenting a comprehensive exploration of how neural and symbolic techniques can be integrated to enhance query processing. Traditional query optimizers in knowledge graphs rely heavily on symbolic methods, utilizing dataset summaries, stati...

Customer demand, regulatory pressure, and engineering efficiency are the driving forces behind the industry-wide trend of moving from siloed engines and services that are optimized in isolation to highly integrated solutions. This is confirmed by the wide adoption of open formats, shared component libraries, and the meteoric success of integrated d...

We introduce the \textit{Extract-Refine-Retrieve-Read} (ERRR) framework, a novel approach designed to bridge the pre-retrieval information gap in Retrieval-Augmented Generation (RAG) systems through query optimization tailored to meet the specific knowledge requirements of Large Language Models (LLMs). Unlike conventional query optimization techniq...

Cloud computing has revolutionized the way data is stored, managed, and accessed. Multi-tenancy, a fundamental characteristic of cloud architecture, enables multiple users or tenants to share computing resources while maintaining data and operational isolation. However, this shared architecture introduces unique challenges, especially in the contex...

Query optimization has become a research area where classical algorithms are being challenged by machine learning algorithms. At the same time, recent trends in learned query optimizers have shown that it is prudent to take advantage of decades of database research and augment classical query optimizers by shrinking the plan search space through di...

Recent work in database query optimization has used complex machine learning strategies, such as customized reinforcement learning schemes. Surprisingly, we show that LLM embeddings of query text contain useful semantic information for query optimization. Specifically, we show that a simple binary classifier deciding between alternative query plans...

Join query optimization is a critical component of database management systems (DBMS), significantly influencing their performance and efficiency. This study delves into advanced optimization techniques by employing the Firefly Algorithm and its hybrid integrations with Deep Q-Network (DQN) and Double Deep Q-Network (DDQN) methodologies. Utilizing...

Query optimization is a well-studied problem in the database industry, with numerous solutions proposed over the last several decades. The success of deep reinforcement learning (DRL) has generated new opportunities in query optimization. One of the most difficult tasks in query optimization and query plan generation is determining the order in whi...

Currently, query and database optimization are very important for database management, especially in distributed systems where data is dispersed and stored across various physical locations. Increasing volume and data distribution makes it more difficult to create an optimal query plan. In cases where data relations are broken down into multiple ta...

Over the last decades, various cost-based optimizers have been proposed to generate optimal plans for SQL queries. These optimizers are key to achieving good performance in database systems and can speed up query execution. Still, they may need enormous expert efforts and perform poorly on complicated queries. Learning-based optimizers have been sh...

The principal component of conventional database query optimizers is a cost model that is used to estimate expected performance of query plans. The accuracy of the cost model has direct impact on the optimality of execution plans selected by the optimizer and thus, on the resulting query latency. Several common parameters of cost models in modern D...

Query optimization is crucial for every database management system (DBMS) to enable fast execution of declarative queries. Most DBMS designs include cost-based query optimization. However, MongoDB implements a different approach to choose an execution plan that we call "first past the post" (FPTP) query optimization. FPTP does not estimate costs fo...

Effective query optimization is essential for improving database management system performance. Using a variety of cutting-edge approaches, including Deep Q-Networks (DQN), Double Deep Q-Networks (DDQN), Genetic Algorithms (GA), and Hybrid DQN-GA and DDQN-GA, we present a thorough examination of join query optimization in this research article. In...

Join query optimization is a critical task in database management systems as it directly impacts the performance and efficiency of query execution. The goal of join query optimization is to determine an optimal execution plan for a given query, considering factors such as cost, resource utilization, and query response time. In recent years, metaheu...

The integration of Explainable AI (XAI) with distributed databases, enhanced by reinforcement learning (RL) and generative models, offers a novel approach to predictive scaling and query optimization. This study explores how these advanced AI techniques can address the challenges of managing and optimizing distributed database systems. Generative m...

The healthcare industry is increasingly relying on data-driven insights to enhance patient care, operational efficiency, and decision-making. Predictive analytics has emerged as a critical tool in optimizing healthcare queries by improving the retrieval, processing, and interpretation of complex medical data. This paper explores the role of predict...

The increasing adoption of the Internet of Things (IoT) in healthcare has led to an explosion of data generated by connected medical devices. This data must be processed efficiently to ensure timely and accurate decision-making in critical healthcare scenarios. Traditional database query optimization techniques struggle to keep pace with the dynami...

Healthcare analytics has witnessed significant advancements with the integration of big data, artificial intelligence, and machine learning. However, the challenge of optimizing query processing remains crucial for extracting meaningful insights efficiently. Semantic query optimization (SQO) emerges as a promising technique that enhances query perf...

The increasing digitization of healthcare has led to the widespread adoption of Electronic Health Records (EHR), which store vast amounts of patient data. However, querying these records efficiently remains a significant challenge due to the complexity and volume of the data. Traditional query optimization techniques often struggle to deliver real-...

The rapid growth of big data in healthcare has necessitated the development of efficient computing paradigms to handle data storage, processing, and analysis. Two prominent computing paradigms-fog computing and cloud computing-offer unique advantages and challenges for big data management. Cloud computing provides scalable, centralized data storage...

The tensor programming abstraction has become the key . This framework allows users to write high performance programs for bulk computation via a high-level imperative interface. Recent work has extended this paradigm to sparse tensors (i.e. tensors where most entries are not explicitly represented) with the use of sparse tensor compilers. These sy...

In the present day, Traditional database management methods are becoming more inadequate for effective data processing as the volume of data created by systems grows. Machine learning approaches have shown promise in optimizing database queries and enhancing database administration functions such as query optimization, workload management, indexing...

Annotation The results of the state assessment are presented … Abstract Query optimization is fundamental to the Productivity of information management systems (DBMS) affecting how queries are Round and Information is accessed. this report presents amp relative psychoanalysis of car acquisition (ml)-based question optimization methods and conventio...

With the ever-increasing volume and complexity of healthcare data, the efficiency of data retrieval mechanisms has become paramount. Healthcare information systems are inundated with electronic health records (EHRs), clinical notes, imaging data, lab reports, and patient histories. Traditional query optimization algorithms often fall short when app...

The rapid growth of the internet has ushered in an era of unprecedented information availability. However, this expansion has also given rise to the formidable challenge of efficiently extracting relevant data from the vast expanse of web pages. Conventional methods have often fallen short, inundating users with an abundance of irrelevant content....

This study investigates the effectiveness of advanced query optimization techniques in SQL databases, focusing on multi-level indexing, query rewriting, and dynamic query execution plans. The research employs a qualitative approach, gathering data from a variety of SQL databases characterized by large datasets typical of big data environments. Thro...

Data dependency-based query optimization techniques can considerably improve database system performance: we apply three such optimization techniques to five database management systems (DBMSs) and observe throughput improvements between 5 % and 33 %. We address two key challenges to achieve these results: (i) efficiently identifying and extracting...

The effectiveness of a cost-based query optimizer relies on the accuracy of selectivity estimates. The execution plan generated by the optimizer can be extremely poor in reality due to uncertainty in these estimates. This paper presents PARQO (Penalty-Aware Robust Query Optimization), a novel system where users can define powerful robustness metric...

Recently, numerous machine learning (ML) techniques have been applied to address database performance management problems, including cardinality estimation, cost modeling, optimal join order prediction, hint generation, etc. In this paper, we focus on query optimizer hints employed by users in their queries in order to mask some Query Optimizer def...

In distributed and cloud database system, large amount of resources are used by complex queries that increase in payment overheads of cloud users. These resource needs can be minimised by evaluating common join operations and caching their results so that they could be applied to the execution of additional queries. In this research, the Query Opti...

Several aspects of query processing in a database management system (DBMS) depend on the distribution of attribute values in the input relations to the query operators. For example, query optimizers choose the most efficient access plan for a query based on its result size which depends to the data distribution in the input relations. Thus, many...

Recent advancements in image segmentation have been notably driven by Vision Transformers. These transformer-based models offer one versatile network structure capable of handling a variety of segmentation tasks. Despite their effectiveness, the pursuit of enhanced capabilities often leads to more intricate architectures and greater computational d...

The advent of artificial intelligence (AI) in optimizing database queries marks a significant milestone in the realmof database management, promising to elevate performance and efficiency to unprecedented levels. Traditional queryoptimization techniques, while effective to a certain extent, struggle to keep pace with the complexities and dynamic na...

Elasticsearch is a widely used distributed search engine, powering applications in enterprise search, e-commerce, security analytics, and knowledge retrieval. As datasets grow, ensuring efficient query execution, accurate ranking, and system scalability becomes a critical challenge. This paper presents a comparative analysis of ranking strategies w...

Query optimization is one of the key factors affecting the performance of database systems that aim to enact the query execution plan with minimum cost. Particularly in distributed database systems, due to the multiple copies of the data that are stored in different data nodes, resulting in the dramatic increase in the feasible query execution plan...

The problem of query optimization in object-oriented databases is addressed. We follow the stack-based approach to query languages, which employs the naming-scoping-binding paradigm of programming languages rather than traditional database concepts such as relational/object algebras or calculi. The classical environment stack is a semantic basis fo...

Large-scale graph processing and Stream processing are two distinct computational paradigms for big data processing. Graph processing deals with computation on graphs of billions of vertices and edges. However, large-scale graph processing frameworks mostly work on graphs that do not change over time, while on the other end of the spectrum, stream...

Query optimization is a cornerstone of efficient database management, crucial for maintaining performance as databases scale in size and complexity. Traditional query optimization techniques, while effective, often rely on static rules and cost-based methods that struggle with dynamic workloads and diverse query patterns. Machine learning (ML) offe...

In today's rapidly evolving business environment, the optimization of query performance in distributed databases is paramount for ensuring resilient and efficient supply chains. As organizations increasingly rely on real-time data to inform their decision-making processes, the ability to swiftly and accurately retrieve relevant information from dis...

Join query optimization aims to find the best join order for tables in a query, which is critical for query processing performance. Recently, reinforcement learning models have been proposed to solve the challenges existing with query processing and join query optimization. However, changes in the data distribution can turn the trained reinforcemen...

Database indexing plays a crucial role in optimizing query performance, particularly in cloud-native and high-performance computing environments. Traditional indexing techniques often struggle to adapt dynamically to varying workloads, leading to suboptimal query execution times and increased computational overhead. This paper presents an AI-augmen...

The nearest neighbors query problem on road networks constitutes a crucial aspect of location-oriented services and has useful practical implications; e.g., it can locate the k-nearest hotels. However, researches who study road networks still encounter obstacles due to the method’s inherent limitations with respect to object mobility. More popular...

The exponential growth of data in modern digital environments has led to the development of sophisticated big data systems capable of storing, processing, and analyzing vast datasets. Query optimization in these systems is critical for ensuring efficient data retrieval and minimizing latency in analytical tasks. Traditional rule-based and cost-base...

Big data has transformed industries by enabling large-scale data storage and analysis, but it also presents challenges in query optimization. Efficient query execution is essential for real-time decision-making, data processing, and analytics. Traditional query optimization techniques struggle to cope with the increasing complexity and scale of mod...

Semantic query optimization (SQO) represents a significant evolution in database technology by transforming user queries into more efficient forms using semantic knowledge of the database. As the paradigm shifts from traditional relational databases to NoSQL and NewSQL systems, there is a pressing need to understand how SQO can be adapted and utili...

Today, due to the rapid development of technology and at the same time, the rapid growth of information, and also, considering that in current computing programs, storing information, processing and maintaining results has become one of the daily and challenging tasks of database managers, discussions such as parallelization and simultaneous execut...

The current boom of learned query optimizers (LQO) can be explained not only by the general continuous improvement of deep learning (DL) methods but also by the straightforward formulation of a query optimization problem (QOP) as a machine learning (ML) one. The idea is often to replace dynamic programming approaches, widespread for solving QOP, wi...

In the era of digital transformation, optimizing cloud computing performance has become a critical focus for organizations striving to leverage the full potential of cloud infrastructures. This study presents a comparative analysis of advanced database management system (DBMS) techniques aimed at enhancing cloud computing performance. By examining...

The traditional and well-established cost-based query optimizer approach enumerates different execution plans for each query, assesses each plan with costs, and selects the plan that promises the lowest costs for execution. However, the optimal execution plan is not always selected. To steer the optimizer in the right direction, many query optimize...

Query optimization is a critical aspect of database systems as it helps to reduce query execution time and improve system performance. In this study, Probabilistic object models to get the specific facts from available statistics and efficient query optimization. Query optimization is a technique that considers potential query plans based on lineag...

In the rapidly evolving fintech domain, the need for high-throughput, low-latency, real-time reporting systems has increased due to rising transaction volumes. This paper examines the architectural design and technical implementation of fintech reporting systems integrating Java Persistence API (JPA) with embedded SQL, ensuring data accuracy, effic...

In recent years, with the increasing complexity of user demands, there is a growing need for hybrid multi-modal queries with both structured and unstructured constraints. A lot of constrained hybrid query methods were proposed to retrieval objects with similar features to the query while satisfying given structured attribute constraints. However, d...

The time necessary for data processing is be-coming shorter and shorter nowadays. This thesis presents a definition of the active data warehousing (ADW) paradigm. One of the data warehouses which is consistent with this paradigm is teradata warehouse. Therefore, the basic ele-ments of the teradata architecture are described, such as processors pars...

Most existing parametric query optimization (PQO) techniques rely on traditional query optimizer cost models, which are often inaccurate and result in suboptimal query performance. We propose Kepler, an end-to-end learning-based approach to PQO that demonstrates significant speedups in query latency over a traditional query optimizer. Central to ou...

Although the many efforts to apply deep reinforcement learning to query optimization in recent years, there remains room for improvement as query optimizers are complex entities that require hand-designed tuning of workloads and datasets. Recent research present learned query optimizations results mostly in bulks of single workloads which focus on...

In this study, we propose three k-nearest neighbor (k-NN) optimization techniques for a distributed, in-memory-based, high-dimensional indexing method to speed up content-based image retrieval. The proposed techniques perform distributed, in-memory, high-dimensional indexing-based k-NN query optimization: a density-based optimization technique that...

Queries in a database application system, have a great influence on the success of application development. Frequently used query forms include subselect and join. Choosing the right query form will affect the performance of the database application. Database applications are sometimes slow to respond because of slowness in their databases. The sel...

Parallel processing and query optimization are two crucial techniques used in the field of database management. Parallel processing involves dividing a large task into smaller sub-tasks executed simultaneously on multiple processors, while query optimization aims to improve the performance of a database system by finding the most efficient way to e...

A recent line of works apply machine learning techniques to assist or rebuild cost based query optimizers in DBMS. While exhibiting superiority in some benchmarks, their deficiencies, e.g., unstable performance, high training cost, and slow model updating, stem from the inherent hardness of predicting the cost or latency of execution plans using ma...

Stream processing engines (SPEs) are widely used for large scale streaming analytics over unbounded time-ordered data streams. Modern day streaming analytics applications exhibit diverse compute characteristics and demand strict latency and throughput requirements. Over the years, there has been significant attention in building hardware-efficient...

Quantum computing promises to solve difficult optimization problems in chemistry, physics and mathematics more efficiently than classical computers. However, it requires fault-tolerant quantum computers with millions of qubits; a technological challenge still not mastered by engineers. To lower the barrier, hybrid algorithms combining classical and...

According to the characteristics of massive, multi-source, heterogeneous, and rapid growth of book literature data information from the perspective of the metaverse, in order to meet the requirements of efficient management and rapid retrieval such as standardized storage, effective extraction, and scientific library construction for unstructured m...

In this paper, the input user Structured Query Language (SQL) query is converted into an optimized SQL query using a hybrid algorithm. The main aim is to reduce query execution time using PHP language and oracle database. These performance has been evaluated using different performance metrics: Cost of individuals, Query execution time. The hybrid...

Cross-layer query optimization in distributed databases has emerged as a critical challenge in big data environments, especially as enterprises require efficient, low-latency data retrieval across disparate, distributed sources. Traditional approaches to query optimization focus on individual layers-such as storage, processing, or network layers-re...

Query optimization is a crucial process across data mining and big data analytics. As the size of the data in the modern applications is increasing due to various sources, types and multi-modal records across databases, there is an urge to optimize lookup and search operations. Therefore, indexes can be utilized to solve the matter of rapid data gr...

It is significant to minimize suboptimal query execution in petabyte scale distributed database systems. This research examines a range of techniques for the improvement of query optimization, namely cost-based optimization, execution in partitioned environment, predicate push-down optimization, dynamic resource management, utilization of data loca...

Placing the processing power near the data, rather than shipping the data to the processor is inevitable and demanding in the era of Big Data. With a given near-data processor, it is important to use it properly to reduce the massive data transfer between data sources and computing nodes. In database query processing, as many operations as possible...

In this paper, we have proposed the intimate environment of multiblockchain optimization algorithm using big data inquiry to mend the effectiveness of association query handling among numerous multihoming blockchains by implementing the big data system. This technique adds semantic evidences to the old-styled multiblockchain prototype and construct...

The query optimizer uses cost-based optimization to create an execution plan with the least cost, which also consumes the least amount of resources. The challenge of query optimization for relational database systems is a combinatorial optimization problem, which renders exhaustive search impossible as query sizes rise. Increases in CPU performance...

Autonomous databases are the new fad in modern database systems. The database systems are managed by machine learning and neural networks for query prediction, self-tuning, and self-indexing. These systems decrease intervention in multi-relational database management systems (multi-RDBMS). This paper analyses the relevance of ML and NN in optimizin...