No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Candidate Solutions for Defining Explainability Requirements of AI Systems
Abstract
[Context and Motivation] Many recent studies highlight explainability as an important requirement that supports in building transparent, trustworthy, and responsible AI systems. As a result, there is an increasing number of solutions that researchers have developed to assist in the definition of explainability requirements. [Question] We conducted a literature study to analyze what kind of candidate solutions are proposed for defining the explainability requirements of AI systems. The focus of this literature review is especially on the field of requirements engineering (RE). [Results] The proposed solutions for defining explainability requirements such as approaches, frameworks, and models are comprehensive. They can be used not only for RE activities but also for testing and evaluating the explainability of AI systems. In addition to the comprehensive solutions, we identified 30 practices that support the development of explainable AI systems. The literature study also revealed that most of the proposed solutions have not been evaluated in real projects, and there is a need for empirical studies. [Contribution] For researchers, the study provides an overview of the candidate solutions and describes research gaps. For practitioners, the paper summarizes potential practices that can help them define and evaluate the explainability requirements of AI systems.
... Despite the increasing interest and number of studies on this topic within the RE and SE literature, there are no empirical case studies on eliciting explainability requirements [1]. This is echoed in Balasubramaniam et al.'s [4] recent literature review in RE where they found 30 practices that support the development of explainability requirements: "most proposed solutions have not been evaluated in real projects, highlighting the need for empirical studies." We have not been able to identify any empirical casestudies as of January 2025. ...
... Researchers are actively engaged in discussions regarding the nature of explainability requirements and how to define them [9,4,6]. To gain a comprehensive understanding of explainability in industrial settings, detailed accounts from real-world cases are essential. ...
This paper analyses how software developers elicit explainability requirements when creating a software application with an AI component, through a case study using AI in the medical context of predicting cerebral palsy (CP) risk in infants. Following a small software development team at a Norwegian hospital, we observe their process of simultaneously developing the AI application and discovering what explanations clinicians require from the AI predictions. Since clinicians struggled to articulate their explainability needs before interacting with the system, an iterative approach proved effective: the team started with minimal explanations and refined these based on clinicians' responses during real patient examinations. Our preliminary findings from the first two iterations show that clinicians valued "interrogative explanations" - i.e., tools that let them explore and compare the AI predictions with their own assessments - over detailed technical explanations of the AI model's inner workings. Based on our analysis, we suggest that successful explainability requirements emerge through iterative collaboration between developers and users rather than being fully specified upfront. To the best of our knowledge, this is the first empirical case study on eliciting explainability requirements in software engineering.
Systems that rely on Machine Learning (ML systems) have differing demands on quality—known as non-functional requirements (NFRs)—from traditional systems. NFRs for ML systems may differ in their definition, measurement, scope, and comparative importance. Despite the importance of NFRs in ensuring the quality ML systems, our understanding of all of these aspects is lacking compared to our understanding of NFRs in traditional domains. We have conducted interviews and a survey to understand how NFRs for ML systems are perceived among practitioners from both industry and academia. We have identified the degree of importance that practitioners place on different NFRs, including cases where practitioners are in agreement or have differences of opinion. We explore how NFRs are defined and measured over different aspects of a ML system (i.e., model, data, or whole system). We also identify challenges associated with NFR definition and measurement. Finally, we explore differences in perspective between practitioners in industry, academia, or a blended context. This knowledge illustrates how NFRs for ML systems are treated in current practice, and helps to guide future RE for ML efforts.
The growing complexity of software systems and the influence of software-supported decisions in our society sparked the need for software that is transparent, accountable, and trustworthy. Explainability has been identified as a means to achieve these qualities. It is recognized as an emerging non-functional requirement (NFR) that has a significant impact on system quality. Accordingly, software engineers need means to assist them in incorporating this NFR into systems. This requires an early analysis of the benefits and possible design issues that arise from interrelationships between different quality aspects. However, explainability is currently under-researched in the domain of requirements engineering, and there is a lack of artifacts that support the requirements engineering process and system design. In this work, we remedy this deficit by proposing four artifacts: a definition of explainability, a conceptual model, a knowledge catalogue, and a reference model for explainable systems. These artifacts should support software and requirements engineers in understanding the definition of explainability and how it interacts with other quality aspects. Besides that, they may be considered a starting point to provide practical value in the refinement of explainability from high-level requirements to concrete design choices, as well as on the identification of methods and metrics for the evaluation of the implemented requirements.
Background
A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which was later complemented with snowball searches. Our conjecture is that we can perform even better searches if combining these two search approaches, referred to as a hybrid search strategy.
Objective
Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared four alternative hybrid search strategies to assess whether we could identify more cost–efficient ways of searching for relevant primary studies.
Methods
To compare and evaluate the hybrid search strategy, we replicated the search procedure in a systematic literature review (SLR) on industry–academia collaboration in software engineering. The SLR used a more “traditional” approach to searching for relevant articles for an SLR, while our replication was executed using a hybrid search strategy.
Results
In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30% more primary studies and even more studies when focusing only on peer–reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when performing systematic literature studies.
Conclusions
The hybrid search strategy is a strong contender for being used when performing systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.
As the capabilities of artificial intelligence (AI) systems constantly grow, so too does their complexity. The explainability toward their users is gaining attention, becoming a requirement that these systems should satisfy. We articulate user requirements for explainable AI systems.
Background
The literature concerning research methodologies and methods has increased in software engineering in the last decade. However, there is limited guidance on selecting an appropriate research methodology for a given research study or project.
Objective
Based on a selection of research methodologies suitable for software engineering research in collaboration between industry and academia, we present, discuss and compare the methodologies aiming to provide guidance on which research methodology to choose in a given situation to ensure successful industry–academia collaboration in research.
Method
Three research methodologies were chosen for two main reasons. Design Science and Action Research were selected for their usage in software engineering. We also chose a model emanating from software engineering, i.e., the Technology Transfer Model. An overview of each methodology is provided. It is followed by a discussion and an illustration concerning their use in industry–academia collaborative research. The three methodologies are then compared using a set of criteria as a basis for our guidance.
Results
The discussion and comparison of the three research methodologies revealed general similarities and distinct differences. All three research methodologies are easily mapped to the general research process describe–solve–practice, while the main driver behind the formulation of the research methodologies is different. Thus, we guide in selecting a research methodology given the primary research objective for a given research study or project in collaboration between industry and academia.
Conclusions
We observe that the three research methodologies have different main objectives and differ in some characteristics, although still having a lot in common. We conclude that it is vital to make an informed decision concerning which research methodology to use. The presentation and comparison aim to guide selecting an appropriate research methodology when conducting research in collaboration between industry and academia.
Software systems are becoming increasingly complex. Their ubiquitous presence makes users more dependent on their correctness in many aspects of daily life. As a result, there is a growing need to make software systems and their decisions more comprehensible, with more transparency in software-based decision making. Transparency is therefore becoming increasingly important as a non-functional requirement. However, the abstract quality aspect of transparency needs to be better understood and related to mechanisms that can foster it. The integration of explanations into software has often been discussed as a solution to mitigate system opacity. Yet, an important first step is to understand user requirements in terms of explainable software behavior: Are users really interested in software transparency and are explanations considered an appropriate way to achieve it? We conducted a survey with 107 end users to assess their opinion on the current level of transparency in software systems and what they consider to be the main advantages and disadvantages of embedded explanations. We assess the relationship between explanations and transparency and analyze its potential impact on software quality. As explainability has become an important issue, researchers and professionals have been discussing how to deal with it in practice. While there are differences of opinion on the need for built-in explanations, understanding this concept and its impact on software is a key step for requirements engineering. Based on our research results and on the study of existing literature, we offer recommendations for the elicitation and analysis of explainability and discuss strategies for the practice.
In the last few years, Artificial Intelligence (AI) has achieved a notable momentum that, if harnessed appropriately, may deliver the best of expectations over many application sectors across the field. For this to occur shortly in Machine Learning, the entire community stands in front of the barrier of explainability, an inherent problem of the latest techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI (namely, expert systems and rule based models). Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is widely acknowledged as a crucial feature for the practical deployment of AI models. The overview presented in this article examines the existing literature and contributions already done in the field of XAI, including a prospect toward what is yet to be reached. For this purpose we summarize previous efforts made to define explainability in Machine Learning, establishing a novel definition of explainable Machine Learning that covers such prior conceptual propositions with a major focus on the audience for which the explainability is sought. Departing from this definition, we propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at explaining Deep Learning methods for which a second dedicated taxonomy is built and examined in detail. This critical literature analysis serves as the motivating background for a series of challenges faced by XAI, such as the interesting crossroads of data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence , namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to the field of XAI with a thorough taxonomy that can serve as reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.
Systematic literature studies have received much attention in empirical software engineering in recent years. They have become a powerful tool to collect and structure reported knowledge in a systematic and reproducible way. We distinguish systematic literature reviews to systematically analyze reported evidence in depth, and systematic mapping studies to structure a field of interest in a broader, usually quantified manner. Due to the rapidly increasing body of knowledge in software engineering, researchers who want to capture the published work in a domain often face an extensive amount of publications, which need to be screened, rated for relevance, classified, and eventually analyzed. Although there are several guidelines to conduct literature studies, they do not yet help researchers coping with the specific difficulties encountered in the practical application of these guidelines. In this article, we present an experience-based guideline to aid researchers in designing systematic literature studies with special emphasis on the data collection and selection procedures. Our guideline aims at providing a blueprint for a practical and pragmatic path through the plethora of currently available practices and deliverables capturing the dependencies among the single steps. The guideline emerges from various mapping studies and literature reviews conducted by the authors and provides recommendations for the general study design, data collection, and study selection procedures. Finally, we share our experiences and lessons learned in applying the different practices of the proposed guideline.
This paper aims at identifying critical factors affecting organization-wide implementation of requirements engineering (RE) processes. The paper is based on a broad literature review and three longitudinal case studies that were carried out using an action research method. The results indicate that RE process implementation is a demanding undertaking, and its success greatly depends on such human factors as motivation, commitment and enthusiasm. Therefore, it is essential that the RE process is useful for its individual users. Furthermore, the results indicate that organizations can gain benefits from RE by defining a simple RE process, by focusing on a small set of RE practices, and by supporting the systematic usage of these practices.
Unlike traditional software development methods, agile methods are marked by extensive collaboration, i.e. face-to-face communication. Although claimed to be beneficial, the software development community as a whole is still unfamiliar with the role of the requirements engineering practices in agile methods. The term “agile requirements engineering” is used to define the “agile way” of planning, executing and reasoning about requirements engineering activities. Moreover, not much is known about the challenges posed by collaboration-oriented agile way of dealing with requirements engineering activities. Our goal is to map the evidence available about requirements engineering practices adopted and challenges faced by agile teams in order to understand how traditional requirements engineering issues are resolved using agile requirements engineering. We conducted a systematic review of literature published between 2002 and June 2013 and identified 21 papers, that discuss agile requirements engineering. We formulated and applied specific inclusion and exclusion criteria in two distinct rounds to determine the most relevant studies for our research goal. The review identified 17 practices of agile requirements engineering, five challenges traceable to traditional requirements engineering that were overcome by agile requirements engineering, and eight challenges posed by the practice of agile requirements engineering. However, our findings suggest that agile requirements engineering as a research context needs additional attention and more empirical results are required to better understand the impact of agile requirements engineering practices e.g. dealing with non-functional requirements and self-organising teams.
This paper presents an overview of the field of software systems requirements engineering (RE). It describes the main areas of RE practice, and highlights some key open research issues for the future. 1 Introduction The primary measure of success of a software system is the degree to which it meets the purpose for which it was intended. Broadly speaking, software systems requirements engineering (RE) is the process of discovering that purpose, by identifying stakeholders and their needs, and documenting these in a form that is amenable to analysis, communication, and subsequent implementation. There are a number of inherent difficulties in this process. Stakeholders (including paying customers, users and developers) may be numerous and distributed. Their goals may vary and conflict, depending on their perspectives of the environment in which they work and the tasks they wish to accomplish. Their goals may not be explicit or may be difficult to articulate, and, inevitably, satisfactio...
Recently reported issues concerning the acceptance of artificial intelligence (AI) solutions after deployment stress the importance of requirements engineering (RE) for designing and delivering responsible AI systems. In this article, we argue that RE should not only be carefully conducted but also tailored for responsible AI.
The anticipated presence of highly automated vehicles and intelligent infrastructure systems in urban traffic will yield new types of demand-driven mobility and value-added services. To provide these services, future transportation systems will consist of large-scale, cooperative ensembles of highly automated and connected systems that operate in mixed traffic and have to interact with human drivers and vulnerable road users while coordinately ensuring traffic efficiency and safety. We posit that the ability to explain processes and decisions is essential for such systems. Adequately addressing the needs of the involved actors will require that explainability and trustworthiness are handled as core properties in the development of highly automated systems. To support explainability-driven design approaches, we identify a set of explainability challenges in the context of a large-scale ongoing endeavor on cooperative transportation and present approaches to target these challenges.
This work focuses on the context of software explainability, which is the production of software capable of explaining to users the dynamics that govern its internal functioning. User models that include information about their requirements and their perceptions of explainability are fundamental when building software with such capability. This study investigates the process of creating personas that include information about users’ explainability perceptions and needs. The proposed approach is based on data collection with questionnaires, modeling of empathy maps, grouping the maps, generating personas from them and evaluation employing the Persona Perception Scale method. In an empirical study, personas are created from 61 users’ response data to a questionnaire. The generated personas are evaluated by 60 users and 38 designers considering attributes of the Persona Perception Scale method. The results include a set of 5 distinct personas that users rate as representative of them at an average level of 3.7 out of 5, and designers rate as having quality 3.5 out of 5. The median rate is 4 out of 5 in the majority of criteria judged by users and designers. Both the personas and their creation and evaluation approach are contributions of this study to the design of software that satisfies the explainability requirement.
Artificial intelligence (AI) has become a fast-growing trend. Increasingly, organizations are interested in developing AI systems, but many of them have realized that the use of AI technologies can raise ethical questions. The goal of this study was to analyze what kind of ethical guidelines companies have for solving potential ethical issues of AI and developing AI systems. This paper presents the results of the case study conducted in three companies. The ethical guidelines defined by the case companies focused on solving potential ethical issues, such as accountability, explainability, fairness, privacy, and transparency. To analyze different viewpoints on critical ethical issues, two of the companies recommended using multi-disciplinary development teams. The companies also considered defining the purposes of their AI systems and analyzing their impacts to be important practices. Based on the results of the study, we suggest that organizations develop and use ethical guidelines to prioritize critical quality requirements of AI. The results also indicate that transparency, explainability, fairness, and privacy can be critical quality requirements of AI systems.
Recent research efforts strive to aid in designing explainable systems. Nevertheless, a systematic and overarching approach to ensure explainability by design is still missing. Often it is not even clear what precisely is meant when demanding explainability. To address this challenge, we investigate the elicitation, specification, and verification of explainablity as a Non-Functional Requirement (NFR) with the long-term vision of establishing a standardized certification process for the explainability of software-driven systems in tandem with appropriate development techniques. In this work, we carve out different notions of explainability and high-level requirements people have in mind when demanding explainability, and sketch how explainability concerns may be approached in a hypothetical hiring scenario. We provide a conceptual analysis which unifies the different notions of explainability and the corresponding explainability demands.
This paper aims at identifying critical factors affecting organization-wide implementation of requirements engineering (RE) processes. The paper is based on a broad literature review and three longitudinal case studies that were carried out using an action research method. The results indicate that RE process implementation is a demanding undertaking, and its success greatly depends on such human factors as motivation, commitment and enthusiasm. Therefore, it is essential that the RE process is useful for its individual users. Furthermore, the results indicate that organizations can gain benefits from RE by defining a simple RE process, by focusing on a small set of RE practices, and by supporting the systematic usage of these practices.
One of the main goals of an applied research field such as requirements engineering is the transfer of research results to
industrial use. To promote industrial adoption of technologies developed in academia, researchers need to provide tangible
evidence of the advantages of using them. This can be done through industry validation, enabling researchers to test and validate
technologies in a real setting with real users and applications. The evidence obtained, together with detailed information
on how the validation was conducted, offers rich decision support material for industrial practitioners seeking to adopt new
technologies. This paper presents a comprehensive systematic literature review of all papers published in the Requirements
Engineering journal containing any type of technology evaluation. The aim is to gauge the support for technology transfer,
i.e., to what degree industrial practitioners can use the reporting of technology evaluations in the journal as decision support
for adopting the technologies in industrial practice. Findings show that very few evaluations offer full technology transfer
support, i.e., have a realistic scale, application or subjects. The major improvement potential concerning support for technology
transfer is found to be the subjects used in the evaluations. Attaining company support, including support for using practitioners
as subjects, is vital for technology transfer and for researchers seeking to validate technologies.
Four principles of explainable artificial intelligence. National Institute of Standards and Technology
- P J Phillips
Phillips, P.J., et al.: Four principles of explainable artificial intelligence. National Institute of
Standards and Technology (U.S.), NIST IR 8312 (2021)