No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Generative AI for Reengineering Variants into Software Product Lines: An Experience Report
... LLMs have revolutionized the fields of natural language processing by mimicking human-like language understanding and generation capabilities [19,22]. These advancements have extended the applications of LLMs into software engineering, notably in automating tasks such as code generation, documentation, and SPL engineering [1,16]. ...
... The steps followed are: (1) Provide ChatGPT with a set of objective functions and ask it to identify and explain correlations; (2) Use a predefined prompt format to guide the model's responses; and (3) Compare ChatGPT's output with the known correlations from the dataset. As a result, ChatGPT should correctly identify significant correlations between (1) (2) ...
... Attempts to integrate machine learning into PLA optimization are familiar but have gained momentum with the advent of more sophisticated models. The work of Acher et al [1] adopts Generative languages for reengineering variants into SPLs. Many studies have explored using SBSE to optimize different SPL tasks [12]. ...
Search-based design of Product Line Architecture (PLA) focuses on enhancing the design and functionality of software product lines through variability management, reuse, and optimization. A particular challenge in this area is the selection of objective functions, which significantly influence the success of the search process. Moreover, many objectives make the analysis and choice of a solution to be used harder. The literature has assigned this task to the PLA designer, i.e., the Decision-Maker (DM), who does not always know all the functions and their impact on the optimization outcomes. On the other hand, recent research shows that Large Language Models (LLMs), particularly the Generative Pre-trained Transformer series (GPT), have obtained promising results to help in various Software Engineering (SE) tasks. Considering this fact, this work explores the integration of such LLMs, notably ChatGPT, into the search-based PLA design. By leveraging LLMs’ capacity to understand/generate human-like text, we investigate their potential to assist DMs and propose an approach for suggesting objective functions, thereby simplifying and improving decision-making in PLA design optimization. Through empirical tests and qualitative feedback from domain experts, this research highlights the application of LLMs in search-based SE. The results demonstrate that integrating ChatGPT into PLA design can significantly enhance decision-making efficiency and solution quality, with a 40% reduction in time required for selecting objective functions and a 25% improvement in solution quality from the DM’s point of view. This study maps out the challenges and opportunities that lie ahead in fully harnessing their potential for PLA search-based design.
... In these cases, they would just accept the code initially and may later on reject it if discovered to be incorrect [68]. (4) No Review at all: In this case, users skipped review of the LLM response all together [42]. It was observed in a study with experts that this rarely happens [36]. ...
... We discovered a lot of variance in the accuracy of the final response. There were significant gains observed when LLMs were used for a wide variety of tasks from basic programming tasks [4,27,36,41,58,80,81], algorithmic tasks [19,50,95], UML modeling [92], software engineering [3][4][5] and Data analysis [58]. Even more gains were observed for specialized tools and frameworks that helped users with reviewing the LLM response (more specifically code review) [26,31,43,103]. ...
... We discovered a lot of variance in the accuracy of the final response. There were significant gains observed when LLMs were used for a wide variety of tasks from basic programming tasks [4,27,36,41,58,80,81], algorithmic tasks [19,50,95], UML modeling [92], software engineering [3][4][5] and Data analysis [58]. Even more gains were observed for specialized tools and frameworks that helped users with reviewing the LLM response (more specifically code review) [26,31,43,103]. ...
Large Language Models (LLMs) are transforming programming practices, offering significant capabilities for code generation activities. While researchers have explored the potential of LLMs in various domains, this paper focuses on their use in programming tasks, drawing insights from user studies that assess the impact of LLMs on programming tasks. We first examined the user interaction behaviors with LLMs observed in these studies, from the types of requests made to task completion strategies. Additionally, our analysis reveals both benefits and weaknesses of LLMs showing mixed effects on the human and task. Lastly, we looked into what factors from the human, LLM or the interaction of both, affect the human's enhancement as well as the task performance. Our findings highlight the variability in human-LLM interactions due to the non-deterministic nature of both parties (humans and LLMs), underscoring the need for a deeper understanding of these interaction patterns. We conclude by providing some practical suggestions for researchers as well as programmers.
... Considering the maintenance part, MBE approaches support consistency checking [36], repairing [37], and overall quality assurance [48]. For the evolution part, approaches target co-evolution [25], software modernization [46], reengineering [3], among others. MBE for maintenance and evolution (MBM&E), however, has received less attention from both research and practice in comparison to programming or testing [33]. ...
... Despite this, there is potential to use GenAI for MBM&E as a means to overcome the challenges and low adoption of the field [9]. This is evidenced by initial efforts from researchers to adopt GenAI techniques to MBM&E [3,10,34]. ...
... The use of AI for model-based techniques is not novel, been explored in the past [15,31,32,50]. The use of GenAI for MBE, however, is still in its early stages, with promising results [2,3,11,19,21,34,41]. That is the reason why we envision that in the next years, the application of GenAI for MBE will only grow, which leads to the possibilities of addressing the challenges of MBM&E. ...
Model-Based Engineering (MBE) has streamlined software development by focusing on abstraction and automation. The adoption of MBE in Maintenance and Evolution (MBM&E), however, is still limited due to poor tool support and a lack of perceived benefits. We argue that Generative Artificial Intelligence (GenAI) can be used as a means to address the limitations of MBM&E. In this sense, we argue that GenAI, driven by Foundation Models, offers promising potential for enhancing MBM&E tasks. With this possibility in mind, we introduce a research vision that contains a classification scheme for GenAI approaches in MBM&E considering two main aspects: (i) the level of augmentation provided by GenAI and (ii) the experience of the engineers involved. We propose that GenAI can be used in MBM&E for: reducing engineers' learning curve, maximizing efficiency with recommendations, or serving as a reasoning tool to understand domain problems. Furthermore, we outline challenges in this field as a research agenda to drive scientific and practical future solutions. With this proposed vision, we aim to bridge the gap between GenAI and MBM&E, presenting a structured and sophisticated way for advancing MBM&E practices.
... Recent research explores the potential of GAI to migrate products into a product line [2], implement 150 % representations [1], and generate meaningful feature configurations [18]. While single files can be handled easily, all explored LLMs are limited by the size of the input and output they can process. ...
... Furthermore, LLMs may produce relatively good results for source code tasks in programming languages they were pre-trained on, but they currently suffer from their non-deterministic nature and may not be applicable for programming languages used in legacy code or specific DSLs [17,41]. The required significant level of human guidance [1,2] contradicts the promised boost in automation and autonomy. ...
... Two studies examine the capabilities of ChatGPT, as one representative of an GAI-assistant to migrate variants to a product line [2] and implement platform source code [1]. While the authors thoroughly document the used prompts and results, they cannot automatically verify the results, but must manually check them. ...
Generative Artificial Intelligence (GAI) promises groundbreaking automation technology - a potential which may raise the management of variability-intensive software systems to a new level of automation. Several activities in maintaining variability-intensive software systems, such as extracting feature traces to updating features consistently, are repetitive and performed mainly manually or semi-automatically. Exploiting the potentials of GAI in maintaining variability-intensive software systems opens a fundamentally new research perspective, where GAI shall solve repetitive and hard-to-automate tasks. In this vision paper, we propose and discuss increasing levels of maintaining variability-intensive software systems automatically enabled through the support of GAI. We sketch actions necessary to reach the next levels of automation while discussing the current state-of-the-art.
... E.g. ChatGPT was used to synthesize SPL on the basis of a set of variants in (Acher and Martinez, 2023). In this latter paper, different types of system variants are considered: Java, UML, GraphML, state charts, and PNG. ...
... When asking LLM to identify design options that group roles and features, the design options are a way to annotate the user-stories, which can be considered as a part of the product line to a certain extent. As suggested in the discussion in (Acher and Martinez, 2023), our proposal combines the use of LLM with a deterministic approach. ...
A widely used Agile practice for requirements is to produce a set of user stories (also called ``agile product backlog''), which roughly includes a list of pairs (role, feature), where the role handles the feature for a certain purpose. In the context of Software Product Lines, the requirements for a family of similar systems is thus a family of user-story sets, one per system, leading to a 3-dimensional dataset composed of sets of triples (system, role, feature). In this paper, we combine Triadic Concept Analysis (TCA) and Large Language Model (LLM) prompting to suggest the user-story set required to develop a new system relying on the variability logic of an existing system family. This process consists in 1) computing 3-dimensional variability expressed as a set of TCA implications, 2) providing the designer with intelligible design options, 3) capturing the designer's selection of options, 4) proposing a first user-story set corresponding to this selection, 5) consolidating its validity according to the implications identified in step 1, while completing it if necessary, and 6) leveraging LLM to have a more comprehensive website. This process is evaluated with a dataset comprising the user-story sets of 67 similar-purpose websites.
... Initial reports regarding automating the development of software systems using LLMs confirm this statement. Acher and Martinez [40] experimented with synthesizing variants into SPLs using GPT-4 [41]. They tried using GPT-4 on five different cases with mixed results. ...
API-Based Generators (ABGs) allow practitioners to achieve the advantages of Model-Driven Engineering (MDE) without making significant changes to their current workflow or the architecture of their solution. However, using ABGs that are defined in terms of the syntax rules of a target language can be cumbersome and impractical. Additionally, manually developing an ABG based on higher-level concepts already used in an existing solution is labor-intensive and time-consuming. This paper introduces a new approach called Semi-automatic API-based Generators Development (SAGED) to expedite the creation of ABGs customized for MDE development of existing solutions. SAGED accomplishes this by (i) providing insight into code idioms that could be considered good candidates for code generation, as they frequently appear in the codebase, and (ii) automating the creation of a code generation API for an ABG, defined in terms of the identified code idioms. The SAGED approach relies on mining code idioms from existing, unlabeled source code based on a machine learning technique called the nonparametric Bayesian Probabilistic Tree Substitution Grammar (PTSG). The main contributions of this paper include the introduction of the SAGED approach, an explanation and optimization of the Type-Based MCMC as a method for approximating the nonparametric Bayesian PTSG, and the development of an open-source inference core for implementing the inference method in different programming languages. Furthermore, the paper presents a solution for implementing SAGED in the C# programming language, along with case studies that demonstrate its effectiveness.
... The autogeneration of high-quality SQL statements is enhancing the efficiency and accuracy of database query formulation (Troy et al., 2023) while GAN and Transformer Language Models are advancing cyber threat-hunting by combining data analysis with automatic report generation (Ferrag et al., 2023). In software engineering LLMs offer support to developers for reengineering software variants into Software Product Lines (Acher and Martinez, 2023). Additionally, in the development of interactive systems, GenAI expedites various stages of the lifecycle, from ideation to testing, by generating rapid prototypes and solutions (Schmidt, 2023). ...
Purpose
This study aims to propose a solution for guiding the configuration of complex product lines by addressing challenges such as high variability, customization issues and decision-making complexity.
Design/methodology/approach
A design science approach was adopted to develop and evaluate ConfiLog, a goal-driven method leveraging process mining techniques. ConfiLog uses event logs to uncover configuration process models, interprets them using a Goal-Question-Metric framework and generates guidance via a recommendation engine. Its effectiveness was tested through controlled experiments involving 226 students configuring a bike from a product line comprising over a billion possible configurations.
Findings
ConfiLog reduced the average time of the configuration process by 81%, achieving a 100% completion rate with customized recommendations. Additionally, ConfiLog demonstrated scalability, maintaining execution times under one second for models with up to 3,000 variants.
Research limitations/implications
The experiments were conducted with students in controlled settings, offering initial insights but limiting applicability to real-world industrial contexts, where dynamic stakeholder goals and diverse user requirements remain unexplored.
Practical implications
ConfiLog enhances customer experience, competitiveness and retention in e-commerce, particularly in high-variability sectors such as automotive, fashion, apparel and electronics, where personalization is crucial. It is also well-suited for other variability contexts, including ERP integration and COTS management. Beyond these applications, ConfiLog demonstrates broad applicability in public services like healthcare and transportation, showcasing its versatility.
Originality/value
ConfiLog supports iterative, interactive and goal-driven configurations. It ensures flexibility and consistency by recommending variants and scheduling the configuration process.
Software-intensive systems emerge in a multitude of variations to meet diverse customer requirements. To develop such variant-rich software systems, software product line (SPL) Engineering has emerged as a key strategy for managing the variability. However, the adoption of SPLs is highly complex due to the diversity of feature model formats and specifications, and the complexity of implementing variability. Leveraging the capabilities of powerful large language models (LLMs) can facilitate the adoption of SPLs. Nonetheless, these LLMs often lack knowledge of the various specifications of potential feature models as well as efficient implementation of different variability mechanisms. To address these challenges, we propose a novel method based on retrieval-augmented generation. This method generates reusable artefacts and a corresponding feature mapping based on a given feature model, thereby aiding system engineers in adopting an SPL.
The effectiveness of model-driven software engineering (MDSE) has been successfully demonstrated in the context of complex software; however, it has not been widely adopted due to the requisite efforts associated with model development and maintenance, as well as the specific modelling competencies required for MDSE. Concurrently, artificial intelligence (AI) methods, particularly deep learning methods, have demonstrated considerable abilities when applied to the huge code bases accessible on open-source coding platforms. The so-called big code provides the basis for significant advances in empirical software engineering, as well as in the automation of coding processes and improvements in software quality with the use of AI. The objective of this paper is to facilitate a synthesis between these two significant domains of software engineering (SE), namely models and AI in SE. The paper provides an overview of the current state of AI-augmented software engineering and develops a corresponding taxonomy, AI4SE. In light of the aforementioned considerations, a vision of AI-assisted Big Models in SE is put forth, with the aim of capitalising on the advantages inherent to both approaches in the context of software development. Finally, the new paradigm of pair modelling in MDSE is proposed.
Context: Over the years, tools have been created to improve
the execution of development process activities. The emergence
of generative Artificial Intelligence (AI) and, more
recently, the launch and dissemination of Copilot, ChatGPT-
3 and other generative tools, have broadened the discussion
about the possibility of using conversational generative AI
tools in diverse development tasks. Problem: There is still a
lack of secondary studies to map the literature about how
software development process activities can be affected by
the usage of generative AI tools. Solution: This study aims
to identify in which activities of the software development
process Natural Language (NL) generative AI tools have been
used and how they can impact requirements specification,
design/architecture, development and testing activities. IS
Theory: The study was developed under the aegis of the
Task Technology Fit theory. Method: This work presents
the results of a Systematic Mapping Review (SMR) carried
out to collect research results that investigate the application
of generative AI tools in the software development process.
Results: Results indicate that the main activities affected are
development and testing and that, although there are still
some issues to be addressed, there are benefits in using AI
generative tools compared to using more traditional methods
like human-human pair programming and code testing made
by software engineering professionals. Contribution: It was possible to collect studies to identify in which activities of
the software development process generative AI tools can be
applied and what are the impacts of using this technology.
Context
Variant-Rich Systems (VRSs), such as Software Product Lines or variants created through clone & own, are created to satisfy different needs while reusing existing assets. The long lifespan of families of variants, and the scale of both the technical side (implementation size) and the organizational side (roles diversity) make their maintenance and evolution a challenge. Visualization tools are a needed companion.
Objective
We aim at mapping the current state of visualization interventions in the area of VRS evolution. We tackle evolution in both the functionality and the variability management architecture. Three research questions are posed: What sort of analysis is being conducted? (Analysis perspective); What sort of visualizations are displayed? (Visualization perspective); What types of research have been reported and how have they been evaluated? (Maturity perspective).
Methods
We performed a systematic mapping study including automated search in digital libraries, expert knowledge, and snowballing.
Results
The study reports on 41 visualization approaches to cope with VRS evolution. Analysis wise, feature identification and location is the most popular scenario, followed by variant integration towards a Software Product Line. As for visualization, nodelink diagram visualization is predominant while researchers have come up with a wealth of ingenious visualization approaches. Finally, maturity wise, almost half of the studies are solution proposals. Most of the studies provide proof-of-concepts, some of them also include public available tools, yet very few face proof-of-value.
Conclusions
This systematic mapping study introduces a comparison framework where to frame future studies. It also points out distinct research gaps worth investigating as well as shortcomings in the evidence about relevance and contextual considerations (e.g., scalability).
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different
prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
The evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone & own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.
Software product line (SPL) approach has been widely adopted to achieve systematic reuse in families of software products. Despite its benefits, developing an SPL from scratch requires high up-front investment. Because of that, organizations commonly create product variants with opportunistic reuse approaches (e.g., copy-and-paste or clone-and-own). However, maintenance and evolution of a large number of product variants is a challenging task. In this context, a family of products developed opportunistically is a good starting point to adopt SPLs, known as extractive approach for SPL adoption. One of the initial phases of the extractive approach is the recovery and definition of a product line architecture (PLA) based on existing software variants, to support variant derivation and also to allow the customization according to customers’ needs. The problem of defining a PLA from existing system variants is that some variants can become highly unrelated to their predecessors, known as outlier variants. The inclusion of outlier variants in the PLA recovery leads to additional effort and noise in the common structure and complicates architectural decisions. In this work, we present an automatic approach to identify and filter outlier variants during the recovery and definition of PLAs. Our approach identifies the minimum subset of cross-product architectural information for an effective PLA recovery. To evaluate our approach, we focus on real-world variants of the Apo-Games family. We recover a PLA taking as input 34 Apo-Game variants developed by using opportunistic reuse. The results provided evidence that our automatic approach is able to identify and filter outlier variants, allowing to eliminate exclusive packages and classes without removing the whole variant. We consider that the recovered PLA can help domain experts to take informed decisions to support SPL adoption.
In the domain of software product lines, Feature Models (FM) play a central role in variability modeling, completed by configuration collections (from concrete software product lines), logical representations, constraint programming or conceptual structures, coming from the field of Formal Concept Analysis (FCA). The development of feature models may take several forms, including their synthesis from configuration collections or their design in several steps (by several teams or with different concerns). FM composition (merge) operators are part of that design activity as they assist their iterative building. In this paper, we describe an approach, based on two main merging semantics (intersection and union), which assists designers in merging several FMs. This approach benefits from the help of FCA to represent all the FMs with the same configuration set through a canonical form. We describe the implementation of our approach and present a set of concrete examples.
Software Product Lines (SPLs) are families of systems that share common assets allowing a disciplined reuse. Rarely SPLs start from scratch, instead they usually start from a set of existing systems that undergo a reengineering process. Many approaches to conduct the reengineering process have been proposed and documented in research literature. This scenario is a clear testament to the interest in this research area. We conducted a systematic mapping study to provide an overview of the current research on reengineering of existing systems to SPLs, identify the community activity in regarding of venues and frequency of publications in this field, and point out trends and open issues that could serve as references for future research. This study identified 119 relevant publications. These primary sources were classified in six different dimensions related to reengineering phases, strategies applied, types of systems used in the evaluation, input artefacts, output artefacts, and tool support. The analysis of the results points out the existence of a consolidate community on this topic and a wide range of strategies to deal with different phases and tasks of the reengineering process, besides the availability of some tools. We identify some open issues and areas for future research such as the implementation of automation and tool support, the use of different sources of information, need for improvements in the feature management, the definition of ways to combine different strategies and methods, lack of sophisticated refactoring, need for new metrics and measures and more robust empirical evaluation. Reengineering of existing systems into SPLs is an active research topic with real benefits in practice. This mapping study motivates new research in this field as well as the adoption of systematic reuse in software companies.
Over more than two decades, numerous variability modeling techniques have been introduced in academia and industry. However, little is known about the actual use of these techniques. While dozens of experience reports on software product line engineering exist, only very few focus on variability modeling. This lack of empirical data threatens the validity of existing techniques, and hinders their improvement. As part of our effort to improve empirical understanding of variability modeling, we present the results of a survey questionnaire distributed to industrial practitioners. These results provide insights into application scenarios and perceived benefits of variability modeling, the notations and tools used, the scale of industrial models, and experienced challenges and mitigation strategies.
Software product line engineering is a paradigm that advocates the reusability of software engineering assets and the rapid development of new applications for a target domain. These objectives are achieved by capturing the commonalities and variabilities between the applications of the target domain and through the development of comprehensive and variability-covering feature models. The feature models developed within the software product line development process need to cover the relevant features and aspects of the target domain. In other words, the feature models should be elaborate representations of the feature space of that domain. Given that feature models, i.e., software product line feature models, are developed mostly by domain analysts by sifting through domain documentation, corporate records and transcribed interviews, the process is a cumbersome and error-prone one. In this paper, we propose a decision support platform that assists domain analysts throughout the domain engineering lifecycle by: (1) automatically performing natural language processing tasks over domain documents and identifying important information for the domain analysts such as the features and integrity constraints that exist in the domain documents; (2) providing a collaboration platform around the domain documents such that multiple domain analysts can collaborate with each other during the process using a Wiki; (3) formulating semantic links between domain terminology with external widely used ontologies such as WordNet in order to disambiguate the terms used in domain documents; and (4) developing traceability links between the unstructured information available in the domain documents and their formal counterparts within the formal feature model representations. Results obtained from our controlled experimentations show that the decision support platform is effective in increasing the performance of the domain analysts during the domain engineering lifecycle in terms of both the coverage and accuracy measures.
In order to migrate software products which are deemed similar into a product line, it is essential to identify the common features and the variations between the product variants. This can however be tedious and error-prone as it may involve browsing complex software and a lot of more or less similar variants. Fortunately, if arte facts of the product variants (source code files and/or models) are available, feature identification can be at least partially automated. In this paper, we thus propose a three-step approach to feature identification from source code of which the first two steps are automated.
Feature Models (FMs) are used extensively in software prod-uct line engineering to help generate and validate individ-ual product configurations and to provide support for do-main analysis. As FM construction can be tedious and time-consuming, researchers have previously developed tech-niques for extracting FMs from sets of formally specified in-dividual configurations, or from software requirements spec-ifications for families of existing products. However, such artifacts are often not available. In this paper we present a novel, automated approach for constructing FMs from pub-licly available product descriptions found in online product repositories and marketing websites such as SoftPedia and CNET. While each individual product description provides only a partial view of features in the domain, a large set of descriptions can provide fairly comprehensive coverage. Our approach utilizes hundreds of partial product descriptions to construct an FM and is described and evaluated against an-tivirus product descriptions mined from SoftPedia.
Understanding the dependency between performance metrics (such as response time) and software configuration or usage parameters is crucial in improving software quality. However, the size of most modern systems makes it nearly impossible to provide a complete performance model. Hence, we focus on scenario-specific problems where software engineers require practical and efficient approaches to draw conclusions, and we propose an automated, measurement-based model inference method to derive goal-oriented performance prediction functions. For the practicability of the approach it is essential to derive functional dependencies with the least possible amount of data. In this paper, we present different strategies for automated improvement of the prediction model through an adaptive selection of new measurement points based on the accuracy of the prediction model. In order to derive the prediction models, we apply and compare different statistical methods. Finally, we evaluate the different combinations based on case studies using SAP and SPEC benchmarks.
Traditional methods characterize a software product line's requirements using either functional or quality criteria. This appears to be inadequate to assess modularity, detect interferences, and analyze trade-offs. We take advantage of both symmetric and asymmetric views of aspects, and perform formal concept analysis to examine the functional and quality requirements of an evolving product line. The resulting concept lattice provides a rich notion which allows remarkable insights into the modularity and interactions of requirements. We formulate a number of problems that aspect-oriented product line requirements engineering should address, and present our solutions according to the concept lattice. We describe a case study applying our approach to analyze a mobile game product line's requirements, and review lessons learned.
Abstract Testing systems with large configurations spaces that chang e often is a challenging problem. The cost and complexity of QA explodes because often there isn’t just one system, but a multitude of related systems. Bugs may appear in certain configurations, but not i n others. The Skoll system and process has been developed,to test these types of systems through distributed, continuous quality assurance, leveraging user resources around-the-world, around-the-clock. It has been shown to be effective in automatically characterizing configurations in which,failures manifest. The derived informa tion helps developers quickly narrow down,the cause of failures which then improves turn around time for fixes. However, this metho d does not scale well. It requires one to exhaustively test eac h configuration in the configuration space. In this paper we examine,an alternative approach. We use a mathematical,object called a covering array to obtain relatively small test schedules with certain coverage properties over the entire configuration space. We empirically assess the effect o f using covering array derived test schedules on the resulting faul t characterizations and provide guidelines to practitioners for th eir use.
Software product line engineering introduces two new dimensions into the traditional engineering of software-based systems: the variability modeling and the product derivation. The variability gathers characteristics that differ from one product to another, while the product derivation is defined as a complete process of building products from the product line. Software Product Line Engineering with the UML has received a lot of attention in recent years. However most of these works only concern variability modeling in UML static models and few works concern behavioral models. In addition, there is very little research on product derivation. This chapter investigates the product derivation in the context of the product line engineering with the UML. First, a set of extensions are proposed to model product line variability in two types of UML models: class diagrams (the static aspect) and sequence diagrams (the behavioral aspect). Then we formalize product derivation using a UML model transformation. An algorithm is given to derive a static model for a product and an algebraic approach is proposed to derive product-specific statecharts from the sequence diagrams of the product line. Two simple case studies are presented, based on a Mercure product line and the banking product line, to illustrate the overall process, from the modeling of the product line to the product derivation.
Domain analysis involves not only looking at standard requirements documents (e.g., use case specifications) but also at customer information packs, market analyses, etc. Looking across all these documents and deriving, in a practical and scalable way, a feature model that is comprised of coherent abstractions is a fundamental and non-trivial challenge. We conduct an exploratory study to investigate the suitability of information retrieval (IR) techniques for scalable identification of commonalities and variabilities in requirement specifications for software product lines. Accordingly, based on observations derived from industrial experience and on state-of-the-art research and practice, we also propose an initial framework, leveraging IR to systematically abstract requirements from existing specifications of a given domain into a feature model. We evaluate this framework, present a roadmap for its further extension, and formulate hypotheses to guide future work in exploring IR techniques for domain analysis.
Most modern software systems (operating systems like Linux or Android, Web browsers like Firefox or Chrome, video encoders like ffmpeg, x264 or VLC, mobile and cloud applications, etc.) are highly configurable. Hundreds of configuration options, features, or plugins can be combined, each potentially with distinct functionality and effects on execution time, security, energy consumption, etc. Due to the combinatorial explosion and the cost of executing software, it is quickly impossible to exhaustively explore the whole configuration space. Hence, numerous works have investigated the idea of learning it from a small sample of configurations’ measurements. The pattern ”sampling, measuring, learning” has emerged in the literature, with several practical interests for both software developers and end-users of configurable systems. In this systematic literature review, we report on the different application objectives (e.g., performance prediction, configuration optimization, constraint mining), use-cases, targeted software systems, and application domains. We review the various strategies employed to gather a representative and cost-effective sample. We describe automated software techniques used to measure functional and non-functional properties of configurations. We classify machine learning algorithms and how they relate to the pursued application. Finally, we also describe how researchers evaluate the quality of the learning process. The findings from this systematic review show that the potential application objective is important; there are a vast number of case studies reported in the literature related to particular domains or software systems. Yet, the huge variant space of configurable systems is still challenging and calls to further investigate the synergies between artificial intelligence and software engineering.
Context: It is common belief that high impact research in software reuse requires assessment in non-trivial, comparable, and reproducible settings. However, software artefacts and common representations are usually unavailable. Also, establishing a representative ground truth is a challenging and debatable subject. Feature location in the context of software families, which is key for software product line adoption, is a research field that is becoming more mature with a high proliferation of techniques. Objective: We present EFLBench, a benchmark and a framework to provide a common ground for the evaluation of feature location techniques in families of systems. Method: EFLBench leverages the efforts made by the Eclipse Community which provides feature-based family artefacts and their plugin-based implementations. Eclipse is an active and non-trivial project and thus, it establishes an unbiased ground truth which is realistic and challenging. Results: EFLBench is publicly available and supports all tasks for feature location techniques integration, benchmark construction and benchmark usage. We demonstrate its usage, simplicity and reproducibility by comparing four techniques in Eclipse releases. As an extension of our previously published work, we consider a decade of Eclipse releases and we also contribute an approach to automatically generate synthetic Eclipse variants to benchmark feature location techniques in tailored settings. We present and discuss three strategies for this automatic generation and we present the results using different settings. Conclusion: EFLBench is a contribution to foster the research in feature location in families of systems providing a common framework and a set of baseline techniques and results.
Modeling how contextual factors relate to a software system’s configuration space is usually a manual, error-prone task that depends highly on expert knowledge. Machine-learning techniques can automatically predict the acceptable software configurations for a given context. Such an approach executes and observes a sample of software configurations within a sample of contexts. It then learns what factors of each context will likely discard or activate some of the software’s features. This lets developers and product managers automatically extract the rules that specialize highly configurable systems for specific contexts.
Building Software Product Lines (SPLs) from existing artefacts is known as the extractive approach for SPL adoption. The traditional case is that variants are created with ad-hoc reuse (e.g., copy-paste-modify to quickly respond to different customer needs) and practitioners want to reengineer them to an SPL. Several industrial cases have been presented in the literature to motivate the interest of the extraction and many case studies are used to validate methods and techniques for different activities during this adoption process.
However, there is no catalog or repository that gather together case studies and artefacts related to extractive SPL adoption. In this paper we present ESPLA, a catalog of Extractive SPL Adoption case studies that aims to foster the advance of this field by providing comprehensive information about case studies that will be otherwise scattered in the literature. Researchers, practitioners and educators can use this catalog to find the case studies that better fit to their particular needs. Currently, ESPLA contains information about 123 case studies and it is intended to be a catalog that can be updated and extended by the community.
Finding the optimally performing configuration of a software system for a given setting is often challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, building an accurate performance model can be very expensive (and is often infeasible in practice). The central insight of this paper is that exact performance values (e.g., the response time of a software system) are not required to rank configurations and to identify the optimal one. As shown by our experiments, performance models that are cheap to learn but inaccurate (with respect to the difference between actual and predicted performance) can still be used rank configurations and hence find the optimal configuration. This novel rank-based approach allows us to significantly reduce the cost (in terms of number of measurements of sample configuration) as well as the time required to build performance models. We evaluate our approach with 21 scenarios based on 9 software systems and demonstrate that our approach is beneficial in 16 scenarios; for the remaining 5 scenarios, an accurate model can be built by using very few samples anyway, without the need for a rank-based approach.
Many software systems provide configuration options relevant to users, which are often called features. Features influence functional properties of software systems as well as non-functional ones, such as performance and memory consumption. Researchers have successfully demonstrated the correlation between feature selection and performance. However, the generality of these performance models across different hardware platforms has not yet been evaluated.
We propose a technique for enhancing generality of performance models across different hardware environments using linear transformation. Empirical studies on three real-world software systems show that our approach is computationally efficient and can achieve high accuracy (less than 10% mean relative error) when predicting system performance across 23 different hardware platforms. Moreover, we investigate why the approach works by comparing performance distributions of systems and structure of performance models across different platforms.
A common and simple way to create custom product variants is to copy and adapt existing software (a. k. a. the clone-and-own approach). Clone-and-own promises low initial costs for creating a new variant as existing code is easily reused. However, clone-and-own also comes with major drawbacks for maintenance and evolution since changes, such as bug fixes, need to be synchronized among several product variants. Software product lines (SPLs) provide solutions to these problems because commonalities are implemented only once. Thus, in an SPL, changes also need to be applied only once. Therefore, the migration of cloned product variants to an SPL would be beneficial. The main tasks of migration are the identification and extraction of commonalities from existing products. However, these tasks are challenging and currently not well-supported. In this paper, we propose a step-wise and semi-automated process to migrate cloned product variants to a feature-oriented SPL. Our process relies on clone detection to identify code that is common to multiple variants and novel, variant-preserving refactorings to extract such common code. We evaluated our approach on five cloned product variants, reducing code clones by 25 %. Moreover, we provide qualitative insights into possible limitations and potentials for removing even more redundant code. We argue that our approach can effectively decrease synchronization effort compared to clone-and-own development and thus reduce the long-term costs for maintenance and evolution.
Reengineering a Software Product Line from legacy variants remains a challenging endeavour. Among various challenges, it is a complex task to retrieve enough information for inferring the variability from experts' domain knowledge and from the semantics of software elements. We propose the VariClouds process that can be leveraged by domain experts to understand the semantics behind the different blocks identified during software variants analysis. VariClouds is based on interactive word cloud visualisations providing name suggestions for these blocks using tf-idf as weighting factor. We evaluate our approach by assessing its added-value to several previous works in the literature where no tool support was provided to domain experts to characterise features from software blocks.
Modern software systems have grown significantly in their size and complexity, therefore understanding how software systems behave when there are many configuration options, also called features, is no longer a trivial task. This is primarily due to the potentially complex interactions among the features. In this paper, we propose a novel mathematical model for performance-relevant, or quantitative in general, feature interactions, based on the theory of Boolean functions. Moreover, we provide two algorithms for detecting all such interactions with little measurement effort and potentially guaranteed accuracy and confidence level. Empirical results on real-world configurable systems demonstrated the feasibility and effectiveness of our approach.
Many applications in Model-Driven Engineering involve processing multiple models, e.g. for comparing and merging of model variants into a common domain model. Despite many sophisticated techniques for model comparison, little attention has been given to the initial data analysis and filtering activities. These are hard to ignore especially in the case of a large dataset, possibly with outliers and sub-groupings. We would like to develop a generic approach for model comparison and analysis for large datasets; using techniques from information retrieval, natural language processing and machine learning. We are implementing our approach as an open framework and have so far evaluated it on public datasets involving domain analysis, repository management and model searching scenarios.
Almost every complex software system today is configurable. While configurability has many benefits, it challenges performance prediction, optimization, and debugging. Often, the influences of individual configuration options on performance are unknown. Worse, configuration options may interact, giving rise to a configuration space of possibly exponential size. Addressing this challenge, we propose an approach that derives a performance-influence model for a given configurable system, describing all relevant influences of configuration options and their interactions. Our approach combines machine-learning and sampling heuristics in a novel way. It improves over standard techniques in that it (1) represents influences of options and their interactions explicitly (which eases debugging), (2) smoothly integrates binary and numeric configuration options for the first time, (3) incorporates domain knowledge, if available (which eases learning and increases accuracy), (4) considers complex constraints among options, and (5) systematically reduces the solution space to a tractable size. A series of experiments demonstrates the feasibility of our approach in terms of the accuracy of the models learned as well as the accuracy of the performance predictions one can make with them.
In this tutorial we will review some ways to automate domain modeling and discuss the boundaries of automation in this context. We will concentrate on an ontological and conceptual approach which examines software behaviors and not just domain terminology and semantic similarity. We will show and demonstrate a tool named SOVA - Semantic and Ontological Variability Analysis, which supports automatic domain modeling from software requirements written in free text. We will further discuss how the existence of different SPLs in the same domain may improve the domain models through cross product line analysis.
Variability analysis in Software Product Line Engineering (SPLE) utilizes various software-related artifacts, including requirements specifications. Currently, measuring the similarity of requirements specifications for analyzing variability of software products mainly takes into account semantic considerations. This might lead to failure to capture important aspects of the software behavior as perceived by users. In this paper we present a tool, called SOVA-Semantic and Ontological Variability Analysis, which introduces ontological considerations to variability analysis, in addition to the semantic ones. The input of the tool is textual requirements statements organized in documents. Each document represents the expectations from or the characteristics of a different software product in a line, while each requirement statement represents an expected behavior of that software product. The output is a feature diagram representing the variability in the input set of requirements documents and setting the ground for behavioral domain analysis.
Model merging is widely recognized as an essential step in a variety of software development activities. During the process of combining a set of related products into a product line or consolidating model views of multiple stakeholders, we need to merge multiple input models into one; yet, most of the existing approaches are applicable to merging only two models. In this paper, we define the n-way merge problem. We show that it can be reduced to the known and widely studied NP-hard problem of weighted set packing. Yet, the approximation solutions for that problem do not scale for real-sized software models. We thus evaluate alternative approaches of merging models that incrementally process input models in small subsets and propose our own algorithm that considerably improves precision over such approaches without sacrificing performance.
The decision-making process in Product Line Engineering (PLE) is often concerned with variant qualities such as cost, battery life, or security. Pareto-optimal variants, with respect to a set of objectives such as minimizing a variant's cost while maximizing battery life and security, are variants in which no single quality can be improved without sacrificing other qualities. We propose a novel method and a tool for visualization and exploration of a multi-dimensional space of optimal variants (i.e., a Pareto front). The visualization method is an integrated, interactive, and synchronized set of complementary views onto a Pareto front specifically designed to support PLE scenarios, including: understanding differences among variants and their positioning with respect to quality dimensions; solving trade-offs; selecting the most desirable variants; and understanding the impact of changes during product line evolution on a variant's qualities. We present an initial experimental evaluation showing that the visualization method is a good basis for supporting these PLE scenarios.
Many companies develop software product lines-collections of similar products-by cloning and adapting artifacts of existing product variants. Transforming such cloned product variants into a "single-copy" software product line representation is considered an important software re-engineering activity, as reflected in numerous tools and methodologies available. However, development practices of companies that use cloning to implement product lines have not been systematically studied. This lack of empirical knowledge threatens the validity and applicability of approaches supporting the transformation, and impedes adoption of advanced solutions for systematic software reuse. It also hinders the attempts to improve the solutions themselves. We address this gap with an empirical study conducted to investigate the cloning culture in six industrial software product lines realized via code cloning. Our study investigates the processes, and the perceived advantages and disadvantages of the approach. We observe that cloning, while widely discouraged in literature, is still perceived as a favorable and natural reuse approach by the majority of practitioners in the studied companies. This is mainly due to its benefits such as simplicity, availability and independence of developers. Based on our observations, we outline issues preventing the adoption of systematic software reuse approaches, and identify future research directions.
Organizations can reduce the costs and enhance the quality of required software by adapting existing software systems. Software adaptation decisions often involve comparing alternatives on two criteria: (1) how well a system meets users' requirements and (2) the effort required for adapting the system. These criteria reflect two points of view — of users and of developers. Common to both views is the notion of functionality, which software developers have traditionally used for effort estimation utilizing concepts such as function points. However, users involved in selecting systems are not necessarily familiar with such concepts. We propose an approach for comparing software functionality from users' point of view. The approach employs ontological concepts to define functionality in terms of system behaviors. To evaluate whether or not the approach is also usable by software developers, we conducted an exploratory experiment. In the experiment, software engineering students ranked descriptions of software systems on the amount of changes needed to adapt the systems to given requirements. The results demonstrated that the ontological approach was usable after a short training and provided results comparable to ranking done by expert software developers. We also compared the ontological approach to a method which employed function point concepts. The results showed no statistically significant differences in performance, but there seemed to be an advantage to the ontological approach for cases that were difficult to analyze. Moreover, it took less time to apply the ontological approach than the function point-based approach, and the difference was statistically significant.
BERT: Pre-training of deep bidirectional transformers for language understanding
- Jacob Devlin
- Ming-Wei Chang
- Kenton Lee
- Kristina Toutanova
- Devlin Jacob
Flashattention: Fast and memory-efficient exact attention with io-awareness
- Tri Dao
- Dan Fu
- Stefano Ermon
- Atri Rudra
- Christopher Ré
- Dao Tri
Alessio Ferrari, Giorgio Oronzo Spagnolo, and Felice dell'Orletta. 2013. Mining commonalities and variabilities from natural language documents
- Alessio Ferrari
- Giorgio Oronzo Spagnolo
- Felice Dell'orletta
- Ferrari Alessio
GitHub Copilot. Your AI pair programmer
- Github
2023 . Handbook of Re-Engineering Software Intensive Systems into Software Product Lines
- Roberto E Lopez-Herrejon
- Jabier Martinez
- Wesley Klewerton Guez Assunção
- Lopez-Herrejon Roberto E.
Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis
- Pooyan Jamshidi
- Norbert Siegmund
- Miguel Velez
- Akshay Patel
- Yuvraj Agarwal
- Jamshidi Pooyan
Natural Language Processing with Transformers Revised Edition
- Thomas Wolf Lewis Tunstall Leandro Von Werra
Bottom-Up Technologies for Reuse: A Framework to Support Extractive Software Product Line Adoption Activities . In Handbook of Re-Engineering Software Intensive Systems into Software Product Lines
- Jabier Martinez
- Tewfik Ziadi
- F Tegawendé
- Jacques Bissyandé
- Yves Le Klein
- Traon
- Martinez Jabier
Julia Rubin and Marsha Chechik. 2013. N-way model merging
- Julia Rubin
- Marsha Chechik
- Rubin Julia