No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A Case Study on Artefact-Based RE Improvement in Practice
Abstract
Most requirements engineering (RE) process improvement approaches are solution-driven and activity-based. They focus on the assessment of the RE of a company against an external norm of best practices. A consequence is that practitioners often have to rely on an improvement approach that skips a profound problem analysis and that results in an RE approach that might be alien to the organisational needs. In recent years, we have developed an RE improvement approach (called \emph{ArtREPI}) that guides a holistic RE improvement against individual goals of a company putting primary attention to the quality of the artefacts. In this paper, we aim at exploring ArtREPI's benefits and limitations. We contribute an industrial evaluation of ArtREPI by relying on a case study research. Our results suggest that ArtREPI is well-suited for the establishment of an RE that reflects a specific organisational culture but to some extent at the cost of efficiency resulting from intensive discussions on a terminology that suits all involved stakeholders. Our results reveal first benefits and limitations, but we can also conclude the need of longitudinal and independent investigations for which we herewith lay the foundation.
... Méndez Fernández and Wagner [16] explored the effect of the RE improvement approach ArtREPI that applies the RE best practice database AMDiRE. The impact of ArtREPI in two case studies is measured via two post-treatment questionnaires: one on the support of process engineers and one about project participants' rating of ArtREPI's output. ...
... Our research incorporates elements from each of the aforementioned studies. We consider the relevance of little direct author involvement [16], allow the companies to choose themselves which quality criteria to apply [17], introduce our treatment to multiple companies [21] and collect both qualitative perception data and quantitative metrics [3]. Differently from Damian's study [3], our metrics include quantitative and qualitative data on indirectly related human processes around software development. ...
... The mildly positive participant perception is not confirmed by software development process metrics. As in similar studies [16,21], a large scale replication is necessary to generalize our conclusions, also to reduce threats to validity by controlling for confounding variables as company and project size. Additionally, we want to study the implications of specific quality criteria; what are the consequences of a minimality or uniformity defect in isolation? ...
Context and motivation: Previous research shows that a considerable amount of real-world user stories contain easily preventable syntactic defects that violate desired qualities of good requirements. However, we still do not know the effect of user stories’ intrinsic quality on practitioners’ work. Question/Problem: We study the effects of introducing the Grimm Method’s Quality User Story framework and the AQUSA tool on the productivity and work deliverable quality of 30 practitioners from 3 companies over a period of 2 months. Principal ideas/results: Our multiple case study delivered mixed findings. Despite an improvement in the intrinsic user story quality, practitioners did not perceive such a change. They explained, however, there was more constructive user story conversation in the post-treatment period leading to less unnecessary rework. Conversely, project management metrics did not result in statistically significant changes in the number of comments, issues, defects, velocity, and rework. Contribution: Introducing our treatment has a mildly positive effect but a larger scale investigation is crucial to decisively assess the impact on work practice. Also, our case study protocol serves as an example for evaluating RE research in practice.
Amaç: Bu makalenin amacı, organizasyonlara değişim için bir ‘Reorganizasyon Modeli’ önermek, modelin uygulanmasına yardımcı olacak bir yol haritası sunmak ve modelin yerel yönetimlerde uygulandığında verimliliği artırdığını, kurumsal performans ve çalışan memnuniyeti sonuçları üzerinden göstermektir.
Yöntem: Reorganizasyon modelindeki her bir kavramın literatür taraması yapılmış, taslak bir model ve yol haritası hazırlanmıştır. Modeli kısmen uygulamış ve EFQM ödülünü kazanarak başarılı olmuş bir kuruluştaki üst düzey yöneticilerle yapılandırılmamış görüşme tekniği yapılarak taslaklara son hali verilmiştir. Modelin kurumsal verimliliği artırdığını göstermek için, uygulama sonuçları, aynı kuruluşta belge analizleri metoduyla incelenmiştir.
Bulgular: Önerilen modeli uygulayan organizasyonun kurumsal performans ve çalışan memnuniyeti göstergelerinde yıllara sâri bir artış gözlemlenmiş ve modelin kurumsal verimliliği artırdığı ispatlanmıştır.
Özgünlük: Bu makaleyi diğerlerinden farklılaştıran nokta, reorganizasyon modelinin literatüre dayalı bütünsel ve kapsayıcı bir şekilde ele alınması, ortak bir dil oluşturacak ve kuruluşlara değişim süreçlerinde yardımcı olacak ilk ve tek model olması, modelin tüm aşamalarının sistemsel bakış açısına sahip olması ve uygulanabilmesi için bir yol haritası önermesidir.
Companies are often challenged to modify and improve their software development processes in order to make them compliant with security standards. The complexity of these processes renders it difficult for practitioners to validate and foresee the effort required for compliance assessments. Further, performing gap analyses when processes are not yet mature enough is costly and involving auditors in early stages is, in our experience, often inefficient. An easier and more productive approach is conducting a self-assessment. However, practitioners, in particular developers, quality engineers, and product owners face difficulties to identify security-relevant process artifacts as required by standards. They would benefit from a proper and light-weight tool to perform early compliance assessments of their processes w.r.t. security standards before entering an in-depth audit. In this paper, we report on our current effort at Siemens Corporate Technology to develop such a light-weight assessment tool to assess the security compliance of software development processes with the IEC 62443-4-1 standard, and we discuss first results from an interview-based evaluation.
The effective communication of the requirements influences the success of software development projects. Achieving effective communication of the requirements is difficult due to the involvement of several persons with different roles, skills, knowledge and responsibilities. Although many studies analyze the communication between clients and system analysts, they do not focus on the communication within the development team. In this research, we propose the creation of a set of artifacts and models to support the communication of requirements. We will base our proposal on the different perspectives of the team members according to their experience in the artifacts and models adopted in the development process within their organization. We will follow a methodology based on Design Science Research guidelines, which will guide us through the creation and evaluation of the artifacts and models to solve problems with the communication of requirements. Our goal is to improve the communication of requirements between the members of a development team, reducing the loss of requirements information during the execution of the software project
Requirements engineering process improvement (REPI) has gained much attention in research and practice. Most REPI approaches are of solution-driven and activity-based nature. They focus on the assessment of company-specific RE reference models against an external norm of best practices, and they propagate an improvement by forecasting the adaptation of the processes and methods in the RE reference model towards that norm. In recent years, we could develop a first problem-driven RE improvement approach that supports an improvement against individual goals and problems of a company putting primary attention to the quality of the RE artefacts (named ArtREPI). In this short paper, we briefly illustrate our resulting approach and report on our initial experiences from ongoing empirical evaluations in practice. We conclude with a summary of planned next steps.
Context: Requirements engineering process improvement (REPI) approaches have gained much attention in research and practice. Goal: So far, there is no comprehensive view on the research in REPI in terms of solutions and current state of reported evidence. We aims to provide an overview on the existing solutions, their underlying principles and their research type facets, i.e. their state of empirical evidence. Method: To this end, we conducted a systematic mapping study of the REPI publication space. Results: This paper reports on the first findings regarding research type facets of the contributions as well as selected methodological principles. We found a strong focus in the existing research on solution proposals for REPI approaches that concentrate on normative assessments and benchmarks of the RE activities rather than on holistic RE improvements according to individual goals of companies. Conclusions: We conclude, so far, that there is a need to broaden the work and to investigate more problem-driven REPI which also targets the improvement of the quality of the underlying RE artefacts, which currently seem out of scope.
Context: For many years, researchers and practitioners have been proposing various methods and approaches to Requirements Engineering (RE). Those contributions remain, however, too often on the level of apodictic discussions without having proper knowledge about the practical problems they propagate to address, or how to measure the success of the contributions when applying them in practical contexts. While the scientific impact of research might not be threatened, the practical impact of the contributions is. Aim: We aim at better understanding practically relevant variables in RE, how those variables relate to each other, and to what extent we can measure those variables. This allows for the establishment of generalisable improvement goals, and the measurement of success of solution proposals. Method: We establish a first empirical basis of dependent variables in RE and means for their measurement. We classify the variables according to their dimension (e.g. RE, company, SW project), their measurability, and their actionability. Results: We reveal 93 variables with 167 dependencies of which a large subset is measurable directly in RE while further variables remain unmeasurable or have too complex dependencies for reliable measurements. We critically reflect on the results and show direct implications for research in the field of RE. Conclusion: We discuss a variety of conclusions we can draw from our results. For example, we show a set of first improvement goals directly usable for evidence-based RE research such as "increase flexibility in the RE process", we discuss suitable study types, and, finally, we can underpin the importance of replication studies to obtain generalisability.
The various influences in the processes and application domains make Requirements Engineering (RE) inherently complex and di�cult to im- plement. In general, we have two options for establishing an RE approach: we can either establish an activity-based RE approach or we can establish an artefact-based one where project participants concentrate on the RE artefacts rather than on the way of creating them. While a number of activity-based RE approaches have been proposed in recent years, we have gained much em- pirical evidence and experiences about the advantages of the artefact-based paradigm for RE. However, artefact orientation is still a young paradigm with various interpretations and practical manifestations whereby we need a clear understanding of its basic concepts and a consolidated and evaluated view on the paradigm.
In this article, we contribute an artefact-based approach to RE (AMDiRE) that emerges from six years of experiences in fundamental and evidence-based research. To this end, we first discuss the basic notion of artefact orientation and its evolution in recent years. We briefly introduce a set of artefact-based RE models we developed in industrial research cooperations for di↵erent ap- plication domains, show their empirical evaluations, and their dissemination into academia and practice, eventually leading to the AMDiRE approach. We conclude with a discussion of experiences we made during the development and di↵erent industrial evaluations, and lessons learnt.
Context: For many years, we have observed industry struggling in defining a high quality requirements engineering (RE) and researchers trying to understand industrial expectations and problems. Although we are investigating the discipline with a plethora of empirical studies, those studies either concentrate on validating specific methods or on single companies or countries. Therefore, they allow only for limited empirical generalisations. Objective: To lay an empirical and generalisable foundation about the state of the practice in RE, we aim at a series of open and reproducible surveys that allow us to steer future research in a problem-driven manner. Method: We designed a globally distributed family of surveys in joint collaborations with different researchers from different countries. The instrument is based on an initial theory inferred from available studies. As a long-term goal, the survey will be regularly replicated to manifest a clear understanding on the status quo and practical needs in RE. In this paper, we present the design of the family of surveys and first results of its start in Germany. Results: Our first results contain responses from 30 German companies. The results are not yet generalisable, but already indicate several trends and problems. For instance, a commonly stated problem respondents see in their company standards are artefacts being underrepresented, and important problems they experience in their projects are incomplete and inconsistent requirements. Conclusion: The results suggest that the survey design and instrument are well-suited to be replicated and, thereby, to create a generalisable empirical basis of RE in practice.
When it comes to software process improvement (SPI), process engineers look for SPI methods to support process analysis, design, realization, deployment, and management. Although a number of different SPI methods and models exist, process engineers tend to view these as too generic, too large, or a poor fit for the organization in which SPI is conducted. A strategy to overcome these shortcomings is to concentrate on the artifacts, which precisely define the de-sired outcomes, rather than on specific methods. In this paper, we present the Artifact-based Software Process Improvement & Management (ArSPI) model that provides a unified perspective on SPI and company-wide software process management (SPM), the required key artifacts, and the life cycle models. ArSPI is shown to be of practical sup-port to industry who called for a practical way to define the interfaces between SPI projects. This paper concludes with an example of how ArSPI paved the way for several organizations through applying the model in real-world SPI-projects.
Requirements engineering (RE) is a key discipline in software development and several methods are available to help assess and improve RE processes. However, these methods rely on prescriptive models of RE; they do not, like other disciplines within software engineering, draw directly on stakeholder perceptions and subjective judgments. Given this backdrop, we present an empirical study in RE process assessment. Our aim was to investigate how stakeholder perceptions and process prescriptions can be combined during assessments to effectively inform RE process improvement. We first describe existing methods for RE process assessment and the role played by stakeholder perceptions and subjective judgments in the software engineering and management literature. We then present a method that combines perceptions and prescriptions in RE assessments together with an industrial case study in which the method was applied and evaluated over a three-year period at TelSoft. The data suggest that the combined method led to a comprehensive and rich assessment and it helped TelSoft consider RE as an important and integral part of the broader engineering context. This, in turn, led to improvements that combined plan-driven and adaptive principles for RE. Overall, the combined method helped TelSoft move from Level 1 to Level 2 in RE maturity, and the employees perceived the resulting engineering practices to be improved. Based on these results, we suggest that software managers and researchers combine stakeholder perceptions and process prescriptions as one way to effectively balance the specificity, comparability, and accuracy of software process assessments.
The importance of continuously improving requirements engineering (RE) has been recognised for many years. Similar to available software process improvement approaches, most RE improvement approaches focus on a normative and solution-driven assessment of companies rather than on a problem-driven RE improvement. The approaches dictate the implementation of a one-size-fits-all reference model without doing a proper problem investigation first, whereas the notion of quality factually depends on whether RE achieves company-specific goals. The approaches furthermore propagate process areas and methods, without proper awareness of the quality in the created artefacts on which the quality of many development phases rely. Little knowledge exists about how to conduct a problem-driven RE improvement that gives attention to the improvement of the artefacts. A promising solution is to start an improvement with an empirical investigation of the RE stakeholders, goals, and artefacts in the company to identify problems while abstracting from inherently complex processes. The RE improvement is then defined and implemented in joint action research workshops with the stakeholders to validate potential solutions while again concentrating on the artefacts. In this paper, we contribute an artefact-based, problem-driven RE improvement approach that emerged from a series of completed RE improvements. We discuss lessons learnt and present first result from an ongoing empirical evaluation at a German company. Our results suggest that our approach supports process engineers in a problem-driven RE improvement, but we need deeper examination of the resulting RE company standard, which is in scope of the final evaluation.
In this paper we discuss our study of the problems 12 software companies experienced in software development. In total we present qualitative data collected from 45 focus groups that involved over 200 software staff. We look at how different practitioner groups respond to software process improvement problems. We show our classification and analysis of this data using correspondence analysis. Correspondence analysis is a graphical data representation method new to software development research. The aim of the work we present is to develop a more holistic understanding of the problems practitioners are experiencing in their attempts to improve their software processes. Our main finding is that there is an association between a company's capability maturity and patterns of reported problems. Organizational problems are more associated with high maturity companies than with low maturity companies. Low maturity companies are closely linked to problems relating directly to projects such as documentation, timescales, tools and technology. Our findings also confirm differences in practitioner group problems. Senior managers cite problems with goals, culture and politics. Project managers are concerned with timescales, change management, budgets and estimates. Developers are experiencing problems with requirements, testing, documentation, communication, tools and technology. These associations are displayed graphically through correspondence analysis maps.
Abstract,Case study is a suitable research methodology,for software engineering,research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims,at providing,an introduction to case study methodology,and,guidelines for researchers,conducting,case studies and,readers studying,reports of such,studies. The content is based on the authors’ own,experience from conducting,and reading case studies. The terminology,and,guidelines are compiled,from,different methodology,handbooks,in other research domains, in particular social science and information systems, and adapted to the needs,in software,engineering. We,present,recommended,practices for software engineering,case studies as well,as empirically,derived,and,evaluated,checklists for researchers and readers of case study research. Keywords,Casestudy.Research methodology.Checklists .Guidelines
Requirements Engineering (RE) processes are highly volatile due to dependencies on customers' capabilities or used process models, both complicating a standardised RE process. A promising solution is given by artefactorientation that emphasises the results rather than dictating a strict development process. At such a basis one is able to incorporate domain-specific methods for producing artefacts without having to take into account the variability of process definitions. Although artefacts are known to support customisable development processes, there still is no common agreement about the structure and semantics of artefact-based methodologies. In this paper we discuss different interpretations of the term artefact considering aspects like process integration capabilities and necessities within individual project environments. We contribute a meta model for artefact-orientation that is inferred from two RE models elaborated within industrial cooperation projects of our research group. We conclude with a discussion of performed case studies and ongoing work.
[Context & motivation] Requirements artifacts, like specifications, diagrams, or user stories, are often used to support various activities related to requirements. How well an artifact can support a specific activity depends on the artifact’s nature. For example, a plain text document can be adequate to provide contextual information, but is not well suited in terms of documenting changes. [Questions / problem] We wanted to understand how practitioners in various roles use requirements artifacts, how they manage to work with multiple artifacts at a time, and whether they use current practices for linking related artifacts. [Principal ideas / results] We have conducted an interview study with 21 practitioners from 6 companies. The interviews indicate that often a variety of artifact types is needed to successfully conduct a project. At the same time, using multiple artifacts causes problems like manual translation effort and inconsistencies. Mapping mechanisms that explicitly relate different artifacts are needed. However, existing methods are often not used. We investigate why these methods challenge developers in practice. [Contribution] We show challenges and chances of requirements artifacts. Our findings are grounded on true experiences from the industry. These experiences can support software developers in planning and improving their processes with regard to better requirements communication and researchers in making mapping methods more applicable in industry.
Context
For many years, we have observed industry struggling in defining a high quality requirements engineering (RE) and researchers trying to understand industrial expectations and problems. Although we are investigating the discipline with a plethora of empirical studies, they still do not allow for empirical generalisations.
Objective
To lay an empirical and externally valid foundation about the state of the practice in RE, we aim at a series of open and reproducible surveys that allow us to steer future research in a problem-driven manner.
Method
We designed a globally distributed family of surveys in joint collaborations with different researchers and completed the first run in Germany. The instrument is based on a theory in the form of a set of hypotheses inferred from our experiences and available studies. We test each hypothesis in our theory and identify further candidates to extend the theory by correlation and Grounded Theory analysis.
Results
In this article, we report on the design of the family of surveys, its underlying theory, and the full results obtained from Germany with participants from 58 companies. The results reveal, for example, a tendency to improve RE via internally defined qualitative methods rather than relying on normative approaches like CMMI. We also discovered various RE problems that are statistically significant in practice. For instance, we could corroborate communication flaws or moving targets as problems in practice. Our results are not yet fully representative but already give first insights into current practices and problems in RE, and they allow us to draw lessons learnt for future replications.
Conclusion
Our results obtained from this first run in Germany make us confident that the survey design and instrument are well-suited to be replicated and, thereby, to create a generalisable empirical basis of RE in practice.
Before technology is transferred to the market, it must be validated empirically by simulating future practical use of the technology. Technology prototypes are first investigated in simplified contexts, and these simulations are scaled up to conditions of practice step by step as more becomes known about the technology. This paper discusses empirical research methods for scaling up new requirements engineering (RE) technology.
When scaling up to practice, researchers want to generalize from validation studies to future practice. An analysis of scaling up technology in drug research reveals two ways to generalize, namely inductive generalization using statistical inference from samples, and analogic generalization using similarity between cases. Both are supported by abductive inference using mechanistic explanations of phenomena observed in the simulations. Illustrations of these inferences both in drug research and empirical RE research are given. Next, four kinds of methods for empirical RE technology validation are given, namely expert opinion, single-case mechanism experiments, technical action research and statistical difference-making experiments. A series of examples from empirical RE will illustrate the use of these methods, and the role of inductive generalization, analogic generalization, and abductive inference in them. Finally, the four kinds of empirical validation methods are compared with lists of validation methods known from empirical software engineering. The lists are combined to give an overview of some of the methods, instruments and data analysis techniques that may be used in empirical RE.
This paper explores why organizations do not adopt CMMI (Capability Maturity Model Integration), by analysing two months of sales data collected by an Australian company selling CMMI appraisal and improvement services. The most frequent reasons given by organizations were: the organization was small; the services were too costly, the organization had no time, and the organization was using another SPI approach. Overall, we found small organizations not adopting CMMI tend to say that adopting it would be infeasible, but do not say it would be unbeneficial. We comment on the significance of our findings and research method for SPI research.
Technology transfer in software engineering involves more than a new idea or evidence that it works. This paper illustrates how technology transfer requires a good idea, the generation of evidence, analysis of that evidence, good packaging and support, and careful consideration of the audience for the technology. By learning from other disciplines, we present a model of technology transfer that can be tailored to a particular organisation's needs.
Software process improvement (SPI) is challenging, particularly for small and medium sized enterprises. Most existing SPI frameworks are either too expensive to deploy, or do not take an organizations’ specific needs into consideration. There is a need for light weight SPI frameworks that enable practitioners to base improvement efforts on the issues that are the most critical for the specific organization.This paper presents a step-by-step guide to process assessment and improvement planning using improvement framework utilizing light weight assessment and improvement planning (iFLAP), aimed at practitioners undertaking SPI initiatives. In addition to the guide itself the industrial application of iFLAP is shown through two industrial cases. iFLAP is a packaged improvement framework, containing both assessment and improvement planning capabilities, explicitly developed to be light weight in nature. Assessment is performed by eliciting improvements issues based on the organization’s experience and knowledge. The findings are validated through triangulation utilizing multiple data sources. iFLAP actively involves practitioners in prioritizing improvement issues and identifying dependencies between them in order to package improvements, and thus establish a, for the organization, realistic improvement plan. The two cases of iFLAP application in industry are presented together with lessons learned in order to exemplify actual use of the framework as well as challenges encountered.
In this paper we present components of a newly developed software process improvement model that aims to represent key practices in requirements engineering (RE). Our model is developed in response to practitioner needs highlighted in our empirical work with UK software development companies. We have now reached the stage in model development where we need some independent feedback as to how well our model meets our objectives. We perform this validation through involving a group of software process improvement and RE experts in examining our RE model components and completing a detailed questionnaire. A major part of this paper is devoted to explaining our validation methodology. There is very little in the literature that directly relates to how process models have been validated, therefore providing this transparency will benefit both the research community and practitioners. The validation methodology and the model itself contribute towards a better understanding of modelling RE processes.
Both software organisations and the academic community are aware that the requirements phase of software development is in need of further support. We address this problem by creating a specialised Requirements Capability Maturity Model (R-CMM1). The model focuses on the requirements engineering process as defined within the established Software Engineering Institute’s (SEI’s) software process improvement framework. Our empirical work with software practitioners is a primary motivation for creating this requirements engineering process improvement model. Although all organisations in our study were involved in software process improvement (SPI), they all showed a lack of control over many requirement engineering activities.
This paper describes how the requirements engineering (RE) process is decomposed and prioritised in accordance with maturity goals set by the SEI’s Software Capability Maturity Model (SW CMM). Our R-CMM builds on the SEI’s framework by identifying and defining recommended RE sub-processes that meet maturity goals. This new focus will help practitioners to define their RE process with a view to setting realistic goals for improvement.
ArtREPI online resources
- Méndez Fernández
A Meta Model for Artefact-Orientation: Fundamentals and Lessons Learned in Requirements Engineering
- D Méndez Fernández
- B Penzenstadler
- M Kuhrmann
- M Broy
Méndez Fernández, D. and Penzenstadler, B. and Kuhrmann, M. and Broy, M.:
A Meta Model for Artefact-Orientation: Fundamentals and Lessons Learned in
Requirements Engineering. In: MoDELS. (2010)
- S Beecham
- T Hall
- A Rainer
Beecham, S., Hall, T., Rainer, A.: Software Process Improvement Problems in
Twelve Software Companies: An Empirical Analysis. EMSE (2003)