No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Supporting Agile Teams with a Test Analytics Platform: A Case Study
Abstract
Continuous improvement, feedback mechanisms and automated testing are cornerstones of agile methods. We introduce the concept of test analytics, which brings these three practices together. We illustrate the concept with an industrial case study and describe the experiments run by a team who had set a goal for itself to get better at testing. Beyond technical aspects, we explain how these experiments have changed the mindset and the behaviour of the team members. We then present an open source test analytics platform, later developed to share the positive learnings with the community. We describe the platform features and architecture and explain how it can be easily put to use. Before the conclusions, we explain how test analytics fits in the broader context of software analytics and present our ideas for future work.
... In total, 540 studies were retrieved. The study selection steps (next described) resulted in the following numbers: Pre-selection: 47 studies; Final selection: 9 studies [1,7,11,14,15,19,24,26,29]; and Backward snowballing: 6 studies [3,6,8,21,25,27]. The final set is composed of 15 studies [1, 3, 6-8, 11, 14, 15, 19, 21, 24-27, 29]. ...
... Liechti et al. [15] discussed and presented an industrial case study about test analytics, which comprehend continuous improvement, feedback mechanisms, and automated testing. The authors defined test analytics as "analytics on test-related data in order to give actionable insights about product quality and agile practices, with the goal to support a continuous improvement process". ...
... The educational context was considered in 10 studies: [1, 3, 6-8, 24-27, 29]. The industrial context was considered in 8 studies: [7,11,14,15,19,21,24,26]. With a few exceptions, all approaches can be applied to both educational and industrial contexts. ...
Context: Testing is a fundamental activity in the software development cycle. Revealing software faults is its main objective. Despite that, testing is considered unpleasant, dull and tedious. As a result, there is a lack of expertise among professionals while many projects fail. Gamification is a promising way to address testing issues; it is a new trend being used mostly to increase motivation, engagement and performance with the use of game elements in non-game contexts. Objective: To describe results of a study that aimed to characterize how gamification has been explored to support software testing. Method: The studies that compose our baseline for analysis and discussion were obtained through a systematic mapping carried out following a research protocol. To retrieve relevant literature, we applied automatic search and backward snowballing. At the end, we selected 15 studies that we analyzed and classified according to six perspectives: application context, used gamifcation elements, gamification goals, testing techniques, testing levels, and testing process phases. Results: The most used gamification elements are points, leader boards, and levels, and unit testing and functional testing are the level and technique most addressed in the studies, respectively. Conclusion: Gamification is a rising research topic, especially in the software testing field. The increasing interest for gamification has the potential do lead to positive outcomes. The map presented in this paper can be a useful resource for the identification of gaps and for triggering new research initiatives.
... The development team implemented agile practices to ensure frequent delivery of deployable software, which was then passed to the quality assurance (QA) team for testing and validation. At present, several organisations are moving towards a holistic team approach [44]. The software development process should therefore involve cross-functional teams consisting of development, quality assurance and operations departments. ...
... The software development process should therefore involve cross-functional teams consisting of development, quality assurance and operations departments. It reduces hand-offs between teams and may lead to faster development cycles and a greater sense of ownership of the product [44]. However, when working under tight deadlines, features are prioritized over quality, and lack of documentation creates a software debt [3][4][5][6][7][8][9][10][11][12]. ...
... Such activities range from mutation testing to the management of the application development lifecycle [4]. Although gamification has primarily been proposed in the educational field, encouraging results have also been provided by industrial case studies [5]. ...
... Few works in the literature have presented evaluations of gamified mechanics in industrial contexts. Liechti et al. [5] presented a case study about continuous improvement, automated testing and feedback mechanisms for developers. The platform integrated an element of gamification in live graphical feedback of users' performance in the assigned task. ...
italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Context
: Gamification appears to improve enjoyment and quality of execution of software engineering activities, including software testing. Though commonly employed in industry, manual exploratory testing of web application GUIs was proven to be mundane and expensive. Gamification applied to that kind of testing activity has the potential to overcome its limitations, though no empirical research has explored this area yet.
Goal : Collect preliminary insights on how gamification, when performed by novice testers, affects the effectiveness, efficiency, test case realism, and user experience in exploratory testing of web applications.
Method : Common gamification features augment an existing exploratory testing tool: Final Score with Leaderboard, Injected Bugs, Progress Bar, and Exploration Highlights. The original tool and the gamified version are then compared in an experiment involving 144 participants. User experience is elicited using the Technology Acceptance Model (TAM) questionnaire instrument.
Results : Statistical analysis identified several significant differences for metrics that represent the effectiveness and efficiency of tests showing an improvement in coverage when they were developed with gamification. Additionally, user experience is improved with gamification.
Conclusions : Gamification of exploratory testing has a tangible effect on how testers create test cases for web applications. While the results are mixed, the effects are most beneficial and interesting and warrant more research in the future. Further research shall be aimed at confirming the presented results in the context of state-of-the-art testing tools and real-world development environments.</p
... Currently, various tools provide structure on top which datadriven improvement processes can be implemented. However, due the perceived complexity and effort required to set up such tools and to establish a measurement program, agreed with the urgency of the product delivery, lack of time, and others reasons, many agile teams end up not systematically using metrics to track product and process performance [8]. There is a lack tools to ease the adoption of software analytics, but also a lack of approaches to effectively change the habits within agile teams towards data-driven decision making. ...
... For Agile teams, the support of different visualization techniques and tools throughout the software development process is crucial to foster awareness and communication. Additionally, this factor can have a positive impact on the sense of purpose [8] when the professionals follow the evolution of their actions within a cycle of continuous improvement. ...
... Albeit the primary application of gamification mechanics is still used primarily in the educational field [7], there is a growing interest by practitioners in implementing them in industrial contexts and tooling [8]. Case studies in the literature have documented encouraging outcomes by such adoption [9]. ...
Software testing is a critical activity in the software development process. Several techniques have been proposed, addressing different levels of granularity from low-level unit testing to higher-level exploratory testing through the software’s graphical user interface (GUI). In modern software development, most test cases are obtained by automated test generation. However, while automation generally achieves high coverage in code-level white-box testing, it does not always generate realistic sequences of interactions with the GUI. By contrast, manual exploratory testing has survived as a costly, error-prone, and tedious yet crucial activity. Gamification is seen as an opportunity to increase user satisfaction and engagement while performing testing activities. It could also enable and encourage crowdsourced testing tasks. The purpose of the study described in this chapter is to provide a framework of gamification mechanics and dynamics that can be applied to the practice of manual exploratory GUI testing. We provide an implementation of the framework as an extension of an existing manual exploratory GUI testing for Web applications, and we provide a preliminary evaluation of the gamified tool in terms of provided efficiency, effectiveness, and user experience. Our results show that the gamified solution makes the testers obtain test suites with higher coverage while reducing slightly the number of bugs signalled while traversing the applications under test. The gamified tool also was considered to provide a positive user experience, and the majority of participants expressed their willingness to use such instruments again in the future. As future work, we foresee the implementation of the framework in a stand-alone tool and in-depth empirical experiment to evaluate quantitatively the benefits and drawbacks provided by such mechanics in real testing scenarios.
... The implementation of agile techniques may occasionally be perceived by developers as an attempt to micromanage (Marchenko & Abrahamsson, 2008). In pseudo agile environments, metrics are often perceived as a threat, make people uncomfortable, and can do more harm than good (Liechti et al., 2017). Selfintensification, for instance, could happen if these problems are not considered when using an agile type of work (Drury et al., 2012;Porschen, 2012). ...
Background: Agile practices have gained popularity in the 21st century. There is also a growing body of research on agile methods. While some earlier research on agile practices and organizational culture exists with the assumption that the optimal combination of agile practices and organizational culture exists, we examine how agile methods and organizational culture interact and are mutually adjusted.
... Practicing a combination of visual and test analytics could help to increase the use of such methods in decision making, improve operational performance and deliver outcomes efficiently. Test analytics is defined as analytics on testrelated data in order to give actionable insights about product quality and development practices, with the goal to support a continuous improvement process [13]. ...
Software testing is an essential activity in all software projects. The key issue in testing is determining the sufficiency of tests and, traditionally this has been done by using the Software Reliability Growth Models (SRGMs). However, SRGMs are not applicable when there are no stabilization phases as required by most software reliability models. Test-Defect Coverage Analytics Model (TDCAM) has been proposed to address this problem. This paper proposes the application of visual analytic techniques as an approach for supporting informed decision making in deciding the sufficiency of tests. Visual analytics research considers interactive visualization as the common platform for combining various computational data analysis techniques to support the analytical reasoning process. We presented four visual representations of TDCAM to demonstrate how analytical models have been applied to indicate the adequacy of tests in relation to sufficient and efficient test coverage. The techniques provide an effective and generally applicable test estimation on the basis of a general trend that higher test coverage correlates with higher probability of detecting more defects.
... Sugere-se analisar também a recuperação destes dados, bem como a coleta dos dados. Outras métricas também podem ser usadas como exemplo a medição e o impacto (Liechti et al., 2017). ...
O objetivo do artigo é identificar quais elementos de coletivos são desenvolvidos com a
gamificação nas organizações. Após a análise de 22 artigos por meio de uma revisão
sistemática da literatura, construiu-se um levantamento da base teórica sobre coletivos e
gamificação, bem como uma agenda de futuras pesquisas. O estudo fornece uma visão ampla sobre os coletivos, olhando para os elementos que foram relacionados aos coletivos como aprendizagem, avaliação, bem-estar social, colaboração, competição, motivação dentre outros. Finalmente, há evidências que a gamificação e os coletivos possuem estudos teóricos nas diversas associações dos coletivos, entretanto, apresentam-se lacunas voltadas aos estudos empíricos e aplicadas em equipes reais e de formação natural.
... It gave them a real sense of mastery. More details about the platform that was developed to support the team are described in [7]. ...
One of the key principles in agile development is that the team should use continuous feedback cycles to maintain awareness both about the product, the process and the involved people. In healthy agile environments, people see a lot of value in the collection of metrics and in their collaborative analysis to support a continuous improvement process. In pseudo agile environments, metrics are often perceived as a threat and they make people uncomfortable. In theses conditions, they can do more harm than good. In this paper, we argue that the acceptance of a data-driven continuous improvement process is heavily influenced by the nature of the channels through which the metrics are published. We argue for the need to go beyond typical dashboards and propose additional engagement channels. We discuss the notions of awareness and story telling and illustrate them with various experiments conducted with agile teams.
Organizational debt (OD) is a major challenge to software organizations that seek to maintain agility, adaptability and sustainable competitiveness in the dynamic business environment. OD can be refers to suboptimal decisions, outdated procedures, misaligned structures and cultural barriers that limit an organization?s ability to adapt and innovate quickly. This multifaceted process includes several factors, including ineffective workflows, knowledge silos, cultural issues, and inadequate resource utilization. When organizations are focused on short-term gains at the expense of long-term organizational health, the symptoms of organizational dysfunction may manifest themselves as reduced output, reduced quality and customer satisfaction. This study aims at assessing the extent of knowledge, factors, and consequences of organizational maladjustment in software organizations. A survey performed in three organizations identified several highly visible issues such as complex code, inconsistent UI, unclear requirements, and outdated processes. These themes often emerge due to exponential growth, prioritizing speed over quality, lack of cooperation and coordination, and outdated processes. The adverse effects of OD are comparable to technical debt, affecting the maintainability, user experience, and project management. This study also offers strategies for identifying, assessing, and mitigating OD through a combination of quantitative metrics, user feedback, and interdepartmental collaboration. By fostering a culture of continuous improvement, open communication, and cross-functional alignment, organizations can proactively address OD and create an environment conducive to innovation, quality, and customer-centricity.
In an era of digital transformation, where in the last decades has increased the need of embrace change rapidly, it’s important that organizations make their transformation in the path of continue being competitive. To this, organizations have started adopting agile methodologies so that they can respond faster to the daily change. DevOps allows companies to gain a leverage in reach their goals and increasing their velocity in react to change. This way, measure DevOps teams’ performance is a bigger and bigger necessity of the organizations so that they can identify improvement points. A Systematic Literature Review was performed and were identified 13 Key Performance Indicators were identified. Among them, those referred to the quality and results of testing are the most referred and implemented.KeywordsDevOpsTeamsPerformance
Context
Over the last decade, Agile methods have changed the software development process in an unparalleled way and with the increasing popularity of Big Data, optimizing development cycles through data analytics is becoming a commodity.
Objective
Although a myriad of research exists on software analytics as well as on Agile software development (ASD) practice on itself, there exists no systematic overview of the research done on ASD from a data analytics perspective. Therefore, the objective of this work is to make progress by linking ASD with Big Data analytics (BDA).
Method
As the primary method to find relevant literature on the topic, we performed manual search and snowballing on papers published between 2011 and 2019.
Results
In total, 88 primary studies were selected and analyzed. Our results show that BDA is employed throughout the whole ASD lifecycle. The results reveal that data-driven software development is focused on the following areas: code repository analytics, defects/bug fixing, testing, project management analytics, and application usage analytics.
Conclusions
As BDA and ASD are fast-developing areas, improving the productivity of software development teams is one of the most important objectives BDA is facing in the industry. This study provides scholars with information about the state of software analytics research and the current trends as well as applications in the business environment. Whereas, thanks to this literature review, practitioners should be able to understand better how to obtain actionable insights from their software artifacts and on which aspects of data analytics to focus when investing in such initiatives.
Findings in several countries show that software testing considered as unattractive work. This may result in low motivation that can lead to lousy testing practice and eventually software defects. For which, we need a way to increase motivation software tester by knowing what motivate and de-motivate software tester. This study presents a systematic literature review that aggregates, summarizes, and discusses the results of 15 relevant primary studies (PSs) focusing about motivation and demotivation factors that affect software testing. The included PSs were selected using inclusion and exclusion criteria applied to relevant articles published between 2015 to 2020. We analyzed the PSs based on a set of classification criteria, including software quality attributes and measures, refactoring scenarios, evaluation approaches, datasets, and impact results. Our study literatur review result show that good management , gamification and time pressure affect motivation factors. While demotivation factor for software testing are lack of good management,recognition and knowledge about long-term benefit of testing. Outcomes of this study are expected to help practitioners know which motivational factor and demotivation factors that affect software testing.
The guest editors of this special issue of IEEE Software invited submissions that reflected the benefits (and drawbacks) of software analytics, an area of explosive growth. They had so many excellent submissions that they had to split this special issue into two volumes--you'll see even more content in the September/October issue. They divided the articles on conceptual grounds, so both volumes will feature equally excellent work. The Web extra at http://youtu.be/nO6X0azR0nw is a video interview in which IEEE Software editor in chief Forrest Shull speaks with Tim Menzies about the growing importance of software analytics.
Behaviour Driven Development (BDD) has gained increasing attention as an agile development approach in recent years. However, characteristics that constitute the BDD approach are not clearly defined. In this paper, we present a set of main BDD characteristics identified through an analysis of relevant literature and current BDD toolkits. Our study can provide a basis for understanding BDD, as well as for extending the exisiting BDD toolkits or developing new ones.
Coding and testing are two activities that are tightly intermingled in agile software development, requiring developers to frequently shift between production code and test artifacts. Unfortunately, links between these artifacts are typically implicitly present in the source code, forcing developers towards time consuming code inspections. In this work, we evaluate the potential of six traceability resolution strategies involving test naming and design conventions, static call graphs, call behavior before asserts, lexical analysis and version log mining to make the relation between developer test cases and units under test explicit. The results show that test conventions yield highly accurate results, yet in their absence capturing the destination type of calls right before assert statements appears as a valuable alternative. Combining these strategies allows the user to find a balance between improved applicability and accuracy.
Engineering software systems is a multidisciplinary activity, whereby a number of artifacts must be created - and maintained - synchronously. In this paper we investigate whether production code and the accompanying tests co- evolve by exploring a project's versioning system, code coverage reports and size-metrics. Our main aim for studying this co-evolution is to create awareness with developers and managers alike about the testing process that is followed. We explore the possibilities of our technique through two open source case studies and observe a number of different co-evolution scenarios. We evaluate our results both with the help of log-messages and the original developers of the software system.
One of the key principles in agile development is that the team should use continuous feedback cycles to maintain awareness both about the product, the process and the involved people. In healthy agile environments, people see a lot of value in the collection of metrics and in their collaborative analysis to support a continuous improvement process. In pseudo agile environments, metrics are often perceived as a threat and they make people uncomfortable. In theses conditions, they can do more harm than good. In this paper, we argue that the acceptance of a data-driven continuous improvement process is heavily influenced by the nature of the channels through which the metrics are published. We argue for the need to go beyond typical dashboards and propose additional engagement channels. We discuss the notions of awareness and story telling and illustrate them with various experiments conducted with agile teams.
Continuous feedback is one of the most important concepts in agile development. We argue for the need to increase the awareness that the team maintains of various facets of the development activity. We then introduce iFLUX, an event-driven middleware designed for the Web of Things. We explain how it provides a platform that facilitates the creation of augmented workplaces that connect various information sources with physical displays, also known as information radiators.
This chapter deals with the ethics of persuasive technology. Ethical issues are especially prominent when computer technology uses novelty as a distraction to increase persuasion. When dealing with a novel experience, people not only lack expertise but they are distracted by the experience, which impedes their ability to focus on the content presented. Being in a novel situation can make people more vulnerable because they are distracted by the newness or complexity of the interaction. When it comes to persuasion, computers also benefit from their traditional reputation of being intelligent and fair, making them seem credible sources of information and advice. Another advantage of computers is persistence. Unlike human persuaders, computers don't get tired; they can implement their persuasive strategies over and over.
This paper presents a new model for understanding human behavior. In this model (FBM), behavior is a product of three factors: motivation, ability, and triggers, each of which has subcomponents. The FBM asserts that for a person to perform a target behavior, he or she must (1) be sufficiently motivated, (2) have the ability to perform the behavior, and (3) be triggered to perform the behavior. These three factors must occur at the same moment, else the behavior will not happen. The FBM is useful in analysis and design of persuasive technologies. The FBM also helps teams work together efficiently because this model gives people a shared way of thinking about behavior change.
CORE ...........................................................................................................................204 Deep Models .......................................................................................................................................205 Refactoring and Distillation ................................................................................................................205 17. Large-Scale Structure.................................................................................................................206 EVOLVING ORDER .........................................................................................................................208 SYSTEM METAPHOR [BECK 2000] .................................................................................................209 PLUGGABLE COMPONENTS.............................................................................................................210 ABSTRACT DOMAIN FRAMEWORK .................................................................................................212 RESPONSIBILITY LAYERS...............................................................................................................212 Large-Scale Structure, Unification Contexts, and Distillation ............................................................223 Refactoring Toward a Fitting Structure...............................................................................................225 Architecture, Architecture Teams, and Large-Scale Structure ............................................................227 18. Game Plans ..................................................................................................................................230 Looking Forward.....
Building a DevOps culture
- M Walls
M. Walls, Building a DevOps culture. O'Reilly Media, Inc., 2013.
Continuous delivery: reliable software releases through build, test, and deployment automation
- J Humble
- D Farley
J. Humble and D. Farley, Continuous delivery: reliable software releases
through build, test, and deployment automation. Pearson Education,
2010.
Game dock: design and implementation of a gamification platform
- Y Kammoun
Y. Kammoun, "Game dock: design and implementation of a gamification platform," University of Applied Sciences Western Switzerland,
Bachelor thesis, 2016.