Conference PaperPDF Available

Implementation of a Software Quality Improvement Project in an SME: A Before and After Comparison

Abstract and Figures

Software process management is a crucial task in small and medium size enterprises (SME) due to scarce resources, small development teams and heavy workload.We have worked with an SME to institutionalize process improvement practices, aligned with CMMI maturity levels2 and 3. Main goals of this project were to achieve effective resource management and increase the product quality with fewer defects. The project had two phases: Phase I conducts an ¿as-is analysis¿ in order to set well-defined processes.Phase II measures the software reliability in terms of time/effort allocation and defect rates. Our before and after analysis showed that: a) time allocation for requirements,coding and testing steps improved from (4%, 31%, 65%) to(47%, 20%, 33%), b) defect rates decreased from 11% to6.5%, c) estimated testing effort also decreased from 47%to 30%.
Content may be subject to copyright.
A preview of the PDF is not available
... As noted in Table 2, in all studies (Wang et al., 2020b;Lin et al., 2020;Kumar and Mishra, 2016;Puri-Jobi, 2015;Hilton et al., 2017;Lee and Hwang, 2012;Collins and de Lucena, 2012;Tosun et al., 2009;Williams et al., 2009;Berner et al., 2005) that have observed how test automation maturity affects product quality, the findings verified the positive impact. These studies view product quality as to which extent a software product free from defects during the quality assurance process or in the production environment. ...
... It is different from scholars who defined product quality as conformance to a standard or to which extent a software product bears on its ability to satisfy given needs (Int'l Standards Organization, 2001;García-Mireles et al., 2015). Studies from Wang et al. (2020b), Hilton et al. (2017), Lee and Hwang (2012), Berner et al. (2005), Puri-Jobi (2015) and Tosun et al. (2009) reported that better product quality can be achieved after improving test automation maturity against certain standard best practices. ...
... An experience study (Ramler et al., 2014) reported that the direct effect of test automation maturity on test automation effort is negative. However, a different school of viewpoint reported by studies from Kumar andMishra (2016), Puri-Jobi (2015), and Tosun et al. (2009) views that, test automation effort will be reduced after improving the level of test automation maturity and product quality after a certain period of time -even though the considerable effort is required to develop and execute rigorous automated tests in attaining high product quality, the later effort savings may offset the invested effort. ...
Article
Full-text available
The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software repositories of the same projects. The main results of regression analysis reveal that, higher levels of test automation maturity are positively associated with higher product quality (p-value=0.000624) and shorter release cycle (p-value=0.01891); There is no statistically significant evidence of increased test automation effort due to higher levels of test automation maturity and product quality. Thus, we conclude that, a potential benefit of improving test automation maturity (using standard best practices) is product quality improvement and release cycle acceleration in the CI context of open source projects. We encourage future research to extend our findings by adding more datasets with different programming languages and CI tools, closed source projects, and large-scale industrial projects. Our recommendation to practitioners (in the similar CI context) is to utilize standard best practices to improve test automation maturity.
... In the case of Mozilla Firefox, there is no significant association between release cycle frequency and product quality. [39] 2009 A software quality improvement project within a health care company in Turkey Experience report Test automation maturity; product quality; test automation effort ...
... [30] 2009 Test automation maturity achieved by using several standard best practices (define a good test automation strategy, carefully design and reuse automated tests, and build a testable SUT) can lead to shorter release cycles and better product quality. Table 2, in all studies [27,33,34,35,28,37,38,39,30,32] that have observed how test automation maturity affects product quality, the findings verified the positive impact. These studies view product quality as to which extent a software product free from defects during the quality assurance process or in the production environment. ...
... It is different from scholars who defined product quality as conformance to a standard or to which extent a software product bears on its ability to satisfy given needs [40,41]. Studies [27,28,37,32,35,39] reported that better product quality can be achieved after improving test automation maturity against certain standard best practices. ...
Preprint
Full-text available
The popularity of continuous integration (CI) is increasing as a result of market pressure to release product features or updates frequently. The ability of CI to deliver quality at speed depends on reliable test automation. In this paper, we present an empirical study to observe the effect of test automation maturity (assessed by standard best practices in the literature) on product quality, test automation effort, and release cycle in the CI context of open source projects. We run our test automation maturity survey and got responses from 37 open source java projects. We also mined software repositories of the same projects. The main results of regression analysis reveal that, higher levels of test automation maturity are positively associated with higher product quality (p-value=0.000624) and shorter release cycle (p-value=0.01891); There is no statistically significant evidence of increased test automation effort due to higher levels of test automation maturity and product quality. Thus, we conclude that, a potential benefit of improving test automation maturity (using standard best practices) is product quality improvement and release cycle acceleration in the CI context of open source projects. We encourage future research to extend our findings by adding more datasets with different programming languages and CI tools, closed source projects, and large-scale industrial projects. Our recommendation to practitioners (in the similar CI context) is to utilize standard best practices to improve test automation maturity.
... [16] 2. Working with tight budgets and resources, which makes the deliverables, depends on the heroic efforts of a few people. [17] 3. Late delivery and Low Productivity, cause a lot of customer complaints, and too much rework. ...
... However, their response time for a change request is long and sometimes the level of quality of services is poor. [17] 5. The employee may play multiple roles in the company, for example a programmer might play the role of a technical architect, developer and tester simultaneously. ...
... [16] 6. Heavy reliance on engineers rather than processes. [17] 7. ...
... At present, the majority of the software companies are small and medium sized companies with 10-50 employees. These SMEs face many challenges to develop the software to a reasonable quality and acceptable standard such as project management, providing training to a new employee, conducting peer reviews and measurement that are quite common in small organizations [2,3]. ...
... Then a second person, probably the head of the project team will test the software after integrating all the modules. However, this is not enforced by any procedure [3]. Incidentally, there are no testing scripts made available and no testware is developed, because most of the SMEs do not apply any formal testing techniques. ...
... There are no proper procedures or methods adopted for the software testing. Since the SME's main objective is to satisfy its stakeholders [3], this constraint resulted in giving too little opportunity for the developers to test the task. Besides that, last minute addition of functionality can also harm the development process. ...
Chapter
Full-text available
In the field of IT, the software industry recognizes the value of the small and medium software (SME’s) enterprises in contributing valuable products and services to the economy. In SME’s the quality of the software is increasingly become a concerned subject. The suitable use of testing standards was not written for the development organizations having less than 25 employees. Currently, most of the SME’s do not follow any standards and do not adopt any well-structured testing methods. In this research paper, the challenges faced by the SME’s towards software testing in order to produce quality products are being addressed. Various testing processes and other factors associated with software quality are analyzed and discussed in the literature review to identify the research gaps in the existing approach. In this research paper, a new framework has been proposed using Exploratory Testing that has not been used in the context of SME’s which aims to improve software quality.
... SSCs 17 [23], [24], [55], [58], [59], [61], [65], [67], [75], [83], [96], [97], [101], [106], [113], [122], [125] 4 SME 16 [56], [57], [63], [66], [76], [85], [86], [93]- [95], [104], [105], [108], [110], [118], [123], [124] (c) Framework/standards 1 None 36 [19], [23], [32], [35], [36], [63], [66], [68], [70], [78], [81], [83]- [87], [93], [94], [96]- [102], [104], [109], [118]- ...
Article
Full-text available
Small software companies face numerous challenges of complexity, unstructured software development processes and scarce resources. This notwithstanding, the companies have dominated the software market by 80 percent. The practice and products of these companies are still persistently marred by quality issues arising from the processes, with evidence indicating that process tools do not fit the unique contexts in which they operate. Significant strides have been made in transforming software development practice; however, the challenges are still evidently apparent. Hence the need to establish how knowledge areas are applied in process practice, understand the context of software development and its implication in practice, how process tools are utilised in practice and evaluate quality of research in software literature The researchers undertook a systematic mapping study to determine the state of practice in empirical literature on software engineering of SSCs by examining and classifying 1096 publications. Other than the finding that research quality was low and affecting generalisation and transferability, the results also revealed interesting findings which we finally consolidated and integrated to develop two contributions (i) a software development process adoption theoretical framework which provides new insights of understanding software development and (ii) a 3-point guideline for research quality.
... In addition, it is necessary to have good documentation from each employee. So, if there are additions, developments, changes to a feature requested by the customer, then the work does not have to be done by the same person [37]. ...
Article
Full-text available
A software house, that established in 2005 based in Indonesia, got 31 projects in 2019. By the end of year, Project Management Officer released documents to inform company’s project health. There are 14 projects confirmed late, 6 projects on time and 11 projects scheduled complete on the next year. That late projects cause serious problem like loses revenue and gets disrupted of company’s cash flow. Based on the root cause analysis, it found that no standardization of software development process in the company. Before designing the standardization to improve process, we need to analyze the obstacles that might be happened. Therefore, this study aims to identify the obstacles on software process improvement in software house. We performed a systematic literature review to determine the obstacles, then we do empirical research to 58 employees on company’s development department to sort the priority of obstacles in the company. From the systematic literature review, we found studies that relevant and there are 13 obstacles of software process improvement, then from the empirical research we got top three obstracles. We also proposed recommendations to solve that obstacles.
... For example, the study by Díaz-Ley et al. [9] reported the experiences of a Spanish SME in implementing an MP and reported that the practitioners could now objectively evaluate the trade-off between on-time releases and software product reliability. Tosun et al. [50] collaborated with a Turkish healthcare SME to institutionalize process improvement practices, and reported improvements in the organization's time allocation for requirements, coding, and testing steps. Furthermore, the authors found that defect rates as well as testing effort estimation decreased. ...
Article
Full-text available
Context: Agile software development has become commonplace in software development companies due to the numerous benefits it provides. However, conducting Agile projects is demanding in Small and Medium Enterprises (SMEs), because projects start and end quickly, but still have to fulfil customers’ quality requirements. Objective: This paper aims at reporting a practical experience on the use of metrics related to the software development process as a means supporting SMEs in the development of software following an Agile methodology. Method: We followed Action-Research principles in a Polish small-size software development company. We developed and executed a study protocol suited to the needs of the company, using a pilot case. Results: A catalogue of Agile development process metrics practically validated in the context of a small-size software development company, adopted by the company in their Agile projects. Conclusions: Practitioners may adopt these metrics in their Agile projects, especially if working in an SME, and customise them to their own needs and tools. Academics may use the findings as a baseline for new research work, including new empirical studies.
... Exploratory Testing is an approach that helps to identify unknown bugs which are missed during traditional testing activities [13][14][15][16][17][18]. In agile development process, exploratory testing plays an important role which can also be combined with other analytical risk-based testing, model-based testing and regression-averse testing [19]. ...
Chapter
Software process improvement (SPI) is the approach to understand the software development process lifecycle and implement necessary changes to achieve a high-quality, maintainable product. The software industry is comprised of small and medium-sized enterprises that adopt agile models as their preferred model of development. The Capability Maturity Model Integration (CMMI) model for software process improvement and quality is used to address aspects of software quality and maturity that Scrum was not initially designed to consider. CMMI is used for its reliable practices and consideration of many aspects of software development in process, project and product. After consideration of literature, three process areas are selected to address necessary elements in Scrum used by software developers in offshoring destinations. The practices are selected based on their practicality in small, agile settings and the ability to be incorporated into Scrum activities without disrupting the models sprint cycles. This opens the way for smaller enterprises to create produce quality.
Conference Paper
Full-text available
In this paper, we present a defect prediction model based on ensemble of classifiers, which has not been fully explored so far in this type of research. We have conducted several experiments on public datasets. Our results reveal that ensemble of classifiers considerably improve the defect detection capability compared to Naive Bayes algorithm. We also conduct a cost-benefit analysis for our ensemble, where it turns out that it is enough to inspect 32% of the code on the average, for detecting 76% of the defects.
Article
Small software development companies need it to ensure they stay on course and are able to respond to the market's ebbs and flows.
Conference Paper
The United States Air Force sponsored research within the Department of Defense (DoD) so~are development community to determine the applicability of the Sojiware Engineering Institute’s (SEI) Capability Maturity Model (CW) for Sojlware to small businesses and small softiare organizations. The research found that small businesses are faced not only with a lack of resources and funds required to implement many of the practices stated in the CM, but also with the task of basing their process improvement initiatives on practices that do not apply to a small business and small sojlware organization. This paper discusses, from industryh perspective, why small businesses and organizations are experiencing dif]culties implementing CMM-based process improvement programs and how they are tailoring their approach to the GUM to meet their quality goals.
Conference Paper
This paper reports on the construction and validation of fault- proneness prediction models in the context of an object-oriented, evolving, legacy system. The goal is to help QA engineers focus their limited verification resources on parts of the system likely to contain faults. A number of measures including code quality, class structure, changes in class structure, and the history of class- level changes and faults are included as candidate predictors of class fault-proneness. A cross-validated classification analysis shows that the obtained model has less than 20% of false positives and false negatives, respectively. However, as shown in this paper, statistics regarding the classification accuracy tend to inflate the potential usefulness of the fault-proneness prediction models. We thus propose a simple and pragmatic methodology for assessing the cost-effectiveness of the predictions to focus verification effort. On the basis of the cost-effectiveness analysis we show that change and fault data from previous releases is paramount to developing a practically useful prediction model. When our model is applied to predict faults in a new release, the estimated potential savings in verification effort is about 29%. In contrast, the estimated savings in verification effort drops to 0% when history data is not included.