ArticlePDF Available

Software Quality: How Much Does It Matter?

Authors:

Abstract and Figures

Interconnected computers and software systems have become an indispensable part of people’s lives in the period of digital transformation. Consequently, software quality research is becoming more and more critical. There have been multiple attempts to synthesise knowledge gained in software quality research; however, they were focused mainly on single aspects of software quality and did not structure the knowledge holistically. To fill this gap, we harvested software quality publications indexed in the Scopus bibliographic database. We analysed them using synthetic content analysis which is a triangulation of bibliometrics and content analysis. The search resulted in 15,468 publications. The performance bibliometric analysis showed that the production of research publications relating to software quality is currently following an exponential growth trend and that the software quality research community is growing. The most productive country was the United States, followed by China. The synthetic content analysis revealed that the published knowledge could be structured into six themes, the most important being the themes regarding software quality improvement by enhancing software engineering, advanced software testing and improved defect and fault prediction with machine learning and data mining.
Content may be subject to copyright.
Citation: Kokol, P. Software Quality:
How Much Does It Matter? Electronics
2022,11, 2485. https://doi.org/
10.3390/electronics11162485
Academic Editor: Maysam Abbod
Received: 6 July 2022
Accepted: 2 August 2022
Published: 10 August 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the author.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
electronics
Review
Software Quality: How Much Does It Matter?
Peter Kokol
Faculty of Electrical Engineering and Computer Science, University of Maribor, SI-2000 Maribor, Slovenia;
peter.kokol@um.si
Abstract:
Interconnected computers and software systems have become an indispensable part of
people’s lives in the period of digital transformation. Consequently, software quality research is
becoming more and more critical. There have been multiple attempts to synthesise knowledge
gained in software quality research; however, they were focused mainly on single aspects of software
quality and did not structure the knowledge holistically. To fill this gap, we harvested software
quality publications indexed in the Scopus bibliographic database. We analysed them using synthetic
content analysis which is a triangulation of bibliometrics and content analysis. The search resulted in
15,468 publications. The performance bibliometric analysis showed that the production of research
publications relating to software quality is currently following an exponential growth trend and that
the software quality research community is growing. The most productive country was the United
States, followed by China. The synthetic content analysis revealed that the published knowledge
could be structured into six themes, the most important being the themes regarding software quality
improvement by enhancing software engineering, advanced software testing and improved defect
and fault prediction with machine learning and data mining.
Keywords:
software engineering; software quality; knowledge synthesis; bibliometrics; synthetic
knowledge synthesis
1. Introduction
The digital transformation caused interconnected computers and software systems
to become an indispensable part of our lives and the necessary toll for performing their
daily personal and business obligations and activities [
1
]. To fulfil global user needs for
storing, retrieving and processing information, knowledge and wisdom, those systems
have to be supported by quality software, which should function correctly and reliably, be
easy, safe and fit to use, test, reuse and maintain, and finally to conform to stakeholders’
requirements. Therefore, software quality is not only one of the most important but also a
multidimensional attribute of computer software [24].
Consequently, there have been multiple attempts to synthesise knowledge gained in
software quality research; however, they were focused mainly on single aspects of software
quality, such as measurement [
5
], human-software interaction [
6
], empirical analysis of
Code smells and refactoring [
7
], design patterns [
8
]; quality models [
9
], human factors [
10
],
testing [
11
] and quality prediction [
12
,
13
] or to specific development approaches such as
agile [14].
Bibliometrics is another approach to synthesise knowledge [
15
]. Similarly to the above,
the only two bibliometrics studies concerned with software quality were focused on specific
aspects of software quality, one on defect prediction [
16
] and the other on code smells [
17
].
To close the gap regarding the lack of holistic knowledge synthesis studies of soft-
ware quality research, we performed a performance bibliometric analysis and synthetic
bibliometric mapping-based content analysis. We aimed to identify the most productive
countries and institutions, most prolific source titles, publication production trends and hot
topics, and structure the research content into themes and trends.
Electronics 2022,11, 2485. https://doi.org/10.3390/electronics11162485 https://www.mdpi.com/journal/electronics
Electronics 2022,11, 2485 2 of 11
The choice to focus on software quality was motivated by the belief that software
quality assurance is a crucial factor in software development for a myriad of stakeholders
such as software developers, theoreticians, practitioners and of course, software users. The
study can help them gain new insights into the topic, deepen their knowledge or inform
them about the trends and essential themes in software quality research. To the best of
our knowledge, a similar quantitative and qualitative bibliometric study that provides
an overview of the current state and development of software quality research from its
beginnings to the present has not been conducted so far. Therefore, we eliminate this
shortcoming through the present study and fill in the current gap.
2. Materials and Methods
Knowledge synthesis is an approach to deal with the explosive growth in research
literature production. Knowledge synthesis roots date back more than 120 years. However,
they became more popular in the 1960s [
18
] and even more commonly used toward the
end of the millennia with the emergence of the evidence-based paradigm [
19
,
20
]. Content
analysis is one of the more popular knowledge synthesis methods used in qualitative and
quantitative research. Its main advantages are that it is content-sensitive, highly flexible
and can be used to analyse many types of data in an inductive or deductive manner [21].
To enable the knowledge synthesis of several thousand or even ten thousand publica-
tions, Kokol et al. [
22
] triangulated descriptive bibliometrics, text mining and bibliometric
mapping and content analysis into Synthetic knowledge synthesis. In addition to semi-
automatic analysis of large corpora, the combination of the above approaches enables one
to combine quantitative and qualitative analysis of the content and production of research
publications. The qualitative part of the synthetic knowledge synthesis was performed
with the Algorithm presented below:
1.
Harvest the research publications concerning software quality. The corpus of retrieved
publications represents the content to analyse and the output of Step 1.
2.
Identify codes in the corpus using text mining and cluster them into an author’s
keyword landscape with bibliometric mapping. Authors’ keywords were selected as
codes since they most concisely present the content of a publication. The author’s
keyword landscape is the output from Step 2.
3.
Condense author keywords with similar meanings into codes for every single cluster
and analyse the links between codes in individual clusters, and then map them into
categories which form the output from Step 3.
4.
Analyse categories and name each cluster with an appropriate theme. The list of
themes is the final output of the qualitative analysis.
In Step 1, the publications were harvested from Scopus (Elsevier, Amsterdam, The
Netherlands), the largest abstract and citation database of peer-reviewed literature. The
corpus was formed on 14 April 2022, for the whole period covered by Scopus. We used the
search string “quality software” OR “quality of software” OR “software quality” in informa-
tion source titles, abstracts, and keywords. The reliability of the search was assessed using
recall (fraction of the documents that are relevant to the query that is successfully retrieved)
information retrieval functions using 20 important software quality publications and ten
eminent authors (19 publications and all authors were retrieved). The publications and
authors were selected based on the discussions with colleagues concerned with software
engineering research. Using Scopus built-in functions, we exported the following metadata
to a CSV formatted corpus file: publication titles, authors’ affiliations’ details, source title,
publication type, publishing year, author keywords and funding data.
Bibliometric mapping in Step 2 has been performed using VOSViewer software version
1.6.16 (Leiden University, The Netherlands). VOSViewer uses text mining to recognise
various text entities in publications such as terms, keywords and country names. Next, it
employs the mapping technique, Visualisation of Similarities (VoS), based on the co-word
analysis. Landscapes are displayed in various ways to present different aspects of the
Electronics 2022,11, 2485 3 of 11
research publications’ content [
23
]. Steps 3 and 4 were performed using the traditional
content analysis process [21].
3. Results and Discussion
3.1. Descriptive Bibliometrics
The search resulted in 15,468 publications. Among them were 9457 conference papers,
4845 journal articles, 401 conference reviews, 294 book chapters, 283 reviews, 74 editorials,
63 books, 31 short papers and 20 other types of publications.
The first paper indexed in Scopus was a conference paper titled Teaching software
development in a studio environment published in 1954 [
24
]. After that (Figure 1), the pro-
duction was low, not exceeding five publications per year till 1978 when the linear growth
trend began, followed by exponential growth starting in 2001. The peak productivity was
reached in 2019 with 1082 publications. The trend of the number of conference proceedings
and journal papers followed a similar pattern till 2001, then the trend split. The number of
journal papers followed a steady exponential trend reaching 400 journal articles in 2020,
while the number of conference papers had an explosive growth period from 2001 till
2007. After that, the number of conference papers remained steady till 2018 and peaked
in 2019 with 701 papers. According to the above trends, we can identify three milestones
in software quality research, namely: (1) The beginning of the seventies when the first
software crisis emerged [
25
]; (2) the end of the seventies when software quality models and
measurements started to gain in importance [
26
]; and finally (3) in the beginning of the
new century with the advent of agile programming [27].
Figure 1.
The dynamics of the softwarequality research expressed by the number of peer-reviewed publications.
Spatial Distribution and Productivity of Literature Production
The most prolific countries in software quality research according to the number of
publications are presented in Table 1. Among them, India, China and Brazil belong among
the 10 best countries to outsource software development [
28
]. The United States, Germany,
India and the United Kingdom are the countries with the largest number of software
engineers [
29
]. As expected, the most prolific institutions (Table 2) are primarily located in
most prolific countries.
Electronics 2022,11, 2485 4 of 11
Table 1. Most productive countries.
Country Number of Publications
United States 2917
China 1684
India 1436
Germany 961
Canada 734
Brazil 698
United Kingdom 626
Italy 510
Spain 428
Japan 427
Table 2. Most productive institutions.
Institution Number of Publications
Florida Atlantic University 227
Beihang University 127
Amity University 99
Peking University 95
Carnegie Mellon University 87
Universidade de São Paulo 86
Technical University of Munich 83
École de Technologie Supérieure 80
Chinese Academy of Sciences 79
Fraunhofer Institute for Experimental Software
Engineering IESE 75
The most prolific source titles (Table 3) are mainly conference and workshop pro-
ceedings. Most prolific journals are prestigious top software engineering journals with
high SNIP 2020 impact factors. SNIP “measures contextual citation impact by weighting
citations based on the total number of citations in a subject field, using Scopus data” [30].
Table 3. Most prolific journals.
Source Title Number of Publications SNIP 2020
Lecture Notes in Computer Science Including Subseries Lecture
Notes in Artificial Intelligence And Lecture Notes
In Bioinformatics
737 0.628
ACM International Conference Proceeding Series 394 0.296
Proceedings International Conference on Software Engineering
324 1.68
Communications In Computer and Information Science 226 0.32
Ceur Workshop Proceedings 221 0.345
Information And Software Technology 214 2.389
Software Quality Journal 191 1.388
Advances In Intelligent Systems and Computing 190 0.428
Journal Of Systems and Software 186 2.16
IEEE Software 152 1.934
According to the number of publications where a funding agency was mentioned in
the funding acknowledgement (Table 4), the most prolific funding agency is the National
Natural Science Foundation of China (n= 318), followed by the US National Science Foun-
dation (n= 182) and European Commission (n= 132). Based on funding acknowledgements
analysis, only 21.5% of publications were sponsored, which is far less than for software
engineering in general (n= 32.3%) [
31
]. That might indicate that software quality research
is underfunded and did not yet reach the status of other software engineering disciplines.
Electronics 2022,11, 2485 5 of 11
Most productive funding institutions are located in China and Europe, and surprisingly
not as expected by other bibliometrics performance attributes, in the United States.
Table 4. Most prolific funding agencies.
Funding Agency Number of Publications
National Natural Science Foundation of China 318
National Science Foundation (USA) 182
European Commission 132
Horizon 2020 Framework Programme (EU) 97
Conselho Nacional de Desenvolvimento
Científico e Tecnológico (Brazil) 89
Natural Sciences and Engineering Research
Council of Canada 77
Coordenação de Aperfeiçoamento de Pessoal
de Nível Superior (Brazil) 76
National Key Research and Development
Program of China 66
European Regional Development Fund 65
Japan Society for the Promotion of Science 61
3.2. Qualitative Synthetic Knowledge Synthesis
3.2.1. Text Mining and Bibliometric Mapping
The 15,468 publications from the corpus were analysed using VOSviewer software
(Step 2 of the algorithm). Text mining identified 19,084 author keywords. All keywords
emerging in at least 40 papers (116 authors’ keywords) were included in the bibliometric
mapping analysis. The resulting author keywords landscape is shown in Figure 2.
Figure 2. Authors’ keywords cluster landscape.
Electronics 2022,11, 2485 6 of 11
3.2.2. Content Analysis
During the 3rd and 4th steps of the algorithm, author keywords were condensed into
34 codes. Those codes were next reduced into 16 categories. By analysing links between
categories and codes, we finally identified six themes of software quality presented below
and in Table 5:
Software process improvement
: The most impactful research regarding the number
of citations was done around the beginning of the new millennia and was related to
software process maturity [
32
,
33
] and the importance of top management leadership,
management infrastructure and stakeholder participation [
34
]. Recent important re-
search is still concerned with process maturity, but CMM(I) is combined with DevOps
and agile approaches [3537].
Metric-based software maintenance
: Most cited papers related to this theme were
published in the period 1993–2010 and are concerned with object-oriented metrics to
predict maintainability [
38
], metrics-based refactoring [
39
], predicting faults [
40
] and
code readability metrics [
41
]. Recent impactful papers deal with technical debt [
42
],
code smells and refactoring [43] and test smells [44].
Software evolution and refactoring:
The most influential research was published in
the past decade and was concerned with the prediction of software evolution [45,46],
automatic detection of bad smells which can trigger refactoring [
47
49
], the association
of software defect and refactoring [50].
Quality assurance in initial phases of software development life cycles:
The impact-
ful papers were published after the end of millennia. The research was mainly focused
on how to assure software quality on the architectural level [
51
,
52
] and software
requirements [5355].
Search-based software engineering for defect prediction and classification:
The re-
search on this theme has become important in the last 15 years. The research mainly
used data mining and machine learning to predict software defects and failures using
static codes and other software documents [5658].
Software quality management and assurance with testing and inspections:
This
theme is the most established, with influential papers starting to be published around
40 years ago [
59
61
]. Recent research is concerned with regression testing [
11
], predic-
tive mutation testing [62], metamorphic testing [63] and modern code reviews [64].
3.3. Hot Topics
To identify hot topics, we extended the methodology developed by Kokol et al. [
65
]
with the synthetic content analysis. We used this extended approach to compare the corpus
of publications published in 2018 and 2019 with the corpus of publications published in
2020 and 2021. All the new categories and themes emerging in 2020/21, and the categories
and themes emerging in 2018/19 mostly cited in 2020/21 were recognised as hot topics. In
that manner, we identified:
A new theme: Improving software development with the Integration of CMMI into
agile approaches [66].
New categories:
o
Natural language processing of software documents to elicit high-quality re-
quirements [67].
o Software quality attributes in agile environments [68].
o Software architectures for the internet of things [69].
o Software maintenance of blockchain-based software [70].
Most cited categories in 2020/21:
o Detecting code smells with genetic algorithms [71].
o Use of Artificial intelligence in risk management [72].
o Use multi-criteria decision-making in software quality modelling [73].
Electronics 2022,11, 2485 7 of 11
Table 5.
Software quality research themes (numbers in parenthesis present the number of publications
in which an author keyword appeared).
Colour Codes Concepts/Categories Themes
Red (26 keywords))
Software engineering (641),
Software quality assurance
and management (330), Agile
software development (319),
Software development (276),
Software process
improvement (203), Software
process (109), Software
reuse (91), Project
management (90)
Software quality assurance
with project and knowledge
management; software
process improvement with
agile approaches and CMMI;
software reuse with
production lines based on
software quality metrics;
Software process
improvement
Green (21 keywords)
Software testing (694),
Metrics (578), Reliability (108);
Maintainability (135),
UML (80); Genetics
algorithms (71)
Metrics-based software testing
supported by genetic
algorithms; using and
predicting maintainability
metrics like reusability,
complexity, cohesion,
and coupling;
Metric-based software
maintenance
Blue (20 keywords)
Software quality (2685),
Empirical studies in software
engineering (281),
Refactoring (226), Software
maintenance (259), Technical
depth and code smells (183),
Software evolution (130),)
Mining software repositories
to support empirical and
search software engineering;
software evolution with
refactoring based on code
smells; technical debts and
code smells in association
with software maintenance;
Software evolution and
refactoring
Yellow (19 keywords)
Software architecture (253),
Software reliability (233),
Requirements engineering
(226), Software quality
models (214), Usability (108),
Quality metrics and
attributes (136), Quality
assessment and
evaluation (99)
Quality attributes of software
requirements and architecture;
quality attributes of software
quality models and standards;
general quality metrics like
reliability; security,
and usability;
Quality assurance in initial
phases of software
development life cycles
Viollet (16 keywords)
Software metrics (568),
Machine learning and data
mining (479); Fault and defect
prediction (228)
Use of software metrics and
data mining in defect
prediction and classification;
Search-based software
engineering for defect
prediction and classification
Light blue (14 keywords)
Quality (221), Software (204);
Quality management and
assurance (164); Testing (148);
Testing and inspection,
verification and validation;
testing automation;
programming productivity
and quality
Software quality management
and assurance with testing
and inspections
3.4. How Much Does the Software Quality Matter?
According to our synthesis, the volume, distribution and scope of quality research,
as well as the research community and the research literature production, are growing,
following the ever-growing importance of software in almost all human activities and
recognising the potentially catastrophic consequences of quality defects, despite the fact
that significant part of software quality research is still presented at conferences a ro-
bust, and well-researched body of evidence is forming but is not yet achieved in yet
to be established core journals. Based on identified trends and best practices in overall
software quality, substantial research in software quality exists but would need to be tai-
Electronics 2022,11, 2485 8 of 11
lored and extended to the specific requirements of software quality sub-disciplines and
application domains.
3.5. Strengths and Limitations
The study’s main strength is that it is the first holistic content analysis of software
quality research. Another strength is that content analysis was performed using a novel
synthetic knowledge synthesis approach. One possible limitation is that the analysis was
limited to publications indexed in Scopus only; however, because Scopus covers the largest
and most complete set of information titles, we believe that we analysed most of the
important peer-reviewed publications.
4. Conclusions
Our bibliometric study showed that the production of research publications relating to
software quality is currently following an exponential growth trend and that the software
quality research community is growing. The synthetic content analysis revealed that the
published knowledge could be structured into six themes, most important being the themes
regarding software quality improvement by enhancing software engineering, advanced
software testing and improved defect and fault prediction with machine learning and data
mining. According to the analysis of the hot topics, it seems that future research will be
directed into developing and using a whole spectre of new artificial intelligence tools (not
just machine learning and data mining) and focusing on ensuring software quality in agile
development paradigms.
The main contributions of or study are first, a holistic and better understanding of
the software quality concept based on the triangulation of quantitative and qualitative
analysis, which provides a landscape, performance and spatial attributes of the software
quality research. Second, this study provides a paradigm for future studies in software
development and digital transformation topics research.
Funding: This research was funded by ARRS, grant number J7-2605.
Data Availability Statement: Not applicable.
Conflicts of Interest: The author declares no conflict of interest.
References
1.
Kokol, P.; Vošner, H.B.; Kokol, M.; Završnik, J. The Quality of Digital Health Software: Should We Be Concerned? Digit. Health
2022,8, 20552076221109055. [CrossRef] [PubMed]
2.
Tian, J. Software Quality Engineering: Testing, Quality Assurance and Quantifiable Improvement; Wiley India Pvt. Limited: New Delhi,
India, 2009; ISBN 978-81-265-0805-1.
3.
Winkler, D.; Biffl, S.; Mendez, D.; Wimmer, M.; Bergsmann, J. (Eds.) Software Quality: Future Perspectives on Software Engineering
Quality: 13th International Conference, SWQD 2021, Vienna, Austria, January 19–21, 2021, Proceedings; Lecture Notes in Business
Information Processing; Springer International Publishing: New York, NY, USA, 2021; ISBN 978-3-030-65853-3.
4. Gillies, A. Software Quality: Theory and Management, 3rd ed.; Lulu Press: Morrisville, NC, USA, 2011.
5.
Suali, A.J.; Fauzi, S.S.M.; Nasir, M.H.N.M.; Sobri, W.A.W.M.; Raharjana, I.K. Software Quality Measurement in Software
Engineering Project: A Systematic Literature Review. J. Theor. Appl. Inf. Technol. 2019,97, 918–929.
6.
Atoum, I. A Novel Framework for Measuring Software Quality-in-Use Based on Semantic Similarity and Sentiment Analysis of
Software Reviews. J. King Saud Univ. Comput. Inf. Sci. 2020,32, 113–125. [CrossRef]
7.
Lacerda, G.; Petrillo, F.; Pimenta, M.; Guéhéneuc, Y.G. Code Smells and Refactoring: A Tertiary Systematic Review of Challenges
and Observations. J. Syst. Softw. 2020,167, 110610. [CrossRef]
8.
Wedyan, F.; Abufakher, S. Impact of Design Patterns on Software Quality: A Systematic Literature Review. IET Softw.
2020
,
14, 1–17. [CrossRef]
9.
Saini, G.L.; Panwar, D.; Kumar, S.; Singh, V. A Systematic Literature Review and Comparative Study of Different Software Quality
Models. J. Discret. Math. Sci. Cryptogr. 2020,23, 585–593. [CrossRef]
10.
Guveyi, E.; Aktas, M.S.; Kalipsiz, O. Human Factor on Software Quality: A Systematic Literature Review. In International
Conference on Computational Science and Its Applications; Springer: Cham, Switzerland, 2020; pp. 918–930. [CrossRef]
11.
Khatibsyarbini, M.; Isa, M.A.; Jawawi, D.N.A.; Tumeng, R. Test Case Prioritization Approaches in Regression Testing: A
Systematic Literature Review. Inf. Softw. Technol. 2018,93, 74–93. [CrossRef]
Electronics 2022,11, 2485 9 of 11
12. Rathore, S.S.; Kumar, S. A Study on Software Fault Prediction Techniques. Artif. Intell. Rev. 2019,51, 255–327. [CrossRef]
13.
Cowlessur, S.K.; Pattnaik, S.; Pattanayak, B.K. A Review of Machine Learning Techniques for Software Quality Prediction. Adv.
Intell. Syst. Comput. 2020,1089, 537–549. [CrossRef]
14.
Kupiainen, E.; Mäntylä, M.V.; Itkonen, J. Using Metrics in Agile and Lean Software Development—A Systematic Literature
Review of Industrial Studies. Inf. Softw. Technol. 2015,62, 143–163. [CrossRef]
15.
Van Raan, A. Measuring Science: Basic Principles and Application of Advanced Bibliometrics. In Springer Handbook of Science
and Technology Indicators; Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M., Eds.; Springer Handbooks; Springer International
Publishing: Cham, Switzerland, 2019; pp. 237–280. ISBN 978-3-030-02511-3.
16.
Pachouly, J.; Ahirrao, S.; Kotecha, K. A Bibliometric Survey on the Reliable Software Delivery Using Predictive Analysis. Libr.
Philos. Pract. 2020,2020, 1–27.
17. Kokol, P.; Kokol, M.; Zagoranski, S. Code Smells: A Synthetic Narrative Review. arXiv 2021, arXiv:2103.01088.
18.
Chalmers, I.; Hedges, L.V.; Cooper, H. A Brief History of Research Synthesis. Eval Health Prof
2002
,25, 12–37. [CrossRef]
[PubMed]
19. Tricco, A.C.; Tetzlaff, J.; Moher, D. The Art and Science of Knowledge Synthesis. J. Clin. Epidemiol. 2011,64, 11–20. [CrossRef]
20.
Whittemore, R.; Chao, A.; Jang, M.; Minges, K.E.; Park, C. Methods for Knowledge Synthesis: An Overview. Heart Lung
2014
,
43, 453–461. [CrossRef]
21.
Kyngäs, H.; Mikkonen, K.; Kääriäinen, M. The Application of Content Analysis in Nursing Science Research; Springer Nature:
Berlin/Heidelberg, Germany, 2020. [CrossRef]
22.
Kokol, P.; Kokol, M.; Zagoranski, S. Machine Learning on Small Size Samples: A Synthetic Knowledge Synthesis. Sci. Prog.
2022
,
105, 00368504211029777. [CrossRef]
23.
Van Eck, N.J.; Waltman, L. Software Survey: VOSviewer, a Computer Program for Bibliometric Mapping. Scientometrics
2010
,
84, 523–538. [CrossRef]
24.
Tomayko, J.E. Teaching Software Development in a Studio Environment. In Proceedings of the SIGCSE 1991, San Antonio, TX,
USA, 7–8 March 1991; pp. 300–303.
25. Dijkstra, E.W. The Humble Programmer. Commun. ACM 1972,15, 859–866. [CrossRef]
26.
Cavano, J.P.; McCall, J.A. A Framework for the Measurement of Software Quality. SIGMETRICS Perform. Eval. Rev.
1978
,
7, 133–139. [CrossRef]
27.
Dingsøyr, T.; Nerur, S.; Balijepally, V.; Moe, N.B. A Decade of Agile Methodologies: Towards Explaining Agile Software
Development. J. Syst. Softw. 2012,85, 1213–1221. [CrossRef]
28.
10 Best Countries to Outsource Software Development, Based on Data. 2019. Available online: https://www.codeinwp.com/
blog/best-countries-to-outsource-software-development/ (accessed on 25 July 2022).
29.
Qubit Labs. How Many Programmers Are There in the World and in the US? 2022. Available online: https://qubit-labs.
com/how-many-programmers-in-the-world/#:~{}:text=As%20per%20the%20information%20above,are%20located%20in%20
North%20America (accessed on 25 July 2022).
30.
Journal Metrics in Scopus: Source Normalized Impact per Paper (SNIP)|Elsevier Scopus Blog. Available online: https://blog.
scopus.com/posts/journal-metrics-in-scopus- source-normalized-impact-per-paper-snip (accessed on 25 July 2022).
31.
Kokol, P. Funded and Non-Funded Research Literature in Software Engineering in Relation to Country Determinants. COLLNET
J. Scientometr. Inf. Manag. 2019,13, 103–109. [CrossRef]
32.
Harter, D.E.; Krishnan, M.S.; Slaughter, S.A. Effects of Process Maturity on Quality, Cycle Time, and Effort in Software Product
Development. Manag. Sci. 2000,46, 451–466. [CrossRef]
33.
Herbsleb, J.; Zubrow, D.; Goldenson, D.; Hayes, W.; Paulk, M. Software Quality and the Capability Maturity Model. Commun.
ACM 1997,40, 30–40. [CrossRef]
34.
Ravichandran, T.; Rai, A. Quality Management in Systems Development: An Organizational System Perspective. MIS Q. Manag.
Inf. Syst. 2000,24, 381–410. [CrossRef]
35.
Alqadri, Y.; Budiardjo, E.K.; Ferdinansyah, A.; Rokhman, M.F. The CMMI-Dev Implementation Factors for Software Quality
Improvement: A Case of XYZ Corporation. In Proceedings of the 2020 2nd Asia Pacific Information Technology Conference, Bali
Island, Indonesia, 17–19 January 2020; pp. 34–40.
36.
Ferdinansyah, A.; Purwandari, B. Challenges in Combining Agile Development and CMMI: A Systematic Literature Review. In
Proceedings of the ICSCA 2021: 2021 10th International Conference on Software and Computer Applications, Kuala Lumpur,
Malaysia, 23–26 February 2021; pp. 63–69.
37.
Rafi, S.; Yu, W.; Akbar, M.A.; Mahmood, S.; Alsanad, A.; Gumaei, A. Readiness Model for DevOps Implementation in Software
Organizations. J. Softw. Evol. Process 2021,33, e2323. [CrossRef]
38. Li, W.; Henry, S. Object-Oriented Metrics That Predict Maintainability. J. Syst. Softw. 1993,23, 111–122. [CrossRef]
39.
Simon, F.; Steinbrückner, F.; Lewerentz, C. Metrics Based Refactoring. In Proceedings of the Fifth European Conference on
Software Maintenance and Reengineering, Lisbon, Portugal, 14–16 March 2001; pp. 30–38.
40.
Olague, H.M.; Etzkorn, L.H.; Gholston, S.; Quattlebaum, S. Empirical Validation of Three Software Metrics Suites to Predict
Fault-Proneness of Object-Oriented Classes Developed Using Highly Iterative or Agile Software Development Processes. IEEE
Trans. Softw. Eng. 2007,33, 402–419. [CrossRef]
41. Buse, R.P.L.; Weimer, W.R. Learning a Metric for Code Readability. IEEE Trans. Softw. Eng. 2010,36, 546–558. [CrossRef]
Electronics 2022,11, 2485 10 of 11
42.
Tsoukalas, D.; Kehagias, D.; Siavvas, M.; Chatzigeorgiou, A. Technical Debt Forecasting: An Empirical Study on Open-Source
Repositories. J. Syst. Softw. 2020,170, 110777. [CrossRef]
43.
Agnihotri, M.; Chug, A. A Systematic Literature Survey of Software Metrics, Code Smells and Refactoring Techniques. J. Inf.
Processing Syst. 2020,16, 915–934. [CrossRef]
44.
Kim, D.J.; Chen, T.-H.; Yang, J. The Secret Life of Test Smells—An Empirical Study on Test Smell Evolution and Maintenance.
Empir. Softw. Eng. 2021,26, 1–47. [CrossRef]
45.
Bhattacharya, P.; Iliofotou, M.; Neamtiu, I.; Faloutsos, M. Graph-Based Analysis and Prediction for Software Evolution. In
Proceedings of the ICSE ‘12: 34th International Conference on Software Engineering, Zurich, Switzerland, 2–9 June 2012;
pp. 419–429.
46.
Wyrich, M.; Bogner, J. Towards an Autonomous Bot for Automatic Source Code Refactoring. In Proceedings of the 2019
IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), Montreal, QC, Canada, 28 May 2019; pp. 24–28.
47.
Fontana, F.A.; Braione, P.; Zanoni, M. Automatic Detection of Bad Smells in Code: An Experimental Assessment. J. Object Technol.
2012,11, 1–38. [CrossRef]
48.
Ouni, A.; Kessentini, M.; Sahraoui, H.; Inoue, K.; Deb, K. Multi-Criteria Code Refactoring Using Search-Based Software
Engineering: An Industrial Case Study. ACM Trans. Softw. Eng. Methodol. 2016,25, 1–53. [CrossRef]
49.
Bavota, G.; De Lucia, A.; Marcus, A.; Oliveto, R. Automating Extract Class Refactoring: An Improved Method and Its Evaluation.
Empir. Softw. Eng. 2014,19, 1617–1664. [CrossRef]
50.
Ratzinger, J.; Sigmund, T.; Gall, H.C. On the Relation of Refactoring and Software Defects. In Proceedings of the 2008 International
Working Conference on Mining Software Repositories, MSR 2008, Leipzig, Germany, 10–11 May 2008; pp. 35–38.
51. Folmer, E.; Bosch, J. Architecting for Usability: A Survey. J. Syst. Softw. 2004,70, 61–78. [CrossRef]
52.
Wang, W.-L.; Pan, D.; Chen, M.-H. Architecture-Based Software Reliability Modeling. J. Syst. Softw.
2006
,79, 132–146. [CrossRef]
53.
Mellado, D.; Fernández-Medina, E.; Piattini, M. A Common Criteria Based Security Requirements Engineering Process for the
Development of Secure Information Systems. Comput. Stand. Interfaces 2007,29, 244–253. [CrossRef]
54.
Sedeño, J.; Schön, E.-M.; Torrecilla-Salinas, C.; Thomaschewski, J.; Escalona, M.J.; Mejias, M. Modelling Agile Requirements Using
Context-Based Persona Stories. In Proceedings of the WEBIST 2017: 13th International Conference on Web Information Systems
and Technologies, Porto, Portugal, 25–27 April 2017; pp. 196–203.
55.
Shull, F.; Rus, I.; Basili, V. How Perspective-Based Reading Can Improve Requirements Inspections. Computer
2000
,33, 73–79.
[CrossRef]
56.
Menzies, T.; Greenwald, J.; Frank, A. Data Mining Static Code Attributes to Learn Defect Predictors. IEEE Trans. Softw. Eng.
2007
,
33, 2–13. [CrossRef]
57.
Lessmann, S.; Baesens, B.; Mues, C.; Pietsch, S. Benchmarking Classification Models for Software Defect Prediction: A Proposed
Framework and Novel Findings. IEEE Trans. Softw. Eng. 2008,34, 485–496. [CrossRef]
58.
Song, Q.; Shepperd, M.; Cartwright, M.; Mair, C. Software Defect Association Mining and Defect Correction Effort Prediction.
IEEE Trans. Softw. Eng. 2006,32, 69–82. [CrossRef]
59.
Boehm, B.W.; Brown, J.R.; Lipow, M. Quantitative Evaluation of Software Quality. In Proceedings of the 2nd International
Conference on Software Engineering, San Francisco, CA, USA, 13–15 October 1976; pp. 592–605.
60. Fagan, M.E. Advances in Software Inspections. IEEE Trans. Softw. Eng. 1986,SE-12, 744–751. [CrossRef]
61. McCabe, T.J.; Butler, C.W. Design Complexity Measurement and Testing. Commun. ACM 1989,32, 1415–1425. [CrossRef]
62.
Zhang, J.; Zhang, L.; Harman, M.; Hao, D.; Jia, Y.; Zhang, L. Predictive Mutation Testing. IEEE Trans. Softw. Eng.
2019
,45, 898–918.
[CrossRef]
63.
Segura, S.; Towey, D.; Zhou, Z.Q.; Chen, T.Y. Metamorphic Testing: Testing the Untestable. IEEE Softw.
2020
,37, 46–53. [CrossRef]
64.
Uchoa, A.; Barbosa, C.; Oizumi, W.; Blenilio, P.; Lima, R.; Garcia, A.; Bezerra, C. How Does Modern Code Review Impact Software
Design Degradation? An In-Depth Empirical Study. In Proceedings of the 2020 IEEE International Conference on Software
Maintenance and Evolution (ICSME), Adelaide, Australia, 28 September–2 October 2020; pp. 511–522.
65. Kokol, P.; Završnik, J.; Blažun Vošner, H. Bibliographic-Based Identification of Hot Future Research Topics: An Opportunity for
Hospital Librarianship. J. Hosp. Librariansh. 2018,18, 1–8. [CrossRef]
66.
Amer, S.K.; Badr, N.; Hamad, A. Combining CMMI Specific Practices with Scrum Model to Address Shortcomings in Process
Maturity. Adv. Intell. Syst. Comput. 2020,921, 898–907. [CrossRef]
67.
Ahmad, A.; Feng, C.; Khan, M.; Khan, A.; Ullah, A.; Nazir, S.; Tahir, A. A Systematic Literature Review on Using Machine
Learning Algorithms for Software Requirements Identification on Stack Overflow. Secur. Commun. Netw.
2020
,2020, 8830683.
[CrossRef]
68.
Oriol, M.; Martínez-Fernández, S.; Behutiye, W.; Farré, C.; Kozik, R.; Seppänen, P.; Vollmer, A.M.; Rodríguez, P.; Franch, X.;
Aaramaa, S.; et al. Data-Driven and Tool-Supported Elicitation of Quality Requirements in Agile Companies. Softw. Qual. J.
2020
,
28, 931–963. [CrossRef]
69.
Minerva, R.; Lee, G.M.; Crespi, N. Digital Twin in the IoT Context: A Survey on Technical Features, Scenarios, and Architectural
Models. Proc. IEEE 2020,108, 1785–1824. [CrossRef]
70. Liu, X.L.; Wang, W.M.; Guo, H.; Barenji, A.V.; Li, Z.; Huang, G.Q. Industrial Blockchain Based Framework for Product Lifecycle
Management in Industry 4.0. Robot. Comput.-Integr. Manuf. 2020,63, 101897. [CrossRef]
Electronics 2022,11, 2485 11 of 11
71.
Caram, F.L.; Rodrigues, B.R.D.O.; Campanelli, A.S.; Parreiras, F.S. Machine Learning Techniques for Code Smells Detection: A
Systematic Mapping Study. Int. J. Softw. Eng. Knowl. Eng. 2019,29, 285–316. [CrossRef]
72.
Winkle, T.; Erbsmehl, C.; Bengler, K. Area-Wide Real-World Test Scenarios of Poor Visibility for Safe Development of Automated
Vehicles. Eur. Transp. Res. Rev. 2018,10, 32. [CrossRef]
73.
Sun, G.; Guan, X.; Yi, X.; Zhou, Z. An Innovative TOPSIS Approach Based on Hesitant Fuzzy Correlation Coefficient and Its
Applications. Appl. Soft Comput. J. 2018,68, 249–267. [CrossRef]
... Notably, the United States emerges as the leading contributor in this domain. Key areas of focus revolve around optimizing software engineering practices to enhance the overall quality of software, implementing advanced techniques for software testing, and harnessing the power of machine learning and data mining to predict defects and faults more effectively [183]. ...
Article
Full-text available
This review focuses on the complex connections between machine learning, mechatronics, and stretch forming, offering valuable insights that can lay the groundwork for future research. It provides an overview of the origins and fundamentals of these fields, emphasizes notable progress, and explores the influence of these fields on society and industry. Also highlighted is the progress of robotics research and particularities in the field of sheet metal forming and its various applications. This review paper focuses on presenting the latest technological advancements and the integrations of these fields from their beginnings to the present days, providing insights into future research directions.
... For software users, quality is related to the usability and implementation of required functionalities. As for software developers, the most important quality issue is that the software meets the specifications and provides services as previously contracted [Tian 2005, Kokol 2022. Therefore, quality is subjectively determined by those who interact with the system, whether to solve their problems through software or are responsible for designing, implementing, and testing the solutions, making judgments about the quality of the product [Gillies 2011, Suali et al. 2019]. ...
Conference Paper
Full-text available
Aspects of software quality (SQ), such as process and product metrics, and assessment techniques, can be taught to computing students during their undergraduate courses, however, there is no consensus on how. In Brazil, computing courses are structured as the Brazilian Computer Society suggests, still, researchers point out that there are few SQ subjects in these courses. This paper aims to analyze the perception of SQ concepts by advanced undergraduate students in the northwest of the Paraná state. We applied a survey and received ninety-nine answers. Our results show that most SQ concepts are taught, but the students feel they did not learn and are not able to apply them. We discuss and suggest guidelines to improve the understanding of SQ concepts.
... Software quality has gained more influence in recent years, as evidenced by the exponential growth of the identified publications in this field by a mapping study [1]. Software quality assessment encompasses two major approaches: (1) the measurement and assessment of the processes that result in a software product, and (2) the measurement and assessment of a software product directly [2,3]. ...
Article
Full-text available
The authors’ previously published research delved into the theory of software product quality modelling, model views, concepts, and terminologies. They also analysed this specific field from the point of view of uncertainty, and possible descriptions based on fuzzy set theory and fuzzy logic. Laying a theoretical foundation was necessary; however, software professionals need a more tangible and practical approach for their everyday work. Consequently, the authors devote this paper to filling in this gap; it aims to illustrate how to interpret and utilise the previous findings, including the established taxonomy of the software product quality models. The developed fuzzy model’s simplification is also presented with a Generalized Additive Model approximation. The paper does not require any formal knowledge of uncertainty computations and reasoning under uncertainty, nor does it need a deep understanding of quality modelling in terms of terminology, concepts, and meta-models, which were necessary to prepare the taxonomy and relevance ranking. The paper investigates how to determine the validity of statements based on a given software product quality model; moreover, it considers how to tailor and adjust quality models to the particular project’s needs. The paper also describes how to apply different software product quality models for different quality views to take advantage of the automation potential offered for the measurement and assessment of source code quality. Furthermore, frequent pitfalls are illustrated with their corresponding resolutions, including an unmeasured quality property that is found to be important and needs to be included in the measurement and assessment process.
Article
In the modern world, where information security becomes a key element of any organization's operations, software security testing automation is more important than ever. The success of an application directly depends on its stability, reliability, and security, which makes the proper implementation of control mechanisms critical. The increase in cyber threats and the growing complexity of software systems make this topic even more relevant. The main role of automation is to provide quick and efficient detection of potential application vulnerabilities during its development phase. Despite its obvious advantages, several issues complicate the implementation of automated solutions, such as the complexity of configuration, high implementation and maintenance costs, lack of expertise in the field, lack of prioritization. This article examines the role of software security testing automation, challenges, methods, tools for testing, and ways to combine them for greater efficiency. Based on the analysis, an architectural solution is proposed that ensures quick, reliable, and regular security testing of the application at different stages of its lifecycle, significantly increasing the likelihood of its resistance to various vulnerabilities. Keywords: software, cybersecurity, automation, testing, containerization, protection.
Article
Aim: This study performed a bibliometric analysis of studies related to mobile learning in the field of nursing education. Methods: The Scopus database was used to determine the most frequently cited studies on mobile learning in nursing education. VOSviewer and Bibliometrix were employed for bibliometric analysis and visualization. Science mapping and performance analysis was adopted from bibliometric analysis techniques. In addition, a synthetic knowledge synthesis approach was used. Results: A total of 234 publications were published in 107 sources in 2002-2023. The publications had 8797 citations, an average of 88 citations per publication. In terms of total link strength (TLS), links, a number of articles and citations, the US led all other countries in the field. Regarding authors, Hwang was the most frequently cited authors (n = 348). According to trend topics analysis, the keywords "gamification", "simulation", "attitude", "clinical competence" and "online learning" have emerged in the field. Conclusion: Research on mobile learning in nursing education has been increasing in recent years. The findings of this study can provide new ideas in the applications of mobile learning in nursing education to researchers or nursing faculties in the field.
Article
Full-text available
The digitalization of healthcare fuelled by advances in technology and the increased prevalence of mobile smart devices and health-related internet of things can offer equitable access to expert-level healthcare globally. Growing demand for telemedicine, mobile health apps, and advanced data analytics have further established their role in a modern information society during the Covid-19 crisis. Digital health is, in essence, powered by software (DHSW), which has to operate in the specific digital health environment characteristics and is therefore highly and intrinsically complex and prone to software defects and faults. Given the lack of standardization regarding DHSW quality, we explored the available reviewed research on this crucial topic in this brief paper, using a synthetic thematic analysis approach. We assert that neither the volume, distribution nor scope of the DHSW quality research content is satisfactory, and significant research gaps exist. Based on the presented evidence, we can only conclude that we should be concerned and that the time to act is now to ensure that the unavoidable increase of usage and prevalence of DHSW will not – in the end – reduce the quality of care due to subpar software and software-based digital health systems.
Article
Full-text available
Machine Learning is an increasingly important technology dealing with the growing complexity of the digitalised world. Despite the fact, that we live in a ‘Big data’ world where, almost ‘everything’ is digitally stored, there are many real-world situations, where researchers are still faced with small data samples. The present bibliometric knowledge synthesis study aims to answer the research question ‘What is the small data problem in machine learning and how it is solved?’ The analysis a positive trend in the number of research publications and substantial growth of the research community, indicating that the research field is reaching maturity. Most productive countries are China, United States and United Kingdom. Despite notable international cooperation, the regional concentration of research literature production in economically more developed countries was observed. Thematic analysis identified four research themes. The themes are concerned with to dimension reduction in complex big data analysis, data augmentation techniques in deep learning, data mining and statistical learning on small datasets.
Article
Full-text available
In recent years, researchers and practitioners have been studying the impact of test smells on test maintenance. However, there is still limited empirical evidence on why developers remove test smells in software maintenance and the mechanism employed for addressing test smells. In this paper, we conduct an empirical study on 12 real-world open-source systems to study the evolution and maintenance of test smells, and how test smells are related to software quality. Our results show that: 1) Although the number of test smell instances increases, test smell density decreases as systems evolve. 2) However, our qualitative analysis on those removed test smells reveals that most test smell removal (83%) is a by-product of feature maintenance activities. 45% of the removed test smells relocate to other test cases due to refactoring, while developers deliberately address the only 17% of the test smell instances, consisting of largely Exception Catch/Throw and Sleepy Test. 3) Our statistical model shows that test smell metrics can provide additional explanatory power on post-release defects over traditional baseline metrics (an average of 8.25% increase in AUC). However, most types of test smells have a minimal effect on post-release defects. Our study provides insight into how developers resolve test smells and current test maintenance practices. Future studies on test smells may consider focusing on the specific types of test smells that may have a higher correlation with defect-proneness when helping developers with test code maintenance.
Chapter
Full-text available
Ensuring software quality is an important step towards a successful project. Since software development is a human-oriented process, it is possible to say that any factor affecting people will directly affect software quality and success. The aim of this study is to reveal which factors affect humans. For this purpose, we conducted a systematic literature review. We identified 80 related primary studies from the literature. We defined 7 research questions. For answering research questions, we extracted data from the primary studies. We researched human factors, methods for data collection and data analysis, publication types and years. Factors are grouped into 3 main groups: Personal factors, interpersonal factors, and organizational factors. The results show that personal factors are the most important category of human factors. It is seen that the most researched factors among personal factors are “experience” and “education”.
Article
Full-text available
DevOps is a new software engineering paradigm adopted by various software organizations to develop the quality software within time and budget. The implementation of DevOps practices is critical, and there are no guidelines to assess and improve the DevOps activities in software organizations. Hence, there is a need to develop a readiness model for DevOps (RMDevOps) with an aim to assist the practitioners for implementation of DevOps practices in software firms. To achieve the study objective , we conducted a systematic literature review (SLR) study to identify the critical challenges and associated best practices of DevOps. A total of 18 challenges and 73 best practices were identified from the 69 primary studies. The identified challenges and best practices were further evaluated by conducting a survey with industry practitioners. The RMDevOps was developed based on other well-established models in software engineering domain, for example, software process improvement readiness model (SPIRM) and software outsourcing vendor readiness model (SOVRM). Finally, case studies were conducted with three different organizations with an aim to validate the developed model. The results show that the RMDevOps is effective to assess and improve the DevOps practices in software organizations. K E Y W O R D S best practices, case study, guidelines, readiness model
Conference Paper
Full-text available
Software design is an important concern in modern code review through which multiple developers actively discuss and improve each single code change. However, there is little understanding of the impact of such developers' reviews on continuously reducing design degradation over time. It is even less clear to what extent and how design degradation is reversed during the process of each single code change's review. In summary, existing studies have not assessed how the process of design degradation evolution is impacted along: (i) within each single review, and (ii) across multiple reviews. As a consequence, one cannot understand how certain code review practices consistently contribute to either reduce or further increase design degradation as the project evolves. We aim at addressing these gaps through a multi-project retrospective study. By investigating 14,971 code reviews from seven software projects, we report the first study that characterizes how the process of design degradation evolves within each review and across multiple reviews. Moreover, we analyze a comprehensive suite of metrics to enable us to observe the influence of certain code review practices on combating or even accelerating design degradation. Our results show that the majority of code reviews had little to no design degradation impact in the analyzed projects. Even worse, this observation also applies, to some extent, to reviews with an explicit concern on design. Surprisingly, the practices of long discussions and high proportion of review disagreement in code reviews were found to increase design degradation. Finally, we also discuss how the study findings shed light on how to improve the research and practice of modern code review.
Book
This book constitutes the refereed proceedings of the 13th Software Quality Days Conference, SWQD 2021, which was planned to be held in Vienna, Austria, during January 19–21, 2021. Due to the COVID-19 pandemic, the conference was cancelled and will be merged with SWQD 2022. The Software Quality Days (SWQD) conference started in 2009 and has grown to the biggest conference on software quality in Europe with a strong community. The program of the SWQD conference is designed to encompass a stimulating mixture of practical presentations and new research topics in scientific presentations. The guiding conference topic of the SWQD 2021 is “Future Perspectives on Software Engineering Quality”. The 3 full papers and 5 short papers presented in this volume were carefully reviewed and selected from 13 submissions. The volume also contains 2 invited talks and one introductory paper for an interactive session. The contributions were organized in topical sections named: automation in software engineering; quality assurance for AI-based systems; machine learning applications; industry-academia collaboration; and experimentation in software engineering.
Article
Delivering a reliable software product is a fairly complex process, which involves proper coordination from the various teams in planning, execution, and testing for delivering software. Most of the development time and the software budget's cost is getting spent finding and fixing bugs. Rework and side effect costs are mostly not visible in the planned estimates, caused by inherent bugs in the modified code, which impact the software delivery timeline and increase the cost. Artificial intelligence advancements can predict the probable defects with classification based on the software code changes, helping the software development team make rational decisions. Optimizing the software cost and improving the software quality is the topmost priority of the industry to remain profitable in the competitive market. Hence, there is a great urge to improve software delivery quality by minimizing defects and having reasonable control over predicted defects. This paper presents the bibliometric study for "Reliable Software Delivery using Predictive analysis" by selecting 450 documents from the Scopus database, choosing keywords like software defect prediction, machine learning, and artificial intelligence. The study is conducted for a year starting from 2010 to 2021. As per the survey, it is observed that Software defect prediction achieved an excellent focus among the researchers. There are great possibilities to predict and improve overall software product quality using artificial intelligence techniques.