ArticlePDF Available

A Retrospective Study of Software Analytics Projects: In-Depth Interviews with Practitioners

Authors:

Abstract and Figures

Software analytics guide practitioners in decision making throughout the software development process. In this context, prediction models help managers efficiently organize their resources and identify problems by analyzing patterns on existing project data in an intelligent and meaningful manner. Over the past decade, the authors have worked with software organizations to build metric repositories and predictive models that address process-, product-, and people-related issues in practice. This article shares their experience over the years, reflecting the expectations and outcomes both from practitioner and researcher viewpoints.
Content may be subject to copyright.
54 IEEE SoftwarE | publIShEd by thE IE EE comput Er SocIE ty 0740 -745 9/13 /$ 31.0 0 © 2 013 I E EE
A Retrospective
Study of Software
Analytics Projects:
In-Depth Interviews
with Practitioners
Ayse Tosun Misirli, University of Oulu
Bora Caglayan, Bogazici University
Ayse Bener, Ryerson University
Burak Turhan, University of Oulu
// Software analytics guide practitioners in decision making
throughout the software development process. In this context,
prediction models can help managers efciently organize their
resources and identify problems by analyzing patterns on
existing project data in an intelligent and meaningful manner. //
THE ULTIMATE BENEFIT of predic-
tion models is realized when they be-
come an integral part of the processes
used to dene policies. However, it
takes a considerable amount of time to
deploy these models into software sys-
tems and observe their benets—for
example, a “cloned code detection” ap-
proach at Microsoft took three years of
effort to move from prototype develop-
ment to full technology transfer.1
Software practitioners look for
solid, convincing evidence to upgrade
software analytics from early prototype
to full technology transfer. However,
“convincing” evidence isn’t easy to nd
and has different interpretations de-
pending on whether you’re a researcher
(the model’s statistical strength or its
results’ replicability2) or a practitioner
(the real value added by the proposed
technology that leads to actionable
changes). Often, we see researcher-
proposed prototypes or early models
that lack the nal deployment step, and
hence, any real-world impacts. How-
ever, practitioners require clear direc-
tion about which model to use, the
tangible benets of using that model in
terms of time and effort, and whether
the model would really help them in de-
cision making.
In the past decade, we’ve worked
with small to large-scale software or-
ganizations to build prediction mod-
els that improve resource planning. In
building these models, we completed
the rst four steps of the software an-
alytics approach that Dongmei Zhang
and her colleagues proposed:3 task
denition, data preparation, analytic-
technology development, and deploy-
ment and feedback gathering. In some
of these projects, we also had the op-
portunity to measure the value added
to the company in terms of reduction
in defect rates and testing effort.4 Some
projects ended without a full technol-
ogy transfer;5 others are still ongoing.6
Our motivations include exploring
practitioners’ expectations of predic-
tion models as software analytics proj-
ects and understanding how these mod-
els can be used in policy making. We
describe here three industrial case stud-
ies, each of which is at a different stage
in terms of its use:
• a deployed defect prediction model
(currently not in use),
• an early effort estimation proto-
type, and
FOCUS: The Many Faces oF soFTware analyTics
SEptEmbEr/octobEr 2013 | IEEE SoftwarE
55
• a quality prediction project that’s
still ongoing.
We conducted in-depth interviews
with the project stakeholders—that is,
development leads and managers—ac-
tively working with us on these proj-
ects. We intend to provide a road map
for research and practice that combines
our experience with the conclusions
drawn from these interviews.
Procedure
Our ndings are based on semistruc-
tured, in-depth interviews that included
open- and closed-ended questions of 12
practitioners from our three case stud-
ies. We interviewed all stakeholders
who participated in our project meet-
ings, helped us with data collection and
analysis, evaluated model performance,
or provided feedback during calibra-
tion. Figure 1 shows detailed informa-
tion about these stakeholders in terms
of degree, education, role, experience,
and gender.
Six respondents were involved in de-
fect prediction projects at TelCo and
TechCo. (Due to condentiality agree-
ments, we use Te lCo, Te chCo, and
BankCo in this article instead of the real
company names.) TelCo provides mo-
bile communication services in Turkey,
whereas TechCo is the Canadian branch
of a multinational technology and con-
sulting corporation. Among these six,
we conducted one-on-one interviews
with ve people from the TelCo devel-
opment team who participated in the
defect prediction project and one qual-
ity assurance expert from TechCo who
currently leads the quality prediction
project. We completed the project with
TelCo and deployed the nal model into
the company’s existing systems. How-
ever, after eight months of usage, the
deployed technology was put on hold
for TechCo. We’re in the initial steps of
prototype development. The rest of the
respondents are from BankCo, the IT
branch of a large European nancial in-
stitution in Turkey, where we set up an
effort estimation model.
Interview Protocol
We developed a protocol to list all the
issues to be explored during the inter-
views to ensure consistency among
them and hence the reliability of our
ndings.7 To guide the interviews, we
prepared two different sets of ques-
tions (http://bit.ly/RDRZAb and
http://bit.ly/R4QaKX) and modied
some of them based on context (qual-
ity prediction compared with effort es-
timation). Other questions were valid
in both contexts because of our aim
of gathering a common understanding
about the usage of prediction models,
the benets of their real usage, and the
reasons preventing their deployment
into daily use. Our questions were both
open- (for example, “What would you
like to add if you were to enrich the
output of these models?”) and closed-
ended (for example, “Were the project
outputs and your expectations from
the project consistent?”).
We followed guidelines described
elsewhere7, 8 during the preparation of
the interviews and paid attention to
question order and the main themes
to be explored, selecting questions to
cover both the negative and positive as-
pects of using software analytics.
Using four themes that had three to
ve questions each, we covered the fol-
lowing: factual information on whether
software analytics projects ended suc-
cessfully and how much effort they re-
quired, the main issues encountered
when building and deploying the tech-
nology, practitioners’ expectations about
the output of these models in terms of
information content and performance,
and the impact of these projects in terms
Electrical engineering Industrial engineering
Gender
024681
01
2
Male Female
Experience <10 10–20 >20
BS MS PhD
Analyst Developer ManagerManager Tester
EE Computer science & engineering IE Math
Role
Education
Degree
Figure 1. Demographic information about stakeholder participants. Participants vary in
terms of roles in software development teams and experience. Two-thirds of the par ticipants
are computer engineers, half with higher degrees (MSs or PhDs).
56 IEEE SoftwarE | www.computEr.org/SoftwarE
FOCUS: The Many Faces oF soFTware analyTics
of permanent changes throughout devel-
opment activities, if any.
Data Collection
We sent our questions to respondents
with a letter explaining our study and
expectations. We then conducted one-
on-one interviews with respondents to
give them some exibility in answering
the open-ended questions as well as to
gather more accurate information. We
recorded all responses anonymously to
make sure that participants could freely
share their thoughts. Some participants
preferred telephone interviews, so in
those cases, we recorded the meetings to
facilitate and improve the analysis pro-
cess. We also provided a list of quality
attributes and their denitions during
defect prediction interviews to agree on
the terminology with the teams. The in-
terviews took three days’ effort in total.
Our post-interview qualitative analy-
sis went as follows: we read all responses
and combined them based on the themes
covered in the questions (the chal-
lenges faced during the projects, ideas
for model improvement, and indirect
project benets), and then we grouped
similar responses to identify consistent
themes as well as any differences in re-
sponses to identify new themes. Finally,
we combined the three projects’ results
to get a generic road map.
Case 1: Effort
Estimation Project
The BankCo project took eight months,
but the nal model wasn’t actually de-
ployed due to major organizational
changes. Our initial agreement with
management was to estimate the effort
for a new project (in man-months) us-
ing a prediction model. As inputs, we
planned to use a list of project metrics
that were periodically collected from
past projects. We also asked the project
team to ll out a questionnaire to pro-
vide any missing data.
During task denition, we identied
problems such as inefcient project and
resource planning, heavy reliance on
experts, and poor estimation accuracy
as they affected both the development
team and senior management through
informal interviews. During prototype
development, we also uncovered data
quality issues and inconsistent values in
different process areas. Ultimately, our
effort estimation model was able to pre-
dict overall effort in 75 percent of proj-
ects with an error rate of less than 25
percent.5 The development team also
asked for phase-based effort estimation
(the effort required for requirements
elicitation, design, coding, and testing)
using the same model, but the estima-
tions for specic phases were poor due
to missing or inaccurate data related to
those phases.
During model calibration, the com-
pany went through a major senior
management change. A new CEO was
appointed, and all the vice presidents
were replaced with mostly external
hires. Although the development team
wanted to deploy our model, the new
management team had a different focus
and business priorities.
Problems Encountered
In this case study, we interviewed the
project team at BankCo to learn their
views about the project, effort esti-
mation models in general, and what
should be done differently next time.
All six respondents agreed that organi-
zational changes before the deployment
disrupted the project. However, they
had different views about the problems
encountered during the project related
to the organizational, data extraction,
and modeling issues.
Due to major changes in senior man-
agement, one respondent stated, “The
project was born dead,” while another
highlighted the signicance of “con-
vincing senior management about the
importance of the project to the com-
pany” for successful implementation of
new decision models in large compa-
nies. She believed that senior-level man-
agement wasn’t fully convinced about
the potential benets of the model,
so it was challenging to escalate some
project- related requests.
Two out of six respondents believed
that problems related to model perfor-
mance played an important role in de-
ployment failure. One respondent ex-
pressed his dissatisfaction about the
model’s performance on a new project
and its effort estimation.
Two of the six also weren’t satised
with the model’s input metrics, stat-
ing that project size estimations such
as lines of code or function points are
too hard to estimate at the beginning
of a project. They argued that every
company needs to dene its own size
measure based on inputs available at
the organization. For BankCo, these
inputs were decided as counts of data-
base tables, transactions, and user in-
terfaces that need to be implemented
in a new project, but these variables
weren’t consistently collected in previ-
ous projects due to data quality prob-
lems, so it was very challenging to ll
in all missing values for past projects.
One respondent believed that the lack
of survey culture in the company made
Our model was able to predict overall
effort in 75 percent of projects with
an error rate of less than 25 percent.
SEptEmbEr/octobEr 2013 | IEEE SoftwarE
57
it very difcult to estimate input pa-
rameters about various process areas
through Cocomo surveys.
Above all, the six respondents
agreed that data collection was a major
challenge throughout the project. None
of the metrics were systematically mea-
sured or reported for past projects. We
stopped the project to x problems in
the data repository, and project leads
managed to change their teams’ work-
ows for a while, but we had to call the
project off due to instability in the or-
ganizational structure.
Ideas for Improvement
Five of the six respondents thought that
the model’s output was insufcient,
but there was no real consensus on
what should be added to the output or
at what level of detail. Suggestions in-
cluded estimation of each requirement’s
effort and an estimation that considers
employee capabilities (each employee’s
effort). The respondents believed that
increasing the model output’s informa-
tion content was the next step in con-
verting effort estimation models into
decision models. Although the sug-
gestions were useful, we believe these
models can only be implemented in
companies with mature data collection
and planning and estimation processes.
Indirect Benets
During the interviews, all six respon-
dents from BankCo mentioned that this
project had led to positive changes in
their way of working, so we asked them
to explain what kind of changes they
had implemented and if any of these
changes had become permanent prac-
tices in the company.
We can group the benets to
BankCo as a mind shift at the organi-
zational level about software measure-
ment and process improvement, experi-
ence in software analytics, and transfer
of this experience to new projects.
During the course of the project, the
respondents had to decide how to quan-
tify the different characteristics of new
software versions as they were discov-
ering the importance of software mea-
surement. The research team conducted
a machine learning seminar, which also
helped the respondents in their profes-
sional development. Some practitio-
ners have learned to use their experi-
ence with data mining and analysis to
make data-driven decisions; other team
members have used their knowledge of
machine learning to build a fault local-
ization tool by mining patterns in ex-
ception logs.
The overall change in organizational
strategies and the effort needed to col-
lect accurate data were the biggest fac-
tors in abandoning the project. How-
ever, respondents indicated that the
project has had long-lasting impacts:
they recently initiated an organization-
wide measurement culture to store ac-
curate project data.
Case 2: Defect
Prediction Projects
Our defect prediction studies in-
volved two different cases (TelCo and
TechCo), but most of our respondents
(ve out of six) were from TelCo, which
participated in two consecutive proj-
ects completed between 2007 and
2011. The respondent from TechCo
has participated in more than 10 soft-
ware quality projects since 2012 and
reported that one of those projects,
whose nal technology is currently in
use, took two to three years.
For TelCo, we initially held kick-off
meetings with senior managers and the
quality assurance team, all of whom
dened the goal tasks as “software
measurement and defect prediction.”
We implemented a tool for collecting
static code metrics from past and cur-
rent projects’ source les. To train the
prediction model, we needed to clas-
sify the les of past projects as defective
or defect-free to understand if changes
were related to a defect x. In data
preparation, we realized that le classi-
cation was missing because matching
issues with le changes wasn’t manda-
tory during commits. To address this,
we initiated a development process
change that asked developers to enter
issue IDs into commit messages.
We built a prototype defect pre-
diction model and calibrated its per-
formance on test projects. Our final
model was evaluated on 10 releases
of nine projects in terms of defect-
detection effectiveness.4 We i nte-
grated the final model into existing
systems and reported our findings
based on its usage.9
The deployed technology was put on
hold after eight months of usage. Al-
though initial expectations were clearly
aligned with management’s long-term
business goals, implementing process
changes was quite challenging due to
resistance from the development teams.
Release schedules were very tight—bi-
weekly—and chaotic, so the develop-
ment team wasn’t eager to implement
additional tasks such as issue-commit
matching and prediction. Furthermore,
we observed that the company changed
its focus, from creating technology
through R&D to buying technology to
Release schedules were very tight and
chaotic, so the development team wasn’t
eager to implement additional tasks.
58 IEEE SoftwarE | www.computEr.org/SoftwarE
FOCUS: The Many Faces oF soFTware analyTics
increase its revenues, which ultimately
increased staff turnover. Other process
and quality improvement initiatives
(for example, transition from waterfall
to agile) were similarly terminated. We
interviewed ve people from the TelCo
team and learned that their initial ex-
pectations weren’t met after deploy-
ment. We also asked them the main
challenges faced after the deployment
and the reasons for stopping a com-
pany-wide deployment.
In contrast, even though we haven’t
completed the technology development
for TechCo yet, the project has gone
very smoothly. There’s a strong dedica-
tion to it because the practitioners in-
volved have built great knowledge of
and experience with measurement and
prediction modeling over the years.
They’re well aware of how software
analytics could improve development
activities in the long run, and they sup-
port the implementation of data-driven
decision making as part of their cor-
porate culture. It’s therefore easier for
us to collect accurate, complete data
and enrich it with additional informa-
tion about issues such as categories or
symptoms. We’ve also received periodic
group-level evaluations and feedback
for our predictions to further calibrate
the model.
Problems Encountered
The TelCo respondents claimed that
different factors put the project on
hold. First, issue-commit matching
was a major challenge: fast develop-
ment cycles and market pressure often
prevented the company from making
time-consuming process changes, and
senior management couldn’t force the
development team to follow certain
policies (for example, writing issue IDs
to commit messages). Consequently,
the model didn’t receive sufcient and
accurate data, which degraded its per-
formance. Second, the respondents
complained about a lack of actionable
outputs from defect prediction mod-
els—for example, a le’s defect prone-
ness didn’t help testers (because testers
primarily run black-box tests) or devel-
opers (because the list of defect-prone
les was too long). Although we pri-
oritized les by size, defect-proneness
probability, and modication time, the
nal list wasn’t condensed enough for
the development team. It wasn’t help-
ful to the testers either unless the les
were matched with the interfaces used
during testing. There was an attempt
within the company to combine the
defect prediction model’s outputs with
these interfaces, but due to high staff
turnover, it wasn’t completed.
In contrast, the TechCo respondent
often participated in successful defect-
prediction projects and still believes
that one of the key factors in successful
projects is tool support.
Ideas for Improvement
Respondents had various ideas for im-
proving the defect-prediction models.
Five of the six recommendations were
related to model integration and model
output, and the remainder was related
to model performance. One respondent
thought that release-based estimations
weren’t appropriate for TelCo, and ve
respondents found the binary classi-
cation (defect-prone/defect-free) to be
insufcient. They wanted to see addi-
tional information about defect causes,
such as the phases introduced, catego-
ries and severity levels, and further “ac-
tionable” recommendations related to a
specic defect, such as a recommenda-
tion about who should x it. They also
believed that the model should be easily
integrated as a plug-in.
The TechCo respondent thought
that defect-prediction models shouldn’t
be considered as a silver bullet to magi-
cally solve all quality assurance prob-
lems—rather, an ideal model imple-
mentation should work in a “setup and
forget” basis.
Indirect Benets
Five of the six TelCo respondents spoke
positively about their project experi-
ence and felt its main benet was in-
troducing a new way of thinking to the
organization. Only one respondent be-
lieved that the project had no contribu-
tions whatsoever.
Two of the ve respondents thought
that the project helped them increase
their awareness about problems and
limitations in their software processes.
They also elaborated on process changes
and concluded that these changes af-
fected team workow in general by
emphasizing the importance of defect
traceability, unit tests, and code reviews.
Two of the ve respondents claimed that
in some groups, these process changes
have been kept up and have had a last-
ing impact.
The TechCo respondent said the
main benets of these projects were the
way they identied weaknesses in com-
pany processes and led to organization-
wide process improvements.
Future Road Map
Table 1 summarizes the main chal-
lenges that the 12 participants stressed
and their suggestions for future
Fast development cycles and market
pressure often prevented the company
from making time-consuming changes.
SEptEmbEr/octobEr 2013 | IEEE SoftwarE
59
improvements. Some of these chal-
lenges resemble the key principles
proposed elsewhere10 to build similar
failure- prediction tools. Jacek Czer-
wonka and his colleagues argued that
information collected from metrics
should be insightful, that users need to
understand how to act based on met-
rics and model outputs, and that hav-
ing statistical experts on development
teams improves model development
and interpretation.10
Senior-level management support
is essential to initiating such projects,
aligning model outputs with business
objectives, integrating them into exist-
ing systems, and measuring their ben-
ets. Similarly, the absence of this sup-
port or changes in business priorities is
a major obstacle for model deployment.
In dynamic environments, these mod-
els are often the rst to be abandoned
because they’re too complex, manage-
ment requires statistics knowledge to
interpret them, or the organization
has lengthy deployment and feedback
cycles. To avoid this, researchers must
provide convincing evidence on the
benets of using these models to lower
levels in the organizational hierarchy
so that they become closely attached to
development processes.
The critical success factors in de-
ploying prediction models include
having mature development processes,
policies on measurement and analy-
sis, and highly qualied profession-
als to calibrate models and analyze
their outputs. Organizations must be
aware that software analytics is data
dependent, so as a rst step, they must
build a measurement culture. In our
own experience, the more mature the
organization in terms of its measure-
ment process, the less prone it is to or-
ganizational volatility. Organizations
should also employ highly qualied
personnel (as in TechCo’s case) who
perform software analytics. These
people, known as data scientists, can
Table 1
Common themes from our in-depth interviews.
Challenge Possible solutions
Model output Increase the model output ’s information content with, for example,
defect-severity/defect-type prediction, defect location, and phase- or
requirement-level effort estimation
Data collection Provide tool support to collect accurate and complete dat a
Lack of
actionable
outputs
Integrate prediction models into existing systems, for example, by
combining the results of defect prediction (defect-prone les) with test
interfaces to decide which interfaces to test rst, or creating a plug-in
that seamlessly works in development and testing environments
AYSE TOSUN M ISI RLI is a postdoctoral research fellow in the
Department of Information Processing Science at the Universit y of Oulu.
Her research interests include empirical software engineering, speci-
cally on mining sof tware data repositories, sof tware measurement,
software process improvement, software quality prediction models,
and applications of AI in sof tware recommendation systems. Misirli
received a PhD in computer engineering from Bogazici Universit y. She’s
a member of IEEE, ACM, and AA AI. C ontact her at ayse.tosunmisirli@
oulu..
BORA CAGL AYAN is a PhD candidate in the Department of Computer
Engineering at Bogazici Univer sity. His research interests include
empirical software engineering, data mining and recommendation sys-
tems. Caglayan received an MS in software engineering from Bogazici
Universit y. He is a student member of the IEEE Computer Society, ACM,
and ACM SIGSOF T. Contact him at bora.caglayan@boun.edu.tr.
AYSE BENE R is the director of the Data Science Lab and a professor
in the Department of Mechanical and Industrial Engineering at Ryerson
Universit y, Canada. Her research interests include intelligent models for
decision making under uncertainty, machine-learning methods to build
predictive models, cognitive science to model human behavior, and
game theoretic models to determine strategies in empirical sof tware
engineering, health sciences, and green IT. She’s a member of IEEE and
ACM. Contact her at ayse.bener@r yerson.ca.
BURAK TURHAN is a researcher and adjunct professor in the Depar t-
ment of Information Processing Science at the University of Oulu. His
research interests include empirical studies of soft ware engineering on
software quality, defect prediction, cost estimation, and data mining
for soft ware engineering. He’s a member of IEEE and the ACM. Contact
him at burak.turhan@oulu..
abouT The auThors
60 IEEE SoftwarE | www.computEr.org/SoftwarE
FOCUS: The Many Faces oF soFTware analyTics
calibrate analytics and present the re-
sults to all levels of the organization.
We recommend that companies create
an internal division on analytics/data
science to be responsible for measure-
ment and analytics.
A prediction model’s output should
lead to tangible actions at all levels in
the organization. Therefore, research-
ers must stop building predictive mod-
els that give binary output. Instead,
software analytics should focus on rea-
sons and casual relationships mapped
to business rules. Predictive models
should focus on managing business
rules to reduce costs, improve decision-
making, and increase business agility.
Finally, running a software analytics
project requires a lot of effort, so orga-
nizations should provide tool support.
Practitioners don’t want to spend time
collecting new data, lling in incomplete
metric values, or interpreting and using
outputs of these models. We believe it’s
necessary to support these projects with
all-in-one tools that manage data collec-
tion, analysis, and calibration, as well
as communicate with existing systems
in software organizations to provide
valuable information.
Our takeaway after so many
years of research is to of-
fer a tool (Dione11) that can
provide developer-level support to an-
swer the “how?” question, so that soft-
ware analytics are no longer perceived
as only “nice to have.” Fortunately, in
all the projects we’ve completed so far,
we’ve seen long-lasting process changes
being implemented and a major mind
shifts starting to happen—the rest will
follow.
References
1. Y. Dang et al., “XI AO: Tuning Code Clones
at Hands of Engineers in Practice,” Proc. 28th
Ann. Computer S ecurity Ap plication s Conf.
(ACSAC 12), ACM, 2012, pp. 369 –378.
2. T. Menzies, and F. Shul l, “Quest to Convi nc-
ing Evidence,” Making Sof tware: W hat Rea lly
Works, an d Why We Bel ieve It, A. Oram and
G. Wilson, eds ., O’Rei lly, 2011, pp. 3–16.
3. D. Zhang et al., “Software An alytics as a
Lear ning Case in Practice: Approache s and
Experiences,” Proc . Int’l Workshop M achine
Lear ning Tech nologies in Softwa re Eng. (MA-
LETS 11), ACM, 2011, pp. 55–58.
4. A. Tosun et a l., “Practica l Considerations
in Deploying Stat istic al Methods for Defect
Predic tion: A Case Study withi n the Turkish
Telecommun ications Industry,” Information
and Software Technology, vol. 52 , no. 11,
2011, pp. 1242–12 57.
5. E. Kocaguneli et al., “AI-Based Models for
Soft ware Ef fort Es timat ion,” Proc. 36th
EUROM ICRO Conf. Software Eng. and
Advanced Applications (SEA A 10), IEEE CS ,
2010, pp. 323 –326 .
6. B. Caglayan et a l., “Usage of Multiple Predic-
tion Models Based on Defect Categories,”
Proc. 6th Int’l Conf. Predicti ve Models in
Software Eng. (PROMI SE 10), ACM, 2010,
article 8.
IEEE TRANSACTIONS ON
CLOUD COMPUTING
The IEEE Transactions on Cloud Computing will publish peer reviewed articles that
provide innovative research ideas and applications results in all areas relating to cloud
computing. Topics relating to novel theory, algorithms, performance analyses and
applications of techniques relating to all areas of cloud computing will be considered
for the transactions.
The transactions will consider submissions specically in the areas of cloud security,
tradeoffs between privacy and utility of cloud, cloud standards, the architecture
of cloud computing, cloud development tools, cloud software, cloud backup and
recovery, cloud interoperability, cloud applications management, cloud data analytics,
cloud communications protocols, mobile cloud, liability issues for data loss on clouds,
data integration on clouds, big data on clouds, cloud education, cloud skill sets, cloud
energy consumption, cloud applications in commerce, education and industry. This
title will also consider submissions on Infrastructure as a Service (IaaS), Platform as a
Service (PaaS), Software as a Service (SaaS), and Business Process as a Service (BPaaS).
Editor-in-Chief: Rajkumar Buyya
For more information please visit: http://www.computer.org/portal/web/tcc
COMING SOON
IN 2013
tcc_cfp_wEIC_Hhalf_mjb.indd 1 1/30/2013 3:02:12 PM
SEptEmbEr/octobEr 2013 | IEEE SoftwarE
61
IEEE Computer Society’s Conference Publishing
Services (CPS) is now offering conference program
mobile apps! Let your attendees have their conference
schedule, conference information, and paper listings in
the palm of their hands.
The conference program mobile app works for
Android devices, iPhone, iPad, and the Kindle Fire.
For more information please contact cps@computer.org
CONFERENCES
in the Palm of Your Hand
Selected CS articles and columns
are also available for free at
http://ComputingNow.computer.org.
7. C. Boyce, and P. Neale, Conducting In-Depth
Intervie ws: A Gu ide for D esigning an d
Cond ucti ng In- Depth Intervie ws for Evalu-
ation Input: Monitoring and Evaluation—2,
Path nder Int ’l Tool Series , Path nder, 2006.
8. R. Czaja, and J. Blair, De signi ng Sur veys: A
Guid e to Deci sion s and Procedures, Sage,
2005.
9. A.T. Misirl i et al., “AI-Based Soft ware Defect
Predic tors: Appl ication s and Benets in a Case
St ud y,” AI, vol. 32, no. 2, 2011, pp. 57– 68.
10. J. Czerwonka et al., CRA NE: Failure Predic-
tion, Change Analysis and Test Prior itizat ion
in Practice—Experiences from W indows,”
Proc. 4th Int’l Conf. Software Testing, Veri-
cation an d Validation (ICST 11), IEE E CS,
2011, pp. 357–366.
11. B. C aglayan et al., “Dione: An I ntegrated
Measurement and D efect P rediction Solut ion,”
Proc. AC M SIGS OFT 2 0th In t’l Sy mp. Fou n-
dations of Software Eng. (FSE 12), ACM,
2012, a rticle 20.
... • Product documentation (31), discussion with colleagues (29), blogs and community forums (27), product websites (27) and books (27) are the relatively most frequently used information sources. • Source code repositories (23), professional training, conferences and workshops (23), information from client and partner companies (23), and online tutorials (23) are moderately used information sources. ...
... • Product documentation (31), discussion with colleagues (29), blogs and community forums (27), product websites (27) and books (27) are the relatively most frequently used information sources. • Source code repositories (23), professional training, conferences and workshops (23), information from client and partner companies (23), and online tutorials (23) are moderately used information sources. ...
... • Research articles (21), online courses (21) are less used information sources. • IT-magazines (27) and social networking sites (23) are the never used information sources. From the above data, we observe that the product documentation, discussion with colleagues and blogs and community forums are the most frequently used information sources. ...
Conference Paper
Software engineering practitioners have information needs to support strategic, tactical and operational decisionmaking. However, there is scarce research on understanding which information needs exist and how they are currently fulfilled in practice. This study aims to identify the information needs, frequency of their occurrence, the sources of information used to satisfy the needs, and the perception of practitioners regarding the usefulness of the sources currently used. For this purpose, a literature review was conducted to aggregate the current state of understanding in this area. We built on the results of the literature review and developed further insights through in-depth interviews with 17 practitioners. The findings from these two investigations were further triangulated by conducting a webbased survey (with 83 completed responses). Based on the results, it is inferred that information regarding product design, product architecture and requirements gathering are most frequently faced needs. Software practitioners mostly use blogs, community forums, product documentation and discussion with colleagues to address their information needs.
... Examples of in vivo CPDP studies for commercial/proprietary systems already exist in the literature, revealing the potential of cross project defect prediction. For example, Turhan et al. [83], [84] and Misirli et al. [91] were able to use the data collected from very different domains for predicting defects across projects, e.g., aerospace to telecommunication and white-goods, and reported reduced inspection efforts by 72% [91]. Their results, even though not as accurate as the within project predictions, demonstrated the applicability of CPDP in practical ways. ...
... Examples of in vivo CPDP studies for commercial/proprietary systems already exist in the literature, revealing the potential of cross project defect prediction. For example, Turhan et al. [83], [84] and Misirli et al. [91] were able to use the data collected from very different domains for predicting defects across projects, e.g., aerospace to telecommunication and white-goods, and reported reduced inspection efforts by 72% [91]. Their results, even though not as accurate as the within project predictions, demonstrated the applicability of CPDP in practical ways. ...
Article
Full-text available
Background:Cross project defect prediction (CPDP) recently gained considerable attention, yet there are no systematic efforts to analyse existing empirical evidence. Objective:To synthesise literature to understand the state-of-the-art in CPDP with respect to metrics, models, data approaches, datasets and associated performances. Further, we aim to assess the performance of CPDP vs. within project DP models. Method: We conducted a systematic literature review. Results from primary studies are synthesised (thematic, meta-analysis) to answer research questions. Results: We identified 30 primary studies passing quality assessment. Performance measures, except precision, vary with the choice of metrics. Recall, precision, f-measure, and AUC are the most common measures. Models based on Nearest-Neighbour and Decision Tree tend to perform well in CPDP, whereas the popular naive Bayes yield average performance. Performance of ensembles varies greatly across f-measure and AUC. Data approaches address CPDP challenges using row/column processing, which improve CPDP in terms of recall at the cost of precision. This is observed in multiple occasions including the meta-analysis of CPDP vs. WPDP. NASA and Jureczko datasets seem to favour CPDP over WPDP more frequently. Conclusion: CPDP is still a challenge and requires more research before trustworthy applications can take place. We provide guidelines for further research.
... In 2013, Misirli et al. [20] conducted in-depth interviews with twelve practitioners who were actively collaborating with them at that time in three industrial software analytics projects. These projects involved defect, effort, and quality prediction. ...
Article
Full-text available
Existing work on the practical impact of software engineering (SE) research examines industrial relevance rather than adoption of study results, hence the question of how results have been practically applied remains open. To answer this and investigate the outcomes of impactful research, we performed a quantitative and qualitative analysis of 4,354 SE patents citing 1,690 SE papers published in four leading SE venues between 1975–2017. Moreover, we conducted a survey on 475 authors of 593 top-cited and awarded publications, achieving 26% response rate. Overall, researchers have equipped practitioners with various tools, processes, and methods, and improved many existing products. SE practice values knowledge-seeking research and is impacted by diverse cross-disciplinary SE areas. Practitioner-oriented publication venues appear more impactful than researcher-oriented ones, while industry-related tracks in conferences could enhance their impact. Some research works did not reach a wide footprint due to limited funding resources or unfavorable cost-benefit trade-off of the proposed solutions. The need for higher SE research funding could be corroborated through a dedicated empirical study. In general, the assessment of impact is subject to its definition. Therefore, academia and industry could jointly agree on a formal description to set a common ground for subsequent research on the topic.
... In 2013, Misirli et al. [18] conducted in-depth interviews with twelve practitioners who were actively collaborating with them at that time in three industrial software analytics projects. These projects involved defect, effort, and quality prediction. ...
Preprint
Full-text available
Existing work on the practical impact of software engineering (SE) research examines industrial relevance rather than adoption of study results, hence the question of how results have been practically applied remains open. To answer this and investigate the outcomes of impactful research, we performed a quantitative and qualitative analysis of 4,335 SE patents citing 1,668 SE papers published between 1975-2017. Moreover, we conducted a survey study on 413 authors of 501 top-cited and awarded publications, achieving 25% response rate. Overall, researchers have equipped practitioners with various tools, processes, and methods, and improved many existing products. SE practice seems to value knowledge-seeking research and is impacted by diverse cross-disciplinary SE areas. Practitioner-oriented publication venues appear more impactful than researcher-, while industry-related tracks in conferences could enhance their impact. Some research works did not reach a wide footprint due to limited funding resources or unfavorable cost-benefit tradeoff of the proposed solutions. The need for higher funding in SE research could be corroborated through a dedicated empirical study. In general, the assessment of impact is subject to its definition. Therefore, academia and industry could jointly agree on a formal description to set a common ground for subsequent research on the topic.
... Researchers have extensively investigated methods and approaches to mine software repositories for a variety of reasons and stakeholders under the term "software analytics" [Buse and Zimmermann, 2010]. Often, these analytics approaches provide prediction models to support software managers to make decisions within their teams [Misirli et al., 2013, Buse and Zimmermann, 2012, Minku et al., 2016. One of the focal points of software analytics is bug and defect prediction. ...
... Misirli et al. [22] mention three common themes for software analytics projects they examined: 1) increase the model outputs information content with, for example, defect-severity or defect-type prediction, defect location, and phase-or requirement-level e ort estimation; 2) provide tool support to collect accurate and complete data; and, 3) integrate prediction models into existing systems, for example, by combining the results of defect prediction with test interfaces to decide which interfaces to test rst, or creating a plug-in that seamlessly works in development and testing environments. ...
Conference Paper
Background: During the period of one year, ING developed an approach for software analytics within an environment of a large number of software engineering teams working in a Continuous Delivery as a Service setting. Goal: Our objective is to examine what factors helped and hindered the implementation of software analytics in such an environment, in order to improve future software analytics activities. Method: We analyzed artifacts delivered by the software analytics project, and performed semi-structured interviews with 15 stakeholders. Results: We identified 16 factors that helped the implementation of software analytics, and 20 factors that hindered the project. Conclusions: Upfront defining and communicating the aims, standardization of data at an early stage, build efficient visualizations, and an empirical approach help companies to improve software analytics projects.
Article
Full-text available
In recent years, the application of artificial intelligence (AI) has become an integral part of a wide range of areas, including software engineering. By analyzing various data sources generated in software engineering, it can provide valuable insights into customer behavior, product performance, bugs and errors, and many more. In practice, however, AI for software analytics and business intelligence often remains at a prototypical stage, and the results are rarely used to make decisions based on data. To understand the underlying causes of this phenomenon, we conduct an explanatory case study consisting of and interview study and a survey on the challenges of realizing and utilizing artificial intelligence in the context of software-intensive businesses. As a result, we identify a vicious circle that prevents practitioners from moving from prototypical AI-based analytics to continuous and productively usable software analytics and business intelligence solutions. In order to break the vicious circle in a targeted manner, we identify a set of solutions based on existing literature as well as the previously conducted interviews and survey. Finally, these solutions are validated by a focus group of experts.
Chapter
This paper investigates factors affecting business analytics (BA) in software and systems development projects. This is the first study to examine business analytics continuance in projects from Pakistani software professional’s perspective. The data was collected from 186 Pakistani software professionals working in software and systems development projects. The data was analyzed using partial least squares structural equation modelling techniques. Our structural model is able to explain 40% variance of BA continuance intention, 62% variance of satisfaction, 69% variance of technological compatibility, and 59% variance of perceived usefulness. Technological compatibility and perceived usefulness are the significant factors that can affect BA continuance intention in software and systems projects. Surprisingly the results show that satisfaction does not affect BA continuance intention.
This volume constitutes the proceedings of the 20th IFIP WG 6.11 Conference on e-Business, e-Services, and e-Society, I3E 2021, held in Galway, Ireland, in September 2021.* The total of 57 full and 8 short papers presented in these volumes were carefully reviewed and selected from 141 submissions. The papers are organized in the following topical sections: AI for Digital Transformation and Public Good; AI & Analytics Decision Making; AI Philosophy, Ethics & Governance; Privacy & Transparency in a Digitized Society; Digital Enabled Sustainable Organizations and Societies; Digital Technologies and Organizational Capabilities; Digitized Supply Chains; Customer Behavior and E-business; Blockchain; Information Systems Development; Social Media & Analytics; and Teaching & Learning. *The conference was held virtually due to the COVID-19 pandemic.
Conference Paper
Full-text available
We are living in the era of the big data. The rapid development of scientific and data technology over the past decade has brought not only new and sophisticated analytical tools into financial and other industries, but also introduced the power of data science application in everyday strategic and operational management. Data analytics and science developments have been particularly valuable to financial organisations that heavily depend on financial information in their decision making processes. The article presents the research that focuses on the impact of the data and technology trends on decision making, particularly in finance industry. It covers an overview of the benefits associated with the decision analytics and the use of big data by financial organisations. The aim of the research is to highlight the areas of impact where the big data trends are creating disruptive changes to the way the finance industry traditionally operates. For example, we can see rapid changes to organisation structures, approach to competition and customer as well as the recognition of the importance of data analytics in strategic and tactical decision making. Investment in data analytics is no longer considered a luxury, but necessity, especially for the financial organisations in developing countries. Technology and data science are both forcing and enabling the financial industry to respond to transformative demands and adapt to rapidly changing market conditions in order to survive and thrive in highly competitive global environment. Financial companies operating in developing countries must develop strong understanding of data-related trends and impacts as well as opportunities. This knowledge should not only be utilised for survival efforts, but also seen as the opportunity to engage at global level through innovation, flexibility, and early adoption of data science benefits. The paper also recommends further studies in related areas, which would provide additional value and awareness to the organisations that are considering their participation in the global data and analytical trends.
Conference Paper
Full-text available
We present an integrated measurement and defect prediction tool: Dione. Our tool enables organizations to measure, monitor, and control product quality through learning based defect prediction. Similar existing tools either provide data collection and analytics, or work just as a prediction engine. Therefore, companies need to deal with multiple tools with incompatible interfaces in order to deploy a complete measurement and prediction solution. Dione provides a fully integrated solution where data extraction, defect prediction and reporting steps fit seamlessly. In this paper, we present the major functionality and architectural elements of Dione followed by an overview of our demonstration.
Conference Paper
Full-text available
Decision making under uncertainty is a critical problem in the field of software engineering. Predicting the software quality or the cost/ effort requires high level expertise. AI based predictor models, on the other hand, are useful decision making tools that learn from past projects' data. In this study, we have built an effort estimation model for a multinational bank to predict the effort prior to projects' development lifecycle. We have collected process, product and resource metrics from past projects together with the effort values distributed among software life cycle phases, i.e. analysis & test, design & development. We have used Clustering approach to form consistent project groups and Support Vector Regression (SVR) to predict the effort. Our results validate the benefits of using AI methods in real life problems. We attain Pred(25) values as high as 78% in predicting future projects.
Conference Paper
We present an integrated measurement and defect prediction tool: Dione. Our tool enables organizations to measure, monitor, and control product quality through learning based defect prediction. Similar existing tools either provide data collection and analytics, or work just as a prediction engine. Therefore, companies need to deal with multiple tools with incompatible interfaces in order to deploy a complete measurement and prediction solution. Dione provides a fully integrated solution where data extraction, defect prediction and reporting steps fit seamlessly. In this paper, we present the major functionality and architectural elements of Dione followed by an overview of our demonstration.
Conference Paper
During software development, engineers often reuse a code fragment via copy-and-paste with or without modifications or adaptations. Such practices lead to a number of the same or similar code fragments spreading within one or many large codebases. Detecting code clones has been shown to be useful towards security such as detection of similar security bugs and, more generally, quality improvement such as refactoring of code clones. A large number of academic research projects have been carried out on empirical studies or tool supports for detecting code clones. In this paper, we report our experiences of carrying out successful technology transfer of our new approach of code-clone detection, called XIAO. XIAO has been integrated into Microsoft Visual Studio 2012, to be benefiting a huge number of developers in industry. The main success factors of XIAO include its high tunability, scalability, compatibility, and explorability. Based on substantial industrial experiences, we present the XIAO approach with emphasis on these success factors of XIAO. We also present empirical results on applying XIAO on real scenarios within Microsoft for the tasks of security-bug detection and refactoring.
Article
Software analytics is to enable software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information for data-driven tasks around software and services. In this position paper, we advocate that when applying analytic technologies in practice of software analytics, one should (1) incorporate a broad spectrum of domain knowledge and expertise, e.g., management, machine learning, large-scale data processing and computing, and information visualization; and (2) investigate how practitioners take actions on the produced information, and provide effective support for such information-based action taking. Our position is based on our experiences of successful technology transfer on software analytics at Microsoft Research Asia.
Article
A huge wealth of various data exists in the software development process, and hidden in the data is information about the quality of software and services as well as the dynamics of software development. With various analytic and computing technologies, software analytics is to enable software practitioners to performance data exploration and analysis in order to obtain insightful and actionable information for data-driven tasks around software and services [1].
Conference Paper
Background: Most of the defect prediction models are built for two purposes: 1) to detect defective and defect-free modules (binary classification), and 2) to estimate the number of defects (regression analysis). It would also be useful to give more information on the nature of defects so that software managers can plan their testing resources more effectively. Aims: In this paper, we propose a defect prediction model that is based on defect categories. Method: We mined the version history of a large-scale enterprise software product to extract churn and static code metrics. and grouped them into three defect categories according to different testing phases. We built a learning-based model for each defect category. We compared the performance of our proposed model with a general one. We conducted statistical techniques to evaluate the relationship between defect categories and software metrics. We also tested our hypothesis by replicating the empirical work on Eclipse data. Results: Our results show that building models that are sensitive to defect categories is cost-effective in the sense that it reveals more information and increases detection rates (pd) by 10% keeping the false alarms (pf) constant. Conclusions: We conclude that slicing defect data and categorizing it for use in a defect prediction model would enable practitioners to take immediate actions. Our results on Eclipse replication showed that haphazard categorization of defects is not worth the effort.
Conference Paper
Building large software systems is difficult. Maintaining large systems is equally hard. Making post-release changes requires not only thorough understanding of the architecture of a software component about to be changed but also its dependencies and interactions with other components in the system. Testing such changes in reasonable time and at a reasonable cost is a difficult problem as infinitely many test cases can be executed for any modification. It is important to obtain a risk assessment of impact of such post-release change fixes. Further, testing of such changes is complicated by the fact that they are applicable to hundreds of millions of users, even the smallest mistakes can translate to a very costly failure and re-work. There has been significant amount of research in the software engineering community on failure prediction, change analysis and test prioritization. Unfortunately, there is little evidence on the use of these techniques in day-to-day software development in industry. In this paper, we present our experiences with CRANE: a failure prediction, change risk analysis and test prioritization system at Microsoft Corporation that leverages existing research for the development and maintenance of Windows Vista. We describe the design of CRANE, validation of its useful-ness and effectiveness in practice and our learnings to help enable other organizations to implement similar tools and practices in their environment.
Article
Building defect prediction models in large organizations has many challenges due to limited resources and tight schedules in the software development lifecycle. It is not easy to collect data, utilize any type of algorithm and build a permanent model at once. We have conducted a study in a large telecommunications company in Turkey to employ a software measurement program and to predict pre-release defects. Based on our prior publication, we have shared our experience in terms of the project steps (i.e. challenges and opportunities). We have further introduced new techniques that improve our earlier results.