Dag I. K. Sjøberg

University of Oslo · Department of Informatics

Background: Construct validity concerns the use of indicators to measure a concept that is not directly measurable. Aim: This study intends to identify, categorize, assess and quantify discussions of threats to construct validity in empirical software engineering literature and use the findings to suggest ways to improve the reporting of construct...

Organisations spend much money on Information Technology (IT) development and maintenance activities with the intention that these activities will create results that enable benefits for the organisations. This paper seeks to understand potential associations between IT development and maintenance activities and the adoption of benefits management...

Many companies migrate to microservices because they help deliver value to customers quickly and continuously. However, like any architectural style, microservices are prone to architectural technical debt (ATD), which can be costly if the debts are not timely identified, avoided, or removed. During the early stages of migration, microservice-speci...

Empirical research aims to establish generalizable claims from data. Such claims may involve concepts that must be measured indirectly by using indicators. Construct validity is concerned with whether one can justifiably make claims at the conceptual level that are supported by results at the operational level. We report a quantitative analysis of...

Background Using a microservices architecture is a popular strategy for software organizations to deliver value to their customers fast and continuously. However, scientific knowledge on how to manage architectural debt in microservices is scarce. Objectives In the context of microservices applications, this paper aims to identify architectural te...

Empirical research aims to establish generalizable claims from data. Such claims involve concepts that often must be measured indirectly by using indicators. Construct validity is concerned with whether one can justifiably make claims at the conceptual level that are supported by results at the operational level. We report a quantitative analysis o...

Empirical research aims to establish generalizable claims from data. Such claims involve concepts that often must be measured indirectly by using indicators. Construct validity is concerned with whether one can justifiably make claims at the conceptual level that are supported by results at the operational level. We report a quantitative analysis o...

Considerable resources are wasted on software projects delivering less than the planned benefits. Herein, the objective is to synthesize empirical evidence of the adoption and impact of benefits management (BM) in software development, and to suggest directions for future research. A systematic review of the literature is performed and identified 4...

Using microservices is a way of supporting an agile architecture. However, if the microservices development is not properly managed, the teams’ development velocity may be affected, reducing agility and increasing architectural technical debt. This paper investigates how to manage the use of shared libraries in microservices to improve agility duri...

This paper presents the main results from a survey investigation performed in Norwegian organizations within the area of software development and maintenance. The focus of the investigation was primarily on maintenance activities, but also a more general portfolio analysis has been undertaken. The results are based on responses from 53 Norwegian or...

Introduction: Software companies aim to achieve continuous delivery to constantly provide value to their customers. A popular strategy is to use microservices architecture. However, such an architecture is also subject to debt, which hinders the continuous delivery process and thus negatively affects the software released to the customers. Objecti...

In a recent article, Falessi et al. (2017) call for a deeper understanding of the pros and cons of using students and professionals in experiments. The authors state: we have observed too many times that our papers were rejected because we used students as subjects. Good experiments with students are certainly a valuable asset in the body of resear...

Members of high performing software teams collaborate, exchange information and coordinate their work on a frequent, regular basis. Most teams have the daily stand-up meeting as a central venue for these activities. Although this kind of meeting is one of the most popular agile practices, it has received little attention from researchers. We observ...

Background: Limiting the amount of Work-In-Progress (WIP) is considered a fundamental principle in Kanban software development. However, no published studies from real cases exist that indicate what an optimal WIP limit should be. Aims: The primary aim is to study the effect of WIP on the performance of a Kanban team. The secondary aim is to illust...

Members of high performing software teams collaborate, exchange information and coordinate their work on a frequent, regular basis. Most teams have the daily stand-up meeting as a central venue for these activities. Although this kind of meeting is one of the most popular agile practices, it has received little attention from researchers. We observ...

Agile principles were originally developed for small projects but are now widely used in larger projects with hundreds of developers. Teamwork quality is essential in any development work, but how does teamwork quality differ in small and large agile projects? We report from an explorative survey with 64 agile teams and 320 team members and team le...

It is relatively easy to generate and acquire much data from software engineering activities. The challenge is to obtain meaning from the data that represents something true, rather than spurious. To increase knowledge and insight, more theories should be built and used.

A decade ago, Kitchenham, Dybå and Jørgensen coined the term and provided the foundations for evidence-based software engineering (EBSE). A trilogy of papers was written for researchers, practitioners, and educators. They suggested that practitioners consider EBSE as a mechanism to support and improve their technology adoption decisions, and that r...

Most practitioners and researchers agree that when developing software, process affects product, and the usefulness of a process depends on the context. However, which processes are most useful for a specific company or project is generally unknown. When studying the relation between context, process and product, one challenge is that experiments o...

Small, self-directed teams are central in agile development. This article investigates the effect of teamwork quality on team performance, learning and work satisfaction in agile software teams, and whether this effect differs from that of traditional software teams. A survey was administered to 477 respondents from 71 agile software teams in 26 co...

Context: The trustworthiness of research results is a growing concern in many empirical disciplines. Aim: The goals of this paper are to assess how much the trustworthiness of results reported in software engineering experiments is affected by researcher and publication bias, given typical statistical power and significance levels, and to suggest i...

Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this res...

Context: Most of the software organizations that use agile methods organize daily team meetings. Aim: Our aim was to understand how daily meetings are conducted and identify obstacles that reduce their efficiency. Method: We observed 56 daily meetings and conducted 21 interviews in three different teams in two countries. We used the repertory grid...

Context: Code smells are assumed to indicate bad design that leads to less maintainable code. However, this assumption has not been investigated in controlled studies with professional software developers. Aim: This paper investigates the relationship between code smells and maintenance effort. Method: Six developers were hired to perform three mai...

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers o...

Context: Many metrics are used in software engineering research as surrogates for maintainability of software systems. Aim: Our aim was to investigate whether such metrics are consistent among themselves and the extent to which they predict maintenance effort at the entire system level. Method: The Maintainability Index, a set of structural measure...

Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an indispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engin...

Proponents of various processes and methods in the agile and lean communities have made many bold claims about usefulness, but those claims are rarely supported by empirical investigations. Data gathered from more than 12,000 work items collected over two years sheds light on Kanban versus Scrum.To acquire more knowledge about the performance of di...

ContextDuring systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines.AimsThis paper inve...

Background: It is trivial that the usefulness of a technology depends on the skill of the user. Several studies have reported an interaction between skill levels and different technologies, but the effect of skill is, for the most part, ignored in empirical, human-centric studies in software engineering. Aim: This paper investigates the usefulness...

The skills of software developers are important to the success of software projects. Also, when studying the general effect of a tool or method, it is important to control for individual differences in skill. However, the way skill is assessed is often ad hoc, or based on unvalidated methods. According to established test theory, validated tests of...

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per pa...

Compu ting technologies are changing everyone's social, political, economic, and cultural worlds.12 Meanwhile, scientists commonly believe that research in computing is advancing more quickly and just as quickly becoming obsolete more quickly than research in other scientific disciplines. A notable indicator is how quickly it stops being cited in t...

Code smells are particular patterns in object-oriented systems that are perceived to lead to difficulties in the maintenance of such systems. It is held that to improve maintainability, code smells should be eliminated by refactoring. It is claimed that classes that are involved in certain code smells are liable to be changed more frequently and ha...

Controlling IT complexity in a large organization's portfolio of constantly evolving software systems is crucial in a business world with tighter economic constraints and an increased focus on business agility. Unfortunately, there are no commonly accepted metrics for software complexity, and the few metrics that exist have not been validated as go...

It is a challenge for usability experts to perform usability inspections of interactive systems that are tailored to work-domains of which these experts have little knowledge. To counter this, usability inspections with work-domain experts have been explored, but little empirical research has been reported on these experts’ performance as evaluator...

Personality tests in various guises are commonly used in recruitment and career counseling industries. Such tests have also been considered as instruments for predicting the job performance of software professionals both individually and in teams. However, research suggests that other human-related factors such as motivation, general mental ability...

Software systems form the foundation of modern information society and many of those systems are among the most complex things ever created by man. The quotation that follows is from the 1999 President’s Information Technology Advisory Committee Report, but is as valid today as it was then: Our ability to construct…needed software systems and our...

Software systems form the foundation of the modern information society. As such, whether stand-alone or embedded in other technological systems, they are crucial to the continuing development of global society in all its spheres: social, political, economic, and cultural. For example, solving large-scale humanitarian and environmental problems call...

Many important phenomena within software engineering are difficult to define and measure. One example is software maintainability, which has been the subject of considerable research and is believed to be a critical determinant of total software costs. We propose using concept mapping, a well-grounded method used in social research, to operationali...

Several experiments on the effects of pair versus solo programming have been reported in the literature. We present a meta-analysis of these studies. The analysis shows a small significant positive overall effect of pair programming on quality, a medium significant positive overall effect on duration, and a medium significant negative overall effec...

The scientific study of a phenomenon requires it to be reproducible. Mature engineering industries are recognized by projects and products that are, to some extent, reproducible. Yet, reproducibility in software engineering (SE) has not been investigated thoroughly, despite the fact that lack of reproducibility has both practical and scientific con...

This paper reports an explorative experimental comparison of (i) an experience-sampling method called feedback collection and (ii) the think-aloud methods with respect to their usefulness in studies on program comprehension. Think-aloud methods are widely used in studies of cognitive processes, including program comprehension. Alternatively, as in...

Background: Experiments in which study units are assigned to experimental groups nonrandomly are called quasi-experiments. They allow investigations of cause-effect relations in settings in which randomization is inappropriate, impractical, or too costly. Problem outline: The procedure by which the nonrandom assignments are made might result in sel...

In mature sciences, building theories is the principal method of acquiring and accumulating knowledge that may be used in a wide range of settings. In software engineering, there is relatively little focus on theories. In particular, there is little use and development of empirically-based theories. We propose, and illustrate with examples, an init...

Empirical studies have become an integral element of software engineering research and practice. This unique text/reference includes chapters from some of the top international empirical software engineering researchers and focuses on the practical knowledge necessary for conducting, reporting and using empirical methods in software engineering. Pa...

Pair programming is a collaborative approach that makes working in pairs rather than individually the primary work style for code development. Because PP is a radically different approach than many developers are used to, it can be hard to predict the effects when a team switches to PP. Because projects focus on different things, this article conce...

An effect size quantifies the effects of an experimental treatment. Conclusions drawn from hypothesis testing results might be erroneous if effect sizes are not judged in addition to statistical significance. This paper reports a systematic review of 92 controlled experiments published in 12 major software engineering journals and conference procee...

We present the vision that for all fields of software engineering (SE), empirical research methods should enable the development of scientific knowledge about how useful different SE technologies are for different kinds of actors, performing different kinds of activities, on different kinds of systems. It is part of the vision that such scientific...

The terminology in the area of databases and data models is inconsistent and inaccurate and thus often confusing. Some fundamental database concepts are described in this paper. The description of these concepts are based on general concepts related to the development of the object oriented languages SIMULA, DELTA and BETA. A database is defined as...

A total of 295 junior, intermediate, and senior professional Java consultants (99 individuals and 98 pairs) from 29 international consultancy companies in Norway, Sweden, and the UK were hired for one day to participate in a controlled experiment on pair programming. The subjects used professional Java tools to perform several change tasks on two a...

Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature, role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software engineering. This article reports a systematic review of the explicit use of theory in a comprehen...

If empirical software engineering is to grow as a valid scientific endeavor, the ability to acquire, use, share, and compare data collected from a variety of sources must be encouraged. This is necessary to validate the formal models being developed within computer science. However, within the empirical software engineering community this has not b...

The amount of work on application systems being taken up by maintenance activities (work done on an IT-system after being put in production) has been one of the arguments of those speaking about a ‘software crisis’. We have earlier investigated the applicability of this notion, and propose to rather look at the percentage of work being done on appl...

Statistical power is an inherent part of empirical studies that employ significance testing and is essential for the planning of studies, for the interpretation of study results, and for the validity of study conclusions. This paper reports a quantitative assessment of the statistical power of empirical software engineering research based on the 10...

IntroductionEstimation LearningEstimation Feedback and Process GuidelinesExperiment: Application of the GuidelinesSummaryAcknowledgementReferences

Trust is considered one of the most important factors for successfully managing software outsourcing relationships. However, there is lack of research into factors that are considered important in establishing and maintaining trust between clients and ...

An important goal of empirical software engineering research is to cumulatively build up knowledge on the basis of our empirical studies, for example, in the form of theories and models (conceptual frameworks). Building useful bodies of knowledge will in general require the combined effort by several research groups over time. To achieve this goal,...

Attendees: Marcus Ciolkowski, Tore Dybå, Frank Houdek, Andreas Jedlitschka, Barbara Kitchenham, Dietmar Pfahl, Dag Sjøberg (chair), Sira Vegas and Laurie Williams There are many arguments in favour of theory use, such as structuring, conciseness, precision, parsimony, abstraction, generalisation, conceptualisation and communication [1,9,11]. Such...

The classical method for identifying cause-effect relationships is to conduct controlled experiments. This paper reports upon the present state of how controlled experiments in software engineering are conducted and the extent to which relevant information is reported. Among the 5,453 scientific articles published in 12 leading software engineering...

The classical method for identifying cause-effect relationships is to conduct controlled experiments. This paper reports upon the present state of how controlled experiments in software engineering are conducted and the extent to which relevant information is reported. Among the 5,453 scientific articles published in 12 leading software engineering...

Several approaches have been proposed for the transition from functional requirements to object-oriented design. In a use case-driven development process, the use cases are important input for the identification of classes and their methods. There is, however, no established, empirically validated technique for the transition from use cases to clas...

Objective: To improve the qualitative data obtained from software engineering experiments by gathering feedback during experiments. Rationale: Existing techniques for collecting quantitative and qualitative data from software engineering experiments do not provide sufficient information to validate or explain all our results. Therefore, we would li...

Software design patterns seek to package proven solutions to design problems in a form that makes it possible to find, adapt and reuse them. To support the industrial use of design patterns, this research investigates when, and how, using patterns is beneficial, and whether some patterns are more difficult to use than others. This paper describes a...

A fundamental question in object-oriented design is how to design maintainable software. According to expert opinion, a delegated control style, typically a result of responsibility-driven design, represents object-oriented design at its best, whereas a centralized control style is reminiscent of a procedural solution, or a "bad" object-oriented de...

The results from the study described in this paper suggest that customer expectations of a project's total cost can have a very large impact on human judgment-based estimates (expert estimates) of the most likely use of software development effort. The information that the customer expectations did not constitute valid input data for making estimat...

Estimation by analogy is, simplified, the process of finding one or more projects that are similar to the one to be estimated and then derive the estimate from the values of these projects. If the selected projects have an unusual high or low productivity, then we should adjust the estimates toward productivity values of more average projects. The...

Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests that software processes improvement should be based on the continuous iteration of characterization, goal setting, selection of improved technology, monitoring and analysis of its effects. This paper describes experiences from the empirical studies in two...

When estimating software development effort, it may be useful to describe the uncertainty of the estimate through an effort prediction interval (PI). An effort PI consists of a minimum and a maximum effort value and a confidence level. We introduce and evaluate a software development effort PI approach that is based on the assumption that the estim...

Use case models capture and describe the functional requirements of a software system. A use case driven development process, where a use case model is the principal basis for constructing an object-oriented design, is recommended when applying UML. There are, however, some problems with use case driven development processes and alternative ways of...

An important goal of most empirical software engineering experiments is the transfer of the research results to industrial applications. To convince industry about the validity and applicability of the results of controlled software engineering experiments, the tasks, subjects and the environments should be as realistic as practically possible. Suc...

Use case models capture and describe the functional requirements of a software system. A use case driven development process, where a use case model is the principal basis for constructing an object-oriented design, is recommended when applying UML. There are, however, some problems with use case driven development processes and alternative ways of...

Several processes have been proposed for the transition from functional requirements to an object-oriented design, but these processes have been subject to little empirical validation. A use case driven development process is often recommended when applying UML. Nevertheless, it has been reported that this process leads to problems, such as the dev...