Martin Höst

• PhD
• Professor (Full) at Lund University

Based on the scoping and planning of the experiment, the experiment design forms the input to the operation step presented in this chapter. Operation includes three sub-steps: preparation, execution, and data validation. In the preparation sub-step, participants need to be committed, and the instrumentation for the operation needs to be prepared. I...

This chapter presents an example to illustrate the experiment process. The presentation is focused on the steps for experimentation. Thus, separate sections for each step in the experiment process are presented to demonstrate the execution of the steps and the decisions made about the experiment presented.

The chapter discusses areas that are essential to empirical research. These include ethical concerns and the importance of replication. Moreover, theory building is presented in the context of empirical research. Furthermore, measurement in software engineering is introduced since it is essential when performing empirical research. The chapter conc...

Systematic literature studies are introduced in this chapter. Terms and concepts are defined, along with steps to go through when conducting systematic literature studies. The chapter continues with a discussion of systematic mapping studies and how systematic literature studies may become outdated as new research findings emerge and the studies ne...

The data collected in the operation step is input to the analysis and interpretation step in the experiment process. This step includes three sub-steps: descriptive statistics, data set reduction, and hypothesis testing. The chapter introduces several statistical analysis methods for the three sub-steps typically used when analyzing data from exper...

The fifth and final step in the experiment process is presented in this chapter. The chapter discusses essential aspects when documenting the findings and conclusions from an experiment. The focus is on the report structure. However, other types of documentation, such as replication packages, educational material, and packaging for industrial chang...

This chapter presents a decision-making structure for determining an appropriate research design for a specific study. A selection of research approaches is introduced to help illustrate the decision-making structure. The research approaches are described briefly to provide a basic understanding of different options. Moreover, the chapter discusses...

This chapter presents case study research. It discusses definitions of case study research and why such research is vital in software engineering. A stepwise case study research process is presented. It includes essential aspects such as design, planning, data collection, data analysis, validity threats, and reporting. Three research methods regula...

This chapter introduces and defines survey research. It discusses the characteristics of survey research and its purpose and objective. The chapter continues with a discussion of essential aspects to consider when designing a survey, including one of the main challenges in survey research, i.e., sampling. The chapter concludes by providing sources...

This chapter presents the first step, out of five, of the experimental process. The first step is scoping the experiment, where the foundation for the experiment is laid. Here, the scoping is done by using a goal template, including the following aspects: the object of study, purpose, quality focus, perspective, and context. The use of the goal tem...

The book’s final chapter includes an experiment published in the Empirical Software Engineering journal. The chapter aims to illustrate how an experiment may be written into a publishable form. The topic of the experiment is perspective-based reading, an inspection/review method, although the topic is of secondary importance in relation to the pres...

This chapter provides an overview of conducting experiments in software engineering. It defines experiment and quasi-experiment and introduces A/B testing. The chapter discusses experimentation principles and introduces essential concepts in experimentation. Moreover, a five-step process for conducting experiments is introduced. The five steps are...

Planning is vital when it comes to running an experiment. This chapter presents seven essential sub-steps to be addressed in planning, i.e., context selection, hypothesis formulation, variable selection, selection of subjects, experiment design type, instrumentation, and validity evaluation. The latter is essential to address upfront. Hence, the ch...

Context: Evidence-based software engineering (EBSE) aims to improve research utilization in practice. It relies on systematic methods to identify, appraise, and synthesize existing research findings to answer questions of interest for practice. However, the lack of practitioners' involvement in these studies' design, execution, and reporting indica...

Empirical software engineering research relies on good communication with industrial partners. Conducting joint research both requires and contributes to bridging the communication gap between industry and academia (IA) in software engineering. This study aims to explore communication between the two parties in such a setting. To better understand...

Evidence-based software engineering (EBSE) aims to improve research utilization in practice. It relies on systematic methods (like systematic literature reviews, systematic mapping studies, and rapid reviews) to identify, appraise, and synthesize existing research findings to answer questions of interest. However, the lack of practitioners’ involve...

Context The changes that are taking place with respect to environmental sensitivity are forcing organizations to adopt a new approach to this problem. Implementing sustainability initiatives has become a priority for the social and environmental awareness of organizations that want to stay ahead of the curve. One of the business areas that has, mor...

Background Assessing and communicating software engineering research can be challenging. Design science is recognized as an appropriate research paradigm for applied research, but is rarely explicitly used as a way to present planned or achieved research contributions in software engineering. Applying the design science lens to software engineering...

Software products are rarely developed from scratch and vulnerabilities in such products might reside in parts that are either open source software or provided by another organization. Hence, the total cybersecurity of a product often depends on cooperation, explicit or implicit, between several organizations. We study the attitudes and practices o...

Background: Communicating software engineering research to industry practitioners and to other researchers can be challenging due to its context dependent nature. Design science is recognized as a pragmatic research paradigm, addressing this and other characteristics of applied and prescriptive research. Applying the design science lens to software...

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations. Diligently performing risk and hazard analysis helps to minimize the potential harm of IT system failures on the society and increases the probability of their undisturbed operation. Risk and hazard analysis is an...

Security has been recognized as a leading barrier for IoT adoption. The growing number of connected devices and reported software vulnerabilities increases the importance firmware updates. Maturity models for software security do include parts of this, but are lacking in several aspects. This paper presents and evaluates a maturity model (HAVOSS) f...

Much empirical software engineering research aims at producing prescriptive knowledge that helps software engineers improve their work or solve their problems. But deriving general knowledge from real world problem solving instances can be challenging. In this paper, we promote design science as a paradigm to support producing and communicating pre...

In the emerging field of Internet of Things (IoT), where computerized devices are combined in creative new ways, there is a need to create Graphical User Interfaces (GUIs) for the systems being built, e.g. in the form of Android “apps”. This is generally a complicated, time consuming task. We report from a controlled experiment that evaluates a new...

A common situation is that an initial architecture has been sufficient and served its purpose in the initial phases of a project, but when the size and complexity of the product scales the architecture must be changed. The main objective of this research is to present experiences from changing an architecture into independent units, providing basic...

Case studies are largely used for investigating software engineering practices. They are characterized by their flexible nature, multiple forms of data collection, and are mostly informed by qualitative data. Synthesis of case studies is necessary to build a body of knowledge from individual cases. There are many methods for such synthe- sis, but t...

Industry involvement in open source software development has become a popular practice among companies which, e.g., share software development costs with other community participants or implement an open source based business model. An increased understanding of the underlying development structure, especially in a case where the community particip...

Software failures in medical devices can lead to catastrophic situations. Therefore, it is crucial to handle software-related risks when developing medical devices, and there is a need for further analysis of how this type of risk management should be conducted. The objective of this paper is to collect and summarise experiences from conducting ris...

The popularity of Open Source Software (OSS) has increased the interest in using it in safety critical applications. The aim of this study is to review research carried out on usage of open source code in development of safety-critical software and systems. We conducted a systematic mapping study through searches in library databases and manual ide...

The IDEM3 maturity model is a process improvement framework that can be used by an organisation to assess and improve their IT dependability management processes. The framework focuses on the coordination of IT management and safety management within an organisation. In this paper, an evaluation plan for the maturity model is presented to evaluate...

The popularity of Open Source Software (OSS) has increased the interest in using it in safety critical applications. The aim of this study is to review research carried out on usage of open source code in development of safety-critical software and systems. We conducted a systematic mapping study through searches in library databases and manual ide...

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations both public and private. Diligently performing risk and hazard analysis helps to minimize the societal harms of IT system failures. In this paper we present experiences gained by applying the System Theoretic Proce...

Involving perspectives into risk analysis brings a potential to increase the efficiency of the risks analysis task and confidence in the identified risks. In this paper, we report the results from an experiment designed to investigate if Perspective-Based Risk Analysis (PBRA) that involves different views and perspectives is more effective and offe...

Software development organizations are continuously looking for better ways to manage their projects. An emerging approach to achieve this is Inner Source, which refers to the adoption of Open Source development practices within the confines of an organization. With an Inner Source approach, individuals, teams, and departments within an organizatio...

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations both public and private. Diligently performing risk and hazard analysis helps to minimize the potential harm of the IT systems failures on the society and increases the probability of their undisturbed operation. I...

Context: Involving perspectives into risk analysis brings a potential to increase the efficiency of the risks analysis task and the confidence in the identified risks. Objective: The objective of the research carried out in this study is to investigate the effectiveness of perspective-based risk analysis (PBRA) method in comparison with a tradition...

Context: Involving perspectives into risk analysis brings a potential to increase the efficiency of the risks analysis task and the confidence in the identified risks. Objective: The objective of the research carried out in this study is to investigate the effectiveness of perspective-based risk analysis (PBRA) method in comparison with a tradition...

This paper presents the findings from a design study on a framework for flexible safety-critical software development, called SimPal. It is an extended version of a paper that was published in SAC'13 Proceedings of the 2013 ACM Symposium on Applied Computing, in which additional details about SimPal as well as a more extensive evaluation of the fra...

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers o...

Context. Development of safety-critical systems is mostly governed by process-heavy paradigms, while increasing demands on flexibility and agility also reach this domain. Objectives. We wanted to explore in more detail the industrial needs and challenges when facing this trend. Method. We launched a qualitative survey, interviewing engineers from f...

Service Level Agreements (SLA) are considered a good practice not only for IT outsourcing but also for IT management within an organisation. In this paper we study the usage of SLAs in municipal IT management. Municipal IT management traditionally involves a large organisation, often with a low IT maturity, but with high requirements on software qu...

Human beings make errors and that is nothing that we can avoid completely. We can however lower the risk of people doing wrong in situations where, for example, medical devices are used. The overall objective of the research presented in this paper is to investigate how usability testing can contribute to software risk management process in the med...

Context: At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. This means that there is a need for risk analysis in the development of this kind of systems. Risk analysis of technical systems has a long history in mechanical and electrical engineer...

This paper presents the findings from a design study of a model-based framework for safety-critical software development, called SimPal. The objective of the study was to better understand the necessary properties of such a framework and to learn more about the challenges of realizing it. Our research approach can be labeled as design research, whi...

The reuse and integration of Open Source Software (OSS) components provided by OSS communities is becoming an economical and strategic need for today's organizations. The integration of OSS components provides many benefits, but also risks and challenges. One of the most important risks is the lack of effective and timely OSS community support for...

ContextDuring systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines.AimsThis paper inve...

In a team, people sometimes leave the team and become replaced by new persons with less experience, and sometimes people participate in new activities and thereby obtain new knowledge. Different processes, in terms of different management strategies, can be followed, e.g., to introduce people to new tasks so they get new knowledge. There is a need...

Maturity models are widely used in process improvement. The users of a maturity model should be confident that the weak points of the assessed processes can be found, and that the most valuable changes are introduced. Therefore, the evaluation of maturity models is an important activity. In this paper, a mapping study of the literature on the evalu...

When an experiment is completed, the findings may be presented for different audiences, as defined in Fig. 11.1. This could, for example, be done in a paper for a conference or a journal, a report for decision-makers, a package for replication of the experiment, or as educational material. The packaging could also be done within companies to improv...

The experiment data from the operation is input to the analysis and interpretation. After collecting experimental data in the operation phase, we want to be able to draw conclusions based on this data. To be able to draw valid conclusions, we must interpret the experiment data.

Perspective-Based Reading (PBR) is a scenario-based inspection technique where several reviewers read a document from different perspectives (e.g. user, designer, tester). The reading is made according to a special scenario, specific for each perspective. The basic assumption behind PBR is that the perspectives find different defects and a combinat...

[Context and motivation] There is considerable flexibility in requirements specifications (both functional and non-functional), as well as in the features of available OSS components. This allows a collaborative matching and negotiation process between stakeholders such as: customers, software contractors and OSS communities, regarding desired requ...

Introduction Design of the Case Study Data Collection Data Analysis Reporting and Dissemination Lessons Learned

Software failures in medical devices can lead to catastrophic situations. Therefore is it crucial to handle software related risks when developing medical devices. This paper presents the experiences gained from an ongoing case study with a medical device development organisation. This part of the study focuses on the two first steps of the risk ma...

Large market-driven software companies continuously receive large numbers of requirements and change requests from multiple sources. The task of analyzing those requests against each other and against already analyzed or implemented functionality then recording similarities between them, also called the requirements consolidation task, may be chall...

The purpose of risk management in the development of safety-critical software is to eliminate or reduce harmful behaviour. In health-care it is essential to manage risk related to software due to its increased use in medical devices and other computer systems. This paper presents some of the experiences gained from an ongoing case study at a large...

Synthesis of case studies is different from synthesis of purely quantitative studies, for example, in that sampling and analysis in primary studies have been carried out differently, and that primary results are of a different nature. The objective of this research is to identify what challenges should be considered when choosing and using a method...

There are two types of research paradigms that have different approaches to empirical studies. Qualitative research is concerned with studying objects in their natural setting. A qualitative researcher attempts to interpret a phenomenon based on explanations that people bring to them [Denzin94]. Qualitative research begins with accepting that there...

Open source components can be used as one type of software component in development of commercial software. In development using this type of component, potential open source components must first be identified, then specific components must be selected, and after that selected components should maybe be adapted before they are included in the deve...

Context: The popularity of the open source software development in the last decade, has brought about an increased interest from the industry on how to use open source components, participate in the open source community, build business models around this type of software development, and learn more about open source development methodologies. Ther...

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per pa...

It is commonly acknowledged that the management of quality requirements is an important and difficult part of the requirements engineering process, which plays a critical role in software product development. In order to identify current research about quality requirements, a systematic literature review was performed. This paper identifies availab...

In many organisations a gap exists between IT management and emergency managemement. This paper illustrates how process improvement based on a maturity model can be used to help organisations to evaluate and improve the way they include IT dependability information in their emergency management. This paper presents the IDEM3 (IT Dependability in E...

Like many engineering programs in Europe, the final part of most Swedish software engineering programs is a longer project in which the students write a Master’s thesis. These projects are often conducted in cooperation between a university and industry, and the students often have two supervisors, one at the university and one in industry. In p...

This paper presents an extensive analysis of static software quality metrics changes for an open source enterprise database management system (DBMS), as the software was moved from the proprietary into open source software development environment. The software quality metrics of special interest for the research are cyclomatic complexity, effective...

When developing software-intensive products for a market-place it is important for a development organisation to create innovative features for coming releases in order to achieve advantage over competitors. This paper focuses on assessment of innovation capability at team level in relation to the requirements engineering that is taking place befor...

This document presents a tutorial on case study research methodology in software engineering, held at the 10th International Conference on Product Focused Software Development and Process Improvement (Profes).

Abstract,Case study is a suitable research methodology,for software engineering,research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims,at providing,an introduction to case study methodology,and,guidel...

Risk management is an important process and risk identification is an important part of this process, especially in development of medical software. This paper presents an experiment where physicians, developers and software developers for medical devices are asked to identify risk in a given scenario describing the procurement of a patient monitor...

In recent years governmental actors have become more and more dependent on IT systems for their responsibilities in a crisis situation. To avoid unexpected problems with the dependability of IT systems in the aftermath of a crisis it is important that such risks are identified and that measures can be taken to reduce the dependence on systems that...

There has been much recent interest in how to help students in higher education develop their generic skills, especially since this is a focus of the Bologna process that aims to standardize European higher education. However, even though the Master thesis is the final and often crucial part of a graduate degree and requires many generic skills...

The objective of the work presented in this paper is to design and develop a framework for simulation of requirements engineering processes. The framework is intended to be a support when simulation models are built by guiding the modeler in which components to use in this type of models and to speed up the process of developing simulation models....

For real time embedded systems software performance is one of the most important quality attributes. Although a variety of general activities has been proposed as a basis for controlling and predicting the software performance in software, these activities are not widely used in the mainte-nance and evolution of embedded platforms. During the maint...

Examensarbetet är kronan på verket på civilingenjörsutbildningen där teori och praktik sätts samman i ett större ingenjörsarbete. Hur kan man agera som handledare och examinator från universitetet för att säkerställa kvaliteten i detta viktiga utbildningsmoment? Vi presenterar en generell processmodell för examensarbete i fyra steg: uppstart, plane...

To measure the reliability of a website from a user's point of view, the uncertainly on the usage of the website has to be taken into account. In this paper we investigate the influence of this uncertainly on the reliability estimate for a web server. For this purpose a session based Markov model is used to model the usage extracted from the server...

Case study is an important research methodology for software engineering. We have identified the need for checklists supporting researchers and reviewers in conducting and reviewing case studies. We derived checklists for researchers and reviewers respectively, using systematic qualitative procedures. Based on nine sources on case studies, checklis...

The medical device industry is a constantly growing domain which makes use of more and more software products. Given the importance to this industry of dependable software components, rigorous software engineering techniques would seem to have an important role to play. However, in a recent survey of the industry we found a lower than expected rate...

A wide variety of the functions provided by today's medical devices relies heavily on software. Most of these capabilities could not be offered without the underlying integrated software solutions. As a result, the medical device industry has become highly interdisciplinary. Medical device manufacturers are finding an increasing need to incorporate...

In market-driven requirements engineering for platform-based development of embedded systems such as mobile phones, it is crucial to market success to find the right balance among competing quality aspects (aka non-functional requirements). This paper presents a conceptual model that incorporates quality as a dimension in addition to the cost and v...

During software project risk management, a number of deci- sions are taken based on discussions and subjective opinions about the importance of identified risks. In this paper, differ- ent people's opinions about the importance of identified risks are investigated in a controlled experiment through the use of utility functions. Engineering students...

Today embedded system development is a complex task. To aid the engineers new methodologies and languages are emerging. During the development the system is modeled using different tools and languages. Transformations between the models are traditionally done manually. We investigate the automation of this process, specifically we are looking at au...

The importance of prioritising requirements is widely recognised. A number of different techniques for prioritising requirements have been proposed, some based on an ordinal scale, others on a ratio scale. Some measurement scales provide more information than others, i.e. the ratio scale is richer than the ordinal scale. This paper aims to investig...

Web and software engineering are not only about technical solutions. They are to a large extent also concerned with organisational issues, project management and human behaviour. For disciplines like Web and software engineering, empirical methods are crucial, since they allow for incorporating human behaviour into the research approach taken. Empi...

During recent years we have seen several large-scale crises. The 9/11 terror attacks, tsunamis, storms, floods and bombings have all caused a great deal of damage. A common factor in these crises has been the need for information and one important source of information is usually web sites. In this work we investigate and design an overload control...

There is a need to identify factors that affect the result of empirical studies in software engineering research. It is still the case that seemingly identical replications of controlled experiments result in different conclusions due to the fact that all factors describing the experiment context are not clearly defined and hence controlled. In thi...

For real time embedded systems software performance is one of the most important quality attributes. Controlling and predicting the software performance in software is associated with a number of chal- lenges. One of the challenges is to tailor the established and rather gen- eral performance activities to the needs and available opportunities of a...

The purpose of this study is to investigate how agile development affects collaboration in an organization. Agile processes have received interest from the software development community during the last years as they address changes, such as new customer requirements or re-prioritization of development tasks, which is important to manage in softwar...

The purpose of this paper is to present results of introducing an agile process based on extreme programming, XP, in an evolutionary and maintenance software development environment. The agile process was introduced to a large software development organization. The process was applied by a team during eight months. The conclusions indicate that it...

Performance estimation is traditionally carried out when measurement from a product can be obtained. In many cases there is, however, a need to start to make predictions earlier in a development project, when for example different architectures are compared. In this paper, two methods for subjective predictions of performance are investigated. With...

During recent years we have seen several large-scale crises. The 9/11 terror attacks, tsunamis, storms, floods and bombings have all been unpredictable and caused a great deal of damage. One common factor in these crises has been the need for information and one important source of information is usually web sites. In this work three new sets of we...

There is a need to identify factors that affect the result of empirical studies in software engineering research. It is still the case that seemingly identical replications of controlled experiments result in different conclusions due to the fact that all factors describing the experiment context are not clearly defined and hence controlled. In thi...

Software architecture is recognized as a critical factor for successful products, but few have studied how organizations decide on architectural changes. We study the topic through several case studies. The changes are in all cases changes to the quality attributes of the system, and follow the same general process. We find that architectural chang...

In market-driven development of product software, a crucial decision for each candidate requirement is whether or not to select it for implementation in the next release. We present an analytical model of the requirements selection process, which is used for reasoning about decision quality. A network of queues, with two classes of jobs, models the...

System verification and validation are important activities in the software development process aiming at testing the software system in various quality aspects. The purpose is to investigate the important characteristics of the verification and validation activities in the software development process in an organization. A qualitative research met...

A good software architecture is becoming recognized as a major factor for successful products. There has been much research on the technical aspects of software archi- tecture and it is recognized that the driving requirements for architectures are "non-functional", but few have stud- ied how organizations decide on architectural changes. In this p...

In order to understand the state of test process practices in the software industry, we have conducted a qualitative survey, covering software development departments at 11 companies in Sweden of different sizes and application domains. The companies develop products in an evolutionary manner, which means either new versions are released regularly,...