Architecture conformance analysis using model-based testing: A case study approach

Full text

Architecture Conformance Analysis using Model-Based Testing:
A Case Study Approach
Burak Uzun, Bedir Tekinerdogan
Information Technology Group, Wageningen University, Wageningen, The Netherlands
{burak.uzun|bedir.tekinerdogan@wur.nl
Context: The architectural drift problem defines the discrepancy between the architecture description and
the code. Deviations of the code from the architecture can occur if architectural constraints as defined in
the architectural models are not implemented in the code. For large-scale systems manually checking the
consistency of the architecture with the code is not trivial and cumbersome.
Objective: The overall objective of this paper is to propose and analyze the effectiveness and practicality
of an architecture conformance analysis approach using model-based testing (ACAMBT) approach for
checking the consistency between architectural models and the code. Hereby, we consider the case whereby
the architecture is assumed correct, and the code needs to align with the architecture.
Method: We propose a model-based testing (MBT) that uses architectural views to automatically derive
test cases for checking the architectural constraints in the code. We have evaluated the approach and the
corresponding toolset for a real industrial case study using a systematic case study protocol. Further we
have adopted exhaustive fault-injection techniques to detect the constraint violations.
Results: The evaluation of the approach on real code showed that deviations with the architectural
constraints could be easily detected in the code.
Conclusion: We can conclude that ACAMBT is effective for identifying inconsistencies between the
architecture views and the code for the defined view constraints. Our survey study with practitioners
showed that adopting the ACAMBT approach is practical and easy to use. The approach as such can be
considered as a complimentary tool to the existing testing and reflexion modeling approaches.
Keywords: model-based testing, software architecture, architecture conformance analysis, case study
1. Introduction
Software systems are rarely static and need to evolve over time due to bug fixes or new requirements. This
situation causes the so-called architectural drift problem which defines the discrepancy between the
architecture description and the resulting implementation [1]. A popular approach for coping with this
problem is reflexion modeling which usually compares an abstract model of the code with the architecture
model to identify the differences [10][30]. Identifying the discrepancies between the code and the
architecture in this manner can be cumbersome for large scale software system. Hence, this form of
conformance checking would benefit from automation to detect the flaws due to the inconsistencies
between the architecture and the code.
Software testing is the traditional approach for the verification of the code to identify possible mismatches
between expected and present requirements of the system [2][32]. Software is correct if and only if each
valid input to the system produces an output according to the system specifications. In general, exhaustive
testing is not practical or tractable for most real programs due to the large number of possible inputs and
sequences of operations [32]. As a result, selecting set of test cases which can detect possible flaws of the
system is the key challenge in software testing.
Model based testing (MBT) addresses this challenge by automating the generation and execution of test
cases using models based on system requirements and behavior [12][42]. MBT relies on models to automate

the generation of the test cases and their execution. A model is usually an abstract, partial presentation of
the desired behavior of a system under test (SUT). MBT can use different representations of the system to
generate testing procedures for different aspects of the software systems. Example models include finite
state machines (FSMs), Petri Nets, I/O automata, and Markov Chains. A recent trend in MBT is to adopt
software architecture models to provide automated support for the test process leading to the notion of
model-driven architecture-based testing. Software architecture is different from the other design
representations since it provides a gross-level representation of the system at higher abstraction levels
[8][13][37].
Model-driven architecture based testing can be used for various purposes in software testing
[6][16][34][45]. In this paper, we propose a model-driven architecture based testing approach for checking
the consistency between the architecture and the code (ACAMBT). When considering architectural
consistency checking in principle three different types of scenarios can be distinguished (1) the architecture
is assumed correct and code needs to conform to architecture (2) the code is considered correct, and
architecture needs to be aligned to the code (3) neither the architecture nor the code is considered definitive
and both can be adapted to realize consistency. Based on the industrial context our study presents an
approach in which we make the assumption that the architecture is correct and that it is required to check
whether the provided code conforms to the architecture. The assumption that the architecture is correct
while the code needs to be aligned with the architecture has also been addressed in various different studies
[6][17][28][29][34][43].
In this paper we report on our experiences using such an ACAMBT approach and evaluate it with respect
to effectiveness and practicality within a real industrial context. To this end, we apply a systematic case
study approach using guidelines provided by Runeson and Höst [36] for a large-scale software company in
the context of a smart e-government system. We have also applied a survey study using guidelines by
Kitchenham’s et al [18][19][20][21][22] to check the practicality of the adopted approach.
The outline of the paper is organized as follows: section 2 provides the background on reflexion modeling,
and model-based testing. Section 3 presents an example industrial case and the problem statement. Section
4 presents the approach for conformance checking using model driven architecture-based testing. Section
5 presents the corresponding tool that implements the approach. Section 6 describes the architecture of the
system that the industrial case study is performed on, a big data e-government system. Section 7 discusses
the case study design along with the survey study design adopting the ACAMBT approach. Section 8
provides the discussion and elaborates on the results, section 9 the related work, and finally, section 10
presents the conclusion.
2. Preliminaries
2.1. Reflexion Modeling
Each software architecture design is the result of a broad set of design decisions and their justifications,
that is, the design rationale [38]. To capture and communicate these architecture decisions a proper
documentation of the software architecture is needed. The architecture documentation usually includes a
set of architecture views which are developed based on the corresponding architecture viewpoint. Each
architecture viewpoint and associated, instantiated architecture view is important for one or more
stakeholders and their concerns. The software architecture documentation as such can be used to serve as a
guideline for the subsequent life cycle activities, to realize and control the design decisions.
Unfortunately, software systems are rarely static and need to evolve over time due to bug fixes or new
requirements. This situation causes the so-called architectural drift problem which defines the discrepancy
between the architecture description and the resulting implementation [10][23][24][30][35][39]. This

architectural drift can be even introduced during the initial implementation of the architecture due to lack
of knowledge about the architecture or stringent time-to-market constraints. The existence of discrepancies
between the architecture description and its implementation is not a merely theoretical issue but may
directly lead to performance, security issues as well as increased maintenance time and cost, because the
original design goals have been lost. In the extreme cases, a system’s architecture may even deteriorate to
a degree where further development is not feasible, leading to a complete system re-implementation.
The architecture and the code can be inconsistent in different ways for different views. If the relations that
are present in the architecture are also found in the implementation, then this is convergent relation. In case
the architecture relation is not present in the implementation then this is called an absence relation. Finally,
if the implementation includes relation that is not present in the architecture, then this is called divergence
relation. Architectural violations are due to absence or divergence relations.
A successful technique that is used for architecture consistency checking is the reflexion modeling
approach, as proposed in the software architecture design community [23][30]. A reflexion model is the
model produced by the technique which allows comparison of the architectural design with a derived,
abstracted model of the code’s dependencies. Typically, a reflexion model highlights the differences
between the code and the architecture and, as such, defines the extent of the architectural drift problem.
Based on the reflexion model either the architecture design, the code or the abstracted model of the code
(the mapping) can be adapted to keep the consistency among both artefacts.
Consistency Checker (Tool) DeveloperArchitect
Create
Architecture Views
Update
Architecture
Reflexion Model
Analysis
Notify
Architect
Notify
Developer
[action
required]
[violations
detected]
Code
Create Reflexion
Model
Develop
Code
Update
Code
Reflexion
Model
[action
required]
Architecture
View
Abstract
Model
Map Code to
Abstract Model
Figure 1. Activity diagram showing the steps in reflexion modeling for architecture-to-code conformance
Figure 1 shows the common workflow representing the reflexion modeling approach. In principle, a
reflexion model allows a software developer to view the structure of a system's source through a chosen
high-level (often architectural) view. To check the consistency between the architecture model and the code,
an abstract model of the code is derived. In the activity diagram it is shown that this is carried out by the
developer. Sometimes this activity can also be carried out by the architect (who could be a developer as
well). This abstract model contains elements such as packages, classes, methods and variables of the source,
grouped together as the 'architectural elements' that might be included in architectural model. The two
models are then compared to each other by parsing of the system and assessing the actual relations between
the architectural elements in the abstract model. The results of the comparison are presented to the user
through a Reflexion Model. The reflexion model explicitly represents the convergence, the divergence, and

the absence relations. If multiple architecture views are adopted then typically for each architecture view a
separate reflexion model can be created.
By analyzing the reflexion model, the architecture, the code or the mapping rules can be altered. Usually
architecture conformance analysis approaches that apply reflexion modeling include tools for modeling the
architecture, modeling the mappings, deriving the abstract model from the source code, checking the
consistency, and generating the resulting reflexion model.
The architecture of a system is usually modeled using multiple architectural views that represents the system
with respect to a particular concern. Because multiple concerns need to be taken into account when
designing a system, multiple architecture views can help to separate the concerns and, as such, support the
modeling, understanding, communication and analysis of the software architecture for different
stakeholders. The existence of multiple architecture views requires that conformance address all these
views, in particular the views that relate to implementation concerns.
2.2. Model Based Testing
Similar to architecture modeling, the primary reason for modeling in general is usually defined as a means
for communication, analysis or guiding the production process. Usually models are characterized based on
their level of precision, and can be considered as a Sketch, a Blueprint, or an Executable [25]. An executable
model is a model that has everything required to produce the desired functionality of a single domain.
Executable models are more precise than sketches or blueprints and can be interpreted by model compilers.
In model-driven software development the concept of models can be considered as executable models. This
is in contrast to model-based software development in which models are used as blueprints at the most. The
language in which models are expressed is defined by meta-models. As such, a model is said to be an
instance of a meta-model, or a model conforms to a meta-model.
Model-driven development has been used in different phases in the lifecycle including software testing. In
the traditional test process, the design of test cases and the oracles as well as the execution of the tests are
performed manually. This manual process is time consuming and less tractable for human testers. Model-
Based Testing (MBT) relies on models of system requirements and behavior to automate the generation of
the test cases and their execution [42]. There are several motivations to perform MBT such as easy test
maintenance, automated test design and enhancing test quality. Figure 2 presents the common workflow
for MBT.
Based on the Test Requirements and the Test Plan a Test Model is constructed. A model is usually an
abstract, partial presentation of the desired behavior of a system under test (SUT). The test model is used
to generate test cases that together form the Abstract Test Suite. The test cases in the abstract test suite lack
the detail needed by the SUT and as such are not directly executable. Hence, in the third step, the abstract
test suite is transformed to a concrete or executable test suite. This is typically done using a transformation
tool, which translates each abstract test case to an executable test case. An advantage of the separation
between abstract test suite and concrete test suite is the platform and language independence of the abstract
test cases. The same abstract test case can be reused in different test execution environments. In the fourth
step, the concrete test cases are executed on the SUT. The test execution will result in a report that contains
the outcome of the execution of the test cases. In the final, fifth step, these results are analyzed and if needed
corrective actions are taken: for each test that reports a failure, the cause of the failure is determined and
the program (or model) is corrected.

Figure 2. Process of model-based testing
3. Industrial Case and Problem Statement
In this section we will describe the industrial case that is used to both illustrate the approach (next section)
and to validate the approach. For confidentiality, the name of the private company is not provided. The case
study concerns an e-government system describing the use of ICT to facilitate the operation of government
to provide better public services to citizens and businesses. In general, e-government includes different
models including government-to-government (G2G), government-to-business (G2B), and government-to-
citizen (G2C). Figure 3 shows the general workflow of the case study system. Citizens can request services
through an internet browser, which is mapped to e-forms that trigger process requests. The process requests
are handled by an intermediate Mid-Office that sends requests to the corresponding government clerk. The
required data is derived from the data administration database.
In our case we are dealing with a G2C e-government system in which the end users of the system are
government clerks that provide services to citizens. There are around six thousand online users on average
throughout the week. Users generate structured data conforming to business rules defined for smart e-
government system. The system which utilizes the relational database solutions handles more than 7 million
database transactions per day. Batch processes run nightly on the real-time data to create batch views and
the required reports and statistics.
KEY artefactprocess step
dataflow controlflow
Test
Requirements
1. Test Model
Construction
Test Plan
Test Model
2. Abstract Test
Case Generator Abstract Test
Suite
3. Concrete Test
Case Generator Concrete Test
Suite
4. Test Execution
Report
5. Analyze Test
Results

Government
Web Server
Government
Clerk
Internet
Browser
E-Form
E-form
Filled out
process
request
send
request
Citizen
request
service
E-Forms
Data
Administration
retrieve
related data
Mid-Office
data access
functions
Figure 3. General Workflow of the G2C E-Government System
To guide the development of the system according to stakeholder concerns, a software architecture design
is provided. For maintenance and consistency purposes it is important that the code of the E-Government
system follows the corresponding architecture. In the following sections we will elaborate on the
architecture models. Figure 4 shows, as an example, the layered architecture that is often used in e-
government systems.
Client Layer Client Applications Webservices
Service Layer
Server-Side Validation Layer
Business Logic Layer
Data Storage Layer
Authentication& Authorization
Relational
Data Base
No-Sql
Database Logs Archives DocumentsMedia
Persistence Layer
Figure 4. Example layered architecture of G2C E-Government System
As shown in the figure the architecture includes the three layers: Client Layer, Service Layer, and Data
Storage Layer. The Client Layer includes the module for e-forms, webservices and the human computer
interaction, the service layer includes the modules for the validation, authorization and e-government
business logic, and finally the data storage layer includes the functionality for storing and accessing the
persistent data.
For consistency it is required that the code conforms to the architectural constraints. This implies that the
defined layers and the relations among the layers are not violated in the code. This is required for the layered
view of the architecture. In case of multiple different architecture views, checking the inconsistencies
requires that conformance analysis must be carried out for the code and each of the selected views.
Likewise, the code should reflect the architectural constraints in each of the views. For small scale systems
with a few architecture views this might be carried out manually. Yet the conformance analysis can soon
become cumbersome and require further automation for larger systems.

4. Conformance Checking using Model-Driven Architecture Based Testing
For verifying whether the code is consistent with the architecture views we propose an architecture
conformance analysis using model-based testing approach (ACAMBT) in which we adopt architecture
views for deriving the test cases for conformance checking of the code with respect to architectural view
constraints. Figure 5 presents the generic workflow for ACAMBT. In the following we discuss the process
in more detail.
MDABT
Consistency Checker (Tool) DeveloperArchitect
Create
Architecture Views
Update
Architecture
Execute Test Cases
Notify
Architect
Notify
Developer
[action
required]
[violations
detected]
Code
Create Test Model for
Architecture View
Develop
Code
Update Code
Concrete Test
Suite View V
[action
required]
Architecture
View
Create View C riteria
Architecture
View Criteria
Test Model for
Architecture View
Create
Test Suite
Figure 5. Process model of architecture conformance analysis using model-based testing
Similar to the earlier reflexion modeling process as shown in Figure 1 the first step consists of developing
the architecture views. The architecture views are the models that will be used to eventually generate the
required test cases for conformance checking for each architecture view constraints. Different architecture
view models can be selected including module views, component and connector views and allocation views
[4]. Figure 4 shows, for example, the layered view of the adopted case study. The activity Create View
Criteria defines the architectural constraints for the corresponding architecture viewpoint. Every viewpoint
will impose different constraints for the conformance with the code. For example, for the given layered
view of the e-government system, the Layered View criteria implies that layers exist in the code, and the

layered property constraints are respected. The latter means that modules in Client Layer can only access,
modules in the Service Layer, and modules in the Service Layer are only allowed to access modules in the
Data Storage Layer. Similarly, for example, conformance checking based on the decomposition view with
the code will require checking whether the code has the same decomposition as defined in the architecture
decomposition view. Conformance checking based on the uses view will require checking the architectural
uses relations in the code. Hence, specific criteria will be defined for each architecture viewpoint.
Once the architecture views and the corresponding view criteria have been defined, in the next activity
Create Test Model for Architecture View a test model for the view will be developed that will be used to
generate the concrete test suite. The concrete test suite will include test cases dedicated for the
corresponding view to check the potential inconsistencies in the code. The concrete test suite as such will
be executed on the code. In case of inconsistencies the architect or the developer will be notified. For
example, based on the Layered View constraint that the modules in the Client Layer can access only
modules in the Service Layer, a test case needs to be created that checks whether this is the case in the code.
Similarly, other test cases need to be defined for the criteria of the corresponding views.
5. ACAMBT Conformance Checking Tool Framework
In the previous section, we have provided a generic process for ACAMBT which is agnostic to the adopted
tools. In principle, the process can be defined using different tool implementations. In this section, we
describe the ACAMBT Checking Tool framework that we have implemented for the generic process. The
implementation is based on the Eclipse Epsilon environment that contains languages and tools for code
generation, model to model transformation, model validation, comparison, migration and refactoring [5].
A snapshot of the tool framework is shown in Figure 6. In the following subsections we explain the activities
of the overall process and the adopted tools.
Figure 6. Snapshot of the toolset that implements the approach
Package Explorer EGL Template Editor Model View
EGX Editor
HUTN Editor

5.1 Modeling Architecture Viewpoints
For modeling the architecture viewpoints, we have first defined the corresponding metamodels for the
viewpoints and then mapped these to domain specific languages using the Eclipse Ecore model [9][11][12].
The metamodels of the selected viewpoints are shown in Figure 7.
Decomposition Viewpoint Metamodel
Shared Data Viewpoint Metamodel
Uses Viewpoint Metamodel
Generalization Viewpoint Metamodel
Layered Viewpoint Metamodel
Figure 7. Metamodels of the selected viewpoints

These metamodels are represented as Ecore models in the Eclipse IDE. Each of these metamodels have
been developed after a thorough domain analysis on the corresponding viewpoints. For this we have
analyzed the viewpoints as discussed by Clements et al. [4][40].
5.2 Modeling Architecture Views
For modeling the architecture views for a particular case, we use the Human-Usable Textual Notation
(HUTN) which is an OMG standard for storing models in a human understandable format. An example of
the decomposition view is shown in Figure 8. In it, a system is illustrated that consists of three sub-modules
X, Y and Z. The module X consists further of sub-modules X1 and X2, while the module Y consists of sub-
modules Y1, Y2, Y3, and Z is not further decomposed. Architecture view models can also be modeled via
Ecore diagram interface.
Figure 8. Example HUTN model for the decomposition view
5.3 Modeling Architecture Viewpoint Criteria
As stated before, each architecture viewpoint will impose a different set of constraints for conformance
checking. Table 1 lists the set of constraints for the selected set of viewpoints.

Table 1. Identified conformance checking constraints for the selected viewpoints
Viewpoint
Constraints
Decomposition
• Does every element in the view model appear in the code?
• Does every sub-element in the view model appear in the code?
• Does every sub-element exist under the corresponding element in the code?
Uses
• Does every uses relation in the view model appear in the code?
• Does every source element appear in the code?
• Does every target element appear in the code?
Generalization
• Does each implementation or inheritance relation exist in the code?
• Does each relation child and parent element exist in the code?
• Are each relation’s child and parent element in the model have same child’s
and parent’s in the code?
Layered
• Does each layer in the relation given in model exist in the code?
• Does each allowed to use below relation in given model exists in the code?
• Does each layer only use functions in the next lower layer?
Shared Data
• Does each data accessor exist in the code?
• Does each attachment of data accessor exist in the code?
To specify these criteria for architecture viewpoints we have applied the Epsilon Generation Language
(EGL) [11] which is a template-based model-to-text language for generating code, documentation and other
textual artefacts from models. View criteria are defined for each view and used in the construction of EGL
templates. An example of an EGL template that implements the constraints for the decomposition viewpoint
is shown in Figure 9. The EGL template checks the three constraints for the decomposition viewpoint as
shown in Table 1. As it can be seen from the EGL template each constraint is implemented as an assertion.
Since we have selected 5 viewpoints we have thus 5 EGL templates where each EGL template
implementing the constraints of the corresponding viewpoints.
Figure 9. EGL Template for conformance checking constraints of the decomposition viewpoint

5.4 Creating Test Model for Architecture View
So far, the architecture views have been defined as HUTN models and the corresponding constraints as
EGL templates. These two inputs will be used to generate the concrete test suite for each view. For this we
use the Epsilon Generation Runner (EGX), which is a runner for EGL templates that can parameterize the
defined templates in which transformation rule is defined. An example of the EGX model for the
decomposition view is shown in Figure 10. The model takes as input decomposition which represent the
decomposition view and executes the template Decomposition2Unit.egl template of Figure 9. The output
is written to the file “gen/TestDecomposition.java” and defines the concrete JUnit test suite for the
decomposition view.
Figure 10. EGX Model incorporating rules for generating test cases for the decomposition viewpoint
5.5 Creating Concrete Test Suite for Architecture View
The execution of EGX file generates a single JUnit test case which can be executed automatically. An
example concrete test suite for the decomposition view is shown in Figure 11.
public class TestDecomposition {
@Test
public void testSDecomposedOfX() {
String decomposedPackageName = "s";
String subPackageName = "s.x";
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(decomposedPackageName), decomposedPackageName));
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(subPackageName), subPackageName));
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(decomposedPackageName), subPackageName));
}
@Test
public void testSDecomposedOfY() {
String decomposedPackageName = "s";
String subPackageName = "s.y";
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(decomposedPackageName), decomposedPackageName));
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(subPackageName), subPackageName));
Assert.assertTrue(isPackageExistsInGivenList(
getSubPackages(decomposedPackageName), subPackageName));
}
…
Figure 11. Example Junit Test Suite (part) for the Decomposition Viewpoint

6. Architectural Models for the E-Government System
To support the communication with the stakeholders, guide the overall development, and analyze the
system, architecture documentation has been provided that includes descriptions of several architecture
views. Figure 12 shows, for example, the deployment view of the system. There are integrated web-services
and client applications that are connected to the server side of the application. Our approach is applied on
the infrastructure of the server-side implementation.
Web Servic es
(Streaming)
SERVER
Oracle DB
1..*
1..*
Stream
Extraction
Batch
Extraction
Stream
Processing
Batch
Processing
Client
1..*
Figure 12. Deployment view of the adopted project
Figure 13 shows the shared data view of the project. The system has two database accessors named PM and
QM whereby each can perform different operations. Moreover, one repository is present which is named
DB. PM can perform both data read and data write operation on DB. However, QM can only perform data
read operations on DB.
Figure 13. Shared data view of adopted project infrastructure
The decomposition view of the system is shown in Figure 14. The decomposition view is similar to that
shown in the figure but, for confidentiality, the names of the modules have been omitted. The adopted
project is composed of one main package A. A is further divided into sub-modules B, C, D, E, F and G.
Furthermore, B is decomposed into B1, B2 and B3. Likewise, C is decomposed into C1 and C2. D is
decomposed into three parts D1, D2 and D3 where D1 is decomposed into one more part called D11.

Figure 14. Decomposition view of adopted project infrastructure
The uses view of the system is shown in Figure 15. Here the arrows represent a uses relation that defines
the dependency of a module to the used module.
B
D3
B1
B2 B3
E
C1
C
C2
F
D
D11
D2
G
Figure 15. Uses view of the adopted project infrastructure
Figure 16 shows the layered view of the system. The adopted project has three main layered views. B1 is
allowed to use B2 and B3, D3 is allowed to use E, F is allowed to use C, C is allowed to use C1 and C2.

Figure 16. Layered view of adopted project infrastructure
In the system we have also defined the generalization view to show the generalization specialization
relations among the modules in the system. Figure 17 shows, for example, the generalization view of
package B. The rounded squares are either interfaces or classes. In the figure, three types of generalizations
are shown including interface extensions, class extension and interface implementation. Class can extend
another class and class can implement an interface. However, interface can only extend another interface.
Similar to module B we have also defined generalization views for other relevant packages which are not
shown due to space constraints. In the figure blue nodes signify interfaces and black nodes signify classes.
Moreover, the dashed arrows are implementation relation and straight lines are extension relations. The
naming convention for the nodes is that nodes with just one number are roots and nodes with other numbers
concatenated onto that number are children.
Figure 17. Generalization view of adopted project infrastructure within package B

The system has been implemented for three years, consisting of tens of thousands of lines of code with
more than one thousand lines of code being added, modified or deleted each month. Many developers and
testers have been assigned who are responsible for continuous maintenance and evolution of the system.
For managing such a large system, it is important that the corresponding code is consistent with the
architecture. Testing is carried out for different quality concerns [26]. One important concern is also the
alignment with the architectural design decisions. For this we have applied the architecture driven model-
based conformance checking approach that we described before.
7. Case Study Design
To validate our ACAMBT approach we have adopted the case study empirical evaluation protocol as
discussed by Runeson and Höst [36]. The protocol consists of the following steps: (1) case study design
(2) preparation for data collection (3) execution with data collection on the studied case (4) analysis of
collected data (5) reporting. Table 2 presents the case study design steps for the selected cases study.
Table 2. Case Study Design
Case Study Design Activity
Case Study
Goal
Assessing the effectiveness and practicality of adapted ACAMBT
approach
Research Questions
RQ1: How effective is the adopted ACAMBT approach?
RQ2: How practical is the adopted ACAMBT approach?
Background and source
Software Engineers (1st degree)
Meetings and interviews (1st degree)
Technical documents and reports (2nd degree)
Source Code (2nd degree)
Official documents (3rd degree)
Data Collection
Direct data collection through semi-structured interviews, meetings
and survey (mix of open and closed questions)
Independent data collection based on document analysis (the papers
and technical reports)
Indirect data collection based on source code analysis.
Data Analysis
Qualitative Data Analysis using Bar Charts
Quantitative Data Analysis using Tables
The case study design approach is in the category of the applied research type and as such the primary
purpose is to understand the impact of the adoption of ACAMBT for conformance analysis within the real
industrial context. Two research questions have been defined. RQ1 relates to how effective the approach is
for conformance analysis, while RQ2 aims to assess the practicality of the approach. The effectiveness of
the proposed approach is calculated by the number of inconsistencies discovered in the implementation of
the system. To this end, we have applied the approach on real code to detect the inconsistencies with the
architectural constraints. Further we have also adopted fault injection techniques to detect potential bugs
that could appear in the code [4][32][43][44]. For assessing the practicality of approach, we have applied
surveys with the team leaders and software engineers and presented the results of the survey. As shown in

Table 2 our 1st degree information sources are software developers, meetings and interviews. We also
analyzed technical reports, technical documents along with the source code as 2nd degree information
source. Finally, we analyzed official documents provided by the company to other stakeholders as 3rd degree
information source. Table 2 also shows the data collection methods from the information sources. We have
conducted semi-structured interviews, meeting and surveys with the software developers. Then, we
independently analyzed technical reports and official documents. Finally, the results of the previous steps
have been reported.
7.1 How effective is the adopted ACAMBT approach?
7.1.1 Existing Architectural Compliance
To assess the effectiveness of the adopted ACAMBT approach we performed data collections for
constructing architecture models for the viewpoints which will be an input to adapted approach. For this,
we analyzed paper and technical reports for the projects architecture models and references which are
maintained throughout the project development. The architecture was documented separately in several
reports. We performed semi-structured interviews with technical leads and server-side software architects.
These analyses led to the architecture presented in previous sections. Subsequently, we analyzed the
project’s source code, and finally we derived the abstract architecture models of the current code that would
be checked against the earlier defined architectural models. In alignment with the steps of the approach we
used these derived architecture models to generate test cases for each viewpoint. The generated test cases
have been executed on the implemented system and the results have been recorded.
Table 3 shows the overall summary of the test execution result for each viewpoint. As it can be observed
from the table, depending on the defined view the number of test cases differs. In total 126 test cases were
defined and only 2 tests failed. The tests for shared data, decomposition and generalization viewpoints have
executed successfully. However, we found inconsistencies between implementation and architecture for
uses and layered viewpoints.
Table 3.Testing results of the real code for each viewpoint
Viewpoint/Test Result
Total possible tests
# of Test Passed
# of Test Failed
Shared Data Viewpoint
3
3
0
Decomposition Viewpoint
15
15
0
Uses Viewpoint
22
21
1
Layered Viewpoint
6
5
1
Generalization Viewpoint
82
82
0
Total
128
126
2
7.1.2 Fault Injection-Based Testing
Besides testing the real existing code as-is, we have also applied fault-based testing techniques in order to
validate our adapted approach. Software testing is labeled as fault-based testing when it aims to demonstrate
of absence of pre-defined faults [27][32]. In order to assess our test cases, we created a mutant copy of our
case study with relevant injected faults. A mutant copy of a program is a copy of the program seeded with
mutations which are structural changes/bugs [33]. We have run the generated test cases on the mutant copy
and aimed to kill all mutants, which implies that at least one test case catches the bug. Killing all the mutants
also implies that the test case is effective at finding real life defects. We introduced the following categories
of faults as our mutation operations:
• Absence relations: removal of expected relation
• Divergence relations: altering a relation
As we have discussed before, each viewpoint includes its own set of criteria, and likewise the absence and
divergence relations will be based on these defined criteria. Different strategies exist in mutation testing

which are based on first order mutants and high order mutants. Jia et al introduced higher order mutation
testing in which mutation operators are applied more than once [15]. In our case, this concept is inapplicable
as our operators are conflicting with each other as one cannot remove and alter the same relation. Therefore,
we applied first order mutation testing.
Since the number of architectural elements is manageable we applied exhaustive testing. Hence, we applied
fault injection for checking the absence and the divergence of each architectural element in each view.
Examples of fault injection for the earlier defined architecture views are shown in Table 4. The pseudo code
for generating faults in the code based on the elements in the architecture views is shown in Table 5.
Table 4. Example Fault-Injections for the adopted viewpoints
Viewpoint
Fault Injections
Shared Data
Viewpoint
• PM_WRITE and PM_READ relations are removed from the implementation of the case
study.
• QM component is removed from the implementation.
Decomposition
Viewpoint
• Package F is removed from the content of package A in the implementation.
• Package E is removed from the implementation.
Uses
Viewpoint
• Package C is removed from the implementation.
• Uses relation of B3 to F is removed from the implementation.
• Uses relation of E to D2 is removed from the implementation.
Layered
Viewpoint
• Uses relation B1 to B2 is removed from the implementation (hence a violation of the
expected allowed-to-use-relation)
• Uses relation from package C to package F is introduced to the implementation of the case
study (and hence a violation of layered property that does not allow call backs).
Generalization
Viewpoint
• B1_6 component is removed from the implementation
• Inheritance by extension relation between B_1 and B3_1 is removed from the
implementation
• Inheritance by implementation relation between B2_7 and B2_8 is removed from the
implementation.

Table 5. Pseudo codes for the fault injections in the architecture views
for each attachment in sharedData.attachments
remove attachment.da from code
endfor
for each attachment in sharedData.attachments
if attachment.type == DataRead
remove attachment.dataRead from code
endif
if attachmet.type == DataWrite
remove attachment.dataWrite from code
endif
endfor
for each element in decomposition.elements
for each subelement in element.subelement
change subelement hierarchy in code
endfor
endfor
for each element in decomposition.elements
for each subelement in element.subelement
remove subelement from the code
endfor
endfor
for each relation in uses.relations
remove.relation from code
endfor
for each relation in uses.relations
alter relation in code
endfor
for each relation in uses.relations
remove relation.source from code
endfor
for each relation in uses.relations
remove relation.target from code
endfor
for each relation in layered.relations
remove relation.source from code
endfor
for each relation in layered.relations
remove relation.target from code
endfor
for each relation in layered.relations
remove relation from code
endfor
for each relation in layered.relations
alter relation in code so that
relation.target uses relation.source
endfor
for each declaration in generalization.declarations
remove declaration.parent from code
endfor
for each declaration in generalization.declarations
remove declaration.child from code
endfor
for each declaration in generalization.declarations
remove declaration.inheritance from code
endfor
for each declaration in generalization.declarations
if declaration.type == Extension
alter declaration.child to implement declaration.parent in code
endif
if declaration.type == Implementation
alter declaration.child to extend declaration.parent in code
endif
endfor
After injecting the faults, we applied the conformance analysis approach and checked the results. We have
injected every possible fault with respect to the architecture view criteria defined, to ensure that our
approach will detect every possible bug that we expect to detect (ones that are presented in architecture
model but not presented in code level implementation). Table 6 shows the fault-based testing results for
each viewpoint. As it can be seen from the table all the injected faults are detected successfully by our
ACAMBT approach. After testing the real code and this fault-based testing approach we can state that the
approach is thus effective.

Table 6. Fault-based testing results for each viewpoint
Viewpoint/Fault Detection
# of Faults Detected
# of Faults Not Detected
Shared Data Viewpoint
5
0
Decomposition Viewpoint
32
0
Uses Viewpoint
78
0
Layered Viewpoint
16
0
Generalization Viewpoint
77
0
7.2 How practical is the adopted ACAMBT approach?
Within an industrial context it is important that the adopted approach and the corresponding tool is also
practical in use. To validate this we have applied designed and conducted a survey study based on the
protocol and the guidelines of Kitchenham’s et al [18][19][20][21][22]. We discuss this in the following
sub-sections.
7.2.1 Survey Study Design
Table 7 shows the description of the survey design. For the survey form we decided to select an
experimental design as we conducted experiments using the adapted approach with survey takers.
Table 7. Survey Study Design
Survey Study Activities
Survey Study
Survey Form
Experimental – Self-control Studies
Data Requirements
Practicality of adapted ACAMBT approach
Population
Managers, Assistants, HR Staff, Other Non-Technical Staff, Software
Developers and Software Development Team Leads
Population Selection Criteria
Exclusion Criteria:
EC1: Employees who do not work in the adopted project
EC2: Employees who are not involved in software development
EC3: Employees who do not have computer science background
Sampling Technique
Non-Probabilistic Sampling – Convenience Sampling
Sample Size
7
Data Collection Technique
Interview
In experimental studies five different study types are defined including concurrent control studies in which
participants can be randomly assigned to groups, concurrent control studies in which participants are not
randomly assigned to groups, self-control studies, historical studies, and studies using a combination of
techniques. Self-control studies are a type of experimental survey form where both pre- and post- treatment
measures are used for obtaining information. For our case, self-control studies were feasible since we
wanted to assess the practicality of the adapted approach with comparison to past practices in the company
for architecture-based testing, if any. The data requirement for this survey study is the research question for
our case study which aims to assess the practicality of adapted ACAMBT approach. The survey study was
conducted in the company and our population size was small. There are 59 employees in the company, and
the employee background varies from technical to non-technical. Different projects are being implemented
within the company. In each project, employees from different backgrounds work on different aspects of
the project. Therefore, we defined our population exclusion criteria according to those stated above. The
population of the company consists of managers, assistants, non-technical staff, software developers and
software development team leaders. The defined exclusion criteria include (a) employees who do not work
in adopted project (b) employees who do not have computer science background and (c) employees are not
involved in software development.

Table 8 shows the overall population and population selection after each exclusion criteria application. At
the remaining sample, we applied convenience sampling, which is one of the non-probabilistic sampling
methods, if any of the resulting population unavailable for survey taking. The sampling technique
established sample size is identified as 7 in which two software development team leads and five software
developers reside. As convenience sampling is applied, we had high response rates and meaningful answers
from the volunteer survey takers. Team leads have a good insight of modelling and analyzing software
architecture and had worked on different projects. Four out of the five software developers were senior
level developers and also had a good insight in designing software architectures. The remaining developer
was working on the current project for two years and had an average knowledge about software architecture.
Table 8. Overall Population Size and Selection
Population
Total
Population
Size
Population Size
After Applying
EC1
Population Size
After Applying
EC2
Population Size
After Applying
EC3
Population
After
Sampling
Company
59
18
13
9
7
7.2.2 Data Collection
Self-control studies depend on pre- and post- studies and are intervened by exposure to the approach
between studies.
For the data collection the following protocol was followed:
▪ First a meeting was scheduled with the survey takers for the initial interview. The goal of this
interview was to capture the initial thoughts and experience on ACAMBT adoption.
▪ In the second step, we gave a short presentation about the goal of the adapted approach.
▪ In the third step engineers applied the ACAMBT approach with the researcher on the government
system, with the models created by the researchers.
▪ In the fourth step, engineers took a survey to assess the practicality of the approach. To collect
data according to pre- and post- questionnaires defined below, we used online free survey tools.
For the pre-survey, we have defined two questions regarding the participant’s initial knowledge and
willingness to adopt the provided approach. The questions were the following:
• What do you know about architecture-based testing?
• Do you apply architecture-based testing techniques?
The survey was conducted on seven subjects according to the exclusion criterion applied on working staff
on adopted project’s company. With the help of pre-survey, the subjects were classified according to their
knowledge level on architecture-based testing. According to the answers of the subjects, it was found that
five of the subjects were not familiar with the architecture-base testing concept, while two of them have a
general idea about the concept. However, none of the subjects had detailed knowledge about the concept.
Moreover, the subjects have never experienced the application of architecture-based testing techniques. The
survey took much more time than it was estimated, since none of the subjects were familiar with the concept.
On the other hand, all of them were very eager to contribute to the study and positively welcomed the
suggested approach.
Table 9 shows the design of the post-survey questionnaire. The questions were grouped as input, execution,
output practicality and general. For each question group, we have defined three purposeful and concrete
questions. As suggested by Kitchenham’s the questions are not contradicting and positively constructed as
it is harder to understand negative questions. Further, the survey has both open and closed questions. Input,
execution and output practicality groups answers are hard coded as strongly agree, agree, disagree and
strongly disagree as suggested in [20]. The last group includes open questions for which answers could be

given. These answers to these questions were recorded and later on processed. The interpretation of the
answers was carried out together by both the researchers. The results of the questions Q1 to Q9 of the survey
are given in Table 10 and visualized as bar charts in Table 11. The answers to questions Q10 to Q12 are
shown in Table 12.
From Table 11 we can see that the answers for the questions for practicality are in general positive. For the
questions 1 to 3 relating to input practicality the majority of the participants agreed that the architecture
models were understandable, and the validation of the architecture models was indeed practical. The
creation of architecture models was also perceived somewhat practical, while three participants found it
less practical, four participants did find it practical. It should be noted that those who found it practical were
also the more experienced employees.
Questions 4 to 6 relate to execution practicality, and from Table 11 we can derive that the majority of the
participants indicated that the approach is easy to use, understandable and useful.
Questions 7 to 9 relate to the output practicality, and from Table 11 we can derive that all participants
indicated that the approach can effectively detects defects, can generate meaningful tests, and generate these
tests quickly.
Table 9. Post-Survey Questionnaire
Strongly
Agree
Agree
Disagree
Strongly
Disagree
INPUT
PRACTICALITY
Q1. Input architecture models are easy to understand
Q2. Input architecture models are easy to create
Q3. Input architecture models are easy to validate
EXECUTION
PRACTICALITY
Q4. Approach is easy to use
Q5. Approach is understandable
Q6. Approach is useful and can be applied in practice
OUTPUT
PRACTICALITY
Q7. Approach can detect defects
Q8. Approach generates meaningful test cases
Q9. Approach generates test cases quickly
GENERAL
Q10. What are the advantages of adapted ACAMBT
approach?
Q11. What are the disadvantages of adapted
ACAMBT approach?
Q12. What are the interesting points in adapted
ACAMBT approach?

Table 10. Results of the Post-Survey Questionnaire
Subject
No
Q1
Answers
Q2
Answers
Q3
Answers
Q4
Answers
Q5
Answers
Q6
Answers
Q7
Answers
Q8
Answers
Q9
Answers
1
Agree
Agree
Disagree
Strongly
Agree
Agree
Strongly
Agree
Strongly
Agree
Agree
Agree
2
Disagree
Disagree
Strongly
Disagree
Agree
Agree
Agree
Agree
Strongly
Agree
Agree
3
Agree
Agree
Agree
Disagree
Agree
Strongly
Agree
Agree
Agree
Agree
4
Agree
Disagree
Agree
Strongly
Agree
Disagree
Agree
Strongly
Agree
Strongly
Agree
Agree
5
Agree
Agree
Agree
Strongly
Agree
Agree
Strongly
Agree
Agree
Strongly
Agree
Agree
6
Disagree
Disagree
Disagree
Disagree
Disagree
Disagree
Agree
Agree
Agree
7
Agree
Agree
Agree
Agree
Strongly
Agree
Agree
Agree
Agree
Agree

Table 11. Results of the Post-Survey Questionnaire represented as bar charts
Q1 – Understandability Architecture Models
Q2 – Creation Architecture Models
Q3 – Validation Architecture Models
Q4. Approach is easy to use
Q5. Approach is understandable
Q6. Approach is Useful
Q7. Approach can detect defects
Q8. Approach generates meaningful tests
Q9. Approach generates tests quickly
0
1
2
3
4
5
6
7
Strongly Agree Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree
0
1
2
3
4
5
6
7
Strongly
Agree
Agree Disagree Stongly
Disagree

Table 12. Results of the Open Questions in the Post-Survey Questionnaire
Subject
No
Q10 – Advantages:
Answers
Q11 – Disadvantages:
Answers
Q12 – Interesting:
Answers
1
It is fast and accurate. It can
actually detect defects
Need to convert already existing
UML models to another model
to generate test cases
Defining models for different
architecture viewpoints so that
you can test them individually
2
It is useful. It detects bugs and
shows the difference between
architecture and implementation
UML models are not accepted. It
cannot be accessed by its own
executable program.
Viewpoints are modelled so that
each can be separately tested.
We can conform the architecture
on the documents to
implemented code.
3
It is easy to apply. It can be
integrated into any test
environment as it simply uses
JUnit library.
Needs programming background
to apply. Need to update
software architecture model in
case of design change.
Uses standard java libraries and
reflection libraries to conform
the architecture and the
implemented code itself.
4
It generates meaningful test
cases in software design
perspective. We did not have
any tool or methodology to
cover this concern as the code
itself changes constantly.
Cannot use existing architecture
models, needs overwork to
translate UML models to
approach's viewpoint based
models.
It is a new methodology that I'm
facing and it has its cost and
benefits as UML models are not
accepted but before the delivery
of product the design document
can be verified
5
Generated test cases cover for
architecture conformance issues
in detail as it uses viewpoints in
isolation.
It uses Eclipse environment but
it can have its own executable.
You have to open Eclipse each
time you want to generate.
I have never used a tool in
Eclipse Epsilon environment.
6
Conforming architecture to code
in every phase of development is
crucial, if a defect in early stages
of development is not detected it
will cause a lot of damage.
Does not accept predefined and
well-known models for software
design.
Input architecture models are
semi-structured format such as
JSON or xml and can be edited
by simple notepad program.
7
Tests the architecture against
code is crucial. The generated
files are actually executing on
code as simple test cases. Fast
and understandable approach.
Extra work need to be done to
maintain approach based
architecture models. Changing
formal model and approach
model can be costly.
Can be improved by adapting
new viewpoints to the approach.
Adapting new viewpoints are
very easy as to create model and
write translation templates for
generation. Approach takes the
rest of the work.
What are the advantages of adapted ACAMBT approach?
The advantages of the approach can be categorized according to the results of the survey. As the first and
more common advantage according to the results, subjects highlight the accuracy and functionality of the
system. They are satisfied with defect detection functionality of the system. They agreed on that the system
detects bugs and shows the difference between architecture and implementation. They found the generated
test cases meaningful from a software-design perspective. Additionally, it is considered as an advantage for
the system that generated test cases cover the architecture conformance issues in detail as it uses viewpoints
in isolation. Moreover, the system tests the architecture against code, and conforms the code in every phase
of development which is interpreted by the subjects as an asset of the system. The subjects are emphasizing
the opportunity cost of an early-detected defect.

In terms of usability, the subjects find the system easy to understand and easy to use. In terms of
accessibility, the subjects highlight that the system can be integrated into any test environment as it simply
uses JUnit library. In terms of availability, subjects are satisfied with the speed of the approach when it is
compared to traditional methods. Additionally, they found the system original while emphasizing the lack
of such a system in the company. They accent the importance of such a system especially in an environment
when changes to the system occur frequently.
What are the disadvantages of adapted ACAMBT approach?
In general, the subjects are concentrated on two main disadvantages of the system. The most mentioned
disadvantage of the system is that the system is not compatible with UML models. The subjects use UML
models in the adopted project they work on. Therefore, the lack of integration with UML models forces
the subjects to rework their models to convert them into the intended viewpoint-based models by the
approach, which means additional time and cost to the subject. Additionally, the subjects underlined the
configuration management challenges of the tested program. Since the approach cannot work with UML
models, it brings the necessity of using two-or-more models in parallel, which makes configuration
management of the code open to individual errors and inadvertencies. As the second disadvantage, the
subjects pointed out that the approach would not work without an executable program. The program uses
the Eclipse environment, which makes the execution process of the program complex and forces the
subjects to have a programming background.
What are the interesting points in adapted ACAMBT approach?
The subjects mentioned as a positive aspect that the program makes it possible that different viewpoints of
the architecture can be separately tested and can be concentrated on. With the help of this functionality, the
architecture can be conformed on the design documents without waiting for the entire implementation phase
of the code. Thus, defects of the program can be detected at earlier phases of the adopted project, and design
can be updated accordingly without extra effort. The program is open to development because it is easy to
integrate new viewpoints to the program. As the suggested approach uses open-source standard java
libraries for testing, it is easy to configure and run on the system. Finally, the ease of editing and simplicity
of viewpoint models were remarked upon by subjects.
Summary
When three aspects of the practicality are analyzed, output practicality is the strongest feature of the
approach according to the survey results. The approach has some features which are open to development
in terms of input practicality, while the approach is good enough in terms of execution practicality.
8. Discussion
Our ACAMBT approach has been evaluated both with respect to effectiveness and practicality. To assess
the effectiveness of our approach, the architecture and the code of a large industrial case has been used for
testing architecture-code conformance. We can state that the approach was effective in checking code
conformance to the specified architecture. It executes successfully on a large system and presents plausible
results.
To assess the practicality of our approach we have conducted a survey study according to protocols we
defined with respect to Kitchenham’s et al [18]. In our survey study, we have conducted both a pre-
operation survey and post operation survey where operation denotes the execution of the approach with
participants and detailing the mentality and logic behind the suggested approach. In our pre-survey, we
have seen that participants never used any ACAMBT tools, on the other hand two of the participants
appeared to be familiar with the concept. Then in our post-survey, we have assessed the practicality of our
approach for three categories including input, execution and output practicalities.

Regarding input practicality we have assessed whether the architecture is easy to understand, to create and
to validate. Model formation, that is, proper architecture modeling is essential for the success of the
approach. Although some effort is needed for architecture modeling, the creation and understanding of
architecture models were largely found easy, the validation of the architecture appeared to be more difficult.
In general, based on our survey study with the practitioners in the project it appeared that the approach was
found practical and that the tool would be used as part of the overall testing effort.
Similar to any case study research our study also has some validity threats. Construct validity refers to
demonstration of measuring what the research has in mind and what the study suggests. Our approach
focused on testing the drifting of the code from the architecture. That is, architecture view models are
validated and verified against the implementation level, but we do not check whether the code is completely
drifting away from the architecture. For example, if a developer adds a relation in the code base and that
relation is not in the architectural model, the relation will not be identified as an inconsistency. In essence,
we consider that our architecture model which is our reference point for testing is both complete and correct.
We do aim to focus on architecture drift analysis from both directions in our future research.
Although each of the participant had a mature experience in design, development and testing, the focus on
testing architectural constraints in the code was not something they had done before explicitly. For coping
with this issue, we organized a meeting where we have first presented the notion of model-based testing,
conformance analysis and deriving test cases from architectural descriptions. Since the focus was still on
testing, the concepts could be easily grasped. This might as such not have a serious impact on the overall
outcome of the study. Still, the potential bias of the participants could be taken into account to soften this
claim.
For the practicality criteria, we have applied survey study before and after applying the approach. In the
survey study, designed to assess the practicality of the suggested approach, we singled out construct
validities which are presented in Table 13.
Table 13. Construct Validity
Threats
Counter Measures
Wrong selection of interviewed persons.
To eliminate this threat, we have defined exclusion
criteria for sample population and applied Non-
Probabilistic Sampling and Convenience Sampling
with respect to Kitchenham’s et al [11].
Wrong interpretation of questions by interviewed
personnel.
Survey questions are prepared using the guidelines
defined in Kitchenham’s et al [11]. We both
provided open and categorized questions.
Number of participants
The total population was analyzed according to the
guidelines defined in Kitchenham’s et al [11].
We searched within the whole company to identify
the participants related to the topics of this study.
For this we discussed with higher level managers
and analyzed the existing projects. In the end, only
participants who could provide relevant insight
were selected.
Difference between measured and what was
intended to be measured
We have applied fault-based testing techniques to
prove that our approach can find the newly
presented defects on defect free environment and
then we applied our approach.

Internal validity refers to casual relation between treatment and outcome. While assessing the effectiveness
of the suggested approach, we have applied formal fault-based testing and reported the results in an isolated
environment that no external variable can affect. Additionally, the practicality of the approach is shown by
a survey study in which we have provided both open and closed questions to discover the participants
explicitly-held and implicitly-held perceptions. Another internal validity threat is the lack of experience in
architecture conformance approaches of the participants. Finally, another issue that should be mentioned is
the possibility of experimenter effects/the Hawthorne effect. This implies that the participants could have
behaved differently due to the researcher’s presence. This Hawtorne effect could have had an impact on
different phases in the case study including the modelling of the architecture, the testing process, and even
the answers to the questions in the survey study. To reduce the effect of this Hawtorne effect, additional
studies could be carried out.
External validity refers to concern of generalizing the results of a scientific study. In our case study, we
have applied our suggested approach on a smart e-government project with a large number of transactions.
The approach has been applied to one case study with 7 participants and this implies a threat to external
validity of the approach. However, we consider the case study as representative for the challenges that many
different systems have to deal with when coping with architecture consistency. What really matters in our
context is not the number of case studies but the size and complexity of the systems. In our approach we
have adopted five different viewpoints, defined the metamodels and domain specific languages for these,
implemented the constraints and generated the test cases. The approach itself is generic and could be also
used for different viewpoints. As stated before the system that is being developed can be a large scale
complex system and hence we could state that we have adopted a sufficiently representative system to
support the external validity. Obviously, in practice even larger systems could be identified, but the
properties of architecture code consistency will be similar. In addition, to enhance the external validity,
although we have one case study we actually carried out two different experiments including the one with
testing the real code, and the other using an exhaustive fault-based testing approach. In our future research
we will focus on further applications of the approach.
9. Related Work
Neto et al. [31] provides the results of a systematic review on model-based testing approaches. In the
literature review 78 primary studies have been selected. The review shows where MBT approaches have
been applied, presents the characteristics and limitations of MBT. The selected studies are analyzed in terms
of application scope of MBT approaches, level of automation, tool support for MBT, models used for test
case generation, test coverage criteria, behavior model limitations, cost and complexity of application of
MBT approach. In addition to these, they also discuss the issues regarding MBT approaches and the
limitation of MBT approaches. Based on this characterization, our approach is general purpose and can be
applied to different application domains. Automation includes the modeling of architecture, test case
generation, test case execution and analysis. We have provided tool support that supports these levels of
automation. The models used for test case generation include executable models of architecture views and
the code. The test coverage criteria relate to the defined architecture constraints. We primarily check static
architectural constraints and do not consider behavioral constraints. The cost and complexity of the MBT
approach was considered manageable based on the output of the survey study.
Besides testing, software architecture has also been used for model-checking. Model checking is a formal
verification technique that aims for the automated analysis of a systems’ behavior with respect to selected
properties. It takes as input a system model and a property specification [7]. The output is ‘‘true”, or false
with a counter-example when an error is detected. Zhang et al. [45] provide a classification and comparison
of model checking software architecture techniques. The authors describe the main activities in a model

checking software architecture process. Then, a classification and comparison framework are provided
which is subsequently used to compare model checking software architecture techniques.
Different approaches are presented for ACAMBT in the current literature. In one of the early studies [6],
the authors present a ACAMBT approach using chemical abstract machine (CHAM) for software
architecture representation and labelled transition systems (LTS) for test model representation. Jin et al.
[16] define software architecture using Wright architectural design language (ADL) and presents testing
criteria based on data flow reachability, control flow reachability, connectivity and concurrency. Behavior
graphs are generated from the architecture descriptions and tests are executed on this graph with respect to
defined criteria. Muccini et al. [29] presents a continuation study for [6] in which CHAM model is replaced
by finite state process (FSP) model and same process of test model of type LTS is generated. Johnsen et al.
[17] presents a ACAMBT approach based on architecture analysis and design language (AADL), and
Uppaal Model (timed automata). AADL is used for representing architecture and timed automata is used
as a test model for generating test cases. Reza et al. [34] presents use of petri nets in ACAMBT in which
software architecture is described in Acme ADL, and test model is generated as Hierarchical Predicate
Transitions Nets (HPrTNs). Javed et al. [14] authors use UML sequence diagrams to model the dynamic
behaviors of software architecture and transform it into xUnit test model from which different types of test
cases can be generated such as jUnit, cUnit, cppUnit and etc. All of these studies use different kinds of
software architecture description and test models, but none of the studies have taken software architecture
viewpoints into consideration. Another way of detecting nonconformance issues is stated in study [3], in
which is traceability between the implementation and code is maintained as the software is being produced.
Tera et al. [41], authors propose a recommendation system for repairing the inconsistencies found between
architecture and code.
10. Conclusion
Model-based testing is a well-known approach for automating the generation and execution of test cases
using models based on system requirements and behavior. Our approach is one of the few approaches that
adopts architecture models to derive test cases. A unique part of our approach is further the generation and
usage of the test cases for checking conformance of the architecture with the code for architectural
constraints. For preparing the model-based testing for architecture models we first had to define the
metamodels and the corresponding domain specific languages. We could do this without serious problems
for all the viewpoints that we considered. The approach for mapping the architecture viewpoints to the
domain specific languages is general and can be also applied to other viewpoints. Our approach builds on
and refines the generic MBT process. We have implemented the corresponding toolset with which
architecture views can be specified, the generators can be developed, and the test cases generated and tested.
As a result, both the approach and the corresponding toolset provides an additional practical toolset for the
testing and the architecture code conformance.
We have adopted a real industrial case study and evaluated our approach based on the guidelines of a
systematic case study design protocol that we planned and executed carefully. We evaluated the
effectiveness of our approach both on the real architecture and the code. Further we also evaluated the
practicality of the approach using a survey study with the real practitioners. From the overall evaluation we
can conclude that the approach and the corresponding toolset is both effective and practical. In our future
work we aim to further enhance the toolset, apply it for other case studies, develop test cases for other
viewpoints, and focus on the integration of the tool with the conventional development tools.

Acknowledgments
The authors would like to thank the anonymous reviewers for their valuable and constructive comments
that greatly helped to improve the quality of the paper.
References
[1]. Ali, N., Baker, S., O’Crowley, R., Herold, S., & Buckley, J. (2018). Architecture consistency: State
of the practice, challenges and requirements. Empirical Software Engineering, 23(1), 224-258.
[2]. [ANSI/IEEE 1059:1993] Software Verification and Validation Plan (ANSI/IEEE 1059), May 2014.
[3]. Antoniol, G., Canfora, G., Casazza, G., & De Lucia, A. (2001). Maintaining traceability links during
object‐oriented software evolution. Software: Practice and Experience, 31(4), 331-355.
[4]. Baier, C., & Katoen, J. P. (2008). Principles of model checking. MIT press.
[5]. Barbosa, R., Silva, N., Duraes, J., & Madeira, H. (2007, February). Verification and validation of
(real time) COTS products using fault injection techniques. In Commercial-off-the-Shelf (COTS)-
Based Software Systems, 2007. ICCBSS'07. Sixth International IEEE Conference on (pp. 233-242).
IEEE.
[6]. Bertolino, A., Corradini, F., Inverardi, P., & Muccini, H. (2000, June). Deriving test plans from
architectural descriptions. In Proceedings of the 22nd international conference on Software
engineering (pp. 220-229). ACM.
[7]. Clarke, E. M., Grumberg, O., & Peled, D. (1999). Model checking. MIT press.
[8]. Clements, P., Bachmann, F., Bass, L., Garlan, D., Ivers, J., Little, R., Merson & Stafford, J. (2011).
Documenting Software Architectures: Views and Beyond. Addison-Wesley Professional.
[9]. Demirli, E., & Tekinerdogan, B. (2011, September). Software language engineering of architectural
viewpoints. In European Conference on Software Architecture (pp. 336-343). Springer, Berlin,
Heidelberg.
[10]. Díaz-Pace, J. A., Soria, Á., Rodríguez, G., & Campo, M. R. (2012). Assisting conformance checks
between architectural scenarios and implementation. Information and Software Technology, 54(5),
448-466.
[11]. Epsilon – model driven development IDE, http://eclipse.org/epsilon, last accessed on April, 2015.
[12]. Gurbuz, H. G., & Tekinerdogan, B. (2017). Model-based testing for software safety: a systematic
mapping study. Software Quality Journal, 1-46.
[13]. ISO/IEC 42010:2007] Recommended practice for architectural description of software-intensive
systems (ISO/IEC 42010), 2011.
[14]. Javed, A. Z., Strooper, P. A., & Watson, G. N. (2007, May). Automated generation of test cases using
model-driven architecture. In Automation of Software Test, 2007. AST'07. Second International
Workshop on (pp. 3-3). IEEE.
[15]. Jia, Y., & Harman, M. (2009). Higher order mutation testing. Information and Software Technology,
51(10), 1379-1393.
[16]. Jin, Z., & Offutt, J. (2001, November). Deriving tests from software architectures. In Software
Reliability Engineering, 2001. ISSRE 2001. Proceedings. 12th International Symposium on (pp. 308-
313). IEEE.

[17]. Johnsen, A., Pettersson, P., & Lundqvist, K. (2011). An architecture-based verification technique for
AADL specifications. Software Architecture, 105-113.
[18]. Kitchenham, B. A. (2002). Principles of Survey Research Part 2: Designing a Survey Barbara A.
Kitchenham Department of Computer Science Keele University, Staffordshire, UK. ACM SIGSOFT
Software Engineering Notes, 27(1), 18..
[19]. Kitchenham, B. A., Pfleeger, S. L.(2002). Principles of survey research part 3: constructing a survey
instrument. Software Engineering Notes, 27(2), 20-24.
[20]. Kitchenham, B., & Pfleeger, S. L. (2002). Principles of survey research part 4: questionnaire
evaluation. SIGSOFT Softw. Eng. Notes, 27(3), 20-23.
[21]. Kitchenham, B., & Pfleeger, S. L. (2002). Principles of survey research: part 5: populations and
samples. ACM SIGSOFT Software Engineering Notes, 27(5), 17-20.
[22]. Kitchenham, B., & Pfleeger, S. L. (2003). Principles of survey research part 6: data analysis. ACM
SIGSOFT Software Engineering Notes, 28(2), 24-27.
[23]. Knodel, J., & Popescu, D. (2007, January). A comparison of static architecture compliance checking
approaches. In Software Architecture, 2007. WICSA'07. The Working IEEE/IFIP Conference on (pp.
12-12). IEEE.
[24]. Koschke, R., & Simon, D. (2003, November). Hierarchical Reflexion Models. In WCRE (Vol. 3, pp.
186-208).
[25]. Mellor, S. J., Scott, K., Uhl, A., & Weise, D. (2004). MDA distilled: principles of model-driven
architecture. Addison-Wesley Professional.
[26]. Mistrik, I., Soley, R., Ali, N., Grundy, J., Tekinerdogan, B. Software quality assurance: in large scale
and complex software-intensive systems. Burlington, Massachusetts, USA : Morgan Kaufmann
Publishers, - p. 420, 2015.
[27]. Morell, L. J. (1990). A theory of fault-based testing. IEEE Transactions on Software Engineering,
16(8), 844-857.
[28]. Tekinerdogan B., Bilir S., Abatlevi C. (2005) Integrating Platform Selection Rules in the Model
Driven Architecture Approach. In: Aßmann U., Aksit M., Rensink A. (eds) Model Driven
Architecture. MDAFA 2004, MDAFA 2003. Lecture Notes in Computer Science, vol 3599. Springer,
Berlin, Heidelberg, 2005.
[29]. Muccini, H., Inverardi, P., & Bertolino, A. (2004). Using software architecture for code testing. IEEE
Transactions on Software Engineering, 30(3), 160-171.
[30]. Murphy, G. C., Notkin, D., & Sullivan, K. J. (2001). Software reflexion models: Bridging the gap
between design and implementation. IEEE Transactions on Software Engineering, 27(4), 364-380.
[31]. Neto, D. A. C., Subramanyan, R., Vieira, M., & Travassos, G. H. (2007, November). A survey on
model-based testing approaches: a systematic review. In Proceedings of the 1st ACM international
workshop on Empirical assessment of software engineering languages and technologies: held in
conjunction with the 22nd IEEE/ACM International Conference on Automated Software
Engineering (ASE) 2007 (pp. 31-36). ACM.
[32]. Michal, Y. (2008). Software testing and analysis: process, principles, and techniques. John Wiley &
Sons.
[33]. Rafi, D.M.; Moses, K.R.K.; Petersen, K.; Mantyla, M.V., Benefits and limitations of automated
software testing: Systematic literature review and practitioner survey, Automation of Software Test
(AST), 2012 7th International Workshop on, vol., no., pp.36, 42, 2 -3, June 2012.

[34]. Reza, H., & Lande, S. (2010, April). Model based testing using software architecture. In Information
Technology: New Generations (ITNG), 2010 Seventh International Conference on (pp. 188-193).
IEEE.
[35]. Rosik, J., Le Gear, A., Buckley, J., Babar, M. A., & Connolly, D. (2011). Assessing architectural
drift in commercial software development: a case study. Software: Practice and Experience, 41(1),
63-86.
[36]. Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in
software engineering. Empirical software engineering, 14(2), 131.
[37]. Tekinerdogan, B. Software Architecture, in: T. Gonzalez and J.L. Díaz-Herrera, Computer Science
Handbook, Second Edition, Volume I: Computer Science and Software Engineering, Taylor and
Francis, 2014.
[38]. Tekinerdogan, B., Sozer, H., & Aksit, M. (2012). Feature-based rationale management system for
supporting software architecture adaptation. International Journal of Software Engineering and
Knowledge Engineering, 22(07), 945-964.
[39]. Tekinerdogan, B. (2015). Architectural drift analysis using design structure reflexion matrices. In
Software Quality Assurance in Large Scale and Complex Software-Intensive Systems (pp. 221-236).
Elsevier.
[40]. Tekinerdogan, B., & Demirli, E. (2013, June). Evaluation framework for software architecture
viewpoint languages. In Proceedings of the 9th international ACM Sigsoft conference on Quality of
software architectures (pp. 89-98). ACM.
[41]. Terra, R., Valente, M. T., Czarnecki, K., & Bigonha, R. S. (2015). A recommendation system for
repairing violations detected by static architecture conformance checking. Software: Practice and
Experience, 45(3), 315-342.
[42]. Utting, M., Pretschner, A., Legeard, B. A taxonomy of model-based testing approaches. Software
Testing, Verification and Reliability, 2011.
[43]. Uzun, B., & Tekinerdogan, B. (2018). Model-Driven Architecture Based Testing: A Systematic
Literature Review. Information and Software Technology.
[44]. Voas, J. M., & McGraw, G. (1997). Software fault injection: inoculating programs against errors.
John Wiley & Sons, Inc.
[45]. Zhang, P., Muccini, H., & Li, B. (2010). A classification and comparison of model checking software
architecture techniques. Journal of Systems and Software, 83(5), 723-744.