Defining and Validating Metrics for Assessing the Maintainability of Entity-Relationship Diagrams

Abstract

Database and data model evolution is a significant problem in the highly dynamic business environment that we experience these days. To support the rapidly changing data requirements of agile companies, conceptual data models, which constitute the foundation of database design, should be sufficiently flexible to be able to incorporate changes easily and smoothly. In order to understand what factors drive the maintainability of conceptual data models and to improve conceptual modelling processes, we need to be able to assess conceptual data model properties and qualities in an objective and cost-efficient manner. The scarcity of early available and thoroughly validated maintainability measurement instruments motivated us to define a set of metrics for Entity-Relationship (ER) diagrams, which are a relevant graphical formalism of the conceptual data modelling method. In this paper we show that these objectives and easily calculated metrics, measuring internal properties of ER diagrams related to their structural complexity, can be used as indirect measures (hereafter called indicators) of the maintainability of the diagrams. These metrics may replace more expensive, subjective, and hence potentially unreliable maintainability measurement instruments that are based on expert judgement. Moreover, these metrics are an alternative to direct measurements that can only be obtained during the actual process of data model maintenance. Another result is that the validation of the metrics as early maintainability indicators opens up the way for an in-depth study of structural complexity as a major determinant of conceptual data model maintainability. Apart from the definition of a metrics suite, a contribution of this study is the methodological approach that was followed to theoretically validate the proposed metrics as structural complexity measures and to empirically validate them as maintainability indicators. This approach is based both on Measurement Theory and on an expe

Full text

Page 1

FACULTEIT ECONOMIE
EN BEDRIJFSKUNDE


HOVENIERSBERG 24
B-9000 GENT
Tel. : 32 - (0)9 – 264.34.61
Fax. : 32 - (0)9 – 264.35.92



WORKING PAPER



Defining and Validating Metrics for Assessing the Maintainability of
Entity-Relationship Diagrams

Marcela Genero 1
Geert Poels 2
Mario Piattini 3



October 2003

2003/199


1 Department of Computer Science, University of Castilla-La Mancha, Ciudad Real, Spain
2 Department of Management Information, Operations Management and Technology Policy, Ghent
University, Ghent, Belgium
3 Department of Computer Science, University of Castilla-La Mancha, Ciudad Real, Spain
Corresponding author is G. Poels, Faculty of Economics and Business Administration, Ghent
University, geert.poels@UGent.be, tel: +32 9 264 34 97.
D/2003/7012/37

Page 2

1
Defining and Validating Metrics for Assessing the Maintainability of
Entity-Relationship Diagrams

Marcela Genero1, Geert Poels2 and Mario Piattini1

1ALARCOS Research Group, Department of Computer Science
University of Castilla-La Mancha
Paseo de la Universidad, 4 - 13071 - Ciudad Real (Spain)
{Marcela.Genero, Mario.Piattini}@uclm.es

2Department of Management Information, Operations Management and Technology Policy
Faculty of Economics and Business Administration
Ghent University
Hoveniersberg 24, 9000 Ghent (Belgium)
geert.poels@ugent.be
Abstract
Database and data model evolution is a significant problem in the highly dynamic business
environment that we experience these days. To support the rapidly changing data requirements
of agile companies, conceptual data models, which constitute the foundation of database design,
should be sufficiently flexible to be able to incorporate changes easily and smoothly. In order to
understand what factors drive the maintainability of conceptual data models and to improve
conceptual modelling processes, we need to be able to assess conceptual data model properties
and qualities in an objective and cost-efficient manner. The scarcity of early available and
thoroughly validated maintainability measurement instruments motivated us to define a set of
metrics for Entity-Relationship (ER) diagrams, which are a relevant graphical formalism of the
conceptual data modelling method. In this paper we show that these objectives and easily
calculated metrics, measuring internal properties of ER diagrams related to their structural
complexity, can be used as indirect measures (hereafter called indicators) of the maintainability
of the diagrams. These metrics may replace more expensive, subjective, and hence potentially
unreliable maintainability measurement instruments that are based on expert judgement.
Moreover, these metrics are an alternative to direct measurements that can only be obtained
during the actual process of data model maintenance. Another result is that the validation of the
metrics as early maintainability indicators opens up the way for an in-depth study of structural
complexity as a major determinant of conceptual data model maintainability.
Apart from the definition of a metrics suite, a contribution of this study is the methodological
approach that was followed to theoretically validate the proposed metrics as structural
complexity measures and to empirically validate them as maintainability indicators. This
approach is based both on Measurement Theory and on an experimental research methodology,
stemming mainly from current research in the field of empirical software engineering. In the
paper we specifically emphasize the need to conduct a family of related experiments, improving
and confirming each other, to produce relevant, empirically supported knowledge on the
validity and usefulness of metrics.

Keywords: conceptual data model, entity relationship diagrams, model evolution, model
quality, maintainability, structural complexity, metrics, prediction, theoretical validation,
empirical validation, experimentation

Page 3

2
1. Introduction
In the highly dynamic business environment we are in these days, existing business
models become obsolete at an ever-increasing pace and must therefore be designed
with flexibility in mind to satisfy the needs of agile companies. The constant reshaping
of business models increases the volatility of the data requirements that must be
supported by companies' data resource management policies and technologies. The
ability to rapidly change databases and their underlying data models to support the
needs of changing business models is a main concern of today's information managers.
Database evolution is a significant problem, high on the research agenda of information
system researchers.
It has been observed recently that conceptual data model quality, which is postulated as
a major determinant of the efficiency and effectiveness of data model evolution, is a
main topic in current conceptual modelling research (Olivé, 2002). Especially the
quality characteristic of model maintainability, i.e. the ability to easily change a model
(ISO, 2001), seems to be a key factor in conceptual data model evolution. In order to
evaluate and, if necessary, improve the maintainability of conceptual data models, data
analysts need instruments to assess the maintainability characteristics of the models they
are producing. The earlier this assessment can be done, the better, as it has proven
much more economical to evaluate and, if necessary, improve quality aspects during the
development process than afterwards (Boehm, 1981).
Maintainability is, however, an external quality property meaning that it can only be
assessed with respect to some operating environment or context of use (Fenton and
Pfleeger, 1997). The maintainability of a conceptual data model will, for instance,
depend on the model's understandability, which depends in turn on model properties
such as structure, clarity and self-expressiveness, but as well on the data analyst's
familiarity with the model (as for instance evidenced by the software cost drivers
included in the COCOMO II model for software cost estimation (Boehm et al., 1995)).
Likewise, the model's modifiability, another maintainability sub-characteristic, may
depend on factors external to the model, like the type of modifications that must be
applied. Therefore, external qualities such as maintainability are hard to measure
objectively early on in the modelling process. They generally need to be assessed in a
subjective way, for instance using expert opinions expressed through a formal or
informal scoring system.
For a more objective (and more cost-efficient) assessment of external quality attributes
such as maintainability, an indirect measurement based on internal model properties is
required (Fenton and Pfleeger, 1997). If a significant relationship with conceptual data
model maintainability can be demonstrated, then measures of internal model properties

Page 4

3
(i.e. “metrics”) can be used as indicators of sufficient or insufficient maintainability.
Once constructed and validated, a measurement-fed maintainability prediction model
can be implemented and employed with relatively few costs compared to an a-posteriori
maintainability assessment.
Therefore, the early availability of metrics would allow database designers to perform:
− a quantitative comparison of design alternatives, and therefore an objective selection
between several conceptual data model alternatives with equivalent semantic
content.
− an early assessment of conceptual data model maintainability, even during the
modelling activity, and therefore a better resource allocation based on this
assessment (e.g. redesigning high-risk models with respect to maintainability).
In this paper we present a set of metrics for measuring the structural complexity of ER
diagrams and present three controlled experiments we carried out in order to gather
empirical evidence that these metrics are related to ER diagram maintainability. Our
focus on ER modelling is justified by the observation that in today's database design
world, it is still the dominant method of conceptual data modelling (Muller, 1999). Our
focus on structural complexity as an internal model property that is potentially related to
ER diagram maintainability is motivated by similar research in the field of empirical
software engineering, where structural complexity has been shown to be a major
determinant of external software qualities, including maintainability (Li and Henry,
1993; Harrison et al., 2000; Briand et al., 2001a; Fioravanti and Nesi, 2001; Poels and
Dedene, 2001; Bandi et al., 2003). Moreover, as a syntactic model property, the
measurement of ER diagram structural complexity can be automated, which assures the
cost-efficiency, the consistency and repeatability of the (indirect) maintainability
assessments.
Apart from the proposal of ER diagram metrics to be used as maintainability indicators,
a contribution of this paper is a methodological approach to metric definition, based on
suggestions provided in Briand et al. (1995) and Calero et al. (2001), paying extensive
attention to the validation of the metrics as measures of internal model properties and as
indicators of external quality. For the validation of the metrics as ER diagram structural
complexity measures we use an approach grounded in Measurement Theory that was
proposed by Poels and Dedene (2000) for the theoretical validation of software metrics.
For the validation of the metrics as ER diagram maintainability indicators a series of
three related experiments was conducted in which the relationship between the metrics
values and direct maintainability assessments was statistically analysed for significance.
As far as we know, this is the first extensive empirical investigation of the relationship
between ER diagram maintainability and ER diagram internal properties related to

Page 5

4
structural complexity. Apart from being able to assess ER diagram maintainability in
an objective and cost-efficient manner, our study may help towards understanding what
factors make a model maintainable and what factors may prove a hindrance to model
evolution properties, in particular maintainability characteristics. From this
understanding, quality-focused model design knowledge may be derived which can then
be incorporated into the modelling language, method and process.
This paper is structured as follows. In section 2 we discuss related work in the field of
metrics-based conceptual data model quality assurance. Our metric definition and
validation methodology is described and motivated in section 3. After proposing our
metrics suite in section 4, we proceed with the theoretical and empirical validation of
the metrics in sections 5 and 6 respectively. Finally, concluding remarks and an outline
of our future work is presented in section 7.

2. Related work
Even though several quality frameworks for conceptual data models have been
proposed (Lindland et al., 1994; Moody and Shanks, 1994; Krogstie et al., 1995;
Schuette and Rotthowe, 1998) most of them do not include quantitative measures (i.e.
metrics) to evaluate the quality of conceptual data models in an objective and cost-
efficient way. Moreover, the metrics that can be found in these frameworks have not
been thoroughly validated. The evidence that they measure the quality properties that
they are purported to measure is scarce.
In order to illustrate this, we will briefly comment the existing studies related to metrics
for conceptual data models.
Eick (1991) proposed a single quality metric applied to S-diagrams1 (Eick and
Lockemann, 1985; Eick and Raupp, 1991). Three quality aspects, expressiveness,
complexity and normalizedness normality, are meant to be captured by this metric. To
our knowledge there is no published work confirming the theoretical or empirical
validity of Eick´s metric.
Gray et al. (1991) proposed a suite of metrics for evaluating the quality characteristics
of ER diagrams (complexity and deviation from third normal form). These authors
commented that empirical validation of these metrics has been performed, but they do
not provide the results. Independently, Ince and Shepperd have carried out a theoretical
validation of the metrics using the algebraic specification language OBJ (Ince and
Shepperd, 1991) demonstrating the correctness of the underlying syntax of the metrics.

1 The S-diagram is a data model which was influenced by the work on the binary
relation model (Abrial, 1974) and The Semantic Database Model (SDM) (Hammer and
McLeod, 1981).