Content uploaded by Stanislav Chren
Author content
All content in this area was uploaded by Stanislav Chren on Mar 06, 2019
Content may be subject to copyright.
Mistakes in UML Diagrams: Analysis of
Student Projects in a Software Engineering Course
Stanislav Chren, Barbora Buhnova, Martin Macak, Lukas Daubner, Bruno Rossi
Faculty of Informatics
Masaryk University
Brno, Czech Republic
{chren, buhnova, macak, daubner, brossi}@mail.muni.cz
Abstract—The Unified Modelling Language (UML) is being
widely accepted as a modelling notation for visualizing software
systems during design and development. UML has thus become
part of many software engineering course curricula at universities
worldwide, providing a recognized tool for practical training of
students in understanding and visualizing software design. It is
however common that students have difficulties in absorbing
UML in its complexity, and often repeat the same mistakes
that have been observed by course tutors in previous years.
Having a catalogue of such mistakes could hence increase the
effectiveness of both teaching and learning of UML diagrams.
In this paper, we introduce such a catalogue, consisting
of 146 types of mistakes in eight types of diagrams. As the
main contribution of this study, we use this catalogue to guide
the analysis of student projects within a software engineering
course. In total, over 2,700 diagrams submitted over 12 weeks
of a semester by 123 students were analysed to identify the
frequency of mistakes (from the catalogue), correlations of the
mistakes between different diagram types, correlation of the
quality of student projects to exam results, student behaviour
in terms of introducing and fixing the mistakes over time, and
other interesting insights. The analysis is described together with
its setup and execution, and all datasets and detailed guidebook
to the catalogue of all mistakes is made available for download.
Index Terms—Software Engineering, Unified Modelling Lan-
guage, Common Mistakes, Software Engineering Course
I. INTRODUCTION
It has been over 20 years since the Unified Modelling Lan-
guage (UML) was introduced by Grady Booch, Ivar Jacobson
and James Rumbaugh as a standard notation for visualizing
system design [1], [2]. Since then, UML found its way to cur-
ricula of most established universities worldwide, and became
a piece of implicit knowledge of many software engineers.
The worldwide popularity of UML does not however make it
any easier for a novice software engineering student to learn
UML and its proper usage, which is why we see the students
repeating the same mistakes in their UML models every year.
Over the six years of experience with teaching UML within
a general software engineering course, we came to realise
that a catalogue of commonly encountered mistakes in UML
diagrams would be invaluable to both the students and tutors
of the course seminars, within which the students work on
their UML modelling assignments. Furthermore, when linked
with the statistics of the occurence of such mistakes in student
projects, it could be of great help to novice UML tutors (of
not only our course), who can this way better understand the
difficulties that students experience during UML usage, and
save time when correcting the assignments thanks to having
structured guidance.
The existing support in this way is however limited. Ex-
isting checklists and catalogues of UML defects have limited
scope [3], [4], [5] (given by a different purpose for which they
were designed), and existing approaches rather focus on the
ability to identify mistakes in UML models [6], [7], [8], [9]
than to understand what mistakes are actually being done by
students and how common they are.
This led us to start collecting a working catalogue of mis-
takes in student projects back in April 2017. The foundation of
the catalogue has been built on the best practices covered by
the book by Arlow and Neustadt [10], which is used as a ref-
erence book in our software engineering course. The catalogue
has been then refined with the findings from surveying existing
works on the topic (see Section II), and then revised by our
course seminar tutors, who contributed with their experience.
The catalogue has been then validated by a bachelor student
(within her bachelor-thesis project between September and
December 2017) against UML models of students delivered
within our course in the Autumn 2016 semester, and further
refined and structured. Then a guidebook of sample examples
(UML model excerpts) to a stable set of 146 types of mistakes
have been created (between January and March 2018) and ob-
jectives have been defined for a detailed quantitative analysis
of student assignments delivered in the Autumn 2017 semester
based on this catalogue (done between May and August 2018).
All together, over 2,700 UML models have been manually
processed and checked against our catalogue of 146 mistakes
defined for eight types of diagrams.
The analysis revealed many interesting insights. Some
brought quantitative confirmation of results of surveys and
questionnaires in related work. For instance, we confirm that
proper understanding of the modelled system’s domain and
the system’s requirements is important as it is a cause for
about 67% of mistakes found in students’ UML diagrams.
Other brought new insights, for instance on the dependencies
between occurrence of mistakes in multiple diagrams, or the
correlations of the quality of student assignments to exam
results, or experience of seminar tutors. Thanks to the analysis
of the evolution of each diagram over the 12 weeks of the
semester, we were able to study also the evolution of mistakes,
and the behaviour of students in terms of how and when they
typically introduce new mistakes and fix those already present.
The remainder of the paper is organized as follows. Sec-
tion II reviews existing resources on the common mistakes
in UML diagrams, which were also used to establish the
initial catalogue of mistakes used for our analysis. Section III
presents the Software Engineering course that served as the
context for our analysis. Section IV describes the setup and
execution of our analysis. Section V outlines our catalogue of
mistakes in UML diagrams, which are analysed and discussed
in Section VI. Section VII sums up the key findings, and
Section VIII concludes the paper.
II. RE LATE D WOR K
Students and teachers of software engineering courses with
practical component of UML modelling are supported by
plethora of books and other resources on UML and its correct
usage [1], [2], [10]. Yet, there have been only a few attempts
to create a catalogue of mistakes in UML models that could
serve as a checklist for either the students themselves or their
tutors.
Cooper et al. presented a survey of the literature on
checklists together with their experiences on using defect
checklists in software engineering classes at both graduate and
undergraduate levels [3]. The checklists used in their study
covered the use case model of the requirements, architecture
model, analysis model, detailed design model, and blackbox
system level test cases. The coverage of the UML diagrams by
the checklists was however quite limited due to a rather general
scope of the study (focusing also on project management,
quality assurance plan, etc.). Lange et al. contributed to the
state of the art with a catalogue of defects in UML models
used primarily to understand how practitioners use UML in
industrial projects [11], [12]. To this end, they implemented
a tool called MetricView that checks the models for defects,
disproportions, and risks of misunderstanding. Yet again, the
catalogue itself covered the UML notation only partially (8
types of mistakes as listed in [4]).
Another body of knowledge is available within the formal
methods community, where approaches are available that
check UML diagrams for possible syntactic and consistency
errors, via automated transformation of UML into formal
notations and their model checking [8], [9]. Campbell et al.
for instance employs SPIN model checker to check errors
and inconsistencies in UML diagrams classified into two cat-
egories: structural errors, which deal with syntactic and static
layout inconsistencies (e.g., variables referenced in a state
diagram must be defined as attributes of the state diagram’s
corresponding class; every state diagram must have an initial
state), and behavioural errors, which deal with the dynamic
aspects of a system (e.g., violation of temporal properties) [8].
These approaches however focus on the methods of automated
analysis rather than the errors themselves, and hence only
a few exaples of the errors are provided.
Multiple approaches study the issues within teaching, learn-
ing and inspecting UML models from the process perspec-
tive [5], [6], [13], [7]. Conradi et al. study the support of UML
model inspections with specialized Object Oriented Reading
Techniques (OORTs), accompanied with checklists and special
reading guidelines, and applied in industrial setup (Ericson
in Norway) [5]. Seven individual reading techniques are dis-
cussed, where each either compares two UML diagrams with
each other, or checks one UML diagram against a Require-
ments Description. The focus is however solely on the review
process, with only a limited view of the defect catalogue
(consistency issues only). The same is true for Laitenberger et
al., who also focus on the process perspective via comparing
perspective-based reading (PBR) for defect detection in UML
models with a more traditional checklist-based reading (CBR)
approach [6]. The comparison was performed in a controlled
experiment with 18 practitioners using these techniques to
identify injected defects in prepared models. Seeded defects
were also used by Labiche in UML models evaluation by
students [7].
While these resources provide a good starting point for
understanding the mistakes that one can make in UML mod-
els (where each became a piece of knowledge used by us
when compiling the catalogue of UML mistakes used in our
study), they do not give much insight into what mistakes are
actually being made by software engineering students when
learning UML. We have not been able to find any published
study reporting on UML mistakes encountered in a significant
number of student assignments, and hence the experience on
the most/least common mistakes and their correlation to other
factors seems to remain unexplored.
Studies only exist on students’ self-perception of learning
difficulties [13], and cognitive dimensions defining the factors
that influence diagram comprehensibility as an important point
in learning [14], but with not much quantitative data support-
ing the findings.
III. SOF TWAR E ENGINEERING COURSE STRUCTURE
The FI:PB007 software engineering course has been taught
at the Faculty of Informatics, Masaryk University since 2002.
Originally, it was called Analysis and Design of Systems and
focused on the structured methods of system analysis and
design. Later, in 2012 it was renamed to Software Engineer-
ing I and its scope changed to cover the whole software
engineering process of software development, accompanied
with practical understanding of UML and its diagrams. The
course has two main goals. First, to give students complex
overview of key activities within software system develop-
ment, including methods of requirements specification, system
analysis and design, architecture design, testing, and system
evolution/maintenance. The second goal is to make students
proficient in using UML to design and model middle-sized
information systems. The course consists of lectures (100
minutes per week, 12 weeks in total) and practical seminars
(100 minutes per week, 12 weeks in total) discussed later in
this section.
The UML diagrams that are covered by the course are
shown in Fig. 1. Besides the common UML diagrams, we
also include the Entity-Relationship Diagram (ERD) in our
course and study, as it is still a key diagram within software
development (although not being a UML diagram).
Fig. 1. Overview of UML diagrams. Blue and red diagrams are covered in
the course. The red diagrams are included in the catalogue of mistakes.
The course is taught yearly and is mandatory in most of the
bachelor study programs at the faculty. Moreover, it can be
freely enrolled by students from any study program including
students from other faculties. At the faculty, majority students
enroll it either in the 5th or 3rd semester of their bachelor
studies. In 2017, we had 123 students available for our study.
93% of them were bachelor students from our faculty, the
rest were students of Faculty of Science and Faculty of
Ecnonomics and Administration.
For passing of the course, it is required that the students
complete a team project at the seminars (via weekly delivery
of individual model assignments), and they collect at least 50
points out of 90 during the semester. The points are awarded
as follows: 70 points can be received for the final exam
which consists of a knowledge test (35 points) in form of
a multiple-choice questionnaire, and practical assignment of
an on-site UML modelling challenge (35 points). The students
have 120 minutes to complete the exam and in case of failure
they are allowed to repeat the exam once. Further points can
be earned from the questionnaires in practical seminars (20
points in total) and up to 5 extra points for activity during
the seminars. The activity might involve carrying out the
weekly assignments flawlessly or to actively participate in the
activities during the seminar sessions.
A. Lectures structure
There are 12 lectures during the semester, one lecture per
week. The content of the lectures is largely inspired by the
books [17] and [10]. In the first 8 weeks, each lecture is
split into two parts. The first one focuses on the software
engineering methods, the second part gives a background
knowledge to a specific UML diagram that is then practically
covered during seminar sessions. The lectures are:
•Week 1 - Software development, UML Use Case diagram
•Week 2 - Requirements specification, UML Activity diagram
•Week 3 - System analysis and design, structured vs. object-oriented A&D
•Week 4 - Object oriented analysis, UML Class, Object and State diagram
•Week 5 - Structured analysis, data modelling, ERD
•Week 6 - High-level design, UML Class diagram in design
•Week 7 - Low-level design and implementation, UML Interaction diagrams
•Week 8 - Architecture design, UML Package, Component and Deployment d.
•Week 9 - Testing, verification and validation
•Week 10 - Operation, maintenance and system evolution
•Week 11 - Software development management
•Week 12 - Advanced software engineering techniques
B. Practical seminars structure
The seminars of the FI:PB007 course focus on the practical
modelling with UML diagrams. The students are enrolled
into seminar groups, each with maximum capacity of 15
students. Each seminar last 100 minutes. It starts with a short
questionnaire at the beginning of the seminar consisting of four
multiple-choice questions related to the diagram to be used at
that seminar. The purpose of the questionnaire is to check entry
knowledge for the seminar. After the questionnaire, the semi-
nar tutor gives a brief summary of the diagram and its usage,
and possibly involves students in some exercise or activity in
order to help them understand practical challenges related to
the diagram. Afterwards, the seminar tutor demonstrates how
the diagram is modelled in the Visual Paradigm tool used in the
seminars, and let the student to use the rest of the seminar time
to start working of the assignment for the week (create model
using the discussed diagram). The content of the seminars is:
•Week 1 - Visual Paradigm introduction, project assignment.
•Week 2 - Initial Use Case diagram.
•Week 3 - Detailed Use Case diagram, textual specification of UC
•Week 4 - Activity diagram.
•Week 5 - Analytical Class diagram
•Week 6 - Finalization of Analytical Class diagram
•Week 7 - State Machine diagram.
•Week 8 - Entity Relationship diagram.
•Week 9 - Design Class diagram
•Week 10 - Interaction diagrams
•Week 11 - Finalization of Interaction diagrams
•Week 12 - Package and Deployment diagram.
•Week 13 - Final consultation, if needed.
All the diagrams are modelled with respect to a given
project assignment, which is given to the students at the first
seminar. In the rest of the semester, students work in groups
of two or three. During that time, the seminar tutor provides
any assistance and consultations that is required. The seminar
tutor also discusses with the students the mistakes in their
solutions submitted in the previous week. Every week the
students submit their diagrams in the form of PDF reports
generated from the Visual Paradigm tool. Only one report per
team is submitted, which contains the new diagram as well as
all the previous ones. The deadline for submission is 48 hours
before the next seminar (where some of the students manage
to complete the diagram already at the seminar). If the students
miss the deadline, they are each penalised with 5 points. The
students are allowed to take the final exam only after the final
report with all diagrams is submitted and accepted (as not
containing any serious mistakes). If their final submission is
not accepted, they are required to correct their solution and
resubmit it again.
C. Tools and resources
In the seminar sessions, the Visual Paradigm1toolset (Stan-
dard edition) is used for modelling UML diagrams and for
1URL https://www.visual-paradigm.com/
generating PDF reports. The generated PDF reports are sub-
mitted to the repository in the university Information System
(IS). Each seminar group and each week has its own repository
folder. The students do not see submitted files of other stu-
dents. Additionally, the students are encouraged to use version
control for their project files. SVN and GitLab accounts are
available for students at the faculty servers. Optionally, it is
possible to use the Teamwork collaboration tool which is part
of the Visual Paradigm suite. There is also online syllabus
in the information system which includes additional related
study resources for each of the diagrams. Moreover, there
are specific discussion fora in the IS for the course and the
individual seminar groups in which students and tutors can
discuss in case of questions or issues.
IV. ANA LYSI S SE TU P AN D EX EC UT IO N
Our study consisted of six phases outlined in this section.
The remainder of the paper then reports on the two key
outcomes of the study, the catalogue of mistakes in UML
diagrams (Section V) and analysis of mistakes identified
during our analysis of student projects from Autumn 2017
semester (Section VI).
Phase 1 – Literature review (April–May, 2017): The initial
inputs for our catalogue of mistakes in UML models have been
collected from existing works on the topic (see Section II) in
combination with violations of the best practices covered by
the book by Arlow and Neustadt [10] (used as the primary
literature within our course).
Phase 2 – Catalogue composition (June–August, 2017):
The inputs have been cleaned (to resolve homonyms, overlaps,
and unify naming) and structured into an initial version of
the catalogue, which has been reviewed by four tutors of
the practical seminars of the course (all with over two years
of experience with teaching the course), who contributed
to the catalogue with their experience. The feedback has
been consolidated and resulted in the second version of the
catalogue.
Phase 3 – Catalogue validation (September–December,
2017): The catalogue has been then validated by a bache-
lor student (within her bachelor-thesis project) against UML
models of students delivered within our course in the Autumn
2016 semester. Within the validation 1020 models have been
checked against the catalogue, and newly encountered types of
mistakes have been added to the catalogue. The resulting third
version of the catalogue has been again reviewed by course
tutors.
Phase 4 – Mistakes guidebook (January–March, 2018):
Based on the third version of the catalogue, a guidebook
with sample examples (UML model excerpts) to the stable
set of mistakes have been created, which then also served as
a documentation of the mistake types during later phases of the
analysis. Example of a mistake illustration from the guidebook
is discussed later in Section V, Fig. 2.
Phase 5 – Analysis preparation (April, 2018): The prepara-
tory phase of the analysis consisted of data collection from
the Autumn 2017 semester, which consisted of over 2,700
individual UML models delivered by over 100 students. The
models were structured by student teams and seminar groups,
as well as the weeks of the semester in which they were
delivered. Within this phase, the scope and objectives of the
analysis were defined.
Phase 6 – Data analysis (May–August, 2018): The final
phase consisted of the analysis itself together with the inter-
pretation and validation of the results. This phase together with
the preparation for it is further elaborated in Section IV.
V. CATALOG UE O F MI STA KE S IN UML DIAGRAMS
The catalogue (in its third and stable version) contains
146 mistakes, which are structured into eight sections, ac-
cording to the diagrams they realate to. We consider the
following diagrams: Use Case diagram, Activity diagram,
Class diagram in two variants (Analytical Class diagram and
Design Class diagram [10]), State Machine diagram, Sequence
diagram, Communication diagram, and Entity-Relationship
diagram. Entity-Relationship diagram was included because
of its important role within software engineering, although it
is not a UML diagram. The mistakes are accompanied with
a short explanation and visual example together with possible
solutions.
The full catalogue of mistakes together with the guidebook
of sample solutions is available at [18]. Since the FI:PB007
course if taught in Czech, the catalogue itself is in Czech.
However, at [18] we give English translation of the key terms
used in the catalogue that should make it possible to navigate
the catalogue even without knowledge of Czech, and we
plan to add English version of the catalogue soon. Due to
space limitation, we cannot describe all the mistakes from the
catalogue here. Instead, in this section, we give a summary of
the main types of mistakes, demonstrated with some examples.
For illustration, Table I lists all the mistakes related to
the Activity diagram, and Fig. 2 gives a graphical example
included in the guidebook of the mistakes and their sample
solution.
Fig. 2. Example of a mistake (no. 2.06) from the guidebook.
We classify the mistakes in the catalogue in two dimensions.
The first dimension represents the severity of the mistake. We
distinguish two types of severity: (1) violations of the UML
specification or (2) violations of the best practice guidelines as
introduced in [10]. We consider both violations important. The
violation of the former might not necessarily have impact on
TABLE I
LIS T OF MI STAK ES FO R ACT IVI TY D IAG RAM .
Id Mistake name
2.01 Missing start or end point
2.02 Output flows from decision node are not disjoint
2.03 Multiple input control flows into action node
2.04 Using decision node also as merge node
2.05 Missing decision node description
2.06 Multiple output control flows from action node
2.07 The condition of decision node should not represent action
2.08 Missing output flows from action node
2.09 Wrong activity workflow
the readability or understandability by the user, customer, soft-
ware architect or developer. However, it might cause problems,
for instance in case of automated model processing. On the
other hand, although the violation of best practice guidelines
might still result in valid UML models, they are very likely to
introduce problems with model understandability or with the
quality of the system analysis and design [14].
The second dimension refers to the nature of the mistakes.
Here we consider four main categories of mistakes, where
each category can contain several subcategories. Each mistake
belongs to one or more categories and to one subcategory per
the main category. The overview of the classification scheme
is in Fig. 3.
The Domain understanding category covers mistakes that
are caused by the lack of knowledge of the modelled system
domain. This can mean, for example lack of detailed require-
ments specification for the system, insufficient understanding
of the system processes, components or concepts present in the
modelled domain, as well as vague boundaries of the system
which result in inconsistent level of detail in the diagrams and
also in the unclear inclusion/exclusion criteria of what should
be considered as part of the system.
The Diagram element category contains mistakes that are
related to a particular UML element in the diagram. The
subcategories of this category represent element types present
in various UML diagrams. For example in the Use Case
diagram, the mistake can be related to a use case, actor or re-
lationship specification. In the Interaction diagrams (Sequence
and Communication diagrams) the mistakes can be related to
a lifeline, message or combined fragment specification.
The Clarity category consists of issues that make the
diagram harder to read and understand. This can be caused, for
example by improper naming of elements or by unclear visual
placement of the elements, such as crossing of relationship
lines or overlapping nameplates, nodes or classes.
The mistakes in the Consistency category are caused by
not maintaining the dependencies between the diagrams. For
example, the State Machine diagrams and Interaction diagrams
should use only methods and attributes present in the Class
diagram.
VI. ANALYSIS OF MISTAKES IN STUDENTS’PROJECTS
In this section, we present our findings from the analysis of
mistakes in UML diagrams that were collected from weekly
Fig. 3. Orthogonal classification scheme for diagram issues.
reports submitted by students of the FI:PB007 course during
the autumn 2017 semester. Our analysis covers reports from 10
seminar groups. Each seminar group consists of 3-5 teams that
are formed by 2-3 students. Overall, we collected data from
42 teams and 123 students in total. Each team submits a report
with modelled UML diagrams every week over the course of
12 weeks. Every week the report contains the main diagram
for that week (see Section III for the schedule) together
with all the diagrams from previous weeks. In each report,
we investigated the presence of mistakes according to the
catalogue. In order to also evaluate the changes in the diagrams
between the weeks (changes due to mistake corrections), we
examined all diagrams in every report. During the analysis,
we processed 504 reports and evaluated over 2,700 UML
diagrams. The complete dataset containing all the data that
was used for the analysis is available at [18].
A. Overall occurrence of mistakes
To get an overview about the occurrence of individual
mistakes in the submitted reports, we computed the total
presence values for each mistake. The total presence represents
number of teams which made the given mistake in at least one
of their reports. In Table II we give a summary of the Top 4
mistakes for each covered diagram. The full dataset contains
also the number of weeks in which the uncorrected mistake
occurred for a given team which we refer to as total mistake
occurrence.
Additionally, in Fig. 4, we provide an overview of the
total presence of mistakes according to the main categories
specified in the Section V. One mistake can belong to multiple
categories. For example, the mistake 2.05 from Table I belongs
to the Element category since it is related to the decision node
element and Clarity category because it might cause confusion
in the diagram understanding. There, one can see that over
65% of all mistakes present in the UML models can be at-
tributed to the insufficient understanding of system’s domain or
specification. Next, approximately 30% of mistakes are related
to decreased visual readability of the diagram. Lastly, 30%
are caused by not maintaining consistency between various
diagrams.
Moreover, in Fig. 5 one can see that in all types of diagrams,
the mistakes are related to some UML element. However, the
distribution of mistakes belonging to also other categories is
varied across the diagrams.
TABLE II
OVE RVIE W OF TO P 4FRE QU ENT M IS TAKE S IN TH E IN DIV ID UAL TY PE S OF DI AGR AM S TOG ETH ER W ITH N UM BER O F TE AMS M AKI NG T HE MI STAK E.
Use Case diagram # % Entity-Relationship diagram # %
1.10 Mistakes in text specification of UC 18 42.9% 5.09 Is not consistent with Class diagram 26 61.9%
1.11 Unsuitable ”include” relationship 14 33.3% 5.01 Redundant or missing relationships 21 50.0%
1.16 Use case naming 14 33.3% 5.03 Replacing ”0” in multiplicities with ”1” 12 28.6%
1.07 Misuse of include/extend/inheritance for functional decomposition 13 31% 5.08 Wrong multiplicities 12 28.6%
Activity diagram # % Design Class diagram # %
2.05 Missing decision node description 18 42.9% 6.11 Missing class operations 25 59.5%
2.03 Multiple control flows enter one action node 13 31% 6.18 Missing dependencies 23 54.8%
2.04 Using the decision node also as a merge node 12 28.6% 6.06 Missing arguments in operations 22 52.4%
2.07 The condition of the decision node should not express an action 11 26.2% 6.24 Wrong/missing type of method 18 42.9%
Analytical Class diagram # % Sequence diagram # %
3.22 Associations have missing navigability 28 66.7% 7.02 Incorrect lifeline naming 26 61.9%
3.09 The diagram contains implementation details 25 59.5% 7.14 Messages are not consistent with Class diagrams 21 50.0%
3.19 Missing description on associations 24 57.1% 7.03 Using asynchronous messages instead of synchronous 20 47.6%
3.08 Operations placed into inappropriate classes 22 52.4% 7.08 Message uses unknown entity 20 47.6%
State Machine diagram # % Communication diagram # %
4.02 Using unknown operation in call event 32 76.2% 8.01 Incorrect lifeline naming 24 57.1%
4.03 Using unknown operations or attributes in actions 30 71.4% 8.03 Wrong message arguments 20 47.6%
4.10 Adding redundant attributes to transitions 22 52.4% 8.02 Sending message to the general class instead of instance 16 38.1%
4.14 Empty transition to the end state 19 45.2% 8.08 Message does not represent method from Class diagram 14 33.3%
Fig. 4. Distribution of main categories for total presence of mistakes
(E=Element, D=Domain knowledge, CL=Clarity, CO=Consistency).
Fig. 5. Total presence of mistakes according to their main categories
(E=Element, D=Domain knowledge, CL=Clarity, CO=Consistency).
The most frequent mistakes in the Use Case diagram are
related to the textual specification of use case details which
was made by almost 43% of all teams. Other problematic
parts are the proper naming of use cases (1/3 of all teams)
and unsuitable usage of include, extend or inheritance rela-
tionships. Mistakes related to the textual specification might be
caused by insufficient domain knowledge, since this is the first
modelled diagram and the students might not be yet familiar
with the system specification in greater detail. The problems
with naming on the other hand negatively impact clarity of
the diagram. Surprisingly, despite the problems with naming
of use cases there was almost no issue with actor naming (only
one occurrence).
In the Activity diagram, three of the top mistakes are related
to the specification of the decision nodes with missing descrip-
tion (43% of teams), which might lead to worse readability of
the diagram. On the other hand, only 12% of teams modelled
a workflow that would make no sense and only 5% forgot to
add the start and end points.
The most frequent mistakes in the Analytical Class diagram
are related to the modelling of associations. Two thirds of all
teams did not specify the navigability of the associations and
57% of teams did not sufficiently describe the associations,
which might lead to lower understandability of the diagram.
Since the Analytical Class diagram is a sort of conceptual
model for the given domain, many mistakes are caused by
problems with the domain understanding, which results in
mistakes such as unsuitable levels of details (60% of teams)
or placing the operations into improper classes (52% of
teams). On the contrary, the least frequent mistakes constitute,
for example reversed direction of generalisation or mistakes
related to improper referencing of objects via attributes.
State Machine diagram mistakes are mainly consistency
issues, for example there are operation calls in the call events,
or on the transition actions are not present in the right class
in the Class diagram (over 70% teams made these mistakes).
Although mistakes related to the call events are very common,
other event types seems to be better understood and mistakes
concerning the time events or change events are the least
frequent mistakes (between 7% and 12%).
The majority of the mistakes in the Entity-Relationship
diagram are related to the specification of relationships and
are again related to the inconsistency with the Class diagram.
They relationships are missing or redundant (50% of teams) or
they have wrong multiplicities (29% of teams). On the other
hand, unlike other diagrams, the Entity-Relationship diagrams
do not suffer from unclear naming of entities or attributes (only
5% of teams).
Design Class diagrams consist of more details than the
Analytical Class diagrams and thus the mistakes are much
frequently focused solely either towards the problems in
diagram elements, such as relationships or to the problems
with the domain knowledge, which is expressed through the
mistakes with detailed specification of methods. Additionally,
while in the Analytical Class diagram, the improper use of
inheritance was very common (1/3 of all teams), in Design
Class diagram, this seems to be an issue no longer (only 10%
of teams).
The mistakes in the Sequence diagram are related to the
domain knowledge of the system, which in this case focuses on
the understanding of the system behaviour. Frequent mistakes
also include improper naming of lifelines (62% of teams) or
misunderstanding the semantics of different message arrow
shapes (50% of teams). On the other hand, the mistakes related
to controlling the behaviour via the combined fragments are
not that prominent, especially when related to specification of
conditions (10%) or iterations in loops (12% of teams).
Communication diagram are related to the Sequence dia-
grams and they also share the most frequent mistake in wrong
naming of the lifelines (57% of teams). Just like the Sequence
diagram, also here there are issues with consistency between
Interaction diagram and the Class diagram (1/3 of all teams).
The least frequent mistakes here are related to the creation of
new objects (only 8% of teams).
The percentages of teams involved in the most frequent
mistakes suggest that the most problematic diagram types are
the Analytical Class diagram (all top mistakes were present
in reports of more than half of the teams) and State Machine
diagram (with consistency related mistakes that are present
in more than 70% of all teams). Interaction diagrams also
show high number of mistakes with Communication diagram
seeming to be a bit less problematic. However, this might be
caused by the selection of use cases that tend to be more
simplistic in case of Communication diagrams.
B. Continuous changes in mistakes
To gain insight into the evolution of mistakes, we have anal-
ysed the changes in numbers of mistakes for each assignment
submission. The goal was to spot the behaviour of students,
how and when they typically introduce new mistakes and fix
those already present. Each diagram in question was processed
and analysed separately. The reason is that students work on
a different diagram each week and that each diagram has
different numbers of mistakes defined in the catalogue.
We define the change in the diagram as a difference between
the number of new mistakes and fixed mistakes compared to
the number of mistakes in the previous week. The data point is
the average change of all teams for each week. A value of zero
means that there is no change in the number of mistakes. Note
that while zero typically suggests that there was no change
in the diagram, it could also mean that the same number
of mistakes were fixed and introduced. A positive number
tells that there were more mistakes introduced than fixed.
A negative number indicates that there were more mistakes
fixed than introduced. The continuous mean changes of errors
can be seen on the visualisations in Fig. 6.
Fig. 6. Continuous changes of mistakes
We have noticed several properties which correspond to the
stages of the assignment. The initial spike of new issues is cor-
responding to the week in which the diagram is introduced to
students. In the following weeks, students are fixing the issues
under the guidance of tutors. For the Use Case, Sequence, and
Communication diagrams, the next week after the introduction
of the diagram has still positive mean change. It may be caused
because the Use Case diagram is the first diagram in the
FI:PB007 course, and because Sequence and Communication
diagrams are the hardest to understand correctly. The changes
tend to stabilise as the diagram is not worked upon, which
is most evident in the first two diagrams. In the final stage,
there is a spike in fixed errors because all diagrams should be
submitted without mistakes. The final report often consists of
multiple iterations until the tutor is contented with the work,
but here we consider only the final one.
C. Dependencies between mistakes
One of the most interesting findings are the dependencies
between mistakes. With well-defined mistakes, thanks to the
catalogue, we can search for the mistakes that often occur
simultaneously. Or in converse, if they typically do not occur
simultaneously. In other words, we are looking for correlated
mistakes. However, the correlation itself is not the only condi-
tion. Dependent mistakes that are considered must be frequent
(i.e., the issue must have been made by at least ten teams) and
must share a category. The motivation for this analysis is to
provide hints about possible mistakes and to spot common
problems.
Here we present notable examples of interesting correlated
mistakes we have found by analyzing the student submissions:
Fig. 7. Correlation between occurrence of mistake 4.3 and 4.10.
1) Using unknown operations or attributes in actions &
Adding redundant attributes to transitions (4.3, 4.10): Both
mistakes are among the most frequent ones for the State
Machine diagram. But most importantly, they have a common
cause. They refer to an ad-hoc introduction of new methods
and attributes that are inconsistent with a Class diagram. Fig. 7
shows the occurrences with the correlation coefficient.
Fig. 8. Correlation between occurrence of mistake 6.06 and 6.24.
2) Missing arguments in operations & Wrong / missing
type of method (6.06, 6.24): In the Design Class diagram,
missing arguments in operation are one of the most frequent
mistakes. The other issue is not as frequent. However, both
of them are pointing to a common problem that is undefined
or insufficiently defined methods in classes. The correlation
coefficient with occurrences of these mistakes is illustrated in
Fig. 8.
Fig. 9. Correlation between occurrence of mistake 7.10 and 7.14.
3) The message leads to the wrong lifeline & Messages are
not consistent with Class diagrams (7.10, 7.14): An example
of Sequence diagram mistakes. Both mistakes are addressing
problems with inconsistencies between methods in a Class
diagram and methods in Sequence diagram. Usually, it means
usage of wrong or made-up method in a Sequence diagram.
The occurrences and correlation coefficient is illustrated in
Fig. 9.
Fig. 10. Correlation between occurrence of mistake 8.04 and 8.07.
4) Modeling multiple communication links between two
objects & Bad / Missing Message Direction (8.04, 8.07):
Finally, for the Communication diagram, we have found two
mistakes regarding very basic syntax problems. Such mistakes
commonly hint poor understanding of a particular diagram.
Fig. 10 illustrates the occurrences and correlation coefficient.
D. Impact of the teacher experience on the quality of diagrams
The FI:PB007 course has multiple seminar tutors whose
task is to guide students in their work. The team of tutors
consists of both university employees and students of various
study programmes. This means that the teaching experience of
the tutors tend to vary from inexperienced to very experienced
teachers.
Fig. 11. Box plots showing differences between the tutors based on the total
presence of mistakes in the final report.
The data form student assignments provide a new possible
way to evaluate teaching quality. We had chosen, for each
student team, the total presence of mistakes, total occurrence
of mistakes and total presence of mistakes in the final report
as possible indicators of quality.
The main question is if the tutors experience positively
influences the number of mistakes in student reports. In other
words, if students guided by more experienced tutor tend to
make fewer mistakes. The tutor’s experience is expressed with
the number of years he or she had participated in the FI:PB007
course. The hypothesis is formulated as follows:
Hypothesis 1 (H0):The correlation of tutor experience and
total presence of mistakes is equal to 0.
Hypotheses for the total occurrence of mistakes and for the
total presence of mistakes in the final report are formulated
analogously.
Based on the test on 95% confidence interval, the null
hypothesis was not rejected for the total presence of mistakes
(ρ=−0.017,p-value = 0.914). On the contrary, the null
hypothesis was rejected in favor of alternative for the total
occurrence of mistakes (ρ=−0.319,p-value = 0.04) and
the total presence of mistakes in the final report (ρ=−0.534,
p-value = 0.00027).
Further analysis was focused on differences between the
tutors using ANOVA. For all three indicators, we had rejected
the hypothesis on equality of expected value. Fig. 11 illustrates
the differences based on the total presence of mistakes in the
final report.
We can argue that the non-rejection in case of total presence
of mistakes is more dependent on the students themselves.
Students naturally made mistakes; it is part of the learning
process. However, the same argument can be used for the
total occurrence of mistakes on which the null hypothesis
was rejected. It should be noted that the p-value is just
slightly below the threshold (0.05) which helps to support our
argument.
The total presence of mistakes in the final report, on the
other hand, seems like a valid indicator of teaching quality.
There usually are multiple iterations of the final report, and
the tutor should accept if it is satisfactory. If there are still
mistakes present, it could mean that the tutor missed them, or
accepted them as minor mistakes.
We can further argue that there might be differences in
tutor acceptance standards and that more experienced tutors
generally have higher standards. The catalogue itself might be
used to formalize a baseline for the final report and thus help
to unify the standard across tutors.
E. Impact of the quality of diagram on exam results
The students are allowed to take the final exam only after
they submit their final report containing all the UML diagrams
without any serious mistakes. The work on the UML diagrams
during the semester should also serve as a preparation for the
practical part of the final exam which consists of modeling of
given UML diagram.
Therefore, we investigated whether there is a dependency
between the quality of the submitted diagrams and the exam
results. Similarly to Section VI.D, as an indicator of quality,
we use the total presence of mistakes for individual student’s
team and the total presence of mistakes in the final report. The
practical part of the exam is evaluated with 0 to 35 points.
The hypothesis is formulated as follows:
Hypothesis 2 (H0):The correlation of practical exam result
and total presence of mistakes is equal to 0.
Hypothesis for the total presence of mistakes in the final
report is formulated analogously.
In Fig. 12 we show both results. In the first case (total
presence of mistakes) we fail to reject the null hypothesis on
95% confidence interval (ρ=−0.12,p-value = 0.18). It
means there is some low negative correlation between the total
presence of mistakes and exam results, but it is not statistically
significant.
On the other hand, in case of the total presence of mistakes
in the final report, we reject the null hypothesis in favor of
the alternative hypothesis (ρ=−0.22,p-value = 0.016). This
means that there is a negative correlation between mistakes in
the final report and the exam results and that the results are
statistically significant.
The results indicate that the students who managed to
correct their UML diagrams at the end of the semester have
a better chance to get good exam results. There is however
considerable scattering in the data which is apparent in Fig. 12.
This could be explained by the fact that during the semester,
students work in groups, while final exams are taken individ-
ually.
VII. DISCUSSION
Our large-scale analysis of UML diagrams revealed several
interesting observations. First, thanks to our classification
scheme, we have better understanding of the nature of possible
mistakes. Using that, we can modify our teaching approach,
Fig. 12. Correlations between the exam results and the total presence of
mistakes (top) as well as the total presence of mistakes in the final reports
(bottom).
for example to spend extra time on discussing the system
requirements or the system domain since the majority of the
mistakes were caused by lack of domain knowledge.
Second, the FI:PB007 course is specific in welcoming sem-
inar tutors with various degree of experience, such as senior
experts with rich teaching experience as well as competent
bachelor or master students without any previous teaching
experience. Our analysis has shown that the teaching expe-
rience does not have significant impact on the quality of the
student models during the semester. On the other hand, more
teaching experience is positively manifested in the quality of
final UML reports. On top of that we discovered that the
quality of the final reports has an impact on the marks obtained
from final exam. Therefore, in the future, we envision creation
of common guidelines with our catalogue as a basis that would
help the seminar tutors to improve their ability to thoroughly
evaluate submitted diagrams.
Third, we identified the most frequent mistakes and also the
diagrams which pose the greatest difficulty for the students
(The State Machine and Analytical Class diagram). We also
managed to reveal several dependencies between different mis-
takes. Additionally, we observed to what extent the diagrams
are worked on continuously during the semester. It turned
out that the students tend to polish the diagrams towards
the submission of the final report rather than focusing on
continuous improvement. All of these insights will help us to
update the seminar structure and the overall schedule in order
to address these specific challenges and thereby improve the
education process of the software engineering course.
VIII. CONCLUSION
The Unified Modelling Language (UML) has over time
became part of the essential software engineering knowledge,
adding to the importance of its effective learning and teaching
at universities, as well as commercial training programs.
In this paper, we support both students and tutors of the
UML notation with a catalogue of 146 types of mistakes for
eight types of diagrams, which can be used as a checklist
during UML modelling as well as during inspections of
submitted UML models. As the main contribution of this
paper, we have used this catalogue to analyse student projects
within a software engineering course, and thanks to the wide
scope of our analysis (over 2,700 analyzed diagrams of 123
students over 12 weeks of semester) bring many interesting
insights summarized in Section VII, which can be used to
create further supportive materials for the students and tutors
of other courses similar to ours.
REFERENCES
[1] Booch, G., Rumbaugh, J., and Jacobson, I. (1999). The Unified Model-
ing Language User Guide. Addison-Wesley.
[2] Rumbaugh, J., Jacobson, I., and Booch, G. (2004). Unified modeling
language reference manual. Pearson Higher Education.
[3] Cooper, K., Liddle, S., and Dascalu, S. (2005, November). Experiences
Using Defect Checklists in Software Engineering Education. In CAINE
(pp. 402-409).
[4] Lange, C. F., and Chaudron, M. R. (2006, May). Effects of defects in
UML models: an experimental investigation. In Proceedings of the 28th
international conference on Software engineering (pp. 401-411). ACM.
[5] Conradi, R., Mohagheghi, P., Arif, T., Hegde, L. C., Bunde, G. A.,
and Pedersen, A. (2003, July). Object-oriented reading techniques for
inspection of UML modelsan industrial experiment. In European Confer-
ence on Object-Oriented Programming (pp. 483-500). Springer, Berlin,
Heidelberg.
[6] Laitenberger, O., Atkinson, C., Schlich, M., and El Emam, K. (2000).
An experimental comparison of reading techniques for defect detection
in UML design documents. Journal of Systems and Software, 53(2),
183-204.
[7] Labiche, Y. (2008, September). The UML is more than boxes and lines.
In International Conference on Model Driven Engineering Languages
and Systems (pp. 375-386). Springer, Berlin, Heidelberg.
[8] Campbell, L. A., Cheng, B. H., McUmber, W. E., and Stirewalt, R. K.
(2002). Automatically detecting and visualising errors in UML diagrams.
Requirements Engineering, 7(4), 264-287.
[9] Lilius, J., and Paltor, I. P. (1999, October). vUML: A tool for verifying
UML models. In Automated Software Engineering, 1999. 14th IEEE
International Conference on. (pp. 255-258). IEEE.
[10] Arlow, J., and Neustadt, I. (2005). UML 2 and the unified process:
practical object-oriented analysis and design. Pearson Higher Education.
[11] Lange, C. F. (2006, May). Improving the quality of UML models in
practice. In Proceedings of the 28th international conference on Software
engineering (pp. 993-996). ACM.
[12] Lange, C. F., Chaudron, M. R., and Muskens, J. (2006). In practice:
UML software architecture and design description. IEEE software,
23(2), 40-46.
[13] Boberic-Krsticev, D., and Tesendic, D. (2013). Experience in Teaching
OOAD to Various Students. Informatics in Education, 12(1), 43-58.
[14] Kutar, M., Britton, C., and Barker, T. (2002). A comparison of em-
pirical study and cognitive dimensions analysis in the evaluation of
UML diagrams. In Procs of the 14th Workshop of the Psychology of
Programming Interest Group (PPIG 14).
[15] Kruchten, P. (2004). The rational unified process: an introduction.
Addison-Wesley Professional.
[16] Eclipse (available online, last accessed October 2018). Open Unified
Process. URL https://www.eclipse.org/epf/general/OpenUP.pdf.
[17] Sommerville, I. (2007). Software engineering. Addison-wesley.
[18] Chren, S., Buhnova, B., Macak, M., Daubner, L., and Rossi, B. (2018).
Catalogue and other materials. URL https://lasaris.fi.muni.cz/uml-
catalogue.
