Conference PaperPDF Available

Teaching Enterprise Modeling Based on Multi-media Simulation: A Pragmatic Approach

Authors:

Abstract and Figures

The paper addresses the problem of how university students can acquire enterprise modeling skills so that they can build high quality models of organizational structure and behavior in practical settings after their graduation. The best way of learning such skills is apprenticeship where the students follow a modeling master in a real business case. However, in a university classroom setting this is difficult to arrange, if even possible. Therefore, the paper suggests the use of a computer-based simulation as a good approximation to apprenticeship. Moreover, it suggests a pragmatic, low-cost approach making the idea accessible even for courses with a low budget. A business case is simulated by providing the students with multi-media information sources that are usually used by system or business analysts when building models. The sources consist of recorded interviews with the stakeholders, a web-site of the enterprise under investigation, internal protocols from management meetings, results of twitter search on the company name, etc. The paper presents practical guidelines on how to build such simulation based on a trial successfully completed at the Department of Computer and System Sciences at Stockholm University.
Content may be subject to copyright.
Published in Benyoucef, M. et al. (Eds.): E-Technologies, LNBIP 209, Springer, 2015
©Copyright Springer 2015
http://link.springer.com/chapter/10.1007/978-3-319-17957-5_16
1
Teaching Enterprise Modeling based on Multi-Media
Simulation: a Pragmatic Approach
Ilia Bider1, Martin Henkel1, Stewart Kowalski1,2, Erik Perjons1
1DSV Stockholm University, Stockholm, Sweden
2Gjøvik University College, Norway
{ilia|martinh|stewart|perjons}@dsv.su.se
Abstract. The paper addresses the problem of how university students can ac-
quire enterprise modeling skills so that they can build high quality models of
organizational structure and behavior in practical settings after their graduation.
The best way of learning such skills is apprenticeship where the students follow
a modeling master in a real business case. However, in a university classroom
setting this is difficult to arrange, if even possible. Therefore, the paper suggests
the use of a computer-based simulation as a good approximation to apprentice-
ship. Moreover, it suggests a pragmatic, low-cost approach making the idea ac-
cessible even for courses with a low budget. A business case is simulated by
providing the students with multi-media information sources that are usually
used by system or business analysts when building models. The sources consist
of recorded interviews with the stakeholders, a web-site of the enterprise under
investigation, internal protocols from management meetings, results of twitter
search on the company name, etc. The paper presents practical guidelines on
how to build such simulation based on a trial successfully completed at the De-
partment of Computer and System Sciences at Stockholm University.
Keywords: eEducation, modeling skills, Information Systems, simulation, mul-
ti-media, apprenticeship
1 Motivation
This paper is devoted to the topic of teaching and learning how to model different
aspects of organizational structure and behavior in order to provide proper IT solu-
tions for businesses and public offices. This topic, which is often referred to as busi-
ness or enterprise modeling, is of special importance for the university programs in
Information System (IS), and other IT-related fields such as Computer Science and
Software Engineering. Its importance for the IT professionals is well understood, and
is discussed in the literature related to university teaching in IT-related fields, see, for
example, [1], [2], [3].
The main problem with acquiring enterprise modeling skills, besides the formal
knowledge on the syntax and semantics of enterprise modeling languages needed, is
that enterprise modeling skills include knowledge on how to extract knowledge from
the unstructured reality of business life. While the first kind of knowledge is quite
2
suitable for teaching in the classroom of a university, the second kind is not, as it
belongs to the area of tacit knowledge [4] or Ways of Thinking and Practicing (WTP)
[5]. The best known solution for acquiring WTP is apprenticeship where the students
follow and help a modeling master in a real business case. However, in a university
classroom setting this is difficult to arrange, if even possible.
Some approximation to a real business case could be achieved by simulating the
enterprise in a computer. This type of simulation of real-life objects is already in use
in some subject of university teaching, e.g. simulation of a patient in medical profes-
sion, where it showed good results [6], [7]. As far as IS and other IT related disci-
plined are concerned, we found only one example of using simulation of an enterprise
for teaching systems analysis, HyperCase, which appeared as early as 1990. Accord-
ing to its designers, HyperCase showed to be more appreciated by the students than
traditional methods [8]. Though HyperCase was introduced in 1990, it is still in use
[9] as an accompanier for an IS course book [10].
The idea used in existing simulation of real objects for teaching is that the object
itself is simulated, e.g. a patient, or an enterprise. With today’s technology, it is cer-
tainly possible to create a virtual reality where a student can work through the enter-
prise, observe what employees are doing, ask them questions, etc. This is the main
idea behind HyperCase. Though technically simulating a real enterprise is possible, it
comes at a considerable cost. While it may be economically justifiable to simulate an
enterprise as an accompanier to a course book that is sold in large volumes, building
such a simulator for a single course would be hardly justifiable.
In the work reported in this paper, we tested another idea of using simulation in
teaching enterprise modeling. Instead of simulating an enterprise, we simulate a situa-
tion of an apprenticeship where the students follow a modeling master and help
him/her to do some part of the work on building models. More specifically, the ana-
lyst chooses the information sources to be used for building a model, and hands the
work of building the model to the students. Such sources may include (but are not
limited to): (a) interviews with stakeholders, e.g. CEO, CIO, (b) samples of relevant
documents, e.g., meetings protocols, forms for managing orders, (c) web-based
sources, e.g. a company web site, results of twitter search on company name. To sim-
ulate such sources, e.g. using video recorded interviews, is much easier than to build a
virtual representation of an enterprise.
The goal of this paper is to present practical recommendations on how to build an
apprenticeship simulator based on the trial completed at the department of Computer
and System Sciences (DSV) at Stockholm University. The simulator has been built
for an introductory course in IS for first year students. One of the main learning activ-
ities in the course is a project assignment that requires the students in groups to build
different types of enterprise models of an imaginary company presented to them. In
earlier occasions of the course, the company was presented in form of a text descrip-
tion. The simulator substituted the text description with a web site that contained mul-
ti-media sources of information as described above and a number of modeling as-
signments left to the students by the master. The site also contained some tips on what
sources to use when completing particular assignments via links made from assign-
3
ments to these sources. The simulator has been used in two course occasions and got
positive feedback from both the students and teachers engaged.
Besides working with the simulator on their own, the students meet a teacher three
times during the project, two of them for consultations, and one for presentation of the
results to the teacher and peers. In the two first meetings, the teacher plays the role of
modeling master, which makes the setting more close to the real apprenticeship.
In the paper, we use our simulator to illustrate the used design principles, and share
our experience obtained when building and using the simulator. We believe that this
experience may be of interest to other teachers in courses that have one of their learn-
ing objectives to acquire enterprise modeling skills.
The rest of the paper is structured in the following manner. Section 2 gives an
overview of the trial course, and the project for which a simulator has been built.
Section 3 describes the way the imaginary company was presented in the project be-
fore the simulator was introduced. Section 4 discusses the new way of presenting the
imaginary companyusing multi-media information sources and our experience of
developing these sources. Section 5 describes the structure of the project site that
integrates all multi-media information sources with the assignments for the students
related to building the models. Section 6 gives some results of the trial evaluation
through surveying the students and the teachers. Section 7 contains our reflections on
using the simulator at two course occasions, and Section 8 contains concluding re-
marks and plans for the future.
2 The Trial Course and the Project Business Case
The trial has been completed in the frame of the first year course called “IT in Organ-
izations” (ITO). Below, we also will refer to this course as the “trial course”. ITO is a
first year course that is mandatory for approximately half of the students who enroll to
bachelor programs at DSV. The number of enrolled students is approximately 250.
The course is given each fall term, and it is the second course for the students. Before
this course they have only completed one other course, “IT for personal use”, which
presents basic theories and concepts in IT and computer science. The trial course ITO
can be classified as an introduction to IS. The overall goal of the course is to give the
students basic knowledge and skills on the analysis, design and development of in-
formation systems, including theories, methods and techniques for this. Additionally,
the course should create an understanding of how organizations - their goals, products
and services, business processes and information - can be changed by using different
forms of IS. The length of the course corresponds to five weeks of full-time work for
the students. The course is 7.5 credits according to the European Credit and Transfer
and accumulation Systems (ECTS). Being an introductory course, the course has a
wide scope. It includes an overview of a number of modeling techniques, such as
functional modeling (with the help of Icam DEFinition for Function Modeling 0, ab-
breviated IDEF0 [11]), process modeling (with Business Process Model and Notation,
abbreviated BPMN [12]), goal modeling (with Business Motivation Model, abbrevi-
ated BMM [13]), and a number of others. The teaching and learning activities in the
4
course include lectures, workshops, tutorials, labs and a project assignment. The pro-
ject assignment is completed in groups of 4-5 students, and it is built around a ficti-
tious company called “AFFE”. The company is said to be engaged in development
and sales of a business game.
3 Presenting the Project Case in a Textual Form
In previous years of the ITO course, the AFFE’s business was presented to the stu-
dents in a textual form. For example, a textual description of how the company han-
dles invoices was available. Based on this textual description, the students needed to
complete a number of tasks. For example, one task was to “Build diagrams of busi-
ness processes currently used in AFFE’s practice using BPMN”. Based on the textual
description, the students were supposed to produce a process diagram. Table 1 gives
examples of text fragment from the textual descriptions. As can be seen, the text con-
tained several types of information to cover the company’s current state of affairs,
their products, working procedures, the IT system they use, and the organizational
changes that the company is planning.
Table 1. Fragments of the text description of AFFE
Type of
information
Example of text fragment
Company
background
information
The company has a game development apartment with 14 game developers
and a sales department with 20 salespersons. The sales department handles
marketing, sales, order processing and delivery.”
Current
state of
affairs
The company needs to borrow money to fund further development of the
game but the banks are unwilling to lend because of the insufficient profita-
bility. Furthermore, the competition is hardening, as many game develop-
ment companies start developing similar business games for the Swedish
market. AFFE's game is still the best-
known business game in Sweden, at
least for manufacturing companies.
Facts on
products
The AFFE game can be played by one user which can take multiple roles in
an enterprise governed in the game. The game can also be played by several
users who then take different roles in the enterprise.
Working
procedures
Today, the development of the next game release is completed in the fol-
lowing way: After the previous version of the game has been released, the
game developers are gathered to determine what requirements must be met in
the next version.
Facts on
systems
used
AFFE currently has the following support systems: a customer record sys-
tem in the form of an excel sheet, an IT-based financial system for account-
ing, a version control system for the game's code, …
5
Type of
information
Example of text fragment
Organiza-
tional
change
Management has discussed various solutions to increase the company's
profitability. They have decided to implement the following five changes:
To offer clients customized variants of the business game.
To offer the game as a cloud service
4 Presenting the Project Case as Multi-media Simulation
When designing a new way of presenting the project assignment to the student, we
used the following guiding design principles:
1. Give the student a more realistic understanding of the work of system and/or busi-
ness analysts. As the authors of this paper had experiences of consulting practice,
we know that textual presentations like described in Section 3 are seldom handed
to an analyst. Thus, using this kind of descriptions may create misconception in the
student’s mind of how an analyst works.
2. Use several types of presentation media. This is in-line with principle 1 above. An
analyst, normally, does not rely on one type of information sources, but combine
several, e.g., interviews with stakeholders, observations, facilitating workshops,
documents analysis, etc.
3. Apply a pragmatic, low cost approach. Our goal was to develop an approach that
could be used in any course that include modeling assignment without requiring a
lot of expensive resources
4. Keep the case already used intact. We tested our approach on the same case as de-
scribed in Section 3, partly to satisfy the principle 3 above, and partly to have a
possibility to make a more adequate evaluation of the impact of the change in case
presentation.
Based on the principles above we have chosen to use the following types of infor-
mation sources and media for presenting the case to the students:
Video recording for simulating interviews with stakeholders. This media could
also be used for simulating a facilitating workshop, or a management group meet-
ing. However, we have not chosen to video recorded these latter types of infor-
mation sources; it would have required much more resources to do (see principle 3
above).
Web-based presentation for simulating the company’s website, or social media
search on the company name. Based on our experience from being analysts, it is
very important to use sources such as the web before the first visit to a company.
For example, web-based presentations such as the company web page and other
public information are important input for an analyst.
Document-based presentation for simulating internal and/or external document
circulating in the company, e.g. protocol from steering committee meeting.
6
In Table 2 below we list all multi-media presentation fragments created for the busi-
ness case outlined in Section 3. The listing is organized according to the media-type
in the first column. The last column shows the approximate costs to produce each
fragment in person-hours. The costs are divided in two sub-columns: Design costs
(D), and Execution costs (E). The design costs are onetime costs that were required to
find out the way to build a component. The execution costs represent the cost of pro-
ducing the component. When developing a new project site, e.g. for another course,
only the execution costs should be taken into consideration. The last row, which is
separated by double line does not represent any presentation fragment, but is related
to creating a site that integrates all fragments and project assignments. It is added to
show the total cost for producing the new way of presenting the project assignment.
Table 2. Multi-media presentation fragments of AFFE
Media type
Presentation fragment
Costs1
D
E
Video recording
Interview 1 with game development manager (simulated)
Interview 2 with sales manager (simulated)
Interview 3 with CEO (simulated)
3
2
2
2
Web-based
sources
Website of the company (AFFE) (simulated)
Twitter feed containing customer opinions (simulated)
Financial info from http://www.proff.se/ (simulated)
IT systems in uselinks to real system vendors websites
1
1
1
0
3
1
1
1
Document-
based sources
Excel sheet 1- sales leads management template (simulated)
Excel sheet 2 - customer management template (simulated)
Protocol of internal management meeting (simulated)
1
1
0
1
1
1
Integration
Course project site
24
8
1All costs are in person-hours
Table 3 shows how various types of information from Table 1 have been incorporated
in the presentation fragments listed in Table 2. For example, we choose to represent
the need for organizational change with a protocol from a management group.
In the paragraphs that follow, we shortly describe each presentation fragment and
the way it has been produced:
1. Interviews 1-3 was created by letting three teachers play the role of CEO, devel-
opment manager and sales manager respectively. A fourth teacher was acting as
the analyst, asking questions to the interviewees. Fig. 1 shows an image from the
resulting interview with the development manager. To save time (principle 3), and
partially to make the interviews more authentic (principle 1), neither the analyst
nor the interviewees had prepared any statements or questions. Instead of preparing
by having a pre-defined script, the interviewees had read the textual description of
the case and studied the intended outcome from the student work with assignments
7
related to the interview. For example, the teacher playing the development manag-
er studied the BPMN process diagram that would be the outcome of the student
work with one of the assignments. The recording of the interview was done in the
university studio, with a professional backdrop and three cameras (left, full scene,
and right). All cutting was done in real-time, so no post-processing was necessary.
The “analyst” started the interview with a short presentation of the aim of the in-
terview as well as involved persons. Since the teacher playing the role of analyst
has extensive experience, he asked the development manager probing questions
such as “How is this performed?”, “If this happens, what do you do then?” etc. A
central part of the interview was also that the analysts made brief summaries of
what was being said, to get confirmations from the interviewee. Introducing these
short summaries, besides being a practical technique applied by experienced ana-
lysts, also had the benefit of allowing the students to take notes without the need to
rewind the video. Each interview lasted between five and seven minutes. The in-
terviews had to be redone 2-3 times until a satisfactory final recording was done.
Each interview took about 40 minutes to record, which in total gives about 6 per-
sons-hours for the three interviews, since three persons where engaged in the in-
terview (interviewee, analyst, technician). Before starting the interview, 3 hours
were spent on design, for example the number and type of interviews was decided.
Table 3. Multi-media fragments usage to provide info needed for completing the project
Presentation fragment
Website
Twitter feed, Financial info, Protocol, Interview 3
Website
Interview 1, Interview 2
IT-systems in use, Excel sheet 1, Excel sheet 2
Meeting protocol
2. Website of the company AFFE was designed using the content management system
WordPress. As any typical company website, the site included pages About us,
Customers, Contact us, Product, etc. Most of the background information from the
textual description was spread through these pages. The WordPress theme Twen-
tyTen with a simple system of horizontal menus has been used for designing the
site. The design took about one hour, while the creation of pages and menus using
WordPress took about 3 hours. We used an existing WordPress installation provid-
ed by the university, and thus had no direct cost for installation.
8
Fig. 1. The Project site interview with the game development manager (left)
3. Twitter feed containing customer opinions was created by getting a list of results
from searching Twitter on an arbitrary search phrase. The search results were first
converted into the PDF format and then edited, so that they represented customers’
opinions and questions on AFFE and its product. Both negative and positive opin-
ions were simulated in the feed, e.g. “the game is good by difficult to learn”, “Oh,
no it went down again”. The PDF format was chosen as it took little time (principle
3) to edit the search results using Acrobat Adobe Pro and thus produce a realistic
(principle 1) simulation. An alternative could be producing a HTML page. In total
the twitter feed took about 2 hours to create.
4. Financial info simulation was created by experimenting with the on-line Swedish
service http://www.proff.se/. This service provides basic financial information
covering the last three years of any Swedish company. By testing the service on
several real companies in the IT sector of the appropriate size, we found a compa-
ny that corresponded roughly to the state in the affairs of AFFE. The information
obtained was then converted to PDF and edited to produce simulated financial in-
formation for AFFE. Again, the PDF format was chosen as it took little time (prin-
ciple 3) to edit the result and thus produce a simulation. In total the financial info
took about 2 hours to create.
5. IT systems in use were not simulated; we have chosen three systems of the right
type on the Swedish market and provided links to the vendors’ sites where these
systems were described: PVCS version management from www.serena.com,
Visma accounting system and wages administration system from
www.vismaspcs.se. This step was straight-forward and required one hour.
9
6. Excel sheet 1, 2 were simulated by searching Internet for Excel based customer
management and downloading available templates, see for example
http://www.vertex42.com/ExcelTemplates/crm-template.html. Then, the templates
where translated to Swedish and filled with simulated data. To find an example of a
sheet took about one hour for each sheet; another hour was spent for translation
and adding data. In total this took 4 hours.
7. Protocol containing meeting notes was simulated by reformatting the text already
existing in the textual description of the project case from the previous year, see
the last row of Table 1. Reformatting consisted of making the text look like a for-
mal protocol; it was done based on the designer’s experience of reading, and,
sometimes, writing such protocols. This activity took less than one hour.
5 The Project Website
The multi-media presentation fragments and the project assignments were integrated
in a project website [14]. The site lists all sources of information via having links to
the fragments listed above, and it lists all assignments, and to each assignment it gives
recommendations on what sources to use when completing the assignments, see, Fig.
2.
Fig. 2. The Project site Description of a task that students need to carry out
Fig. 2 shows an example of an assignment the assignment with the title: “Describe
graphically the as-is business processes of the company in BPMN”. The assignment
page also provides links to sources needed to carry out the assignment.
The project website has been designed under the content management system
WordPress using a theme called “Responsive” that allows several levels of menus,
Links to all
sources
Links to all
assignments
Title for
Assignment D
Assignment
description
Links to sources
needed to carry
out Assignment D
Links to all
assignments
10
both in the top of the pages and on the sides. The menu system was used to connect
presentation fragments to the assignments and arrange the fragments in groups. It took
relatively long time to build the project site, 32 hours. However, most of this time
(about 24 hours) went into designing the site structure. Now, when the structure is in
place, it takes very little time to set up a website for a new project, provided that all
presentation fragments are created, and the instructions for the project assignments
are written.
6 Evaluation
The project website with multi-media presentation of a business case was used in two
course occasions, fall 2013 and fall 2014. To understand how the change in case
presentation impacts teaching/learning of modeling skills, a number of surveys have
been completed. After the occasion of fall 2013 the following survey evaluations has
been completed:
1. A project website evaluation by the students on the issue whether the multi-media
presentation of the company had enough information to carry out the project tasks.
Most of the students who participated agreed on that the information content of the
site was satisfactory.
2. A project website evaluation by the teachers. The teachers agreed that the engage-
ment of the students became higher and the quality of results in the project some-
what increased. They disagreed on other parameters that were related to their work,
e.g. their workload for mentoring the project work did not decrease.
3. An evaluation by the students based on comparison with another course that used
the textual presentation of the project. The majority of the students agreed that the
multi-media presentation of project assignment was to be preferred in all courses
that include business/enterprise modeling.
4. Comparative analysis of examination results for years 2012 (textual presentation)
and 2013 (multi-media presentation). The result showed that the “tail” of the dis-
tribution of lower grades improved for 2013 compared to 2012. This means that the
less performing students showed better results after the project site has been intro-
duced (no change for the better performing students were detected).
After the occasion of fall 2014 a new evaluation has been conducted:
5. A project website evaluation by the students with the focus on comparative inves-
tigation of different media types used in the case presentation.
All first four evaluations pointed towards some improvement in learning environment
and results. Due to the limited space, we do not present the results of the evaluations
1-41, but focus on the latest evaluation, evaluation 5 from the list above, as it is more
relevant to the topic of this paper.
1 The results of evaluations 3,4 are to be published separately in [19]. Note that [19] is com-
plementary to this paper, as it does not contain details on the simulation design.
11
When designing the survey for the latest evaluation, we used a combination of pa-
rameters from different frameworks, e.g. [15,16], as we failed to find a ready-made
framework for our purpose; in addition to the used frameworks we also added a pa-
rameter to measure student engagement.. The design process resulted in selecting the
following parameters:
1. Usefulnessthe degree to which the project site helped to attain the goals set. This
parameter corresponds to Perceived Usefulness (PU) of the TAM framework [15].
2. Completeness the degree to which the information provided is sufficient for
completing the assignments. This parameter corresponds to parameter Complete-
ness from the data model quality framework [16].
3. Effort the amount of time it takes for processing information in order to complete
assignments. This parameter corresponds to Perceived Ease Of Use (PEOU) of the
TAM framework [15]. However in our case measuring this parameter has another
purpose than in TAM. Making it too easy to complete the assignments contradicts
the learning goals of obtaining modeling skills nearing the level of acting practi-
tioner. The time spent should be long enough but not too long so that the students
lost their interest in completing assignments. The latter can be measured through
parameter Engagement, see below.
4. Engagement the degree to which working with the site was deemed interesting.
This parameter interplays with the effort parameter, the idea is that if an assign-
ment is interesting, increasing the student’s motivation, the assignment could also
be difficult and require an effort to be completed.
Based on the four parameters above, we created survey questions to measure them in
relation to the whole project website (e.g. overall usefulness), and to the different
types of media used for the case presentation (e.g. comparative usefulness). Out of the
210 students working with project assignments 49 responded to the survey. The re-
sponse rate (23%) is to be considered normal for this type of survey in our depart-
ment; the regular course evaluation surveys at DSV usually get about a 15-20% re-
sponse rate. The results are presented in Table 4.
The survey was designed to answer two questions:
1. Find out to what degree the project website was suitable as educational material
2. Find out the ways for improvement
As far as suitability is concerned, the answers show the following:
Usefulness. The first two questions (Q1, Q2) about overall usefulness show that the
students to a very large degree consider that the site helped them learn the subject
matter, and understand how an analyst works. Note that in the second question
(Q2) regarding the site’s ability to aid the student understand the work analysts do,
got a high degree (33%) of students who were undecided. This is quite natural
since many students do not have work experience and thus cannot compare it with
previous experiences. Several comments related to Q1 pointed out that what helped
the student to learn to use models was the site in combination with lectures.
Completeness. Based on Q5, the site was clearly deemed as containing enough
information (86%).
12
Table 4. Results of Evaluation 5
Q1 - Overall usefulness (q3)
The site helped me to learn to analyze existing
organizations and their IT support with the aid
of different kinds of models.
Q2 - Overall usefulness (q4)
The site helped me understand the work a
business/systems analyst performs.
Q3 - Overall usefulness (q5)
If the project site is to be re-designed, what
should the purpose of the re-design be?
Q4 - Comparative usefulness (q8)
If we change the project site, which infor-
mation sources would you like to have
more of?
Q5 - Overall completeness (q1)
Did the project web site contain enough infor-
mation to complete the project tasks?
Q6 - Overall effort (q2)
In general, was it easy to find information
needed to complete the project tasks?
Strongly
Agree
21%
Agree
67%
Undecided
8%
Disagree
4%
Strongly Disagree
0%
Strongly
Agree
14%
Agree
43%
Undecided
33%
Disagree
10%
Strongly Disagree
0%
Easier
14%
More
realistic
64%
More
challenging
12%
More challenging,
realistic and easy
7%
Other
3%
More video
interviews
29%
More web-
based sources
51%
More internal
documents
20%
Not enough
0%
Barely enough
10%
Enough
86%
Little too much
2%
Much too much
2%
Very easy
14%
Easy
47%
Neither easy
nor difficult
37%
Difficult
0%
Very difficult
2%
13
Q7 - Comparative effort (q6)
Which information source required the MOST
work to find information needed to complete the
project tasks?
Q8 - Comparative engagement (q7)
Which information source was most inter-
esting to work with?
Effort. Question Q6 and Q7 shows that 61% of the students thought it was easy or
very easy to find the information on the site. As described before, it is not neces-
sarily good that the information is too easy to find, since that would not reflect a
realistic working situation. Thus, the site could probably be changed to actually in-
crease the effort needed from the students. This is also reflected in a comment
made in the survey, where a student pointed out that it was actually too easy to find
relevant information in the video interviews. On the other hand when students
needed to choose the information source that comparatively required the most
work (Q7), the video interviews was pointed out by 30% of the students as requir-
ing the most work.
Engagement. From the responses on the question about which information source
that was most interesting (Q8), 59 % of the students found the video interviews
most interesting and 33 % found all sources equally interesting.
As far as ways for improvement are concerned, the answers show the following:
From the answers of Q3 and Q4 it can be said that a future re-design targeting use-
fulness would be to make the site more realistic (64%) with a focus on the “AFFE”
company’s simulated web site (51%).
The video interviews was considered the most interesting information source (Q8),
at the same time they were considered the most challenging (Q7) - and 29% stated
that they would like to have more video interviews in the future (Q4). This combi-
nation of challenging and engaging is something to tap into for future revisions of
the site.
The survey gave us a clear indication of that the site design was successful. During
the course the teachers also had the same subjective impression, as will be discussed
in the next section.
7 Reflections
As stated in the beginning of the paper we set out to change a IS modeling course to
make it more realistic, but we also aimed at doing it by using a low-cost pragmatic
approach. Regarding the cost, we spent in total 32 person-hours on re-designing the
Video
interviews
30%
Web-based
sources
11%
Internal
documents
23%
All required equal
amount of work
36%
Video
interviews
59%
Web-based
sources
6%
Internal
documents
2%
All was equally
interesting
33%
14
course and 23 hours on executing/implementing the design. Thus, in total 55 hours
were spent on remaking the course from using a text-only representation of the project
assignment into a simulation using video, reports and a company website. Even
though 55 hours is over a week of full-time work, it can still be said to be low-cost
compared to the courses overall budget.
When doing the re-design of the course, and when teaching the course we also
made several subjective observations about the effects of the re-designed course:
The first observation is that the students like the use of several media types, and are
subjectively deemed as more engaged in the course. The survey showed that the stu-
dent thought the usefulness of the project web site was high (see previous section on
evaluation). This was also seen when observing the students present and discuss their
work. For example, one teacher observed that several groups strongly objected when
one group presented the idea that the fictitious “AFFE” company should buy an ex-
pensive ERP system. They simply felt that this would ruin the company’s fragile
economy without having any positive effect. This was one of several signs of an in-
creased engagement in the project assignment.
An observation from the design work was that a part of adding a realistic touch to
the information sources was to add layers of information. For example, the simulated
protocol from the internal meeting was “decorated” with a formal introduction and
signatures. Likewise, in the video interview with the development manager the man-
ager expressed positive opinions on some parts of the development process. In both
these examples, the added information was not used by the students in the assignment,
but was added to give a feeling of authenticity.
It was furthermore observed that some unintended information was conveyed in the
video recordings. While the main subject of the video interviews was the description
of a business process or the company’s current economic state, the students also in-
terpreted how the interviewees behaved. For example, the sales manager was consid-
ered as being somewhat formal and “square” and some groups of students speculated
that this could be the culprit of the issues the fictitious company had. The knowledge
of different leadership styles and analysis thereof is not part of the current course, but
observations like this may be of interest for future courses.
Another, positive, unintended effect that was observed during the survey was that
one student had problems reading long text because of dyslexia. For this student, us-
ing mixed information sources made it easier to grasp the course content. This is one
example of how mixing media types may be a way to appeal to a wider range of stu-
dents.
8 Conclusion
The evaluations of the trial has indicated that pedagogical gains can be made in teach-
ing modeling skills with relative simple and cost effective use of video technology
paired with a straight-forward web site. The key to creating the web site has been to
have a simulated apprenticeship approach, where the student examines an enterprise
based on simulated material in form of information fragments, such as economic in-
15
formation and protocols. Furthermore the students, as “apprentices”, were able to see
a “master” analyst performing interviews.
Our approach to using simulation in teaching/learning modeling skills is via simu-
lating apprenticeship. As such, it is complementary to other usages of simulation for
the same purpose, e.g. simulation of an object to be modeled, as in case of HyperCase
[9]. The latter approach can require considerable costs and can be justified only if the
volume of usage of such a simulator is high, which is the case with HeperCase dis-
tributed as a book companion. Our approach based on simulation of apprenticeship is
a low-cost approach and can be used in, more or less, any course.
The research presented in this paper is being conducted as Action Research (AR),
i.e. in cycles. There are several ways of representing an AR cycle with the number of
phases varying from 4 to 6, dependent on how detailed one wants to present the cycle.
We use the version suggested by Kolb for experimental learning style theory [17],
which can be represented as four stage learning cycle: Concrete experience Reflec-
tive Observation Abstract Conceptualization Active Experimentation Con-
crete Experience …. We consider that Kolb’s cycle is a good representation not
only for student learning, but also for AR in a situation when researchers are also
practitioners in the field where the given research project is conducted. In this case,
learning is equal to obtaining new knowledge not only for the researchers involved,
but also for the community of researchers and practitioners in the discipline.
We are currently on the second cycle, having applied the new course design for the
second time, and now doing “Reflective Observation”. In the next phase of our re-
search (Abstract Conceptualization) we will investigate means to make the created
information fragments reusable, so that several courses can share the same material. A
start would be to see if the information fragments developed in the studied ITO course
can be labeled in such a manner as to make them useful in another IS course where
modeling skill are also developed. For this labeling different ontologies/taxonomies
will be tested, including Bloom taxonomy [18], to map the formal intended learning
objectives of the IS courses in the department to see which courses that may use the
created information fragments. This would give economies of scale when reusing
video and textual information fragments. Furthermore, video annotation could be used
to make the video interviews more reusable by creating searchable libraries.
Acknowledgments. The project was sponsored by Stockholm University’s program
“Future Learn”. The authors are grateful to our colleagues Jelena Zdravkovic and
Anders Thelemyr who participated in interviews recording, and to the anonymous
reviewers whose comments helped to improve the text. The first author is also grate-
ful to Anthony Burden who encouraged him in writing texts related to teach-
ing/learning.
References
1.
Bezivin, J., France, R., Gogolla, M., Haugen, O., Taentzer, G., D., V.: Teaching Modeling:
Why, When, What? In Ghosh, S., ed. : MODELS 2009 Workshops, LNCS 6002, pp.55
62
(2010)
16
2.
Engels, G., Hausmann, J. H., Lohmann, M., Sauer, S.: Teaching UML is Teaching
Software Engineering is Teaching Abstraction. In Bruel, J. M., ed. : MoDELS 2005
Workshops, LNCS 3844, pp.306319 (2006)
3.
Fenstermacher, K. D.: If I had a Model, I’d Model in the Morning. In : Proceeding of
OOPSLA'04, pp.88-89 (2004)
4.
Polanyi, M. S.: Knowing and Being. University of Chicago, Chicago (1969)
5.
McCune, V., Hounsell, D.: The development of students' ways of thinking and practising in
three final-year biology courses. Higher Education 49(3), 255-289 (2005)
6.
Bergin, R., Youngblood, Y., Ayers, M., Boberg, J., Bolander, K., Courteille, O., Dev, P.,
Hindbeck, H., Stringer, J., Thalme, A., Fors, U.: Interactive simulated patient: experiences
with collaborative e-learning in medicine. J Educ. Comput. Res 29(3), 387-400 (2003)
7.
Bergin, R. A., Fors, U.: Interactive simulated patientan advanced tool for student
activated learning in medicine and healthcare. Computers & Education 40, 61376. (2003)
8.
Kendall, J., Kendal, K. E., Baskerville, R., Barnes, R. J.: An Empirical Comparison of a
Hypertext
-
Based Systems Analysis Case with Conventional Cases and Role Playing.
DATA BASE ADVANCES, 27(1), 58 77 (1996)
9.
Kendall, J., Kendall, K. E., Schmidt, A., Baskerville, R., Barnes, R. J.: HyperCase: A
Hypertext
-Based
Case for Training Systems Analysts. (Accessed 2013) Available at:
http://www.pearsonhighered.com/hypercase/hypercase2.9
10.
Kendall, J., Kendall, E.: Systems Analysis and Design 9th edn. Pearson (2013)
11.
NIST: Integration definition for function modeling (IDEF0), Draft Federal Information
Processing Standards, Publication 183, 1993. (Accessed February 2015) Available at:
www.idef.com/downloads/pdf/idef0.pdf
12.
OMG: Business Process Model and Notation (BPMN), Version 2.0.2, Object Management
Group (OMG), Document formal/2013
-12-
09, December 2013. (Accessed February 2015)
Available at: http://www.omg.org/spec/BPMN/2.0.2/PDF
13.
OMG: Business Motivation Model, Version 1.2, Object Management Group (OMG),
Document formal/2014
-05-
01, May 2014. (Accessed February 2015) Available at:
http://www.omg.org/spec/BMM/1.2/PDF
14.
Bider, I., Perjons, E., Henkel, M., Kowalski, S.: Projektplatsen för ITO 2013. (Accessed
2013) Available at: http://ito2013.blogs.dsv.su.se/
15.
Davis, F.: Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS quarterly 13(3), 319-340 (1989)
16.
Moody, D.: Improving the quality of data models: empirical validation of a quality
management framework. Information systems 28(6), 619-650 (2003)
17.
Kolb, D.: Experiential learning: Experience as the source of learning and development 1.
Prentice-Hall., Englewood Cliffs, NJ, US (1984 )
18.
Bloom, B.: Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain.
David McKay Co Inc., New York (1956)
19.
Bider, I., Henkel, M., Kowalski, S., Perjons, E.: Technology Enhanced Learning of
Modeling Skills in the Field. In : To apear in proceedings of 8th IADIS International
Conference on Information systems, Funchal (2015)
... To solve the problem, we decided to simulate a situation of apprenticeship using multimedia for presenting a realistic business case to the students as the basis for creating enterprise models [12]. The idea was implemented as a project web site that included multi-media presentation of the business case and a list of modeling tasks [13]. The site was tested at several occasions of the same course and was evaluated by students and teachers afterwards. ...
... The site was tested in a first year course that approximately corresponds to "Introduction to IS". The site was used in three occasions of this course (in 2013, 2014 and 2015), and it was evaluated with the help of surveys and interviews by both students and teachers [12], [13]. The evaluation showed that the students appreciated the site and even suggested some improvements. ...
... This, however, requires considerable resources to convert all current case descriptions into multi-media fragments and put them in the developed structure of the website. Though building a site for one case does not require much resources [13], the sum for all cases for all courses that include teaching/learning modeling skills may be quite high. To solve the resource problem, we are currently investigating possibility of reusing the multi-media case presentations between the courses. ...
Article
Full-text available
The article is devoted to developing a methodological support for planning Design Science (DS) research projects. More exactly, it is aimed at developing a classification of a particular kind of cycles, inherent to DS research projects, and guidelines on how to choose the next cycle in a specific project. The classification and guidelines are based on results from an analysis of DS literature and a reflective analysis of our own research practice. As far as own research practice is concerned, two DS projects have been analyzed in detail. The analysis revealed that though both projects included cycles, the nature of these cycles was different. In the first project, cycles concerned finding a better problem to solve, while in the second project, cycles concerned finding a better solution for more or less the same problem. Both projects concern developing methodologies in the area of socio-technical systems, and the guidelines on how to choose the next cycle have special provisions related to such systems. In conclusion, the article presents examples of other projects that followed the suggested guidelines and discusses the difference of the suggested approach from other approaches to conduct DS research projects.
... When graduating, most of the students lack the most important part of the modelling skillshow to capture the reality to build a model, through making field observations, interviewing people, and analysing diverse documents. A solution to this problem that the authors have developed is the use of Apprenticeship Simulation (Bider Henkel, Kowalski & Perjons, 2015b;Bider, Henkel, Kowalski & Perjons, 2015a). ...
... • More realistic. The students perceive that they get a more realistic understanding of the work that systems analysts do (Bider, Henkel, Kowalski, & Perjons, 2015b). • Preferred by the students. ...
... • Preferred by the students. Apprenticeship simulation is preferred by the students, compared to traditional case descriptions (Bider et al., 2015b). • Are reusable. ...
Article
Full-text available
Teaching enterprise modelling is a complex task, as it requires both teaching modelling syntax as well as how to select and extract information from several sources, such as IT systems, employees, and existing documents. To aid teaching enterprise modelling Apprenticeship Simulation (AS) may be used. AS is a form of case-based learning where the students examine a case by following a virtual expert who gives them information sources with which they should work. A case as presented to the students can be quite complex, consisting of numerous types of diverse sources, including recorded interviews and links to real-world dynamic sources such as web pages. In this paper, we present a set of principles for the design of AS case presentations. The principles are based on our experiences in applying AS in four courses in the area of enterprise modelling.
... AS is a tool for students to acquire enterprise modelling skills, which also helps the students to obtain a holistic view on the IS field. AS has been used in four different courses at the Department of Computer and Systems Sciences (DSV) of Stockholm University, see for example [8,9,10]. ...
... The ACMDI matrix for AS is represented in Table III and IV. Table III includes the first 5 context elements (columns) from Table I, while Table IV Table III and IV are based on evaluations of AS carried out and presented in a number of research papers [8,9,10]. Table III and IV show that ACMDI could be used to explicate how added digital features change the use and development of an artefact, including effects (positive and negative) they produce. ...
Conference Paper
Full-text available
This is draft. We are working on a journal version, which will be more readable. Many artefacts of today are non-digital or only partly digital. Digital features can often be added to these artefacts to make them more digital. However, for designers, it can be challenging to figure out what digital features to add. In this paper, we outline an artefact centric approach that addresses the challenge called Artefact Centric Model for Digital Innovation (ACMDI). For the time being, ACMDI works only for information artefacts. In the paper, ACMDI is demonstrated on an example of an educational tool called Apprenticeship Simulation (AS). AS is a tool facilitating students acquiring modelling skills; AS was developed by digitalizing an ordinary text description of a business case.
... The survey was performed on a large bachelor level course, where 49 out of the 210 students completed the survey. More results from the survey are reported on in [17]. ...
... Having the same case for several courses was intentional, as we wanted to achieve a tacit-based connection between the subject matters taught in these courses based on the common case. The usage of AS and CBILE was evaluated through surveys and the results were positive, see [1,2,17]. ...
Chapter
Full-text available
Apprenticeship Simulation (AS) is a form of case-based learning where the students follow a virtual expert who selects and hands to them information sources with which they should work. A case as presented to the students can be quite complex, consisting of numerous types of diverse sources, including recorded interviews and links to real-world dynamic sources such as web pages. A teacher who designs an AS case needs to decide on the structure of the case presentation, including which sources should be used and how they and associated tasks should be presented to the students. In this paper, we present a set of principles for the design of AS case presentations. The principles are based on our experiences in applying AS in four courses in the area of enterprise modelling.
... A usual way of presenting a business case to the students is giving them a text description of the case. For testing our idea, however, we use a different way of case presentation, the one that comes from our previous projects (Bider et al., 2015a;Bider et al., 2015b), which is called apprenticeship simulation. The apprenticeship simulation is based on using multi-media capability to present a case to the students, such as recorded interviews with the stakeholders, web-sites of the companies included in the business case, examples of internal documents, email correspondence, etc. ...
... However, such simulations may be complex to create, especially, if the case should cover a multitude of perspectives needed for IS analysis and design. As an alternative, apprenticeship simulation (Bider et al., 2015a;Bider et al., 2015b) has been developed. Rather than simulating a complete business, apprenticeship simulation aims at simulating the situation of an apprentice being guided by a master analyst. ...
Conference Paper
Full-text available
The paper considers the problem of students acquiring a holistic view on the IS discipline via a set of not explicitly connected subjects taught in disparate courses. The main idea is based on M. Polanyi's works on a structure of tacit knowing that can produce "a stereoscopic image from two separate pictures". The images that are used for creating a stereoscopic picture give different perspectives on the same reality, but they do not explicitly refer to each other, the 3-d picture is being created unconsciously by the human mind. This paper demonstrates that a connection between subjects can be created by using the same or tightly connected business cases in different courses that use case based learning combined with computer-based apprenticeship simulation. The paper discusses the main idea, the trial settings, and preliminary results.
... Having the same case for several courses was intentional, as we wanted to achieve a tacit-based connection between the subject matters taught in these courses based on the common case. The idea and results achieved are discussed in [3]. The usage of AS and CBILE was evaluated through surveys and the results were positive, see [1,2,3]. ...
... The idea and results achieved are discussed in [3]. The usage of AS and CBILE was evaluated through surveys and the results were positive, see [1,2,3]. ...
Conference Paper
Full-text available
CBILE is an immersive environment that supports Apprenticeship Simulation (AS) – a method for teaching/learning modeling skills in university education. AS is aimed at the students acquiring not only explicitly defined syntax and semantics of modeling languages, but also skills of analyzing various types of sources for obtaining information to build a model. AS is a kind of case-based learning that presents a case to the students using multimedia sources, such as recorded interviews and web-based sources. CBILE integrates all sources, simulated as well as real, to be used in the case in one place and connects them to the project tasks. The environment uses WordPress as an underlying platform for both integration and simulation of web sources.
... Even papers on teaching Enterprise Modeling are scarce, with some notable but isolated works, e.g. [9]. One direct consequence of this state of affairs is that students graduate bachelor programs with the perception that Conceptual Modeling is a chapter/technique of Software Engineering and has no relevance outside the goals associates with that discipline; Business Process Management is gaining some attraction in this respect (less than a quarter of the surveyed education-focused papers according to [7]), however it's still not sufficient to establish a general notion of "model value" that can inspire junior research work taking on Conceptual Modeling as a standalone discipline that uses or creates conceptualizations for any domains. ...
Article
Full-text available
This article frames Conceptual Modeling education as a design problem, in the sense of the Design Science research framework, motivated by student preconceptions and oversimplifications causing a gap between how the discipline is perceived at the bachelor level and the holistic understanding of model value that is required for research work. The treatment to this design problem must comprise teaching approaches and artifacts capable of positioning Conceptual Modeling as a standalone discipline having a value proposition for any application domain, rather than a technique subordinated to other disciplines. The underpinning thesis is that modeling languages should be primarily understood as purposeful knowledge schemas that can be subjected to agile adaptations in support of model-driven systems or knowledge processes, by analogy to how a database schema is evolved in response to changing requirements of a data-driven system or data analytics needs. This thesis is supported by enablers provided by the Open Models Laboratory and the Agile Modeling Method Engineering framework – resources that support the development of treatments to the design problem framed by the article.
... We also believe that it can be used in other areas of academic and professional education. To promote using computers for simulating apprenticeship, we have published technical details of the project (Bider et al., 2015), and are ready to share with others the project site design implemented with the help of WordPress. ...
Article
Full-text available
Purpose – This paper aims to report on a project aimed at using simulation for improving the quality of teaching and learning modeling skills. More specifically, the project goal was to facilitate the students to acquire skills of building models of organizational structure and behavior through analysis of internal and external documents, and interviews with employees and management. An important skill that practitioners in the information systems field need to possess is the skill of modeling information systems. The main problem with acquiring modeling skills is to learn how to extract knowledge from the unstructured reality of business life. Design/methodology/approach – To achieve the goal, a solution was introduced in the form of a computerized environment utilizing multimedia to simulate a case of an apprenticeship situation. The paper gives an overview of the problem that the solution addresses, presents the solution and reports on the trial completed in a first-year undergraduate course at Stockholm University. Findings – The results of the trial indicate that using rich multimedia along with a case-based learning approach did improve the overall performance of the students. It was also shown that both students’ and the teachers’ attitudes toward the solution were positive. Originality/value – The solution presented in this paper, using computer simulation in teaching/learning by focusing on an apprenticeship situation, can be reused by other university teachers, especially in the Information Systems discipline. This solution can thus be used in teaching, system design, requirements engineering, business analysis and other courses typical for information systems.
Chapter
This paper proposes a teaching method and artifact for Conceptual Modeling education, motivated by a challenge in the authors’ university of bridging the gap between bachelor-level studies and research work on topics related to Conceptual Modeling. At bachelor-level, Conceptual Modeling is subordinated to Software Engineering or Business Process Management topics, making extensive use of available standards for graphical documentation purposes. However, at doctoral level and in project-based work, modeling methods must be scientifically framed within wider-scoped paradigms – e.g. Knowledge Management, Enterprise Modeling – or tailored for domain-specific scenarios. The teaching artifact presented in this paper is an example of an “agile modeling method” that can be iteratively evolved together with students through a metamodeling approach in support of a course flow that argues for a generalized model value proposition and modeling languages acting as “schema” that can be tailored and migrated to accommodate explicit requirements from any application domain.
Conference Paper
Case-based teaching/learning is widely used in Information Systems (IS) education in general, and in teaching/learning modeling, in particular. A case presents to the students a real or imaginary business situation asking them to build a model of it, or showing how such model can be built. In situations where a business case is presented in a text form, reusing it as is, or with modification in the same or a different course does not constitute much of a problem. However, using textual description for presenting cases has drawbacks on its own, as it does not help the students to acquire the skills of dissecting analyzing and analyzing the reality when building a model. The latter can be better achieved when a case is presented in a multimedia form, e.g. recorded interviews, website of a company, etc. As the previous works of the same authors show, such case presentations gives the students better understanding of the essence of modeling, which is appreciated by the students. The dark side of the multimedia presentation is that such a case presentation requires more time to build compared to using a textual form, and it is not easy to change it. This paper is a preliminary inquiry into the problem of reusing of cases presented with the help of multimedia. It presents a conceptual model of the domain aimed at discussing the potential of reuse of the whole case or its parts, and concludes with considerations on reusability that need to be covered when building multimedia presentations of cases.
Article
Full-text available
Findings are presented from an ongoing study of three final-year, honours-level course units in the biosciences with a combined intake of 85 students. The data on which the analysis draws comprise semi-structured interviews with students together with findings from an Experiences of Teaching and Learning Questionnaire. The investigation forms part of a wider project concerned with the enhancement of teaching–learning environments in undergraduate courses in contrasting subject areas. The findings presented in the paper focus on two interrelated aspects of the students experiences within the teaching–learning environments represented by the three course settings. First, salient aspects of high-quality undergraduate-level learning are identified in the form of the students evolving grasp of characteristic ways of thinking and practising in the subject. These ways of thinking and practising in the biosciences were evident in the students engagement with the research literature and with experimental data, and in their efforts to master the requirements and conventions of the subject for written and oral discourse. The second part of the analyses focuses on the teaching–learning environments represented by the three course settings. It was found that, despite taking markedly different forms in the three settings, the teaching–learning and assessment strategies pursued appeared to be broadly congruent with the promotion of ways of thinking and practising in the subject. The quality of feedback to students also seemed pivotal, but needed to be understood in terms of the interplay of various factors, including opportunities for intrinsic as well as extrinsic feedback. Finally, the later years of undergraduate studies had called for a significant process of adjustment on the students part to a step-change in study demands.
Conference Paper
This paper reports on a project aimed at using simulation for improving the quality of teaching and learning in the field of Information Systems. More specifically, the project goal was facilitating the students to acquire skills of building models of organizational structure and behavior through analysis of internal and external documents, and interviewing employees and management. The solution tested in the project was a computerized environment utilizing multi-media to simulate a business case. The paper gives an overview of the problem that the solution addresses, presents the solution, and reports on a trial completed in a first year undergraduate course at Stockholm University. The results of the trial indicate that using rich multi-media along with a case based learning approach did improve the overall performance of the students. It was also shown that both students' and the teachers' attitude toward the solution was positive.
Article
Interactive Simulated Patient (ISP) is a computer-based simulation tool designed to provide medical students with the opportunity to practice their clinical problem solving skills. The ISP system allows students to perform most clinical decision-making procedures in a simulated environment, including history taking in natural language, many hundreds of laboratory tests (e.g., images and endoscopy), and physical examination procedures. The system has been evaluated in a number of courses at three universities, Karolinska Institutet and Uppsala University in Sweden, and Stanford University in the United States. This article describes a study conducted in 2002, with an emphasis on results that pertain to collaboration between students. Results indicate that ISP is engaging and stimulates more active student involvement than traditional paper-based case presentation methods and that students seem to collaborate more easily when using ISP compared to traditional paper-based methods.
Article
This paper describes the results of a 5-year research programme into evaluating and improving the quality of data models. The theoretical base for this work was a data model quality management framework proposed by Moody and Shanks (In: P. Loucopolous (Ed.), Proceedings of the 13th International Conference on the Entity Relationship Approach, Manchester, England, December 14–17, 1994). A combination of field and laboratory research methods (action research, laboratory experiments and systems development) was used to empirically validate the framework. This paper describes how the framework was used to: (a) quality assure a data model in a large application development project (product quality); (b) reengineer application development processes to build quality into the data analysis process (process quality); (c) investigate differences between data models produced by experts and novices; (d) provide automated support for the evaluation process (the Data Model Quality Advisor). The results of the research have been used to refine and extend the framework, to the point that it is now a stable and mature approach.