Content uploaded by Ilia Bider
Author content
All content in this area was uploaded by Ilia Bider on Sep 27, 2017
Content may be subject to copyright.
Simulating Apprenticeship Using Multimedia in Higher
Education: a Case from the Information Systems Field
Ilia Bider, Martin Henkel, Stewart Kowalski, Erik Perjons
Stockholm University, Department of Computer and Systems Sciences, Postbox 7003, 164 07 Kista, Sweden
{ilia|martinh|stewart|perjons}.@dsv.su.se
ABSTRACT
Purpose
An important skill that practitioners in the information systems field need to possess is the skill of
modeling information systems. The main problem with acquiring modeling skills is to learn how to
extract knowledge from the unstructured reality of business life. This paper reports on a project aimed at
using simulation for improving the quality of teaching and learning modeling skills. More specifically,
the project goal was to facilitate the students to acquire skills of building models of organizational
structure and behavior through analysis of internal and external documents, and interviews with
employees and management.
Design/methodology/approach
In order to achieve the goal, a solution was introduced in the form of a computerized environment
utilizing multi-media to simulate a case of an apprenticeship situation. The paper gives an overview of the
problem that the solution addresses, presents the solution, and reports on the trial completed in a first year
undergraduate course at Stockholm University.
Findings
The results of the trial indicate that using rich multi-media along with a case based learning approach did
improve the overall performance of the students. It was also shown that both students’ and the teachers’
attitude toward the solution was positive.
Originality/value
The solution presented in this paper, using computer simulation in teaching/learning by focusing on an
apprenticeship situation, can be reused by other university teachers, especially in the IS discipline. This
solution can thus be used in teaching, system design, requirements engineering, business analysis and
other courses typical for information systems.
KEYWORDS
Case based learning, modeling skills, information systems, IS, simulation, education, multi-media
1. Introduction
Educational theory and practice identifies several types of so-called “difficult knowledge” (Perkins, 2007),
acquiring of which is critical for the students to become professionals. Among them, there is one particular
type that is difficult to acquire in a class rum settings, namely, tacit knowledge (Polanyi, 1969), or Ways of
Thinking and Practicing (WTP) (McCune & Hounsell, 2005). This knowledge can be acquired not by
learning theories and methods, but via socialization (Nonaka, 1994) when students do something practical
together with experts in the field in question. However, it is practically impossible to arrange apprenticeship
in real practice for all students for each course that requires acquiring tacit knowledge or WTP. The question
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
1
arises whether it is possible to use a modern approach, where computers are used for creating an artificial
environment that could give a reasonably good approximation of real apprenticeship? This paper report on a
trial project completed in the field of teaching in the Information Systems (IS) discipline that gives a positive
answer on this question. The trial has been completed at the department of Computer and Systems Sciences
(DSV) at Stockholm University.
One of the important skills that practitioners in the IS field need to possess is the skill of modeling
information systems or their parts. For example, practitioners draws graphical models to represent the
information structures that the system contains. The importance of modeling skills for IS professionals is well
understood, and many courses in IS devote all or part of their time to teaching and learning modeling skills.
This is true for IS education in general, and for DSV in particular. The main problem in teaching and learning
modeling in a university environment of today is that it mainly focuses on acquiring a formal part of
modeling skills. The students learn the syntax and, partly, semantics of the formal languages used for
modeling, as well as some techniques of translating a model from one modeling language into another. When
graduating, most of the students lack the most important part of the modeling skills – how to capture the
reality to build a model, through making field observations, interviewing people, and analyzing diverse
documents.
The difficulties of acquiring the modeling skills described above in university settings comes as no
surprise, as these skills belong to the area of tacit knowledge, or WTP, and thus require some kind of
apprenticeship. Another factor that increases the level of difficulty of learning modeling skills is that many
students do not have prior work experience. They do not have their own tacit knowledge about organizations
that could facilitate their learning of how to model a real-life organization. What makes the problem more
grave in IS education is that not all teachers have real experience of modeling in practice and thus do not
possess the tacit knowledge to transfer via socialization.
The goal of the project reported in this paper has been to propose and test a computerized environment
that would give a possibility to acquire a higher level of modeling skills than it is possible to achieve in the
standard university settings. The environment simulates a situation of apprenticeship when an apprentice
follows the master, observes what he/she is doing, and then is handed some work to do. In our case, the
“master” performs interviewing, makes observations and selects diverse sources of information, and the
students are creating models based on the inner structure of organization “revealed” in the interviews and
other information sources. The simulation is done by utilizing multi-media presentation of a business case.
The presentation consists of recorded interviews with key stakeholders, web-site of organization under
investigation, its internal documents, etc. The project included a trial which consisted of building a multi-
media simulator for one course, running this course, and evaluating the results.
The course chosen for the trial is an introductory course in IS for the first year students, one of the main
learning activities in the course being a project assignment. The project requires that the students, in groups,
create a number of different models of a company whose business situation is described in a textual form.
During the trial, the textual description has been substituted by a multi-media simulation implemented as a
project site. Initial assessment of the trial has shown that the students appreciated the new way of presenting
a business case, and achieved somewhat better results in the exams than during the previous years.
The rest of the paper is structured in the following manner. Section 2 gives a literary overview of the state
of the art in teaching and learning modeling skills, and using multi-media for teaching and learning. Section 3
focuses on the state of affairs at DSV prior our research project. The next two sections report on the trial
completed in the frame of an actual course. The trial is described in some details in Section 4, while Section 5
is devoted to presenting the evaluation of the trial. Section 6 summarizes the results achieved and draws plans
for the future.
As far as the research approach is concerned, the project has been completed according to the action
research guidelines. More exactly, we followed the Kolb’s experimental learning style theory (Kolb & Fry,
1975), which can be represented as repeated execution of the four stage learning cycle: Concrete experience
Reflective observation Abstract conceptualization Active experimentation Concrete experience
… . All authors teach modeling of IS at DSV, and had experience of this activity before starting the
project. Based on our observations, we found the problem addressed in this paper in our teaching activities,
and conceptualized it in the need of introducing simulation as a means for enhancing teaching/learning
modeling skills. The idea was realized in a real-life setting, i.e. a course at DSV, and the paper presents the
experiences of teaching modeling using this new environment. A more detailed discussion on the approach to
research used in the project is discussed in a special subsection of Section 6.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
2
2. Literature overview
The relevant literature for this work has been obtained via three ways:
1. Search on Google Scholar for “teaching modeling skills” in the areas of computer science, software
engineering and IS.
2. Sending request to the community of practice in the IS discipline (about 1 000 academics strong)
regarding “teaching modeling skills”.
3. Search on papers related to using patient simulation in teaching the medical profession. This area, though
not directly related to IS discipline, was known to us through personal contacts.
The search resulted in more than ten works representative for the topic under investigation in this paper. As
far as the literature in the IS discipline is concerned, the following picture emerged:
The importance of acquiring modeling skills is well understood in the IS and related disciplines, such as
Computer Science and Software Engineering, see, for example (Bezivin et al., 2010; Engels et al., 2006;
Fenstermacher, 2004).
The differences between the novice modelers and real experts who tacitly know WTP (McCune &
Hounsell, 2005) are also well understood and some methods of transferring this knowledge have been
suggested and tested (Venable, 1996). Some work on using computerized tools for helping the students to
acquire tacit knowledge has been tested, e.g. in teaching programming (Lane & VanLehn, 2005).
Several ways of using case based learning is applied in teaching situations in IS. Teaching modeling
skills in IS often uses case-based techniques where a student builds a model of a real or imaginary
organization. Traditional methods of presenting this organization to the students are as follows:
1. Textual description of the organization (Fenstermacher, 2004), which is often used at DSV.
2. Role play when one group of students plays roles of managers/employees of an organization, which
the other group is then trying to model (Costain & McKenna, 2011). Some courses at DSV use this
technique as well.
3. Modeling an on-line or real business which the students can easily access, e.g. Amazon, PayPal, etc.
(Recker & Rosemann, 2009). Some courses at DSV use this technique.
4. Modeling a real organization of which the student is part (e.g. as an employee). This is used in
vocational programs (MA, MBA) with students having working life experience (Truex et al., 2010),
but is not used at DSV.
The teaching profession of IS has understood the capability of rich-media provided by Internet relatively
early. The first organizational simulator, HyperCase, for teaching system analysis appeared as early as
1990. According to its designers, HyperCase showed to be more appreciated by the students than the
traditional methods described above (Kendall et al., 1996). Though HyperCase was introduced in 1990, it
is still in use (Kendall et al., 2013) as an accompanier for an IS course book (Kendall & Kendall, 2013).
Though the possibility of using rich media has been discussed in IS education literature (Guy et al.,
2000), no other environment than HyperCase has been reported in the literature.
Investigating the literature on using simulation in teaching other disciplines, revealed that the medical
profession uses teaching supported by a simulator – e.g. a virtual patient - in much greater scope than IS, see
for example, (Ellaway et al., 2008). There are reports of successful usage of this approach for students’
activated learning (Bergin & Fors, 2003), and for students’ collaborative learning (Bergin et al., 2003).
Based on the analysis of the literature, we can conclude that there are positive evidences of using
computer simulation in teaching, both in IS discipline (e.g. simulation of an enterprise/organization) and
other disciplines (e.g. simulation of a patient in the medical profession). However, using such simulations is
not widely spread, at least not in IS. This can be partly explained by the fact that in the cases described in
literature, the object of investigation itself is simulated, e.g. a patient, or an enterprise. With today’s
technology, it is certainly possible to create a virtual reality where a student can work through the enterprise,
observe what employees are doing, ask them questions, etc. This is the main idea behind HyperCase. Though
technically simulating a real enterprise is possible, it comes at a considerable cost. While it may be
economically justifiable to simulate an enterprise as an accompanier to a course book that is sold in large
volumes, building such a simulator for a single course would hardly be justifiable.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
3
In the work reported in this paper, we tested another idea of using simulation in teaching enterprise
modeling. Instead of simulating an enterprise, we simulate a situation of an apprenticeship where the students
follow a modeling master and help him/her to do some part of the work on building models. More
specifically, the analyst chooses the information sources to be used for building a model, and hands the work
of building the model to the students. Such sources may include (but are not limited to): interviews with
stakeholders; samples of relevant documents, e.g., meetings protocols, forms for managing orders; web-based
sources, e.g. a company web site, results of twitter search on company name. To simulate such sources, e.g.
using video recorded interviews, is much easier than to build a virtual representation of an enterprise.
3. Analysis of the practice of teaching modeling skills at DSV
DSV is using a case-based method for teaching/learning modeling skills. Cases are artificial and they are
created by teachers as the needs arise. Cases are often reused in the same or different courses often with
modifications. A case represents an imaginary organization, e.g. an enterprise, a hospital, etc., and it consists
of two basic components: case presentation and suggested solution. Case presentation is in the form of a text
that describes the behavior and structure of the imaginary organization, and can include requirements on an
IT system that the organization would like to introduce, or an organizational change that it wants to carry out.
Sometimes, the text is presented as a transcript of an interview with the representatives of the imaginary
organization. The suggested solution is a model of the imaginary organization built based on case
presentation.
Cases are used in different teaching/learning activities. Smaller cases are used in lectures and tutorial
sessions where a teacher shows how to build a suggested solution based on a case presentation. Cases of the
same kind are used during examinations when students are required to build a model based on a short case
description.
More complicated cases are used in project-based teaching/learning, which constitute an essential part of
each course that includes in its learning objectives that the students should acquire modeling skills. A project
is carried out in groups where each group gets a case presentation and needs to build a solution of the same
kind as the suggested solution. However, there is no requirement that the students’ solution should be exactly
the same as the suggested one. Deviations are allowed as case presentation, usually, allows different
interpretations.
A student group can include from 2 to 5 students depending on the course and the number of enrolled
students. Each group gets tutoring during the project, including milestones reviews with the teacher. The
project is graded, often just on the basis pass/failed. The goal with the project is not so much examination,
but learning (acquiring modeling skills) through doing.
The problem with the current practice is that it does not explicitly focus on teaching/learning of how to
obtain the information needed for building a model. The main goal is to teach/learn how to build a
syntactically and semantically correct model given the text description.
4. Trial
This section contains the details of a proof of concept trial that has been completed in the frame of the
project. The goal of the trial has been:
1. To investigate whether there are any particular problems in introducing a multi-media project
presentation in a course at DSV.
2. To evaluate whether the change from the textual description to multi-media presentation affects the
learning results achieved in the course.
4.1 Trial settings
The trial has been completed in the frame of the first year course called “IT in Organizations” (ITO). Below,
we also will refer to this course as the “trial course”. ITO is a first year course that is mandatory for
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
4
approximately half of the students who enroll to bachelor programs at DSV. The number of enrolled students
is approximately 250. The course is given each fall term, and it is the second course for the students. Before
this course they have only completed one other course, “IT for personal use”, which presents basic theories
and concepts in IT and computer science. The trial course ITO can be classified as an introduction to IS. The
overall goal of the course is to give the students basic knowledge and skills on the analysis, design and
development of information systems, including theories, methods and techniques for this. Additionally, the
course should create an understanding of how organizations - their goals, products and services, business
processes and information - can be changed by using different forms of IS. The length of the course
corresponds to five weeks of full-time work for the students. Its ECTS credits are 7.5. Being an introductory
course, the course has a wide scope. It includes an overview of a number of modeling techniques, such as
functional modeling (with the help of Icam DEFinition for Function Modeling 0 (NIST, 1993), abbreviated
IDEF0), process modeling (with Business Process Model and Notation, abbreviated BPMN (OMG, 2013),
see Fig. 1), goal modeling (with Business Motivation Model (OMG, 2014), abbreviated BMM), and a
number of others. The teaching and learning activities in the course include lectures, workshops, tutorials,
labs and a project assignment. The project assignment is completed in groups of 4-5 students, and it is built
around a fictitious company called “AFFE”. The company is said to be engaged in development and sales of
a business game.
4.2 The textual way of presenting the project assignment
In earlier year’s occasions of the ITO course, the AFFE’s business was presented to the students in a textual
form. For example, a textual description of how the company handles invoices was available in the following
manner:
“Today, the sales process is relatively simple and involves a customer calling in to the sales
department and ordering a license of the game, or an upgrade of the existing license to the latest
version of the game. The sales department sends the game or the new version of it via mail along with
an invoice. A copy of the invoice is also sent to the finance department that takes responsibility for the
continuation of the process. If the customer does not pay by a certain date the accounting department
sends a reminder. If the customer still does not pay despite the reminder the case is handed to a
collection agency who takes the responsibility of getting money from the customer. When and if the
debt collection agency acquires the payment, the money is transferred to the AFFE's bank account.”
Based on the textual description, the students complete a number of tasks. For example, one task is to “Build
BPMN diagrams of the business processes currently used in AFFE’s practice”. Based on the textual
description above, the students were supposed to produce a process diagram of the kind presented in Fig. 1.
Figure 1. A BPMN diagram that corresponds to the fragment of a textual description
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
5
4.3 The multi-media way of presenting assignment
In the trial completed in the fall term of 2013, the textual assignments have been totally substituted by a
project site that utilized multi-media (Bider et al., 2013). The site included the following presentation
fragments:
Web-site of the company AFFE that presents the company, its products, customers and organizational
structure.
Recorded interviews with AFFE’s management (three interviews) who describe business processes
currently in use (i.e. a sales process and game development process) and problems faced by the company.
The recording was done with some of DSV teachers playing roles of business analysts and stakeholders.
Results of twitter search on the company name that presents the opinions of AFFE’s existing and potential
customers about their product.
Financial information obtained via http://www.proff.se/ (an online business finder service).
List of internal information systems in use by AFFE. The list included references to the web sites of IT
system vendors, and Excel spread sheet templates used in the sales process.
Protocol of the management meeting where a list of organizational changes where discussed and decided
upon.
These fragments and the assignments were integrated in a project web site. The site lists all sources of
information having links to the fragments listed above, and it lists all assignments, and to each assignment it
gives recommendations on what sources to use when completing the assignments. Fig. 2 shows an
assignment with the title: “Draw BPMN diagrams of current business processes”. The assignment page also
provides links to sources needed to carry out the assignment. Fig. 3 shows such a source, an interview with
the manager of the game development department, in which the manager is describing the game development
process. The project site was the only source of information that students received, no text description of the
company AFFE was handed out.
Figure 2. The project site – description of a task that students need to carry out
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
6
Figure 3. The Project site – interview with the game development manager
5. Evaluation of the trial
The trial was completed in a real course environment where we had no possibility to divide the students into
two groups: first would use the project site using multi-media, and second would use the textual description.
Instead, the evaluation of the trial was done indirectly in form of four sub-evaluations, listed below and
presented in the following subsections
1. A project site evaluation by the students
2. An evaluation by the students based on comparison with another course
3. A project site evaluation by the teachers
4. A comparative analysis of examination results
All evaluations pointed towards some improvements in learning environment and results.
5.1 A project site evaluation by the students
In the project site evaluation by the student, the students’ opinions on using multi-media for describing the
project assignments were gathered. The method used for the evaluation was a survey, using a questionnaire
that consisted of two types of questions. The first type of questions was statements to agree/disagree for
which the students gave their opinion using a value between 1 (completely disagree) and 5 (completely
agree), i.e. using a Linkert scale. The second type of questions was questions with open ended answers
regarding improvements of the solution.
The questionnaire was available to all students that have been registered to the ITO course via the student
administrative IT system. Totally, 68 of 210 students active in the ITO course have completed the
questionnaire.
The first type of questions in the questionnaire, the ones using the Linkert scale, contained both detailed
questions related to the use of process models, information models and financial information, and more
general ones. These questions and answers received are summarized in Table 1 below.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
7
Table 1. Results of evaluation by the students (”svar” = answer in Swedish)
1. The project assignment’s site gave a good picture
of the AFFE company and how it works.
59 students answered with an average of 3,9
2. The information in general on the project site was
enough to carry out the tasks.
58 students answered with an average of 4.1:
3. The information about the business processes of
AFFE on the project site was enough to carry out
the tasks.
58 students answered with an average of 4.1
4. The information about the IT systems of AFFE
on the project site was enough to carry out the tasks.
56 students answered with the average of 3.9:
5. The information about the problems of AFFE on
the project site was enough to carry out the tasks.
55 students answered with the average of 4.3
6. The information about the customers and their
view of products of AFFE on the project site was
enough to carry out the tasks.
55 students answered with the average of 4.0
7. The information about the economic situation of
AFFE on the project site was enough to carry out
the tasks.
55 students answered with the average of 3.8
8. The information on project site gave a good
understanding of what it meant to work as business
consult with a system science background.
56 students answered with the average of 3.9
Summarizing the analysis of the answers on the first type of questions in the survey, we can conclude that
according to the students, the project site had enough information about AFFE business for carrying out the
tasks in the project assignment. The average values were between 3.8 and 4.3 with somewhat higher values
for information about the problems (4.3), business processes (4.1) and customers and their view of products
(4.0) and somewhat lower for information about the economic situation of AFFE (3.8) and whether the
information on project assignment’s site gave a good understanding of what it meant to work as business
consult with a system science background (3.9).
The second part of the questionnaire contained two questions; “Which further information on the project
assignment’s site do you think is needed to carry out the tasks of the assignment?” and “How could the
project assignment’s site be further improved?”. These two questions only received three answers each.
Regarding further information one student requested more trustworthy information about the IT systems, two
students requested more financial information, and one stated that the information about AFFE was not
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
8
always consistent. Regarding suggestions for improvements there was a request for more pictures, a nicer
interface of the site, and more links between topics on the site.
Two of the comments, the comments on more trustworthy information about the IT systems and more
financial information, will be managed in the next improved version of the project site. An interesting
comment was the one about inconsistencies in the information about AFFE. When doing analysis and
modeling in reality, such inconsistencies are common, and, therefore, the students need to manage these
inconsistencies during their work with the project assignment. It could even be possible to add a task to the
existing assignment, expressed such as: “Which inconsistencies in the information about AFFE did you find,
and how did you manage these?”.
5.2 An evaluation by the students based on comparison with another course
In the evaluation by the students based on comparison with another course, the students’ opinions about the
use of the project site with simulation were obtained contrasting it to a traditional presentation of project
assignments, the one that uses textual descriptions. This was done by surveying the students that had
experiences of both types of presentations of project assignments. For the majority of the students, the next
course after ITO in the curriculum was the course on Object Oriented Analysis and Design (OOS). The ITO
course used multi-media to describe the project assignment, while the OOS course used a traditional textual
description to describe the project assignment. Therefore, the students who participated in both courses could
compare two different presentations of project assignments. The method used for the comparison was a
survey using a questionnaire that included two types of questions. The first type of questions was statements
to which the students gave their opinion using a value between 1 (completely disagree) and 5 (completely
agree). The second type of questions was the possibility for the students to post a general comment at the end
of the questionnaire.
The questionnaire was available to all students that have been registered on both the ITO and OOS
courses. Totally, 46 students of around 150 have filled in the questionnaire. In total, 210 students were active
in both the ITO and the OOS course but around 50 of these students were carrying out another type of
assignment in the OOS course that was not comparable with the project assignment in the ITO course.
Table 2 summarizes the results of the three questions that were part of the first type of questions in the
questionnaire.
Table 2. Evaluation by the students based on comparison with another course
1. A project site (similar to the one used for ITO
course) would facilitate the completion of the OOS
project because it would provide a more
comprehensive picture of the business.
46 students answered with the average of 4.2
2. A project site (similar to the one used for ITO
course) gave us better understanding of how a
professional consultant worked than just having a
textual description of the business.
46 students answered with the average of 4.2
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
9
3. A project site (similar to the one used for ITO
course) is to be preferred for all courses that include
business/enterprise modeling.
46 students answered with the average of 4.2
The answers of the first type of questions in the questionnaire shows that the students’ opinions, on average,
were that the multi-media presentation of project assignment similar to the one used in the ITO course was to
be preferred in all courses that include enterprise modeling.
The second type of questions in the questionnaire was a general comment that students could post at the
end of the questionnaire. This resulted in several comments from the students. Below, we present some of
them:
“The work using a project assignment site like the one in the ITO course is more challenging for the
students. For example, the students need to transcribe and interpret the video recorded interviews, which
takes time” (5 students with similar comments)
“The project assignment site like the one in the ITO course gave a more realistic view of an organization,
and is therefore to be preferred” (2 students with similar comments)
“The project assignment site like the one in the ITO course made the work with the assignment more
interesting and fun” (2 students with similar comments)
“Information should be given both as a web site and in text form. To use just videos may lead to
misunderstanding and to use just text may not give enough information to carry out the tasks” (1 student)
5.3 A project site evaluation by the teachers
In the project site evaluation by the teachers, we examined the teachers’ opinions about using a case based
learning repository for project assignment in comparison with using a traditional approach of a textual
description. This was done by letting teachers that have experiences of both approaches carrying out the
comparison. Therefore, teachers that have participated in the ITO course during several occasions were asked
to participate in the comparison. The method used for the comparison was the same as the one described
earlier, using a survey and a questionnaire with a number of statements to which the teachers gave their
opinion using a value between 1 and 5. Totally, three teachers of five engaged in the course participated in
the survey; two teachers were excluded as they were part of the site design team and could have bias.
A summary of the answers received from the teachers is presented below:
Statement 1: The ITO project site in 2013 using multi and hyper media helped you as a teacher to
supervise your groups of students compared to previous years when the project assignments were
expressed in textual form. The three answers where equally distributed between option 1, 2 and 3. Thus, it
can be concluded that the teachers disagreed to the statement.
Statement 2: The students in the ITO course 2013 using the multi and hyper media project site have
shown more engagement when completing the project in comparison with previous years' students when
the project assignments were expressed in the textual form. Two teachers set option “5” (completely
agree), while one selected “4”.
Statement 3: The quality of project results achieved by the ITO students 2013 were higher in comparison
with the previous years because of the use of the new ITO course site describing the project assignment
using multi and hyper media. Two teachers selected “5” (completely agree), while one teacher selected
“4”.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
10
Statement 4: You feel that your workload per student group in the 2013 year's ITO course decreased in
comparison with the previous years because of the use of the new ITO course site describing the project
assignment using multi and hyper media. Two teachers to some extent disagreed, selecting option 2 and 3,
while one completely agreed (selected “5”).
Statement 5: All projects at DSV that concern modeling of business activities and concepts should use
similar project sites using multi and hyper media for describing the project assignments. Two teachers
were undecided (option 3) while one completely agreed (selected “5”).
As follows from the results, there are only two questions where full consensus can be seen, i.e. questions 2 –
students’ engagement, and 3 – quality of the project results. In both cases two teachers selected value “5” and
one teacher selected value “4”. Therefore, all three teachers consider that the new way of presenting the
project assignment positively affected the most important parameters of the teaching/learning environment:
learners’ engagement and their achievements in the project. This opinion coincides with the reflections of
two of the authors of this report who supervised the projects in the frame of ITO course. Their opinions was
not been recorded in the survey to avoid possible bias.
Note that the teachers were less enthusiastic than the students in their assessment whether a multi-media
presentation of project assignments should be accepted for all courses that include teaching modeling skills.
Note also that the teachers did not agree that the use of multi and hyper media helped them as teachers to
supervise groups of students compared to previous years.
5.4 A comparative analysis of examination results
In the comparative analysis of examination results, we compared the exam results achieved by ITO students
in the year 2012, when the textual version of the project assignments was used, with the exam results of the
year 2013, when the multi-media version of the project assignments was used. We used the results of the
written exams for making the comparison, as the exams had a detailed scale of grading, while the project was
assessed on the pass/fail scale. The scale for the exams included: A (highest) – E (lowest) as passed, and Fx
(failed on some components) – F (failed) as not passed. The comparative results of students from years 2012
and 2013 are presented in Fig 4. As can be seen from this figure, the main difference starts at grade C 31%
for 2013 vs. 23% for 2012, and shows that the “tail” of the distribution of lower grades looks better for the
year 2013 than for the year 2012. We can, cautiously interpret this change as less strong students achieving
better results in the year 2013 than the students in the year 2012.
Year 2012
Year 2013
Figure 4. Comparison between the results of written examination
To exclude the possibility of the year 2013 enrollment having “brighter” students than the year 2012, we
checked whether the results of the written examination in the next course, for the same students, differed
between the years 2012 and 2013. According to the course supervisor in that course there was no substantial
difference between the students results in that course between the years 2012 and 2013.
Comparing the distribution curves of years 2012 and 2013 it could be observed that in 2012 there is a
small grouping of students at the lower grade scales that shows a student skill sets that is slightly bimodal in
nature. Bimodal distribution in skill sets among first year computer science students have been observed in
other study (McCracken et al., 2001):
“The result about bi-modality is troubling. There are two distinct groups of performance in our datasets.
This result suggests that our current teaching approach is leading to one kind of performance for one
sizable group of students and another kind of performance for another sizable group. We need to keep in
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
11
mind that different groups of students have different needs and strengths; we must ensure that the results
from one group do not obscure our view of the other.”
It can be suggested that the difference between the change in distribution of grades between 2012 and 2013
can be due to the fact that the computerized environment used in the 2013 course was more beneficial to the
D and C group of students than it was to the A and B group of students. That is to say that the computerized
environment was able to help the lower skill group without negatively affect the high skill group and
consequently produced a more unimodal distribution of grades in 2013.
6. Conclusion
6.1 Summary and potential impact
As was discussed in the Introduction, this paper is devoted to teaching/learning “difficult knowledge” of the
WPT (Ways of Thinking and Practicing) type (McCune & Hounsell, 2005) in a university setting. More
specifically, we investigated this problem on a specific case in IS discipline – teaching/learning modeling
skills. The main achievements of the project described in this paper consist in the following:
Explicitly formulating the problem of teaching/learning modeling skills in the university setting in
general, and in DSV at Stockholm University in particular.
Considering a solution, not widely used in the field of Information System
Conducting a trial that have shown that the solution suggested can be introduced in the IS courses at
DSV, and, according to evaluation, affects positively the students learning environment.
Our solution to using computer simulation for teaching/learning in IS discipline differs from the ones
suggested and used in IS teaching practice, like HyperCase (Kendall et al., 2013). Instead of simulating an
object of investigation/modeling, we simulated the situation of apprenticeship using multi-media presentation
of a business case. The main difference here is the costs. Our solution has low-level costs which make it
feasible even for a single university course. The total resources spent for creating multi-media presentation of
the case and building a project site was 55 person-hours, where 23 went to creating multi-media fragments
and 32 for designing and implementing the project site. The resources above do not include time spent for
research and dissemination (like writing articles and presenting them in the conferences). Neither do they
include the time that was needed for the development of business case. We took an existing business case and
converted its text presentation into a multi-media presentation. This was done for two reasons: (a) to save
time, and (b) being able to make comparisons when conducting evaluation.
Note that of 32 person-hours spent for developing the project-site, 24 went to designing the site and 8 for
filling the content. The design is generic and can be re-used for any other course project. Thus for building a
new simulator of the same complexity would require resources in the range of 30-40 person-hours, provided
that the business case for the project has already been designed.
Considering the low-cost pragmatic approach to using computer simulation in teaching/learning WTP, we
believe that our approach could be reused by other university teachers, especially in the IS discipline. This
approach can be used in teaching, system design, requirements engineering, business analysis and other
courses typical for Information Systems, Software Engineering, Business Administration, etc. We also
believe that it can be used in other areas of academic and professional education. To promote using
computers for simulating apprenticeship, we have published technical details of the project (Bider et al.,
2015), and are ready to share with others the project site design implemented with the help of WordPress.
6.2 Methodology and plans for the future
The research presented in this paper is being conducted as Action Research (AR) (Lewin, 1947), the
characteristics feature of which is its direction on solving a problem in a local practice, while solution is of
generic nature and can be used in similar local practices. Another characteristic feature of AR is that research
is being conducted in cycles. There are many ways of representing an AR cycle with the number of phases
varying from 2 to 6, dependent on how detailed one wants to present the cycle. The original Kurt Lewin’s
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
12
cycle included 6 stages: (1) Analysis, (2) Fact finding, (3) Conceptualization, (4) Planning, (5)
Implementation of action, and (6) Evaluation. Others use 4 phases cycle (1) Plan, (2), Act, (3) Observe, (4)
Reflect. The minimum AR cycle can be found in (ALARA, 2002) which consists of repetitions of two
activities act review act review ... .
We use the version suggested by Kolb for experimental learning style theory (Kolb, 1984), which can be
represented as four stage learning cycle: Concrete experience Reflective Observation Abstract
Conceptualization Active Experimentation Concrete Experience …, see Fig. 5. We consider that
Kolb’s cycle is a good representation not only for student learning, but also for AR in a situation when
researchers are also practitioners in the field where the given research project is conducted. In this case,
learning is equal to obtaining new knowledge not only for the researchers involved, but also for the
community of researchers and practitioners in the discipline.
Figure 5. Kolb’s cycle of experimental learning (Kolb, 1984)
We are currently on the second cycle, having applied the new course design for the second time in autumn
2014, and now doing “Reflective Observation”. In the next phase of our research (Abstract
Conceptualization) we will investigate means to make the created information fragments reusable, so that
several courses can share the same material. A start would be to see if the information fragments developed
in the studied ITO course can be labeled in such a manner as to make them useful in another IS course where
modeling skill are also developed. For this labeling different ontologies/taxonomies will be tested, including
Bloom taxonomy (Bloom, 1956), to map the formal intended learning objectives of the IS courses in the
department to see which courses that may use the created information fragments. This would give economies
of scale when reusing video and textual information fragments. Furthermore, video annotation could be used
to make the video interviews more reusable by creating searchable libraries.
Acknowledgments. The authors are grateful to our colleagues Jelena Zdravkovic and Anders Thelemyr who
participated in interviews recording. The project was sponsored by Stockholm University’s program “Future
Learn”. The first author is also grateful to Anthony Burden who encouraged him in becoming engaged in
pedagogical research.
References
ALARA, 2002. Action Research. [Online] Available at: http://www.alarassociation.org/pages/al-and-ar/action-research
[Accessed April 2015].
Bergin, R.A. & Fors, U., 2003. Interactive simulated patient—an advanced tool for student activated learning in medicine
and healthcare. Computers & Education, 40, pp.61–376.
Bergin, R. et al., 2003. Interactive simulated patient: experiences with collaborative e-learning in medicine. J Educ.
Comput. Res, 29(3), pp.387-400.
Bezivin, J. et al., 2010. Teaching Modeling: Why, When, What?. In Ghosh, S., ed. MODELS 2009 Workshops, LNCS
6002. Springer, pp.55–62.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
13
Bider, I., Henkel, M., Kowalski, S. & Perjons, E., 2015. Teaching Enterprise Modeling based on Multi-Media
Simulation: a Pragmatic Approach. In Benyoucef, M., Weiss, M. & Mili, H., eds. 2015 MCETECH Conference.
Montreal. Springer, LNBIP 209.
Bider, I., Perjons, E., Henkel, M. & Kowalski, S., 2013. Projektplatsen för ITO 2013. [Online] Available at:
http://ito2013.blogs.dsv.su.se/ [Accessed October 2014].
Bloom, B.S., 1956. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay
Co Inc.
Costain, G. & McKenna, B., 2011. Experiencing the Elicitation of User Requirements and Recording them in Use Case
Diagrams through Role Play. Journal of Information Systems Education, 22(4), pp.368-82.
Ellaway, R. et al., 2008. Building a virtual patient commons. Med Teach, 30, pp.170-74.
Engels, G., Hausmann, J.H., Lohmann, M. & Sauer, S., 2006. Teaching UML is Teaching Software Engineering is
Teaching Abstraction. In Bruel, J.M., ed. MoDELS 2005 Workshops, LNCS 3844. Springer, pp.306–19.
Fenstermacher, K.D., 2004. If I had a Model, I’d Model in the Morning. In Proceeding of OOPSLA'04. ACM, pp.88-89.
Guy, E., Pemberton, L. & Knight, J., 2000. Rich Cases: a framework for interactive case studies in Information Systems
teaching. European Journal of Open, Distance and E-Learning.
Kendall, J.E., Kendal, K.E., Baskerville, R.L. & Barnes, R.J., 1996. An Empirical Comparison of a Hypertext-Based
Systems Analysis Case with Conventional Cases and Role Playing. DATA BASE ADVANCES, 27(1), pp.58 – 77.
Kendall, J.E. & Kendall, E.K., 2013. Systems Analysis and Design. 9th ed. Pearson.
Kendall, J.E. et al., 2013. HyperCase: A Hypertext-Based Case for Training Systems Analysts. [Online] Available at:
http://www.pearsonhighered.com/hypercase/hypercase2.9 [Accessed December 2013].
Kolb, D.A., 1984. Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ,
US: Prentice-Hall.
Kolb, D.A. & Fry, R., 1975. Toward an applied theory of experiential learning. In Cooper & C., eds. Theories of Group
Process. London: John Wiley. pp.33-57.
Lane, H.C. & VanLehn, K., 2005. Teaching the tacit knowledge of programming to novices with natural language
tutoring. Computer Science Education, 5(3), pp.183-201.
Lewin, K., 1947. Frontiers in Group Dynamics II. Human Relations, 1(2), pp.143-53.
McCracken, M. et al., 2001. A multi-national, multi-institutional study of assessment of programming skills of first-year
CS students. Working group reports from ITiCSE on Innovation and technology in computer science education
(ITiCSE-WGR '01). New York, NY, USA: ACM ITiCSE, pp. 125-180.
McCune, V. & Hounsell, D., 2005. The development of students' ways of thinking and practising in three final-year
biology courses. Higher Education, 49(3), pp. 255-289.
NIST, 1993. Integration definition for function modeling (IDEF0), Draft Federal Information Processing Standards,
Publication 183, 1993. [Online] Available at: www.idef.com/downloads/pdf/idef0.pdf [Accessed February 2015].
Nonaka, I., 1994. A dynamic theory of organizational knowledge creation. Organ. Sci., 5(1), pp.14–37.
OMG, 2013. Business Process Model and Notation (BPMN), Version 2.0.2, Object Management Group (OMG),
Document formal/2013-12-09, December 2013. [Online] Available at: http://www.omg.org/spec/BPMN/2.0.2/PDF
[Accessed February 2015].
OMG, 2014. Business Motivation Model, Version 1.2, Object Management Group (OMG), Document formal/2014-05-01,
May 2014. [Online] Available at: http://www.omg.org/spec/BMM/1.2/PDF [Accessed February 2015].
Perkins, D., 2007. Theories of difficulty. In BJEP Monograph Series II, No 4 - Student Learning and University
Teaching. Leicester: The British Psychological Society. pp.31-48.
Polanyi, M.S., 1969. Knowing and Being. Chicago: University of Chicago.
Recker, J. & Rosemann, M., 2009. Teaching Business Process Modelling: Experiences and Recommendations.
Communications of the AIS, 25, pp.379-94.
Truex, D.P., Alter, S. & Cherie, L., 2010. Systems Analysis for Everyone Else: Empowering Business Professionals
through a Systems Analysis Method that Fits Their Needs. In Proceedings of ECIS 2010. AIS.
Venable, J., 1996. Teaching novice conceptual data modelers to become experts. In Software Engineering Education and
Practice., pp.411-18.
Published in Interactive Technology and Smart Education Vol.12 No.2, 2015 pp.137-154
©Emerald Group Publishing Limited 1741-5659 DOI 10.1108/ITSE-04-2015-0004
14