Conference PaperPDF Available

Abstract and Figures

Previous researches in pedagogy and project management have already underlined the positive contribution of serious games on project management courses. However, the empirical outcome of their studies has not been translated yet into functional and technical specifications for "serious games" designers. Our study aims at obtaining a set of technical and functional design guidelines for serious game scenario editors to be used in large classes of project management students. We have conceived a framework to assess the influence of different serious games components over student’s perceived acquired competency. Such frameworks will allow us to develop a software module for reflective learning, which is meant to extend theory of serious games design.
Content may be subject to copyright.
Analysis of serious games implementation for
project management courses
R. Bonazzi
1
, S. Missonier
1
, D. Jaccard
2
, P. Bienz
1
, B. Fritscher
1
, E.
Fernandes
1
Abstract Previous researches in pedagogy and project management have already
underlined the positive contribution of serious games on project management
courses. However, the empirical outcome of their studies has not been translated
yet into functional and technical specifications for "serious games" designers. Our
study aims at obtaining a set of technical and functional design guidelines for seri-
ous game scenario editors to be used in large classes of project management stu-
dents. We have conceived a framework to assess the influence of different serious
games components over student’s perceived acquired competency. Such frame-
works will allow us to develop a software module for reflective learning, which is
meant to extend theory of serious games design.
Introduction
Information system (IS) project management courses are known to be challenging
to conceive, since most of the skills required for project managers cannot be
achieved ex cathedra. Problems in IS are characterized by incomplete, contradic-
tory and changing requirements, and solutions are often difficult to recognize be-
cause of complex interdependencies. This leads to an educational dilemma in
teaching such problems because a rich background of knowledge and intuition are
needed for effective problem-solving. Hence complexity is added rather than re-
duced with increased understanding of the problem (Connolly and Stanfield,
2006).
As a consequence of the large number of failed projects a strong challenge to
traditional methods of project management based on universal best practices (such
1
University of Lausanne, School of Economics, Lausanne, Switzerland
stephanie.missonier@unil.ch
2
HEIG-VD, Yverdon-les-Bains, Switzerland dominique.jaccard@heig-vd.ch
as the Project Management Institute) emerges in the academic world and among
practitioners (Hodgson and Cicmil, 2007; Sauer and Reich, 2009). Traditional
approaches are part of a very instrumental and functionalist vision of promoting
project management principles that do not reflect the reality of the projects, which
is ambiguous, fragmented, complex, socio-technical built, and with a strong politi-
cal character. Therefore, project management is a discipline that requires knowl-
edge and reflective practice that allows players to lead the project team in an
emergent way. This kind of frameworks requires a high degree of interaction be-
tween teacher and students. But face-to-face exchanges are hard to manage when
the number of students is greater than forty (Smith and Kampf, 2004).
Game-based learning (also known as serious games) uses simulation to allow
students to actively acquire competences required to solve problems. Hence game-
based learning scenarios might be the solution to introduce large classes to IS pro-
ject management since they are known to have an effect on student's self-efficacy
as well as acquisition and retention of declarative and procedural knowledge
(Sitzmann, 2011). Yet little interest has been given so far on how to design a sce-
nario editor to support an IS project management course by means of game-based
learning. In software engineering courses, game-based simulations are far less
used than other types of educational approaches (e.g. industrial partnership or
team learning) and they lack to incorporate model-based instruction and reflective
learning (Navarro and Van Der Hoek, 2009). We expect a similar trend in IS pro-
ject management courses. Therefore our research question is:
How to design a game-based learning scenario editor to support an infor-
mation system project management course for more than forty students?
By adopting a design science methodology this study aims at obtaining a
framework to design game-based learning scenario editors to enhances project
management competences for students attending the course. Such framework is
induced by testing different software components to assess their influences of stu-
dents’ acquired competency. Therefore the creation of a model to assess the soft-
ware components described in this paper is the initial step of such study. We start
here by assessing the gaps in the existing literature and by deriving a conceptual
model in the next section. The third section illustrates the methodology we adopt
to test our conceptual model and to assess the pedagogical effect of different soft-
ware tools. The results of a first assessment performed in one of these teaching
courses are presented in the fourth section as example. The paper ends by discuss-
ing the results obtained and by highlighting the next steps of our study.
Literature review
This section briefly assesses the state of the art in game-based learning for project
management course. We are looking for concrete evidences regarding the link
between game-based learning and performance of the IS project management
course. Hence we use the guidelines of Okoli and Schabram (2010) for a protocol
to assess the existing literature. For sake of simplicity we decide to limit our
Google Scholar search to articles published in the period 2005 -2010. Using the
selected keywords (“project management”; “information systems”; “game-based
learning”) we obtain 59 results, among which 21 are cited by at least another pa-
per and accessible to us. Since we are interested in articles that have assessed the
performance of the serious game analysed, we skim our set of articles to only a
few. For those papers we perform forward and backward analysis, i.e. we assess
the papers that cite/are cited by them. At the end we obtain two streams of re-
search: ex-ante evaluation and ex-post evaluation. Since we wish to connect these
two stream of research we derive three concepts: the student’s perceived acquired
competency (1), which is the set of measured capabilities that the student acquires
in class; the perception of the serious game design (2), which we consider here as
the set of features that the game-based learning software possesses to empower the
teacher; the student's engagement (3), i.e. the student's will to take part actively to
the game-based learning experience.
The first stream of research focuses on the ex ante evaluation of the effect that
serious game design has on student’s perceived acquired competency. This group
of papers claims that while traditional methods are based on an instructivist meth-
odology, game-based learning provides a constructivist learning environment
where learners can practice the formulation of requirements specification through
requirements elicitation and learning by doing (Hainey and Connolly, 2010). In
addition to that game-based learning provides a challenging and complex real-
world environment within which to apply their theoretical knowledge to overcome
difficulties in dealing with ambiguity and vagueness, while developing self-
confidence and increased motivation (De Freitas et al., 2006).
The second stream of research focuses on the ex post evaluation of the effect
that student's engagement having played the serious game has on the student’s
perceived acquired competency. Researchers collect student's suggestions for
game changes (Pfahl et al., 2004; Dantas et al., 2004) and perceived competences
needed (Pfahl et al., 2004; Greese von Wangenheim et al., 2009; Zapata, 2010;
Mawdesley et al., 2011).
To link these two streams of research we suggest considering the student's en-
gagement as a mediator between serious game design and student’s perceived ac-
quired competency. At the end we derive the following set of hypotheses: (H1)
the perception of the serious game design influence the student’s perceived
acquired competency; (H2) the student's engagement influences the student’s
perceived acquired competency; (H3) the perception of the serious game de-
sign influences the student's engagement.
In the next section we illustrate how we intend to design an experiment to test
our hypotheses.
Methodology
In this section we briefly describe the methodology we use to perform our experi-
ment. Design science seeks for outcomes that can be relevant for practitioners and
that have been obtained in a rigorous way. The purpose in this kind of study is
usefulness rather than truth. Although design science has been used since many
decades, it has been officially accepted in information system since the Manage-
ment Information System Quarterly article of Hevner et al. (2004). In our study
and in this paper we adopt the methodology suggested by Peffers et al. (2008),
which proposes a process composed of six steps. Following the first step we
clearly identify our problem, using the literature review, as summarized by our
research question.
The second step of the methodology identifies the objectives of the solution. In
this sense, in the previous section we have identified two gaps in the literature: the
first one concerns the link among ex ante and ex post evaluation criteria, whereas
the second one regards the use of reflective learning by means of serious games.
Thus our study should start by conceiving a framework to assess the correlation
among ex ante and ex post evaluation criteria. Then we will move towards the
development of an additional module for reflective learning over the student's
achieved skills and towards the assessment of its added value.
Design and development: In the third step of the methodology the design and
development of the new component occurs. Yet in the first part of our study de-
scribed here, the development is minimal since we have decided to reuse an exist-
ing serious game. The selected platform to test our assessment framework is a
game-based learning scenario editor called Albasim. The main reason underlying
the choice of such platform is its large set of existing features and the direct link
that the authors have with the development team of the software. This is going to
be very useful during the second part of the study, when we will be developing an
additional component. Figure 1 illustrates the dashboard used by the game players
by means of a web browser. On the top right corner there are the key performance
indicators. On the top left corner of the screen the four stages of the game are il-
lustrated: the players start by the project initiation (1), then they move on by plan-
ning the project (2) and executing it (3) before closing it (4). The central part of
the screen is multifunctional, whereas the right side of the central screen allows
the player to manage resources and task, and to read e-mails send by the central
system. For what concerns the reflective learning, the system does not have a
dedicated feature, leaving to the teachers the task to arrange students’ presenta-
tions to share lessons learned, as explained in the following section.
Figure 1: Dashboard of Albasim (Source: www.albasim.com)
The pedagogical scenario implemented: The fourth step of the methodology of
Peffers et al. (2008) requires a demonstration of the artefact. In our case the game
requires two four-hour sessions, for a total of eight class hours over two weeks.
Before the first session the students receive the software manual and the business
case. At the beginning of the first session students get familiar with the idea of
serious game and to the functionalities of the software (e.g. the dashboard). Then
the students are asked to gather in group and to collect and process information
own by the different fictive players in the game, in order to deliver a project pro-
posal to be validated with the client (i.e. the professor). During the rest of the
week the students are supposed to work in group to complete the assignment and
send the improved project proposal to the professor, who choses two proposals
among them. At the beginning of the second game session the chosen grooups are
asked to do a short presentation of their project proposal to the rest of the class.
Once two student groups have presented the teachers gives them a constructive
feedback and add some remarks about the overall performance of the other groups
(best and worst practices). After the presentations the teacher recalls to the class
key theoretical concepts regarding project planning. Then the students are asked to
work in group to make and to justify their planning decisions, while taking into
account a set of constraints (time, cost, quality, resources availability and risks). In
the rest of the week student groups are ask to finalize the WorkBreakdown Struc-
ture, Program Evaluation and Review Technique and Gantt diagrams, together
with cost estimations.
Evaluation: the fifth step of the methodology concerns the evaluation of the ar-
tifact. To operationalize our constructs we reuse existing items from the two
streams of literature whenver possible and we obtain a set of five-point Likert
scale items, which are meant to be collected by questionnaire to be handed once
the students have completed the assignments of the second game session. For the
student’s perceived acquired competency we derive four items inspired by Zapata
(2010) and Mawdesley et al. (2011). For the serious game design we implement
seven items inspired by Hainey and Connolly (2010) and de Freita and Oliver
(2004). For the student's engagement we use seven items inspired by Gresse von
Wangenheim et al. (2009) and Dantas et al. (2004). A set of open questions has
been collected as well, but their answers will be not presented here for sake of
brevity.
Current results
We have tested the serious game with a sample of bachelor students enrolled in a
project management course with a special focus on information systems. We have
collected students' perception by means of an electronic survey. We have ob-
tained 74 answers out of the total of 104 students. Although limited in size, we
consider this sample as representative for our study and a good starting point to
perform statistical analysis using Stata 11. We started by performing the Cron-
bach’s alpha test over each set of items to measure how well each set of items was
representing the concepts. A Cronbach’s alpha value of 1.00 would be optimal,
whereas a value below 0.70 should be rejected. In our case we obtained the fol-
lowing results: acquired competency = 0.79; design = 0.80; engagement = 0.77.
While testing the causality effect we have performed seemingly unrelated re-
gressions among the three constructs obtained by performing the average of each
set of items (i.e tau-equivalent factor loadings). In other words we have asked
Stata 11 to tests all the regressions at once.
Figure 2 represents the results that we obtained and it shows that serious game
design has also a direct effect over student's acquired competencies, which is sta-
tistically significant (p<0.01). It also appears that the student's engagement has an
effect over the student's perceived acquired competency that is statistically signifi-
cant (p <0.05). Finally the serious game design has an effect over the student's
engagement that is statistically significant (p <0.01). Thus all hypotheses are
confirmed.
Figure 2: Results of the preliminary test
The direct and indirect effect of serious game design explains almost 50% of
student’s perceived acquired competency variance among students (R
2
=0.47). This
is to say that none of the two effects should be neglected. In addition to that the
student's engagement variability among students is largely explained by serious
game design (R
2
=0.56), which leads us to believe this model has a good explana-
tory power. We have also controlled for the effect of sex and nationality and the
results were not statistically relevant.
Conclusions and further works
We start this section by recalling our research question: How to design a game-
based learning scenario editor to support an information system project manage-
ment course for more than forty students? In this paper we present our framework
to link ex ante and ex post evaluation criteria to assess a game-based learning edi-
tor. Now that the framework is in place we can develop the reflective learning
module and we can assess its added value by using such module on a subset of the
overall students’ sample, treating the rest of the class as control group. The results
we obtained so far lead us to believe that serious game design has a direct and
indirect effect over student’s perceived acquired competency, which is mediated
by student's engagement. The module we wish to develop has a graphical interface
that allows the scenario designer to represent the scenario as a graph. The module
is expected to be able to mine the log of student groups’ actions and to represent
them under the shape of graphs, in order to benchmark the different groups’ ex-
perience.
In the next iteration we intend to have students groups playing different ver-
sions of the same game, whereas the student’s acquired competency will be tested
with a set of questions in the final exam of the course. These improvements should
increase the reliability of our results against endogeneity due to common method
variance (Antonakis et al., 2010).
References
1. Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. (2010). On making causal claims: A
review and recommendations. The Leadership Quarterly, 21(6), 1086-1120.
2. Connolly, T., & Stanfield, M. (2006). Using Games-Based eLearning Technologies in Over-
coming Difficulties in Teaching Information Systems. Journal of Information Technology Edu-
cation, 5(2006), 459-476.
3. Dantas, A. R., Barros, M. O., & Werner, C. M. L. (2004). A simulation-based game for pro-
ject management experiential learning. 2004 International Conference on Software Engineering
and Knowledge Engineering.
4. De Freitas, S., & Oliver, M. (2006). How can exploratory learning with games and simula-
tions within the curriculum be most effectively evaluated? Computers & Education, 46(3), 249–
264.
5. Gresse von Wangenheim, C., Thiry, M., & Kochanski, D. (2009a). Empirical evaluation of
an educational game on software measurement. Empirical Software Engineering, 14(4), 418–
452.
6. Hainey, T., & Connolly, T. M. (2010). Evaluation of a game to teach requirements collection
and analysis in software engineering at tertiary education level. Computers & Education, 56(1),
21-35.
7. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information sys-
tems research. Management Information Systems Quarterly, 28(1), 75–106.
8. Hodgson, D., & Cicmil, S. (2007). The Politics of Standards in Modern Management: Mak-
ing “The Project”a Reality*. Journal of Management Studies, 44(3), 431–450.
9. Mawdesley, M., Long, G., Al-jibouri, S., & Scott, D. (2011). The enhancement of simulation
based learning exercises through formalised reflection, focus groups and group presentation.
Computers & Education, 56(1), 44–52.
10. Navarro, E. O., & Van Der Hoek, A. (2009). On the Role of Learning Theories in Furthering
Software Engineering Education. Ellis, H. J.C. ,Demurjian, S.A. and Naveda J. F., eds. Software
Engineering: Effective Teaching and Learning Approaches and Practices (pp. 38-59). IGI Glob-
al.
11. Okoli, C., & Schabram, K. (2010). A Guide to Conducting a Systematic Literature Review of
Information Systems Research. Retrieved from http://sprouts.aisnet.org/10-26/
12. Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2008). A Design Science
Research Methodology for Information Systems Research. Journal of Management Information
Systems, 24(3), 45-77.
13. Pfahl, D., Laitenberger, O., Ruhe, G., Dorsch, J., & Krivobokova, T. (2004). Evaluating the
learning effectiveness of using simulations in software project management education: results
from a twice replicated experiment. Information and Software Technology, 46(2), 127-147.
14. Sauer, C., & Reich, B. H. (2009). Rethinking IT project management: Evidence of a new
mindset and its implications. International Journal of Project Management, 27(2), 182–193.
15. Sitzmann, T. (2011). A Meta-analytic Examination of the Instructional Effectiveness of
Computer-based Simulation Games. Personnel Psychology, 64(2), 489–528.
16. Smith, K., & Kampf, C. (2004). Developing writing assignments and feedback strategies for
maximum effectiveness in large classroom environments. Professional Communication Confer-
ence, 2004. IPCC 2004, Minneapolis, Minnesota, USA.
17. Zapata, C. M. (2010). A Classroom Game for teaching Management of Software Companies.
Dyna, 77(163), 290–299.
... In supply chain management, the beer distribution game (see Forrester 1958) exposes students to the importance of coordination in supply chain management and the so-called " bull-whip effect. " Project management curricula (Bonazzi et al. 2012; Pfahl et al. 2004) make use of the complexity that can be created by simulation games to assess and stress different methods of managing projects. Serious games in the form of ERP-based simulations such as ERPsim (Léger 2006) also prove useful in the area of information systems, mainly to teach students the dimensions of enterprise systems and business process integration through a learning-bydoing approach (Cronan and Douglas 2012; Foster and Hopkins 2011; Seethamraju 2011). ...
... Existing research shows that simulation games provide positive learning experience in many academic disciplines. They raise the awareness of students regarding specific aspects such as the importance of coordination in supply chain management (Forrester 1958) or the assessment of different project management methods (Bonazzi et al. 2012; Pfahl et al. 2004). These related studies primarily focus on the opportunities provided by simulation games, as well as their benefits compared to traditional learning strategies. ...
Conference Paper
Full-text available
Serious games are becoming increasingly popular as learning tools in management education. Owing to their engaging nature, they enhance students’ intrinsic motivation for learning and provide challenging environments for problem-based and experiential learning. However, they also require changes to instructional and pedagogical strategies in order to leverage the full potential of problem-based learning. This paper suggests using simulation games based on enterprise resource planning systems (ERP) to activate and integrate knowledge from different management disciplines. Our research contributions are threefold: First, we suggest a set of learning goals and a curriculum design for a capstone course that leverages ERP-based simulation games. Second, we present empirical insights based on a systematic evaluation of the learning outcomes with two cohorts of 32 students. Third, we derive a set of recommendations related to the role of ERP-based simulation games and course design in general management curricula.
... Three of the authors (DJ, ML and AH) belong to the AlbaSim research lab (Media Engineering Institute, University of Applied Sciences of Western Switzerland), which has been developing serious games for more than a decade. This lab has conducted serious games projects from design to implementation in many different educational fields such as emergency triage at hospital, cardiac clinical evaluation, oncology care, project management, computer education, energy management, or crime scene investigation [17][18][19][20][21][22]. Another author (LS) has conducted serious games projects in fields like resuscitation and COVID-19 infection prevention and control at the Geneva University Hospitals [23,24]. ...
Preprint
BACKGROUND Serious games offer teachers the opportunity to create meaningful learning scenarios and are increasingly used at all levels of education. Designing efficient and engaging serious games is a difficult process which requires a collaborative approach. Many design frameworks have been described, most of which are dedicated to the development of specific types of serious games. OBJECTIVE Our aim was to create a general serious game design framework which could be adapted to all kinds of serious games and implemented in a collaborative web platform. METHODS We combined the results of a literature review with our experience in serious game development to determine the basic building blocks of a design framework. We then organized these building blocks into categories and determined the features that a generic design framework should include. Finally, based on the paradigm of complex systems and systemic modelling, we created the co.LAB generic design framework and specifications to allow its implementation in a collaborative web platform. RESULTS Based on a total of 10 existing design methodologies or frameworks, 23 building blocks were identified and represent the foundation of the co.LAB framework. These blocks were organized into five categories: "context and objectives", "game design", "mechanics", "learning design" and "assessment". The arrangement by categories provides a structure which can be visualized in multiple and complementary ways. The classical view links game and learning design while other views offer project, systemic and process visualizations. For the implementation of the co.LAB framework in a web platform, we propose to convert the building blocks into “cards”. Each card would constitute a collaborative working space for the design of the corresponding block. To make the framework adaptive, cards could be added, adapted or removed according to the kind of serious game intended. Enhancing the visualization of relationships between cards should support a systemic implementation of the framework. CONCLUSIONS By offering a structured view of the fundamental design elements required to create serious games, the co.LAB framework can facilitate the design and development of such games by virtue of a collaborative, adaptive and systemic approach. The different visualizations of the building blocks should allow for a shared understanding and a consistent approach throughout the design and development process. The implementation of the co.LAB framework in a collaborative web platform should now be performed and its actual usability and effectiveness tested.
Conference Paper
Full-text available
The knowledge and the implementation of patients' rights are fundamental in health professional practices. Courses for healthcare students are often taught using a lecture format. Yet, students find it difficult to understand the importance of legal concepts and apply it to their future practice. In order to promote learner centered teaching practices and encourage knowledge acquisition and practical skills development, the "Patient's Rights & Innovative Teaching Strategy (PRITS)" project aimed at developing a serious game dedicated to patients' rights education. To design and develop this serious game and the associated learning concept, we set up a multidisciplinary team of experts from the fields of healthcare, law, education and engineering. A collaborative methodology was used to ensure the coherent development of all games and learning elements. The PRITS serious game integrates knowledge about patients' rights into dialogues with virtual patients. It also provides additional resources and exercises related to the topics. The learning experience intends to challenge students' mental models and support a transition from "quality of care only "to "quality of care and respect of patients' rights".
Article
Full-text available
Background Serious games are increasingly used at all levels of education. However, research shows that serious games do not always fulfill all the targeted pedagogical objectives. Designing efficient and engaging serious games is a difficult and multidisciplinary process that requires a collaborative approach. Many design frameworks have been described, most of which are dedicated to the development of specific types of serious games and take the collaborative dimension into account only to a limited extent. Objective Our aim was to create a generic serious game design framework that could be adapted to all kinds of serious games and implemented in a collaborative web platform. Methods We combined the results of a literature review with our experience in serious game design and development to determine the basic building blocks of a collaborative design framework. We then organized these building blocks into categories and determined the features that a generic design framework should include. Finally, based on the paradigm of complex systems and systemic modelling, we created the co.LAB generic design framework and specifications to allow its implementation in a collaborative web platform. Results Based on a total of 10 existing design methodologies or frameworks, 23 building blocks were identified and represent the foundation of the co.LAB framework. These blocks were organized into 5 categories: “context and objectives,” “game design,” “mechanics,” “learning design,” and “assessment.” The arrangement by categories provides a structure that can be visualized in multiple and complementary ways. The classical view links game and learning design while other views offer project, systemic, and process visualizations. For the implementation of the co.LAB framework in a web platform, we propose to convert the building blocks into “cards.” Each card would constitute a collaborative working space for the design of the corresponding block. To make the framework adaptive, cards could be added, adapted, or removed according to the kind of serious game intended. Enhancing the visualization of relationships between cards should support a systemic implementation of the framework. Conclusions By offering a structured view of the fundamental design elements required to create serious games, the co.LAB framework can facilitate the design and development of such games by virtue of a collaborative, adaptive, and systemic approach. The different visualizations of the building blocks should allow for a shared understanding and a consistent approach throughout the design and development process. The implementation of the co.LAB framework in a collaborative web platform should now be performed and its actual usability and effectiveness tested.
Chapter
Full-text available
In this paper, we present a framework for the systematic classification of management research methods. We propose a scalable model based on six classification criteria: Type of Contribution (e.g. theory testing, context-specific description, etc.); Research Paradigm (e.g. positivist, interpretivist, etc.); Research Design (e.g. case study, survey, etc.); Time and Distance Lag Strategies (e.g. longitudinal, cross-cultural, etc.); Information Elaboration Methods (e.g. statistical analysis, text analysis, etc.); and Information Gathering and Preparation Methods (e.g. interviews, secondary data collection, etc.). We suggest that the six-level framework we present here has the potential to allow the systematic and granular monitoring of methodological trends in scholarly communities. In the final part of the paper, the framework is utilized to classify the 62 papers of the itAIS 2011 selected papers collection, and the methodological attitudes of the itAIS community are briefly commented.
Article
Full-text available
The paper motivates, presents, demonstrates in use, and evaluates a methodology for conducting design science (DS) research in information systems (IS). DS is of importance in a discipline oriented to the creation of successful artifacts. Several researchers have pioneered DS research in IS, yet over the past 15 years, little DS research has been done within the discipline. The lack of a methodology to serve as a commonly accepted framework for DS research and of a template for its presentation may have contributed to its slow adoption. The design science research methodology (DSRM) presented here incorporates principles, practices, and procedures required to carry out such research and meets three objectives: it is consistent with prior literature, it provides a nominal process model for doing DS research, and it provides a mental model for presenting and evaluating DS research in IS. The DS process includes six steps: problem identification and motivation, definition of the objectives for a solution, design and development, demonstration, evaluation, and communication. We demonstrate and evaluate the methodology by presenting four case studies in terms of the DSRM, including cases that present the design of a database to support health assessment methods, a software reuse measure, an Internet video telephony application, and an IS planning method. The designed methodology effectively satisfies the three objectives and has the potential to help aid the acceptance of DS research in the IS discipline.
Article
Full-text available
Interactive cognitive complexity theory suggests that simulation games are more effective than other instructional methods because they simultaneously engage trainees’ affective and cognitive processes (Tennyson & Jorczak, 2008). Meta-analytic techniques were used to examine the instructional effectiveness of computer-based simulation games relative to a comparison group (k= 65, N= 6,476). Consistent with theory, posttraining self-efficacy was 20% higher, declarative knowledge was 11% higher, procedural knowledge was 14% higher, and retention was 9% higher for trainees taught with simulation games, relative to a comparison group. However, the results provide strong evidence of publication bias in simulation games research. Characteristics of simulation games and the instructional context also moderated the effectiveness of simulation games. Trainees learned more, relative to a comparison group, when simulation games conveyed course material actively rather than passively, trainees could access the simulation game as many times as desired, and the simulation game was a supplement to other instructional methods rather than stand-alone instruction. However, trainees learned less from simulation games than comparison instructional methods when the instruction the comparison group received as a substitute for the simulation game actively engaged them in the learning experience.
Article
Full-text available
Learning theories describe how people learn. There is a large body of work concerning learning theories on which to draw, a valuable resource of which the domain of software engineering educational research has thus far not taken full advantage. In this chapter, we explore what role learning theories could play in software engineering education. We propose that learning theories can move the field of software engineering education forward by helping us to categorize, design, evaluate, and communicate about software engineering educational approaches. We demonstrate this by: (1) surveying a set of relevant learning theories, (2) presenting a categorization of common software engineering educational approaches in terms of learning theories, and (3) using one such approach (SimSE) as a case study to explore how learning theories can be used to improve existing approaches, design new approaches, and structure and guide the evaluation of an approach.
Article
Full-text available
This working paper has been thoroughly revised and superseded by two distinct articles. The first is a revised and peer-reviewed version of the original article: Okoli, Chitu (2015), A Guide to Conducting a Standalone Systematic Literature Review. Communications of the Association for Information Systems (37:43), November 2015, pp. 879-910. This article presents a methodology for conducting a systematic literature review with many examples from IS research and references to guides with further helpful details. The article is available from Google Scholar or from the author's website. The second extension article focuses on developing theory with literature reviews: Okoli, Chitu (2015), The View from Giants’ Shoulders: Developing Theory with Theory-Mining Systematic Literature Reviews. SSRN Working Paper Series, December 8, 2015. This article identifies theory-mining reviews, which are literature reviews that extract and synthesize theoretical concepts from the source primary studies. The article demonstrates by citation analysis that, in information systems research, this kind of literature review is more highly cited than other kinds of literature review. The article provides detailed guidelines to writing a high-quality theory-mining review.
Data
Two paradigms characterize much of the research in the Information Systems discipline: behavioral science and design science. The behavioral-science paradigm seeks to develop and verify theories that explain or predict human or organizational behavior. The design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts. Both paradigms are foundational to the IS discipline, positioned as it is at the confluence of people, organizations, and technology. Our objective is to describe the performance of design-science research in Information Systems via a concise conceptual framework and clear guidelines for understanding, executing, and evaluating the research. In the design-science paradigm, knowledge and understanding of a problem domain and its solution are achieved in the building and application of the designed artifact. Three recent exemplars in the research literature are used to demonstrate the application of these guidelines. We conclude with an analysis of the challenges of performing high-quality design-science research in the context of the broader IS community.
Article
This paper contributes to the rethinking project management agenda in relation to the information technology (IT) sector. Our analysis of the evolution of thinking and practice among leading IT project managers across four countries elicits nine principles and four personal qualities that constitute the core of a mindset that facilitates rethinking the practice of IT project management. We compare this with the Rethinking Project Management research agenda [Winter M, Smith C, Morris P, Cicmil S. Directions for future research in Project management: the main findings of a UK government-funded research network. Int J Project Manage 2006;24(8):638-649.]. Our contribution is to (1) validate the directions defined in that agenda; (2) identify elements not incorporated in it and (3) provide examples that crystallise the agenda for the domain of IT project management.
Article
A highly important part of software engineering education is requirements collection and analysis which is one of the initial stages of the Database Application Lifecycle and arguably the most important stage of the Software Development Lifecycle. No other conceptual work is as difficult to rectify at a later stage or as damaging to the overall system if performed incorrectly. As software engineering is a field with a reputation for producing graduates who are inappropriately prepared for applying their skills in real life software engineering scenarios, it suggests that traditional educational techniques such as role-play, live-through case studies and paper-based case studies are insufficient preparation and that other approaches are required. To attempt to combat this problem we have developed a games-based learning application to teach requirements collection and analysis at tertiary education level as games-based learning is seen as a highly motivating, engaging form of media and is a rapidly expanding field. This paper will describe the evaluation of the requirements collection and analysis game particularly from a pedagogical perspective. The game will be compared to traditional methods of software engineering education using a pre-test/post-test, control group/experimental group design to assess if the game can act as a suitable supplement to traditional techniques and assess if it can potentially overcome shortcomings. The game will be evaluated in five separate experiments at tertiary education level.
Article
Social scientists often estimate models from correlational data, where the independent variable has not been exogenously manipulated; they also make implicit or explicit causal claims based on these models. When can these claims be made? We answer this question by first discussing design and estimation conditions under which model estimates can be interpreted, using the randomized experiment as the gold standard. We show how endogeneity – which includes omitted variables, omitted selection, simultaneity, common-method variance, and measurement error – renders estimates causally uninterpretable. Second, we present methods that allow researchers to test causal claims in situations where randomization is not possible or when causal interpretation could be confounded; these methods include fixed-effects panel, sample selection, instrumental variable, regression discontinuity, and difference-in-differences models. Third, we take stock of the methodological rigor with which causal claims are being made in a social sciences discipline by reviewing a representative sample of 110 articles on leadership published in the previous 10 years in top-tier journals. Our key finding is that researchers fail to address at least 66% and up to 90% of design and estimation conditions that make causal claims invalid. We conclude by offering 10 suggestions on how to improve non-experimental research.
Article
There have been few attempts to introduce frameworks that can help support tutors evaluate educational games and simulations that can be most effective in their particular learning context and subject area. The lack of a dedicated framework has produced a significant impediment for the uptake of games and simulations particularly in formal learning contexts. This paper addresses this shortcoming by introducing a four-dimensional framework for helping tutors to evaluate the potential of using games- and simulation-based learning in their practice, and to support more critical approaches to this form of games and simulations. The four-dimensional framework is applied to two examples from practice to test its efficacy and structure critical reflection upon practice.
Article
The increasing demand for software project managers in industry requires strategies for the development of management-related knowledge and skills of the current and future software workforce. Although several educational approaches help to develop the necessary skills in a university setting, few empirical studies are currently available to characterise and compare their effects.This paper presents the results of a twice replicated experiment that evaluates the learning effectiveness of using a process simulation model for educating computer science students in software project management. While the experimental group applied a System Dynamics simulation model, the control group used the well-known COCOMO model as a predictive tool for project planning.The results of each empirical study indicate that students using the simulation model gain a better understanding about typical behaviour patterns of software development projects. The combination of the results from the initial experiment and the two replications with meta-analysis techniques corroborates this finding. Additional analysis shows that the observed effect can mainly be attributed to the use of the simulation model in combination with a web-based role-play scenario. This finding is strongly supported by information gathered from the debriefing questionnaires of subjects in the experimental group. They consistently rated the simulation-based role-play scenario as a very useful approach for learning about issues in software project management.