Content uploaded by Jason Mcdonald
Author content
All content in this area was uploaded by Jason Mcdonald on Mar 14, 2023
Content may be subject to copyright.
34
“ARE THESE PEOPLE REAL?”: DESIGNING AND PLAYTESTING AN
ALTERNATIVE REALITY, EDUCATIONAL SIMULATION
Jason K. McDonald, Jonathan Balzotti, Melissa Franklin, Jessica Haws, & Jamin Rowan, Brigham Young University
In this design case, we report our design and playtest of a
form of alternative reality, educational simulation that we
call a playable case study (PCS). One of the features that
make our simulations unique is how they are designed to
implement a principle called This Is Not a Game, or TINAG,
meaning that the affordances we design into the simulation
suggest to students that the experience they are having is
real, in contrast to the way the artificial nature of the expe-
rience is highlighted in many computer games. In this case,
we describe some challenges we encountered in designing
a PCS to align with TINAG, along with how the situation in
which we play tested the simulation highlighted other ways
in which the principle of TINAG was challenging to achieve.
Jason K. McDonald is a professor in the department of
Instructional Psychology and Technology at Brigham Young
University.
Jonathan Balzotti is an associate professor in the English
department at Brigham Young University.
Melissa Franklin is a graduate student in the department of
Instructional Psychology and Technology at Brigham Young
University.
Jessica Haws is a graduate student in the English department at
Brigham Young University.
Jamin Rowan is an associate professor in the English department
at Brigham Young University.
INTRODUCTION
Since 2016, our interdisciplinary design team at Brigham
Young University has developed a type of educational
simulation we call a playable case study (PCS). Our team is
comprised of both faculty members and students, drawn
from the departments of English, Information Technology,
and Instructional Technology. Three faculty members (one
from each department) act as a core, persistent team, two of
which have been involved since the beginning (and one of
whom is this paper’s second author), and the other joined
in 2017 (this paper’s first author). Other faculty members
and students join the team on a temporary basis, often to
provide specific expertise, to serve as a liaison for a specific
course in which the PCS will be used, or, in the case of
students, to use it as internships or capstone project expe-
riences for their degree programs. The other authors of this
paper were involved for such reasons, as will be described as
the narrative unfolds.
A PCS is a web-based simulation that students experience
through their interaction with common tools including
email, video conferencing, chatbots, and technical docu-
ments. The framing device for a PCS is the conceit that stu-
dents have been hired as a member of a professional team
(e.g., an intern in a cybersecurity firm or a new employee at a
museum), and then role-play through a realistic scenario that
provides them opportunities to learn relevant knowledge
and practice applicable skills while experiencing some of
the affective responses that often accompany common
work situations. This occurs through a structure that blends
fictional elements with real-world assignments. Two of the
most common features are that students (a) communicate
with in-game characters who interact as would actual
supervisors, peers, clients, and so on; and (b) are assigned
professional tasks they complete as both in-class activities
and homework. For more background on the PCS format see
Giboney et al. (2021) and Winters et al. (2020).
One of the principles that guide our design of a PCS is the
“This Is Not a Game” ethos (TINAG). Borrowed from the
genre of alternate reality gaming (Flushman et al., 2015),
designing for TINAG means that the affordances we design
Copyright © 2023 by the International Journal of Designs for Learning,
a publication of the Association of Educational Communications and
Technology. (AECT). Permission to make digital or hard copies of portions of
this work for personal or classroom use is granted without fee provided that
the copies are not made or distributed for profit or commercial advantage
and that copies bear this notice and the full citation on the first page in print
or the first screen in digital media. Copyrights for components of this work
owned by others than IJDL or AECT must be honored. Abstracting with
credit is permitted.
https://doi.org/10.14434/ijdl.v14i1.34682
2023 | Volume 14, Issue 1 | Pages 34-42
34
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 35
into the simulation suggest to students that the experience
they are having is real, in contrast to the way the artificial
nature of the experience is highlighted in many computer
games. Certainly, students completing a PCS know they are
engaged in an educational simulation, but through the way
characters talk to them, or how tasks are framed, they are
encouraged to suspend their disbelief in the simulation’s
artificiality and fully embody the identity they are given. So,
for instance, when TINAG has been achieved the interface
will not instruct students to push a button to continue to the
next video, but rather will direct them to call their supervisor
whenever they are ready. Or, instead of requesting that they
complete a background survey to collect relevant informa-
tion about their readiness for the simulation, they will receive
an email from the human resources department inviting
them to complete an onboarding questionnaire.
Designing a PCS so that it aligns with TINAG is not always
easy, however. Since the simulation is, in fact, an artificial
environment there are sometimes realities of the genre
that only allow TINAG to be taken so far; as a result, it is
sometimes not clear how to frame a game mechanic or
technological constraint in terms of real-world activities and
situations. Further, TINAG is only one principle among many
that guides our design; most notably, as an educational
experience, a PCS aims towards certain learning outcomes
that sometimes are best facilitated through learning
activities that are clearly contrived. Nevertheless, we have
found TINAG to be a helpful principle in many situations,
and worth the effort to balance with the realities of our PCS
technology and the other principles that direct our efforts.
In this design case, we describe how we have attempted to
achieve this balance in the design of a particular PCS that
we call the Seneca simulation. We also highlight some of
the challenges we experienced in doing so and how we
attempted to overcome them. This includes our report of our
playtest of the simulation, where the nature of students’ prior
experience highlighted aspects of the simulation’s artificiality
in ways we did not expect.
BACKGROUND
A PCS is designed to nurture students’ development of
professional identities, and learning of professional practices,
within a particular domain of knowledge (Giboney et al.,
2021). The primary goal is to help students gain a more
realistic view of professional work and understand the
importance of situational constraints when communicating
within these contexts. In addition to exposing students to
professional contexts, the PCS simulations are also designed
to help develop career intentions by increasing self-efficacy
(I can do this) and career exploration (do I want to do this?).
The format is based in part on the broader educational
simulation genre (Gredler, 2004), case study instruction
(Heitzmann, 2008), and educational Alternate Reality Games
(or ARGS; Battles et al, 2011; Jagoda et al., 2015; Johnston
et al., 2012; Niemeyer et al., 2009). The PCS experience is
one of students “playing” through a scenario, or case study,
as a member of a professional team, completing tasks and
communicating with fictional characters through a realistic
interface (Hansen et al., 2017). Additionally, each simulation
contains a learning analytic platform that we included in an
attempt to present teachers with real-time data on students’
trajectories within the technology and their engagement
with in-game tasks. Students also write and reflect on their
experiences within the simulation. We intend that this ap-
proach gives teachers data points that will inform classroom
discussion, helping teachers decide what to emphasize and
how to respond to students’ experiences within the simu-
lation. We have used the PCS format to develop students’
interest in exploring a potential discipline as a possible
career (Giboney et al., 2021), help novices develop a sense
of self-efficacy as they start to explore a discipline (Winters
et al., 2020), improve writing skills (Balzotti et al., 2022), and
prepare them for important professional realities like how to
address ethical dilemmas (Neupane et al., 2021).
Critical to the students’ experience in a PCS is the use of
what we call a time-released narrative (a term that appears
to be unique to our work, although precedent for the
concept can be found in Kim et al., 2008, and Whitton et
al., 2008). Students access the web-based simulation either
through their personal devices (laptops or tablets being
common) or through computers provided through campus
computer labs (most students use their own devices). They
move the story forward through their interaction with other
characters and through interaction with other students in
the simulation. The challenges students face are released
as they complete different tasks and assignments given to
them by characters in the story. The events are triggered by
our PCS authoring tool as students send emails or reports to
other characters. The time-released narrative is embedded
in a web-based experience in which students adopt a
specific persona. These devices - the student persona and
the time-released narrative - are the primary mechanisms
through which we attempt to create a sense of TINAG.
The situation out of which students’ identity is presented
to them mimics those common to real work settings, and
the narrative immerses students in the simulated situation
through how tasks are represented, along with how they are
instructed to interact with the environment.
THE SENECA PCS
The Seneca PCS places students in the role of an intern at a
city government. Throughout the simulation, they prepare,
and eventually deliver, an oral presentation for the annual
budget meeting of a small town called Seneca. At the outset,
students access the simulation through a website designed
to simulate a dashboard for municipal employees, where
they choose to represent one of two departments: Library, or
Parks and Recreation. After making this choice the rest of the
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 36
PCS is organized into four “days”—each of which is centered
around a learning objective, with the learning material
customized based on the department for which they are
working:
DAY 1: Complete introductory tasks to become familiar with
the simulation
DAY 2: Learn about the audience (city council members)
DAY 3: Review and analyze research related to the depart-
ment’s proposed project
DAY 4: Present to the city council
Each day features 3-5 tasks that the students must complete.
Some tasks involve sending daily report emails to a supervi-
sor, reviewing city council members’ bios, reviewing project
timelines and community feedback, and developing a
handout for the final presentation. The tasks are designed to
help students work towards the outlined learning objectives
and move forward in the simulation.
Students may only work on one task at a time and may not
skip ahead. A checkmark appears beside each listed task
after it is completed, and, after the final task of the day is
finished, the simulation automatically enters the next day.
Though students could work through the entire simulation
in 1-3 hours, breaking the simulation into days creates
natural resting points for both students and instructors.
Instructors may assign students to work through the entire
simulation or assign a certain number of tasks or days to
complete as homework, depending on how they choose to
integrate the PCS into their course schedule.
The interface of the PCS is designed to engage students in
the simulated experience. The “dashboard” includes all the
information and applications that students will need as they
work on tasks. As Figure 1 illustrates, the dashboard allows
students to see their progress in the simulation, review what
tasks they have or will complete throughout the day, read
and send emails, review departments’ project initiatives,
study city council members’ bios, interact with characters
through video chat as directed, and stay apprised of city
happenings via the social media feed. These artificial aspects
of the simulation were all developed to mimic reality. For
example, the videos of the city councilors used actors, play-
ing the role of councilors in a scenario written specifically to
advance the PCS narrative and filmed in a conference-style
classroom located in the university’s Law School, which had
the appearance of common city council chambers.
Many aspects of the interface also mimic programs that stu-
dents regularly encounter (e.g., email forms and social media
feeds; see Figure 2). This is meant to increase the simulation’s
TINAG ethos and navigability. Because the design of the in-
terface does not feel entirely foreign to student participants,
they are more likely to feel immersed in the tasks and overall
experiences.
FIGURE 1. The Seneca PCS dashboard.
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 37
As previously stated, each task is designed to guide students
through the simulation and to direct their attention toward
the learning objectives. As one example, the detailed
breakdown of tasks that students must complete on “Day 2,”
as they appear in the simulation, are:
1. Read the email from the department director. Follow
Wesley’s guidance to begin analyzing your audience
[author note: Wesley is a city employee students work
with to learn about the city council members]. Find
ways to organize your presentation that will best
persuade each council member to vote for the [parks/
library] proposal.
2. Explore city council member bios. Read the council
members’ biographies on the “Council” tab. Take note
of each person’s background, agendas, and current
initiatives. Develop a sense of what each member will
want to hear during your presentation.
3. Read Marcus’s email. See when Marcus is available to
talk [author note: Marcus is a city council member who
provides additional background information about his
colleagues]. Follow his instructions to join his video call.
4. Meet with your committee’s council member represen-
tative. Call Marcus Rosales. Take note of any relevant
inside information he shares regarding the other city
council members.
5. Email the department director. Organize your notes
from the day and send an email to Wesley with key
insights you have learned about the council members.
Each task was created to both move the simulation forward
and help students learn and practice skills related to giving
oral presentations.
The culminating assessment of the Seneca PCS is an oral
presentation to the city council, that students record inside
the simulation. Students must use the information they
gathered about their audience and their department’s
project to compose an effective presentation. In the simula-
tion, the students join a “Livestream” city council meeting (a
pre-recorded video) and have three minutes to present their
argument for why their department should receive funding.
(Figure 3 shows the interface during the city council meeting
where students present.)
Additionally, after students record and submit their presenta-
tions, they view additional videos from city council members
that ask them two, unexpected questions to which they
must respond by recording another video (the questions are
the same for each student). The simulation allows students
to re-record their responses before submitting them, so they
have the option of retrying until they are confident in their
presentations. Both aspects of this final presentation—the
presentation and the responses—are designed to help
students increase awareness of their audience’s various
needs and expectations. Additionally, unlike live presenta-
tions, the simulation allows students to practice addressing
an audience in a low-stakes setting. Students can use this
exercise to boost their confidence and self-efficacy without
FIGURE 2. Examples of email and social feeds.
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 38
the heightened anxiety that many instructors at our universi-
ty have observed in them when they present.
DESIGN PROCESS AND CHALLENGES
The Seneca PCS was the seventh simulation we designed.
We began the project to address gaps many writing teachers
at our university had observed: students lacked confidence
and ability in preparing and delivering professional pre-
sentations. Because this challenge aligned so well with the
purpose of a PCS (as discussed in the Background section
of this paper), our design intent was to address a practical
problem in writing courses, using an educational technology
that had shown promise in similar settings.
In some ways, we considered the project a “return to our
roots.” Over time our designs had become more intricate
and complicated (adding in required teamwork, for instance,
or developing complex interfaces for students to create
different kinds of artifacts). On the one hand, we were
pleased with such efforts because they allowed us to push
the boundaries of the PCS format and provide students
with learning experiences that were difficult to achieve in
other ways. Yet, at the same time, as our designs became
more elaborate, we found ourselves making concessions on
a number of principles, including TINAG, to accommodate
additional features. In some of these cases, this meant that
students were not having as immersive an experience with
the simulation as we intended. Given the connection we had
found between immersion and the PCS’s affective outcomes
(for instance, the connection between students’ immersion
in the scenario and their interest in the continued study of
a discipline; see Giboney et al., 2021), we expected that this
trend would eventually impede some of the most unique
educational affordances the PCS offered. So, as we started
to plan the Seneca project, we took steps to ensure that our
structure and process would facilitate our ability to align the
simulation with the principle of TINAG.
One early decision was to keep the PCS design team small.
This was in part a pragmatic decision since one of our team
members who was usually involved was playing a lead role
on another project, and so only had time to take an advising
role on a new PCS. But we also assumed that a small team
could help us focus on our most essential processes and
goals. In contrast to the PCS we had previously completed
before the Seneca project, which had input from six faculty
members from three institutions, as well as dozens of
students, we limited our team to the three core faculty mem-
bers, an instructional design student, two creative writing
students, and two student software developers. Another way
we kept our team small was to only use actors from a local
talent agency for major speaking roles, with all video extras
or actors with one or two lines being taken from the existing
PCS team.
Further, some of the early decisions we made about the
simulation story were intended to provide a context in which
it would be easy to develop a sense of TINAG. For instance,
once we had decided that our learning outcomes would aim
FIGURE 3. City council meeting interface.
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 39
toward oral presentation skills, one of the faculty members
suggested that we place students in the role of new em-
ployees in a city department preparing a budget proposal
for the city council because he had a contact in a local city
government that could act as an advisor for the project. Our
instructional designer and writer met with this individual
(who provided background information and answered
questions but did not participate further in the design),
and much of what they learned became the foundation for
the simulation’s daily tasks, such as students’ assignments
to develop detailed descriptions about how their proposal
would impact residents, and what kind of feedback they had
received from other citizens through surveys and door-to-
door research. They were also able to use his input for minor
details that added additional touches of authenticity. One
example was how important it was for many cities to accom-
modate the sentimental value residents placed on different
locations or even specific objects like trees. In response, our
writer and designer included citizen feedback that made
these kinds of requests, to see if and how students consid-
ered such appeals in their eventual proposals.
Even with these steps, however, we still encountered chal-
lenges in aligning the Seneca PCS with TINAG. For example,
when students “interviewed” members of the city council,
or interacted with the council during the budget meeting,
they were expected to ask several questions. The limits of our
technology did not allow us to create a verbal interface for
students to speak their questions using their natural lan-
guage, so we resorted to an artificial, visual interaction where
they chose questions from a question bank. Also, when it
came time for students to record their presentations, we
were unable to develop a plausible reason why they were
supposedly live before the city council but were also being
instructed to record their presentations using the in-game
interface (both concessions were illustrated in Figure 3).
Yet given the many other TINAG affordances we were able
to successfully create, we ultimately judged that these two
concessions did not do irreparable damage to the overall
sense of realism towards which we strove.
As challenges arose during the development of the Seneca
PCS, we also used TINAG as part of our problem-solving
process. One example was related to our video production.
During the fact-finding phase of the simulation, students
“interviewed” one of two city council members to learn more
about dynamics within Seneca, along with what various
council members cared about when approving projects
(the interviews were actually pre-recorded video segments
that played in response to students choosing an audience
analysis question from a list provided in the user interface).
Each interviewee also offered advice on how to prepare
a successful presentation for the city council. The specific
council member that the students heard from depended on
whether they selected the library or the park project at the
beginning of the simulation. But even though the stories
each person told were unique to them as characters, the
actual information students were given about the city and
other council members was essentially the same. This was
intended to simplify instructors’ eventual grading process
by eliminating any need for them to keep track of different
success criteria that might arise if students were being
given different details by their informants in the alternative
storylines.
As we were adding the video clips into the simulation, we
discovered that a script for one of the council members had
been overlooked, and so had not been shot. Specifically,
this clip would have given a key piece of information about
one of the other members of the council. Consequently,
we had a complete set of information for one storyline,
but the other was missing some details students needed
to complete their final presentation. Our first response was
to return to the video production office and ask them to
shoot the missing video. Ultimately, however, this would not
have been as simple as it appeared. The video team we had
contracted with was a small unit housed in Brigham Young
University’s Center for Teaching and Learning. Given the
demand for their services, getting a slot on their schedule
was challenging. They were sympathetic to our position that
they had inadvertently missed the script and agreed to take
necessary actions to deliver us the product we were expect-
ing. However, since we were a relatively small project, with
smaller enrollments compared to many of the courses for
which they were shooting videos, the priority of their other
deadlines meant that we would have to wait for at least a
month (and likely more) before they could shoot the missing
script. As it turned out, we could have waited, since we did
not playtest the simulation for six more months. But at the
time we were hoping to conduct our first test much sooner,
so we began looking for alternatives.
As we considered other options, our primary criteria were
(a) the simplicity of the alternative and (b) whether it
maintained a sense of TINAG. TINAG remained important in
this case because we anticipated that many of the ways we
could respond might undermine students’ sense of the simu-
lation’s authenticity, which, inadvertently, might damage
their sense of trust in what they were learning. We judged
that it would be advantageous to find an in-game justifica-
tion for why some information was being delivered through
a mechanism other than the simulated “face-to-face” manner
in which students had learned everything else.
The most feasible alternative was to add a new email that
students received after they completed their interviews. To
maintain TINAG, we tied the email into how the students’
interview with the council member had ended; the council
member abruptly told students that he was out of time
and had to leave for his next meeting. Building on this, the
new email we wrote apologized for ending the interview
so quickly, then revealed that the council member was in
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 40
such a hurry he forgot to tell the student about one of his
colleagues. The message then proceeded to convey the
same information that was found in the original script. In
addition, even though we had all the videos for the other
branch of the simulation we decided to cut out the clips that
described this council member and add a similar email as
in the other branch. This decision was based on our belief
at the time that it would ultimately be easier for teachers if
the media in each narrative were consistent (e.g., if teachers
referred back to the information about city council members,
they did not have to remember that some students had read
an email while others had seen a video).
PREPARING FOR THE PCS PLAYTEST
During the Fall of 2021, we prepared to playtest the Seneca
PCS in a class at our campus. For previous simulations, we
had developed the product for a specific course, and so as
part of the design process, we asked the relevant instructors
if we could playtest the simulation using their class(es) as
a trial for us to gather evaluative data. In the case of the
Seneca PCS, we developed the simulation for a broader
purpose, specifically to fit in any course that had a writing
or presentation component (a skill many writing instructors
had observed students struggle with, as described earlier).
Since it could apply to multiple courses, to playtest it we
approached an instructor in the English department (the
last author of the paper), who taught a practicum course
where students put their skills as persuasive communicators
to use in projects for a local community. His reflection on
the course background, along with why he was willing to
playtest the simulation, is reproduced below:
“The Provo City Lab (PCL) is an internship-like course in
which students collaborate with city planners, elected
representatives, residents, and other stakeholders to improve
urban design, economic vitality, environmental responsibil-
ity, social equality, and overall livability of the community in
which our campus is situated. In Fall 2021, eighteen students
worked in teams to redesign five of Provo City’s gateways
(the areas where residents and visitors enter and exit the
city). A project like this one gives students opportunities to
discover how the competencies they have developed in
their various majors are useful beyond the university and to
further develop competencies that their courses have not
helped them acquire.”
“While our university has explicitly directed its resources to-
ward helping students become better writers, it has paid less
attention to helping them develop their ability to communi-
cate orally. In designing the curriculum for the PCL, I wanted
to provide students with the opportunities and training
to improve their oral communication skills that they may
not have received in other courses. To this end, I scheduled
three different presentations that students would give over
the course of the semester to Provo City committees. The
sequencing of these three presentations allowed students
to both receive feedback on their ideas throughout the
semester and to reflect on the quality of their oral presenta-
tions with an eye toward improving for the final and most
significant presentation to the City Council and Mayor.”
“In previous iterations of the PCL, students prepared for their
presentations to Provo City in relatively informal ways—the
curriculum did not contain any assignments or other types
of learning activities that students completed prior to their
presentations. The class would spend some time discussing
an upcoming presentation, designate a few students to give
the presentation, listen to and provide feedback on a dry
run of the presentation, and then gather for a debrief after
the presentation. While this process certainly gave students
opportunities to improve their oral communication compe-
tencies, it did not help them as thoroughly and thoughtfully
as it should have. When the Seneca PCS team approached
me about the possibility of piloting the simulation, I saw the
potential it had to help students improve their oral commu-
nication in more structured and deliberate ways.”
RESULTS OF THE PLAYTEST
At this point in our process, it was difficult for us to imagine
a more ideal curricular context for the Seneca PCS than the
PCL—a course in which students could prepare to present
to an actual city council by participating in a PCS organized
around preparing for and presenting to a simulated city
council. The reality of this, however, proved to be more
complex than we had hoped. The initial run-through of the
simulation was conducted towards the end of the semester -
a solely practical decision based on how quickly our software
developers could complete the project. By that point,
however, students had already prepared and given several
required presentations to a real city council. Consequently,
for at least some students the PCS seemed less like a learning
experience that would prepare them for the more difficult
assignment of a real presentation, and more like a simplified
review of skills they had already learned by doing. Combined
with some technical issues that arose during the playtest, our
evaluation results suggested that while some students were
engaged with the simulation, the overall experience was not
as effective in preparing them for the realities of professional
practice as we had anticipated, based on the results of our
prior work (as discussed earlier in this paper). This is illustrat-
ed through data taken from the evaluation report prepared
by two student researchers (the remaining authors of this
paper), which forms the basis of the account that follows.
After the instructor and student researchers introduced
the PCS, students were instructed to complete as much
of the simulation as they could during the current class
session and to finish the work—including recording the oral
presentation—before coming to the next class. However, a
technology failure resulted in only one student receiving the
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 41
simulated entrance email and being able to start the PCS. We
had anticipated the possibility of this type of challenge. We
had our software developers on hand (remotely) to trouble-
shoot. While the developers worked to solve the problem,
students began chatting and working on other assignments.
After about 15 minutes, but before the developers were able
to correct the problem, one student found a workaround
and was able to trigger the starting email by clicking out
of, and then back into, the simulation’s email environment.
However, by this time some of the other students seemed to
have lost interest and confidence in the simulation. About 10
minutes later, after the simulation had officially “started” and
students were anticipated to have made some noticeable
progress, the researchers observed that some students were
still talking to each other, and only casually completing the
simulation’s initial tasks. When questioned, some students
indicated little initial enthusiasm for working through the
simulation.
Once students did start the simulation, at least some of them
seemed to approach it with a skeptical attitude. Some initial
comments the student researchers recorded included:
“Are these people real? Is this a real email? It’s giving me
a little anxiety thinking this is a real person and they’re
expecting to talk to me.”
“My role isn’t really clear, I’m not sure why I was selected to
do this.”
“For each step, it’s not really clear how to proceed.”
Eventually, most students started working through the
simulation tasks as they had been directed. Overall, as they
proceeded through the initial tasks, many of them did seem
to engage with the simulation, although several others
continued to have questions about how to proceed. An
exception was encountered with a team of students who
were actively working on a different presentation for the real
city council, that would be delivered the following week. The
student researcher who observed this recorded, “the stu-
dents who are presenting next week were more distracted
and not as engaged in the PCS. Their concern for the real-life
context superseded the concern for the simulation.”
During the next class session, the instructor and one of
the student researchers conducted a focus group to elicit
student comments about their experience with the PCS.
Questions included, “What was the value of this simulation?”
What strategies did you learn? What did the simulation
offer?”
The focus group did provide some findings with which we
were pleased. Some students commented on authentic
touches in the simulation which drew them in, such as
one student who reported “the city council bios were very
compelling,” which made another class member wonder, “if
they were real people.” Another student described how she
learned the importance of “research and making a presenta-
tion specific to the interests of the audience,” which aligned
with one of our primary learning outcomes.
Several other students, however, commented on aspects
of the PCS that were not authentic to them. While they did
not directly compare the PCS to their prior experiences of
presenting to a real city council, the manner in which they
described completing the simulation suggested that they
did not find it as compelling because it was more contrived
than work in which they had already been involved. One stu-
dent noted how he “missed some elements that are inherent
in live presentations. [He] wanted to be forced to defend
his ideas because there was no pushback,” a judgment he
presumably based on his prior experience presenting to a
similar audience. And another mentioned how one of the
PCS tasks - to prepare a handout that summarized the main
points of their presentation - was not realistic because it was
not a requirement “for the [real city] presentation.”
Further evaluative insights were gathered from the students
who responded to the simulation’s exit survey (15 of the 18
students in the course responded to the survey). Students
were asked to respond to a series of statements and rate
their level of agreement or disagreement (5-point scale, 5
being strongly agree, 1 being strongly disagree).
“The interface helped me believe in the experience.” The
mean response was 4.07 out of 5.0.
“I felt responsible for the tasks I was given.” The mean
response was 3.40 out of 5.0.
“I believed my tasks were consequential.” The mean
response was 3.13 out of 5.0.
When asked how their confidence in presenting improved
through the simulation, eight of the 15 students reported
feeling that they were somewhat more confident, while
three told us there were no changes.
From these findings, we concluded two things. First, the
simulation itself (the interface, email, video interactions, and
so on) were perceived as authentic by many (although not
all) of the students. This was affirming, given some of the
challenges we had encountered in maintaining a sense of
TINAG during development. A slight majority of students
reported it being at least somewhat helpful to them in
preparing for future presentations.
Second, we discovered that the quality of the simulation
itself was not the only factor in whether TINAG was created
for the students. In our prior work, we used simulations with
students who were true novices in a subject (e.g., Winters
et al., 2020); in this case students already had some prior
experience with real presentations that had offered them
more authentic learning experiences than the PCS offered.
This was not only evident in the focus groups but seemed
IJDL | 2023 | Volume 14, Issue 1 | Pages 34-42 42
to be a factor in some of the observational data gathered,
and also related to why students perceived that their actions
were not consequential in the simulation’s outcomes (a
finding that was different from our evaluations of prior
simulations; Winters et al., 2020). In retrospect, this seems to
be an obvious and predictable result, however, during the
development and planning of our playtest, we did not suffi-
ciently consider it. While the possibility had been discussed,
we had concluded that students would appreciate the more
structured opportunity we provided them to practice before
they engaged in another high-stakes presentation. While
this seemed to have been the case for a few, for a number of
others they perceived the PCS to be somewhat of a distrac-
tion, and it was not very consequential to their learning. As a
result, we speculated that the quality of the simulation itself
was only part of what created a sense of TINAG for students.
The prior experiences, expectations, and skills that students
brought to the experience were as important.
Based on these findings, we are preparing to use the Seneca
PCS in the next offering of the same course. Rather than
using it at the end of the semester, we plan for students
to experience it during the initial weeks of class, where it
can truly be a preparation for their first real presentation.
Although our evaluative findings suggested possible
improvements we could make to the simulation, our plan
at the time of this writing is to use it unchanged, other
than completing critical bug fixes (such as the email failure
described earlier). Since our software development team
is shared with other projects, we do not have the ability to
make other improvements at this time.
CONCLUSION
In this design case, we have reported the results of our de-
sign and playtest of a playable case study that we called the
Seneca simulation. We highlighted some of the challenges
encountered in adhering to our ideal of creating a simulation
where students experienced a sense of TINAG, or a sense
that “This [simulation] Is Not a Game,” that, through earlier
work we had found to be useful in preparing students for
some of the aspects of real-world practice. These included
both development challenges, as well as how the circum-
stances in which we tested the PCS drew out the effects
of students’ prior experience on whether they were able to
accept the simulation as an authentic situation.
REFERENCES
Balzotti, J., Haw, K., Rogers, A., McDonald, J. K., & Baker, M.J. (2022).
Microcore: using online playable cases to increase student
engagement in online writing environments. The Journal of Applied
Instructional Design, 11(3). https://dx.doi.org/10.51869/113/bhrmb1
Battles, J., Glenn, V., & Shedd, L. (2011). Rethinking the library game:
Creating an alternate reality with social media. Journal of Web
Librarianship, 5(2), 114–131. https://doi.org/10.1080/19322909.201
1.569922
Flushman, T., Gondree, M., Peterson, Z. N. (2015, August). This is
not a game: Early observations on using alternate reality games
for teaching security concepts to first-year undergraduates. In 8th
Workshop on Cyber Security Experimentation and Test ({CSET} 15).
Berkley, CA: USENIX.
Giboney, J. S., McDonald, J. K., Balzotti, J., Hansen, D. L., Winters, D.
M., & Bonsignore, E. (2021). Increasing cybersecurity career interest
through playable case studies. TechTrends, 65(4), 496–510. https://
doi.org/10.1007/s11528-021-00585-w
Gredler, M. E. (2004). Games and simulations and their relationship
to learning. In D. H. Jonassen (Ed.), Handbook of research on
educational communications and technology (2nd ed., pp. 571–582).
Mahwah, NJ: Lawrence Erlbaum Associates Inc.
Hansen, D. L., Balzotti, J., Fine, L., & Ebeling, D. (2017). Microcore:
A playable case study for improving adolescents’ argumentative
writing in a workplace context. 50th Hawaii International Conference
on System Sciences.
Heitzmann, R. (2008). Case study instruction in teacher education:
Opportunity to develop students’ critical thinking, school smarts
and decision making. Education, 128(4), 523–542. https://doi.
org/10.1177/0098628320977267
Jagoda, P., Gilliam, M., McDonald, P., & Russell, C. (2015). Worlding
through play: Alternate reality games, large-scale learning, and
“The Source.” American Journal of Play, 8(1), 74-100. https://doi.
org/10.3390/educsci10040095
Johnston, J. D., Massey, A. P., & Marker-Hoffman, R. L. (2012).
Using an alternate reality game to increase physical activity
and decrease obesity risk of college students. Journal of
Diabetes Science and Technology, 6(4), 828–838. https://doi.
org/10.1177/193229681200600414
Kim, J. Y., Allen, J. P., & Lee, E. (2008). Alternate reality gaming.
Communications of the ACM, 51(2), 36-42. https://doi.
org/10.1145/1314215.1314222
Neupane, A., Gedris, K., McDonald, J. K., Hansen, D. L., & Balzotti, J.
(2021). Balancing competing goods: Design challenges associated
with complex learning. In B. Hokanson, M. Exter, A. Grincewicz, S.
Matthew, & A. A. Tawfik (Eds.), Learning: Design, engagement and
definition (pp. 181–190). Springer Nature Switzerland AG. https://
doi.org/10.1007/978-3-030-85078-4_14
Niemeyer, G., Garcia, A., & Naima, R. (2009, October). Black cloud:
patterns towards da future. In Proceedings of the 17th ACM
International Conference on Multimedia (pp. 1073–1082). New York,
NY: ACM. https://doi.org/10.1145/1631272.1631514
Whitton, N., Wilson, S., Jones, R., & Whitton, P. (2008, October).
Innovative induction with alternate reality games. In Proceedings of
the second European conference on game-based learning.
Winters, D. M., McDonald, J. K., Hansen, D. L., Johnson, T. W.,
Balzotti, J., Bonsignore, E., & Giboney, J. S. (2020). The playable case
study: An online simulation for skill and attitudinal learning. In B.
Hokanson, G. Clinton, A. A. Tawfik, A. Grincewicz, & M. Schmidt
(Eds.), Educational technology beyond content: A new focus for
learning (pp. 127–140). Springer Nature Switzerland AG. https://doi.
org/10.1007/978-3-030-37254-5_11
