Content uploaded by Vu Minh Chieu
Author content
All content in this area was uploaded by Vu Minh Chieu on Apr 01, 2017
Content may be subject to copyright.
Content uploaded by Vu Minh Chieu
Author content
All content in this area was uploaded by Vu Minh Chieu on Dec 07, 2016
Content may be subject to copyright.
Can a Teaching Simulation Predict Novice
and Expert Teachers’ Decision-Making?
Vu Minh Chieu, Nicolas Boileau, Mollee Shultz, Patricio Herbst, and Amanda Milewski
University of Michigan - School of Education
610 East University Avenue, Ann Arbor, MI 48109-1259
vmchieu@umich.edu, nboilea@umich.edu, mollee@umich.edu, pgherbst@umich.edu, amilewsk@umich.edu
Abstract: The primary goal of this paper is to investigate whether a computer-based simulation
can detect the difference between novice and expert teachers’ decision-making in mathematics
instruction, which is complex in nature. The design of the simulation is grounded in a
sociological perspective on practical rationality of mathematics teaching. The simulation consists
of classroom scenarios, in the form of cartoon-based storyboards, with a series of decision
moments to simulate the instructional situation of doing proofs in geometry. Empirical data
helped verify and revise our design hypotheses/principles and showed that the simulation was
able to detect some differences between novice-teacher and expert-teacher decision-making.
Results of this study could inform the development of more advanced, computational models of
mathematics teachers’ decision-making.
Introduction
Teaching is complex in nature (Grossman et al., 2009; Lampert, 2010; Leinhardt & Ohlsson,
1990), because the teacher must attend to and interpret many simultaneous events, and
orchestrate various types of resources to make moment-to-moment teaching decisions. Chieu
and Herbst (2011) define a teaching simulation as a virtual environment that simulates the
practice of teaching, reducing some complexities but instantiating others. A teaching simulation
must enable practice from a first-person perspective. It could be used to assess a teacher
practitioner's knowledge state and to support practice-based learning. For example, Figure 1
illustrates the architecture of SimTeach, which can be described as follows: The apprentice
interacts with the Simulation User Interface, which captures their input (e.g., what action of
teaching to do next). The component of Apprentice Diagnosis and Modeling analyzes that input
and updates the apprentice’s current state of knowledge, using the Teaching Expertise
component. Then, the Educational Feedback component searches for relevant feedback from
the Teaching Expertise component, on the basis of their current state of knowledge. Finally, the
feedback is provided to the apprentice through the Simulation User Interface, and they continue
to play the simulation. This paper focuses on the assessment role of the simulation.
Existing teaching simulations (e.g., Dieker et al., 2013; Gibson, Aldrich, & Prensky, 2006;
Girod & Girod, 2008) have only demonstrated that they can enable and assess the practice of
basic teaching and classroom management skills, connecting general theoretical principles with
classroom practices (e.g., monitoring students’ behaviors). The development of SimTeach is
strongly grounded in recent research on mathematics teaching knowledge. Such development,
however, is very difficult because of the complexity of teaching expertise. Thus, a reasonable
first step, and the primary goal of this paper, is to investigate if a preliminary version of
SimTeach can detect the difference between novice- and expert-teacher decision-making,
before further development can be done to diagnose more fine-grained knowledge status (e.g.,
if they master a certain piece of knowledge or not).
Figure 1. Architecture of SimTeach, an intelligent teaching simulation (adapted from Luengo et
al., 2007 and Chieu & Herbst, 2011).
Theoretical Framework
The design of the first version of SimTeach is grounded in Herbst and Chazan’s (2011, 2012,
2015) account of the practical rationality
of mathematics teaching
. According to that account,
mathematics teachers’ instructional decisions can be understood as requiring mathematical
knowledge for teaching (MKT; Ball, Thames & Phelps, 2008), as based on their beliefs and
instructional goals (Schoenfeld, 2012), and as regulated by instructional norms
(i.e., routine and
tacitly-expected ways of working on routine mathematical tasks) and professional obligations
(to
their students, the class, the institutions that they work in, and to the discipline of mathematics),
in an instructional situation
— recurrent segments of classroom interaction in which students
work on routine mathematical tasks. Many of these norms are specific to the instructional
situation that the class is in (i.e., to the type of task assigned) and they regulate how student
work is exchanged for the teacher’s claims about the extent to which students have acquired the
knowledge and/or skills that the task was assigned to assess. See Figure 2 for examples of
norms of the situation of doing proofs.
Although these norms are default ways of behaving in instructional situations, they are
sometimes breached. Herbst and Chazan (2012) include knowledge, beliefs, and professional
obligations in their model of teacher decision-making as possible sources of justification for such
breaches. Below, we explain how these ideas influenced our hypotheses about how
instructional decisions of novice and expert teachers may differ and how we design the
simulation to test those hypotheses.
● The students identify the reason for each statement after it is made.
● The justification of a statement needs to be a previously studied theorem, definition,
postulate, or the given.
● Each of the reasons is stated in a conceptual register.
● After a statement in a proof is made and before the next statement is made, a reason
for the first statement is needed.
● The duration of the proof production is gauged in terms of the number of steps.
● Every single statement or reason is produced in a handful of seconds.
Figure 2. Examples of norms that regulate behavior in the instructional situation of doing proofs
in geometry, taken from a longer list provided by Herbst, Chen, Weiss, & González (2009); see
also Nachlieli & Herbst, (2009).
A Preliminary Version of SimTeach
In this study, we explore the hypothesis that novice teachers and expert teachers differ in
terms of their behavior towards norms and professional obligations. Specifically, we
conjecture that novice teachers would comply with instructional norms more often than
expert teachers.
To test the above hypotheses, we designed a computer-based teaching simulation in
which participants are asked to play the role of the teacher in a high school geometry class,
working on a proof problem, represented using a cartoon-based storyboard. The scenario
consists of parts that the participant is simply asked to view, which we will henceforth refer to as
stems
, and parts where participants are asked to indicate what they would do next, which we
will refer to as decision points
. The first step in designing this scenario was selecting a proof
problem for the class to work on. We chose what we hypothesized to be a normative proof
problem, chosen from commonly-used high school geometry resources, in order to cue
participants to the instructional situation. This was important because, if we chose a task that
most participants would perceive as novel (i.e., non-normative), they might breach norms that
they would not normally, simply because they would not have their common expectations about
how work on such proof problems should unfold. The next step in the design was to write a
story that we thought would comply with (at least) experienced high school geometry teachers’
expectations for how work on that problem might unfold. We did this by having the teacher
comply with all of the instructional norms described in Herbst, Chen, Weiss, & González (2009);
Figure 3 shows its beginning. Then, we identified points in that story where we hypothesized an
expert teacher might see reason to breach a norm (e.g., in order to comply with one of their
professional obligations) and mapped out what alternative decisions would be made at those
point; see Figure 4). At each decision point, we created four close-ended options: two options
that we hypothesize represent less productive
moves (by using the above conjecture), which we
hypothesize novice teachers would be more likely to choose — option 1 (Figure 5) and option 2
— and two options that we hypothesize represent more productive
moves (gain by using the
above conjecture), which we hypothesize expert teachers would be more likely to choose —
option 3 and option 4 (Figure 6). Last, we imagined how students would react to each of those
four moves (at each decision point) and used those to design branches of the scenario leading
to other decision points. The preliminary version of the simulation included 20 decision points,
although one would only be presented with a subset of these each time they play the simulation,
depending on the choices that they make.
Figure 3. Beginning of the simulation.
To provide us with insights into our design of the simulation (e.g., whether the simulation,
including the stems and the decision options, can capture essential aspects of teaching
practice), we selected four expert teachers from four different local high schools and one
pre-service teacher (PST) at our institution. We conducted four separate study sessions with
four experts. In each session, we asked the expert to complete at least two different branching
scenarios of the simulation. The PST sat next to the expert to facilitate the sharing of their
thinking. Before choosing a close-ended option at each decision point, the expert was asked to
explain and justify to the PST what they would do next. While they were making a choice, the
PST asked them to explain their analysis of the teaching event and the justification of their
choice. Because of the branching characteristic of the simulation, there are several decision
points that were not visited by any participant. So, we used a similar protocol to invite other two
expert teachers to navigate through the simulation, but we guided them to those specific
decision points and interviewed them about the design of the decision options at each of those
decision points.
Figure 4. The first decision point of the simulation.
Figure 5. Option 1 that we hypothesized as less productive
for the first decision point.
Figure 6. Option 4 that we hypothesized as more productive
for the first decision point.
Then, we used a constant comparative method (Glaser & Strauss, 1967; Fram, 2013;
Morse et al., 2009) to analyze videos and screencasts that captured the interaction between
each expert and the simulation and the PST. We used an iterative analysis process to look for
the similarities and differences of patterns of decision-making across decision points, scenarios,
and participants. This analysis showed that the simulation is able to capture many aspects of
teaching practice by the expert teachers described earlier. This analysis also helped us revise
the characteristic of the options (i.e., less productive
or more productive
) of three decision
points. A demo of the preliminary version of SimTeach is available at
https://www.lessonsketch.org/viewer.php?e=knqcMj9B0fS.
Methods
Research Questions
Our research questions, related to our hypotheses about how one might distinguish the
instructional decisions of novice and expert teachers stated above, are the following:
1. What differences exist between the patterns of novice teachers and expert teachers’
responses at decision points?
2. Are these patterns consistent with our hypothesis (i.e., that novice teachers tend to
choose less productive
options than expert teachers do)? In other words, can the
novice/expert status predict their performance in the simulation?
Procedure
To investigate the main hypothesis described above, we selected 30 novice teachers and 30
expert teachers from a national sample of 341 secondary mathematics teachers who had
previously completed a background questionnaire and MKT assessment (Herbst et al., 2017),
which we used to determine these two groups (“novice teachers” were teachers with less than
six years of geometry teaching experience and low MKT scores and “expert teachers” are
teachers more than five years of geometry teaching experience and high MKT scores). We
asked them to complete four different branching scenarios of the simulation so that they could
go through most decision points we designed.
Data Analysis
We used linear regression model to investigate whether teachers’ expertise could predict their
choice to make what we hypothesized to be more productive (or less productive) decisions. In
order to do this, we dichotomized the variables representing the choice that they made at each
decision option as 0 = less productive
move and 1 = more productive
move. We then calculated
the proportion of more productive choices chosen by each participant, and we name this
proportion the performance score
in the simulation. Then, we regressed that performance score
on the dichotomous variable indicating whether the participant was an expert teacher or a
novice teacher.
Results and Discussion
As certain choices at certain decision points were rarely chosen, only 13 out of the total 20
decision points were visited by enough participants to statistically compare the differences
between the novice teachers’ and expert teachers’ choices. The patterns of participants’ choice
at 10 out of those 13 (about 77%) decision points were somewhat similar to the one illustrated in
Figure 7, which shows that their choices varied for both novice teachers and expert teachers,
which is predictable as the literature points out that teachers’ decision-making is influenced by
many attributes (e.g., Schoenfeld, 2012; Westerman, 2010). Yet, there existed an important
pattern that was consistent with the hypothesis stated above: novice teachers were more
likely to choose less productive
options than expert teachers were, or in other words,
expert teachers were more likely to choose more productive
options than novice
teachers were. To strengthen that hypothesis further, we also run a linear regression model to
investigate the relationship between participants’ overall performance score in the simulation
and their expert / novice status. The model indicated that the expert / novice status positively
predicted participants’ performance score (effect size = 0.09, p < 0.001; see also Figure 8).
Figure 7. Distribution of choices (1, 2, 3, 4, from left to right) of a decision point in the middle of
the simulation (the above part represents novice teachers’ responses and the below part
represent expert teachers’).
Figure 8. Difference between novice teachers’ and expert teachers’ performance score.
This study, however, is still preliminary because it has not looked into the difference
between the chains of decisions that novice teachers and expert teachers made in a whole
scenario. For example, in some subsequent, exploratory analysis, we have found significant
correlations between expert teachers’ choices in some decision points and their choices in the
preceding ones (p < 0.05). Thus, it would be useful to model the temporal dimension
in
teachers’ decision-making, for instance, by using temporal / dynamic bayesian networks (see
Russell & Norvig, 2009), to detect if there is any difference in the chains of decisions made by
novice teachers and expert teachers.
Scientific Significance of the Study
This paper shows evidence that a simulation can be used to test and refine hypotheses about
differences between novice teachers’ and expert teachers’ decision-making in mathematics
instruction, from a sociological perspective on practical rationality (Herbst & Chazan, 2011,
2012, 2015). Results of this study could inform the development of more advanced,
computational models of mathematics teachers’ decision-making.
Acknowledgments
The work reported in this paper is supported by NSF grant DRL-1420102 to Patricio Herbst and
Vu Minh Chieu. Opinions expressed here are the sole responsibility of the authors and do not
necessarily reflect the views of the Foundation.
SimTeach has been designed and developed by using resources and tools of
LessonSketch
(https://www.lessonsketch.org), a multimedia platform that enables teachers and
other human service professionals to represent, examine, share, and discuss their own
practices, as well as the practices of other members of their profession.
References
Ball, D. L., Thames, M., & Phelps, G. (2008). Content knowledge for teaching: What makes it
special? Journal of Teacher Education, 59
(5), 389–407.
Chieu, V. M., & Herbst, P. (2011). Designing an intelligent teaching simulator for learning by
practicing in the practice of mathematics teaching. ZDM—The International Journal of
Mathematics Education, 43
, 105-117.
Dieker, L. A., Rodriguez, J. A., Lignugaris, B., Hynes, M. C., & Hughes, C. E. (2013). The
potential of simulated environments in teacher education: Current and future possibilities.
Teacher Education and Special Education: The Journal of the Teacher Education Division of
the Council for Exceptional Children
, DOI: 10.1177/0888406413512683.
Fram, S. (2013). The constant comparative analysis method outside of grounded theory. The
Qualitative Report, 18
, 1–25.
Gibson, D., Aldrich, C., & Prensky, M. (Eds.) (2006). Games and simulations in online learning.
Hershey, PA: Ideas Group.
Girod, M., Girod, G.R. (2008). Simulation and the need for practice in teacher preparation.
Journal of Technology and Teacher Education, 16
(3), 307–337.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for
qualitative research
. New York, NY: Aldine Publishing Company.
Grossman, P., Compton, C., Igra, D., Ronfeldt, M., Shahan, E., & Williamson, P. (2009).
Teaching practice: A cross-professional perspective. Teachers College Record, 111
(9),
2055–2100.
Herbst, P., Chazan, D., Dimmel, J. K., Erickson, A., Boileau, N., & Inah, K. (2017, April).
Understanding the rationality of teaching using multimedia questionnaires
. Paper to be
presented at the Research Pre-session of the Annual Meeting of the National Council of
Teachers of Mathematics, San Antonio, TX.
Herbst, P., & Chazan, D. (2015). Using multimedia scenarios delivered online to study
professional knowledge use in practice. International Journal of Research and Method in
Education, 38
(3), 272-287.
Herbst, P., & Chazan, D. (2012). On the instructional triangle and sources of justification for
actions in mathematics teaching. ZDM The International Journal of Mathematics Education,
44
(5), 601-612.
Herbst, P. & Chazan, D. (2011). Research on practical rationality: Studying the justification of
actions in mathematics teaching. The Mathematics Enthusiast, 8
(3), 405-462.
Herbst, P., Chen, C., Weiss, M., & González, G., with Nachlieli, T., Hamlin, M., & Brach, C.
(2009). “Doing proofs” in geometry classrooms. In M. Blanton, D. Stylianou, & E. Knuth
(Eds.), Teaching and learning of proof across the grades: A K-16 perspective
(pp. 250-268).
New York, NY: Routledge.
Lampert, M. (2010). Learning teaching in, from, and for practice: What do we mean? Journal of
Teacher Education, 61
(1-2), 21–34.
Leinhardt, G., & Ohlsson, S. (1990). Tutorials on the structure of tutoring from teachers. Journal
of Artificial Intelligence in Education, 2
, 21–46.
Luengo, V., Mufti-Alchawafa, D., & Vadcard, L. (2007). Design of adaptive surgery learning
environment with bayesian network. Proceedings of the 2007 International Technology,
Education and Development Conference
(CD publication, ISBN: 978-84-611-4517-1).
Valencia, Spain: International Association of Technology, Education and Development.
Morse, J. M., Stern, P. N., Stern, P. N., Faan, N., Corbin, J., Bowers, B., ... & Clarke, A. E.
(2009). Developing grounded theory: The second generation
. Routledge.
Nachlieli, T., & Herbst, P. with González, G. (2009). Seeing a colleague encourage a student to
make an assumption while proving: What teachers put to play in casting an episode of
geometry instruction. Journal for Research in Mathematics Education, 40
(4), 427-459.
Russell, S., & Norvig, P. (2009). Artificial intelligence: A modern approach
(3rd ed.). Upper
Saddle River, NJ: Prentice Hall.
Schoenfeld, A. H. (2012). “How we think: A theory of human-decision making, with a focus on
teaching”. In S. J. Cho (ed.), The Proceedings of the 12th International Congress on
Mathematical Education
, Seoul, Korea, DOI 10.1007/978-3-319-12688-3_16.
Westerman, D. A. (2010). Expert and novice teacher decision making. Journal of Teacher
Education, 42
(4), 292–305.
