Beyond Correctness: What Do Teachers Notice About Student Work Produced in Problem-Based Lessons?

Abstract

A key characteristic of problem-based lessons in mathematics is centering students' ideas in classroom discussions. Thus, teachers need to decide which pieces of student work (SW) to share with the class. Previous studies suggest that this decision is informed by teachers' attention to the features of SW. This study focuses on this preliminary stage, examining the aspects mathematics teachers noticed when they were asked to annotate SW from four problems-based lessons. We found that teachers noticed aspects of SW that were serviceable to the lesson goals more than those that were not serviceable. Interestingly, we also found they noticed non-normative aspects of SW more often than normative aspects. These results contribute to understanding teachers' multidimensional noticing in problem-based lessons.

Full text

Beyond correctness: What do teachers notice about student work produced in problem-
based lessons?
Gil Schwarts, Patricio Herbst, Irma Stevens, Amanda Brown
Abstract
A key characteristic of problem-based lessons in mathematics is centering students’ ideas in classroom
discussions. Thus, teachers need to decide which pieces of student work (SW) to share with the class.
Previous studies suggest that this decision is informed by teachers’ attention to the features of SW. This
study focuses on this preliminary stage, examining the aspects mathematics teachers noticed when they
were asked to annotate SW from four problems-based lessons. We found that teachers noticed aspects of
SW that were serviceable to the lesson goals more than those that were not serviceable. Interestingly, we
also found they noticed non-normative aspects of SW more often than normative aspects. These results
contribute to understanding teachers’ multidimensional noticing in problem-based lessons.
Objectives
A key task of mathematics teachers in problem-based lessons is to elicit, connect, and build on students’
ideas during classroom discussion to move the lesson toward the instructional goal (Stein et al., 2008). It
follows that teachers’ decision-making when selecting pieces of student work (SW) to share publicly is
pivotal for the success of the lesson. In this vein, the PSSM stated that: “Worthwhile tasks alone are not
sufficient for effective teaching. Teachers must also decide what aspects of a task to highlight, [and] how
to organize and orchestrate the work of the students” (NCTM, 2000, p. 19). While naming that teachers’
interactions with tasks is of high importance, the PSSM did not elaborate what aspects of tasks or SW are
those that are worth highlighting. Literature on problem-based instruction oftentimes focuses on the
works’ correctness, and debates whether using erroneous solutions prevents students’ sense-making
(Booth et al., 2013; Pillai et al., 2020). Other considerations for selecting SW to present on the board
include their representativeness (how many students have used similar strategies), ensuring equitable
participation, or their mathematical “sophistication” (Ayalon & Rubel, 2022; Stein et al., 2008). Although
these considerations go beyond the correct-incorrect dichotomy, they either do not attend to features of
SW but to social purposes (in the case of the former two), or they are ill-defined (in the latter). Thus,
much uncertainty still exists about the categories of perception teachers might use in their interpretations
of SW. This paper aims to show how additional categories, beyond correctness, can inform teachers’
decisions in problem-based lessons. We do so by employing a novel framework that specifies categories
of perception teachers use when commenting on SW: normativity and serviceability. Below, we define
these categories based on a broader theoretical framing.
Theoretical framework
This research draws on mathematics teachers’ practical rationality (Herbst & Chazan, 2011, 2012;
Chazan et al., 2016), a framework that describes teachers’ rationality using the notions of instructional

norms, instructional situations, and professional obligations. Instructional situations are “the distinct types
of problems used in a course of studies insofar as they appeal to particular norms for mathematical work”
(Herbst et al., 2021, p. 5). In the US high-school context, common instructional situations include
construction and proof in geometry and solving equations and doing word problems in algebra (Chazan et
al., 2012; Herbst et al., 2018). Instructional norms are expectations that teachers have about who should
do what in lessons – some of those norms are specific to instructional situations (e.g., expecting students
to use given objects to construct new objects in a construction situation). This framework has been
instrumental in identifying categories of perception that teachers use when they attend SW: normativity,
serviceability, and responsiveness (Boileau et al., 2020; Herbst et al., 2021; Herbst et al., submitted). This
paper focuses on the first two categories. Normativity alludes to aspects of SW that comply with teachers’
expectations for the instructional situation used to frame work on the problem. For example, if a problem
has been framed as a construction, a work in which the student sketched geometrical figures would be
less normative than work in which they used construction tools. Another category of perception is the
serviceability of the work, namely if the SW has features that can move the classroom discourse toward
the lesson’s instructional goal. For example, a sketch that includes elements that are part of the
instructional goal would be considered more serviceable (even though it is non-normative) than a
construction that does not give leverage for advancing toward the goal.
Previous works on this emerging framework related it to the ideas of teacher noticing (Herbst et al.,
submitted), showed how the categories tacitly inform teachers’ decisions considering different framings
of problems (Boileau et al., 2020), and suggested that in different stages of the lesson, normativity and
serviceability differ in importance (Stevens et al., 2022).
We consider normativity and serviceability as continua rather than as dichotomies. Thus, a SW may have
features that make it normative and others that make it non-normative (and the same holds for
serviceability). Whether teachers see it as more or less normative/serviceable involves their noticing.
Therefore, an additional framework guides our analysis: The “learning to notice” framework decomposes
noticing into attending, interpreting, and shaping (van Es & Sherin, 2002; 2021; see table 1 for definitions
and adaptations). Since our study participants were both attending and interpreting, hereafter we use
“noticing” to describe their impressions. We follow the revised definition of attending (emphasis added):
“attending involves not only looking closely at some features of the classroom environment, but also
disregarding other aspects of that environment” (van Es & Sherin, 2021, p. 20). Along those lines, we
ask:
What aspects of SW do teachers notice, in terms of normativity and serviceability?
In particular, do teachers notice some aspects more often than others? What aspects are disregarded?
Methods and data sources
We report on data collected during four cycles of StoryCircles that took place during 2021. StoryCircles
is a form of professional learning through collaboration that brings mathematics teachers together to
collaboratively anticipate a lesson through iterative phases of scripting, visualizing, and arguing, in
synchronous and asynchronous online interactions (Herbst & Milewski 2018, 2020). There were a total of
four cycles (each cycle was six-weeks long): two Algebra and two Geometry. The Algebra teachers
discussed lessons whose goal was to introduce inverse functions (the Softball Problem, cycle 1) and
solving difference equations (the Walkie Talkie Problem, cycle 2). The Geometry teachers discussed

lessons whose goal was to learn the Tangent Segments Theorem (the Tangent Circle Problem, cycle 1)
and the Midpoint of the Hypotenuse Theorem (the Pool Problem, cycle 2). Figure 1 presents the
problems and lessons’ goals.
This paper reports on asynchronous activities where teachers annotated sets of SW related to the specific
problem associated with each cycle of StoryCircles, using a collaborative annotation software
(Anotemos). The data corpus for this paper comprises 253 annotations on 36 pieces of SW (see Table 2).
The SW were either taken as is from implementations of the lessons in classrooms, or were adapted based
on literature about student conceptions.
Data analysis process was as follows:
First, we coded the SW as normative/non-normativity and serviceable/on-serviceable. Although we see
these categories as continua, we reduced the spectrum of possibilities to binary classifications to support
the data analysis and because we do not yet have measures of serviceability or normativity. That is, when
a SW had fundamentally more “non-serviceable” than serviceable aspects, it was coded as non-
serviceable (and similarly for the other three possibilities). To define normativity, we identified the
relevant instructional situation: In the geometry lessons these were construction in the beginning of the
lesson and then proof, and in the algebra lessons the situations were doing word problems and solving
equations. This means that SW that include, for example, sketches in geometry or tables in algebra (as
opposed to constructions or equations, respectively), were coded as non-normative. The serviceability of
SW was based on each lesson’s instructional goal (see Figure 1). This stage ensured that normative and
non-normative aspects had similar frequency, as did serviceability and non-serviceability. We then coded
participants’ annotations, applying both bottom-up and top-down analysis (we report here on the latter).
To answer the research question, we used the codes: normativity, non-normativity, serviceability, non-
serviceability, coding each annotation with 0-4 of them (multiple codes could be applied to the same
annotation). Counting the codes we applied assisted us in identifying patterns.
Results
Among other differences across the constructs, we identified that:
1. Non-normativity was noticed more frequently than normativity (in particular in the algebra
lessons).
2. Serviceability was noticed more frequently than non-serviceability.
Figure 2 shows the frequency (in percentages) of the four codes across annotations of SW in the four
lessons. Below, we demonstrate the findings with illustrative annotations and suggest possible
explanations. For ease of reading, we focus on one cycle/lesson – the softball problem in algebra (see
Figure 1), which was used to introduce inverse functions. Within the instructional situation of doing word
problems, the canonical approach in US textbooks is translating the story into equations, and then solving
them. Thus, other representations that were included in SW such as graphs and tables were coded as non-
normative. Table 3 presents the SW and how they were coded.

1. Non-normativity was noticed more frequently than normativity
We found that teachers noticed non-normative aspects in SW more often than normative aspects,
particularly in algebra. For example, in the softball problem, three out of eight SW were coded normative,
but 13 comments (out of 36) noticed non-normativity while only one attended to normativity. When
noticing non-normativity, participants often raised considerations of efficiency, for example:
“The guess and check method is used a lot [...]. If you were using small numbers; yes. But with larger
numbers; it could take a while to figure out.” [SW6, see table 3].
[when commenting on SW4 that used table:]
● “This is a great visual. What if someone needed to get the answer very quickly? I am wondering
how long this might take.”
● “Legit approach; but we could use it to spark discussion about efficiency and using the equation
to solve”.
All three responses did not invalidate the strategies proposed in the SW, and pointed out their advantages
(“a great visual”). However, all of them implied or explicitly suggested that equations were the fastest,
most desirable strategy. These comments illustrate that although teachers are accepting alternative
representations in algebra lessons (Chazan et al., 2012), they still perceive equations as the preferable
strategy for word problems, and notice when this instructional norm is breached.
In contrast, when teachers commented on normative SW (SW1, SW7, SW8, see table 3), they mostly did
not mention the fact that students used equations, or the appropriateness of their approach. This trends
was similar across all four lessons, as Figure 2 shows.
Overall, this finding is in line with previous studies, including ours, suggesting that people tend to
overlook normative classroom behaviors, but react to breaches of norms (Herbst & Chazan, 2011).
2. Serviceability was noticed more frequently than non-serviceability
In contrast to the previous finding, here the presence of the construct was noticed by teachers, and they
disregarded details that did not assist in reaching the lesson goal. For example, the idea of switching
output and input that was included in SW2 was mentioned as useful for the lesson goal:
● This feels like an important [work] and a sort of informal start to the notion of inverse ... going
from "output" to "input"…
● Although there isn't much to go off of here I think it can be an enlightening example of another
way to remember the inverse process […]. With the inverse starting with y and tracing to the line
to find x could help this student connect switching x and y in the process that the teacher was
heading toward.
In contrast, there were very few comments that alluded to non-serviceability. A possible explanation for
this finding is that a main component of teachers’ tacit knowledge when observing SW before selecting
them for a whole-class discussion is their ability to find serviceable aspects even when it is challenging.

That is, we suggest that the annotation activity oriented teachers to fulfill a task of teaching they are well-
familiar with, and in which they disregard non-serviceable aspects.
Significance
This study is part of a broader effort to understand teachers’ considerations when noticing and selecting
SW, under a subject-specific lens that focuses on mathematical aspects. The emerging framework helped
us showing that teachers notice aspects of SW that go beyond correctness. In particular, we found that
teachers attend to normativity and serviceability, but in different ways: while lack of normativity is more
discernible than its presence, with serviceability the opposite was found true. Our future research will
examine these findings at scale while exploring the reasons for the identified difference. Another future
direction is scrutinizing the differences presented in Figure 2 between annotations of SW in algebra and
geometry.
Overall, this study enabled us to identify patterns and formulate conjectures. Its contribution to the
literature on teachers’ decision-making is in providing multidimensional subject-specific descriptions of
SW, and of teachers’ considerations when relating to them.
References
Ayalon, M., & Rubel, L. H. (2022). Selecting and sequencing for a whole-class discussion: Teachers’
considerations. The Journal of Mathematical Behavior, 66, 100958, 1-13.
https://doi.org/10.1016/j.jmathb.2022.100958
Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to
improve student learning in algebra: Differentiating between correct and incorrect examples. Learning
and Instruction, 25, 24-34.
Boileau, N., Herbst, P., & Milewski, A. (2020). Framing novel tasks as familiar: Some reasons why and a
resulting tension. Annual Meeting of the American Educational Research Association, San Francisco,
CA.
Chazan, D., Herbst, P., and Clark, L. (2016). Research on the Teaching of Mathematics: A Call to
Theorize the Role of Society and Schooling in Mathematics. In D. Gitomer and C. Bell (Eds.),
Handbook of research on teaching (5th ed., pp. 1039-1097). Washington, DC: AERA.
Chazan, D., Sela, H., & Herbst, P. (2012). Is the role of equations in the doing of word problems in school
algebra changing? Initial indications from teacher study groups. Cognition and Instruction, 30(1), 1-
38.
van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers’ interpretations of
classroom interactions. Journal of technology and teacher education, 10(4), 571-596.
van Es, E.A., & Sherin, M.G. (2021). Expanding on prior conceptualizations of teacher noticing. ZDM -
Mathematics Education, 53, 17–27 (2021). https://doi.org/10.1007/s11858-020-01211-4

Herbst, P., Boileau, N., & Gürsel, U. (2018). Examining the work of teaching geometry as a subject
specific phenomenon. In P. Herbst, U. Cheah, K. Jones, & P. Richard (Eds.), International
Perspectives on the Teaching and Learning of Geometry in Secondary Schools (pp. 87-110). Springer.
Herbst, P., Brown, A., Chazan, D., Boileau, N., & Stevens, I. (Submitted) Framing, responsiveness,
serviceability, and normativity: Categories of perception teachers use to relate to students’
mathematical work in problem-based lessons.
Herbst, P. and Chazan, D. (2011). Research on Practical Rationality: Studying the Justification of Actions
in Mathematics Teaching. The Mathematics Enthusiast, 8(3), 405-462.
Herbst, P., & Chazan, D. (2012). On the instructional triangle and sources of justification for
actions in mathematics teaching. ZDM, 44(5), 601–612. https://doi.org/10/ghqjcz
Herbst, P., Chazan, D., & Boileau, N. (2021). Framing, normativity, and serviceability in teachers’
decision making during lessons. Paper accepted for presentation at TSG 37, ICME 14, Shanghai,
China, 2021, later withdrawn. Available at https://hdl.handle.net/2027.42/171072
Herbst, P. & Milewski, A. (2018). What StoryCircles can do for mathematics teaching and teacher
education. In R. Zazkis and P. Herbst (Eds.), Scripting approaches in mathematics education:
Mathematical dialogues in research and practice (pp. 321-364). Springer.
Herbst, P. & Milewski, A. (2020). Using StoryCircles to inquire into the social and representational
infrastructure of lesson-centered teacher collaboration. In H. Borko & D. Potari (Eds.), Teachers of
Mathematics Working and Learning in Collaborative Groups: Proceedings from ICMI Study 25 (pp.
629-636). University of Lisbon.
National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics.
NCTM.
Pillai, R. M., Loehr, A. M., Yeo, D. J., Hong, M. K., & Fazio, L. K. (2020). Are there costs to using
incorrect worked examples in mathematics education?. Journal of Applied Research in Memory and
Cognition, 9(4), 519-531.
Stein, M. K., Engle, R., Smith, M., & Hughes, E. (2008). Orchestrating productive mathematical
discussions: Five practices for helping teachers move beyond show and tell. Mathematical Thinking
and Learning, 10(4), 313–340. https://doi.org/10.1080/10986060802229675
Stevens, I. E., Boileau, N., Bridges, S., Brown, A., & Herbst, P. (2022). Timing as a Complexity in
Teaching. Paper presented for Round Table at the 2022 Annual Meeting for the American Educational
Research Association (AERA). San Diego, CA.

Tables
Van Es & Sherin, 2021
Our adaptation
Attending
Identify noteworthy features of
classroom interactions
Disregard selected features of
classroom interactions
Identify noteworthy features of
student work
Disregard selected features of
student work
Interpreting
Use one’s knowledge and
experiences to make sense of
what is observed
Adopt a stance of inquiry
Use one’s knowledge and
experiences to make sense of
what is observed in student
work
Adopt a stance of inquiry
Shaping
Construct interactions and
contexts that provide access to
additional information
-
Table 1. The “Learning to Notice” revised framework (Van Es & Sherin, 2021) and its adaptations for
this study.
Prompt for annotating student work: How do you react to these pieces of student work?
Would using this work on the board serve an important purpose? Why or why not?
Are there parts of any of the student work that are unanticipated?
Would you bring up the work to the board as is?
Does this work make you think of any other work that might exist?
Algebra cycle 1
Algebra cycle 2
Geometry cycle 1
Geometry cycle 2
No. of teachers
5
6
11
8
No. of SW
8
9
10
9
Total no. of
annotations
36
28
130
59
Table 2. Instructions for the activities; number of teachers who participated in each activity; the number
of pieces of work; and the number of annotations for each activity.

No.
SW
Normative
Serviceable
SW1
Normative:
equation
Non-
serviceable: the
equation does
not move the
lesson towards
inverse function.
SW2
Non-normative:
not an equation
Serviceable:
arrows and text
indicate the
reverse direction
SW3
Non-normative:
not an equation
Non-
serviceable: text
and figure do
not relate to the
inverse function

SW4
Non-normative:
not an equation
Serviceable:
flipping the
columns of the
table
SW5
Non-normative:
The increments
by 1 result in
uninterpretable
values for
number of
players
Serviceable:
“solve for x”
SW6
Non-normative:
the student has
not created a
new equation
Non-
serviceable:
does not relate
to the inverse
function
SW7
Normative:
equation
Serviceable: the
student is
attempting to
switch and solve

SW8
Normative:
equation
Serviceable:
switch and solve
Table 3. Pieces of student work for the Softball Problem and their coding according to normativity and
serviceability.

Figures
The lesson/cycle
Task presented at the beginning of the lesson
Final frame of the lesson - the instructional
goal
Algebra Cycle 1 -
The Softball
Problem
Algebra Cycle 2 -
The Walkie Talkie
Problem

Geometry cycle 1 -
the Tangent Circle
Problem
Geometry cycle 2 -
the Pool Problem
Figure 1. The problems posed in each storyboarded lesson, and the instructional goals (© 2021,
BLINDED, used with permission)

Figure 2. Frequency of normative and serviceable aspects in teachers’ annotations of student work from
four problem-based lessons.