Content uploaded by Samira Alirezabeigi
Author content
All content in this area was uploaded by Samira Alirezabeigi on Jan 23, 2025
Content may be subject to copyright.
Content uploaded by Samira Alirezabeigi
Author content
All content in this area was uploaded by Samira Alirezabeigi on Jan 23, 2025
Content may be subject to copyright.
Critical Studies in Education
ISSN: (Print) (Online) Journal homepage: www.tandfonline.com/journals/rcse20
Evaluating datafied time: exploring the multiple
temporalities of teachers’ work with digital
assessment
Samira Alirezabeigi & Mathias Decuypere
To cite this article: Samira Alirezabeigi & Mathias Decuypere (23 Jan 2025): Evaluating datafied
time: exploring the multiple temporalities of teachers’ work with digital assessment, Critical
Studies in Education, DOI: 10.1080/17508487.2025.2451886
To link to this article: https://doi.org/10.1080/17508487.2025.2451886
Published online: 23 Jan 2025.
Submit your article to this journal
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=rcse20
Evaluating dataed time: exploring the multiple temporalities
of teachers’ work with digital assessment
Samira Alirezabeigi
a
and Mathias Decuypere
b
a
Methodology of Educational Sciences Research Group, KU Leuven, Leuven, Belgium;
b
Center for School
Development, Zurich University of Teacher Education, Zurich Switzerland
ABSTRACT
The recent emphasis of educational policies on the development of
AI tools for education is aimed to save teachers’ time of mundane
and repetitive tasks for meaningful pedagogical practices. These so-
called burdensome tasks (e.g. students’ roll-in presences, plagiarism
detection, feedback, and grading) which are incorporated within
platform infrastructures and delegated to algorithms are heavily
data-driven. This paper explores the question of how data-driven
automated processes of evaluation govern teachers’ evaluative
practices with respect to professional judgement and decision-
making by producing plural temporalities. This paper empirically
scrutinizes how teachers conduct evaluative practices through digi-
tal evaluation embedded in the Moodle platform, in which practices
of correction, grading and feedback giving are partially automated.
By analysing the screen recordings of ten digital tests from the
moment of genesis until the moment they are archived, as well as
a group discussion with teachers, we identify that one of the sig-
nicant performances of automated grading in student evaluation
is to condense teachers’ correction time and produce immediate
results. This, however, is only made possible through the extensive
data work that teachers need to do while making a test.
ARTICLE HISTORY
Received 31 January 2024
Accepted 7 January 2025
Keywords
temporality; automated
grading; digital assessment;
teacher’s work; automation
of work
1. Introduction
The integration of digital technologies and specifically Artificial Intelligence (AI) in
school-based assessments accelerated after the COVID-19 pandemic, with the ubiquitous
implementation of online learning platforms at schools in most western countries
(Ivanov, 2023). Since then, formative assessments which evaluate students’ learning
throughout the year, and summative assessments which evaluate students’ integral
achievement in a subject matter, have increasingly been conducted through these plat-
forms and embedded within learning management systems (LMS) (Bergviken Rensfeldt
& Player-Koro, 2024; Laursen & Jensen, 2024; Lingard et al., 2021). The main governing
rationalities behind the expansion of digital evaluation and using AI in assessment
policies is the alleged time-efficiency, real-time feedback, and objective validity that the
automation of grading would provide (Andrejevic, 2020; Blundell, 2021). The
CONTACT Samira Alirezabeigi samira.alirezabeigi@kuleuven.be Methodology of Educational Sciences Research
Group, KU Leuven, Belgium
CRITICAL STUDIES IN EDUCATION
https://doi.org/10.1080/17508487.2025.2451886
© 2025 Informa UK Limited, trading as Taylor & Francis Group
automation of grading varies from basic forms that make use of keyword detection in
student responses to more advanced forms that deploy Large Language Models (LLM) to
synthesize students’ responses and grade their exams (Chaudhry & Kazim, 2022;
Farazouli et al., 2024; Lingard et al., 2021).
However, the promise of liberating teachers from certain responsibilities through
automation is based on the idea of saving and reducing the cost of labor, which,
immediately and inevitably implies new arrangements of human and non-human labor
(Lingard et al., 2021; Rensfeldt & Rahm, 2022). More specifically, turning to digital
evaluation and automatic grading equally affects the organization of teachers’ work,
the relevance of their professional judgement and, more importantly, the educational
value and significance of evaluative practices as such (Bradbury, 2019; Wyatt-Smith et al.,
2019). In this respect, first of all, Bergviken Rensfeldt and Player-Koro’s (2024) ethno-
graphic study on online and platform-based school assessment shows how the main-
tenance and repairing work of making online assessment has become an additional task
for teachers. Secondly, in terms of time-efficiency, Farazouli (2024) argues that speeding
up the process of assessment through automation replaces or limits human functions
while relying on pattern recognition and isolated language features that overlook subtle
interactions of humor, irony, and sarcasm as well as innovative and unique ways of
thinking that students express, and that can only be detected, analyzed and understood
by their teacher. Thirdly, the automation of grading can lead to the devaluation of
teachers, for instance by introducing a depreciation of teachers’ expertise and profes-
sionalism. Lastly, the automation of grading also brings about the issue of transparency of
who decides for what: the platform decides for the teacher or the teacher decides how the
platform grades (Gallagher & Breines, 2021).
In these new evolutions, the emergence of new temporal orders for teachers’ work is
rendered increasingly obvious. That is, ‘teaching processes are initiated at various points,
inserted at one point and executed at other points, placing pressure on existing temporal
orders of teaching practice, suggesting a necessary process of recalibration’ (Gallagher &
Breines, 2021, p. 88). Connected to that, Bergviken Rensfeldt and Player-Koro (2024) also
discuss that working with platforms for evaluative practices has made teachers’ time and
space of work flexible. Nevertheless, they observed that in online evaluations, teachers
operate as time managers, synchronizing shared calendars not only with students, but
also with other colleagues in order to make their time visible for the collective. In
a similar vein, Gallagher and Breines (2021) argue that the reconfiguration of teachers’
work, which impacts existing dominant temporal orders (e.g. timetables), should be
investigated in terms of their potential for engendering different velocities of flows and
relations (also see Perrotta, 2023). What is crucial and common with school education in
all this, is how implementing digital evaluation with the presuming promise and logic of
optimizing teachers’ work, inevitably creates multi-faceted temporal implications for
how teachers organize their work (Holmes et al., 2022; Selwyn, 2022).
These aspects on the one hand influence how teachers’ professional expertise, judge-
ment and responsibility is manifested (or not) while routinely working with automation,
and on the other hand how a temporal governance (Landahl, 2020 of evaluative practices
comes into existence (cf. Timmis et al., 2016). For these reasons, educational researchers
(Holmes et al., 2022; Perrotta, 2023) call for empirical examination of the consequences
of automating educators’ work and their pedagogical decision-making. To date, however,
2S. ALIREZABEIGI AND M. DECUYPERE
empirical and critical studies on school education have mainly focused on the role of
digital technologies in large scale assessments through data infrastructures, standardized
testing and comparing systems, leaving the use of school-based digital evaluation with
relatively limited attention (Blundell, 2021; Lingard et al., 2021). For instance, Landahl
(2020) shows how the PISA tests structure a new temporal regime that is characterized by
‘high pace, simultaneous publication of results around the world and regular, recurrent
studies making the analysis and comparison of trends possible’ (Landahl, 2020, p. 636).
Nevertheless, how teachers’ time in its day-to-day nature is organized, how their role is
reconfigured and how a different regime of working with evaluation is installed, and what
forms of knowledge are favored through these time regimes, have remained understudied
(Cerratto Pargman et al., 2023; Selwyn et al., 2023).
This paper aims to address the above illustrated gaps and seeks to identify the
emerging temporalities of teachers’ work with respect to digital evaluation and auto-
mated grading. Moreover, it aims to trace the implications of these temporalities for
evaluation, namely through the different forms of knowledge and epistemic values that
digital evaluation and automatic grading puts forward and favors. It will do so by
meticulously looking into the journey of digital tests through which digital formative
tests are made and graded. We conclude how these (temporal) rearrangements for
teachers’ work end up governing their tasks through introducing the oscillating and
decompressing time of evaluation.
2. Tracing temporalities of digital evaluation through data journeys
From a methodological perspective and in order to understand the emerging temporal-
ities that digital evaluation installs for (and with) teachers, we have sought to capture the
journey of digital tests, inspired by the concept of data journeys (Bates et al., 2016). Data
journey is relevant in this context, as one of the main shifts when moving to digital
evaluation, is the production of a wide range of digital data, including log-file data, test
data analysis, and feedback, which have all led to the growing datafication of school-
based assessments. Datafication, defined as ‘the transformation of different aspects of
education into digital data’ (Williamson, 2017, p. 5), is profoundly related to data
infrastructures which can be described as ‘the sociotechnical assemblages (i.e. people
and computing hardware/software) that laboriously translate human activity into data,
which are then able to be collected, stored, visualized and mediated between otherwise
disparate, diverse and disconnected actors and spaces (e.g. schools, districts, states,
systems)’ (Holloway & Lewis, 2022, p.4). Even though datafication is not limited to
evaluative practices, it has gained particular importance in evaluation. Bradbury (2019)
demonstrates how the datafication of education, introduced through tablet-based assess-
ments of young students, makes data a key part of schools’ presentation to the outside
world. She shows that datafication necessitates a change of how ‘normal’ classroom
practices are traditionally understood, favoring measurement over building meaningful
pedagogical relationships between the teacher and students. The shift in the nature of
teachers’ work is also shown in how their role as decision makers is increasingly delegated
to testing companies and online learning platforms, which are said to serve teachers as
a guide in educational decision-making through providing automatic, objective, and
valid scoring, saving their time and energy for more valuable educational tasks
CRITICAL STUDIES IN EDUCATION 3
(Lingard et al., 2021). As such, datafication and the development of data infrastructures
in school education provide the means to not only ‘fabricate supposedly objective
knowledge’ (Lewis & Hartong, 2021, p. 947), but also to ‘define the types of knowledge,
expertise and discourses that are valued and authorized within that space (i.e. epistemi-
cally) (p.949)’.
From this perspective, the term ‘journey’ makes critical educational studies sensitive to
concrete sites in which data practices take place, and it allows for tracing the situated
movements of data in connection to physical places – rather than assuming their abstract
and frictionless flow within spaces (see Lewis & Hartong, 2021). Whereas the term
‘journey’ generally evokes the spatial movement of data over its temporal movement
and texture, in this study, we are especially cognizant of the temporal aspect of the
journey, as we follow the sequence of events and actions in which digital tests are
centered. Here, digital tests are considered as programmable data objects that journey
and, thus, are mutable in the sense that they are adapted in different sites and contexts (cf.
Bates et al., 2016).
Using this approach, Howard and colleagues (2022) unpack the journey of educational
data as they go through ranges of transformations within educational policy documents
and illustrate how far these data can travel with the participation of different stake-
holders. They conclude that educational research needs to ‘study data journeys in relation
to actual school and classroom practices’, specifically in the field of automation and
AIED (Artificial Intelligence in Education) in order to reveal the full implications of data
work and use in practice (p.7). Responding to this call, we trace how the implementation
of digital evaluation occurs in one school in Flanders. Our focus on the life of digital tests,
from the moment of their creation up to the moment when they are archived, not only
allows us to understand the journey of actual school-based tests, but equally illuminates
how working with these tests temporally governs teachers’ work in specific ways.
We adopted the method of ‘data diaries’ to observe the journey of school-based digital
tests, akin to the work of Tkacz and colleagues (2021), who took up a narrative approach
to recount the story of what data did in the specific context of a research lab and over
a period of time. Defined as ‘a strategy of notation’ (p.4), the data diary method
repurposes traditional ethnographic methods to those that can capture data practices.
As such, ‘the role of humans recedes such that data can come into focus’ (ibid.).
Narrating data diaries should not be conflated with anthropomorphizing data, but should
instead be seen as an ethnographic method of understanding how data constitute space
and time.
3. Teachers’ work with digital tests
For this study, we focus on digital formative assessments at one Flemish (Belgium)
secondary school. The school has been using the Moodle platform as their Learning
Management System (LMS) for a period of seven years. Being one of the functionalities of
Moodle, enthusiastic teachers have already been experimenting with digital formative
assessments – which they call digital tests – for some time. The time of conducting the
study in 2022 coincided with a new decision of the school leadership to embark on the
integration of digital evaluation for summative assessments in the near future. Within
this context, digital evaluation was a hot topic of discussion among teachers during that
4S. ALIREZABEIGI AND M. DECUYPERE
time. For the purpose of this research, we asked ten teachers to make a diary of one
selected digital test. Teachers were purposefully selected according to the usage of digital
tests. The first author had a first meeting with each teacher individually, where she
elaborated the purpose and the method of the study, as well as the specificities of the
diary. The diary consisted, firstly, of teachers’ screen recordings whenever they engaged
with a different evaluative practice on their screen. The practices that were identified by
researchers were based on the chronological sequences for testing and included 1)
making the test, 2) performing the test – referring to the activities teachers did when
students take the test, 3) grading the test, and lastly 4) giving feedback on a test. The
length of screen recordings varied in different teachers’ practices from 20 minutes to
one hour for each practice.
These screen recordings, which aimed to show the life of a digital test on the screen,
were complemented by a written diary that contained a templated form for each
indicated moment concerning multiple aspects of teachers’ practices. The template
inquired into 1) teachers’ description of the task at hand as well as the offscreen material
they used, 2) the physical place and the time they chose to be engaged with digital tests, as
well as their motivation for choosing this time and place, and finally 3) if there were any
interruptions during their screen work. The template also asked teachers to describe how
these practices were conducted if they were not done on the screen. Whereas some
teachers dedicated the time to describe at length how their off-screen practices were
shaped, some other teachers limited their diary to short answers. These screen recordings
were subsequently annotated by the researcher, with dedicated analytical focus on how
teachers worked with the produced data and the sequence of their activities on the screen.
To complete the journey of digital tests and discuss about the missing aspects that
were not captured in the written and screen diaries, the analysis was afterwards presented
to teachers in one group conversation which lasted one hour and 30 minutes. This group
conversation was then transcribed and analyzed. To present the findings of our research,
we constructed the story of one digital test through descriptive vignettes. Even though
each of these vignettes is taken from our actual collected data, the described narrative is
not the story of one single digital test. These vignettes are chosen because they showcase
key features of working with digital tests, as we observed it in our dataset.
4. The spatiotemporal mutability of questions as data objects
Test T6CH3 is born as a renamed duplicate of an already existing test from the Moodle’s
question pool. The window of its temporal visibility is immediately decided at that very
moment. T6CH3 has a duration of 40 minutes, but for deciding when it needs to be visible
for which group of students, the teacher checks the digital agenda of her classes. T6CH3 is
coded to be visible if all conditions set by the teacher are met. The test already contains
questions as it is a duplicate of an old test. On the platform, the overview of the questions is
displayed as a mix of codes and tags. First of all, questions are visually categorized in terms
of the type of response required, such as a response with one word, or a response with
a more elaborate text. The visual categories provide a quick overview of types of questions in
the test. Next, the questions are tagged by means of a headline that describes what kind of
concept the question addresses, such as ‘ready knowledge’, ‘understanding’, or ‘adapting
techniques’. These headlines bundle similar concept questions together. Lastly, questions are
defined with keywords that summarize the question text. The keywords allow for a quick
CRITICAL STUDIES IN EDUCATION 5
insight into the question without having to see the full question (the text and its possible
responses). On the overview section, the teacher deletes some questions based on their
headlines and changes the headlines of other questions. Rather than working on the
questions’ text, the teacher first works with these categories, tags and headlines. The more
precise questions are tagged, the easier and smoother it gets to use a question. All questions
of T6CH3 belong to a question pool that is a database of all questions ever made by the
teachers of this subject matter in this school. When the teacher scrolls through the question
pool, the questions are categorized by topics as well, namely indigenous folk, or Brazilian
agriculture. The teacher programs 10 randomly selected questions from the category of
‘concept ready knowledge’ to be added to the test. The teacher searches between the
categories and subcategories of the question pool and adds a ‘self-reflection’ question to
T6CH3. Then, she commands 10 randomized questions to appear in the test from the
category ‘understanding’ in the question pool.
In our collected diaries and screen recordings, almost no test is made completely from
scratch. Either questions already exist in the question pool and teachers select and add
them to each new test, or teachers digitalize already existing paper questions of their
handbooks into Moodle’s question pool. Our data also show that questions existing in
other online learning environments can end up being added in Moodle’s question pool
by teachers. This makes the practice of making a test, a task of manually transferring
existing tests from other online learning platforms to the Moodle platform.
As we see in the vignette, each of these options requires a series of infrastructural task
without which the question cannot exist on the platform. Lingard et al. (2021) distinguish
between digitizing a test, which is a way of reproducing pen-and-paper tests on screen,
and digitalizing tests. Digitalizing involves rearticulating and redesigning the test in
terms of how it appears on the screen, how students engage with it, and the possibility
of integrating different multimedia resources into the design of the test. The above
vignette illustrates how digitalizing a test is practiced through the work of tagging,
categorizing, naming and ordering. As such, a digital question is not merely
a pedagogical medium of evaluation, but it has equally morphed into a data object.
Bates et al. (2016) introduce the concept of ‘data object’ by arguing that data can be
understood as a material-semiotic thing. By this, they highlight that the material condi-
tions of data production and practices within which this production takes place, as well as
the material properties of data objects such as durability, mutability, and size, impact
what data represent and how people encounter, use, and transform them. For example, as
seen in the vignette, the question’s data object contains different properties such as: 1)
visual tags that identify the type of the answer (a word, a number, one sentence or a text);
2) tags that categorize the type of knowledge or competence the question addresses,
namely understanding, self-reflection, and analysis; 3) naming questions’ title which
summarizes the topic of the question. Next to these properties that teachers manually
craft, the question data object also includes properties that are filled automatically, such
as the date the question was first created, and the number of times it was used in a test.
The above vignette precisely shows how meticulous, time-consuming, and laborious
the work of tagging, naming and categorizing questions is. As Selwyn (2021) discusses,
the efforts of teachers to produce ‘functioning’ and ‘useable’ data, requires serious
‘behind-the-scenes’ labor (see equally Perrotta, 2023). In this respect, the abovemen-
tioned distinction between digitizing versus digitalizing a test explicitly entails all the
data work of coding, tagging, and categorizing, which is a key character of digital tests. It
6S. ALIREZABEIGI AND M. DECUYPERE
shapes teachers’ evaluative work to a great extent, it makes questions appear in a specific
way on Moodle, and it makes the test viable for automatic grading. In this process, the
data work is so fundamental and infrastructural that, in our recordings, working on the
actual ‘text’ (i.e. the ‘content’) of questions only appears after this work is sorted out. This
signifies that the content-related work of creating questions is greatly interwoven with its
infrastructural data condition: content-related work cannot happen separated from, or
before, the data work is done.
Even though the practice of tagging, categorizing and ordering may seem
a straightforward and mundane practice, this practice is not only essential for digital
tests but equally, and more importantly, significantly demonstrates how, through data
work, evaluation is modelled into a form of digital and eventually automated evaluation
(cf. Edwards, 2015). Question data objects are made based on several categories such as
concrete knowledge constructs (‘understanding’, ‘adapting’ and ‘self-reflection’), cate-
gories of types of responses (one word answer, multiple-choice, and open question), and
finally the mode of grading (manually or automatically). Said differently, this vignette
shows how the data infrastructure of digital evaluation is practically developed through
the practice of making question data objects.
The digital test data infrastructure equally allows questions to be used over and over
again. The development of tags and categories are significant in that they help teachers
find and re-find questions faster and render questions re-usable over the years (and also
amongst different teachers). This means that questions can only move in time and space
due to teachers’ data work. Bates and colleagues (2016) elaborate on this aspect as the
mutability of data objects, which points to the repurposing, adaptation, and remixing of
data for different ends. The consequence of this laborious data work, however, is that
teachers are more encouraged to re-use once-made questions, instead of having to spend
each time a whole amount of their time for adding questions on Moodle from scratch, as
we observed in our video recordings. In this sense, once the time needed to fill all the data
entries of a question is spent, the questions will be reused, either by different teachers
and/or in different school years. Hence, a new test can be seen as a temporal reconfigura-
tion of what existed before in the question pool. This specifically also highlights the
temporality that is interwoven with the evaluation data infrastructure. This means that
the intensive data work for making questions will be compensated when teachers reuse
these questions.
5. The oscillating time of making the data infrastructure of digital
evaluation
As a copy of an old test, T6CH3 already entails several questions, among which, one that
needs to be answered with the use of an atlas. The question appears on the screen as a mix of
codes and texts, which is due to the fact that the teacher has a specific user interface view that
is different from what students eventually get to see. As the question is in the editing mode,
the teacher adds a subquestion: ‘What is the number of cold months?’. After all the tagging
and categorization work, this is the first time the full text of a question appears on the screen.
Responses to the questions are coded in front of the question text, in a line of code. Since the
teacher doesn’t know the code by heart, she searches in other questions for the code, copies
it and finally pastes and adds the correct answer for her newly added question. Ultimately,
CRITICAL STUDIES IN EDUCATION 7
the teacher turns the interface into the student view and checks how the question looks like
for the students. After having done so, she modifies the formulation of another already
existing question: ‘Write the [full] name of the country [in Dutch]’. Words in [] are added to
make this already existing question more specific and its responses more precise. She checks
the different responses that are already coded into the question as a correct response and
deletes ‘Z-Afrika’ [a shorthand for South Africa in Dutch]. Then, the teacher opens her
course module in Moodle in a new tab and opens a PDF sheet from the lesson. She spends
some time scrolling on the page and eventually she copies a piece of the text. Afterward, she
returns to the exam-making tab and adds a new category of ‘child marriage’, then chooses
the question response type as ‘open question’ and tags the question as ‘understanding’. In
the section of the question, the question name, its text, the general feedback and points are
shown. She pastes the part of the lesson sheet which she had copied in the general feedback
box and divides the text in three lines. She adds numbers to each line to make the text
appears as a correct response. Afterwards, she chooses keywords of each response to be
recognized by the platform and highlighted in a different color. She assigns 6 points for the
question, and on the question text, she pastes the sentence: ‘Child marriage is a deep-rooted
Ethiopian tradition. The communities even see it as a good way to protect a girl and her
family. Explain this’. She, then, chooses the size of the answer box with the number of lines.
Turning everything such as knowledge categories, topic of the question, the type of the
response, as well as the correct answer and the anticipation of students’ mistakes and the
general feedback into question data object changes the evaluative practices of teachers. In
this vignette, the formulation of questions become decisively crucial, as short answer
questions will be automatically corrected. As such, question data objects are not only
directed to students for the evaluation of their learning; they are also a means of
communication with the platform to be able to grade questions automatically. Here,
the teacher needs to safeguard that the responses are ‘comprehensible’ for the platform.
For automation to happen, responses need to be typed exactly as the teacher has coded in
Moodle, that is, in a uniform standardized way. In this sense, question data objects do not
merely contain an educational query; they equally incorporate all the possible responses.
That is, next to translating what is pedagogically relevant for the teacher to a language
form understandable for the machine, the teacher uses the features of the platform such
as coding her anticipation of student responses for short text questions and assigning
keywords for the responses of an open question to fasten and shorten the process of
grading the test. By anticipating spelling mistakes and different possible correct answers,
the teacher incorporates the predictive logics that automation operates with into her
evaluative practices (Perrotta, 2024).
Similar to Spina’s argument (Spina, 2021), we argue that the reconfigurations of
evaluative practices of teachers as well as incorporating data work, not only standardizes
the procedures of making test, but also the evaluation itself. This constellation shows not
only that an automatically gradable test necessitates the teacher’s time to make the
questions, but equally the extensive time that is needed in order to mediate the responses
between the student and the machine, as well as troubleshooting potential glitches and
inoperativeness (Selwyn, 2021).
As a result, what we clearly see in the vignette, is how particular types of questions and
forms of writing are favored. That is, questions which are more in line with the
standardizing and anticipating logics of automation are easier used than question types
which require students’ unique and subjective ways of writing and thinking. Reflecting
on the role of forms of knowledge, Perrotta (n.d.) highlights the distinction between
8S. ALIREZABEIGI AND M. DECUYPERE
‘knowledge’ and ‘understanding’. He argues that understanding goes beyond declarative
knowledge (knowing that) and procedural knowledge (knowing how) and it entails the
act of reaching a superior grasp of something. He continues to argue for the possibility of
novel forms of knowing and understanding to appear which are associated with auto-
mation and predictive governance (p. 76). In our case, the use of copy-pasting logic to
both formulate a question and its responses in the above vignette rather shows how the
standardization of forms of knowledge has a notable manifestation in evaluative practices
of teachers. As such, the process of coding and data work configures the evaluative work
of teachers into standardization and mediation, but they also configure the favored forms
of knowledge as well as redefinition of what form becomes valuable knowledge instead of
the emergence of novel forms of knowledge and understanding.
Our group discussion however, shed light on how teachers seek to preserve their
pedagogical and evaluative goals while contributing to the making of the data infra-
structure of digital evaluation. In case of short answer questions, a teacher mentioned
how she needs to secure a single solution approach to each question, while keeping the
difficulty level. ‘There are questions in which students need to put things in the right
chronological order, it seems like an easy question, but I can really evaluate their
understanding, because in fact it’s not very easy. And it also automatically corrects
everything. But for these questions, I definitely make sure that there is only one solution
and there is no discussion possible’.
Choosing the suitable form of question to evaluate students is also shown with respect
to non-textual ways of evaluation, such as drawing an electric circuit or drawing
a geographical phenomenon (which cannot easily be automatically graded). In these
cases, teachers pointed that the attempt to render these types of questions automatically
gradable lowers the bar of evaluation, as they need to reformulate the question and
suggest possible answers from which students can choose from. Another teacher adds
that ‘Whenever you have something in mind that you want to ask, you need to be quite
creative to fit that in the standardized types of questions that exist in Moodle, certainly if
you use figures and pictures on which students have to mark things. It is not always that
easy, it takes a lot of time and I think that the question sometimes changes a little bit,
because you want to fit it into the types of the questions that exist in the system’. Even
though teachers highlighted that this is certainly the effect of these types of the question,
which is not exclusive to digital evaluation, they unanimously agreed that the digital
format favors the use of automatically gradable questions (multiple choice and closed
questions), and hence changes what is being actually evaluated: ‘In digital evaluation you
make a test for it to be time-saving, otherwise why do it digitally?’ is the attestation of one
teacher, which was continued by another teacher: ‘That’s why if you engage with it
[digital format], you have to fight with this urge [making automatically gradable ques-
tions] and experiment on how to make questions evenly difficult’.
This immediately highlights the professional responsibility that is associated with
automation. Teachers refer to their evaluative responsibility by stating that ‘I am still
the teacher and have the responsibility to make sure that I ask a question that directs the
right level’, as one teacher for instance noted during the focus group. Moreover, discuss-
ing the possibility of automizing the grading of open questions through AI functions in
the future, a teacher stated that ‘as a Dutch teacher, the answer to questions is not only
about what students write, but it is also about how they write it. I want to read their
CRITICAL STUDIES IN EDUCATION 9
answers even twice by myself to know that. It is often even difficult to annotate these
things digitally. Students can use the digital format as a way of structuring their answers
and the only advantage for me is being able to read the typed answers better’. This
highlights that teachers’ work towards making the digital evaluation infrastructure and
exteriorizing judgment (either by means of AI or highlighting keywords in Moodle)
which inevitably comes with standardization of questions or expressions of thought,
influences teachers’ knowledge of students. As this quote expresses, gaining this knowl-
edge is one of the crucial parts of teacher’s evaluative practice beyond student perfor-
mance in terms of correctness of answers.
The process of making digital tests highlighted not only the reconfiguration of
teachers’ activities in terms of data work, but also showed how this time becomes an
oscillating time between standardization for the sake of automation and time-efficiency
and keeping their pedagogical goals of evaluation and their professional knowledge and
responsibility. As our analysis shows, achieving this is a challenge that not only requires
teachers’ time and experimentation with different forms of questions, but also
a resistance for not falling into the logics of automation without safeguarding evaluative
values. What we can also see, is how automation logics of standardization and anticipa-
tion can govern this oscillating time in a way that on the one hand teachers still seek to
gain contextualized knowledge from digital tests that are corrected by the Moodle plat-
form, and at the same time account for students’ individual and unique ways of expres-
sing their thoughts.
6. Decompressing the immediacy of automatic grading and feedback
Immediately after the test, T6CH3 is displayed as a list with two sections. A first
section contains questions that are automatically graded, whereas a second section
contains questions that need to be manually graded by the teacher. The teacher
clicks on a question that is already graded. By clicking on a question, the responses
of all students are displayed on top of each other in form of repetitive boxes. Each
box is framed with the name of the student, the question text and the type of
question (e.g. ‘Answer in one word’). Right below, the question reads: ‘What is the
last name of the Brazilian president during years 2018–2022?’. Below that line, the
response of the student is marked as ‘Bonsonaro’, which is scored with a little red
cross indicating a wrong answer. The right answer is shown then as general feed-
back, indicating two possible actions for the teacher: either ‘make a comment’ or
‘overwrite the grade’. Right below these options, the ‘history of the answer’ is
recorded in a table recording the time, the action, status, and the grade of the
question. The history tracks the hour at which the question was opened, the time
the answer was saved, and the moment the answer was finalized (and thus its
automatic grading are indicated as well). The teacher changes the grade from zero
to one point (the full point that can be scored being two). It does not take much
before she finishes reviewing all the already graded questions. The page moves very
fast, and the changing of grades remains the teacher’s decision. Sometimes the same
error is made by two students, but only one of the grades is changed. After
finishing the review, she opens an open question which is not graded and starts
scrolling down the answers. The page moves smoothly as she grades. Once in
10 S. ALIREZABEIGI AND M. DECUYPERE
a while, she adds comments such as ‘too vague’ or ‘not clear’ in the personal
comments. T6CH3 entails two open questions that are evaluated manually. The
grading is over after 12 minutes. She then displays the test results based on the
names of students, where the total time of finishing the test, their total performance
as well as the grade they received for each question is listed.
The pace of grading, and the time teachers spend on the actual reading of responses, is
noteworthy. Even though automatically graded questions still require review time of
teachers, the division of tasks between teachers and the platform’s automatic correction
functionalities, makes that the time of grading is considerably shorter than the time of
making tests. As we saw in the previous section, part of the grading practice is already
done during the making of the digital test and through the extensive data work of
incorporating the anticipation of different responses and allocating grades to these
possible responses on the Moodle platform. In this constellation, it can be argued that
digital evaluation only accelerates the time of grading as long as we do not account for the
time of test making and the elaborate and laborious data work of teachers in order to
automize grading. As such, we further argue that the speed of grading is built upon the
slowness of test-making.
Moreover, time-efficiency is only achieved by the continuous reworking of each
question data object. In this respect, during the group discussion, one teacher stated
that ‘digital evaluation is quite flexible when making a test. Sometimes, based on students’
responses, I realize that the question is not formulated very well and I can immediately
correct the formulation for future usage, already while correcting tests of this year’. Here,
the intention of reusing a question in the future is already put into operation in the
present as the teacher grades a test. This also happens in case of coding new possible
responses into the Moodle platform, in a way that automatic grading can adapt to the
judgement of the teacher if a response is right or wrong. ‘I code multiple answers that
I consider correct in Moodle, things that I know they will write differently, and I can
predict, like using an article or having typos. I update the list of possible answers while
grading, it goes very fast and for the future I don’t need to do that anymore’, said a Dutch
teacher during the group discussion. Here again, we trace the significance of adaptation
of automatic grading for future reuse and, most importantly, for the speed at which
automatic grading can be made reliable based on each teacher’s entries.
Automatic grading also means immediate results and feedback. Teachers attest that
through automatic feedback, students receive more feedback on a test as they program
correct responses, but also insert occasional notes on key points students need to reflect
upon beforehand (based on the anticipation of their responses). As such, this type of
feedback is no longer a matter of feeding back a personalized comment to students after
the test, but it is a general feeding forward, already premade in form of an answer sheet
immediately accessible to students. What can be argued here is the anticipatory logic that
automation incorporates, in the sense that immediate and automatic grading and feed-
back can happen through teachers’ anticipations of students’ performances.
One teacher mentions about the educational value of immediate feedback in her diary:
Students receive a first feedback moment immediately after submitting their test: they
can see for the closed questions whether they are correct (green), incorrect (red) or partly
correct (orange). Students can also view the expected answers. This is a huge learning
moment. Students check this immediately after taking the test and see immediately what
CRITICAL STUDIES IN EDUCATION 11
they got right and what they got wrong. The learning gain at this moment is very high
(Clara, geography teacher).
However, teachers mention that they incorporate multiple moments of giving
feedback to students, only one of which is the immediate result and feedback. One
English teacher mentioned in the diary ‘that apart from the automatic feedback
I immediately provide written feedback while grading. Afterwards, I give the stu-
dents time in class to read the feedback and ask questions orally’. We noticed
different strategies that each teacher uses in order to bring the individual and
immediate feedback into the class and make it part of the lesson. For example, in
the group discussion, a teacher explained how she de-automizes automatic grading,
while asking students to review Moodle’s automatic grading and argue why they
think the automatic grade should have been otherwise. In these occasions, teachers
create pedagogical moments during the lesson time to learn from a test by either
including students in the reviewing process of grading or by using the test analytics
to address common mistakes collectively. The collective feedback giving moments
highlight that the immediacy of receiving test results necessitates teachers to de-
compress and expand this immediacy during the lesson, by making pedagogical and
collective learning moments out of the test results and the feedback.
In our group discussion, teachers declared how the datafied history of each question
can provide clues to teacher’s holistic evaluation of students beyond their performance
and grade. In this respect, a history teacher stated that ‘If a student spends only six active
minutes on a test, I think that she might have written something very fast, might have not
cared about the results, or she might not be doing well. It gives me some information, but
not the real information’. Teachers further add that the interpretation of these data are
taken up by them to parents, the principal, and other colleagues if needed. As such, the
datafication of assessment goes beyond the testing itself, permeating into the pedagogical
order of school about care, conduct and overall wellbeing and performance of the
student.
In terms of temporality, we observe here that grading practices are intertwined with
test making practices in the sense that they are both based on the automation logics of
anticipation of mistakes and feed forwards before the test takes place (cf. Perrotta, n.d.).
As such, on the one hand the immediacy of grading is built on the laborious anticipatory
work of teachers while making tests. On the other hand, even though this immediacy
brings helpful insights to students about their performance, this general and immediate
feedback still requires to be ‘decompressed’ by the teacher and turned into a collective
pedagogical moment during the lesson. Here, it can be argued that digital evaluation
governs teachers’ time by installing automation logics of immediacy and anticipation to
which teachers respond with implementing different pedagogical strategies.
Decompressing time is one of those strategies when teachers dedicate the collective
time of the class to make sense of the immediate and automatic feedbacks and feedfor-
wards and turn them into a learning moment.
7. Concluding thoughts
Following a journey of digital tests, this paper demonstrated how digital evaluation
and automatic grading has evoked a substantial reconfiguration of teachers´
12 S. ALIREZABEIGI AND M. DECUYPERE
evaluative practices. Notably, by elaborating on different components of making and
grading digital tests, we showed how digital evaluation installs a temporal regime in
which teachers’ work is governed by automation logics of standardization, anticipa-
tion, and time-efficiency. These logics favor specific forms of evaluation which as
discussed, risk standardizing forms of students’ thoughts and understanding by
mainly favoring short answer questions and keyword-oriented responses
(Farazouli, 2024). This specifically becomes important since teachers’ time for
making tests becomes a oscillating time of benefiting the advantages of mutability
of question data objects and automatic grading while at the same time preserving
their pedagogical values and evaluative responsibilities.
As such, the promise of time-efficiency and relief from repetitive tasks cannot be
regarded as pedagogically valuable without accounting for teachers’ slow and continuous
data work that is done at the time of making digital tests. In this sense, our study shows
how teachers’ work is restructured and reshuffled, in the sense that they already need to
engage in practices of grading and feedforward while making tests. The implication of
this reshuffling is the risk of losing individual and contextual knowledge that teachers
gain from being directly engaged with students’ tests. Next to the slowness of making
tests, we also argued that the fastness of grading gains pedagogical relevance in light of
decompressing strategies that teachers integrate in their classroom practices to engage
with immediate feedback collectively.
This paper equally engages with assumptions and promises of automation through the
lens of automatic grading, by highlighting that the influence of automation on time-
efficiency cannot be reflected upon in isolation and in separation from other teaching
activities. For automation to function in the ensemble of evaluative practices of teachers,
not only teachers evaluative time needs to be restructured which makes them opt for
different oscillating and decompressing strategies. More importantly, teachers’ profes-
sional knowledge and judgement which makes them the primary responsible figures of
evaluation limits the scope in which automation can be integrated in a pedagogically
meaningful way in evaluation.
In this light, we would like to conclude this paper by reflecting on the implication
of this study on the more recent upsurge of GenAI (Generative Artificial
Intelligence) technologies, which has raised serious concerns from educational
scholars (Williamson et al., 2024; Ritzer et al., 2024). As we demonstrated in this
paper, the complex character of evaluative practices, which changes through, and
simultaneously resist, technology in multiple ways still holds teachers responsible as
the main (but not only) figures of students’ evaluation. As such, the laborious work
of reviewing technological output by teachers is not only unlikely to diminish. At
last, an intensive delegation of teacher’s professional judgement to these technolo-
gies strongly limits their in-depth, individual and contextual knowledge they
embody of students, which holds a central role in the relational character of
education.
Disclosure statement
No potential conflict of interest was reported by the author(s).
CRITICAL STUDIES IN EDUCATION 13
Funding
This work was supported by the This paper is funded by dtec.bw – Digitalization and Technology
Research Center of the Bundeswehr. dtec.bw is funded by the European Union – NextGenerationEU;
This research/paper is funded by dtec.bw – Digitalization and Technology Research Center of the
Bundeswehr. dtec.bw is funded by the European Union – NextGenerationEU.
Notes on contributors
Samira Ali Reza Beigi is post-doctoral researcher at the research groups of Methodology of
Educational Sciences, University of Leuven (Belgium). Her research interests include digitization of
education, qualitative research methodologies, sociomaterial approaches, and philosophy of education.
Mathias Decuypere is professor in education with a focus on school development and governance at
the Zurich University of Teacher Education (Switzerland) and is equally affiliated to KU Leuven
(Belgium). His main interests are situated at the digitization, datafication and platformization of
education; and how these evolutions enact distinct forms of education policy, governance, and practice.
ORCID
Samira Alirezabeigi http://orcid.org/0000-0003-2783-4438
Ethical statement
Sociaal Maatschappelijke Ethische Commissie [Social and Societal Ethics Committee] (SMEC).
Approval number: G-2021–4217-R3 (AMD).
References
Andrejevic, M. (2020). Automated media. Routledge.
Bates, J., Lin, Y.-W., & Goodale, P. (2016). Data journeys: Capturing the socio-material constitu-
tion of data objects and flows. Big Data & Society, 3(2), 2053951716654502. https://doi.org/10.
1177/2053951716654502
Bergviken Rensfeldt, A., & Player-Koro, C. (2024). Platformized teacher work - obstacles and
diffractions in assessment work practices. In A. Buch, Y. Lindberg, & T. C. Pargman (Eds.),
Framing futures in postdigital education: Critical concepts for data-driven practices (pp. 39–58).
Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-58622-4_3
Blundell, C. N. (2021). Teacher use of digital technologies for school-based assessment: A scoping
review. Assessment in Education Principles, Policy & Practice, 28(3), 279–300. https://doi.org/10.
1080/0969594X.2021.1929828
Bradbury, A. (2019). Datafied at four: The role of data in the ‘schoolification’ of early childhood
education in England. Learning, Media and Technology, 44(1), 7–21. https://doi.org/10.1080/
17439884.2018.1511577
Cerratto Pargman, T., Lindberg, Y., & Buch, A. (2023). Automation is coming! Exploring
future(s)-oriented methods in education. Postdigital Science & Education, 5(1), 171–194.
https://doi.org/10.1007/s42438-022-00349-6
Chaudhry, M. A., & Kazim, E. (2022). Artificial intelligence in education (AIEd): A high-level
academic and industry note 2021. AI and Ethics, 2(1), 157–165. https://doi.org/10.1007/s43681-
021-00074-z
Edwards, R. (2015). Knowledge infrastructures and the inscrutability of openness in education.
Learning, Media and Technology, 40(3), 251–264. https://doi.org/10.1080/17439884.2015.1006131
14 S. ALIREZABEIGI AND M. DECUYPERE
Farazouli, A. (2024). Automation and assessment: Exploring ethical issues of automated grading
systems from a relational ethics approach. In A. Buch, Y. Lindberg, & T. C. Pargman (Eds.),
Framing futures in postdigital education: Critical concepts for data-driven practices (pp.
209–226). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-58622-4_12
Farazouli, A., Cerratto-Pargman, T., Bolander-Laksov, K., & McGrath, C. (2024). Hello GPT!
Goodbye home examination? An exploratory study of AI chatbots impact on university
teachers’ assessment practices. Assessment & Evaluation in Higher Education, 49(3), 363–375.
https://doi.org/10.1080/02602938.2023.2241676
Gallagher, M., & Breines, M. (2021). Surfacing knowledge mobilities in higher education:
Reconfiguring the teacher function through automation. Learning, Media and Technology, 46
(1), 78–90. https://doi.org/10.1080/17439884.2021.1823411
Holloway, J., & Lewis, S. (2022). Governing teachers through datafication: Physical–virtual
hybridity and language interoperability in teacher accountability. Big Data & Society, 9(2).
https://doi.org/10.1177/20539517221137553
Holmes, W., Porayska-Pomsta, K., Holstein, K., Sutherland, E., Baker, T., Shum, S. B.,
Santos, O. C., Rodrigo, M. T., Cukurova, M., Bittencourt, I. I., & Koedinger, K. R. (2022).
Ethics of AI in education: Towards a community-wide framework. International Journal of
Artificial Intelligence in Education, 32(3), 504–526. https://doi.org/10.1007/s40593-021-00239-1
Howard, S. K., Swist, T., Gasevic, D., Bartimote, K., Knight, S., Gulson, K., Apps, T., Peloche, J.,
Hutchinson, N., & Selwyn, N. (2022). Educational data journeys: Where are we going, what are
we taking and making for AI?. Computers and Education: Artificial Intelligence, 3. https://doi.
org/10.1016/j.caeai.2022.100073
Ivanov, S. (2023). The dark side of artificial intelligence in higher education. The Service Industries
Journal, 43(15–16), 1055–1082. https://doi.org/10.1080/02642069.2023.2258799
Landahl, J. (2020). The PISA calendar: Temporal governance and international large-scale
assessments. Educational Philosophy and Theory, 52(6), 625–639. https://doi.org/10.1080/
00131857.2020.1731686
Laursen, R., & Jensen, R. (2024). The governance of teachers’ time allocation and data usage
through a learning management system: A biopolitical perspective. Critical Studies in
Education, 1–20. https://doi.org/10.1080/17508487.2024.2373239
Lewis, S., & Hartong, S. (2021). New shadow professionals and infrastructures around the datafied
school: Topological thinking as an analytical device. European Educational Research Journal, 21
(6), 946–960. https://doi.org/10.1177/14749041211007496
Lingard, B., Wyatt-Smith, C., & Heck, E. (2021). Digital disruption in teaching and Testing; assess-
ments, big data, and the transformation of schooling (First ed.). https://www.routledge.com/
Perrotta, C. (2023). Afterword: Platformed professional(itie)s and the ongoing digital transformation
of education. Tertium Comparationis, 29(1), 117–130. https://doi.org/10.31244/tc.2023.01.06
Perrotta, C. (2024). Plug-and-play Education; knowledge and learning in the age of platforms and
artificial intelligence.
Rensfeldt, A. B., & Rahm, L. (2022). Automating Teacher work? A history of the politics of
automation and artificial intelligence in education. Postdigital Science & Education, 5(1),
25–43. https://doi.org/10.1007/s42438-022-00344-x
Ritzer, G., Ryan, J. M., Hayes, S., Elliot, M., & Jandrić, P. (2024). McDonaldization and artificial
intelligence. Postdigital Science & Education, 6(4), 1320–1333. https://doi.org/10.1007/s42438-024-
00475-3
Selwyn, N. (2021). The human labour of school data: Exploring the production of digital data in schools.
Oxford Review of Education, 47(3), 353–368. https://doi.org/10.1080/03054985.2020.1835628
Selwyn, N. (2022). Less work for teacher? The ironies of automated decision-making in schools. In
S. Pink, M. Berg, D. Lupton, & M. Ruckenstein (Eds.), Everyday automation: Experiencing and
anticipating emerging technologies (pp. 73–86). Routledge.
Selwyn, N., Hillman, T., Bergviken Rensfeldt, A., & Perrotta, C. (2023). Digital technologies and
the automation of education — key questions and concerns. Postdigital Science & Education, 5
(1), 15–24. https://doi.org/10.1007/s42438-021-00263-3
CRITICAL STUDIES IN EDUCATION 15
Spina, N. (2021). Data culture and the organisation of teachers’ Work; an institutional ethnography
(First ed.). Routledge.
Timmis, S., Broadfoot, P., Sutherland, R., & Oldfield, A. (2016). Rethinking assessment in a digital
age: Opportunities, challenges and risks. British Educational Research Journal, 42(3), 454–476.
https://doi.org/10.1002/berj.3215
Tkacz, N., Henrique da Mata Martins, M., Porto de Albuquerque, J., Horita, F., & Dolif Neto, G.
(2021). Data diaries: A situated approach to the study of data. Big Data & Society, 8(1). https://
doi.org/10.1177/2053951721996036
Williamson, B. (2017). Big data in education: The digital future of learning, policy and practice. Sage
Publications Ltd: London.
Williamson, B., Molnar, A., & Boninger, F. (2024). Time for a pause: Without effective public
oversight, AI in Schools Will Do More Harm Than Good. National Education Policy Center.
http://nepc.colorado.edu
Wyatt-Smith, C., Lingard, B., & Heck, E. (2019). Digital learning assessments and big data:
Implications for teacher professionalism education, research and foresight: working papers,
25(1): 1–23. https://unesdoc.unesco.org/ark:/48223/pf0000370940
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of
research on artificial intelligence applications in higher education – where are the educators?
International Journal of Educational Technology in Higher Education, 16(1). https://doi.org/10.
1186/s41239-019-0171-0
16 S. ALIREZABEIGI AND M. DECUYPERE
