Content uploaded by Sergio Araneda
Author content
All content in this area was uploaded by Sergio Araneda on Jan 04, 2023
Content may be subject to copyright.
An Experiential Approach to
Test Design and Validation
Sergio Araneda
Dissertation Defense
University of Massachusetts Amherst
December 2022
The experience of
taking a Test and being
Assessed.
Origins in Game design
We still need valid
interpretation and uses!
An Experiential Approach
to Test Validation.
A phenomenological take in
Test Validation.
A test is a device with a purpose, that elicits experiences
that allow an observer to make inferences about an
individual.
An Assessment Technology is an ecosystem of devices
that allow a test to be used according to its purposes.
Test validity is the set of Situated Knowledges constructed
by a researcher and the communities affected by the uses
of the test about the appropriateness, adequacy and
preferability of future instantiations of a test, the
interpretations to be done with those instantiations; and
the procedures in which those instantiations will be used
for.
Test validation is the process by which a researcher builds
experiences that well-explain the validity of a test to the
communities affected by the uses of a test.
Outline of the
Dissertation
Dissertation // series of papers.
Three papers with literature reviews
and conceptual frameworks.
Two papers with Empirical studies.
A Conceptual Framework for Assessment
Experience in Educational Testing.
Author: Sergio Araneda, more authors?
Situating the Validity Argument: Lessons
from Feminist Epistemologies.
Author: Sergio Araneda
An Experiential Approach to Test
Validation.
Author: Sergio Araneda, more authors?
An Experiential Approach in Classroom
Assessment: Centering Test Design around
the Experiences Elicited by a Test.
Authors: Sergio Araneda, Lisa Keller
Tell me about your Experience: A Study
about Invalidity Hypotheses mentioned by
Users on Twitter
Authors: Sergio Araneda, Stephen Sireci
Conceptual Hypotheses
of the Dissertation
a) The experiential approach will provide
valuable information to improve test design
and validation practices above and beyond
information provided by traditional methods.
b) By implementing aspects of the experiential
approach, we can evaluate its feasibility and
practicality.
Theoretical foundations of the
Experiential Approach
A Conceptual Framework for Assessment
Experience in Educational Testing.
A Conceptual Framework for
Assessment Experience in
Educational Testing.
What is Experience?
(a) the more immediate and
instantaneous stream of information a
person can perceive.
(b) a narration about something that
happened, which represents an
integration of the raw stream of
information into a more cohesive and
curated narrative of what is happening or
just happened.
A Conceptual Framework for
Assessment Experience in
Educational Testing.
Dewey’s ideas about experience define the
conceptual framework around three principles:
•A principle of interaction, assuming that experience
happens in the interaction of a person with its
environment and cannot happen without this
interaction.
•A principle of continuity, assuming all experience is
affected by a continuum with past experiences and
expectations of future experiences.
•A principle of sense-making, in which we assume
that we make sense of the experiences, and in this
sense-making process, we can discover, re-discover
and interact in new ways.
A Conceptual Framework for
Assessment Experience in
Educational Testing.
Principle of Interaction
Cognitive
Emotional
Expressive
Fluent
Physical
The examinee interacts
with the test in a
cognitive level.
He/She/they react
using their cognitive
capacities.
The examinee interacts
with the test in an
emotional level.
He/She/they react
experiencing certain
types of emotions.
The examinee interacts
with the test in an
expressive level.
He/She/they react
experiencing the need
to express themselves
in an opinion or other
forms of engagement
with the test as a device.
The examinee interacts
with the test in
an fluent
way. He/She/they react
experiencing actions
that do not happen in a
conscious level.
The examinee interacts
with the test in
an
physical level.
He/She/they react
experiencing physical
sensations the need to
do certain body
movements.
Principle of continuity
Ante
Pre
Zero
Post
Long
-term
Experiences lived even
before knowing that the
examinee will take a
test.
Experiences lived in
anticipation to test a
test.
The experience itself of
taking a test.
The experiences after
the test and their results
were given.
The long
-term
experiences related with
consequences
associated with a test.
Principle of sense-making
Free
-stream
Immediate
Integrative
Co
-integrative
Dialogical
The examinee does not
make sense of the full
stream of information in
Experience.
The examinee barely
makes sense of the
stream of information in
Experience.
The examinee makes
sense of the stream of
information in
Experience, being able
to explain it as a story or
narration.
The examinee makes
sense of the stream of
information in
Experience, being able
to explain it as a story or
narration, including the
experiences of peers or
other people around the
examinee.
The examinee makes
sense of the stream of
information in
Experience, being able
to explain it as a story or
narration, including the
information and
Dialogue with the test
developer about the
assessment Experience.
Situating the Validity Argument: Lessons
from Feminist Epistemologies.
Situating the Validity Argument:
Lessons from Feminist
Epistemologies.
Why is necessary an
experiential approach?
The moral is simple: only partial perspective promises
objective vision. All Western cultural narratives about
objectivity are allegories of the ideologies governing
the relations of what we call mind and body, distance,
and responsibility. Feminist objectivity is about limited
location and situated knowledge, not about
transcendence and splitting of subject and object. It
allows us to become answerable for what we learn how
to see. (Haraway, 1988; p. 583)
De-located Knowledges Situated Knowledges
Universal rationality Ethnophilosophies
Common Language Heteroglossia
New organon Deconstruction
Unified field theory Oppositional positioning
World system Local knowledges
Master theory Webbed accounts
Validity as an Evaluative
Judgement
vs
Validity as a set of Situated
Knowledges
An ethnographic /
participatory /
phenomenological take for
Test Design and Validation
An Experiential Approach
to Test Validation.
An Experiential Approach
to Test Validation.
Is it possible to detect any source of
invalidity in a test in the extensive study
of user experiences?
Argument sketched as a
proof-by-contradiction in Formal Logic.
Reductio ad Absurdum argument
strategy.
An Experiential Approach
to Test Validation.
A phenomenological take in
Test Validation.
A test is a device with a purpose, that elicits experiences
that allow an observer to make inferences about an
individual.
An Assessment Technology is an ecosystem of devices
that allow a test to be used according to its purposes.
Test validity is the set of Situated Knowledges constructed
by a researcher and the communities affected by the uses
of the test about the appropriateness, adequacy and
preferability of future instantiations of a test, the
interpretations to be done with those instantiations; and
the procedures in which those instantiations will be used
for.
Test validation is the process by which a researcher builds
experiences that well-explain the validity of a test to the
communities affected by the uses of a test.
An Experiential Approach
to Test Validation.
The P.A.A.P. Minimum
Partiality → God-Trick
Appropriateness → Moral Philosophy.
Adequacy → Scientific correctness.
Preferability → Expectations
An Experiential Approach
to Test Validation.
General procedure
1. Explore All users and All user experience.
2. Create a list of invalidity hypotheses.
3. Prioritize hypotheses with the input of
users.
4. Refute or Fail to Refute the invalidity
Hypotheses.
5. Make the communities vote about the
issues that are not solved.
6. Sketch and Interpretation and Use
Argument (IUA) with the evidence obtained
in this process.
Empirical studies and
Proof-of-concepts for the
Experiential approach
An Experiential Approach in Classroom
Assessment: Centering Test Design
around the Experiences Elicited by a Test.
An Experiential Approach in
Classroom Assessment: Centering
Test Design around the Experiences
Elicited by a Test.
Proof-of-concept of the experiential
approach to test design.
Diagnostic Test for Professor Keller’s
Regression Analysis Class.
Introduction to the concept of
Experience Specification.
Experience
Specification
Research questions
1. What are the student’s preferences
regarding their assessment
experiences?
2. How congruent is the initial level of
skill and knowledge of Professor
Keller’s class relative to the content of
the Regression Class she will teach?
3. Did the Assessment Experience elicit
good experiences for the students?
Centering Test Design around
the Experiences Elicited by a
Test in Classroom Assessment.
Method
1. What are the student’s preferences
regarding their assessment
experiences?
•Question in the introductory survey
asking about their preferences in
terms of item format.
•Presenting two mock items with
different ways for doing the
diagnostic task and asking them for
written feedback about each item
format.
Centering Test Design around
the Experiences Elicited by a
Test in Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
“I feel that I understand the concepts and I could manage
them in class, but it is difficult to define them.”
“I prefer the second one [Multiple choice question] If some
terms in the definition can be overstriking which can make
me understand better.”
“I prefer the second way to answer the question [Multiple
choice question]. Although I can find the description of any
concept on Google or from other resources, sometimes I
do not necessarily fully understand what this concept
means. If it is a multiple-choice question, I must understand
the meanings of all the words in the question before I can
choose the correct answer. I think this kind of learning is
more effective.”
Centering Test Design around
the Experiences Elicited by a
Test in Classroom Assessment.
Method
2. How congruent is the initial level of
skill and knowledge of Professor
Keller’s class relative to the content of
the Regression Class she will teach?
▪Questions asking for what concepts
they DO recognize in a definition.
Centering Test Design around
the Experiences Elicited by a
Test in Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Method
3. Did the Assessment Experience elicit
good experiences for the students?
▪Final question at the end of the
diagnostic test asking for 5-star
rating and written feedback.
Centering Test Design around
the Experiences Elicited by a
Test in Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Centering Test Design
around the Experiences
Elicited by a Test in
Classroom Assessment.
Tell me about your Experience: A Study
about Invalidity Hypotheses mentioned
by Users on Twitter
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Proof-of-concept of the experiential
approach to test validation.
Study of experiences of Students taking
the test used for College Admission in
Chile, shared in Twitter.
Instance of the test done during the
pandemic (January 2021).
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Method
1. Topic identification on a Sample of the Tweets
(10%)
2. Creation of a list of invalidity hypotheses. Each
hypothesis correspond to one “device” and to
one of the P.A.A.P. questions.
3. Survey asking Students and members of the
public about priorities (1 to 5 Likert) to study
the invalidity hypotheses found.
4. Creation of overarching hypotheses (clusters of
invalidity hypotheses).
5. Sketch of a validity argument that can respond
to the concerns of the public.
6. Visualization adding a heatmap with the
public’s priorities.
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Tell me about your Experience: A
Study about Invalidity Hypotheses
mentioned by Users on Twitter
Discussion
Proof
-of-concept
Design
part
Proof
-of-concept
Validation part
The experiential
approach will provide
valuable information to
improve test design and
validation practices
above and beyond
information provided by
traditional methods.
Feedback on each of the stages
of the Assessment Experience
provided valuable insights, that
helped to construct a better test
and to interpret its results in a
better way. Classical testing does
not include student feedback in
its process.
The tweets provided interesting
insights. First, a map of what
devices are relevant for test
scores interpretation. Second,
what problems specific to that
year are necessary to check
before interpreting results.
Classical validation practices use
expert knowledge only to sketch
a validity argument.
By implementing aspects
of the experiential
approach, we can
evaluate its feasibility
and practicality.
It was feasible and practical. We
could find a way to apply the
Experiential Approach in a low
-
budget context. The mock items
served as an “invitational
argument” of the test.
Data on Twitter is public and
ready to be used. Surveying
participants of the public can be
difficult. Targeting students can
add extra complexity to the task.
Overall, it was feasible.
User Experience
Research.
Validity as Situated
Knowledges
Participatory Validation
Ethnographic take
Validity argument
designed around the
experience that elicits
across different
users/stakeholders.
•Conceptual framework for Assessment Experience
•Redefining the concepts of Test, Assessment Technology, Test Validity
and Test Validation from an Experiential/Phenomenological perspective.
•P.A.A.P. Minimum as guiding questions of Test Validation
•Experience Specification
•Emergent IUA from UXR
Innovations of the Experiential Approach
•A proposal for practices that close the distance between ourselves and
the public.
•More tools in the toolkit for “the Resistance.” Giving back agency to the
communities.
•Experiential Requirements: Engagement, Positive Assessment, Delightful
Assessment Experiences.
Why the Experiential Approach matters?
The End
Contact: sondaxius@gmail.com
An Experiential Approach to
Test Design and Validation
Sergio Araneda
Dissertation Defense
University of Massachusetts Amherst
December 2022