MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES USING THINK-ALOUD TO EXPLORE PROBLEM-SOLVING PROCEDURES FOR ANATOMY STUDENTS

Abstract

This chapter addresses qualitative methods that enable educators to examine the learning processes of individuals. We describe methods used to make learning visible, specifically think-alouds and mind mapping. The first case describes the use of think-alouds to uncover problem-solving approaches in a multiple-choice test. The second case draws on mind maps, collaborative dialogue, and video documentation to enable elaboration of research topics.

Full text

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER
LEARNING PROCESSES
This chapter addresses qualitative methods that enable educators to examine the learning processes of individuals.
We describe methods used to make learning visible, specifically think-alouds and mind mapping. The first case
describes the use of think-alouds to uncover problem-solving approaches in a multiple-choice test. The second case
draws on mind maps, collaborative dialogue, and video documentation to enable elaboration of research topics.
USING THINK-ALOUD TO EXPLORE PROBLEM-SOLVING PROCEDURES FOR ANATOMY STUDENTS
Introduction
This case study presents the use of think-alouds as a powerful qualitative method to unravel student thought
processes as they complete multiple-choice assessments in anatomy. Multiple-choice assessments in the anatomical
sciences are often perceived to be targeting recall of facts and regurgitation of trivial details. Moving away from this
assumption requires the design of purposeful multiple-choice questions that focus on higher-order cognitive
functions as opposed to rote memorization. In order to develop such questions, we needed to first understand the
strategies that students use in solving multiple-choice questions. Our goal was to uncover patterns in the reasoning
process that students used when solving multiple-choice questions. Our study required participants to verbalize their
thought processes when solving six multiple-choice questions covering five key areas of anatomy. The multiple-
choice questions targeted three levels of cognitive functioning based on the ICE framework (Fostaty Young &
Wilson, 2000), which stands for Ideas, Connections, and Extensions. The ICE framework represents the gradual
progression and growth of the learner towards deeper understanding. Ideas are the fundamental, distinct pieces of
information that make up the building blocks of learning. Connections are the relationships that students can form

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
among discrete ideas, and associating new concepts to prior learning. Extensions constitute creating new learning
and applying knowledge to completely new and novel situations (Fostaty Young & Wilson, 2000).
Think-Alouds as our “Best Fit for Purpose” Method
We chose the think-aloud interview (Ericsson & Simon, 1984) as the initial approach to gathering data as it provided
an opportunity to obtain rich, deep, and descriptive information from the participant’s perceptions, meanings, and
misconceptions in solving multiple-choice questions. This was the most suitable methodology to help us answer the
following research question: What procedures do students use to solve multiple-choice anatomical questions?
Through think-alouds we were aiming to uncover the thinking patterns and strategies used by students and obtain
rich insights in the choices students make as they encounter those choices. Think-alouds can be a powerful vehicle
in making visible metacognitive processes that often remain hidden to learners. Studying metacognition has proved
to be difficult as researchers cannot directly infer cognitive processes from observing behaviour, and must take into
account the specific context and mediating processes in play (Durning et al., 2013).
Data Collection
We recruited participants from a second-year undergraduate anatomy course offered in the Winter semester and
conducted think-aloud interviews with 10 participants. This was carried out to develop the framework for our later
data collection using a survey. The one-on-one think-aloud interviews ranged from 40-60 minutes in length, and
were audio-recorded and transcribed. We used concurrent think-alouds which involves verbalizations taking place
during the task, rather than articulating reflections after the task has been completed as would be the case in
retrospective think-alouds. The focus was on the cognitive processes, rather than the final product, with the goal of
making these processes as explicit as possible during task performance. Hence, students were frequently reminded
that the purpose of the activity was not to evaluate whether they got the correct answer, but rather to articulate their
thought processes as they interpreted the question and arrived at a conclusion. We were looking for patterns in the
reasoning process that the participants used when solving the multiple-choice questions. Prior to the think-alouds, all
students were asked to complete a practice activity in order to better understand the depth of responses that we were
looking for in this study and to make students feel comfortable with this approach. The practice activity consisted of
reading the following scenario: “You open the door to your apartment and need to put the milk you just bought in
the fridge. What are the steps you take to complete this task?” Participants were asked to read the question and say

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
aloud everything they were thinking as they were reading the question. After this practice and feedback session,
participants were asked to read each multiple-choice question and follow a similar process. During the data
collection we were also considering how we might code the responses. We didn’t have standard questions, because
we didn’t know how students would respond but the goal was to unpack student thinking.
Navigating Meaning
In order to be able to make sense of the students’ thought processes and at the same time offer the ‘right’ level of
probing, we designed a rubric with prompts corresponding to the different strategies that might be adopted by the
students. We transcribed the think-alouds and followed the qualitative content analysis protocol (Patton, 1990) to
identify operators that students followed when working through the multiple-choice questions. These operators
explained the predominant reasoning processes used by the students. Once we became more clear and confident in
these emerging procedures, we clustered similar coded units together into categories. This process helped us to
compare segments of data and to confirm whether or not the data mapped onto existing categories (Charmaz,
2006). Next we compared our list of categories to the ICE Framework and mapped the categories to the appropriate
level in the ICE taxonomy. Responses to the individual think-aloud sessions were used to generate a survey that was
distributed across anatomy courses at Queen’s University. We analyzed and categorized reasoning processes that
were used by the 82 students who responded to our survey as well as those 10 students who participated in the think-
alouds. At the same time we explored the relationship amongst the level of the question, the strategies that were
being used, and the likeliness of students getting the answer correct.
Initially we had contemplated interviews as an alternative to think-alouds, however, we rejected that format as it
was important that the participants immediately verbalized their reaction to the question, rather than being given
sufficient time to prepare an answer. Instantaneous feedback is a unique feature of think alouds and it allowed us to
capture thinking processes as they occurred. It also enabled us to see the different approaches students draw upon.
Some focused on the connection between the content and their own knowledge: “what do I know about that”, while
others had a more holistic approach, “when I first read this means I need to choose what the question is demanding
as opposed to what I know”. Some students realized that the factual questions at the Ideas level were simple recall
questions. One of the questions we had at the ideas level asked the following: Fissures divide the lungs into,
followed by four options to choose from (a. lobules, b. lobes, c. alveolar sacs, and d. segments). Most students
instantaneously made comments similar to:

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
"I just like pick out certain words that kind of cue an answer, so what I noticed is I looked at the question and
as soon as I saw “fissures” then I’m like “it has to be lobes” like just because like we have learned that
fissures divide the lungs into lobes so I just automatically associate the word fissure with lobes"
Similarly, the other ideas-level questions asked students about trochlea and specifically which of the bones it was
a part of. The students responded by noting that they “remembered the bones”, “recalled the definition of trochle a”,
looked at the answer to refresh their memory, then they picked their answer out of the four options. At the same
time, as instructors, we were reflecting on the purpose of those multiple-choice questions and whether our goal
should have been to have students apply their learning rather than recall straight facts, then the questions would have
been designed quite differently.
We were curious to find out if their problem-solving strategies changed as they encountered connections-type
questions. Examples included: 1) The muscles that were most likely damaged if a 20-year-old patient cannot abduct
and medially rotate the thigh while running and climbing; and 2) Finding the length of the sarcomere when given the
length of the “I” and “A” bands. These types of case study questions are asking students to identify pertinent
information, understand the context, and associate actions of the muscles with their proper function and intricacies
of muscle contraction. As soon as the students started verbalizing their thought process, we were able to identify if
connections were made as the think aloud procedure allowed us to hear some of those associations amongst different
ideas. We could hear them say: "First I went through the motions of abduction and medial rotation, like pictured it
on my body” and,
"I took a couple of seconds to just be like okay don’t make an automatic assumption, then I went back and
actually read the actual sentence, read out the answers and then I thought of pictures that I saw in the textbook
and just knowing that the key words in this were like “I band A band and sarcomere”, then I associated the
words with the diagram and I could just see the diagram in my head”
Another question triggered more complex thinking patterns: Kyphosis affects the structure of vertebrae causing
forward rounding and abnormal curvature of the spine. Regarding the anatomy of the spine and associated axial
skeleton, what may be the functional implications of this bony disorder? Answer options were: a. shorter stature, b.
change in shape of the thoracic cavity, c. odd shaped stomach, and d. two of the above options. When students
started to extrapolate and make predictions, we could infer that this type of question was more demanding and at the

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
extensions-level. Many students remarked on the multiple thinking processes needed to answer the question, as
exemplified by this quotation:
“the idea is that I am going to have to use a level of higher thinking, I’m going to have to integrate concepts that
we have learned regarding the spine and apply it to a case that we haven’t seen before."
Students were trying to think of people they knew that had such a condition to fully understand what this meant, and
tried to predict what could happen in the future as a result of having this condition. This was highlighted by another
student when they stated that:
"It’s not something that we like may have explicitly went over, like what effects it has on the whole body, we
just know what kyphosis is and what it does to the spine, so you kind of have to like assume from that what
effects it has on the overall body, like we wouldn’t have talked about the stomach or anything, we just have like
pictures of the spine and which portions of the spine, so you would have to go above that and to see how that
would look from the outside."
The students’ answers revealed the procedures used and allowed us to associate these with a particular level of
thinking as defined by the ICE model. It was a challenging process to probe the students as we felt somewhat limited
in the amount of prompting we could do without distracting them from the act of thinking. We had to wait for what
students had to say and honour their voice, but at the same time we were trying to understand their interpretations
without making an assumption. In anticipation of some of these challenges, we created a rubric to help us identify
patterns in students’ thinking processes and also establish some clarity around what we were hearing (Table 1)
Table 1. Rubric Created to Capture Strategies and Prompts in Solving Multiple-Choice Questions
Clarifying
Restated or paraphrased the problem stem or one of
the multiple-choice options.
What did you think when you first
saw the question? What do you think
the question is asking?
Comparing
language of
options
Detected similarities and differences in the language
of two multiple-choice options.
What differences or similarities do
you see in how the options are
worded?
Correcting
Pointed out that they had been thinking incorrectly
about the problem earlier in the written think-aloud
and now see the correct way to think about the
problem.
What don’t you understand about
this question?

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
Delaying
Considered one of the multiple-choice options and
decide that it should not be eliminated. Rather, the
quality of that option should be evaluated later, after
the other multiple-choice options are considered.
What are the steps you are taking to
review the multiple-choice options?
Will you come back to that option
later?
Recognizing
Noted that a multiple-choice option is correct or
incorrect without any rationale.
Noted an option as correct or
incorrect.
Adding
Information
Provided more information about one of the multiple-
choice options, such as additional facts that were
omitted or corrections to incorrect statements (i.e.,
presented incorrectly to serve as distractors).
Is there certain relevant information
omitted from that option or
presented as incorrect?
Asking a
question
Asked a question about the problem stem or multiple-
choice options.
Clarifying the question stem or
multiple- choice options.
Checking
Explained why an option is correct or incorrect by
comparing the option with their knowledge or with
the data provided in the problem.
How does it fit in with what you
already know?
Predicting
As an early step in the written think-aloud, predicted
what they expected the answer to be (i.e., what
multiple-choice option they were looking for).
What do you anticipate the answer to
be?
Recalling
Retrieved basic facts or concepts from class, notes, or
the textbook (i.e., declarative knowledge).
What do you know about this topic?
Visual
presentation
Convert the written information to a visual that have
encountered previously
What images/pictures you create in
your mind connected to the words
you are reading?
Revisiting Think-Alouds as “Best Fit for Purpose” Approach
Amongst the challenges experienced with this approach were: students’ tendencies to get distracted and go off topic;
their inability to vocalize their thoughts or provide in-depth information; finding the “appropriate” level of
researcher prompting; as well as the labour-intensive analysis process. Think-aloud procedures can be quite time
consuming, and the participants need to have a model of what thinking aloud looks like. One of the limitations of
our study was not having an opportunity to meet with the students beforehand and properly mentor them on how to
“think-aloud”. The mentoring was done during the actual think-aloud and it might not have been sufficient for some
students. As a result, while we identified several strategies used by the students, there might have been other

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
thoughts or procedures that were not captured. Although, we used the practice activity, there was a tendency for
some participants to go off task, although with some gentle reminders we were able to redirect them to the task at
hand. It was difficult to find what was the “right” amount of probing as we noticed in the cases where we provided
more prompts that the participants changed their subsequent performance on the task which made us question the
accuracy of the cognitive processes being investigated. We had to be careful with the amount of information we
supplied and how much prompting to provide. At the same time, in cases where students were shy or distracted there
was a need to supply more information. Although it was critical to understand their thought process, we did not want
them to focus more on the verbal reports than the task at hand. We are mindful that the verbalized thoughts are
attributed to working memory and don’t always capture the automatic thoughts that might not necessarily be
vocalized. It could also be possible that some students do not have the conceptual framework or vocabulary for
describing the event in a way that helps us understand the processes at play. This poses another limitation for think-
alouds. The think-aloud interview approach promised to provide rich, deep, and descriptive data from students as it
allowed them to articulate their patterns of reasoning throughout the course of solving multiple-choice questions.
This can make visible metacognitive processes that often remain hidden to both the participants and researcher. In
our case, the benefit of using think-alouds outweighed the challenges as only through this methodology would we
have been able to uncover patterns in the reasoning process that students used when solving multiple-choice
questions. A better understanding of students’ decision-making processes can help us educators redesign our
assessment practices and offer better support for student learning.
USING VISUAL MAPPING AND DIALOGUE TO EXPAND CONCEPT ELABORATION
The second case study used a qualitative paradigm involving three forms of data collection not commonly
combined. The scenario was dialogue about a thesis topic among a graduate student, the thesis supervisor, and a
research librarian. Visualization in the form of a mind map was used by the student to track the dynamic evolution
of key ideas, concepts, and their inter-relationships. In tandem with the map construction, collaborative dialogue was
used to unravel and clarify ideas. Sessions for five students (one male, five female) were videotaped and a survey
was later distributed to gather feedback on the process.
This research emerged from the authors’ practical experience of supporting graduate research at a Faculty of
Education. One author is a librarian who assists graduate students in locating research materials. The other is a
professor who supervises graduate theses using ongoing meetings to discuss research questions, methodology, and

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
to set goals. The impetus for the study resulted from the librarian’s experiences in research consultations using a
mind map to record and clarify ideas, concepts, and resources. Students consistently requested their map for future
reference triggering curiosity about the value of the interaction and the resulting visual aid. Both authors are
instructors and the overarching goal for the study was to understand how to better support student information
gathering and subsequent clarification of research topics. The research literature documents the need to support
graduate information search processes (Delaney & Bates, 2018; Spezi, 2016; Catalano, 2013) and the mapping
approach appeared to be a strategic tool to facilitate negotiating and sharing knowledge associated with these
challenges (Hay et al, 2008). Drawing upon these experiences we sought to examine how visualization and dialogue
interact to shape and extend student research topics.
• How does visual mapping and collaborative dialogue help students clarify and extend conceptions of their
research?
• What aspects of verbal support enable students to extend their thinking during collaborative dialogue?
Research methods were selected on the basis of their similarity to authentic interactions between a graduate
student, librarian, and thesis supervisor, and to mimic the context in which student-researcher interactions typically
took place. Rather than adopt an approach providing reflection on a past event, such as a formal interview, the
researchers wanted to capture the human and learning dynamics taking place in situ. We intended this approach to
promote open and unscripted dialogue with the student enabling them to take the lead in setting conversation
directions. Consequently the study took place in the familiar setting of the library and combined the mind map and
conversational style of both the librarian and supervisor.
Three forms of data collection were combined:
• Students independently created a mind map of their thesis topic using a concurrent think-aloud protocol
(audio and videotaped with visual map)
• Collaborative dialogue between the student and researchers with continued student construction of the mind
map (audio and videotaped with visual map)
• Follow-up survey of students’ perceptions of the exercise
Selection of Research Methods
Our study combined a mind map, participant-researcher dialogue, video capture, and a survey. It was the first time
we used the mind map and collaborative dialogue as data capture tools. The next sections of this paper outline our

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
reasons for choosing these methods and link them to the literature describing their functionality as techniques that
help to make learning visible.
Visualization as Research Method
Capturing data visually was a key consideration in our study. Visual images are used in qualitative research to
gather or interpret data and can be pre-existent or generated by the researcher and/or participant during an interview
(Drew & Guillemin, 2014; Renfro, 2017). They can take the form of diagrams, video, drawings, photographs, and
maps (Weber, 2008). While interviews provide an opportunity to explore and probe themes of interest as they arise,
they rely on mutual and immediate understanding and interpretation. As exchanges take place, previously discussed
ideas recede while others claim a focus in the conversation. Given our limited capacity to recall aural information, it
may be difficult to remember what was previously discussed and how it relates to the topic at hand. Listeners may
not be aware of individual frameworks for prior knowledge and expert understanding nor do they have a way to
illustrate shared understanding as it develops. Recording ideas visually offers a mechanism to make ideas
transparent and to capture conversation over time, thereby enabling better communication, shared understanding,
and depiction of data (Glegg, 2018). The English idiom “a picture is worth a thousand words” captures the essence
of this idea: complex thoughts can be translated into essential ideas through images.
Psychologists of cognition represent information in different forms (e.g., illustrations, diagrams, flow-charts,
concept maps) to examine how they affect learning (Nesbit & Adesope, 2006). In this study, we used a free form of
visual mapping (mind map) as the students were at an early stage in developing their research proposal. Traditional
concept maps are difficult to make because they necessitate the identification and naming of ideas and their multiple
relationships within a hierarchical structure that applies formal rules (Eppler, 2006). In his comparison of concept
maps, mind maps, conceptual diagrams, and visual metaphors he ranked mind maps as “easy to learn and apply”
versus concepts maps “not easy to apply by novices; requires extensive training” (Eppler, 2006, p. 206). Mind maps
allow a freer form of visual expression in which ideas and relationships are evolving and shifting. As a data
collection tool mind maps provided an objective visual and permanent representation of the dynamic evolution of a
student’s thoughts and served as a focus for the collaborative dialogue. When individuals are asked to draw maps on
the same topic at different stages of learning, changes in learning can be made visible (Hay, 2007). The collaborative

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
construction of this type of diagram reveals both the underlying structure of a topic alongside deeper understandings
captured during the idea elaboration process (Adesope, & Nesbit, 2010).
Collaborative Dialogue
The use of multiple tools for data collection allowed us to better answer our research questions as each tool had
benefits and limitations. Dialogue is auditory, impermanent, and linear while mapping is visual, permanent and non-
linear, while video-taping captured the changes in both. We viewed unstructured dialogue as a form of collaboration
that “is the result of a continued attempt to construct and maintain a shared conception of a problem” (Roschelle &
Teasley, 1995, p. 70). Although our scenario was naturalistic, the synchronous interaction did require intensive
attention (Baker, 2015). One very pertinent observation during our dialogue was an awareness of Vygotsky’s zone
of proximal development (ZPD) in action (1978). Vygotsky focused on the role of social interaction and the use of
cultural tools (e.g. language, diagrams, images) in the learning process. The ZPD refers to the learning that takes
place when students interact with and receive support from partners who are more knowledgeable in an area. As the
map changed during conversation, we observed knowledge being actively constructed first hand. The map
evidenced clarification of new ideas and enhanced understanding of the inter-relationships among them through the
extended focused discussion as outlined by Murphy et al. (2009). The reiterative mapping process encouraged
revision of ideas and interaction among participants revealing the underlying topic structure and subsequent deeper
understanding and co-construction of knowledge, as documented by other researchers (Basque & Pudelko, 2010;
Bereiter & Scardamalia, 2012; Wells, 1999, 2002). Given that each interview lasted from 1-1.5 hours, it was
important to remain focused on developing ideas without trying to solve particular issues such as decisions about
research methods. This meant that researchers had to remain focused on the task at hand and acknowledge when the
conversation was moving beyond its purpose, such as making concrete decisions for preferred research methods.
Visualization and Dialogue in Combination
Visualization combined with dialogue offers a dual approach to communicating shared understanding. Individuals’
navigation of the external world is through the information gathered by their perceptual systems (Kellman,
2002). Dual coding theory (Pavio, 1986) argues that when visual and auditory stimuli are perceived simultaneously
it is encoded in both the auditory and visual cortices and these are linked (Craik & Lockhart, 1972; Pavio, 1986).
Perceptual information is processed in working memory which has a limited capacity for the number of items that

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
can be kept active at one time, that is, the cognitive load (Sweller, 2011). This limitation means that only some of
the information is transferred from working memory to long-term memory, the rest being discarded. Information
that is processed together results in multiple pathways for long-term memory storage and retrieval, hence the benefit
of dual coding (Baddeley, 2000; Pavio, 1986). In generating a mind map, an individual retrieves stored information
from long-term memory and creates a visual representation of their conceptual understanding. The map transfers
some of the cognitive load from working memory, freeing capacity in working memory for the continual retrieval of
other aspects of a complex concept (Baddeley, 2000). The collaborative dialogue functioned in tandem with the map
creation to enable the student to return to past ideas, connect new with old, and draw relationships among them as
well.
Concurrent Think- Aloud as Research Method
As described previously, concurrent think-alouds invite students to articulate whatever comes to mind as they carry
out a task (Afflerbach, 2000; Ericsson & Simon, 1980). This allows greater insight into individuals’ active ongoing
cognitive processes, that is, the content of their working memory in comparison to a summary after completing the
task (retrospective think-aloud). Our study traced student prior knowledge and depiction of a topic and followed its
elaboration during conversation. To differentiate between the two stages, we should have asked that a single black
marker be used to illustrate the first map stage and then required other colours to express new ideas. This would
have made it easier to distinguish between the independent map versus the collaborative one. Fortunately, the
transition was captured on videotape. Further discussion about the purpose of a mind map and practice in creating
one using an everyday topic would also have helped as some students had little recent experience with the technique.
Analysis and Interpretation of the Mind Maps
The data collection methods appeared to be straightforward at the outset. What was unanticipated was how difficult
it would be to code and then integrate three distinct data sets: transcripts of the dialogue, maps of the ideas, and the
video of the process as it transpired which revealed moments when a particular aspect of the dialogue caused the
student to make a change to the mind map.
Four formats of data were collected for each of the five students (audio transcript, mind map (Figure 1 provides an
example of a before and after map), videotape, and survey). The analysis was completed in stages. The audio of the
student think-aloud and follow-up collaborative dialogue was transcribed verbatim. Transcripts averaged 30 pages

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
and were uploaded into NVivo (Version 11, 2015) for thematic analysis. The intention was to openly code the
dialogue for significant categories and emerging patterns (Heath & Cowley, 2004; Glaser & Strauss, 1967).
However, initial attempts to code the transcripts did not provide a coherent picture of the data. Each transcript was
unique and addressed a different topic. It was difficult to determine what was relevant in the absence of seeing
where a change was made to the map. In retrospect, what was missing was a framework for coding. We
inadvertently came to this later in the analysis process. Consequently, we abandoned NVivo and switched to
independently hand-coding transcripts in an attempt to identify emerging themes. This did not generate a coherent
picture but we could see that what transcripts had in common were discussions around different aspects of the
research process. We then attempted a quantitative analysis of the independently generated map in comparison to
the expanded map generated by counting map nodes and sub-nodes. However, given the complexity of some of the
maps (Figure 1), this proved to be a dead end because it didn’t inform our research questions regarding how the
dialogue and visual mapping enabled clarification of the research topic. In an aha moment, we realized that because
we had five separate cases in which each student described a unique topic, it was difficult to find common themes
across disparate content. We had failed to consider the interaction between the dialogue and the map as captured on
video revealing student-initiated changes in response to the dialogue. This led to analysis of the videotapes. We
viewed each videotape and marked on the transcript whenever a change to the map was made. Then we
independently examined and highlighted each segment of dialogue that prompted the change. Collaboratively, we
discussed the segments that elicited visual additions and categorized the types of dialogue that led them. These were
referred to as “prompts”. Further re-reading and comparison enabled definitions for two types of prompts: clarifying
and knowledge prompts. Clarifying prompts occurred when researchers asked questions to unravel the verbal
description or visual representation given by the student. Knowledge prompts occurred when researchers offered
information to extend student thinking or analysis. This proved to be the level of analysis needed to produce a
meaningful picture of the data.
Following analysis of the prompts, we returned to examining the data for commonalities across all cases. We
identified the need for a coding framework that could be applied across transcripts. In stepping back from the
detailed analysis, we decided to apply a standard social sciences model of research stages (Berg, 2001) as each case
had this in common. Each researcher independently identified segments of the dialogue by the stages of the research
process. Codes were compared resulting in three themes: defining the research question, information gathering and

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
evaluation as part of the literature review, and study methodology. This employed a deductive model of analysis
using a template method (Blair, 2015).
Figure 1. Student-generated maps. The left map was created independently by the student and the right map is the
same map with additions made by the student at the close of the collaborative dialogue.
Reflection on the Methods: Challenges and Benefits
The benefits of our methods were also the challenges. We were inspired by the notion of capturing learning as it
happens but were unprepared for the complexity of analysis.
Each student brought their own style to the mapping exercise and used a free form of visual representation. The
dialogue was responsive to the student’s map and the resulting ongoing changes to the map by the student. As we

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
did not follow a script, each transcript was unique with different prompts and varying amounts of time spent on
different aspects of the map. While this provided freedom and choice to the student, it resulted in greater difficulty
in analysis. The first challenge was working with the different data formats: printed transcript, hand-drawn mind
map, live-action video, and survey. Next there was the need to code and analyze them as an integrated whole which
required multiple iterations and approaches. With five distinct cases, we also needed to apply a common framework
which was the research process model. Getting beyond the surface level representation in the mind maps was
difficult and likely is why NVivo and the initial thematic coding attempts were not successful.
Multiple forms of data collection allowed for triangulation in data analysis. The dialogue prompted changes to the
map and these were captured both visually and over time on the video enabling us to identify trigger points for
student conceptual understanding; the survey data gave student perceptions of the process. Using an open-ended
approach, mind maps and unscripted collaborative dialogue afforded each student the opportunity to represent their
understanding and conceptualization of the thesis topic. Collaborative dialogue that followed the student’s map and
think-aloud provided a very rich dataset. Students were very engaged in the process because they generated the map
that was the focal point of the collaborative dialogue. They were in control of the process, responding or not as they
wished, to the prompts from the researchers that had been elicited by their map. The visual additions to the maps
enabled us to capture the moments that had meaning to the students. Having five distinct cases provided unique
perspectives on how students’ conceptualize the research process. The extensive reiterative re-coding and
consequent familiarity with the data eventually allowed us to perceive deeper levels of commonality across
seemingly disparate cases. In response to the follow-up surveys, students were very positive stating that the
visualization and dialogue enabled them to develop and deepen conceptualization of their topics.
Conclusions
This chapter focused on qualitative methods that helped make learning visible for both researchers and participants.
In the first case, think-alouds allowed students to deconstruct their problem-solving strategies by articulating their
thought processes in action. This enabled the researchers to identify types of strategies that aligned with different
levels of sophistication in the multiple-choice questions. It also exposed cognitive biases, assumptions, and attitudes
towards multiple-choice questions. In the second case, visualization offered study participants an opportunity to
externalize some of their thoughts reducing cognitive load. The mind map served as a truncated form of expression
offering a snapshot and consolidation of ideas using single words and phrases, unlike oral exchange in typical

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES
sentence format. Mind maps build an overarching picture of a multi-faceted concept and reveal elaboration of ideas
over time. Non-scripted collaborative dialogue changes the power dynamic from a typical interview making the
student a conversation partner who directs discussion and consequently remains more engaged with the task.
The choice of data collection methodologies determined the analysis process and is a reminder that we keep the end
in mind when selecting which forms of evidence are gathered. Researchers must anticipate how that evidence will
inform their specific research questions. Research is an iterative process and this critical reflection on our work has
similarly sparked further insights into our choice of methodologies.
References
Adesope, O. O., & Nesbit, J. C. (2010). A systematic review of research on collaborative learning with concept maps. In R. de Cassia Veiga
Marriott & P. Lupion Torres, (Eds.), Handbook of research on collaborative learning using concept mapping (pp. 238-251). Hershey, NY:
Information Science Reference.
Afflerbach, P. (2000). Verbal reports and protocol analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr. (Eds.), Handbook of
reading research, Vol. Ill, (pp. 171-71). Mahwah, NJ: Lawrence Erlbaum.
Baddeley, A. (2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4(11), 417-423.
doi.org/10.1016/S1364-6613(00)01538-2
Baker, M. J. (2015). Collaboration in collaborative learning. Interaction Studies: Social Behaviour and Communication in Biological and
Artificial Systems, 16(3), 451-473.
Basque, J., & Pudelko, B. (2010). lntersubjective meaning-mapping in dyads
using object-typed concept mapping. In R. de Cassia Veiga Marriott & P. Lupion Torres, (Eds.), Handbook of research on collaborative
learning using concept mapping (pp. 187-209). Hershey, PA: Information Science Reference.
Bereiter, C., & Scardamalia, M. (2012). Theory building and the pursuit of understanding in history, social studies, and literature. In J. Kirby &
M. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction and learning (pp. 160-177). Cambridge, UK: Cambridge
University Press.
Berg, B. L. (2001). Qualitative research methods for the social sciences. 4th ed. Boston, MA:
Allyn and Bacon.
Blair, E. (2015). A reflexive exploration of two qualitative data coding techniques. Journal of Methods and Measurement in the Social Sciences,
6(1), 14-29. doi:10.2458/azu_jmmss_v6i1_blair
Catalano, A. (2013). Patterns of graduate students' information seeking behavior: A meta-synthesis of the literature. Journal of Documentation,
69(2), 243-274. doi.org/10.1108/00220411311300066
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: Sage.
Craik, F., & Lockhart, R. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior,
11(6), 671-684. doi.org/10.1016/S0022-5371(72)80001-X
Delaney, G., & Bates, J. (2018). How can the university library better meet the information needs of research students? Experiences from Ulster
University. New Review of Academic Librarianship, 24(1), 63-89. doi: 10.1080/13614533.2017.1384267
Drew, S., & Guillemin, M. (2014). From photographs to findings: Visual meaning-making and interpretive engagement in the analysis of
participant-generated images. Visual Studies, 29(1), 54-67. doi: 10.1080/1472586X.2014.862994
Durning, S. J., Artino, A. R., Beckman, T. J., Graner, J., van der Vleuten, C., Holmboe, E., & Schuwirth, L. (2013). Does the think-aloud protocol
reflect thinking? Exploring functional neuroimaging differences with thinking (answering multiple choice) versus thinking aloud. Medical
Teacher, 35(9), 720-726.
Eppler, M. J. (2006). A comparison between concept maps, mind maps, conceptual diagrams, and visual metaphors as complementary tools for
knowledge construction and sharing. Information Visualization, 5(3), 202-210. doi.org/10.1057/palgrave.ivs.9500131
Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.

KLODIANA KOLOMITRO, CORINNE LAVERTY, ELIZABETH LEE
Ericsson, K. A., & Simon, H. (1980). Verbal reports as data. Psychological Review, 87(3), 215-251. doi.org/10.1037/0033-295X.87.3.215
Fostaty Young, S., & Wilson, R. J. (2000). Assessment and learning: The ICE approach. Winnipeg, MB: Portage & Main Press.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New Brunswick, NJ: Aldine
Transaction.
Glegg, S. M. (2019). Facilitating interviews in qualitative research with visual tools: A typology. Qualitative Health Research, 29(2), 301-310.
doi:1049732318786485
Hay, D. B. (2007). Using concept maps to measure deep, surface and non learning outcomes. Studies in Higher Education, 32(1), 39-57.
doi:10.1080/03075070601099432.
Hay, D., Kinchin, I., & Lygo‐Baker, S. (2008). Making learning visible: The role of concept mapping in higher education. Studies in Higher
Education, 33(3), 295-311. doi:10.1080/03075070802049251.
Heath, H., & Cowley, S. (2004). Developing a grounded theory approach: a comparison of Glaser and Strauss. International Journal of Nursing
Studies, 41(2), 141-150.
https://doi.org/10.1016/S0020-7489(03)00113-5
Kellman, P. J. (2002). Perceptual learning. In H. Pashler & R. Gallistel (Eds.), Stevens’ handbook of experimental psychology. 3: Learning,
motivation, and emotion (3rd ed.). New York: Wiley. doi:10.1002/0471214426.pas0307.
Murphy, P. K., Wilkinson, I. A. G., Soter, A. O., Hennessey, M. N., & Alexander, J. F. (2009). Examining the effects of classroom discussion on
students’ comprehension of text: A meta-analysis. Journal of Educational Psychology, 101(3), 740-764.
doi.org/10.1037/a0015576
Nesbit, J. C., & Adesope, O. O. (2006). Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research, 76, 413-
448. doi:10.3102/00346543076003413
Patton, M. Q. (1990). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage.
Pavio, A. (1986). Mental representation: A dual coding approach. Oxford, England: Oxford University Press.
Renfro, C. (2017). The use of visual tools in the academic research process: A literature review. The Journal of Academic Librarianship, 43(2),
95-99. doi.org/10.1016/j.acalib.2017.02.004
Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.),
computer supported collaborative learning (pp. 69-97). Berlin: Springer Verlag.
Spezi, V. (2016). Is information-seeking behavior of doctoral students changing? A review of the literature (2010–2015). New Review of
Academic Librarianship, 22(1), 78-106. doi:1080/13614533.2015.1127831
Sweller, J. (2011). Cognitive load theory. In J. P. Mestre, & B. H. Ross (Eds.), Psychology of learning and motivation Vol. 55 (pp. 37-76).
Elsevier. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Vygotsky, L.V. (1978). Mind and psychology. Cambridge MA: Harvard University Press.
Weber, S. (2008). Visual images in research. In J. G. Knowles & A. L. Cole (Eds.), Handbook of the arts in qualitative research: Perspectives,
methodologies, examples, and issues. (pp. 42-54). London: Sage.
Wells, G. (1999). Dialogic inquiry in education: Building on the legacy of Vygotsky. In C. D. Lee & P. Smagorinsky (Eds.), Vygotskian
perspectives on literacy research (pp.51-85). New York: Cambridge University Press.
Wells, G. (2002). Dialogue about knowledge building. In B. Smith, (Ed.), Liberal education in a knowledge society (pp.111-138). Chicago: Open
Court.
Klodiana Kolomitro
Centre for Teaching and Learning
Queen’s University
Corinne Laverty
Centre for Teaching and Learning
Queen’s University
Elizabeth Lee
Faculty of Education (PhD retired)
Queen’s University

MAKING LEARNING VISIBLE: RESEARCH METHODS TO UNCOVER LEARNING PROCESSES